Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[parth brahmbhatt <br...@gmail.com>]

Bumping this up one more time, can other committers review?

Thanks
Parth

On Tue, Apr 26, 2016 at 9:07 AM, Harsha <ka...@harsha.io> wrote:

> Parth,
>           Overall current design looks good to me. I am +1 on the KIP.
>
> Gwen , Jun can you review this as well.
>
> -Harsha
>
> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote:
> > Thanks for review Jitendra.
> >
> > I don't like the idea of infinite lifetime but I see the Streaming use
> > case. Even for Streaming use case I was hoping there will be some notion
> > of
> > master/driver that can get new delegation tokens at fixed interval and
> > distribute to workers. If that is not the case for we can discuss
> > delegation tokens renewing them self and the security implications of the
> > same.
> >
> > I did not want clients to fetch tokens from zookeeper, overall I think
> > its
> > better if clients don't rely on our metadata store and I think we are
> > moving in that direction with all the KIP-4 improvements.  I chose
> > zookeeper as in this case the client will still just talk to broker , its
> > the brokers that will use zookeeper which we already do for a lot of
> > other
> > usecases + ease of development + and the ability so tokens will survive
> > even a rolling restart/cluster failure. if a majority agrees the added
> > complexity to have controller forwarding keys to all broker is justified
> > as
> > it provides tighter security , I am fine with that option too.
> >
> > Given zookeeper does not support SSL we can not store master keys in
> > zookeeper as master keys will be exposed on wire. To support rotation
> > without affecting current clients is something I need to put more thought
> > in. My current proposal assumes the rotation will invalidate all current
> > tokens.
> >
> > I request committers to also review and post their comments so we can
> > make
> > progress on this KIP.
> >
> > Thanks
> > Parth
> >
> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <as...@cloudera.com>
> > wrote:
> >
> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <ka...@harsha.io> wrote:
> > >
> > > > Unifying the two discussion threads on this KIP.
> > > >
> > > > Here is the response from Jitendra
> > > >
> > > > "The need for a large number of clients that are running all over the
> > > > cluster that authenticate with Kafka brokers, is very similar to the
> > > > Hadoop use case of large number of tasks running across the cluster
> that
> > > > need authentication to Hdfs Namenode. Therefore, the delegation token
> > > > approach does seem like a good fit for this use case as we have seen
> it
> > > > working at large scale in HDFS and YARN.
> > > >
> > > >   The proposed design is very much inline with Hadoop approach. A few
> > > >   comments:
> > > >
> > > > 1) Why do you guys want to allow infinite renewable lifetime for a
> > > > token? HDFS restricts a token to a max life time (default 7 days).  A
> > > > token's vulnerability is believed to increase with time.
> > > >
> > > I agree that having infinite lifetime might not be the best idea.
> > >
> > > >
> > > > 2) As I understand the tokens are stored in zookeeper as well, and
> can
> > > > be updated there. This is clever as it can allow replacing the tokens
> > > > once they run out of max life time, and clients can download new
> tokens
> > > > from zookeeper. It shouldn't be a big load on zookeeper as a client
> will
> > > > need to get a new token once in several days. In this approach you
> don't
> > > > need infinite lifetime on the token even for long running clients.
> > > >
> > > > 3) The token password are generated using a master key. The master
> key
> > > > should also be periodically changed. In Hadoop, the default renewal
> > > > period is 1 day.?
> > > >
> > > IIUC, this will require brokers maintaining a list of X most recent
> master
> > > keys. This list will have to be persisted somewhere, as if a broker
> goes
> > > down it will have to get that list again and storing master keys on ZK
> is
> > > not the best idea. However, if a broker goes down then we have much
> bigger
> > > issue to deal with and client can always re-authenticate is such
> events.
> > >
> > > Did you happen to take a look at other alternatives this list has
> > > suggested?
> > >
> > > >
> > > > Thanks for a thorough proposal, great work!"
> > > >
> > > >
> > > >
> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
> > > > > Makes sense to me. Thanks!
> > > > >
> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <ka...@harsha.io> wrote:
> > > > > > It doesn't need any release vehicle but still the work can move
> > > > forward.
> > > > > > If anyone is interested in the KIP please do the review and
> provide
> > > the
> > > > > > comments.
> > > > > >
> > > > > > -Harsha
> > > > > >
> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma wrote:
> > > > > >> I agree that it would be good to have more time to review and
> > > discuss
> > > > > >> KIP-48.
> > > > > >>
> > > > > >> Ismael
> > > > > >>
> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <
> gwen@confluent.io>
> > > > wrote:
> > > > > >>
> > > > > >> > Hi Team,
> > > > > >> >
> > > > > >> > Since KIP-48 depends on KIP-43, which is already a bit of a
> risk
> > > for
> > > > > >> > the next release - any chance we can delay delegation tokens
> to
> > > > Kafka
> > > > > >> > 0.10.1?
> > > > > >> > With the community working on a release every 3 month, this
> is not
> > > > a huge
> > > > > >> > delay.
> > > > > >> >
> > > > > >> > Gwen
> > > > > >> >
> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh <
> > > asingh@cloudera.com>
> > > > wrote:
> > > > > >> > > Parth,
> > > > > >> > >
> > > > > >> > > Thanks again for the awesome write up. Following our
> discussion
> > > > from the
> > > > > >> > > JIRA, I think it will be easier to compare various
> alternatives
> > > > if they
> > > > > >> > are
> > > > > >> > > listed together. I am stating below a few alternatives along
> > > with
> > > > a the
> > > > > >> > > current proposal.
> > > > > >> > > (Current proposal) Store Delegation Token, DT, on ZK.
> > > > > >> > >
> > > > > >> > >    1. Client authenticates with a broker.
> > > > > >> > >    2. Once a client is authenticated, it will make a broker
> side
> > > > call to
> > > > > >> > >    issue a delegation token.
> > > > > >> > >    3. The broker generates a shared secret based on
> > > HMAC-SHA256(a
> > > > > >> > >    Password/Secret shared between all brokers, randomly
> > > generated
> > > > > >> > tokenId).
> > > > > >> > >    4. Broker stores this token in its in memory cache.
> Broker
> > > > also stores
> > > > > >> > >    the DelegationToken without the hmac in the zookeeper.
> > > > > >> > >    5. All brokers will have a cache backed by zookeeper so
> they
> > > > will all
> > > > > >> > >    get notified whenever a new token is generated and they
> will
> > > > update
> > > > > >> > their
> > > > > >> > >    local cache whenever token state changes.
> > > > > >> > >    6. Broker returns the token to Client.
> > > > > >> > >
> > > > > >> > > Probable issues and fixes
> > > > > >> > >
> > > > > >> > >    1. Probable race condition, client tries to authenticate
> with
> > > > a broker
> > > > > >> > >    that is yet to be updated with the newly generated DT.
> This
> > > can
> > > > > >> > probably be
> > > > > >> > >    dealt with making dtRequest block until all brokers have
> > > > updated
> > > > > >> > their DT
> > > > > >> > >    cache. Zk barrier or similar mechanism can be used.
> However,
> > > > all such
> > > > > >> > >    mechanisms will increase complexity.
> > > > > >> > >    2. Using a static secret key from config file. Will
> require
> > > yet
> > > > > >> > another
> > > > > >> > >    config and uses a static secret key. It is advised to
> rotate
> > > > secret
> > > > > >> > keys
> > > > > >> > >    periodically. This can be avoided with controller
> generating
> > > > > >> > secretKey and
> > > > > >> > >    passing to brokers periodically. However, this will
> require
> > > > brokers to
> > > > > >> > >    maintain certain counts of secretKeys.
> > > > > >> > >
> > > > > >> > > (Alternative 1) Have controller generate delegation token.
> > > > > >> > >
> > > > > >> > >    1. Client authenticates with a broker.
> > > > > >> > >    2. Once a client is authenticated, it will make a broker
> side
> > > > call to
> > > > > >> > >    issue a delegation token.
> > > > > >> > >    3. Broker forwards the request to controller.
> > > > > >> > >    4. Controller generates a DT and broadcasts to all
> brokers.
> > > > > >> > >    5. Broker stores this token in its memory cache.
> > > > > >> > >    6. Controller responds to broker’s DT req.
> > > > > >> > >    7. Broker returns the token to Client.
> > > > > >> > >
> > > > > >> > > Probable issues and fixes
> > > > > >> > >
> > > > > >> > >    1. We will have to add new APIs to support controller
> pushing
> > > > tokens
> > > > > >> > to
> > > > > >> > >    brokers on top of the minimal APIs that are currently
> > > proposed.
> > > > > >> > >    2. We will also have to add APIs to support the
> bootstrapping
> > > > case,
> > > > > >> > i.e,
> > > > > >> > >    when a new broker comes up it will have to get all
> delegation
> > > > tokens
> > > > > >> > from
> > > > > >> > >    the controller.
> > > > > >> > >    3. In catastrophic failures where all brokers go down,
> the
> > > > tokens will
> > > > > >> > >    be lost even if servers are restarted as tokens are not
> > > > persisted
> > > > > >> > anywhere.
> > > > > >> > >    If this happens, then there are more important things to
> > > worry
> > > > about
> > > > > >> > and
> > > > > >> > >    maybe it is better to re-authenticate.
> > > > > >> > >
> > > > > >> > > (Alternative 2) Do not distribute DT to other brokers at
> all.
> > > > > >> > >
> > > > > >> > >    1. Client authenticates with a broker.
> > > > > >> > >    2. Once a client is authenticated, it will make a broker
> side
> > > > call to
> > > > > >> > >    issue a delegation token.
> > > > > >> > >    3. The broker generates DT of form, [hmac + (owner,
> renewer,
> > > > > >> > >    maxLifeTime, id, hmac, expirationTime)] and passes back
> this
> > > > DT to
> > > > > >> > client.
> > > > > >> > >    hmac is generated via {HMAC-SHA256(owner, renewer,
> > > > maxLifeTime, id,
> > > > > >> > hmac,
> > > > > >> > >    expirationTime) using SecretKey}. Note that all brokers
> have
> > > > this
> > > > > >> > SecretKey.
> > > > > >> > >    4. Client then goes to any broker and to authenticate
> sends
> > > > the DT.
> > > > > >> > >    Broker recalculates hmac using (owner, renewer,
> maxLifeTime,
> > > > id, hmac,
> > > > > >> > >    expirationTime) info from DT and its SecretKey. If it
> matches
> > > > with
> > > > > >> > hmac of
> > > > > >> > >    DT, client is authenticated. Yes, it will do other
> obvious
> > > > checks of
> > > > > >> > >    timestamp expiry and such.
> > > > > >> > >
> > > > > >> > > Note that secret key will be generated by controller and
> passed
> > > to
> > > > > >> > brokers
> > > > > >> > > periodically.
> > > > > >> > > Probable issues and fixes
> > > > > >> > >
> > > > > >> > >    1. How to delete a DT? Yes, that is a downside here.
> However,
> > > > this can
> > > > > >> > >    be handled with brokers maintaining a blacklist of DTs,
> DTs
> > > > from this
> > > > > >> > list
> > > > > >> > >    can be removed after expiry.
> > > > > >> > >    2. In catastrophic failures where all brokers go down,
> the
> > > > tokens will
> > > > > >> > >    be lost even if servers are restarted as tokens are not
> > > > persisted
> > > > > >> > anywhere.
> > > > > >> > >    If this happens, then there are more important things to
> > > worry
> > > > about
> > > > > >> > and
> > > > > >> > >    maybe it is better to re-authenticate.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth Brahmbhatt <
> > > > > >> > > pbrahmbhatt@hortonworks.com> wrote:
> > > > > >> > >
> > > > > >> > >> Hi,
> > > > > >> > >>
> > > > > >> > >> I have filed KIP-48 so we can offer hadoop like delegation
> > > > tokens in
> > > > > >> > >> kafka. You can review the design
> > > > > >> > >>
> > > > > >> >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > > > > >> > .
> > > > > >> > >> This KIP depends on KIP-43 and we have also discussed an
> > > > alternative to
> > > > > >> > >> proposed design here<
> > > > > >> > >>
> > > > > >> >
> > > >
> > >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > > > > >> > >> >.
> > > > > >> > >>
> > > > > >> > >> Thanks
> > > > > >> > >> Parth
> > > > > >> > >>
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > --
> > > > > >> > >
> > > > > >> > > Regards,
> > > > > >> > > Ashish
> > > > > >> >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Regards,
> > > Ashish
> > >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Becket Qin <be...@gmail.com>]

According to the meeting minutes of KIP hangout on 8/30, it seems the KIP
wiki needs some update?

KIP48 (delegation tokens): Harsha will update the wiki with more details on
how to use delegation tokens and how to configure it.

Not sure if that has been done or not.

On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <gw...@confluent.io> wrote:

> Hi Guys,
>
> This discussion was dead for a while. Are there still contentious
> points? If not, why are there no votes?
>
> On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <ju...@confluent.io> wrote:
> > Ashish,
> >
> > Yes, I will send out a KIP invite for next week to discuss KIP-48 and
> other
> > remaining KIPs.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh <as...@cloudera.com>
> wrote:
> >
> >> Thanks Harsha!
> >>
> >> Jun, can we add KIP-48 to next KIP hangout's agenda. Also, we did not
> >> actually make a call on when we should have next KIP call. As there are
> a
> >> few outstanding KIPs that could not be discussed this week, can we have
> a
> >> KIP hangout call next week?
> >>
> >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha Chintalapani <ka...@harsha.io>
> >> wrote:
> >>
> >>> Ashish,
> >>>         Yes we are working on it. Lets discuss in the next KIP meeting.
> >>> I'll join.
> >>> -Harsha
> >>>
> >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh <as...@cloudera.com>
> >>> wrote:
> >>>
> >>> > Hello Harsha,
> >>> >
> >>> > Are you still working on this? Wondering if we can discuss this in
> next
> >>> KIP
> >>> > meeting, if you can join.
> >>> >
> >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha Chintalapani <
> kafka@harsha.io>
> >>> > wrote:
> >>> >
> >>> > > Hi Grant,
> >>> > >           We are working on it. Will add the details to KIP about
> the
> >>> > > request protocol.
> >>> > >
> >>> > > Thanks,
> >>> > > Harsha
> >>> > >
> >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant Henke <gh...@cloudera.com>
> >>> wrote:
> >>> > >
> >>> > > > Hi Parth,
> >>> > > >
> >>> > > > Are you still working on this? If you need any help please don't
> >>> > hesitate
> >>> > > > to ask.
> >>> > > >
> >>> > > > Thanks,
> >>> > > > Grant
> >>> > > >
> >>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <ju...@confluent.io>
> wrote:
> >>> > > >
> >>> > > > > Parth,
> >>> > > > >
> >>> > > > > Thanks for the reply.
> >>> > > > >
> >>> > > > > It makes sense to only allow the renewal by users that
> >>> authenticated
> >>> > > > using
> >>> > > > > *non* delegation token mechanism. Then, should we make the
> >>> renewal a
> >>> > > > list?
> >>> > > > > For example, in the case of rest proxy, it will be useful for
> >>> every
> >>> > > > > instance of rest proxy to be able to renew the tokens.
> >>> > > > >
> >>> > > > > It would be clearer if we can document the request protocol
> like
> >>> > > > >
> >>> > > > >
> >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> > > 4+-+Command+line+and+centralized+administrative+operations#KIP-4-
> >>> > > Commandlineandcentralizedadministrativeoperations-
> >>> > > CreateTopicsRequest(KAFKA-2945):(VotedandPlannedforin0.10.1.0)
> >>> > > > > .
> >>> > > > >
> >>> > > > > It would also be useful to document the client APIs.
> >>> > > > >
> >>> > > > > Thanks,
> >>> > > > >
> >>> > > > > Jun
> >>> > > > >
> >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM, parth brahmbhatt <
> >>> > > > > brahmbhatt.parth@gmail.com> wrote:
> >>> > > > >
> >>> > > > > > Hi,
> >>> > > > > >
> >>> > > > > > I am suggesting that we will only allow the renewal by users
> >>> that
> >>> > > > > > authenticated using *non* delegation token mechanism. For
> >>> example,
> >>> > If
> >>> > > > > user
> >>> > > > > > Alice authenticated using kerberos and requested delegation
> >>> tokens,
> >>> > > > only
> >>> > > > > > user Alice authenticated via non delegation token mechanism
> can
> >>> > > renew.
> >>> > > > > > Clients that have  access to delegation tokens can not issue
> >>> > renewal
> >>> > > > > > request for renewing their own token and this is primarily
> >>> > important
> >>> > > to
> >>> > > > > > reduce the time window for which a compromised token will be
> >>> valid.
> >>> > > > > >
> >>> > > > > > To clarify, Yes any authenticated user can request delegation
> >>> > tokens
> >>> > > > but
> >>> > > > > > even here I would recommend to avoid creating a chain where a
> >>> > client
> >>> > > > > > authenticated via delegation token request for more
> delegation
> >>> > > tokens.
> >>> > > > > > Basically anyone can request delegation token, as long as
> they
> >>> > > > > authenticate
> >>> > > > > > via a non delegation token mechanism.
> >>> > > > > >
> >>> > > > > > Aren't classes listed here
> >>> > > > > > <
> >>> > > > > >
> >>> > > > >
> >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> > > 48+Delegation+token+support+for+Kafka#KIP-48Delegationtokens
> >>> upportforKaf
> >>> > > ka-PublicInterfaces
> >>> > > > > > >
> >>> > > > > > sufficient?
> >>> > > > > >
> >>> > > > > > Thanks
> >>> > > > > > Parth
> >>> > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao <ju...@confluent.io>
> >>> wrote:
> >>> > > > > >
> >>> > > > > > > Parth,
> >>> > > > > > >
> >>> > > > > > > Thanks for the reply. A couple of comments inline below.
> >>> > > > > > >
> >>> > > > > > > On Tue, Jun 21, 2016 at 10:36 AM, parth brahmbhatt <
> >>> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> >>> > > > > > >
> >>> > > > > > > > 1. Who / how are tokens renewed? By original requester
> >>> only? or
> >>> > > > using
> >>> > > > > > > > Kerberos
> >>> > > > > > > > auth only?
> >>> > > > > > > > My recommendation is to do this only using Kerberos auth
> and
> >>> > only
> >>> > > > > threw
> >>> > > > > > > the
> >>> > > > > > > > renewer specified during the acquisition request.
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > Hmm, not sure that I follow this. Are you saying that any
> >>> client
> >>> > > > > > > authenticated with the delegation token can renew, i.e.
> there
> >>> is
> >>> > no
> >>> > > > > > renewer
> >>> > > > > > > needed?
> >>> > > > > > >
> >>> > > > > > > Also, just to be clear, any authenticated client (either
> >>> through
> >>> > > SASL
> >>> > > > > or
> >>> > > > > > > SSL) can request a delegation token for the authenticated
> >>> user,
> >>> > > > right?
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > > > 2. Are tokens stored on each broker or in ZK?
> >>> > > > > > > > My recommendation is still to store in ZK or not store
> them
> >>> at
> >>> > > all.
> >>> > > > > The
> >>> > > > > > > > whole controller based distribution is too much overhead
> >>> with
> >>> > not
> >>> > > > > much
> >>> > > > > > to
> >>> > > > > > > > achieve.
> >>> > > > > > > >
> >>> > > > > > > > 3. How are tokens invalidated / expired?
> >>> > > > > > > > Either by expiration time out or through an explicit
> >>> request to
> >>> > > > > > > invalidate.
> >>> > > > > > > >
> >>> > > > > > > > 4. Which encryption algorithm is used?
> >>> > > > > > > > SCRAM
> >>> > > > > > > >
> >>> > > > > > > > 5. What is the impersonation proposal (it wasn't in the
> KIP
> >>> but
> >>> > > was
> >>> > > > > > > > discussed
> >>> > > > > > > > in this thread)?
> >>> > > > > > > > There is no imperonation proposal. I tried and explained
> how
> >>> > its
> >>> > > a
> >>> > > > > > > > different problem and why its not really necessary to
> >>> discuss
> >>> > > that
> >>> > > > as
> >>> > > > > > > part
> >>> > > > > > > > of this KIP.  This KIP will not support any
> impersonation,
> >>> it
> >>> > > will
> >>> > > > > just
> >>> > > > > > > be
> >>> > > > > > > > another way to authenticate.
> >>> > > > > > > >
> >>> > > > > > > > 6. Do we need new ACLs, if so - for what actions?
> >>> > > > > > > > We do not need new ACLs.
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > Could we document the format of the new request/response
> and
> >>> > their
> >>> > > > > > > associated Resource and Operation for ACL?
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > > > 7. How would the delegation token be configured in the
> >>> client?
> >>> > > > > > > > Should be through config. I wasn't planning on supporting
> >>> JAAS
> >>> > > for
> >>> > > > > > > tokens.
> >>> > > > > > > > I don't believe hadoop does this either.
> >>> > > > > > > >
> >>> > > > > > > > Thanks
> >>> > > > > > > > Parth
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <
> jun@confluent.io>
> >>> > > wrote:
> >>> > > > > > > >
> >>> > > > > > > > > Harsha,
> >>> > > > > > > > >
> >>> > > > > > > > > Another question.
> >>> > > > > > > > >
> >>> > > > > > > > > 9. How would the delegation token be configured in the
> >>> > client?
> >>> > > > The
> >>> > > > > > > > standard
> >>> > > > > > > > > way is to do this through JAAS. However, we will need
> to
> >>> > think
> >>> > > > > > through
> >>> > > > > > > if
> >>> > > > > > > > > this is convenient in a shared environment. For
> example,
> >>> > when a
> >>> > > > new
> >>> > > > > > > task
> >>> > > > > > > > is
> >>> > > > > > > > > added to a Storm worker node, do we need to dynamically
> >>> add a
> >>> > > new
> >>> > > > > > > section
> >>> > > > > > > > > in the JAAS file? It may be more convenient if we can
> >>> pass in
> >>> > > the
> >>> > > > > > token
> >>> > > > > > > > > through the config directly w/o going through JAAS.
> >>> > > > > > > > >
> >>> > > > > > > > > Are you or Parth still actively working on this KIP?
> >>> > > > > > > > >
> >>> > > > > > > > > Thanks,
> >>> > > > > > > > >
> >>> > > > > > > > > Jun
> >>> > > > > > > > >
> >>> > > > > > > > >
> >>> > > > > > > > >
> >>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <
> >>> jun@confluent.io>
> >>> > > > wrote:
> >>> > > > > > > > >
> >>> > > > > > > > > > Just to add on that list.
> >>> > > > > > > > > >
> >>> > > > > > > > > > 2. It would be good to document the format of the
> data
> >>> > stored
> >>> > > > in
> >>> > > > > > ZK.
> >>> > > > > > > > > > 7. Earlier, there was a discussion on whether the
> tokens
> >>> > > should
> >>> > > > > be
> >>> > > > > > > > > > propagated through ZK like config/acl/quota, or
> through
> >>> the
> >>> > > > > > > controller.
> >>> > > > > > > > > > Currently, the controller is only designed for
> >>> propagating
> >>> > > > topic
> >>> > > > > > > > > metadata,
> >>> > > > > > > > > > but not other data.
> >>> > > > > > > > > > 8. Should we use SCRAM to send the token instead of
> >>> > > DIGEST-MD5
> >>> > > > > > since
> >>> > > > > > > > it's
> >>> > > > > > > > > > deprecated?
> >>> > > > > > > > > >
> >>> > > > > > > > > > Also, the images in the wiki seem broken.
> >>> > > > > > > > > >
> >>> > > > > > > > > > Thanks,
> >>> > > > > > > > > >
> >>> > > > > > > > > > Jun
> >>> > > > > > > > > >
> >>> > > > > > > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <
> >>> > > > > gwen@confluent.io>
> >>> > > > > > > > > wrote:
> >>> > > > > > > > > >
> >>> > > > > > > > > >> From what I can see, remaining questions are:
> >>> > > > > > > > > >>
> >>> > > > > > > > > >> 1. Who / how are tokens renewed? By original
> requester
> >>> > only?
> >>> > > > or
> >>> > > > > > > using
> >>> > > > > > > > > >> Kerberos auth only?
> >>> > > > > > > > > >> 2. Are tokens stored on each broker or in ZK?
> >>> > > > > > > > > >> 3. How are tokens invalidated / expired?
> >>> > > > > > > > > >> 4. Which encryption algorithm is used?
> >>> > > > > > > > > >> 5. What is the impersonation proposal (it wasn't in
> the
> >>> > KIP
> >>> > > > but
> >>> > > > > > was
> >>> > > > > > > > > >> discussed in this thread)?
> >>> > > > > > > > > >> 6. Do we need new ACLs, if so - for what actions?
> >>> > > > > > > > > >>
> >>> > > > > > > > > >> Gwen
> >>> > > > > > > > > >>
> >>> > > > > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <
> >>> kafka@harsha.io>
> >>> > > > wrote:
> >>> > > > > > > > > >> > Jun & Ismael,
> >>> > > > > > > > > >> >                          Unfortunately I couldn't
> >>> attend
> >>> > > the
> >>> > > > > KIP
> >>> > > > > > > > > meeting
> >>> > > > > > > > > >> >                          when delegation tokens
> >>> > discussed.
> >>> > > > > > > > Appreciate
> >>> > > > > > > > > if
> >>> > > > > > > > > >> >                          you can update the
> thread if
> >>> > you
> >>> > > > have
> >>> > > > > > any
> >>> > > > > > > > > >> >                          further questions.
> >>> > > > > > > > > >> > Thanks,
> >>> > > > > > > > > >> > Harsha
> >>> > > > > > > > > >> >
> >>> > > > > > > > > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei
> wrote:
> >>> > > > > > > > > >> >> It seems that the links to images in the KIP are
> >>> > broken.
> >>> > > > > > > > > >> >>
> >>> > > > > > > > > >> >> Liquan
> >>> > > > > > > > > >> >>
> >>> > > > > > > > > >> >> On Tue, May 24, 2016 at 9:33 AM, parth
> brahmbhatt <
> >>> > > > > > > > > >> >> brahmbhatt.parth@gmail.com> wrote:
> >>> > > > > > > > > >> >>
> >>> > > > > > > > > >> >> > 110. What does getDelegationTokenAs mean?
> >>> > > > > > > > > >> >> > In the current proposal we only allow a user to
> >>> get
> >>> > > > > > delegation
> >>> > > > > > > > > token
> >>> > > > > > > > > >> for
> >>> > > > > > > > > >> >> > the identity that it authenticated as using
> >>> another
> >>> > > > > > mechanism,
> >>> > > > > > > > i.e.
> >>> > > > > > > > > >> A user
> >>> > > > > > > > > >> >> > that authenticate using a keytab for principal
> >>> > > > > > > user1@EXAMPLE.COM
> >>> > > > > > > > > >> will get
> >>> > > > > > > > > >> >> > delegation tokens for that user only. In
> future I
> >>> > think
> >>> > > > we
> >>> > > > > > will
> >>> > > > > > > > > have
> >>> > > > > > > > > >> to
> >>> > > > > > > > > >> >> > extend support such that we allow some set of
> >>> users (
> >>> > > > > > > > > >> >> > kafka-rest-user@EXAMPLE.COM,
> >>> > storm-nimbus@EXAMPLE.COM)
> >>> > > > to
> >>> > > > > > > > acquire
> >>> > > > > > > > > >> >> > delegation tokens on behalf of other users
> whose
> >>> > > identity
> >>> > > > > > they
> >>> > > > > > > > have
> >>> > > > > > > > > >> >> > verified independently.  Kafka brokers will
> have
> >>> ACLs
> >>> > > to
> >>> > > > > > > control
> >>> > > > > > > > > >> which
> >>> > > > > > > > > >> >> > users are allowed to impersonate other users
> and
> >>> get
> >>> > > > tokens
> >>> > > > > > on
> >>> > > > > > > > > >> behalf of
> >>> > > > > > > > > >> >> > them. Overall Impersonation is a whole
> different
> >>> > > problem
> >>> > > > in
> >>> > > > > > my
> >>> > > > > > > > > >> opinion and
> >>> > > > > > > > > >> >> > I think we can tackle it in separate KIP.
> >>> > > > > > > > > >> >> >
> >>> > > > > > > > > >> >> > 111. What's the typical rate of getting and
> >>> renewing
> >>> > > > > > delegation
> >>> > > > > > > > > >> tokens?
> >>> > > > > > > > > >> >> > Typically this should be very very low, 1
> request
> >>> per
> >>> > > > > minute
> >>> > > > > > > is a
> >>> > > > > > > > > >> >> > relatively high estimate. However it depends on
> >>> the
> >>> > > token
> >>> > > > > > > > > >> expiration. I am
> >>> > > > > > > > > >> >> > less worried about the extra load it puts on
> >>> > controller
> >>> > > > vs
> >>> > > > > > the
> >>> > > > > > > > > added
> >>> > > > > > > > > >> >> > complexity and the value it offers.
> >>> > > > > > > > > >> >> >
> >>> > > > > > > > > >> >> > Thanks
> >>> > > > > > > > > >> >> > Parth
> >>> > > > > > > > > >> >> >
> >>> > > > > > > > > >> >> >
> >>> > > > > > > > > >> >> >
> >>> > > > > > > > > >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <
> >>> > > > > > > ismael@juma.me.uk>
> >>> > > > > > > > > >> wrote:
> >>> > > > > > > > > >> >> >
> >>> > > > > > > > > >> >> > > Thanks Rajini. It would probably require a
> >>> separate
> >>> > > KIP
> >>> > > > > as
> >>> > > > > > it
> >>> > > > > > > > > will
> >>> > > > > > > > > >> >> > > introduce user visible changes. We could also
> >>> > update
> >>> > > > > KIP-48
> >>> > > > > > > to
> >>> > > > > > > > > >> have this
> >>> > > > > > > > > >> >> > > information, but it seems cleaner to do it
> >>> > > separately.
> >>> > > > We
> >>> > > > > > can
> >>> > > > > > > > > >> discuss
> >>> > > > > > > > > >> >> > that
> >>> > > > > > > > > >> >> > > in the KIP call today.
> >>> > > > > > > > > >> >> > >
> >>> > > > > > > > > >> >> > > Ismael
> >>> > > > > > > > > >> >> > >
> >>> > > > > > > > > >> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini
> Sivaram
> >>> <
> >>> > > > > > > > > >> >> > > rajinisivaram@googlemail.com> wrote:
> >>> > > > > > > > > >> >> > >
> >>> > > > > > > > > >> >> > > > Ismael,
> >>> > > > > > > > > >> >> > > >
> >>> > > > > > > > > >> >> > > > I have created a JIRA (
> >>> > > > > > > > > >> >> > https://issues.apache.org/
> jira/browse/KAFKA-3751)
> >>> > > > > > > > > >> >> > > > for adding SCRAM as a SASL mechanism. Would
> >>> that
> >>> > > need
> >>> > > > > > > another
> >>> > > > > > > > > >> KIP? If
> >>> > > > > > > > > >> >> > > > KIP-48 will use this mechanism, can this
> just
> >>> be
> >>> > a
> >>> > > > JIRA
> >>> > > > > > > that
> >>> > > > > > > > > gets
> >>> > > > > > > > > >> >> > > reviewed
> >>> > > > > > > > > >> >> > > > when the PR is ready?
> >>> > > > > > > > > >> >> > > >
> >>> > > > > > > > > >> >> > > > Thank you,
> >>> > > > > > > > > >> >> > > >
> >>> > > > > > > > > >> >> > > > Rajini
> >>> > > > > > > > > >> >> > > >
> >>> > > > > > > > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael
> Juma <
> >>> > > > > > > > > ismael@juma.me.uk>
> >>> > > > > > > > > >> >> > wrote:
> >>> > > > > > > > > >> >> > > >
> >>> > > > > > > > > >> >> > > > > Thanks Rajini, SCRAM seems like a good
> >>> > candidate.
> >>> > > > > > > > > >> >> > > > >
> >>> > > > > > > > > >> >> > > > > Gwen had independently mentioned this as
> a
> >>> SASL
> >>> > > > > > mechanism
> >>> > > > > > > > > that
> >>> > > > > > > > > >> might
> >>> > > > > > > > > >> >> > be
> >>> > > > > > > > > >> >> > > > > useful for Kafka and I have been meaning
> to
> >>> > check
> >>> > > > it
> >>> > > > > in
> >>> > > > > > > > more
> >>> > > > > > > > > >> detail.
> >>> > > > > > > > > >> >> > > Good
> >>> > > > > > > > > >> >> > > > > to know that you are willing to
> contribute
> >>> an
> >>> > > > > > > > implementation.
> >>> > > > > > > > > >> Maybe
> >>> > > > > > > > > >> >> > we
> >>> > > > > > > > > >> >> > > > > should file a separate JIRA for this?
> >>> > > > > > > > > >> >> > > > >
> >>> > > > > > > > > >> >> > > > > Ismael
> >>> > > > > > > > > >> >> > > > >
> >>> > > > > > > > > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini
> >>> > Sivaram <
> >>> > > > > > > > > >> >> > > > > rajinisivaram@googlemail.com> wrote:
> >>> > > > > > > > > >> >> > > > >
> >>> > > > > > > > > >> >> > > > > > SCRAM (Salted Challenge Response
> >>> > Authentication
> >>> > > > > > > > Mechanism)
> >>> > > > > > > > > >> is a
> >>> > > > > > > > > >> >> > > better
> >>> > > > > > > > > >> >> > > > > > mechanism than Digest-MD5. Java doesn't
> >>> come
> >>> > > > with a
> >>> > > > > > > > > built-in
> >>> > > > > > > > > >> SCRAM
> >>> > > > > > > > > >> >> > > > > > SaslServer or SaslClient, but I will be
> >>> happy
> >>> > > to
> >>> > > > > add
> >>> > > > > > > > > support
> >>> > > > > > > > > >> in
> >>> > > > > > > > > >> >> > Kafka
> >>> > > > > > > > > >> >> > > > > since
> >>> > > > > > > > > >> >> > > > > > it would be a useful mechanism to
> support
> >>> > > anyway.
> >>> > > > > > > > > >> >> > > > > > https://tools.ietf.org/html/rfc7677
> >>> > describes
> >>> > > > the
> >>> > > > > > > > protocol
> >>> > > > > > > > > >> for
> >>> > > > > > > > > >> >> > > > > > SCRAM-SHA-256.
> >>> > > > > > > > > >> >> > > > > >
> >>> > > > > > > > > >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun
> Rao <
> >>> > > > > > > > jun@confluent.io
> >>> > > > > > > > > >
> >>> > > > > > > > > >> wrote:
> >>> > > > > > > > > >> >> > > > > >
> >>> > > > > > > > > >> >> > > > > > > Parth,
> >>> > > > > > > > > >> >> > > > > > >
> >>> > > > > > > > > >> >> > > > > > > Thanks for the explanation. A couple
> of
> >>> > more
> >>> > > > > > > questions.
> >>> > > > > > > > > >> >> > > > > > >
> >>> > > > > > > > > >> >> > > > > > > 110. What does getDelegationTokenAs
> >>> mean?
> >>> > > > > > > > > >> >> > > > > > >
> >>> > > > > > > > > >> >> > > > > > > 111. What's the typical rate of
> getting
> >>> and
> >>> > > > > > renewing
> >>> > > > > > > > > >> delegation
> >>> > > > > > > > > >> >> > > > tokens?
> >>> > > > > > > > > >> >> > > > > > > That may have an impact on whether
> they
> >>> > > should
> >>> > > > be
> >>> > > > > > > > > directed
> >>> > > > > > > > > >> to the
> >>> > > > > > > > > >> >> > > > > > > controller.
> >>> > > > > > > > > >> >> > > > > > >
> >>> > > > > > > > > >> >> > > > > > > Jun
> >>> > > > > > > > > >> >> > > > > > >
> >>> > > > > > > > > >> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM,
> parth
> >>> > > > > brahmbhatt <
> >>> > > > > > > > > >> >> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> >>> > > > > > > > > >> >> > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > Hi Jun,
> >>> > > > > > > > > >> >> > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > Thanks for reviewing.
> >>> > > > > > > > > >> >> > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > * We could add a Cluster action to
> add
> >>> > acls
> >>> > > > on
> >>> > > > > > who
> >>> > > > > > > > can
> >>> > > > > > > > > >> request
> >>> > > > > > > > > >> >> > > > > > delegation
> >>> > > > > > > > > >> >> > > > > > > > tokens. I don't see the use case
> for
> >>> that
> >>> > > yet
> >>> > > > > but
> >>> > > > > > > > down
> >>> > > > > > > > > >> the line
> >>> > > > > > > > > >> >> > > > when
> >>> > > > > > > > > >> >> > > > > we
> >>> > > > > > > > > >> >> > > > > > > > start supporting
> getDelegationTokenAs
> >>> it
> >>> > > will
> >>> > > > > be
> >>> > > > > > > > > >> necessary.
> >>> > > > > > > > > >> >> > > > > > > > * Yes we recommend tokens to be
> only
> >>> > > > > > > used/distributed
> >>> > > > > > > > > >> over
> >>> > > > > > > > > >> >> > secure
> >>> > > > > > > > > >> >> > > > > > > channels.
> >>> > > > > > > > > >> >> > > > > > > > * Depending on what design we end
> up
> >>> > > choosing
> >>> > > > > > > > > >> Invalidation will
> >>> > > > > > > > > >> >> > > be
> >>> > > > > > > > > >> >> > > > > > > > responsibility of every broker or
> >>> > > controller.
> >>> > > > > > > > > >> >> > > > > > > > * I am not sure if I documented
> >>> somewhere
> >>> > > > that
> >>> > > > > > > > > >> invalidation
> >>> > > > > > > > > >> >> > will
> >>> > > > > > > > > >> >> > > > > > directly
> >>> > > > > > > > > >> >> > > > > > > > go through zookeeper but that is
> not
> >>> the
> >>> > > > > intent.
> >>> > > > > > > > > >> Invalidation
> >>> > > > > > > > > >> >> > > will
> >>> > > > > > > > > >> >> > > > > > either
> >>> > > > > > > > > >> >> > > > > > > > be request based or due to
> >>> expiration. No
> >>> > > > > direct
> >>> > > > > > > > > >> zookeeper
> >>> > > > > > > > > >> >> > > > > interaction
> >>> > > > > > > > > >> >> > > > > > > from
> >>> > > > > > > > > >> >> > > > > > > > any client.
> >>> > > > > > > > > >> >> > > > > > > > * "Broker also stores the
> >>> DelegationToken
> >>> > > > > without
> >>> > > > > > > the
> >>> > > > > > > > > >> hmac in
> >>> > > > > > > > > >> >> > the
> >>> > > > > > > > > >> >> > > > > > > > zookeeper." : Sorry about the
> >>> confusion.
> >>> > > The
> >>> > > > > sole
> >>> > > > > > > > > >> purpose of
> >>> > > > > > > > > >> >> > > > > zookeeper
> >>> > > > > > > > > >> >> > > > > > in
> >>> > > > > > > > > >> >> > > > > > > > this design is as distribution
> channel
> >>> > for
> >>> > > > > tokens
> >>> > > > > > > > > >> between all
> >>> > > > > > > > > >> >> > > > brokers
> >>> > > > > > > > > >> >> > > > > > > and a
> >>> > > > > > > > > >> >> > > > > > > > layer that ensures only tokens that
> >>> were
> >>> > > > > > generated
> >>> > > > > > > by
> >>> > > > > > > > > >> making a
> >>> > > > > > > > > >> >> > > > > request
> >>> > > > > > > > > >> >> > > > > > > to a
> >>> > > > > > > > > >> >> > > > > > > > broker will be accepted (more on
> this
> >>> in
> >>> > > > second
> >>> > > > > > > > > >> paragraph). The
> >>> > > > > > > > > >> >> > > > token
> >>> > > > > > > > > >> >> > > > > > > > consists of few elements (owner,
> >>> renewer,
> >>> > > > uuid
> >>> > > > > ,
> >>> > > > > > > > > >> expiration,
> >>> > > > > > > > > >> >> > > hmac)
> >>> > > > > > > > > >> >> > > > ,
> >>> > > > > > > > > >> >> > > > > > one
> >>> > > > > > > > > >> >> > > > > > > of
> >>> > > > > > > > > >> >> > > > > > > > which is the finally generated hmac
> >>> but
> >>> > > hmac
> >>> > > > it
> >>> > > > > > > self
> >>> > > > > > > > is
> >>> > > > > > > > > >> >> > derivable
> >>> > > > > > > > > >> >> > > > if
> >>> > > > > > > > > >> >> > > > > > you
> >>> > > > > > > > > >> >> > > > > > > > have all the other elements of the
> >>> token
> >>> > +
> >>> > > > > secret
> >>> > > > > > > key
> >>> > > > > > > > > to
> >>> > > > > > > > > >> >> > generate
> >>> > > > > > > > > >> >> > > > > hmac.
> >>> > > > > > > > > >> >> > > > > > > > Given zookeeper does not provide
> SSL
> >>> > > support
> >>> > > > we
> >>> > > > > > do
> >>> > > > > > > > not
> >>> > > > > > > > > >> want the
> >>> > > > > > > > > >> >> > > > > entire
> >>> > > > > > > > > >> >> > > > > > > > token to be wire transferred to
> >>> zookeeper
> >>> > > as
> >>> > > > > that
> >>> > > > > > > > will
> >>> > > > > > > > > >> be an
> >>> > > > > > > > > >> >> > > > insecure
> >>> > > > > > > > > >> >> > > > > > > wire
> >>> > > > > > > > > >> >> > > > > > > > transfer. Instead we only store all
> >>> the
> >>> > > other
> >>> > > > > > > > elements
> >>> > > > > > > > > >> of a
> >>> > > > > > > > > >> >> > > > > delegation
> >>> > > > > > > > > >> >> > > > > > > > tokens. Brokers can read these
> >>> elements
> >>> > and
> >>> > > > > > because
> >>> > > > > > > > > they
> >>> > > > > > > > > >> also
> >>> > > > > > > > > >> >> > > have
> >>> > > > > > > > > >> >> > > > > > access
> >>> > > > > > > > > >> >> > > > > > > > to secret key they will be able to
> >>> > generate
> >>> > > > > hmac
> >>> > > > > > on
> >>> > > > > > > > > >> their end.
> >>> > > > > > > > > >> >> > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > One of the alternative proposed is
> to
> >>> > avoid
> >>> > > > > > > zookeeper
> >>> > > > > > > > > >> >> > > altogether. A
> >>> > > > > > > > > >> >> > > > > > > Client
> >>> > > > > > > > > >> >> > > > > > > > will call broker with required
> >>> > information
> >>> > > > > > (owner,
> >>> > > > > > > > > >> renwer,
> >>> > > > > > > > > >> >> > > > > expiration)
> >>> > > > > > > > > >> >> > > > > > > and
> >>> > > > > > > > > >> >> > > > > > > > get back (signed hmac, uuid).
> Broker
> >>> > won't
> >>> > > > > store
> >>> > > > > > > this
> >>> > > > > > > > > in
> >>> > > > > > > > > >> >> > > zookeeper.
> >>> > > > > > > > > >> >> > > > > > From
> >>> > > > > > > > > >> >> > > > > > > > this point a client can contact any
> >>> > broker
> >>> > > > with
> >>> > > > > > all
> >>> > > > > > > > the
> >>> > > > > > > > > >> >> > > delegation
> >>> > > > > > > > > >> >> > > > > > token
> >>> > > > > > > > > >> >> > > > > > > > info (owner, rewner, expiration,
> hmac,
> >>> > > uuid)
> >>> > > > > the
> >>> > > > > > > > borker
> >>> > > > > > > > > >> will
> >>> > > > > > > > > >> >> > > > > regenerate
> >>> > > > > > > > > >> >> > > > > > > the
> >>> > > > > > > > > >> >> > > > > > > > hmac and as long as it matches with
> >>> hmac
> >>> > > > > > presented
> >>> > > > > > > by
> >>> > > > > > > > > >> client ,
> >>> > > > > > > > > >> >> > > > broker
> >>> > > > > > > > > >> >> > > > > > > will
> >>> > > > > > > > > >> >> > > > > > > > allow the request to authenticate.
> >>> Only
> >>> > > > > problem
> >>> > > > > > > with
> >>> > > > > > > > > >> this
> >>> > > > > > > > > >> >> > > approach
> >>> > > > > > > > > >> >> > > > > is
> >>> > > > > > > > > >> >> > > > > > if
> >>> > > > > > > > > >> >> > > > > > > > the secret key is compromised any
> >>> client
> >>> > > can
> >>> > > > > now
> >>> > > > > > > > > generate
> >>> > > > > > > > > >> >> > random
> >>> > > > > > > > > >> >> > > > > tokens
> >>> > > > > > > > > >> >> > > > > > > and
> >>> > > > > > > > > >> >> > > > > > > > they will still be able to
> >>> authenticate
> >>> > as
> >>> > > > any
> >>> > > > > > user
> >>> > > > > > > > > they
> >>> > > > > > > > > >> like.
> >>> > > > > > > > > >> >> > > with
> >>> > > > > > > > > >> >> > > > > > > > zookeeper we guarantee that only
> >>> tokens
> >>> > > > > acquired
> >>> > > > > > > via
> >>> > > > > > > > a
> >>> > > > > > > > > >> broker
> >>> > > > > > > > > >> >> > > > (using
> >>> > > > > > > > > >> >> > > > > > some
> >>> > > > > > > > > >> >> > > > > > > > auth scheme other than delegation
> >>> token)
> >>> > > will
> >>> > > > > be
> >>> > > > > > > > > >> accepted. We
> >>> > > > > > > > > >> >> > > need
> >>> > > > > > > > > >> >> > > > to
> >>> > > > > > > > > >> >> > > > > > > > discuss which proposal makes more
> >>> sense
> >>> > and
> >>> > > > we
> >>> > > > > > can
> >>> > > > > > > go
> >>> > > > > > > > > >> over it
> >>> > > > > > > > > >> >> > in
> >>> > > > > > > > > >> >> > > > > > > tomorrow's
> >>> > > > > > > > > >> >> > > > > > > > meeting.
> >>> > > > > > > > > >> >> > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > Also, can you forward the invite to
> >>> me?
> >>> > > > > > > > > >> >> > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > Thanks
> >>> > > > > > > > > >> >> > > > > > > > Parth
> >>> > > > > > > > > >> >> > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM,
> Jun
> >>> > Rao <
> >>> > > > > > > > > >> jun@confluent.io>
> >>> > > > > > > > > >> >> > > > wrote:
> >>> > > > > > > > > >> >> > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > Thanks for the KIP. A few
> comments.
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > 100. This potentially can be
> useful
> >>> for
> >>> > > > Kafka
> >>> > > > > > > > Connect
> >>> > > > > > > > > >> and
> >>> > > > > > > > > >> >> > Kafka
> >>> > > > > > > > > >> >> > > > > rest
> >>> > > > > > > > > >> >> > > > > > > > proxy
> >>> > > > > > > > > >> >> > > > > > > > > where a worker agent will need to
> >>> run a
> >>> > > > task
> >>> > > > > on
> >>> > > > > > > > > behalf
> >>> > > > > > > > > >> of a
> >>> > > > > > > > > >> >> > > > client.
> >>> > > > > > > > > >> >> > > > > > We
> >>> > > > > > > > > >> >> > > > > > > > will
> >>> > > > > > > > > >> >> > > > > > > > > likely need to change how those
> >>> > services
> >>> > > > use
> >>> > > > > > > Kafka
> >>> > > > > > > > > >> clients
> >>> > > > > > > > > >> >> > > > > > > > > (producer/consumer). Instead of a
> >>> > shared
> >>> > > > > client
> >>> > > > > > > per
> >>> > > > > > > > > >> worker,
> >>> > > > > > > > > >> >> > we
> >>> > > > > > > > > >> >> > > > will
> >>> > > > > > > > > >> >> > > > > > > need
> >>> > > > > > > > > >> >> > > > > > > > a
> >>> > > > > > > > > >> >> > > > > > > > > client per user task since the
> >>> > > > authentication
> >>> > > > > > > > happens
> >>> > > > > > > > > >> at the
> >>> > > > > > > > > >> >> > > > > > connection
> >>> > > > > > > > > >> >> > > > > > > > > level. For Kafka Connect, the
> >>> renewer
> >>> > > will
> >>> > > > be
> >>> > > > > > the
> >>> > > > > > > > > >> workers.
> >>> > > > > > > > > >> >> > So,
> >>> > > > > > > > > >> >> > > we
> >>> > > > > > > > > >> >> > > > > > > > probably
> >>> > > > > > > > > >> >> > > > > > > > > need to allow multiple renewers.
> For
> >>> > > Kafka
> >>> > > > > rest
> >>> > > > > > > > > proxy,
> >>> > > > > > > > > >> the
> >>> > > > > > > > > >> >> > > > renewer
> >>> > > > > > > > > >> >> > > > > > can
> >>> > > > > > > > > >> >> > > > > > > > > probably just be the creator of
> the
> >>> > > token.
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > 101. Do we need new acl on who
> can
> >>> > > request
> >>> > > > > > > > delegation
> >>> > > > > > > > > >> tokens?
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > 102. Do we recommend people to
> send
> >>> > > > > delegation
> >>> > > > > > > > tokens
> >>> > > > > > > > > >> in an
> >>> > > > > > > > > >> >> > > > > encrypted
> >>> > > > > > > > > >> >> > > > > > > > > channel?
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > 103. Who is responsible for
> expiring
> >>> > > > tokens,
> >>> > > > > > > every
> >>> > > > > > > > > >> broker?
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > 104. For invalidating tokens,
> would
> >>> it
> >>> > be
> >>> > > > > > better
> >>> > > > > > > to
> >>> > > > > > > > > do
> >>> > > > > > > > > >> it in
> >>> > > > > > > > > >> >> > a
> >>> > > > > > > > > >> >> > > > > > request
> >>> > > > > > > > > >> >> > > > > > > > > instead of going to ZK directly?
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > 105. The terminology of client in
> >>> the
> >>> > > wiki
> >>> > > > > > > > sometimes
> >>> > > > > > > > > >> refers
> >>> > > > > > > > > >> >> > to
> >>> > > > > > > > > >> >> > > > the
> >>> > > > > > > > > >> >> > > > > > end
> >>> > > > > > > > > >> >> > > > > > > > > client and some other times
> refers
> >>> to
> >>> > the
> >>> > > > > > client
> >>> > > > > > > > > using
> >>> > > > > > > > > >> the
> >>> > > > > > > > > >> >> > > > > delegation
> >>> > > > > > > > > >> >> > > > > > > > > tokens. It would be useful to
> >>> > distinguish
> >>> > > > > > between
> >>> > > > > > > > the
> >>> > > > > > > > > >> two.
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > 106. Could you explain the
> sentence
> >>> > > "Broker
> >>> > > > > > also
> >>> > > > > > > > > >> stores the
> >>> > > > > > > > > >> >> > > > > > > > DelegationToken
> >>> > > > > > > > > >> >> > > > > > > > > without the hmac in the
> zookeeper."
> >>> a
> >>> > bit
> >>> > > > > > more? I
> >>> > > > > > > > > >> thought the
> >>> > > > > > > > > >> >> > > > > > > delegation
> >>> > > > > > > > > >> >> > > > > > > > > token is the hmac.
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > Thanks,
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > Jun
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM,
> Jun
> >>> > Rao
> >>> > > <
> >>> > > > > > > > > >> jun@confluent.io>
> >>> > > > > > > > > >> >> > > > wrote:
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > > Hi, Harsha,
> >>> > > > > > > > > >> >> > > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > > Just sent out a KIP meeting
> >>> invite.
> >>> > We
> >>> > > > can
> >>> > > > > > > > discuss
> >>> > > > > > > > > >> this in
> >>> > > > > > > > > >> >> > > the
> >>> > > > > > > > > >> >> > > > > > > meeting
> >>> > > > > > > > > >> >> > > > > > > > > > tomorrow.
> >>> > > > > > > > > >> >> > > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > > Thanks,
> >>> > > > > > > > > >> >> > > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > > Jun
> >>> > > > > > > > > >> >> > > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47
> AM,
> >>> > > Harsha <
> >>> > > > > > > > > >> kafka@harsha.io>
> >>> > > > > > > > > >> >> > > > wrote:
> >>> > > > > > > > > >> >> > > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> Hi All,
> >>> > > > > > > > > >> >> > > > > > > > > >>            Can we have a KIP
> >>> meeting
> >>> > > > > around
> >>> > > > > > > > this.
> >>> > > > > > > > > >> The KIP
> >>> > > > > > > > > >> >> > is
> >>> > > > > > > > > >> >> > > > up
> >>> > > > > > > > > >> >> > > > > > for
> >>> > > > > > > > > >> >> > > > > > > > > >>            sometime and if
> there
> >>> are
> >>> > > any
> >>> > > > > > > > questions
> >>> > > > > > > > > >> lets
> >>> > > > > > > > > >> >> > > > quickly
> >>> > > > > > > > > >> >> > > > > > hash
> >>> > > > > > > > > >> >> > > > > > > > out
> >>> > > > > > > > > >> >> > > > > > > > > >>            details.
> >>> > > > > > > > > >> >> > > > > > > > > >>
> >>> > > > > > > > > >> >> > > > > > > > > >> Thanks,
> >>> > > > > > > > > >> >> > > > > > > > > >> Harsha
> >>> > > > > > > > > >> >> > > > > > > > > >>
> >>> > > > > > > > > >> >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40
> >>> AM,
> >>> > > parth
> >>> > > > > > > > > brahmbhatt
> >>> > > > > > > > > >> wrote:
> >>> > > > > > > > > >> >> > > > > > > > > >> > That is what the hadoop echo
> >>> > system
> >>> > > > uses
> >>> > > > > > so
> >>> > > > > > > no
> >>> > > > > > > > > >> good
> >>> > > > > > > > > >> >> > reason
> >>> > > > > > > > > >> >> > > > > > really.
> >>> > > > > > > > > >> >> > > > > > > > We
> >>> > > > > > > > > >> >> > > > > > > > > >> > could
> >>> > > > > > > > > >> >> > > > > > > > > >> > change it to whatever is the
> >>> > newest
> >>> > > > > > > > recommended
> >>> > > > > > > > > >> standard
> >>> > > > > > > > > >> >> > > is.
> >>> > > > > > > > > >> >> > > > > > > > > >> >
> >>> > > > > > > > > >> >> > > > > > > > > >> > Thanks
> >>> > > > > > > > > >> >> > > > > > > > > >> > Parth
> >>> > > > > > > > > >> >> > > > > > > > > >> >
> >>> > > > > > > > > >> >> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33
> >>> AM,
> >>> > > > Ismael
> >>> > > > > > > Juma <
> >>> > > > > > > > > >> >> > > > > ismael@juma.me.uk
> >>> > > > > > > > > >> >> > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > wrote:
> >>> > > > > > > > > >> >> > > > > > > > > >> >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > Hi Parth,
> >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > Thanks for the KIP. I only
> >>> > started
> >>> > > > > > > reviewing
> >>> > > > > > > > > >> this and
> >>> > > > > > > > > >> >> > > may
> >>> > > > > > > > > >> >> > > > > have
> >>> > > > > > > > > >> >> > > > > > > > > >> additional
> >>> > > > > > > > > >> >> > > > > > > > > >> > > questions later. The
> >>> immediate
> >>> > > > > question
> >>> > > > > > > that
> >>> > > > > > > > > >> came to
> >>> > > > > > > > > >> >> > > mind
> >>> > > > > > > > > >> >> > > > is
> >>> > > > > > > > > >> >> > > > > > our
> >>> > > > > > > > > >> >> > > > > > > > > >> choice of
> >>> > > > > > > > > >> >> > > > > > > > > >> > > "DIGEST-MD5" even though
> it's
> >>> > > marked
> >>> > > > > as
> >>> > > > > > > > > >> OBSOLETE in
> >>> > > > > > > > > >> >> > the
> >>> > > > > > > > > >> >> > > > IANA
> >>> > > > > > > > > >> >> > > > > > > > > Registry
> >>> > > > > > > > > >> >> > > > > > > > > >> of
> >>> > > > > > > > > >> >> > > > > > > > > >> > > SASL mechanisms and the
> >>> original
> >>> > > RFC
> >>> > > > > > > (2831)
> >>> > > > > > > > > has
> >>> > > > > > > > > >> been
> >>> > > > > > > > > >> >> > > moved
> >>> > > > > > > > > >> >> > > > > to
> >>> > > > > > > > > >> >> > > > > > > > > Historic
> >>> > > > > > > > > >> >> > > > > > > > > >> > > status:
> >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > >
> https://tools.ietf.org/html/
> >>> > > rfc6331
> >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > >
> >>> > > > > > > > > >> >> > >
> >>> > > > > > > > > >>
> >>> > > > > > >
> >>> > > > http://www.iana.org/assignments/sasl-mechanisms/sasl-
> >>> mechanisms.xhtml
> >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > What is the reasoning
> behind
> >>> > that
> >>> > > > > > choice?
> >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > Thanks,
> >>> > > > > > > > > >> >> > > > > > > > > >> > > Ismael
> >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > On Fri, May 13, 2016 at
> 11:29
> >>> > PM,
> >>> > > > Gwen
> >>> > > > > > > > > Shapira <
> >>> > > > > > > > > >> >> > > > > > > gwen@confluent.io
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> wrote:
> >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > Also comments inline :)
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > * I want to emphasize
> >>> that
> >>> > > even
> >>> > > > > > though
> >>> > > > > > > > > >> delegation
> >>> > > > > > > > > >> >> > > > tokens
> >>> > > > > > > > > >> >> > > > > > > are a
> >>> > > > > > > > > >> >> > > > > > > > > >> Hadoop
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > innovation, I feel
> very
> >>> > > strongly
> >>> > > > > > about
> >>> > > > > > > > not
> >>> > > > > > > > > >> adding
> >>> > > > > > > > > >> >> > > > > > dependency
> >>> > > > > > > > > >> >> > > > > > > > on
> >>> > > > > > > > > >> >> > > > > > > > > >> Hadoop
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > when implementing
> >>> delegation
> >>> > > > > tokens
> >>> > > > > > > for
> >>> > > > > > > > > >> Kafka. The
> >>> > > > > > > > > >> >> > > KIP
> >>> > > > > > > > > >> >> > > > > > > doesn't
> >>> > > > > > > > > >> >> > > > > > > > > >> imply
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > such dependency, but
> if
> >>> you
> >>> > > can
> >>> > > > > > > > clarify...
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *No hadoop
> dependency.*
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > Yay! Just add this to
> the
> >>> KIP
> >>> > so
> >>> > > > no
> >>> > > > > > one
> >>> > > > > > > > will
> >>> > > > > > > > > >> read
> >>> > > > > > > > > >> >> > the
> >>> > > > > > > > > >> >> > > > KIP
> >>> > > > > > > > > >> >> > > > > > and
> >>> > > > > > > > > >> >> > > > > > > > > panic
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > three weeks before the
> next
> >>> > > > > release...
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > * Can we get
> delegation
> >>> > token
> >>> > > at
> >>> > > > > any
> >>> > > > > > > > time
> >>> > > > > > > > > >> after
> >>> > > > > > > > > >> >> > > > > > > > authenticating?
> >>> > > > > > > > > >> >> > > > > > > > > >> only
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > immediately after?
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *As long as you are
> >>> > > > authenticated
> >>> > > > > > you
> >>> > > > > > > > can
> >>> > > > > > > > > >> get
> >>> > > > > > > > > >> >> > > > delegation
> >>> > > > > > > > > >> >> > > > > > > > tokens.
> >>> > > > > > > > > >> >> > > > > > > > > >> We
> >>> > > > > > > > > >> >> > > > > > > > > >> > > need
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > to
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > discuss if a client
> >>> > > > authenticated
> >>> > > > > > > using
> >>> > > > > > > > > >> delegation
> >>> > > > > > > > > >> >> > > > > token,
> >>> > > > > > > > > >> >> > > > > > > can
> >>> > > > > > > > > >> >> > > > > > > > > also
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > acquire
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > delegation token
> again or
> >>> > not.
> >>> > > > > Also
> >>> > > > > > > > there
> >>> > > > > > > > > >> is the
> >>> > > > > > > > > >> >> > > > > question
> >>> > > > > > > > > >> >> > > > > > of
> >>> > > > > > > > > >> >> > > > > > > > do
> >>> > > > > > > > > >> >> > > > > > > > > we
> >>> > > > > > > > > >> >> > > > > > > > > >> > > allow
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > anyone to acquire
> >>> delegation
> >>> > > > token
> >>> > > > > > or
> >>> > > > > > > we
> >>> > > > > > > > > >> want
> >>> > > > > > > > > >> >> > > specific
> >>> > > > > > > > > >> >> > > > > > ACLs
> >>> > > > > > > > > >> >> > > > > > > (I
> >>> > > > > > > > > >> >> > > > > > > > > >> think
> >>> > > > > > > > > >> >> > > > > > > > > >> > > its
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > an
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > overkill.)*
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > I agree that ACLs is an
> >>> > > overkill.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > I think we are debating
> two
> >>> > > > options:
> >>> > > > > > > > Either
> >>> > > > > > > > > >> require
> >>> > > > > > > > > >> >> > > > > Kerberos
> >>> > > > > > > > > >> >> > > > > > > > auth
> >>> > > > > > > > > >> >> > > > > > > > > >> for
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > renewal or require
> >>> non-owners
> >>> > to
> >>> > > > > > renew.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > I *think* the latter is
> >>> > simpler
> >>> > > > (it
> >>> > > > > > > > > basically
> >>> > > > > > > > > >> >> > require
> >>> > > > > > > > > >> >> > > a
> >>> > > > > > > > > >> >> > > > > "job
> >>> > > > > > > > > >> >> > > > > > > > > master"
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > to take responsibility
> for
> >>> the
> >>> > > > > > renewal,
> >>> > > > > > > it
> >>> > > > > > > > > >> will have
> >>> > > > > > > > > >> >> > > its
> >>> > > > > > > > > >> >> > > > > own
> >>> > > > > > > > > >> >> > > > > > > > > >> identity
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > anyway and I think this
> is
> >>> the
> >>> > > > > correct
> >>> > > > > > > > > design
> >>> > > > > > > > > >> >> > pattern
> >>> > > > > > > > > >> >> > > > > > anyway.
> >>> > > > > > > > > >> >> > > > > > > > For
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > storm, I'd expect
> Nimbus to
> >>> > > > > coordinate
> >>> > > > > > > > > >> renewals?),
> >>> > > > > > > > > >> >> > but
> >>> > > > > > > > > >> >> > > > it
> >>> > > > > > > > > >> >> > > > > is
> >>> > > > > > > > > >> >> > > > > > > > hard
> >>> > > > > > > > > >> >> > > > > > > > > to
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > debate simplicity
> without
> >>> > > looking
> >>> > > > at
> >>> > > > > > the
> >>> > > > > > > > > code
> >>> > > > > > > > > >> >> > changes
> >>> > > > > > > > > >> >> > > > > > > required.
> >>> > > > > > > > > >> >> > > > > > > > If
> >>> > > > > > > > > >> >> > > > > > > > > >> you
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > have a draft of how the
> >>> > "require
> >>> > > > > > > Kerberos"
> >>> > > > > > > > > >> will look
> >>> > > > > > > > > >> >> > > in
> >>> > > > > > > > > >> >> > > > > > Kafka
> >>> > > > > > > > > >> >> > > > > > > > > code,
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > I'll be happy to take a
> >>> look.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > * My understanding is
> >>> that
> >>> > > > tokens
> >>> > > > > > will
> >>> > > > > > > > > >> propagate
> >>> > > > > > > > > >> >> > via
> >>> > > > > > > > > >> >> > > > ZK
> >>> > > > > > > > > >> >> > > > > > but
> >>> > > > > > > > > >> >> > > > > > > > > >> without
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > additional changes to
> >>> > > > > UpdateMetadata
> >>> > > > > > > > > >> protocol,
> >>> > > > > > > > > >> >> > > > correct?
> >>> > > > > > > > > >> >> > > > > > > > Clients
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > currently don't retry
> on
> >>> > SASL
> >>> > > > auth
> >>> > > > > > > > failure
> >>> > > > > > > > > >> (IIRC),
> >>> > > > > > > > > >> >> > > but
> >>> > > > > > > > > >> >> > > > > > since
> >>> > > > > > > > > >> >> > > > > > > > the
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > tokens propagate
> between
> >>> > > brokers
> >>> > > > > > > asynch,
> >>> > > > > > > > > we
> >>> > > > > > > > > >> will
> >>> > > > > > > > > >> >> > > need
> >>> > > > > > > > > >> >> > > > to
> >>> > > > > > > > > >> >> > > > > > > > retry a
> >>> > > > > > > > > >> >> > > > > > > > > >> bit
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > to avoid clients
> failing
> >>> > auth
> >>> > > > due
> >>> > > > > to
> >>> > > > > > > > > timing
> >>> > > > > > > > > >> >> > issues.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *I am considering 2
> >>> > > alternatives
> >>> > > > > > right
> >>> > > > > > > > > now.
> >>> > > > > > > > > >> The
> >>> > > > > > > > > >> >> > > > current
> >>> > > > > > > > > >> >> > > > > > > > > documented
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > approach
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > is zookeeper based
> and it
> >>> > does
> >>> > > > not
> >>> > > > > > > > require
> >>> > > > > > > > > >> any
> >>> > > > > > > > > >> >> > > changes
> >>> > > > > > > > > >> >> > > > > to
> >>> > > > > > > > > >> >> > > > > > > > > >> > > UpdateMetadata
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > protocol. An
> alternative
> >>> > > > approach
> >>> > > > > > can
> >>> > > > > > > > > remove
> >>> > > > > > > > > >> >> > > zookeeper
> >>> > > > > > > > > >> >> > > > > > > > > dependency
> >>> > > > > > > > > >> >> > > > > > > > > >> as
> >>> > > > > > > > > >> >> > > > > > > > > >> > > well
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > but we can discuss
> that
> >>> in
> >>> > KIP
> >>> > > > > > > > discussion
> >>> > > > > > > > > >> call.*
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > Oooh! Sounds
> interesting.
> >>> Do
> >>> > you
> >>> > > > > want
> >>> > > > > > to
> >>> > > > > > > > > ping
> >>> > > > > > > > > >> Jun to
> >>> > > > > > > > > >> >> > > > > > arrange a
> >>> > > > > > > > > >> >> > > > > > > > > call?
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > * I liked Ashish's
> >>> > suggestion
> >>> > > of
> >>> > > > > > > having
> >>> > > > > > > > > >> just the
> >>> > > > > > > > > >> >> > > > > > controller
> >>> > > > > > > > > >> >> > > > > > > > > issue
> >>> > > > > > > > > >> >> > > > > > > > > >> the
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > delegation tokens, to
> >>> avoid
> >>> > > > > syncing
> >>> > > > > > a
> >>> > > > > > > > > shared
> >>> > > > > > > > > >> >> > secret.
> >>> > > > > > > > > >> >> > > > Not
> >>> > > > > > > > > >> >> > > > > > > sure
> >>> > > > > > > > > >> >> > > > > > > > if
> >>> > > > > > > > > >> >> > > > > > > > > >> we
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > want to continue the
> >>> > > discussion
> >>> > > > > here
> >>> > > > > > > or
> >>> > > > > > > > on
> >>> > > > > > > > > >> the
> >>> > > > > > > > > >> >> > > wiki. I
> >>> > > > > > > > > >> >> > > > > > think
> >>> > > > > > > > > >> >> > > > > > > > > that
> >>> > > > > > > > > >> >> > > > > > > > > >> we
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > can decouple the
> problem
> >>> of
> >>> > > > "token
> >>> > > > > > > > > >> distribution"
> >>> > > > > > > > > >> >> > > from
> >>> > > > > > > > > >> >> > > > > > > "shared
> >>> > > > > > > > > >> >> > > > > > > > > >> secret
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > distribution" and use
> the
> >>> > > > > controller
> >>> > > > > > > as
> >>> > > > > > > > > the
> >>> > > > > > > > > >> only
> >>> > > > > > > > > >> >> > > token
> >>> > > > > > > > > >> >> > > > > > > > generator
> >>> > > > > > > > > >> >> > > > > > > > > >> to
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > solve the second
> issue,
> >>> > while
> >>> > > > > still
> >>> > > > > > > > using
> >>> > > > > > > > > >> ZK async
> >>> > > > > > > > > >> >> > > to
> >>> > > > > > > > > >> >> > > > > > > > distribute
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > tokens.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *As mentioned in the
> >>> > previous
> >>> > > > > Email
> >>> > > > > > I
> >>> > > > > > > am
> >>> > > > > > > > > >> fine with
> >>> > > > > > > > > >> >> > > > that
> >>> > > > > > > > > >> >> > > > > > > > approach
> >>> > > > > > > > > >> >> > > > > > > > > >> as
> >>> > > > > > > > > >> >> > > > > > > > > >> > > long
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > as
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > we agree that the
> extra
> >>> > > > complexity
> >>> > > > > > of
> >>> > > > > > > > > >> >> > > adding/updating
> >>> > > > > > > > > >> >> > > > > APIS
> >>> > > > > > > > > >> >> > > > > > > > adds
> >>> > > > > > > > > >> >> > > > > > > > > >> enough
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > value. The advantage
> with
> >>> > the
> >>> > > > > > > controller
> >>> > > > > > > > > >> approach
> >>> > > > > > > > > >> >> > is
> >>> > > > > > > > > >> >> > > > > > secret
> >>> > > > > > > > > >> >> > > > > > > > > >> rotation
> >>> > > > > > > > > >> >> > > > > > > > > >> > > can
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > be
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > automated,frequent and
> >>> would
> >>> > > not
> >>> > > > > > > require
> >>> > > > > > > > > >> >> > > deployment. *
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > Can you detail the extra
> >>> > > > complexity
> >>> > > > > > (or
> >>> > > > > > > > > point
> >>> > > > > > > > > >> me to
> >>> > > > > > > > > >> >> > > the
> >>> > > > > > > > > >> >> > > > > > email
> >>> > > > > > > > > >> >> > > > > > > I
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > missed?) - which Apis
> are
> >>> > > > required?
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > As far as I can tell,
> >>> clients
> >>> > > can
> >>> > > > > > > already
> >>> > > > > > > > > >> find the
> >>> > > > > > > > > >> >> > > > > > controller
> >>> > > > > > > > > >> >> > > > > > > > from
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > metadata. I'm a bit more
> >>> > > concerned
> >>> > > > > > about
> >>> > > > > > > > > >> controller
> >>> > > > > > > > > >> >> > > > load.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > * While I like the
> idea
> >>> of
> >>> > > > forcing
> >>> > > > > > > > > kerberos
> >>> > > > > > > > > >> auth
> >>> > > > > > > > > >> >> > for
> >>> > > > > > > > > >> >> > > > > > > renwal, I
> >>> > > > > > > > > >> >> > > > > > > > > >> think
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > it mixes the transport
> >>> layer
> >>> > > the
> >>> > > > > the
> >>> > > > > > > > > request
> >>> > > > > > > > > >> >> > content
> >>> > > > > > > > > >> >> > > > in
> >>> > > > > > > > > >> >> > > > > a
> >>> > > > > > > > > >> >> > > > > > > > pretty
> >>> > > > > > > > > >> >> > > > > > > > > >> ugly
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > way. Perhaps limiting
> >>> > renewer
> >>> > > to
> >>> > > > > > > > non-owner
> >>> > > > > > > > > >> is
> >>> > > > > > > > > >> >> > > better.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *I feel this is a
> >>> necessary
> >>> > > > evil.
> >>> > > > > > > While
> >>> > > > > > > > > >> this will
> >>> > > > > > > > > >> >> > > make
> >>> > > > > > > > > >> >> > > > > the
> >>> > > > > > > > > >> >> > > > > > > > kafka
> >>> > > > > > > > > >> >> > > > > > > > > >> code
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > pretty straight
> forward ,
> >>> > > > forcing
> >>> > > > > > > > renewer
> >>> > > > > > > > > >> to
> >>> > > > > > > > > >> >> > > > non-owner
> >>> > > > > > > > > >> >> > > > > > > pushes
> >>> > > > > > > > > >> >> > > > > > > > > >> the code
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > ugliness to client and
> >>> makes
> >>> > > it
> >>> > > > > even
> >>> > > > > > > > > harder
> >>> > > > > > > > > >> to
> >>> > > > > > > > > >> >> > > > > > integrate.  *
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > As mentioned before, I
> >>> don't
> >>> > > think
> >>> > > > > the
> >>> > > > > > > > > >> "renewal by
> >>> > > > > > > > > >> >> > > > other"
> >>> > > > > > > > > >> >> > > > > > > > approach
> >>> > > > > > > > > >> >> > > > > > > > > >> is
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > that ugly for the
> clients
> >>> we
> >>> > > > expect
> >>> > > > > to
> >>> > > > > > > use
> >>> > > > > > > > > >> >> > delegation
> >>> > > > > > > > > >> >> > > > > tokens
> >>> > > > > > > > > >> >> > > > > > > > since
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > they will have an
> >>> app-master
> >>> > of
> >>> > > > some
> >>> > > > > > > sort
> >>> > > > > > > > > who
> >>> > > > > > > > > >> >> > > requested
> >>> > > > > > > > > >> >> > > > > the
> >>> > > > > > > > > >> >> > > > > > > > token
> >>> > > > > > > > > >> >> > > > > > > > > to
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > begin with.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > The response for my
> >>> question
> >>> > > on
> >>> > > > > how
> >>> > > > > > > > > multiple
> >>> > > > > > > > > >> >> > > > identities
> >>> > > > > > > > > >> >> > > > > > will
> >>> > > > > > > > > >> >> > > > > > > > be
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > handled wasn't super
> >>> clear
> >>> > to
> >>> > > > me -
> >>> > > > > > > > AFAIK,
> >>> > > > > > > > > >> we don't
> >>> > > > > > > > > >> >> > > > > > > > authenticate
> >>> > > > > > > > > >> >> > > > > > > > > >> each
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > request, we
> authenticate
> >>> > > > > > connections.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *We authenticate
> >>> > connections,
> >>> > > > and
> >>> > > > > > only
> >>> > > > > > > > > when
> >>> > > > > > > > > >> they
> >>> > > > > > > > > >> >> > are
> >>> > > > > > > > > >> >> > > > > being
> >>> > > > > > > > > >> >> > > > > > > > > >> established.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > Let
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > me try to phrase this
> as
> >>> a
> >>> > > > > question,
> >>> > > > > > > in
> >>> > > > > > > > > >> absence of
> >>> > > > > > > > > >> >> > > > > > > delegation
> >>> > > > > > > > > >> >> > > > > > > > > >> tokens if
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > we
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > had to support the use
> >>> case
> >>> > > > using
> >>> > > > > > user
> >>> > > > > > > > > >> TGT's how
> >>> > > > > > > > > >> >> > > would
> >>> > > > > > > > > >> >> > > > > we
> >>> > > > > > > > > >> >> > > > > > do
> >>> > > > > > > > > >> >> > > > > > > > it?
> >>> > > > > > > > > >> >> > > > > > > > > >> My
> >>> > > > > > > > > >> >> > > > > > > > > >> > > point
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > was it would be no
> >>> different
> >>> > > > with
> >>> > > > > > > > > delegation
> >>> > > > > > > > > >> >> > tokens.
> >>> > > > > > > > > >> >> > > > The
> >>> > > > > > > > > >> >> > > > > > use
> >>> > > > > > > > > >> >> > > > > > > > > case
> >>> > > > > > > > > >> >> > > > > > > > > >> you
> >>> > > > > > > > > >> >> > > > > > > > > >> > > are
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > describing seems more
> >>> like
> >>> > > > > > > > impersonation.*
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > Yeah, I thought that
> one of
> >>> > the
> >>> > > > > things
> >>> > > > > > > > that
> >>> > > > > > > > > >> >> > del
> >>>
> >>
> >>
> >>
> >> --
> >>
> >> Regards,
> >> Ashish
> >>
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Jun Rao <ju...@confluent.io>]

Thanks for the KIP.

Request-level impersonation is going to make the client implementation more
complicated. On the producer side, we batch messages per partition. Now, do
we have to batch per partition, per user? The consumer client is designed
to be single-threaded. So, I am not sure if request-level impersonation
makes sense there. The main benefit of request-level impersonation seems to
be saving the number of socket connections to the broker. However, the
broker is capable of handling lots of connections.

Jun

On Tue, Dec 20, 2016 at 8:42 AM, Manikumar <ma...@gmail.com>
wrote:

> Hi Devs,
>
> If there are no more comments, I will start vote on this KIP later this
> week.
>
>
> Thanks
>
> On Fri, Dec 16, 2016 at 12:28 PM, Manikumar <ma...@gmail.com>
> wrote:
>
> > Hi,
> >
> >
> >> Can you add a sample Jaas configuration using delegation tokens to the
> >> KIP?
> >>
> >
> > Will add sample Jaas configuration.
> >
> >
> >> To make sure I have understood correctly, KAFKA-3712 is aimed at
> enabling
> >> a
> >> superuser to impersonate another (single) user, say alice. A producer
> >> using
> >> impersonation will authenticate with superuser credentials. All requests
> >> from the producer will be run with the principal alice. But alice is not
> >> involved in the authentication and alice's credentials are not actually
> >> provided to the broker?
> >>
> >>
> >  Yes, this matches with my understanding of impersonation work . Even in
> > this approach
> >  we have to handle all producer related issues mentioned by you. Yes,
> this
> > is big change
> >  and can be implemented if there is a pressing need. I hope we are all in
> > agreement, that
> >  this can be done in a separate KIP.
> >
> >
> >  I request others give any suggestions/concerns on this KIP.
> >
> >
> > Thanks,
> >
> >
> >
> >>
> >> On Thu, Dec 15, 2016 at 9:04 AM, Manikumar <ma...@gmail.com>
> >> wrote:
> >>
> >> > @Gwen, @Rajini,
> >> >
> >> > As mentioned in the KIP, main motivation for this KIP is to reduce
> load
> >> on
> >> > Kerberos
> >> > server on large kafka deployments with large number of clients.
> >> >
> >> > Also it looks like we are combining two overlapping concepts
> >> > 1. Single client sending requests with multiple users/authentications
> >> > 2. Impersonation
> >> >
> >> > Option 1, is definitely useful in some use cases and can be used to
> >> > implement workaround for
> >> > impersonation
> >> >
> >> > In Impersonation, a super user can send requests on behalf of another
> >> > user(Alice) in a secured way.
> >> > superuser has credentials but user Alice doesn't have any. The
> requests
> >> are
> >> > required
> >> > to run as user Alice and accesses/ACLs on Broker are required to be
> >> done as
> >> > user Alice.
> >> > It is required that user Alice can connect to the Broker on a
> connection
> >> > authenticated with
> >> > superuser's credentials. In other words superuser is impersonating the
> >> user
> >> > Alice.
> >> >
> >> > The approach mentioned by Harsha in previous mail is implemented in
> >> hadoop,
> >> > storm etc..
> >> >
> >> > Some more details here:
> >> > https://hadoop.apache.org/docs/r2.7.2/hadoop-project-
> >> > dist/hadoop-common/Superusers.html
> >> >
> >> >
> >> > @Rajini
> >> >
> >> > Thanks for your comments on SASL/SCRAM usage. I am thinking to send
> >> > tokenHmac (salted-hashed version)
> >> > as password for authentication and tokenID for retrial of tokenHmac at
> >> > server side.
> >> > Does above sound OK?
> >> >
> >> >
> >> > Thanks,
> >> > Manikumar
> >> >
> >> > On Wed, Dec 14, 2016 at 10:33 PM, Harsha Chintalapani <
> kafka@harsha.io>
> >> > wrote:
> >> >
> >> > > @Gwen @Mani  Not sure why we want to authenticate at every request.
> >> Even
> >> > if
> >> > > the token exchange is cheap it still a few calls that need to go
> >> through
> >> > > round trip.  Impersonation doesn't require authentication for every
> >> > > request.
> >> > >
> >> > > "So a centralized app can create few producers, do the metadata
> >> request
> >> > and
> >> > > broker discovery with its own user auth, but then use delegation
> >> tokens
> >> > to
> >> > > allow performing produce/fetch requests as different users? Instead
> of
> >> > > having to re-connect for each impersonated user?"
> >> > >
> >> > > Yes. But what we will have is this centralized user as impersonation
> >> user
> >> > > on behalf of other users. When it authenticates initially we will
> >> create
> >> > a
> >> > > "Subject" and from there on wards centralized user can do
> >> > > Subject.doAsPrivileged
> >> > > on behalf, other users.
> >> > > On the server side, we can retrieve two principals out of this one
> is
> >> the
> >> > > authenticated user (centralized user) and another is impersonated
> >> user.
> >> > We
> >> > > will first check if the authenticated user allowed to impersonate
> and
> >> > then
> >> > > move on to check if the user Alice has access to the topic "X" to
> >> > > read/write.
> >> > >
> >> > > @Rajini Intention of this KIP is to support token auth via
> SASL/SCRAM,
> >> > not
> >> > > just with TLS.  What you raised is a good point let me take a look
> and
> >> > add
> >> > > details.
> >> > >
> >> > > It will be easier to add impersonation once we reach agreement on
> this
> >> > KIP.
> >> > >
> >> > >
> >> > > On Wed, Dec 14, 2016 at 5:51 AM Ismael Juma <is...@juma.me.uk>
> >> wrote:
> >> > >
> >> > > > Hi Rajini,
> >> > > >
> >> > > > I think it would definitely be valuable to have a KIP for
> >> > impersonation.
> >> > > >
> >> > > > Ismael
> >> > > >
> >> > > > On Wed, Dec 14, 2016 at 4:03 AM, Rajini Sivaram <
> >> rsivaram@pivotal.io>
> >> > > > wrote:
> >> > > >
> >> > > > > It would clearly be very useful to enable clients to send
> >> requests on
> >> > > > > behalf of multiple users. A separate KIP makes sense, but it may
> >> be
> >> > > worth
> >> > > > > thinking through some of the implications now, especially if the
> >> main
> >> > > > > interest in delegation tokens comes from its potential to enable
> >> > > > > impersonation.
> >> > > > >
> >> > > > > I understand that delegation tokens are only expected to be used
> >> with
> >> > > > TLS.
> >> > > > > But the choice of SASL/SCRAM for authentication must be based
> on a
> >> > > > > requirement to protect the tokenHmac - otherwise you could just
> >> use
> >> > > > > SASL/PLAIN. With SASL/SCRAM the tokenHmac is never propagated
> >> > > > on-the-wire,
> >> > > > > only a salted-hashed version of it is used in the SASL
> >> authentication
> >> > > > > exchange. If impersonation is based on sending tokenHmac in
> >> requests,
> >> > > any
> >> > > > > benefit of using SCRAM is lost.
> >> > > > >
> >> > > > > An alternative may be to allow clients to authenticate multiple
> >> times
> >> > > > using
> >> > > > > SASL and include one of its authenticated principals in each
> >> request
> >> > > > > (optionally). I haven't thought it through yet, obviously. But
> if
> >> the
> >> > > > > approach is of interest and no one is working on a KIP for
> >> > > impersonation
> >> > > > at
> >> > > > > the moment, I am happy to write one. It may provide something
> for
> >> > > > > comparison at least.
> >> > > > >
> >> > > > > Thoughts?
> >> > > > >
> >> > > > >
> >> > > > > On Wed, Dec 14, 2016 at 9:53 AM, Manikumar <
> >> > manikumar.reddy@gmail.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > That's a good idea. Authenticating every request with
> delegation
> >> > > token
> >> > > > > will
> >> > > > > > be useful for
> >> > > > > > impersonation use-cases. But as of now, we are thinking
> >> delegation
> >> > > > token
> >> > > > > as
> >> > > > > > just another way
> >> > > > > > to authenticate the users. We haven't think through all the
> use
> >> > cases
> >> > > > > > related to
> >> > > > > > impersonation or using delegation token for impersonation. We
> >> want
> >> > to
> >> > > > > > handle impersonation
> >> > > > > > (KAFKA-3712) as part of separate KIP.
> >> > > > > >
> >> > > > > > Will that be Ok?
> >> > > > > >
> >> > > > > >
> >> > > > > > On Wed, Dec 14, 2016 at 8:09 AM, Gwen Shapira <
> >> gwen@confluent.io>
> >> > > > wrote:
> >> > > > > >
> >> > > > > > > Thinking out loud here:
> >> > > > > > >
> >> > > > > > > It looks like authentication with a delegation token is
> going
> >> to
> >> > be
> >> > > > > > > super-cheap, right? We just compare the token to a value in
> >> the
> >> > > > broker
> >> > > > > > > cache?
> >> > > > > > >
> >> > > > > > > If I understood the KIP correctly, right now it suggests
> that
> >> > > > > > > authentication happens when establishing the client-broker
> >> > > connection
> >> > > > > (as
> >> > > > > > > normal for Kafka. But perhaps we want to consider
> >> authenticating
> >> > > > every
> >> > > > > > > request with delegation token (if exists)?
> >> > > > > > >
> >> > > > > > > So a centralized app can create few producers, do the
> metadata
> >> > > > request
> >> > > > > > and
> >> > > > > > > broker discovery with its own user auth, but then use
> >> delegation
> >> > > > tokens
> >> > > > > > to
> >> > > > > > > allow performing produce/fetch requests as different users?
> >> > Instead
> >> > > > of
> >> > > > > > > having to re-connect for each impersonated user?
> >> > > > > > >
> >> > > > > > > This may over-complicate things quite a bit (basically
> adding
> >> > extra
> >> > > > > > > information in every request), but maybe it will be useful
> for
> >> > > > > > > impersonation use-cases (which seem to drive much of the
> >> interest
> >> > > in
> >> > > > > this
> >> > > > > > > KIP)?
> >> > > > > > > Kafka Connect, NiFi and friends can probably use this to
> share
> >> > > > clients
> >> > > > > > > between multiple jobs, tasks, etc.
> >> > > > > > >
> >> > > > > > > What do you think?
> >> > > > > > >
> >> > > > > > > Gwen
> >> > > > > > >
> >> > > > > > > On Tue, Dec 13, 2016 at 12:43 AM, Manikumar <
> >> > > > manikumar.reddy@gmail.com
> >> > > > > >
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Ashish,
> >> > > > > > > >
> >> > > > > > > > Thank you for reviewing the KIP.  Please see the replies
> >> > inline.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > > 1. How to disable delegation token authentication?
> >> > > > > > > > >
> >> > > > > > > > > This can be achieved in various ways, however I think
> >> reusing
> >> > > > > > > delegation
> >> > > > > > > > > token secret config for this makes sense here. Avoids
> >> > creating
> >> > > > yet
> >> > > > > > > > another
> >> > > > > > > > > config and forces delegation token users to consciously
> >> set
> >> > the
> >> > > > > > secret.
> >> > > > > > > > If
> >> > > > > > > > > the secret is not set or set to empty string, brokers
> >> should
> >> > > turn
> >> > > > > off
> >> > > > > > > > > delegation token support. This will however require a
> new
> >> > error
> >> > > > > code
> >> > > > > > to
> >> > > > > > > > > indicate delegation token support is turned off on
> broker.
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > >   Thanks for the suggestion. Option to turnoff delegation
> >> token
> >> > > > > > > > authentication will be useful.
> >> > > > > > > >   I'll update the KIP.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > 2. ACLs on delegation token?
> >> > > > > > > > >
> >> > > > > > > > > Do we need to have ACLs defined for tokens? I do not
> >> think it
> >> > > > buys
> >> > > > > us
> >> > > > > > > > > anything, as delegation token can be treated as
> >> impersonation
> >> > > of
> >> > > > > the
> >> > > > > > > > owner.
> >> > > > > > > > > Any thing the owner has permission to do, delegation
> >> tokens
> >> > > > should
> >> > > > > be
> >> > > > > > > > > allowed to do as well. If so, we probably won't need to
> >> > return
> >> > > > > > > > > authorization exception error code while creating
> >> delegation
> >> > > > token.
> >> > > > > > It
> >> > > > > > > > > however would make sense to check renew and expire
> >> requests
> >> > are
> >> > > > > > coming
> >> > > > > > > > from
> >> > > > > > > > > owner or renewers of the token, but that does not
> require
> >> > > > explicit
> >> > > > > > > acls.
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Yes, We agreed to not have new acl on who can request
> >> > delegation
> >> > > > > token.
> >> > > > > > > >  I'll update the KIP.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > 3. How to restrict max life time of a token?
> >> > > > > > > > >
> >> > > > > > > > > Admins might want to restrict max life time of tokens
> >> created
> >> > > on
> >> > > > a
> >> > > > > > > > cluster,
> >> > > > > > > > > and this can very from cluster to cluster based on
> >> use-cases.
> >> > > > This
> >> > > > > > > might
> >> > > > > > > > > warrant a separate broker config.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > Currently we  have "delegation.token.max.lifetime.sec"
> >> server
> >> > > > config
> >> > > > > > > > property
> >> > > > > > > > May be we can take min(User supplied MaxTime, Server
> >> MaxTime)
> >> > as
> >> > > > max
> >> > > > > > life
> >> > > > > > > > time.
> >> > > > > > > > I am open to add new config property.
> >> > > > > > > >
> >> > > > > > > > Few more comments based on recent KIP update.
> >> > > > > > > > >
> >> > > > > > > > > 1. Do we need a separate {{InvalidateTokenRequest}}?
> >> Can't we
> >> > > use
> >> > > > > > > > > {{ExpireTokenRequest}} with with expiryDate set to
> >> anything
> >> > > > before
> >> > > > > > > > current
> >> > > > > > > > > date?
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > > makes sense. we don't need special request to cancel the
> >> token.
> >> > > We
> >> > > > > can
> >> > > > > > > use
> >> > > > > > > > ExpireTokenRequest.
> >> > > > > > > > I'll update the KIP.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > > 2. Can we change time field names to indicate their unit
> >> is
> >> > > > > > > milliseconds,
> >> > > > > > > > > like, IssueDateMs, ExpiryDateMs, etc.?
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >   Done.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > > 3. Can we allow users to renew a token for a specified
> >> amount
> >> > > of
> >> > > > > > time?
> >> > > > > > > In
> >> > > > > > > > > current version of KIP, renew request does not take time
> >> as a
> >> > > > > param,
> >> > > > > > > not
> >> > > > > > > > > sure what is expiry time set to after renewal.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >  Yes, we need to specify renew period.  I'll update the
> KIP.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > > Mankumar
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Mon, Dec 12, 2016 at 9:08 AM Manikumar <
> >> > > > > manikumar.reddy@gmail.com
> >> > > > > > >
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi,
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > I would like to reinitiate the discussion on
> Delegation
> >> > token
> >> > > > > > support
> >> > > > > > > > for
> >> > > > > > > > > >
> >> > > > > > > > > > Kafka.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Brief summary of the past discussion:
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > 1) Broker stores delegation tokens in zookeeper.  All
> >> > brokers
> >> > > > > will
> >> > > > > > > > have a
> >> > > > > > > > > >
> >> > > > > > > > > > cache backed by
> >> > > > > > > > > >
> >> > > > > > > > > >    zookeeper so they will all get notified whenever a
> >> new
> >> > > token
> >> > > > > is
> >> > > > > > > > > >
> >> > > > > > > > > > generated and they will
> >> > > > > > > > > >
> >> > > > > > > > > >    update their local cache whenever token state
> >> changes.
> >> > > > > > > > > >
> >> > > > > > > > > > 2) The current proposal does not support rotation of
> >> secret
> >> > > > > > > > > >
> >> > > > > > > > > > 3) Only allow the renewal by users that authenticated
> >> using
> >> > > > *non*
> >> > > > > > > > > >
> >> > > > > > > > > > delegation token mechanism
> >> > > > > > > > > >
> >> > > > > > > > > > 4) KIP-84 proposes to support  SASL SCRAM mechanisms.
> >> Kafka
> >> > > > > clients
> >> > > > > > > can
> >> > > > > > > > > >
> >> > > > > > > > > > authenticate using
> >> > > > > > > > > >
> >> > > > > > > > > >    SCRAM-SHA-256, providing the delegation token HMAC
> as
> >> > > > > password.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Updated the KIP with the following:
> >> > > > > > > > > >
> >> > > > > > > > > > 1. Protocol and Config changes
> >> > > > > > > > > >
> >> > > > > > > > > > 2. format of the data stored in ZK.
> >> > > > > > > > > >
> >> > > > > > > > > > 3. Changes to Java Clients/Usage of SASL SCRAM
> mechanism
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > https://cwiki.apache.org/confl
> uence/display/KAFKA/KIP-
> >> > > > > > > > > >
> >> > > > > > > > > > 48+Delegation+token+support+for+Kafka
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Jun, Ashish, Gwen,
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Pl review the updated KIP.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks
> >> > > > > > > > > >
> >> > > > > > > > > > Manikumar
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > On Thu, Sep 29, 2016 at 9:56 PM, Ashish Singh <
> >> > > > > asingh@cloudera.com
> >> > > > > > >
> >> > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > Harsha/ Gwen,
> >> > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > How do we proceed here? I am willing to help out
> with
> >> > here.
> >> > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > On Fri, Sep 23, 2016 at 11:41 AM, Gwen Shapira <
> >> > > > > > gwen@confluent.io>
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > Is it updated? are all concerns addressed? do you
> >> want
> >> > to
> >> > > > > > start a
> >> > > > > > > > > vote?
> >> > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > Sorry for being pushy, I do appreciate that we are
> >> all
> >> > > > > > volunteers
> >> > > > > > > > and
> >> > > > > > > > > >
> >> > > > > > > > > > > > finding time is difficult. This feature is
> important
> >> > for
> >> > > > > > anything
> >> > > > > > > > > that
> >> > > > > > > > > >
> >> > > > > > > > > > > > integrates with Kafka (stream processors, Flume,
> >> NiFi,
> >> > > etc)
> >> > > > > > and I
> >> > > > > > > > > >
> >> > > > > > > > > > > > don't want to see this getting stuck because we
> lack
> >> > > > > > coordination
> >> > > > > > > > > >
> >> > > > > > > > > > > > within the community.
> >> > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > On Thu, Sep 15, 2016 at 6:39 PM, Harsha
> >> Chintalapani <
> >> > > > > > > > > kafka@harsha.io>
> >> > > > > > > > > >
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > > The only pending update for the KIP is to write
> up
> >> > the
> >> > > > > > protocol
> >> > > > > > > > > > changes
> >> > > > > > > > > >
> >> > > > > > > > > > > > like
> >> > > > > > > > > >
> >> > > > > > > > > > > > > we've it KIP-4. I'll update the wiki.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > > On Thu, Sep 15, 2016 at 4:27 PM Ashish Singh <
> >> > > > > > > > asingh@cloudera.com>
> >> > > > > > > > > >
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> I think we decided to not support secret
> >> rotation, I
> >> > > > guess
> >> > > > > > > this
> >> > > > > > > > > can
> >> > > > > > > > > > be
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> stated clearly on the KIP. Also, more details
> on
> >> how
> >> > > > > clients
> >> > > > > > > > will
> >> > > > > > > > > >
> >> > > > > > > > > > > > perform
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> token distribution and how CLI will look like
> >> will
> >> > be
> >> > > > > > helpful.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <
> >> > > > > > > > gwen@confluent.io>
> >> > > > > > > > > >
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > Hi Guys,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > This discussion was dead for a while. Are
> there
> >> > > still
> >> > > > > > > > > contentious
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > points? If not, why are there no votes?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <
> >> > > > > > jun@confluent.io>
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > > Ashish,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > > Yes, I will send out a KIP invite for next
> >> week
> >> > to
> >> > > > > > discuss
> >> > > > > > > > > > KIP-48
> >> > > > > > > > > >
> >> > > > > > > > > > > > and
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > other
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > > remaining KIPs.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > > Thanks,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > > Jun
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > > On Tue, Aug 23, 2016 at 1:22 PM, Ashish
> >> Singh <
> >> > > > > > > > > >
> >> > > > > > > > > > > asingh@cloudera.com>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >> Thanks Harsha!
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >> Jun, can we add KIP-48 to next KIP
> hangout's
> >> > > > agenda.
> >> > > > > > > Also,
> >> > > > > > > > we
> >> > > > > > > > > > did
> >> > > > > > > > > >
> >> > > > > > > > > > > > not
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >> actually make a call on when we should
> have
> >> > next
> >> > > > KIP
> >> > > > > > > call.
> >> > > > > > > > As
> >> > > > > > > > > >
> >> > > > > > > > > > > there
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >> are
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > a
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >> few outstanding KIPs that could not be
> >> > discussed
> >> > > > this
> >> > > > > > > week,
> >> > > > > > > > > can
> >> > > > > > > > > >
> >> > > > > > > > > > > we
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >> have
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > a
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >> KIP hangout call next week?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha
> >> > > > Chintalapani
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >> <ka...@harsha.io>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >> wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> Ashish,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>>         Yes we are working on it. Lets
> >> discuss
> >> > > in
> >> > > > > the
> >> > > > > > > next
> >> > > > > > > > > KIP
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> meeting.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> I'll join.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> -Harsha
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish
> >> Singh
> >> > <
> >> > > > > > > > > >
> >> > > > > > > > > > > > asingh@cloudera.com>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > Hello Harsha,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > Are you still working on this?
> Wondering
> >> if
> >> > we
> >> > > > can
> >> > > > > > > > discuss
> >> > > > > > > > > >
> >> > > > > > > > > > > this
> >> > > > > > > > > >
> >> > > > > > > > > > > > in
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > next
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> KIP
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > meeting, if you can join.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha
> >> > > > > > Chintalapani <
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > kafka@harsha.io>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > Hi Grant,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > >           We are working on it. Will
> >> add
> >> > the
> >> > > > > > details
> >> > > > > > > > to
> >> > > > > > > > > > KIP
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > about
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > the
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > request protocol.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > Thanks,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > Harsha
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant
> >> > Henke
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > <gh...@cloudera.com>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > Hi Parth,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > Are you still working on this? If
> you
> >> > need
> >> > > > any
> >> > > > > > > help
> >> > > > > > > > > > please
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > don't
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > hesitate
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > to ask.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > Thanks,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > Grant
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > On Thu, Jun 30, 2016 at 4:35 PM,
> Jun
> >> > Rao <
> >> > > > > > > > > >
> >> > > > > > > > > > > jun@confluent.io>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > Parth,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > Thanks for the reply.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > It makes sense to only allow the
> >> > renewal
> >> > > > by
> >> > > > > > > users
> >> > > > > > > > > that
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> authenticated
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > using
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > *non* delegation token mechanism.
> >> > Then,
> >> > > > > should
> >> > > > > > > we
> >> > > > > > > > > make
> >> > > > > > > > > >
> >> > > > > > > > > > > the
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> renewal a
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > list?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > For example, in the case of rest
> >> > proxy,
> >> > > it
> >> > > > > > will
> >> > > > > > > be
> >> > > > > > > > > >
> >> > > > > > > > > > > useful
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > for
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> every
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > instance of rest proxy to be able
> >> to
> >> > > renew
> >> > > > > the
> >> > > > > > > > > tokens.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > It would be clearer if we can
> >> document
> >> > > the
> >> > > > > > > request
> >> > > > > > > > > >
> >> > > > > > > > > > > > protocol
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > like
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > https://cwiki.apache.org/confl
> >> > > > > > > > > uence/display/KAFKA/KIP-
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > 4+-+Command+line+and+
> >> > > > > centralized+administrative+
> >> > > > > > > > > >
> >> > > > > > > > > > > > operations#KIP-4-
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > Commandlineandcentralizedadmin
> >> > > > > > istrativeoperations-
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > CreateTopicsRequest(KAFKA-
> >> > > > > > > > 2945):(VotedandPlannedforin0.
> >> > > > > > > > > >
> >> > > > > > > > > > > > 10.1.0)
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > .
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > It would also be useful to
> document
> >> > the
> >> > > > > client
> >> > > > > > > > APIs.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > Thanks,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > Jun
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM,
> >> parth
> >> > > > > > > brahmbhatt
> >> > > > > > > > <
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > brahmbhatt.parth@gmail.com>
> wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > Hi,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > I am suggesting that we will
> only
> >> > > allow
> >> > > > > the
> >> > > > > > > > > renewal
> >> > > > > > > > > > by
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > users
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> that
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > authenticated using *non*
> >> delegation
> >> > > > token
> >> > > > > > > > > > mechanism.
> >> > > > > > > > > >
> >> > > > > > > > > > > > For
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> example,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > If
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > user
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > Alice authenticated using
> >> kerberos
> >> > and
> >> > > > > > > requested
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > delegation
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> tokens,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > only
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > user Alice authenticated via
> non
> >> > > > > delegation
> >> > > > > > > > token
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > mechanism
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > can
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > renew.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > Clients that have  access to
> >> > > delegation
> >> > > > > > tokens
> >> > > > > > > > can
> >> > > > > > > > > > not
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > issue
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > renewal
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > request for renewing their own
> >> token
> >> > > and
> >> > > > > > this
> >> > > > > > > is
> >> > > > > > > > > >
> >> > > > > > > > > > > > primarily
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > important
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > to
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > reduce the time window for
> which
> >> a
> >> > > > > > compromised
> >> > > > > > > > > token
> >> > > > > > > > > >
> >> > > > > > > > > > > > will
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > be
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> valid.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > To clarify, Yes any
> authenticated
> >> > user
> >> > > > can
> >> > > > > > > > request
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > delegation
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > tokens
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > but
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > even here I would recommend to
> >> avoid
> >> > > > > > creating
> >> > > > > > > a
> >> > > > > > > > > > chain
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > where a
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > client
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > authenticated via delegation
> >> token
> >> > > > request
> >> > > > > > for
> >> > > > > > > > > more
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > delegation
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > tokens.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > Basically anyone can request
> >> > > delegation
> >> > > > > > token,
> >> > > > > > > > as
> >> > > > > > > > > > long
> >> > > > > > > > > >
> >> > > > > > > > > > > > as
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > they
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > authenticate
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > via a non delegation token
> >> > mechanism.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > Aren't classes listed here
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > <
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > https://cwiki.apache.org/confl
> >> > > > > > > > > uence/display/KAFKA/KIP-
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > 48+Delegation+token+support+fo
> >> > > > > > > > > >
> >> > > > > > > > > > > r+Kafka#KIP-48Delegationtokens
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> upportforKaf
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > ka-PublicInterfaces
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > sufficient?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > Thanks
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > Parth
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > On Tue, Jun 21, 2016 at 4:33
> PM,
> >> Jun
> >> > > Rao
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > <ju...@confluent.io>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > Parth,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > Thanks for the reply. A
> couple
> >> of
> >> > > > > comments
> >> > > > > > > > > inline
> >> > > > > > > > > >
> >> > > > > > > > > > > > below.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > On Tue, Jun 21, 2016 at 10:36
> >> AM,
> >> > > > parth
> >> > > > > > > > > > brahmbhatt <
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > brahmbhatt.parth@gmail.com>
> >> > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > 1. Who / how are tokens
> >> renewed?
> >> > > By
> >> > > > > > > original
> >> > > > > > > > > >
> >> > > > > > > > > > > > requester
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> only? or
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > using
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > Kerberos
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > auth only?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > My recommendation is to do
> >> this
> >> > > only
> >> > > > > > using
> >> > > > > > > > > >
> >> > > > > > > > > > > Kerberos
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > auth
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > and
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > only
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > threw
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > the
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > renewer specified during
> the
> >> > > > > acquisition
> >> > > > > > > > > > request.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > Hmm, not sure that I follow
> >> this.
> >> > > Are
> >> > > > > you
> >> > > > > > > > saying
> >> > > > > > > > > >
> >> > > > > > > > > > > that
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > any
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> client
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > authenticated with the
> >> delegation
> >> > > > token
> >> > > > > > can
> >> > > > > > > > > renew,
> >> > > > > > > > > >
> >> > > > > > > > > > > > i.e.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > there
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> is
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > no
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > renewer
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > needed?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > Also, just to be clear, any
> >> > > > > authenticated
> >> > > > > > > > client
> >> > > > > > > > > >
> >> > > > > > > > > > > > (either
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> through
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > SASL
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > or
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > SSL) can request a delegation
> >> > token
> >> > > > for
> >> > > > > > the
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > authenticated
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> user,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > right?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > 2. Are tokens stored on
> each
> >> > > broker
> >> > > > or
> >> > > > > > in
> >> > > > > > > > ZK?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > My recommendation is still
> to
> >> > > store
> >> > > > in
> >> > > > > > ZK
> >> > > > > > > or
> >> > > > > > > > > not
> >> > > > > > > > > >
> >> > > > > > > > > > > > store
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > them
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> at
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > all.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > The
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > whole controller based
> >> > > distribution
> >> > > > is
> >> > > > > > too
> >> > > > > > > > > much
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > overhead
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> with
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > not
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > much
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > to
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > achieve.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > 3. How are tokens
> >> invalidated /
> >> > > > > expired?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > Either by expiration time
> >> out or
> >> > > > > through
> >> > > > > > > an
> >> > > > > > > > > >
> >> > > > > > > > > > > explicit
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> request to
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > invalidate.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > 4. Which encryption
> >> algorithm is
> >> > > > used?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > SCRAM
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > 5. What is the
> impersonation
> >> > > > proposal
> >> > > > > > (it
> >> > > > > > > > > wasn't
> >> > > > > > > > > >
> >> > > > > > > > > > > in
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > the
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > KIP
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> but
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > was
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > discussed
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > in this thread)?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > There is no imperonation
> >> > > proposal. I
> >> > > > > > tried
> >> > > > > > > > and
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > explained
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > how
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > its
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > a
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > different problem and why
> its
> >> > not
> >> > > > > really
> >> > > > > > > > > > necessary
> >> > > > > > > > > >
> >> > > > > > > > > > > > to
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> discuss
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > that
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > as
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > part
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > of this KIP.  This KIP will
> >> not
> >> > > > > support
> >> > > > > > > any
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > impersonation,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> it
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > will
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > just
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > be
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > another way to
> authenticate.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > 6. Do we need new ACLs, if
> >> so -
> >> > > for
> >> > > > > what
> >> > > > > > > > > > actions?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > We do not need new ACLs.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > Could we document the format
> of
> >> > the
> >> > > > new
> >> > > > > > > > > >
> >> > > > > > > > > > > > request/response
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > and
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > their
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > associated Resource and
> >> Operation
> >> > > for
> >> > > > > ACL?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > 7. How would the delegation
> >> > token
> >> > > be
> >> > > > > > > > > configured
> >> > > > > > > > > > in
> >> > > > > > > > > >
> >> > > > > > > > > > > > the
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> client?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > Should be through config. I
> >> > wasn't
> >> > > > > > > planning
> >> > > > > > > > on
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > supporting
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> JAAS
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > for
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > tokens.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > I don't believe hadoop does
> >> this
> >> > > > > either.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > Thanks
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > Parth
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > On Thu, Jun 16, 2016 at
> 4:03
> >> PM,
> >> > > Jun
> >> > > > > > Rao <
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > jun@confluent.io>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > Harsha,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > Another question.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > 9. How would the
> delegation
> >> > > token
> >> > > > be
> >> > > > > > > > > > configured
> >> > > > > > > > > >
> >> > > > > > > > > > > in
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > the
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > client?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > The
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > standard
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > way is to do this through
> >> > JAAS.
> >> > > > > > However,
> >> > > > > > > > we
> >> > > > > > > > > > will
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > need
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > to
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > think
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > through
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > if
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > this is convenient in a
> >> shared
> >> > > > > > > > environment.
> >> > > > > > > > > > For
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > example,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > when a
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > new
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > task
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > is
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > added to a Storm worker
> >> node,
> >> > do
> >> > > > we
> >> > > > > > need
> >> > > > > > > > to
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > dynamically
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> add a
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > new
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > section
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > in the JAAS file? It may
> be
> >> > more
> >> > > > > > > > convenient
> >> > > > > > > > > if
> >> > > > > > > > > >
> >> > > > > > > > > > > we
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > can
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> pass in
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > the
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > token
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > through the config
> directly
> >> > w/o
> >> > > > > going
> >> > > > > > > > > through
> >> > > > > > > > > >
> >> > > > > > > > > > > > JAAS.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > Are you or Parth still
> >> > actively
> >> > > > > > working
> >> > > > > > > on
> >> > > > > > > > > > this
> >> > > > > > > > > >
> >> > > > > > > > > > > > KIP?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > Thanks,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > Jun
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > On Sun, Jun 12, 2016 at
> >> 2:18
> >> > PM,
> >> > > > Jun
> >> > > > > > > Rao <
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> jun@confluent.io>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > > Just to add on that
> list.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > > 2. It would be good to
> >> > > document
> >> > > > > the
> >> > > > > > > > format
> >> > > > > > > > > > of
> >> > > > > > > > > >
> >> > > > > > > > > > > > the
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > data
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > stored
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > in
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > ZK.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > > 7. Earlier, there was a
> >> > > > discussion
> >> > > > > > on
> >> > > > > > > > > > whether
> >> > > > > > > > > >
> >> > > > > > > > > > > > the
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > tokens
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > should
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > be
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > > propagated through ZK
> >> like
> >> > > > > > > > > config/acl/quota,
> >> > > > > > > > > >
> >> > > > > > > > > > > or
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > through
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> the
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > controller.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > > Currently, the
> >> controller is
> >> > > > only
> >> > > > > > > > designed
> >> > > > > > > > > > for
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> propagating
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > topic
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > metadata,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > > but not other data.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > > 8. Should we use SCRAM
> to
> >> > send
> >> > > > the
> >> > > > > > > token
> >> > > > > > > > > >
> >> > > > > > > > > > > instead
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > > of
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > DIGEST-MD5
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > since
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > it's
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > > deprecated?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > > Also, the images in the
> >> wiki
> >> > > > seem
> >> > > > > > > > broken.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > > Thanks,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > > Jun
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > > On Fri, Jun 10, 2016 at
> >> > 10:02
> >> > > > AM,
> >> > > > > > Gwen
> >> > > > > > > > > >
> >> > > > > > > > > > > Shapira <
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > gwen@confluent.io>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> From what I can see,
> >> > > remaining
> >> > > > > > > > questions
> >> > > > > > > > > > are:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> 1. Who / how are
> tokens
> >> > > > renewed?
> >> > > > > By
> >> > > > > > > > > > original
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > requester
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > only?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > or
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > using
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> Kerberos auth only?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> 2. Are tokens stored
> on
> >> > each
> >> > > > > broker
> >> > > > > > > or
> >> > > > > > > > in
> >> > > > > > > > > > ZK?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> 3. How are tokens
> >> > > invalidated /
> >> > > > > > > > expired?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> 4. Which encryption
> >> > algorithm
> >> > > > is
> >> > > > > > > used?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> 5. What is the
> >> > impersonation
> >> > > > > > proposal
> >> > > > > > > > (it
> >> > > > > > > > > >
> >> > > > > > > > > > > > wasn't
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> in
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > the
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > KIP
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > but
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > was
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> discussed in this
> >> thread)?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> 6. Do we need new
> ACLs,
> >> if
> >> > > so -
> >> > > > > for
> >> > > > > > > > what
> >> > > > > > > > > >
> >> > > > > > > > > > > > actions?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> Gwen
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> On Thu, Jun 9, 2016 at
> >> 7:48
> >> > > PM,
> >> > > > > > > Harsha
> >> > > > > > > > <
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> kafka@harsha.io>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> > Jun & Ismael,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
> >> > > > > > > > Unfortunately
> >> > > > > > > > > I
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> > couldn't
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> attend
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > the
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > KIP
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > meeting
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
> >> > > when
> >> > > > > > > > > delegation
> >> > > > > > > > > >
> >> > > > > > > > > > > > tokens
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > discussed.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > Appreciate
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > if
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
> >> > > you
> >> > > > > can
> >> > > > > > > > update
> >> > > > > > > > > > the
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > thread if
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > you
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > have
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > any
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
> >> > > > > further
> >> > > > > > > > > > questions.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> > Thanks,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> > Harsha
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> > On Tue, May 24,
> 2016,
> >> at
> >> > > > 11:32
> >> > > > > > AM,
> >> > > > > > > > > Liquan
> >> > > > > > > > > >
> >> > > > > > > > > > > Pei
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> It seems that the
> >> links
> >> > to
> >> > > > > > images
> >> > > > > > > in
> >> > > > > > > > > the
> >> > > > > > > > > >
> >> > > > > > > > > > > KIP
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> are
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > broken.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> Liquan
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> On Tue, May 24,
> 2016
> >> at
> >> > > 9:33
> >> > > > > AM,
> >> > > > > > > > parth
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > brahmbhatt <
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >>
> >> > > brahmbhatt.parth@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >>
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > 110. What does
> >> > > > > > > > getDelegationTokenAs
> >> > > > > > > > > >
> >> > > > > > > > > > > mean?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > In the current
> >> > proposal
> >> > > we
> >> > > > > > only
> >> > > > > > > > > allow
> >> > > > > > > > > > a
> >> > > > > > > > > >
> >> > > > > > > > > > > > user
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > to
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> get
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > delegation
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > token
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> for
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > the identity that
> >> it
> >> > > > > > > authenticated
> >> > > > > > > > > as
> >> > > > > > > > > >
> >> > > > > > > > > > > > using
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> another
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > mechanism,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > i.e.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> A user
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > that authenticate
> >> > using
> >> > > a
> >> > > > > > keytab
> >> > > > > > > > for
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > principal
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > user1@EXAMPLE.COM
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> will get
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens
> >> for
> >> > > that
> >> > > > > > user
> >> > > > > > > > > only.
> >> > > > > > > > > > In
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > future I
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > think
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > we
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > will
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > have
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> to
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > extend support
> such
> >> > that
> >> > > > we
> >> > > > > > > allow
> >> > > > > > > > > some
> >> > > > > > > > > >
> >> > > > > > > > > > > set
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > of
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> users (
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> >
> >> > > > kafka-rest-user@EXAMPLE.COM
> >> > > > > ,
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > storm-nimbus@EXAMPLE.COM)
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > to
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > acquire
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens
> >> on
> >> > > > behalf
> >> > > > > of
> >> > > > > > > > other
> >> > > > > > > > > >
> >> > > > > > > > > > > users
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > whose
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > identity
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > they
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > have
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > verified
> >> > independently.
> >> > > > > Kafka
> >> > > > > > > > > brokers
> >> > > > > > > > > >
> >> > > > > > > > > > > > will
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > have
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> ACLs
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > to
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > control
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> which
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > users are allowed
> >> to
> >> > > > > > impersonate
> >> > > > > > > > > other
> >> > > > > > > > > >
> >> > > > > > > > > > > > users
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > and
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> get
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > tokens
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > on
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> behalf of
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > them. Overall
> >> > > > Impersonation
> >> > > > > > is a
> >> > > > > > > > > whole
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > different
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > problem
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > in
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > my
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> opinion and
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > I think we can
> >> tackle
> >> > it
> >> > > > in
> >> > > > > > > > separate
> >> > > > > > > > > >
> >> > > > > > > > > > > KIP.
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> >
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > 111. What's the
> >> > typical
> >> > > > rate
> >> > > > > > of
> >> > > > > > > > > > getting
> >> > > > > > > > > >
> >> > > > > > > > > > > > and
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> renewing
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > delegation
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> tokens?
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > Typically this
> >> should
> >> > be
> >> > > > > very
> >> > > > > > > very
> >> > > > > > > > > > low,
> >> > > > > > > > > >
> >> > > > > > > > > > > 1
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > request
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> per
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > minute
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > is a
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > relatively high
> >> > > estimate.
> >> > > > > > > However
> >> > > > > > > > it
> >> > > > > > > > > >
> >> > > > > > > > > > > > depends
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > on
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> the
> >> > > > > > > > > >
> >> > > > > > > > > > > > >> > >>> > > token
> >> > > > > > > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Manikumar <ma...@gmail.com>]

Hi Devs,

If there are no more comments, I will start vote on this KIP later this
week.


Thanks

On Fri, Dec 16, 2016 at 12:28 PM, Manikumar <ma...@gmail.com>
wrote:

> Hi,
>
>
>> Can you add a sample Jaas configuration using delegation tokens to the
>> KIP?
>>
>
> Will add sample Jaas configuration.
>
>
>> To make sure I have understood correctly, KAFKA-3712 is aimed at enabling
>> a
>> superuser to impersonate another (single) user, say alice. A producer
>> using
>> impersonation will authenticate with superuser credentials. All requests
>> from the producer will be run with the principal alice. But alice is not
>> involved in the authentication and alice's credentials are not actually
>> provided to the broker?
>>
>>
>  Yes, this matches with my understanding of impersonation work . Even in
> this approach
>  we have to handle all producer related issues mentioned by you. Yes, this
> is big change
>  and can be implemented if there is a pressing need. I hope we are all in
> agreement, that
>  this can be done in a separate KIP.
>
>
>  I request others give any suggestions/concerns on this KIP.
>
>
> Thanks,
>
>
>
>>
>> On Thu, Dec 15, 2016 at 9:04 AM, Manikumar <ma...@gmail.com>
>> wrote:
>>
>> > @Gwen, @Rajini,
>> >
>> > As mentioned in the KIP, main motivation for this KIP is to reduce load
>> on
>> > Kerberos
>> > server on large kafka deployments with large number of clients.
>> >
>> > Also it looks like we are combining two overlapping concepts
>> > 1. Single client sending requests with multiple users/authentications
>> > 2. Impersonation
>> >
>> > Option 1, is definitely useful in some use cases and can be used to
>> > implement workaround for
>> > impersonation
>> >
>> > In Impersonation, a super user can send requests on behalf of another
>> > user(Alice) in a secured way.
>> > superuser has credentials but user Alice doesn't have any. The requests
>> are
>> > required
>> > to run as user Alice and accesses/ACLs on Broker are required to be
>> done as
>> > user Alice.
>> > It is required that user Alice can connect to the Broker on a connection
>> > authenticated with
>> > superuser's credentials. In other words superuser is impersonating the
>> user
>> > Alice.
>> >
>> > The approach mentioned by Harsha in previous mail is implemented in
>> hadoop,
>> > storm etc..
>> >
>> > Some more details here:
>> > https://hadoop.apache.org/docs/r2.7.2/hadoop-project-
>> > dist/hadoop-common/Superusers.html
>> >
>> >
>> > @Rajini
>> >
>> > Thanks for your comments on SASL/SCRAM usage. I am thinking to send
>> > tokenHmac (salted-hashed version)
>> > as password for authentication and tokenID for retrial of tokenHmac at
>> > server side.
>> > Does above sound OK?
>> >
>> >
>> > Thanks,
>> > Manikumar
>> >
>> > On Wed, Dec 14, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
>> > wrote:
>> >
>> > > @Gwen @Mani  Not sure why we want to authenticate at every request.
>> Even
>> > if
>> > > the token exchange is cheap it still a few calls that need to go
>> through
>> > > round trip.  Impersonation doesn't require authentication for every
>> > > request.
>> > >
>> > > "So a centralized app can create few producers, do the metadata
>> request
>> > and
>> > > broker discovery with its own user auth, but then use delegation
>> tokens
>> > to
>> > > allow performing produce/fetch requests as different users? Instead of
>> > > having to re-connect for each impersonated user?"
>> > >
>> > > Yes. But what we will have is this centralized user as impersonation
>> user
>> > > on behalf of other users. When it authenticates initially we will
>> create
>> > a
>> > > "Subject" and from there on wards centralized user can do
>> > > Subject.doAsPrivileged
>> > > on behalf, other users.
>> > > On the server side, we can retrieve two principals out of this one is
>> the
>> > > authenticated user (centralized user) and another is impersonated
>> user.
>> > We
>> > > will first check if the authenticated user allowed to impersonate and
>> > then
>> > > move on to check if the user Alice has access to the topic "X" to
>> > > read/write.
>> > >
>> > > @Rajini Intention of this KIP is to support token auth via SASL/SCRAM,
>> > not
>> > > just with TLS.  What you raised is a good point let me take a look and
>> > add
>> > > details.
>> > >
>> > > It will be easier to add impersonation once we reach agreement on this
>> > KIP.
>> > >
>> > >
>> > > On Wed, Dec 14, 2016 at 5:51 AM Ismael Juma <is...@juma.me.uk>
>> wrote:
>> > >
>> > > > Hi Rajini,
>> > > >
>> > > > I think it would definitely be valuable to have a KIP for
>> > impersonation.
>> > > >
>> > > > Ismael
>> > > >
>> > > > On Wed, Dec 14, 2016 at 4:03 AM, Rajini Sivaram <
>> rsivaram@pivotal.io>
>> > > > wrote:
>> > > >
>> > > > > It would clearly be very useful to enable clients to send
>> requests on
>> > > > > behalf of multiple users. A separate KIP makes sense, but it may
>> be
>> > > worth
>> > > > > thinking through some of the implications now, especially if the
>> main
>> > > > > interest in delegation tokens comes from its potential to enable
>> > > > > impersonation.
>> > > > >
>> > > > > I understand that delegation tokens are only expected to be used
>> with
>> > > > TLS.
>> > > > > But the choice of SASL/SCRAM for authentication must be based on a
>> > > > > requirement to protect the tokenHmac - otherwise you could just
>> use
>> > > > > SASL/PLAIN. With SASL/SCRAM the tokenHmac is never propagated
>> > > > on-the-wire,
>> > > > > only a salted-hashed version of it is used in the SASL
>> authentication
>> > > > > exchange. If impersonation is based on sending tokenHmac in
>> requests,
>> > > any
>> > > > > benefit of using SCRAM is lost.
>> > > > >
>> > > > > An alternative may be to allow clients to authenticate multiple
>> times
>> > > > using
>> > > > > SASL and include one of its authenticated principals in each
>> request
>> > > > > (optionally). I haven't thought it through yet, obviously. But if
>> the
>> > > > > approach is of interest and no one is working on a KIP for
>> > > impersonation
>> > > > at
>> > > > > the moment, I am happy to write one. It may provide something for
>> > > > > comparison at least.
>> > > > >
>> > > > > Thoughts?
>> > > > >
>> > > > >
>> > > > > On Wed, Dec 14, 2016 at 9:53 AM, Manikumar <
>> > manikumar.reddy@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > That's a good idea. Authenticating every request with delegation
>> > > token
>> > > > > will
>> > > > > > be useful for
>> > > > > > impersonation use-cases. But as of now, we are thinking
>> delegation
>> > > > token
>> > > > > as
>> > > > > > just another way
>> > > > > > to authenticate the users. We haven't think through all the use
>> > cases
>> > > > > > related to
>> > > > > > impersonation or using delegation token for impersonation. We
>> want
>> > to
>> > > > > > handle impersonation
>> > > > > > (KAFKA-3712) as part of separate KIP.
>> > > > > >
>> > > > > > Will that be Ok?
>> > > > > >
>> > > > > >
>> > > > > > On Wed, Dec 14, 2016 at 8:09 AM, Gwen Shapira <
>> gwen@confluent.io>
>> > > > wrote:
>> > > > > >
>> > > > > > > Thinking out loud here:
>> > > > > > >
>> > > > > > > It looks like authentication with a delegation token is going
>> to
>> > be
>> > > > > > > super-cheap, right? We just compare the token to a value in
>> the
>> > > > broker
>> > > > > > > cache?
>> > > > > > >
>> > > > > > > If I understood the KIP correctly, right now it suggests that
>> > > > > > > authentication happens when establishing the client-broker
>> > > connection
>> > > > > (as
>> > > > > > > normal for Kafka. But perhaps we want to consider
>> authenticating
>> > > > every
>> > > > > > > request with delegation token (if exists)?
>> > > > > > >
>> > > > > > > So a centralized app can create few producers, do the metadata
>> > > > request
>> > > > > > and
>> > > > > > > broker discovery with its own user auth, but then use
>> delegation
>> > > > tokens
>> > > > > > to
>> > > > > > > allow performing produce/fetch requests as different users?
>> > Instead
>> > > > of
>> > > > > > > having to re-connect for each impersonated user?
>> > > > > > >
>> > > > > > > This may over-complicate things quite a bit (basically adding
>> > extra
>> > > > > > > information in every request), but maybe it will be useful for
>> > > > > > > impersonation use-cases (which seem to drive much of the
>> interest
>> > > in
>> > > > > this
>> > > > > > > KIP)?
>> > > > > > > Kafka Connect, NiFi and friends can probably use this to share
>> > > > clients
>> > > > > > > between multiple jobs, tasks, etc.
>> > > > > > >
>> > > > > > > What do you think?
>> > > > > > >
>> > > > > > > Gwen
>> > > > > > >
>> > > > > > > On Tue, Dec 13, 2016 at 12:43 AM, Manikumar <
>> > > > manikumar.reddy@gmail.com
>> > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Ashish,
>> > > > > > > >
>> > > > > > > > Thank you for reviewing the KIP.  Please see the replies
>> > inline.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > > 1. How to disable delegation token authentication?
>> > > > > > > > >
>> > > > > > > > > This can be achieved in various ways, however I think
>> reusing
>> > > > > > > delegation
>> > > > > > > > > token secret config for this makes sense here. Avoids
>> > creating
>> > > > yet
>> > > > > > > > another
>> > > > > > > > > config and forces delegation token users to consciously
>> set
>> > the
>> > > > > > secret.
>> > > > > > > > If
>> > > > > > > > > the secret is not set or set to empty string, brokers
>> should
>> > > turn
>> > > > > off
>> > > > > > > > > delegation token support. This will however require a new
>> > error
>> > > > > code
>> > > > > > to
>> > > > > > > > > indicate delegation token support is turned off on broker.
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >   Thanks for the suggestion. Option to turnoff delegation
>> token
>> > > > > > > > authentication will be useful.
>> > > > > > > >   I'll update the KIP.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > >
>> > > > > > > > > 2. ACLs on delegation token?
>> > > > > > > > >
>> > > > > > > > > Do we need to have ACLs defined for tokens? I do not
>> think it
>> > > > buys
>> > > > > us
>> > > > > > > > > anything, as delegation token can be treated as
>> impersonation
>> > > of
>> > > > > the
>> > > > > > > > owner.
>> > > > > > > > > Any thing the owner has permission to do, delegation
>> tokens
>> > > > should
>> > > > > be
>> > > > > > > > > allowed to do as well. If so, we probably won't need to
>> > return
>> > > > > > > > > authorization exception error code while creating
>> delegation
>> > > > token.
>> > > > > > It
>> > > > > > > > > however would make sense to check renew and expire
>> requests
>> > are
>> > > > > > coming
>> > > > > > > > from
>> > > > > > > > > owner or renewers of the token, but that does not require
>> > > > explicit
>> > > > > > > acls.
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Yes, We agreed to not have new acl on who can request
>> > delegation
>> > > > > token.
>> > > > > > > >  I'll update the KIP.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > >
>> > > > > > > > > 3. How to restrict max life time of a token?
>> > > > > > > > >
>> > > > > > > > > Admins might want to restrict max life time of tokens
>> created
>> > > on
>> > > > a
>> > > > > > > > cluster,
>> > > > > > > > > and this can very from cluster to cluster based on
>> use-cases.
>> > > > This
>> > > > > > > might
>> > > > > > > > > warrant a separate broker config.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > Currently we  have "delegation.token.max.lifetime.sec"
>> server
>> > > > config
>> > > > > > > > property
>> > > > > > > > May be we can take min(User supplied MaxTime, Server
>> MaxTime)
>> > as
>> > > > max
>> > > > > > life
>> > > > > > > > time.
>> > > > > > > > I am open to add new config property.
>> > > > > > > >
>> > > > > > > > Few more comments based on recent KIP update.
>> > > > > > > > >
>> > > > > > > > > 1. Do we need a separate {{InvalidateTokenRequest}}?
>> Can't we
>> > > use
>> > > > > > > > > {{ExpireTokenRequest}} with with expiryDate set to
>> anything
>> > > > before
>> > > > > > > > current
>> > > > > > > > > date?
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > > makes sense. we don't need special request to cancel the
>> token.
>> > > We
>> > > > > can
>> > > > > > > use
>> > > > > > > > ExpireTokenRequest.
>> > > > > > > > I'll update the KIP.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > > 2. Can we change time field names to indicate their unit
>> is
>> > > > > > > milliseconds,
>> > > > > > > > > like, IssueDateMs, ExpiryDateMs, etc.?
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >   Done.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > > 3. Can we allow users to renew a token for a specified
>> amount
>> > > of
>> > > > > > time?
>> > > > > > > In
>> > > > > > > > > current version of KIP, renew request does not take time
>> as a
>> > > > > param,
>> > > > > > > not
>> > > > > > > > > sure what is expiry time set to after renewal.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >  Yes, we need to specify renew period.  I'll update the KIP.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > Mankumar
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Mon, Dec 12, 2016 at 9:08 AM Manikumar <
>> > > > > manikumar.reddy@gmail.com
>> > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi,
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > I would like to reinitiate the discussion on Delegation
>> > token
>> > > > > > support
>> > > > > > > > for
>> > > > > > > > > >
>> > > > > > > > > > Kafka.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Brief summary of the past discussion:
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > 1) Broker stores delegation tokens in zookeeper.  All
>> > brokers
>> > > > > will
>> > > > > > > > have a
>> > > > > > > > > >
>> > > > > > > > > > cache backed by
>> > > > > > > > > >
>> > > > > > > > > >    zookeeper so they will all get notified whenever a
>> new
>> > > token
>> > > > > is
>> > > > > > > > > >
>> > > > > > > > > > generated and they will
>> > > > > > > > > >
>> > > > > > > > > >    update their local cache whenever token state
>> changes.
>> > > > > > > > > >
>> > > > > > > > > > 2) The current proposal does not support rotation of
>> secret
>> > > > > > > > > >
>> > > > > > > > > > 3) Only allow the renewal by users that authenticated
>> using
>> > > > *non*
>> > > > > > > > > >
>> > > > > > > > > > delegation token mechanism
>> > > > > > > > > >
>> > > > > > > > > > 4) KIP-84 proposes to support  SASL SCRAM mechanisms.
>> Kafka
>> > > > > clients
>> > > > > > > can
>> > > > > > > > > >
>> > > > > > > > > > authenticate using
>> > > > > > > > > >
>> > > > > > > > > >    SCRAM-SHA-256, providing the delegation token HMAC as
>> > > > > password.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Updated the KIP with the following:
>> > > > > > > > > >
>> > > > > > > > > > 1. Protocol and Config changes
>> > > > > > > > > >
>> > > > > > > > > > 2. format of the data stored in ZK.
>> > > > > > > > > >
>> > > > > > > > > > 3. Changes to Java Clients/Usage of SASL SCRAM mechanism
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > > > > > > >
>> > > > > > > > > > 48+Delegation+token+support+for+Kafka
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Jun, Ashish, Gwen,
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Pl review the updated KIP.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Thanks
>> > > > > > > > > >
>> > > > > > > > > > Manikumar
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Thu, Sep 29, 2016 at 9:56 PM, Ashish Singh <
>> > > > > asingh@cloudera.com
>> > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > Harsha/ Gwen,
>> > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > How do we proceed here? I am willing to help out with
>> > here.
>> > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > On Fri, Sep 23, 2016 at 11:41 AM, Gwen Shapira <
>> > > > > > gwen@confluent.io>
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > Is it updated? are all concerns addressed? do you
>> want
>> > to
>> > > > > > start a
>> > > > > > > > > vote?
>> > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > Sorry for being pushy, I do appreciate that we are
>> all
>> > > > > > volunteers
>> > > > > > > > and
>> > > > > > > > > >
>> > > > > > > > > > > > finding time is difficult. This feature is important
>> > for
>> > > > > > anything
>> > > > > > > > > that
>> > > > > > > > > >
>> > > > > > > > > > > > integrates with Kafka (stream processors, Flume,
>> NiFi,
>> > > etc)
>> > > > > > and I
>> > > > > > > > > >
>> > > > > > > > > > > > don't want to see this getting stuck because we lack
>> > > > > > coordination
>> > > > > > > > > >
>> > > > > > > > > > > > within the community.
>> > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > On Thu, Sep 15, 2016 at 6:39 PM, Harsha
>> Chintalapani <
>> > > > > > > > > kafka@harsha.io>
>> > > > > > > > > >
>> > > > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > > The only pending update for the KIP is to write up
>> > the
>> > > > > > protocol
>> > > > > > > > > > changes
>> > > > > > > > > >
>> > > > > > > > > > > > like
>> > > > > > > > > >
>> > > > > > > > > > > > > we've it KIP-4. I'll update the wiki.
>> > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > > On Thu, Sep 15, 2016 at 4:27 PM Ashish Singh <
>> > > > > > > > asingh@cloudera.com>
>> > > > > > > > > >
>> > > > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > > > > > >> I think we decided to not support secret
>> rotation, I
>> > > > guess
>> > > > > > > this
>> > > > > > > > > can
>> > > > > > > > > > be
>> > > > > > > > > >
>> > > > > > > > > > > > >> stated clearly on the KIP. Also, more details on
>> how
>> > > > > clients
>> > > > > > > > will
>> > > > > > > > > >
>> > > > > > > > > > > > perform
>> > > > > > > > > >
>> > > > > > > > > > > > >> token distribution and how CLI will look like
>> will
>> > be
>> > > > > > helpful.
>> > > > > > > > > >
>> > > > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > > > > > >> On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <
>> > > > > > > > gwen@confluent.io>
>> > > > > > > > > >
>> > > > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > Hi Guys,
>> > > > > > > > > >
>> > > > > > > > > > > > >> >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > This discussion was dead for a while. Are there
>> > > still
>> > > > > > > > > contentious
>> > > > > > > > > >
>> > > > > > > > > > > > >> > points? If not, why are there no votes?
>> > > > > > > > > >
>> > > > > > > > > > > > >> >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <
>> > > > > > jun@confluent.io>
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > >> > > Ashish,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > > Yes, I will send out a KIP invite for next
>> week
>> > to
>> > > > > > discuss
>> > > > > > > > > > KIP-48
>> > > > > > > > > >
>> > > > > > > > > > > > and
>> > > > > > > > > >
>> > > > > > > > > > > > >> > other
>> > > > > > > > > >
>> > > > > > > > > > > > >> > > remaining KIPs.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > > Thanks,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > > Jun
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > > On Tue, Aug 23, 2016 at 1:22 PM, Ashish
>> Singh <
>> > > > > > > > > >
>> > > > > > > > > > > asingh@cloudera.com>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >> Thanks Harsha!
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >> Jun, can we add KIP-48 to next KIP hangout's
>> > > > agenda.
>> > > > > > > Also,
>> > > > > > > > we
>> > > > > > > > > > did
>> > > > > > > > > >
>> > > > > > > > > > > > not
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >> actually make a call on when we should have
>> > next
>> > > > KIP
>> > > > > > > call.
>> > > > > > > > As
>> > > > > > > > > >
>> > > > > > > > > > > there
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >> are
>> > > > > > > > > >
>> > > > > > > > > > > > >> > a
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >> few outstanding KIPs that could not be
>> > discussed
>> > > > this
>> > > > > > > week,
>> > > > > > > > > can
>> > > > > > > > > >
>> > > > > > > > > > > we
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >> have
>> > > > > > > > > >
>> > > > > > > > > > > > >> > a
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >> KIP hangout call next week?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha
>> > > > Chintalapani
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >> <ka...@harsha.io>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >> wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> Ashish,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>>         Yes we are working on it. Lets
>> discuss
>> > > in
>> > > > > the
>> > > > > > > next
>> > > > > > > > > KIP
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> meeting.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> I'll join.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> -Harsha
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish
>> Singh
>> > <
>> > > > > > > > > >
>> > > > > > > > > > > > asingh@cloudera.com>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > Hello Harsha,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > Are you still working on this? Wondering
>> if
>> > we
>> > > > can
>> > > > > > > > discuss
>> > > > > > > > > >
>> > > > > > > > > > > this
>> > > > > > > > > >
>> > > > > > > > > > > > in
>> > > > > > > > > >
>> > > > > > > > > > > > >> > next
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> KIP
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > meeting, if you can join.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha
>> > > > > > Chintalapani <
>> > > > > > > > > >
>> > > > > > > > > > > > >> > kafka@harsha.io>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > Hi Grant,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > >           We are working on it. Will
>> add
>> > the
>> > > > > > details
>> > > > > > > > to
>> > > > > > > > > > KIP
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > about
>> > > > > > > > > >
>> > > > > > > > > > > > >> > the
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > request protocol.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > Thanks,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > Harsha
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant
>> > Henke
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > <gh...@cloudera.com>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > Hi Parth,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > Are you still working on this? If you
>> > need
>> > > > any
>> > > > > > > help
>> > > > > > > > > > please
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > don't
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > hesitate
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > to ask.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > Thanks,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > Grant
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun
>> > Rao <
>> > > > > > > > > >
>> > > > > > > > > > > jun@confluent.io>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > Parth,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > Thanks for the reply.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > It makes sense to only allow the
>> > renewal
>> > > > by
>> > > > > > > users
>> > > > > > > > > that
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> authenticated
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > using
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > *non* delegation token mechanism.
>> > Then,
>> > > > > should
>> > > > > > > we
>> > > > > > > > > make
>> > > > > > > > > >
>> > > > > > > > > > > the
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> renewal a
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > list?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > For example, in the case of rest
>> > proxy,
>> > > it
>> > > > > > will
>> > > > > > > be
>> > > > > > > > > >
>> > > > > > > > > > > useful
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > for
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> every
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > instance of rest proxy to be able
>> to
>> > > renew
>> > > > > the
>> > > > > > > > > tokens.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > It would be clearer if we can
>> document
>> > > the
>> > > > > > > request
>> > > > > > > > > >
>> > > > > > > > > > > > protocol
>> > > > > > > > > >
>> > > > > > > > > > > > >> > like
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > https://cwiki.apache.org/confl
>> > > > > > > > > uence/display/KAFKA/KIP-
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > 4+-+Command+line+and+
>> > > > > centralized+administrative+
>> > > > > > > > > >
>> > > > > > > > > > > > operations#KIP-4-
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > Commandlineandcentralizedadmin
>> > > > > > istrativeoperations-
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > CreateTopicsRequest(KAFKA-
>> > > > > > > > 2945):(VotedandPlannedforin0.
>> > > > > > > > > >
>> > > > > > > > > > > > 10.1.0)
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > .
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > It would also be useful to document
>> > the
>> > > > > client
>> > > > > > > > APIs.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > Thanks,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > Jun
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM,
>> parth
>> > > > > > > brahmbhatt
>> > > > > > > > <
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > brahmbhatt.parth@gmail.com> wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > Hi,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > I am suggesting that we will only
>> > > allow
>> > > > > the
>> > > > > > > > > renewal
>> > > > > > > > > > by
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > users
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> that
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > authenticated using *non*
>> delegation
>> > > > token
>> > > > > > > > > > mechanism.
>> > > > > > > > > >
>> > > > > > > > > > > > For
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> example,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > If
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > user
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > Alice authenticated using
>> kerberos
>> > and
>> > > > > > > requested
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > delegation
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> tokens,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > only
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > user Alice authenticated via non
>> > > > > delegation
>> > > > > > > > token
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > mechanism
>> > > > > > > > > >
>> > > > > > > > > > > > >> > can
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > renew.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > Clients that have  access to
>> > > delegation
>> > > > > > tokens
>> > > > > > > > can
>> > > > > > > > > > not
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > issue
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > renewal
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > request for renewing their own
>> token
>> > > and
>> > > > > > this
>> > > > > > > is
>> > > > > > > > > >
>> > > > > > > > > > > > primarily
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > important
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > to
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > reduce the time window for which
>> a
>> > > > > > compromised
>> > > > > > > > > token
>> > > > > > > > > >
>> > > > > > > > > > > > will
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > be
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> valid.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > To clarify, Yes any authenticated
>> > user
>> > > > can
>> > > > > > > > request
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > delegation
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > tokens
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > but
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > even here I would recommend to
>> avoid
>> > > > > > creating
>> > > > > > > a
>> > > > > > > > > > chain
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > where a
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > client
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > authenticated via delegation
>> token
>> > > > request
>> > > > > > for
>> > > > > > > > > more
>> > > > > > > > > >
>> > > > > > > > > > > > >> > delegation
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > tokens.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > Basically anyone can request
>> > > delegation
>> > > > > > token,
>> > > > > > > > as
>> > > > > > > > > > long
>> > > > > > > > > >
>> > > > > > > > > > > > as
>> > > > > > > > > >
>> > > > > > > > > > > > >> > they
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > authenticate
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > via a non delegation token
>> > mechanism.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > Aren't classes listed here
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > <
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > https://cwiki.apache.org/confl
>> > > > > > > > > uence/display/KAFKA/KIP-
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > 48+Delegation+token+support+fo
>> > > > > > > > > >
>> > > > > > > > > > > r+Kafka#KIP-48Delegationtokens
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> upportforKaf
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > ka-PublicInterfaces
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > sufficient?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > Thanks
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > Parth
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM,
>> Jun
>> > > Rao
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > <ju...@confluent.io>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > Parth,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > Thanks for the reply. A couple
>> of
>> > > > > comments
>> > > > > > > > > inline
>> > > > > > > > > >
>> > > > > > > > > > > > below.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > On Tue, Jun 21, 2016 at 10:36
>> AM,
>> > > > parth
>> > > > > > > > > > brahmbhatt <
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > brahmbhatt.parth@gmail.com>
>> > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > 1. Who / how are tokens
>> renewed?
>> > > By
>> > > > > > > original
>> > > > > > > > > >
>> > > > > > > > > > > > requester
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> only? or
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > using
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > Kerberos
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > auth only?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > My recommendation is to do
>> this
>> > > only
>> > > > > > using
>> > > > > > > > > >
>> > > > > > > > > > > Kerberos
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > auth
>> > > > > > > > > >
>> > > > > > > > > > > > >> > and
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > only
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > threw
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > the
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > renewer specified during the
>> > > > > acquisition
>> > > > > > > > > > request.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > Hmm, not sure that I follow
>> this.
>> > > Are
>> > > > > you
>> > > > > > > > saying
>> > > > > > > > > >
>> > > > > > > > > > > that
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > any
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> client
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > authenticated with the
>> delegation
>> > > > token
>> > > > > > can
>> > > > > > > > > renew,
>> > > > > > > > > >
>> > > > > > > > > > > > i.e.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > there
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> is
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > no
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > renewer
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > needed?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > Also, just to be clear, any
>> > > > > authenticated
>> > > > > > > > client
>> > > > > > > > > >
>> > > > > > > > > > > > (either
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> through
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > SASL
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > or
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > SSL) can request a delegation
>> > token
>> > > > for
>> > > > > > the
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > authenticated
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> user,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > right?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > 2. Are tokens stored on each
>> > > broker
>> > > > or
>> > > > > > in
>> > > > > > > > ZK?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > My recommendation is still to
>> > > store
>> > > > in
>> > > > > > ZK
>> > > > > > > or
>> > > > > > > > > not
>> > > > > > > > > >
>> > > > > > > > > > > > store
>> > > > > > > > > >
>> > > > > > > > > > > > >> > them
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> at
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > all.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > The
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > whole controller based
>> > > distribution
>> > > > is
>> > > > > > too
>> > > > > > > > > much
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > overhead
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> with
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > not
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > much
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > to
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > achieve.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > 3. How are tokens
>> invalidated /
>> > > > > expired?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > Either by expiration time
>> out or
>> > > > > through
>> > > > > > > an
>> > > > > > > > > >
>> > > > > > > > > > > explicit
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> request to
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > invalidate.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > 4. Which encryption
>> algorithm is
>> > > > used?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > SCRAM
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > 5. What is the impersonation
>> > > > proposal
>> > > > > > (it
>> > > > > > > > > wasn't
>> > > > > > > > > >
>> > > > > > > > > > > in
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > the
>> > > > > > > > > >
>> > > > > > > > > > > > >> > KIP
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> but
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > was
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > discussed
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > in this thread)?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > There is no imperonation
>> > > proposal. I
>> > > > > > tried
>> > > > > > > > and
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > explained
>> > > > > > > > > >
>> > > > > > > > > > > > >> > how
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > its
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > a
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > different problem and why its
>> > not
>> > > > > really
>> > > > > > > > > > necessary
>> > > > > > > > > >
>> > > > > > > > > > > > to
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> discuss
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > that
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > as
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > part
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > of this KIP.  This KIP will
>> not
>> > > > > support
>> > > > > > > any
>> > > > > > > > > >
>> > > > > > > > > > > > >> > impersonation,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> it
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > will
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > just
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > be
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > another way to authenticate.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > 6. Do we need new ACLs, if
>> so -
>> > > for
>> > > > > what
>> > > > > > > > > > actions?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > We do not need new ACLs.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > Could we document the format of
>> > the
>> > > > new
>> > > > > > > > > >
>> > > > > > > > > > > > request/response
>> > > > > > > > > >
>> > > > > > > > > > > > >> > and
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > their
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > associated Resource and
>> Operation
>> > > for
>> > > > > ACL?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > 7. How would the delegation
>> > token
>> > > be
>> > > > > > > > > configured
>> > > > > > > > > > in
>> > > > > > > > > >
>> > > > > > > > > > > > the
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> client?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > Should be through config. I
>> > wasn't
>> > > > > > > planning
>> > > > > > > > on
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > supporting
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> JAAS
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > for
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > tokens.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > I don't believe hadoop does
>> this
>> > > > > either.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > Thanks
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > Parth
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > On Thu, Jun 16, 2016 at 4:03
>> PM,
>> > > Jun
>> > > > > > Rao <
>> > > > > > > > > >
>> > > > > > > > > > > > >> > jun@confluent.io>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > Harsha,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > Another question.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > 9. How would the delegation
>> > > token
>> > > > be
>> > > > > > > > > > configured
>> > > > > > > > > >
>> > > > > > > > > > > in
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > the
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > client?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > The
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > standard
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > way is to do this through
>> > JAAS.
>> > > > > > However,
>> > > > > > > > we
>> > > > > > > > > > will
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > need
>> > > > > > > > > >
>> > > > > > > > > > > > >> > to
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > think
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > through
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > if
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > this is convenient in a
>> shared
>> > > > > > > > environment.
>> > > > > > > > > > For
>> > > > > > > > > >
>> > > > > > > > > > > > >> > example,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > when a
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > new
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > task
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > is
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > added to a Storm worker
>> node,
>> > do
>> > > > we
>> > > > > > need
>> > > > > > > > to
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > dynamically
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> add a
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > new
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > section
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > in the JAAS file? It may be
>> > more
>> > > > > > > > convenient
>> > > > > > > > > if
>> > > > > > > > > >
>> > > > > > > > > > > we
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > can
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> pass in
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > the
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > token
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > through the config directly
>> > w/o
>> > > > > going
>> > > > > > > > > through
>> > > > > > > > > >
>> > > > > > > > > > > > JAAS.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > Are you or Parth still
>> > actively
>> > > > > > working
>> > > > > > > on
>> > > > > > > > > > this
>> > > > > > > > > >
>> > > > > > > > > > > > KIP?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > Thanks,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > Jun
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > On Sun, Jun 12, 2016 at
>> 2:18
>> > PM,
>> > > > Jun
>> > > > > > > Rao <
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> jun@confluent.io>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > > Just to add on that list.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > > 2. It would be good to
>> > > document
>> > > > > the
>> > > > > > > > format
>> > > > > > > > > > of
>> > > > > > > > > >
>> > > > > > > > > > > > the
>> > > > > > > > > >
>> > > > > > > > > > > > >> > data
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > stored
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > in
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > ZK.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > > 7. Earlier, there was a
>> > > > discussion
>> > > > > > on
>> > > > > > > > > > whether
>> > > > > > > > > >
>> > > > > > > > > > > > the
>> > > > > > > > > >
>> > > > > > > > > > > > >> > tokens
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > should
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > be
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > > propagated through ZK
>> like
>> > > > > > > > > config/acl/quota,
>> > > > > > > > > >
>> > > > > > > > > > > or
>> > > > > > > > > >
>> > > > > > > > > > > > >> > through
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> the
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > controller.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > > Currently, the
>> controller is
>> > > > only
>> > > > > > > > designed
>> > > > > > > > > > for
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> propagating
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > topic
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > metadata,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > > but not other data.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > > 8. Should we use SCRAM to
>> > send
>> > > > the
>> > > > > > > token
>> > > > > > > > > >
>> > > > > > > > > > > instead
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > > of
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > DIGEST-MD5
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > since
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > it's
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > > deprecated?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > > Also, the images in the
>> wiki
>> > > > seem
>> > > > > > > > broken.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > > Thanks,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > > Jun
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > > On Fri, Jun 10, 2016 at
>> > 10:02
>> > > > AM,
>> > > > > > Gwen
>> > > > > > > > > >
>> > > > > > > > > > > Shapira <
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > gwen@confluent.io>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> From what I can see,
>> > > remaining
>> > > > > > > > questions
>> > > > > > > > > > are:
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> 1. Who / how are tokens
>> > > > renewed?
>> > > > > By
>> > > > > > > > > > original
>> > > > > > > > > >
>> > > > > > > > > > > > >> > requester
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > only?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > or
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > using
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> Kerberos auth only?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> 2. Are tokens stored on
>> > each
>> > > > > broker
>> > > > > > > or
>> > > > > > > > in
>> > > > > > > > > > ZK?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> 3. How are tokens
>> > > invalidated /
>> > > > > > > > expired?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> 4. Which encryption
>> > algorithm
>> > > > is
>> > > > > > > used?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> 5. What is the
>> > impersonation
>> > > > > > proposal
>> > > > > > > > (it
>> > > > > > > > > >
>> > > > > > > > > > > > wasn't
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> in
>> > > > > > > > > >
>> > > > > > > > > > > > >> > the
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > KIP
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > but
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > was
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> discussed in this
>> thread)?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> 6. Do we need new ACLs,
>> if
>> > > so -
>> > > > > for
>> > > > > > > > what
>> > > > > > > > > >
>> > > > > > > > > > > > actions?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> Gwen
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> On Thu, Jun 9, 2016 at
>> 7:48
>> > > PM,
>> > > > > > > Harsha
>> > > > > > > > <
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> kafka@harsha.io>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> > Jun & Ismael,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
>> > > > > > > > Unfortunately
>> > > > > > > > > I
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> > couldn't
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> attend
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > the
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > KIP
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > meeting
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
>> > > when
>> > > > > > > > > delegation
>> > > > > > > > > >
>> > > > > > > > > > > > tokens
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > discussed.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > Appreciate
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > if
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
>> > > you
>> > > > > can
>> > > > > > > > update
>> > > > > > > > > > the
>> > > > > > > > > >
>> > > > > > > > > > > > >> > thread if
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > you
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > have
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > any
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
>> > > > > further
>> > > > > > > > > > questions.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> > Thanks,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> > Harsha
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> > On Tue, May 24, 2016,
>> at
>> > > > 11:32
>> > > > > > AM,
>> > > > > > > > > Liquan
>> > > > > > > > > >
>> > > > > > > > > > > Pei
>> > > > > > > > > >
>> > > > > > > > > > > > >> > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> It seems that the
>> links
>> > to
>> > > > > > images
>> > > > > > > in
>> > > > > > > > > the
>> > > > > > > > > >
>> > > > > > > > > > > KIP
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> are
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > broken.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> Liquan
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> On Tue, May 24, 2016
>> at
>> > > 9:33
>> > > > > AM,
>> > > > > > > > parth
>> > > > > > > > > >
>> > > > > > > > > > > > >> > brahmbhatt <
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >>
>> > > brahmbhatt.parth@gmail.com>
>> > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >>
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > 110. What does
>> > > > > > > > getDelegationTokenAs
>> > > > > > > > > >
>> > > > > > > > > > > mean?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > In the current
>> > proposal
>> > > we
>> > > > > > only
>> > > > > > > > > allow
>> > > > > > > > > > a
>> > > > > > > > > >
>> > > > > > > > > > > > user
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > to
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> get
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > delegation
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > token
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> for
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > the identity that
>> it
>> > > > > > > authenticated
>> > > > > > > > > as
>> > > > > > > > > >
>> > > > > > > > > > > > using
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> another
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > mechanism,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > i.e.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> A user
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > that authenticate
>> > using
>> > > a
>> > > > > > keytab
>> > > > > > > > for
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > principal
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > user1@EXAMPLE.COM
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> will get
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens
>> for
>> > > that
>> > > > > > user
>> > > > > > > > > only.
>> > > > > > > > > > In
>> > > > > > > > > >
>> > > > > > > > > > > > >> > future I
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > think
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > we
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > will
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > have
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> to
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > extend support such
>> > that
>> > > > we
>> > > > > > > allow
>> > > > > > > > > some
>> > > > > > > > > >
>> > > > > > > > > > > set
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > of
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> users (
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> >
>> > > > kafka-rest-user@EXAMPLE.COM
>> > > > > ,
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > storm-nimbus@EXAMPLE.COM)
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > to
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > acquire
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens
>> on
>> > > > behalf
>> > > > > of
>> > > > > > > > other
>> > > > > > > > > >
>> > > > > > > > > > > users
>> > > > > > > > > >
>> > > > > > > > > > > > >> > whose
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > identity
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > they
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > have
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > verified
>> > independently.
>> > > > > Kafka
>> > > > > > > > > brokers
>> > > > > > > > > >
>> > > > > > > > > > > > will
>> > > > > > > > > >
>> > > > > > > > > > > > >> > have
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> ACLs
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > to
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > control
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> which
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > users are allowed
>> to
>> > > > > > impersonate
>> > > > > > > > > other
>> > > > > > > > > >
>> > > > > > > > > > > > users
>> > > > > > > > > >
>> > > > > > > > > > > > >> > and
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> get
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > tokens
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > on
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> behalf of
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > them. Overall
>> > > > Impersonation
>> > > > > > is a
>> > > > > > > > > whole
>> > > > > > > > > >
>> > > > > > > > > > > > >> > different
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > problem
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > in
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > my
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> opinion and
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > I think we can
>> tackle
>> > it
>> > > > in
>> > > > > > > > separate
>> > > > > > > > > >
>> > > > > > > > > > > KIP.
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> >
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > 111. What's the
>> > typical
>> > > > rate
>> > > > > > of
>> > > > > > > > > > getting
>> > > > > > > > > >
>> > > > > > > > > > > > and
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> renewing
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > delegation
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> tokens?
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > Typically this
>> should
>> > be
>> > > > > very
>> > > > > > > very
>> > > > > > > > > > low,
>> > > > > > > > > >
>> > > > > > > > > > > 1
>> > > > > > > > > >
>> > > > > > > > > > > > >> > request
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> per
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > minute
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > is a
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > relatively high
>> > > estimate.
>> > > > > > > However
>> > > > > > > > it
>> > > > > > > > > >
>> > > > > > > > > > > > depends
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > on
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> the
>> > > > > > > > > >
>> > > > > > > > > > > > >> > >>> > > token
>> > > > > > > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Manikumar <ma...@gmail.com>]

Hi,


> Can you add a sample Jaas configuration using delegation tokens to the KIP?
>

Will add sample Jaas configuration.


> To make sure I have understood correctly, KAFKA-3712 is aimed at enabling a
> superuser to impersonate another (single) user, say alice. A producer using
> impersonation will authenticate with superuser credentials. All requests
> from the producer will be run with the principal alice. But alice is not
> involved in the authentication and alice's credentials are not actually
> provided to the broker?
>
>
 Yes, this matches with my understanding of impersonation work . Even in
this approach
 we have to handle all producer related issues mentioned by you. Yes, this
is big change
 and can be implemented if there is a pressing need. I hope we are all in
agreement, that
 this can be done in a separate KIP.


 I request others give any suggestions/concerns on this KIP.


Thanks,



>
> On Thu, Dec 15, 2016 at 9:04 AM, Manikumar <ma...@gmail.com>
> wrote:
>
> > @Gwen, @Rajini,
> >
> > As mentioned in the KIP, main motivation for this KIP is to reduce load
> on
> > Kerberos
> > server on large kafka deployments with large number of clients.
> >
> > Also it looks like we are combining two overlapping concepts
> > 1. Single client sending requests with multiple users/authentications
> > 2. Impersonation
> >
> > Option 1, is definitely useful in some use cases and can be used to
> > implement workaround for
> > impersonation
> >
> > In Impersonation, a super user can send requests on behalf of another
> > user(Alice) in a secured way.
> > superuser has credentials but user Alice doesn't have any. The requests
> are
> > required
> > to run as user Alice and accesses/ACLs on Broker are required to be done
> as
> > user Alice.
> > It is required that user Alice can connect to the Broker on a connection
> > authenticated with
> > superuser's credentials. In other words superuser is impersonating the
> user
> > Alice.
> >
> > The approach mentioned by Harsha in previous mail is implemented in
> hadoop,
> > storm etc..
> >
> > Some more details here:
> > https://hadoop.apache.org/docs/r2.7.2/hadoop-project-
> > dist/hadoop-common/Superusers.html
> >
> >
> > @Rajini
> >
> > Thanks for your comments on SASL/SCRAM usage. I am thinking to send
> > tokenHmac (salted-hashed version)
> > as password for authentication and tokenID for retrial of tokenHmac at
> > server side.
> > Does above sound OK?
> >
> >
> > Thanks,
> > Manikumar
> >
> > On Wed, Dec 14, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > > @Gwen @Mani  Not sure why we want to authenticate at every request.
> Even
> > if
> > > the token exchange is cheap it still a few calls that need to go
> through
> > > round trip.  Impersonation doesn't require authentication for every
> > > request.
> > >
> > > "So a centralized app can create few producers, do the metadata request
> > and
> > > broker discovery with its own user auth, but then use delegation tokens
> > to
> > > allow performing produce/fetch requests as different users? Instead of
> > > having to re-connect for each impersonated user?"
> > >
> > > Yes. But what we will have is this centralized user as impersonation
> user
> > > on behalf of other users. When it authenticates initially we will
> create
> > a
> > > "Subject" and from there on wards centralized user can do
> > > Subject.doAsPrivileged
> > > on behalf, other users.
> > > On the server side, we can retrieve two principals out of this one is
> the
> > > authenticated user (centralized user) and another is impersonated user.
> > We
> > > will first check if the authenticated user allowed to impersonate and
> > then
> > > move on to check if the user Alice has access to the topic "X" to
> > > read/write.
> > >
> > > @Rajini Intention of this KIP is to support token auth via SASL/SCRAM,
> > not
> > > just with TLS.  What you raised is a good point let me take a look and
> > add
> > > details.
> > >
> > > It will be easier to add impersonation once we reach agreement on this
> > KIP.
> > >
> > >
> > > On Wed, Dec 14, 2016 at 5:51 AM Ismael Juma <is...@juma.me.uk> wrote:
> > >
> > > > Hi Rajini,
> > > >
> > > > I think it would definitely be valuable to have a KIP for
> > impersonation.
> > > >
> > > > Ismael
> > > >
> > > > On Wed, Dec 14, 2016 at 4:03 AM, Rajini Sivaram <rsivaram@pivotal.io
> >
> > > > wrote:
> > > >
> > > > > It would clearly be very useful to enable clients to send requests
> on
> > > > > behalf of multiple users. A separate KIP makes sense, but it may be
> > > worth
> > > > > thinking through some of the implications now, especially if the
> main
> > > > > interest in delegation tokens comes from its potential to enable
> > > > > impersonation.
> > > > >
> > > > > I understand that delegation tokens are only expected to be used
> with
> > > > TLS.
> > > > > But the choice of SASL/SCRAM for authentication must be based on a
> > > > > requirement to protect the tokenHmac - otherwise you could just use
> > > > > SASL/PLAIN. With SASL/SCRAM the tokenHmac is never propagated
> > > > on-the-wire,
> > > > > only a salted-hashed version of it is used in the SASL
> authentication
> > > > > exchange. If impersonation is based on sending tokenHmac in
> requests,
> > > any
> > > > > benefit of using SCRAM is lost.
> > > > >
> > > > > An alternative may be to allow clients to authenticate multiple
> times
> > > > using
> > > > > SASL and include one of its authenticated principals in each
> request
> > > > > (optionally). I haven't thought it through yet, obviously. But if
> the
> > > > > approach is of interest and no one is working on a KIP for
> > > impersonation
> > > > at
> > > > > the moment, I am happy to write one. It may provide something for
> > > > > comparison at least.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > >
> > > > > On Wed, Dec 14, 2016 at 9:53 AM, Manikumar <
> > manikumar.reddy@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > That's a good idea. Authenticating every request with delegation
> > > token
> > > > > will
> > > > > > be useful for
> > > > > > impersonation use-cases. But as of now, we are thinking
> delegation
> > > > token
> > > > > as
> > > > > > just another way
> > > > > > to authenticate the users. We haven't think through all the use
> > cases
> > > > > > related to
> > > > > > impersonation or using delegation token for impersonation. We
> want
> > to
> > > > > > handle impersonation
> > > > > > (KAFKA-3712) as part of separate KIP.
> > > > > >
> > > > > > Will that be Ok?
> > > > > >
> > > > > >
> > > > > > On Wed, Dec 14, 2016 at 8:09 AM, Gwen Shapira <gwen@confluent.io
> >
> > > > wrote:
> > > > > >
> > > > > > > Thinking out loud here:
> > > > > > >
> > > > > > > It looks like authentication with a delegation token is going
> to
> > be
> > > > > > > super-cheap, right? We just compare the token to a value in the
> > > > broker
> > > > > > > cache?
> > > > > > >
> > > > > > > If I understood the KIP correctly, right now it suggests that
> > > > > > > authentication happens when establishing the client-broker
> > > connection
> > > > > (as
> > > > > > > normal for Kafka. But perhaps we want to consider
> authenticating
> > > > every
> > > > > > > request with delegation token (if exists)?
> > > > > > >
> > > > > > > So a centralized app can create few producers, do the metadata
> > > > request
> > > > > > and
> > > > > > > broker discovery with its own user auth, but then use
> delegation
> > > > tokens
> > > > > > to
> > > > > > > allow performing produce/fetch requests as different users?
> > Instead
> > > > of
> > > > > > > having to re-connect for each impersonated user?
> > > > > > >
> > > > > > > This may over-complicate things quite a bit (basically adding
> > extra
> > > > > > > information in every request), but maybe it will be useful for
> > > > > > > impersonation use-cases (which seem to drive much of the
> interest
> > > in
> > > > > this
> > > > > > > KIP)?
> > > > > > > Kafka Connect, NiFi and friends can probably use this to share
> > > > clients
> > > > > > > between multiple jobs, tasks, etc.
> > > > > > >
> > > > > > > What do you think?
> > > > > > >
> > > > > > > Gwen
> > > > > > >
> > > > > > > On Tue, Dec 13, 2016 at 12:43 AM, Manikumar <
> > > > manikumar.reddy@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Ashish,
> > > > > > > >
> > > > > > > > Thank you for reviewing the KIP.  Please see the replies
> > inline.
> > > > > > > >
> > > > > > > >
> > > > > > > > > 1. How to disable delegation token authentication?
> > > > > > > > >
> > > > > > > > > This can be achieved in various ways, however I think
> reusing
> > > > > > > delegation
> > > > > > > > > token secret config for this makes sense here. Avoids
> > creating
> > > > yet
> > > > > > > > another
> > > > > > > > > config and forces delegation token users to consciously set
> > the
> > > > > > secret.
> > > > > > > > If
> > > > > > > > > the secret is not set or set to empty string, brokers
> should
> > > turn
> > > > > off
> > > > > > > > > delegation token support. This will however require a new
> > error
> > > > > code
> > > > > > to
> > > > > > > > > indicate delegation token support is turned off on broker.
> > > > > > > > >
> > > > > > > >
> > > > > > > >   Thanks for the suggestion. Option to turnoff delegation
> token
> > > > > > > > authentication will be useful.
> > > > > > > >   I'll update the KIP.
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 2. ACLs on delegation token?
> > > > > > > > >
> > > > > > > > > Do we need to have ACLs defined for tokens? I do not think
> it
> > > > buys
> > > > > us
> > > > > > > > > anything, as delegation token can be treated as
> impersonation
> > > of
> > > > > the
> > > > > > > > owner.
> > > > > > > > > Any thing the owner has permission to do, delegation tokens
> > > > should
> > > > > be
> > > > > > > > > allowed to do as well. If so, we probably won't need to
> > return
> > > > > > > > > authorization exception error code while creating
> delegation
> > > > token.
> > > > > > It
> > > > > > > > > however would make sense to check renew and expire requests
> > are
> > > > > > coming
> > > > > > > > from
> > > > > > > > > owner or renewers of the token, but that does not require
> > > > explicit
> > > > > > > acls.
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Yes, We agreed to not have new acl on who can request
> > delegation
> > > > > token.
> > > > > > > >  I'll update the KIP.
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 3. How to restrict max life time of a token?
> > > > > > > > >
> > > > > > > > > Admins might want to restrict max life time of tokens
> created
> > > on
> > > > a
> > > > > > > > cluster,
> > > > > > > > > and this can very from cluster to cluster based on
> use-cases.
> > > > This
> > > > > > > might
> > > > > > > > > warrant a separate broker config.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > Currently we  have "delegation.token.max.lifetime.sec"
> server
> > > > config
> > > > > > > > property
> > > > > > > > May be we can take min(User supplied MaxTime, Server MaxTime)
> > as
> > > > max
> > > > > > life
> > > > > > > > time.
> > > > > > > > I am open to add new config property.
> > > > > > > >
> > > > > > > > Few more comments based on recent KIP update.
> > > > > > > > >
> > > > > > > > > 1. Do we need a separate {{InvalidateTokenRequest}}? Can't
> we
> > > use
> > > > > > > > > {{ExpireTokenRequest}} with with expiryDate set to anything
> > > > before
> > > > > > > > current
> > > > > > > > > date?
> > > > > > > > >
> > > > > > > >
> > > > > > > > makes sense. we don't need special request to cancel the
> token.
> > > We
> > > > > can
> > > > > > > use
> > > > > > > > ExpireTokenRequest.
> > > > > > > > I'll update the KIP.
> > > > > > > >
> > > > > > > >
> > > > > > > > > 2. Can we change time field names to indicate their unit is
> > > > > > > milliseconds,
> > > > > > > > > like, IssueDateMs, ExpiryDateMs, etc.?
> > > > > > > > >
> > > > > > > > >
> > > > > > > >   Done.
> > > > > > > >
> > > > > > > >
> > > > > > > > > 3. Can we allow users to renew a token for a specified
> amount
> > > of
> > > > > > time?
> > > > > > > In
> > > > > > > > > current version of KIP, renew request does not take time
> as a
> > > > > param,
> > > > > > > not
> > > > > > > > > sure what is expiry time set to after renewal.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >  Yes, we need to specify renew period.  I'll update the KIP.
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Mankumar
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Dec 12, 2016 at 9:08 AM Manikumar <
> > > > > manikumar.reddy@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I would like to reinitiate the discussion on Delegation
> > token
> > > > > > support
> > > > > > > > for
> > > > > > > > > >
> > > > > > > > > > Kafka.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Brief summary of the past discussion:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 1) Broker stores delegation tokens in zookeeper.  All
> > brokers
> > > > > will
> > > > > > > > have a
> > > > > > > > > >
> > > > > > > > > > cache backed by
> > > > > > > > > >
> > > > > > > > > >    zookeeper so they will all get notified whenever a new
> > > token
> > > > > is
> > > > > > > > > >
> > > > > > > > > > generated and they will
> > > > > > > > > >
> > > > > > > > > >    update their local cache whenever token state changes.
> > > > > > > > > >
> > > > > > > > > > 2) The current proposal does not support rotation of
> secret
> > > > > > > > > >
> > > > > > > > > > 3) Only allow the renewal by users that authenticated
> using
> > > > *non*
> > > > > > > > > >
> > > > > > > > > > delegation token mechanism
> > > > > > > > > >
> > > > > > > > > > 4) KIP-84 proposes to support  SASL SCRAM mechanisms.
> Kafka
> > > > > clients
> > > > > > > can
> > > > > > > > > >
> > > > > > > > > > authenticate using
> > > > > > > > > >
> > > > > > > > > >    SCRAM-SHA-256, providing the delegation token HMAC as
> > > > > password.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Updated the KIP with the following:
> > > > > > > > > >
> > > > > > > > > > 1. Protocol and Config changes
> > > > > > > > > >
> > > > > > > > > > 2. format of the data stored in ZK.
> > > > > > > > > >
> > > > > > > > > > 3. Changes to Java Clients/Usage of SASL SCRAM mechanism
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > > > >
> > > > > > > > > > 48+Delegation+token+support+for+Kafka
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Jun, Ashish, Gwen,
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Pl review the updated KIP.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > Manikumar
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 29, 2016 at 9:56 PM, Ashish Singh <
> > > > > asingh@cloudera.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > Harsha/ Gwen,
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > How do we proceed here? I am willing to help out with
> > here.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > On Fri, Sep 23, 2016 at 11:41 AM, Gwen Shapira <
> > > > > > gwen@confluent.io>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > Is it updated? are all concerns addressed? do you
> want
> > to
> > > > > > start a
> > > > > > > > > vote?
> > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > Sorry for being pushy, I do appreciate that we are
> all
> > > > > > volunteers
> > > > > > > > and
> > > > > > > > > >
> > > > > > > > > > > > finding time is difficult. This feature is important
> > for
> > > > > > anything
> > > > > > > > > that
> > > > > > > > > >
> > > > > > > > > > > > integrates with Kafka (stream processors, Flume,
> NiFi,
> > > etc)
> > > > > > and I
> > > > > > > > > >
> > > > > > > > > > > > don't want to see this getting stuck because we lack
> > > > > > coordination
> > > > > > > > > >
> > > > > > > > > > > > within the community.
> > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 15, 2016 at 6:39 PM, Harsha Chintalapani
> <
> > > > > > > > > kafka@harsha.io>
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > > The only pending update for the KIP is to write up
> > the
> > > > > > protocol
> > > > > > > > > > changes
> > > > > > > > > >
> > > > > > > > > > > > like
> > > > > > > > > >
> > > > > > > > > > > > > we've it KIP-4. I'll update the wiki.
> > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 15, 2016 at 4:27 PM Ashish Singh <
> > > > > > > > asingh@cloudera.com>
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > > > > >> I think we decided to not support secret
> rotation, I
> > > > guess
> > > > > > > this
> > > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > >
> > > > > > > > > > > > >> stated clearly on the KIP. Also, more details on
> how
> > > > > clients
> > > > > > > > will
> > > > > > > > > >
> > > > > > > > > > > > perform
> > > > > > > > > >
> > > > > > > > > > > > >> token distribution and how CLI will look like will
> > be
> > > > > > helpful.
> > > > > > > > > >
> > > > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > > > > >> On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <
> > > > > > > > gwen@confluent.io>
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > > > > >> > Hi Guys,
> > > > > > > > > >
> > > > > > > > > > > > >> >
> > > > > > > > > >
> > > > > > > > > > > > >> > This discussion was dead for a while. Are there
> > > still
> > > > > > > > > contentious
> > > > > > > > > >
> > > > > > > > > > > > >> > points? If not, why are there no votes?
> > > > > > > > > >
> > > > > > > > > > > > >> >
> > > > > > > > > >
> > > > > > > > > > > > >> > On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <
> > > > > > jun@confluent.io>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > > Ashish,
> > > > > > > > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > >
> > > > > > > > > > > > >> > > Yes, I will send out a KIP invite for next
> week
> > to
> > > > > > discuss
> > > > > > > > > > KIP-48
> > > > > > > > > >
> > > > > > > > > > > > and
> > > > > > > > > >
> > > > > > > > > > > > >> > other
> > > > > > > > > >
> > > > > > > > > > > > >> > > remaining KIPs.
> > > > > > > > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > >
> > > > > > > > > > > > >> > > Thanks,
> > > > > > > > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > >
> > > > > > > > > > > > >> > > Jun
> > > > > > > > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > >
> > > > > > > > > > > > >> > > On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh
> <
> > > > > > > > > >
> > > > > > > > > > > asingh@cloudera.com>
> > > > > > > > > >
> > > > > > > > > > > > >> > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >> Thanks Harsha!
> > > > > > > > > >
> > > > > > > > > > > > >> > >>
> > > > > > > > > >
> > > > > > > > > > > > >> > >> Jun, can we add KIP-48 to next KIP hangout's
> > > > agenda.
> > > > > > > Also,
> > > > > > > > we
> > > > > > > > > > did
> > > > > > > > > >
> > > > > > > > > > > > not
> > > > > > > > > >
> > > > > > > > > > > > >> > >> actually make a call on when we should have
> > next
> > > > KIP
> > > > > > > call.
> > > > > > > > As
> > > > > > > > > >
> > > > > > > > > > > there
> > > > > > > > > >
> > > > > > > > > > > > >> > >> are
> > > > > > > > > >
> > > > > > > > > > > > >> > a
> > > > > > > > > >
> > > > > > > > > > > > >> > >> few outstanding KIPs that could not be
> > discussed
> > > > this
> > > > > > > week,
> > > > > > > > > can
> > > > > > > > > >
> > > > > > > > > > > we
> > > > > > > > > >
> > > > > > > > > > > > >> > >> have
> > > > > > > > > >
> > > > > > > > > > > > >> > a
> > > > > > > > > >
> > > > > > > > > > > > >> > >> KIP hangout call next week?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>
> > > > > > > > > >
> > > > > > > > > > > > >> > >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha
> > > > Chintalapani
> > > > > > > > > >
> > > > > > > > > > > > >> > >> <ka...@harsha.io>
> > > > > > > > > >
> > > > > > > > > > > > >> > >> wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> Ashish,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>>         Yes we are working on it. Lets
> discuss
> > > in
> > > > > the
> > > > > > > next
> > > > > > > > > KIP
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> meeting.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> I'll join.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> -Harsha
> > > > > > > > > >
> > > > > > > > > > > > >> > >>>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish
> Singh
> > <
> > > > > > > > > >
> > > > > > > > > > > > asingh@cloudera.com>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > Hello Harsha,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > Are you still working on this? Wondering
> if
> > we
> > > > can
> > > > > > > > discuss
> > > > > > > > > >
> > > > > > > > > > > this
> > > > > > > > > >
> > > > > > > > > > > > in
> > > > > > > > > >
> > > > > > > > > > > > >> > next
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> KIP
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > meeting, if you can join.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha
> > > > > > Chintalapani <
> > > > > > > > > >
> > > > > > > > > > > > >> > kafka@harsha.io>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > Hi Grant,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > >           We are working on it. Will add
> > the
> > > > > > details
> > > > > > > > to
> > > > > > > > > > KIP
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > about
> > > > > > > > > >
> > > > > > > > > > > > >> > the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > request protocol.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > Thanks,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > Harsha
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant
> > Henke
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > <gh...@cloudera.com>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > Hi Parth,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > Are you still working on this? If you
> > need
> > > > any
> > > > > > > help
> > > > > > > > > > please
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > don't
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > hesitate
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > to ask.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > Grant
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun
> > Rao <
> > > > > > > > > >
> > > > > > > > > > > jun@confluent.io>
> > > > > > > > > >
> > > > > > > > > > > > >> > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > Parth,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > Thanks for the reply.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > It makes sense to only allow the
> > renewal
> > > > by
> > > > > > > users
> > > > > > > > > that
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> authenticated
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > using
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > *non* delegation token mechanism.
> > Then,
> > > > > should
> > > > > > > we
> > > > > > > > > make
> > > > > > > > > >
> > > > > > > > > > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> renewal a
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > list?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > For example, in the case of rest
> > proxy,
> > > it
> > > > > > will
> > > > > > > be
> > > > > > > > > >
> > > > > > > > > > > useful
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > for
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> every
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > instance of rest proxy to be able to
> > > renew
> > > > > the
> > > > > > > > > tokens.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > It would be clearer if we can
> document
> > > the
> > > > > > > request
> > > > > > > > > >
> > > > > > > > > > > > protocol
> > > > > > > > > >
> > > > > > > > > > > > >> > like
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > https://cwiki.apache.org/confl
> > > > > > > > > uence/display/KAFKA/KIP-
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > 4+-+Command+line+and+
> > > > > centralized+administrative+
> > > > > > > > > >
> > > > > > > > > > > > operations#KIP-4-
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > Commandlineandcentralizedadmin
> > > > > > istrativeoperations-
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > CreateTopicsRequest(KAFKA-
> > > > > > > > 2945):(VotedandPlannedforin0.
> > > > > > > > > >
> > > > > > > > > > > > 10.1.0)
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > .
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > It would also be useful to document
> > the
> > > > > client
> > > > > > > > APIs.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > Jun
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM,
> parth
> > > > > > > brahmbhatt
> > > > > > > > <
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > I am suggesting that we will only
> > > allow
> > > > > the
> > > > > > > > > renewal
> > > > > > > > > > by
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > users
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> that
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > authenticated using *non*
> delegation
> > > > token
> > > > > > > > > > mechanism.
> > > > > > > > > >
> > > > > > > > > > > > For
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> example,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > If
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > user
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > Alice authenticated using kerberos
> > and
> > > > > > > requested
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > delegation
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> tokens,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > only
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > user Alice authenticated via non
> > > > > delegation
> > > > > > > > token
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > mechanism
> > > > > > > > > >
> > > > > > > > > > > > >> > can
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > renew.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > Clients that have  access to
> > > delegation
> > > > > > tokens
> > > > > > > > can
> > > > > > > > > > not
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > issue
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > renewal
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > request for renewing their own
> token
> > > and
> > > > > > this
> > > > > > > is
> > > > > > > > > >
> > > > > > > > > > > > primarily
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > important
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > reduce the time window for which a
> > > > > > compromised
> > > > > > > > > token
> > > > > > > > > >
> > > > > > > > > > > > will
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > be
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> valid.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > To clarify, Yes any authenticated
> > user
> > > > can
> > > > > > > > request
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > delegation
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > tokens
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > but
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > even here I would recommend to
> avoid
> > > > > > creating
> > > > > > > a
> > > > > > > > > > chain
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > where a
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > client
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > authenticated via delegation token
> > > > request
> > > > > > for
> > > > > > > > > more
> > > > > > > > > >
> > > > > > > > > > > > >> > delegation
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > tokens.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > Basically anyone can request
> > > delegation
> > > > > > token,
> > > > > > > > as
> > > > > > > > > > long
> > > > > > > > > >
> > > > > > > > > > > > as
> > > > > > > > > >
> > > > > > > > > > > > >> > they
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > authenticate
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > via a non delegation token
> > mechanism.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > Aren't classes listed here
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > <
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > https://cwiki.apache.org/confl
> > > > > > > > > uence/display/KAFKA/KIP-
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > 48+Delegation+token+support+fo
> > > > > > > > > >
> > > > > > > > > > > r+Kafka#KIP-48Delegationtokens
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> upportforKaf
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > ka-PublicInterfaces
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > sufficient?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > Parth
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM,
> Jun
> > > Rao
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > <ju...@confluent.io>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > Parth,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > Thanks for the reply. A couple
> of
> > > > > comments
> > > > > > > > > inline
> > > > > > > > > >
> > > > > > > > > > > > below.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > On Tue, Jun 21, 2016 at 10:36
> AM,
> > > > parth
> > > > > > > > > > brahmbhatt <
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > brahmbhatt.parth@gmail.com>
> > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > 1. Who / how are tokens
> renewed?
> > > By
> > > > > > > original
> > > > > > > > > >
> > > > > > > > > > > > requester
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> only? or
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > using
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > Kerberos
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > auth only?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > My recommendation is to do
> this
> > > only
> > > > > > using
> > > > > > > > > >
> > > > > > > > > > > Kerberos
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > auth
> > > > > > > > > >
> > > > > > > > > > > > >> > and
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > only
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > threw
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > renewer specified during the
> > > > > acquisition
> > > > > > > > > > request.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > Hmm, not sure that I follow
> this.
> > > Are
> > > > > you
> > > > > > > > saying
> > > > > > > > > >
> > > > > > > > > > > that
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > any
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> client
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > authenticated with the
> delegation
> > > > token
> > > > > > can
> > > > > > > > > renew,
> > > > > > > > > >
> > > > > > > > > > > > i.e.
> > > > > > > > > >
> > > > > > > > > > > > >> > there
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> is
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > no
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > renewer
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > needed?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > Also, just to be clear, any
> > > > > authenticated
> > > > > > > > client
> > > > > > > > > >
> > > > > > > > > > > > (either
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> through
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > SASL
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > or
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > SSL) can request a delegation
> > token
> > > > for
> > > > > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > authenticated
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> user,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > right?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > 2. Are tokens stored on each
> > > broker
> > > > or
> > > > > > in
> > > > > > > > ZK?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > My recommendation is still to
> > > store
> > > > in
> > > > > > ZK
> > > > > > > or
> > > > > > > > > not
> > > > > > > > > >
> > > > > > > > > > > > store
> > > > > > > > > >
> > > > > > > > > > > > >> > them
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> at
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > all.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > The
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > whole controller based
> > > distribution
> > > > is
> > > > > > too
> > > > > > > > > much
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > overhead
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> with
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > not
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > much
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > achieve.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > 3. How are tokens invalidated
> /
> > > > > expired?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > Either by expiration time out
> or
> > > > > through
> > > > > > > an
> > > > > > > > > >
> > > > > > > > > > > explicit
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> request to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > invalidate.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > 4. Which encryption algorithm
> is
> > > > used?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > SCRAM
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > 5. What is the impersonation
> > > > proposal
> > > > > > (it
> > > > > > > > > wasn't
> > > > > > > > > >
> > > > > > > > > > > in
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > KIP
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> but
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > was
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > discussed
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > in this thread)?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > There is no imperonation
> > > proposal. I
> > > > > > tried
> > > > > > > > and
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > explained
> > > > > > > > > >
> > > > > > > > > > > > >> > how
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > its
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > a
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > different problem and why its
> > not
> > > > > really
> > > > > > > > > > necessary
> > > > > > > > > >
> > > > > > > > > > > > to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> discuss
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > that
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > as
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > part
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > of this KIP.  This KIP will
> not
> > > > > support
> > > > > > > any
> > > > > > > > > >
> > > > > > > > > > > > >> > impersonation,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> it
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > will
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > just
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > be
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > another way to authenticate.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > 6. Do we need new ACLs, if so
> -
> > > for
> > > > > what
> > > > > > > > > > actions?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > We do not need new ACLs.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > Could we document the format of
> > the
> > > > new
> > > > > > > > > >
> > > > > > > > > > > > request/response
> > > > > > > > > >
> > > > > > > > > > > > >> > and
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > their
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > associated Resource and
> Operation
> > > for
> > > > > ACL?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > 7. How would the delegation
> > token
> > > be
> > > > > > > > > configured
> > > > > > > > > > in
> > > > > > > > > >
> > > > > > > > > > > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> client?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > Should be through config. I
> > wasn't
> > > > > > > planning
> > > > > > > > on
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > supporting
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> JAAS
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > for
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > tokens.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > I don't believe hadoop does
> this
> > > > > either.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > Parth
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > On Thu, Jun 16, 2016 at 4:03
> PM,
> > > Jun
> > > > > > Rao <
> > > > > > > > > >
> > > > > > > > > > > > >> > jun@confluent.io>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > Harsha,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > Another question.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > 9. How would the delegation
> > > token
> > > > be
> > > > > > > > > > configured
> > > > > > > > > >
> > > > > > > > > > > in
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > client?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > The
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > standard
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > way is to do this through
> > JAAS.
> > > > > > However,
> > > > > > > > we
> > > > > > > > > > will
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > need
> > > > > > > > > >
> > > > > > > > > > > > >> > to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > think
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > through
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > if
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > this is convenient in a
> shared
> > > > > > > > environment.
> > > > > > > > > > For
> > > > > > > > > >
> > > > > > > > > > > > >> > example,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > when a
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > new
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > task
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > is
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > added to a Storm worker
> node,
> > do
> > > > we
> > > > > > need
> > > > > > > > to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > dynamically
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> add a
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > new
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > section
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > in the JAAS file? It may be
> > more
> > > > > > > > convenient
> > > > > > > > > if
> > > > > > > > > >
> > > > > > > > > > > we
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > can
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> pass in
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > token
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > through the config directly
> > w/o
> > > > > going
> > > > > > > > > through
> > > > > > > > > >
> > > > > > > > > > > > JAAS.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > Are you or Parth still
> > actively
> > > > > > working
> > > > > > > on
> > > > > > > > > > this
> > > > > > > > > >
> > > > > > > > > > > > KIP?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18
> > PM,
> > > > Jun
> > > > > > > Rao <
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> jun@confluent.io>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > Just to add on that list.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > 2. It would be good to
> > > document
> > > > > the
> > > > > > > > format
> > > > > > > > > > of
> > > > > > > > > >
> > > > > > > > > > > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > data
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > stored
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > in
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > ZK.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > 7. Earlier, there was a
> > > > discussion
> > > > > > on
> > > > > > > > > > whether
> > > > > > > > > >
> > > > > > > > > > > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > tokens
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > should
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > be
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > propagated through ZK like
> > > > > > > > > config/acl/quota,
> > > > > > > > > >
> > > > > > > > > > > or
> > > > > > > > > >
> > > > > > > > > > > > >> > through
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > controller.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > Currently, the controller
> is
> > > > only
> > > > > > > > designed
> > > > > > > > > > for
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> propagating
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > topic
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > metadata,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > but not other data.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > 8. Should we use SCRAM to
> > send
> > > > the
> > > > > > > token
> > > > > > > > > >
> > > > > > > > > > > instead
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > of
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > DIGEST-MD5
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > since
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > it's
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > deprecated?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > Also, the images in the
> wiki
> > > > seem
> > > > > > > > broken.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > On Fri, Jun 10, 2016 at
> > 10:02
> > > > AM,
> > > > > > Gwen
> > > > > > > > > >
> > > > > > > > > > > Shapira <
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > gwen@confluent.io>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> From what I can see,
> > > remaining
> > > > > > > > questions
> > > > > > > > > > are:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> 1. Who / how are tokens
> > > > renewed?
> > > > > By
> > > > > > > > > > original
> > > > > > > > > >
> > > > > > > > > > > > >> > requester
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > only?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > or
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > using
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> Kerberos auth only?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> 2. Are tokens stored on
> > each
> > > > > broker
> > > > > > > or
> > > > > > > > in
> > > > > > > > > > ZK?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> 3. How are tokens
> > > invalidated /
> > > > > > > > expired?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> 4. Which encryption
> > algorithm
> > > > is
> > > > > > > used?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> 5. What is the
> > impersonation
> > > > > > proposal
> > > > > > > > (it
> > > > > > > > > >
> > > > > > > > > > > > wasn't
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> in
> > > > > > > > > >
> > > > > > > > > > > > >> > the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > KIP
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > but
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > was
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> discussed in this
> thread)?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> 6. Do we need new ACLs,
> if
> > > so -
> > > > > for
> > > > > > > > what
> > > > > > > > > >
> > > > > > > > > > > > actions?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> Gwen
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> On Thu, Jun 9, 2016 at
> 7:48
> > > PM,
> > > > > > > Harsha
> > > > > > > > <
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> kafka@harsha.io>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> > Jun & Ismael,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
> > > > > > > > Unfortunately
> > > > > > > > > I
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> > couldn't
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> attend
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > KIP
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > meeting
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
> > > when
> > > > > > > > > delegation
> > > > > > > > > >
> > > > > > > > > > > > tokens
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > discussed.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > Appreciate
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > if
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
> > > you
> > > > > can
> > > > > > > > update
> > > > > > > > > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > thread if
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > you
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > have
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > any
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
> > > > > further
> > > > > > > > > > questions.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> > Thanks,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> > Harsha
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> > On Tue, May 24, 2016,
> at
> > > > 11:32
> > > > > > AM,
> > > > > > > > > Liquan
> > > > > > > > > >
> > > > > > > > > > > Pei
> > > > > > > > > >
> > > > > > > > > > > > >> > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> It seems that the
> links
> > to
> > > > > > images
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > >
> > > > > > > > > > > KIP
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> are
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > broken.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> Liquan
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> On Tue, May 24, 2016
> at
> > > 9:33
> > > > > AM,
> > > > > > > > parth
> > > > > > > > > >
> > > > > > > > > > > > >> > brahmbhatt <
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >>
> > > brahmbhatt.parth@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > 110. What does
> > > > > > > > getDelegationTokenAs
> > > > > > > > > >
> > > > > > > > > > > mean?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > In the current
> > proposal
> > > we
> > > > > > only
> > > > > > > > > allow
> > > > > > > > > > a
> > > > > > > > > >
> > > > > > > > > > > > user
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> get
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > delegation
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > token
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> for
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > the identity that it
> > > > > > > authenticated
> > > > > > > > > as
> > > > > > > > > >
> > > > > > > > > > > > using
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> another
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > mechanism,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > i.e.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> A user
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > that authenticate
> > using
> > > a
> > > > > > keytab
> > > > > > > > for
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > principal
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > user1@EXAMPLE.COM
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> will get
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens
> for
> > > that
> > > > > > user
> > > > > > > > > only.
> > > > > > > > > > In
> > > > > > > > > >
> > > > > > > > > > > > >> > future I
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > think
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > we
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > will
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > have
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > extend support such
> > that
> > > > we
> > > > > > > allow
> > > > > > > > > some
> > > > > > > > > >
> > > > > > > > > > > set
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > of
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> users (
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > kafka-rest-user@EXAMPLE.COM
> > > > > ,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > storm-nimbus@EXAMPLE.COM)
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > acquire
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens on
> > > > behalf
> > > > > of
> > > > > > > > other
> > > > > > > > > >
> > > > > > > > > > > users
> > > > > > > > > >
> > > > > > > > > > > > >> > whose
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > identity
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > they
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > have
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > verified
> > independently.
> > > > > Kafka
> > > > > > > > > brokers
> > > > > > > > > >
> > > > > > > > > > > > will
> > > > > > > > > >
> > > > > > > > > > > > >> > have
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> ACLs
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > control
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> which
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > users are allowed to
> > > > > > impersonate
> > > > > > > > > other
> > > > > > > > > >
> > > > > > > > > > > > users
> > > > > > > > > >
> > > > > > > > > > > > >> > and
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> get
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > tokens
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > on
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> behalf of
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > them. Overall
> > > > Impersonation
> > > > > > is a
> > > > > > > > > whole
> > > > > > > > > >
> > > > > > > > > > > > >> > different
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > problem
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > in
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > my
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> opinion and
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > I think we can
> tackle
> > it
> > > > in
> > > > > > > > separate
> > > > > > > > > >
> > > > > > > > > > > KIP.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > 111. What's the
> > typical
> > > > rate
> > > > > > of
> > > > > > > > > > getting
> > > > > > > > > >
> > > > > > > > > > > > and
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> renewing
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > delegation
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> tokens?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > Typically this
> should
> > be
> > > > > very
> > > > > > > very
> > > > > > > > > > low,
> > > > > > > > > >
> > > > > > > > > > > 1
> > > > > > > > > >
> > > > > > > > > > > > >> > request
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> per
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > minute
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > is a
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > relatively high
> > > estimate.
> > > > > > > However
> > > > > > > > it
> > > > > > > > > >
> > > > > > > > > > > > depends
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > on
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > token
> > > > > > > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Ismael Juma <is...@juma.me.uk>]

Hi Rajini,

The use case you outline is indeed the one I was thinking of. The concern
is indeed what you pointed out, it could be a large change.

Ismael

On Thu, Dec 15, 2016 at 3:34 AM, Rajini Sivaram <rs...@pivotal.io> wrote:

> @Mani
>
> Can you add a sample Jaas configuration using delegation tokens to the KIP?
> Since delegation tokens will be handled differently from other SCRAM
> credentials, it should work anyway, but it will be good to see an example
> of the configuration the user provides. It sounds like users provide both
> tokenID and delegation token, but I wasn't sure.
>
> @Gwen, @Ismael, @Harsha, @Mani
>
> To make sure I have understood correctly, KAFKA-3712 is aimed at enabling a
> superuser to impersonate another (single) user, say alice. A producer using
> impersonation will authenticate with superuser credentials. All requests
> from the producer will be run with the principal alice. But alice is not
> involved in the authentication and alice's credentials are not actually
> provided to the broker?
>
> The use case I was thinking of was the other one on Mani's list above. I
> want to make sure that my understanding matches Gwen's and Ismael's for
> this one. Using REST proxy as an example, users sending requests to the
> REST proxy provide a delegation token as an auth token. REST proxy itself
> authenticates as a different user, perhaps with access to read metadata of
> all topics. But Kafka requests corresponding to each REST request should be
> authenticated based on the authentication token provided in the request.
> At the moment, the REST proxy has to create producers for each user (or
> perform authentication and authorization for the user). Instead we want to
> create a single producer which has the ability to send requests on behalf
> of multiple users where the user is authenticated by Kafka and each request
> from the producer is authorized based on the user who initiated the
> request.
>
> 1a) I think Gwen's suggestion was something along the lines of:
> producer.send(record1, delegationToken1);
> producer.send(record2, delegationToken2);
> 1b) I was thinking more along the lines of:
> subject1 = producer.authenticate(user1Credentials);
> subject2 = producer.authenticate(user2Credentials);
> ....
> Subject.doAs(subject1, (PrivilegedExceptionAction<Future<RecordMetadata>>)
> () -> producer.send(record1));
> Subject.doAs(subject2, (PrivilegedExceptionAction<Future<RecordMetadata>>)
> () -> producer.send(record2));
>
> To make either approach work, each request needs to be associated with a
> user. This would be delegation token in 1a) and user principal in 1b). And
> both need to ensure that producers dont send records from multiple users in
> one request. And if producers hold one metadata rather than one per-user,
> calls like partitionsFor() shouldn't return metadata for unauthorized
> topics. It is a very big change, so it will be good to know whether there
> is a real requirement to support this.
>
>
> On Thu, Dec 15, 2016 at 9:04 AM, Manikumar <ma...@gmail.com>
> wrote:
>
> > @Gwen, @Rajini,
> >
> > As mentioned in the KIP, main motivation for this KIP is to reduce load
> on
> > Kerberos
> > server on large kafka deployments with large number of clients.
> >
> > Also it looks like we are combining two overlapping concepts
> > 1. Single client sending requests with multiple users/authentications
> > 2. Impersonation
> >
> > Option 1, is definitely useful in some use cases and can be used to
> > implement workaround for
> > impersonation
> >
> > In Impersonation, a super user can send requests on behalf of another
> > user(Alice) in a secured way.
> > superuser has credentials but user Alice doesn't have any. The requests
> are
> > required
> > to run as user Alice and accesses/ACLs on Broker are required to be done
> as
> > user Alice.
> > It is required that user Alice can connect to the Broker on a connection
> > authenticated with
> > superuser's credentials. In other words superuser is impersonating the
> user
> > Alice.
> >
> > The approach mentioned by Harsha in previous mail is implemented in
> hadoop,
> > storm etc..
> >
> > Some more details here:
> > https://hadoop.apache.org/docs/r2.7.2/hadoop-project-
> > dist/hadoop-common/Superusers.html
> >
> >
> > @Rajini
> >
> > Thanks for your comments on SASL/SCRAM usage. I am thinking to send
> > tokenHmac (salted-hashed version)
> > as password for authentication and tokenID for retrial of tokenHmac at
> > server side.
> > Does above sound OK?
> >
> >
> > Thanks,
> > Manikumar
> >
> > On Wed, Dec 14, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > > @Gwen @Mani  Not sure why we want to authenticate at every request.
> Even
> > if
> > > the token exchange is cheap it still a few calls that need to go
> through
> > > round trip.  Impersonation doesn't require authentication for every
> > > request.
> > >
> > > "So a centralized app can create few producers, do the metadata request
> > and
> > > broker discovery with its own user auth, but then use delegation tokens
> > to
> > > allow performing produce/fetch requests as different users? Instead of
> > > having to re-connect for each impersonated user?"
> > >
> > > Yes. But what we will have is this centralized user as impersonation
> user
> > > on behalf of other users. When it authenticates initially we will
> create
> > a
> > > "Subject" and from there on wards centralized user can do
> > > Subject.doAsPrivileged
> > > on behalf, other users.
> > > On the server side, we can retrieve two principals out of this one is
> the
> > > authenticated user (centralized user) and another is impersonated user.
> > We
> > > will first check if the authenticated user allowed to impersonate and
> > then
> > > move on to check if the user Alice has access to the topic "X" to
> > > read/write.
> > >
> > > @Rajini Intention of this KIP is to support token auth via SASL/SCRAM,
> > not
> > > just with TLS.  What you raised is a good point let me take a look and
> > add
> > > details.
> > >
> > > It will be easier to add impersonation once we reach agreement on this
> > KIP.
> > >
> > >
> > > On Wed, Dec 14, 2016 at 5:51 AM Ismael Juma <is...@juma.me.uk> wrote:
> > >
> > > > Hi Rajini,
> > > >
> > > > I think it would definitely be valuable to have a KIP for
> > impersonation.
> > > >
> > > > Ismael
> > > >
> > > > On Wed, Dec 14, 2016 at 4:03 AM, Rajini Sivaram <rsivaram@pivotal.io
> >
> > > > wrote:
> > > >
> > > > > It would clearly be very useful to enable clients to send requests
> on
> > > > > behalf of multiple users. A separate KIP makes sense, but it may be
> > > worth
> > > > > thinking through some of the implications now, especially if the
> main
> > > > > interest in delegation tokens comes from its potential to enable
> > > > > impersonation.
> > > > >
> > > > > I understand that delegation tokens are only expected to be used
> with
> > > > TLS.
> > > > > But the choice of SASL/SCRAM for authentication must be based on a
> > > > > requirement to protect the tokenHmac - otherwise you could just use
> > > > > SASL/PLAIN. With SASL/SCRAM the tokenHmac is never propagated
> > > > on-the-wire,
> > > > > only a salted-hashed version of it is used in the SASL
> authentication
> > > > > exchange. If impersonation is based on sending tokenHmac in
> requests,
> > > any
> > > > > benefit of using SCRAM is lost.
> > > > >
> > > > > An alternative may be to allow clients to authenticate multiple
> times
> > > > using
> > > > > SASL and include one of its authenticated principals in each
> request
> > > > > (optionally). I haven't thought it through yet, obviously. But if
> the
> > > > > approach is of interest and no one is working on a KIP for
> > > impersonation
> > > > at
> > > > > the moment, I am happy to write one. It may provide something for
> > > > > comparison at least.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > >
> > > > > On Wed, Dec 14, 2016 at 9:53 AM, Manikumar <
> > manikumar.reddy@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > That's a good idea. Authenticating every request with delegation
> > > token
> > > > > will
> > > > > > be useful for
> > > > > > impersonation use-cases. But as of now, we are thinking
> delegation
> > > > token
> > > > > as
> > > > > > just another way
> > > > > > to authenticate the users. We haven't think through all the use
> > cases
> > > > > > related to
> > > > > > impersonation or using delegation token for impersonation. We
> want
> > to
> > > > > > handle impersonation
> > > > > > (KAFKA-3712) as part of separate KIP.
> > > > > >
> > > > > > Will that be Ok?
> > > > > >
> > > > > >
> > > > > > On Wed, Dec 14, 2016 at 8:09 AM, Gwen Shapira <gwen@confluent.io
> >
> > > > wrote:
> > > > > >
> > > > > > > Thinking out loud here:
> > > > > > >
> > > > > > > It looks like authentication with a delegation token is going
> to
> > be
> > > > > > > super-cheap, right? We just compare the token to a value in the
> > > > broker
> > > > > > > cache?
> > > > > > >
> > > > > > > If I understood the KIP correctly, right now it suggests that
> > > > > > > authentication happens when establishing the client-broker
> > > connection
> > > > > (as
> > > > > > > normal for Kafka. But perhaps we want to consider
> authenticating
> > > > every
> > > > > > > request with delegation token (if exists)?
> > > > > > >
> > > > > > > So a centralized app can create few producers, do the metadata
> > > > request
> > > > > > and
> > > > > > > broker discovery with its own user auth, but then use
> delegation
> > > > tokens
> > > > > > to
> > > > > > > allow performing produce/fetch requests as different users?
> > Instead
> > > > of
> > > > > > > having to re-connect for each impersonated user?
> > > > > > >
> > > > > > > This may over-complicate things quite a bit (basically adding
> > extra
> > > > > > > information in every request), but maybe it will be useful for
> > > > > > > impersonation use-cases (which seem to drive much of the
> interest
> > > in
> > > > > this
> > > > > > > KIP)?
> > > > > > > Kafka Connect, NiFi and friends can probably use this to share
> > > > clients
> > > > > > > between multiple jobs, tasks, etc.
> > > > > > >
> > > > > > > What do you think?
> > > > > > >
> > > > > > > Gwen
> > > > > > >
> > > > > > > On Tue, Dec 13, 2016 at 12:43 AM, Manikumar <
> > > > manikumar.reddy@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Ashish,
> > > > > > > >
> > > > > > > > Thank you for reviewing the KIP.  Please see the replies
> > inline.
> > > > > > > >
> > > > > > > >
> > > > > > > > > 1. How to disable delegation token authentication?
> > > > > > > > >
> > > > > > > > > This can be achieved in various ways, however I think
> reusing
> > > > > > > delegation
> > > > > > > > > token secret config for this makes sense here. Avoids
> > creating
> > > > yet
> > > > > > > > another
> > > > > > > > > config and forces delegation token users to consciously set
> > the
> > > > > > secret.
> > > > > > > > If
> > > > > > > > > the secret is not set or set to empty string, brokers
> should
> > > turn
> > > > > off
> > > > > > > > > delegation token support. This will however require a new
> > error
> > > > > code
> > > > > > to
> > > > > > > > > indicate delegation token support is turned off on broker.
> > > > > > > > >
> > > > > > > >
> > > > > > > >   Thanks for the suggestion. Option to turnoff delegation
> token
> > > > > > > > authentication will be useful.
> > > > > > > >   I'll update the KIP.
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 2. ACLs on delegation token?
> > > > > > > > >
> > > > > > > > > Do we need to have ACLs defined for tokens? I do not think
> it
> > > > buys
> > > > > us
> > > > > > > > > anything, as delegation token can be treated as
> impersonation
> > > of
> > > > > the
> > > > > > > > owner.
> > > > > > > > > Any thing the owner has permission to do, delegation tokens
> > > > should
> > > > > be
> > > > > > > > > allowed to do as well. If so, we probably won't need to
> > return
> > > > > > > > > authorization exception error code while creating
> delegation
> > > > token.
> > > > > > It
> > > > > > > > > however would make sense to check renew and expire requests
> > are
> > > > > > coming
> > > > > > > > from
> > > > > > > > > owner or renewers of the token, but that does not require
> > > > explicit
> > > > > > > acls.
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Yes, We agreed to not have new acl on who can request
> > delegation
> > > > > token.
> > > > > > > >  I'll update the KIP.
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 3. How to restrict max life time of a token?
> > > > > > > > >
> > > > > > > > > Admins might want to restrict max life time of tokens
> created
> > > on
> > > > a
> > > > > > > > cluster,
> > > > > > > > > and this can very from cluster to cluster based on
> use-cases.
> > > > This
> > > > > > > might
> > > > > > > > > warrant a separate broker config.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > Currently we  have "delegation.token.max.lifetime.sec"
> server
> > > > config
> > > > > > > > property
> > > > > > > > May be we can take min(User supplied MaxTime, Server MaxTime)
> > as
> > > > max
> > > > > > life
> > > > > > > > time.
> > > > > > > > I am open to add new config property.
> > > > > > > >
> > > > > > > > Few more comments based on recent KIP update.
> > > > > > > > >
> > > > > > > > > 1. Do we need a separate {{InvalidateTokenRequest}}? Can't
> we
> > > use
> > > > > > > > > {{ExpireTokenRequest}} with with expiryDate set to anything
> > > > before
> > > > > > > > current
> > > > > > > > > date?
> > > > > > > > >
> > > > > > > >
> > > > > > > > makes sense. we don't need special request to cancel the
> token.
> > > We
> > > > > can
> > > > > > > use
> > > > > > > > ExpireTokenRequest.
> > > > > > > > I'll update the KIP.
> > > > > > > >
> > > > > > > >
> > > > > > > > > 2. Can we change time field names to indicate their unit is
> > > > > > > milliseconds,
> > > > > > > > > like, IssueDateMs, ExpiryDateMs, etc.?
> > > > > > > > >
> > > > > > > > >
> > > > > > > >   Done.
> > > > > > > >
> > > > > > > >
> > > > > > > > > 3. Can we allow users to renew a token for a specified
> amount
> > > of
> > > > > > time?
> > > > > > > In
> > > > > > > > > current version of KIP, renew request does not take time
> as a
> > > > > param,
> > > > > > > not
> > > > > > > > > sure what is expiry time set to after renewal.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >  Yes, we need to specify renew period.  I'll update the KIP.
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Mankumar
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Dec 12, 2016 at 9:08 AM Manikumar <
> > > > > manikumar.reddy@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I would like to reinitiate the discussion on Delegation
> > token
> > > > > > support
> > > > > > > > for
> > > > > > > > > >
> > > > > > > > > > Kafka.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Brief summary of the past discussion:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 1) Broker stores delegation tokens in zookeeper.  All
> > brokers
> > > > > will
> > > > > > > > have a
> > > > > > > > > >
> > > > > > > > > > cache backed by
> > > > > > > > > >
> > > > > > > > > >    zookeeper so they will all get notified whenever a new
> > > token
> > > > > is
> > > > > > > > > >
> > > > > > > > > > generated and they will
> > > > > > > > > >
> > > > > > > > > >    update their local cache whenever token state changes.
> > > > > > > > > >
> > > > > > > > > > 2) The current proposal does not support rotation of
> secret
> > > > > > > > > >
> > > > > > > > > > 3) Only allow the renewal by users that authenticated
> using
> > > > *non*
> > > > > > > > > >
> > > > > > > > > > delegation token mechanism
> > > > > > > > > >
> > > > > > > > > > 4) KIP-84 proposes to support  SASL SCRAM mechanisms.
> Kafka
> > > > > clients
> > > > > > > can
> > > > > > > > > >
> > > > > > > > > > authenticate using
> > > > > > > > > >
> > > > > > > > > >    SCRAM-SHA-256, providing the delegation token HMAC as
> > > > > password.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Updated the KIP with the following:
> > > > > > > > > >
> > > > > > > > > > 1. Protocol and Config changes
> > > > > > > > > >
> > > > > > > > > > 2. format of the data stored in ZK.
> > > > > > > > > >
> > > > > > > > > > 3. Changes to Java Clients/Usage of SASL SCRAM mechanism
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > > > >
> > > > > > > > > > 48+Delegation+token+support+for+Kafka
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Jun, Ashish, Gwen,
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Pl review the updated KIP.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > Manikumar
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 29, 2016 at 9:56 PM, Ashish Singh <
> > > > > asingh@cloudera.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > Harsha/ Gwen,
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > How do we proceed here? I am willing to help out with
> > here.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > On Fri, Sep 23, 2016 at 11:41 AM, Gwen Shapira <
> > > > > > gwen@confluent.io>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > Is it updated? are all concerns addressed? do you
> want
> > to
> > > > > > start a
> > > > > > > > > vote?
> > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > Sorry for being pushy, I do appreciate that we are
> all
> > > > > > volunteers
> > > > > > > > and
> > > > > > > > > >
> > > > > > > > > > > > finding time is difficult. This feature is important
> > for
> > > > > > anything
> > > > > > > > > that
> > > > > > > > > >
> > > > > > > > > > > > integrates with Kafka (stream processors, Flume,
> NiFi,
> > > etc)
> > > > > > and I
> > > > > > > > > >
> > > > > > > > > > > > don't want to see this getting stuck because we lack
> > > > > > coordination
> > > > > > > > > >
> > > > > > > > > > > > within the community.
> > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 15, 2016 at 6:39 PM, Harsha Chintalapani
> <
> > > > > > > > > kafka@harsha.io>
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > > The only pending update for the KIP is to write up
> > the
> > > > > > protocol
> > > > > > > > > > changes
> > > > > > > > > >
> > > > > > > > > > > > like
> > > > > > > > > >
> > > > > > > > > > > > > we've it KIP-4. I'll update the wiki.
> > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 15, 2016 at 4:27 PM Ashish Singh <
> > > > > > > > asingh@cloudera.com>
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > > > > >> I think we decided to not support secret
> rotation, I
> > > > guess
> > > > > > > this
> > > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > >
> > > > > > > > > > > > >> stated clearly on the KIP. Also, more details on
> how
> > > > > clients
> > > > > > > > will
> > > > > > > > > >
> > > > > > > > > > > > perform
> > > > > > > > > >
> > > > > > > > > > > > >> token distribution and how CLI will look like will
> > be
> > > > > > helpful.
> > > > > > > > > >
> > > > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > > > > >> On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <
> > > > > > > > gwen@confluent.io>
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > > > > >> > Hi Guys,
> > > > > > > > > >
> > > > > > > > > > > > >> >
> > > > > > > > > >
> > > > > > > > > > > > >> > This discussion was dead for a while. Are there
> > > still
> > > > > > > > > contentious
> > > > > > > > > >
> > > > > > > > > > > > >> > points? If not, why are there no votes?
> > > > > > > > > >
> > > > > > > > > > > > >> >
> > > > > > > > > >
> > > > > > > > > > > > >> > On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <
> > > > > > jun@confluent.io>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > > Ashish,
> > > > > > > > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > >
> > > > > > > > > > > > >> > > Yes, I will send out a KIP invite for next
> week
> > to
> > > > > > discuss
> > > > > > > > > > KIP-48
> > > > > > > > > >
> > > > > > > > > > > > and
> > > > > > > > > >
> > > > > > > > > > > > >> > other
> > > > > > > > > >
> > > > > > > > > > > > >> > > remaining KIPs.
> > > > > > > > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > >
> > > > > > > > > > > > >> > > Thanks,
> > > > > > > > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > >
> > > > > > > > > > > > >> > > Jun
> > > > > > > > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > >
> > > > > > > > > > > > >> > > On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh
> <
> > > > > > > > > >
> > > > > > > > > > > asingh@cloudera.com>
> > > > > > > > > >
> > > > > > > > > > > > >> > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >> Thanks Harsha!
> > > > > > > > > >
> > > > > > > > > > > > >> > >>
> > > > > > > > > >
> > > > > > > > > > > > >> > >> Jun, can we add KIP-48 to next KIP hangout's
> > > > agenda.
> > > > > > > Also,
> > > > > > > > we
> > > > > > > > > > did
> > > > > > > > > >
> > > > > > > > > > > > not
> > > > > > > > > >
> > > > > > > > > > > > >> > >> actually make a call on when we should have
> > next
> > > > KIP
> > > > > > > call.
> > > > > > > > As
> > > > > > > > > >
> > > > > > > > > > > there
> > > > > > > > > >
> > > > > > > > > > > > >> > >> are
> > > > > > > > > >
> > > > > > > > > > > > >> > a
> > > > > > > > > >
> > > > > > > > > > > > >> > >> few outstanding KIPs that could not be
> > discussed
> > > > this
> > > > > > > week,
> > > > > > > > > can
> > > > > > > > > >
> > > > > > > > > > > we
> > > > > > > > > >
> > > > > > > > > > > > >> > >> have
> > > > > > > > > >
> > > > > > > > > > > > >> > a
> > > > > > > > > >
> > > > > > > > > > > > >> > >> KIP hangout call next week?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>
> > > > > > > > > >
> > > > > > > > > > > > >> > >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha
> > > > Chintalapani
> > > > > > > > > >
> > > > > > > > > > > > >> > >> <ka...@harsha.io>
> > > > > > > > > >
> > > > > > > > > > > > >> > >> wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> Ashish,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>>         Yes we are working on it. Lets
> discuss
> > > in
> > > > > the
> > > > > > > next
> > > > > > > > > KIP
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> meeting.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> I'll join.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> -Harsha
> > > > > > > > > >
> > > > > > > > > > > > >> > >>>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish
> Singh
> > <
> > > > > > > > > >
> > > > > > > > > > > > asingh@cloudera.com>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > Hello Harsha,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > Are you still working on this? Wondering
> if
> > we
> > > > can
> > > > > > > > discuss
> > > > > > > > > >
> > > > > > > > > > > this
> > > > > > > > > >
> > > > > > > > > > > > in
> > > > > > > > > >
> > > > > > > > > > > > >> > next
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> KIP
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > meeting, if you can join.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha
> > > > > > Chintalapani <
> > > > > > > > > >
> > > > > > > > > > > > >> > kafka@harsha.io>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > Hi Grant,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > >           We are working on it. Will add
> > the
> > > > > > details
> > > > > > > > to
> > > > > > > > > > KIP
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > about
> > > > > > > > > >
> > > > > > > > > > > > >> > the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > request protocol.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > Thanks,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > Harsha
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant
> > Henke
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > <gh...@cloudera.com>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > Hi Parth,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > Are you still working on this? If you
> > need
> > > > any
> > > > > > > help
> > > > > > > > > > please
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > don't
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > hesitate
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > to ask.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > Grant
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun
> > Rao <
> > > > > > > > > >
> > > > > > > > > > > jun@confluent.io>
> > > > > > > > > >
> > > > > > > > > > > > >> > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > Parth,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > Thanks for the reply.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > It makes sense to only allow the
> > renewal
> > > > by
> > > > > > > users
> > > > > > > > > that
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> authenticated
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > using
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > *non* delegation token mechanism.
> > Then,
> > > > > should
> > > > > > > we
> > > > > > > > > make
> > > > > > > > > >
> > > > > > > > > > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> renewal a
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > list?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > For example, in the case of rest
> > proxy,
> > > it
> > > > > > will
> > > > > > > be
> > > > > > > > > >
> > > > > > > > > > > useful
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > for
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> every
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > instance of rest proxy to be able to
> > > renew
> > > > > the
> > > > > > > > > tokens.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > It would be clearer if we can
> document
> > > the
> > > > > > > request
> > > > > > > > > >
> > > > > > > > > > > > protocol
> > > > > > > > > >
> > > > > > > > > > > > >> > like
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > https://cwiki.apache.org/confl
> > > > > > > > > uence/display/KAFKA/KIP-
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > 4+-+Command+line+and+
> > > > > centralized+administrative+
> > > > > > > > > >
> > > > > > > > > > > > operations#KIP-4-
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > Commandlineandcentralizedadmin
> > > > > > istrativeoperations-
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > CreateTopicsRequest(KAFKA-
> > > > > > > > 2945):(VotedandPlannedforin0.
> > > > > > > > > >
> > > > > > > > > > > > 10.1.0)
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > .
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > It would also be useful to document
> > the
> > > > > client
> > > > > > > > APIs.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > Jun
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM,
> parth
> > > > > > > brahmbhatt
> > > > > > > > <
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > I am suggesting that we will only
> > > allow
> > > > > the
> > > > > > > > > renewal
> > > > > > > > > > by
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > users
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> that
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > authenticated using *non*
> delegation
> > > > token
> > > > > > > > > > mechanism.
> > > > > > > > > >
> > > > > > > > > > > > For
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> example,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > If
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > user
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > Alice authenticated using kerberos
> > and
> > > > > > > requested
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > delegation
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> tokens,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > only
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > user Alice authenticated via non
> > > > > delegation
> > > > > > > > token
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > mechanism
> > > > > > > > > >
> > > > > > > > > > > > >> > can
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > renew.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > Clients that have  access to
> > > delegation
> > > > > > tokens
> > > > > > > > can
> > > > > > > > > > not
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > issue
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > renewal
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > request for renewing their own
> token
> > > and
> > > > > > this
> > > > > > > is
> > > > > > > > > >
> > > > > > > > > > > > primarily
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > important
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > reduce the time window for which a
> > > > > > compromised
> > > > > > > > > token
> > > > > > > > > >
> > > > > > > > > > > > will
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > be
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> valid.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > To clarify, Yes any authenticated
> > user
> > > > can
> > > > > > > > request
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > delegation
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > tokens
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > but
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > even here I would recommend to
> avoid
> > > > > > creating
> > > > > > > a
> > > > > > > > > > chain
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > where a
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > client
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > authenticated via delegation token
> > > > request
> > > > > > for
> > > > > > > > > more
> > > > > > > > > >
> > > > > > > > > > > > >> > delegation
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > tokens.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > Basically anyone can request
> > > delegation
> > > > > > token,
> > > > > > > > as
> > > > > > > > > > long
> > > > > > > > > >
> > > > > > > > > > > > as
> > > > > > > > > >
> > > > > > > > > > > > >> > they
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > authenticate
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > via a non delegation token
> > mechanism.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > Aren't classes listed here
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > <
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > https://cwiki.apache.org/confl
> > > > > > > > > uence/display/KAFKA/KIP-
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > 48+Delegation+token+support+fo
> > > > > > > > > >
> > > > > > > > > > > r+Kafka#KIP-48Delegationtokens
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> upportforKaf
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > ka-PublicInterfaces
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > sufficient?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > Parth
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM,
> Jun
> > > Rao
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > <ju...@confluent.io>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > Parth,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > Thanks for the reply. A couple
> of
> > > > > comments
> > > > > > > > > inline
> > > > > > > > > >
> > > > > > > > > > > > below.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > On Tue, Jun 21, 2016 at 10:36
> AM,
> > > > parth
> > > > > > > > > > brahmbhatt <
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > brahmbhatt.parth@gmail.com>
> > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > 1. Who / how are tokens
> renewed?
> > > By
> > > > > > > original
> > > > > > > > > >
> > > > > > > > > > > > requester
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> only? or
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > using
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > Kerberos
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > auth only?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > My recommendation is to do
> this
> > > only
> > > > > > using
> > > > > > > > > >
> > > > > > > > > > > Kerberos
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > auth
> > > > > > > > > >
> > > > > > > > > > > > >> > and
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > only
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > threw
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > renewer specified during the
> > > > > acquisition
> > > > > > > > > > request.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > Hmm, not sure that I follow
> this.
> > > Are
> > > > > you
> > > > > > > > saying
> > > > > > > > > >
> > > > > > > > > > > that
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > any
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> client
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > authenticated with the
> delegation
> > > > token
> > > > > > can
> > > > > > > > > renew,
> > > > > > > > > >
> > > > > > > > > > > > i.e.
> > > > > > > > > >
> > > > > > > > > > > > >> > there
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> is
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > no
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > renewer
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > needed?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > Also, just to be clear, any
> > > > > authenticated
> > > > > > > > client
> > > > > > > > > >
> > > > > > > > > > > > (either
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> through
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > SASL
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > or
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > SSL) can request a delegation
> > token
> > > > for
> > > > > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > authenticated
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> user,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > right?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > 2. Are tokens stored on each
> > > broker
> > > > or
> > > > > > in
> > > > > > > > ZK?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > My recommendation is still to
> > > store
> > > > in
> > > > > > ZK
> > > > > > > or
> > > > > > > > > not
> > > > > > > > > >
> > > > > > > > > > > > store
> > > > > > > > > >
> > > > > > > > > > > > >> > them
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> at
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > all.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > The
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > whole controller based
> > > distribution
> > > > is
> > > > > > too
> > > > > > > > > much
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > overhead
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> with
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > not
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > much
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > achieve.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > 3. How are tokens invalidated
> /
> > > > > expired?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > Either by expiration time out
> or
> > > > > through
> > > > > > > an
> > > > > > > > > >
> > > > > > > > > > > explicit
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> request to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > invalidate.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > 4. Which encryption algorithm
> is
> > > > used?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > SCRAM
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > 5. What is the impersonation
> > > > proposal
> > > > > > (it
> > > > > > > > > wasn't
> > > > > > > > > >
> > > > > > > > > > > in
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > KIP
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> but
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > was
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > discussed
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > in this thread)?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > There is no imperonation
> > > proposal. I
> > > > > > tried
> > > > > > > > and
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > explained
> > > > > > > > > >
> > > > > > > > > > > > >> > how
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > its
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > a
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > different problem and why its
> > not
> > > > > really
> > > > > > > > > > necessary
> > > > > > > > > >
> > > > > > > > > > > > to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> discuss
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > that
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > as
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > part
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > of this KIP.  This KIP will
> not
> > > > > support
> > > > > > > any
> > > > > > > > > >
> > > > > > > > > > > > >> > impersonation,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> it
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > will
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > just
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > be
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > another way to authenticate.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > 6. Do we need new ACLs, if so
> -
> > > for
> > > > > what
> > > > > > > > > > actions?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > We do not need new ACLs.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > Could we document the format of
> > the
> > > > new
> > > > > > > > > >
> > > > > > > > > > > > request/response
> > > > > > > > > >
> > > > > > > > > > > > >> > and
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > their
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > associated Resource and
> Operation
> > > for
> > > > > ACL?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > 7. How would the delegation
> > token
> > > be
> > > > > > > > > configured
> > > > > > > > > > in
> > > > > > > > > >
> > > > > > > > > > > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> client?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > Should be through config. I
> > wasn't
> > > > > > > planning
> > > > > > > > on
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > supporting
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> JAAS
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > for
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > tokens.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > I don't believe hadoop does
> this
> > > > > either.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > Parth
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > On Thu, Jun 16, 2016 at 4:03
> PM,
> > > Jun
> > > > > > Rao <
> > > > > > > > > >
> > > > > > > > > > > > >> > jun@confluent.io>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > Harsha,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > Another question.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > 9. How would the delegation
> > > token
> > > > be
> > > > > > > > > > configured
> > > > > > > > > >
> > > > > > > > > > > in
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > client?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > The
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > standard
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > way is to do this through
> > JAAS.
> > > > > > However,
> > > > > > > > we
> > > > > > > > > > will
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > need
> > > > > > > > > >
> > > > > > > > > > > > >> > to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > think
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > through
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > if
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > this is convenient in a
> shared
> > > > > > > > environment.
> > > > > > > > > > For
> > > > > > > > > >
> > > > > > > > > > > > >> > example,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > when a
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > new
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > task
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > is
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > added to a Storm worker
> node,
> > do
> > > > we
> > > > > > need
> > > > > > > > to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > dynamically
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> add a
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > new
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > section
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > in the JAAS file? It may be
> > more
> > > > > > > > convenient
> > > > > > > > > if
> > > > > > > > > >
> > > > > > > > > > > we
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > can
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> pass in
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > token
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > through the config directly
> > w/o
> > > > > going
> > > > > > > > > through
> > > > > > > > > >
> > > > > > > > > > > > JAAS.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > Are you or Parth still
> > actively
> > > > > > working
> > > > > > > on
> > > > > > > > > > this
> > > > > > > > > >
> > > > > > > > > > > > KIP?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18
> > PM,
> > > > Jun
> > > > > > > Rao <
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> jun@confluent.io>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > Just to add on that list.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > 2. It would be good to
> > > document
> > > > > the
> > > > > > > > format
> > > > > > > > > > of
> > > > > > > > > >
> > > > > > > > > > > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > data
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > stored
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > in
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > ZK.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > 7. Earlier, there was a
> > > > discussion
> > > > > > on
> > > > > > > > > > whether
> > > > > > > > > >
> > > > > > > > > > > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > tokens
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > should
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > be
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > propagated through ZK like
> > > > > > > > > config/acl/quota,
> > > > > > > > > >
> > > > > > > > > > > or
> > > > > > > > > >
> > > > > > > > > > > > >> > through
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > controller.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > Currently, the controller
> is
> > > > only
> > > > > > > > designed
> > > > > > > > > > for
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> propagating
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > topic
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > metadata,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > but not other data.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > 8. Should we use SCRAM to
> > send
> > > > the
> > > > > > > token
> > > > > > > > > >
> > > > > > > > > > > instead
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > of
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > DIGEST-MD5
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > since
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > it's
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > deprecated?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > Also, the images in the
> wiki
> > > > seem
> > > > > > > > broken.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > > On Fri, Jun 10, 2016 at
> > 10:02
> > > > AM,
> > > > > > Gwen
> > > > > > > > > >
> > > > > > > > > > > Shapira <
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > gwen@confluent.io>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> From what I can see,
> > > remaining
> > > > > > > > questions
> > > > > > > > > > are:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> 1. Who / how are tokens
> > > > renewed?
> > > > > By
> > > > > > > > > > original
> > > > > > > > > >
> > > > > > > > > > > > >> > requester
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > only?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > or
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > using
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> Kerberos auth only?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> 2. Are tokens stored on
> > each
> > > > > broker
> > > > > > > or
> > > > > > > > in
> > > > > > > > > > ZK?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> 3. How are tokens
> > > invalidated /
> > > > > > > > expired?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> 4. Which encryption
> > algorithm
> > > > is
> > > > > > > used?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> 5. What is the
> > impersonation
> > > > > > proposal
> > > > > > > > (it
> > > > > > > > > >
> > > > > > > > > > > > wasn't
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> in
> > > > > > > > > >
> > > > > > > > > > > > >> > the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > KIP
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > but
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > was
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> discussed in this
> thread)?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> 6. Do we need new ACLs,
> if
> > > so -
> > > > > for
> > > > > > > > what
> > > > > > > > > >
> > > > > > > > > > > > actions?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> Gwen
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> On Thu, Jun 9, 2016 at
> 7:48
> > > PM,
> > > > > > > Harsha
> > > > > > > > <
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> kafka@harsha.io>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> > Jun & Ismael,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
> > > > > > > > Unfortunately
> > > > > > > > > I
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> > couldn't
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> attend
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > KIP
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > meeting
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
> > > when
> > > > > > > > > delegation
> > > > > > > > > >
> > > > > > > > > > > > tokens
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > discussed.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > Appreciate
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > if
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
> > > you
> > > > > can
> > > > > > > > update
> > > > > > > > > > the
> > > > > > > > > >
> > > > > > > > > > > > >> > thread if
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > you
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > have
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > any
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
> > > > > further
> > > > > > > > > > questions.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> > Thanks,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> > Harsha
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> > On Tue, May 24, 2016,
> at
> > > > 11:32
> > > > > > AM,
> > > > > > > > > Liquan
> > > > > > > > > >
> > > > > > > > > > > Pei
> > > > > > > > > >
> > > > > > > > > > > > >> > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> It seems that the
> links
> > to
> > > > > > images
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > >
> > > > > > > > > > > KIP
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> are
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > broken.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> Liquan
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> On Tue, May 24, 2016
> at
> > > 9:33
> > > > > AM,
> > > > > > > > parth
> > > > > > > > > >
> > > > > > > > > > > > >> > brahmbhatt <
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >>
> > > brahmbhatt.parth@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >>
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > 110. What does
> > > > > > > > getDelegationTokenAs
> > > > > > > > > >
> > > > > > > > > > > mean?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > In the current
> > proposal
> > > we
> > > > > > only
> > > > > > > > > allow
> > > > > > > > > > a
> > > > > > > > > >
> > > > > > > > > > > > user
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> get
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > delegation
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > token
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> for
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > the identity that it
> > > > > > > authenticated
> > > > > > > > > as
> > > > > > > > > >
> > > > > > > > > > > > using
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> another
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > mechanism,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > i.e.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> A user
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > that authenticate
> > using
> > > a
> > > > > > keytab
> > > > > > > > for
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > principal
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > user1@EXAMPLE.COM
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> will get
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens
> for
> > > that
> > > > > > user
> > > > > > > > > only.
> > > > > > > > > > In
> > > > > > > > > >
> > > > > > > > > > > > >> > future I
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > think
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > we
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > will
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > have
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > extend support such
> > that
> > > > we
> > > > > > > allow
> > > > > > > > > some
> > > > > > > > > >
> > > > > > > > > > > set
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > of
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> users (
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > kafka-rest-user@EXAMPLE.COM
> > > > > ,
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > storm-nimbus@EXAMPLE.COM)
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > acquire
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens on
> > > > behalf
> > > > > of
> > > > > > > > other
> > > > > > > > > >
> > > > > > > > > > > users
> > > > > > > > > >
> > > > > > > > > > > > >> > whose
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > identity
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > they
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > have
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > verified
> > independently.
> > > > > Kafka
> > > > > > > > > brokers
> > > > > > > > > >
> > > > > > > > > > > > will
> > > > > > > > > >
> > > > > > > > > > > > >> > have
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> ACLs
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > to
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > control
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> which
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > users are allowed to
> > > > > > impersonate
> > > > > > > > > other
> > > > > > > > > >
> > > > > > > > > > > > users
> > > > > > > > > >
> > > > > > > > > > > > >> > and
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> get
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > tokens
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > on
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> behalf of
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > them. Overall
> > > > Impersonation
> > > > > > is a
> > > > > > > > > whole
> > > > > > > > > >
> > > > > > > > > > > > >> > different
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > problem
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > in
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > my
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> opinion and
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > I think we can
> tackle
> > it
> > > > in
> > > > > > > > separate
> > > > > > > > > >
> > > > > > > > > > > KIP.
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > 111. What's the
> > typical
> > > > rate
> > > > > > of
> > > > > > > > > > getting
> > > > > > > > > >
> > > > > > > > > > > > and
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> renewing
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > delegation
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> tokens?
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > Typically this
> should
> > be
> > > > > very
> > > > > > > very
> > > > > > > > > > low,
> > > > > > > > > >
> > > > > > > > > > > 1
> > > > > > > > > >
> > > > > > > > > > > > >> > request
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> per
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > minute
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > is a
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > relatively high
> > > estimate.
> > > > > > > However
> > > > > > > > it
> > > > > > > > > >
> > > > > > > > > > > > depends
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > > > > > > > >> >> > on
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> the
> > > > > > > > > >
> > > > > > > > > > > > >> > >>> > > token
> > > > > > > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Rajini Sivaram <rs...@pivotal.io>]

@Mani

Can you add a sample Jaas configuration using delegation tokens to the KIP?
Since delegation tokens will be handled differently from other SCRAM
credentials, it should work anyway, but it will be good to see an example
of the configuration the user provides. It sounds like users provide both
tokenID and delegation token, but I wasn't sure.

@Gwen, @Ismael, @Harsha, @Mani

To make sure I have understood correctly, KAFKA-3712 is aimed at enabling a
superuser to impersonate another (single) user, say alice. A producer using
impersonation will authenticate with superuser credentials. All requests
from the producer will be run with the principal alice. But alice is not
involved in the authentication and alice's credentials are not actually
provided to the broker?

The use case I was thinking of was the other one on Mani's list above. I
want to make sure that my understanding matches Gwen's and Ismael's for
this one. Using REST proxy as an example, users sending requests to the
REST proxy provide a delegation token as an auth token. REST proxy itself
authenticates as a different user, perhaps with access to read metadata of
all topics. But Kafka requests corresponding to each REST request should be
authenticated based on the authentication token provided in the request.
At the moment, the REST proxy has to create producers for each user (or
perform authentication and authorization for the user). Instead we want to
create a single producer which has the ability to send requests on behalf
of multiple users where the user is authenticated by Kafka and each request
from the producer is authorized based on the user who initiated the request.

1a) I think Gwen's suggestion was something along the lines of:
producer.send(record1, delegationToken1);
producer.send(record2, delegationToken2);
1b) I was thinking more along the lines of:
subject1 = producer.authenticate(user1Credentials);
subject2 = producer.authenticate(user2Credentials);
....
Subject.doAs(subject1, (PrivilegedExceptionAction<Future<RecordMetadata>>)
() -> producer.send(record1));
Subject.doAs(subject2, (PrivilegedExceptionAction<Future<RecordMetadata>>)
() -> producer.send(record2));

To make either approach work, each request needs to be associated with a
user. This would be delegation token in 1a) and user principal in 1b). And
both need to ensure that producers dont send records from multiple users in
one request. And if producers hold one metadata rather than one per-user,
calls like partitionsFor() shouldn't return metadata for unauthorized
topics. It is a very big change, so it will be good to know whether there
is a real requirement to support this.


On Thu, Dec 15, 2016 at 9:04 AM, Manikumar <ma...@gmail.com>
wrote:

> @Gwen, @Rajini,
>
> As mentioned in the KIP, main motivation for this KIP is to reduce load on
> Kerberos
> server on large kafka deployments with large number of clients.
>
> Also it looks like we are combining two overlapping concepts
> 1. Single client sending requests with multiple users/authentications
> 2. Impersonation
>
> Option 1, is definitely useful in some use cases and can be used to
> implement workaround for
> impersonation
>
> In Impersonation, a super user can send requests on behalf of another
> user(Alice) in a secured way.
> superuser has credentials but user Alice doesn't have any. The requests are
> required
> to run as user Alice and accesses/ACLs on Broker are required to be done as
> user Alice.
> It is required that user Alice can connect to the Broker on a connection
> authenticated with
> superuser's credentials. In other words superuser is impersonating the user
> Alice.
>
> The approach mentioned by Harsha in previous mail is implemented in hadoop,
> storm etc..
>
> Some more details here:
> https://hadoop.apache.org/docs/r2.7.2/hadoop-project-
> dist/hadoop-common/Superusers.html
>
>
> @Rajini
>
> Thanks for your comments on SASL/SCRAM usage. I am thinking to send
> tokenHmac (salted-hashed version)
> as password for authentication and tokenID for retrial of tokenHmac at
> server side.
> Does above sound OK?
>
>
> Thanks,
> Manikumar
>
> On Wed, Dec 14, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> > @Gwen @Mani  Not sure why we want to authenticate at every request. Even
> if
> > the token exchange is cheap it still a few calls that need to go through
> > round trip.  Impersonation doesn't require authentication for every
> > request.
> >
> > "So a centralized app can create few producers, do the metadata request
> and
> > broker discovery with its own user auth, but then use delegation tokens
> to
> > allow performing produce/fetch requests as different users? Instead of
> > having to re-connect for each impersonated user?"
> >
> > Yes. But what we will have is this centralized user as impersonation user
> > on behalf of other users. When it authenticates initially we will create
> a
> > "Subject" and from there on wards centralized user can do
> > Subject.doAsPrivileged
> > on behalf, other users.
> > On the server side, we can retrieve two principals out of this one is the
> > authenticated user (centralized user) and another is impersonated user.
> We
> > will first check if the authenticated user allowed to impersonate and
> then
> > move on to check if the user Alice has access to the topic "X" to
> > read/write.
> >
> > @Rajini Intention of this KIP is to support token auth via SASL/SCRAM,
> not
> > just with TLS.  What you raised is a good point let me take a look and
> add
> > details.
> >
> > It will be easier to add impersonation once we reach agreement on this
> KIP.
> >
> >
> > On Wed, Dec 14, 2016 at 5:51 AM Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Rajini,
> > >
> > > I think it would definitely be valuable to have a KIP for
> impersonation.
> > >
> > > Ismael
> > >
> > > On Wed, Dec 14, 2016 at 4:03 AM, Rajini Sivaram <rs...@pivotal.io>
> > > wrote:
> > >
> > > > It would clearly be very useful to enable clients to send requests on
> > > > behalf of multiple users. A separate KIP makes sense, but it may be
> > worth
> > > > thinking through some of the implications now, especially if the main
> > > > interest in delegation tokens comes from its potential to enable
> > > > impersonation.
> > > >
> > > > I understand that delegation tokens are only expected to be used with
> > > TLS.
> > > > But the choice of SASL/SCRAM for authentication must be based on a
> > > > requirement to protect the tokenHmac - otherwise you could just use
> > > > SASL/PLAIN. With SASL/SCRAM the tokenHmac is never propagated
> > > on-the-wire,
> > > > only a salted-hashed version of it is used in the SASL authentication
> > > > exchange. If impersonation is based on sending tokenHmac in requests,
> > any
> > > > benefit of using SCRAM is lost.
> > > >
> > > > An alternative may be to allow clients to authenticate multiple times
> > > using
> > > > SASL and include one of its authenticated principals in each request
> > > > (optionally). I haven't thought it through yet, obviously. But if the
> > > > approach is of interest and no one is working on a KIP for
> > impersonation
> > > at
> > > > the moment, I am happy to write one. It may provide something for
> > > > comparison at least.
> > > >
> > > > Thoughts?
> > > >
> > > >
> > > > On Wed, Dec 14, 2016 at 9:53 AM, Manikumar <
> manikumar.reddy@gmail.com>
> > > > wrote:
> > > >
> > > > > That's a good idea. Authenticating every request with delegation
> > token
> > > > will
> > > > > be useful for
> > > > > impersonation use-cases. But as of now, we are thinking delegation
> > > token
> > > > as
> > > > > just another way
> > > > > to authenticate the users. We haven't think through all the use
> cases
> > > > > related to
> > > > > impersonation or using delegation token for impersonation. We want
> to
> > > > > handle impersonation
> > > > > (KAFKA-3712) as part of separate KIP.
> > > > >
> > > > > Will that be Ok?
> > > > >
> > > > >
> > > > > On Wed, Dec 14, 2016 at 8:09 AM, Gwen Shapira <gw...@confluent.io>
> > > wrote:
> > > > >
> > > > > > Thinking out loud here:
> > > > > >
> > > > > > It looks like authentication with a delegation token is going to
> be
> > > > > > super-cheap, right? We just compare the token to a value in the
> > > broker
> > > > > > cache?
> > > > > >
> > > > > > If I understood the KIP correctly, right now it suggests that
> > > > > > authentication happens when establishing the client-broker
> > connection
> > > > (as
> > > > > > normal for Kafka. But perhaps we want to consider authenticating
> > > every
> > > > > > request with delegation token (if exists)?
> > > > > >
> > > > > > So a centralized app can create few producers, do the metadata
> > > request
> > > > > and
> > > > > > broker discovery with its own user auth, but then use delegation
> > > tokens
> > > > > to
> > > > > > allow performing produce/fetch requests as different users?
> Instead
> > > of
> > > > > > having to re-connect for each impersonated user?
> > > > > >
> > > > > > This may over-complicate things quite a bit (basically adding
> extra
> > > > > > information in every request), but maybe it will be useful for
> > > > > > impersonation use-cases (which seem to drive much of the interest
> > in
> > > > this
> > > > > > KIP)?
> > > > > > Kafka Connect, NiFi and friends can probably use this to share
> > > clients
> > > > > > between multiple jobs, tasks, etc.
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > > Gwen
> > > > > >
> > > > > > On Tue, Dec 13, 2016 at 12:43 AM, Manikumar <
> > > manikumar.reddy@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Ashish,
> > > > > > >
> > > > > > > Thank you for reviewing the KIP.  Please see the replies
> inline.
> > > > > > >
> > > > > > >
> > > > > > > > 1. How to disable delegation token authentication?
> > > > > > > >
> > > > > > > > This can be achieved in various ways, however I think reusing
> > > > > > delegation
> > > > > > > > token secret config for this makes sense here. Avoids
> creating
> > > yet
> > > > > > > another
> > > > > > > > config and forces delegation token users to consciously set
> the
> > > > > secret.
> > > > > > > If
> > > > > > > > the secret is not set or set to empty string, brokers should
> > turn
> > > > off
> > > > > > > > delegation token support. This will however require a new
> error
> > > > code
> > > > > to
> > > > > > > > indicate delegation token support is turned off on broker.
> > > > > > > >
> > > > > > >
> > > > > > >   Thanks for the suggestion. Option to turnoff delegation token
> > > > > > > authentication will be useful.
> > > > > > >   I'll update the KIP.
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > 2. ACLs on delegation token?
> > > > > > > >
> > > > > > > > Do we need to have ACLs defined for tokens? I do not think it
> > > buys
> > > > us
> > > > > > > > anything, as delegation token can be treated as impersonation
> > of
> > > > the
> > > > > > > owner.
> > > > > > > > Any thing the owner has permission to do, delegation tokens
> > > should
> > > > be
> > > > > > > > allowed to do as well. If so, we probably won't need to
> return
> > > > > > > > authorization exception error code while creating delegation
> > > token.
> > > > > It
> > > > > > > > however would make sense to check renew and expire requests
> are
> > > > > coming
> > > > > > > from
> > > > > > > > owner or renewers of the token, but that does not require
> > > explicit
> > > > > > acls.
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Yes, We agreed to not have new acl on who can request
> delegation
> > > > token.
> > > > > > >  I'll update the KIP.
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > 3. How to restrict max life time of a token?
> > > > > > > >
> > > > > > > > Admins might want to restrict max life time of tokens created
> > on
> > > a
> > > > > > > cluster,
> > > > > > > > and this can very from cluster to cluster based on use-cases.
> > > This
> > > > > > might
> > > > > > > > warrant a separate broker config.
> > > > > > > >
> > > > > > > >
> > > > > > > Currently we  have "delegation.token.max.lifetime.sec" server
> > > config
> > > > > > > property
> > > > > > > May be we can take min(User supplied MaxTime, Server MaxTime)
> as
> > > max
> > > > > life
> > > > > > > time.
> > > > > > > I am open to add new config property.
> > > > > > >
> > > > > > > Few more comments based on recent KIP update.
> > > > > > > >
> > > > > > > > 1. Do we need a separate {{InvalidateTokenRequest}}? Can't we
> > use
> > > > > > > > {{ExpireTokenRequest}} with with expiryDate set to anything
> > > before
> > > > > > > current
> > > > > > > > date?
> > > > > > > >
> > > > > > >
> > > > > > > makes sense. we don't need special request to cancel the token.
> > We
> > > > can
> > > > > > use
> > > > > > > ExpireTokenRequest.
> > > > > > > I'll update the KIP.
> > > > > > >
> > > > > > >
> > > > > > > > 2. Can we change time field names to indicate their unit is
> > > > > > milliseconds,
> > > > > > > > like, IssueDateMs, ExpiryDateMs, etc.?
> > > > > > > >
> > > > > > > >
> > > > > > >   Done.
> > > > > > >
> > > > > > >
> > > > > > > > 3. Can we allow users to renew a token for a specified amount
> > of
> > > > > time?
> > > > > > In
> > > > > > > > current version of KIP, renew request does not take time as a
> > > > param,
> > > > > > not
> > > > > > > > sure what is expiry time set to after renewal.
> > > > > > > >
> > > > > > > >
> > > > > > >  Yes, we need to specify renew period.  I'll update the KIP.
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mankumar
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Dec 12, 2016 at 9:08 AM Manikumar <
> > > > manikumar.reddy@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I would like to reinitiate the discussion on Delegation
> token
> > > > > support
> > > > > > > for
> > > > > > > > >
> > > > > > > > > Kafka.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Brief summary of the past discussion:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 1) Broker stores delegation tokens in zookeeper.  All
> brokers
> > > > will
> > > > > > > have a
> > > > > > > > >
> > > > > > > > > cache backed by
> > > > > > > > >
> > > > > > > > >    zookeeper so they will all get notified whenever a new
> > token
> > > > is
> > > > > > > > >
> > > > > > > > > generated and they will
> > > > > > > > >
> > > > > > > > >    update their local cache whenever token state changes.
> > > > > > > > >
> > > > > > > > > 2) The current proposal does not support rotation of secret
> > > > > > > > >
> > > > > > > > > 3) Only allow the renewal by users that authenticated using
> > > *non*
> > > > > > > > >
> > > > > > > > > delegation token mechanism
> > > > > > > > >
> > > > > > > > > 4) KIP-84 proposes to support  SASL SCRAM mechanisms. Kafka
> > > > clients
> > > > > > can
> > > > > > > > >
> > > > > > > > > authenticate using
> > > > > > > > >
> > > > > > > > >    SCRAM-SHA-256, providing the delegation token HMAC as
> > > > password.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Updated the KIP with the following:
> > > > > > > > >
> > > > > > > > > 1. Protocol and Config changes
> > > > > > > > >
> > > > > > > > > 2. format of the data stored in ZK.
> > > > > > > > >
> > > > > > > > > 3. Changes to Java Clients/Usage of SASL SCRAM mechanism
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > > >
> > > > > > > > > 48+Delegation+token+support+for+Kafka
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Jun, Ashish, Gwen,
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Pl review the updated KIP.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > Manikumar
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Sep 29, 2016 at 9:56 PM, Ashish Singh <
> > > > asingh@cloudera.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > Harsha/ Gwen,
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > > How do we proceed here? I am willing to help out with
> here.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > > On Fri, Sep 23, 2016 at 11:41 AM, Gwen Shapira <
> > > > > gwen@confluent.io>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > Is it updated? are all concerns addressed? do you want
> to
> > > > > start a
> > > > > > > > vote?
> > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > Sorry for being pushy, I do appreciate that we are all
> > > > > volunteers
> > > > > > > and
> > > > > > > > >
> > > > > > > > > > > finding time is difficult. This feature is important
> for
> > > > > anything
> > > > > > > > that
> > > > > > > > >
> > > > > > > > > > > integrates with Kafka (stream processors, Flume, NiFi,
> > etc)
> > > > > and I
> > > > > > > > >
> > > > > > > > > > > don't want to see this getting stuck because we lack
> > > > > coordination
> > > > > > > > >
> > > > > > > > > > > within the community.
> > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > On Thu, Sep 15, 2016 at 6:39 PM, Harsha Chintalapani <
> > > > > > > > kafka@harsha.io>
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > > > The only pending update for the KIP is to write up
> the
> > > > > protocol
> > > > > > > > > changes
> > > > > > > > >
> > > > > > > > > > > like
> > > > > > > > >
> > > > > > > > > > > > we've it KIP-4. I'll update the wiki.
> > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 15, 2016 at 4:27 PM Ashish Singh <
> > > > > > > asingh@cloudera.com>
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > > >>
> > > > > > > > >
> > > > > > > > > > > >> I think we decided to not support secret rotation, I
> > > guess
> > > > > > this
> > > > > > > > can
> > > > > > > > > be
> > > > > > > > >
> > > > > > > > > > > >> stated clearly on the KIP. Also, more details on how
> > > > clients
> > > > > > > will
> > > > > > > > >
> > > > > > > > > > > perform
> > > > > > > > >
> > > > > > > > > > > >> token distribution and how CLI will look like will
> be
> > > > > helpful.
> > > > > > > > >
> > > > > > > > > > > >>
> > > > > > > > >
> > > > > > > > > > > >> On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <
> > > > > > > gwen@confluent.io>
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > > >>
> > > > > > > > >
> > > > > > > > > > > >> > Hi Guys,
> > > > > > > > >
> > > > > > > > > > > >> >
> > > > > > > > >
> > > > > > > > > > > >> > This discussion was dead for a while. Are there
> > still
> > > > > > > > contentious
> > > > > > > > >
> > > > > > > > > > > >> > points? If not, why are there no votes?
> > > > > > > > >
> > > > > > > > > > > >> >
> > > > > > > > >
> > > > > > > > > > > >> > On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <
> > > > > jun@confluent.io>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > > >> > > Ashish,
> > > > > > > > >
> > > > > > > > > > > >> > >
> > > > > > > > >
> > > > > > > > > > > >> > > Yes, I will send out a KIP invite for next week
> to
> > > > > discuss
> > > > > > > > > KIP-48
> > > > > > > > >
> > > > > > > > > > > and
> > > > > > > > >
> > > > > > > > > > > >> > other
> > > > > > > > >
> > > > > > > > > > > >> > > remaining KIPs.
> > > > > > > > >
> > > > > > > > > > > >> > >
> > > > > > > > >
> > > > > > > > > > > >> > > Thanks,
> > > > > > > > >
> > > > > > > > > > > >> > >
> > > > > > > > >
> > > > > > > > > > > >> > > Jun
> > > > > > > > >
> > > > > > > > > > > >> > >
> > > > > > > > >
> > > > > > > > > > > >> > > On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh <
> > > > > > > > >
> > > > > > > > > > asingh@cloudera.com>
> > > > > > > > >
> > > > > > > > > > > >> > wrote:
> > > > > > > > >
> > > > > > > > > > > >> > >
> > > > > > > > >
> > > > > > > > > > > >> > >> Thanks Harsha!
> > > > > > > > >
> > > > > > > > > > > >> > >>
> > > > > > > > >
> > > > > > > > > > > >> > >> Jun, can we add KIP-48 to next KIP hangout's
> > > agenda.
> > > > > > Also,
> > > > > > > we
> > > > > > > > > did
> > > > > > > > >
> > > > > > > > > > > not
> > > > > > > > >
> > > > > > > > > > > >> > >> actually make a call on when we should have
> next
> > > KIP
> > > > > > call.
> > > > > > > As
> > > > > > > > >
> > > > > > > > > > there
> > > > > > > > >
> > > > > > > > > > > >> > >> are
> > > > > > > > >
> > > > > > > > > > > >> > a
> > > > > > > > >
> > > > > > > > > > > >> > >> few outstanding KIPs that could not be
> discussed
> > > this
> > > > > > week,
> > > > > > > > can
> > > > > > > > >
> > > > > > > > > > we
> > > > > > > > >
> > > > > > > > > > > >> > >> have
> > > > > > > > >
> > > > > > > > > > > >> > a
> > > > > > > > >
> > > > > > > > > > > >> > >> KIP hangout call next week?
> > > > > > > > >
> > > > > > > > > > > >> > >>
> > > > > > > > >
> > > > > > > > > > > >> > >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha
> > > Chintalapani
> > > > > > > > >
> > > > > > > > > > > >> > >> <ka...@harsha.io>
> > > > > > > > >
> > > > > > > > > > > >> > >> wrote:
> > > > > > > > >
> > > > > > > > > > > >> > >>
> > > > > > > > >
> > > > > > > > > > > >> > >>> Ashish,
> > > > > > > > >
> > > > > > > > > > > >> > >>>         Yes we are working on it. Lets discuss
> > in
> > > > the
> > > > > > next
> > > > > > > > KIP
> > > > > > > > >
> > > > > > > > > > > >> > >>> meeting.
> > > > > > > > >
> > > > > > > > > > > >> > >>> I'll join.
> > > > > > > > >
> > > > > > > > > > > >> > >>> -Harsha
> > > > > > > > >
> > > > > > > > > > > >> > >>>
> > > > > > > > >
> > > > > > > > > > > >> > >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh
> <
> > > > > > > > >
> > > > > > > > > > > asingh@cloudera.com>
> > > > > > > > >
> > > > > > > > > > > >> > >>> wrote:
> > > > > > > > >
> > > > > > > > > > > >> > >>>
> > > > > > > > >
> > > > > > > > > > > >> > >>> > Hello Harsha,
> > > > > > > > >
> > > > > > > > > > > >> > >>> >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > Are you still working on this? Wondering if
> we
> > > can
> > > > > > > discuss
> > > > > > > > >
> > > > > > > > > > this
> > > > > > > > >
> > > > > > > > > > > in
> > > > > > > > >
> > > > > > > > > > > >> > next
> > > > > > > > >
> > > > > > > > > > > >> > >>> KIP
> > > > > > > > >
> > > > > > > > > > > >> > >>> > meeting, if you can join.
> > > > > > > > >
> > > > > > > > > > > >> > >>> >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha
> > > > > Chintalapani <
> > > > > > > > >
> > > > > > > > > > > >> > kafka@harsha.io>
> > > > > > > > >
> > > > > > > > > > > >> > >>> > wrote:
> > > > > > > > >
> > > > > > > > > > > >> > >>> >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > Hi Grant,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > >           We are working on it. Will add
> the
> > > > > details
> > > > > > > to
> > > > > > > > > KIP
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > about
> > > > > > > > >
> > > > > > > > > > > >> > the
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > request protocol.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > Thanks,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > Harsha
> > > > > > > > >
> > > > > > > > > > > >> > >>> > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant
> Henke
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > <gh...@cloudera.com>
> > > > > > > > >
> > > > > > > > > > > >> > >>> wrote:
> > > > > > > > >
> > > > > > > > > > > >> > >>> > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > Hi Parth,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > Are you still working on this? If you
> need
> > > any
> > > > > > help
> > > > > > > > > please
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > don't
> > > > > > > > >
> > > > > > > > > > > >> > >>> > hesitate
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > to ask.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > Thanks,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > Grant
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun
> Rao <
> > > > > > > > >
> > > > > > > > > > jun@confluent.io>
> > > > > > > > >
> > > > > > > > > > > >> > wrote:
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > Parth,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > Thanks for the reply.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > It makes sense to only allow the
> renewal
> > > by
> > > > > > users
> > > > > > > > that
> > > > > > > > >
> > > > > > > > > > > >> > >>> authenticated
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > using
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > *non* delegation token mechanism.
> Then,
> > > > should
> > > > > > we
> > > > > > > > make
> > > > > > > > >
> > > > > > > > > > the
> > > > > > > > >
> > > > > > > > > > > >> > >>> renewal a
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > list?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > For example, in the case of rest
> proxy,
> > it
> > > > > will
> > > > > > be
> > > > > > > > >
> > > > > > > > > > useful
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > for
> > > > > > > > >
> > > > > > > > > > > >> > >>> every
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > instance of rest proxy to be able to
> > renew
> > > > the
> > > > > > > > tokens.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > It would be clearer if we can document
> > the
> > > > > > request
> > > > > > > > >
> > > > > > > > > > > protocol
> > > > > > > > >
> > > > > > > > > > > >> > like
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > https://cwiki.apache.org/confl
> > > > > > > > uence/display/KAFKA/KIP-
> > > > > > > > >
> > > > > > > > > > > >> > >>> > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > 4+-+Command+line+and+
> > > > centralized+administrative+
> > > > > > > > >
> > > > > > > > > > > operations#KIP-4-
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > Commandlineandcentralizedadmin
> > > > > istrativeoperations-
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > CreateTopicsRequest(KAFKA-
> > > > > > > 2945):(VotedandPlannedforin0.
> > > > > > > > >
> > > > > > > > > > > 10.1.0)
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > .
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > It would also be useful to document
> the
> > > > client
> > > > > > > APIs.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > Thanks,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > Jun
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM, parth
> > > > > > brahmbhatt
> > > > > > > <
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > Hi,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > I am suggesting that we will only
> > allow
> > > > the
> > > > > > > > renewal
> > > > > > > > > by
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > users
> > > > > > > > >
> > > > > > > > > > > >> > >>> that
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > authenticated using *non* delegation
> > > token
> > > > > > > > > mechanism.
> > > > > > > > >
> > > > > > > > > > > For
> > > > > > > > >
> > > > > > > > > > > >> > >>> example,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > If
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > user
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > Alice authenticated using kerberos
> and
> > > > > > requested
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > delegation
> > > > > > > > >
> > > > > > > > > > > >> > >>> tokens,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > only
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > user Alice authenticated via non
> > > > delegation
> > > > > > > token
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > mechanism
> > > > > > > > >
> > > > > > > > > > > >> > can
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > renew.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > Clients that have  access to
> > delegation
> > > > > tokens
> > > > > > > can
> > > > > > > > > not
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > issue
> > > > > > > > >
> > > > > > > > > > > >> > >>> > renewal
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > request for renewing their own token
> > and
> > > > > this
> > > > > > is
> > > > > > > > >
> > > > > > > > > > > primarily
> > > > > > > > >
> > > > > > > > > > > >> > >>> > important
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > to
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > reduce the time window for which a
> > > > > compromised
> > > > > > > > token
> > > > > > > > >
> > > > > > > > > > > will
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > be
> > > > > > > > >
> > > > > > > > > > > >> > >>> valid.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > To clarify, Yes any authenticated
> user
> > > can
> > > > > > > request
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > delegation
> > > > > > > > >
> > > > > > > > > > > >> > >>> > tokens
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > but
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > even here I would recommend to avoid
> > > > > creating
> > > > > > a
> > > > > > > > > chain
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > where a
> > > > > > > > >
> > > > > > > > > > > >> > >>> > client
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > authenticated via delegation token
> > > request
> > > > > for
> > > > > > > > more
> > > > > > > > >
> > > > > > > > > > > >> > delegation
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > tokens.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > Basically anyone can request
> > delegation
> > > > > token,
> > > > > > > as
> > > > > > > > > long
> > > > > > > > >
> > > > > > > > > > > as
> > > > > > > > >
> > > > > > > > > > > >> > they
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > authenticate
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > via a non delegation token
> mechanism.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > Aren't classes listed here
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > <
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > https://cwiki.apache.org/confl
> > > > > > > > uence/display/KAFKA/KIP-
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > 48+Delegation+token+support+fo
> > > > > > > > >
> > > > > > > > > > r+Kafka#KIP-48Delegationtokens
> > > > > > > > >
> > > > > > > > > > > >> > >>> upportforKaf
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > ka-PublicInterfaces
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > sufficient?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > Thanks
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > Parth
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun
> > Rao
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > <ju...@confluent.io>
> > > > > > > > >
> > > > > > > > > > > >> > >>> wrote:
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > Parth,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > Thanks for the reply. A couple of
> > > > comments
> > > > > > > > inline
> > > > > > > > >
> > > > > > > > > > > below.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > On Tue, Jun 21, 2016 at 10:36 AM,
> > > parth
> > > > > > > > > brahmbhatt <
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > brahmbhatt.parth@gmail.com>
> wrote:
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > 1. Who / how are tokens renewed?
> > By
> > > > > > original
> > > > > > > > >
> > > > > > > > > > > requester
> > > > > > > > >
> > > > > > > > > > > >> > >>> only? or
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > using
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > Kerberos
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > auth only?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > My recommendation is to do this
> > only
> > > > > using
> > > > > > > > >
> > > > > > > > > > Kerberos
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > auth
> > > > > > > > >
> > > > > > > > > > > >> > and
> > > > > > > > >
> > > > > > > > > > > >> > >>> > only
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > threw
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > the
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > renewer specified during the
> > > > acquisition
> > > > > > > > > request.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > Hmm, not sure that I follow this.
> > Are
> > > > you
> > > > > > > saying
> > > > > > > > >
> > > > > > > > > > that
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > any
> > > > > > > > >
> > > > > > > > > > > >> > >>> client
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > authenticated with the delegation
> > > token
> > > > > can
> > > > > > > > renew,
> > > > > > > > >
> > > > > > > > > > > i.e.
> > > > > > > > >
> > > > > > > > > > > >> > there
> > > > > > > > >
> > > > > > > > > > > >> > >>> is
> > > > > > > > >
> > > > > > > > > > > >> > >>> > no
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > renewer
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > needed?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > Also, just to be clear, any
> > > > authenticated
> > > > > > > client
> > > > > > > > >
> > > > > > > > > > > (either
> > > > > > > > >
> > > > > > > > > > > >> > >>> through
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > SASL
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > or
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > SSL) can request a delegation
> token
> > > for
> > > > > the
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > authenticated
> > > > > > > > >
> > > > > > > > > > > >> > >>> user,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > right?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > 2. Are tokens stored on each
> > broker
> > > or
> > > > > in
> > > > > > > ZK?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > My recommendation is still to
> > store
> > > in
> > > > > ZK
> > > > > > or
> > > > > > > > not
> > > > > > > > >
> > > > > > > > > > > store
> > > > > > > > >
> > > > > > > > > > > >> > them
> > > > > > > > >
> > > > > > > > > > > >> > >>> at
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > all.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > The
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > whole controller based
> > distribution
> > > is
> > > > > too
> > > > > > > > much
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > overhead
> > > > > > > > >
> > > > > > > > > > > >> > >>> with
> > > > > > > > >
> > > > > > > > > > > >> > >>> > not
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > much
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > to
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > achieve.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > 3. How are tokens invalidated /
> > > > expired?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > Either by expiration time out or
> > > > through
> > > > > > an
> > > > > > > > >
> > > > > > > > > > explicit
> > > > > > > > >
> > > > > > > > > > > >> > >>> request to
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > invalidate.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > 4. Which encryption algorithm is
> > > used?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > SCRAM
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > 5. What is the impersonation
> > > proposal
> > > > > (it
> > > > > > > > wasn't
> > > > > > > > >
> > > > > > > > > > in
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > the
> > > > > > > > >
> > > > > > > > > > > >> > KIP
> > > > > > > > >
> > > > > > > > > > > >> > >>> but
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > was
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > discussed
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > in this thread)?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > There is no imperonation
> > proposal. I
> > > > > tried
> > > > > > > and
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > explained
> > > > > > > > >
> > > > > > > > > > > >> > how
> > > > > > > > >
> > > > > > > > > > > >> > >>> > its
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > a
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > different problem and why its
> not
> > > > really
> > > > > > > > > necessary
> > > > > > > > >
> > > > > > > > > > > to
> > > > > > > > >
> > > > > > > > > > > >> > >>> discuss
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > that
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > as
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > part
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > of this KIP.  This KIP will not
> > > > support
> > > > > > any
> > > > > > > > >
> > > > > > > > > > > >> > impersonation,
> > > > > > > > >
> > > > > > > > > > > >> > >>> it
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > will
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > just
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > be
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > another way to authenticate.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > 6. Do we need new ACLs, if so -
> > for
> > > > what
> > > > > > > > > actions?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > We do not need new ACLs.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > Could we document the format of
> the
> > > new
> > > > > > > > >
> > > > > > > > > > > request/response
> > > > > > > > >
> > > > > > > > > > > >> > and
> > > > > > > > >
> > > > > > > > > > > >> > >>> > their
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > associated Resource and Operation
> > for
> > > > ACL?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > 7. How would the delegation
> token
> > be
> > > > > > > > configured
> > > > > > > > > in
> > > > > > > > >
> > > > > > > > > > > the
> > > > > > > > >
> > > > > > > > > > > >> > >>> client?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > Should be through config. I
> wasn't
> > > > > > planning
> > > > > > > on
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > supporting
> > > > > > > > >
> > > > > > > > > > > >> > >>> JAAS
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > for
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > tokens.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > I don't believe hadoop does this
> > > > either.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > Parth
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM,
> > Jun
> > > > > Rao <
> > > > > > > > >
> > > > > > > > > > > >> > jun@confluent.io>
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > wrote:
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > Harsha,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > Another question.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > 9. How would the delegation
> > token
> > > be
> > > > > > > > > configured
> > > > > > > > >
> > > > > > > > > > in
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > the
> > > > > > > > >
> > > > > > > > > > > >> > >>> > client?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > The
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > standard
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > way is to do this through
> JAAS.
> > > > > However,
> > > > > > > we
> > > > > > > > > will
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > need
> > > > > > > > >
> > > > > > > > > > > >> > to
> > > > > > > > >
> > > > > > > > > > > >> > >>> > think
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > through
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > if
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > this is convenient in a shared
> > > > > > > environment.
> > > > > > > > > For
> > > > > > > > >
> > > > > > > > > > > >> > example,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > when a
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > new
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > task
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > is
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > added to a Storm worker node,
> do
> > > we
> > > > > need
> > > > > > > to
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > dynamically
> > > > > > > > >
> > > > > > > > > > > >> > >>> add a
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > new
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > section
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > in the JAAS file? It may be
> more
> > > > > > > convenient
> > > > > > > > if
> > > > > > > > >
> > > > > > > > > > we
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > can
> > > > > > > > >
> > > > > > > > > > > >> > >>> pass in
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > the
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > token
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > through the config directly
> w/o
> > > > going
> > > > > > > > through
> > > > > > > > >
> > > > > > > > > > > JAAS.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > Are you or Parth still
> actively
> > > > > working
> > > > > > on
> > > > > > > > > this
> > > > > > > > >
> > > > > > > > > > > KIP?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18
> PM,
> > > Jun
> > > > > > Rao <
> > > > > > > > >
> > > > > > > > > > > >> > >>> jun@confluent.io>
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > wrote:
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > > Just to add on that list.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > > 2. It would be good to
> > document
> > > > the
> > > > > > > format
> > > > > > > > > of
> > > > > > > > >
> > > > > > > > > > > the
> > > > > > > > >
> > > > > > > > > > > >> > data
> > > > > > > > >
> > > > > > > > > > > >> > >>> > stored
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > in
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > ZK.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > > 7. Earlier, there was a
> > > discussion
> > > > > on
> > > > > > > > > whether
> > > > > > > > >
> > > > > > > > > > > the
> > > > > > > > >
> > > > > > > > > > > >> > tokens
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > should
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > be
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > > propagated through ZK like
> > > > > > > > config/acl/quota,
> > > > > > > > >
> > > > > > > > > > or
> > > > > > > > >
> > > > > > > > > > > >> > through
> > > > > > > > >
> > > > > > > > > > > >> > >>> the
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > controller.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > > Currently, the controller is
> > > only
> > > > > > > designed
> > > > > > > > > for
> > > > > > > > >
> > > > > > > > > > > >> > >>> propagating
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > topic
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > metadata,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > > but not other data.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > > 8. Should we use SCRAM to
> send
> > > the
> > > > > > token
> > > > > > > > >
> > > > > > > > > > instead
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > > of
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > DIGEST-MD5
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > since
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > it's
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > > deprecated?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > > Also, the images in the wiki
> > > seem
> > > > > > > broken.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > > On Fri, Jun 10, 2016 at
> 10:02
> > > AM,
> > > > > Gwen
> > > > > > > > >
> > > > > > > > > > Shapira <
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > gwen@confluent.io>
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> From what I can see,
> > remaining
> > > > > > > questions
> > > > > > > > > are:
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >>
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> 1. Who / how are tokens
> > > renewed?
> > > > By
> > > > > > > > > original
> > > > > > > > >
> > > > > > > > > > > >> > requester
> > > > > > > > >
> > > > > > > > > > > >> > >>> > only?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > or
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > using
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> Kerberos auth only?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> 2. Are tokens stored on
> each
> > > > broker
> > > > > > or
> > > > > > > in
> > > > > > > > > ZK?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> 3. How are tokens
> > invalidated /
> > > > > > > expired?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> 4. Which encryption
> algorithm
> > > is
> > > > > > used?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> 5. What is the
> impersonation
> > > > > proposal
> > > > > > > (it
> > > > > > > > >
> > > > > > > > > > > wasn't
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> in
> > > > > > > > >
> > > > > > > > > > > >> > the
> > > > > > > > >
> > > > > > > > > > > >> > >>> > KIP
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > but
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > was
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> discussed in this thread)?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> 6. Do we need new ACLs, if
> > so -
> > > > for
> > > > > > > what
> > > > > > > > >
> > > > > > > > > > > actions?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >>
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> Gwen
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >>
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> On Thu, Jun 9, 2016 at 7:48
> > PM,
> > > > > > Harsha
> > > > > > > <
> > > > > > > > >
> > > > > > > > > > > >> > >>> kafka@harsha.io>
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > wrote:
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> > Jun & Ismael,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >
> > > > > > > Unfortunately
> > > > > > > > I
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> > couldn't
> > > > > > > > >
> > > > > > > > > > > >> > >>> attend
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > the
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > KIP
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > meeting
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >
> > when
> > > > > > > > delegation
> > > > > > > > >
> > > > > > > > > > > tokens
> > > > > > > > >
> > > > > > > > > > > >> > >>> > discussed.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > Appreciate
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > if
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >
> > you
> > > > can
> > > > > > > update
> > > > > > > > > the
> > > > > > > > >
> > > > > > > > > > > >> > thread if
> > > > > > > > >
> > > > > > > > > > > >> > >>> > you
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > have
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > any
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >
> > > > further
> > > > > > > > > questions.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> > Thanks,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> > Harsha
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> > On Tue, May 24, 2016, at
> > > 11:32
> > > > > AM,
> > > > > > > > Liquan
> > > > > > > > >
> > > > > > > > > > Pei
> > > > > > > > >
> > > > > > > > > > > >> > wrote:
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> It seems that the links
> to
> > > > > images
> > > > > > in
> > > > > > > > the
> > > > > > > > >
> > > > > > > > > > KIP
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> are
> > > > > > > > >
> > > > > > > > > > > >> > >>> > broken.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >>
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> Liquan
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >>
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> On Tue, May 24, 2016 at
> > 9:33
> > > > AM,
> > > > > > > parth
> > > > > > > > >
> > > > > > > > > > > >> > brahmbhatt <
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >>
> > brahmbhatt.parth@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >>
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> > 110. What does
> > > > > > > getDelegationTokenAs
> > > > > > > > >
> > > > > > > > > > mean?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> > In the current
> proposal
> > we
> > > > > only
> > > > > > > > allow
> > > > > > > > > a
> > > > > > > > >
> > > > > > > > > > > user
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> > to
> > > > > > > > >
> > > > > > > > > > > >> > >>> get
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > delegation
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > token
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> for
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> > the identity that it
> > > > > > authenticated
> > > > > > > > as
> > > > > > > > >
> > > > > > > > > > > using
> > > > > > > > >
> > > > > > > > > > > >> > >>> another
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > mechanism,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > i.e.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> A user
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> > that authenticate
> using
> > a
> > > > > keytab
> > > > > > > for
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> > principal
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > user1@EXAMPLE.COM
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> will get
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens for
> > that
> > > > > user
> > > > > > > > only.
> > > > > > > > > In
> > > > > > > > >
> > > > > > > > > > > >> > future I
> > > > > > > > >
> > > > > > > > > > > >> > >>> > think
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > we
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > will
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > have
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> to
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> > extend support such
> that
> > > we
> > > > > > allow
> > > > > > > > some
> > > > > > > > >
> > > > > > > > > > set
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> > of
> > > > > > > > >
> > > > > > > > > > > >> > >>> users (
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> >
> > > kafka-rest-user@EXAMPLE.COM
> > > > ,
> > > > > > > > >
> > > > > > > > > > > >> > >>> > storm-nimbus@EXAMPLE.COM)
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > to
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > acquire
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens on
> > > behalf
> > > > of
> > > > > > > other
> > > > > > > > >
> > > > > > > > > > users
> > > > > > > > >
> > > > > > > > > > > >> > whose
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > identity
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > they
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > have
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> > verified
> independently.
> > > > Kafka
> > > > > > > > brokers
> > > > > > > > >
> > > > > > > > > > > will
> > > > > > > > >
> > > > > > > > > > > >> > have
> > > > > > > > >
> > > > > > > > > > > >> > >>> ACLs
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > to
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > control
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> which
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> > users are allowed to
> > > > > impersonate
> > > > > > > > other
> > > > > > > > >
> > > > > > > > > > > users
> > > > > > > > >
> > > > > > > > > > > >> > and
> > > > > > > > >
> > > > > > > > > > > >> > >>> get
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > tokens
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > on
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> behalf of
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> > them. Overall
> > > Impersonation
> > > > > is a
> > > > > > > > whole
> > > > > > > > >
> > > > > > > > > > > >> > different
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > problem
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > in
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > my
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> opinion and
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> > I think we can tackle
> it
> > > in
> > > > > > > separate
> > > > > > > > >
> > > > > > > > > > KIP.
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> > 111. What's the
> typical
> > > rate
> > > > > of
> > > > > > > > > getting
> > > > > > > > >
> > > > > > > > > > > and
> > > > > > > > >
> > > > > > > > > > > >> > >>> renewing
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > delegation
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> tokens?
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> > Typically this should
> be
> > > > very
> > > > > > very
> > > > > > > > > low,
> > > > > > > > >
> > > > > > > > > > 1
> > > > > > > > >
> > > > > > > > > > > >> > request
> > > > > > > > >
> > > > > > > > > > > >> > >>> per
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > minute
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > is a
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> > relatively high
> > estimate.
> > > > > > However
> > > > > > > it
> > > > > > > > >
> > > > > > > > > > > depends
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > > > > > > > >> >> > on
> > > > > > > > >
> > > > > > > > > > > >> > >>> the
> > > > > > > > >
> > > > > > > > > > > >> > >>> > > token
> > > > > > > > >
> > >
> >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Manikumar <ma...@gmail.com>]

@Gwen, @Rajini,

As mentioned in the KIP, main motivation for this KIP is to reduce load on
Kerberos
server on large kafka deployments with large number of clients.

Also it looks like we are combining two overlapping concepts
1. Single client sending requests with multiple users/authentications
2. Impersonation

Option 1, is definitely useful in some use cases and can be used to
implement workaround for
impersonation

In Impersonation, a super user can send requests on behalf of another
user(Alice) in a secured way.
superuser has credentials but user Alice doesn't have any. The requests are
required
to run as user Alice and accesses/ACLs on Broker are required to be done as
user Alice.
It is required that user Alice can connect to the Broker on a connection
authenticated with
superuser's credentials. In other words superuser is impersonating the user
Alice.

The approach mentioned by Harsha in previous mail is implemented in hadoop,
storm etc..

Some more details here:
https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/Superusers.html


@Rajini

Thanks for your comments on SASL/SCRAM usage. I am thinking to send
tokenHmac (salted-hashed version)
as password for authentication and tokenID for retrial of tokenHmac at
server side.
Does above sound OK?


Thanks,
Manikumar

On Wed, Dec 14, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
wrote:

> @Gwen @Mani  Not sure why we want to authenticate at every request. Even if
> the token exchange is cheap it still a few calls that need to go through
> round trip.  Impersonation doesn't require authentication for every
> request.
>
> "So a centralized app can create few producers, do the metadata request and
> broker discovery with its own user auth, but then use delegation tokens to
> allow performing produce/fetch requests as different users? Instead of
> having to re-connect for each impersonated user?"
>
> Yes. But what we will have is this centralized user as impersonation user
> on behalf of other users. When it authenticates initially we will create a
> "Subject" and from there on wards centralized user can do
> Subject.doAsPrivileged
> on behalf, other users.
> On the server side, we can retrieve two principals out of this one is the
> authenticated user (centralized user) and another is impersonated user. We
> will first check if the authenticated user allowed to impersonate and then
> move on to check if the user Alice has access to the topic "X" to
> read/write.
>
> @Rajini Intention of this KIP is to support token auth via SASL/SCRAM, not
> just with TLS.  What you raised is a good point let me take a look and add
> details.
>
> It will be easier to add impersonation once we reach agreement on this KIP.
>
>
> On Wed, Dec 14, 2016 at 5:51 AM Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Rajini,
> >
> > I think it would definitely be valuable to have a KIP for impersonation.
> >
> > Ismael
> >
> > On Wed, Dec 14, 2016 at 4:03 AM, Rajini Sivaram <rs...@pivotal.io>
> > wrote:
> >
> > > It would clearly be very useful to enable clients to send requests on
> > > behalf of multiple users. A separate KIP makes sense, but it may be
> worth
> > > thinking through some of the implications now, especially if the main
> > > interest in delegation tokens comes from its potential to enable
> > > impersonation.
> > >
> > > I understand that delegation tokens are only expected to be used with
> > TLS.
> > > But the choice of SASL/SCRAM for authentication must be based on a
> > > requirement to protect the tokenHmac - otherwise you could just use
> > > SASL/PLAIN. With SASL/SCRAM the tokenHmac is never propagated
> > on-the-wire,
> > > only a salted-hashed version of it is used in the SASL authentication
> > > exchange. If impersonation is based on sending tokenHmac in requests,
> any
> > > benefit of using SCRAM is lost.
> > >
> > > An alternative may be to allow clients to authenticate multiple times
> > using
> > > SASL and include one of its authenticated principals in each request
> > > (optionally). I haven't thought it through yet, obviously. But if the
> > > approach is of interest and no one is working on a KIP for
> impersonation
> > at
> > > the moment, I am happy to write one. It may provide something for
> > > comparison at least.
> > >
> > > Thoughts?
> > >
> > >
> > > On Wed, Dec 14, 2016 at 9:53 AM, Manikumar <ma...@gmail.com>
> > > wrote:
> > >
> > > > That's a good idea. Authenticating every request with delegation
> token
> > > will
> > > > be useful for
> > > > impersonation use-cases. But as of now, we are thinking delegation
> > token
> > > as
> > > > just another way
> > > > to authenticate the users. We haven't think through all the use cases
> > > > related to
> > > > impersonation or using delegation token for impersonation. We want to
> > > > handle impersonation
> > > > (KAFKA-3712) as part of separate KIP.
> > > >
> > > > Will that be Ok?
> > > >
> > > >
> > > > On Wed, Dec 14, 2016 at 8:09 AM, Gwen Shapira <gw...@confluent.io>
> > wrote:
> > > >
> > > > > Thinking out loud here:
> > > > >
> > > > > It looks like authentication with a delegation token is going to be
> > > > > super-cheap, right? We just compare the token to a value in the
> > broker
> > > > > cache?
> > > > >
> > > > > If I understood the KIP correctly, right now it suggests that
> > > > > authentication happens when establishing the client-broker
> connection
> > > (as
> > > > > normal for Kafka. But perhaps we want to consider authenticating
> > every
> > > > > request with delegation token (if exists)?
> > > > >
> > > > > So a centralized app can create few producers, do the metadata
> > request
> > > > and
> > > > > broker discovery with its own user auth, but then use delegation
> > tokens
> > > > to
> > > > > allow performing produce/fetch requests as different users? Instead
> > of
> > > > > having to re-connect for each impersonated user?
> > > > >
> > > > > This may over-complicate things quite a bit (basically adding extra
> > > > > information in every request), but maybe it will be useful for
> > > > > impersonation use-cases (which seem to drive much of the interest
> in
> > > this
> > > > > KIP)?
> > > > > Kafka Connect, NiFi and friends can probably use this to share
> > clients
> > > > > between multiple jobs, tasks, etc.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > Gwen
> > > > >
> > > > > On Tue, Dec 13, 2016 at 12:43 AM, Manikumar <
> > manikumar.reddy@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Ashish,
> > > > > >
> > > > > > Thank you for reviewing the KIP.  Please see the replies inline.
> > > > > >
> > > > > >
> > > > > > > 1. How to disable delegation token authentication?
> > > > > > >
> > > > > > > This can be achieved in various ways, however I think reusing
> > > > > delegation
> > > > > > > token secret config for this makes sense here. Avoids creating
> > yet
> > > > > > another
> > > > > > > config and forces delegation token users to consciously set the
> > > > secret.
> > > > > > If
> > > > > > > the secret is not set or set to empty string, brokers should
> turn
> > > off
> > > > > > > delegation token support. This will however require a new error
> > > code
> > > > to
> > > > > > > indicate delegation token support is turned off on broker.
> > > > > > >
> > > > > >
> > > > > >   Thanks for the suggestion. Option to turnoff delegation token
> > > > > > authentication will be useful.
> > > > > >   I'll update the KIP.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > 2. ACLs on delegation token?
> > > > > > >
> > > > > > > Do we need to have ACLs defined for tokens? I do not think it
> > buys
> > > us
> > > > > > > anything, as delegation token can be treated as impersonation
> of
> > > the
> > > > > > owner.
> > > > > > > Any thing the owner has permission to do, delegation tokens
> > should
> > > be
> > > > > > > allowed to do as well. If so, we probably won't need to return
> > > > > > > authorization exception error code while creating delegation
> > token.
> > > > It
> > > > > > > however would make sense to check renew and expire requests are
> > > > coming
> > > > > > from
> > > > > > > owner or renewers of the token, but that does not require
> > explicit
> > > > > acls.
> > > > > > >
> > > > > >
> > > > > >
> > > > > > Yes, We agreed to not have new acl on who can request delegation
> > > token.
> > > > > >  I'll update the KIP.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > 3. How to restrict max life time of a token?
> > > > > > >
> > > > > > > Admins might want to restrict max life time of tokens created
> on
> > a
> > > > > > cluster,
> > > > > > > and this can very from cluster to cluster based on use-cases.
> > This
> > > > > might
> > > > > > > warrant a separate broker config.
> > > > > > >
> > > > > > >
> > > > > > Currently we  have "delegation.token.max.lifetime.sec" server
> > config
> > > > > > property
> > > > > > May be we can take min(User supplied MaxTime, Server MaxTime) as
> > max
> > > > life
> > > > > > time.
> > > > > > I am open to add new config property.
> > > > > >
> > > > > > Few more comments based on recent KIP update.
> > > > > > >
> > > > > > > 1. Do we need a separate {{InvalidateTokenRequest}}? Can't we
> use
> > > > > > > {{ExpireTokenRequest}} with with expiryDate set to anything
> > before
> > > > > > current
> > > > > > > date?
> > > > > > >
> > > > > >
> > > > > > makes sense. we don't need special request to cancel the token.
> We
> > > can
> > > > > use
> > > > > > ExpireTokenRequest.
> > > > > > I'll update the KIP.
> > > > > >
> > > > > >
> > > > > > > 2. Can we change time field names to indicate their unit is
> > > > > milliseconds,
> > > > > > > like, IssueDateMs, ExpiryDateMs, etc.?
> > > > > > >
> > > > > > >
> > > > > >   Done.
> > > > > >
> > > > > >
> > > > > > > 3. Can we allow users to renew a token for a specified amount
> of
> > > > time?
> > > > > In
> > > > > > > current version of KIP, renew request does not take time as a
> > > param,
> > > > > not
> > > > > > > sure what is expiry time set to after renewal.
> > > > > > >
> > > > > > >
> > > > > >  Yes, we need to specify renew period.  I'll update the KIP.
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Mankumar
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > On Mon, Dec 12, 2016 at 9:08 AM Manikumar <
> > > manikumar.reddy@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > I would like to reinitiate the discussion on Delegation token
> > > > support
> > > > > > for
> > > > > > > >
> > > > > > > > Kafka.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Brief summary of the past discussion:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > 1) Broker stores delegation tokens in zookeeper.  All brokers
> > > will
> > > > > > have a
> > > > > > > >
> > > > > > > > cache backed by
> > > > > > > >
> > > > > > > >    zookeeper so they will all get notified whenever a new
> token
> > > is
> > > > > > > >
> > > > > > > > generated and they will
> > > > > > > >
> > > > > > > >    update their local cache whenever token state changes.
> > > > > > > >
> > > > > > > > 2) The current proposal does not support rotation of secret
> > > > > > > >
> > > > > > > > 3) Only allow the renewal by users that authenticated using
> > *non*
> > > > > > > >
> > > > > > > > delegation token mechanism
> > > > > > > >
> > > > > > > > 4) KIP-84 proposes to support  SASL SCRAM mechanisms. Kafka
> > > clients
> > > > > can
> > > > > > > >
> > > > > > > > authenticate using
> > > > > > > >
> > > > > > > >    SCRAM-SHA-256, providing the delegation token HMAC as
> > > password.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Updated the KIP with the following:
> > > > > > > >
> > > > > > > > 1. Protocol and Config changes
> > > > > > > >
> > > > > > > > 2. format of the data stored in ZK.
> > > > > > > >
> > > > > > > > 3. Changes to Java Clients/Usage of SASL SCRAM mechanism
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > >
> > > > > > > > 48+Delegation+token+support+for+Kafka
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Jun, Ashish, Gwen,
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Pl review the updated KIP.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > Manikumar
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Sep 29, 2016 at 9:56 PM, Ashish Singh <
> > > asingh@cloudera.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > > Harsha/ Gwen,
> > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > > How do we proceed here? I am willing to help out with here.
> > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > > On Fri, Sep 23, 2016 at 11:41 AM, Gwen Shapira <
> > > > gwen@confluent.io>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > > > Is it updated? are all concerns addressed? do you want to
> > > > start a
> > > > > > > vote?
> > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > > > > Sorry for being pushy, I do appreciate that we are all
> > > > volunteers
> > > > > > and
> > > > > > > >
> > > > > > > > > > finding time is difficult. This feature is important for
> > > > anything
> > > > > > > that
> > > > > > > >
> > > > > > > > > > integrates with Kafka (stream processors, Flume, NiFi,
> etc)
> > > > and I
> > > > > > > >
> > > > > > > > > > don't want to see this getting stuck because we lack
> > > > coordination
> > > > > > > >
> > > > > > > > > > within the community.
> > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > > > > On Thu, Sep 15, 2016 at 6:39 PM, Harsha Chintalapani <
> > > > > > > kafka@harsha.io>
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > > > The only pending update for the KIP is to write up the
> > > > protocol
> > > > > > > > changes
> > > > > > > >
> > > > > > > > > > like
> > > > > > > >
> > > > > > > > > > > we've it KIP-4. I'll update the wiki.
> > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > > > > > > > > On Thu, Sep 15, 2016 at 4:27 PM Ashish Singh <
> > > > > > asingh@cloudera.com>
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > > >>
> > > > > > > >
> > > > > > > > > > >> I think we decided to not support secret rotation, I
> > guess
> > > > > this
> > > > > > > can
> > > > > > > > be
> > > > > > > >
> > > > > > > > > > >> stated clearly on the KIP. Also, more details on how
> > > clients
> > > > > > will
> > > > > > > >
> > > > > > > > > > perform
> > > > > > > >
> > > > > > > > > > >> token distribution and how CLI will look like will be
> > > > helpful.
> > > > > > > >
> > > > > > > > > > >>
> > > > > > > >
> > > > > > > > > > >> On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <
> > > > > > gwen@confluent.io>
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > > >>
> > > > > > > >
> > > > > > > > > > >> > Hi Guys,
> > > > > > > >
> > > > > > > > > > >> >
> > > > > > > >
> > > > > > > > > > >> > This discussion was dead for a while. Are there
> still
> > > > > > > contentious
> > > > > > > >
> > > > > > > > > > >> > points? If not, why are there no votes?
> > > > > > > >
> > > > > > > > > > >> >
> > > > > > > >
> > > > > > > > > > >> > On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <
> > > > jun@confluent.io>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > > >> > > Ashish,
> > > > > > > >
> > > > > > > > > > >> > >
> > > > > > > >
> > > > > > > > > > >> > > Yes, I will send out a KIP invite for next week to
> > > > discuss
> > > > > > > > KIP-48
> > > > > > > >
> > > > > > > > > > and
> > > > > > > >
> > > > > > > > > > >> > other
> > > > > > > >
> > > > > > > > > > >> > > remaining KIPs.
> > > > > > > >
> > > > > > > > > > >> > >
> > > > > > > >
> > > > > > > > > > >> > > Thanks,
> > > > > > > >
> > > > > > > > > > >> > >
> > > > > > > >
> > > > > > > > > > >> > > Jun
> > > > > > > >
> > > > > > > > > > >> > >
> > > > > > > >
> > > > > > > > > > >> > > On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh <
> > > > > > > >
> > > > > > > > > asingh@cloudera.com>
> > > > > > > >
> > > > > > > > > > >> > wrote:
> > > > > > > >
> > > > > > > > > > >> > >
> > > > > > > >
> > > > > > > > > > >> > >> Thanks Harsha!
> > > > > > > >
> > > > > > > > > > >> > >>
> > > > > > > >
> > > > > > > > > > >> > >> Jun, can we add KIP-48 to next KIP hangout's
> > agenda.
> > > > > Also,
> > > > > > we
> > > > > > > > did
> > > > > > > >
> > > > > > > > > > not
> > > > > > > >
> > > > > > > > > > >> > >> actually make a call on when we should have next
> > KIP
> > > > > call.
> > > > > > As
> > > > > > > >
> > > > > > > > > there
> > > > > > > >
> > > > > > > > > > >> > >> are
> > > > > > > >
> > > > > > > > > > >> > a
> > > > > > > >
> > > > > > > > > > >> > >> few outstanding KIPs that could not be discussed
> > this
> > > > > week,
> > > > > > > can
> > > > > > > >
> > > > > > > > > we
> > > > > > > >
> > > > > > > > > > >> > >> have
> > > > > > > >
> > > > > > > > > > >> > a
> > > > > > > >
> > > > > > > > > > >> > >> KIP hangout call next week?
> > > > > > > >
> > > > > > > > > > >> > >>
> > > > > > > >
> > > > > > > > > > >> > >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha
> > Chintalapani
> > > > > > > >
> > > > > > > > > > >> > >> <ka...@harsha.io>
> > > > > > > >
> > > > > > > > > > >> > >> wrote:
> > > > > > > >
> > > > > > > > > > >> > >>
> > > > > > > >
> > > > > > > > > > >> > >>> Ashish,
> > > > > > > >
> > > > > > > > > > >> > >>>         Yes we are working on it. Lets discuss
> in
> > > the
> > > > > next
> > > > > > > KIP
> > > > > > > >
> > > > > > > > > > >> > >>> meeting.
> > > > > > > >
> > > > > > > > > > >> > >>> I'll join.
> > > > > > > >
> > > > > > > > > > >> > >>> -Harsha
> > > > > > > >
> > > > > > > > > > >> > >>>
> > > > > > > >
> > > > > > > > > > >> > >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh <
> > > > > > > >
> > > > > > > > > > asingh@cloudera.com>
> > > > > > > >
> > > > > > > > > > >> > >>> wrote:
> > > > > > > >
> > > > > > > > > > >> > >>>
> > > > > > > >
> > > > > > > > > > >> > >>> > Hello Harsha,
> > > > > > > >
> > > > > > > > > > >> > >>> >
> > > > > > > >
> > > > > > > > > > >> > >>> > Are you still working on this? Wondering if we
> > can
> > > > > > discuss
> > > > > > > >
> > > > > > > > > this
> > > > > > > >
> > > > > > > > > > in
> > > > > > > >
> > > > > > > > > > >> > next
> > > > > > > >
> > > > > > > > > > >> > >>> KIP
> > > > > > > >
> > > > > > > > > > >> > >>> > meeting, if you can join.
> > > > > > > >
> > > > > > > > > > >> > >>> >
> > > > > > > >
> > > > > > > > > > >> > >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha
> > > > Chintalapani <
> > > > > > > >
> > > > > > > > > > >> > kafka@harsha.io>
> > > > > > > >
> > > > > > > > > > >> > >>> > wrote:
> > > > > > > >
> > > > > > > > > > >> > >>> >
> > > > > > > >
> > > > > > > > > > >> > >>> > > Hi Grant,
> > > > > > > >
> > > > > > > > > > >> > >>> > >           We are working on it. Will add the
> > > > details
> > > > > > to
> > > > > > > > KIP
> > > > > > > >
> > > > > > > > > > >> > >>> > > about
> > > > > > > >
> > > > > > > > > > >> > the
> > > > > > > >
> > > > > > > > > > >> > >>> > > request protocol.
> > > > > > > >
> > > > > > > > > > >> > >>> > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > Thanks,
> > > > > > > >
> > > > > > > > > > >> > >>> > > Harsha
> > > > > > > >
> > > > > > > > > > >> > >>> > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant Henke
> > > > > > > >
> > > > > > > > > > >> > >>> > > <gh...@cloudera.com>
> > > > > > > >
> > > > > > > > > > >> > >>> wrote:
> > > > > > > >
> > > > > > > > > > >> > >>> > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > Hi Parth,
> > > > > > > >
> > > > > > > > > > >> > >>> > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > Are you still working on this? If you need
> > any
> > > > > help
> > > > > > > > please
> > > > > > > >
> > > > > > > > > > >> > >>> > > > don't
> > > > > > > >
> > > > > > > > > > >> > >>> > hesitate
> > > > > > > >
> > > > > > > > > > >> > >>> > > > to ask.
> > > > > > > >
> > > > > > > > > > >> > >>> > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > Thanks,
> > > > > > > >
> > > > > > > > > > >> > >>> > > > Grant
> > > > > > > >
> > > > > > > > > > >> > >>> > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <
> > > > > > > >
> > > > > > > > > jun@confluent.io>
> > > > > > > >
> > > > > > > > > > >> > wrote:
> > > > > > > >
> > > > > > > > > > >> > >>> > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > Parth,
> > > > > > > >
> > > > > > > > > > >> > >>> > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > Thanks for the reply.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > It makes sense to only allow the renewal
> > by
> > > > > users
> > > > > > > that
> > > > > > > >
> > > > > > > > > > >> > >>> authenticated
> > > > > > > >
> > > > > > > > > > >> > >>> > > > using
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > *non* delegation token mechanism. Then,
> > > should
> > > > > we
> > > > > > > make
> > > > > > > >
> > > > > > > > > the
> > > > > > > >
> > > > > > > > > > >> > >>> renewal a
> > > > > > > >
> > > > > > > > > > >> > >>> > > > list?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > For example, in the case of rest proxy,
> it
> > > > will
> > > > > be
> > > > > > > >
> > > > > > > > > useful
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > for
> > > > > > > >
> > > > > > > > > > >> > >>> every
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > instance of rest proxy to be able to
> renew
> > > the
> > > > > > > tokens.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > It would be clearer if we can document
> the
> > > > > request
> > > > > > > >
> > > > > > > > > > protocol
> > > > > > > >
> > > > > > > > > > >> > like
> > > > > > > >
> > > > > > > > > > >> > >>> > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > https://cwiki.apache.org/confl
> > > > > > > uence/display/KAFKA/KIP-
> > > > > > > >
> > > > > > > > > > >> > >>> > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > 4+-+Command+line+and+
> > > centralized+administrative+
> > > > > > > >
> > > > > > > > > > operations#KIP-4-
> > > > > > > >
> > > > > > > > > > >> > >>> > > Commandlineandcentralizedadmin
> > > > istrativeoperations-
> > > > > > > >
> > > > > > > > > > >> > >>> > > CreateTopicsRequest(KAFKA-
> > > > > > 2945):(VotedandPlannedforin0.
> > > > > > > >
> > > > > > > > > > 10.1.0)
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > .
> > > > > > > >
> > > > > > > > > > >> > >>> > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > It would also be useful to document the
> > > client
> > > > > > APIs.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > Thanks,
> > > > > > > >
> > > > > > > > > > >> > >>> > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > Jun
> > > > > > > >
> > > > > > > > > > >> > >>> > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM, parth
> > > > > brahmbhatt
> > > > > > <
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > > >> > >>> > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > Hi,
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > I am suggesting that we will only
> allow
> > > the
> > > > > > > renewal
> > > > > > > > by
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > users
> > > > > > > >
> > > > > > > > > > >> > >>> that
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > authenticated using *non* delegation
> > token
> > > > > > > > mechanism.
> > > > > > > >
> > > > > > > > > > For
> > > > > > > >
> > > > > > > > > > >> > >>> example,
> > > > > > > >
> > > > > > > > > > >> > >>> > If
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > user
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > Alice authenticated using kerberos and
> > > > > requested
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > delegation
> > > > > > > >
> > > > > > > > > > >> > >>> tokens,
> > > > > > > >
> > > > > > > > > > >> > >>> > > > only
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > user Alice authenticated via non
> > > delegation
> > > > > > token
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > mechanism
> > > > > > > >
> > > > > > > > > > >> > can
> > > > > > > >
> > > > > > > > > > >> > >>> > > renew.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > Clients that have  access to
> delegation
> > > > tokens
> > > > > > can
> > > > > > > > not
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > issue
> > > > > > > >
> > > > > > > > > > >> > >>> > renewal
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > request for renewing their own token
> and
> > > > this
> > > > > is
> > > > > > > >
> > > > > > > > > > primarily
> > > > > > > >
> > > > > > > > > > >> > >>> > important
> > > > > > > >
> > > > > > > > > > >> > >>> > > to
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > reduce the time window for which a
> > > > compromised
> > > > > > > token
> > > > > > > >
> > > > > > > > > > will
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > be
> > > > > > > >
> > > > > > > > > > >> > >>> valid.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > To clarify, Yes any authenticated user
> > can
> > > > > > request
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > delegation
> > > > > > > >
> > > > > > > > > > >> > >>> > tokens
> > > > > > > >
> > > > > > > > > > >> > >>> > > > but
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > even here I would recommend to avoid
> > > > creating
> > > > > a
> > > > > > > > chain
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > where a
> > > > > > > >
> > > > > > > > > > >> > >>> > client
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > authenticated via delegation token
> > request
> > > > for
> > > > > > > more
> > > > > > > >
> > > > > > > > > > >> > delegation
> > > > > > > >
> > > > > > > > > > >> > >>> > > tokens.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > Basically anyone can request
> delegation
> > > > token,
> > > > > > as
> > > > > > > > long
> > > > > > > >
> > > > > > > > > > as
> > > > > > > >
> > > > > > > > > > >> > they
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > authenticate
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > via a non delegation token mechanism.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > Aren't classes listed here
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > <
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > https://cwiki.apache.org/confl
> > > > > > > uence/display/KAFKA/KIP-
> > > > > > > >
> > > > > > > > > > >> > >>> > > 48+Delegation+token+support+fo
> > > > > > > >
> > > > > > > > > r+Kafka#KIP-48Delegationtokens
> > > > > > > >
> > > > > > > > > > >> > >>> upportforKaf
> > > > > > > >
> > > > > > > > > > >> > >>> > > ka-PublicInterfaces
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > sufficient?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > Thanks
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > Parth
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun
> Rao
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > <ju...@confluent.io>
> > > > > > > >
> > > > > > > > > > >> > >>> wrote:
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > Parth,
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > Thanks for the reply. A couple of
> > > comments
> > > > > > > inline
> > > > > > > >
> > > > > > > > > > below.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > On Tue, Jun 21, 2016 at 10:36 AM,
> > parth
> > > > > > > > brahmbhatt <
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > 1. Who / how are tokens renewed?
> By
> > > > > original
> > > > > > > >
> > > > > > > > > > requester
> > > > > > > >
> > > > > > > > > > >> > >>> only? or
> > > > > > > >
> > > > > > > > > > >> > >>> > > > using
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > Kerberos
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > auth only?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > My recommendation is to do this
> only
> > > > using
> > > > > > > >
> > > > > > > > > Kerberos
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > auth
> > > > > > > >
> > > > > > > > > > >> > and
> > > > > > > >
> > > > > > > > > > >> > >>> > only
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > threw
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > the
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > renewer specified during the
> > > acquisition
> > > > > > > > request.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > Hmm, not sure that I follow this.
> Are
> > > you
> > > > > > saying
> > > > > > > >
> > > > > > > > > that
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > any
> > > > > > > >
> > > > > > > > > > >> > >>> client
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > authenticated with the delegation
> > token
> > > > can
> > > > > > > renew,
> > > > > > > >
> > > > > > > > > > i.e.
> > > > > > > >
> > > > > > > > > > >> > there
> > > > > > > >
> > > > > > > > > > >> > >>> is
> > > > > > > >
> > > > > > > > > > >> > >>> > no
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > renewer
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > needed?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > Also, just to be clear, any
> > > authenticated
> > > > > > client
> > > > > > > >
> > > > > > > > > > (either
> > > > > > > >
> > > > > > > > > > >> > >>> through
> > > > > > > >
> > > > > > > > > > >> > >>> > > SASL
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > or
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > SSL) can request a delegation token
> > for
> > > > the
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > authenticated
> > > > > > > >
> > > > > > > > > > >> > >>> user,
> > > > > > > >
> > > > > > > > > > >> > >>> > > > right?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > 2. Are tokens stored on each
> broker
> > or
> > > > in
> > > > > > ZK?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > My recommendation is still to
> store
> > in
> > > > ZK
> > > > > or
> > > > > > > not
> > > > > > > >
> > > > > > > > > > store
> > > > > > > >
> > > > > > > > > > >> > them
> > > > > > > >
> > > > > > > > > > >> > >>> at
> > > > > > > >
> > > > > > > > > > >> > >>> > > all.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > The
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > whole controller based
> distribution
> > is
> > > > too
> > > > > > > much
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > overhead
> > > > > > > >
> > > > > > > > > > >> > >>> with
> > > > > > > >
> > > > > > > > > > >> > >>> > not
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > much
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > to
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > achieve.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > 3. How are tokens invalidated /
> > > expired?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > Either by expiration time out or
> > > through
> > > > > an
> > > > > > > >
> > > > > > > > > explicit
> > > > > > > >
> > > > > > > > > > >> > >>> request to
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > invalidate.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > 4. Which encryption algorithm is
> > used?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > SCRAM
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > 5. What is the impersonation
> > proposal
> > > > (it
> > > > > > > wasn't
> > > > > > > >
> > > > > > > > > in
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > the
> > > > > > > >
> > > > > > > > > > >> > KIP
> > > > > > > >
> > > > > > > > > > >> > >>> but
> > > > > > > >
> > > > > > > > > > >> > >>> > > was
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > discussed
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > in this thread)?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > There is no imperonation
> proposal. I
> > > > tried
> > > > > > and
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > explained
> > > > > > > >
> > > > > > > > > > >> > how
> > > > > > > >
> > > > > > > > > > >> > >>> > its
> > > > > > > >
> > > > > > > > > > >> > >>> > > a
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > different problem and why its not
> > > really
> > > > > > > > necessary
> > > > > > > >
> > > > > > > > > > to
> > > > > > > >
> > > > > > > > > > >> > >>> discuss
> > > > > > > >
> > > > > > > > > > >> > >>> > > that
> > > > > > > >
> > > > > > > > > > >> > >>> > > > as
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > part
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > of this KIP.  This KIP will not
> > > support
> > > > > any
> > > > > > > >
> > > > > > > > > > >> > impersonation,
> > > > > > > >
> > > > > > > > > > >> > >>> it
> > > > > > > >
> > > > > > > > > > >> > >>> > > will
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > just
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > be
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > another way to authenticate.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > 6. Do we need new ACLs, if so -
> for
> > > what
> > > > > > > > actions?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > We do not need new ACLs.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > Could we document the format of the
> > new
> > > > > > > >
> > > > > > > > > > request/response
> > > > > > > >
> > > > > > > > > > >> > and
> > > > > > > >
> > > > > > > > > > >> > >>> > their
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > associated Resource and Operation
> for
> > > ACL?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > 7. How would the delegation token
> be
> > > > > > > configured
> > > > > > > > in
> > > > > > > >
> > > > > > > > > > the
> > > > > > > >
> > > > > > > > > > >> > >>> client?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > Should be through config. I wasn't
> > > > > planning
> > > > > > on
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > supporting
> > > > > > > >
> > > > > > > > > > >> > >>> JAAS
> > > > > > > >
> > > > > > > > > > >> > >>> > > for
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > tokens.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > I don't believe hadoop does this
> > > either.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > Parth
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM,
> Jun
> > > > Rao <
> > > > > > > >
> > > > > > > > > > >> > jun@confluent.io>
> > > > > > > >
> > > > > > > > > > >> > >>> > > wrote:
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > Harsha,
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > Another question.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > 9. How would the delegation
> token
> > be
> > > > > > > > configured
> > > > > > > >
> > > > > > > > > in
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > the
> > > > > > > >
> > > > > > > > > > >> > >>> > client?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > The
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > standard
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > way is to do this through JAAS.
> > > > However,
> > > > > > we
> > > > > > > > will
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > need
> > > > > > > >
> > > > > > > > > > >> > to
> > > > > > > >
> > > > > > > > > > >> > >>> > think
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > through
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > if
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > this is convenient in a shared
> > > > > > environment.
> > > > > > > > For
> > > > > > > >
> > > > > > > > > > >> > example,
> > > > > > > >
> > > > > > > > > > >> > >>> > when a
> > > > > > > >
> > > > > > > > > > >> > >>> > > > new
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > task
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > is
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > added to a Storm worker node, do
> > we
> > > > need
> > > > > > to
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > dynamically
> > > > > > > >
> > > > > > > > > > >> > >>> add a
> > > > > > > >
> > > > > > > > > > >> > >>> > > new
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > section
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > in the JAAS file? It may be more
> > > > > > convenient
> > > > > > > if
> > > > > > > >
> > > > > > > > > we
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > can
> > > > > > > >
> > > > > > > > > > >> > >>> pass in
> > > > > > > >
> > > > > > > > > > >> > >>> > > the
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > token
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > through the config directly w/o
> > > going
> > > > > > > through
> > > > > > > >
> > > > > > > > > > JAAS.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > Are you or Parth still actively
> > > > working
> > > > > on
> > > > > > > > this
> > > > > > > >
> > > > > > > > > > KIP?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > Jun
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM,
> > Jun
> > > > > Rao <
> > > > > > > >
> > > > > > > > > > >> > >>> jun@confluent.io>
> > > > > > > >
> > > > > > > > > > >> > >>> > > > wrote:
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > > Just to add on that list.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > > 2. It would be good to
> document
> > > the
> > > > > > format
> > > > > > > > of
> > > > > > > >
> > > > > > > > > > the
> > > > > > > >
> > > > > > > > > > >> > data
> > > > > > > >
> > > > > > > > > > >> > >>> > stored
> > > > > > > >
> > > > > > > > > > >> > >>> > > > in
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > ZK.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > > 7. Earlier, there was a
> > discussion
> > > > on
> > > > > > > > whether
> > > > > > > >
> > > > > > > > > > the
> > > > > > > >
> > > > > > > > > > >> > tokens
> > > > > > > >
> > > > > > > > > > >> > >>> > > should
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > be
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > > propagated through ZK like
> > > > > > > config/acl/quota,
> > > > > > > >
> > > > > > > > > or
> > > > > > > >
> > > > > > > > > > >> > through
> > > > > > > >
> > > > > > > > > > >> > >>> the
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > controller.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > > Currently, the controller is
> > only
> > > > > > designed
> > > > > > > > for
> > > > > > > >
> > > > > > > > > > >> > >>> propagating
> > > > > > > >
> > > > > > > > > > >> > >>> > > > topic
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > metadata,
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > > but not other data.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > > 8. Should we use SCRAM to send
> > the
> > > > > token
> > > > > > > >
> > > > > > > > > instead
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > > of
> > > > > > > >
> > > > > > > > > > >> > >>> > > DIGEST-MD5
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > since
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > it's
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > > deprecated?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > > Also, the images in the wiki
> > seem
> > > > > > broken.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > > Jun
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > > On Fri, Jun 10, 2016 at 10:02
> > AM,
> > > > Gwen
> > > > > > > >
> > > > > > > > > Shapira <
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > gwen@confluent.io>
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> From what I can see,
> remaining
> > > > > > questions
> > > > > > > > are:
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >>
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> 1. Who / how are tokens
> > renewed?
> > > By
> > > > > > > > original
> > > > > > > >
> > > > > > > > > > >> > requester
> > > > > > > >
> > > > > > > > > > >> > >>> > only?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > or
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > using
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> Kerberos auth only?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> 2. Are tokens stored on each
> > > broker
> > > > > or
> > > > > > in
> > > > > > > > ZK?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> 3. How are tokens
> invalidated /
> > > > > > expired?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> 4. Which encryption algorithm
> > is
> > > > > used?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> 5. What is the impersonation
> > > > proposal
> > > > > > (it
> > > > > > > >
> > > > > > > > > > wasn't
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> in
> > > > > > > >
> > > > > > > > > > >> > the
> > > > > > > >
> > > > > > > > > > >> > >>> > KIP
> > > > > > > >
> > > > > > > > > > >> > >>> > > > but
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > was
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> discussed in this thread)?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> 6. Do we need new ACLs, if
> so -
> > > for
> > > > > > what
> > > > > > > >
> > > > > > > > > > actions?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >>
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> Gwen
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >>
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> On Thu, Jun 9, 2016 at 7:48
> PM,
> > > > > Harsha
> > > > > > <
> > > > > > > >
> > > > > > > > > > >> > >>> kafka@harsha.io>
> > > > > > > >
> > > > > > > > > > >> > >>> > > > wrote:
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> > Jun & Ismael,
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >
> > > > > > Unfortunately
> > > > > > > I
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> > couldn't
> > > > > > > >
> > > > > > > > > > >> > >>> attend
> > > > > > > >
> > > > > > > > > > >> > >>> > > the
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > KIP
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > meeting
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >
> when
> > > > > > > delegation
> > > > > > > >
> > > > > > > > > > tokens
> > > > > > > >
> > > > > > > > > > >> > >>> > discussed.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > Appreciate
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > if
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >
> you
> > > can
> > > > > > update
> > > > > > > > the
> > > > > > > >
> > > > > > > > > > >> > thread if
> > > > > > > >
> > > > > > > > > > >> > >>> > you
> > > > > > > >
> > > > > > > > > > >> > >>> > > > have
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > any
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >
> > > further
> > > > > > > > questions.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> > Thanks,
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> > Harsha
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> > On Tue, May 24, 2016, at
> > 11:32
> > > > AM,
> > > > > > > Liquan
> > > > > > > >
> > > > > > > > > Pei
> > > > > > > >
> > > > > > > > > > >> > wrote:
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> It seems that the links to
> > > > images
> > > > > in
> > > > > > > the
> > > > > > > >
> > > > > > > > > KIP
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> are
> > > > > > > >
> > > > > > > > > > >> > >>> > broken.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >>
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> Liquan
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >>
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> On Tue, May 24, 2016 at
> 9:33
> > > AM,
> > > > > > parth
> > > > > > > >
> > > > > > > > > > >> > brahmbhatt <
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >>
> brahmbhatt.parth@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >>
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> > 110. What does
> > > > > > getDelegationTokenAs
> > > > > > > >
> > > > > > > > > mean?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> > In the current proposal
> we
> > > > only
> > > > > > > allow
> > > > > > > > a
> > > > > > > >
> > > > > > > > > > user
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> > to
> > > > > > > >
> > > > > > > > > > >> > >>> get
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > delegation
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > token
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> for
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> > the identity that it
> > > > > authenticated
> > > > > > > as
> > > > > > > >
> > > > > > > > > > using
> > > > > > > >
> > > > > > > > > > >> > >>> another
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > mechanism,
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > i.e.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> A user
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> > that authenticate using
> a
> > > > keytab
> > > > > > for
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> > principal
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > user1@EXAMPLE.COM
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> will get
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens for
> that
> > > > user
> > > > > > > only.
> > > > > > > > In
> > > > > > > >
> > > > > > > > > > >> > future I
> > > > > > > >
> > > > > > > > > > >> > >>> > think
> > > > > > > >
> > > > > > > > > > >> > >>> > > > we
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > will
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > have
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> to
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> > extend support such that
> > we
> > > > > allow
> > > > > > > some
> > > > > > > >
> > > > > > > > > set
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> > of
> > > > > > > >
> > > > > > > > > > >> > >>> users (
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> >
> > kafka-rest-user@EXAMPLE.COM
> > > ,
> > > > > > > >
> > > > > > > > > > >> > >>> > storm-nimbus@EXAMPLE.COM)
> > > > > > > >
> > > > > > > > > > >> > >>> > > > to
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > acquire
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens on
> > behalf
> > > of
> > > > > > other
> > > > > > > >
> > > > > > > > > users
> > > > > > > >
> > > > > > > > > > >> > whose
> > > > > > > >
> > > > > > > > > > >> > >>> > > identity
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > they
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > have
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> > verified independently.
> > > Kafka
> > > > > > > brokers
> > > > > > > >
> > > > > > > > > > will
> > > > > > > >
> > > > > > > > > > >> > have
> > > > > > > >
> > > > > > > > > > >> > >>> ACLs
> > > > > > > >
> > > > > > > > > > >> > >>> > > to
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > control
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> which
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> > users are allowed to
> > > > impersonate
> > > > > > > other
> > > > > > > >
> > > > > > > > > > users
> > > > > > > >
> > > > > > > > > > >> > and
> > > > > > > >
> > > > > > > > > > >> > >>> get
> > > > > > > >
> > > > > > > > > > >> > >>> > > > tokens
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > on
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> behalf of
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> > them. Overall
> > Impersonation
> > > > is a
> > > > > > > whole
> > > > > > > >
> > > > > > > > > > >> > different
> > > > > > > >
> > > > > > > > > > >> > >>> > > problem
> > > > > > > >
> > > > > > > > > > >> > >>> > > > in
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > my
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> opinion and
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> > I think we can tackle it
> > in
> > > > > > separate
> > > > > > > >
> > > > > > > > > KIP.
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> > 111. What's the typical
> > rate
> > > > of
> > > > > > > > getting
> > > > > > > >
> > > > > > > > > > and
> > > > > > > >
> > > > > > > > > > >> > >>> renewing
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > delegation
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> tokens?
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> > Typically this should be
> > > very
> > > > > very
> > > > > > > > low,
> > > > > > > >
> > > > > > > > > 1
> > > > > > > >
> > > > > > > > > > >> > request
> > > > > > > >
> > > > > > > > > > >> > >>> per
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > minute
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > is a
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> > relatively high
> estimate.
> > > > > However
> > > > > > it
> > > > > > > >
> > > > > > > > > > depends
> > > > > > > >
> > > > > > > > > > >> > >>> > > > > > > > > >> >> > on
> > > > > > > >
> > > > > > > > > > >> > >>> the
> > > > > > > >
> > > > > > > > > > >> > >>> > > token
> > > > > > > >
> >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Harsha Chintalapani <ka...@harsha.io>]

@Gwen @Mani  Not sure why we want to authenticate at every request. Even if
the token exchange is cheap it still a few calls that need to go through
round trip.  Impersonation doesn't require authentication for every
request.

"So a centralized app can create few producers, do the metadata request and
broker discovery with its own user auth, but then use delegation tokens to
allow performing produce/fetch requests as different users? Instead of
having to re-connect for each impersonated user?"

Yes. But what we will have is this centralized user as impersonation user
on behalf of other users. When it authenticates initially we will create a
"Subject" and from there on wards centralized user can do
Subject.doAsPrivileged
on behalf, other users.
On the server side, we can retrieve two principals out of this one is the
authenticated user (centralized user) and another is impersonated user. We
will first check if the authenticated user allowed to impersonate and then
move on to check if the user Alice has access to the topic "X" to
read/write.

@Rajini Intention of this KIP is to support token auth via SASL/SCRAM, not
just with TLS.  What you raised is a good point let me take a look and add
details.

It will be easier to add impersonation once we reach agreement on this KIP.


On Wed, Dec 14, 2016 at 5:51 AM Ismael Juma <is...@juma.me.uk> wrote:

> Hi Rajini,
>
> I think it would definitely be valuable to have a KIP for impersonation.
>
> Ismael
>
> On Wed, Dec 14, 2016 at 4:03 AM, Rajini Sivaram <rs...@pivotal.io>
> wrote:
>
> > It would clearly be very useful to enable clients to send requests on
> > behalf of multiple users. A separate KIP makes sense, but it may be worth
> > thinking through some of the implications now, especially if the main
> > interest in delegation tokens comes from its potential to enable
> > impersonation.
> >
> > I understand that delegation tokens are only expected to be used with
> TLS.
> > But the choice of SASL/SCRAM for authentication must be based on a
> > requirement to protect the tokenHmac - otherwise you could just use
> > SASL/PLAIN. With SASL/SCRAM the tokenHmac is never propagated
> on-the-wire,
> > only a salted-hashed version of it is used in the SASL authentication
> > exchange. If impersonation is based on sending tokenHmac in requests, any
> > benefit of using SCRAM is lost.
> >
> > An alternative may be to allow clients to authenticate multiple times
> using
> > SASL and include one of its authenticated principals in each request
> > (optionally). I haven't thought it through yet, obviously. But if the
> > approach is of interest and no one is working on a KIP for impersonation
> at
> > the moment, I am happy to write one. It may provide something for
> > comparison at least.
> >
> > Thoughts?
> >
> >
> > On Wed, Dec 14, 2016 at 9:53 AM, Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > That's a good idea. Authenticating every request with delegation token
> > will
> > > be useful for
> > > impersonation use-cases. But as of now, we are thinking delegation
> token
> > as
> > > just another way
> > > to authenticate the users. We haven't think through all the use cases
> > > related to
> > > impersonation or using delegation token for impersonation. We want to
> > > handle impersonation
> > > (KAFKA-3712) as part of separate KIP.
> > >
> > > Will that be Ok?
> > >
> > >
> > > On Wed, Dec 14, 2016 at 8:09 AM, Gwen Shapira <gw...@confluent.io>
> wrote:
> > >
> > > > Thinking out loud here:
> > > >
> > > > It looks like authentication with a delegation token is going to be
> > > > super-cheap, right? We just compare the token to a value in the
> broker
> > > > cache?
> > > >
> > > > If I understood the KIP correctly, right now it suggests that
> > > > authentication happens when establishing the client-broker connection
> > (as
> > > > normal for Kafka. But perhaps we want to consider authenticating
> every
> > > > request with delegation token (if exists)?
> > > >
> > > > So a centralized app can create few producers, do the metadata
> request
> > > and
> > > > broker discovery with its own user auth, but then use delegation
> tokens
> > > to
> > > > allow performing produce/fetch requests as different users? Instead
> of
> > > > having to re-connect for each impersonated user?
> > > >
> > > > This may over-complicate things quite a bit (basically adding extra
> > > > information in every request), but maybe it will be useful for
> > > > impersonation use-cases (which seem to drive much of the interest in
> > this
> > > > KIP)?
> > > > Kafka Connect, NiFi and friends can probably use this to share
> clients
> > > > between multiple jobs, tasks, etc.
> > > >
> > > > What do you think?
> > > >
> > > > Gwen
> > > >
> > > > On Tue, Dec 13, 2016 at 12:43 AM, Manikumar <
> manikumar.reddy@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Ashish,
> > > > >
> > > > > Thank you for reviewing the KIP.  Please see the replies inline.
> > > > >
> > > > >
> > > > > > 1. How to disable delegation token authentication?
> > > > > >
> > > > > > This can be achieved in various ways, however I think reusing
> > > > delegation
> > > > > > token secret config for this makes sense here. Avoids creating
> yet
> > > > > another
> > > > > > config and forces delegation token users to consciously set the
> > > secret.
> > > > > If
> > > > > > the secret is not set or set to empty string, brokers should turn
> > off
> > > > > > delegation token support. This will however require a new error
> > code
> > > to
> > > > > > indicate delegation token support is turned off on broker.
> > > > > >
> > > > >
> > > > >   Thanks for the suggestion. Option to turnoff delegation token
> > > > > authentication will be useful.
> > > > >   I'll update the KIP.
> > > > >
> > > > >
> > > > > >
> > > > > > 2. ACLs on delegation token?
> > > > > >
> > > > > > Do we need to have ACLs defined for tokens? I do not think it
> buys
> > us
> > > > > > anything, as delegation token can be treated as impersonation of
> > the
> > > > > owner.
> > > > > > Any thing the owner has permission to do, delegation tokens
> should
> > be
> > > > > > allowed to do as well. If so, we probably won't need to return
> > > > > > authorization exception error code while creating delegation
> token.
> > > It
> > > > > > however would make sense to check renew and expire requests are
> > > coming
> > > > > from
> > > > > > owner or renewers of the token, but that does not require
> explicit
> > > > acls.
> > > > > >
> > > > >
> > > > >
> > > > > Yes, We agreed to not have new acl on who can request delegation
> > token.
> > > > >  I'll update the KIP.
> > > > >
> > > > >
> > > > > >
> > > > > > 3. How to restrict max life time of a token?
> > > > > >
> > > > > > Admins might want to restrict max life time of tokens created on
> a
> > > > > cluster,
> > > > > > and this can very from cluster to cluster based on use-cases.
> This
> > > > might
> > > > > > warrant a separate broker config.
> > > > > >
> > > > > >
> > > > > Currently we  have "delegation.token.max.lifetime.sec" server
> config
> > > > > property
> > > > > May be we can take min(User supplied MaxTime, Server MaxTime) as
> max
> > > life
> > > > > time.
> > > > > I am open to add new config property.
> > > > >
> > > > > Few more comments based on recent KIP update.
> > > > > >
> > > > > > 1. Do we need a separate {{InvalidateTokenRequest}}? Can't we use
> > > > > > {{ExpireTokenRequest}} with with expiryDate set to anything
> before
> > > > > current
> > > > > > date?
> > > > > >
> > > > >
> > > > > makes sense. we don't need special request to cancel the token. We
> > can
> > > > use
> > > > > ExpireTokenRequest.
> > > > > I'll update the KIP.
> > > > >
> > > > >
> > > > > > 2. Can we change time field names to indicate their unit is
> > > > milliseconds,
> > > > > > like, IssueDateMs, ExpiryDateMs, etc.?
> > > > > >
> > > > > >
> > > > >   Done.
> > > > >
> > > > >
> > > > > > 3. Can we allow users to renew a token for a specified amount of
> > > time?
> > > > In
> > > > > > current version of KIP, renew request does not take time as a
> > param,
> > > > not
> > > > > > sure what is expiry time set to after renewal.
> > > > > >
> > > > > >
> > > > >  Yes, we need to specify renew period.  I'll update the KIP.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Mankumar
> > > > >
> > > > >
> > > > >
> > > > > >
> > > > > > On Mon, Dec 12, 2016 at 9:08 AM Manikumar <
> > manikumar.reddy@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I would like to reinitiate the discussion on Delegation token
> > > support
> > > > > for
> > > > > > >
> > > > > > > Kafka.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Brief summary of the past discussion:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 1) Broker stores delegation tokens in zookeeper.  All brokers
> > will
> > > > > have a
> > > > > > >
> > > > > > > cache backed by
> > > > > > >
> > > > > > >    zookeeper so they will all get notified whenever a new token
> > is
> > > > > > >
> > > > > > > generated and they will
> > > > > > >
> > > > > > >    update their local cache whenever token state changes.
> > > > > > >
> > > > > > > 2) The current proposal does not support rotation of secret
> > > > > > >
> > > > > > > 3) Only allow the renewal by users that authenticated using
> *non*
> > > > > > >
> > > > > > > delegation token mechanism
> > > > > > >
> > > > > > > 4) KIP-84 proposes to support  SASL SCRAM mechanisms. Kafka
> > clients
> > > > can
> > > > > > >
> > > > > > > authenticate using
> > > > > > >
> > > > > > >    SCRAM-SHA-256, providing the delegation token HMAC as
> > password.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Updated the KIP with the following:
> > > > > > >
> > > > > > > 1. Protocol and Config changes
> > > > > > >
> > > > > > > 2. format of the data stored in ZK.
> > > > > > >
> > > > > > > 3. Changes to Java Clients/Usage of SASL SCRAM mechanism
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > >
> > > > > > > 48+Delegation+token+support+for+Kafka
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Jun, Ashish, Gwen,
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Pl review the updated KIP.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > Manikumar
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Sep 29, 2016 at 9:56 PM, Ashish Singh <
> > asingh@cloudera.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > Harsha/ Gwen,
> > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > > How do we proceed here? I am willing to help out with here.
> > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > > On Fri, Sep 23, 2016 at 11:41 AM, Gwen Shapira <
> > > gwen@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > > > Is it updated? are all concerns addressed? do you want to
> > > start a
> > > > > > vote?
> > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > > > > Sorry for being pushy, I do appreciate that we are all
> > > volunteers
> > > > > and
> > > > > > >
> > > > > > > > > finding time is difficult. This feature is important for
> > > anything
> > > > > > that
> > > > > > >
> > > > > > > > > integrates with Kafka (stream processors, Flume, NiFi, etc)
> > > and I
> > > > > > >
> > > > > > > > > don't want to see this getting stuck because we lack
> > > coordination
> > > > > > >
> > > > > > > > > within the community.
> > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > > > > On Thu, Sep 15, 2016 at 6:39 PM, Harsha Chintalapani <
> > > > > > kafka@harsha.io>
> > > > > > >
> > > > > > > > > wrote:
> > > > > > >
> > > > > > > > > > The only pending update for the KIP is to write up the
> > > protocol
> > > > > > > changes
> > > > > > >
> > > > > > > > > like
> > > > > > >
> > > > > > > > > > we've it KIP-4. I'll update the wiki.
> > > > > > >
> > > > > > > > > >
> > > > > > >
> > > > > > > > > >
> > > > > > >
> > > > > > > > > > On Thu, Sep 15, 2016 at 4:27 PM Ashish Singh <
> > > > > asingh@cloudera.com>
> > > > > > >
> > > > > > > > > wrote:
> > > > > > >
> > > > > > > > > >>
> > > > > > >
> > > > > > > > > >> I think we decided to not support secret rotation, I
> guess
> > > > this
> > > > > > can
> > > > > > > be
> > > > > > >
> > > > > > > > > >> stated clearly on the KIP. Also, more details on how
> > clients
> > > > > will
> > > > > > >
> > > > > > > > > perform
> > > > > > >
> > > > > > > > > >> token distribution and how CLI will look like will be
> > > helpful.
> > > > > > >
> > > > > > > > > >>
> > > > > > >
> > > > > > > > > >> On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <
> > > > > gwen@confluent.io>
> > > > > > >
> > > > > > > > > wrote:
> > > > > > >
> > > > > > > > > >>
> > > > > > >
> > > > > > > > > >> > Hi Guys,
> > > > > > >
> > > > > > > > > >> >
> > > > > > >
> > > > > > > > > >> > This discussion was dead for a while. Are there still
> > > > > > contentious
> > > > > > >
> > > > > > > > > >> > points? If not, why are there no votes?
> > > > > > >
> > > > > > > > > >> >
> > > > > > >
> > > > > > > > > >> > On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <
> > > jun@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > > >> > > Ashish,
> > > > > > >
> > > > > > > > > >> > >
> > > > > > >
> > > > > > > > > >> > > Yes, I will send out a KIP invite for next week to
> > > discuss
> > > > > > > KIP-48
> > > > > > >
> > > > > > > > > and
> > > > > > >
> > > > > > > > > >> > other
> > > > > > >
> > > > > > > > > >> > > remaining KIPs.
> > > > > > >
> > > > > > > > > >> > >
> > > > > > >
> > > > > > > > > >> > > Thanks,
> > > > > > >
> > > > > > > > > >> > >
> > > > > > >
> > > > > > > > > >> > > Jun
> > > > > > >
> > > > > > > > > >> > >
> > > > > > >
> > > > > > > > > >> > > On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh <
> > > > > > >
> > > > > > > > asingh@cloudera.com>
> > > > > > >
> > > > > > > > > >> > wrote:
> > > > > > >
> > > > > > > > > >> > >
> > > > > > >
> > > > > > > > > >> > >> Thanks Harsha!
> > > > > > >
> > > > > > > > > >> > >>
> > > > > > >
> > > > > > > > > >> > >> Jun, can we add KIP-48 to next KIP hangout's
> agenda.
> > > > Also,
> > > > > we
> > > > > > > did
> > > > > > >
> > > > > > > > > not
> > > > > > >
> > > > > > > > > >> > >> actually make a call on when we should have next
> KIP
> > > > call.
> > > > > As
> > > > > > >
> > > > > > > > there
> > > > > > >
> > > > > > > > > >> > >> are
> > > > > > >
> > > > > > > > > >> > a
> > > > > > >
> > > > > > > > > >> > >> few outstanding KIPs that could not be discussed
> this
> > > > week,
> > > > > > can
> > > > > > >
> > > > > > > > we
> > > > > > >
> > > > > > > > > >> > >> have
> > > > > > >
> > > > > > > > > >> > a
> > > > > > >
> > > > > > > > > >> > >> KIP hangout call next week?
> > > > > > >
> > > > > > > > > >> > >>
> > > > > > >
> > > > > > > > > >> > >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha
> Chintalapani
> > > > > > >
> > > > > > > > > >> > >> <ka...@harsha.io>
> > > > > > >
> > > > > > > > > >> > >> wrote:
> > > > > > >
> > > > > > > > > >> > >>
> > > > > > >
> > > > > > > > > >> > >>> Ashish,
> > > > > > >
> > > > > > > > > >> > >>>         Yes we are working on it. Lets discuss in
> > the
> > > > next
> > > > > > KIP
> > > > > > >
> > > > > > > > > >> > >>> meeting.
> > > > > > >
> > > > > > > > > >> > >>> I'll join.
> > > > > > >
> > > > > > > > > >> > >>> -Harsha
> > > > > > >
> > > > > > > > > >> > >>>
> > > > > > >
> > > > > > > > > >> > >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh <
> > > > > > >
> > > > > > > > > asingh@cloudera.com>
> > > > > > >
> > > > > > > > > >> > >>> wrote:
> > > > > > >
> > > > > > > > > >> > >>>
> > > > > > >
> > > > > > > > > >> > >>> > Hello Harsha,
> > > > > > >
> > > > > > > > > >> > >>> >
> > > > > > >
> > > > > > > > > >> > >>> > Are you still working on this? Wondering if we
> can
> > > > > discuss
> > > > > > >
> > > > > > > > this
> > > > > > >
> > > > > > > > > in
> > > > > > >
> > > > > > > > > >> > next
> > > > > > >
> > > > > > > > > >> > >>> KIP
> > > > > > >
> > > > > > > > > >> > >>> > meeting, if you can join.
> > > > > > >
> > > > > > > > > >> > >>> >
> > > > > > >
> > > > > > > > > >> > >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha
> > > Chintalapani <
> > > > > > >
> > > > > > > > > >> > kafka@harsha.io>
> > > > > > >
> > > > > > > > > >> > >>> > wrote:
> > > > > > >
> > > > > > > > > >> > >>> >
> > > > > > >
> > > > > > > > > >> > >>> > > Hi Grant,
> > > > > > >
> > > > > > > > > >> > >>> > >           We are working on it. Will add the
> > > details
> > > > > to
> > > > > > > KIP
> > > > > > >
> > > > > > > > > >> > >>> > > about
> > > > > > >
> > > > > > > > > >> > the
> > > > > > >
> > > > > > > > > >> > >>> > > request protocol.
> > > > > > >
> > > > > > > > > >> > >>> > >
> > > > > > >
> > > > > > > > > >> > >>> > > Thanks,
> > > > > > >
> > > > > > > > > >> > >>> > > Harsha
> > > > > > >
> > > > > > > > > >> > >>> > >
> > > > > > >
> > > > > > > > > >> > >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant Henke
> > > > > > >
> > > > > > > > > >> > >>> > > <gh...@cloudera.com>
> > > > > > >
> > > > > > > > > >> > >>> wrote:
> > > > > > >
> > > > > > > > > >> > >>> > >
> > > > > > >
> > > > > > > > > >> > >>> > > > Hi Parth,
> > > > > > >
> > > > > > > > > >> > >>> > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > Are you still working on this? If you need
> any
> > > > help
> > > > > > > please
> > > > > > >
> > > > > > > > > >> > >>> > > > don't
> > > > > > >
> > > > > > > > > >> > >>> > hesitate
> > > > > > >
> > > > > > > > > >> > >>> > > > to ask.
> > > > > > >
> > > > > > > > > >> > >>> > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > Thanks,
> > > > > > >
> > > > > > > > > >> > >>> > > > Grant
> > > > > > >
> > > > > > > > > >> > >>> > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <
> > > > > > >
> > > > > > > > jun@confluent.io>
> > > > > > >
> > > > > > > > > >> > wrote:
> > > > > > >
> > > > > > > > > >> > >>> > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > Parth,
> > > > > > >
> > > > > > > > > >> > >>> > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > Thanks for the reply.
> > > > > > >
> > > > > > > > > >> > >>> > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > It makes sense to only allow the renewal
> by
> > > > users
> > > > > > that
> > > > > > >
> > > > > > > > > >> > >>> authenticated
> > > > > > >
> > > > > > > > > >> > >>> > > > using
> > > > > > >
> > > > > > > > > >> > >>> > > > > *non* delegation token mechanism. Then,
> > should
> > > > we
> > > > > > make
> > > > > > >
> > > > > > > > the
> > > > > > >
> > > > > > > > > >> > >>> renewal a
> > > > > > >
> > > > > > > > > >> > >>> > > > list?
> > > > > > >
> > > > > > > > > >> > >>> > > > > For example, in the case of rest proxy, it
> > > will
> > > > be
> > > > > > >
> > > > > > > > useful
> > > > > > >
> > > > > > > > > >> > >>> > > > > for
> > > > > > >
> > > > > > > > > >> > >>> every
> > > > > > >
> > > > > > > > > >> > >>> > > > > instance of rest proxy to be able to renew
> > the
> > > > > > tokens.
> > > > > > >
> > > > > > > > > >> > >>> > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > It would be clearer if we can document the
> > > > request
> > > > > > >
> > > > > > > > > protocol
> > > > > > >
> > > > > > > > > >> > like
> > > > > > >
> > > > > > > > > >> > >>> > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > https://cwiki.apache.org/confl
> > > > > > uence/display/KAFKA/KIP-
> > > > > > >
> > > > > > > > > >> > >>> > >
> > > > > > >
> > > > > > > > > >> > >>> > > 4+-+Command+line+and+
> > centralized+administrative+
> > > > > > >
> > > > > > > > > operations#KIP-4-
> > > > > > >
> > > > > > > > > >> > >>> > > Commandlineandcentralizedadmin
> > > istrativeoperations-
> > > > > > >
> > > > > > > > > >> > >>> > > CreateTopicsRequest(KAFKA-
> > > > > 2945):(VotedandPlannedforin0.
> > > > > > >
> > > > > > > > > 10.1.0)
> > > > > > >
> > > > > > > > > >> > >>> > > > > .
> > > > > > >
> > > > > > > > > >> > >>> > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > It would also be useful to document the
> > client
> > > > > APIs.
> > > > > > >
> > > > > > > > > >> > >>> > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > Thanks,
> > > > > > >
> > > > > > > > > >> > >>> > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > Jun
> > > > > > >
> > > > > > > > > >> > >>> > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM, parth
> > > > brahmbhatt
> > > > > <
> > > > > > >
> > > > > > > > > >> > >>> > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > > > >
> > > > > > > > > >> > >>> > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > Hi,
> > > > > > >
> > > > > > > > > >> > >>> > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > I am suggesting that we will only allow
> > the
> > > > > > renewal
> > > > > > > by
> > > > > > >
> > > > > > > > > >> > >>> > > > > > users
> > > > > > >
> > > > > > > > > >> > >>> that
> > > > > > >
> > > > > > > > > >> > >>> > > > > > authenticated using *non* delegation
> token
> > > > > > > mechanism.
> > > > > > >
> > > > > > > > > For
> > > > > > >
> > > > > > > > > >> > >>> example,
> > > > > > >
> > > > > > > > > >> > >>> > If
> > > > > > >
> > > > > > > > > >> > >>> > > > > user
> > > > > > >
> > > > > > > > > >> > >>> > > > > > Alice authenticated using kerberos and
> > > > requested
> > > > > > >
> > > > > > > > > >> > >>> > > > > > delegation
> > > > > > >
> > > > > > > > > >> > >>> tokens,
> > > > > > >
> > > > > > > > > >> > >>> > > > only
> > > > > > >
> > > > > > > > > >> > >>> > > > > > user Alice authenticated via non
> > delegation
> > > > > token
> > > > > > >
> > > > > > > > > >> > >>> > > > > > mechanism
> > > > > > >
> > > > > > > > > >> > can
> > > > > > >
> > > > > > > > > >> > >>> > > renew.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > Clients that have  access to delegation
> > > tokens
> > > > > can
> > > > > > > not
> > > > > > >
> > > > > > > > > >> > >>> > > > > > issue
> > > > > > >
> > > > > > > > > >> > >>> > renewal
> > > > > > >
> > > > > > > > > >> > >>> > > > > > request for renewing their own token and
> > > this
> > > > is
> > > > > > >
> > > > > > > > > primarily
> > > > > > >
> > > > > > > > > >> > >>> > important
> > > > > > >
> > > > > > > > > >> > >>> > > to
> > > > > > >
> > > > > > > > > >> > >>> > > > > > reduce the time window for which a
> > > compromised
> > > > > > token
> > > > > > >
> > > > > > > > > will
> > > > > > >
> > > > > > > > > >> > >>> > > > > > be
> > > > > > >
> > > > > > > > > >> > >>> valid.
> > > > > > >
> > > > > > > > > >> > >>> > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > To clarify, Yes any authenticated user
> can
> > > > > request
> > > > > > >
> > > > > > > > > >> > >>> > > > > > delegation
> > > > > > >
> > > > > > > > > >> > >>> > tokens
> > > > > > >
> > > > > > > > > >> > >>> > > > but
> > > > > > >
> > > > > > > > > >> > >>> > > > > > even here I would recommend to avoid
> > > creating
> > > > a
> > > > > > > chain
> > > > > > >
> > > > > > > > > >> > >>> > > > > > where a
> > > > > > >
> > > > > > > > > >> > >>> > client
> > > > > > >
> > > > > > > > > >> > >>> > > > > > authenticated via delegation token
> request
> > > for
> > > > > > more
> > > > > > >
> > > > > > > > > >> > delegation
> > > > > > >
> > > > > > > > > >> > >>> > > tokens.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > Basically anyone can request delegation
> > > token,
> > > > > as
> > > > > > > long
> > > > > > >
> > > > > > > > > as
> > > > > > >
> > > > > > > > > >> > they
> > > > > > >
> > > > > > > > > >> > >>> > > > > authenticate
> > > > > > >
> > > > > > > > > >> > >>> > > > > > via a non delegation token mechanism.
> > > > > > >
> > > > > > > > > >> > >>> > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > Aren't classes listed here
> > > > > > >
> > > > > > > > > >> > >>> > > > > > <
> > > > > > >
> > > > > > > > > >> > >>> > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > https://cwiki.apache.org/confl
> > > > > > uence/display/KAFKA/KIP-
> > > > > > >
> > > > > > > > > >> > >>> > > 48+Delegation+token+support+fo
> > > > > > >
> > > > > > > > r+Kafka#KIP-48Delegationtokens
> > > > > > >
> > > > > > > > > >> > >>> upportforKaf
> > > > > > >
> > > > > > > > > >> > >>> > > ka-PublicInterfaces
> > > > > > >
> > > > > > > > > >> > >>> > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > sufficient?
> > > > > > >
> > > > > > > > > >> > >>> > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > Thanks
> > > > > > >
> > > > > > > > > >> > >>> > > > > > Parth
> > > > > > >
> > > > > > > > > >> > >>> > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao
> > > > > > >
> > > > > > > > > >> > >>> > > > > > <ju...@confluent.io>
> > > > > > >
> > > > > > > > > >> > >>> wrote:
> > > > > > >
> > > > > > > > > >> > >>> > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > Parth,
> > > > > > >
> > > > > > > > > >> > >>> > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > Thanks for the reply. A couple of
> > comments
> > > > > > inline
> > > > > > >
> > > > > > > > > below.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > On Tue, Jun 21, 2016 at 10:36 AM,
> parth
> > > > > > > brahmbhatt <
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > > > >
> > > > > > > > > >> > >>> > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > 1. Who / how are tokens renewed? By
> > > > original
> > > > > > >
> > > > > > > > > requester
> > > > > > >
> > > > > > > > > >> > >>> only? or
> > > > > > >
> > > > > > > > > >> > >>> > > > using
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > Kerberos
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > auth only?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > My recommendation is to do this only
> > > using
> > > > > > >
> > > > > > > > Kerberos
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > auth
> > > > > > >
> > > > > > > > > >> > and
> > > > > > >
> > > > > > > > > >> > >>> > only
> > > > > > >
> > > > > > > > > >> > >>> > > > > threw
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > the
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > renewer specified during the
> > acquisition
> > > > > > > request.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > Hmm, not sure that I follow this. Are
> > you
> > > > > saying
> > > > > > >
> > > > > > > > that
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > any
> > > > > > >
> > > > > > > > > >> > >>> client
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > authenticated with the delegation
> token
> > > can
> > > > > > renew,
> > > > > > >
> > > > > > > > > i.e.
> > > > > > >
> > > > > > > > > >> > there
> > > > > > >
> > > > > > > > > >> > >>> is
> > > > > > >
> > > > > > > > > >> > >>> > no
> > > > > > >
> > > > > > > > > >> > >>> > > > > > renewer
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > needed?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > Also, just to be clear, any
> > authenticated
> > > > > client
> > > > > > >
> > > > > > > > > (either
> > > > > > >
> > > > > > > > > >> > >>> through
> > > > > > >
> > > > > > > > > >> > >>> > > SASL
> > > > > > >
> > > > > > > > > >> > >>> > > > > or
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > SSL) can request a delegation token
> for
> > > the
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > authenticated
> > > > > > >
> > > > > > > > > >> > >>> user,
> > > > > > >
> > > > > > > > > >> > >>> > > > right?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > 2. Are tokens stored on each broker
> or
> > > in
> > > > > ZK?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > My recommendation is still to store
> in
> > > ZK
> > > > or
> > > > > > not
> > > > > > >
> > > > > > > > > store
> > > > > > >
> > > > > > > > > >> > them
> > > > > > >
> > > > > > > > > >> > >>> at
> > > > > > >
> > > > > > > > > >> > >>> > > all.
> > > > > > >
> > > > > > > > > >> > >>> > > > > The
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > whole controller based distribution
> is
> > > too
> > > > > > much
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > overhead
> > > > > > >
> > > > > > > > > >> > >>> with
> > > > > > >
> > > > > > > > > >> > >>> > not
> > > > > > >
> > > > > > > > > >> > >>> > > > > much
> > > > > > >
> > > > > > > > > >> > >>> > > > > > to
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > achieve.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > 3. How are tokens invalidated /
> > expired?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > Either by expiration time out or
> > through
> > > > an
> > > > > > >
> > > > > > > > explicit
> > > > > > >
> > > > > > > > > >> > >>> request to
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > invalidate.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > 4. Which encryption algorithm is
> used?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > SCRAM
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > 5. What is the impersonation
> proposal
> > > (it
> > > > > > wasn't
> > > > > > >
> > > > > > > > in
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > the
> > > > > > >
> > > > > > > > > >> > KIP
> > > > > > >
> > > > > > > > > >> > >>> but
> > > > > > >
> > > > > > > > > >> > >>> > > was
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > discussed
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > in this thread)?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > There is no imperonation proposal. I
> > > tried
> > > > > and
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > explained
> > > > > > >
> > > > > > > > > >> > how
> > > > > > >
> > > > > > > > > >> > >>> > its
> > > > > > >
> > > > > > > > > >> > >>> > > a
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > different problem and why its not
> > really
> > > > > > > necessary
> > > > > > >
> > > > > > > > > to
> > > > > > >
> > > > > > > > > >> > >>> discuss
> > > > > > >
> > > > > > > > > >> > >>> > > that
> > > > > > >
> > > > > > > > > >> > >>> > > > as
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > part
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > of this KIP.  This KIP will not
> > support
> > > > any
> > > > > > >
> > > > > > > > > >> > impersonation,
> > > > > > >
> > > > > > > > > >> > >>> it
> > > > > > >
> > > > > > > > > >> > >>> > > will
> > > > > > >
> > > > > > > > > >> > >>> > > > > just
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > be
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > another way to authenticate.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > 6. Do we need new ACLs, if so - for
> > what
> > > > > > > actions?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > We do not need new ACLs.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > Could we document the format of the
> new
> > > > > > >
> > > > > > > > > request/response
> > > > > > >
> > > > > > > > > >> > and
> > > > > > >
> > > > > > > > > >> > >>> > their
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > associated Resource and Operation for
> > ACL?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > 7. How would the delegation token be
> > > > > > configured
> > > > > > > in
> > > > > > >
> > > > > > > > > the
> > > > > > >
> > > > > > > > > >> > >>> client?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > Should be through config. I wasn't
> > > > planning
> > > > > on
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > supporting
> > > > > > >
> > > > > > > > > >> > >>> JAAS
> > > > > > >
> > > > > > > > > >> > >>> > > for
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > tokens.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > I don't believe hadoop does this
> > either.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > Thanks
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > Parth
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun
> > > Rao <
> > > > > > >
> > > > > > > > > >> > jun@confluent.io>
> > > > > > >
> > > > > > > > > >> > >>> > > wrote:
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > Harsha,
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > Another question.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > 9. How would the delegation token
> be
> > > > > > > configured
> > > > > > >
> > > > > > > > in
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > the
> > > > > > >
> > > > > > > > > >> > >>> > client?
> > > > > > >
> > > > > > > > > >> > >>> > > > The
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > standard
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > way is to do this through JAAS.
> > > However,
> > > > > we
> > > > > > > will
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > need
> > > > > > >
> > > > > > > > > >> > to
> > > > > > >
> > > > > > > > > >> > >>> > think
> > > > > > >
> > > > > > > > > >> > >>> > > > > > through
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > if
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > this is convenient in a shared
> > > > > environment.
> > > > > > > For
> > > > > > >
> > > > > > > > > >> > example,
> > > > > > >
> > > > > > > > > >> > >>> > when a
> > > > > > >
> > > > > > > > > >> > >>> > > > new
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > task
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > is
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > added to a Storm worker node, do
> we
> > > need
> > > > > to
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > dynamically
> > > > > > >
> > > > > > > > > >> > >>> add a
> > > > > > >
> > > > > > > > > >> > >>> > > new
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > section
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > in the JAAS file? It may be more
> > > > > convenient
> > > > > > if
> > > > > > >
> > > > > > > > we
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > can
> > > > > > >
> > > > > > > > > >> > >>> pass in
> > > > > > >
> > > > > > > > > >> > >>> > > the
> > > > > > >
> > > > > > > > > >> > >>> > > > > > token
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > through the config directly w/o
> > going
> > > > > > through
> > > > > > >
> > > > > > > > > JAAS.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > Are you or Parth still actively
> > > working
> > > > on
> > > > > > > this
> > > > > > >
> > > > > > > > > KIP?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > Thanks,
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > Jun
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM,
> Jun
> > > > Rao <
> > > > > > >
> > > > > > > > > >> > >>> jun@confluent.io>
> > > > > > >
> > > > > > > > > >> > >>> > > > wrote:
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > > Just to add on that list.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > > 2. It would be good to document
> > the
> > > > > format
> > > > > > > of
> > > > > > >
> > > > > > > > > the
> > > > > > >
> > > > > > > > > >> > data
> > > > > > >
> > > > > > > > > >> > >>> > stored
> > > > > > >
> > > > > > > > > >> > >>> > > > in
> > > > > > >
> > > > > > > > > >> > >>> > > > > > ZK.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > > 7. Earlier, there was a
> discussion
> > > on
> > > > > > > whether
> > > > > > >
> > > > > > > > > the
> > > > > > >
> > > > > > > > > >> > tokens
> > > > > > >
> > > > > > > > > >> > >>> > > should
> > > > > > >
> > > > > > > > > >> > >>> > > > > be
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > > propagated through ZK like
> > > > > > config/acl/quota,
> > > > > > >
> > > > > > > > or
> > > > > > >
> > > > > > > > > >> > through
> > > > > > >
> > > > > > > > > >> > >>> the
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > controller.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > > Currently, the controller is
> only
> > > > > designed
> > > > > > > for
> > > > > > >
> > > > > > > > > >> > >>> propagating
> > > > > > >
> > > > > > > > > >> > >>> > > > topic
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > metadata,
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > > but not other data.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > > 8. Should we use SCRAM to send
> the
> > > > token
> > > > > > >
> > > > > > > > instead
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > > of
> > > > > > >
> > > > > > > > > >> > >>> > > DIGEST-MD5
> > > > > > >
> > > > > > > > > >> > >>> > > > > > since
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > it's
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > > deprecated?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > > Also, the images in the wiki
> seem
> > > > > broken.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > > Thanks,
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > > Jun
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > > On Fri, Jun 10, 2016 at 10:02
> AM,
> > > Gwen
> > > > > > >
> > > > > > > > Shapira <
> > > > > > >
> > > > > > > > > >> > >>> > > > > gwen@confluent.io>
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > wrote:
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> From what I can see, remaining
> > > > > questions
> > > > > > > are:
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >>
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> 1. Who / how are tokens
> renewed?
> > By
> > > > > > > original
> > > > > > >
> > > > > > > > > >> > requester
> > > > > > >
> > > > > > > > > >> > >>> > only?
> > > > > > >
> > > > > > > > > >> > >>> > > > or
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > using
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> Kerberos auth only?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> 2. Are tokens stored on each
> > broker
> > > > or
> > > > > in
> > > > > > > ZK?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> 3. How are tokens invalidated /
> > > > > expired?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> 4. Which encryption algorithm
> is
> > > > used?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> 5. What is the impersonation
> > > proposal
> > > > > (it
> > > > > > >
> > > > > > > > > wasn't
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> in
> > > > > > >
> > > > > > > > > >> > the
> > > > > > >
> > > > > > > > > >> > >>> > KIP
> > > > > > >
> > > > > > > > > >> > >>> > > > but
> > > > > > >
> > > > > > > > > >> > >>> > > > > > was
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> discussed in this thread)?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> 6. Do we need new ACLs, if so -
> > for
> > > > > what
> > > > > > >
> > > > > > > > > actions?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >>
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> Gwen
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >>
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM,
> > > > Harsha
> > > > > <
> > > > > > >
> > > > > > > > > >> > >>> kafka@harsha.io>
> > > > > > >
> > > > > > > > > >> > >>> > > > wrote:
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> > Jun & Ismael,
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >
> > > > > Unfortunately
> > > > > > I
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> > couldn't
> > > > > > >
> > > > > > > > > >> > >>> attend
> > > > > > >
> > > > > > > > > >> > >>> > > the
> > > > > > >
> > > > > > > > > >> > >>> > > > > KIP
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > meeting
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >                          when
> > > > > > delegation
> > > > > > >
> > > > > > > > > tokens
> > > > > > >
> > > > > > > > > >> > >>> > discussed.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > Appreciate
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > if
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >                          you
> > can
> > > > > update
> > > > > > > the
> > > > > > >
> > > > > > > > > >> > thread if
> > > > > > >
> > > > > > > > > >> > >>> > you
> > > > > > >
> > > > > > > > > >> > >>> > > > have
> > > > > > >
> > > > > > > > > >> > >>> > > > > > any
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >
> > further
> > > > > > > questions.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> > Thanks,
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> > Harsha
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> > On Tue, May 24, 2016, at
> 11:32
> > > AM,
> > > > > > Liquan
> > > > > > >
> > > > > > > > Pei
> > > > > > >
> > > > > > > > > >> > wrote:
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> It seems that the links to
> > > images
> > > > in
> > > > > > the
> > > > > > >
> > > > > > > > KIP
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> are
> > > > > > >
> > > > > > > > > >> > >>> > broken.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >>
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> Liquan
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >>
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> On Tue, May 24, 2016 at 9:33
> > AM,
> > > > > parth
> > > > > > >
> > > > > > > > > >> > brahmbhatt <
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> brahmbhatt.parth@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >>
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> > 110. What does
> > > > > getDelegationTokenAs
> > > > > > >
> > > > > > > > mean?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> > In the current proposal we
> > > only
> > > > > > allow
> > > > > > > a
> > > > > > >
> > > > > > > > > user
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> > to
> > > > > > >
> > > > > > > > > >> > >>> get
> > > > > > >
> > > > > > > > > >> > >>> > > > > > delegation
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > token
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> for
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> > the identity that it
> > > > authenticated
> > > > > > as
> > > > > > >
> > > > > > > > > using
> > > > > > >
> > > > > > > > > >> > >>> another
> > > > > > >
> > > > > > > > > >> > >>> > > > > > mechanism,
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > i.e.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> A user
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> > that authenticate using a
> > > keytab
> > > > > for
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> > principal
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > user1@EXAMPLE.COM
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> will get
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens for that
> > > user
> > > > > > only.
> > > > > > > In
> > > > > > >
> > > > > > > > > >> > future I
> > > > > > >
> > > > > > > > > >> > >>> > think
> > > > > > >
> > > > > > > > > >> > >>> > > > we
> > > > > > >
> > > > > > > > > >> > >>> > > > > > will
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > have
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> to
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> > extend support such that
> we
> > > > allow
> > > > > > some
> > > > > > >
> > > > > > > > set
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> > of
> > > > > > >
> > > > > > > > > >> > >>> users (
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> >
> kafka-rest-user@EXAMPLE.COM
> > ,
> > > > > > >
> > > > > > > > > >> > >>> > storm-nimbus@EXAMPLE.COM)
> > > > > > >
> > > > > > > > > >> > >>> > > > to
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > acquire
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens on
> behalf
> > of
> > > > > other
> > > > > > >
> > > > > > > > users
> > > > > > >
> > > > > > > > > >> > whose
> > > > > > >
> > > > > > > > > >> > >>> > > identity
> > > > > > >
> > > > > > > > > >> > >>> > > > > > they
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > have
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> > verified independently.
> > Kafka
> > > > > > brokers
> > > > > > >
> > > > > > > > > will
> > > > > > >
> > > > > > > > > >> > have
> > > > > > >
> > > > > > > > > >> > >>> ACLs
> > > > > > >
> > > > > > > > > >> > >>> > > to
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > control
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> which
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> > users are allowed to
> > > impersonate
> > > > > > other
> > > > > > >
> > > > > > > > > users
> > > > > > >
> > > > > > > > > >> > and
> > > > > > >
> > > > > > > > > >> > >>> get
> > > > > > >
> > > > > > > > > >> > >>> > > > tokens
> > > > > > >
> > > > > > > > > >> > >>> > > > > > on
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> behalf of
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> > them. Overall
> Impersonation
> > > is a
> > > > > > whole
> > > > > > >
> > > > > > > > > >> > different
> > > > > > >
> > > > > > > > > >> > >>> > > problem
> > > > > > >
> > > > > > > > > >> > >>> > > > in
> > > > > > >
> > > > > > > > > >> > >>> > > > > > my
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> opinion and
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> > I think we can tackle it
> in
> > > > > separate
> > > > > > >
> > > > > > > > KIP.
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> > 111. What's the typical
> rate
> > > of
> > > > > > > getting
> > > > > > >
> > > > > > > > > and
> > > > > > >
> > > > > > > > > >> > >>> renewing
> > > > > > >
> > > > > > > > > >> > >>> > > > > > delegation
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> tokens?
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> > Typically this should be
> > very
> > > > very
> > > > > > > low,
> > > > > > >
> > > > > > > > 1
> > > > > > >
> > > > > > > > > >> > request
> > > > > > >
> > > > > > > > > >> > >>> per
> > > > > > >
> > > > > > > > > >> > >>> > > > > minute
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > is a
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> > relatively high estimate.
> > > > However
> > > > > it
> > > > > > >
> > > > > > > > > depends
> > > > > > >
> > > > > > > > > >> > >>> > > > > > > > > >> >> > on
> > > > > > >
> > > > > > > > > >> > >>> the
> > > > > > >
> > > > > > > > > >> > >>> > > token
> > > > > > >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Ismael Juma <is...@juma.me.uk>]

Hi Rajini,

I think it would definitely be valuable to have a KIP for impersonation.

Ismael

On Wed, Dec 14, 2016 at 4:03 AM, Rajini Sivaram <rs...@pivotal.io> wrote:

> It would clearly be very useful to enable clients to send requests on
> behalf of multiple users. A separate KIP makes sense, but it may be worth
> thinking through some of the implications now, especially if the main
> interest in delegation tokens comes from its potential to enable
> impersonation.
>
> I understand that delegation tokens are only expected to be used with TLS.
> But the choice of SASL/SCRAM for authentication must be based on a
> requirement to protect the tokenHmac - otherwise you could just use
> SASL/PLAIN. With SASL/SCRAM the tokenHmac is never propagated on-the-wire,
> only a salted-hashed version of it is used in the SASL authentication
> exchange. If impersonation is based on sending tokenHmac in requests, any
> benefit of using SCRAM is lost.
>
> An alternative may be to allow clients to authenticate multiple times using
> SASL and include one of its authenticated principals in each request
> (optionally). I haven't thought it through yet, obviously. But if the
> approach is of interest and no one is working on a KIP for impersonation at
> the moment, I am happy to write one. It may provide something for
> comparison at least.
>
> Thoughts?
>
>
> On Wed, Dec 14, 2016 at 9:53 AM, Manikumar <ma...@gmail.com>
> wrote:
>
> > That's a good idea. Authenticating every request with delegation token
> will
> > be useful for
> > impersonation use-cases. But as of now, we are thinking delegation token
> as
> > just another way
> > to authenticate the users. We haven't think through all the use cases
> > related to
> > impersonation or using delegation token for impersonation. We want to
> > handle impersonation
> > (KAFKA-3712) as part of separate KIP.
> >
> > Will that be Ok?
> >
> >
> > On Wed, Dec 14, 2016 at 8:09 AM, Gwen Shapira <gw...@confluent.io> wrote:
> >
> > > Thinking out loud here:
> > >
> > > It looks like authentication with a delegation token is going to be
> > > super-cheap, right? We just compare the token to a value in the broker
> > > cache?
> > >
> > > If I understood the KIP correctly, right now it suggests that
> > > authentication happens when establishing the client-broker connection
> (as
> > > normal for Kafka. But perhaps we want to consider authenticating every
> > > request with delegation token (if exists)?
> > >
> > > So a centralized app can create few producers, do the metadata request
> > and
> > > broker discovery with its own user auth, but then use delegation tokens
> > to
> > > allow performing produce/fetch requests as different users? Instead of
> > > having to re-connect for each impersonated user?
> > >
> > > This may over-complicate things quite a bit (basically adding extra
> > > information in every request), but maybe it will be useful for
> > > impersonation use-cases (which seem to drive much of the interest in
> this
> > > KIP)?
> > > Kafka Connect, NiFi and friends can probably use this to share clients
> > > between multiple jobs, tasks, etc.
> > >
> > > What do you think?
> > >
> > > Gwen
> > >
> > > On Tue, Dec 13, 2016 at 12:43 AM, Manikumar <manikumar.reddy@gmail.com
> >
> > > wrote:
> > >
> > > > Ashish,
> > > >
> > > > Thank you for reviewing the KIP.  Please see the replies inline.
> > > >
> > > >
> > > > > 1. How to disable delegation token authentication?
> > > > >
> > > > > This can be achieved in various ways, however I think reusing
> > > delegation
> > > > > token secret config for this makes sense here. Avoids creating yet
> > > > another
> > > > > config and forces delegation token users to consciously set the
> > secret.
> > > > If
> > > > > the secret is not set or set to empty string, brokers should turn
> off
> > > > > delegation token support. This will however require a new error
> code
> > to
> > > > > indicate delegation token support is turned off on broker.
> > > > >
> > > >
> > > >   Thanks for the suggestion. Option to turnoff delegation token
> > > > authentication will be useful.
> > > >   I'll update the KIP.
> > > >
> > > >
> > > > >
> > > > > 2. ACLs on delegation token?
> > > > >
> > > > > Do we need to have ACLs defined for tokens? I do not think it buys
> us
> > > > > anything, as delegation token can be treated as impersonation of
> the
> > > > owner.
> > > > > Any thing the owner has permission to do, delegation tokens should
> be
> > > > > allowed to do as well. If so, we probably won't need to return
> > > > > authorization exception error code while creating delegation token.
> > It
> > > > > however would make sense to check renew and expire requests are
> > coming
> > > > from
> > > > > owner or renewers of the token, but that does not require explicit
> > > acls.
> > > > >
> > > >
> > > >
> > > > Yes, We agreed to not have new acl on who can request delegation
> token.
> > > >  I'll update the KIP.
> > > >
> > > >
> > > > >
> > > > > 3. How to restrict max life time of a token?
> > > > >
> > > > > Admins might want to restrict max life time of tokens created on a
> > > > cluster,
> > > > > and this can very from cluster to cluster based on use-cases. This
> > > might
> > > > > warrant a separate broker config.
> > > > >
> > > > >
> > > > Currently we  have "delegation.token.max.lifetime.sec" server config
> > > > property
> > > > May be we can take min(User supplied MaxTime, Server MaxTime) as max
> > life
> > > > time.
> > > > I am open to add new config property.
> > > >
> > > > Few more comments based on recent KIP update.
> > > > >
> > > > > 1. Do we need a separate {{InvalidateTokenRequest}}? Can't we use
> > > > > {{ExpireTokenRequest}} with with expiryDate set to anything before
> > > > current
> > > > > date?
> > > > >
> > > >
> > > > makes sense. we don't need special request to cancel the token. We
> can
> > > use
> > > > ExpireTokenRequest.
> > > > I'll update the KIP.
> > > >
> > > >
> > > > > 2. Can we change time field names to indicate their unit is
> > > milliseconds,
> > > > > like, IssueDateMs, ExpiryDateMs, etc.?
> > > > >
> > > > >
> > > >   Done.
> > > >
> > > >
> > > > > 3. Can we allow users to renew a token for a specified amount of
> > time?
> > > In
> > > > > current version of KIP, renew request does not take time as a
> param,
> > > not
> > > > > sure what is expiry time set to after renewal.
> > > > >
> > > > >
> > > >  Yes, we need to specify renew period.  I'll update the KIP.
> > > >
> > > >
> > > > Thanks,
> > > > Mankumar
> > > >
> > > >
> > > >
> > > > >
> > > > > On Mon, Dec 12, 2016 at 9:08 AM Manikumar <
> manikumar.reddy@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > >
> > > > > >
> > > > > > I would like to reinitiate the discussion on Delegation token
> > support
> > > > for
> > > > > >
> > > > > > Kafka.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Brief summary of the past discussion:
> > > > > >
> > > > > >
> > > > > >
> > > > > > 1) Broker stores delegation tokens in zookeeper.  All brokers
> will
> > > > have a
> > > > > >
> > > > > > cache backed by
> > > > > >
> > > > > >    zookeeper so they will all get notified whenever a new token
> is
> > > > > >
> > > > > > generated and they will
> > > > > >
> > > > > >    update their local cache whenever token state changes.
> > > > > >
> > > > > > 2) The current proposal does not support rotation of secret
> > > > > >
> > > > > > 3) Only allow the renewal by users that authenticated using *non*
> > > > > >
> > > > > > delegation token mechanism
> > > > > >
> > > > > > 4) KIP-84 proposes to support  SASL SCRAM mechanisms. Kafka
> clients
> > > can
> > > > > >
> > > > > > authenticate using
> > > > > >
> > > > > >    SCRAM-SHA-256, providing the delegation token HMAC as
> password.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Updated the KIP with the following:
> > > > > >
> > > > > > 1. Protocol and Config changes
> > > > > >
> > > > > > 2. format of the data stored in ZK.
> > > > > >
> > > > > > 3. Changes to Java Clients/Usage of SASL SCRAM mechanism
> > > > > >
> > > > > >
> > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > >
> > > > > > 48+Delegation+token+support+for+Kafka
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Jun, Ashish, Gwen,
> > > > > >
> > > > > >
> > > > > >
> > > > > > Pl review the updated KIP.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > Manikumar
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 29, 2016 at 9:56 PM, Ashish Singh <
> asingh@cloudera.com
> > >
> > > > > wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > > Harsha/ Gwen,
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > How do we proceed here? I am willing to help out with here.
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > On Fri, Sep 23, 2016 at 11:41 AM, Gwen Shapira <
> > gwen@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > > Is it updated? are all concerns addressed? do you want to
> > start a
> > > > > vote?
> > > > > >
> > > > > > > >
> > > > > >
> > > > > > > > Sorry for being pushy, I do appreciate that we are all
> > volunteers
> > > > and
> > > > > >
> > > > > > > > finding time is difficult. This feature is important for
> > anything
> > > > > that
> > > > > >
> > > > > > > > integrates with Kafka (stream processors, Flume, NiFi, etc)
> > and I
> > > > > >
> > > > > > > > don't want to see this getting stuck because we lack
> > coordination
> > > > > >
> > > > > > > > within the community.
> > > > > >
> > > > > > > >
> > > > > >
> > > > > > > > On Thu, Sep 15, 2016 at 6:39 PM, Harsha Chintalapani <
> > > > > kafka@harsha.io>
> > > > > >
> > > > > > > > wrote:
> > > > > >
> > > > > > > > > The only pending update for the KIP is to write up the
> > protocol
> > > > > > changes
> > > > > >
> > > > > > > > like
> > > > > >
> > > > > > > > > we've it KIP-4. I'll update the wiki.
> > > > > >
> > > > > > > > >
> > > > > >
> > > > > > > > >
> > > > > >
> > > > > > > > > On Thu, Sep 15, 2016 at 4:27 PM Ashish Singh <
> > > > asingh@cloudera.com>
> > > > > >
> > > > > > > > wrote:
> > > > > >
> > > > > > > > >>
> > > > > >
> > > > > > > > >> I think we decided to not support secret rotation, I guess
> > > this
> > > > > can
> > > > > > be
> > > > > >
> > > > > > > > >> stated clearly on the KIP. Also, more details on how
> clients
> > > > will
> > > > > >
> > > > > > > > perform
> > > > > >
> > > > > > > > >> token distribution and how CLI will look like will be
> > helpful.
> > > > > >
> > > > > > > > >>
> > > > > >
> > > > > > > > >> On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <
> > > > gwen@confluent.io>
> > > > > >
> > > > > > > > wrote:
> > > > > >
> > > > > > > > >>
> > > > > >
> > > > > > > > >> > Hi Guys,
> > > > > >
> > > > > > > > >> >
> > > > > >
> > > > > > > > >> > This discussion was dead for a while. Are there still
> > > > > contentious
> > > > > >
> > > > > > > > >> > points? If not, why are there no votes?
> > > > > >
> > > > > > > > >> >
> > > > > >
> > > > > > > > >> > On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <
> > jun@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > > > > >> > > Ashish,
> > > > > >
> > > > > > > > >> > >
> > > > > >
> > > > > > > > >> > > Yes, I will send out a KIP invite for next week to
> > discuss
> > > > > > KIP-48
> > > > > >
> > > > > > > > and
> > > > > >
> > > > > > > > >> > other
> > > > > >
> > > > > > > > >> > > remaining KIPs.
> > > > > >
> > > > > > > > >> > >
> > > > > >
> > > > > > > > >> > > Thanks,
> > > > > >
> > > > > > > > >> > >
> > > > > >
> > > > > > > > >> > > Jun
> > > > > >
> > > > > > > > >> > >
> > > > > >
> > > > > > > > >> > > On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh <
> > > > > >
> > > > > > > asingh@cloudera.com>
> > > > > >
> > > > > > > > >> > wrote:
> > > > > >
> > > > > > > > >> > >
> > > > > >
> > > > > > > > >> > >> Thanks Harsha!
> > > > > >
> > > > > > > > >> > >>
> > > > > >
> > > > > > > > >> > >> Jun, can we add KIP-48 to next KIP hangout's agenda.
> > > Also,
> > > > we
> > > > > > did
> > > > > >
> > > > > > > > not
> > > > > >
> > > > > > > > >> > >> actually make a call on when we should have next KIP
> > > call.
> > > > As
> > > > > >
> > > > > > > there
> > > > > >
> > > > > > > > >> > >> are
> > > > > >
> > > > > > > > >> > a
> > > > > >
> > > > > > > > >> > >> few outstanding KIPs that could not be discussed this
> > > week,
> > > > > can
> > > > > >
> > > > > > > we
> > > > > >
> > > > > > > > >> > >> have
> > > > > >
> > > > > > > > >> > a
> > > > > >
> > > > > > > > >> > >> KIP hangout call next week?
> > > > > >
> > > > > > > > >> > >>
> > > > > >
> > > > > > > > >> > >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha Chintalapani
> > > > > >
> > > > > > > > >> > >> <ka...@harsha.io>
> > > > > >
> > > > > > > > >> > >> wrote:
> > > > > >
> > > > > > > > >> > >>
> > > > > >
> > > > > > > > >> > >>> Ashish,
> > > > > >
> > > > > > > > >> > >>>         Yes we are working on it. Lets discuss in
> the
> > > next
> > > > > KIP
> > > > > >
> > > > > > > > >> > >>> meeting.
> > > > > >
> > > > > > > > >> > >>> I'll join.
> > > > > >
> > > > > > > > >> > >>> -Harsha
> > > > > >
> > > > > > > > >> > >>>
> > > > > >
> > > > > > > > >> > >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh <
> > > > > >
> > > > > > > > asingh@cloudera.com>
> > > > > >
> > > > > > > > >> > >>> wrote:
> > > > > >
> > > > > > > > >> > >>>
> > > > > >
> > > > > > > > >> > >>> > Hello Harsha,
> > > > > >
> > > > > > > > >> > >>> >
> > > > > >
> > > > > > > > >> > >>> > Are you still working on this? Wondering if we can
> > > > discuss
> > > > > >
> > > > > > > this
> > > > > >
> > > > > > > > in
> > > > > >
> > > > > > > > >> > next
> > > > > >
> > > > > > > > >> > >>> KIP
> > > > > >
> > > > > > > > >> > >>> > meeting, if you can join.
> > > > > >
> > > > > > > > >> > >>> >
> > > > > >
> > > > > > > > >> > >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha
> > Chintalapani <
> > > > > >
> > > > > > > > >> > kafka@harsha.io>
> > > > > >
> > > > > > > > >> > >>> > wrote:
> > > > > >
> > > > > > > > >> > >>> >
> > > > > >
> > > > > > > > >> > >>> > > Hi Grant,
> > > > > >
> > > > > > > > >> > >>> > >           We are working on it. Will add the
> > details
> > > > to
> > > > > > KIP
> > > > > >
> > > > > > > > >> > >>> > > about
> > > > > >
> > > > > > > > >> > the
> > > > > >
> > > > > > > > >> > >>> > > request protocol.
> > > > > >
> > > > > > > > >> > >>> > >
> > > > > >
> > > > > > > > >> > >>> > > Thanks,
> > > > > >
> > > > > > > > >> > >>> > > Harsha
> > > > > >
> > > > > > > > >> > >>> > >
> > > > > >
> > > > > > > > >> > >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant Henke
> > > > > >
> > > > > > > > >> > >>> > > <gh...@cloudera.com>
> > > > > >
> > > > > > > > >> > >>> wrote:
> > > > > >
> > > > > > > > >> > >>> > >
> > > > > >
> > > > > > > > >> > >>> > > > Hi Parth,
> > > > > >
> > > > > > > > >> > >>> > > >
> > > > > >
> > > > > > > > >> > >>> > > > Are you still working on this? If you need any
> > > help
> > > > > > please
> > > > > >
> > > > > > > > >> > >>> > > > don't
> > > > > >
> > > > > > > > >> > >>> > hesitate
> > > > > >
> > > > > > > > >> > >>> > > > to ask.
> > > > > >
> > > > > > > > >> > >>> > > >
> > > > > >
> > > > > > > > >> > >>> > > > Thanks,
> > > > > >
> > > > > > > > >> > >>> > > > Grant
> > > > > >
> > > > > > > > >> > >>> > > >
> > > > > >
> > > > > > > > >> > >>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <
> > > > > >
> > > > > > > jun@confluent.io>
> > > > > >
> > > > > > > > >> > wrote:
> > > > > >
> > > > > > > > >> > >>> > > >
> > > > > >
> > > > > > > > >> > >>> > > > > Parth,
> > > > > >
> > > > > > > > >> > >>> > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > Thanks for the reply.
> > > > > >
> > > > > > > > >> > >>> > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > It makes sense to only allow the renewal by
> > > users
> > > > > that
> > > > > >
> > > > > > > > >> > >>> authenticated
> > > > > >
> > > > > > > > >> > >>> > > > using
> > > > > >
> > > > > > > > >> > >>> > > > > *non* delegation token mechanism. Then,
> should
> > > we
> > > > > make
> > > > > >
> > > > > > > the
> > > > > >
> > > > > > > > >> > >>> renewal a
> > > > > >
> > > > > > > > >> > >>> > > > list?
> > > > > >
> > > > > > > > >> > >>> > > > > For example, in the case of rest proxy, it
> > will
> > > be
> > > > > >
> > > > > > > useful
> > > > > >
> > > > > > > > >> > >>> > > > > for
> > > > > >
> > > > > > > > >> > >>> every
> > > > > >
> > > > > > > > >> > >>> > > > > instance of rest proxy to be able to renew
> the
> > > > > tokens.
> > > > > >
> > > > > > > > >> > >>> > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > It would be clearer if we can document the
> > > request
> > > > > >
> > > > > > > > protocol
> > > > > >
> > > > > > > > >> > like
> > > > > >
> > > > > > > > >> > >>> > > > >
> > > > > >
> > > > > > > > >> > >>> > > > >
> > > > > >
> > > > > > > > >> > >>> > > > https://cwiki.apache.org/confl
> > > > > uence/display/KAFKA/KIP-
> > > > > >
> > > > > > > > >> > >>> > >
> > > > > >
> > > > > > > > >> > >>> > > 4+-+Command+line+and+
> centralized+administrative+
> > > > > >
> > > > > > > > operations#KIP-4-
> > > > > >
> > > > > > > > >> > >>> > > Commandlineandcentralizedadmin
> > istrativeoperations-
> > > > > >
> > > > > > > > >> > >>> > > CreateTopicsRequest(KAFKA-
> > > > 2945):(VotedandPlannedforin0.
> > > > > >
> > > > > > > > 10.1.0)
> > > > > >
> > > > > > > > >> > >>> > > > > .
> > > > > >
> > > > > > > > >> > >>> > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > It would also be useful to document the
> client
> > > > APIs.
> > > > > >
> > > > > > > > >> > >>> > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > Thanks,
> > > > > >
> > > > > > > > >> > >>> > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > Jun
> > > > > >
> > > > > > > > >> > >>> > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM, parth
> > > brahmbhatt
> > > > <
> > > > > >
> > > > > > > > >> > >>> > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > > >
> > > > > > > > >> > >>> > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > Hi,
> > > > > >
> > > > > > > > >> > >>> > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > I am suggesting that we will only allow
> the
> > > > > renewal
> > > > > > by
> > > > > >
> > > > > > > > >> > >>> > > > > > users
> > > > > >
> > > > > > > > >> > >>> that
> > > > > >
> > > > > > > > >> > >>> > > > > > authenticated using *non* delegation token
> > > > > > mechanism.
> > > > > >
> > > > > > > > For
> > > > > >
> > > > > > > > >> > >>> example,
> > > > > >
> > > > > > > > >> > >>> > If
> > > > > >
> > > > > > > > >> > >>> > > > > user
> > > > > >
> > > > > > > > >> > >>> > > > > > Alice authenticated using kerberos and
> > > requested
> > > > > >
> > > > > > > > >> > >>> > > > > > delegation
> > > > > >
> > > > > > > > >> > >>> tokens,
> > > > > >
> > > > > > > > >> > >>> > > > only
> > > > > >
> > > > > > > > >> > >>> > > > > > user Alice authenticated via non
> delegation
> > > > token
> > > > > >
> > > > > > > > >> > >>> > > > > > mechanism
> > > > > >
> > > > > > > > >> > can
> > > > > >
> > > > > > > > >> > >>> > > renew.
> > > > > >
> > > > > > > > >> > >>> > > > > > Clients that have  access to delegation
> > tokens
> > > > can
> > > > > > not
> > > > > >
> > > > > > > > >> > >>> > > > > > issue
> > > > > >
> > > > > > > > >> > >>> > renewal
> > > > > >
> > > > > > > > >> > >>> > > > > > request for renewing their own token and
> > this
> > > is
> > > > > >
> > > > > > > > primarily
> > > > > >
> > > > > > > > >> > >>> > important
> > > > > >
> > > > > > > > >> > >>> > > to
> > > > > >
> > > > > > > > >> > >>> > > > > > reduce the time window for which a
> > compromised
> > > > > token
> > > > > >
> > > > > > > > will
> > > > > >
> > > > > > > > >> > >>> > > > > > be
> > > > > >
> > > > > > > > >> > >>> valid.
> > > > > >
> > > > > > > > >> > >>> > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > To clarify, Yes any authenticated user can
> > > > request
> > > > > >
> > > > > > > > >> > >>> > > > > > delegation
> > > > > >
> > > > > > > > >> > >>> > tokens
> > > > > >
> > > > > > > > >> > >>> > > > but
> > > > > >
> > > > > > > > >> > >>> > > > > > even here I would recommend to avoid
> > creating
> > > a
> > > > > > chain
> > > > > >
> > > > > > > > >> > >>> > > > > > where a
> > > > > >
> > > > > > > > >> > >>> > client
> > > > > >
> > > > > > > > >> > >>> > > > > > authenticated via delegation token request
> > for
> > > > > more
> > > > > >
> > > > > > > > >> > delegation
> > > > > >
> > > > > > > > >> > >>> > > tokens.
> > > > > >
> > > > > > > > >> > >>> > > > > > Basically anyone can request delegation
> > token,
> > > > as
> > > > > > long
> > > > > >
> > > > > > > > as
> > > > > >
> > > > > > > > >> > they
> > > > > >
> > > > > > > > >> > >>> > > > > authenticate
> > > > > >
> > > > > > > > >> > >>> > > > > > via a non delegation token mechanism.
> > > > > >
> > > > > > > > >> > >>> > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > Aren't classes listed here
> > > > > >
> > > > > > > > >> > >>> > > > > > <
> > > > > >
> > > > > > > > >> > >>> > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > >
> > > > > >
> > > > > > > > >> > >>> > > > https://cwiki.apache.org/confl
> > > > > uence/display/KAFKA/KIP-
> > > > > >
> > > > > > > > >> > >>> > > 48+Delegation+token+support+fo
> > > > > >
> > > > > > > r+Kafka#KIP-48Delegationtokens
> > > > > >
> > > > > > > > >> > >>> upportforKaf
> > > > > >
> > > > > > > > >> > >>> > > ka-PublicInterfaces
> > > > > >
> > > > > > > > >> > >>> > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > sufficient?
> > > > > >
> > > > > > > > >> > >>> > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > Thanks
> > > > > >
> > > > > > > > >> > >>> > > > > > Parth
> > > > > >
> > > > > > > > >> > >>> > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao
> > > > > >
> > > > > > > > >> > >>> > > > > > <ju...@confluent.io>
> > > > > >
> > > > > > > > >> > >>> wrote:
> > > > > >
> > > > > > > > >> > >>> > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > Parth,
> > > > > >
> > > > > > > > >> > >>> > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > Thanks for the reply. A couple of
> comments
> > > > > inline
> > > > > >
> > > > > > > > below.
> > > > > >
> > > > > > > > >> > >>> > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > On Tue, Jun 21, 2016 at 10:36 AM, parth
> > > > > > brahmbhatt <
> > > > > >
> > > > > > > > >> > >>> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > > >
> > > > > > > > >> > >>> > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > 1. Who / how are tokens renewed? By
> > > original
> > > > > >
> > > > > > > > requester
> > > > > >
> > > > > > > > >> > >>> only? or
> > > > > >
> > > > > > > > >> > >>> > > > using
> > > > > >
> > > > > > > > >> > >>> > > > > > > > Kerberos
> > > > > >
> > > > > > > > >> > >>> > > > > > > > auth only?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > My recommendation is to do this only
> > using
> > > > > >
> > > > > > > Kerberos
> > > > > >
> > > > > > > > >> > >>> > > > > > > > auth
> > > > > >
> > > > > > > > >> > and
> > > > > >
> > > > > > > > >> > >>> > only
> > > > > >
> > > > > > > > >> > >>> > > > > threw
> > > > > >
> > > > > > > > >> > >>> > > > > > > the
> > > > > >
> > > > > > > > >> > >>> > > > > > > > renewer specified during the
> acquisition
> > > > > > request.
> > > > > >
> > > > > > > > >> > >>> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > Hmm, not sure that I follow this. Are
> you
> > > > saying
> > > > > >
> > > > > > > that
> > > > > >
> > > > > > > > >> > >>> > > > > > > any
> > > > > >
> > > > > > > > >> > >>> client
> > > > > >
> > > > > > > > >> > >>> > > > > > > authenticated with the delegation token
> > can
> > > > > renew,
> > > > > >
> > > > > > > > i.e.
> > > > > >
> > > > > > > > >> > there
> > > > > >
> > > > > > > > >> > >>> is
> > > > > >
> > > > > > > > >> > >>> > no
> > > > > >
> > > > > > > > >> > >>> > > > > > renewer
> > > > > >
> > > > > > > > >> > >>> > > > > > > needed?
> > > > > >
> > > > > > > > >> > >>> > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > Also, just to be clear, any
> authenticated
> > > > client
> > > > > >
> > > > > > > > (either
> > > > > >
> > > > > > > > >> > >>> through
> > > > > >
> > > > > > > > >> > >>> > > SASL
> > > > > >
> > > > > > > > >> > >>> > > > > or
> > > > > >
> > > > > > > > >> > >>> > > > > > > SSL) can request a delegation token for
> > the
> > > > > >
> > > > > > > > >> > >>> > > > > > > authenticated
> > > > > >
> > > > > > > > >> > >>> user,
> > > > > >
> > > > > > > > >> > >>> > > > right?
> > > > > >
> > > > > > > > >> > >>> > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > 2. Are tokens stored on each broker or
> > in
> > > > ZK?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > My recommendation is still to store in
> > ZK
> > > or
> > > > > not
> > > > > >
> > > > > > > > store
> > > > > >
> > > > > > > > >> > them
> > > > > >
> > > > > > > > >> > >>> at
> > > > > >
> > > > > > > > >> > >>> > > all.
> > > > > >
> > > > > > > > >> > >>> > > > > The
> > > > > >
> > > > > > > > >> > >>> > > > > > > > whole controller based distribution is
> > too
> > > > > much
> > > > > >
> > > > > > > > >> > >>> > > > > > > > overhead
> > > > > >
> > > > > > > > >> > >>> with
> > > > > >
> > > > > > > > >> > >>> > not
> > > > > >
> > > > > > > > >> > >>> > > > > much
> > > > > >
> > > > > > > > >> > >>> > > > > > to
> > > > > >
> > > > > > > > >> > >>> > > > > > > > achieve.
> > > > > >
> > > > > > > > >> > >>> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > 3. How are tokens invalidated /
> expired?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > Either by expiration time out or
> through
> > > an
> > > > > >
> > > > > > > explicit
> > > > > >
> > > > > > > > >> > >>> request to
> > > > > >
> > > > > > > > >> > >>> > > > > > > invalidate.
> > > > > >
> > > > > > > > >> > >>> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > 4. Which encryption algorithm is used?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > SCRAM
> > > > > >
> > > > > > > > >> > >>> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > 5. What is the impersonation proposal
> > (it
> > > > > wasn't
> > > > > >
> > > > > > > in
> > > > > >
> > > > > > > > >> > >>> > > > > > > > the
> > > > > >
> > > > > > > > >> > KIP
> > > > > >
> > > > > > > > >> > >>> but
> > > > > >
> > > > > > > > >> > >>> > > was
> > > > > >
> > > > > > > > >> > >>> > > > > > > > discussed
> > > > > >
> > > > > > > > >> > >>> > > > > > > > in this thread)?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > There is no imperonation proposal. I
> > tried
> > > > and
> > > > > >
> > > > > > > > >> > >>> > > > > > > > explained
> > > > > >
> > > > > > > > >> > how
> > > > > >
> > > > > > > > >> > >>> > its
> > > > > >
> > > > > > > > >> > >>> > > a
> > > > > >
> > > > > > > > >> > >>> > > > > > > > different problem and why its not
> really
> > > > > > necessary
> > > > > >
> > > > > > > > to
> > > > > >
> > > > > > > > >> > >>> discuss
> > > > > >
> > > > > > > > >> > >>> > > that
> > > > > >
> > > > > > > > >> > >>> > > > as
> > > > > >
> > > > > > > > >> > >>> > > > > > > part
> > > > > >
> > > > > > > > >> > >>> > > > > > > > of this KIP.  This KIP will not
> support
> > > any
> > > > > >
> > > > > > > > >> > impersonation,
> > > > > >
> > > > > > > > >> > >>> it
> > > > > >
> > > > > > > > >> > >>> > > will
> > > > > >
> > > > > > > > >> > >>> > > > > just
> > > > > >
> > > > > > > > >> > >>> > > > > > > be
> > > > > >
> > > > > > > > >> > >>> > > > > > > > another way to authenticate.
> > > > > >
> > > > > > > > >> > >>> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > 6. Do we need new ACLs, if so - for
> what
> > > > > > actions?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > We do not need new ACLs.
> > > > > >
> > > > > > > > >> > >>> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > Could we document the format of the new
> > > > > >
> > > > > > > > request/response
> > > > > >
> > > > > > > > >> > and
> > > > > >
> > > > > > > > >> > >>> > their
> > > > > >
> > > > > > > > >> > >>> > > > > > > associated Resource and Operation for
> ACL?
> > > > > >
> > > > > > > > >> > >>> > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > 7. How would the delegation token be
> > > > > configured
> > > > > > in
> > > > > >
> > > > > > > > the
> > > > > >
> > > > > > > > >> > >>> client?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > Should be through config. I wasn't
> > > planning
> > > > on
> > > > > >
> > > > > > > > >> > >>> > > > > > > > supporting
> > > > > >
> > > > > > > > >> > >>> JAAS
> > > > > >
> > > > > > > > >> > >>> > > for
> > > > > >
> > > > > > > > >> > >>> > > > > > > tokens.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > I don't believe hadoop does this
> either.
> > > > > >
> > > > > > > > >> > >>> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > Thanks
> > > > > >
> > > > > > > > >> > >>> > > > > > > > Parth
> > > > > >
> > > > > > > > >> > >>> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun
> > Rao <
> > > > > >
> > > > > > > > >> > jun@confluent.io>
> > > > > >
> > > > > > > > >> > >>> > > wrote:
> > > > > >
> > > > > > > > >> > >>> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > Harsha,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > Another question.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > 9. How would the delegation token be
> > > > > > configured
> > > > > >
> > > > > > > in
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > the
> > > > > >
> > > > > > > > >> > >>> > client?
> > > > > >
> > > > > > > > >> > >>> > > > The
> > > > > >
> > > > > > > > >> > >>> > > > > > > > standard
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > way is to do this through JAAS.
> > However,
> > > > we
> > > > > > will
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > need
> > > > > >
> > > > > > > > >> > to
> > > > > >
> > > > > > > > >> > >>> > think
> > > > > >
> > > > > > > > >> > >>> > > > > > through
> > > > > >
> > > > > > > > >> > >>> > > > > > > if
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > this is convenient in a shared
> > > > environment.
> > > > > > For
> > > > > >
> > > > > > > > >> > example,
> > > > > >
> > > > > > > > >> > >>> > when a
> > > > > >
> > > > > > > > >> > >>> > > > new
> > > > > >
> > > > > > > > >> > >>> > > > > > > task
> > > > > >
> > > > > > > > >> > >>> > > > > > > > is
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > added to a Storm worker node, do we
> > need
> > > > to
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > dynamically
> > > > > >
> > > > > > > > >> > >>> add a
> > > > > >
> > > > > > > > >> > >>> > > new
> > > > > >
> > > > > > > > >> > >>> > > > > > > section
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > in the JAAS file? It may be more
> > > > convenient
> > > > > if
> > > > > >
> > > > > > > we
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > can
> > > > > >
> > > > > > > > >> > >>> pass in
> > > > > >
> > > > > > > > >> > >>> > > the
> > > > > >
> > > > > > > > >> > >>> > > > > > token
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > through the config directly w/o
> going
> > > > > through
> > > > > >
> > > > > > > > JAAS.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > Are you or Parth still actively
> > working
> > > on
> > > > > > this
> > > > > >
> > > > > > > > KIP?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > Thanks,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > Jun
> > > > > >
> > > > > > > > >> > >>> > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun
> > > Rao <
> > > > > >
> > > > > > > > >> > >>> jun@confluent.io>
> > > > > >
> > > > > > > > >> > >>> > > > wrote:
> > > > > >
> > > > > > > > >> > >>> > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > > Just to add on that list.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > > 2. It would be good to document
> the
> > > > format
> > > > > > of
> > > > > >
> > > > > > > > the
> > > > > >
> > > > > > > > >> > data
> > > > > >
> > > > > > > > >> > >>> > stored
> > > > > >
> > > > > > > > >> > >>> > > > in
> > > > > >
> > > > > > > > >> > >>> > > > > > ZK.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > > 7. Earlier, there was a discussion
> > on
> > > > > > whether
> > > > > >
> > > > > > > > the
> > > > > >
> > > > > > > > >> > tokens
> > > > > >
> > > > > > > > >> > >>> > > should
> > > > > >
> > > > > > > > >> > >>> > > > > be
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > > propagated through ZK like
> > > > > config/acl/quota,
> > > > > >
> > > > > > > or
> > > > > >
> > > > > > > > >> > through
> > > > > >
> > > > > > > > >> > >>> the
> > > > > >
> > > > > > > > >> > >>> > > > > > > controller.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > > Currently, the controller is only
> > > > designed
> > > > > > for
> > > > > >
> > > > > > > > >> > >>> propagating
> > > > > >
> > > > > > > > >> > >>> > > > topic
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > metadata,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > > but not other data.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > > 8. Should we use SCRAM to send the
> > > token
> > > > > >
> > > > > > > instead
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > > of
> > > > > >
> > > > > > > > >> > >>> > > DIGEST-MD5
> > > > > >
> > > > > > > > >> > >>> > > > > > since
> > > > > >
> > > > > > > > >> > >>> > > > > > > > it's
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > > deprecated?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > > Also, the images in the wiki seem
> > > > broken.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > > Thanks,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > > Jun
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > > On Fri, Jun 10, 2016 at 10:02 AM,
> > Gwen
> > > > > >
> > > > > > > Shapira <
> > > > > >
> > > > > > > > >> > >>> > > > > gwen@confluent.io>
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > wrote:
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> From what I can see, remaining
> > > > questions
> > > > > > are:
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >>
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> 1. Who / how are tokens renewed?
> By
> > > > > > original
> > > > > >
> > > > > > > > >> > requester
> > > > > >
> > > > > > > > >> > >>> > only?
> > > > > >
> > > > > > > > >> > >>> > > > or
> > > > > >
> > > > > > > > >> > >>> > > > > > > using
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> Kerberos auth only?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> 2. Are tokens stored on each
> broker
> > > or
> > > > in
> > > > > > ZK?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> 3. How are tokens invalidated /
> > > > expired?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> 4. Which encryption algorithm is
> > > used?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> 5. What is the impersonation
> > proposal
> > > > (it
> > > > > >
> > > > > > > > wasn't
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> in
> > > > > >
> > > > > > > > >> > the
> > > > > >
> > > > > > > > >> > >>> > KIP
> > > > > >
> > > > > > > > >> > >>> > > > but
> > > > > >
> > > > > > > > >> > >>> > > > > > was
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> discussed in this thread)?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> 6. Do we need new ACLs, if so -
> for
> > > > what
> > > > > >
> > > > > > > > actions?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >>
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> Gwen
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >>
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM,
> > > Harsha
> > > > <
> > > > > >
> > > > > > > > >> > >>> kafka@harsha.io>
> > > > > >
> > > > > > > > >> > >>> > > > wrote:
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> > Jun & Ismael,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >
> > > > Unfortunately
> > > > > I
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> > couldn't
> > > > > >
> > > > > > > > >> > >>> attend
> > > > > >
> > > > > > > > >> > >>> > > the
> > > > > >
> > > > > > > > >> > >>> > > > > KIP
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > meeting
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >                          when
> > > > > delegation
> > > > > >
> > > > > > > > tokens
> > > > > >
> > > > > > > > >> > >>> > discussed.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > Appreciate
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > if
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >                          you
> can
> > > > update
> > > > > > the
> > > > > >
> > > > > > > > >> > thread if
> > > > > >
> > > > > > > > >> > >>> > you
> > > > > >
> > > > > > > > >> > >>> > > > have
> > > > > >
> > > > > > > > >> > >>> > > > > > any
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >
> further
> > > > > > questions.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> > Thanks,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> > Harsha
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> > On Tue, May 24, 2016, at 11:32
> > AM,
> > > > > Liquan
> > > > > >
> > > > > > > Pei
> > > > > >
> > > > > > > > >> > wrote:
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> It seems that the links to
> > images
> > > in
> > > > > the
> > > > > >
> > > > > > > KIP
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> are
> > > > > >
> > > > > > > > >> > >>> > broken.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >>
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> Liquan
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >>
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> On Tue, May 24, 2016 at 9:33
> AM,
> > > > parth
> > > > > >
> > > > > > > > >> > brahmbhatt <
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> brahmbhatt.parth@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >>
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > 110. What does
> > > > getDelegationTokenAs
> > > > > >
> > > > > > > mean?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > In the current proposal we
> > only
> > > > > allow
> > > > > > a
> > > > > >
> > > > > > > > user
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > to
> > > > > >
> > > > > > > > >> > >>> get
> > > > > >
> > > > > > > > >> > >>> > > > > > delegation
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > token
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> for
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > the identity that it
> > > authenticated
> > > > > as
> > > > > >
> > > > > > > > using
> > > > > >
> > > > > > > > >> > >>> another
> > > > > >
> > > > > > > > >> > >>> > > > > > mechanism,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > i.e.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> A user
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > that authenticate using a
> > keytab
> > > > for
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > principal
> > > > > >
> > > > > > > > >> > >>> > > > > > > user1@EXAMPLE.COM
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> will get
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens for that
> > user
> > > > > only.
> > > > > > In
> > > > > >
> > > > > > > > >> > future I
> > > > > >
> > > > > > > > >> > >>> > think
> > > > > >
> > > > > > > > >> > >>> > > > we
> > > > > >
> > > > > > > > >> > >>> > > > > > will
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > have
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> to
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > extend support such that we
> > > allow
> > > > > some
> > > > > >
> > > > > > > set
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > of
> > > > > >
> > > > > > > > >> > >>> users (
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > kafka-rest-user@EXAMPLE.COM
> ,
> > > > > >
> > > > > > > > >> > >>> > storm-nimbus@EXAMPLE.COM)
> > > > > >
> > > > > > > > >> > >>> > > > to
> > > > > >
> > > > > > > > >> > >>> > > > > > > > acquire
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens on behalf
> of
> > > > other
> > > > > >
> > > > > > > users
> > > > > >
> > > > > > > > >> > whose
> > > > > >
> > > > > > > > >> > >>> > > identity
> > > > > >
> > > > > > > > >> > >>> > > > > > they
> > > > > >
> > > > > > > > >> > >>> > > > > > > > have
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > verified independently.
> Kafka
> > > > > brokers
> > > > > >
> > > > > > > > will
> > > > > >
> > > > > > > > >> > have
> > > > > >
> > > > > > > > >> > >>> ACLs
> > > > > >
> > > > > > > > >> > >>> > > to
> > > > > >
> > > > > > > > >> > >>> > > > > > > control
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> which
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > users are allowed to
> > impersonate
> > > > > other
> > > > > >
> > > > > > > > users
> > > > > >
> > > > > > > > >> > and
> > > > > >
> > > > > > > > >> > >>> get
> > > > > >
> > > > > > > > >> > >>> > > > tokens
> > > > > >
> > > > > > > > >> > >>> > > > > > on
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> behalf of
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > them. Overall Impersonation
> > is a
> > > > > whole
> > > > > >
> > > > > > > > >> > different
> > > > > >
> > > > > > > > >> > >>> > > problem
> > > > > >
> > > > > > > > >> > >>> > > > in
> > > > > >
> > > > > > > > >> > >>> > > > > > my
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> opinion and
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > I think we can tackle it in
> > > > separate
> > > > > >
> > > > > > > KIP.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > 111. What's the typical rate
> > of
> > > > > > getting
> > > > > >
> > > > > > > > and
> > > > > >
> > > > > > > > >> > >>> renewing
> > > > > >
> > > > > > > > >> > >>> > > > > > delegation
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> tokens?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > Typically this should be
> very
> > > very
> > > > > > low,
> > > > > >
> > > > > > > 1
> > > > > >
> > > > > > > > >> > request
> > > > > >
> > > > > > > > >> > >>> per
> > > > > >
> > > > > > > > >> > >>> > > > > minute
> > > > > >
> > > > > > > > >> > >>> > > > > > > is a
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > relatively high estimate.
> > > However
> > > > it
> > > > > >
> > > > > > > > depends
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > on
> > > > > >
> > > > > > > > >> > >>> the
> > > > > >
> > > > > > > > >> > >>> > > token
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> expiration. I am
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > less worried about the extra
> > > load
> > > > it
> > > > > >
> > > > > > > puts
> > > > > >
> > > > > > > > on
> > > > > >
> > > > > > > > >> > >>> > controller
> > > > > >
> > > > > > > > >> > >>> > > > vs
> > > > > >
> > > > > > > > >> > >>> > > > > > the
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > added
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > complexity and the value it
> > > > offers.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > Thanks
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > Parth
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > On Tue, May 24, 2016 at 7:30
> > AM,
> > > > > > Ismael
> > > > > >
> > > > > > > > Juma
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > <
> > > > > >
> > > > > > > > >> > >>> > > > > > > ismael@juma.me.uk>
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> wrote:
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > Thanks Rajini. It would
> > > probably
> > > > > >
> > > > > > > > require a
> > > > > >
> > > > > > > > >> > >>> separate
> > > > > >
> > > > > > > > >> > >>> > > KIP
> > > > > >
> > > > > > > > >> > >>> > > > > as
> > > > > >
> > > > > > > > >> > >>> > > > > > it
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > will
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > introduce user visible
> > > changes.
> > > > We
> > > > > >
> > > > > > > could
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > also
> > > > > >
> > > > > > > > >> > >>> > update
> > > > > >
> > > > > > > > >> > >>> > > > > KIP-48
> > > > > >
> > > > > > > > >> > >>> > > > > > > to
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> have this
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > information, but it seems
> > > > cleaner
> > > > > to
> > > > > >
> > > > > > > do
> > > > > >
> > > > > > > > it
> > > > > >
> > > > > > > > >> > >>> > > separately.
> > > > > >
> > > > > > > > >> > >>> > > > We
> > > > > >
> > > > > > > > >> > >>> > > > > > can
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> discuss
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > that
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > in the KIP call today.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > Ismael
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > On Tue, May 24, 2016 at
> 3:19
> > > PM,
> > > > > >
> > > > > > > Rajini
> > > > > >
> > > > > > > > >> > Sivaram
> > > > > >
> > > > > > > > >> > >>> <
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > >
> > rajinisivaram@googlemail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > Ismael,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > I have created a JIRA (
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > https://issues.apache.org/
> > > > > >
> > > > > > > > >> > jira/browse/KAFKA-3751)
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > for adding SCRAM as a
> SASL
> > > > > >
> > > > > > > mechanism.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > Would
> > > > > >
> > > > > > > > >> > >>> that
> > > > > >
> > > > > > > > >> > >>> > > need
> > > > > >
> > > > > > > > >> > >>> > > > > > > another
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> KIP? If
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > KIP-48 will use this
> > > > mechanism,
> > > > > > can
> > > > > >
> > > > > > > > this
> > > > > >
> > > > > > > > >> > just
> > > > > >
> > > > > > > > >> > >>> be
> > > > > >
> > > > > > > > >> > >>> > a
> > > > > >
> > > > > > > > >> > >>> > > > JIRA
> > > > > >
> > > > > > > > >> > >>> > > > > > > that
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > gets
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > reviewed
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > when the PR is ready?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > Thank you,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > Rajini
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > On Tue, May 24, 2016 at
> > 2:46
> > > > PM,
> > > > > >
> > > > > > > > Ismael
> > > > > >
> > > > > > > > >> > Juma <
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > ismael@juma.me.uk>
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > wrote:
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > Thanks Rajini, SCRAM
> > seems
> > > > > like
> > > > > > a
> > > > > >
> > > > > > > > good
> > > > > >
> > > > > > > > >> > >>> > candidate.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > Gwen had independently
> > > > > mentioned
> > > > > >
> > > > > > > > this
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > as
> > > > > >
> > > > > > > > >> > a
> > > > > >
> > > > > > > > >> > >>> SASL
> > > > > >
> > > > > > > > >> > >>> > > > > > mechanism
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > that
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> might
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > be
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > useful for Kafka and I
> > > have
> > > > > been
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > meaning
> > > > > >
> > > > > > > > >> > to
> > > > > >
> > > > > > > > >> > >>> > check
> > > > > >
> > > > > > > > >> > >>> > > > it
> > > > > >
> > > > > > > > >> > >>> > > > > in
> > > > > >
> > > > > > > > >> > >>> > > > > > > > more
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> detail.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > Good
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > to know that you are
> > > willing
> > > > > to
> > > > > >
> > > > > > > > >> > contribute
> > > > > >
> > > > > > > > >> > >>> an
> > > > > >
> > > > > > > > >> > >>> > > > > > > > implementation.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> Maybe
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > we
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > should file a separate
> > > JIRA
> > > > > for
> > > > > >
> > > > > > > > this?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > Ismael
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > On Tue, May 24, 2016
> at
> > > 2:12
> > > > > PM,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > Rajini
> > > > > >
> > > > > > > > >> > >>> > Sivaram <
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > >
> > > > rajinisivaram@googlemail.com>
> > > > > >
> > > > > > > > wrote:
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > SCRAM (Salted
> > Challenge
> > > > > > Response
> > > > > >
> > > > > > > > >> > >>> > Authentication
> > > > > >
> > > > > > > > >> > >>> > > > > > > > Mechanism)
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> is a
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > better
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > mechanism than
> > > Digest-MD5.
> > > > > > Java
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > doesn't
> > > > > >
> > > > > > > > >> > >>> come
> > > > > >
> > > > > > > > >> > >>> > > > with a
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > built-in
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> SCRAM
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > SaslServer or
> > > SaslClient,
> > > > > but
> > > > > > I
> > > > > >
> > > > > > > > will
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > be
> > > > > >
> > > > > > > > >> > >>> happy
> > > > > >
> > > > > > > > >> > >>> > > to
> > > > > >
> > > > > > > > >> > >>> > > > > add
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > support
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> in
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > Kafka
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > since
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > it would be a useful
> > > > > mechanism
> > > > > >
> > > > > > > to
> > > > > >
> > > > > > > > >> > support
> > > > > >
> > > > > > > > >> > >>> > > anyway.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > >
> > > > > https://tools.ietf.org/html/
> > > > > >
> > > > > > > > rfc7677
> > > > > >
> > > > > > > > >> > >>> > describes
> > > > > >
> > > > > > > > >> > >>> > > > the
> > > > > >
> > > > > > > > >> > >>> > > > > > > > protocol
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> for
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > SCRAM-SHA-256.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > On Tue, May 24, 2016
> > at
> > > > 2:37
> > > > > > AM,
> > > > > >
> > > > > > > > Jun
> > > > > >
> > > > > > > > >> > Rao <
> > > > > >
> > > > > > > > >> > >>> > > > > > > > jun@confluent.io
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> wrote:
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > Parth,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > Thanks for the
> > > > > explanation.
> > > > > > A
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > couple
> > > > > >
> > > > > > > > >> > of
> > > > > >
> > > > > > > > >> > >>> > more
> > > > > >
> > > > > > > > >> > >>> > > > > > > questions.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > 110. What does
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> getDelegationTokenAs
> > > > > >
> > > > > > > > >> > >>> mean?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > 111. What's the
> > > typical
> > > > > rate
> > > > > >
> > > > > > > of
> > > > > >
> > > > > > > > >> > getting
> > > > > >
> > > > > > > > >> > >>> and
> > > > > >
> > > > > > > > >> > >>> > > > > > renewing
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> delegation
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > tokens?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > That may have an
> > > impact
> > > > on
> > > > > >
> > > > > > > > whether
> > > > > >
> > > > > > > > >> > they
> > > > > >
> > > > > > > > >> > >>> > > should
> > > > > >
> > > > > > > > >> > >>> > > > be
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > directed
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> to the
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > controller.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > Jun
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > On Mon, May 23,
> 2016
> > > at
> > > > > 1:19
> > > > > >
> > > > > > > PM,
> > > > > >
> > > > > > > > >> > parth
> > > > > >
> > > > > > > > >> > >>> > > > > brahmbhatt <
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > > > brahmbhatt.parth@gmail.com>
> > > > > >
> > > > > > > > wrote:
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Hi Jun,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Thanks for
> > > reviewing.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * We could add a
> > > > Cluster
> > > > > >
> > > > > > > > action
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > to
> > > > > >
> > > > > > > > >> > add
> > > > > >
> > > > > > > > >> > >>> > acls
> > > > > >
> > > > > > > > >> > >>> > > > on
> > > > > >
> > > > > > > > >> > >>> > > > > > who
> > > > > >
> > > > > > > > >> > >>> > > > > > > > can
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> request
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > delegation
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > tokens. I don't
> > see
> > > > the
> > > > > > use
> > > > > >
> > > > > > > > case
> > > > > >
> > > > > > > > >> > for
> > > > > >
> > > > > > > > >> > >>> that
> > > > > >
> > > > > > > > >> > >>> > > yet
> > > > > >
> > > > > > > > >> > >>> > > > > but
> > > > > >
> > > > > > > > >> > >>> > > > > > > > down
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> the line
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > when
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > we
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > start supporting
> > > > > >
> > > > > > > > >> > getDelegationTokenAs
> > > > > >
> > > > > > > > >> > >>> it
> > > > > >
> > > > > > > > >> > >>> > > will
> > > > > >
> > > > > > > > >> > >>> > > > > be
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> necessary.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * Yes we
> recommend
> > > > > tokens
> > > > > > to
> > > > > >
> > > > > > > > be
> > > > > >
> > > > > > > > >> > only
> > > > > >
> > > > > > > > >> > >>> > > > > > > used/distributed
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> over
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > secure
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > channels.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * Depending on
> > what
> > > > > design
> > > > > >
> > > > > > > we
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > end
> > > > > >
> > > > > > > > >> > up
> > > > > >
> > > > > > > > >> > >>> > > choosing
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> Invalidation will
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > be
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > responsibility
> of
> > > > every
> > > > > >
> > > > > > > broker
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > or
> > > > > >
> > > > > > > > >> > >>> > > controller.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * I am not sure
> > if I
> > > > > >
> > > > > > > > documented
> > > > > >
> > > > > > > > >> > >>> somewhere
> > > > > >
> > > > > > > > >> > >>> > > > that
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> invalidation
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > will
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > directly
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > go through
> > zookeeper
> > > > but
> > > > > >
> > > > > > > that
> > > > > >
> > > > > > > > is
> > > > > >
> > > > > > > > >> > not
> > > > > >
> > > > > > > > >> > >>> the
> > > > > >
> > > > > > > > >> > >>> > > > > intent.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> Invalidation
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > will
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > either
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > be request based
> > or
> > > > due
> > > > > to
> > > > > >
> > > > > > > > >> > >>> expiration. No
> > > > > >
> > > > > > > > >> > >>> > > > > direct
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> zookeeper
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > interaction
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > from
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > any client.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * "Broker also
> > > stores
> > > > > the
> > > > > >
> > > > > > > > >> > >>> DelegationToken
> > > > > >
> > > > > > > > >> > >>> > > > > without
> > > > > >
> > > > > > > > >> > >>> > > > > > > the
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> hmac in
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > the
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > zookeeper." :
> > Sorry
> > > > > about
> > > > > >
> > > > > > > the
> > > > > >
> > > > > > > > >> > >>> confusion.
> > > > > >
> > > > > > > > >> > >>> > > The
> > > > > >
> > > > > > > > >> > >>> > > > > sole
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> purpose of
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > zookeeper
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > in
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > this design is
> as
> > > > > >
> > > > > > > distribution
> > > > > >
> > > > > > > > >> > channel
> > > > > >
> > > > > > > > >> > >>> > for
> > > > > >
> > > > > > > > >> > >>> > > > > tokens
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> between all
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > brokers
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > and a
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > layer that
> ensures
> > > > only
> > > > > >
> > > > > > > tokens
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > that
> > > > > >
> > > > > > > > >> > >>> were
> > > > > >
> > > > > > > > >> > >>> > > > > > generated
> > > > > >
> > > > > > > > >> > >>> > > > > > > by
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> making a
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > request
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > to a
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > broker will be
> > > > accepted
> > > > > >
> > > > > > > (more
> > > > > >
> > > > > > > > on
> > > > > >
> > > > > > > > >> > this
> > > > > >
> > > > > > > > >> > >>> in
> > > > > >
> > > > > > > > >> > >>> > > > second
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> paragraph). The
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > token
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > consists of few
> > > > elements
> > > > > >
> > > > > > > > (owner,
> > > > > >
> > > > > > > > >> > >>> renewer,
> > > > > >
> > > > > > > > >> > >>> > > > uuid
> > > > > >
> > > > > > > > >> > >>> > > > > ,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> expiration,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > hmac)
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > ,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > one
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > of
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > which is the
> > finally
> > > > > >
> > > > > > > generated
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > hmac
> > > > > >
> > > > > > > > >> > >>> but
> > > > > >
> > > > > > > > >> > >>> > > hmac
> > > > > >
> > > > > > > > >> > >>> > > > it
> > > > > >
> > > > > > > > >> > >>> > > > > > > self
> > > > > >
> > > > > > > > >> > >>> > > > > > > > is
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > derivable
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > if
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > you
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > have all the
> other
> > > > > > elements
> > > > > >
> > > > > > > of
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > the
> > > > > >
> > > > > > > > >> > >>> token
> > > > > >
> > > > > > > > >> > >>> > +
> > > > > >
> > > > > > > > >> > >>> > > > > secret
> > > > > >
> > > > > > > > >> > >>> > > > > > > key
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > to
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > generate
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > hmac.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Given zookeeper
> > does
> > > > not
> > > > > >
> > > > > > > > provide
> > > > > >
> > > > > > > > >> > SSL
> > > > > >
> > > > > > > > >> > >>> > > support
> > > > > >
> > > > > > > > >> > >>> > > > we
> > > > > >
> > > > > > > > >> > >>> > > > > > do
> > > > > >
> > > > > > > > >> > >>> > > > > > > > not
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> want the
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > entire
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > token to be wire
> > > > > > transferred
> > > > > >
> > > > > > > > to
> > > > > >
> > > > > > > > >> > >>> zookeeper
> > > > > >
> > > > > > > > >> > >>> > > as
> > > > > >
> > > > > > > > >> > >>> > > > > that
> > > > > >
> > > > > > > > >> > >>> > > > > > > > will
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> be an
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > insecure
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > wire
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > transfer.
> Instead
> > we
> > > > > only
> > > > > >
> > > > > > > > store
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > all
> > > > > >
> > > > > > > > >> > >>> the
> > > > > >
> > > > > > > > >> > >>> > > other
> > > > > >
> > > > > > > > >> > >>> > > > > > > > elements
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> of a
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > delegation
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > tokens. Brokers
> > can
> > > > read
> > > > > >
> > > > > > > these
> > > > > >
> > > > > > > > >> > >>> elements
> > > > > >
> > > > > > > > >> > >>> > and
> > > > > >
> > > > > > > > >> > >>> > > > > > because
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > they
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> also
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > have
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > access
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > to secret key
> they
> > > > will
> > > > > be
> > > > > >
> > > > > > > > able
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > to
> > > > > >
> > > > > > > > >> > >>> > generate
> > > > > >
> > > > > > > > >> > >>> > > > > hmac
> > > > > >
> > > > > > > > >> > >>> > > > > > on
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> their end.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > One of the
> > > alternative
> > > > > >
> > > > > > > > proposed
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > is
> > > > > >
> > > > > > > > >> > to
> > > > > >
> > > > > > > > >> > >>> > avoid
> > > > > >
> > > > > > > > >> > >>> > > > > > > zookeeper
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > altogether. A
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > Client
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > will call broker
> > > with
> > > > > >
> > > > > > > required
> > > > > >
> > > > > > > > >> > >>> > information
> > > > > >
> > > > > > > > >> > >>> > > > > > (owner,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> renwer,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > expiration)
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > and
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > get back (signed
> > > hmac,
> > > > > >
> > > > > > > uuid).
> > > > > >
> > > > > > > > >> > Broker
> > > > > >
> > > > > > > > >> > >>> > won't
> > > > > >
> > > > > > > > >> > >>> > > > > store
> > > > > >
> > > > > > > > >> > >>> > > > > > > this
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > in
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > zookeeper.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > From
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > this point a
> > client
> > > > can
> > > > > >
> > > > > > > > contact
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > any
> > > > > >
> > > > > > > > >> > >>> > broker
> > > > > >
> > > > > > > > >> > >>> > > > with
> > > > > >
> > > > > > > > >> > >>> > > > > > all
> > > > > >
> > > > > > > > >> > >>> > > > > > > > the
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > delegation
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > token
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > info (owner,
> > rewner,
> > > > > >
> > > > > > > > expiration,
> > > > > >
> > > > > > > > >> > hmac,
> > > > > >
> > > > > > > > >> > >>> > > uuid)
> > > > > >
> > > > > > > > >> > >>> > > > > the
> > > > > >
> > > > > > > > >> > >>> > > > > > > > borker
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> will
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > regenerate
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > the
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > hmac and as long
> > as
> > > it
> > > > > >
> > > > > > > matches
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > with
> > > > > >
> > > > > > > > >> > >>> hmac
> > > > > >
> > > > > > > > >> > >>> > > > > > presented
> > > > > >
> > > > > > > > >> > >>> > > > > > > by
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> client ,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > broker
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > will
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > allow the
> request
> > to
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > authenticate.
> > > > > >
> > > > > > > > >> > >>> Only
> > > > > >
> > > > > > > > >> > >>> > > > > problem
> > > > > >
> > > > > > > > >> > >>> > > > > > > with
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> this
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > approach
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > is
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > if
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > the secret key
> is
> > > > > >
> > > > > > > compromised
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > any
> > > > > >
> > > > > > > > >> > >>> client
> > > > > >
> > > > > > > > >> > >>> > > can
> > > > > >
> > > > > > > > >> > >>> > > > > now
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > generate
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > random
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > tokens
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > and
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > they will still
> be
> > > > able
> > > > > to
> > > > > >
> > > > > > > > >> > >>> authenticate
> > > > > >
> > > > > > > > >> > >>> > as
> > > > > >
> > > > > > > > >> > >>> > > > any
> > > > > >
> > > > > > > > >> > >>> > > > > > user
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > they
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> like.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > with
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > zookeeper we
> > > guarantee
> > > > > > that
> > > > > >
> > > > > > > > only
> > > > > >
> > > > > > > > >> > >>> tokens
> > > > > >
> > > > > > > > >> > >>> > > > > acquired
> > > > > >
> > > > > > > > >> > >>> > > > > > > via
> > > > > >
> > > > > > > > >> > >>> > > > > > > > a
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> broker
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > (using
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > some
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > auth scheme
> other
> > > than
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > delegation
> > > > > >
> > > > > > > > >> > >>> token)
> > > > > >
> > > > > > > > >> > >>> > > will
> > > > > >
> > > > > > > > >> > >>> > > > > be
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> accepted. We
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > need
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > to
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > discuss which
> > > proposal
> > > > > > makes
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > more
> > > > > >
> > > > > > > > >> > >>> sense
> > > > > >
> > > > > > > > >> > >>> > and
> > > > > >
> > > > > > > > >> > >>> > > > we
> > > > > >
> > > > > > > > >> > >>> > > > > > can
> > > > > >
> > > > > > > > >> > >>> > > > > > > go
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> over it
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > in
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > tomorrow's
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > meeting.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Also, can you
> > > forward
> > > > > the
> > > > > >
> > > > > > > > invite
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > to
> > > > > >
> > > > > > > > >> > >>> me?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Thanks
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Parth
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > On Mon, May 23,
> > 2016
> > > > at
> > > > > >
> > > > > > > 10:35
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > AM,
> > > > > >
> > > > > > > > >> > Jun
> > > > > >
> > > > > > > > >> > >>> > Rao <
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> jun@confluent.io>
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > wrote:
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > Thanks for the
> > > KIP.
> > > > A
> > > > > > few
> > > > > >
> > > > > > > > >> > comments.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > 100. This
> > > > potentially
> > > > > > can
> > > > > >
> > > > > > > be
> > > > > >
> > > > > > > > >> > useful
> > > > > >
> > > > > > > > >> > >>> for
> > > > > >
> > > > > > > > >> > >>> > > > Kafka
> > > > > >
> > > > > > > > >> > >>> > > > > > > > Connect
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> and
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > Kafka
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > rest
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > proxy
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > where a worker
> > > agent
> > > > > > will
> > > > > >
> > > > > > > > need
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > to
> > > > > >
> > > > > > > > >> > >>> run a
> > > > > >
> > > > > > > > >> > >>> > > > task
> > > > > >
> > > > > > > > >> > >>> > > > > on
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > behalf
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> of a
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > client.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > We
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > will
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > likely need to
> > > > change
> > > > > > how
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > those
> > > > > >
> > > > > > > > >> > >>> > services
> > > > > >
> > > > > > > > >> > >>> > > > use
> > > > > >
> > > > > > > > >> > >>> > > > > > > Kafka
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> clients
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > > (producer/consumer).
> > > > > >
> > > > > > > Instead
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > of a
> > > > > >
> > > > > > > > >> > >>> > shared
> > > > > >
> > > > > > > > >> > >>> > > > > client
> > > > > >
> > > > > > > > >> > >>> > > > > > > per
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> worker,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > we
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > will
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > need
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > a
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > client per
> user
> > > task
> > > > > > since
> > > > > >
> > > > > > > > the
> > > > > >
> > > > > > > > >> > >>> > > > authentication
> > > > > >
> > > > > > > > >> > >>> > > > > > > > happens
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> at the
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > connection
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > level. For
> Kafka
> > > > > > Connect,
> > > > > >
> > > > > > > > the
> > > > > >
> > > > > > > > >> > >>> renewer
> > > > > >
> > > > > > > > >> > >>> > > will
> > > > > >
> > > > > > > > >> > >>> > > > be
> > > > > >
> > > > > > > > >> > >>> > > > > > the
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> workers.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > So,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > we
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > probably
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > need to allow
> > > > multiple
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > renewers.
> > > > > >
> > > > > > > > >> > For
> > > > > >
> > > > > > > > >> > >>> > > Kafka
> > > > > >
> > > > > > > > >> > >>> > > > > rest
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > proxy,
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> the
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > renewer
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > can
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > probably just
> be
> > > the
> > > > > >
> > > > > > > creator
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > of
> > > > > >
> > > > > > > > >> > the
> > > > > >
> > > > > > > > >> > >>> > > token.
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > 101. Do we
> need
> > > new
> > > > > acl
> > > > > > on
> > > > > >
> > > > > > > > who
> > > > > >
> > > > > > > > >> > can
> > > > > >
> > > > > > > > >> > >>> > > request
> > > > > >
> > > > > > > > >> > >>> > > > > > > > delegation
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> tokens?
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > 102. Do we
> > > recommend
> > > > > >
> > > > > > > people
> > > > > >
> > > > > > > > to
> > > > > >
> > > > > > > > >> > send
> > > > > >
> > > > > > > > >> > >>> > > > > delegation
> > > > > >
> > > > > > > > >> > >>> > > > > > > > tokens
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> in an
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > encrypted
> > > > > >
> > > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > channel?
> > > > > >
> > > > > > > > >> > >>>
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Gwen Shapira*
> > > Product Manager | Confluent
> > > 650.450.2760 | @gwenshap
> > > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> > > <http://www.confluent.io/blog>
> > >
> >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Rajini Sivaram <rs...@pivotal.io>]

It would clearly be very useful to enable clients to send requests on
behalf of multiple users. A separate KIP makes sense, but it may be worth
thinking through some of the implications now, especially if the main
interest in delegation tokens comes from its potential to enable
impersonation.

I understand that delegation tokens are only expected to be used with TLS.
But the choice of SASL/SCRAM for authentication must be based on a
requirement to protect the tokenHmac - otherwise you could just use
SASL/PLAIN. With SASL/SCRAM the tokenHmac is never propagated on-the-wire,
only a salted-hashed version of it is used in the SASL authentication
exchange. If impersonation is based on sending tokenHmac in requests, any
benefit of using SCRAM is lost.

An alternative may be to allow clients to authenticate multiple times using
SASL and include one of its authenticated principals in each request
(optionally). I haven't thought it through yet, obviously. But if the
approach is of interest and no one is working on a KIP for impersonation at
the moment, I am happy to write one. It may provide something for
comparison at least.

Thoughts?


On Wed, Dec 14, 2016 at 9:53 AM, Manikumar <ma...@gmail.com>
wrote:

> That's a good idea. Authenticating every request with delegation token will
> be useful for
> impersonation use-cases. But as of now, we are thinking delegation token as
> just another way
> to authenticate the users. We haven't think through all the use cases
> related to
> impersonation or using delegation token for impersonation. We want to
> handle impersonation
> (KAFKA-3712) as part of separate KIP.
>
> Will that be Ok?
>
>
> On Wed, Dec 14, 2016 at 8:09 AM, Gwen Shapira <gw...@confluent.io> wrote:
>
> > Thinking out loud here:
> >
> > It looks like authentication with a delegation token is going to be
> > super-cheap, right? We just compare the token to a value in the broker
> > cache?
> >
> > If I understood the KIP correctly, right now it suggests that
> > authentication happens when establishing the client-broker connection (as
> > normal for Kafka. But perhaps we want to consider authenticating every
> > request with delegation token (if exists)?
> >
> > So a centralized app can create few producers, do the metadata request
> and
> > broker discovery with its own user auth, but then use delegation tokens
> to
> > allow performing produce/fetch requests as different users? Instead of
> > having to re-connect for each impersonated user?
> >
> > This may over-complicate things quite a bit (basically adding extra
> > information in every request), but maybe it will be useful for
> > impersonation use-cases (which seem to drive much of the interest in this
> > KIP)?
> > Kafka Connect, NiFi and friends can probably use this to share clients
> > between multiple jobs, tasks, etc.
> >
> > What do you think?
> >
> > Gwen
> >
> > On Tue, Dec 13, 2016 at 12:43 AM, Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Ashish,
> > >
> > > Thank you for reviewing the KIP.  Please see the replies inline.
> > >
> > >
> > > > 1. How to disable delegation token authentication?
> > > >
> > > > This can be achieved in various ways, however I think reusing
> > delegation
> > > > token secret config for this makes sense here. Avoids creating yet
> > > another
> > > > config and forces delegation token users to consciously set the
> secret.
> > > If
> > > > the secret is not set or set to empty string, brokers should turn off
> > > > delegation token support. This will however require a new error code
> to
> > > > indicate delegation token support is turned off on broker.
> > > >
> > >
> > >   Thanks for the suggestion. Option to turnoff delegation token
> > > authentication will be useful.
> > >   I'll update the KIP.
> > >
> > >
> > > >
> > > > 2. ACLs on delegation token?
> > > >
> > > > Do we need to have ACLs defined for tokens? I do not think it buys us
> > > > anything, as delegation token can be treated as impersonation of the
> > > owner.
> > > > Any thing the owner has permission to do, delegation tokens should be
> > > > allowed to do as well. If so, we probably won't need to return
> > > > authorization exception error code while creating delegation token.
> It
> > > > however would make sense to check renew and expire requests are
> coming
> > > from
> > > > owner or renewers of the token, but that does not require explicit
> > acls.
> > > >
> > >
> > >
> > > Yes, We agreed to not have new acl on who can request delegation token.
> > >  I'll update the KIP.
> > >
> > >
> > > >
> > > > 3. How to restrict max life time of a token?
> > > >
> > > > Admins might want to restrict max life time of tokens created on a
> > > cluster,
> > > > and this can very from cluster to cluster based on use-cases. This
> > might
> > > > warrant a separate broker config.
> > > >
> > > >
> > > Currently we  have "delegation.token.max.lifetime.sec" server config
> > > property
> > > May be we can take min(User supplied MaxTime, Server MaxTime) as max
> life
> > > time.
> > > I am open to add new config property.
> > >
> > > Few more comments based on recent KIP update.
> > > >
> > > > 1. Do we need a separate {{InvalidateTokenRequest}}? Can't we use
> > > > {{ExpireTokenRequest}} with with expiryDate set to anything before
> > > current
> > > > date?
> > > >
> > >
> > > makes sense. we don't need special request to cancel the token. We can
> > use
> > > ExpireTokenRequest.
> > > I'll update the KIP.
> > >
> > >
> > > > 2. Can we change time field names to indicate their unit is
> > milliseconds,
> > > > like, IssueDateMs, ExpiryDateMs, etc.?
> > > >
> > > >
> > >   Done.
> > >
> > >
> > > > 3. Can we allow users to renew a token for a specified amount of
> time?
> > In
> > > > current version of KIP, renew request does not take time as a param,
> > not
> > > > sure what is expiry time set to after renewal.
> > > >
> > > >
> > >  Yes, we need to specify renew period.  I'll update the KIP.
> > >
> > >
> > > Thanks,
> > > Mankumar
> > >
> > >
> > >
> > > >
> > > > On Mon, Dec 12, 2016 at 9:08 AM Manikumar <manikumar.reddy@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > >
> > > > >
> > > > > I would like to reinitiate the discussion on Delegation token
> support
> > > for
> > > > >
> > > > > Kafka.
> > > > >
> > > > >
> > > > >
> > > > > Brief summary of the past discussion:
> > > > >
> > > > >
> > > > >
> > > > > 1) Broker stores delegation tokens in zookeeper.  All brokers will
> > > have a
> > > > >
> > > > > cache backed by
> > > > >
> > > > >    zookeeper so they will all get notified whenever a new token is
> > > > >
> > > > > generated and they will
> > > > >
> > > > >    update their local cache whenever token state changes.
> > > > >
> > > > > 2) The current proposal does not support rotation of secret
> > > > >
> > > > > 3) Only allow the renewal by users that authenticated using *non*
> > > > >
> > > > > delegation token mechanism
> > > > >
> > > > > 4) KIP-84 proposes to support  SASL SCRAM mechanisms. Kafka clients
> > can
> > > > >
> > > > > authenticate using
> > > > >
> > > > >    SCRAM-SHA-256, providing the delegation token HMAC as password.
> > > > >
> > > > >
> > > > >
> > > > > Updated the KIP with the following:
> > > > >
> > > > > 1. Protocol and Config changes
> > > > >
> > > > > 2. format of the data stored in ZK.
> > > > >
> > > > > 3. Changes to Java Clients/Usage of SASL SCRAM mechanism
> > > > >
> > > > >
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >
> > > > > 48+Delegation+token+support+for+Kafka
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Jun, Ashish, Gwen,
> > > > >
> > > > >
> > > > >
> > > > > Pl review the updated KIP.
> > > > >
> > > > >
> > > > >
> > > > > Thanks
> > > > >
> > > > > Manikumar
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Sep 29, 2016 at 9:56 PM, Ashish Singh <asingh@cloudera.com
> >
> > > > wrote:
> > > > >
> > > > >
> > > > >
> > > > > > Harsha/ Gwen,
> > > > >
> > > > > >
> > > > >
> > > > > > How do we proceed here? I am willing to help out with here.
> > > > >
> > > > > >
> > > > >
> > > > > > On Fri, Sep 23, 2016 at 11:41 AM, Gwen Shapira <
> gwen@confluent.io>
> > > > > wrote:
> > > > >
> > > > > >
> > > > >
> > > > > > > Is it updated? are all concerns addressed? do you want to
> start a
> > > > vote?
> > > > >
> > > > > > >
> > > > >
> > > > > > > Sorry for being pushy, I do appreciate that we are all
> volunteers
> > > and
> > > > >
> > > > > > > finding time is difficult. This feature is important for
> anything
> > > > that
> > > > >
> > > > > > > integrates with Kafka (stream processors, Flume, NiFi, etc)
> and I
> > > > >
> > > > > > > don't want to see this getting stuck because we lack
> coordination
> > > > >
> > > > > > > within the community.
> > > > >
> > > > > > >
> > > > >
> > > > > > > On Thu, Sep 15, 2016 at 6:39 PM, Harsha Chintalapani <
> > > > kafka@harsha.io>
> > > > >
> > > > > > > wrote:
> > > > >
> > > > > > > > The only pending update for the KIP is to write up the
> protocol
> > > > > changes
> > > > >
> > > > > > > like
> > > > >
> > > > > > > > we've it KIP-4. I'll update the wiki.
> > > > >
> > > > > > > >
> > > > >
> > > > > > > >
> > > > >
> > > > > > > > On Thu, Sep 15, 2016 at 4:27 PM Ashish Singh <
> > > asingh@cloudera.com>
> > > > >
> > > > > > > wrote:
> > > > >
> > > > > > > >>
> > > > >
> > > > > > > >> I think we decided to not support secret rotation, I guess
> > this
> > > > can
> > > > > be
> > > > >
> > > > > > > >> stated clearly on the KIP. Also, more details on how clients
> > > will
> > > > >
> > > > > > > perform
> > > > >
> > > > > > > >> token distribution and how CLI will look like will be
> helpful.
> > > > >
> > > > > > > >>
> > > > >
> > > > > > > >> On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <
> > > gwen@confluent.io>
> > > > >
> > > > > > > wrote:
> > > > >
> > > > > > > >>
> > > > >
> > > > > > > >> > Hi Guys,
> > > > >
> > > > > > > >> >
> > > > >
> > > > > > > >> > This discussion was dead for a while. Are there still
> > > > contentious
> > > > >
> > > > > > > >> > points? If not, why are there no votes?
> > > > >
> > > > > > > >> >
> > > > >
> > > > > > > >> > On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <
> jun@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > > >> > > Ashish,
> > > > >
> > > > > > > >> > >
> > > > >
> > > > > > > >> > > Yes, I will send out a KIP invite for next week to
> discuss
> > > > > KIP-48
> > > > >
> > > > > > > and
> > > > >
> > > > > > > >> > other
> > > > >
> > > > > > > >> > > remaining KIPs.
> > > > >
> > > > > > > >> > >
> > > > >
> > > > > > > >> > > Thanks,
> > > > >
> > > > > > > >> > >
> > > > >
> > > > > > > >> > > Jun
> > > > >
> > > > > > > >> > >
> > > > >
> > > > > > > >> > > On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh <
> > > > >
> > > > > > asingh@cloudera.com>
> > > > >
> > > > > > > >> > wrote:
> > > > >
> > > > > > > >> > >
> > > > >
> > > > > > > >> > >> Thanks Harsha!
> > > > >
> > > > > > > >> > >>
> > > > >
> > > > > > > >> > >> Jun, can we add KIP-48 to next KIP hangout's agenda.
> > Also,
> > > we
> > > > > did
> > > > >
> > > > > > > not
> > > > >
> > > > > > > >> > >> actually make a call on when we should have next KIP
> > call.
> > > As
> > > > >
> > > > > > there
> > > > >
> > > > > > > >> > >> are
> > > > >
> > > > > > > >> > a
> > > > >
> > > > > > > >> > >> few outstanding KIPs that could not be discussed this
> > week,
> > > > can
> > > > >
> > > > > > we
> > > > >
> > > > > > > >> > >> have
> > > > >
> > > > > > > >> > a
> > > > >
> > > > > > > >> > >> KIP hangout call next week?
> > > > >
> > > > > > > >> > >>
> > > > >
> > > > > > > >> > >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha Chintalapani
> > > > >
> > > > > > > >> > >> <ka...@harsha.io>
> > > > >
> > > > > > > >> > >> wrote:
> > > > >
> > > > > > > >> > >>
> > > > >
> > > > > > > >> > >>> Ashish,
> > > > >
> > > > > > > >> > >>>         Yes we are working on it. Lets discuss in the
> > next
> > > > KIP
> > > > >
> > > > > > > >> > >>> meeting.
> > > > >
> > > > > > > >> > >>> I'll join.
> > > > >
> > > > > > > >> > >>> -Harsha
> > > > >
> > > > > > > >> > >>>
> > > > >
> > > > > > > >> > >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh <
> > > > >
> > > > > > > asingh@cloudera.com>
> > > > >
> > > > > > > >> > >>> wrote:
> > > > >
> > > > > > > >> > >>>
> > > > >
> > > > > > > >> > >>> > Hello Harsha,
> > > > >
> > > > > > > >> > >>> >
> > > > >
> > > > > > > >> > >>> > Are you still working on this? Wondering if we can
> > > discuss
> > > > >
> > > > > > this
> > > > >
> > > > > > > in
> > > > >
> > > > > > > >> > next
> > > > >
> > > > > > > >> > >>> KIP
> > > > >
> > > > > > > >> > >>> > meeting, if you can join.
> > > > >
> > > > > > > >> > >>> >
> > > > >
> > > > > > > >> > >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha
> Chintalapani <
> > > > >
> > > > > > > >> > kafka@harsha.io>
> > > > >
> > > > > > > >> > >>> > wrote:
> > > > >
> > > > > > > >> > >>> >
> > > > >
> > > > > > > >> > >>> > > Hi Grant,
> > > > >
> > > > > > > >> > >>> > >           We are working on it. Will add the
> details
> > > to
> > > > > KIP
> > > > >
> > > > > > > >> > >>> > > about
> > > > >
> > > > > > > >> > the
> > > > >
> > > > > > > >> > >>> > > request protocol.
> > > > >
> > > > > > > >> > >>> > >
> > > > >
> > > > > > > >> > >>> > > Thanks,
> > > > >
> > > > > > > >> > >>> > > Harsha
> > > > >
> > > > > > > >> > >>> > >
> > > > >
> > > > > > > >> > >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant Henke
> > > > >
> > > > > > > >> > >>> > > <gh...@cloudera.com>
> > > > >
> > > > > > > >> > >>> wrote:
> > > > >
> > > > > > > >> > >>> > >
> > > > >
> > > > > > > >> > >>> > > > Hi Parth,
> > > > >
> > > > > > > >> > >>> > > >
> > > > >
> > > > > > > >> > >>> > > > Are you still working on this? If you need any
> > help
> > > > > please
> > > > >
> > > > > > > >> > >>> > > > don't
> > > > >
> > > > > > > >> > >>> > hesitate
> > > > >
> > > > > > > >> > >>> > > > to ask.
> > > > >
> > > > > > > >> > >>> > > >
> > > > >
> > > > > > > >> > >>> > > > Thanks,
> > > > >
> > > > > > > >> > >>> > > > Grant
> > > > >
> > > > > > > >> > >>> > > >
> > > > >
> > > > > > > >> > >>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <
> > > > >
> > > > > > jun@confluent.io>
> > > > >
> > > > > > > >> > wrote:
> > > > >
> > > > > > > >> > >>> > > >
> > > > >
> > > > > > > >> > >>> > > > > Parth,
> > > > >
> > > > > > > >> > >>> > > > >
> > > > >
> > > > > > > >> > >>> > > > > Thanks for the reply.
> > > > >
> > > > > > > >> > >>> > > > >
> > > > >
> > > > > > > >> > >>> > > > > It makes sense to only allow the renewal by
> > users
> > > > that
> > > > >
> > > > > > > >> > >>> authenticated
> > > > >
> > > > > > > >> > >>> > > > using
> > > > >
> > > > > > > >> > >>> > > > > *non* delegation token mechanism. Then, should
> > we
> > > > make
> > > > >
> > > > > > the
> > > > >
> > > > > > > >> > >>> renewal a
> > > > >
> > > > > > > >> > >>> > > > list?
> > > > >
> > > > > > > >> > >>> > > > > For example, in the case of rest proxy, it
> will
> > be
> > > > >
> > > > > > useful
> > > > >
> > > > > > > >> > >>> > > > > for
> > > > >
> > > > > > > >> > >>> every
> > > > >
> > > > > > > >> > >>> > > > > instance of rest proxy to be able to renew the
> > > > tokens.
> > > > >
> > > > > > > >> > >>> > > > >
> > > > >
> > > > > > > >> > >>> > > > > It would be clearer if we can document the
> > request
> > > > >
> > > > > > > protocol
> > > > >
> > > > > > > >> > like
> > > > >
> > > > > > > >> > >>> > > > >
> > > > >
> > > > > > > >> > >>> > > > >
> > > > >
> > > > > > > >> > >>> > > > https://cwiki.apache.org/confl
> > > > uence/display/KAFKA/KIP-
> > > > >
> > > > > > > >> > >>> > >
> > > > >
> > > > > > > >> > >>> > > 4+-+Command+line+and+centralized+administrative+
> > > > >
> > > > > > > operations#KIP-4-
> > > > >
> > > > > > > >> > >>> > > Commandlineandcentralizedadmin
> istrativeoperations-
> > > > >
> > > > > > > >> > >>> > > CreateTopicsRequest(KAFKA-
> > > 2945):(VotedandPlannedforin0.
> > > > >
> > > > > > > 10.1.0)
> > > > >
> > > > > > > >> > >>> > > > > .
> > > > >
> > > > > > > >> > >>> > > > >
> > > > >
> > > > > > > >> > >>> > > > > It would also be useful to document the client
> > > APIs.
> > > > >
> > > > > > > >> > >>> > > > >
> > > > >
> > > > > > > >> > >>> > > > > Thanks,
> > > > >
> > > > > > > >> > >>> > > > >
> > > > >
> > > > > > > >> > >>> > > > > Jun
> > > > >
> > > > > > > >> > >>> > > > >
> > > > >
> > > > > > > >> > >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM, parth
> > brahmbhatt
> > > <
> > > > >
> > > > > > > >> > >>> > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > >
> > > > > > > >> > >>> > > > >
> > > > >
> > > > > > > >> > >>> > > > > > Hi,
> > > > >
> > > > > > > >> > >>> > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > I am suggesting that we will only allow the
> > > > renewal
> > > > > by
> > > > >
> > > > > > > >> > >>> > > > > > users
> > > > >
> > > > > > > >> > >>> that
> > > > >
> > > > > > > >> > >>> > > > > > authenticated using *non* delegation token
> > > > > mechanism.
> > > > >
> > > > > > > For
> > > > >
> > > > > > > >> > >>> example,
> > > > >
> > > > > > > >> > >>> > If
> > > > >
> > > > > > > >> > >>> > > > > user
> > > > >
> > > > > > > >> > >>> > > > > > Alice authenticated using kerberos and
> > requested
> > > > >
> > > > > > > >> > >>> > > > > > delegation
> > > > >
> > > > > > > >> > >>> tokens,
> > > > >
> > > > > > > >> > >>> > > > only
> > > > >
> > > > > > > >> > >>> > > > > > user Alice authenticated via non delegation
> > > token
> > > > >
> > > > > > > >> > >>> > > > > > mechanism
> > > > >
> > > > > > > >> > can
> > > > >
> > > > > > > >> > >>> > > renew.
> > > > >
> > > > > > > >> > >>> > > > > > Clients that have  access to delegation
> tokens
> > > can
> > > > > not
> > > > >
> > > > > > > >> > >>> > > > > > issue
> > > > >
> > > > > > > >> > >>> > renewal
> > > > >
> > > > > > > >> > >>> > > > > > request for renewing their own token and
> this
> > is
> > > > >
> > > > > > > primarily
> > > > >
> > > > > > > >> > >>> > important
> > > > >
> > > > > > > >> > >>> > > to
> > > > >
> > > > > > > >> > >>> > > > > > reduce the time window for which a
> compromised
> > > > token
> > > > >
> > > > > > > will
> > > > >
> > > > > > > >> > >>> > > > > > be
> > > > >
> > > > > > > >> > >>> valid.
> > > > >
> > > > > > > >> > >>> > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > To clarify, Yes any authenticated user can
> > > request
> > > > >
> > > > > > > >> > >>> > > > > > delegation
> > > > >
> > > > > > > >> > >>> > tokens
> > > > >
> > > > > > > >> > >>> > > > but
> > > > >
> > > > > > > >> > >>> > > > > > even here I would recommend to avoid
> creating
> > a
> > > > > chain
> > > > >
> > > > > > > >> > >>> > > > > > where a
> > > > >
> > > > > > > >> > >>> > client
> > > > >
> > > > > > > >> > >>> > > > > > authenticated via delegation token request
> for
> > > > more
> > > > >
> > > > > > > >> > delegation
> > > > >
> > > > > > > >> > >>> > > tokens.
> > > > >
> > > > > > > >> > >>> > > > > > Basically anyone can request delegation
> token,
> > > as
> > > > > long
> > > > >
> > > > > > > as
> > > > >
> > > > > > > >> > they
> > > > >
> > > > > > > >> > >>> > > > > authenticate
> > > > >
> > > > > > > >> > >>> > > > > > via a non delegation token mechanism.
> > > > >
> > > > > > > >> > >>> > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > Aren't classes listed here
> > > > >
> > > > > > > >> > >>> > > > > > <
> > > > >
> > > > > > > >> > >>> > > > > >
> > > > >
> > > > > > > >> > >>> > > > >
> > > > >
> > > > > > > >> > >>> > > > https://cwiki.apache.org/confl
> > > > uence/display/KAFKA/KIP-
> > > > >
> > > > > > > >> > >>> > > 48+Delegation+token+support+fo
> > > > >
> > > > > > r+Kafka#KIP-48Delegationtokens
> > > > >
> > > > > > > >> > >>> upportforKaf
> > > > >
> > > > > > > >> > >>> > > ka-PublicInterfaces
> > > > >
> > > > > > > >> > >>> > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > sufficient?
> > > > >
> > > > > > > >> > >>> > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > Thanks
> > > > >
> > > > > > > >> > >>> > > > > > Parth
> > > > >
> > > > > > > >> > >>> > > > > >
> > > > >
> > > > > > > >> > >>> > > > > >
> > > > >
> > > > > > > >> > >>> > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao
> > > > >
> > > > > > > >> > >>> > > > > > <ju...@confluent.io>
> > > > >
> > > > > > > >> > >>> wrote:
> > > > >
> > > > > > > >> > >>> > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > Parth,
> > > > >
> > > > > > > >> > >>> > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > Thanks for the reply. A couple of comments
> > > > inline
> > > > >
> > > > > > > below.
> > > > >
> > > > > > > >> > >>> > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > On Tue, Jun 21, 2016 at 10:36 AM, parth
> > > > > brahmbhatt <
> > > > >
> > > > > > > >> > >>> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > >
> > > > > > > >> > >>> > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > 1. Who / how are tokens renewed? By
> > original
> > > > >
> > > > > > > requester
> > > > >
> > > > > > > >> > >>> only? or
> > > > >
> > > > > > > >> > >>> > > > using
> > > > >
> > > > > > > >> > >>> > > > > > > > Kerberos
> > > > >
> > > > > > > >> > >>> > > > > > > > auth only?
> > > > >
> > > > > > > >> > >>> > > > > > > > My recommendation is to do this only
> using
> > > > >
> > > > > > Kerberos
> > > > >
> > > > > > > >> > >>> > > > > > > > auth
> > > > >
> > > > > > > >> > and
> > > > >
> > > > > > > >> > >>> > only
> > > > >
> > > > > > > >> > >>> > > > > threw
> > > > >
> > > > > > > >> > >>> > > > > > > the
> > > > >
> > > > > > > >> > >>> > > > > > > > renewer specified during the acquisition
> > > > > request.
> > > > >
> > > > > > > >> > >>> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > Hmm, not sure that I follow this. Are you
> > > saying
> > > > >
> > > > > > that
> > > > >
> > > > > > > >> > >>> > > > > > > any
> > > > >
> > > > > > > >> > >>> client
> > > > >
> > > > > > > >> > >>> > > > > > > authenticated with the delegation token
> can
> > > > renew,
> > > > >
> > > > > > > i.e.
> > > > >
> > > > > > > >> > there
> > > > >
> > > > > > > >> > >>> is
> > > > >
> > > > > > > >> > >>> > no
> > > > >
> > > > > > > >> > >>> > > > > > renewer
> > > > >
> > > > > > > >> > >>> > > > > > > needed?
> > > > >
> > > > > > > >> > >>> > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > Also, just to be clear, any authenticated
> > > client
> > > > >
> > > > > > > (either
> > > > >
> > > > > > > >> > >>> through
> > > > >
> > > > > > > >> > >>> > > SASL
> > > > >
> > > > > > > >> > >>> > > > > or
> > > > >
> > > > > > > >> > >>> > > > > > > SSL) can request a delegation token for
> the
> > > > >
> > > > > > > >> > >>> > > > > > > authenticated
> > > > >
> > > > > > > >> > >>> user,
> > > > >
> > > > > > > >> > >>> > > > right?
> > > > >
> > > > > > > >> > >>> > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > 2. Are tokens stored on each broker or
> in
> > > ZK?
> > > > >
> > > > > > > >> > >>> > > > > > > > My recommendation is still to store in
> ZK
> > or
> > > > not
> > > > >
> > > > > > > store
> > > > >
> > > > > > > >> > them
> > > > >
> > > > > > > >> > >>> at
> > > > >
> > > > > > > >> > >>> > > all.
> > > > >
> > > > > > > >> > >>> > > > > The
> > > > >
> > > > > > > >> > >>> > > > > > > > whole controller based distribution is
> too
> > > > much
> > > > >
> > > > > > > >> > >>> > > > > > > > overhead
> > > > >
> > > > > > > >> > >>> with
> > > > >
> > > > > > > >> > >>> > not
> > > > >
> > > > > > > >> > >>> > > > > much
> > > > >
> > > > > > > >> > >>> > > > > > to
> > > > >
> > > > > > > >> > >>> > > > > > > > achieve.
> > > > >
> > > > > > > >> > >>> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > 3. How are tokens invalidated / expired?
> > > > >
> > > > > > > >> > >>> > > > > > > > Either by expiration time out or through
> > an
> > > > >
> > > > > > explicit
> > > > >
> > > > > > > >> > >>> request to
> > > > >
> > > > > > > >> > >>> > > > > > > invalidate.
> > > > >
> > > > > > > >> > >>> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > 4. Which encryption algorithm is used?
> > > > >
> > > > > > > >> > >>> > > > > > > > SCRAM
> > > > >
> > > > > > > >> > >>> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > 5. What is the impersonation proposal
> (it
> > > > wasn't
> > > > >
> > > > > > in
> > > > >
> > > > > > > >> > >>> > > > > > > > the
> > > > >
> > > > > > > >> > KIP
> > > > >
> > > > > > > >> > >>> but
> > > > >
> > > > > > > >> > >>> > > was
> > > > >
> > > > > > > >> > >>> > > > > > > > discussed
> > > > >
> > > > > > > >> > >>> > > > > > > > in this thread)?
> > > > >
> > > > > > > >> > >>> > > > > > > > There is no imperonation proposal. I
> tried
> > > and
> > > > >
> > > > > > > >> > >>> > > > > > > > explained
> > > > >
> > > > > > > >> > how
> > > > >
> > > > > > > >> > >>> > its
> > > > >
> > > > > > > >> > >>> > > a
> > > > >
> > > > > > > >> > >>> > > > > > > > different problem and why its not really
> > > > > necessary
> > > > >
> > > > > > > to
> > > > >
> > > > > > > >> > >>> discuss
> > > > >
> > > > > > > >> > >>> > > that
> > > > >
> > > > > > > >> > >>> > > > as
> > > > >
> > > > > > > >> > >>> > > > > > > part
> > > > >
> > > > > > > >> > >>> > > > > > > > of this KIP.  This KIP will not support
> > any
> > > > >
> > > > > > > >> > impersonation,
> > > > >
> > > > > > > >> > >>> it
> > > > >
> > > > > > > >> > >>> > > will
> > > > >
> > > > > > > >> > >>> > > > > just
> > > > >
> > > > > > > >> > >>> > > > > > > be
> > > > >
> > > > > > > >> > >>> > > > > > > > another way to authenticate.
> > > > >
> > > > > > > >> > >>> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > 6. Do we need new ACLs, if so - for what
> > > > > actions?
> > > > >
> > > > > > > >> > >>> > > > > > > > We do not need new ACLs.
> > > > >
> > > > > > > >> > >>> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > Could we document the format of the new
> > > > >
> > > > > > > request/response
> > > > >
> > > > > > > >> > and
> > > > >
> > > > > > > >> > >>> > their
> > > > >
> > > > > > > >> > >>> > > > > > > associated Resource and Operation for ACL?
> > > > >
> > > > > > > >> > >>> > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > 7. How would the delegation token be
> > > > configured
> > > > > in
> > > > >
> > > > > > > the
> > > > >
> > > > > > > >> > >>> client?
> > > > >
> > > > > > > >> > >>> > > > > > > > Should be through config. I wasn't
> > planning
> > > on
> > > > >
> > > > > > > >> > >>> > > > > > > > supporting
> > > > >
> > > > > > > >> > >>> JAAS
> > > > >
> > > > > > > >> > >>> > > for
> > > > >
> > > > > > > >> > >>> > > > > > > tokens.
> > > > >
> > > > > > > >> > >>> > > > > > > > I don't believe hadoop does this either.
> > > > >
> > > > > > > >> > >>> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > Thanks
> > > > >
> > > > > > > >> > >>> > > > > > > > Parth
> > > > >
> > > > > > > >> > >>> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun
> Rao <
> > > > >
> > > > > > > >> > jun@confluent.io>
> > > > >
> > > > > > > >> > >>> > > wrote:
> > > > >
> > > > > > > >> > >>> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > Harsha,
> > > > >
> > > > > > > >> > >>> > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > Another question.
> > > > >
> > > > > > > >> > >>> > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > 9. How would the delegation token be
> > > > > configured
> > > > >
> > > > > > in
> > > > >
> > > > > > > >> > >>> > > > > > > > > the
> > > > >
> > > > > > > >> > >>> > client?
> > > > >
> > > > > > > >> > >>> > > > The
> > > > >
> > > > > > > >> > >>> > > > > > > > standard
> > > > >
> > > > > > > >> > >>> > > > > > > > > way is to do this through JAAS.
> However,
> > > we
> > > > > will
> > > > >
> > > > > > > >> > >>> > > > > > > > > need
> > > > >
> > > > > > > >> > to
> > > > >
> > > > > > > >> > >>> > think
> > > > >
> > > > > > > >> > >>> > > > > > through
> > > > >
> > > > > > > >> > >>> > > > > > > if
> > > > >
> > > > > > > >> > >>> > > > > > > > > this is convenient in a shared
> > > environment.
> > > > > For
> > > > >
> > > > > > > >> > example,
> > > > >
> > > > > > > >> > >>> > when a
> > > > >
> > > > > > > >> > >>> > > > new
> > > > >
> > > > > > > >> > >>> > > > > > > task
> > > > >
> > > > > > > >> > >>> > > > > > > > is
> > > > >
> > > > > > > >> > >>> > > > > > > > > added to a Storm worker node, do we
> need
> > > to
> > > > >
> > > > > > > >> > >>> > > > > > > > > dynamically
> > > > >
> > > > > > > >> > >>> add a
> > > > >
> > > > > > > >> > >>> > > new
> > > > >
> > > > > > > >> > >>> > > > > > > section
> > > > >
> > > > > > > >> > >>> > > > > > > > > in the JAAS file? It may be more
> > > convenient
> > > > if
> > > > >
> > > > > > we
> > > > >
> > > > > > > >> > >>> > > > > > > > > can
> > > > >
> > > > > > > >> > >>> pass in
> > > > >
> > > > > > > >> > >>> > > the
> > > > >
> > > > > > > >> > >>> > > > > > token
> > > > >
> > > > > > > >> > >>> > > > > > > > > through the config directly w/o going
> > > > through
> > > > >
> > > > > > > JAAS.
> > > > >
> > > > > > > >> > >>> > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > Are you or Parth still actively
> working
> > on
> > > > > this
> > > > >
> > > > > > > KIP?
> > > > >
> > > > > > > >> > >>> > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > Thanks,
> > > > >
> > > > > > > >> > >>> > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > Jun
> > > > >
> > > > > > > >> > >>> > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun
> > Rao <
> > > > >
> > > > > > > >> > >>> jun@confluent.io>
> > > > >
> > > > > > > >> > >>> > > > wrote:
> > > > >
> > > > > > > >> > >>> > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > > Just to add on that list.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > > 2. It would be good to document the
> > > format
> > > > > of
> > > > >
> > > > > > > the
> > > > >
> > > > > > > >> > data
> > > > >
> > > > > > > >> > >>> > stored
> > > > >
> > > > > > > >> > >>> > > > in
> > > > >
> > > > > > > >> > >>> > > > > > ZK.
> > > > >
> > > > > > > >> > >>> > > > > > > > > > 7. Earlier, there was a discussion
> on
> > > > > whether
> > > > >
> > > > > > > the
> > > > >
> > > > > > > >> > tokens
> > > > >
> > > > > > > >> > >>> > > should
> > > > >
> > > > > > > >> > >>> > > > > be
> > > > >
> > > > > > > >> > >>> > > > > > > > > > propagated through ZK like
> > > > config/acl/quota,
> > > > >
> > > > > > or
> > > > >
> > > > > > > >> > through
> > > > >
> > > > > > > >> > >>> the
> > > > >
> > > > > > > >> > >>> > > > > > > controller.
> > > > >
> > > > > > > >> > >>> > > > > > > > > > Currently, the controller is only
> > > designed
> > > > > for
> > > > >
> > > > > > > >> > >>> propagating
> > > > >
> > > > > > > >> > >>> > > > topic
> > > > >
> > > > > > > >> > >>> > > > > > > > > metadata,
> > > > >
> > > > > > > >> > >>> > > > > > > > > > but not other data.
> > > > >
> > > > > > > >> > >>> > > > > > > > > > 8. Should we use SCRAM to send the
> > token
> > > > >
> > > > > > instead
> > > > >
> > > > > > > >> > >>> > > > > > > > > > of
> > > > >
> > > > > > > >> > >>> > > DIGEST-MD5
> > > > >
> > > > > > > >> > >>> > > > > > since
> > > > >
> > > > > > > >> > >>> > > > > > > > it's
> > > > >
> > > > > > > >> > >>> > > > > > > > > > deprecated?
> > > > >
> > > > > > > >> > >>> > > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > > Also, the images in the wiki seem
> > > broken.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > > Thanks,
> > > > >
> > > > > > > >> > >>> > > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > > Jun
> > > > >
> > > > > > > >> > >>> > > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > > On Fri, Jun 10, 2016 at 10:02 AM,
> Gwen
> > > > >
> > > > > > Shapira <
> > > > >
> > > > > > > >> > >>> > > > > gwen@confluent.io>
> > > > >
> > > > > > > >> > >>> > > > > > > > > wrote:
> > > > >
> > > > > > > >> > >>> > > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> From what I can see, remaining
> > > questions
> > > > > are:
> > > > >
> > > > > > > >> > >>> > > > > > > > > >>
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> 1. Who / how are tokens renewed? By
> > > > > original
> > > > >
> > > > > > > >> > requester
> > > > >
> > > > > > > >> > >>> > only?
> > > > >
> > > > > > > >> > >>> > > > or
> > > > >
> > > > > > > >> > >>> > > > > > > using
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> Kerberos auth only?
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> 2. Are tokens stored on each broker
> > or
> > > in
> > > > > ZK?
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> 3. How are tokens invalidated /
> > > expired?
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> 4. Which encryption algorithm is
> > used?
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> 5. What is the impersonation
> proposal
> > > (it
> > > > >
> > > > > > > wasn't
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> in
> > > > >
> > > > > > > >> > the
> > > > >
> > > > > > > >> > >>> > KIP
> > > > >
> > > > > > > >> > >>> > > > but
> > > > >
> > > > > > > >> > >>> > > > > > was
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> discussed in this thread)?
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> 6. Do we need new ACLs, if so - for
> > > what
> > > > >
> > > > > > > actions?
> > > > >
> > > > > > > >> > >>> > > > > > > > > >>
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> Gwen
> > > > >
> > > > > > > >> > >>> > > > > > > > > >>
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM,
> > Harsha
> > > <
> > > > >
> > > > > > > >> > >>> kafka@harsha.io>
> > > > >
> > > > > > > >> > >>> > > > wrote:
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> > Jun & Ismael,
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >
> > > Unfortunately
> > > > I
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> > couldn't
> > > > >
> > > > > > > >> > >>> attend
> > > > >
> > > > > > > >> > >>> > > the
> > > > >
> > > > > > > >> > >>> > > > > KIP
> > > > >
> > > > > > > >> > >>> > > > > > > > > meeting
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >                          when
> > > > delegation
> > > > >
> > > > > > > tokens
> > > > >
> > > > > > > >> > >>> > discussed.
> > > > >
> > > > > > > >> > >>> > > > > > > > Appreciate
> > > > >
> > > > > > > >> > >>> > > > > > > > > if
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >                          you can
> > > update
> > > > > the
> > > > >
> > > > > > > >> > thread if
> > > > >
> > > > > > > >> > >>> > you
> > > > >
> > > > > > > >> > >>> > > > have
> > > > >
> > > > > > > >> > >>> > > > > > any
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >                          further
> > > > > questions.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> > Thanks,
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> > Harsha
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> > On Tue, May 24, 2016, at 11:32
> AM,
> > > > Liquan
> > > > >
> > > > > > Pei
> > > > >
> > > > > > > >> > wrote:
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> It seems that the links to
> images
> > in
> > > > the
> > > > >
> > > > > > KIP
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> are
> > > > >
> > > > > > > >> > >>> > broken.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >>
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> Liquan
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >>
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> On Tue, May 24, 2016 at 9:33 AM,
> > > parth
> > > > >
> > > > > > > >> > brahmbhatt <
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> brahmbhatt.parth@gmail.com>
> > wrote:
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >>
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > 110. What does
> > > getDelegationTokenAs
> > > > >
> > > > > > mean?
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > In the current proposal we
> only
> > > > allow
> > > > > a
> > > > >
> > > > > > > user
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > to
> > > > >
> > > > > > > >> > >>> get
> > > > >
> > > > > > > >> > >>> > > > > > delegation
> > > > >
> > > > > > > >> > >>> > > > > > > > > token
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> for
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > the identity that it
> > authenticated
> > > > as
> > > > >
> > > > > > > using
> > > > >
> > > > > > > >> > >>> another
> > > > >
> > > > > > > >> > >>> > > > > > mechanism,
> > > > >
> > > > > > > >> > >>> > > > > > > > i.e.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> A user
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > that authenticate using a
> keytab
> > > for
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > principal
> > > > >
> > > > > > > >> > >>> > > > > > > user1@EXAMPLE.COM
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> will get
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens for that
> user
> > > > only.
> > > > > In
> > > > >
> > > > > > > >> > future I
> > > > >
> > > > > > > >> > >>> > think
> > > > >
> > > > > > > >> > >>> > > > we
> > > > >
> > > > > > > >> > >>> > > > > > will
> > > > >
> > > > > > > >> > >>> > > > > > > > > have
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> to
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > extend support such that we
> > allow
> > > > some
> > > > >
> > > > > > set
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > of
> > > > >
> > > > > > > >> > >>> users (
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > kafka-rest-user@EXAMPLE.COM,
> > > > >
> > > > > > > >> > >>> > storm-nimbus@EXAMPLE.COM)
> > > > >
> > > > > > > >> > >>> > > > to
> > > > >
> > > > > > > >> > >>> > > > > > > > acquire
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens on behalf of
> > > other
> > > > >
> > > > > > users
> > > > >
> > > > > > > >> > whose
> > > > >
> > > > > > > >> > >>> > > identity
> > > > >
> > > > > > > >> > >>> > > > > > they
> > > > >
> > > > > > > >> > >>> > > > > > > > have
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > verified independently.  Kafka
> > > > brokers
> > > > >
> > > > > > > will
> > > > >
> > > > > > > >> > have
> > > > >
> > > > > > > >> > >>> ACLs
> > > > >
> > > > > > > >> > >>> > > to
> > > > >
> > > > > > > >> > >>> > > > > > > control
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> which
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > users are allowed to
> impersonate
> > > > other
> > > > >
> > > > > > > users
> > > > >
> > > > > > > >> > and
> > > > >
> > > > > > > >> > >>> get
> > > > >
> > > > > > > >> > >>> > > > tokens
> > > > >
> > > > > > > >> > >>> > > > > > on
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> behalf of
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > them. Overall Impersonation
> is a
> > > > whole
> > > > >
> > > > > > > >> > different
> > > > >
> > > > > > > >> > >>> > > problem
> > > > >
> > > > > > > >> > >>> > > > in
> > > > >
> > > > > > > >> > >>> > > > > > my
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> opinion and
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > I think we can tackle it in
> > > separate
> > > > >
> > > > > > KIP.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > 111. What's the typical rate
> of
> > > > > getting
> > > > >
> > > > > > > and
> > > > >
> > > > > > > >> > >>> renewing
> > > > >
> > > > > > > >> > >>> > > > > > delegation
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> tokens?
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > Typically this should be very
> > very
> > > > > low,
> > > > >
> > > > > > 1
> > > > >
> > > > > > > >> > request
> > > > >
> > > > > > > >> > >>> per
> > > > >
> > > > > > > >> > >>> > > > > minute
> > > > >
> > > > > > > >> > >>> > > > > > > is a
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > relatively high estimate.
> > However
> > > it
> > > > >
> > > > > > > depends
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > on
> > > > >
> > > > > > > >> > >>> the
> > > > >
> > > > > > > >> > >>> > > token
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> expiration. I am
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > less worried about the extra
> > load
> > > it
> > > > >
> > > > > > puts
> > > > >
> > > > > > > on
> > > > >
> > > > > > > >> > >>> > controller
> > > > >
> > > > > > > >> > >>> > > > vs
> > > > >
> > > > > > > >> > >>> > > > > > the
> > > > >
> > > > > > > >> > >>> > > > > > > > > added
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > complexity and the value it
> > > offers.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > Thanks
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > Parth
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > On Tue, May 24, 2016 at 7:30
> AM,
> > > > > Ismael
> > > > >
> > > > > > > Juma
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > <
> > > > >
> > > > > > > >> > >>> > > > > > > ismael@juma.me.uk>
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> wrote:
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > Thanks Rajini. It would
> > probably
> > > > >
> > > > > > > require a
> > > > >
> > > > > > > >> > >>> separate
> > > > >
> > > > > > > >> > >>> > > KIP
> > > > >
> > > > > > > >> > >>> > > > > as
> > > > >
> > > > > > > >> > >>> > > > > > it
> > > > >
> > > > > > > >> > >>> > > > > > > > > will
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > introduce user visible
> > changes.
> > > We
> > > > >
> > > > > > could
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > also
> > > > >
> > > > > > > >> > >>> > update
> > > > >
> > > > > > > >> > >>> > > > > KIP-48
> > > > >
> > > > > > > >> > >>> > > > > > > to
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> have this
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > information, but it seems
> > > cleaner
> > > > to
> > > > >
> > > > > > do
> > > > >
> > > > > > > it
> > > > >
> > > > > > > >> > >>> > > separately.
> > > > >
> > > > > > > >> > >>> > > > We
> > > > >
> > > > > > > >> > >>> > > > > > can
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> discuss
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > that
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > in the KIP call today.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > Ismael
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > On Tue, May 24, 2016 at 3:19
> > PM,
> > > > >
> > > > > > Rajini
> > > > >
> > > > > > > >> > Sivaram
> > > > >
> > > > > > > >> > >>> <
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > >
> rajinisivaram@googlemail.com>
> > > > > wrote:
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > Ismael,
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > I have created a JIRA (
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > https://issues.apache.org/
> > > > >
> > > > > > > >> > jira/browse/KAFKA-3751)
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > for adding SCRAM as a SASL
> > > > >
> > > > > > mechanism.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > Would
> > > > >
> > > > > > > >> > >>> that
> > > > >
> > > > > > > >> > >>> > > need
> > > > >
> > > > > > > >> > >>> > > > > > > another
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> KIP? If
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > KIP-48 will use this
> > > mechanism,
> > > > > can
> > > > >
> > > > > > > this
> > > > >
> > > > > > > >> > just
> > > > >
> > > > > > > >> > >>> be
> > > > >
> > > > > > > >> > >>> > a
> > > > >
> > > > > > > >> > >>> > > > JIRA
> > > > >
> > > > > > > >> > >>> > > > > > > that
> > > > >
> > > > > > > >> > >>> > > > > > > > > gets
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > reviewed
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > when the PR is ready?
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > Thank you,
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > Rajini
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > On Tue, May 24, 2016 at
> 2:46
> > > PM,
> > > > >
> > > > > > > Ismael
> > > > >
> > > > > > > >> > Juma <
> > > > >
> > > > > > > >> > >>> > > > > > > > > ismael@juma.me.uk>
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > wrote:
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > Thanks Rajini, SCRAM
> seems
> > > > like
> > > > > a
> > > > >
> > > > > > > good
> > > > >
> > > > > > > >> > >>> > candidate.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > Gwen had independently
> > > > mentioned
> > > > >
> > > > > > > this
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > as
> > > > >
> > > > > > > >> > a
> > > > >
> > > > > > > >> > >>> SASL
> > > > >
> > > > > > > >> > >>> > > > > > mechanism
> > > > >
> > > > > > > >> > >>> > > > > > > > > that
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> might
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > be
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > useful for Kafka and I
> > have
> > > > been
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > meaning
> > > > >
> > > > > > > >> > to
> > > > >
> > > > > > > >> > >>> > check
> > > > >
> > > > > > > >> > >>> > > > it
> > > > >
> > > > > > > >> > >>> > > > > in
> > > > >
> > > > > > > >> > >>> > > > > > > > more
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> detail.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > Good
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > to know that you are
> > willing
> > > > to
> > > > >
> > > > > > > >> > contribute
> > > > >
> > > > > > > >> > >>> an
> > > > >
> > > > > > > >> > >>> > > > > > > > implementation.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> Maybe
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > we
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > should file a separate
> > JIRA
> > > > for
> > > > >
> > > > > > > this?
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > Ismael
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > On Tue, May 24, 2016 at
> > 2:12
> > > > PM,
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > Rajini
> > > > >
> > > > > > > >> > >>> > Sivaram <
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > >
> > > rajinisivaram@googlemail.com>
> > > > >
> > > > > > > wrote:
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > SCRAM (Salted
> Challenge
> > > > > Response
> > > > >
> > > > > > > >> > >>> > Authentication
> > > > >
> > > > > > > >> > >>> > > > > > > > Mechanism)
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> is a
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > better
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > mechanism than
> > Digest-MD5.
> > > > > Java
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > doesn't
> > > > >
> > > > > > > >> > >>> come
> > > > >
> > > > > > > >> > >>> > > > with a
> > > > >
> > > > > > > >> > >>> > > > > > > > > built-in
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> SCRAM
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > SaslServer or
> > SaslClient,
> > > > but
> > > > > I
> > > > >
> > > > > > > will
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > be
> > > > >
> > > > > > > >> > >>> happy
> > > > >
> > > > > > > >> > >>> > > to
> > > > >
> > > > > > > >> > >>> > > > > add
> > > > >
> > > > > > > >> > >>> > > > > > > > > support
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> in
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > Kafka
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > since
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > it would be a useful
> > > > mechanism
> > > > >
> > > > > > to
> > > > >
> > > > > > > >> > support
> > > > >
> > > > > > > >> > >>> > > anyway.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > >
> > > > https://tools.ietf.org/html/
> > > > >
> > > > > > > rfc7677
> > > > >
> > > > > > > >> > >>> > describes
> > > > >
> > > > > > > >> > >>> > > > the
> > > > >
> > > > > > > >> > >>> > > > > > > > protocol
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> for
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > SCRAM-SHA-256.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > On Tue, May 24, 2016
> at
> > > 2:37
> > > > > AM,
> > > > >
> > > > > > > Jun
> > > > >
> > > > > > > >> > Rao <
> > > > >
> > > > > > > >> > >>> > > > > > > > jun@confluent.io
> > > > >
> > > > > > > >> > >>> > > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> wrote:
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > Parth,
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > Thanks for the
> > > > explanation.
> > > > > A
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > couple
> > > > >
> > > > > > > >> > of
> > > > >
> > > > > > > >> > >>> > more
> > > > >
> > > > > > > >> > >>> > > > > > > questions.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > 110. What does
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > getDelegationTokenAs
> > > > >
> > > > > > > >> > >>> mean?
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > 111. What's the
> > typical
> > > > rate
> > > > >
> > > > > > of
> > > > >
> > > > > > > >> > getting
> > > > >
> > > > > > > >> > >>> and
> > > > >
> > > > > > > >> > >>> > > > > > renewing
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> delegation
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > tokens?
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > That may have an
> > impact
> > > on
> > > > >
> > > > > > > whether
> > > > >
> > > > > > > >> > they
> > > > >
> > > > > > > >> > >>> > > should
> > > > >
> > > > > > > >> > >>> > > > be
> > > > >
> > > > > > > >> > >>> > > > > > > > > directed
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> to the
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > controller.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > Jun
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > On Mon, May 23, 2016
> > at
> > > > 1:19
> > > > >
> > > > > > PM,
> > > > >
> > > > > > > >> > parth
> > > > >
> > > > > > > >> > >>> > > > > brahmbhatt <
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > > brahmbhatt.parth@gmail.com>
> > > > >
> > > > > > > wrote:
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Hi Jun,
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Thanks for
> > reviewing.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * We could add a
> > > Cluster
> > > > >
> > > > > > > action
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > to
> > > > >
> > > > > > > >> > add
> > > > >
> > > > > > > >> > >>> > acls
> > > > >
> > > > > > > >> > >>> > > > on
> > > > >
> > > > > > > >> > >>> > > > > > who
> > > > >
> > > > > > > >> > >>> > > > > > > > can
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> request
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > delegation
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > tokens. I don't
> see
> > > the
> > > > > use
> > > > >
> > > > > > > case
> > > > >
> > > > > > > >> > for
> > > > >
> > > > > > > >> > >>> that
> > > > >
> > > > > > > >> > >>> > > yet
> > > > >
> > > > > > > >> > >>> > > > > but
> > > > >
> > > > > > > >> > >>> > > > > > > > down
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> the line
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > when
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > we
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > start supporting
> > > > >
> > > > > > > >> > getDelegationTokenAs
> > > > >
> > > > > > > >> > >>> it
> > > > >
> > > > > > > >> > >>> > > will
> > > > >
> > > > > > > >> > >>> > > > > be
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> necessary.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * Yes we recommend
> > > > tokens
> > > > > to
> > > > >
> > > > > > > be
> > > > >
> > > > > > > >> > only
> > > > >
> > > > > > > >> > >>> > > > > > > used/distributed
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> over
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > secure
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > channels.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * Depending on
> what
> > > > design
> > > > >
> > > > > > we
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > end
> > > > >
> > > > > > > >> > up
> > > > >
> > > > > > > >> > >>> > > choosing
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> Invalidation will
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > be
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > responsibility of
> > > every
> > > > >
> > > > > > broker
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > or
> > > > >
> > > > > > > >> > >>> > > controller.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * I am not sure
> if I
> > > > >
> > > > > > > documented
> > > > >
> > > > > > > >> > >>> somewhere
> > > > >
> > > > > > > >> > >>> > > > that
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> invalidation
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > will
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > directly
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > go through
> zookeeper
> > > but
> > > > >
> > > > > > that
> > > > >
> > > > > > > is
> > > > >
> > > > > > > >> > not
> > > > >
> > > > > > > >> > >>> the
> > > > >
> > > > > > > >> > >>> > > > > intent.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> Invalidation
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > will
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > either
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > be request based
> or
> > > due
> > > > to
> > > > >
> > > > > > > >> > >>> expiration. No
> > > > >
> > > > > > > >> > >>> > > > > direct
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> zookeeper
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > interaction
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > from
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > any client.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * "Broker also
> > stores
> > > > the
> > > > >
> > > > > > > >> > >>> DelegationToken
> > > > >
> > > > > > > >> > >>> > > > > without
> > > > >
> > > > > > > >> > >>> > > > > > > the
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> hmac in
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > the
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > zookeeper." :
> Sorry
> > > > about
> > > > >
> > > > > > the
> > > > >
> > > > > > > >> > >>> confusion.
> > > > >
> > > > > > > >> > >>> > > The
> > > > >
> > > > > > > >> > >>> > > > > sole
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> purpose of
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > zookeeper
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > in
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > this design is as
> > > > >
> > > > > > distribution
> > > > >
> > > > > > > >> > channel
> > > > >
> > > > > > > >> > >>> > for
> > > > >
> > > > > > > >> > >>> > > > > tokens
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> between all
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > brokers
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > and a
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > layer that ensures
> > > only
> > > > >
> > > > > > tokens
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > that
> > > > >
> > > > > > > >> > >>> were
> > > > >
> > > > > > > >> > >>> > > > > > generated
> > > > >
> > > > > > > >> > >>> > > > > > > by
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> making a
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > request
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > to a
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > broker will be
> > > accepted
> > > > >
> > > > > > (more
> > > > >
> > > > > > > on
> > > > >
> > > > > > > >> > this
> > > > >
> > > > > > > >> > >>> in
> > > > >
> > > > > > > >> > >>> > > > second
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> paragraph). The
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > token
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > consists of few
> > > elements
> > > > >
> > > > > > > (owner,
> > > > >
> > > > > > > >> > >>> renewer,
> > > > >
> > > > > > > >> > >>> > > > uuid
> > > > >
> > > > > > > >> > >>> > > > > ,
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> expiration,
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > hmac)
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > ,
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > one
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > of
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > which is the
> finally
> > > > >
> > > > > > generated
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > hmac
> > > > >
> > > > > > > >> > >>> but
> > > > >
> > > > > > > >> > >>> > > hmac
> > > > >
> > > > > > > >> > >>> > > > it
> > > > >
> > > > > > > >> > >>> > > > > > > self
> > > > >
> > > > > > > >> > >>> > > > > > > > is
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > derivable
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > if
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > you
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > have all the other
> > > > > elements
> > > > >
> > > > > > of
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > the
> > > > >
> > > > > > > >> > >>> token
> > > > >
> > > > > > > >> > >>> > +
> > > > >
> > > > > > > >> > >>> > > > > secret
> > > > >
> > > > > > > >> > >>> > > > > > > key
> > > > >
> > > > > > > >> > >>> > > > > > > > > to
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > generate
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > hmac.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Given zookeeper
> does
> > > not
> > > > >
> > > > > > > provide
> > > > >
> > > > > > > >> > SSL
> > > > >
> > > > > > > >> > >>> > > support
> > > > >
> > > > > > > >> > >>> > > > we
> > > > >
> > > > > > > >> > >>> > > > > > do
> > > > >
> > > > > > > >> > >>> > > > > > > > not
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> want the
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > entire
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > token to be wire
> > > > > transferred
> > > > >
> > > > > > > to
> > > > >
> > > > > > > >> > >>> zookeeper
> > > > >
> > > > > > > >> > >>> > > as
> > > > >
> > > > > > > >> > >>> > > > > that
> > > > >
> > > > > > > >> > >>> > > > > > > > will
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> be an
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > insecure
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > wire
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > transfer. Instead
> we
> > > > only
> > > > >
> > > > > > > store
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > all
> > > > >
> > > > > > > >> > >>> the
> > > > >
> > > > > > > >> > >>> > > other
> > > > >
> > > > > > > >> > >>> > > > > > > > elements
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> of a
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > delegation
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > tokens. Brokers
> can
> > > read
> > > > >
> > > > > > these
> > > > >
> > > > > > > >> > >>> elements
> > > > >
> > > > > > > >> > >>> > and
> > > > >
> > > > > > > >> > >>> > > > > > because
> > > > >
> > > > > > > >> > >>> > > > > > > > > they
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> also
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > have
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > access
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > to secret key they
> > > will
> > > > be
> > > > >
> > > > > > > able
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > to
> > > > >
> > > > > > > >> > >>> > generate
> > > > >
> > > > > > > >> > >>> > > > > hmac
> > > > >
> > > > > > > >> > >>> > > > > > on
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> their end.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > One of the
> > alternative
> > > > >
> > > > > > > proposed
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > is
> > > > >
> > > > > > > >> > to
> > > > >
> > > > > > > >> > >>> > avoid
> > > > >
> > > > > > > >> > >>> > > > > > > zookeeper
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > altogether. A
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > Client
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > will call broker
> > with
> > > > >
> > > > > > required
> > > > >
> > > > > > > >> > >>> > information
> > > > >
> > > > > > > >> > >>> > > > > > (owner,
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> renwer,
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > expiration)
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > and
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > get back (signed
> > hmac,
> > > > >
> > > > > > uuid).
> > > > >
> > > > > > > >> > Broker
> > > > >
> > > > > > > >> > >>> > won't
> > > > >
> > > > > > > >> > >>> > > > > store
> > > > >
> > > > > > > >> > >>> > > > > > > this
> > > > >
> > > > > > > >> > >>> > > > > > > > > in
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > zookeeper.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > From
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > this point a
> client
> > > can
> > > > >
> > > > > > > contact
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > any
> > > > >
> > > > > > > >> > >>> > broker
> > > > >
> > > > > > > >> > >>> > > > with
> > > > >
> > > > > > > >> > >>> > > > > > all
> > > > >
> > > > > > > >> > >>> > > > > > > > the
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > delegation
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > token
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > info (owner,
> rewner,
> > > > >
> > > > > > > expiration,
> > > > >
> > > > > > > >> > hmac,
> > > > >
> > > > > > > >> > >>> > > uuid)
> > > > >
> > > > > > > >> > >>> > > > > the
> > > > >
> > > > > > > >> > >>> > > > > > > > borker
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> will
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > regenerate
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > the
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > hmac and as long
> as
> > it
> > > > >
> > > > > > matches
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > with
> > > > >
> > > > > > > >> > >>> hmac
> > > > >
> > > > > > > >> > >>> > > > > > presented
> > > > >
> > > > > > > >> > >>> > > > > > > by
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> client ,
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > broker
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > will
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > allow the request
> to
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > authenticate.
> > > > >
> > > > > > > >> > >>> Only
> > > > >
> > > > > > > >> > >>> > > > > problem
> > > > >
> > > > > > > >> > >>> > > > > > > with
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> this
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > approach
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > is
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > if
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > the secret key is
> > > > >
> > > > > > compromised
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > any
> > > > >
> > > > > > > >> > >>> client
> > > > >
> > > > > > > >> > >>> > > can
> > > > >
> > > > > > > >> > >>> > > > > now
> > > > >
> > > > > > > >> > >>> > > > > > > > > generate
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > random
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > tokens
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > and
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > they will still be
> > > able
> > > > to
> > > > >
> > > > > > > >> > >>> authenticate
> > > > >
> > > > > > > >> > >>> > as
> > > > >
> > > > > > > >> > >>> > > > any
> > > > >
> > > > > > > >> > >>> > > > > > user
> > > > >
> > > > > > > >> > >>> > > > > > > > > they
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> like.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > with
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > zookeeper we
> > guarantee
> > > > > that
> > > > >
> > > > > > > only
> > > > >
> > > > > > > >> > >>> tokens
> > > > >
> > > > > > > >> > >>> > > > > acquired
> > > > >
> > > > > > > >> > >>> > > > > > > via
> > > > >
> > > > > > > >> > >>> > > > > > > > a
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> broker
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > (using
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > some
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > auth scheme other
> > than
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > delegation
> > > > >
> > > > > > > >> > >>> token)
> > > > >
> > > > > > > >> > >>> > > will
> > > > >
> > > > > > > >> > >>> > > > > be
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> accepted. We
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > need
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > to
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > discuss which
> > proposal
> > > > > makes
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > more
> > > > >
> > > > > > > >> > >>> sense
> > > > >
> > > > > > > >> > >>> > and
> > > > >
> > > > > > > >> > >>> > > > we
> > > > >
> > > > > > > >> > >>> > > > > > can
> > > > >
> > > > > > > >> > >>> > > > > > > go
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> over it
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > in
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > tomorrow's
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > meeting.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Also, can you
> > forward
> > > > the
> > > > >
> > > > > > > invite
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > to
> > > > >
> > > > > > > >> > >>> me?
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Thanks
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Parth
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > On Mon, May 23,
> 2016
> > > at
> > > > >
> > > > > > 10:35
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > AM,
> > > > >
> > > > > > > >> > Jun
> > > > >
> > > > > > > >> > >>> > Rao <
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> jun@confluent.io>
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > wrote:
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > Thanks for the
> > KIP.
> > > A
> > > > > few
> > > > >
> > > > > > > >> > comments.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > 100. This
> > > potentially
> > > > > can
> > > > >
> > > > > > be
> > > > >
> > > > > > > >> > useful
> > > > >
> > > > > > > >> > >>> for
> > > > >
> > > > > > > >> > >>> > > > Kafka
> > > > >
> > > > > > > >> > >>> > > > > > > > Connect
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> and
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > Kafka
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > rest
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > proxy
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > where a worker
> > agent
> > > > > will
> > > > >
> > > > > > > need
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > to
> > > > >
> > > > > > > >> > >>> run a
> > > > >
> > > > > > > >> > >>> > > > task
> > > > >
> > > > > > > >> > >>> > > > > on
> > > > >
> > > > > > > >> > >>> > > > > > > > > behalf
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> of a
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > client.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > We
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > will
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > likely need to
> > > change
> > > > > how
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > those
> > > > >
> > > > > > > >> > >>> > services
> > > > >
> > > > > > > >> > >>> > > > use
> > > > >
> > > > > > > >> > >>> > > > > > > Kafka
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> clients
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > (producer/consumer).
> > > > >
> > > > > > Instead
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > of a
> > > > >
> > > > > > > >> > >>> > shared
> > > > >
> > > > > > > >> > >>> > > > > client
> > > > >
> > > > > > > >> > >>> > > > > > > per
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> worker,
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > we
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > will
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > need
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > a
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > client per user
> > task
> > > > > since
> > > > >
> > > > > > > the
> > > > >
> > > > > > > >> > >>> > > > authentication
> > > > >
> > > > > > > >> > >>> > > > > > > > happens
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> at the
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > connection
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > level. For Kafka
> > > > > Connect,
> > > > >
> > > > > > > the
> > > > >
> > > > > > > >> > >>> renewer
> > > > >
> > > > > > > >> > >>> > > will
> > > > >
> > > > > > > >> > >>> > > > be
> > > > >
> > > > > > > >> > >>> > > > > > the
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> workers.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > So,
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > we
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > probably
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > need to allow
> > > multiple
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > renewers.
> > > > >
> > > > > > > >> > For
> > > > >
> > > > > > > >> > >>> > > Kafka
> > > > >
> > > > > > > >> > >>> > > > > rest
> > > > >
> > > > > > > >> > >>> > > > > > > > > proxy,
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> the
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > renewer
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > can
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > probably just be
> > the
> > > > >
> > > > > > creator
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > of
> > > > >
> > > > > > > >> > the
> > > > >
> > > > > > > >> > >>> > > token.
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > 101. Do we need
> > new
> > > > acl
> > > > > on
> > > > >
> > > > > > > who
> > > > >
> > > > > > > >> > can
> > > > >
> > > > > > > >> > >>> > > request
> > > > >
> > > > > > > >> > >>> > > > > > > > delegation
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> tokens?
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > 102. Do we
> > recommend
> > > > >
> > > > > > people
> > > > >
> > > > > > > to
> > > > >
> > > > > > > >> > send
> > > > >
> > > > > > > >> > >>> > > > > delegation
> > > > >
> > > > > > > >> > >>> > > > > > > > tokens
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> in an
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > encrypted
> > > > >
> > > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > channel?
> > > > >
> > > > > > > >> > >>>
> > > >
> > >
> >
> >
> >
> > --
> > *Gwen Shapira*
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> > <http://www.confluent.io/blog>
> >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Manikumar <ma...@gmail.com>]

That's a good idea. Authenticating every request with delegation token will
be useful for
impersonation use-cases. But as of now, we are thinking delegation token as
just another way
to authenticate the users. We haven't think through all the use cases
related to
impersonation or using delegation token for impersonation. We want to
handle impersonation
(KAFKA-3712) as part of separate KIP.

Will that be Ok?


On Wed, Dec 14, 2016 at 8:09 AM, Gwen Shapira <gw...@confluent.io> wrote:

> Thinking out loud here:
>
> It looks like authentication with a delegation token is going to be
> super-cheap, right? We just compare the token to a value in the broker
> cache?
>
> If I understood the KIP correctly, right now it suggests that
> authentication happens when establishing the client-broker connection (as
> normal for Kafka. But perhaps we want to consider authenticating every
> request with delegation token (if exists)?
>
> So a centralized app can create few producers, do the metadata request and
> broker discovery with its own user auth, but then use delegation tokens to
> allow performing produce/fetch requests as different users? Instead of
> having to re-connect for each impersonated user?
>
> This may over-complicate things quite a bit (basically adding extra
> information in every request), but maybe it will be useful for
> impersonation use-cases (which seem to drive much of the interest in this
> KIP)?
> Kafka Connect, NiFi and friends can probably use this to share clients
> between multiple jobs, tasks, etc.
>
> What do you think?
>
> Gwen
>
> On Tue, Dec 13, 2016 at 12:43 AM, Manikumar <ma...@gmail.com>
> wrote:
>
> > Ashish,
> >
> > Thank you for reviewing the KIP.  Please see the replies inline.
> >
> >
> > > 1. How to disable delegation token authentication?
> > >
> > > This can be achieved in various ways, however I think reusing
> delegation
> > > token secret config for this makes sense here. Avoids creating yet
> > another
> > > config and forces delegation token users to consciously set the secret.
> > If
> > > the secret is not set or set to empty string, brokers should turn off
> > > delegation token support. This will however require a new error code to
> > > indicate delegation token support is turned off on broker.
> > >
> >
> >   Thanks for the suggestion. Option to turnoff delegation token
> > authentication will be useful.
> >   I'll update the KIP.
> >
> >
> > >
> > > 2. ACLs on delegation token?
> > >
> > > Do we need to have ACLs defined for tokens? I do not think it buys us
> > > anything, as delegation token can be treated as impersonation of the
> > owner.
> > > Any thing the owner has permission to do, delegation tokens should be
> > > allowed to do as well. If so, we probably won't need to return
> > > authorization exception error code while creating delegation token. It
> > > however would make sense to check renew and expire requests are coming
> > from
> > > owner or renewers of the token, but that does not require explicit
> acls.
> > >
> >
> >
> > Yes, We agreed to not have new acl on who can request delegation token.
> >  I'll update the KIP.
> >
> >
> > >
> > > 3. How to restrict max life time of a token?
> > >
> > > Admins might want to restrict max life time of tokens created on a
> > cluster,
> > > and this can very from cluster to cluster based on use-cases. This
> might
> > > warrant a separate broker config.
> > >
> > >
> > Currently we  have "delegation.token.max.lifetime.sec" server config
> > property
> > May be we can take min(User supplied MaxTime, Server MaxTime) as max life
> > time.
> > I am open to add new config property.
> >
> > Few more comments based on recent KIP update.
> > >
> > > 1. Do we need a separate {{InvalidateTokenRequest}}? Can't we use
> > > {{ExpireTokenRequest}} with with expiryDate set to anything before
> > current
> > > date?
> > >
> >
> > makes sense. we don't need special request to cancel the token. We can
> use
> > ExpireTokenRequest.
> > I'll update the KIP.
> >
> >
> > > 2. Can we change time field names to indicate their unit is
> milliseconds,
> > > like, IssueDateMs, ExpiryDateMs, etc.?
> > >
> > >
> >   Done.
> >
> >
> > > 3. Can we allow users to renew a token for a specified amount of time?
> In
> > > current version of KIP, renew request does not take time as a param,
> not
> > > sure what is expiry time set to after renewal.
> > >
> > >
> >  Yes, we need to specify renew period.  I'll update the KIP.
> >
> >
> > Thanks,
> > Mankumar
> >
> >
> >
> > >
> > > On Mon, Dec 12, 2016 at 9:08 AM Manikumar <ma...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > > >
> > > > I would like to reinitiate the discussion on Delegation token support
> > for
> > > >
> > > > Kafka.
> > > >
> > > >
> > > >
> > > > Brief summary of the past discussion:
> > > >
> > > >
> > > >
> > > > 1) Broker stores delegation tokens in zookeeper.  All brokers will
> > have a
> > > >
> > > > cache backed by
> > > >
> > > >    zookeeper so they will all get notified whenever a new token is
> > > >
> > > > generated and they will
> > > >
> > > >    update their local cache whenever token state changes.
> > > >
> > > > 2) The current proposal does not support rotation of secret
> > > >
> > > > 3) Only allow the renewal by users that authenticated using *non*
> > > >
> > > > delegation token mechanism
> > > >
> > > > 4) KIP-84 proposes to support  SASL SCRAM mechanisms. Kafka clients
> can
> > > >
> > > > authenticate using
> > > >
> > > >    SCRAM-SHA-256, providing the delegation token HMAC as password.
> > > >
> > > >
> > > >
> > > > Updated the KIP with the following:
> > > >
> > > > 1. Protocol and Config changes
> > > >
> > > > 2. format of the data stored in ZK.
> > > >
> > > > 3. Changes to Java Clients/Usage of SASL SCRAM mechanism
> > > >
> > > >
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >
> > > > 48+Delegation+token+support+for+Kafka
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Jun, Ashish, Gwen,
> > > >
> > > >
> > > >
> > > > Pl review the updated KIP.
> > > >
> > > >
> > > >
> > > > Thanks
> > > >
> > > > Manikumar
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Sep 29, 2016 at 9:56 PM, Ashish Singh <as...@cloudera.com>
> > > wrote:
> > > >
> > > >
> > > >
> > > > > Harsha/ Gwen,
> > > >
> > > > >
> > > >
> > > > > How do we proceed here? I am willing to help out with here.
> > > >
> > > > >
> > > >
> > > > > On Fri, Sep 23, 2016 at 11:41 AM, Gwen Shapira <gw...@confluent.io>
> > > > wrote:
> > > >
> > > > >
> > > >
> > > > > > Is it updated? are all concerns addressed? do you want to start a
> > > vote?
> > > >
> > > > > >
> > > >
> > > > > > Sorry for being pushy, I do appreciate that we are all volunteers
> > and
> > > >
> > > > > > finding time is difficult. This feature is important for anything
> > > that
> > > >
> > > > > > integrates with Kafka (stream processors, Flume, NiFi, etc) and I
> > > >
> > > > > > don't want to see this getting stuck because we lack coordination
> > > >
> > > > > > within the community.
> > > >
> > > > > >
> > > >
> > > > > > On Thu, Sep 15, 2016 at 6:39 PM, Harsha Chintalapani <
> > > kafka@harsha.io>
> > > >
> > > > > > wrote:
> > > >
> > > > > > > The only pending update for the KIP is to write up the protocol
> > > > changes
> > > >
> > > > > > like
> > > >
> > > > > > > we've it KIP-4. I'll update the wiki.
> > > >
> > > > > > >
> > > >
> > > > > > >
> > > >
> > > > > > > On Thu, Sep 15, 2016 at 4:27 PM Ashish Singh <
> > asingh@cloudera.com>
> > > >
> > > > > > wrote:
> > > >
> > > > > > >>
> > > >
> > > > > > >> I think we decided to not support secret rotation, I guess
> this
> > > can
> > > > be
> > > >
> > > > > > >> stated clearly on the KIP. Also, more details on how clients
> > will
> > > >
> > > > > > perform
> > > >
> > > > > > >> token distribution and how CLI will look like will be helpful.
> > > >
> > > > > > >>
> > > >
> > > > > > >> On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <
> > gwen@confluent.io>
> > > >
> > > > > > wrote:
> > > >
> > > > > > >>
> > > >
> > > > > > >> > Hi Guys,
> > > >
> > > > > > >> >
> > > >
> > > > > > >> > This discussion was dead for a while. Are there still
> > > contentious
> > > >
> > > > > > >> > points? If not, why are there no votes?
> > > >
> > > > > > >> >
> > > >
> > > > > > >> > On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <ju...@confluent.io>
> > > > wrote:
> > > >
> > > > > > >> > > Ashish,
> > > >
> > > > > > >> > >
> > > >
> > > > > > >> > > Yes, I will send out a KIP invite for next week to discuss
> > > > KIP-48
> > > >
> > > > > > and
> > > >
> > > > > > >> > other
> > > >
> > > > > > >> > > remaining KIPs.
> > > >
> > > > > > >> > >
> > > >
> > > > > > >> > > Thanks,
> > > >
> > > > > > >> > >
> > > >
> > > > > > >> > > Jun
> > > >
> > > > > > >> > >
> > > >
> > > > > > >> > > On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh <
> > > >
> > > > > asingh@cloudera.com>
> > > >
> > > > > > >> > wrote:
> > > >
> > > > > > >> > >
> > > >
> > > > > > >> > >> Thanks Harsha!
> > > >
> > > > > > >> > >>
> > > >
> > > > > > >> > >> Jun, can we add KIP-48 to next KIP hangout's agenda.
> Also,
> > we
> > > > did
> > > >
> > > > > > not
> > > >
> > > > > > >> > >> actually make a call on when we should have next KIP
> call.
> > As
> > > >
> > > > > there
> > > >
> > > > > > >> > >> are
> > > >
> > > > > > >> > a
> > > >
> > > > > > >> > >> few outstanding KIPs that could not be discussed this
> week,
> > > can
> > > >
> > > > > we
> > > >
> > > > > > >> > >> have
> > > >
> > > > > > >> > a
> > > >
> > > > > > >> > >> KIP hangout call next week?
> > > >
> > > > > > >> > >>
> > > >
> > > > > > >> > >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha Chintalapani
> > > >
> > > > > > >> > >> <ka...@harsha.io>
> > > >
> > > > > > >> > >> wrote:
> > > >
> > > > > > >> > >>
> > > >
> > > > > > >> > >>> Ashish,
> > > >
> > > > > > >> > >>>         Yes we are working on it. Lets discuss in the
> next
> > > KIP
> > > >
> > > > > > >> > >>> meeting.
> > > >
> > > > > > >> > >>> I'll join.
> > > >
> > > > > > >> > >>> -Harsha
> > > >
> > > > > > >> > >>>
> > > >
> > > > > > >> > >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh <
> > > >
> > > > > > asingh@cloudera.com>
> > > >
> > > > > > >> > >>> wrote:
> > > >
> > > > > > >> > >>>
> > > >
> > > > > > >> > >>> > Hello Harsha,
> > > >
> > > > > > >> > >>> >
> > > >
> > > > > > >> > >>> > Are you still working on this? Wondering if we can
> > discuss
> > > >
> > > > > this
> > > >
> > > > > > in
> > > >
> > > > > > >> > next
> > > >
> > > > > > >> > >>> KIP
> > > >
> > > > > > >> > >>> > meeting, if you can join.
> > > >
> > > > > > >> > >>> >
> > > >
> > > > > > >> > >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha Chintalapani <
> > > >
> > > > > > >> > kafka@harsha.io>
> > > >
> > > > > > >> > >>> > wrote:
> > > >
> > > > > > >> > >>> >
> > > >
> > > > > > >> > >>> > > Hi Grant,
> > > >
> > > > > > >> > >>> > >           We are working on it. Will add the details
> > to
> > > > KIP
> > > >
> > > > > > >> > >>> > > about
> > > >
> > > > > > >> > the
> > > >
> > > > > > >> > >>> > > request protocol.
> > > >
> > > > > > >> > >>> > >
> > > >
> > > > > > >> > >>> > > Thanks,
> > > >
> > > > > > >> > >>> > > Harsha
> > > >
> > > > > > >> > >>> > >
> > > >
> > > > > > >> > >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant Henke
> > > >
> > > > > > >> > >>> > > <gh...@cloudera.com>
> > > >
> > > > > > >> > >>> wrote:
> > > >
> > > > > > >> > >>> > >
> > > >
> > > > > > >> > >>> > > > Hi Parth,
> > > >
> > > > > > >> > >>> > > >
> > > >
> > > > > > >> > >>> > > > Are you still working on this? If you need any
> help
> > > > please
> > > >
> > > > > > >> > >>> > > > don't
> > > >
> > > > > > >> > >>> > hesitate
> > > >
> > > > > > >> > >>> > > > to ask.
> > > >
> > > > > > >> > >>> > > >
> > > >
> > > > > > >> > >>> > > > Thanks,
> > > >
> > > > > > >> > >>> > > > Grant
> > > >
> > > > > > >> > >>> > > >
> > > >
> > > > > > >> > >>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <
> > > >
> > > > > jun@confluent.io>
> > > >
> > > > > > >> > wrote:
> > > >
> > > > > > >> > >>> > > >
> > > >
> > > > > > >> > >>> > > > > Parth,
> > > >
> > > > > > >> > >>> > > > >
> > > >
> > > > > > >> > >>> > > > > Thanks for the reply.
> > > >
> > > > > > >> > >>> > > > >
> > > >
> > > > > > >> > >>> > > > > It makes sense to only allow the renewal by
> users
> > > that
> > > >
> > > > > > >> > >>> authenticated
> > > >
> > > > > > >> > >>> > > > using
> > > >
> > > > > > >> > >>> > > > > *non* delegation token mechanism. Then, should
> we
> > > make
> > > >
> > > > > the
> > > >
> > > > > > >> > >>> renewal a
> > > >
> > > > > > >> > >>> > > > list?
> > > >
> > > > > > >> > >>> > > > > For example, in the case of rest proxy, it will
> be
> > > >
> > > > > useful
> > > >
> > > > > > >> > >>> > > > > for
> > > >
> > > > > > >> > >>> every
> > > >
> > > > > > >> > >>> > > > > instance of rest proxy to be able to renew the
> > > tokens.
> > > >
> > > > > > >> > >>> > > > >
> > > >
> > > > > > >> > >>> > > > > It would be clearer if we can document the
> request
> > > >
> > > > > > protocol
> > > >
> > > > > > >> > like
> > > >
> > > > > > >> > >>> > > > >
> > > >
> > > > > > >> > >>> > > > >
> > > >
> > > > > > >> > >>> > > > https://cwiki.apache.org/confl
> > > uence/display/KAFKA/KIP-
> > > >
> > > > > > >> > >>> > >
> > > >
> > > > > > >> > >>> > > 4+-+Command+line+and+centralized+administrative+
> > > >
> > > > > > operations#KIP-4-
> > > >
> > > > > > >> > >>> > > Commandlineandcentralizedadministrativeoperations-
> > > >
> > > > > > >> > >>> > > CreateTopicsRequest(KAFKA-
> > 2945):(VotedandPlannedforin0.
> > > >
> > > > > > 10.1.0)
> > > >
> > > > > > >> > >>> > > > > .
> > > >
> > > > > > >> > >>> > > > >
> > > >
> > > > > > >> > >>> > > > > It would also be useful to document the client
> > APIs.
> > > >
> > > > > > >> > >>> > > > >
> > > >
> > > > > > >> > >>> > > > > Thanks,
> > > >
> > > > > > >> > >>> > > > >
> > > >
> > > > > > >> > >>> > > > > Jun
> > > >
> > > > > > >> > >>> > > > >
> > > >
> > > > > > >> > >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM, parth
> brahmbhatt
> > <
> > > >
> > > > > > >> > >>> > > > > brahmbhatt.parth@gmail.com> wrote:
> > > >
> > > > > > >> > >>> > > > >
> > > >
> > > > > > >> > >>> > > > > > Hi,
> > > >
> > > > > > >> > >>> > > > > >
> > > >
> > > > > > >> > >>> > > > > > I am suggesting that we will only allow the
> > > renewal
> > > > by
> > > >
> > > > > > >> > >>> > > > > > users
> > > >
> > > > > > >> > >>> that
> > > >
> > > > > > >> > >>> > > > > > authenticated using *non* delegation token
> > > > mechanism.
> > > >
> > > > > > For
> > > >
> > > > > > >> > >>> example,
> > > >
> > > > > > >> > >>> > If
> > > >
> > > > > > >> > >>> > > > > user
> > > >
> > > > > > >> > >>> > > > > > Alice authenticated using kerberos and
> requested
> > > >
> > > > > > >> > >>> > > > > > delegation
> > > >
> > > > > > >> > >>> tokens,
> > > >
> > > > > > >> > >>> > > > only
> > > >
> > > > > > >> > >>> > > > > > user Alice authenticated via non delegation
> > token
> > > >
> > > > > > >> > >>> > > > > > mechanism
> > > >
> > > > > > >> > can
> > > >
> > > > > > >> > >>> > > renew.
> > > >
> > > > > > >> > >>> > > > > > Clients that have  access to delegation tokens
> > can
> > > > not
> > > >
> > > > > > >> > >>> > > > > > issue
> > > >
> > > > > > >> > >>> > renewal
> > > >
> > > > > > >> > >>> > > > > > request for renewing their own token and this
> is
> > > >
> > > > > > primarily
> > > >
> > > > > > >> > >>> > important
> > > >
> > > > > > >> > >>> > > to
> > > >
> > > > > > >> > >>> > > > > > reduce the time window for which a compromised
> > > token
> > > >
> > > > > > will
> > > >
> > > > > > >> > >>> > > > > > be
> > > >
> > > > > > >> > >>> valid.
> > > >
> > > > > > >> > >>> > > > > >
> > > >
> > > > > > >> > >>> > > > > > To clarify, Yes any authenticated user can
> > request
> > > >
> > > > > > >> > >>> > > > > > delegation
> > > >
> > > > > > >> > >>> > tokens
> > > >
> > > > > > >> > >>> > > > but
> > > >
> > > > > > >> > >>> > > > > > even here I would recommend to avoid creating
> a
> > > > chain
> > > >
> > > > > > >> > >>> > > > > > where a
> > > >
> > > > > > >> > >>> > client
> > > >
> > > > > > >> > >>> > > > > > authenticated via delegation token request for
> > > more
> > > >
> > > > > > >> > delegation
> > > >
> > > > > > >> > >>> > > tokens.
> > > >
> > > > > > >> > >>> > > > > > Basically anyone can request delegation token,
> > as
> > > > long
> > > >
> > > > > > as
> > > >
> > > > > > >> > they
> > > >
> > > > > > >> > >>> > > > > authenticate
> > > >
> > > > > > >> > >>> > > > > > via a non delegation token mechanism.
> > > >
> > > > > > >> > >>> > > > > >
> > > >
> > > > > > >> > >>> > > > > > Aren't classes listed here
> > > >
> > > > > > >> > >>> > > > > > <
> > > >
> > > > > > >> > >>> > > > > >
> > > >
> > > > > > >> > >>> > > > >
> > > >
> > > > > > >> > >>> > > > https://cwiki.apache.org/confl
> > > uence/display/KAFKA/KIP-
> > > >
> > > > > > >> > >>> > > 48+Delegation+token+support+fo
> > > >
> > > > > r+Kafka#KIP-48Delegationtokens
> > > >
> > > > > > >> > >>> upportforKaf
> > > >
> > > > > > >> > >>> > > ka-PublicInterfaces
> > > >
> > > > > > >> > >>> > > > > > >
> > > >
> > > > > > >> > >>> > > > > > sufficient?
> > > >
> > > > > > >> > >>> > > > > >
> > > >
> > > > > > >> > >>> > > > > > Thanks
> > > >
> > > > > > >> > >>> > > > > > Parth
> > > >
> > > > > > >> > >>> > > > > >
> > > >
> > > > > > >> > >>> > > > > >
> > > >
> > > > > > >> > >>> > > > > >
> > > >
> > > > > > >> > >>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao
> > > >
> > > > > > >> > >>> > > > > > <ju...@confluent.io>
> > > >
> > > > > > >> > >>> wrote:
> > > >
> > > > > > >> > >>> > > > > >
> > > >
> > > > > > >> > >>> > > > > > > Parth,
> > > >
> > > > > > >> > >>> > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > Thanks for the reply. A couple of comments
> > > inline
> > > >
> > > > > > below.
> > > >
> > > > > > >> > >>> > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > On Tue, Jun 21, 2016 at 10:36 AM, parth
> > > > brahmbhatt <
> > > >
> > > > > > >> > >>> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > > >
> > > > > > >> > >>> > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > 1. Who / how are tokens renewed? By
> original
> > > >
> > > > > > requester
> > > >
> > > > > > >> > >>> only? or
> > > >
> > > > > > >> > >>> > > > using
> > > >
> > > > > > >> > >>> > > > > > > > Kerberos
> > > >
> > > > > > >> > >>> > > > > > > > auth only?
> > > >
> > > > > > >> > >>> > > > > > > > My recommendation is to do this only using
> > > >
> > > > > Kerberos
> > > >
> > > > > > >> > >>> > > > > > > > auth
> > > >
> > > > > > >> > and
> > > >
> > > > > > >> > >>> > only
> > > >
> > > > > > >> > >>> > > > > threw
> > > >
> > > > > > >> > >>> > > > > > > the
> > > >
> > > > > > >> > >>> > > > > > > > renewer specified during the acquisition
> > > > request.
> > > >
> > > > > > >> > >>> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > Hmm, not sure that I follow this. Are you
> > saying
> > > >
> > > > > that
> > > >
> > > > > > >> > >>> > > > > > > any
> > > >
> > > > > > >> > >>> client
> > > >
> > > > > > >> > >>> > > > > > > authenticated with the delegation token can
> > > renew,
> > > >
> > > > > > i.e.
> > > >
> > > > > > >> > there
> > > >
> > > > > > >> > >>> is
> > > >
> > > > > > >> > >>> > no
> > > >
> > > > > > >> > >>> > > > > > renewer
> > > >
> > > > > > >> > >>> > > > > > > needed?
> > > >
> > > > > > >> > >>> > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > Also, just to be clear, any authenticated
> > client
> > > >
> > > > > > (either
> > > >
> > > > > > >> > >>> through
> > > >
> > > > > > >> > >>> > > SASL
> > > >
> > > > > > >> > >>> > > > > or
> > > >
> > > > > > >> > >>> > > > > > > SSL) can request a delegation token for the
> > > >
> > > > > > >> > >>> > > > > > > authenticated
> > > >
> > > > > > >> > >>> user,
> > > >
> > > > > > >> > >>> > > > right?
> > > >
> > > > > > >> > >>> > > > > > >
> > > >
> > > > > > >> > >>> > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > 2. Are tokens stored on each broker or in
> > ZK?
> > > >
> > > > > > >> > >>> > > > > > > > My recommendation is still to store in ZK
> or
> > > not
> > > >
> > > > > > store
> > > >
> > > > > > >> > them
> > > >
> > > > > > >> > >>> at
> > > >
> > > > > > >> > >>> > > all.
> > > >
> > > > > > >> > >>> > > > > The
> > > >
> > > > > > >> > >>> > > > > > > > whole controller based distribution is too
> > > much
> > > >
> > > > > > >> > >>> > > > > > > > overhead
> > > >
> > > > > > >> > >>> with
> > > >
> > > > > > >> > >>> > not
> > > >
> > > > > > >> > >>> > > > > much
> > > >
> > > > > > >> > >>> > > > > > to
> > > >
> > > > > > >> > >>> > > > > > > > achieve.
> > > >
> > > > > > >> > >>> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > 3. How are tokens invalidated / expired?
> > > >
> > > > > > >> > >>> > > > > > > > Either by expiration time out or through
> an
> > > >
> > > > > explicit
> > > >
> > > > > > >> > >>> request to
> > > >
> > > > > > >> > >>> > > > > > > invalidate.
> > > >
> > > > > > >> > >>> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > 4. Which encryption algorithm is used?
> > > >
> > > > > > >> > >>> > > > > > > > SCRAM
> > > >
> > > > > > >> > >>> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > 5. What is the impersonation proposal (it
> > > wasn't
> > > >
> > > > > in
> > > >
> > > > > > >> > >>> > > > > > > > the
> > > >
> > > > > > >> > KIP
> > > >
> > > > > > >> > >>> but
> > > >
> > > > > > >> > >>> > > was
> > > >
> > > > > > >> > >>> > > > > > > > discussed
> > > >
> > > > > > >> > >>> > > > > > > > in this thread)?
> > > >
> > > > > > >> > >>> > > > > > > > There is no imperonation proposal. I tried
> > and
> > > >
> > > > > > >> > >>> > > > > > > > explained
> > > >
> > > > > > >> > how
> > > >
> > > > > > >> > >>> > its
> > > >
> > > > > > >> > >>> > > a
> > > >
> > > > > > >> > >>> > > > > > > > different problem and why its not really
> > > > necessary
> > > >
> > > > > > to
> > > >
> > > > > > >> > >>> discuss
> > > >
> > > > > > >> > >>> > > that
> > > >
> > > > > > >> > >>> > > > as
> > > >
> > > > > > >> > >>> > > > > > > part
> > > >
> > > > > > >> > >>> > > > > > > > of this KIP.  This KIP will not support
> any
> > > >
> > > > > > >> > impersonation,
> > > >
> > > > > > >> > >>> it
> > > >
> > > > > > >> > >>> > > will
> > > >
> > > > > > >> > >>> > > > > just
> > > >
> > > > > > >> > >>> > > > > > > be
> > > >
> > > > > > >> > >>> > > > > > > > another way to authenticate.
> > > >
> > > > > > >> > >>> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > 6. Do we need new ACLs, if so - for what
> > > > actions?
> > > >
> > > > > > >> > >>> > > > > > > > We do not need new ACLs.
> > > >
> > > > > > >> > >>> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > Could we document the format of the new
> > > >
> > > > > > request/response
> > > >
> > > > > > >> > and
> > > >
> > > > > > >> > >>> > their
> > > >
> > > > > > >> > >>> > > > > > > associated Resource and Operation for ACL?
> > > >
> > > > > > >> > >>> > > > > > >
> > > >
> > > > > > >> > >>> > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > 7. How would the delegation token be
> > > configured
> > > > in
> > > >
> > > > > > the
> > > >
> > > > > > >> > >>> client?
> > > >
> > > > > > >> > >>> > > > > > > > Should be through config. I wasn't
> planning
> > on
> > > >
> > > > > > >> > >>> > > > > > > > supporting
> > > >
> > > > > > >> > >>> JAAS
> > > >
> > > > > > >> > >>> > > for
> > > >
> > > > > > >> > >>> > > > > > > tokens.
> > > >
> > > > > > >> > >>> > > > > > > > I don't believe hadoop does this either.
> > > >
> > > > > > >> > >>> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > Thanks
> > > >
> > > > > > >> > >>> > > > > > > > Parth
> > > >
> > > > > > >> > >>> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <
> > > >
> > > > > > >> > jun@confluent.io>
> > > >
> > > > > > >> > >>> > > wrote:
> > > >
> > > > > > >> > >>> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > Harsha,
> > > >
> > > > > > >> > >>> > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > Another question.
> > > >
> > > > > > >> > >>> > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > 9. How would the delegation token be
> > > > configured
> > > >
> > > > > in
> > > >
> > > > > > >> > >>> > > > > > > > > the
> > > >
> > > > > > >> > >>> > client?
> > > >
> > > > > > >> > >>> > > > The
> > > >
> > > > > > >> > >>> > > > > > > > standard
> > > >
> > > > > > >> > >>> > > > > > > > > way is to do this through JAAS. However,
> > we
> > > > will
> > > >
> > > > > > >> > >>> > > > > > > > > need
> > > >
> > > > > > >> > to
> > > >
> > > > > > >> > >>> > think
> > > >
> > > > > > >> > >>> > > > > > through
> > > >
> > > > > > >> > >>> > > > > > > if
> > > >
> > > > > > >> > >>> > > > > > > > > this is convenient in a shared
> > environment.
> > > > For
> > > >
> > > > > > >> > example,
> > > >
> > > > > > >> > >>> > when a
> > > >
> > > > > > >> > >>> > > > new
> > > >
> > > > > > >> > >>> > > > > > > task
> > > >
> > > > > > >> > >>> > > > > > > > is
> > > >
> > > > > > >> > >>> > > > > > > > > added to a Storm worker node, do we need
> > to
> > > >
> > > > > > >> > >>> > > > > > > > > dynamically
> > > >
> > > > > > >> > >>> add a
> > > >
> > > > > > >> > >>> > > new
> > > >
> > > > > > >> > >>> > > > > > > section
> > > >
> > > > > > >> > >>> > > > > > > > > in the JAAS file? It may be more
> > convenient
> > > if
> > > >
> > > > > we
> > > >
> > > > > > >> > >>> > > > > > > > > can
> > > >
> > > > > > >> > >>> pass in
> > > >
> > > > > > >> > >>> > > the
> > > >
> > > > > > >> > >>> > > > > > token
> > > >
> > > > > > >> > >>> > > > > > > > > through the config directly w/o going
> > > through
> > > >
> > > > > > JAAS.
> > > >
> > > > > > >> > >>> > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > Are you or Parth still actively working
> on
> > > > this
> > > >
> > > > > > KIP?
> > > >
> > > > > > >> > >>> > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > Thanks,
> > > >
> > > > > > >> > >>> > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > Jun
> > > >
> > > > > > >> > >>> > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun
> Rao <
> > > >
> > > > > > >> > >>> jun@confluent.io>
> > > >
> > > > > > >> > >>> > > > wrote:
> > > >
> > > > > > >> > >>> > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > > Just to add on that list.
> > > >
> > > > > > >> > >>> > > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > > 2. It would be good to document the
> > format
> > > > of
> > > >
> > > > > > the
> > > >
> > > > > > >> > data
> > > >
> > > > > > >> > >>> > stored
> > > >
> > > > > > >> > >>> > > > in
> > > >
> > > > > > >> > >>> > > > > > ZK.
> > > >
> > > > > > >> > >>> > > > > > > > > > 7. Earlier, there was a discussion on
> > > > whether
> > > >
> > > > > > the
> > > >
> > > > > > >> > tokens
> > > >
> > > > > > >> > >>> > > should
> > > >
> > > > > > >> > >>> > > > > be
> > > >
> > > > > > >> > >>> > > > > > > > > > propagated through ZK like
> > > config/acl/quota,
> > > >
> > > > > or
> > > >
> > > > > > >> > through
> > > >
> > > > > > >> > >>> the
> > > >
> > > > > > >> > >>> > > > > > > controller.
> > > >
> > > > > > >> > >>> > > > > > > > > > Currently, the controller is only
> > designed
> > > > for
> > > >
> > > > > > >> > >>> propagating
> > > >
> > > > > > >> > >>> > > > topic
> > > >
> > > > > > >> > >>> > > > > > > > > metadata,
> > > >
> > > > > > >> > >>> > > > > > > > > > but not other data.
> > > >
> > > > > > >> > >>> > > > > > > > > > 8. Should we use SCRAM to send the
> token
> > > >
> > > > > instead
> > > >
> > > > > > >> > >>> > > > > > > > > > of
> > > >
> > > > > > >> > >>> > > DIGEST-MD5
> > > >
> > > > > > >> > >>> > > > > > since
> > > >
> > > > > > >> > >>> > > > > > > > it's
> > > >
> > > > > > >> > >>> > > > > > > > > > deprecated?
> > > >
> > > > > > >> > >>> > > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > > Also, the images in the wiki seem
> > broken.
> > > >
> > > > > > >> > >>> > > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > > Thanks,
> > > >
> > > > > > >> > >>> > > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > > Jun
> > > >
> > > > > > >> > >>> > > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen
> > > >
> > > > > Shapira <
> > > >
> > > > > > >> > >>> > > > > gwen@confluent.io>
> > > >
> > > > > > >> > >>> > > > > > > > > wrote:
> > > >
> > > > > > >> > >>> > > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> From what I can see, remaining
> > questions
> > > > are:
> > > >
> > > > > > >> > >>> > > > > > > > > >>
> > > >
> > > > > > >> > >>> > > > > > > > > >> 1. Who / how are tokens renewed? By
> > > > original
> > > >
> > > > > > >> > requester
> > > >
> > > > > > >> > >>> > only?
> > > >
> > > > > > >> > >>> > > > or
> > > >
> > > > > > >> > >>> > > > > > > using
> > > >
> > > > > > >> > >>> > > > > > > > > >> Kerberos auth only?
> > > >
> > > > > > >> > >>> > > > > > > > > >> 2. Are tokens stored on each broker
> or
> > in
> > > > ZK?
> > > >
> > > > > > >> > >>> > > > > > > > > >> 3. How are tokens invalidated /
> > expired?
> > > >
> > > > > > >> > >>> > > > > > > > > >> 4. Which encryption algorithm is
> used?
> > > >
> > > > > > >> > >>> > > > > > > > > >> 5. What is the impersonation proposal
> > (it
> > > >
> > > > > > wasn't
> > > >
> > > > > > >> > >>> > > > > > > > > >> in
> > > >
> > > > > > >> > the
> > > >
> > > > > > >> > >>> > KIP
> > > >
> > > > > > >> > >>> > > > but
> > > >
> > > > > > >> > >>> > > > > > was
> > > >
> > > > > > >> > >>> > > > > > > > > >> discussed in this thread)?
> > > >
> > > > > > >> > >>> > > > > > > > > >> 6. Do we need new ACLs, if so - for
> > what
> > > >
> > > > > > actions?
> > > >
> > > > > > >> > >>> > > > > > > > > >>
> > > >
> > > > > > >> > >>> > > > > > > > > >> Gwen
> > > >
> > > > > > >> > >>> > > > > > > > > >>
> > > >
> > > > > > >> > >>> > > > > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM,
> Harsha
> > <
> > > >
> > > > > > >> > >>> kafka@harsha.io>
> > > >
> > > > > > >> > >>> > > > wrote:
> > > >
> > > > > > >> > >>> > > > > > > > > >> > Jun & Ismael,
> > > >
> > > > > > >> > >>> > > > > > > > > >> >
> > Unfortunately
> > > I
> > > >
> > > > > > >> > >>> > > > > > > > > >> > couldn't
> > > >
> > > > > > >> > >>> attend
> > > >
> > > > > > >> > >>> > > the
> > > >
> > > > > > >> > >>> > > > > KIP
> > > >
> > > > > > >> > >>> > > > > > > > > meeting
> > > >
> > > > > > >> > >>> > > > > > > > > >> >                          when
> > > delegation
> > > >
> > > > > > tokens
> > > >
> > > > > > >> > >>> > discussed.
> > > >
> > > > > > >> > >>> > > > > > > > Appreciate
> > > >
> > > > > > >> > >>> > > > > > > > > if
> > > >
> > > > > > >> > >>> > > > > > > > > >> >                          you can
> > update
> > > > the
> > > >
> > > > > > >> > thread if
> > > >
> > > > > > >> > >>> > you
> > > >
> > > > > > >> > >>> > > > have
> > > >
> > > > > > >> > >>> > > > > > any
> > > >
> > > > > > >> > >>> > > > > > > > > >> >                          further
> > > > questions.
> > > >
> > > > > > >> > >>> > > > > > > > > >> > Thanks,
> > > >
> > > > > > >> > >>> > > > > > > > > >> > Harsha
> > > >
> > > > > > >> > >>> > > > > > > > > >> >
> > > >
> > > > > > >> > >>> > > > > > > > > >> > On Tue, May 24, 2016, at 11:32 AM,
> > > Liquan
> > > >
> > > > > Pei
> > > >
> > > > > > >> > wrote:
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> It seems that the links to images
> in
> > > the
> > > >
> > > > > KIP
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> are
> > > >
> > > > > > >> > >>> > broken.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >>
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> Liquan
> > > >
> > > > > > >> > >>> > > > > > > > > >> >>
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> On Tue, May 24, 2016 at 9:33 AM,
> > parth
> > > >
> > > > > > >> > brahmbhatt <
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> brahmbhatt.parth@gmail.com>
> wrote:
> > > >
> > > > > > >> > >>> > > > > > > > > >> >>
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > 110. What does
> > getDelegationTokenAs
> > > >
> > > > > mean?
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > In the current proposal we only
> > > allow
> > > > a
> > > >
> > > > > > user
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > to
> > > >
> > > > > > >> > >>> get
> > > >
> > > > > > >> > >>> > > > > > delegation
> > > >
> > > > > > >> > >>> > > > > > > > > token
> > > >
> > > > > > >> > >>> > > > > > > > > >> for
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > the identity that it
> authenticated
> > > as
> > > >
> > > > > > using
> > > >
> > > > > > >> > >>> another
> > > >
> > > > > > >> > >>> > > > > > mechanism,
> > > >
> > > > > > >> > >>> > > > > > > > i.e.
> > > >
> > > > > > >> > >>> > > > > > > > > >> A user
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > that authenticate using a keytab
> > for
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > principal
> > > >
> > > > > > >> > >>> > > > > > > user1@EXAMPLE.COM
> > > >
> > > > > > >> > >>> > > > > > > > > >> will get
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens for that user
> > > only.
> > > > In
> > > >
> > > > > > >> > future I
> > > >
> > > > > > >> > >>> > think
> > > >
> > > > > > >> > >>> > > > we
> > > >
> > > > > > >> > >>> > > > > > will
> > > >
> > > > > > >> > >>> > > > > > > > > have
> > > >
> > > > > > >> > >>> > > > > > > > > >> to
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > extend support such that we
> allow
> > > some
> > > >
> > > > > set
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > of
> > > >
> > > > > > >> > >>> users (
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > kafka-rest-user@EXAMPLE.COM,
> > > >
> > > > > > >> > >>> > storm-nimbus@EXAMPLE.COM)
> > > >
> > > > > > >> > >>> > > > to
> > > >
> > > > > > >> > >>> > > > > > > > acquire
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > delegation tokens on behalf of
> > other
> > > >
> > > > > users
> > > >
> > > > > > >> > whose
> > > >
> > > > > > >> > >>> > > identity
> > > >
> > > > > > >> > >>> > > > > > they
> > > >
> > > > > > >> > >>> > > > > > > > have
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > verified independently.  Kafka
> > > brokers
> > > >
> > > > > > will
> > > >
> > > > > > >> > have
> > > >
> > > > > > >> > >>> ACLs
> > > >
> > > > > > >> > >>> > > to
> > > >
> > > > > > >> > >>> > > > > > > control
> > > >
> > > > > > >> > >>> > > > > > > > > >> which
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > users are allowed to impersonate
> > > other
> > > >
> > > > > > users
> > > >
> > > > > > >> > and
> > > >
> > > > > > >> > >>> get
> > > >
> > > > > > >> > >>> > > > tokens
> > > >
> > > > > > >> > >>> > > > > > on
> > > >
> > > > > > >> > >>> > > > > > > > > >> behalf of
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > them. Overall Impersonation is a
> > > whole
> > > >
> > > > > > >> > different
> > > >
> > > > > > >> > >>> > > problem
> > > >
> > > > > > >> > >>> > > > in
> > > >
> > > > > > >> > >>> > > > > > my
> > > >
> > > > > > >> > >>> > > > > > > > > >> opinion and
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > I think we can tackle it in
> > separate
> > > >
> > > > > KIP.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > 111. What's the typical rate of
> > > > getting
> > > >
> > > > > > and
> > > >
> > > > > > >> > >>> renewing
> > > >
> > > > > > >> > >>> > > > > > delegation
> > > >
> > > > > > >> > >>> > > > > > > > > >> tokens?
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > Typically this should be very
> very
> > > > low,
> > > >
> > > > > 1
> > > >
> > > > > > >> > request
> > > >
> > > > > > >> > >>> per
> > > >
> > > > > > >> > >>> > > > > minute
> > > >
> > > > > > >> > >>> > > > > > > is a
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > relatively high estimate.
> However
> > it
> > > >
> > > > > > depends
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > on
> > > >
> > > > > > >> > >>> the
> > > >
> > > > > > >> > >>> > > token
> > > >
> > > > > > >> > >>> > > > > > > > > >> expiration. I am
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > less worried about the extra
> load
> > it
> > > >
> > > > > puts
> > > >
> > > > > > on
> > > >
> > > > > > >> > >>> > controller
> > > >
> > > > > > >> > >>> > > > vs
> > > >
> > > > > > >> > >>> > > > > > the
> > > >
> > > > > > >> > >>> > > > > > > > > added
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > complexity and the value it
> > offers.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > Thanks
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > Parth
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > On Tue, May 24, 2016 at 7:30 AM,
> > > > Ismael
> > > >
> > > > > > Juma
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > <
> > > >
> > > > > > >> > >>> > > > > > > ismael@juma.me.uk>
> > > >
> > > > > > >> > >>> > > > > > > > > >> wrote:
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > Thanks Rajini. It would
> probably
> > > >
> > > > > > require a
> > > >
> > > > > > >> > >>> separate
> > > >
> > > > > > >> > >>> > > KIP
> > > >
> > > > > > >> > >>> > > > > as
> > > >
> > > > > > >> > >>> > > > > > it
> > > >
> > > > > > >> > >>> > > > > > > > > will
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > introduce user visible
> changes.
> > We
> > > >
> > > > > could
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > also
> > > >
> > > > > > >> > >>> > update
> > > >
> > > > > > >> > >>> > > > > KIP-48
> > > >
> > > > > > >> > >>> > > > > > > to
> > > >
> > > > > > >> > >>> > > > > > > > > >> have this
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > information, but it seems
> > cleaner
> > > to
> > > >
> > > > > do
> > > >
> > > > > > it
> > > >
> > > > > > >> > >>> > > separately.
> > > >
> > > > > > >> > >>> > > > We
> > > >
> > > > > > >> > >>> > > > > > can
> > > >
> > > > > > >> > >>> > > > > > > > > >> discuss
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > that
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > in the KIP call today.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > Ismael
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > On Tue, May 24, 2016 at 3:19
> PM,
> > > >
> > > > > Rajini
> > > >
> > > > > > >> > Sivaram
> > > >
> > > > > > >> > >>> <
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > rajinisivaram@googlemail.com>
> > > > wrote:
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > Ismael,
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > I have created a JIRA (
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > https://issues.apache.org/
> > > >
> > > > > > >> > jira/browse/KAFKA-3751)
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > for adding SCRAM as a SASL
> > > >
> > > > > mechanism.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > Would
> > > >
> > > > > > >> > >>> that
> > > >
> > > > > > >> > >>> > > need
> > > >
> > > > > > >> > >>> > > > > > > another
> > > >
> > > > > > >> > >>> > > > > > > > > >> KIP? If
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > KIP-48 will use this
> > mechanism,
> > > > can
> > > >
> > > > > > this
> > > >
> > > > > > >> > just
> > > >
> > > > > > >> > >>> be
> > > >
> > > > > > >> > >>> > a
> > > >
> > > > > > >> > >>> > > > JIRA
> > > >
> > > > > > >> > >>> > > > > > > that
> > > >
> > > > > > >> > >>> > > > > > > > > gets
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > reviewed
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > when the PR is ready?
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > Thank you,
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > Rajini
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > On Tue, May 24, 2016 at 2:46
> > PM,
> > > >
> > > > > > Ismael
> > > >
> > > > > > >> > Juma <
> > > >
> > > > > > >> > >>> > > > > > > > > ismael@juma.me.uk>
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > wrote:
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > Thanks Rajini, SCRAM seems
> > > like
> > > > a
> > > >
> > > > > > good
> > > >
> > > > > > >> > >>> > candidate.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > Gwen had independently
> > > mentioned
> > > >
> > > > > > this
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > as
> > > >
> > > > > > >> > a
> > > >
> > > > > > >> > >>> SASL
> > > >
> > > > > > >> > >>> > > > > > mechanism
> > > >
> > > > > > >> > >>> > > > > > > > > that
> > > >
> > > > > > >> > >>> > > > > > > > > >> might
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > be
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > useful for Kafka and I
> have
> > > been
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > meaning
> > > >
> > > > > > >> > to
> > > >
> > > > > > >> > >>> > check
> > > >
> > > > > > >> > >>> > > > it
> > > >
> > > > > > >> > >>> > > > > in
> > > >
> > > > > > >> > >>> > > > > > > > more
> > > >
> > > > > > >> > >>> > > > > > > > > >> detail.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > Good
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > to know that you are
> willing
> > > to
> > > >
> > > > > > >> > contribute
> > > >
> > > > > > >> > >>> an
> > > >
> > > > > > >> > >>> > > > > > > > implementation.
> > > >
> > > > > > >> > >>> > > > > > > > > >> Maybe
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > we
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > should file a separate
> JIRA
> > > for
> > > >
> > > > > > this?
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > Ismael
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > On Tue, May 24, 2016 at
> 2:12
> > > PM,
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > Rajini
> > > >
> > > > > > >> > >>> > Sivaram <
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > >
> > rajinisivaram@googlemail.com>
> > > >
> > > > > > wrote:
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > SCRAM (Salted Challenge
> > > > Response
> > > >
> > > > > > >> > >>> > Authentication
> > > >
> > > > > > >> > >>> > > > > > > > Mechanism)
> > > >
> > > > > > >> > >>> > > > > > > > > >> is a
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > better
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > mechanism than
> Digest-MD5.
> > > > Java
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > doesn't
> > > >
> > > > > > >> > >>> come
> > > >
> > > > > > >> > >>> > > > with a
> > > >
> > > > > > >> > >>> > > > > > > > > built-in
> > > >
> > > > > > >> > >>> > > > > > > > > >> SCRAM
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > SaslServer or
> SaslClient,
> > > but
> > > > I
> > > >
> > > > > > will
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > be
> > > >
> > > > > > >> > >>> happy
> > > >
> > > > > > >> > >>> > > to
> > > >
> > > > > > >> > >>> > > > > add
> > > >
> > > > > > >> > >>> > > > > > > > > support
> > > >
> > > > > > >> > >>> > > > > > > > > >> in
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > Kafka
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > since
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > it would be a useful
> > > mechanism
> > > >
> > > > > to
> > > >
> > > > > > >> > support
> > > >
> > > > > > >> > >>> > > anyway.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > >
> > > https://tools.ietf.org/html/
> > > >
> > > > > > rfc7677
> > > >
> > > > > > >> > >>> > describes
> > > >
> > > > > > >> > >>> > > > the
> > > >
> > > > > > >> > >>> > > > > > > > protocol
> > > >
> > > > > > >> > >>> > > > > > > > > >> for
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > SCRAM-SHA-256.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > On Tue, May 24, 2016 at
> > 2:37
> > > > AM,
> > > >
> > > > > > Jun
> > > >
> > > > > > >> > Rao <
> > > >
> > > > > > >> > >>> > > > > > > > jun@confluent.io
> > > >
> > > > > > >> > >>> > > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> wrote:
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > Parth,
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > Thanks for the
> > > explanation.
> > > > A
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > couple
> > > >
> > > > > > >> > of
> > > >
> > > > > > >> > >>> > more
> > > >
> > > > > > >> > >>> > > > > > > questions.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > 110. What does
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > getDelegationTokenAs
> > > >
> > > > > > >> > >>> mean?
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > 111. What's the
> typical
> > > rate
> > > >
> > > > > of
> > > >
> > > > > > >> > getting
> > > >
> > > > > > >> > >>> and
> > > >
> > > > > > >> > >>> > > > > > renewing
> > > >
> > > > > > >> > >>> > > > > > > > > >> delegation
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > tokens?
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > That may have an
> impact
> > on
> > > >
> > > > > > whether
> > > >
> > > > > > >> > they
> > > >
> > > > > > >> > >>> > > should
> > > >
> > > > > > >> > >>> > > > be
> > > >
> > > > > > >> > >>> > > > > > > > > directed
> > > >
> > > > > > >> > >>> > > > > > > > > >> to the
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > controller.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > Jun
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > On Mon, May 23, 2016
> at
> > > 1:19
> > > >
> > > > > PM,
> > > >
> > > > > > >> > parth
> > > >
> > > > > > >> > >>> > > > > brahmbhatt <
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > brahmbhatt.parth@gmail.com>
> > > >
> > > > > > wrote:
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Hi Jun,
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Thanks for
> reviewing.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * We could add a
> > Cluster
> > > >
> > > > > > action
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > to
> > > >
> > > > > > >> > add
> > > >
> > > > > > >> > >>> > acls
> > > >
> > > > > > >> > >>> > > > on
> > > >
> > > > > > >> > >>> > > > > > who
> > > >
> > > > > > >> > >>> > > > > > > > can
> > > >
> > > > > > >> > >>> > > > > > > > > >> request
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > delegation
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > tokens. I don't see
> > the
> > > > use
> > > >
> > > > > > case
> > > >
> > > > > > >> > for
> > > >
> > > > > > >> > >>> that
> > > >
> > > > > > >> > >>> > > yet
> > > >
> > > > > > >> > >>> > > > > but
> > > >
> > > > > > >> > >>> > > > > > > > down
> > > >
> > > > > > >> > >>> > > > > > > > > >> the line
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > when
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > we
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > start supporting
> > > >
> > > > > > >> > getDelegationTokenAs
> > > >
> > > > > > >> > >>> it
> > > >
> > > > > > >> > >>> > > will
> > > >
> > > > > > >> > >>> > > > > be
> > > >
> > > > > > >> > >>> > > > > > > > > >> necessary.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * Yes we recommend
> > > tokens
> > > > to
> > > >
> > > > > > be
> > > >
> > > > > > >> > only
> > > >
> > > > > > >> > >>> > > > > > > used/distributed
> > > >
> > > > > > >> > >>> > > > > > > > > >> over
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > secure
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > channels.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * Depending on what
> > > design
> > > >
> > > > > we
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > end
> > > >
> > > > > > >> > up
> > > >
> > > > > > >> > >>> > > choosing
> > > >
> > > > > > >> > >>> > > > > > > > > >> Invalidation will
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > be
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > responsibility of
> > every
> > > >
> > > > > broker
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > or
> > > >
> > > > > > >> > >>> > > controller.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * I am not sure if I
> > > >
> > > > > > documented
> > > >
> > > > > > >> > >>> somewhere
> > > >
> > > > > > >> > >>> > > > that
> > > >
> > > > > > >> > >>> > > > > > > > > >> invalidation
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > will
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > directly
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > go through zookeeper
> > but
> > > >
> > > > > that
> > > >
> > > > > > is
> > > >
> > > > > > >> > not
> > > >
> > > > > > >> > >>> the
> > > >
> > > > > > >> > >>> > > > > intent.
> > > >
> > > > > > >> > >>> > > > > > > > > >> Invalidation
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > will
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > either
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > be request based or
> > due
> > > to
> > > >
> > > > > > >> > >>> expiration. No
> > > >
> > > > > > >> > >>> > > > > direct
> > > >
> > > > > > >> > >>> > > > > > > > > >> zookeeper
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > interaction
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > from
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > any client.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * "Broker also
> stores
> > > the
> > > >
> > > > > > >> > >>> DelegationToken
> > > >
> > > > > > >> > >>> > > > > without
> > > >
> > > > > > >> > >>> > > > > > > the
> > > >
> > > > > > >> > >>> > > > > > > > > >> hmac in
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > the
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > zookeeper." : Sorry
> > > about
> > > >
> > > > > the
> > > >
> > > > > > >> > >>> confusion.
> > > >
> > > > > > >> > >>> > > The
> > > >
> > > > > > >> > >>> > > > > sole
> > > >
> > > > > > >> > >>> > > > > > > > > >> purpose of
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > zookeeper
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > in
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > this design is as
> > > >
> > > > > distribution
> > > >
> > > > > > >> > channel
> > > >
> > > > > > >> > >>> > for
> > > >
> > > > > > >> > >>> > > > > tokens
> > > >
> > > > > > >> > >>> > > > > > > > > >> between all
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > brokers
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > and a
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > layer that ensures
> > only
> > > >
> > > > > tokens
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > that
> > > >
> > > > > > >> > >>> were
> > > >
> > > > > > >> > >>> > > > > > generated
> > > >
> > > > > > >> > >>> > > > > > > by
> > > >
> > > > > > >> > >>> > > > > > > > > >> making a
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > request
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > to a
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > broker will be
> > accepted
> > > >
> > > > > (more
> > > >
> > > > > > on
> > > >
> > > > > > >> > this
> > > >
> > > > > > >> > >>> in
> > > >
> > > > > > >> > >>> > > > second
> > > >
> > > > > > >> > >>> > > > > > > > > >> paragraph). The
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > token
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > consists of few
> > elements
> > > >
> > > > > > (owner,
> > > >
> > > > > > >> > >>> renewer,
> > > >
> > > > > > >> > >>> > > > uuid
> > > >
> > > > > > >> > >>> > > > > ,
> > > >
> > > > > > >> > >>> > > > > > > > > >> expiration,
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > hmac)
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > ,
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > one
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > of
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > which is the finally
> > > >
> > > > > generated
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > hmac
> > > >
> > > > > > >> > >>> but
> > > >
> > > > > > >> > >>> > > hmac
> > > >
> > > > > > >> > >>> > > > it
> > > >
> > > > > > >> > >>> > > > > > > self
> > > >
> > > > > > >> > >>> > > > > > > > is
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > derivable
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > if
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > you
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > have all the other
> > > > elements
> > > >
> > > > > of
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > the
> > > >
> > > > > > >> > >>> token
> > > >
> > > > > > >> > >>> > +
> > > >
> > > > > > >> > >>> > > > > secret
> > > >
> > > > > > >> > >>> > > > > > > key
> > > >
> > > > > > >> > >>> > > > > > > > > to
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > generate
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > hmac.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Given zookeeper does
> > not
> > > >
> > > > > > provide
> > > >
> > > > > > >> > SSL
> > > >
> > > > > > >> > >>> > > support
> > > >
> > > > > > >> > >>> > > > we
> > > >
> > > > > > >> > >>> > > > > > do
> > > >
> > > > > > >> > >>> > > > > > > > not
> > > >
> > > > > > >> > >>> > > > > > > > > >> want the
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > entire
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > token to be wire
> > > > transferred
> > > >
> > > > > > to
> > > >
> > > > > > >> > >>> zookeeper
> > > >
> > > > > > >> > >>> > > as
> > > >
> > > > > > >> > >>> > > > > that
> > > >
> > > > > > >> > >>> > > > > > > > will
> > > >
> > > > > > >> > >>> > > > > > > > > >> be an
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > insecure
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > wire
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > transfer. Instead we
> > > only
> > > >
> > > > > > store
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > all
> > > >
> > > > > > >> > >>> the
> > > >
> > > > > > >> > >>> > > other
> > > >
> > > > > > >> > >>> > > > > > > > elements
> > > >
> > > > > > >> > >>> > > > > > > > > >> of a
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > delegation
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > tokens. Brokers can
> > read
> > > >
> > > > > these
> > > >
> > > > > > >> > >>> elements
> > > >
> > > > > > >> > >>> > and
> > > >
> > > > > > >> > >>> > > > > > because
> > > >
> > > > > > >> > >>> > > > > > > > > they
> > > >
> > > > > > >> > >>> > > > > > > > > >> also
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > have
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > access
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > to secret key they
> > will
> > > be
> > > >
> > > > > > able
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > to
> > > >
> > > > > > >> > >>> > generate
> > > >
> > > > > > >> > >>> > > > > hmac
> > > >
> > > > > > >> > >>> > > > > > on
> > > >
> > > > > > >> > >>> > > > > > > > > >> their end.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > One of the
> alternative
> > > >
> > > > > > proposed
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > is
> > > >
> > > > > > >> > to
> > > >
> > > > > > >> > >>> > avoid
> > > >
> > > > > > >> > >>> > > > > > > zookeeper
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > altogether. A
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > Client
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > will call broker
> with
> > > >
> > > > > required
> > > >
> > > > > > >> > >>> > information
> > > >
> > > > > > >> > >>> > > > > > (owner,
> > > >
> > > > > > >> > >>> > > > > > > > > >> renwer,
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > expiration)
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > and
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > get back (signed
> hmac,
> > > >
> > > > > uuid).
> > > >
> > > > > > >> > Broker
> > > >
> > > > > > >> > >>> > won't
> > > >
> > > > > > >> > >>> > > > > store
> > > >
> > > > > > >> > >>> > > > > > > this
> > > >
> > > > > > >> > >>> > > > > > > > > in
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > zookeeper.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > From
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > this point a client
> > can
> > > >
> > > > > > contact
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > any
> > > >
> > > > > > >> > >>> > broker
> > > >
> > > > > > >> > >>> > > > with
> > > >
> > > > > > >> > >>> > > > > > all
> > > >
> > > > > > >> > >>> > > > > > > > the
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > delegation
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > token
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > info (owner, rewner,
> > > >
> > > > > > expiration,
> > > >
> > > > > > >> > hmac,
> > > >
> > > > > > >> > >>> > > uuid)
> > > >
> > > > > > >> > >>> > > > > the
> > > >
> > > > > > >> > >>> > > > > > > > borker
> > > >
> > > > > > >> > >>> > > > > > > > > >> will
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > regenerate
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > the
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > hmac and as long as
> it
> > > >
> > > > > matches
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > with
> > > >
> > > > > > >> > >>> hmac
> > > >
> > > > > > >> > >>> > > > > > presented
> > > >
> > > > > > >> > >>> > > > > > > by
> > > >
> > > > > > >> > >>> > > > > > > > > >> client ,
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > broker
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > will
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > allow the request to
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > authenticate.
> > > >
> > > > > > >> > >>> Only
> > > >
> > > > > > >> > >>> > > > > problem
> > > >
> > > > > > >> > >>> > > > > > > with
> > > >
> > > > > > >> > >>> > > > > > > > > >> this
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > approach
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > is
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > if
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > the secret key is
> > > >
> > > > > compromised
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > any
> > > >
> > > > > > >> > >>> client
> > > >
> > > > > > >> > >>> > > can
> > > >
> > > > > > >> > >>> > > > > now
> > > >
> > > > > > >> > >>> > > > > > > > > generate
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > random
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > tokens
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > and
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > they will still be
> > able
> > > to
> > > >
> > > > > > >> > >>> authenticate
> > > >
> > > > > > >> > >>> > as
> > > >
> > > > > > >> > >>> > > > any
> > > >
> > > > > > >> > >>> > > > > > user
> > > >
> > > > > > >> > >>> > > > > > > > > they
> > > >
> > > > > > >> > >>> > > > > > > > > >> like.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > with
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > zookeeper we
> guarantee
> > > > that
> > > >
> > > > > > only
> > > >
> > > > > > >> > >>> tokens
> > > >
> > > > > > >> > >>> > > > > acquired
> > > >
> > > > > > >> > >>> > > > > > > via
> > > >
> > > > > > >> > >>> > > > > > > > a
> > > >
> > > > > > >> > >>> > > > > > > > > >> broker
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > (using
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > some
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > auth scheme other
> than
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > delegation
> > > >
> > > > > > >> > >>> token)
> > > >
> > > > > > >> > >>> > > will
> > > >
> > > > > > >> > >>> > > > > be
> > > >
> > > > > > >> > >>> > > > > > > > > >> accepted. We
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > need
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > to
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > discuss which
> proposal
> > > > makes
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > more
> > > >
> > > > > > >> > >>> sense
> > > >
> > > > > > >> > >>> > and
> > > >
> > > > > > >> > >>> > > > we
> > > >
> > > > > > >> > >>> > > > > > can
> > > >
> > > > > > >> > >>> > > > > > > go
> > > >
> > > > > > >> > >>> > > > > > > > > >> over it
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > in
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > tomorrow's
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > meeting.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Also, can you
> forward
> > > the
> > > >
> > > > > > invite
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > to
> > > >
> > > > > > >> > >>> me?
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Thanks
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Parth
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > On Mon, May 23, 2016
> > at
> > > >
> > > > > 10:35
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > AM,
> > > >
> > > > > > >> > Jun
> > > >
> > > > > > >> > >>> > Rao <
> > > >
> > > > > > >> > >>> > > > > > > > > >> jun@confluent.io>
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > wrote:
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > Thanks for the
> KIP.
> > A
> > > > few
> > > >
> > > > > > >> > comments.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > 100. This
> > potentially
> > > > can
> > > >
> > > > > be
> > > >
> > > > > > >> > useful
> > > >
> > > > > > >> > >>> for
> > > >
> > > > > > >> > >>> > > > Kafka
> > > >
> > > > > > >> > >>> > > > > > > > Connect
> > > >
> > > > > > >> > >>> > > > > > > > > >> and
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > Kafka
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > rest
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > proxy
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > where a worker
> agent
> > > > will
> > > >
> > > > > > need
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > to
> > > >
> > > > > > >> > >>> run a
> > > >
> > > > > > >> > >>> > > > task
> > > >
> > > > > > >> > >>> > > > > on
> > > >
> > > > > > >> > >>> > > > > > > > > behalf
> > > >
> > > > > > >> > >>> > > > > > > > > >> of a
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > client.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > We
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > will
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > likely need to
> > change
> > > > how
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > those
> > > >
> > > > > > >> > >>> > services
> > > >
> > > > > > >> > >>> > > > use
> > > >
> > > > > > >> > >>> > > > > > > Kafka
> > > >
> > > > > > >> > >>> > > > > > > > > >> clients
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > >
> (producer/consumer).
> > > >
> > > > > Instead
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > of a
> > > >
> > > > > > >> > >>> > shared
> > > >
> > > > > > >> > >>> > > > > client
> > > >
> > > > > > >> > >>> > > > > > > per
> > > >
> > > > > > >> > >>> > > > > > > > > >> worker,
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > we
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > will
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > need
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > a
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > client per user
> task
> > > > since
> > > >
> > > > > > the
> > > >
> > > > > > >> > >>> > > > authentication
> > > >
> > > > > > >> > >>> > > > > > > > happens
> > > >
> > > > > > >> > >>> > > > > > > > > >> at the
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > connection
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > level. For Kafka
> > > > Connect,
> > > >
> > > > > > the
> > > >
> > > > > > >> > >>> renewer
> > > >
> > > > > > >> > >>> > > will
> > > >
> > > > > > >> > >>> > > > be
> > > >
> > > > > > >> > >>> > > > > > the
> > > >
> > > > > > >> > >>> > > > > > > > > >> workers.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > So,
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > we
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > probably
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > need to allow
> > multiple
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > renewers.
> > > >
> > > > > > >> > For
> > > >
> > > > > > >> > >>> > > Kafka
> > > >
> > > > > > >> > >>> > > > > rest
> > > >
> > > > > > >> > >>> > > > > > > > > proxy,
> > > >
> > > > > > >> > >>> > > > > > > > > >> the
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > renewer
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > can
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > probably just be
> the
> > > >
> > > > > creator
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > of
> > > >
> > > > > > >> > the
> > > >
> > > > > > >> > >>> > > token.
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > 101. Do we need
> new
> > > acl
> > > > on
> > > >
> > > > > > who
> > > >
> > > > > > >> > can
> > > >
> > > > > > >> > >>> > > request
> > > >
> > > > > > >> > >>> > > > > > > > delegation
> > > >
> > > > > > >> > >>> > > > > > > > > >> tokens?
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > 102. Do we
> recommend
> > > >
> > > > > people
> > > >
> > > > > > to
> > > >
> > > > > > >> > send
> > > >
> > > > > > >> > >>> > > > > delegation
> > > >
> > > > > > >> > >>> > > > > > > > tokens
> > > >
> > > > > > >> > >>> > > > > > > > > >> in an
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > encrypted
> > > >
> > > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > channel?
> > > >
> > > > > > >> > >>>
> > >
> >
>
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> <http://www.confluent.io/blog>
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Gwen Shapira <gw...@confluent.io>]

Thinking out loud here:

It looks like authentication with a delegation token is going to be
super-cheap, right? We just compare the token to a value in the broker
cache?

If I understood the KIP correctly, right now it suggests that
authentication happens when establishing the client-broker connection (as
normal for Kafka. But perhaps we want to consider authenticating every
request with delegation token (if exists)?

So a centralized app can create few producers, do the metadata request and
broker discovery with its own user auth, but then use delegation tokens to
allow performing produce/fetch requests as different users? Instead of
having to re-connect for each impersonated user?

This may over-complicate things quite a bit (basically adding extra
information in every request), but maybe it will be useful for
impersonation use-cases (which seem to drive much of the interest in this
KIP)?
Kafka Connect, NiFi and friends can probably use this to share clients
between multiple jobs, tasks, etc.

What do you think?

Gwen

On Tue, Dec 13, 2016 at 12:43 AM, Manikumar <ma...@gmail.com>
wrote:

> Ashish,
>
> Thank you for reviewing the KIP.  Please see the replies inline.
>
>
> > 1. How to disable delegation token authentication?
> >
> > This can be achieved in various ways, however I think reusing delegation
> > token secret config for this makes sense here. Avoids creating yet
> another
> > config and forces delegation token users to consciously set the secret.
> If
> > the secret is not set or set to empty string, brokers should turn off
> > delegation token support. This will however require a new error code to
> > indicate delegation token support is turned off on broker.
> >
>
>   Thanks for the suggestion. Option to turnoff delegation token
> authentication will be useful.
>   I'll update the KIP.
>
>
> >
> > 2. ACLs on delegation token?
> >
> > Do we need to have ACLs defined for tokens? I do not think it buys us
> > anything, as delegation token can be treated as impersonation of the
> owner.
> > Any thing the owner has permission to do, delegation tokens should be
> > allowed to do as well. If so, we probably won't need to return
> > authorization exception error code while creating delegation token. It
> > however would make sense to check renew and expire requests are coming
> from
> > owner or renewers of the token, but that does not require explicit acls.
> >
>
>
> Yes, We agreed to not have new acl on who can request delegation token.
>  I'll update the KIP.
>
>
> >
> > 3. How to restrict max life time of a token?
> >
> > Admins might want to restrict max life time of tokens created on a
> cluster,
> > and this can very from cluster to cluster based on use-cases. This might
> > warrant a separate broker config.
> >
> >
> Currently we  have "delegation.token.max.lifetime.sec" server config
> property
> May be we can take min(User supplied MaxTime, Server MaxTime) as max life
> time.
> I am open to add new config property.
>
> Few more comments based on recent KIP update.
> >
> > 1. Do we need a separate {{InvalidateTokenRequest}}? Can't we use
> > {{ExpireTokenRequest}} with with expiryDate set to anything before
> current
> > date?
> >
>
> makes sense. we don't need special request to cancel the token. We can use
> ExpireTokenRequest.
> I'll update the KIP.
>
>
> > 2. Can we change time field names to indicate their unit is milliseconds,
> > like, IssueDateMs, ExpiryDateMs, etc.?
> >
> >
>   Done.
>
>
> > 3. Can we allow users to renew a token for a specified amount of time? In
> > current version of KIP, renew request does not take time as a param, not
> > sure what is expiry time set to after renewal.
> >
> >
>  Yes, we need to specify renew period.  I'll update the KIP.
>
>
> Thanks,
> Mankumar
>
>
>
> >
> > On Mon, Dec 12, 2016 at 9:08 AM Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > >
> > >
> > > I would like to reinitiate the discussion on Delegation token support
> for
> > >
> > > Kafka.
> > >
> > >
> > >
> > > Brief summary of the past discussion:
> > >
> > >
> > >
> > > 1) Broker stores delegation tokens in zookeeper.  All brokers will
> have a
> > >
> > > cache backed by
> > >
> > >    zookeeper so they will all get notified whenever a new token is
> > >
> > > generated and they will
> > >
> > >    update their local cache whenever token state changes.
> > >
> > > 2) The current proposal does not support rotation of secret
> > >
> > > 3) Only allow the renewal by users that authenticated using *non*
> > >
> > > delegation token mechanism
> > >
> > > 4) KIP-84 proposes to support  SASL SCRAM mechanisms. Kafka clients can
> > >
> > > authenticate using
> > >
> > >    SCRAM-SHA-256, providing the delegation token HMAC as password.
> > >
> > >
> > >
> > > Updated the KIP with the following:
> > >
> > > 1. Protocol and Config changes
> > >
> > > 2. format of the data stored in ZK.
> > >
> > > 3. Changes to Java Clients/Usage of SASL SCRAM mechanism
> > >
> > >
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >
> > > 48+Delegation+token+support+for+Kafka
> > >
> > >
> > >
> > >
> > >
> > > Jun, Ashish, Gwen,
> > >
> > >
> > >
> > > Pl review the updated KIP.
> > >
> > >
> > >
> > > Thanks
> > >
> > > Manikumar
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Sep 29, 2016 at 9:56 PM, Ashish Singh <as...@cloudera.com>
> > wrote:
> > >
> > >
> > >
> > > > Harsha/ Gwen,
> > >
> > > >
> > >
> > > > How do we proceed here? I am willing to help out with here.
> > >
> > > >
> > >
> > > > On Fri, Sep 23, 2016 at 11:41 AM, Gwen Shapira <gw...@confluent.io>
> > > wrote:
> > >
> > > >
> > >
> > > > > Is it updated? are all concerns addressed? do you want to start a
> > vote?
> > >
> > > > >
> > >
> > > > > Sorry for being pushy, I do appreciate that we are all volunteers
> and
> > >
> > > > > finding time is difficult. This feature is important for anything
> > that
> > >
> > > > > integrates with Kafka (stream processors, Flume, NiFi, etc) and I
> > >
> > > > > don't want to see this getting stuck because we lack coordination
> > >
> > > > > within the community.
> > >
> > > > >
> > >
> > > > > On Thu, Sep 15, 2016 at 6:39 PM, Harsha Chintalapani <
> > kafka@harsha.io>
> > >
> > > > > wrote:
> > >
> > > > > > The only pending update for the KIP is to write up the protocol
> > > changes
> > >
> > > > > like
> > >
> > > > > > we've it KIP-4. I'll update the wiki.
> > >
> > > > > >
> > >
> > > > > >
> > >
> > > > > > On Thu, Sep 15, 2016 at 4:27 PM Ashish Singh <
> asingh@cloudera.com>
> > >
> > > > > wrote:
> > >
> > > > > >>
> > >
> > > > > >> I think we decided to not support secret rotation, I guess this
> > can
> > > be
> > >
> > > > > >> stated clearly on the KIP. Also, more details on how clients
> will
> > >
> > > > > perform
> > >
> > > > > >> token distribution and how CLI will look like will be helpful.
> > >
> > > > > >>
> > >
> > > > > >> On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <
> gwen@confluent.io>
> > >
> > > > > wrote:
> > >
> > > > > >>
> > >
> > > > > >> > Hi Guys,
> > >
> > > > > >> >
> > >
> > > > > >> > This discussion was dead for a while. Are there still
> > contentious
> > >
> > > > > >> > points? If not, why are there no votes?
> > >
> > > > > >> >
> > >
> > > > > >> > On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <ju...@confluent.io>
> > > wrote:
> > >
> > > > > >> > > Ashish,
> > >
> > > > > >> > >
> > >
> > > > > >> > > Yes, I will send out a KIP invite for next week to discuss
> > > KIP-48
> > >
> > > > > and
> > >
> > > > > >> > other
> > >
> > > > > >> > > remaining KIPs.
> > >
> > > > > >> > >
> > >
> > > > > >> > > Thanks,
> > >
> > > > > >> > >
> > >
> > > > > >> > > Jun
> > >
> > > > > >> > >
> > >
> > > > > >> > > On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh <
> > >
> > > > asingh@cloudera.com>
> > >
> > > > > >> > wrote:
> > >
> > > > > >> > >
> > >
> > > > > >> > >> Thanks Harsha!
> > >
> > > > > >> > >>
> > >
> > > > > >> > >> Jun, can we add KIP-48 to next KIP hangout's agenda. Also,
> we
> > > did
> > >
> > > > > not
> > >
> > > > > >> > >> actually make a call on when we should have next KIP call.
> As
> > >
> > > > there
> > >
> > > > > >> > >> are
> > >
> > > > > >> > a
> > >
> > > > > >> > >> few outstanding KIPs that could not be discussed this week,
> > can
> > >
> > > > we
> > >
> > > > > >> > >> have
> > >
> > > > > >> > a
> > >
> > > > > >> > >> KIP hangout call next week?
> > >
> > > > > >> > >>
> > >
> > > > > >> > >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha Chintalapani
> > >
> > > > > >> > >> <ka...@harsha.io>
> > >
> > > > > >> > >> wrote:
> > >
> > > > > >> > >>
> > >
> > > > > >> > >>> Ashish,
> > >
> > > > > >> > >>>         Yes we are working on it. Lets discuss in the next
> > KIP
> > >
> > > > > >> > >>> meeting.
> > >
> > > > > >> > >>> I'll join.
> > >
> > > > > >> > >>> -Harsha
> > >
> > > > > >> > >>>
> > >
> > > > > >> > >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh <
> > >
> > > > > asingh@cloudera.com>
> > >
> > > > > >> > >>> wrote:
> > >
> > > > > >> > >>>
> > >
> > > > > >> > >>> > Hello Harsha,
> > >
> > > > > >> > >>> >
> > >
> > > > > >> > >>> > Are you still working on this? Wondering if we can
> discuss
> > >
> > > > this
> > >
> > > > > in
> > >
> > > > > >> > next
> > >
> > > > > >> > >>> KIP
> > >
> > > > > >> > >>> > meeting, if you can join.
> > >
> > > > > >> > >>> >
> > >
> > > > > >> > >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha Chintalapani <
> > >
> > > > > >> > kafka@harsha.io>
> > >
> > > > > >> > >>> > wrote:
> > >
> > > > > >> > >>> >
> > >
> > > > > >> > >>> > > Hi Grant,
> > >
> > > > > >> > >>> > >           We are working on it. Will add the details
> to
> > > KIP
> > >
> > > > > >> > >>> > > about
> > >
> > > > > >> > the
> > >
> > > > > >> > >>> > > request protocol.
> > >
> > > > > >> > >>> > >
> > >
> > > > > >> > >>> > > Thanks,
> > >
> > > > > >> > >>> > > Harsha
> > >
> > > > > >> > >>> > >
> > >
> > > > > >> > >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant Henke
> > >
> > > > > >> > >>> > > <gh...@cloudera.com>
> > >
> > > > > >> > >>> wrote:
> > >
> > > > > >> > >>> > >
> > >
> > > > > >> > >>> > > > Hi Parth,
> > >
> > > > > >> > >>> > > >
> > >
> > > > > >> > >>> > > > Are you still working on this? If you need any help
> > > please
> > >
> > > > > >> > >>> > > > don't
> > >
> > > > > >> > >>> > hesitate
> > >
> > > > > >> > >>> > > > to ask.
> > >
> > > > > >> > >>> > > >
> > >
> > > > > >> > >>> > > > Thanks,
> > >
> > > > > >> > >>> > > > Grant
> > >
> > > > > >> > >>> > > >
> > >
> > > > > >> > >>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <
> > >
> > > > jun@confluent.io>
> > >
> > > > > >> > wrote:
> > >
> > > > > >> > >>> > > >
> > >
> > > > > >> > >>> > > > > Parth,
> > >
> > > > > >> > >>> > > > >
> > >
> > > > > >> > >>> > > > > Thanks for the reply.
> > >
> > > > > >> > >>> > > > >
> > >
> > > > > >> > >>> > > > > It makes sense to only allow the renewal by users
> > that
> > >
> > > > > >> > >>> authenticated
> > >
> > > > > >> > >>> > > > using
> > >
> > > > > >> > >>> > > > > *non* delegation token mechanism. Then, should we
> > make
> > >
> > > > the
> > >
> > > > > >> > >>> renewal a
> > >
> > > > > >> > >>> > > > list?
> > >
> > > > > >> > >>> > > > > For example, in the case of rest proxy, it will be
> > >
> > > > useful
> > >
> > > > > >> > >>> > > > > for
> > >
> > > > > >> > >>> every
> > >
> > > > > >> > >>> > > > > instance of rest proxy to be able to renew the
> > tokens.
> > >
> > > > > >> > >>> > > > >
> > >
> > > > > >> > >>> > > > > It would be clearer if we can document the request
> > >
> > > > > protocol
> > >
> > > > > >> > like
> > >
> > > > > >> > >>> > > > >
> > >
> > > > > >> > >>> > > > >
> > >
> > > > > >> > >>> > > > https://cwiki.apache.org/confl
> > uence/display/KAFKA/KIP-
> > >
> > > > > >> > >>> > >
> > >
> > > > > >> > >>> > > 4+-+Command+line+and+centralized+administrative+
> > >
> > > > > operations#KIP-4-
> > >
> > > > > >> > >>> > > Commandlineandcentralizedadministrativeoperations-
> > >
> > > > > >> > >>> > > CreateTopicsRequest(KAFKA-
> 2945):(VotedandPlannedforin0.
> > >
> > > > > 10.1.0)
> > >
> > > > > >> > >>> > > > > .
> > >
> > > > > >> > >>> > > > >
> > >
> > > > > >> > >>> > > > > It would also be useful to document the client
> APIs.
> > >
> > > > > >> > >>> > > > >
> > >
> > > > > >> > >>> > > > > Thanks,
> > >
> > > > > >> > >>> > > > >
> > >
> > > > > >> > >>> > > > > Jun
> > >
> > > > > >> > >>> > > > >
> > >
> > > > > >> > >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM, parth brahmbhatt
> <
> > >
> > > > > >> > >>> > > > > brahmbhatt.parth@gmail.com> wrote:
> > >
> > > > > >> > >>> > > > >
> > >
> > > > > >> > >>> > > > > > Hi,
> > >
> > > > > >> > >>> > > > > >
> > >
> > > > > >> > >>> > > > > > I am suggesting that we will only allow the
> > renewal
> > > by
> > >
> > > > > >> > >>> > > > > > users
> > >
> > > > > >> > >>> that
> > >
> > > > > >> > >>> > > > > > authenticated using *non* delegation token
> > > mechanism.
> > >
> > > > > For
> > >
> > > > > >> > >>> example,
> > >
> > > > > >> > >>> > If
> > >
> > > > > >> > >>> > > > > user
> > >
> > > > > >> > >>> > > > > > Alice authenticated using kerberos and requested
> > >
> > > > > >> > >>> > > > > > delegation
> > >
> > > > > >> > >>> tokens,
> > >
> > > > > >> > >>> > > > only
> > >
> > > > > >> > >>> > > > > > user Alice authenticated via non delegation
> token
> > >
> > > > > >> > >>> > > > > > mechanism
> > >
> > > > > >> > can
> > >
> > > > > >> > >>> > > renew.
> > >
> > > > > >> > >>> > > > > > Clients that have  access to delegation tokens
> can
> > > not
> > >
> > > > > >> > >>> > > > > > issue
> > >
> > > > > >> > >>> > renewal
> > >
> > > > > >> > >>> > > > > > request for renewing their own token and this is
> > >
> > > > > primarily
> > >
> > > > > >> > >>> > important
> > >
> > > > > >> > >>> > > to
> > >
> > > > > >> > >>> > > > > > reduce the time window for which a compromised
> > token
> > >
> > > > > will
> > >
> > > > > >> > >>> > > > > > be
> > >
> > > > > >> > >>> valid.
> > >
> > > > > >> > >>> > > > > >
> > >
> > > > > >> > >>> > > > > > To clarify, Yes any authenticated user can
> request
> > >
> > > > > >> > >>> > > > > > delegation
> > >
> > > > > >> > >>> > tokens
> > >
> > > > > >> > >>> > > > but
> > >
> > > > > >> > >>> > > > > > even here I would recommend to avoid creating a
> > > chain
> > >
> > > > > >> > >>> > > > > > where a
> > >
> > > > > >> > >>> > client
> > >
> > > > > >> > >>> > > > > > authenticated via delegation token request for
> > more
> > >
> > > > > >> > delegation
> > >
> > > > > >> > >>> > > tokens.
> > >
> > > > > >> > >>> > > > > > Basically anyone can request delegation token,
> as
> > > long
> > >
> > > > > as
> > >
> > > > > >> > they
> > >
> > > > > >> > >>> > > > > authenticate
> > >
> > > > > >> > >>> > > > > > via a non delegation token mechanism.
> > >
> > > > > >> > >>> > > > > >
> > >
> > > > > >> > >>> > > > > > Aren't classes listed here
> > >
> > > > > >> > >>> > > > > > <
> > >
> > > > > >> > >>> > > > > >
> > >
> > > > > >> > >>> > > > >
> > >
> > > > > >> > >>> > > > https://cwiki.apache.org/confl
> > uence/display/KAFKA/KIP-
> > >
> > > > > >> > >>> > > 48+Delegation+token+support+fo
> > >
> > > > r+Kafka#KIP-48Delegationtokens
> > >
> > > > > >> > >>> upportforKaf
> > >
> > > > > >> > >>> > > ka-PublicInterfaces
> > >
> > > > > >> > >>> > > > > > >
> > >
> > > > > >> > >>> > > > > > sufficient?
> > >
> > > > > >> > >>> > > > > >
> > >
> > > > > >> > >>> > > > > > Thanks
> > >
> > > > > >> > >>> > > > > > Parth
> > >
> > > > > >> > >>> > > > > >
> > >
> > > > > >> > >>> > > > > >
> > >
> > > > > >> > >>> > > > > >
> > >
> > > > > >> > >>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao
> > >
> > > > > >> > >>> > > > > > <ju...@confluent.io>
> > >
> > > > > >> > >>> wrote:
> > >
> > > > > >> > >>> > > > > >
> > >
> > > > > >> > >>> > > > > > > Parth,
> > >
> > > > > >> > >>> > > > > > >
> > >
> > > > > >> > >>> > > > > > > Thanks for the reply. A couple of comments
> > inline
> > >
> > > > > below.
> > >
> > > > > >> > >>> > > > > > >
> > >
> > > > > >> > >>> > > > > > > On Tue, Jun 21, 2016 at 10:36 AM, parth
> > > brahmbhatt <
> > >
> > > > > >> > >>> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > >
> > > > > >> > >>> > > > > > >
> > >
> > > > > >> > >>> > > > > > > > 1. Who / how are tokens renewed? By original
> > >
> > > > > requester
> > >
> > > > > >> > >>> only? or
> > >
> > > > > >> > >>> > > > using
> > >
> > > > > >> > >>> > > > > > > > Kerberos
> > >
> > > > > >> > >>> > > > > > > > auth only?
> > >
> > > > > >> > >>> > > > > > > > My recommendation is to do this only using
> > >
> > > > Kerberos
> > >
> > > > > >> > >>> > > > > > > > auth
> > >
> > > > > >> > and
> > >
> > > > > >> > >>> > only
> > >
> > > > > >> > >>> > > > > threw
> > >
> > > > > >> > >>> > > > > > > the
> > >
> > > > > >> > >>> > > > > > > > renewer specified during the acquisition
> > > request.
> > >
> > > > > >> > >>> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > Hmm, not sure that I follow this. Are you
> saying
> > >
> > > > that
> > >
> > > > > >> > >>> > > > > > > any
> > >
> > > > > >> > >>> client
> > >
> > > > > >> > >>> > > > > > > authenticated with the delegation token can
> > renew,
> > >
> > > > > i.e.
> > >
> > > > > >> > there
> > >
> > > > > >> > >>> is
> > >
> > > > > >> > >>> > no
> > >
> > > > > >> > >>> > > > > > renewer
> > >
> > > > > >> > >>> > > > > > > needed?
> > >
> > > > > >> > >>> > > > > > >
> > >
> > > > > >> > >>> > > > > > > Also, just to be clear, any authenticated
> client
> > >
> > > > > (either
> > >
> > > > > >> > >>> through
> > >
> > > > > >> > >>> > > SASL
> > >
> > > > > >> > >>> > > > > or
> > >
> > > > > >> > >>> > > > > > > SSL) can request a delegation token for the
> > >
> > > > > >> > >>> > > > > > > authenticated
> > >
> > > > > >> > >>> user,
> > >
> > > > > >> > >>> > > > right?
> > >
> > > > > >> > >>> > > > > > >
> > >
> > > > > >> > >>> > > > > > >
> > >
> > > > > >> > >>> > > > > > > > 2. Are tokens stored on each broker or in
> ZK?
> > >
> > > > > >> > >>> > > > > > > > My recommendation is still to store in ZK or
> > not
> > >
> > > > > store
> > >
> > > > > >> > them
> > >
> > > > > >> > >>> at
> > >
> > > > > >> > >>> > > all.
> > >
> > > > > >> > >>> > > > > The
> > >
> > > > > >> > >>> > > > > > > > whole controller based distribution is too
> > much
> > >
> > > > > >> > >>> > > > > > > > overhead
> > >
> > > > > >> > >>> with
> > >
> > > > > >> > >>> > not
> > >
> > > > > >> > >>> > > > > much
> > >
> > > > > >> > >>> > > > > > to
> > >
> > > > > >> > >>> > > > > > > > achieve.
> > >
> > > > > >> > >>> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > 3. How are tokens invalidated / expired?
> > >
> > > > > >> > >>> > > > > > > > Either by expiration time out or through an
> > >
> > > > explicit
> > >
> > > > > >> > >>> request to
> > >
> > > > > >> > >>> > > > > > > invalidate.
> > >
> > > > > >> > >>> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > 4. Which encryption algorithm is used?
> > >
> > > > > >> > >>> > > > > > > > SCRAM
> > >
> > > > > >> > >>> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > 5. What is the impersonation proposal (it
> > wasn't
> > >
> > > > in
> > >
> > > > > >> > >>> > > > > > > > the
> > >
> > > > > >> > KIP
> > >
> > > > > >> > >>> but
> > >
> > > > > >> > >>> > > was
> > >
> > > > > >> > >>> > > > > > > > discussed
> > >
> > > > > >> > >>> > > > > > > > in this thread)?
> > >
> > > > > >> > >>> > > > > > > > There is no imperonation proposal. I tried
> and
> > >
> > > > > >> > >>> > > > > > > > explained
> > >
> > > > > >> > how
> > >
> > > > > >> > >>> > its
> > >
> > > > > >> > >>> > > a
> > >
> > > > > >> > >>> > > > > > > > different problem and why its not really
> > > necessary
> > >
> > > > > to
> > >
> > > > > >> > >>> discuss
> > >
> > > > > >> > >>> > > that
> > >
> > > > > >> > >>> > > > as
> > >
> > > > > >> > >>> > > > > > > part
> > >
> > > > > >> > >>> > > > > > > > of this KIP.  This KIP will not support any
> > >
> > > > > >> > impersonation,
> > >
> > > > > >> > >>> it
> > >
> > > > > >> > >>> > > will
> > >
> > > > > >> > >>> > > > > just
> > >
> > > > > >> > >>> > > > > > > be
> > >
> > > > > >> > >>> > > > > > > > another way to authenticate.
> > >
> > > > > >> > >>> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > 6. Do we need new ACLs, if so - for what
> > > actions?
> > >
> > > > > >> > >>> > > > > > > > We do not need new ACLs.
> > >
> > > > > >> > >>> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > Could we document the format of the new
> > >
> > > > > request/response
> > >
> > > > > >> > and
> > >
> > > > > >> > >>> > their
> > >
> > > > > >> > >>> > > > > > > associated Resource and Operation for ACL?
> > >
> > > > > >> > >>> > > > > > >
> > >
> > > > > >> > >>> > > > > > >
> > >
> > > > > >> > >>> > > > > > > > 7. How would the delegation token be
> > configured
> > > in
> > >
> > > > > the
> > >
> > > > > >> > >>> client?
> > >
> > > > > >> > >>> > > > > > > > Should be through config. I wasn't planning
> on
> > >
> > > > > >> > >>> > > > > > > > supporting
> > >
> > > > > >> > >>> JAAS
> > >
> > > > > >> > >>> > > for
> > >
> > > > > >> > >>> > > > > > > tokens.
> > >
> > > > > >> > >>> > > > > > > > I don't believe hadoop does this either.
> > >
> > > > > >> > >>> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > Thanks
> > >
> > > > > >> > >>> > > > > > > > Parth
> > >
> > > > > >> > >>> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <
> > >
> > > > > >> > jun@confluent.io>
> > >
> > > > > >> > >>> > > wrote:
> > >
> > > > > >> > >>> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > Harsha,
> > >
> > > > > >> > >>> > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > Another question.
> > >
> > > > > >> > >>> > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > 9. How would the delegation token be
> > > configured
> > >
> > > > in
> > >
> > > > > >> > >>> > > > > > > > > the
> > >
> > > > > >> > >>> > client?
> > >
> > > > > >> > >>> > > > The
> > >
> > > > > >> > >>> > > > > > > > standard
> > >
> > > > > >> > >>> > > > > > > > > way is to do this through JAAS. However,
> we
> > > will
> > >
> > > > > >> > >>> > > > > > > > > need
> > >
> > > > > >> > to
> > >
> > > > > >> > >>> > think
> > >
> > > > > >> > >>> > > > > > through
> > >
> > > > > >> > >>> > > > > > > if
> > >
> > > > > >> > >>> > > > > > > > > this is convenient in a shared
> environment.
> > > For
> > >
> > > > > >> > example,
> > >
> > > > > >> > >>> > when a
> > >
> > > > > >> > >>> > > > new
> > >
> > > > > >> > >>> > > > > > > task
> > >
> > > > > >> > >>> > > > > > > > is
> > >
> > > > > >> > >>> > > > > > > > > added to a Storm worker node, do we need
> to
> > >
> > > > > >> > >>> > > > > > > > > dynamically
> > >
> > > > > >> > >>> add a
> > >
> > > > > >> > >>> > > new
> > >
> > > > > >> > >>> > > > > > > section
> > >
> > > > > >> > >>> > > > > > > > > in the JAAS file? It may be more
> convenient
> > if
> > >
> > > > we
> > >
> > > > > >> > >>> > > > > > > > > can
> > >
> > > > > >> > >>> pass in
> > >
> > > > > >> > >>> > > the
> > >
> > > > > >> > >>> > > > > > token
> > >
> > > > > >> > >>> > > > > > > > > through the config directly w/o going
> > through
> > >
> > > > > JAAS.
> > >
> > > > > >> > >>> > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > Are you or Parth still actively working on
> > > this
> > >
> > > > > KIP?
> > >
> > > > > >> > >>> > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > Thanks,
> > >
> > > > > >> > >>> > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > Jun
> > >
> > > > > >> > >>> > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <
> > >
> > > > > >> > >>> jun@confluent.io>
> > >
> > > > > >> > >>> > > > wrote:
> > >
> > > > > >> > >>> > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > > Just to add on that list.
> > >
> > > > > >> > >>> > > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > > 2. It would be good to document the
> format
> > > of
> > >
> > > > > the
> > >
> > > > > >> > data
> > >
> > > > > >> > >>> > stored
> > >
> > > > > >> > >>> > > > in
> > >
> > > > > >> > >>> > > > > > ZK.
> > >
> > > > > >> > >>> > > > > > > > > > 7. Earlier, there was a discussion on
> > > whether
> > >
> > > > > the
> > >
> > > > > >> > tokens
> > >
> > > > > >> > >>> > > should
> > >
> > > > > >> > >>> > > > > be
> > >
> > > > > >> > >>> > > > > > > > > > propagated through ZK like
> > config/acl/quota,
> > >
> > > > or
> > >
> > > > > >> > through
> > >
> > > > > >> > >>> the
> > >
> > > > > >> > >>> > > > > > > controller.
> > >
> > > > > >> > >>> > > > > > > > > > Currently, the controller is only
> designed
> > > for
> > >
> > > > > >> > >>> propagating
> > >
> > > > > >> > >>> > > > topic
> > >
> > > > > >> > >>> > > > > > > > > metadata,
> > >
> > > > > >> > >>> > > > > > > > > > but not other data.
> > >
> > > > > >> > >>> > > > > > > > > > 8. Should we use SCRAM to send the token
> > >
> > > > instead
> > >
> > > > > >> > >>> > > > > > > > > > of
> > >
> > > > > >> > >>> > > DIGEST-MD5
> > >
> > > > > >> > >>> > > > > > since
> > >
> > > > > >> > >>> > > > > > > > it's
> > >
> > > > > >> > >>> > > > > > > > > > deprecated?
> > >
> > > > > >> > >>> > > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > > Also, the images in the wiki seem
> broken.
> > >
> > > > > >> > >>> > > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > > Thanks,
> > >
> > > > > >> > >>> > > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > > Jun
> > >
> > > > > >> > >>> > > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen
> > >
> > > > Shapira <
> > >
> > > > > >> > >>> > > > > gwen@confluent.io>
> > >
> > > > > >> > >>> > > > > > > > > wrote:
> > >
> > > > > >> > >>> > > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> From what I can see, remaining
> questions
> > > are:
> > >
> > > > > >> > >>> > > > > > > > > >>
> > >
> > > > > >> > >>> > > > > > > > > >> 1. Who / how are tokens renewed? By
> > > original
> > >
> > > > > >> > requester
> > >
> > > > > >> > >>> > only?
> > >
> > > > > >> > >>> > > > or
> > >
> > > > > >> > >>> > > > > > > using
> > >
> > > > > >> > >>> > > > > > > > > >> Kerberos auth only?
> > >
> > > > > >> > >>> > > > > > > > > >> 2. Are tokens stored on each broker or
> in
> > > ZK?
> > >
> > > > > >> > >>> > > > > > > > > >> 3. How are tokens invalidated /
> expired?
> > >
> > > > > >> > >>> > > > > > > > > >> 4. Which encryption algorithm is used?
> > >
> > > > > >> > >>> > > > > > > > > >> 5. What is the impersonation proposal
> (it
> > >
> > > > > wasn't
> > >
> > > > > >> > >>> > > > > > > > > >> in
> > >
> > > > > >> > the
> > >
> > > > > >> > >>> > KIP
> > >
> > > > > >> > >>> > > > but
> > >
> > > > > >> > >>> > > > > > was
> > >
> > > > > >> > >>> > > > > > > > > >> discussed in this thread)?
> > >
> > > > > >> > >>> > > > > > > > > >> 6. Do we need new ACLs, if so - for
> what
> > >
> > > > > actions?
> > >
> > > > > >> > >>> > > > > > > > > >>
> > >
> > > > > >> > >>> > > > > > > > > >> Gwen
> > >
> > > > > >> > >>> > > > > > > > > >>
> > >
> > > > > >> > >>> > > > > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha
> <
> > >
> > > > > >> > >>> kafka@harsha.io>
> > >
> > > > > >> > >>> > > > wrote:
> > >
> > > > > >> > >>> > > > > > > > > >> > Jun & Ismael,
> > >
> > > > > >> > >>> > > > > > > > > >> >
> Unfortunately
> > I
> > >
> > > > > >> > >>> > > > > > > > > >> > couldn't
> > >
> > > > > >> > >>> attend
> > >
> > > > > >> > >>> > > the
> > >
> > > > > >> > >>> > > > > KIP
> > >
> > > > > >> > >>> > > > > > > > > meeting
> > >
> > > > > >> > >>> > > > > > > > > >> >                          when
> > delegation
> > >
> > > > > tokens
> > >
> > > > > >> > >>> > discussed.
> > >
> > > > > >> > >>> > > > > > > > Appreciate
> > >
> > > > > >> > >>> > > > > > > > > if
> > >
> > > > > >> > >>> > > > > > > > > >> >                          you can
> update
> > > the
> > >
> > > > > >> > thread if
> > >
> > > > > >> > >>> > you
> > >
> > > > > >> > >>> > > > have
> > >
> > > > > >> > >>> > > > > > any
> > >
> > > > > >> > >>> > > > > > > > > >> >                          further
> > > questions.
> > >
> > > > > >> > >>> > > > > > > > > >> > Thanks,
> > >
> > > > > >> > >>> > > > > > > > > >> > Harsha
> > >
> > > > > >> > >>> > > > > > > > > >> >
> > >
> > > > > >> > >>> > > > > > > > > >> > On Tue, May 24, 2016, at 11:32 AM,
> > Liquan
> > >
> > > > Pei
> > >
> > > > > >> > wrote:
> > >
> > > > > >> > >>> > > > > > > > > >> >> It seems that the links to images in
> > the
> > >
> > > > KIP
> > >
> > > > > >> > >>> > > > > > > > > >> >> are
> > >
> > > > > >> > >>> > broken.
> > >
> > > > > >> > >>> > > > > > > > > >> >>
> > >
> > > > > >> > >>> > > > > > > > > >> >> Liquan
> > >
> > > > > >> > >>> > > > > > > > > >> >>
> > >
> > > > > >> > >>> > > > > > > > > >> >> On Tue, May 24, 2016 at 9:33 AM,
> parth
> > >
> > > > > >> > brahmbhatt <
> > >
> > > > > >> > >>> > > > > > > > > >> >> brahmbhatt.parth@gmail.com> wrote:
> > >
> > > > > >> > >>> > > > > > > > > >> >>
> > >
> > > > > >> > >>> > > > > > > > > >> >> > 110. What does
> getDelegationTokenAs
> > >
> > > > mean?
> > >
> > > > > >> > >>> > > > > > > > > >> >> > In the current proposal we only
> > allow
> > > a
> > >
> > > > > user
> > >
> > > > > >> > >>> > > > > > > > > >> >> > to
> > >
> > > > > >> > >>> get
> > >
> > > > > >> > >>> > > > > > delegation
> > >
> > > > > >> > >>> > > > > > > > > token
> > >
> > > > > >> > >>> > > > > > > > > >> for
> > >
> > > > > >> > >>> > > > > > > > > >> >> > the identity that it authenticated
> > as
> > >
> > > > > using
> > >
> > > > > >> > >>> another
> > >
> > > > > >> > >>> > > > > > mechanism,
> > >
> > > > > >> > >>> > > > > > > > i.e.
> > >
> > > > > >> > >>> > > > > > > > > >> A user
> > >
> > > > > >> > >>> > > > > > > > > >> >> > that authenticate using a keytab
> for
> > >
> > > > > >> > >>> > > > > > > > > >> >> > principal
> > >
> > > > > >> > >>> > > > > > > user1@EXAMPLE.COM
> > >
> > > > > >> > >>> > > > > > > > > >> will get
> > >
> > > > > >> > >>> > > > > > > > > >> >> > delegation tokens for that user
> > only.
> > > In
> > >
> > > > > >> > future I
> > >
> > > > > >> > >>> > think
> > >
> > > > > >> > >>> > > > we
> > >
> > > > > >> > >>> > > > > > will
> > >
> > > > > >> > >>> > > > > > > > > have
> > >
> > > > > >> > >>> > > > > > > > > >> to
> > >
> > > > > >> > >>> > > > > > > > > >> >> > extend support such that we allow
> > some
> > >
> > > > set
> > >
> > > > > >> > >>> > > > > > > > > >> >> > of
> > >
> > > > > >> > >>> users (
> > >
> > > > > >> > >>> > > > > > > > > >> >> > kafka-rest-user@EXAMPLE.COM,
> > >
> > > > > >> > >>> > storm-nimbus@EXAMPLE.COM)
> > >
> > > > > >> > >>> > > > to
> > >
> > > > > >> > >>> > > > > > > > acquire
> > >
> > > > > >> > >>> > > > > > > > > >> >> > delegation tokens on behalf of
> other
> > >
> > > > users
> > >
> > > > > >> > whose
> > >
> > > > > >> > >>> > > identity
> > >
> > > > > >> > >>> > > > > > they
> > >
> > > > > >> > >>> > > > > > > > have
> > >
> > > > > >> > >>> > > > > > > > > >> >> > verified independently.  Kafka
> > brokers
> > >
> > > > > will
> > >
> > > > > >> > have
> > >
> > > > > >> > >>> ACLs
> > >
> > > > > >> > >>> > > to
> > >
> > > > > >> > >>> > > > > > > control
> > >
> > > > > >> > >>> > > > > > > > > >> which
> > >
> > > > > >> > >>> > > > > > > > > >> >> > users are allowed to impersonate
> > other
> > >
> > > > > users
> > >
> > > > > >> > and
> > >
> > > > > >> > >>> get
> > >
> > > > > >> > >>> > > > tokens
> > >
> > > > > >> > >>> > > > > > on
> > >
> > > > > >> > >>> > > > > > > > > >> behalf of
> > >
> > > > > >> > >>> > > > > > > > > >> >> > them. Overall Impersonation is a
> > whole
> > >
> > > > > >> > different
> > >
> > > > > >> > >>> > > problem
> > >
> > > > > >> > >>> > > > in
> > >
> > > > > >> > >>> > > > > > my
> > >
> > > > > >> > >>> > > > > > > > > >> opinion and
> > >
> > > > > >> > >>> > > > > > > > > >> >> > I think we can tackle it in
> separate
> > >
> > > > KIP.
> > >
> > > > > >> > >>> > > > > > > > > >> >> >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > 111. What's the typical rate of
> > > getting
> > >
> > > > > and
> > >
> > > > > >> > >>> renewing
> > >
> > > > > >> > >>> > > > > > delegation
> > >
> > > > > >> > >>> > > > > > > > > >> tokens?
> > >
> > > > > >> > >>> > > > > > > > > >> >> > Typically this should be very very
> > > low,
> > >
> > > > 1
> > >
> > > > > >> > request
> > >
> > > > > >> > >>> per
> > >
> > > > > >> > >>> > > > > minute
> > >
> > > > > >> > >>> > > > > > > is a
> > >
> > > > > >> > >>> > > > > > > > > >> >> > relatively high estimate. However
> it
> > >
> > > > > depends
> > >
> > > > > >> > >>> > > > > > > > > >> >> > on
> > >
> > > > > >> > >>> the
> > >
> > > > > >> > >>> > > token
> > >
> > > > > >> > >>> > > > > > > > > >> expiration. I am
> > >
> > > > > >> > >>> > > > > > > > > >> >> > less worried about the extra load
> it
> > >
> > > > puts
> > >
> > > > > on
> > >
> > > > > >> > >>> > controller
> > >
> > > > > >> > >>> > > > vs
> > >
> > > > > >> > >>> > > > > > the
> > >
> > > > > >> > >>> > > > > > > > > added
> > >
> > > > > >> > >>> > > > > > > > > >> >> > complexity and the value it
> offers.
> > >
> > > > > >> > >>> > > > > > > > > >> >> >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > Thanks
> > >
> > > > > >> > >>> > > > > > > > > >> >> > Parth
> > >
> > > > > >> > >>> > > > > > > > > >> >> >
> > >
> > > > > >> > >>> > > > > > > > > >> >> >
> > >
> > > > > >> > >>> > > > > > > > > >> >> >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > On Tue, May 24, 2016 at 7:30 AM,
> > > Ismael
> > >
> > > > > Juma
> > >
> > > > > >> > >>> > > > > > > > > >> >> > <
> > >
> > > > > >> > >>> > > > > > > ismael@juma.me.uk>
> > >
> > > > > >> > >>> > > > > > > > > >> wrote:
> > >
> > > > > >> > >>> > > > > > > > > >> >> >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > Thanks Rajini. It would probably
> > >
> > > > > require a
> > >
> > > > > >> > >>> separate
> > >
> > > > > >> > >>> > > KIP
> > >
> > > > > >> > >>> > > > > as
> > >
> > > > > >> > >>> > > > > > it
> > >
> > > > > >> > >>> > > > > > > > > will
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > introduce user visible changes.
> We
> > >
> > > > could
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > also
> > >
> > > > > >> > >>> > update
> > >
> > > > > >> > >>> > > > > KIP-48
> > >
> > > > > >> > >>> > > > > > > to
> > >
> > > > > >> > >>> > > > > > > > > >> have this
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > information, but it seems
> cleaner
> > to
> > >
> > > > do
> > >
> > > > > it
> > >
> > > > > >> > >>> > > separately.
> > >
> > > > > >> > >>> > > > We
> > >
> > > > > >> > >>> > > > > > can
> > >
> > > > > >> > >>> > > > > > > > > >> discuss
> > >
> > > > > >> > >>> > > > > > > > > >> >> > that
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > in the KIP call today.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > Ismael
> > >
> > > > > >> > >>> > > > > > > > > >> >> > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > On Tue, May 24, 2016 at 3:19 PM,
> > >
> > > > Rajini
> > >
> > > > > >> > Sivaram
> > >
> > > > > >> > >>> <
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > rajinisivaram@googlemail.com>
> > > wrote:
> > >
> > > > > >> > >>> > > > > > > > > >> >> > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > Ismael,
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > I have created a JIRA (
> > >
> > > > > >> > >>> > > > > > > > > >> >> > https://issues.apache.org/
> > >
> > > > > >> > jira/browse/KAFKA-3751)
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > for adding SCRAM as a SASL
> > >
> > > > mechanism.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > Would
> > >
> > > > > >> > >>> that
> > >
> > > > > >> > >>> > > need
> > >
> > > > > >> > >>> > > > > > > another
> > >
> > > > > >> > >>> > > > > > > > > >> KIP? If
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > KIP-48 will use this
> mechanism,
> > > can
> > >
> > > > > this
> > >
> > > > > >> > just
> > >
> > > > > >> > >>> be
> > >
> > > > > >> > >>> > a
> > >
> > > > > >> > >>> > > > JIRA
> > >
> > > > > >> > >>> > > > > > > that
> > >
> > > > > >> > >>> > > > > > > > > gets
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > reviewed
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > when the PR is ready?
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > Thank you,
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > Rajini
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > On Tue, May 24, 2016 at 2:46
> PM,
> > >
> > > > > Ismael
> > >
> > > > > >> > Juma <
> > >
> > > > > >> > >>> > > > > > > > > ismael@juma.me.uk>
> > >
> > > > > >> > >>> > > > > > > > > >> >> > wrote:
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > Thanks Rajini, SCRAM seems
> > like
> > > a
> > >
> > > > > good
> > >
> > > > > >> > >>> > candidate.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > Gwen had independently
> > mentioned
> > >
> > > > > this
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > as
> > >
> > > > > >> > a
> > >
> > > > > >> > >>> SASL
> > >
> > > > > >> > >>> > > > > > mechanism
> > >
> > > > > >> > >>> > > > > > > > > that
> > >
> > > > > >> > >>> > > > > > > > > >> might
> > >
> > > > > >> > >>> > > > > > > > > >> >> > be
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > useful for Kafka and I have
> > been
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > meaning
> > >
> > > > > >> > to
> > >
> > > > > >> > >>> > check
> > >
> > > > > >> > >>> > > > it
> > >
> > > > > >> > >>> > > > > in
> > >
> > > > > >> > >>> > > > > > > > more
> > >
> > > > > >> > >>> > > > > > > > > >> detail.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > Good
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > to know that you are willing
> > to
> > >
> > > > > >> > contribute
> > >
> > > > > >> > >>> an
> > >
> > > > > >> > >>> > > > > > > > implementation.
> > >
> > > > > >> > >>> > > > > > > > > >> Maybe
> > >
> > > > > >> > >>> > > > > > > > > >> >> > we
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > should file a separate JIRA
> > for
> > >
> > > > > this?
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > Ismael
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > On Tue, May 24, 2016 at 2:12
> > PM,
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > Rajini
> > >
> > > > > >> > >>> > Sivaram <
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > >
> rajinisivaram@googlemail.com>
> > >
> > > > > wrote:
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > SCRAM (Salted Challenge
> > > Response
> > >
> > > > > >> > >>> > Authentication
> > >
> > > > > >> > >>> > > > > > > > Mechanism)
> > >
> > > > > >> > >>> > > > > > > > > >> is a
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > better
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > mechanism than Digest-MD5.
> > > Java
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > doesn't
> > >
> > > > > >> > >>> come
> > >
> > > > > >> > >>> > > > with a
> > >
> > > > > >> > >>> > > > > > > > > built-in
> > >
> > > > > >> > >>> > > > > > > > > >> SCRAM
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > SaslServer or SaslClient,
> > but
> > > I
> > >
> > > > > will
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > be
> > >
> > > > > >> > >>> happy
> > >
> > > > > >> > >>> > > to
> > >
> > > > > >> > >>> > > > > add
> > >
> > > > > >> > >>> > > > > > > > > support
> > >
> > > > > >> > >>> > > > > > > > > >> in
> > >
> > > > > >> > >>> > > > > > > > > >> >> > Kafka
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > since
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > it would be a useful
> > mechanism
> > >
> > > > to
> > >
> > > > > >> > support
> > >
> > > > > >> > >>> > > anyway.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > >
> > https://tools.ietf.org/html/
> > >
> > > > > rfc7677
> > >
> > > > > >> > >>> > describes
> > >
> > > > > >> > >>> > > > the
> > >
> > > > > >> > >>> > > > > > > > protocol
> > >
> > > > > >> > >>> > > > > > > > > >> for
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > SCRAM-SHA-256.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > On Tue, May 24, 2016 at
> 2:37
> > > AM,
> > >
> > > > > Jun
> > >
> > > > > >> > Rao <
> > >
> > > > > >> > >>> > > > > > > > jun@confluent.io
> > >
> > > > > >> > >>> > > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> wrote:
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > Parth,
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > Thanks for the
> > explanation.
> > > A
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > couple
> > >
> > > > > >> > of
> > >
> > > > > >> > >>> > more
> > >
> > > > > >> > >>> > > > > > > questions.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > 110. What does
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > getDelegationTokenAs
> > >
> > > > > >> > >>> mean?
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > 111. What's the typical
> > rate
> > >
> > > > of
> > >
> > > > > >> > getting
> > >
> > > > > >> > >>> and
> > >
> > > > > >> > >>> > > > > > renewing
> > >
> > > > > >> > >>> > > > > > > > > >> delegation
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > tokens?
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > That may have an impact
> on
> > >
> > > > > whether
> > >
> > > > > >> > they
> > >
> > > > > >> > >>> > > should
> > >
> > > > > >> > >>> > > > be
> > >
> > > > > >> > >>> > > > > > > > > directed
> > >
> > > > > >> > >>> > > > > > > > > >> to the
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > controller.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > Jun
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > On Mon, May 23, 2016 at
> > 1:19
> > >
> > > > PM,
> > >
> > > > > >> > parth
> > >
> > > > > >> > >>> > > > > brahmbhatt <
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > brahmbhatt.parth@gmail.com>
> > >
> > > > > wrote:
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Hi Jun,
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Thanks for reviewing.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * We could add a
> Cluster
> > >
> > > > > action
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > to
> > >
> > > > > >> > add
> > >
> > > > > >> > >>> > acls
> > >
> > > > > >> > >>> > > > on
> > >
> > > > > >> > >>> > > > > > who
> > >
> > > > > >> > >>> > > > > > > > can
> > >
> > > > > >> > >>> > > > > > > > > >> request
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > delegation
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > tokens. I don't see
> the
> > > use
> > >
> > > > > case
> > >
> > > > > >> > for
> > >
> > > > > >> > >>> that
> > >
> > > > > >> > >>> > > yet
> > >
> > > > > >> > >>> > > > > but
> > >
> > > > > >> > >>> > > > > > > > down
> > >
> > > > > >> > >>> > > > > > > > > >> the line
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > when
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > we
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > start supporting
> > >
> > > > > >> > getDelegationTokenAs
> > >
> > > > > >> > >>> it
> > >
> > > > > >> > >>> > > will
> > >
> > > > > >> > >>> > > > > be
> > >
> > > > > >> > >>> > > > > > > > > >> necessary.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * Yes we recommend
> > tokens
> > > to
> > >
> > > > > be
> > >
> > > > > >> > only
> > >
> > > > > >> > >>> > > > > > > used/distributed
> > >
> > > > > >> > >>> > > > > > > > > >> over
> > >
> > > > > >> > >>> > > > > > > > > >> >> > secure
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > channels.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * Depending on what
> > design
> > >
> > > > we
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > end
> > >
> > > > > >> > up
> > >
> > > > > >> > >>> > > choosing
> > >
> > > > > >> > >>> > > > > > > > > >> Invalidation will
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > be
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > responsibility of
> every
> > >
> > > > broker
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > or
> > >
> > > > > >> > >>> > > controller.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * I am not sure if I
> > >
> > > > > documented
> > >
> > > > > >> > >>> somewhere
> > >
> > > > > >> > >>> > > > that
> > >
> > > > > >> > >>> > > > > > > > > >> invalidation
> > >
> > > > > >> > >>> > > > > > > > > >> >> > will
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > directly
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > go through zookeeper
> but
> > >
> > > > that
> > >
> > > > > is
> > >
> > > > > >> > not
> > >
> > > > > >> > >>> the
> > >
> > > > > >> > >>> > > > > intent.
> > >
> > > > > >> > >>> > > > > > > > > >> Invalidation
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > will
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > either
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > be request based or
> due
> > to
> > >
> > > > > >> > >>> expiration. No
> > >
> > > > > >> > >>> > > > > direct
> > >
> > > > > >> > >>> > > > > > > > > >> zookeeper
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > interaction
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > from
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > any client.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > * "Broker also stores
> > the
> > >
> > > > > >> > >>> DelegationToken
> > >
> > > > > >> > >>> > > > > without
> > >
> > > > > >> > >>> > > > > > > the
> > >
> > > > > >> > >>> > > > > > > > > >> hmac in
> > >
> > > > > >> > >>> > > > > > > > > >> >> > the
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > zookeeper." : Sorry
> > about
> > >
> > > > the
> > >
> > > > > >> > >>> confusion.
> > >
> > > > > >> > >>> > > The
> > >
> > > > > >> > >>> > > > > sole
> > >
> > > > > >> > >>> > > > > > > > > >> purpose of
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > zookeeper
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > in
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > this design is as
> > >
> > > > distribution
> > >
> > > > > >> > channel
> > >
> > > > > >> > >>> > for
> > >
> > > > > >> > >>> > > > > tokens
> > >
> > > > > >> > >>> > > > > > > > > >> between all
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > brokers
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > and a
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > layer that ensures
> only
> > >
> > > > tokens
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > that
> > >
> > > > > >> > >>> were
> > >
> > > > > >> > >>> > > > > > generated
> > >
> > > > > >> > >>> > > > > > > by
> > >
> > > > > >> > >>> > > > > > > > > >> making a
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > request
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > to a
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > broker will be
> accepted
> > >
> > > > (more
> > >
> > > > > on
> > >
> > > > > >> > this
> > >
> > > > > >> > >>> in
> > >
> > > > > >> > >>> > > > second
> > >
> > > > > >> > >>> > > > > > > > > >> paragraph). The
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > token
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > consists of few
> elements
> > >
> > > > > (owner,
> > >
> > > > > >> > >>> renewer,
> > >
> > > > > >> > >>> > > > uuid
> > >
> > > > > >> > >>> > > > > ,
> > >
> > > > > >> > >>> > > > > > > > > >> expiration,
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > hmac)
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > ,
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > one
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > of
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > which is the finally
> > >
> > > > generated
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > hmac
> > >
> > > > > >> > >>> but
> > >
> > > > > >> > >>> > > hmac
> > >
> > > > > >> > >>> > > > it
> > >
> > > > > >> > >>> > > > > > > self
> > >
> > > > > >> > >>> > > > > > > > is
> > >
> > > > > >> > >>> > > > > > > > > >> >> > derivable
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > if
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > you
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > have all the other
> > > elements
> > >
> > > > of
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > the
> > >
> > > > > >> > >>> token
> > >
> > > > > >> > >>> > +
> > >
> > > > > >> > >>> > > > > secret
> > >
> > > > > >> > >>> > > > > > > key
> > >
> > > > > >> > >>> > > > > > > > > to
> > >
> > > > > >> > >>> > > > > > > > > >> >> > generate
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > hmac.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Given zookeeper does
> not
> > >
> > > > > provide
> > >
> > > > > >> > SSL
> > >
> > > > > >> > >>> > > support
> > >
> > > > > >> > >>> > > > we
> > >
> > > > > >> > >>> > > > > > do
> > >
> > > > > >> > >>> > > > > > > > not
> > >
> > > > > >> > >>> > > > > > > > > >> want the
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > entire
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > token to be wire
> > > transferred
> > >
> > > > > to
> > >
> > > > > >> > >>> zookeeper
> > >
> > > > > >> > >>> > > as
> > >
> > > > > >> > >>> > > > > that
> > >
> > > > > >> > >>> > > > > > > > will
> > >
> > > > > >> > >>> > > > > > > > > >> be an
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > insecure
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > wire
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > transfer. Instead we
> > only
> > >
> > > > > store
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > all
> > >
> > > > > >> > >>> the
> > >
> > > > > >> > >>> > > other
> > >
> > > > > >> > >>> > > > > > > > elements
> > >
> > > > > >> > >>> > > > > > > > > >> of a
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > delegation
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > tokens. Brokers can
> read
> > >
> > > > these
> > >
> > > > > >> > >>> elements
> > >
> > > > > >> > >>> > and
> > >
> > > > > >> > >>> > > > > > because
> > >
> > > > > >> > >>> > > > > > > > > they
> > >
> > > > > >> > >>> > > > > > > > > >> also
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > have
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > access
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > to secret key they
> will
> > be
> > >
> > > > > able
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > to
> > >
> > > > > >> > >>> > generate
> > >
> > > > > >> > >>> > > > > hmac
> > >
> > > > > >> > >>> > > > > > on
> > >
> > > > > >> > >>> > > > > > > > > >> their end.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > One of the alternative
> > >
> > > > > proposed
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > is
> > >
> > > > > >> > to
> > >
> > > > > >> > >>> > avoid
> > >
> > > > > >> > >>> > > > > > > zookeeper
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > altogether. A
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > Client
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > will call broker with
> > >
> > > > required
> > >
> > > > > >> > >>> > information
> > >
> > > > > >> > >>> > > > > > (owner,
> > >
> > > > > >> > >>> > > > > > > > > >> renwer,
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > expiration)
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > and
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > get back (signed hmac,
> > >
> > > > uuid).
> > >
> > > > > >> > Broker
> > >
> > > > > >> > >>> > won't
> > >
> > > > > >> > >>> > > > > store
> > >
> > > > > >> > >>> > > > > > > this
> > >
> > > > > >> > >>> > > > > > > > > in
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > zookeeper.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > From
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > this point a client
> can
> > >
> > > > > contact
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > any
> > >
> > > > > >> > >>> > broker
> > >
> > > > > >> > >>> > > > with
> > >
> > > > > >> > >>> > > > > > all
> > >
> > > > > >> > >>> > > > > > > > the
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > delegation
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > token
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > info (owner, rewner,
> > >
> > > > > expiration,
> > >
> > > > > >> > hmac,
> > >
> > > > > >> > >>> > > uuid)
> > >
> > > > > >> > >>> > > > > the
> > >
> > > > > >> > >>> > > > > > > > borker
> > >
> > > > > >> > >>> > > > > > > > > >> will
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > regenerate
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > the
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > hmac and as long as it
> > >
> > > > matches
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > with
> > >
> > > > > >> > >>> hmac
> > >
> > > > > >> > >>> > > > > > presented
> > >
> > > > > >> > >>> > > > > > > by
> > >
> > > > > >> > >>> > > > > > > > > >> client ,
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > broker
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > will
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > allow the request to
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > authenticate.
> > >
> > > > > >> > >>> Only
> > >
> > > > > >> > >>> > > > > problem
> > >
> > > > > >> > >>> > > > > > > with
> > >
> > > > > >> > >>> > > > > > > > > >> this
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > approach
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > is
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > if
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > the secret key is
> > >
> > > > compromised
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > any
> > >
> > > > > >> > >>> client
> > >
> > > > > >> > >>> > > can
> > >
> > > > > >> > >>> > > > > now
> > >
> > > > > >> > >>> > > > > > > > > generate
> > >
> > > > > >> > >>> > > > > > > > > >> >> > random
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > tokens
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > and
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > they will still be
> able
> > to
> > >
> > > > > >> > >>> authenticate
> > >
> > > > > >> > >>> > as
> > >
> > > > > >> > >>> > > > any
> > >
> > > > > >> > >>> > > > > > user
> > >
> > > > > >> > >>> > > > > > > > > they
> > >
> > > > > >> > >>> > > > > > > > > >> like.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > with
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > zookeeper we guarantee
> > > that
> > >
> > > > > only
> > >
> > > > > >> > >>> tokens
> > >
> > > > > >> > >>> > > > > acquired
> > >
> > > > > >> > >>> > > > > > > via
> > >
> > > > > >> > >>> > > > > > > > a
> > >
> > > > > >> > >>> > > > > > > > > >> broker
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > (using
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > some
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > auth scheme other than
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > delegation
> > >
> > > > > >> > >>> token)
> > >
> > > > > >> > >>> > > will
> > >
> > > > > >> > >>> > > > > be
> > >
> > > > > >> > >>> > > > > > > > > >> accepted. We
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > need
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > to
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > discuss which proposal
> > > makes
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > more
> > >
> > > > > >> > >>> sense
> > >
> > > > > >> > >>> > and
> > >
> > > > > >> > >>> > > > we
> > >
> > > > > >> > >>> > > > > > can
> > >
> > > > > >> > >>> > > > > > > go
> > >
> > > > > >> > >>> > > > > > > > > >> over it
> > >
> > > > > >> > >>> > > > > > > > > >> >> > in
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > tomorrow's
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > meeting.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Also, can you forward
> > the
> > >
> > > > > invite
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > to
> > >
> > > > > >> > >>> me?
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Thanks
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > Parth
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > On Mon, May 23, 2016
> at
> > >
> > > > 10:35
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > AM,
> > >
> > > > > >> > Jun
> > >
> > > > > >> > >>> > Rao <
> > >
> > > > > >> > >>> > > > > > > > > >> jun@confluent.io>
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > wrote:
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > Thanks for the KIP.
> A
> > > few
> > >
> > > > > >> > comments.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > 100. This
> potentially
> > > can
> > >
> > > > be
> > >
> > > > > >> > useful
> > >
> > > > > >> > >>> for
> > >
> > > > > >> > >>> > > > Kafka
> > >
> > > > > >> > >>> > > > > > > > Connect
> > >
> > > > > >> > >>> > > > > > > > > >> and
> > >
> > > > > >> > >>> > > > > > > > > >> >> > Kafka
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > rest
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > proxy
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > where a worker agent
> > > will
> > >
> > > > > need
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > to
> > >
> > > > > >> > >>> run a
> > >
> > > > > >> > >>> > > > task
> > >
> > > > > >> > >>> > > > > on
> > >
> > > > > >> > >>> > > > > > > > > behalf
> > >
> > > > > >> > >>> > > > > > > > > >> of a
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > client.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > We
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > will
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > likely need to
> change
> > > how
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > those
> > >
> > > > > >> > >>> > services
> > >
> > > > > >> > >>> > > > use
> > >
> > > > > >> > >>> > > > > > > Kafka
> > >
> > > > > >> > >>> > > > > > > > > >> clients
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > (producer/consumer).
> > >
> > > > Instead
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > of a
> > >
> > > > > >> > >>> > shared
> > >
> > > > > >> > >>> > > > > client
> > >
> > > > > >> > >>> > > > > > > per
> > >
> > > > > >> > >>> > > > > > > > > >> worker,
> > >
> > > > > >> > >>> > > > > > > > > >> >> > we
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > will
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > need
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > a
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > client per user task
> > > since
> > >
> > > > > the
> > >
> > > > > >> > >>> > > > authentication
> > >
> > > > > >> > >>> > > > > > > > happens
> > >
> > > > > >> > >>> > > > > > > > > >> at the
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > connection
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > level. For Kafka
> > > Connect,
> > >
> > > > > the
> > >
> > > > > >> > >>> renewer
> > >
> > > > > >> > >>> > > will
> > >
> > > > > >> > >>> > > > be
> > >
> > > > > >> > >>> > > > > > the
> > >
> > > > > >> > >>> > > > > > > > > >> workers.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > So,
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > we
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > probably
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > need to allow
> multiple
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > renewers.
> > >
> > > > > >> > For
> > >
> > > > > >> > >>> > > Kafka
> > >
> > > > > >> > >>> > > > > rest
> > >
> > > > > >> > >>> > > > > > > > > proxy,
> > >
> > > > > >> > >>> > > > > > > > > >> the
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > renewer
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > can
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > probably just be the
> > >
> > > > creator
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > of
> > >
> > > > > >> > the
> > >
> > > > > >> > >>> > > token.
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > 101. Do we need new
> > acl
> > > on
> > >
> > > > > who
> > >
> > > > > >> > can
> > >
> > > > > >> > >>> > > request
> > >
> > > > > >> > >>> > > > > > > > delegation
> > >
> > > > > >> > >>> > > > > > > > > >> tokens?
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > 102. Do we recommend
> > >
> > > > people
> > >
> > > > > to
> > >
> > > > > >> > send
> > >
> > > > > >> > >>> > > > > delegation
> > >
> > > > > >> > >>> > > > > > > > tokens
> > >
> > > > > >> > >>> > > > > > > > > >> in an
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > encrypted
> > >
> > > > > >> > >>> > > > > > > > > >> >> > > > > > > > > channel?
> > >
> > > > > >> > >>>
> >
>



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
<http://www.confluent.io/blog>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Manikumar <ma...@gmail.com>]

Ashish,

Thank you for reviewing the KIP.  Please see the replies inline.


> 1. How to disable delegation token authentication?
>
> This can be achieved in various ways, however I think reusing delegation
> token secret config for this makes sense here. Avoids creating yet another
> config and forces delegation token users to consciously set the secret. If
> the secret is not set or set to empty string, brokers should turn off
> delegation token support. This will however require a new error code to
> indicate delegation token support is turned off on broker.
>

  Thanks for the suggestion. Option to turnoff delegation token
authentication will be useful.
  I'll update the KIP.


>
> 2. ACLs on delegation token?
>
> Do we need to have ACLs defined for tokens? I do not think it buys us
> anything, as delegation token can be treated as impersonation of the owner.
> Any thing the owner has permission to do, delegation tokens should be
> allowed to do as well. If so, we probably won't need to return
> authorization exception error code while creating delegation token. It
> however would make sense to check renew and expire requests are coming from
> owner or renewers of the token, but that does not require explicit acls.
>


Yes, We agreed to not have new acl on who can request delegation token.
 I'll update the KIP.


>
> 3. How to restrict max life time of a token?
>
> Admins might want to restrict max life time of tokens created on a cluster,
> and this can very from cluster to cluster based on use-cases. This might
> warrant a separate broker config.
>
>
Currently we  have "delegation.token.max.lifetime.sec" server config
property
May be we can take min(User supplied MaxTime, Server MaxTime) as max life
time.
I am open to add new config property.

Few more comments based on recent KIP update.
>
> 1. Do we need a separate {{InvalidateTokenRequest}}? Can't we use
> {{ExpireTokenRequest}} with with expiryDate set to anything before current
> date?
>

makes sense. we don't need special request to cancel the token. We can use
ExpireTokenRequest.
I'll update the KIP.


> 2. Can we change time field names to indicate their unit is milliseconds,
> like, IssueDateMs, ExpiryDateMs, etc.?
>
>
  Done.


> 3. Can we allow users to renew a token for a specified amount of time? In
> current version of KIP, renew request does not take time as a param, not
> sure what is expiry time set to after renewal.
>
>
 Yes, we need to specify renew period.  I'll update the KIP.


Thanks,
Mankumar



>
> On Mon, Dec 12, 2016 at 9:08 AM Manikumar <ma...@gmail.com>
> wrote:
>
> > Hi,
> >
> >
> >
> > I would like to reinitiate the discussion on Delegation token support for
> >
> > Kafka.
> >
> >
> >
> > Brief summary of the past discussion:
> >
> >
> >
> > 1) Broker stores delegation tokens in zookeeper.  All brokers will have a
> >
> > cache backed by
> >
> >    zookeeper so they will all get notified whenever a new token is
> >
> > generated and they will
> >
> >    update their local cache whenever token state changes.
> >
> > 2) The current proposal does not support rotation of secret
> >
> > 3) Only allow the renewal by users that authenticated using *non*
> >
> > delegation token mechanism
> >
> > 4) KIP-84 proposes to support  SASL SCRAM mechanisms. Kafka clients can
> >
> > authenticate using
> >
> >    SCRAM-SHA-256, providing the delegation token HMAC as password.
> >
> >
> >
> > Updated the KIP with the following:
> >
> > 1. Protocol and Config changes
> >
> > 2. format of the data stored in ZK.
> >
> > 3. Changes to Java Clients/Usage of SASL SCRAM mechanism
> >
> >
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >
> > 48+Delegation+token+support+for+Kafka
> >
> >
> >
> >
> >
> > Jun, Ashish, Gwen,
> >
> >
> >
> > Pl review the updated KIP.
> >
> >
> >
> > Thanks
> >
> > Manikumar
> >
> >
> >
> >
> >
> > On Thu, Sep 29, 2016 at 9:56 PM, Ashish Singh <as...@cloudera.com>
> wrote:
> >
> >
> >
> > > Harsha/ Gwen,
> >
> > >
> >
> > > How do we proceed here? I am willing to help out with here.
> >
> > >
> >
> > > On Fri, Sep 23, 2016 at 11:41 AM, Gwen Shapira <gw...@confluent.io>
> > wrote:
> >
> > >
> >
> > > > Is it updated? are all concerns addressed? do you want to start a
> vote?
> >
> > > >
> >
> > > > Sorry for being pushy, I do appreciate that we are all volunteers and
> >
> > > > finding time is difficult. This feature is important for anything
> that
> >
> > > > integrates with Kafka (stream processors, Flume, NiFi, etc) and I
> >
> > > > don't want to see this getting stuck because we lack coordination
> >
> > > > within the community.
> >
> > > >
> >
> > > > On Thu, Sep 15, 2016 at 6:39 PM, Harsha Chintalapani <
> kafka@harsha.io>
> >
> > > > wrote:
> >
> > > > > The only pending update for the KIP is to write up the protocol
> > changes
> >
> > > > like
> >
> > > > > we've it KIP-4. I'll update the wiki.
> >
> > > > >
> >
> > > > >
> >
> > > > > On Thu, Sep 15, 2016 at 4:27 PM Ashish Singh <as...@cloudera.com>
> >
> > > > wrote:
> >
> > > > >>
> >
> > > > >> I think we decided to not support secret rotation, I guess this
> can
> > be
> >
> > > > >> stated clearly on the KIP. Also, more details on how clients will
> >
> > > > perform
> >
> > > > >> token distribution and how CLI will look like will be helpful.
> >
> > > > >>
> >
> > > > >> On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <gw...@confluent.io>
> >
> > > > wrote:
> >
> > > > >>
> >
> > > > >> > Hi Guys,
> >
> > > > >> >
> >
> > > > >> > This discussion was dead for a while. Are there still
> contentious
> >
> > > > >> > points? If not, why are there no votes?
> >
> > > > >> >
> >
> > > > >> > On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <ju...@confluent.io>
> > wrote:
> >
> > > > >> > > Ashish,
> >
> > > > >> > >
> >
> > > > >> > > Yes, I will send out a KIP invite for next week to discuss
> > KIP-48
> >
> > > > and
> >
> > > > >> > other
> >
> > > > >> > > remaining KIPs.
> >
> > > > >> > >
> >
> > > > >> > > Thanks,
> >
> > > > >> > >
> >
> > > > >> > > Jun
> >
> > > > >> > >
> >
> > > > >> > > On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh <
> >
> > > asingh@cloudera.com>
> >
> > > > >> > wrote:
> >
> > > > >> > >
> >
> > > > >> > >> Thanks Harsha!
> >
> > > > >> > >>
> >
> > > > >> > >> Jun, can we add KIP-48 to next KIP hangout's agenda. Also, we
> > did
> >
> > > > not
> >
> > > > >> > >> actually make a call on when we should have next KIP call. As
> >
> > > there
> >
> > > > >> > >> are
> >
> > > > >> > a
> >
> > > > >> > >> few outstanding KIPs that could not be discussed this week,
> can
> >
> > > we
> >
> > > > >> > >> have
> >
> > > > >> > a
> >
> > > > >> > >> KIP hangout call next week?
> >
> > > > >> > >>
> >
> > > > >> > >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha Chintalapani
> >
> > > > >> > >> <ka...@harsha.io>
> >
> > > > >> > >> wrote:
> >
> > > > >> > >>
> >
> > > > >> > >>> Ashish,
> >
> > > > >> > >>>         Yes we are working on it. Lets discuss in the next
> KIP
> >
> > > > >> > >>> meeting.
> >
> > > > >> > >>> I'll join.
> >
> > > > >> > >>> -Harsha
> >
> > > > >> > >>>
> >
> > > > >> > >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh <
> >
> > > > asingh@cloudera.com>
> >
> > > > >> > >>> wrote:
> >
> > > > >> > >>>
> >
> > > > >> > >>> > Hello Harsha,
> >
> > > > >> > >>> >
> >
> > > > >> > >>> > Are you still working on this? Wondering if we can discuss
> >
> > > this
> >
> > > > in
> >
> > > > >> > next
> >
> > > > >> > >>> KIP
> >
> > > > >> > >>> > meeting, if you can join.
> >
> > > > >> > >>> >
> >
> > > > >> > >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha Chintalapani <
> >
> > > > >> > kafka@harsha.io>
> >
> > > > >> > >>> > wrote:
> >
> > > > >> > >>> >
> >
> > > > >> > >>> > > Hi Grant,
> >
> > > > >> > >>> > >           We are working on it. Will add the details to
> > KIP
> >
> > > > >> > >>> > > about
> >
> > > > >> > the
> >
> > > > >> > >>> > > request protocol.
> >
> > > > >> > >>> > >
> >
> > > > >> > >>> > > Thanks,
> >
> > > > >> > >>> > > Harsha
> >
> > > > >> > >>> > >
> >
> > > > >> > >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant Henke
> >
> > > > >> > >>> > > <gh...@cloudera.com>
> >
> > > > >> > >>> wrote:
> >
> > > > >> > >>> > >
> >
> > > > >> > >>> > > > Hi Parth,
> >
> > > > >> > >>> > > >
> >
> > > > >> > >>> > > > Are you still working on this? If you need any help
> > please
> >
> > > > >> > >>> > > > don't
> >
> > > > >> > >>> > hesitate
> >
> > > > >> > >>> > > > to ask.
> >
> > > > >> > >>> > > >
> >
> > > > >> > >>> > > > Thanks,
> >
> > > > >> > >>> > > > Grant
> >
> > > > >> > >>> > > >
> >
> > > > >> > >>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <
> >
> > > jun@confluent.io>
> >
> > > > >> > wrote:
> >
> > > > >> > >>> > > >
> >
> > > > >> > >>> > > > > Parth,
> >
> > > > >> > >>> > > > >
> >
> > > > >> > >>> > > > > Thanks for the reply.
> >
> > > > >> > >>> > > > >
> >
> > > > >> > >>> > > > > It makes sense to only allow the renewal by users
> that
> >
> > > > >> > >>> authenticated
> >
> > > > >> > >>> > > > using
> >
> > > > >> > >>> > > > > *non* delegation token mechanism. Then, should we
> make
> >
> > > the
> >
> > > > >> > >>> renewal a
> >
> > > > >> > >>> > > > list?
> >
> > > > >> > >>> > > > > For example, in the case of rest proxy, it will be
> >
> > > useful
> >
> > > > >> > >>> > > > > for
> >
> > > > >> > >>> every
> >
> > > > >> > >>> > > > > instance of rest proxy to be able to renew the
> tokens.
> >
> > > > >> > >>> > > > >
> >
> > > > >> > >>> > > > > It would be clearer if we can document the request
> >
> > > > protocol
> >
> > > > >> > like
> >
> > > > >> > >>> > > > >
> >
> > > > >> > >>> > > > >
> >
> > > > >> > >>> > > > https://cwiki.apache.org/confl
> uence/display/KAFKA/KIP-
> >
> > > > >> > >>> > >
> >
> > > > >> > >>> > > 4+-+Command+line+and+centralized+administrative+
> >
> > > > operations#KIP-4-
> >
> > > > >> > >>> > > Commandlineandcentralizedadministrativeoperations-
> >
> > > > >> > >>> > > CreateTopicsRequest(KAFKA-2945):(VotedandPlannedforin0.
> >
> > > > 10.1.0)
> >
> > > > >> > >>> > > > > .
> >
> > > > >> > >>> > > > >
> >
> > > > >> > >>> > > > > It would also be useful to document the client APIs.
> >
> > > > >> > >>> > > > >
> >
> > > > >> > >>> > > > > Thanks,
> >
> > > > >> > >>> > > > >
> >
> > > > >> > >>> > > > > Jun
> >
> > > > >> > >>> > > > >
> >
> > > > >> > >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM, parth brahmbhatt <
> >
> > > > >> > >>> > > > > brahmbhatt.parth@gmail.com> wrote:
> >
> > > > >> > >>> > > > >
> >
> > > > >> > >>> > > > > > Hi,
> >
> > > > >> > >>> > > > > >
> >
> > > > >> > >>> > > > > > I am suggesting that we will only allow the
> renewal
> > by
> >
> > > > >> > >>> > > > > > users
> >
> > > > >> > >>> that
> >
> > > > >> > >>> > > > > > authenticated using *non* delegation token
> > mechanism.
> >
> > > > For
> >
> > > > >> > >>> example,
> >
> > > > >> > >>> > If
> >
> > > > >> > >>> > > > > user
> >
> > > > >> > >>> > > > > > Alice authenticated using kerberos and requested
> >
> > > > >> > >>> > > > > > delegation
> >
> > > > >> > >>> tokens,
> >
> > > > >> > >>> > > > only
> >
> > > > >> > >>> > > > > > user Alice authenticated via non delegation token
> >
> > > > >> > >>> > > > > > mechanism
> >
> > > > >> > can
> >
> > > > >> > >>> > > renew.
> >
> > > > >> > >>> > > > > > Clients that have  access to delegation tokens can
> > not
> >
> > > > >> > >>> > > > > > issue
> >
> > > > >> > >>> > renewal
> >
> > > > >> > >>> > > > > > request for renewing their own token and this is
> >
> > > > primarily
> >
> > > > >> > >>> > important
> >
> > > > >> > >>> > > to
> >
> > > > >> > >>> > > > > > reduce the time window for which a compromised
> token
> >
> > > > will
> >
> > > > >> > >>> > > > > > be
> >
> > > > >> > >>> valid.
> >
> > > > >> > >>> > > > > >
> >
> > > > >> > >>> > > > > > To clarify, Yes any authenticated user can request
> >
> > > > >> > >>> > > > > > delegation
> >
> > > > >> > >>> > tokens
> >
> > > > >> > >>> > > > but
> >
> > > > >> > >>> > > > > > even here I would recommend to avoid creating a
> > chain
> >
> > > > >> > >>> > > > > > where a
> >
> > > > >> > >>> > client
> >
> > > > >> > >>> > > > > > authenticated via delegation token request for
> more
> >
> > > > >> > delegation
> >
> > > > >> > >>> > > tokens.
> >
> > > > >> > >>> > > > > > Basically anyone can request delegation token, as
> > long
> >
> > > > as
> >
> > > > >> > they
> >
> > > > >> > >>> > > > > authenticate
> >
> > > > >> > >>> > > > > > via a non delegation token mechanism.
> >
> > > > >> > >>> > > > > >
> >
> > > > >> > >>> > > > > > Aren't classes listed here
> >
> > > > >> > >>> > > > > > <
> >
> > > > >> > >>> > > > > >
> >
> > > > >> > >>> > > > >
> >
> > > > >> > >>> > > > https://cwiki.apache.org/confl
> uence/display/KAFKA/KIP-
> >
> > > > >> > >>> > > 48+Delegation+token+support+fo
> >
> > > r+Kafka#KIP-48Delegationtokens
> >
> > > > >> > >>> upportforKaf
> >
> > > > >> > >>> > > ka-PublicInterfaces
> >
> > > > >> > >>> > > > > > >
> >
> > > > >> > >>> > > > > > sufficient?
> >
> > > > >> > >>> > > > > >
> >
> > > > >> > >>> > > > > > Thanks
> >
> > > > >> > >>> > > > > > Parth
> >
> > > > >> > >>> > > > > >
> >
> > > > >> > >>> > > > > >
> >
> > > > >> > >>> > > > > >
> >
> > > > >> > >>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao
> >
> > > > >> > >>> > > > > > <ju...@confluent.io>
> >
> > > > >> > >>> wrote:
> >
> > > > >> > >>> > > > > >
> >
> > > > >> > >>> > > > > > > Parth,
> >
> > > > >> > >>> > > > > > >
> >
> > > > >> > >>> > > > > > > Thanks for the reply. A couple of comments
> inline
> >
> > > > below.
> >
> > > > >> > >>> > > > > > >
> >
> > > > >> > >>> > > > > > > On Tue, Jun 21, 2016 at 10:36 AM, parth
> > brahmbhatt <
> >
> > > > >> > >>> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> >
> > > > >> > >>> > > > > > >
> >
> > > > >> > >>> > > > > > > > 1. Who / how are tokens renewed? By original
> >
> > > > requester
> >
> > > > >> > >>> only? or
> >
> > > > >> > >>> > > > using
> >
> > > > >> > >>> > > > > > > > Kerberos
> >
> > > > >> > >>> > > > > > > > auth only?
> >
> > > > >> > >>> > > > > > > > My recommendation is to do this only using
> >
> > > Kerberos
> >
> > > > >> > >>> > > > > > > > auth
> >
> > > > >> > and
> >
> > > > >> > >>> > only
> >
> > > > >> > >>> > > > > threw
> >
> > > > >> > >>> > > > > > > the
> >
> > > > >> > >>> > > > > > > > renewer specified during the acquisition
> > request.
> >
> > > > >> > >>> > > > > > > >
> >
> > > > >> > >>> > > > > > > >
> >
> > > > >> > >>> > > > > > > Hmm, not sure that I follow this. Are you saying
> >
> > > that
> >
> > > > >> > >>> > > > > > > any
> >
> > > > >> > >>> client
> >
> > > > >> > >>> > > > > > > authenticated with the delegation token can
> renew,
> >
> > > > i.e.
> >
> > > > >> > there
> >
> > > > >> > >>> is
> >
> > > > >> > >>> > no
> >
> > > > >> > >>> > > > > > renewer
> >
> > > > >> > >>> > > > > > > needed?
> >
> > > > >> > >>> > > > > > >
> >
> > > > >> > >>> > > > > > > Also, just to be clear, any authenticated client
> >
> > > > (either
> >
> > > > >> > >>> through
> >
> > > > >> > >>> > > SASL
> >
> > > > >> > >>> > > > > or
> >
> > > > >> > >>> > > > > > > SSL) can request a delegation token for the
> >
> > > > >> > >>> > > > > > > authenticated
> >
> > > > >> > >>> user,
> >
> > > > >> > >>> > > > right?
> >
> > > > >> > >>> > > > > > >
> >
> > > > >> > >>> > > > > > >
> >
> > > > >> > >>> > > > > > > > 2. Are tokens stored on each broker or in ZK?
> >
> > > > >> > >>> > > > > > > > My recommendation is still to store in ZK or
> not
> >
> > > > store
> >
> > > > >> > them
> >
> > > > >> > >>> at
> >
> > > > >> > >>> > > all.
> >
> > > > >> > >>> > > > > The
> >
> > > > >> > >>> > > > > > > > whole controller based distribution is too
> much
> >
> > > > >> > >>> > > > > > > > overhead
> >
> > > > >> > >>> with
> >
> > > > >> > >>> > not
> >
> > > > >> > >>> > > > > much
> >
> > > > >> > >>> > > > > > to
> >
> > > > >> > >>> > > > > > > > achieve.
> >
> > > > >> > >>> > > > > > > >
> >
> > > > >> > >>> > > > > > > > 3. How are tokens invalidated / expired?
> >
> > > > >> > >>> > > > > > > > Either by expiration time out or through an
> >
> > > explicit
> >
> > > > >> > >>> request to
> >
> > > > >> > >>> > > > > > > invalidate.
> >
> > > > >> > >>> > > > > > > >
> >
> > > > >> > >>> > > > > > > > 4. Which encryption algorithm is used?
> >
> > > > >> > >>> > > > > > > > SCRAM
> >
> > > > >> > >>> > > > > > > >
> >
> > > > >> > >>> > > > > > > > 5. What is the impersonation proposal (it
> wasn't
> >
> > > in
> >
> > > > >> > >>> > > > > > > > the
> >
> > > > >> > KIP
> >
> > > > >> > >>> but
> >
> > > > >> > >>> > > was
> >
> > > > >> > >>> > > > > > > > discussed
> >
> > > > >> > >>> > > > > > > > in this thread)?
> >
> > > > >> > >>> > > > > > > > There is no imperonation proposal. I tried and
> >
> > > > >> > >>> > > > > > > > explained
> >
> > > > >> > how
> >
> > > > >> > >>> > its
> >
> > > > >> > >>> > > a
> >
> > > > >> > >>> > > > > > > > different problem and why its not really
> > necessary
> >
> > > > to
> >
> > > > >> > >>> discuss
> >
> > > > >> > >>> > > that
> >
> > > > >> > >>> > > > as
> >
> > > > >> > >>> > > > > > > part
> >
> > > > >> > >>> > > > > > > > of this KIP.  This KIP will not support any
> >
> > > > >> > impersonation,
> >
> > > > >> > >>> it
> >
> > > > >> > >>> > > will
> >
> > > > >> > >>> > > > > just
> >
> > > > >> > >>> > > > > > > be
> >
> > > > >> > >>> > > > > > > > another way to authenticate.
> >
> > > > >> > >>> > > > > > > >
> >
> > > > >> > >>> > > > > > > > 6. Do we need new ACLs, if so - for what
> > actions?
> >
> > > > >> > >>> > > > > > > > We do not need new ACLs.
> >
> > > > >> > >>> > > > > > > >
> >
> > > > >> > >>> > > > > > > >
> >
> > > > >> > >>> > > > > > > Could we document the format of the new
> >
> > > > request/response
> >
> > > > >> > and
> >
> > > > >> > >>> > their
> >
> > > > >> > >>> > > > > > > associated Resource and Operation for ACL?
> >
> > > > >> > >>> > > > > > >
> >
> > > > >> > >>> > > > > > >
> >
> > > > >> > >>> > > > > > > > 7. How would the delegation token be
> configured
> > in
> >
> > > > the
> >
> > > > >> > >>> client?
> >
> > > > >> > >>> > > > > > > > Should be through config. I wasn't planning on
> >
> > > > >> > >>> > > > > > > > supporting
> >
> > > > >> > >>> JAAS
> >
> > > > >> > >>> > > for
> >
> > > > >> > >>> > > > > > > tokens.
> >
> > > > >> > >>> > > > > > > > I don't believe hadoop does this either.
> >
> > > > >> > >>> > > > > > > >
> >
> > > > >> > >>> > > > > > > > Thanks
> >
> > > > >> > >>> > > > > > > > Parth
> >
> > > > >> > >>> > > > > > > >
> >
> > > > >> > >>> > > > > > > >
> >
> > > > >> > >>> > > > > > > >
> >
> > > > >> > >>> > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <
> >
> > > > >> > jun@confluent.io>
> >
> > > > >> > >>> > > wrote:
> >
> > > > >> > >>> > > > > > > >
> >
> > > > >> > >>> > > > > > > > > Harsha,
> >
> > > > >> > >>> > > > > > > > >
> >
> > > > >> > >>> > > > > > > > > Another question.
> >
> > > > >> > >>> > > > > > > > >
> >
> > > > >> > >>> > > > > > > > > 9. How would the delegation token be
> > configured
> >
> > > in
> >
> > > > >> > >>> > > > > > > > > the
> >
> > > > >> > >>> > client?
> >
> > > > >> > >>> > > > The
> >
> > > > >> > >>> > > > > > > > standard
> >
> > > > >> > >>> > > > > > > > > way is to do this through JAAS. However, we
> > will
> >
> > > > >> > >>> > > > > > > > > need
> >
> > > > >> > to
> >
> > > > >> > >>> > think
> >
> > > > >> > >>> > > > > > through
> >
> > > > >> > >>> > > > > > > if
> >
> > > > >> > >>> > > > > > > > > this is convenient in a shared environment.
> > For
> >
> > > > >> > example,
> >
> > > > >> > >>> > when a
> >
> > > > >> > >>> > > > new
> >
> > > > >> > >>> > > > > > > task
> >
> > > > >> > >>> > > > > > > > is
> >
> > > > >> > >>> > > > > > > > > added to a Storm worker node, do we need to
> >
> > > > >> > >>> > > > > > > > > dynamically
> >
> > > > >> > >>> add a
> >
> > > > >> > >>> > > new
> >
> > > > >> > >>> > > > > > > section
> >
> > > > >> > >>> > > > > > > > > in the JAAS file? It may be more convenient
> if
> >
> > > we
> >
> > > > >> > >>> > > > > > > > > can
> >
> > > > >> > >>> pass in
> >
> > > > >> > >>> > > the
> >
> > > > >> > >>> > > > > > token
> >
> > > > >> > >>> > > > > > > > > through the config directly w/o going
> through
> >
> > > > JAAS.
> >
> > > > >> > >>> > > > > > > > >
> >
> > > > >> > >>> > > > > > > > > Are you or Parth still actively working on
> > this
> >
> > > > KIP?
> >
> > > > >> > >>> > > > > > > > >
> >
> > > > >> > >>> > > > > > > > > Thanks,
> >
> > > > >> > >>> > > > > > > > >
> >
> > > > >> > >>> > > > > > > > > Jun
> >
> > > > >> > >>> > > > > > > > >
> >
> > > > >> > >>> > > > > > > > >
> >
> > > > >> > >>> > > > > > > > >
> >
> > > > >> > >>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <
> >
> > > > >> > >>> jun@confluent.io>
> >
> > > > >> > >>> > > > wrote:
> >
> > > > >> > >>> > > > > > > > >
> >
> > > > >> > >>> > > > > > > > > > Just to add on that list.
> >
> > > > >> > >>> > > > > > > > > >
> >
> > > > >> > >>> > > > > > > > > > 2. It would be good to document the format
> > of
> >
> > > > the
> >
> > > > >> > data
> >
> > > > >> > >>> > stored
> >
> > > > >> > >>> > > > in
> >
> > > > >> > >>> > > > > > ZK.
> >
> > > > >> > >>> > > > > > > > > > 7. Earlier, there was a discussion on
> > whether
> >
> > > > the
> >
> > > > >> > tokens
> >
> > > > >> > >>> > > should
> >
> > > > >> > >>> > > > > be
> >
> > > > >> > >>> > > > > > > > > > propagated through ZK like
> config/acl/quota,
> >
> > > or
> >
> > > > >> > through
> >
> > > > >> > >>> the
> >
> > > > >> > >>> > > > > > > controller.
> >
> > > > >> > >>> > > > > > > > > > Currently, the controller is only designed
> > for
> >
> > > > >> > >>> propagating
> >
> > > > >> > >>> > > > topic
> >
> > > > >> > >>> > > > > > > > > metadata,
> >
> > > > >> > >>> > > > > > > > > > but not other data.
> >
> > > > >> > >>> > > > > > > > > > 8. Should we use SCRAM to send the token
> >
> > > instead
> >
> > > > >> > >>> > > > > > > > > > of
> >
> > > > >> > >>> > > DIGEST-MD5
> >
> > > > >> > >>> > > > > > since
> >
> > > > >> > >>> > > > > > > > it's
> >
> > > > >> > >>> > > > > > > > > > deprecated?
> >
> > > > >> > >>> > > > > > > > > >
> >
> > > > >> > >>> > > > > > > > > > Also, the images in the wiki seem broken.
> >
> > > > >> > >>> > > > > > > > > >
> >
> > > > >> > >>> > > > > > > > > > Thanks,
> >
> > > > >> > >>> > > > > > > > > >
> >
> > > > >> > >>> > > > > > > > > > Jun
> >
> > > > >> > >>> > > > > > > > > >
> >
> > > > >> > >>> > > > > > > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen
> >
> > > Shapira <
> >
> > > > >> > >>> > > > > gwen@confluent.io>
> >
> > > > >> > >>> > > > > > > > > wrote:
> >
> > > > >> > >>> > > > > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> From what I can see, remaining questions
> > are:
> >
> > > > >> > >>> > > > > > > > > >>
> >
> > > > >> > >>> > > > > > > > > >> 1. Who / how are tokens renewed? By
> > original
> >
> > > > >> > requester
> >
> > > > >> > >>> > only?
> >
> > > > >> > >>> > > > or
> >
> > > > >> > >>> > > > > > > using
> >
> > > > >> > >>> > > > > > > > > >> Kerberos auth only?
> >
> > > > >> > >>> > > > > > > > > >> 2. Are tokens stored on each broker or in
> > ZK?
> >
> > > > >> > >>> > > > > > > > > >> 3. How are tokens invalidated / expired?
> >
> > > > >> > >>> > > > > > > > > >> 4. Which encryption algorithm is used?
> >
> > > > >> > >>> > > > > > > > > >> 5. What is the impersonation proposal (it
> >
> > > > wasn't
> >
> > > > >> > >>> > > > > > > > > >> in
> >
> > > > >> > the
> >
> > > > >> > >>> > KIP
> >
> > > > >> > >>> > > > but
> >
> > > > >> > >>> > > > > > was
> >
> > > > >> > >>> > > > > > > > > >> discussed in this thread)?
> >
> > > > >> > >>> > > > > > > > > >> 6. Do we need new ACLs, if so - for what
> >
> > > > actions?
> >
> > > > >> > >>> > > > > > > > > >>
> >
> > > > >> > >>> > > > > > > > > >> Gwen
> >
> > > > >> > >>> > > > > > > > > >>
> >
> > > > >> > >>> > > > > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <
> >
> > > > >> > >>> kafka@harsha.io>
> >
> > > > >> > >>> > > > wrote:
> >
> > > > >> > >>> > > > > > > > > >> > Jun & Ismael,
> >
> > > > >> > >>> > > > > > > > > >> >                          Unfortunately
> I
> >
> > > > >> > >>> > > > > > > > > >> > couldn't
> >
> > > > >> > >>> attend
> >
> > > > >> > >>> > > the
> >
> > > > >> > >>> > > > > KIP
> >
> > > > >> > >>> > > > > > > > > meeting
> >
> > > > >> > >>> > > > > > > > > >> >                          when
> delegation
> >
> > > > tokens
> >
> > > > >> > >>> > discussed.
> >
> > > > >> > >>> > > > > > > > Appreciate
> >
> > > > >> > >>> > > > > > > > > if
> >
> > > > >> > >>> > > > > > > > > >> >                          you can update
> > the
> >
> > > > >> > thread if
> >
> > > > >> > >>> > you
> >
> > > > >> > >>> > > > have
> >
> > > > >> > >>> > > > > > any
> >
> > > > >> > >>> > > > > > > > > >> >                          further
> > questions.
> >
> > > > >> > >>> > > > > > > > > >> > Thanks,
> >
> > > > >> > >>> > > > > > > > > >> > Harsha
> >
> > > > >> > >>> > > > > > > > > >> >
> >
> > > > >> > >>> > > > > > > > > >> > On Tue, May 24, 2016, at 11:32 AM,
> Liquan
> >
> > > Pei
> >
> > > > >> > wrote:
> >
> > > > >> > >>> > > > > > > > > >> >> It seems that the links to images in
> the
> >
> > > KIP
> >
> > > > >> > >>> > > > > > > > > >> >> are
> >
> > > > >> > >>> > broken.
> >
> > > > >> > >>> > > > > > > > > >> >>
> >
> > > > >> > >>> > > > > > > > > >> >> Liquan
> >
> > > > >> > >>> > > > > > > > > >> >>
> >
> > > > >> > >>> > > > > > > > > >> >> On Tue, May 24, 2016 at 9:33 AM, parth
> >
> > > > >> > brahmbhatt <
> >
> > > > >> > >>> > > > > > > > > >> >> brahmbhatt.parth@gmail.com> wrote:
> >
> > > > >> > >>> > > > > > > > > >> >>
> >
> > > > >> > >>> > > > > > > > > >> >> > 110. What does getDelegationTokenAs
> >
> > > mean?
> >
> > > > >> > >>> > > > > > > > > >> >> > In the current proposal we only
> allow
> > a
> >
> > > > user
> >
> > > > >> > >>> > > > > > > > > >> >> > to
> >
> > > > >> > >>> get
> >
> > > > >> > >>> > > > > > delegation
> >
> > > > >> > >>> > > > > > > > > token
> >
> > > > >> > >>> > > > > > > > > >> for
> >
> > > > >> > >>> > > > > > > > > >> >> > the identity that it authenticated
> as
> >
> > > > using
> >
> > > > >> > >>> another
> >
> > > > >> > >>> > > > > > mechanism,
> >
> > > > >> > >>> > > > > > > > i.e.
> >
> > > > >> > >>> > > > > > > > > >> A user
> >
> > > > >> > >>> > > > > > > > > >> >> > that authenticate using a keytab for
> >
> > > > >> > >>> > > > > > > > > >> >> > principal
> >
> > > > >> > >>> > > > > > > user1@EXAMPLE.COM
> >
> > > > >> > >>> > > > > > > > > >> will get
> >
> > > > >> > >>> > > > > > > > > >> >> > delegation tokens for that user
> only.
> > In
> >
> > > > >> > future I
> >
> > > > >> > >>> > think
> >
> > > > >> > >>> > > > we
> >
> > > > >> > >>> > > > > > will
> >
> > > > >> > >>> > > > > > > > > have
> >
> > > > >> > >>> > > > > > > > > >> to
> >
> > > > >> > >>> > > > > > > > > >> >> > extend support such that we allow
> some
> >
> > > set
> >
> > > > >> > >>> > > > > > > > > >> >> > of
> >
> > > > >> > >>> users (
> >
> > > > >> > >>> > > > > > > > > >> >> > kafka-rest-user@EXAMPLE.COM,
> >
> > > > >> > >>> > storm-nimbus@EXAMPLE.COM)
> >
> > > > >> > >>> > > > to
> >
> > > > >> > >>> > > > > > > > acquire
> >
> > > > >> > >>> > > > > > > > > >> >> > delegation tokens on behalf of other
> >
> > > users
> >
> > > > >> > whose
> >
> > > > >> > >>> > > identity
> >
> > > > >> > >>> > > > > > they
> >
> > > > >> > >>> > > > > > > > have
> >
> > > > >> > >>> > > > > > > > > >> >> > verified independently.  Kafka
> brokers
> >
> > > > will
> >
> > > > >> > have
> >
> > > > >> > >>> ACLs
> >
> > > > >> > >>> > > to
> >
> > > > >> > >>> > > > > > > control
> >
> > > > >> > >>> > > > > > > > > >> which
> >
> > > > >> > >>> > > > > > > > > >> >> > users are allowed to impersonate
> other
> >
> > > > users
> >
> > > > >> > and
> >
> > > > >> > >>> get
> >
> > > > >> > >>> > > > tokens
> >
> > > > >> > >>> > > > > > on
> >
> > > > >> > >>> > > > > > > > > >> behalf of
> >
> > > > >> > >>> > > > > > > > > >> >> > them. Overall Impersonation is a
> whole
> >
> > > > >> > different
> >
> > > > >> > >>> > > problem
> >
> > > > >> > >>> > > > in
> >
> > > > >> > >>> > > > > > my
> >
> > > > >> > >>> > > > > > > > > >> opinion and
> >
> > > > >> > >>> > > > > > > > > >> >> > I think we can tackle it in separate
> >
> > > KIP.
> >
> > > > >> > >>> > > > > > > > > >> >> >
> >
> > > > >> > >>> > > > > > > > > >> >> > 111. What's the typical rate of
> > getting
> >
> > > > and
> >
> > > > >> > >>> renewing
> >
> > > > >> > >>> > > > > > delegation
> >
> > > > >> > >>> > > > > > > > > >> tokens?
> >
> > > > >> > >>> > > > > > > > > >> >> > Typically this should be very very
> > low,
> >
> > > 1
> >
> > > > >> > request
> >
> > > > >> > >>> per
> >
> > > > >> > >>> > > > > minute
> >
> > > > >> > >>> > > > > > > is a
> >
> > > > >> > >>> > > > > > > > > >> >> > relatively high estimate. However it
> >
> > > > depends
> >
> > > > >> > >>> > > > > > > > > >> >> > on
> >
> > > > >> > >>> the
> >
> > > > >> > >>> > > token
> >
> > > > >> > >>> > > > > > > > > >> expiration. I am
> >
> > > > >> > >>> > > > > > > > > >> >> > less worried about the extra load it
> >
> > > puts
> >
> > > > on
> >
> > > > >> > >>> > controller
> >
> > > > >> > >>> > > > vs
> >
> > > > >> > >>> > > > > > the
> >
> > > > >> > >>> > > > > > > > > added
> >
> > > > >> > >>> > > > > > > > > >> >> > complexity and the value it offers.
> >
> > > > >> > >>> > > > > > > > > >> >> >
> >
> > > > >> > >>> > > > > > > > > >> >> > Thanks
> >
> > > > >> > >>> > > > > > > > > >> >> > Parth
> >
> > > > >> > >>> > > > > > > > > >> >> >
> >
> > > > >> > >>> > > > > > > > > >> >> >
> >
> > > > >> > >>> > > > > > > > > >> >> >
> >
> > > > >> > >>> > > > > > > > > >> >> > On Tue, May 24, 2016 at 7:30 AM,
> > Ismael
> >
> > > > Juma
> >
> > > > >> > >>> > > > > > > > > >> >> > <
> >
> > > > >> > >>> > > > > > > ismael@juma.me.uk>
> >
> > > > >> > >>> > > > > > > > > >> wrote:
> >
> > > > >> > >>> > > > > > > > > >> >> >
> >
> > > > >> > >>> > > > > > > > > >> >> > > Thanks Rajini. It would probably
> >
> > > > require a
> >
> > > > >> > >>> separate
> >
> > > > >> > >>> > > KIP
> >
> > > > >> > >>> > > > > as
> >
> > > > >> > >>> > > > > > it
> >
> > > > >> > >>> > > > > > > > > will
> >
> > > > >> > >>> > > > > > > > > >> >> > > introduce user visible changes. We
> >
> > > could
> >
> > > > >> > >>> > > > > > > > > >> >> > > also
> >
> > > > >> > >>> > update
> >
> > > > >> > >>> > > > > KIP-48
> >
> > > > >> > >>> > > > > > > to
> >
> > > > >> > >>> > > > > > > > > >> have this
> >
> > > > >> > >>> > > > > > > > > >> >> > > information, but it seems cleaner
> to
> >
> > > do
> >
> > > > it
> >
> > > > >> > >>> > > separately.
> >
> > > > >> > >>> > > > We
> >
> > > > >> > >>> > > > > > can
> >
> > > > >> > >>> > > > > > > > > >> discuss
> >
> > > > >> > >>> > > > > > > > > >> >> > that
> >
> > > > >> > >>> > > > > > > > > >> >> > > in the KIP call today.
> >
> > > > >> > >>> > > > > > > > > >> >> > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > Ismael
> >
> > > > >> > >>> > > > > > > > > >> >> > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > On Tue, May 24, 2016 at 3:19 PM,
> >
> > > Rajini
> >
> > > > >> > Sivaram
> >
> > > > >> > >>> <
> >
> > > > >> > >>> > > > > > > > > >> >> > > rajinisivaram@googlemail.com>
> > wrote:
> >
> > > > >> > >>> > > > > > > > > >> >> > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > Ismael,
> >
> > > > >> > >>> > > > > > > > > >> >> > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > I have created a JIRA (
> >
> > > > >> > >>> > > > > > > > > >> >> > https://issues.apache.org/
> >
> > > > >> > jira/browse/KAFKA-3751)
> >
> > > > >> > >>> > > > > > > > > >> >> > > > for adding SCRAM as a SASL
> >
> > > mechanism.
> >
> > > > >> > >>> > > > > > > > > >> >> > > > Would
> >
> > > > >> > >>> that
> >
> > > > >> > >>> > > need
> >
> > > > >> > >>> > > > > > > another
> >
> > > > >> > >>> > > > > > > > > >> KIP? If
> >
> > > > >> > >>> > > > > > > > > >> >> > > > KIP-48 will use this mechanism,
> > can
> >
> > > > this
> >
> > > > >> > just
> >
> > > > >> > >>> be
> >
> > > > >> > >>> > a
> >
> > > > >> > >>> > > > JIRA
> >
> > > > >> > >>> > > > > > > that
> >
> > > > >> > >>> > > > > > > > > gets
> >
> > > > >> > >>> > > > > > > > > >> >> > > reviewed
> >
> > > > >> > >>> > > > > > > > > >> >> > > > when the PR is ready?
> >
> > > > >> > >>> > > > > > > > > >> >> > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > Thank you,
> >
> > > > >> > >>> > > > > > > > > >> >> > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > Rajini
> >
> > > > >> > >>> > > > > > > > > >> >> > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM,
> >
> > > > Ismael
> >
> > > > >> > Juma <
> >
> > > > >> > >>> > > > > > > > > ismael@juma.me.uk>
> >
> > > > >> > >>> > > > > > > > > >> >> > wrote:
> >
> > > > >> > >>> > > > > > > > > >> >> > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > Thanks Rajini, SCRAM seems
> like
> > a
> >
> > > > good
> >
> > > > >> > >>> > candidate.
> >
> > > > >> > >>> > > > > > > > > >> >> > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > Gwen had independently
> mentioned
> >
> > > > this
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > as
> >
> > > > >> > a
> >
> > > > >> > >>> SASL
> >
> > > > >> > >>> > > > > > mechanism
> >
> > > > >> > >>> > > > > > > > > that
> >
> > > > >> > >>> > > > > > > > > >> might
> >
> > > > >> > >>> > > > > > > > > >> >> > be
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > useful for Kafka and I have
> been
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > meaning
> >
> > > > >> > to
> >
> > > > >> > >>> > check
> >
> > > > >> > >>> > > > it
> >
> > > > >> > >>> > > > > in
> >
> > > > >> > >>> > > > > > > > more
> >
> > > > >> > >>> > > > > > > > > >> detail.
> >
> > > > >> > >>> > > > > > > > > >> >> > > Good
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > to know that you are willing
> to
> >
> > > > >> > contribute
> >
> > > > >> > >>> an
> >
> > > > >> > >>> > > > > > > > implementation.
> >
> > > > >> > >>> > > > > > > > > >> Maybe
> >
> > > > >> > >>> > > > > > > > > >> >> > we
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > should file a separate JIRA
> for
> >
> > > > this?
> >
> > > > >> > >>> > > > > > > > > >> >> > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > Ismael
> >
> > > > >> > >>> > > > > > > > > >> >> > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > On Tue, May 24, 2016 at 2:12
> PM,
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > Rajini
> >
> > > > >> > >>> > Sivaram <
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > rajinisivaram@googlemail.com>
> >
> > > > wrote:
> >
> > > > >> > >>> > > > > > > > > >> >> > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > SCRAM (Salted Challenge
> > Response
> >
> > > > >> > >>> > Authentication
> >
> > > > >> > >>> > > > > > > > Mechanism)
> >
> > > > >> > >>> > > > > > > > > >> is a
> >
> > > > >> > >>> > > > > > > > > >> >> > > better
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > mechanism than Digest-MD5.
> > Java
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > doesn't
> >
> > > > >> > >>> come
> >
> > > > >> > >>> > > > with a
> >
> > > > >> > >>> > > > > > > > > built-in
> >
> > > > >> > >>> > > > > > > > > >> SCRAM
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > SaslServer or SaslClient,
> but
> > I
> >
> > > > will
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > be
> >
> > > > >> > >>> happy
> >
> > > > >> > >>> > > to
> >
> > > > >> > >>> > > > > add
> >
> > > > >> > >>> > > > > > > > > support
> >
> > > > >> > >>> > > > > > > > > >> in
> >
> > > > >> > >>> > > > > > > > > >> >> > Kafka
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > since
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > it would be a useful
> mechanism
> >
> > > to
> >
> > > > >> > support
> >
> > > > >> > >>> > > anyway.
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > >
> https://tools.ietf.org/html/
> >
> > > > rfc7677
> >
> > > > >> > >>> > describes
> >
> > > > >> > >>> > > > the
> >
> > > > >> > >>> > > > > > > > protocol
> >
> > > > >> > >>> > > > > > > > > >> for
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > SCRAM-SHA-256.
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > On Tue, May 24, 2016 at 2:37
> > AM,
> >
> > > > Jun
> >
> > > > >> > Rao <
> >
> > > > >> > >>> > > > > > > > jun@confluent.io
> >
> > > > >> > >>> > > > > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> wrote:
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > Parth,
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > Thanks for the
> explanation.
> > A
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > couple
> >
> > > > >> > of
> >
> > > > >> > >>> > more
> >
> > > > >> > >>> > > > > > > questions.
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > 110. What does
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > getDelegationTokenAs
> >
> > > > >> > >>> mean?
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > 111. What's the typical
> rate
> >
> > > of
> >
> > > > >> > getting
> >
> > > > >> > >>> and
> >
> > > > >> > >>> > > > > > renewing
> >
> > > > >> > >>> > > > > > > > > >> delegation
> >
> > > > >> > >>> > > > > > > > > >> >> > > > tokens?
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > That may have an impact on
> >
> > > > whether
> >
> > > > >> > they
> >
> > > > >> > >>> > > should
> >
> > > > >> > >>> > > > be
> >
> > > > >> > >>> > > > > > > > > directed
> >
> > > > >> > >>> > > > > > > > > >> to the
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > controller.
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > Jun
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > On Mon, May 23, 2016 at
> 1:19
> >
> > > PM,
> >
> > > > >> > parth
> >
> > > > >> > >>> > > > > brahmbhatt <
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > >
> brahmbhatt.parth@gmail.com>
> >
> > > > wrote:
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > Hi Jun,
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > Thanks for reviewing.
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > * We could add a Cluster
> >
> > > > action
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > to
> >
> > > > >> > add
> >
> > > > >> > >>> > acls
> >
> > > > >> > >>> > > > on
> >
> > > > >> > >>> > > > > > who
> >
> > > > >> > >>> > > > > > > > can
> >
> > > > >> > >>> > > > > > > > > >> request
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > delegation
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > tokens. I don't see the
> > use
> >
> > > > case
> >
> > > > >> > for
> >
> > > > >> > >>> that
> >
> > > > >> > >>> > > yet
> >
> > > > >> > >>> > > > > but
> >
> > > > >> > >>> > > > > > > > down
> >
> > > > >> > >>> > > > > > > > > >> the line
> >
> > > > >> > >>> > > > > > > > > >> >> > > > when
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > we
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > start supporting
> >
> > > > >> > getDelegationTokenAs
> >
> > > > >> > >>> it
> >
> > > > >> > >>> > > will
> >
> > > > >> > >>> > > > > be
> >
> > > > >> > >>> > > > > > > > > >> necessary.
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > * Yes we recommend
> tokens
> > to
> >
> > > > be
> >
> > > > >> > only
> >
> > > > >> > >>> > > > > > > used/distributed
> >
> > > > >> > >>> > > > > > > > > >> over
> >
> > > > >> > >>> > > > > > > > > >> >> > secure
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > channels.
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > * Depending on what
> design
> >
> > > we
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > end
> >
> > > > >> > up
> >
> > > > >> > >>> > > choosing
> >
> > > > >> > >>> > > > > > > > > >> Invalidation will
> >
> > > > >> > >>> > > > > > > > > >> >> > > be
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > responsibility of every
> >
> > > broker
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > or
> >
> > > > >> > >>> > > controller.
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > * I am not sure if I
> >
> > > > documented
> >
> > > > >> > >>> somewhere
> >
> > > > >> > >>> > > > that
> >
> > > > >> > >>> > > > > > > > > >> invalidation
> >
> > > > >> > >>> > > > > > > > > >> >> > will
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > directly
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > go through zookeeper but
> >
> > > that
> >
> > > > is
> >
> > > > >> > not
> >
> > > > >> > >>> the
> >
> > > > >> > >>> > > > > intent.
> >
> > > > >> > >>> > > > > > > > > >> Invalidation
> >
> > > > >> > >>> > > > > > > > > >> >> > > will
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > either
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > be request based or due
> to
> >
> > > > >> > >>> expiration. No
> >
> > > > >> > >>> > > > > direct
> >
> > > > >> > >>> > > > > > > > > >> zookeeper
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > interaction
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > from
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > any client.
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > * "Broker also stores
> the
> >
> > > > >> > >>> DelegationToken
> >
> > > > >> > >>> > > > > without
> >
> > > > >> > >>> > > > > > > the
> >
> > > > >> > >>> > > > > > > > > >> hmac in
> >
> > > > >> > >>> > > > > > > > > >> >> > the
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > zookeeper." : Sorry
> about
> >
> > > the
> >
> > > > >> > >>> confusion.
> >
> > > > >> > >>> > > The
> >
> > > > >> > >>> > > > > sole
> >
> > > > >> > >>> > > > > > > > > >> purpose of
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > zookeeper
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > in
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > this design is as
> >
> > > distribution
> >
> > > > >> > channel
> >
> > > > >> > >>> > for
> >
> > > > >> > >>> > > > > tokens
> >
> > > > >> > >>> > > > > > > > > >> between all
> >
> > > > >> > >>> > > > > > > > > >> >> > > > brokers
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > and a
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > layer that ensures only
> >
> > > tokens
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > that
> >
> > > > >> > >>> were
> >
> > > > >> > >>> > > > > > generated
> >
> > > > >> > >>> > > > > > > by
> >
> > > > >> > >>> > > > > > > > > >> making a
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > request
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > to a
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > broker will be accepted
> >
> > > (more
> >
> > > > on
> >
> > > > >> > this
> >
> > > > >> > >>> in
> >
> > > > >> > >>> > > > second
> >
> > > > >> > >>> > > > > > > > > >> paragraph). The
> >
> > > > >> > >>> > > > > > > > > >> >> > > > token
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > consists of few elements
> >
> > > > (owner,
> >
> > > > >> > >>> renewer,
> >
> > > > >> > >>> > > > uuid
> >
> > > > >> > >>> > > > > ,
> >
> > > > >> > >>> > > > > > > > > >> expiration,
> >
> > > > >> > >>> > > > > > > > > >> >> > > hmac)
> >
> > > > >> > >>> > > > > > > > > >> >> > > > ,
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > one
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > of
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > which is the finally
> >
> > > generated
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > hmac
> >
> > > > >> > >>> but
> >
> > > > >> > >>> > > hmac
> >
> > > > >> > >>> > > > it
> >
> > > > >> > >>> > > > > > > self
> >
> > > > >> > >>> > > > > > > > is
> >
> > > > >> > >>> > > > > > > > > >> >> > derivable
> >
> > > > >> > >>> > > > > > > > > >> >> > > > if
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > you
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > have all the other
> > elements
> >
> > > of
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > the
> >
> > > > >> > >>> token
> >
> > > > >> > >>> > +
> >
> > > > >> > >>> > > > > secret
> >
> > > > >> > >>> > > > > > > key
> >
> > > > >> > >>> > > > > > > > > to
> >
> > > > >> > >>> > > > > > > > > >> >> > generate
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > hmac.
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > Given zookeeper does not
> >
> > > > provide
> >
> > > > >> > SSL
> >
> > > > >> > >>> > > support
> >
> > > > >> > >>> > > > we
> >
> > > > >> > >>> > > > > > do
> >
> > > > >> > >>> > > > > > > > not
> >
> > > > >> > >>> > > > > > > > > >> want the
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > entire
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > token to be wire
> > transferred
> >
> > > > to
> >
> > > > >> > >>> zookeeper
> >
> > > > >> > >>> > > as
> >
> > > > >> > >>> > > > > that
> >
> > > > >> > >>> > > > > > > > will
> >
> > > > >> > >>> > > > > > > > > >> be an
> >
> > > > >> > >>> > > > > > > > > >> >> > > > insecure
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > wire
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > transfer. Instead we
> only
> >
> > > > store
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > all
> >
> > > > >> > >>> the
> >
> > > > >> > >>> > > other
> >
> > > > >> > >>> > > > > > > > elements
> >
> > > > >> > >>> > > > > > > > > >> of a
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > delegation
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > tokens. Brokers can read
> >
> > > these
> >
> > > > >> > >>> elements
> >
> > > > >> > >>> > and
> >
> > > > >> > >>> > > > > > because
> >
> > > > >> > >>> > > > > > > > > they
> >
> > > > >> > >>> > > > > > > > > >> also
> >
> > > > >> > >>> > > > > > > > > >> >> > > have
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > access
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > to secret key they will
> be
> >
> > > > able
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > to
> >
> > > > >> > >>> > generate
> >
> > > > >> > >>> > > > > hmac
> >
> > > > >> > >>> > > > > > on
> >
> > > > >> > >>> > > > > > > > > >> their end.
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > One of the alternative
> >
> > > > proposed
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > is
> >
> > > > >> > to
> >
> > > > >> > >>> > avoid
> >
> > > > >> > >>> > > > > > > zookeeper
> >
> > > > >> > >>> > > > > > > > > >> >> > > altogether. A
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > Client
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > will call broker with
> >
> > > required
> >
> > > > >> > >>> > information
> >
> > > > >> > >>> > > > > > (owner,
> >
> > > > >> > >>> > > > > > > > > >> renwer,
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > expiration)
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > and
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > get back (signed hmac,
> >
> > > uuid).
> >
> > > > >> > Broker
> >
> > > > >> > >>> > won't
> >
> > > > >> > >>> > > > > store
> >
> > > > >> > >>> > > > > > > this
> >
> > > > >> > >>> > > > > > > > > in
> >
> > > > >> > >>> > > > > > > > > >> >> > > zookeeper.
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > From
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > this point a client can
> >
> > > > contact
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > any
> >
> > > > >> > >>> > broker
> >
> > > > >> > >>> > > > with
> >
> > > > >> > >>> > > > > > all
> >
> > > > >> > >>> > > > > > > > the
> >
> > > > >> > >>> > > > > > > > > >> >> > > delegation
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > token
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > info (owner, rewner,
> >
> > > > expiration,
> >
> > > > >> > hmac,
> >
> > > > >> > >>> > > uuid)
> >
> > > > >> > >>> > > > > the
> >
> > > > >> > >>> > > > > > > > borker
> >
> > > > >> > >>> > > > > > > > > >> will
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > regenerate
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > the
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > hmac and as long as it
> >
> > > matches
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > with
> >
> > > > >> > >>> hmac
> >
> > > > >> > >>> > > > > > presented
> >
> > > > >> > >>> > > > > > > by
> >
> > > > >> > >>> > > > > > > > > >> client ,
> >
> > > > >> > >>> > > > > > > > > >> >> > > > broker
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > will
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > allow the request to
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > authenticate.
> >
> > > > >> > >>> Only
> >
> > > > >> > >>> > > > > problem
> >
> > > > >> > >>> > > > > > > with
> >
> > > > >> > >>> > > > > > > > > >> this
> >
> > > > >> > >>> > > > > > > > > >> >> > > approach
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > is
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > if
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > the secret key is
> >
> > > compromised
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > any
> >
> > > > >> > >>> client
> >
> > > > >> > >>> > > can
> >
> > > > >> > >>> > > > > now
> >
> > > > >> > >>> > > > > > > > > generate
> >
> > > > >> > >>> > > > > > > > > >> >> > random
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > tokens
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > and
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > they will still be able
> to
> >
> > > > >> > >>> authenticate
> >
> > > > >> > >>> > as
> >
> > > > >> > >>> > > > any
> >
> > > > >> > >>> > > > > > user
> >
> > > > >> > >>> > > > > > > > > they
> >
> > > > >> > >>> > > > > > > > > >> like.
> >
> > > > >> > >>> > > > > > > > > >> >> > > with
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > zookeeper we guarantee
> > that
> >
> > > > only
> >
> > > > >> > >>> tokens
> >
> > > > >> > >>> > > > > acquired
> >
> > > > >> > >>> > > > > > > via
> >
> > > > >> > >>> > > > > > > > a
> >
> > > > >> > >>> > > > > > > > > >> broker
> >
> > > > >> > >>> > > > > > > > > >> >> > > > (using
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > some
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > auth scheme other than
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > delegation
> >
> > > > >> > >>> token)
> >
> > > > >> > >>> > > will
> >
> > > > >> > >>> > > > > be
> >
> > > > >> > >>> > > > > > > > > >> accepted. We
> >
> > > > >> > >>> > > > > > > > > >> >> > > need
> >
> > > > >> > >>> > > > > > > > > >> >> > > > to
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > discuss which proposal
> > makes
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > more
> >
> > > > >> > >>> sense
> >
> > > > >> > >>> > and
> >
> > > > >> > >>> > > > we
> >
> > > > >> > >>> > > > > > can
> >
> > > > >> > >>> > > > > > > go
> >
> > > > >> > >>> > > > > > > > > >> over it
> >
> > > > >> > >>> > > > > > > > > >> >> > in
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > tomorrow's
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > meeting.
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > Also, can you forward
> the
> >
> > > > invite
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > to
> >
> > > > >> > >>> me?
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > Thanks
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > Parth
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > On Mon, May 23, 2016 at
> >
> > > 10:35
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > AM,
> >
> > > > >> > Jun
> >
> > > > >> > >>> > Rao <
> >
> > > > >> > >>> > > > > > > > > >> jun@confluent.io>
> >
> > > > >> > >>> > > > > > > > > >> >> > > > wrote:
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > > Thanks for the KIP. A
> > few
> >
> > > > >> > comments.
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > > 100. This potentially
> > can
> >
> > > be
> >
> > > > >> > useful
> >
> > > > >> > >>> for
> >
> > > > >> > >>> > > > Kafka
> >
> > > > >> > >>> > > > > > > > Connect
> >
> > > > >> > >>> > > > > > > > > >> and
> >
> > > > >> > >>> > > > > > > > > >> >> > Kafka
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > rest
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > proxy
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > > where a worker agent
> > will
> >
> > > > need
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > > to
> >
> > > > >> > >>> run a
> >
> > > > >> > >>> > > > task
> >
> > > > >> > >>> > > > > on
> >
> > > > >> > >>> > > > > > > > > behalf
> >
> > > > >> > >>> > > > > > > > > >> of a
> >
> > > > >> > >>> > > > > > > > > >> >> > > > client.
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > We
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > will
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > > likely need to change
> > how
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > > those
> >
> > > > >> > >>> > services
> >
> > > > >> > >>> > > > use
> >
> > > > >> > >>> > > > > > > Kafka
> >
> > > > >> > >>> > > > > > > > > >> clients
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > > (producer/consumer).
> >
> > > Instead
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > > of a
> >
> > > > >> > >>> > shared
> >
> > > > >> > >>> > > > > client
> >
> > > > >> > >>> > > > > > > per
> >
> > > > >> > >>> > > > > > > > > >> worker,
> >
> > > > >> > >>> > > > > > > > > >> >> > we
> >
> > > > >> > >>> > > > > > > > > >> >> > > > will
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > need
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > a
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > > client per user task
> > since
> >
> > > > the
> >
> > > > >> > >>> > > > authentication
> >
> > > > >> > >>> > > > > > > > happens
> >
> > > > >> > >>> > > > > > > > > >> at the
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > connection
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > > level. For Kafka
> > Connect,
> >
> > > > the
> >
> > > > >> > >>> renewer
> >
> > > > >> > >>> > > will
> >
> > > > >> > >>> > > > be
> >
> > > > >> > >>> > > > > > the
> >
> > > > >> > >>> > > > > > > > > >> workers.
> >
> > > > >> > >>> > > > > > > > > >> >> > So,
> >
> > > > >> > >>> > > > > > > > > >> >> > > we
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > probably
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > > need to allow multiple
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > > renewers.
> >
> > > > >> > For
> >
> > > > >> > >>> > > Kafka
> >
> > > > >> > >>> > > > > rest
> >
> > > > >> > >>> > > > > > > > > proxy,
> >
> > > > >> > >>> > > > > > > > > >> the
> >
> > > > >> > >>> > > > > > > > > >> >> > > > renewer
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > can
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > > probably just be the
> >
> > > creator
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > > of
> >
> > > > >> > the
> >
> > > > >> > >>> > > token.
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > > 101. Do we need new
> acl
> > on
> >
> > > > who
> >
> > > > >> > can
> >
> > > > >> > >>> > > request
> >
> > > > >> > >>> > > > > > > > delegation
> >
> > > > >> > >>> > > > > > > > > >> tokens?
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > >
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > > 102. Do we recommend
> >
> > > people
> >
> > > > to
> >
> > > > >> > send
> >
> > > > >> > >>> > > > > delegation
> >
> > > > >> > >>> > > > > > > > tokens
> >
> > > > >> > >>> > > > > > > > > >> in an
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > encrypted
> >
> > > > >> > >>> > > > > > > > > >> >> > > > > > > > > channel?
> >
> > > > >> > >>>
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Ashish Singh <as...@cloudera.com>]

Manikumar,

Thanks for the update. As its been a while since any progress was made
here, I started working on POC for this. Will be happy to share the
progress I have made so far, let's discuss that on parent JIRA. However,
more importantly while working on POC, I had following questions that I
think are worth considering.

1. How to disable delegation token authentication?

Not everyone would want to authenticate via delegation token. If someone is
not interested in this, then this unnecessarily increases threat vector for
such users. Having a way to turn it off will be appreciated by these users.
Rather I think by default this should be kept off.

This can be achieved in various ways, however I think reusing delegation
token secret config for this makes sense here. Avoids creating yet another
config and forces delegation token users to consciously set the secret. If
the secret is not set or set to empty string, brokers should turn off
delegation token support. This will however require a new error code to
indicate delegation token support is turned off on broker.

2. ACLs on delegation token?

Do we need to have ACLs defined for tokens? I do not think it buys us
anything, as delegation token can be treated as impersonation of the owner.
Any thing the owner has permission to do, delegation tokens should be
allowed to do as well. If so, we probably won't need to return
authorization exception error code while creating delegation token. It
however would make sense to check renew and expire requests are coming from
owner or renewers of the token, but that does not require explicit acls.

3. How to restrict max life time of a token?

Admins might want to restrict max life time of tokens created on a cluster,
and this can very from cluster to cluster based on use-cases. This might
warrant a separate broker config.

Few more comments based on recent KIP update.

1. Do we need a separate {{InvalidateTokenRequest}}? Can't we use
{{ExpireTokenRequest}} with with expiryDate set to anything before current
date?

2. Can we change time field names to indicate their unit is milliseconds,
like, IssueDateMs, ExpiryDateMs, etc.?

3. Can we allow users to renew a token for a specified amount of time? In
current version of KIP, renew request does not take time as a param, not
sure what is expiry time set to after renewal.


On Mon, Dec 12, 2016 at 9:08 AM Manikumar <ma...@gmail.com> wrote:

> Hi,
>
>
>
> I would like to reinitiate the discussion on Delegation token support for
>
> Kafka.
>
>
>
> Brief summary of the past discussion:
>
>
>
> 1) Broker stores delegation tokens in zookeeper.  All brokers will have a
>
> cache backed by
>
>    zookeeper so they will all get notified whenever a new token is
>
> generated and they will
>
>    update their local cache whenever token state changes.
>
> 2) The current proposal does not support rotation of secret
>
> 3) Only allow the renewal by users that authenticated using *non*
>
> delegation token mechanism
>
> 4) KIP-84 proposes to support  SASL SCRAM mechanisms. Kafka clients can
>
> authenticate using
>
>    SCRAM-SHA-256, providing the delegation token HMAC as password.
>
>
>
> Updated the KIP with the following:
>
> 1. Protocol and Config changes
>
> 2. format of the data stored in ZK.
>
> 3. Changes to Java Clients/Usage of SASL SCRAM mechanism
>
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> 48+Delegation+token+support+for+Kafka
>
>
>
>
>
> Jun, Ashish, Gwen,
>
>
>
> Pl review the updated KIP.
>
>
>
> Thanks
>
> Manikumar
>
>
>
>
>
> On Thu, Sep 29, 2016 at 9:56 PM, Ashish Singh <as...@cloudera.com> wrote:
>
>
>
> > Harsha/ Gwen,
>
> >
>
> > How do we proceed here? I am willing to help out with here.
>
> >
>
> > On Fri, Sep 23, 2016 at 11:41 AM, Gwen Shapira <gw...@confluent.io>
> wrote:
>
> >
>
> > > Is it updated? are all concerns addressed? do you want to start a vote?
>
> > >
>
> > > Sorry for being pushy, I do appreciate that we are all volunteers and
>
> > > finding time is difficult. This feature is important for anything that
>
> > > integrates with Kafka (stream processors, Flume, NiFi, etc) and I
>
> > > don't want to see this getting stuck because we lack coordination
>
> > > within the community.
>
> > >
>
> > > On Thu, Sep 15, 2016 at 6:39 PM, Harsha Chintalapani <ka...@harsha.io>
>
> > > wrote:
>
> > > > The only pending update for the KIP is to write up the protocol
> changes
>
> > > like
>
> > > > we've it KIP-4. I'll update the wiki.
>
> > > >
>
> > > >
>
> > > > On Thu, Sep 15, 2016 at 4:27 PM Ashish Singh <as...@cloudera.com>
>
> > > wrote:
>
> > > >>
>
> > > >> I think we decided to not support secret rotation, I guess this can
> be
>
> > > >> stated clearly on the KIP. Also, more details on how clients will
>
> > > perform
>
> > > >> token distribution and how CLI will look like will be helpful.
>
> > > >>
>
> > > >> On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <gw...@confluent.io>
>
> > > wrote:
>
> > > >>
>
> > > >> > Hi Guys,
>
> > > >> >
>
> > > >> > This discussion was dead for a while. Are there still contentious
>
> > > >> > points? If not, why are there no votes?
>
> > > >> >
>
> > > >> > On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <ju...@confluent.io>
> wrote:
>
> > > >> > > Ashish,
>
> > > >> > >
>
> > > >> > > Yes, I will send out a KIP invite for next week to discuss
> KIP-48
>
> > > and
>
> > > >> > other
>
> > > >> > > remaining KIPs.
>
> > > >> > >
>
> > > >> > > Thanks,
>
> > > >> > >
>
> > > >> > > Jun
>
> > > >> > >
>
> > > >> > > On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh <
>
> > asingh@cloudera.com>
>
> > > >> > wrote:
>
> > > >> > >
>
> > > >> > >> Thanks Harsha!
>
> > > >> > >>
>
> > > >> > >> Jun, can we add KIP-48 to next KIP hangout's agenda. Also, we
> did
>
> > > not
>
> > > >> > >> actually make a call on when we should have next KIP call. As
>
> > there
>
> > > >> > >> are
>
> > > >> > a
>
> > > >> > >> few outstanding KIPs that could not be discussed this week, can
>
> > we
>
> > > >> > >> have
>
> > > >> > a
>
> > > >> > >> KIP hangout call next week?
>
> > > >> > >>
>
> > > >> > >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha Chintalapani
>
> > > >> > >> <ka...@harsha.io>
>
> > > >> > >> wrote:
>
> > > >> > >>
>
> > > >> > >>> Ashish,
>
> > > >> > >>>         Yes we are working on it. Lets discuss in the next KIP
>
> > > >> > >>> meeting.
>
> > > >> > >>> I'll join.
>
> > > >> > >>> -Harsha
>
> > > >> > >>>
>
> > > >> > >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh <
>
> > > asingh@cloudera.com>
>
> > > >> > >>> wrote:
>
> > > >> > >>>
>
> > > >> > >>> > Hello Harsha,
>
> > > >> > >>> >
>
> > > >> > >>> > Are you still working on this? Wondering if we can discuss
>
> > this
>
> > > in
>
> > > >> > next
>
> > > >> > >>> KIP
>
> > > >> > >>> > meeting, if you can join.
>
> > > >> > >>> >
>
> > > >> > >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha Chintalapani <
>
> > > >> > kafka@harsha.io>
>
> > > >> > >>> > wrote:
>
> > > >> > >>> >
>
> > > >> > >>> > > Hi Grant,
>
> > > >> > >>> > >           We are working on it. Will add the details to
> KIP
>
> > > >> > >>> > > about
>
> > > >> > the
>
> > > >> > >>> > > request protocol.
>
> > > >> > >>> > >
>
> > > >> > >>> > > Thanks,
>
> > > >> > >>> > > Harsha
>
> > > >> > >>> > >
>
> > > >> > >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant Henke
>
> > > >> > >>> > > <gh...@cloudera.com>
>
> > > >> > >>> wrote:
>
> > > >> > >>> > >
>
> > > >> > >>> > > > Hi Parth,
>
> > > >> > >>> > > >
>
> > > >> > >>> > > > Are you still working on this? If you need any help
> please
>
> > > >> > >>> > > > don't
>
> > > >> > >>> > hesitate
>
> > > >> > >>> > > > to ask.
>
> > > >> > >>> > > >
>
> > > >> > >>> > > > Thanks,
>
> > > >> > >>> > > > Grant
>
> > > >> > >>> > > >
>
> > > >> > >>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <
>
> > jun@confluent.io>
>
> > > >> > wrote:
>
> > > >> > >>> > > >
>
> > > >> > >>> > > > > Parth,
>
> > > >> > >>> > > > >
>
> > > >> > >>> > > > > Thanks for the reply.
>
> > > >> > >>> > > > >
>
> > > >> > >>> > > > > It makes sense to only allow the renewal by users that
>
> > > >> > >>> authenticated
>
> > > >> > >>> > > > using
>
> > > >> > >>> > > > > *non* delegation token mechanism. Then, should we make
>
> > the
>
> > > >> > >>> renewal a
>
> > > >> > >>> > > > list?
>
> > > >> > >>> > > > > For example, in the case of rest proxy, it will be
>
> > useful
>
> > > >> > >>> > > > > for
>
> > > >> > >>> every
>
> > > >> > >>> > > > > instance of rest proxy to be able to renew the tokens.
>
> > > >> > >>> > > > >
>
> > > >> > >>> > > > > It would be clearer if we can document the request
>
> > > protocol
>
> > > >> > like
>
> > > >> > >>> > > > >
>
> > > >> > >>> > > > >
>
> > > >> > >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> > > >> > >>> > >
>
> > > >> > >>> > > 4+-+Command+line+and+centralized+administrative+
>
> > > operations#KIP-4-
>
> > > >> > >>> > > Commandlineandcentralizedadministrativeoperations-
>
> > > >> > >>> > > CreateTopicsRequest(KAFKA-2945):(VotedandPlannedforin0.
>
> > > 10.1.0)
>
> > > >> > >>> > > > > .
>
> > > >> > >>> > > > >
>
> > > >> > >>> > > > > It would also be useful to document the client APIs.
>
> > > >> > >>> > > > >
>
> > > >> > >>> > > > > Thanks,
>
> > > >> > >>> > > > >
>
> > > >> > >>> > > > > Jun
>
> > > >> > >>> > > > >
>
> > > >> > >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM, parth brahmbhatt <
>
> > > >> > >>> > > > > brahmbhatt.parth@gmail.com> wrote:
>
> > > >> > >>> > > > >
>
> > > >> > >>> > > > > > Hi,
>
> > > >> > >>> > > > > >
>
> > > >> > >>> > > > > > I am suggesting that we will only allow the renewal
> by
>
> > > >> > >>> > > > > > users
>
> > > >> > >>> that
>
> > > >> > >>> > > > > > authenticated using *non* delegation token
> mechanism.
>
> > > For
>
> > > >> > >>> example,
>
> > > >> > >>> > If
>
> > > >> > >>> > > > > user
>
> > > >> > >>> > > > > > Alice authenticated using kerberos and requested
>
> > > >> > >>> > > > > > delegation
>
> > > >> > >>> tokens,
>
> > > >> > >>> > > > only
>
> > > >> > >>> > > > > > user Alice authenticated via non delegation token
>
> > > >> > >>> > > > > > mechanism
>
> > > >> > can
>
> > > >> > >>> > > renew.
>
> > > >> > >>> > > > > > Clients that have  access to delegation tokens can
> not
>
> > > >> > >>> > > > > > issue
>
> > > >> > >>> > renewal
>
> > > >> > >>> > > > > > request for renewing their own token and this is
>
> > > primarily
>
> > > >> > >>> > important
>
> > > >> > >>> > > to
>
> > > >> > >>> > > > > > reduce the time window for which a compromised token
>
> > > will
>
> > > >> > >>> > > > > > be
>
> > > >> > >>> valid.
>
> > > >> > >>> > > > > >
>
> > > >> > >>> > > > > > To clarify, Yes any authenticated user can request
>
> > > >> > >>> > > > > > delegation
>
> > > >> > >>> > tokens
>
> > > >> > >>> > > > but
>
> > > >> > >>> > > > > > even here I would recommend to avoid creating a
> chain
>
> > > >> > >>> > > > > > where a
>
> > > >> > >>> > client
>
> > > >> > >>> > > > > > authenticated via delegation token request for more
>
> > > >> > delegation
>
> > > >> > >>> > > tokens.
>
> > > >> > >>> > > > > > Basically anyone can request delegation token, as
> long
>
> > > as
>
> > > >> > they
>
> > > >> > >>> > > > > authenticate
>
> > > >> > >>> > > > > > via a non delegation token mechanism.
>
> > > >> > >>> > > > > >
>
> > > >> > >>> > > > > > Aren't classes listed here
>
> > > >> > >>> > > > > > <
>
> > > >> > >>> > > > > >
>
> > > >> > >>> > > > >
>
> > > >> > >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> > > >> > >>> > > 48+Delegation+token+support+fo
>
> > r+Kafka#KIP-48Delegationtokens
>
> > > >> > >>> upportforKaf
>
> > > >> > >>> > > ka-PublicInterfaces
>
> > > >> > >>> > > > > > >
>
> > > >> > >>> > > > > > sufficient?
>
> > > >> > >>> > > > > >
>
> > > >> > >>> > > > > > Thanks
>
> > > >> > >>> > > > > > Parth
>
> > > >> > >>> > > > > >
>
> > > >> > >>> > > > > >
>
> > > >> > >>> > > > > >
>
> > > >> > >>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao
>
> > > >> > >>> > > > > > <ju...@confluent.io>
>
> > > >> > >>> wrote:
>
> > > >> > >>> > > > > >
>
> > > >> > >>> > > > > > > Parth,
>
> > > >> > >>> > > > > > >
>
> > > >> > >>> > > > > > > Thanks for the reply. A couple of comments inline
>
> > > below.
>
> > > >> > >>> > > > > > >
>
> > > >> > >>> > > > > > > On Tue, Jun 21, 2016 at 10:36 AM, parth
> brahmbhatt <
>
> > > >> > >>> > > > > > > brahmbhatt.parth@gmail.com> wrote:
>
> > > >> > >>> > > > > > >
>
> > > >> > >>> > > > > > > > 1. Who / how are tokens renewed? By original
>
> > > requester
>
> > > >> > >>> only? or
>
> > > >> > >>> > > > using
>
> > > >> > >>> > > > > > > > Kerberos
>
> > > >> > >>> > > > > > > > auth only?
>
> > > >> > >>> > > > > > > > My recommendation is to do this only using
>
> > Kerberos
>
> > > >> > >>> > > > > > > > auth
>
> > > >> > and
>
> > > >> > >>> > only
>
> > > >> > >>> > > > > threw
>
> > > >> > >>> > > > > > > the
>
> > > >> > >>> > > > > > > > renewer specified during the acquisition
> request.
>
> > > >> > >>> > > > > > > >
>
> > > >> > >>> > > > > > > >
>
> > > >> > >>> > > > > > > Hmm, not sure that I follow this. Are you saying
>
> > that
>
> > > >> > >>> > > > > > > any
>
> > > >> > >>> client
>
> > > >> > >>> > > > > > > authenticated with the delegation token can renew,
>
> > > i.e.
>
> > > >> > there
>
> > > >> > >>> is
>
> > > >> > >>> > no
>
> > > >> > >>> > > > > > renewer
>
> > > >> > >>> > > > > > > needed?
>
> > > >> > >>> > > > > > >
>
> > > >> > >>> > > > > > > Also, just to be clear, any authenticated client
>
> > > (either
>
> > > >> > >>> through
>
> > > >> > >>> > > SASL
>
> > > >> > >>> > > > > or
>
> > > >> > >>> > > > > > > SSL) can request a delegation token for the
>
> > > >> > >>> > > > > > > authenticated
>
> > > >> > >>> user,
>
> > > >> > >>> > > > right?
>
> > > >> > >>> > > > > > >
>
> > > >> > >>> > > > > > >
>
> > > >> > >>> > > > > > > > 2. Are tokens stored on each broker or in ZK?
>
> > > >> > >>> > > > > > > > My recommendation is still to store in ZK or not
>
> > > store
>
> > > >> > them
>
> > > >> > >>> at
>
> > > >> > >>> > > all.
>
> > > >> > >>> > > > > The
>
> > > >> > >>> > > > > > > > whole controller based distribution is too much
>
> > > >> > >>> > > > > > > > overhead
>
> > > >> > >>> with
>
> > > >> > >>> > not
>
> > > >> > >>> > > > > much
>
> > > >> > >>> > > > > > to
>
> > > >> > >>> > > > > > > > achieve.
>
> > > >> > >>> > > > > > > >
>
> > > >> > >>> > > > > > > > 3. How are tokens invalidated / expired?
>
> > > >> > >>> > > > > > > > Either by expiration time out or through an
>
> > explicit
>
> > > >> > >>> request to
>
> > > >> > >>> > > > > > > invalidate.
>
> > > >> > >>> > > > > > > >
>
> > > >> > >>> > > > > > > > 4. Which encryption algorithm is used?
>
> > > >> > >>> > > > > > > > SCRAM
>
> > > >> > >>> > > > > > > >
>
> > > >> > >>> > > > > > > > 5. What is the impersonation proposal (it wasn't
>
> > in
>
> > > >> > >>> > > > > > > > the
>
> > > >> > KIP
>
> > > >> > >>> but
>
> > > >> > >>> > > was
>
> > > >> > >>> > > > > > > > discussed
>
> > > >> > >>> > > > > > > > in this thread)?
>
> > > >> > >>> > > > > > > > There is no imperonation proposal. I tried and
>
> > > >> > >>> > > > > > > > explained
>
> > > >> > how
>
> > > >> > >>> > its
>
> > > >> > >>> > > a
>
> > > >> > >>> > > > > > > > different problem and why its not really
> necessary
>
> > > to
>
> > > >> > >>> discuss
>
> > > >> > >>> > > that
>
> > > >> > >>> > > > as
>
> > > >> > >>> > > > > > > part
>
> > > >> > >>> > > > > > > > of this KIP.  This KIP will not support any
>
> > > >> > impersonation,
>
> > > >> > >>> it
>
> > > >> > >>> > > will
>
> > > >> > >>> > > > > just
>
> > > >> > >>> > > > > > > be
>
> > > >> > >>> > > > > > > > another way to authenticate.
>
> > > >> > >>> > > > > > > >
>
> > > >> > >>> > > > > > > > 6. Do we need new ACLs, if so - for what
> actions?
>
> > > >> > >>> > > > > > > > We do not need new ACLs.
>
> > > >> > >>> > > > > > > >
>
> > > >> > >>> > > > > > > >
>
> > > >> > >>> > > > > > > Could we document the format of the new
>
> > > request/response
>
> > > >> > and
>
> > > >> > >>> > their
>
> > > >> > >>> > > > > > > associated Resource and Operation for ACL?
>
> > > >> > >>> > > > > > >
>
> > > >> > >>> > > > > > >
>
> > > >> > >>> > > > > > > > 7. How would the delegation token be configured
> in
>
> > > the
>
> > > >> > >>> client?
>
> > > >> > >>> > > > > > > > Should be through config. I wasn't planning on
>
> > > >> > >>> > > > > > > > supporting
>
> > > >> > >>> JAAS
>
> > > >> > >>> > > for
>
> > > >> > >>> > > > > > > tokens.
>
> > > >> > >>> > > > > > > > I don't believe hadoop does this either.
>
> > > >> > >>> > > > > > > >
>
> > > >> > >>> > > > > > > > Thanks
>
> > > >> > >>> > > > > > > > Parth
>
> > > >> > >>> > > > > > > >
>
> > > >> > >>> > > > > > > >
>
> > > >> > >>> > > > > > > >
>
> > > >> > >>> > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <
>
> > > >> > jun@confluent.io>
>
> > > >> > >>> > > wrote:
>
> > > >> > >>> > > > > > > >
>
> > > >> > >>> > > > > > > > > Harsha,
>
> > > >> > >>> > > > > > > > >
>
> > > >> > >>> > > > > > > > > Another question.
>
> > > >> > >>> > > > > > > > >
>
> > > >> > >>> > > > > > > > > 9. How would the delegation token be
> configured
>
> > in
>
> > > >> > >>> > > > > > > > > the
>
> > > >> > >>> > client?
>
> > > >> > >>> > > > The
>
> > > >> > >>> > > > > > > > standard
>
> > > >> > >>> > > > > > > > > way is to do this through JAAS. However, we
> will
>
> > > >> > >>> > > > > > > > > need
>
> > > >> > to
>
> > > >> > >>> > think
>
> > > >> > >>> > > > > > through
>
> > > >> > >>> > > > > > > if
>
> > > >> > >>> > > > > > > > > this is convenient in a shared environment.
> For
>
> > > >> > example,
>
> > > >> > >>> > when a
>
> > > >> > >>> > > > new
>
> > > >> > >>> > > > > > > task
>
> > > >> > >>> > > > > > > > is
>
> > > >> > >>> > > > > > > > > added to a Storm worker node, do we need to
>
> > > >> > >>> > > > > > > > > dynamically
>
> > > >> > >>> add a
>
> > > >> > >>> > > new
>
> > > >> > >>> > > > > > > section
>
> > > >> > >>> > > > > > > > > in the JAAS file? It may be more convenient if
>
> > we
>
> > > >> > >>> > > > > > > > > can
>
> > > >> > >>> pass in
>
> > > >> > >>> > > the
>
> > > >> > >>> > > > > > token
>
> > > >> > >>> > > > > > > > > through the config directly w/o going through
>
> > > JAAS.
>
> > > >> > >>> > > > > > > > >
>
> > > >> > >>> > > > > > > > > Are you or Parth still actively working on
> this
>
> > > KIP?
>
> > > >> > >>> > > > > > > > >
>
> > > >> > >>> > > > > > > > > Thanks,
>
> > > >> > >>> > > > > > > > >
>
> > > >> > >>> > > > > > > > > Jun
>
> > > >> > >>> > > > > > > > >
>
> > > >> > >>> > > > > > > > >
>
> > > >> > >>> > > > > > > > >
>
> > > >> > >>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <
>
> > > >> > >>> jun@confluent.io>
>
> > > >> > >>> > > > wrote:
>
> > > >> > >>> > > > > > > > >
>
> > > >> > >>> > > > > > > > > > Just to add on that list.
>
> > > >> > >>> > > > > > > > > >
>
> > > >> > >>> > > > > > > > > > 2. It would be good to document the format
> of
>
> > > the
>
> > > >> > data
>
> > > >> > >>> > stored
>
> > > >> > >>> > > > in
>
> > > >> > >>> > > > > > ZK.
>
> > > >> > >>> > > > > > > > > > 7. Earlier, there was a discussion on
> whether
>
> > > the
>
> > > >> > tokens
>
> > > >> > >>> > > should
>
> > > >> > >>> > > > > be
>
> > > >> > >>> > > > > > > > > > propagated through ZK like config/acl/quota,
>
> > or
>
> > > >> > through
>
> > > >> > >>> the
>
> > > >> > >>> > > > > > > controller.
>
> > > >> > >>> > > > > > > > > > Currently, the controller is only designed
> for
>
> > > >> > >>> propagating
>
> > > >> > >>> > > > topic
>
> > > >> > >>> > > > > > > > > metadata,
>
> > > >> > >>> > > > > > > > > > but not other data.
>
> > > >> > >>> > > > > > > > > > 8. Should we use SCRAM to send the token
>
> > instead
>
> > > >> > >>> > > > > > > > > > of
>
> > > >> > >>> > > DIGEST-MD5
>
> > > >> > >>> > > > > > since
>
> > > >> > >>> > > > > > > > it's
>
> > > >> > >>> > > > > > > > > > deprecated?
>
> > > >> > >>> > > > > > > > > >
>
> > > >> > >>> > > > > > > > > > Also, the images in the wiki seem broken.
>
> > > >> > >>> > > > > > > > > >
>
> > > >> > >>> > > > > > > > > > Thanks,
>
> > > >> > >>> > > > > > > > > >
>
> > > >> > >>> > > > > > > > > > Jun
>
> > > >> > >>> > > > > > > > > >
>
> > > >> > >>> > > > > > > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen
>
> > Shapira <
>
> > > >> > >>> > > > > gwen@confluent.io>
>
> > > >> > >>> > > > > > > > > wrote:
>
> > > >> > >>> > > > > > > > > >
>
> > > >> > >>> > > > > > > > > >> From what I can see, remaining questions
> are:
>
> > > >> > >>> > > > > > > > > >>
>
> > > >> > >>> > > > > > > > > >> 1. Who / how are tokens renewed? By
> original
>
> > > >> > requester
>
> > > >> > >>> > only?
>
> > > >> > >>> > > > or
>
> > > >> > >>> > > > > > > using
>
> > > >> > >>> > > > > > > > > >> Kerberos auth only?
>
> > > >> > >>> > > > > > > > > >> 2. Are tokens stored on each broker or in
> ZK?
>
> > > >> > >>> > > > > > > > > >> 3. How are tokens invalidated / expired?
>
> > > >> > >>> > > > > > > > > >> 4. Which encryption algorithm is used?
>
> > > >> > >>> > > > > > > > > >> 5. What is the impersonation proposal (it
>
> > > wasn't
>
> > > >> > >>> > > > > > > > > >> in
>
> > > >> > the
>
> > > >> > >>> > KIP
>
> > > >> > >>> > > > but
>
> > > >> > >>> > > > > > was
>
> > > >> > >>> > > > > > > > > >> discussed in this thread)?
>
> > > >> > >>> > > > > > > > > >> 6. Do we need new ACLs, if so - for what
>
> > > actions?
>
> > > >> > >>> > > > > > > > > >>
>
> > > >> > >>> > > > > > > > > >> Gwen
>
> > > >> > >>> > > > > > > > > >>
>
> > > >> > >>> > > > > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <
>
> > > >> > >>> kafka@harsha.io>
>
> > > >> > >>> > > > wrote:
>
> > > >> > >>> > > > > > > > > >> > Jun & Ismael,
>
> > > >> > >>> > > > > > > > > >> >                          Unfortunately I
>
> > > >> > >>> > > > > > > > > >> > couldn't
>
> > > >> > >>> attend
>
> > > >> > >>> > > the
>
> > > >> > >>> > > > > KIP
>
> > > >> > >>> > > > > > > > > meeting
>
> > > >> > >>> > > > > > > > > >> >                          when delegation
>
> > > tokens
>
> > > >> > >>> > discussed.
>
> > > >> > >>> > > > > > > > Appreciate
>
> > > >> > >>> > > > > > > > > if
>
> > > >> > >>> > > > > > > > > >> >                          you can update
> the
>
> > > >> > thread if
>
> > > >> > >>> > you
>
> > > >> > >>> > > > have
>
> > > >> > >>> > > > > > any
>
> > > >> > >>> > > > > > > > > >> >                          further
> questions.
>
> > > >> > >>> > > > > > > > > >> > Thanks,
>
> > > >> > >>> > > > > > > > > >> > Harsha
>
> > > >> > >>> > > > > > > > > >> >
>
> > > >> > >>> > > > > > > > > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan
>
> > Pei
>
> > > >> > wrote:
>
> > > >> > >>> > > > > > > > > >> >> It seems that the links to images in the
>
> > KIP
>
> > > >> > >>> > > > > > > > > >> >> are
>
> > > >> > >>> > broken.
>
> > > >> > >>> > > > > > > > > >> >>
>
> > > >> > >>> > > > > > > > > >> >> Liquan
>
> > > >> > >>> > > > > > > > > >> >>
>
> > > >> > >>> > > > > > > > > >> >> On Tue, May 24, 2016 at 9:33 AM, parth
>
> > > >> > brahmbhatt <
>
> > > >> > >>> > > > > > > > > >> >> brahmbhatt.parth@gmail.com> wrote:
>
> > > >> > >>> > > > > > > > > >> >>
>
> > > >> > >>> > > > > > > > > >> >> > 110. What does getDelegationTokenAs
>
> > mean?
>
> > > >> > >>> > > > > > > > > >> >> > In the current proposal we only allow
> a
>
> > > user
>
> > > >> > >>> > > > > > > > > >> >> > to
>
> > > >> > >>> get
>
> > > >> > >>> > > > > > delegation
>
> > > >> > >>> > > > > > > > > token
>
> > > >> > >>> > > > > > > > > >> for
>
> > > >> > >>> > > > > > > > > >> >> > the identity that it authenticated as
>
> > > using
>
> > > >> > >>> another
>
> > > >> > >>> > > > > > mechanism,
>
> > > >> > >>> > > > > > > > i.e.
>
> > > >> > >>> > > > > > > > > >> A user
>
> > > >> > >>> > > > > > > > > >> >> > that authenticate using a keytab for
>
> > > >> > >>> > > > > > > > > >> >> > principal
>
> > > >> > >>> > > > > > > user1@EXAMPLE.COM
>
> > > >> > >>> > > > > > > > > >> will get
>
> > > >> > >>> > > > > > > > > >> >> > delegation tokens for that user only.
> In
>
> > > >> > future I
>
> > > >> > >>> > think
>
> > > >> > >>> > > > we
>
> > > >> > >>> > > > > > will
>
> > > >> > >>> > > > > > > > > have
>
> > > >> > >>> > > > > > > > > >> to
>
> > > >> > >>> > > > > > > > > >> >> > extend support such that we allow some
>
> > set
>
> > > >> > >>> > > > > > > > > >> >> > of
>
> > > >> > >>> users (
>
> > > >> > >>> > > > > > > > > >> >> > kafka-rest-user@EXAMPLE.COM,
>
> > > >> > >>> > storm-nimbus@EXAMPLE.COM)
>
> > > >> > >>> > > > to
>
> > > >> > >>> > > > > > > > acquire
>
> > > >> > >>> > > > > > > > > >> >> > delegation tokens on behalf of other
>
> > users
>
> > > >> > whose
>
> > > >> > >>> > > identity
>
> > > >> > >>> > > > > > they
>
> > > >> > >>> > > > > > > > have
>
> > > >> > >>> > > > > > > > > >> >> > verified independently.  Kafka brokers
>
> > > will
>
> > > >> > have
>
> > > >> > >>> ACLs
>
> > > >> > >>> > > to
>
> > > >> > >>> > > > > > > control
>
> > > >> > >>> > > > > > > > > >> which
>
> > > >> > >>> > > > > > > > > >> >> > users are allowed to impersonate other
>
> > > users
>
> > > >> > and
>
> > > >> > >>> get
>
> > > >> > >>> > > > tokens
>
> > > >> > >>> > > > > > on
>
> > > >> > >>> > > > > > > > > >> behalf of
>
> > > >> > >>> > > > > > > > > >> >> > them. Overall Impersonation is a whole
>
> > > >> > different
>
> > > >> > >>> > > problem
>
> > > >> > >>> > > > in
>
> > > >> > >>> > > > > > my
>
> > > >> > >>> > > > > > > > > >> opinion and
>
> > > >> > >>> > > > > > > > > >> >> > I think we can tackle it in separate
>
> > KIP.
>
> > > >> > >>> > > > > > > > > >> >> >
>
> > > >> > >>> > > > > > > > > >> >> > 111. What's the typical rate of
> getting
>
> > > and
>
> > > >> > >>> renewing
>
> > > >> > >>> > > > > > delegation
>
> > > >> > >>> > > > > > > > > >> tokens?
>
> > > >> > >>> > > > > > > > > >> >> > Typically this should be very very
> low,
>
> > 1
>
> > > >> > request
>
> > > >> > >>> per
>
> > > >> > >>> > > > > minute
>
> > > >> > >>> > > > > > > is a
>
> > > >> > >>> > > > > > > > > >> >> > relatively high estimate. However it
>
> > > depends
>
> > > >> > >>> > > > > > > > > >> >> > on
>
> > > >> > >>> the
>
> > > >> > >>> > > token
>
> > > >> > >>> > > > > > > > > >> expiration. I am
>
> > > >> > >>> > > > > > > > > >> >> > less worried about the extra load it
>
> > puts
>
> > > on
>
> > > >> > >>> > controller
>
> > > >> > >>> > > > vs
>
> > > >> > >>> > > > > > the
>
> > > >> > >>> > > > > > > > > added
>
> > > >> > >>> > > > > > > > > >> >> > complexity and the value it offers.
>
> > > >> > >>> > > > > > > > > >> >> >
>
> > > >> > >>> > > > > > > > > >> >> > Thanks
>
> > > >> > >>> > > > > > > > > >> >> > Parth
>
> > > >> > >>> > > > > > > > > >> >> >
>
> > > >> > >>> > > > > > > > > >> >> >
>
> > > >> > >>> > > > > > > > > >> >> >
>
> > > >> > >>> > > > > > > > > >> >> > On Tue, May 24, 2016 at 7:30 AM,
> Ismael
>
> > > Juma
>
> > > >> > >>> > > > > > > > > >> >> > <
>
> > > >> > >>> > > > > > > ismael@juma.me.uk>
>
> > > >> > >>> > > > > > > > > >> wrote:
>
> > > >> > >>> > > > > > > > > >> >> >
>
> > > >> > >>> > > > > > > > > >> >> > > Thanks Rajini. It would probably
>
> > > require a
>
> > > >> > >>> separate
>
> > > >> > >>> > > KIP
>
> > > >> > >>> > > > > as
>
> > > >> > >>> > > > > > it
>
> > > >> > >>> > > > > > > > > will
>
> > > >> > >>> > > > > > > > > >> >> > > introduce user visible changes. We
>
> > could
>
> > > >> > >>> > > > > > > > > >> >> > > also
>
> > > >> > >>> > update
>
> > > >> > >>> > > > > KIP-48
>
> > > >> > >>> > > > > > > to
>
> > > >> > >>> > > > > > > > > >> have this
>
> > > >> > >>> > > > > > > > > >> >> > > information, but it seems cleaner to
>
> > do
>
> > > it
>
> > > >> > >>> > > separately.
>
> > > >> > >>> > > > We
>
> > > >> > >>> > > > > > can
>
> > > >> > >>> > > > > > > > > >> discuss
>
> > > >> > >>> > > > > > > > > >> >> > that
>
> > > >> > >>> > > > > > > > > >> >> > > in the KIP call today.
>
> > > >> > >>> > > > > > > > > >> >> > >
>
> > > >> > >>> > > > > > > > > >> >> > > Ismael
>
> > > >> > >>> > > > > > > > > >> >> > >
>
> > > >> > >>> > > > > > > > > >> >> > > On Tue, May 24, 2016 at 3:19 PM,
>
> > Rajini
>
> > > >> > Sivaram
>
> > > >> > >>> <
>
> > > >> > >>> > > > > > > > > >> >> > > rajinisivaram@googlemail.com>
> wrote:
>
> > > >> > >>> > > > > > > > > >> >> > >
>
> > > >> > >>> > > > > > > > > >> >> > > > Ismael,
>
> > > >> > >>> > > > > > > > > >> >> > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > I have created a JIRA (
>
> > > >> > >>> > > > > > > > > >> >> > https://issues.apache.org/
>
> > > >> > jira/browse/KAFKA-3751)
>
> > > >> > >>> > > > > > > > > >> >> > > > for adding SCRAM as a SASL
>
> > mechanism.
>
> > > >> > >>> > > > > > > > > >> >> > > > Would
>
> > > >> > >>> that
>
> > > >> > >>> > > need
>
> > > >> > >>> > > > > > > another
>
> > > >> > >>> > > > > > > > > >> KIP? If
>
> > > >> > >>> > > > > > > > > >> >> > > > KIP-48 will use this mechanism,
> can
>
> > > this
>
> > > >> > just
>
> > > >> > >>> be
>
> > > >> > >>> > a
>
> > > >> > >>> > > > JIRA
>
> > > >> > >>> > > > > > > that
>
> > > >> > >>> > > > > > > > > gets
>
> > > >> > >>> > > > > > > > > >> >> > > reviewed
>
> > > >> > >>> > > > > > > > > >> >> > > > when the PR is ready?
>
> > > >> > >>> > > > > > > > > >> >> > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > Thank you,
>
> > > >> > >>> > > > > > > > > >> >> > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > Rajini
>
> > > >> > >>> > > > > > > > > >> >> > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM,
>
> > > Ismael
>
> > > >> > Juma <
>
> > > >> > >>> > > > > > > > > ismael@juma.me.uk>
>
> > > >> > >>> > > > > > > > > >> >> > wrote:
>
> > > >> > >>> > > > > > > > > >> >> > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > Thanks Rajini, SCRAM seems like
> a
>
> > > good
>
> > > >> > >>> > candidate.
>
> > > >> > >>> > > > > > > > > >> >> > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > Gwen had independently mentioned
>
> > > this
>
> > > >> > >>> > > > > > > > > >> >> > > > > as
>
> > > >> > a
>
> > > >> > >>> SASL
>
> > > >> > >>> > > > > > mechanism
>
> > > >> > >>> > > > > > > > > that
>
> > > >> > >>> > > > > > > > > >> might
>
> > > >> > >>> > > > > > > > > >> >> > be
>
> > > >> > >>> > > > > > > > > >> >> > > > > useful for Kafka and I have been
>
> > > >> > >>> > > > > > > > > >> >> > > > > meaning
>
> > > >> > to
>
> > > >> > >>> > check
>
> > > >> > >>> > > > it
>
> > > >> > >>> > > > > in
>
> > > >> > >>> > > > > > > > more
>
> > > >> > >>> > > > > > > > > >> detail.
>
> > > >> > >>> > > > > > > > > >> >> > > Good
>
> > > >> > >>> > > > > > > > > >> >> > > > > to know that you are willing to
>
> > > >> > contribute
>
> > > >> > >>> an
>
> > > >> > >>> > > > > > > > implementation.
>
> > > >> > >>> > > > > > > > > >> Maybe
>
> > > >> > >>> > > > > > > > > >> >> > we
>
> > > >> > >>> > > > > > > > > >> >> > > > > should file a separate JIRA for
>
> > > this?
>
> > > >> > >>> > > > > > > > > >> >> > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > Ismael
>
> > > >> > >>> > > > > > > > > >> >> > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM,
>
> > > >> > >>> > > > > > > > > >> >> > > > > Rajini
>
> > > >> > >>> > Sivaram <
>
> > > >> > >>> > > > > > > > > >> >> > > > > rajinisivaram@googlemail.com>
>
> > > wrote:
>
> > > >> > >>> > > > > > > > > >> >> > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > SCRAM (Salted Challenge
> Response
>
> > > >> > >>> > Authentication
>
> > > >> > >>> > > > > > > > Mechanism)
>
> > > >> > >>> > > > > > > > > >> is a
>
> > > >> > >>> > > > > > > > > >> >> > > better
>
> > > >> > >>> > > > > > > > > >> >> > > > > > mechanism than Digest-MD5.
> Java
>
> > > >> > >>> > > > > > > > > >> >> > > > > > doesn't
>
> > > >> > >>> come
>
> > > >> > >>> > > > with a
>
> > > >> > >>> > > > > > > > > built-in
>
> > > >> > >>> > > > > > > > > >> SCRAM
>
> > > >> > >>> > > > > > > > > >> >> > > > > > SaslServer or SaslClient, but
> I
>
> > > will
>
> > > >> > >>> > > > > > > > > >> >> > > > > > be
>
> > > >> > >>> happy
>
> > > >> > >>> > > to
>
> > > >> > >>> > > > > add
>
> > > >> > >>> > > > > > > > > support
>
> > > >> > >>> > > > > > > > > >> in
>
> > > >> > >>> > > > > > > > > >> >> > Kafka
>
> > > >> > >>> > > > > > > > > >> >> > > > > since
>
> > > >> > >>> > > > > > > > > >> >> > > > > > it would be a useful mechanism
>
> > to
>
> > > >> > support
>
> > > >> > >>> > > anyway.
>
> > > >> > >>> > > > > > > > > >> >> > > > > > https://tools.ietf.org/html/
>
> > > rfc7677
>
> > > >> > >>> > describes
>
> > > >> > >>> > > > the
>
> > > >> > >>> > > > > > > > protocol
>
> > > >> > >>> > > > > > > > > >> for
>
> > > >> > >>> > > > > > > > > >> >> > > > > > SCRAM-SHA-256.
>
> > > >> > >>> > > > > > > > > >> >> > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > On Tue, May 24, 2016 at 2:37
> AM,
>
> > > Jun
>
> > > >> > Rao <
>
> > > >> > >>> > > > > > > > jun@confluent.io
>
> > > >> > >>> > > > > > > > > >
>
> > > >> > >>> > > > > > > > > >> wrote:
>
> > > >> > >>> > > > > > > > > >> >> > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > Parth,
>
> > > >> > >>> > > > > > > > > >> >> > > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > Thanks for the explanation.
> A
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > couple
>
> > > >> > of
>
> > > >> > >>> > more
>
> > > >> > >>> > > > > > > questions.
>
> > > >> > >>> > > > > > > > > >> >> > > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > 110. What does
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > getDelegationTokenAs
>
> > > >> > >>> mean?
>
> > > >> > >>> > > > > > > > > >> >> > > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > 111. What's the typical rate
>
> > of
>
> > > >> > getting
>
> > > >> > >>> and
>
> > > >> > >>> > > > > > renewing
>
> > > >> > >>> > > > > > > > > >> delegation
>
> > > >> > >>> > > > > > > > > >> >> > > > tokens?
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > That may have an impact on
>
> > > whether
>
> > > >> > they
>
> > > >> > >>> > > should
>
> > > >> > >>> > > > be
>
> > > >> > >>> > > > > > > > > directed
>
> > > >> > >>> > > > > > > > > >> to the
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > controller.
>
> > > >> > >>> > > > > > > > > >> >> > > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > Jun
>
> > > >> > >>> > > > > > > > > >> >> > > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > On Mon, May 23, 2016 at 1:19
>
> > PM,
>
> > > >> > parth
>
> > > >> > >>> > > > > brahmbhatt <
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > brahmbhatt.parth@gmail.com>
>
> > > wrote:
>
> > > >> > >>> > > > > > > > > >> >> > > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > Hi Jun,
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > Thanks for reviewing.
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > * We could add a Cluster
>
> > > action
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > to
>
> > > >> > add
>
> > > >> > >>> > acls
>
> > > >> > >>> > > > on
>
> > > >> > >>> > > > > > who
>
> > > >> > >>> > > > > > > > can
>
> > > >> > >>> > > > > > > > > >> request
>
> > > >> > >>> > > > > > > > > >> >> > > > > > delegation
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > tokens. I don't see the
> use
>
> > > case
>
> > > >> > for
>
> > > >> > >>> that
>
> > > >> > >>> > > yet
>
> > > >> > >>> > > > > but
>
> > > >> > >>> > > > > > > > down
>
> > > >> > >>> > > > > > > > > >> the line
>
> > > >> > >>> > > > > > > > > >> >> > > > when
>
> > > >> > >>> > > > > > > > > >> >> > > > > we
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > start supporting
>
> > > >> > getDelegationTokenAs
>
> > > >> > >>> it
>
> > > >> > >>> > > will
>
> > > >> > >>> > > > > be
>
> > > >> > >>> > > > > > > > > >> necessary.
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > * Yes we recommend tokens
> to
>
> > > be
>
> > > >> > only
>
> > > >> > >>> > > > > > > used/distributed
>
> > > >> > >>> > > > > > > > > >> over
>
> > > >> > >>> > > > > > > > > >> >> > secure
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > channels.
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > * Depending on what design
>
> > we
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > end
>
> > > >> > up
>
> > > >> > >>> > > choosing
>
> > > >> > >>> > > > > > > > > >> Invalidation will
>
> > > >> > >>> > > > > > > > > >> >> > > be
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > responsibility of every
>
> > broker
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > or
>
> > > >> > >>> > > controller.
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > * I am not sure if I
>
> > > documented
>
> > > >> > >>> somewhere
>
> > > >> > >>> > > > that
>
> > > >> > >>> > > > > > > > > >> invalidation
>
> > > >> > >>> > > > > > > > > >> >> > will
>
> > > >> > >>> > > > > > > > > >> >> > > > > > directly
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > go through zookeeper but
>
> > that
>
> > > is
>
> > > >> > not
>
> > > >> > >>> the
>
> > > >> > >>> > > > > intent.
>
> > > >> > >>> > > > > > > > > >> Invalidation
>
> > > >> > >>> > > > > > > > > >> >> > > will
>
> > > >> > >>> > > > > > > > > >> >> > > > > > either
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > be request based or due to
>
> > > >> > >>> expiration. No
>
> > > >> > >>> > > > > direct
>
> > > >> > >>> > > > > > > > > >> zookeeper
>
> > > >> > >>> > > > > > > > > >> >> > > > > interaction
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > from
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > any client.
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > * "Broker also stores the
>
> > > >> > >>> DelegationToken
>
> > > >> > >>> > > > > without
>
> > > >> > >>> > > > > > > the
>
> > > >> > >>> > > > > > > > > >> hmac in
>
> > > >> > >>> > > > > > > > > >> >> > the
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > zookeeper." : Sorry about
>
> > the
>
> > > >> > >>> confusion.
>
> > > >> > >>> > > The
>
> > > >> > >>> > > > > sole
>
> > > >> > >>> > > > > > > > > >> purpose of
>
> > > >> > >>> > > > > > > > > >> >> > > > > zookeeper
>
> > > >> > >>> > > > > > > > > >> >> > > > > > in
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > this design is as
>
> > distribution
>
> > > >> > channel
>
> > > >> > >>> > for
>
> > > >> > >>> > > > > tokens
>
> > > >> > >>> > > > > > > > > >> between all
>
> > > >> > >>> > > > > > > > > >> >> > > > brokers
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > and a
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > layer that ensures only
>
> > tokens
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > that
>
> > > >> > >>> were
>
> > > >> > >>> > > > > > generated
>
> > > >> > >>> > > > > > > by
>
> > > >> > >>> > > > > > > > > >> making a
>
> > > >> > >>> > > > > > > > > >> >> > > > > request
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > to a
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > broker will be accepted
>
> > (more
>
> > > on
>
> > > >> > this
>
> > > >> > >>> in
>
> > > >> > >>> > > > second
>
> > > >> > >>> > > > > > > > > >> paragraph). The
>
> > > >> > >>> > > > > > > > > >> >> > > > token
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > consists of few elements
>
> > > (owner,
>
> > > >> > >>> renewer,
>
> > > >> > >>> > > > uuid
>
> > > >> > >>> > > > > ,
>
> > > >> > >>> > > > > > > > > >> expiration,
>
> > > >> > >>> > > > > > > > > >> >> > > hmac)
>
> > > >> > >>> > > > > > > > > >> >> > > > ,
>
> > > >> > >>> > > > > > > > > >> >> > > > > > one
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > of
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > which is the finally
>
> > generated
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > hmac
>
> > > >> > >>> but
>
> > > >> > >>> > > hmac
>
> > > >> > >>> > > > it
>
> > > >> > >>> > > > > > > self
>
> > > >> > >>> > > > > > > > is
>
> > > >> > >>> > > > > > > > > >> >> > derivable
>
> > > >> > >>> > > > > > > > > >> >> > > > if
>
> > > >> > >>> > > > > > > > > >> >> > > > > > you
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > have all the other
> elements
>
> > of
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > the
>
> > > >> > >>> token
>
> > > >> > >>> > +
>
> > > >> > >>> > > > > secret
>
> > > >> > >>> > > > > > > key
>
> > > >> > >>> > > > > > > > > to
>
> > > >> > >>> > > > > > > > > >> >> > generate
>
> > > >> > >>> > > > > > > > > >> >> > > > > hmac.
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > Given zookeeper does not
>
> > > provide
>
> > > >> > SSL
>
> > > >> > >>> > > support
>
> > > >> > >>> > > > we
>
> > > >> > >>> > > > > > do
>
> > > >> > >>> > > > > > > > not
>
> > > >> > >>> > > > > > > > > >> want the
>
> > > >> > >>> > > > > > > > > >> >> > > > > entire
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > token to be wire
> transferred
>
> > > to
>
> > > >> > >>> zookeeper
>
> > > >> > >>> > > as
>
> > > >> > >>> > > > > that
>
> > > >> > >>> > > > > > > > will
>
> > > >> > >>> > > > > > > > > >> be an
>
> > > >> > >>> > > > > > > > > >> >> > > > insecure
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > wire
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > transfer. Instead we only
>
> > > store
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > all
>
> > > >> > >>> the
>
> > > >> > >>> > > other
>
> > > >> > >>> > > > > > > > elements
>
> > > >> > >>> > > > > > > > > >> of a
>
> > > >> > >>> > > > > > > > > >> >> > > > > delegation
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > tokens. Brokers can read
>
> > these
>
> > > >> > >>> elements
>
> > > >> > >>> > and
>
> > > >> > >>> > > > > > because
>
> > > >> > >>> > > > > > > > > they
>
> > > >> > >>> > > > > > > > > >> also
>
> > > >> > >>> > > > > > > > > >> >> > > have
>
> > > >> > >>> > > > > > > > > >> >> > > > > > access
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > to secret key they will be
>
> > > able
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > to
>
> > > >> > >>> > generate
>
> > > >> > >>> > > > > hmac
>
> > > >> > >>> > > > > > on
>
> > > >> > >>> > > > > > > > > >> their end.
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > One of the alternative
>
> > > proposed
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > is
>
> > > >> > to
>
> > > >> > >>> > avoid
>
> > > >> > >>> > > > > > > zookeeper
>
> > > >> > >>> > > > > > > > > >> >> > > altogether. A
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > Client
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > will call broker with
>
> > required
>
> > > >> > >>> > information
>
> > > >> > >>> > > > > > (owner,
>
> > > >> > >>> > > > > > > > > >> renwer,
>
> > > >> > >>> > > > > > > > > >> >> > > > > expiration)
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > and
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > get back (signed hmac,
>
> > uuid).
>
> > > >> > Broker
>
> > > >> > >>> > won't
>
> > > >> > >>> > > > > store
>
> > > >> > >>> > > > > > > this
>
> > > >> > >>> > > > > > > > > in
>
> > > >> > >>> > > > > > > > > >> >> > > zookeeper.
>
> > > >> > >>> > > > > > > > > >> >> > > > > > From
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > this point a client can
>
> > > contact
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > any
>
> > > >> > >>> > broker
>
> > > >> > >>> > > > with
>
> > > >> > >>> > > > > > all
>
> > > >> > >>> > > > > > > > the
>
> > > >> > >>> > > > > > > > > >> >> > > delegation
>
> > > >> > >>> > > > > > > > > >> >> > > > > > token
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > info (owner, rewner,
>
> > > expiration,
>
> > > >> > hmac,
>
> > > >> > >>> > > uuid)
>
> > > >> > >>> > > > > the
>
> > > >> > >>> > > > > > > > borker
>
> > > >> > >>> > > > > > > > > >> will
>
> > > >> > >>> > > > > > > > > >> >> > > > > regenerate
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > the
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > hmac and as long as it
>
> > matches
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > with
>
> > > >> > >>> hmac
>
> > > >> > >>> > > > > > presented
>
> > > >> > >>> > > > > > > by
>
> > > >> > >>> > > > > > > > > >> client ,
>
> > > >> > >>> > > > > > > > > >> >> > > > broker
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > will
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > allow the request to
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > authenticate.
>
> > > >> > >>> Only
>
> > > >> > >>> > > > > problem
>
> > > >> > >>> > > > > > > with
>
> > > >> > >>> > > > > > > > > >> this
>
> > > >> > >>> > > > > > > > > >> >> > > approach
>
> > > >> > >>> > > > > > > > > >> >> > > > > is
>
> > > >> > >>> > > > > > > > > >> >> > > > > > if
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > the secret key is
>
> > compromised
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > any
>
> > > >> > >>> client
>
> > > >> > >>> > > can
>
> > > >> > >>> > > > > now
>
> > > >> > >>> > > > > > > > > generate
>
> > > >> > >>> > > > > > > > > >> >> > random
>
> > > >> > >>> > > > > > > > > >> >> > > > > tokens
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > and
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > they will still be able to
>
> > > >> > >>> authenticate
>
> > > >> > >>> > as
>
> > > >> > >>> > > > any
>
> > > >> > >>> > > > > > user
>
> > > >> > >>> > > > > > > > > they
>
> > > >> > >>> > > > > > > > > >> like.
>
> > > >> > >>> > > > > > > > > >> >> > > with
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > zookeeper we guarantee
> that
>
> > > only
>
> > > >> > >>> tokens
>
> > > >> > >>> > > > > acquired
>
> > > >> > >>> > > > > > > via
>
> > > >> > >>> > > > > > > > a
>
> > > >> > >>> > > > > > > > > >> broker
>
> > > >> > >>> > > > > > > > > >> >> > > > (using
>
> > > >> > >>> > > > > > > > > >> >> > > > > > some
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > auth scheme other than
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > delegation
>
> > > >> > >>> token)
>
> > > >> > >>> > > will
>
> > > >> > >>> > > > > be
>
> > > >> > >>> > > > > > > > > >> accepted. We
>
> > > >> > >>> > > > > > > > > >> >> > > need
>
> > > >> > >>> > > > > > > > > >> >> > > > to
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > discuss which proposal
> makes
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > more
>
> > > >> > >>> sense
>
> > > >> > >>> > and
>
> > > >> > >>> > > > we
>
> > > >> > >>> > > > > > can
>
> > > >> > >>> > > > > > > go
>
> > > >> > >>> > > > > > > > > >> over it
>
> > > >> > >>> > > > > > > > > >> >> > in
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > tomorrow's
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > meeting.
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > Also, can you forward the
>
> > > invite
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > to
>
> > > >> > >>> me?
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > Thanks
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > Parth
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > On Mon, May 23, 2016 at
>
> > 10:35
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > AM,
>
> > > >> > Jun
>
> > > >> > >>> > Rao <
>
> > > >> > >>> > > > > > > > > >> jun@confluent.io>
>
> > > >> > >>> > > > > > > > > >> >> > > > wrote:
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > > Thanks for the KIP. A
> few
>
> > > >> > comments.
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > > 100. This potentially
> can
>
> > be
>
> > > >> > useful
>
> > > >> > >>> for
>
> > > >> > >>> > > > Kafka
>
> > > >> > >>> > > > > > > > Connect
>
> > > >> > >>> > > > > > > > > >> and
>
> > > >> > >>> > > > > > > > > >> >> > Kafka
>
> > > >> > >>> > > > > > > > > >> >> > > > > rest
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > proxy
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > > where a worker agent
> will
>
> > > need
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > > to
>
> > > >> > >>> run a
>
> > > >> > >>> > > > task
>
> > > >> > >>> > > > > on
>
> > > >> > >>> > > > > > > > > behalf
>
> > > >> > >>> > > > > > > > > >> of a
>
> > > >> > >>> > > > > > > > > >> >> > > > client.
>
> > > >> > >>> > > > > > > > > >> >> > > > > > We
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > will
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > > likely need to change
> how
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > > those
>
> > > >> > >>> > services
>
> > > >> > >>> > > > use
>
> > > >> > >>> > > > > > > Kafka
>
> > > >> > >>> > > > > > > > > >> clients
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > > (producer/consumer).
>
> > Instead
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > > of a
>
> > > >> > >>> > shared
>
> > > >> > >>> > > > > client
>
> > > >> > >>> > > > > > > per
>
> > > >> > >>> > > > > > > > > >> worker,
>
> > > >> > >>> > > > > > > > > >> >> > we
>
> > > >> > >>> > > > > > > > > >> >> > > > will
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > need
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > a
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > > client per user task
> since
>
> > > the
>
> > > >> > >>> > > > authentication
>
> > > >> > >>> > > > > > > > happens
>
> > > >> > >>> > > > > > > > > >> at the
>
> > > >> > >>> > > > > > > > > >> >> > > > > > connection
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > > level. For Kafka
> Connect,
>
> > > the
>
> > > >> > >>> renewer
>
> > > >> > >>> > > will
>
> > > >> > >>> > > > be
>
> > > >> > >>> > > > > > the
>
> > > >> > >>> > > > > > > > > >> workers.
>
> > > >> > >>> > > > > > > > > >> >> > So,
>
> > > >> > >>> > > > > > > > > >> >> > > we
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > probably
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > > need to allow multiple
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > > renewers.
>
> > > >> > For
>
> > > >> > >>> > > Kafka
>
> > > >> > >>> > > > > rest
>
> > > >> > >>> > > > > > > > > proxy,
>
> > > >> > >>> > > > > > > > > >> the
>
> > > >> > >>> > > > > > > > > >> >> > > > renewer
>
> > > >> > >>> > > > > > > > > >> >> > > > > > can
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > > probably just be the
>
> > creator
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > > of
>
> > > >> > the
>
> > > >> > >>> > > token.
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > > 101. Do we need new acl
> on
>
> > > who
>
> > > >> > can
>
> > > >> > >>> > > request
>
> > > >> > >>> > > > > > > > delegation
>
> > > >> > >>> > > > > > > > > >> tokens?
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > >
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > > 102. Do we recommend
>
> > people
>
> > > to
>
> > > >> > send
>
> > > >> > >>> > > > > delegation
>
> > > >> > >>> > > > > > > > tokens
>
> > > >> > >>> > > > > > > > > >> in an
>
> > > >> > >>> > > > > > > > > >> >> > > > > encrypted
>
> > > >> > >>> > > > > > > > > >> >> > > > > > > > > channel?
>
> > > >> > >>>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Manikumar <ma...@gmail.com>]

Hi,

I would like to reinitiate the discussion on Delegation token support for
Kafka.

Brief summary of the past discussion:

1) Broker stores delegation tokens in zookeeper.  All brokers will have a
cache backed by
   zookeeper so they will all get notified whenever a new token is
generated and they will
   update their local cache whenever token state changes.
2) The current proposal does not support rotation of secret
3) Only allow the renewal by users that authenticated using *non*
delegation token mechanism
4) KIP-84 proposes to support  SASL SCRAM mechanisms. Kafka clients can
authenticate using
   SCRAM-SHA-256, providing the delegation token HMAC as password.

Updated the KIP with the following:
1. Protocol and Config changes
2. format of the data stored in ZK.
3. Changes to Java Clients/Usage of SASL SCRAM mechanism

https://cwiki.apache.org/confluence/display/KAFKA/KIP-
48+Delegation+token+support+for+Kafka


Jun, Ashish, Gwen,

Pl review the updated KIP.

Thanks
Manikumar


On Thu, Sep 29, 2016 at 9:56 PM, Ashish Singh <as...@cloudera.com> wrote:

> Harsha/ Gwen,
>
> How do we proceed here? I am willing to help out with here.
>
> On Fri, Sep 23, 2016 at 11:41 AM, Gwen Shapira <gw...@confluent.io> wrote:
>
> > Is it updated? are all concerns addressed? do you want to start a vote?
> >
> > Sorry for being pushy, I do appreciate that we are all volunteers and
> > finding time is difficult. This feature is important for anything that
> > integrates with Kafka (stream processors, Flume, NiFi, etc) and I
> > don't want to see this getting stuck because we lack coordination
> > within the community.
> >
> > On Thu, Sep 15, 2016 at 6:39 PM, Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> > > The only pending update for the KIP is to write up the protocol changes
> > like
> > > we've it KIP-4. I'll update the wiki.
> > >
> > >
> > > On Thu, Sep 15, 2016 at 4:27 PM Ashish Singh <as...@cloudera.com>
> > wrote:
> > >>
> > >> I think we decided to not support secret rotation, I guess this can be
> > >> stated clearly on the KIP. Also, more details on how clients will
> > perform
> > >> token distribution and how CLI will look like will be helpful.
> > >>
> > >> On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <gw...@confluent.io>
> > wrote:
> > >>
> > >> > Hi Guys,
> > >> >
> > >> > This discussion was dead for a while. Are there still contentious
> > >> > points? If not, why are there no votes?
> > >> >
> > >> > On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <ju...@confluent.io> wrote:
> > >> > > Ashish,
> > >> > >
> > >> > > Yes, I will send out a KIP invite for next week to discuss KIP-48
> > and
> > >> > other
> > >> > > remaining KIPs.
> > >> > >
> > >> > > Thanks,
> > >> > >
> > >> > > Jun
> > >> > >
> > >> > > On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh <
> asingh@cloudera.com>
> > >> > wrote:
> > >> > >
> > >> > >> Thanks Harsha!
> > >> > >>
> > >> > >> Jun, can we add KIP-48 to next KIP hangout's agenda. Also, we did
> > not
> > >> > >> actually make a call on when we should have next KIP call. As
> there
> > >> > >> are
> > >> > a
> > >> > >> few outstanding KIPs that could not be discussed this week, can
> we
> > >> > >> have
> > >> > a
> > >> > >> KIP hangout call next week?
> > >> > >>
> > >> > >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha Chintalapani
> > >> > >> <ka...@harsha.io>
> > >> > >> wrote:
> > >> > >>
> > >> > >>> Ashish,
> > >> > >>>         Yes we are working on it. Lets discuss in the next KIP
> > >> > >>> meeting.
> > >> > >>> I'll join.
> > >> > >>> -Harsha
> > >> > >>>
> > >> > >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh <
> > asingh@cloudera.com>
> > >> > >>> wrote:
> > >> > >>>
> > >> > >>> > Hello Harsha,
> > >> > >>> >
> > >> > >>> > Are you still working on this? Wondering if we can discuss
> this
> > in
> > >> > next
> > >> > >>> KIP
> > >> > >>> > meeting, if you can join.
> > >> > >>> >
> > >> > >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha Chintalapani <
> > >> > kafka@harsha.io>
> > >> > >>> > wrote:
> > >> > >>> >
> > >> > >>> > > Hi Grant,
> > >> > >>> > >           We are working on it. Will add the details to KIP
> > >> > >>> > > about
> > >> > the
> > >> > >>> > > request protocol.
> > >> > >>> > >
> > >> > >>> > > Thanks,
> > >> > >>> > > Harsha
> > >> > >>> > >
> > >> > >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant Henke
> > >> > >>> > > <gh...@cloudera.com>
> > >> > >>> wrote:
> > >> > >>> > >
> > >> > >>> > > > Hi Parth,
> > >> > >>> > > >
> > >> > >>> > > > Are you still working on this? If you need any help please
> > >> > >>> > > > don't
> > >> > >>> > hesitate
> > >> > >>> > > > to ask.
> > >> > >>> > > >
> > >> > >>> > > > Thanks,
> > >> > >>> > > > Grant
> > >> > >>> > > >
> > >> > >>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <
> jun@confluent.io>
> > >> > wrote:
> > >> > >>> > > >
> > >> > >>> > > > > Parth,
> > >> > >>> > > > >
> > >> > >>> > > > > Thanks for the reply.
> > >> > >>> > > > >
> > >> > >>> > > > > It makes sense to only allow the renewal by users that
> > >> > >>> authenticated
> > >> > >>> > > > using
> > >> > >>> > > > > *non* delegation token mechanism. Then, should we make
> the
> > >> > >>> renewal a
> > >> > >>> > > > list?
> > >> > >>> > > > > For example, in the case of rest proxy, it will be
> useful
> > >> > >>> > > > > for
> > >> > >>> every
> > >> > >>> > > > > instance of rest proxy to be able to renew the tokens.
> > >> > >>> > > > >
> > >> > >>> > > > > It would be clearer if we can document the request
> > protocol
> > >> > like
> > >> > >>> > > > >
> > >> > >>> > > > >
> > >> > >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >> > >>> > >
> > >> > >>> > > 4+-+Command+line+and+centralized+administrative+
> > operations#KIP-4-
> > >> > >>> > > Commandlineandcentralizedadministrativeoperations-
> > >> > >>> > > CreateTopicsRequest(KAFKA-2945):(VotedandPlannedforin0.
> > 10.1.0)
> > >> > >>> > > > > .
> > >> > >>> > > > >
> > >> > >>> > > > > It would also be useful to document the client APIs.
> > >> > >>> > > > >
> > >> > >>> > > > > Thanks,
> > >> > >>> > > > >
> > >> > >>> > > > > Jun
> > >> > >>> > > > >
> > >> > >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM, parth brahmbhatt <
> > >> > >>> > > > > brahmbhatt.parth@gmail.com> wrote:
> > >> > >>> > > > >
> > >> > >>> > > > > > Hi,
> > >> > >>> > > > > >
> > >> > >>> > > > > > I am suggesting that we will only allow the renewal by
> > >> > >>> > > > > > users
> > >> > >>> that
> > >> > >>> > > > > > authenticated using *non* delegation token mechanism.
> > For
> > >> > >>> example,
> > >> > >>> > If
> > >> > >>> > > > > user
> > >> > >>> > > > > > Alice authenticated using kerberos and requested
> > >> > >>> > > > > > delegation
> > >> > >>> tokens,
> > >> > >>> > > > only
> > >> > >>> > > > > > user Alice authenticated via non delegation token
> > >> > >>> > > > > > mechanism
> > >> > can
> > >> > >>> > > renew.
> > >> > >>> > > > > > Clients that have  access to delegation tokens can not
> > >> > >>> > > > > > issue
> > >> > >>> > renewal
> > >> > >>> > > > > > request for renewing their own token and this is
> > primarily
> > >> > >>> > important
> > >> > >>> > > to
> > >> > >>> > > > > > reduce the time window for which a compromised token
> > will
> > >> > >>> > > > > > be
> > >> > >>> valid.
> > >> > >>> > > > > >
> > >> > >>> > > > > > To clarify, Yes any authenticated user can request
> > >> > >>> > > > > > delegation
> > >> > >>> > tokens
> > >> > >>> > > > but
> > >> > >>> > > > > > even here I would recommend to avoid creating a chain
> > >> > >>> > > > > > where a
> > >> > >>> > client
> > >> > >>> > > > > > authenticated via delegation token request for more
> > >> > delegation
> > >> > >>> > > tokens.
> > >> > >>> > > > > > Basically anyone can request delegation token, as long
> > as
> > >> > they
> > >> > >>> > > > > authenticate
> > >> > >>> > > > > > via a non delegation token mechanism.
> > >> > >>> > > > > >
> > >> > >>> > > > > > Aren't classes listed here
> > >> > >>> > > > > > <
> > >> > >>> > > > > >
> > >> > >>> > > > >
> > >> > >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >> > >>> > > 48+Delegation+token+support+fo
> r+Kafka#KIP-48Delegationtokens
> > >> > >>> upportforKaf
> > >> > >>> > > ka-PublicInterfaces
> > >> > >>> > > > > > >
> > >> > >>> > > > > > sufficient?
> > >> > >>> > > > > >
> > >> > >>> > > > > > Thanks
> > >> > >>> > > > > > Parth
> > >> > >>> > > > > >
> > >> > >>> > > > > >
> > >> > >>> > > > > >
> > >> > >>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao
> > >> > >>> > > > > > <ju...@confluent.io>
> > >> > >>> wrote:
> > >> > >>> > > > > >
> > >> > >>> > > > > > > Parth,
> > >> > >>> > > > > > >
> > >> > >>> > > > > > > Thanks for the reply. A couple of comments inline
> > below.
> > >> > >>> > > > > > >
> > >> > >>> > > > > > > On Tue, Jun 21, 2016 at 10:36 AM, parth brahmbhatt <
> > >> > >>> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > >> > >>> > > > > > >
> > >> > >>> > > > > > > > 1. Who / how are tokens renewed? By original
> > requester
> > >> > >>> only? or
> > >> > >>> > > > using
> > >> > >>> > > > > > > > Kerberos
> > >> > >>> > > > > > > > auth only?
> > >> > >>> > > > > > > > My recommendation is to do this only using
> Kerberos
> > >> > >>> > > > > > > > auth
> > >> > and
> > >> > >>> > only
> > >> > >>> > > > > threw
> > >> > >>> > > > > > > the
> > >> > >>> > > > > > > > renewer specified during the acquisition request.
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > Hmm, not sure that I follow this. Are you saying
> that
> > >> > >>> > > > > > > any
> > >> > >>> client
> > >> > >>> > > > > > > authenticated with the delegation token can renew,
> > i.e.
> > >> > there
> > >> > >>> is
> > >> > >>> > no
> > >> > >>> > > > > > renewer
> > >> > >>> > > > > > > needed?
> > >> > >>> > > > > > >
> > >> > >>> > > > > > > Also, just to be clear, any authenticated client
> > (either
> > >> > >>> through
> > >> > >>> > > SASL
> > >> > >>> > > > > or
> > >> > >>> > > > > > > SSL) can request a delegation token for the
> > >> > >>> > > > > > > authenticated
> > >> > >>> user,
> > >> > >>> > > > right?
> > >> > >>> > > > > > >
> > >> > >>> > > > > > >
> > >> > >>> > > > > > > > 2. Are tokens stored on each broker or in ZK?
> > >> > >>> > > > > > > > My recommendation is still to store in ZK or not
> > store
> > >> > them
> > >> > >>> at
> > >> > >>> > > all.
> > >> > >>> > > > > The
> > >> > >>> > > > > > > > whole controller based distribution is too much
> > >> > >>> > > > > > > > overhead
> > >> > >>> with
> > >> > >>> > not
> > >> > >>> > > > > much
> > >> > >>> > > > > > to
> > >> > >>> > > > > > > > achieve.
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > 3. How are tokens invalidated / expired?
> > >> > >>> > > > > > > > Either by expiration time out or through an
> explicit
> > >> > >>> request to
> > >> > >>> > > > > > > invalidate.
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > 4. Which encryption algorithm is used?
> > >> > >>> > > > > > > > SCRAM
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > 5. What is the impersonation proposal (it wasn't
> in
> > >> > >>> > > > > > > > the
> > >> > KIP
> > >> > >>> but
> > >> > >>> > > was
> > >> > >>> > > > > > > > discussed
> > >> > >>> > > > > > > > in this thread)?
> > >> > >>> > > > > > > > There is no imperonation proposal. I tried and
> > >> > >>> > > > > > > > explained
> > >> > how
> > >> > >>> > its
> > >> > >>> > > a
> > >> > >>> > > > > > > > different problem and why its not really necessary
> > to
> > >> > >>> discuss
> > >> > >>> > > that
> > >> > >>> > > > as
> > >> > >>> > > > > > > part
> > >> > >>> > > > > > > > of this KIP.  This KIP will not support any
> > >> > impersonation,
> > >> > >>> it
> > >> > >>> > > will
> > >> > >>> > > > > just
> > >> > >>> > > > > > > be
> > >> > >>> > > > > > > > another way to authenticate.
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > 6. Do we need new ACLs, if so - for what actions?
> > >> > >>> > > > > > > > We do not need new ACLs.
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > Could we document the format of the new
> > request/response
> > >> > and
> > >> > >>> > their
> > >> > >>> > > > > > > associated Resource and Operation for ACL?
> > >> > >>> > > > > > >
> > >> > >>> > > > > > >
> > >> > >>> > > > > > > > 7. How would the delegation token be configured in
> > the
> > >> > >>> client?
> > >> > >>> > > > > > > > Should be through config. I wasn't planning on
> > >> > >>> > > > > > > > supporting
> > >> > >>> JAAS
> > >> > >>> > > for
> > >> > >>> > > > > > > tokens.
> > >> > >>> > > > > > > > I don't believe hadoop does this either.
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > Thanks
> > >> > >>> > > > > > > > Parth
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <
> > >> > jun@confluent.io>
> > >> > >>> > > wrote:
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > > Harsha,
> > >> > >>> > > > > > > > >
> > >> > >>> > > > > > > > > Another question.
> > >> > >>> > > > > > > > >
> > >> > >>> > > > > > > > > 9. How would the delegation token be configured
> in
> > >> > >>> > > > > > > > > the
> > >> > >>> > client?
> > >> > >>> > > > The
> > >> > >>> > > > > > > > standard
> > >> > >>> > > > > > > > > way is to do this through JAAS. However, we will
> > >> > >>> > > > > > > > > need
> > >> > to
> > >> > >>> > think
> > >> > >>> > > > > > through
> > >> > >>> > > > > > > if
> > >> > >>> > > > > > > > > this is convenient in a shared environment. For
> > >> > example,
> > >> > >>> > when a
> > >> > >>> > > > new
> > >> > >>> > > > > > > task
> > >> > >>> > > > > > > > is
> > >> > >>> > > > > > > > > added to a Storm worker node, do we need to
> > >> > >>> > > > > > > > > dynamically
> > >> > >>> add a
> > >> > >>> > > new
> > >> > >>> > > > > > > section
> > >> > >>> > > > > > > > > in the JAAS file? It may be more convenient if
> we
> > >> > >>> > > > > > > > > can
> > >> > >>> pass in
> > >> > >>> > > the
> > >> > >>> > > > > > token
> > >> > >>> > > > > > > > > through the config directly w/o going through
> > JAAS.
> > >> > >>> > > > > > > > >
> > >> > >>> > > > > > > > > Are you or Parth still actively working on this
> > KIP?
> > >> > >>> > > > > > > > >
> > >> > >>> > > > > > > > > Thanks,
> > >> > >>> > > > > > > > >
> > >> > >>> > > > > > > > > Jun
> > >> > >>> > > > > > > > >
> > >> > >>> > > > > > > > >
> > >> > >>> > > > > > > > >
> > >> > >>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <
> > >> > >>> jun@confluent.io>
> > >> > >>> > > > wrote:
> > >> > >>> > > > > > > > >
> > >> > >>> > > > > > > > > > Just to add on that list.
> > >> > >>> > > > > > > > > >
> > >> > >>> > > > > > > > > > 2. It would be good to document the format of
> > the
> > >> > data
> > >> > >>> > stored
> > >> > >>> > > > in
> > >> > >>> > > > > > ZK.
> > >> > >>> > > > > > > > > > 7. Earlier, there was a discussion on whether
> > the
> > >> > tokens
> > >> > >>> > > should
> > >> > >>> > > > > be
> > >> > >>> > > > > > > > > > propagated through ZK like config/acl/quota,
> or
> > >> > through
> > >> > >>> the
> > >> > >>> > > > > > > controller.
> > >> > >>> > > > > > > > > > Currently, the controller is only designed for
> > >> > >>> propagating
> > >> > >>> > > > topic
> > >> > >>> > > > > > > > > metadata,
> > >> > >>> > > > > > > > > > but not other data.
> > >> > >>> > > > > > > > > > 8. Should we use SCRAM to send the token
> instead
> > >> > >>> > > > > > > > > > of
> > >> > >>> > > DIGEST-MD5
> > >> > >>> > > > > > since
> > >> > >>> > > > > > > > it's
> > >> > >>> > > > > > > > > > deprecated?
> > >> > >>> > > > > > > > > >
> > >> > >>> > > > > > > > > > Also, the images in the wiki seem broken.
> > >> > >>> > > > > > > > > >
> > >> > >>> > > > > > > > > > Thanks,
> > >> > >>> > > > > > > > > >
> > >> > >>> > > > > > > > > > Jun
> > >> > >>> > > > > > > > > >
> > >> > >>> > > > > > > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen
> Shapira <
> > >> > >>> > > > > gwen@confluent.io>
> > >> > >>> > > > > > > > > wrote:
> > >> > >>> > > > > > > > > >
> > >> > >>> > > > > > > > > >> From what I can see, remaining questions are:
> > >> > >>> > > > > > > > > >>
> > >> > >>> > > > > > > > > >> 1. Who / how are tokens renewed? By original
> > >> > requester
> > >> > >>> > only?
> > >> > >>> > > > or
> > >> > >>> > > > > > > using
> > >> > >>> > > > > > > > > >> Kerberos auth only?
> > >> > >>> > > > > > > > > >> 2. Are tokens stored on each broker or in ZK?
> > >> > >>> > > > > > > > > >> 3. How are tokens invalidated / expired?
> > >> > >>> > > > > > > > > >> 4. Which encryption algorithm is used?
> > >> > >>> > > > > > > > > >> 5. What is the impersonation proposal (it
> > wasn't
> > >> > >>> > > > > > > > > >> in
> > >> > the
> > >> > >>> > KIP
> > >> > >>> > > > but
> > >> > >>> > > > > > was
> > >> > >>> > > > > > > > > >> discussed in this thread)?
> > >> > >>> > > > > > > > > >> 6. Do we need new ACLs, if so - for what
> > actions?
> > >> > >>> > > > > > > > > >>
> > >> > >>> > > > > > > > > >> Gwen
> > >> > >>> > > > > > > > > >>
> > >> > >>> > > > > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <
> > >> > >>> kafka@harsha.io>
> > >> > >>> > > > wrote:
> > >> > >>> > > > > > > > > >> > Jun & Ismael,
> > >> > >>> > > > > > > > > >> >                          Unfortunately I
> > >> > >>> > > > > > > > > >> > couldn't
> > >> > >>> attend
> > >> > >>> > > the
> > >> > >>> > > > > KIP
> > >> > >>> > > > > > > > > meeting
> > >> > >>> > > > > > > > > >> >                          when delegation
> > tokens
> > >> > >>> > discussed.
> > >> > >>> > > > > > > > Appreciate
> > >> > >>> > > > > > > > > if
> > >> > >>> > > > > > > > > >> >                          you can update the
> > >> > thread if
> > >> > >>> > you
> > >> > >>> > > > have
> > >> > >>> > > > > > any
> > >> > >>> > > > > > > > > >> >                          further questions.
> > >> > >>> > > > > > > > > >> > Thanks,
> > >> > >>> > > > > > > > > >> > Harsha
> > >> > >>> > > > > > > > > >> >
> > >> > >>> > > > > > > > > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan
> Pei
> > >> > wrote:
> > >> > >>> > > > > > > > > >> >> It seems that the links to images in the
> KIP
> > >> > >>> > > > > > > > > >> >> are
> > >> > >>> > broken.
> > >> > >>> > > > > > > > > >> >>
> > >> > >>> > > > > > > > > >> >> Liquan
> > >> > >>> > > > > > > > > >> >>
> > >> > >>> > > > > > > > > >> >> On Tue, May 24, 2016 at 9:33 AM, parth
> > >> > brahmbhatt <
> > >> > >>> > > > > > > > > >> >> brahmbhatt.parth@gmail.com> wrote:
> > >> > >>> > > > > > > > > >> >>
> > >> > >>> > > > > > > > > >> >> > 110. What does getDelegationTokenAs
> mean?
> > >> > >>> > > > > > > > > >> >> > In the current proposal we only allow a
> > user
> > >> > >>> > > > > > > > > >> >> > to
> > >> > >>> get
> > >> > >>> > > > > > delegation
> > >> > >>> > > > > > > > > token
> > >> > >>> > > > > > > > > >> for
> > >> > >>> > > > > > > > > >> >> > the identity that it authenticated as
> > using
> > >> > >>> another
> > >> > >>> > > > > > mechanism,
> > >> > >>> > > > > > > > i.e.
> > >> > >>> > > > > > > > > >> A user
> > >> > >>> > > > > > > > > >> >> > that authenticate using a keytab for
> > >> > >>> > > > > > > > > >> >> > principal
> > >> > >>> > > > > > > user1@EXAMPLE.COM
> > >> > >>> > > > > > > > > >> will get
> > >> > >>> > > > > > > > > >> >> > delegation tokens for that user only. In
> > >> > future I
> > >> > >>> > think
> > >> > >>> > > > we
> > >> > >>> > > > > > will
> > >> > >>> > > > > > > > > have
> > >> > >>> > > > > > > > > >> to
> > >> > >>> > > > > > > > > >> >> > extend support such that we allow some
> set
> > >> > >>> > > > > > > > > >> >> > of
> > >> > >>> users (
> > >> > >>> > > > > > > > > >> >> > kafka-rest-user@EXAMPLE.COM,
> > >> > >>> > storm-nimbus@EXAMPLE.COM)
> > >> > >>> > > > to
> > >> > >>> > > > > > > > acquire
> > >> > >>> > > > > > > > > >> >> > delegation tokens on behalf of other
> users
> > >> > whose
> > >> > >>> > > identity
> > >> > >>> > > > > > they
> > >> > >>> > > > > > > > have
> > >> > >>> > > > > > > > > >> >> > verified independently.  Kafka brokers
> > will
> > >> > have
> > >> > >>> ACLs
> > >> > >>> > > to
> > >> > >>> > > > > > > control
> > >> > >>> > > > > > > > > >> which
> > >> > >>> > > > > > > > > >> >> > users are allowed to impersonate other
> > users
> > >> > and
> > >> > >>> get
> > >> > >>> > > > tokens
> > >> > >>> > > > > > on
> > >> > >>> > > > > > > > > >> behalf of
> > >> > >>> > > > > > > > > >> >> > them. Overall Impersonation is a whole
> > >> > different
> > >> > >>> > > problem
> > >> > >>> > > > in
> > >> > >>> > > > > > my
> > >> > >>> > > > > > > > > >> opinion and
> > >> > >>> > > > > > > > > >> >> > I think we can tackle it in separate
> KIP.
> > >> > >>> > > > > > > > > >> >> >
> > >> > >>> > > > > > > > > >> >> > 111. What's the typical rate of getting
> > and
> > >> > >>> renewing
> > >> > >>> > > > > > delegation
> > >> > >>> > > > > > > > > >> tokens?
> > >> > >>> > > > > > > > > >> >> > Typically this should be very very low,
> 1
> > >> > request
> > >> > >>> per
> > >> > >>> > > > > minute
> > >> > >>> > > > > > > is a
> > >> > >>> > > > > > > > > >> >> > relatively high estimate. However it
> > depends
> > >> > >>> > > > > > > > > >> >> > on
> > >> > >>> the
> > >> > >>> > > token
> > >> > >>> > > > > > > > > >> expiration. I am
> > >> > >>> > > > > > > > > >> >> > less worried about the extra load it
> puts
> > on
> > >> > >>> > controller
> > >> > >>> > > > vs
> > >> > >>> > > > > > the
> > >> > >>> > > > > > > > > added
> > >> > >>> > > > > > > > > >> >> > complexity and the value it offers.
> > >> > >>> > > > > > > > > >> >> >
> > >> > >>> > > > > > > > > >> >> > Thanks
> > >> > >>> > > > > > > > > >> >> > Parth
> > >> > >>> > > > > > > > > >> >> >
> > >> > >>> > > > > > > > > >> >> >
> > >> > >>> > > > > > > > > >> >> >
> > >> > >>> > > > > > > > > >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael
> > Juma
> > >> > >>> > > > > > > > > >> >> > <
> > >> > >>> > > > > > > ismael@juma.me.uk>
> > >> > >>> > > > > > > > > >> wrote:
> > >> > >>> > > > > > > > > >> >> >
> > >> > >>> > > > > > > > > >> >> > > Thanks Rajini. It would probably
> > require a
> > >> > >>> separate
> > >> > >>> > > KIP
> > >> > >>> > > > > as
> > >> > >>> > > > > > it
> > >> > >>> > > > > > > > > will
> > >> > >>> > > > > > > > > >> >> > > introduce user visible changes. We
> could
> > >> > >>> > > > > > > > > >> >> > > also
> > >> > >>> > update
> > >> > >>> > > > > KIP-48
> > >> > >>> > > > > > > to
> > >> > >>> > > > > > > > > >> have this
> > >> > >>> > > > > > > > > >> >> > > information, but it seems cleaner to
> do
> > it
> > >> > >>> > > separately.
> > >> > >>> > > > We
> > >> > >>> > > > > > can
> > >> > >>> > > > > > > > > >> discuss
> > >> > >>> > > > > > > > > >> >> > that
> > >> > >>> > > > > > > > > >> >> > > in the KIP call today.
> > >> > >>> > > > > > > > > >> >> > >
> > >> > >>> > > > > > > > > >> >> > > Ismael
> > >> > >>> > > > > > > > > >> >> > >
> > >> > >>> > > > > > > > > >> >> > > On Tue, May 24, 2016 at 3:19 PM,
> Rajini
> > >> > Sivaram
> > >> > >>> <
> > >> > >>> > > > > > > > > >> >> > > rajinisivaram@googlemail.com> wrote:
> > >> > >>> > > > > > > > > >> >> > >
> > >> > >>> > > > > > > > > >> >> > > > Ismael,
> > >> > >>> > > > > > > > > >> >> > > >
> > >> > >>> > > > > > > > > >> >> > > > I have created a JIRA (
> > >> > >>> > > > > > > > > >> >> > https://issues.apache.org/
> > >> > jira/browse/KAFKA-3751)
> > >> > >>> > > > > > > > > >> >> > > > for adding SCRAM as a SASL
> mechanism.
> > >> > >>> > > > > > > > > >> >> > > > Would
> > >> > >>> that
> > >> > >>> > > need
> > >> > >>> > > > > > > another
> > >> > >>> > > > > > > > > >> KIP? If
> > >> > >>> > > > > > > > > >> >> > > > KIP-48 will use this mechanism, can
> > this
> > >> > just
> > >> > >>> be
> > >> > >>> > a
> > >> > >>> > > > JIRA
> > >> > >>> > > > > > > that
> > >> > >>> > > > > > > > > gets
> > >> > >>> > > > > > > > > >> >> > > reviewed
> > >> > >>> > > > > > > > > >> >> > > > when the PR is ready?
> > >> > >>> > > > > > > > > >> >> > > >
> > >> > >>> > > > > > > > > >> >> > > > Thank you,
> > >> > >>> > > > > > > > > >> >> > > >
> > >> > >>> > > > > > > > > >> >> > > > Rajini
> > >> > >>> > > > > > > > > >> >> > > >
> > >> > >>> > > > > > > > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM,
> > Ismael
> > >> > Juma <
> > >> > >>> > > > > > > > > ismael@juma.me.uk>
> > >> > >>> > > > > > > > > >> >> > wrote:
> > >> > >>> > > > > > > > > >> >> > > >
> > >> > >>> > > > > > > > > >> >> > > > > Thanks Rajini, SCRAM seems like a
> > good
> > >> > >>> > candidate.
> > >> > >>> > > > > > > > > >> >> > > > >
> > >> > >>> > > > > > > > > >> >> > > > > Gwen had independently mentioned
> > this
> > >> > >>> > > > > > > > > >> >> > > > > as
> > >> > a
> > >> > >>> SASL
> > >> > >>> > > > > > mechanism
> > >> > >>> > > > > > > > > that
> > >> > >>> > > > > > > > > >> might
> > >> > >>> > > > > > > > > >> >> > be
> > >> > >>> > > > > > > > > >> >> > > > > useful for Kafka and I have been
> > >> > >>> > > > > > > > > >> >> > > > > meaning
> > >> > to
> > >> > >>> > check
> > >> > >>> > > > it
> > >> > >>> > > > > in
> > >> > >>> > > > > > > > more
> > >> > >>> > > > > > > > > >> detail.
> > >> > >>> > > > > > > > > >> >> > > Good
> > >> > >>> > > > > > > > > >> >> > > > > to know that you are willing to
> > >> > contribute
> > >> > >>> an
> > >> > >>> > > > > > > > implementation.
> > >> > >>> > > > > > > > > >> Maybe
> > >> > >>> > > > > > > > > >> >> > we
> > >> > >>> > > > > > > > > >> >> > > > > should file a separate JIRA for
> > this?
> > >> > >>> > > > > > > > > >> >> > > > >
> > >> > >>> > > > > > > > > >> >> > > > > Ismael
> > >> > >>> > > > > > > > > >> >> > > > >
> > >> > >>> > > > > > > > > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM,
> > >> > >>> > > > > > > > > >> >> > > > > Rajini
> > >> > >>> > Sivaram <
> > >> > >>> > > > > > > > > >> >> > > > > rajinisivaram@googlemail.com>
> > wrote:
> > >> > >>> > > > > > > > > >> >> > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > SCRAM (Salted Challenge Response
> > >> > >>> > Authentication
> > >> > >>> > > > > > > > Mechanism)
> > >> > >>> > > > > > > > > >> is a
> > >> > >>> > > > > > > > > >> >> > > better
> > >> > >>> > > > > > > > > >> >> > > > > > mechanism than Digest-MD5. Java
> > >> > >>> > > > > > > > > >> >> > > > > > doesn't
> > >> > >>> come
> > >> > >>> > > > with a
> > >> > >>> > > > > > > > > built-in
> > >> > >>> > > > > > > > > >> SCRAM
> > >> > >>> > > > > > > > > >> >> > > > > > SaslServer or SaslClient, but I
> > will
> > >> > >>> > > > > > > > > >> >> > > > > > be
> > >> > >>> happy
> > >> > >>> > > to
> > >> > >>> > > > > add
> > >> > >>> > > > > > > > > support
> > >> > >>> > > > > > > > > >> in
> > >> > >>> > > > > > > > > >> >> > Kafka
> > >> > >>> > > > > > > > > >> >> > > > > since
> > >> > >>> > > > > > > > > >> >> > > > > > it would be a useful mechanism
> to
> > >> > support
> > >> > >>> > > anyway.
> > >> > >>> > > > > > > > > >> >> > > > > > https://tools.ietf.org/html/
> > rfc7677
> > >> > >>> > describes
> > >> > >>> > > > the
> > >> > >>> > > > > > > > protocol
> > >> > >>> > > > > > > > > >> for
> > >> > >>> > > > > > > > > >> >> > > > > > SCRAM-SHA-256.
> > >> > >>> > > > > > > > > >> >> > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM,
> > Jun
> > >> > Rao <
> > >> > >>> > > > > > > > jun@confluent.io
> > >> > >>> > > > > > > > > >
> > >> > >>> > > > > > > > > >> wrote:
> > >> > >>> > > > > > > > > >> >> > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > Parth,
> > >> > >>> > > > > > > > > >> >> > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > Thanks for the explanation. A
> > >> > >>> > > > > > > > > >> >> > > > > > > couple
> > >> > of
> > >> > >>> > more
> > >> > >>> > > > > > > questions.
> > >> > >>> > > > > > > > > >> >> > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > 110. What does
> > >> > >>> > > > > > > > > >> >> > > > > > > getDelegationTokenAs
> > >> > >>> mean?
> > >> > >>> > > > > > > > > >> >> > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > 111. What's the typical rate
> of
> > >> > getting
> > >> > >>> and
> > >> > >>> > > > > > renewing
> > >> > >>> > > > > > > > > >> delegation
> > >> > >>> > > > > > > > > >> >> > > > tokens?
> > >> > >>> > > > > > > > > >> >> > > > > > > That may have an impact on
> > whether
> > >> > they
> > >> > >>> > > should
> > >> > >>> > > > be
> > >> > >>> > > > > > > > > directed
> > >> > >>> > > > > > > > > >> to the
> > >> > >>> > > > > > > > > >> >> > > > > > > controller.
> > >> > >>> > > > > > > > > >> >> > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > Jun
> > >> > >>> > > > > > > > > >> >> > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > On Mon, May 23, 2016 at 1:19
> PM,
> > >> > parth
> > >> > >>> > > > > brahmbhatt <
> > >> > >>> > > > > > > > > >> >> > > > > > > brahmbhatt.parth@gmail.com>
> > wrote:
> > >> > >>> > > > > > > > > >> >> > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > Hi Jun,
> > >> > >>> > > > > > > > > >> >> > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > Thanks for reviewing.
> > >> > >>> > > > > > > > > >> >> > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > * We could add a Cluster
> > action
> > >> > >>> > > > > > > > > >> >> > > > > > > > to
> > >> > add
> > >> > >>> > acls
> > >> > >>> > > > on
> > >> > >>> > > > > > who
> > >> > >>> > > > > > > > can
> > >> > >>> > > > > > > > > >> request
> > >> > >>> > > > > > > > > >> >> > > > > > delegation
> > >> > >>> > > > > > > > > >> >> > > > > > > > tokens. I don't see the use
> > case
> > >> > for
> > >> > >>> that
> > >> > >>> > > yet
> > >> > >>> > > > > but
> > >> > >>> > > > > > > > down
> > >> > >>> > > > > > > > > >> the line
> > >> > >>> > > > > > > > > >> >> > > > when
> > >> > >>> > > > > > > > > >> >> > > > > we
> > >> > >>> > > > > > > > > >> >> > > > > > > > start supporting
> > >> > getDelegationTokenAs
> > >> > >>> it
> > >> > >>> > > will
> > >> > >>> > > > > be
> > >> > >>> > > > > > > > > >> necessary.
> > >> > >>> > > > > > > > > >> >> > > > > > > > * Yes we recommend tokens to
> > be
> > >> > only
> > >> > >>> > > > > > > used/distributed
> > >> > >>> > > > > > > > > >> over
> > >> > >>> > > > > > > > > >> >> > secure
> > >> > >>> > > > > > > > > >> >> > > > > > > channels.
> > >> > >>> > > > > > > > > >> >> > > > > > > > * Depending on what design
> we
> > >> > >>> > > > > > > > > >> >> > > > > > > > end
> > >> > up
> > >> > >>> > > choosing
> > >> > >>> > > > > > > > > >> Invalidation will
> > >> > >>> > > > > > > > > >> >> > > be
> > >> > >>> > > > > > > > > >> >> > > > > > > > responsibility of every
> broker
> > >> > >>> > > > > > > > > >> >> > > > > > > > or
> > >> > >>> > > controller.
> > >> > >>> > > > > > > > > >> >> > > > > > > > * I am not sure if I
> > documented
> > >> > >>> somewhere
> > >> > >>> > > > that
> > >> > >>> > > > > > > > > >> invalidation
> > >> > >>> > > > > > > > > >> >> > will
> > >> > >>> > > > > > > > > >> >> > > > > > directly
> > >> > >>> > > > > > > > > >> >> > > > > > > > go through zookeeper but
> that
> > is
> > >> > not
> > >> > >>> the
> > >> > >>> > > > > intent.
> > >> > >>> > > > > > > > > >> Invalidation
> > >> > >>> > > > > > > > > >> >> > > will
> > >> > >>> > > > > > > > > >> >> > > > > > either
> > >> > >>> > > > > > > > > >> >> > > > > > > > be request based or due to
> > >> > >>> expiration. No
> > >> > >>> > > > > direct
> > >> > >>> > > > > > > > > >> zookeeper
> > >> > >>> > > > > > > > > >> >> > > > > interaction
> > >> > >>> > > > > > > > > >> >> > > > > > > from
> > >> > >>> > > > > > > > > >> >> > > > > > > > any client.
> > >> > >>> > > > > > > > > >> >> > > > > > > > * "Broker also stores the
> > >> > >>> DelegationToken
> > >> > >>> > > > > without
> > >> > >>> > > > > > > the
> > >> > >>> > > > > > > > > >> hmac in
> > >> > >>> > > > > > > > > >> >> > the
> > >> > >>> > > > > > > > > >> >> > > > > > > > zookeeper." : Sorry about
> the
> > >> > >>> confusion.
> > >> > >>> > > The
> > >> > >>> > > > > sole
> > >> > >>> > > > > > > > > >> purpose of
> > >> > >>> > > > > > > > > >> >> > > > > zookeeper
> > >> > >>> > > > > > > > > >> >> > > > > > in
> > >> > >>> > > > > > > > > >> >> > > > > > > > this design is as
> distribution
> > >> > channel
> > >> > >>> > for
> > >> > >>> > > > > tokens
> > >> > >>> > > > > > > > > >> between all
> > >> > >>> > > > > > > > > >> >> > > > brokers
> > >> > >>> > > > > > > > > >> >> > > > > > > and a
> > >> > >>> > > > > > > > > >> >> > > > > > > > layer that ensures only
> tokens
> > >> > >>> > > > > > > > > >> >> > > > > > > > that
> > >> > >>> were
> > >> > >>> > > > > > generated
> > >> > >>> > > > > > > by
> > >> > >>> > > > > > > > > >> making a
> > >> > >>> > > > > > > > > >> >> > > > > request
> > >> > >>> > > > > > > > > >> >> > > > > > > to a
> > >> > >>> > > > > > > > > >> >> > > > > > > > broker will be accepted
> (more
> > on
> > >> > this
> > >> > >>> in
> > >> > >>> > > > second
> > >> > >>> > > > > > > > > >> paragraph). The
> > >> > >>> > > > > > > > > >> >> > > > token
> > >> > >>> > > > > > > > > >> >> > > > > > > > consists of few elements
> > (owner,
> > >> > >>> renewer,
> > >> > >>> > > > uuid
> > >> > >>> > > > > ,
> > >> > >>> > > > > > > > > >> expiration,
> > >> > >>> > > > > > > > > >> >> > > hmac)
> > >> > >>> > > > > > > > > >> >> > > > ,
> > >> > >>> > > > > > > > > >> >> > > > > > one
> > >> > >>> > > > > > > > > >> >> > > > > > > of
> > >> > >>> > > > > > > > > >> >> > > > > > > > which is the finally
> generated
> > >> > >>> > > > > > > > > >> >> > > > > > > > hmac
> > >> > >>> but
> > >> > >>> > > hmac
> > >> > >>> > > > it
> > >> > >>> > > > > > > self
> > >> > >>> > > > > > > > is
> > >> > >>> > > > > > > > > >> >> > derivable
> > >> > >>> > > > > > > > > >> >> > > > if
> > >> > >>> > > > > > > > > >> >> > > > > > you
> > >> > >>> > > > > > > > > >> >> > > > > > > > have all the other elements
> of
> > >> > >>> > > > > > > > > >> >> > > > > > > > the
> > >> > >>> token
> > >> > >>> > +
> > >> > >>> > > > > secret
> > >> > >>> > > > > > > key
> > >> > >>> > > > > > > > > to
> > >> > >>> > > > > > > > > >> >> > generate
> > >> > >>> > > > > > > > > >> >> > > > > hmac.
> > >> > >>> > > > > > > > > >> >> > > > > > > > Given zookeeper does not
> > provide
> > >> > SSL
> > >> > >>> > > support
> > >> > >>> > > > we
> > >> > >>> > > > > > do
> > >> > >>> > > > > > > > not
> > >> > >>> > > > > > > > > >> want the
> > >> > >>> > > > > > > > > >> >> > > > > entire
> > >> > >>> > > > > > > > > >> >> > > > > > > > token to be wire transferred
> > to
> > >> > >>> zookeeper
> > >> > >>> > > as
> > >> > >>> > > > > that
> > >> > >>> > > > > > > > will
> > >> > >>> > > > > > > > > >> be an
> > >> > >>> > > > > > > > > >> >> > > > insecure
> > >> > >>> > > > > > > > > >> >> > > > > > > wire
> > >> > >>> > > > > > > > > >> >> > > > > > > > transfer. Instead we only
> > store
> > >> > >>> > > > > > > > > >> >> > > > > > > > all
> > >> > >>> the
> > >> > >>> > > other
> > >> > >>> > > > > > > > elements
> > >> > >>> > > > > > > > > >> of a
> > >> > >>> > > > > > > > > >> >> > > > > delegation
> > >> > >>> > > > > > > > > >> >> > > > > > > > tokens. Brokers can read
> these
> > >> > >>> elements
> > >> > >>> > and
> > >> > >>> > > > > > because
> > >> > >>> > > > > > > > > they
> > >> > >>> > > > > > > > > >> also
> > >> > >>> > > > > > > > > >> >> > > have
> > >> > >>> > > > > > > > > >> >> > > > > > access
> > >> > >>> > > > > > > > > >> >> > > > > > > > to secret key they will be
> > able
> > >> > >>> > > > > > > > > >> >> > > > > > > > to
> > >> > >>> > generate
> > >> > >>> > > > > hmac
> > >> > >>> > > > > > on
> > >> > >>> > > > > > > > > >> their end.
> > >> > >>> > > > > > > > > >> >> > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > One of the alternative
> > proposed
> > >> > >>> > > > > > > > > >> >> > > > > > > > is
> > >> > to
> > >> > >>> > avoid
> > >> > >>> > > > > > > zookeeper
> > >> > >>> > > > > > > > > >> >> > > altogether. A
> > >> > >>> > > > > > > > > >> >> > > > > > > Client
> > >> > >>> > > > > > > > > >> >> > > > > > > > will call broker with
> required
> > >> > >>> > information
> > >> > >>> > > > > > (owner,
> > >> > >>> > > > > > > > > >> renwer,
> > >> > >>> > > > > > > > > >> >> > > > > expiration)
> > >> > >>> > > > > > > > > >> >> > > > > > > and
> > >> > >>> > > > > > > > > >> >> > > > > > > > get back (signed hmac,
> uuid).
> > >> > Broker
> > >> > >>> > won't
> > >> > >>> > > > > store
> > >> > >>> > > > > > > this
> > >> > >>> > > > > > > > > in
> > >> > >>> > > > > > > > > >> >> > > zookeeper.
> > >> > >>> > > > > > > > > >> >> > > > > > From
> > >> > >>> > > > > > > > > >> >> > > > > > > > this point a client can
> > contact
> > >> > >>> > > > > > > > > >> >> > > > > > > > any
> > >> > >>> > broker
> > >> > >>> > > > with
> > >> > >>> > > > > > all
> > >> > >>> > > > > > > > the
> > >> > >>> > > > > > > > > >> >> > > delegation
> > >> > >>> > > > > > > > > >> >> > > > > > token
> > >> > >>> > > > > > > > > >> >> > > > > > > > info (owner, rewner,
> > expiration,
> > >> > hmac,
> > >> > >>> > > uuid)
> > >> > >>> > > > > the
> > >> > >>> > > > > > > > borker
> > >> > >>> > > > > > > > > >> will
> > >> > >>> > > > > > > > > >> >> > > > > regenerate
> > >> > >>> > > > > > > > > >> >> > > > > > > the
> > >> > >>> > > > > > > > > >> >> > > > > > > > hmac and as long as it
> matches
> > >> > >>> > > > > > > > > >> >> > > > > > > > with
> > >> > >>> hmac
> > >> > >>> > > > > > presented
> > >> > >>> > > > > > > by
> > >> > >>> > > > > > > > > >> client ,
> > >> > >>> > > > > > > > > >> >> > > > broker
> > >> > >>> > > > > > > > > >> >> > > > > > > will
> > >> > >>> > > > > > > > > >> >> > > > > > > > allow the request to
> > >> > >>> > > > > > > > > >> >> > > > > > > > authenticate.
> > >> > >>> Only
> > >> > >>> > > > > problem
> > >> > >>> > > > > > > with
> > >> > >>> > > > > > > > > >> this
> > >> > >>> > > > > > > > > >> >> > > approach
> > >> > >>> > > > > > > > > >> >> > > > > is
> > >> > >>> > > > > > > > > >> >> > > > > > if
> > >> > >>> > > > > > > > > >> >> > > > > > > > the secret key is
> compromised
> > >> > >>> > > > > > > > > >> >> > > > > > > > any
> > >> > >>> client
> > >> > >>> > > can
> > >> > >>> > > > > now
> > >> > >>> > > > > > > > > generate
> > >> > >>> > > > > > > > > >> >> > random
> > >> > >>> > > > > > > > > >> >> > > > > tokens
> > >> > >>> > > > > > > > > >> >> > > > > > > and
> > >> > >>> > > > > > > > > >> >> > > > > > > > they will still be able to
> > >> > >>> authenticate
> > >> > >>> > as
> > >> > >>> > > > any
> > >> > >>> > > > > > user
> > >> > >>> > > > > > > > > they
> > >> > >>> > > > > > > > > >> like.
> > >> > >>> > > > > > > > > >> >> > > with
> > >> > >>> > > > > > > > > >> >> > > > > > > > zookeeper we guarantee that
> > only
> > >> > >>> tokens
> > >> > >>> > > > > acquired
> > >> > >>> > > > > > > via
> > >> > >>> > > > > > > > a
> > >> > >>> > > > > > > > > >> broker
> > >> > >>> > > > > > > > > >> >> > > > (using
> > >> > >>> > > > > > > > > >> >> > > > > > some
> > >> > >>> > > > > > > > > >> >> > > > > > > > auth scheme other than
> > >> > >>> > > > > > > > > >> >> > > > > > > > delegation
> > >> > >>> token)
> > >> > >>> > > will
> > >> > >>> > > > > be
> > >> > >>> > > > > > > > > >> accepted. We
> > >> > >>> > > > > > > > > >> >> > > need
> > >> > >>> > > > > > > > > >> >> > > > to
> > >> > >>> > > > > > > > > >> >> > > > > > > > discuss which proposal makes
> > >> > >>> > > > > > > > > >> >> > > > > > > > more
> > >> > >>> sense
> > >> > >>> > and
> > >> > >>> > > > we
> > >> > >>> > > > > > can
> > >> > >>> > > > > > > go
> > >> > >>> > > > > > > > > >> over it
> > >> > >>> > > > > > > > > >> >> > in
> > >> > >>> > > > > > > > > >> >> > > > > > > tomorrow's
> > >> > >>> > > > > > > > > >> >> > > > > > > > meeting.
> > >> > >>> > > > > > > > > >> >> > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > Also, can you forward the
> > invite
> > >> > >>> > > > > > > > > >> >> > > > > > > > to
> > >> > >>> me?
> > >> > >>> > > > > > > > > >> >> > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > Thanks
> > >> > >>> > > > > > > > > >> >> > > > > > > > Parth
> > >> > >>> > > > > > > > > >> >> > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > On Mon, May 23, 2016 at
> 10:35
> > >> > >>> > > > > > > > > >> >> > > > > > > > AM,
> > >> > Jun
> > >> > >>> > Rao <
> > >> > >>> > > > > > > > > >> jun@confluent.io>
> > >> > >>> > > > > > > > > >> >> > > > wrote:
> > >> > >>> > > > > > > > > >> >> > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > Thanks for the KIP. A few
> > >> > comments.
> > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > 100. This potentially can
> be
> > >> > useful
> > >> > >>> for
> > >> > >>> > > > Kafka
> > >> > >>> > > > > > > > Connect
> > >> > >>> > > > > > > > > >> and
> > >> > >>> > > > > > > > > >> >> > Kafka
> > >> > >>> > > > > > > > > >> >> > > > > rest
> > >> > >>> > > > > > > > > >> >> > > > > > > > proxy
> > >> > >>> > > > > > > > > >> >> > > > > > > > > where a worker agent will
> > need
> > >> > >>> > > > > > > > > >> >> > > > > > > > > to
> > >> > >>> run a
> > >> > >>> > > > task
> > >> > >>> > > > > on
> > >> > >>> > > > > > > > > behalf
> > >> > >>> > > > > > > > > >> of a
> > >> > >>> > > > > > > > > >> >> > > > client.
> > >> > >>> > > > > > > > > >> >> > > > > > We
> > >> > >>> > > > > > > > > >> >> > > > > > > > will
> > >> > >>> > > > > > > > > >> >> > > > > > > > > likely need to change how
> > >> > >>> > > > > > > > > >> >> > > > > > > > > those
> > >> > >>> > services
> > >> > >>> > > > use
> > >> > >>> > > > > > > Kafka
> > >> > >>> > > > > > > > > >> clients
> > >> > >>> > > > > > > > > >> >> > > > > > > > > (producer/consumer).
> Instead
> > >> > >>> > > > > > > > > >> >> > > > > > > > > of a
> > >> > >>> > shared
> > >> > >>> > > > > client
> > >> > >>> > > > > > > per
> > >> > >>> > > > > > > > > >> worker,
> > >> > >>> > > > > > > > > >> >> > we
> > >> > >>> > > > > > > > > >> >> > > > will
> > >> > >>> > > > > > > > > >> >> > > > > > > need
> > >> > >>> > > > > > > > > >> >> > > > > > > > a
> > >> > >>> > > > > > > > > >> >> > > > > > > > > client per user task since
> > the
> > >> > >>> > > > authentication
> > >> > >>> > > > > > > > happens
> > >> > >>> > > > > > > > > >> at the
> > >> > >>> > > > > > > > > >> >> > > > > > connection
> > >> > >>> > > > > > > > > >> >> > > > > > > > > level. For Kafka Connect,
> > the
> > >> > >>> renewer
> > >> > >>> > > will
> > >> > >>> > > > be
> > >> > >>> > > > > > the
> > >> > >>> > > > > > > > > >> workers.
> > >> > >>> > > > > > > > > >> >> > So,
> > >> > >>> > > > > > > > > >> >> > > we
> > >> > >>> > > > > > > > > >> >> > > > > > > > probably
> > >> > >>> > > > > > > > > >> >> > > > > > > > > need to allow multiple
> > >> > >>> > > > > > > > > >> >> > > > > > > > > renewers.
> > >> > For
> > >> > >>> > > Kafka
> > >> > >>> > > > > rest
> > >> > >>> > > > > > > > > proxy,
> > >> > >>> > > > > > > > > >> the
> > >> > >>> > > > > > > > > >> >> > > > renewer
> > >> > >>> > > > > > > > > >> >> > > > > > can
> > >> > >>> > > > > > > > > >> >> > > > > > > > > probably just be the
> creator
> > >> > >>> > > > > > > > > >> >> > > > > > > > > of
> > >> > the
> > >> > >>> > > token.
> > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > 101. Do we need new acl on
> > who
> > >> > can
> > >> > >>> > > request
> > >> > >>> > > > > > > > delegation
> > >> > >>> > > > > > > > > >> tokens?
> > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > 102. Do we recommend
> people
> > to
> > >> > send
> > >> > >>> > > > > delegation
> > >> > >>> > > > > > > > tokens
> > >> > >>> > > > > > > > > >> in an
> > >> > >>> > > > > > > > > >> >> > > > > encrypted
> > >> > >>> > > > > > > > > >> >> > > > > > > > > channel?
> > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > 103. Who is responsible
> for
> > >> > expiring
> > >> > >>> > > > tokens,
> > >> > >>> > > > > > > every
> > >> > >>> > > > > > > > > >> broker?
> > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > 104. For invalidating
> > tokens,
> > >> > would
> > >> > >>> it
> > >> > >>> > be
> > >> > >>> > > > > > better
> > >> > >>> > > > > > > to
> > >> > >>> > > > > > > > > do
> > >> > >>> > > > > > > > > >> it in
> > >> > >>> > > > > > > > > >> >> > a
> > >> > >>> > > > > > > > > >> >> > > > > > request
> > >> > >>> > > > > > > > > >> >> > > > > > > > > instead of going to ZK
> > >> > >>> > > > > > > > > >> >> > > > > > > > > directly?
> > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > 105. The terminology of
> > client
> > >> > >>> > > > > > > > > >> >> > > > > > > > > in
> > >> > >>> the
> > >> > >>> > > wiki
> > >> > >>> > > > > > > > sometimes
> > >> > >>> > > > > > > > > >> refers
> > >> > >>> > > > > > > > > >> >> > to
> > >> > >>> > > > > > > > > >> >> > > > the
> > >> > >>> > > > > > > > > >> >> > > > > > end
> > >> > >>> > > > > > > > > >> >> > > > > > > > > client and some other
> times
> > >> > refers
> > >> > >>> to
> > >> > >>> > the
> > >> > >>> > > > > > client
> > >> > >>> > > > > > > > > using
> > >> > >>> > > > > > > > > >> the
> > >> > >>> > > > > > > > > >> >> > > > > delegation
> > >> > >>> > > > > > > > > >> >> > > > > > > > > tokens. It would be useful
> > to
> > >> > >>> > distinguish
> > >> > >>> > > > > > between
> > >> > >>> > > > > > > > the
> > >> > >>> > > > > > > > > >> two.
> > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > 106. Could you explain the
> > >> > sentence
> > >> > >>> > > "Broker
> > >> > >>> > > > > > also
> > >> > >>> > > > > > > > > >> stores the
> > >> > >>> > > > > > > > > >> >> > > > > > > > DelegationToken
> > >> > >>> > > > > > > > > >> >> > > > > > > > > without the hmac in the
> > >> > zookeeper."
> > >> > >>> a
> > >> > >>> > bit
> > >> > >>> > > > > > more? I
> > >> > >>> > > > > > > > > >> thought the
> > >> > >>> > > > > > > > > >> >> > > > > > > delegation
> > >> > >>> > > > > > > > > >> >> > > > > > > > > token is the hmac.
> > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > Thanks,
> > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > Jun
> > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > On Mon, May 23, 2016 at
> 9:22
> > >> > >>> > > > > > > > > >> >> > > > > > > > > AM,
> > >> > Jun
> > >> > >>> > Rao
> > >> > >>> > > <
> > >> > >>> > > > > > > > > >> jun@confluent.io>
> > >> > >>> > > > > > > > > >> >> > > > wrote:
> > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > > Hi, Harsha,
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > > Just sent out a KIP
> > meeting
> > >> > >>> invite.
> > >> > >>> > We
> > >> > >>> > > > can
> > >> > >>> > > > > > > > discuss
> > >> > >>> > > > > > > > > >> this in
> > >> > >>> > > > > > > > > >> >> > > the
> > >> > >>> > > > > > > > > >> >> > > > > > > meeting
> > >> > >>> > > > > > > > > >> >> > > > > > > > > > tomorrow.
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > > Thanks,
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > > Jun
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > > On Thu, May 19, 2016 at
> > 8:47
> > >> > AM,
> > >> > >>> > > Harsha <
> > >> > >>> > > > > > > > > >> kafka@harsha.io>
> > >> > >>> > > > > > > > > >> >> > > > wrote:
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> Hi All,
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >>            Can we have
> a
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> KIP
> > >> > >>> meeting
> > >> > >>> > > > > around
> > >> > >>> > > > > > > > this.
> > >> > >>> > > > > > > > > >> The KIP
> > >> > >>> > > > > > > > > >> >> > is
> > >> > >>> > > > > > > > > >> >> > > > up
> > >> > >>> > > > > > > > > >> >> > > > > > for
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >>            sometime and
> > if
> > >> > there
> > >> > >>> are
> > >> > >>> > > any
> > >> > >>> > > > > > > > questions
> > >> > >>> > > > > > > > > >> lets
> > >> > >>> > > > > > > > > >> >> > > > quickly
> > >> > >>> > > > > > > > > >> >> > > > > > hash
> > >> > >>> > > > > > > > > >> >> > > > > > > > out
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >>            details.
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >>
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> Thanks,
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> Harsha
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >>
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> On Thu, May 19, 2016,
> at
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> 08:40
> > >> > >>> AM,
> > >> > >>> > > parth
> > >> > >>> > > > > > > > > brahmbhatt
> > >> > >>> > > > > > > > > >> wrote:
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > That is what the
> hadoop
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > echo
> > >> > >>> > system
> > >> > >>> > > > uses
> > >> > >>> > > > > > so
> > >> > >>> > > > > > > no
> > >> > >>> > > > > > > > > >> good
> > >> > >>> > > > > > > > > >> >> > reason
> > >> > >>> > > > > > > > > >> >> > > > > > really.
> > >> > >>> > > > > > > > > >> >> > > > > > > > We
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > could
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > change it to whatever
> > is
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > the
> > >> > >>> > newest
> > >> > >>> > > > > > > > recommended
> > >> > >>> > > > > > > > > >> standard
> > >> > >>> > > > > > > > > >> >> > > is.
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > Thanks
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > Parth
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > On Thu, May 19, 2016
> at
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > 3:33
> > >> > >>> AM,
> > >> > >>> > > > Ismael
> > >> > >>> > > > > > > Juma <
> > >> > >>> > > > > > > > > >> >> > > > > ismael@juma.me.uk
> > >> > >>> > > > > > > > > >> >> > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > wrote:
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > Hi Parth,
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > Thanks for the
> KIP. I
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > only
> > >> > >>> > started
> > >> > >>> > > > > > > reviewing
> > >> > >>> > > > > > > > > >> this and
> > >> > >>> > > > > > > > > >> >> > > may
> > >> > >>> > > > > > > > > >> >> > > > > have
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> additional
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > questions later.
> The
> > >> > >>> immediate
> > >> > >>> > > > > question
> > >> > >>> > > > > > > that
> > >> > >>> > > > > > > > > >> came to
> > >> > >>> > > > > > > > > >> >> > > mind
> > >> > >>> > > > > > > > > >> >> > > > is
> > >> > >>> > > > > > > > > >> >> > > > > > our
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> choice of
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > "DIGEST-MD5" even
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > though
> > >> > it's
> > >> > >>> > > marked
> > >> > >>> > > > > as
> > >> > >>> > > > > > > > > >> OBSOLETE in
> > >> > >>> > > > > > > > > >> >> > the
> > >> > >>> > > > > > > > > >> >> > > > IANA
> > >> > >>> > > > > > > > > >> >> > > > > > > > > Registry
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> of
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > SASL mechanisms and
> > the
> > >> > >>> original
> > >> > >>> > > RFC
> > >> > >>> > > > > > > (2831)
> > >> > >>> > > > > > > > > has
> > >> > >>> > > > > > > > > >> been
> > >> > >>> > > > > > > > > >> >> > > moved
> > >> > >>> > > > > > > > > >> >> > > > > to
> > >> > >>> > > > > > > > > >> >> > > > > > > > > Historic
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > status:
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> > >> > https://tools.ietf.org/html/
> > >> > >>> > > rfc6331
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > >
> > >> > >>> > > > > > > > > >> >> > >
> > >> > >>> > > > > > > > > >>
> > >> > >>> > > > > > >
> > >> > >>> > > > http://www.iana.org/assignments/sasl-mechanisms/sasl-
> > >> > >>> mechanisms.xhtml
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > What is the
> reasoning
> > >> > behind
> > >> > >>> > that
> > >> > >>> > > > > > choice?
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > Thanks,
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > Ismael
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > On Fri, May 13,
> 2016
> > at
> > >> > 11:29
> > >> > >>> > PM,
> > >> > >>> > > > Gwen
> > >> > >>> > > > > > > > > Shapira <
> > >> > >>> > > > > > > > > >> >> > > > > > > gwen@confluent.io
> > >> > >>> > > > > > > > > >> >> > > > > > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> wrote:
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > Also comments
> > inline
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > :)
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > * I want to
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > emphasize
> > >> > >>> that
> > >> > >>> > > even
> > >> > >>> > > > > > though
> > >> > >>> > > > > > > > > >> delegation
> > >> > >>> > > > > > > > > >> >> > > > tokens
> > >> > >>> > > > > > > > > >> >> > > > > > > are a
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> Hadoop
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > innovation, I
> > feel
> > >> > very
> > >> > >>> > > strongly
> > >> > >>> > > > > > about
> > >> > >>> > > > > > > > not
> > >> > >>> > > > > > > > > >> adding
> > >> > >>> > > > > > > > > >> >> > > > > > dependency
> > >> > >>> > > > > > > > > >> >> > > > > > > > on
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> Hadoop
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > when
> implementing
> > >> > >>> delegation
> > >> > >>> > > > > tokens
> > >> > >>> > > > > > > for
> > >> > >>> > > > > > > > > >> Kafka. The
> > >> > >>> > > > > > > > > >> >> > > KIP
> > >> > >>> > > > > > > > > >> >> > > > > > > doesn't
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> imply
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > such
> dependency,
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > but
> > >> > if
> > >> > >>> you
> > >> > >>> > > can
> > >> > >>> > > > > > > > clarify...
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *No hadoop
> > >> > dependency.*
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > Yay! Just add
> this
> > to
> > >> > the
> > >> > >>> KIP
> > >> > >>> > so
> > >> > >>> > > > no
> > >> > >>> > > > > > one
> > >> > >>> > > > > > > > will
> > >> > >>> > > > > > > > > >> read
> > >> > >>> > > > > > > > > >> >> > the
> > >> > >>> > > > > > > > > >> >> > > > KIP
> > >> > >>> > > > > > > > > >> >> > > > > > and
> > >> > >>> > > > > > > > > >> >> > > > > > > > > panic
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > three weeks
> before
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > the
> > >> > next
> > >> > >>> > > > > release...
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > * Can we get
> > >> > delegation
> > >> > >>> > token
> > >> > >>> > > at
> > >> > >>> > > > > any
> > >> > >>> > > > > > > > time
> > >> > >>> > > > > > > > > >> after
> > >> > >>> > > > > > > > > >> >> > > > > > > > authenticating?
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> only
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > immediately
> > after?
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *As long as you
> > are
> > >> > >>> > > > authenticated
> > >> > >>> > > > > > you
> > >> > >>> > > > > > > > can
> > >> > >>> > > > > > > > > >> get
> > >> > >>> > > > > > > > > >> >> > > > delegation
> > >> > >>> > > > > > > > > >> >> > > > > > > > tokens.
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> We
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > need
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > to
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > discuss if a
> > client
> > >> > >>> > > > authenticated
> > >> > >>> > > > > > > using
> > >> > >>> > > > > > > > > >> delegation
> > >> > >>> > > > > > > > > >> >> > > > > token,
> > >> > >>> > > > > > > > > >> >> > > > > > > can
> > >> > >>> > > > > > > > > >> >> > > > > > > > > also
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > acquire
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > delegation
> token
> > >> > again or
> > >> > >>> > not.
> > >> > >>> > > > > Also
> > >> > >>> > > > > > > > there
> > >> > >>> > > > > > > > > >> is the
> > >> > >>> > > > > > > > > >> >> > > > > question
> > >> > >>> > > > > > > > > >> >> > > > > > of
> > >> > >>> > > > > > > > > >> >> > > > > > > > do
> > >> > >>> > > > > > > > > >> >> > > > > > > > > we
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > allow
> > >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > anyone to
> acquire
> > >> > >>> delegation
> > >> > >>> > > > token
> > >> > >>> > > > > > or
> > >> > >>> > > > > > > we
> > >> > >>> > > > > > > > > >> want
> > >> > >>> > > > > > > > > >> >> > > specific
> > >> > >>> > > > > > > > > >>
> >
> >
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>
>
>
> --
>
> Regards,
> Ashish
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Ashish Singh <as...@cloudera.com>]

Harsha/ Gwen,

How do we proceed here? I am willing to help out with here.

On Fri, Sep 23, 2016 at 11:41 AM, Gwen Shapira <gw...@confluent.io> wrote:

> Is it updated? are all concerns addressed? do you want to start a vote?
>
> Sorry for being pushy, I do appreciate that we are all volunteers and
> finding time is difficult. This feature is important for anything that
> integrates with Kafka (stream processors, Flume, NiFi, etc) and I
> don't want to see this getting stuck because we lack coordination
> within the community.
>
> On Thu, Sep 15, 2016 at 6:39 PM, Harsha Chintalapani <ka...@harsha.io>
> wrote:
> > The only pending update for the KIP is to write up the protocol changes
> like
> > we've it KIP-4. I'll update the wiki.
> >
> >
> > On Thu, Sep 15, 2016 at 4:27 PM Ashish Singh <as...@cloudera.com>
> wrote:
> >>
> >> I think we decided to not support secret rotation, I guess this can be
> >> stated clearly on the KIP. Also, more details on how clients will
> perform
> >> token distribution and how CLI will look like will be helpful.
> >>
> >> On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <gw...@confluent.io>
> wrote:
> >>
> >> > Hi Guys,
> >> >
> >> > This discussion was dead for a while. Are there still contentious
> >> > points? If not, why are there no votes?
> >> >
> >> > On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <ju...@confluent.io> wrote:
> >> > > Ashish,
> >> > >
> >> > > Yes, I will send out a KIP invite for next week to discuss KIP-48
> and
> >> > other
> >> > > remaining KIPs.
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Jun
> >> > >
> >> > > On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh <as...@cloudera.com>
> >> > wrote:
> >> > >
> >> > >> Thanks Harsha!
> >> > >>
> >> > >> Jun, can we add KIP-48 to next KIP hangout's agenda. Also, we did
> not
> >> > >> actually make a call on when we should have next KIP call. As there
> >> > >> are
> >> > a
> >> > >> few outstanding KIPs that could not be discussed this week, can we
> >> > >> have
> >> > a
> >> > >> KIP hangout call next week?
> >> > >>
> >> > >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha Chintalapani
> >> > >> <ka...@harsha.io>
> >> > >> wrote:
> >> > >>
> >> > >>> Ashish,
> >> > >>>         Yes we are working on it. Lets discuss in the next KIP
> >> > >>> meeting.
> >> > >>> I'll join.
> >> > >>> -Harsha
> >> > >>>
> >> > >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh <
> asingh@cloudera.com>
> >> > >>> wrote:
> >> > >>>
> >> > >>> > Hello Harsha,
> >> > >>> >
> >> > >>> > Are you still working on this? Wondering if we can discuss this
> in
> >> > next
> >> > >>> KIP
> >> > >>> > meeting, if you can join.
> >> > >>> >
> >> > >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha Chintalapani <
> >> > kafka@harsha.io>
> >> > >>> > wrote:
> >> > >>> >
> >> > >>> > > Hi Grant,
> >> > >>> > >           We are working on it. Will add the details to KIP
> >> > >>> > > about
> >> > the
> >> > >>> > > request protocol.
> >> > >>> > >
> >> > >>> > > Thanks,
> >> > >>> > > Harsha
> >> > >>> > >
> >> > >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant Henke
> >> > >>> > > <gh...@cloudera.com>
> >> > >>> wrote:
> >> > >>> > >
> >> > >>> > > > Hi Parth,
> >> > >>> > > >
> >> > >>> > > > Are you still working on this? If you need any help please
> >> > >>> > > > don't
> >> > >>> > hesitate
> >> > >>> > > > to ask.
> >> > >>> > > >
> >> > >>> > > > Thanks,
> >> > >>> > > > Grant
> >> > >>> > > >
> >> > >>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <ju...@confluent.io>
> >> > wrote:
> >> > >>> > > >
> >> > >>> > > > > Parth,
> >> > >>> > > > >
> >> > >>> > > > > Thanks for the reply.
> >> > >>> > > > >
> >> > >>> > > > > It makes sense to only allow the renewal by users that
> >> > >>> authenticated
> >> > >>> > > > using
> >> > >>> > > > > *non* delegation token mechanism. Then, should we make the
> >> > >>> renewal a
> >> > >>> > > > list?
> >> > >>> > > > > For example, in the case of rest proxy, it will be useful
> >> > >>> > > > > for
> >> > >>> every
> >> > >>> > > > > instance of rest proxy to be able to renew the tokens.
> >> > >>> > > > >
> >> > >>> > > > > It would be clearer if we can document the request
> protocol
> >> > like
> >> > >>> > > > >
> >> > >>> > > > >
> >> > >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> > >>> > >
> >> > >>> > > 4+-+Command+line+and+centralized+administrative+
> operations#KIP-4-
> >> > >>> > > Commandlineandcentralizedadministrativeoperations-
> >> > >>> > > CreateTopicsRequest(KAFKA-2945):(VotedandPlannedforin0.
> 10.1.0)
> >> > >>> > > > > .
> >> > >>> > > > >
> >> > >>> > > > > It would also be useful to document the client APIs.
> >> > >>> > > > >
> >> > >>> > > > > Thanks,
> >> > >>> > > > >
> >> > >>> > > > > Jun
> >> > >>> > > > >
> >> > >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM, parth brahmbhatt <
> >> > >>> > > > > brahmbhatt.parth@gmail.com> wrote:
> >> > >>> > > > >
> >> > >>> > > > > > Hi,
> >> > >>> > > > > >
> >> > >>> > > > > > I am suggesting that we will only allow the renewal by
> >> > >>> > > > > > users
> >> > >>> that
> >> > >>> > > > > > authenticated using *non* delegation token mechanism.
> For
> >> > >>> example,
> >> > >>> > If
> >> > >>> > > > > user
> >> > >>> > > > > > Alice authenticated using kerberos and requested
> >> > >>> > > > > > delegation
> >> > >>> tokens,
> >> > >>> > > > only
> >> > >>> > > > > > user Alice authenticated via non delegation token
> >> > >>> > > > > > mechanism
> >> > can
> >> > >>> > > renew.
> >> > >>> > > > > > Clients that have  access to delegation tokens can not
> >> > >>> > > > > > issue
> >> > >>> > renewal
> >> > >>> > > > > > request for renewing their own token and this is
> primarily
> >> > >>> > important
> >> > >>> > > to
> >> > >>> > > > > > reduce the time window for which a compromised token
> will
> >> > >>> > > > > > be
> >> > >>> valid.
> >> > >>> > > > > >
> >> > >>> > > > > > To clarify, Yes any authenticated user can request
> >> > >>> > > > > > delegation
> >> > >>> > tokens
> >> > >>> > > > but
> >> > >>> > > > > > even here I would recommend to avoid creating a chain
> >> > >>> > > > > > where a
> >> > >>> > client
> >> > >>> > > > > > authenticated via delegation token request for more
> >> > delegation
> >> > >>> > > tokens.
> >> > >>> > > > > > Basically anyone can request delegation token, as long
> as
> >> > they
> >> > >>> > > > > authenticate
> >> > >>> > > > > > via a non delegation token mechanism.
> >> > >>> > > > > >
> >> > >>> > > > > > Aren't classes listed here
> >> > >>> > > > > > <
> >> > >>> > > > > >
> >> > >>> > > > >
> >> > >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> > >>> > > 48+Delegation+token+support+for+Kafka#KIP-48Delegationtokens
> >> > >>> upportforKaf
> >> > >>> > > ka-PublicInterfaces
> >> > >>> > > > > > >
> >> > >>> > > > > > sufficient?
> >> > >>> > > > > >
> >> > >>> > > > > > Thanks
> >> > >>> > > > > > Parth
> >> > >>> > > > > >
> >> > >>> > > > > >
> >> > >>> > > > > >
> >> > >>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao
> >> > >>> > > > > > <ju...@confluent.io>
> >> > >>> wrote:
> >> > >>> > > > > >
> >> > >>> > > > > > > Parth,
> >> > >>> > > > > > >
> >> > >>> > > > > > > Thanks for the reply. A couple of comments inline
> below.
> >> > >>> > > > > > >
> >> > >>> > > > > > > On Tue, Jun 21, 2016 at 10:36 AM, parth brahmbhatt <
> >> > >>> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> >> > >>> > > > > > >
> >> > >>> > > > > > > > 1. Who / how are tokens renewed? By original
> requester
> >> > >>> only? or
> >> > >>> > > > using
> >> > >>> > > > > > > > Kerberos
> >> > >>> > > > > > > > auth only?
> >> > >>> > > > > > > > My recommendation is to do this only using Kerberos
> >> > >>> > > > > > > > auth
> >> > and
> >> > >>> > only
> >> > >>> > > > > threw
> >> > >>> > > > > > > the
> >> > >>> > > > > > > > renewer specified during the acquisition request.
> >> > >>> > > > > > > >
> >> > >>> > > > > > > >
> >> > >>> > > > > > > Hmm, not sure that I follow this. Are you saying that
> >> > >>> > > > > > > any
> >> > >>> client
> >> > >>> > > > > > > authenticated with the delegation token can renew,
> i.e.
> >> > there
> >> > >>> is
> >> > >>> > no
> >> > >>> > > > > > renewer
> >> > >>> > > > > > > needed?
> >> > >>> > > > > > >
> >> > >>> > > > > > > Also, just to be clear, any authenticated client
> (either
> >> > >>> through
> >> > >>> > > SASL
> >> > >>> > > > > or
> >> > >>> > > > > > > SSL) can request a delegation token for the
> >> > >>> > > > > > > authenticated
> >> > >>> user,
> >> > >>> > > > right?
> >> > >>> > > > > > >
> >> > >>> > > > > > >
> >> > >>> > > > > > > > 2. Are tokens stored on each broker or in ZK?
> >> > >>> > > > > > > > My recommendation is still to store in ZK or not
> store
> >> > them
> >> > >>> at
> >> > >>> > > all.
> >> > >>> > > > > The
> >> > >>> > > > > > > > whole controller based distribution is too much
> >> > >>> > > > > > > > overhead
> >> > >>> with
> >> > >>> > not
> >> > >>> > > > > much
> >> > >>> > > > > > to
> >> > >>> > > > > > > > achieve.
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > 3. How are tokens invalidated / expired?
> >> > >>> > > > > > > > Either by expiration time out or through an explicit
> >> > >>> request to
> >> > >>> > > > > > > invalidate.
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > 4. Which encryption algorithm is used?
> >> > >>> > > > > > > > SCRAM
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > 5. What is the impersonation proposal (it wasn't in
> >> > >>> > > > > > > > the
> >> > KIP
> >> > >>> but
> >> > >>> > > was
> >> > >>> > > > > > > > discussed
> >> > >>> > > > > > > > in this thread)?
> >> > >>> > > > > > > > There is no imperonation proposal. I tried and
> >> > >>> > > > > > > > explained
> >> > how
> >> > >>> > its
> >> > >>> > > a
> >> > >>> > > > > > > > different problem and why its not really necessary
> to
> >> > >>> discuss
> >> > >>> > > that
> >> > >>> > > > as
> >> > >>> > > > > > > part
> >> > >>> > > > > > > > of this KIP.  This KIP will not support any
> >> > impersonation,
> >> > >>> it
> >> > >>> > > will
> >> > >>> > > > > just
> >> > >>> > > > > > > be
> >> > >>> > > > > > > > another way to authenticate.
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > 6. Do we need new ACLs, if so - for what actions?
> >> > >>> > > > > > > > We do not need new ACLs.
> >> > >>> > > > > > > >
> >> > >>> > > > > > > >
> >> > >>> > > > > > > Could we document the format of the new
> request/response
> >> > and
> >> > >>> > their
> >> > >>> > > > > > > associated Resource and Operation for ACL?
> >> > >>> > > > > > >
> >> > >>> > > > > > >
> >> > >>> > > > > > > > 7. How would the delegation token be configured in
> the
> >> > >>> client?
> >> > >>> > > > > > > > Should be through config. I wasn't planning on
> >> > >>> > > > > > > > supporting
> >> > >>> JAAS
> >> > >>> > > for
> >> > >>> > > > > > > tokens.
> >> > >>> > > > > > > > I don't believe hadoop does this either.
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > Thanks
> >> > >>> > > > > > > > Parth
> >> > >>> > > > > > > >
> >> > >>> > > > > > > >
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <
> >> > jun@confluent.io>
> >> > >>> > > wrote:
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > > Harsha,
> >> > >>> > > > > > > > >
> >> > >>> > > > > > > > > Another question.
> >> > >>> > > > > > > > >
> >> > >>> > > > > > > > > 9. How would the delegation token be configured in
> >> > >>> > > > > > > > > the
> >> > >>> > client?
> >> > >>> > > > The
> >> > >>> > > > > > > > standard
> >> > >>> > > > > > > > > way is to do this through JAAS. However, we will
> >> > >>> > > > > > > > > need
> >> > to
> >> > >>> > think
> >> > >>> > > > > > through
> >> > >>> > > > > > > if
> >> > >>> > > > > > > > > this is convenient in a shared environment. For
> >> > example,
> >> > >>> > when a
> >> > >>> > > > new
> >> > >>> > > > > > > task
> >> > >>> > > > > > > > is
> >> > >>> > > > > > > > > added to a Storm worker node, do we need to
> >> > >>> > > > > > > > > dynamically
> >> > >>> add a
> >> > >>> > > new
> >> > >>> > > > > > > section
> >> > >>> > > > > > > > > in the JAAS file? It may be more convenient if we
> >> > >>> > > > > > > > > can
> >> > >>> pass in
> >> > >>> > > the
> >> > >>> > > > > > token
> >> > >>> > > > > > > > > through the config directly w/o going through
> JAAS.
> >> > >>> > > > > > > > >
> >> > >>> > > > > > > > > Are you or Parth still actively working on this
> KIP?
> >> > >>> > > > > > > > >
> >> > >>> > > > > > > > > Thanks,
> >> > >>> > > > > > > > >
> >> > >>> > > > > > > > > Jun
> >> > >>> > > > > > > > >
> >> > >>> > > > > > > > >
> >> > >>> > > > > > > > >
> >> > >>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <
> >> > >>> jun@confluent.io>
> >> > >>> > > > wrote:
> >> > >>> > > > > > > > >
> >> > >>> > > > > > > > > > Just to add on that list.
> >> > >>> > > > > > > > > >
> >> > >>> > > > > > > > > > 2. It would be good to document the format of
> the
> >> > data
> >> > >>> > stored
> >> > >>> > > > in
> >> > >>> > > > > > ZK.
> >> > >>> > > > > > > > > > 7. Earlier, there was a discussion on whether
> the
> >> > tokens
> >> > >>> > > should
> >> > >>> > > > > be
> >> > >>> > > > > > > > > > propagated through ZK like config/acl/quota, or
> >> > through
> >> > >>> the
> >> > >>> > > > > > > controller.
> >> > >>> > > > > > > > > > Currently, the controller is only designed for
> >> > >>> propagating
> >> > >>> > > > topic
> >> > >>> > > > > > > > > metadata,
> >> > >>> > > > > > > > > > but not other data.
> >> > >>> > > > > > > > > > 8. Should we use SCRAM to send the token instead
> >> > >>> > > > > > > > > > of
> >> > >>> > > DIGEST-MD5
> >> > >>> > > > > > since
> >> > >>> > > > > > > > it's
> >> > >>> > > > > > > > > > deprecated?
> >> > >>> > > > > > > > > >
> >> > >>> > > > > > > > > > Also, the images in the wiki seem broken.
> >> > >>> > > > > > > > > >
> >> > >>> > > > > > > > > > Thanks,
> >> > >>> > > > > > > > > >
> >> > >>> > > > > > > > > > Jun
> >> > >>> > > > > > > > > >
> >> > >>> > > > > > > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <
> >> > >>> > > > > gwen@confluent.io>
> >> > >>> > > > > > > > > wrote:
> >> > >>> > > > > > > > > >
> >> > >>> > > > > > > > > >> From what I can see, remaining questions are:
> >> > >>> > > > > > > > > >>
> >> > >>> > > > > > > > > >> 1. Who / how are tokens renewed? By original
> >> > requester
> >> > >>> > only?
> >> > >>> > > > or
> >> > >>> > > > > > > using
> >> > >>> > > > > > > > > >> Kerberos auth only?
> >> > >>> > > > > > > > > >> 2. Are tokens stored on each broker or in ZK?
> >> > >>> > > > > > > > > >> 3. How are tokens invalidated / expired?
> >> > >>> > > > > > > > > >> 4. Which encryption algorithm is used?
> >> > >>> > > > > > > > > >> 5. What is the impersonation proposal (it
> wasn't
> >> > >>> > > > > > > > > >> in
> >> > the
> >> > >>> > KIP
> >> > >>> > > > but
> >> > >>> > > > > > was
> >> > >>> > > > > > > > > >> discussed in this thread)?
> >> > >>> > > > > > > > > >> 6. Do we need new ACLs, if so - for what
> actions?
> >> > >>> > > > > > > > > >>
> >> > >>> > > > > > > > > >> Gwen
> >> > >>> > > > > > > > > >>
> >> > >>> > > > > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <
> >> > >>> kafka@harsha.io>
> >> > >>> > > > wrote:
> >> > >>> > > > > > > > > >> > Jun & Ismael,
> >> > >>> > > > > > > > > >> >                          Unfortunately I
> >> > >>> > > > > > > > > >> > couldn't
> >> > >>> attend
> >> > >>> > > the
> >> > >>> > > > > KIP
> >> > >>> > > > > > > > > meeting
> >> > >>> > > > > > > > > >> >                          when delegation
> tokens
> >> > >>> > discussed.
> >> > >>> > > > > > > > Appreciate
> >> > >>> > > > > > > > > if
> >> > >>> > > > > > > > > >> >                          you can update the
> >> > thread if
> >> > >>> > you
> >> > >>> > > > have
> >> > >>> > > > > > any
> >> > >>> > > > > > > > > >> >                          further questions.
> >> > >>> > > > > > > > > >> > Thanks,
> >> > >>> > > > > > > > > >> > Harsha
> >> > >>> > > > > > > > > >> >
> >> > >>> > > > > > > > > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei
> >> > wrote:
> >> > >>> > > > > > > > > >> >> It seems that the links to images in the KIP
> >> > >>> > > > > > > > > >> >> are
> >> > >>> > broken.
> >> > >>> > > > > > > > > >> >>
> >> > >>> > > > > > > > > >> >> Liquan
> >> > >>> > > > > > > > > >> >>
> >> > >>> > > > > > > > > >> >> On Tue, May 24, 2016 at 9:33 AM, parth
> >> > brahmbhatt <
> >> > >>> > > > > > > > > >> >> brahmbhatt.parth@gmail.com> wrote:
> >> > >>> > > > > > > > > >> >>
> >> > >>> > > > > > > > > >> >> > 110. What does getDelegationTokenAs mean?
> >> > >>> > > > > > > > > >> >> > In the current proposal we only allow a
> user
> >> > >>> > > > > > > > > >> >> > to
> >> > >>> get
> >> > >>> > > > > > delegation
> >> > >>> > > > > > > > > token
> >> > >>> > > > > > > > > >> for
> >> > >>> > > > > > > > > >> >> > the identity that it authenticated as
> using
> >> > >>> another
> >> > >>> > > > > > mechanism,
> >> > >>> > > > > > > > i.e.
> >> > >>> > > > > > > > > >> A user
> >> > >>> > > > > > > > > >> >> > that authenticate using a keytab for
> >> > >>> > > > > > > > > >> >> > principal
> >> > >>> > > > > > > user1@EXAMPLE.COM
> >> > >>> > > > > > > > > >> will get
> >> > >>> > > > > > > > > >> >> > delegation tokens for that user only. In
> >> > future I
> >> > >>> > think
> >> > >>> > > > we
> >> > >>> > > > > > will
> >> > >>> > > > > > > > > have
> >> > >>> > > > > > > > > >> to
> >> > >>> > > > > > > > > >> >> > extend support such that we allow some set
> >> > >>> > > > > > > > > >> >> > of
> >> > >>> users (
> >> > >>> > > > > > > > > >> >> > kafka-rest-user@EXAMPLE.COM,
> >> > >>> > storm-nimbus@EXAMPLE.COM)
> >> > >>> > > > to
> >> > >>> > > > > > > > acquire
> >> > >>> > > > > > > > > >> >> > delegation tokens on behalf of other users
> >> > whose
> >> > >>> > > identity
> >> > >>> > > > > > they
> >> > >>> > > > > > > > have
> >> > >>> > > > > > > > > >> >> > verified independently.  Kafka brokers
> will
> >> > have
> >> > >>> ACLs
> >> > >>> > > to
> >> > >>> > > > > > > control
> >> > >>> > > > > > > > > >> which
> >> > >>> > > > > > > > > >> >> > users are allowed to impersonate other
> users
> >> > and
> >> > >>> get
> >> > >>> > > > tokens
> >> > >>> > > > > > on
> >> > >>> > > > > > > > > >> behalf of
> >> > >>> > > > > > > > > >> >> > them. Overall Impersonation is a whole
> >> > different
> >> > >>> > > problem
> >> > >>> > > > in
> >> > >>> > > > > > my
> >> > >>> > > > > > > > > >> opinion and
> >> > >>> > > > > > > > > >> >> > I think we can tackle it in separate KIP.
> >> > >>> > > > > > > > > >> >> >
> >> > >>> > > > > > > > > >> >> > 111. What's the typical rate of getting
> and
> >> > >>> renewing
> >> > >>> > > > > > delegation
> >> > >>> > > > > > > > > >> tokens?
> >> > >>> > > > > > > > > >> >> > Typically this should be very very low, 1
> >> > request
> >> > >>> per
> >> > >>> > > > > minute
> >> > >>> > > > > > > is a
> >> > >>> > > > > > > > > >> >> > relatively high estimate. However it
> depends
> >> > >>> > > > > > > > > >> >> > on
> >> > >>> the
> >> > >>> > > token
> >> > >>> > > > > > > > > >> expiration. I am
> >> > >>> > > > > > > > > >> >> > less worried about the extra load it puts
> on
> >> > >>> > controller
> >> > >>> > > > vs
> >> > >>> > > > > > the
> >> > >>> > > > > > > > > added
> >> > >>> > > > > > > > > >> >> > complexity and the value it offers.
> >> > >>> > > > > > > > > >> >> >
> >> > >>> > > > > > > > > >> >> > Thanks
> >> > >>> > > > > > > > > >> >> > Parth
> >> > >>> > > > > > > > > >> >> >
> >> > >>> > > > > > > > > >> >> >
> >> > >>> > > > > > > > > >> >> >
> >> > >>> > > > > > > > > >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael
> Juma
> >> > >>> > > > > > > > > >> >> > <
> >> > >>> > > > > > > ismael@juma.me.uk>
> >> > >>> > > > > > > > > >> wrote:
> >> > >>> > > > > > > > > >> >> >
> >> > >>> > > > > > > > > >> >> > > Thanks Rajini. It would probably
> require a
> >> > >>> separate
> >> > >>> > > KIP
> >> > >>> > > > > as
> >> > >>> > > > > > it
> >> > >>> > > > > > > > > will
> >> > >>> > > > > > > > > >> >> > > introduce user visible changes. We could
> >> > >>> > > > > > > > > >> >> > > also
> >> > >>> > update
> >> > >>> > > > > KIP-48
> >> > >>> > > > > > > to
> >> > >>> > > > > > > > > >> have this
> >> > >>> > > > > > > > > >> >> > > information, but it seems cleaner to do
> it
> >> > >>> > > separately.
> >> > >>> > > > We
> >> > >>> > > > > > can
> >> > >>> > > > > > > > > >> discuss
> >> > >>> > > > > > > > > >> >> > that
> >> > >>> > > > > > > > > >> >> > > in the KIP call today.
> >> > >>> > > > > > > > > >> >> > >
> >> > >>> > > > > > > > > >> >> > > Ismael
> >> > >>> > > > > > > > > >> >> > >
> >> > >>> > > > > > > > > >> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini
> >> > Sivaram
> >> > >>> <
> >> > >>> > > > > > > > > >> >> > > rajinisivaram@googlemail.com> wrote:
> >> > >>> > > > > > > > > >> >> > >
> >> > >>> > > > > > > > > >> >> > > > Ismael,
> >> > >>> > > > > > > > > >> >> > > >
> >> > >>> > > > > > > > > >> >> > > > I have created a JIRA (
> >> > >>> > > > > > > > > >> >> > https://issues.apache.org/
> >> > jira/browse/KAFKA-3751)
> >> > >>> > > > > > > > > >> >> > > > for adding SCRAM as a SASL mechanism.
> >> > >>> > > > > > > > > >> >> > > > Would
> >> > >>> that
> >> > >>> > > need
> >> > >>> > > > > > > another
> >> > >>> > > > > > > > > >> KIP? If
> >> > >>> > > > > > > > > >> >> > > > KIP-48 will use this mechanism, can
> this
> >> > just
> >> > >>> be
> >> > >>> > a
> >> > >>> > > > JIRA
> >> > >>> > > > > > > that
> >> > >>> > > > > > > > > gets
> >> > >>> > > > > > > > > >> >> > > reviewed
> >> > >>> > > > > > > > > >> >> > > > when the PR is ready?
> >> > >>> > > > > > > > > >> >> > > >
> >> > >>> > > > > > > > > >> >> > > > Thank you,
> >> > >>> > > > > > > > > >> >> > > >
> >> > >>> > > > > > > > > >> >> > > > Rajini
> >> > >>> > > > > > > > > >> >> > > >
> >> > >>> > > > > > > > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM,
> Ismael
> >> > Juma <
> >> > >>> > > > > > > > > ismael@juma.me.uk>
> >> > >>> > > > > > > > > >> >> > wrote:
> >> > >>> > > > > > > > > >> >> > > >
> >> > >>> > > > > > > > > >> >> > > > > Thanks Rajini, SCRAM seems like a
> good
> >> > >>> > candidate.
> >> > >>> > > > > > > > > >> >> > > > >
> >> > >>> > > > > > > > > >> >> > > > > Gwen had independently mentioned
> this
> >> > >>> > > > > > > > > >> >> > > > > as
> >> > a
> >> > >>> SASL
> >> > >>> > > > > > mechanism
> >> > >>> > > > > > > > > that
> >> > >>> > > > > > > > > >> might
> >> > >>> > > > > > > > > >> >> > be
> >> > >>> > > > > > > > > >> >> > > > > useful for Kafka and I have been
> >> > >>> > > > > > > > > >> >> > > > > meaning
> >> > to
> >> > >>> > check
> >> > >>> > > > it
> >> > >>> > > > > in
> >> > >>> > > > > > > > more
> >> > >>> > > > > > > > > >> detail.
> >> > >>> > > > > > > > > >> >> > > Good
> >> > >>> > > > > > > > > >> >> > > > > to know that you are willing to
> >> > contribute
> >> > >>> an
> >> > >>> > > > > > > > implementation.
> >> > >>> > > > > > > > > >> Maybe
> >> > >>> > > > > > > > > >> >> > we
> >> > >>> > > > > > > > > >> >> > > > > should file a separate JIRA for
> this?
> >> > >>> > > > > > > > > >> >> > > > >
> >> > >>> > > > > > > > > >> >> > > > > Ismael
> >> > >>> > > > > > > > > >> >> > > > >
> >> > >>> > > > > > > > > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM,
> >> > >>> > > > > > > > > >> >> > > > > Rajini
> >> > >>> > Sivaram <
> >> > >>> > > > > > > > > >> >> > > > > rajinisivaram@googlemail.com>
> wrote:
> >> > >>> > > > > > > > > >> >> > > > >
> >> > >>> > > > > > > > > >> >> > > > > > SCRAM (Salted Challenge Response
> >> > >>> > Authentication
> >> > >>> > > > > > > > Mechanism)
> >> > >>> > > > > > > > > >> is a
> >> > >>> > > > > > > > > >> >> > > better
> >> > >>> > > > > > > > > >> >> > > > > > mechanism than Digest-MD5. Java
> >> > >>> > > > > > > > > >> >> > > > > > doesn't
> >> > >>> come
> >> > >>> > > > with a
> >> > >>> > > > > > > > > built-in
> >> > >>> > > > > > > > > >> SCRAM
> >> > >>> > > > > > > > > >> >> > > > > > SaslServer or SaslClient, but I
> will
> >> > >>> > > > > > > > > >> >> > > > > > be
> >> > >>> happy
> >> > >>> > > to
> >> > >>> > > > > add
> >> > >>> > > > > > > > > support
> >> > >>> > > > > > > > > >> in
> >> > >>> > > > > > > > > >> >> > Kafka
> >> > >>> > > > > > > > > >> >> > > > > since
> >> > >>> > > > > > > > > >> >> > > > > > it would be a useful mechanism to
> >> > support
> >> > >>> > > anyway.
> >> > >>> > > > > > > > > >> >> > > > > > https://tools.ietf.org/html/
> rfc7677
> >> > >>> > describes
> >> > >>> > > > the
> >> > >>> > > > > > > > protocol
> >> > >>> > > > > > > > > >> for
> >> > >>> > > > > > > > > >> >> > > > > > SCRAM-SHA-256.
> >> > >>> > > > > > > > > >> >> > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM,
> Jun
> >> > Rao <
> >> > >>> > > > > > > > jun@confluent.io
> >> > >>> > > > > > > > > >
> >> > >>> > > > > > > > > >> wrote:
> >> > >>> > > > > > > > > >> >> > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > Parth,
> >> > >>> > > > > > > > > >> >> > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > Thanks for the explanation. A
> >> > >>> > > > > > > > > >> >> > > > > > > couple
> >> > of
> >> > >>> > more
> >> > >>> > > > > > > questions.
> >> > >>> > > > > > > > > >> >> > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > 110. What does
> >> > >>> > > > > > > > > >> >> > > > > > > getDelegationTokenAs
> >> > >>> mean?
> >> > >>> > > > > > > > > >> >> > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > 111. What's the typical rate of
> >> > getting
> >> > >>> and
> >> > >>> > > > > > renewing
> >> > >>> > > > > > > > > >> delegation
> >> > >>> > > > > > > > > >> >> > > > tokens?
> >> > >>> > > > > > > > > >> >> > > > > > > That may have an impact on
> whether
> >> > they
> >> > >>> > > should
> >> > >>> > > > be
> >> > >>> > > > > > > > > directed
> >> > >>> > > > > > > > > >> to the
> >> > >>> > > > > > > > > >> >> > > > > > > controller.
> >> > >>> > > > > > > > > >> >> > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > Jun
> >> > >>> > > > > > > > > >> >> > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM,
> >> > parth
> >> > >>> > > > > brahmbhatt <
> >> > >>> > > > > > > > > >> >> > > > > > > brahmbhatt.parth@gmail.com>
> wrote:
> >> > >>> > > > > > > > > >> >> > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > Hi Jun,
> >> > >>> > > > > > > > > >> >> > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > Thanks for reviewing.
> >> > >>> > > > > > > > > >> >> > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > * We could add a Cluster
> action
> >> > >>> > > > > > > > > >> >> > > > > > > > to
> >> > add
> >> > >>> > acls
> >> > >>> > > > on
> >> > >>> > > > > > who
> >> > >>> > > > > > > > can
> >> > >>> > > > > > > > > >> request
> >> > >>> > > > > > > > > >> >> > > > > > delegation
> >> > >>> > > > > > > > > >> >> > > > > > > > tokens. I don't see the use
> case
> >> > for
> >> > >>> that
> >> > >>> > > yet
> >> > >>> > > > > but
> >> > >>> > > > > > > > down
> >> > >>> > > > > > > > > >> the line
> >> > >>> > > > > > > > > >> >> > > > when
> >> > >>> > > > > > > > > >> >> > > > > we
> >> > >>> > > > > > > > > >> >> > > > > > > > start supporting
> >> > getDelegationTokenAs
> >> > >>> it
> >> > >>> > > will
> >> > >>> > > > > be
> >> > >>> > > > > > > > > >> necessary.
> >> > >>> > > > > > > > > >> >> > > > > > > > * Yes we recommend tokens to
> be
> >> > only
> >> > >>> > > > > > > used/distributed
> >> > >>> > > > > > > > > >> over
> >> > >>> > > > > > > > > >> >> > secure
> >> > >>> > > > > > > > > >> >> > > > > > > channels.
> >> > >>> > > > > > > > > >> >> > > > > > > > * Depending on what design we
> >> > >>> > > > > > > > > >> >> > > > > > > > end
> >> > up
> >> > >>> > > choosing
> >> > >>> > > > > > > > > >> Invalidation will
> >> > >>> > > > > > > > > >> >> > > be
> >> > >>> > > > > > > > > >> >> > > > > > > > responsibility of every broker
> >> > >>> > > > > > > > > >> >> > > > > > > > or
> >> > >>> > > controller.
> >> > >>> > > > > > > > > >> >> > > > > > > > * I am not sure if I
> documented
> >> > >>> somewhere
> >> > >>> > > > that
> >> > >>> > > > > > > > > >> invalidation
> >> > >>> > > > > > > > > >> >> > will
> >> > >>> > > > > > > > > >> >> > > > > > directly
> >> > >>> > > > > > > > > >> >> > > > > > > > go through zookeeper but that
> is
> >> > not
> >> > >>> the
> >> > >>> > > > > intent.
> >> > >>> > > > > > > > > >> Invalidation
> >> > >>> > > > > > > > > >> >> > > will
> >> > >>> > > > > > > > > >> >> > > > > > either
> >> > >>> > > > > > > > > >> >> > > > > > > > be request based or due to
> >> > >>> expiration. No
> >> > >>> > > > > direct
> >> > >>> > > > > > > > > >> zookeeper
> >> > >>> > > > > > > > > >> >> > > > > interaction
> >> > >>> > > > > > > > > >> >> > > > > > > from
> >> > >>> > > > > > > > > >> >> > > > > > > > any client.
> >> > >>> > > > > > > > > >> >> > > > > > > > * "Broker also stores the
> >> > >>> DelegationToken
> >> > >>> > > > > without
> >> > >>> > > > > > > the
> >> > >>> > > > > > > > > >> hmac in
> >> > >>> > > > > > > > > >> >> > the
> >> > >>> > > > > > > > > >> >> > > > > > > > zookeeper." : Sorry about the
> >> > >>> confusion.
> >> > >>> > > The
> >> > >>> > > > > sole
> >> > >>> > > > > > > > > >> purpose of
> >> > >>> > > > > > > > > >> >> > > > > zookeeper
> >> > >>> > > > > > > > > >> >> > > > > > in
> >> > >>> > > > > > > > > >> >> > > > > > > > this design is as distribution
> >> > channel
> >> > >>> > for
> >> > >>> > > > > tokens
> >> > >>> > > > > > > > > >> between all
> >> > >>> > > > > > > > > >> >> > > > brokers
> >> > >>> > > > > > > > > >> >> > > > > > > and a
> >> > >>> > > > > > > > > >> >> > > > > > > > layer that ensures only tokens
> >> > >>> > > > > > > > > >> >> > > > > > > > that
> >> > >>> were
> >> > >>> > > > > > generated
> >> > >>> > > > > > > by
> >> > >>> > > > > > > > > >> making a
> >> > >>> > > > > > > > > >> >> > > > > request
> >> > >>> > > > > > > > > >> >> > > > > > > to a
> >> > >>> > > > > > > > > >> >> > > > > > > > broker will be accepted (more
> on
> >> > this
> >> > >>> in
> >> > >>> > > > second
> >> > >>> > > > > > > > > >> paragraph). The
> >> > >>> > > > > > > > > >> >> > > > token
> >> > >>> > > > > > > > > >> >> > > > > > > > consists of few elements
> (owner,
> >> > >>> renewer,
> >> > >>> > > > uuid
> >> > >>> > > > > ,
> >> > >>> > > > > > > > > >> expiration,
> >> > >>> > > > > > > > > >> >> > > hmac)
> >> > >>> > > > > > > > > >> >> > > > ,
> >> > >>> > > > > > > > > >> >> > > > > > one
> >> > >>> > > > > > > > > >> >> > > > > > > of
> >> > >>> > > > > > > > > >> >> > > > > > > > which is the finally generated
> >> > >>> > > > > > > > > >> >> > > > > > > > hmac
> >> > >>> but
> >> > >>> > > hmac
> >> > >>> > > > it
> >> > >>> > > > > > > self
> >> > >>> > > > > > > > is
> >> > >>> > > > > > > > > >> >> > derivable
> >> > >>> > > > > > > > > >> >> > > > if
> >> > >>> > > > > > > > > >> >> > > > > > you
> >> > >>> > > > > > > > > >> >> > > > > > > > have all the other elements of
> >> > >>> > > > > > > > > >> >> > > > > > > > the
> >> > >>> token
> >> > >>> > +
> >> > >>> > > > > secret
> >> > >>> > > > > > > key
> >> > >>> > > > > > > > > to
> >> > >>> > > > > > > > > >> >> > generate
> >> > >>> > > > > > > > > >> >> > > > > hmac.
> >> > >>> > > > > > > > > >> >> > > > > > > > Given zookeeper does not
> provide
> >> > SSL
> >> > >>> > > support
> >> > >>> > > > we
> >> > >>> > > > > > do
> >> > >>> > > > > > > > not
> >> > >>> > > > > > > > > >> want the
> >> > >>> > > > > > > > > >> >> > > > > entire
> >> > >>> > > > > > > > > >> >> > > > > > > > token to be wire transferred
> to
> >> > >>> zookeeper
> >> > >>> > > as
> >> > >>> > > > > that
> >> > >>> > > > > > > > will
> >> > >>> > > > > > > > > >> be an
> >> > >>> > > > > > > > > >> >> > > > insecure
> >> > >>> > > > > > > > > >> >> > > > > > > wire
> >> > >>> > > > > > > > > >> >> > > > > > > > transfer. Instead we only
> store
> >> > >>> > > > > > > > > >> >> > > > > > > > all
> >> > >>> the
> >> > >>> > > other
> >> > >>> > > > > > > > elements
> >> > >>> > > > > > > > > >> of a
> >> > >>> > > > > > > > > >> >> > > > > delegation
> >> > >>> > > > > > > > > >> >> > > > > > > > tokens. Brokers can read these
> >> > >>> elements
> >> > >>> > and
> >> > >>> > > > > > because
> >> > >>> > > > > > > > > they
> >> > >>> > > > > > > > > >> also
> >> > >>> > > > > > > > > >> >> > > have
> >> > >>> > > > > > > > > >> >> > > > > > access
> >> > >>> > > > > > > > > >> >> > > > > > > > to secret key they will be
> able
> >> > >>> > > > > > > > > >> >> > > > > > > > to
> >> > >>> > generate
> >> > >>> > > > > hmac
> >> > >>> > > > > > on
> >> > >>> > > > > > > > > >> their end.
> >> > >>> > > > > > > > > >> >> > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > One of the alternative
> proposed
> >> > >>> > > > > > > > > >> >> > > > > > > > is
> >> > to
> >> > >>> > avoid
> >> > >>> > > > > > > zookeeper
> >> > >>> > > > > > > > > >> >> > > altogether. A
> >> > >>> > > > > > > > > >> >> > > > > > > Client
> >> > >>> > > > > > > > > >> >> > > > > > > > will call broker with required
> >> > >>> > information
> >> > >>> > > > > > (owner,
> >> > >>> > > > > > > > > >> renwer,
> >> > >>> > > > > > > > > >> >> > > > > expiration)
> >> > >>> > > > > > > > > >> >> > > > > > > and
> >> > >>> > > > > > > > > >> >> > > > > > > > get back (signed hmac, uuid).
> >> > Broker
> >> > >>> > won't
> >> > >>> > > > > store
> >> > >>> > > > > > > this
> >> > >>> > > > > > > > > in
> >> > >>> > > > > > > > > >> >> > > zookeeper.
> >> > >>> > > > > > > > > >> >> > > > > > From
> >> > >>> > > > > > > > > >> >> > > > > > > > this point a client can
> contact
> >> > >>> > > > > > > > > >> >> > > > > > > > any
> >> > >>> > broker
> >> > >>> > > > with
> >> > >>> > > > > > all
> >> > >>> > > > > > > > the
> >> > >>> > > > > > > > > >> >> > > delegation
> >> > >>> > > > > > > > > >> >> > > > > > token
> >> > >>> > > > > > > > > >> >> > > > > > > > info (owner, rewner,
> expiration,
> >> > hmac,
> >> > >>> > > uuid)
> >> > >>> > > > > the
> >> > >>> > > > > > > > borker
> >> > >>> > > > > > > > > >> will
> >> > >>> > > > > > > > > >> >> > > > > regenerate
> >> > >>> > > > > > > > > >> >> > > > > > > the
> >> > >>> > > > > > > > > >> >> > > > > > > > hmac and as long as it matches
> >> > >>> > > > > > > > > >> >> > > > > > > > with
> >> > >>> hmac
> >> > >>> > > > > > presented
> >> > >>> > > > > > > by
> >> > >>> > > > > > > > > >> client ,
> >> > >>> > > > > > > > > >> >> > > > broker
> >> > >>> > > > > > > > > >> >> > > > > > > will
> >> > >>> > > > > > > > > >> >> > > > > > > > allow the request to
> >> > >>> > > > > > > > > >> >> > > > > > > > authenticate.
> >> > >>> Only
> >> > >>> > > > > problem
> >> > >>> > > > > > > with
> >> > >>> > > > > > > > > >> this
> >> > >>> > > > > > > > > >> >> > > approach
> >> > >>> > > > > > > > > >> >> > > > > is
> >> > >>> > > > > > > > > >> >> > > > > > if
> >> > >>> > > > > > > > > >> >> > > > > > > > the secret key is compromised
> >> > >>> > > > > > > > > >> >> > > > > > > > any
> >> > >>> client
> >> > >>> > > can
> >> > >>> > > > > now
> >> > >>> > > > > > > > > generate
> >> > >>> > > > > > > > > >> >> > random
> >> > >>> > > > > > > > > >> >> > > > > tokens
> >> > >>> > > > > > > > > >> >> > > > > > > and
> >> > >>> > > > > > > > > >> >> > > > > > > > they will still be able to
> >> > >>> authenticate
> >> > >>> > as
> >> > >>> > > > any
> >> > >>> > > > > > user
> >> > >>> > > > > > > > > they
> >> > >>> > > > > > > > > >> like.
> >> > >>> > > > > > > > > >> >> > > with
> >> > >>> > > > > > > > > >> >> > > > > > > > zookeeper we guarantee that
> only
> >> > >>> tokens
> >> > >>> > > > > acquired
> >> > >>> > > > > > > via
> >> > >>> > > > > > > > a
> >> > >>> > > > > > > > > >> broker
> >> > >>> > > > > > > > > >> >> > > > (using
> >> > >>> > > > > > > > > >> >> > > > > > some
> >> > >>> > > > > > > > > >> >> > > > > > > > auth scheme other than
> >> > >>> > > > > > > > > >> >> > > > > > > > delegation
> >> > >>> token)
> >> > >>> > > will
> >> > >>> > > > > be
> >> > >>> > > > > > > > > >> accepted. We
> >> > >>> > > > > > > > > >> >> > > need
> >> > >>> > > > > > > > > >> >> > > > to
> >> > >>> > > > > > > > > >> >> > > > > > > > discuss which proposal makes
> >> > >>> > > > > > > > > >> >> > > > > > > > more
> >> > >>> sense
> >> > >>> > and
> >> > >>> > > > we
> >> > >>> > > > > > can
> >> > >>> > > > > > > go
> >> > >>> > > > > > > > > >> over it
> >> > >>> > > > > > > > > >> >> > in
> >> > >>> > > > > > > > > >> >> > > > > > > tomorrow's
> >> > >>> > > > > > > > > >> >> > > > > > > > meeting.
> >> > >>> > > > > > > > > >> >> > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > Also, can you forward the
> invite
> >> > >>> > > > > > > > > >> >> > > > > > > > to
> >> > >>> me?
> >> > >>> > > > > > > > > >> >> > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > Thanks
> >> > >>> > > > > > > > > >> >> > > > > > > > Parth
> >> > >>> > > > > > > > > >> >> > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > On Mon, May 23, 2016 at 10:35
> >> > >>> > > > > > > > > >> >> > > > > > > > AM,
> >> > Jun
> >> > >>> > Rao <
> >> > >>> > > > > > > > > >> jun@confluent.io>
> >> > >>> > > > > > > > > >> >> > > > wrote:
> >> > >>> > > > > > > > > >> >> > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > Thanks for the KIP. A few
> >> > comments.
> >> > >>> > > > > > > > > >> >> > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > 100. This potentially can be
> >> > useful
> >> > >>> for
> >> > >>> > > > Kafka
> >> > >>> > > > > > > > Connect
> >> > >>> > > > > > > > > >> and
> >> > >>> > > > > > > > > >> >> > Kafka
> >> > >>> > > > > > > > > >> >> > > > > rest
> >> > >>> > > > > > > > > >> >> > > > > > > > proxy
> >> > >>> > > > > > > > > >> >> > > > > > > > > where a worker agent will
> need
> >> > >>> > > > > > > > > >> >> > > > > > > > > to
> >> > >>> run a
> >> > >>> > > > task
> >> > >>> > > > > on
> >> > >>> > > > > > > > > behalf
> >> > >>> > > > > > > > > >> of a
> >> > >>> > > > > > > > > >> >> > > > client.
> >> > >>> > > > > > > > > >> >> > > > > > We
> >> > >>> > > > > > > > > >> >> > > > > > > > will
> >> > >>> > > > > > > > > >> >> > > > > > > > > likely need to change how
> >> > >>> > > > > > > > > >> >> > > > > > > > > those
> >> > >>> > services
> >> > >>> > > > use
> >> > >>> > > > > > > Kafka
> >> > >>> > > > > > > > > >> clients
> >> > >>> > > > > > > > > >> >> > > > > > > > > (producer/consumer). Instead
> >> > >>> > > > > > > > > >> >> > > > > > > > > of a
> >> > >>> > shared
> >> > >>> > > > > client
> >> > >>> > > > > > > per
> >> > >>> > > > > > > > > >> worker,
> >> > >>> > > > > > > > > >> >> > we
> >> > >>> > > > > > > > > >> >> > > > will
> >> > >>> > > > > > > > > >> >> > > > > > > need
> >> > >>> > > > > > > > > >> >> > > > > > > > a
> >> > >>> > > > > > > > > >> >> > > > > > > > > client per user task since
> the
> >> > >>> > > > authentication
> >> > >>> > > > > > > > happens
> >> > >>> > > > > > > > > >> at the
> >> > >>> > > > > > > > > >> >> > > > > > connection
> >> > >>> > > > > > > > > >> >> > > > > > > > > level. For Kafka Connect,
> the
> >> > >>> renewer
> >> > >>> > > will
> >> > >>> > > > be
> >> > >>> > > > > > the
> >> > >>> > > > > > > > > >> workers.
> >> > >>> > > > > > > > > >> >> > So,
> >> > >>> > > > > > > > > >> >> > > we
> >> > >>> > > > > > > > > >> >> > > > > > > > probably
> >> > >>> > > > > > > > > >> >> > > > > > > > > need to allow multiple
> >> > >>> > > > > > > > > >> >> > > > > > > > > renewers.
> >> > For
> >> > >>> > > Kafka
> >> > >>> > > > > rest
> >> > >>> > > > > > > > > proxy,
> >> > >>> > > > > > > > > >> the
> >> > >>> > > > > > > > > >> >> > > > renewer
> >> > >>> > > > > > > > > >> >> > > > > > can
> >> > >>> > > > > > > > > >> >> > > > > > > > > probably just be the creator
> >> > >>> > > > > > > > > >> >> > > > > > > > > of
> >> > the
> >> > >>> > > token.
> >> > >>> > > > > > > > > >> >> > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > 101. Do we need new acl on
> who
> >> > can
> >> > >>> > > request
> >> > >>> > > > > > > > delegation
> >> > >>> > > > > > > > > >> tokens?
> >> > >>> > > > > > > > > >> >> > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > 102. Do we recommend people
> to
> >> > send
> >> > >>> > > > > delegation
> >> > >>> > > > > > > > tokens
> >> > >>> > > > > > > > > >> in an
> >> > >>> > > > > > > > > >> >> > > > > encrypted
> >> > >>> > > > > > > > > >> >> > > > > > > > > channel?
> >> > >>> > > > > > > > > >> >> > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > 103. Who is responsible for
> >> > expiring
> >> > >>> > > > tokens,
> >> > >>> > > > > > > every
> >> > >>> > > > > > > > > >> broker?
> >> > >>> > > > > > > > > >> >> > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > 104. For invalidating
> tokens,
> >> > would
> >> > >>> it
> >> > >>> > be
> >> > >>> > > > > > better
> >> > >>> > > > > > > to
> >> > >>> > > > > > > > > do
> >> > >>> > > > > > > > > >> it in
> >> > >>> > > > > > > > > >> >> > a
> >> > >>> > > > > > > > > >> >> > > > > > request
> >> > >>> > > > > > > > > >> >> > > > > > > > > instead of going to ZK
> >> > >>> > > > > > > > > >> >> > > > > > > > > directly?
> >> > >>> > > > > > > > > >> >> > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > 105. The terminology of
> client
> >> > >>> > > > > > > > > >> >> > > > > > > > > in
> >> > >>> the
> >> > >>> > > wiki
> >> > >>> > > > > > > > sometimes
> >> > >>> > > > > > > > > >> refers
> >> > >>> > > > > > > > > >> >> > to
> >> > >>> > > > > > > > > >> >> > > > the
> >> > >>> > > > > > > > > >> >> > > > > > end
> >> > >>> > > > > > > > > >> >> > > > > > > > > client and some other times
> >> > refers
> >> > >>> to
> >> > >>> > the
> >> > >>> > > > > > client
> >> > >>> > > > > > > > > using
> >> > >>> > > > > > > > > >> the
> >> > >>> > > > > > > > > >> >> > > > > delegation
> >> > >>> > > > > > > > > >> >> > > > > > > > > tokens. It would be useful
> to
> >> > >>> > distinguish
> >> > >>> > > > > > between
> >> > >>> > > > > > > > the
> >> > >>> > > > > > > > > >> two.
> >> > >>> > > > > > > > > >> >> > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > 106. Could you explain the
> >> > sentence
> >> > >>> > > "Broker
> >> > >>> > > > > > also
> >> > >>> > > > > > > > > >> stores the
> >> > >>> > > > > > > > > >> >> > > > > > > > DelegationToken
> >> > >>> > > > > > > > > >> >> > > > > > > > > without the hmac in the
> >> > zookeeper."
> >> > >>> a
> >> > >>> > bit
> >> > >>> > > > > > more? I
> >> > >>> > > > > > > > > >> thought the
> >> > >>> > > > > > > > > >> >> > > > > > > delegation
> >> > >>> > > > > > > > > >> >> > > > > > > > > token is the hmac.
> >> > >>> > > > > > > > > >> >> > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > Thanks,
> >> > >>> > > > > > > > > >> >> > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > Jun
> >> > >>> > > > > > > > > >> >> > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > On Mon, May 23, 2016 at 9:22
> >> > >>> > > > > > > > > >> >> > > > > > > > > AM,
> >> > Jun
> >> > >>> > Rao
> >> > >>> > > <
> >> > >>> > > > > > > > > >> jun@confluent.io>
> >> > >>> > > > > > > > > >> >> > > > wrote:
> >> > >>> > > > > > > > > >> >> > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > > Hi, Harsha,
> >> > >>> > > > > > > > > >> >> > > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > > Just sent out a KIP
> meeting
> >> > >>> invite.
> >> > >>> > We
> >> > >>> > > > can
> >> > >>> > > > > > > > discuss
> >> > >>> > > > > > > > > >> this in
> >> > >>> > > > > > > > > >> >> > > the
> >> > >>> > > > > > > > > >> >> > > > > > > meeting
> >> > >>> > > > > > > > > >> >> > > > > > > > > > tomorrow.
> >> > >>> > > > > > > > > >> >> > > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > > Thanks,
> >> > >>> > > > > > > > > >> >> > > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > > Jun
> >> > >>> > > > > > > > > >> >> > > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > > On Thu, May 19, 2016 at
> 8:47
> >> > AM,
> >> > >>> > > Harsha <
> >> > >>> > > > > > > > > >> kafka@harsha.io>
> >> > >>> > > > > > > > > >> >> > > > wrote:
> >> > >>> > > > > > > > > >> >> > > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> Hi All,
> >> > >>> > > > > > > > > >> >> > > > > > > > > >>            Can we have a
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> KIP
> >> > >>> meeting
> >> > >>> > > > > around
> >> > >>> > > > > > > > this.
> >> > >>> > > > > > > > > >> The KIP
> >> > >>> > > > > > > > > >> >> > is
> >> > >>> > > > > > > > > >> >> > > > up
> >> > >>> > > > > > > > > >> >> > > > > > for
> >> > >>> > > > > > > > > >> >> > > > > > > > > >>            sometime and
> if
> >> > there
> >> > >>> are
> >> > >>> > > any
> >> > >>> > > > > > > > questions
> >> > >>> > > > > > > > > >> lets
> >> > >>> > > > > > > > > >> >> > > > quickly
> >> > >>> > > > > > > > > >> >> > > > > > hash
> >> > >>> > > > > > > > > >> >> > > > > > > > out
> >> > >>> > > > > > > > > >> >> > > > > > > > > >>            details.
> >> > >>> > > > > > > > > >> >> > > > > > > > > >>
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> Thanks,
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> Harsha
> >> > >>> > > > > > > > > >> >> > > > > > > > > >>
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> On Thu, May 19, 2016, at
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> 08:40
> >> > >>> AM,
> >> > >>> > > parth
> >> > >>> > > > > > > > > brahmbhatt
> >> > >>> > > > > > > > > >> wrote:
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > That is what the hadoop
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > echo
> >> > >>> > system
> >> > >>> > > > uses
> >> > >>> > > > > > so
> >> > >>> > > > > > > no
> >> > >>> > > > > > > > > >> good
> >> > >>> > > > > > > > > >> >> > reason
> >> > >>> > > > > > > > > >> >> > > > > > really.
> >> > >>> > > > > > > > > >> >> > > > > > > > We
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > could
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > change it to whatever
> is
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > the
> >> > >>> > newest
> >> > >>> > > > > > > > recommended
> >> > >>> > > > > > > > > >> standard
> >> > >>> > > > > > > > > >> >> > > is.
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> >
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > Thanks
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > Parth
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> >
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > On Thu, May 19, 2016 at
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > 3:33
> >> > >>> AM,
> >> > >>> > > > Ismael
> >> > >>> > > > > > > Juma <
> >> > >>> > > > > > > > > >> >> > > > > ismael@juma.me.uk
> >> > >>> > > > > > > > > >> >> > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > wrote:
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> >
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > Hi Parth,
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > Thanks for the KIP. I
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > only
> >> > >>> > started
> >> > >>> > > > > > > reviewing
> >> > >>> > > > > > > > > >> this and
> >> > >>> > > > > > > > > >> >> > > may
> >> > >>> > > > > > > > > >> >> > > > > have
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> additional
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > questions later. The
> >> > >>> immediate
> >> > >>> > > > > question
> >> > >>> > > > > > > that
> >> > >>> > > > > > > > > >> came to
> >> > >>> > > > > > > > > >> >> > > mind
> >> > >>> > > > > > > > > >> >> > > > is
> >> > >>> > > > > > > > > >> >> > > > > > our
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> choice of
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > "DIGEST-MD5" even
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > though
> >> > it's
> >> > >>> > > marked
> >> > >>> > > > > as
> >> > >>> > > > > > > > > >> OBSOLETE in
> >> > >>> > > > > > > > > >> >> > the
> >> > >>> > > > > > > > > >> >> > > > IANA
> >> > >>> > > > > > > > > >> >> > > > > > > > > Registry
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> of
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > SASL mechanisms and
> the
> >> > >>> original
> >> > >>> > > RFC
> >> > >>> > > > > > > (2831)
> >> > >>> > > > > > > > > has
> >> > >>> > > > > > > > > >> been
> >> > >>> > > > > > > > > >> >> > > moved
> >> > >>> > > > > > > > > >> >> > > > > to
> >> > >>> > > > > > > > > >> >> > > > > > > > > Historic
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > status:
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >> > https://tools.ietf.org/html/
> >> > >>> > > rfc6331
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >> > >>> > > > > > > > > >> >> > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > >
> >> > >>> > > > > > > > > >> >> > >
> >> > >>> > > > > > > > > >>
> >> > >>> > > > > > >
> >> > >>> > > > http://www.iana.org/assignments/sasl-mechanisms/sasl-
> >> > >>> mechanisms.xhtml
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > What is the reasoning
> >> > behind
> >> > >>> > that
> >> > >>> > > > > > choice?
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > Thanks,
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > Ismael
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > On Fri, May 13, 2016
> at
> >> > 11:29
> >> > >>> > PM,
> >> > >>> > > > Gwen
> >> > >>> > > > > > > > > Shapira <
> >> > >>> > > > > > > > > >> >> > > > > > > gwen@confluent.io
> >> > >>> > > > > > > > > >> >> > > > > > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> wrote:
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > Also comments
> inline
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > :)
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > * I want to
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > emphasize
> >> > >>> that
> >> > >>> > > even
> >> > >>> > > > > > though
> >> > >>> > > > > > > > > >> delegation
> >> > >>> > > > > > > > > >> >> > > > tokens
> >> > >>> > > > > > > > > >> >> > > > > > > are a
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> Hadoop
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > innovation, I
> feel
> >> > very
> >> > >>> > > strongly
> >> > >>> > > > > > about
> >> > >>> > > > > > > > not
> >> > >>> > > > > > > > > >> adding
> >> > >>> > > > > > > > > >> >> > > > > > dependency
> >> > >>> > > > > > > > > >> >> > > > > > > > on
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> Hadoop
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > when implementing
> >> > >>> delegation
> >> > >>> > > > > tokens
> >> > >>> > > > > > > for
> >> > >>> > > > > > > > > >> Kafka. The
> >> > >>> > > > > > > > > >> >> > > KIP
> >> > >>> > > > > > > > > >> >> > > > > > > doesn't
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> imply
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > such dependency,
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > but
> >> > if
> >> > >>> you
> >> > >>> > > can
> >> > >>> > > > > > > > clarify...
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *No hadoop
> >> > dependency.*
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > Yay! Just add this
> to
> >> > the
> >> > >>> KIP
> >> > >>> > so
> >> > >>> > > > no
> >> > >>> > > > > > one
> >> > >>> > > > > > > > will
> >> > >>> > > > > > > > > >> read
> >> > >>> > > > > > > > > >> >> > the
> >> > >>> > > > > > > > > >> >> > > > KIP
> >> > >>> > > > > > > > > >> >> > > > > > and
> >> > >>> > > > > > > > > >> >> > > > > > > > > panic
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > three weeks before
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > the
> >> > next
> >> > >>> > > > > release...
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > * Can we get
> >> > delegation
> >> > >>> > token
> >> > >>> > > at
> >> > >>> > > > > any
> >> > >>> > > > > > > > time
> >> > >>> > > > > > > > > >> after
> >> > >>> > > > > > > > > >> >> > > > > > > > authenticating?
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> only
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > immediately
> after?
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *As long as you
> are
> >> > >>> > > > authenticated
> >> > >>> > > > > > you
> >> > >>> > > > > > > > can
> >> > >>> > > > > > > > > >> get
> >> > >>> > > > > > > > > >> >> > > > delegation
> >> > >>> > > > > > > > > >> >> > > > > > > > tokens.
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> We
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > need
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > to
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > discuss if a
> client
> >> > >>> > > > authenticated
> >> > >>> > > > > > > using
> >> > >>> > > > > > > > > >> delegation
> >> > >>> > > > > > > > > >> >> > > > > token,
> >> > >>> > > > > > > > > >> >> > > > > > > can
> >> > >>> > > > > > > > > >> >> > > > > > > > > also
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > acquire
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > delegation token
> >> > again or
> >> > >>> > not.
> >> > >>> > > > > Also
> >> > >>> > > > > > > > there
> >> > >>> > > > > > > > > >> is the
> >> > >>> > > > > > > > > >> >> > > > > question
> >> > >>> > > > > > > > > >> >> > > > > > of
> >> > >>> > > > > > > > > >> >> > > > > > > > do
> >> > >>> > > > > > > > > >> >> > > > > > > > > we
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > allow
> >> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > anyone to acquire
> >> > >>> delegation
> >> > >>> > > > token
> >> > >>> > > > > > or
> >> > >>> > > > > > > we
> >> > >>> > > > > > > > > >> want
> >> > >>> > > > > > > > > >> >> > > specific
> >> > >>> > > > > > > > > >>
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Gwen Shapira <gw...@confluent.io>]

Is it updated? are all concerns addressed? do you want to start a vote?

Sorry for being pushy, I do appreciate that we are all volunteers and
finding time is difficult. This feature is important for anything that
integrates with Kafka (stream processors, Flume, NiFi, etc) and I
don't want to see this getting stuck because we lack coordination
within the community.

On Thu, Sep 15, 2016 at 6:39 PM, Harsha Chintalapani <ka...@harsha.io> wrote:
> The only pending update for the KIP is to write up the protocol changes like
> we've it KIP-4. I'll update the wiki.
>
>
> On Thu, Sep 15, 2016 at 4:27 PM Ashish Singh <as...@cloudera.com> wrote:
>>
>> I think we decided to not support secret rotation, I guess this can be
>> stated clearly on the KIP. Also, more details on how clients will perform
>> token distribution and how CLI will look like will be helpful.
>>
>> On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <gw...@confluent.io> wrote:
>>
>> > Hi Guys,
>> >
>> > This discussion was dead for a while. Are there still contentious
>> > points? If not, why are there no votes?
>> >
>> > On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <ju...@confluent.io> wrote:
>> > > Ashish,
>> > >
>> > > Yes, I will send out a KIP invite for next week to discuss KIP-48 and
>> > other
>> > > remaining KIPs.
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > > On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh <as...@cloudera.com>
>> > wrote:
>> > >
>> > >> Thanks Harsha!
>> > >>
>> > >> Jun, can we add KIP-48 to next KIP hangout's agenda. Also, we did not
>> > >> actually make a call on when we should have next KIP call. As there
>> > >> are
>> > a
>> > >> few outstanding KIPs that could not be discussed this week, can we
>> > >> have
>> > a
>> > >> KIP hangout call next week?
>> > >>
>> > >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha Chintalapani
>> > >> <ka...@harsha.io>
>> > >> wrote:
>> > >>
>> > >>> Ashish,
>> > >>>         Yes we are working on it. Lets discuss in the next KIP
>> > >>> meeting.
>> > >>> I'll join.
>> > >>> -Harsha
>> > >>>
>> > >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh <as...@cloudera.com>
>> > >>> wrote:
>> > >>>
>> > >>> > Hello Harsha,
>> > >>> >
>> > >>> > Are you still working on this? Wondering if we can discuss this in
>> > next
>> > >>> KIP
>> > >>> > meeting, if you can join.
>> > >>> >
>> > >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha Chintalapani <
>> > kafka@harsha.io>
>> > >>> > wrote:
>> > >>> >
>> > >>> > > Hi Grant,
>> > >>> > >           We are working on it. Will add the details to KIP
>> > >>> > > about
>> > the
>> > >>> > > request protocol.
>> > >>> > >
>> > >>> > > Thanks,
>> > >>> > > Harsha
>> > >>> > >
>> > >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant Henke
>> > >>> > > <gh...@cloudera.com>
>> > >>> wrote:
>> > >>> > >
>> > >>> > > > Hi Parth,
>> > >>> > > >
>> > >>> > > > Are you still working on this? If you need any help please
>> > >>> > > > don't
>> > >>> > hesitate
>> > >>> > > > to ask.
>> > >>> > > >
>> > >>> > > > Thanks,
>> > >>> > > > Grant
>> > >>> > > >
>> > >>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <ju...@confluent.io>
>> > wrote:
>> > >>> > > >
>> > >>> > > > > Parth,
>> > >>> > > > >
>> > >>> > > > > Thanks for the reply.
>> > >>> > > > >
>> > >>> > > > > It makes sense to only allow the renewal by users that
>> > >>> authenticated
>> > >>> > > > using
>> > >>> > > > > *non* delegation token mechanism. Then, should we make the
>> > >>> renewal a
>> > >>> > > > list?
>> > >>> > > > > For example, in the case of rest proxy, it will be useful
>> > >>> > > > > for
>> > >>> every
>> > >>> > > > > instance of rest proxy to be able to renew the tokens.
>> > >>> > > > >
>> > >>> > > > > It would be clearer if we can document the request protocol
>> > like
>> > >>> > > > >
>> > >>> > > > >
>> > >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > >>> > >
>> > >>> > > 4+-+Command+line+and+centralized+administrative+operations#KIP-4-
>> > >>> > > Commandlineandcentralizedadministrativeoperations-
>> > >>> > > CreateTopicsRequest(KAFKA-2945):(VotedandPlannedforin0.10.1.0)
>> > >>> > > > > .
>> > >>> > > > >
>> > >>> > > > > It would also be useful to document the client APIs.
>> > >>> > > > >
>> > >>> > > > > Thanks,
>> > >>> > > > >
>> > >>> > > > > Jun
>> > >>> > > > >
>> > >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM, parth brahmbhatt <
>> > >>> > > > > brahmbhatt.parth@gmail.com> wrote:
>> > >>> > > > >
>> > >>> > > > > > Hi,
>> > >>> > > > > >
>> > >>> > > > > > I am suggesting that we will only allow the renewal by
>> > >>> > > > > > users
>> > >>> that
>> > >>> > > > > > authenticated using *non* delegation token mechanism. For
>> > >>> example,
>> > >>> > If
>> > >>> > > > > user
>> > >>> > > > > > Alice authenticated using kerberos and requested
>> > >>> > > > > > delegation
>> > >>> tokens,
>> > >>> > > > only
>> > >>> > > > > > user Alice authenticated via non delegation token
>> > >>> > > > > > mechanism
>> > can
>> > >>> > > renew.
>> > >>> > > > > > Clients that have  access to delegation tokens can not
>> > >>> > > > > > issue
>> > >>> > renewal
>> > >>> > > > > > request for renewing their own token and this is primarily
>> > >>> > important
>> > >>> > > to
>> > >>> > > > > > reduce the time window for which a compromised token will
>> > >>> > > > > > be
>> > >>> valid.
>> > >>> > > > > >
>> > >>> > > > > > To clarify, Yes any authenticated user can request
>> > >>> > > > > > delegation
>> > >>> > tokens
>> > >>> > > > but
>> > >>> > > > > > even here I would recommend to avoid creating a chain
>> > >>> > > > > > where a
>> > >>> > client
>> > >>> > > > > > authenticated via delegation token request for more
>> > delegation
>> > >>> > > tokens.
>> > >>> > > > > > Basically anyone can request delegation token, as long as
>> > they
>> > >>> > > > > authenticate
>> > >>> > > > > > via a non delegation token mechanism.
>> > >>> > > > > >
>> > >>> > > > > > Aren't classes listed here
>> > >>> > > > > > <
>> > >>> > > > > >
>> > >>> > > > >
>> > >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > >>> > > 48+Delegation+token+support+for+Kafka#KIP-48Delegationtokens
>> > >>> upportforKaf
>> > >>> > > ka-PublicInterfaces
>> > >>> > > > > > >
>> > >>> > > > > > sufficient?
>> > >>> > > > > >
>> > >>> > > > > > Thanks
>> > >>> > > > > > Parth
>> > >>> > > > > >
>> > >>> > > > > >
>> > >>> > > > > >
>> > >>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao
>> > >>> > > > > > <ju...@confluent.io>
>> > >>> wrote:
>> > >>> > > > > >
>> > >>> > > > > > > Parth,
>> > >>> > > > > > >
>> > >>> > > > > > > Thanks for the reply. A couple of comments inline below.
>> > >>> > > > > > >
>> > >>> > > > > > > On Tue, Jun 21, 2016 at 10:36 AM, parth brahmbhatt <
>> > >>> > > > > > > brahmbhatt.parth@gmail.com> wrote:
>> > >>> > > > > > >
>> > >>> > > > > > > > 1. Who / how are tokens renewed? By original requester
>> > >>> only? or
>> > >>> > > > using
>> > >>> > > > > > > > Kerberos
>> > >>> > > > > > > > auth only?
>> > >>> > > > > > > > My recommendation is to do this only using Kerberos
>> > >>> > > > > > > > auth
>> > and
>> > >>> > only
>> > >>> > > > > threw
>> > >>> > > > > > > the
>> > >>> > > > > > > > renewer specified during the acquisition request.
>> > >>> > > > > > > >
>> > >>> > > > > > > >
>> > >>> > > > > > > Hmm, not sure that I follow this. Are you saying that
>> > >>> > > > > > > any
>> > >>> client
>> > >>> > > > > > > authenticated with the delegation token can renew, i.e.
>> > there
>> > >>> is
>> > >>> > no
>> > >>> > > > > > renewer
>> > >>> > > > > > > needed?
>> > >>> > > > > > >
>> > >>> > > > > > > Also, just to be clear, any authenticated client (either
>> > >>> through
>> > >>> > > SASL
>> > >>> > > > > or
>> > >>> > > > > > > SSL) can request a delegation token for the
>> > >>> > > > > > > authenticated
>> > >>> user,
>> > >>> > > > right?
>> > >>> > > > > > >
>> > >>> > > > > > >
>> > >>> > > > > > > > 2. Are tokens stored on each broker or in ZK?
>> > >>> > > > > > > > My recommendation is still to store in ZK or not store
>> > them
>> > >>> at
>> > >>> > > all.
>> > >>> > > > > The
>> > >>> > > > > > > > whole controller based distribution is too much
>> > >>> > > > > > > > overhead
>> > >>> with
>> > >>> > not
>> > >>> > > > > much
>> > >>> > > > > > to
>> > >>> > > > > > > > achieve.
>> > >>> > > > > > > >
>> > >>> > > > > > > > 3. How are tokens invalidated / expired?
>> > >>> > > > > > > > Either by expiration time out or through an explicit
>> > >>> request to
>> > >>> > > > > > > invalidate.
>> > >>> > > > > > > >
>> > >>> > > > > > > > 4. Which encryption algorithm is used?
>> > >>> > > > > > > > SCRAM
>> > >>> > > > > > > >
>> > >>> > > > > > > > 5. What is the impersonation proposal (it wasn't in
>> > >>> > > > > > > > the
>> > KIP
>> > >>> but
>> > >>> > > was
>> > >>> > > > > > > > discussed
>> > >>> > > > > > > > in this thread)?
>> > >>> > > > > > > > There is no imperonation proposal. I tried and
>> > >>> > > > > > > > explained
>> > how
>> > >>> > its
>> > >>> > > a
>> > >>> > > > > > > > different problem and why its not really necessary to
>> > >>> discuss
>> > >>> > > that
>> > >>> > > > as
>> > >>> > > > > > > part
>> > >>> > > > > > > > of this KIP.  This KIP will not support any
>> > impersonation,
>> > >>> it
>> > >>> > > will
>> > >>> > > > > just
>> > >>> > > > > > > be
>> > >>> > > > > > > > another way to authenticate.
>> > >>> > > > > > > >
>> > >>> > > > > > > > 6. Do we need new ACLs, if so - for what actions?
>> > >>> > > > > > > > We do not need new ACLs.
>> > >>> > > > > > > >
>> > >>> > > > > > > >
>> > >>> > > > > > > Could we document the format of the new request/response
>> > and
>> > >>> > their
>> > >>> > > > > > > associated Resource and Operation for ACL?
>> > >>> > > > > > >
>> > >>> > > > > > >
>> > >>> > > > > > > > 7. How would the delegation token be configured in the
>> > >>> client?
>> > >>> > > > > > > > Should be through config. I wasn't planning on
>> > >>> > > > > > > > supporting
>> > >>> JAAS
>> > >>> > > for
>> > >>> > > > > > > tokens.
>> > >>> > > > > > > > I don't believe hadoop does this either.
>> > >>> > > > > > > >
>> > >>> > > > > > > > Thanks
>> > >>> > > > > > > > Parth
>> > >>> > > > > > > >
>> > >>> > > > > > > >
>> > >>> > > > > > > >
>> > >>> > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <
>> > jun@confluent.io>
>> > >>> > > wrote:
>> > >>> > > > > > > >
>> > >>> > > > > > > > > Harsha,
>> > >>> > > > > > > > >
>> > >>> > > > > > > > > Another question.
>> > >>> > > > > > > > >
>> > >>> > > > > > > > > 9. How would the delegation token be configured in
>> > >>> > > > > > > > > the
>> > >>> > client?
>> > >>> > > > The
>> > >>> > > > > > > > standard
>> > >>> > > > > > > > > way is to do this through JAAS. However, we will
>> > >>> > > > > > > > > need
>> > to
>> > >>> > think
>> > >>> > > > > > through
>> > >>> > > > > > > if
>> > >>> > > > > > > > > this is convenient in a shared environment. For
>> > example,
>> > >>> > when a
>> > >>> > > > new
>> > >>> > > > > > > task
>> > >>> > > > > > > > is
>> > >>> > > > > > > > > added to a Storm worker node, do we need to
>> > >>> > > > > > > > > dynamically
>> > >>> add a
>> > >>> > > new
>> > >>> > > > > > > section
>> > >>> > > > > > > > > in the JAAS file? It may be more convenient if we
>> > >>> > > > > > > > > can
>> > >>> pass in
>> > >>> > > the
>> > >>> > > > > > token
>> > >>> > > > > > > > > through the config directly w/o going through JAAS.
>> > >>> > > > > > > > >
>> > >>> > > > > > > > > Are you or Parth still actively working on this KIP?
>> > >>> > > > > > > > >
>> > >>> > > > > > > > > Thanks,
>> > >>> > > > > > > > >
>> > >>> > > > > > > > > Jun
>> > >>> > > > > > > > >
>> > >>> > > > > > > > >
>> > >>> > > > > > > > >
>> > >>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <
>> > >>> jun@confluent.io>
>> > >>> > > > wrote:
>> > >>> > > > > > > > >
>> > >>> > > > > > > > > > Just to add on that list.
>> > >>> > > > > > > > > >
>> > >>> > > > > > > > > > 2. It would be good to document the format of the
>> > data
>> > >>> > stored
>> > >>> > > > in
>> > >>> > > > > > ZK.
>> > >>> > > > > > > > > > 7. Earlier, there was a discussion on whether the
>> > tokens
>> > >>> > > should
>> > >>> > > > > be
>> > >>> > > > > > > > > > propagated through ZK like config/acl/quota, or
>> > through
>> > >>> the
>> > >>> > > > > > > controller.
>> > >>> > > > > > > > > > Currently, the controller is only designed for
>> > >>> propagating
>> > >>> > > > topic
>> > >>> > > > > > > > > metadata,
>> > >>> > > > > > > > > > but not other data.
>> > >>> > > > > > > > > > 8. Should we use SCRAM to send the token instead
>> > >>> > > > > > > > > > of
>> > >>> > > DIGEST-MD5
>> > >>> > > > > > since
>> > >>> > > > > > > > it's
>> > >>> > > > > > > > > > deprecated?
>> > >>> > > > > > > > > >
>> > >>> > > > > > > > > > Also, the images in the wiki seem broken.
>> > >>> > > > > > > > > >
>> > >>> > > > > > > > > > Thanks,
>> > >>> > > > > > > > > >
>> > >>> > > > > > > > > > Jun
>> > >>> > > > > > > > > >
>> > >>> > > > > > > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <
>> > >>> > > > > gwen@confluent.io>
>> > >>> > > > > > > > > wrote:
>> > >>> > > > > > > > > >
>> > >>> > > > > > > > > >> From what I can see, remaining questions are:
>> > >>> > > > > > > > > >>
>> > >>> > > > > > > > > >> 1. Who / how are tokens renewed? By original
>> > requester
>> > >>> > only?
>> > >>> > > > or
>> > >>> > > > > > > using
>> > >>> > > > > > > > > >> Kerberos auth only?
>> > >>> > > > > > > > > >> 2. Are tokens stored on each broker or in ZK?
>> > >>> > > > > > > > > >> 3. How are tokens invalidated / expired?
>> > >>> > > > > > > > > >> 4. Which encryption algorithm is used?
>> > >>> > > > > > > > > >> 5. What is the impersonation proposal (it wasn't
>> > >>> > > > > > > > > >> in
>> > the
>> > >>> > KIP
>> > >>> > > > but
>> > >>> > > > > > was
>> > >>> > > > > > > > > >> discussed in this thread)?
>> > >>> > > > > > > > > >> 6. Do we need new ACLs, if so - for what actions?
>> > >>> > > > > > > > > >>
>> > >>> > > > > > > > > >> Gwen
>> > >>> > > > > > > > > >>
>> > >>> > > > > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <
>> > >>> kafka@harsha.io>
>> > >>> > > > wrote:
>> > >>> > > > > > > > > >> > Jun & Ismael,
>> > >>> > > > > > > > > >> >                          Unfortunately I
>> > >>> > > > > > > > > >> > couldn't
>> > >>> attend
>> > >>> > > the
>> > >>> > > > > KIP
>> > >>> > > > > > > > > meeting
>> > >>> > > > > > > > > >> >                          when delegation tokens
>> > >>> > discussed.
>> > >>> > > > > > > > Appreciate
>> > >>> > > > > > > > > if
>> > >>> > > > > > > > > >> >                          you can update the
>> > thread if
>> > >>> > you
>> > >>> > > > have
>> > >>> > > > > > any
>> > >>> > > > > > > > > >> >                          further questions.
>> > >>> > > > > > > > > >> > Thanks,
>> > >>> > > > > > > > > >> > Harsha
>> > >>> > > > > > > > > >> >
>> > >>> > > > > > > > > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei
>> > wrote:
>> > >>> > > > > > > > > >> >> It seems that the links to images in the KIP
>> > >>> > > > > > > > > >> >> are
>> > >>> > broken.
>> > >>> > > > > > > > > >> >>
>> > >>> > > > > > > > > >> >> Liquan
>> > >>> > > > > > > > > >> >>
>> > >>> > > > > > > > > >> >> On Tue, May 24, 2016 at 9:33 AM, parth
>> > brahmbhatt <
>> > >>> > > > > > > > > >> >> brahmbhatt.parth@gmail.com> wrote:
>> > >>> > > > > > > > > >> >>
>> > >>> > > > > > > > > >> >> > 110. What does getDelegationTokenAs mean?
>> > >>> > > > > > > > > >> >> > In the current proposal we only allow a user
>> > >>> > > > > > > > > >> >> > to
>> > >>> get
>> > >>> > > > > > delegation
>> > >>> > > > > > > > > token
>> > >>> > > > > > > > > >> for
>> > >>> > > > > > > > > >> >> > the identity that it authenticated as using
>> > >>> another
>> > >>> > > > > > mechanism,
>> > >>> > > > > > > > i.e.
>> > >>> > > > > > > > > >> A user
>> > >>> > > > > > > > > >> >> > that authenticate using a keytab for
>> > >>> > > > > > > > > >> >> > principal
>> > >>> > > > > > > user1@EXAMPLE.COM
>> > >>> > > > > > > > > >> will get
>> > >>> > > > > > > > > >> >> > delegation tokens for that user only. In
>> > future I
>> > >>> > think
>> > >>> > > > we
>> > >>> > > > > > will
>> > >>> > > > > > > > > have
>> > >>> > > > > > > > > >> to
>> > >>> > > > > > > > > >> >> > extend support such that we allow some set
>> > >>> > > > > > > > > >> >> > of
>> > >>> users (
>> > >>> > > > > > > > > >> >> > kafka-rest-user@EXAMPLE.COM,
>> > >>> > storm-nimbus@EXAMPLE.COM)
>> > >>> > > > to
>> > >>> > > > > > > > acquire
>> > >>> > > > > > > > > >> >> > delegation tokens on behalf of other users
>> > whose
>> > >>> > > identity
>> > >>> > > > > > they
>> > >>> > > > > > > > have
>> > >>> > > > > > > > > >> >> > verified independently.  Kafka brokers will
>> > have
>> > >>> ACLs
>> > >>> > > to
>> > >>> > > > > > > control
>> > >>> > > > > > > > > >> which
>> > >>> > > > > > > > > >> >> > users are allowed to impersonate other users
>> > and
>> > >>> get
>> > >>> > > > tokens
>> > >>> > > > > > on
>> > >>> > > > > > > > > >> behalf of
>> > >>> > > > > > > > > >> >> > them. Overall Impersonation is a whole
>> > different
>> > >>> > > problem
>> > >>> > > > in
>> > >>> > > > > > my
>> > >>> > > > > > > > > >> opinion and
>> > >>> > > > > > > > > >> >> > I think we can tackle it in separate KIP.
>> > >>> > > > > > > > > >> >> >
>> > >>> > > > > > > > > >> >> > 111. What's the typical rate of getting and
>> > >>> renewing
>> > >>> > > > > > delegation
>> > >>> > > > > > > > > >> tokens?
>> > >>> > > > > > > > > >> >> > Typically this should be very very low, 1
>> > request
>> > >>> per
>> > >>> > > > > minute
>> > >>> > > > > > > is a
>> > >>> > > > > > > > > >> >> > relatively high estimate. However it depends
>> > >>> > > > > > > > > >> >> > on
>> > >>> the
>> > >>> > > token
>> > >>> > > > > > > > > >> expiration. I am
>> > >>> > > > > > > > > >> >> > less worried about the extra load it puts on
>> > >>> > controller
>> > >>> > > > vs
>> > >>> > > > > > the
>> > >>> > > > > > > > > added
>> > >>> > > > > > > > > >> >> > complexity and the value it offers.
>> > >>> > > > > > > > > >> >> >
>> > >>> > > > > > > > > >> >> > Thanks
>> > >>> > > > > > > > > >> >> > Parth
>> > >>> > > > > > > > > >> >> >
>> > >>> > > > > > > > > >> >> >
>> > >>> > > > > > > > > >> >> >
>> > >>> > > > > > > > > >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma
>> > >>> > > > > > > > > >> >> > <
>> > >>> > > > > > > ismael@juma.me.uk>
>> > >>> > > > > > > > > >> wrote:
>> > >>> > > > > > > > > >> >> >
>> > >>> > > > > > > > > >> >> > > Thanks Rajini. It would probably require a
>> > >>> separate
>> > >>> > > KIP
>> > >>> > > > > as
>> > >>> > > > > > it
>> > >>> > > > > > > > > will
>> > >>> > > > > > > > > >> >> > > introduce user visible changes. We could
>> > >>> > > > > > > > > >> >> > > also
>> > >>> > update
>> > >>> > > > > KIP-48
>> > >>> > > > > > > to
>> > >>> > > > > > > > > >> have this
>> > >>> > > > > > > > > >> >> > > information, but it seems cleaner to do it
>> > >>> > > separately.
>> > >>> > > > We
>> > >>> > > > > > can
>> > >>> > > > > > > > > >> discuss
>> > >>> > > > > > > > > >> >> > that
>> > >>> > > > > > > > > >> >> > > in the KIP call today.
>> > >>> > > > > > > > > >> >> > >
>> > >>> > > > > > > > > >> >> > > Ismael
>> > >>> > > > > > > > > >> >> > >
>> > >>> > > > > > > > > >> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini
>> > Sivaram
>> > >>> <
>> > >>> > > > > > > > > >> >> > > rajinisivaram@googlemail.com> wrote:
>> > >>> > > > > > > > > >> >> > >
>> > >>> > > > > > > > > >> >> > > > Ismael,
>> > >>> > > > > > > > > >> >> > > >
>> > >>> > > > > > > > > >> >> > > > I have created a JIRA (
>> > >>> > > > > > > > > >> >> > https://issues.apache.org/
>> > jira/browse/KAFKA-3751)
>> > >>> > > > > > > > > >> >> > > > for adding SCRAM as a SASL mechanism.
>> > >>> > > > > > > > > >> >> > > > Would
>> > >>> that
>> > >>> > > need
>> > >>> > > > > > > another
>> > >>> > > > > > > > > >> KIP? If
>> > >>> > > > > > > > > >> >> > > > KIP-48 will use this mechanism, can this
>> > just
>> > >>> be
>> > >>> > a
>> > >>> > > > JIRA
>> > >>> > > > > > > that
>> > >>> > > > > > > > > gets
>> > >>> > > > > > > > > >> >> > > reviewed
>> > >>> > > > > > > > > >> >> > > > when the PR is ready?
>> > >>> > > > > > > > > >> >> > > >
>> > >>> > > > > > > > > >> >> > > > Thank you,
>> > >>> > > > > > > > > >> >> > > >
>> > >>> > > > > > > > > >> >> > > > Rajini
>> > >>> > > > > > > > > >> >> > > >
>> > >>> > > > > > > > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael
>> > Juma <
>> > >>> > > > > > > > > ismael@juma.me.uk>
>> > >>> > > > > > > > > >> >> > wrote:
>> > >>> > > > > > > > > >> >> > > >
>> > >>> > > > > > > > > >> >> > > > > Thanks Rajini, SCRAM seems like a good
>> > >>> > candidate.
>> > >>> > > > > > > > > >> >> > > > >
>> > >>> > > > > > > > > >> >> > > > > Gwen had independently mentioned this
>> > >>> > > > > > > > > >> >> > > > > as
>> > a
>> > >>> SASL
>> > >>> > > > > > mechanism
>> > >>> > > > > > > > > that
>> > >>> > > > > > > > > >> might
>> > >>> > > > > > > > > >> >> > be
>> > >>> > > > > > > > > >> >> > > > > useful for Kafka and I have been
>> > >>> > > > > > > > > >> >> > > > > meaning
>> > to
>> > >>> > check
>> > >>> > > > it
>> > >>> > > > > in
>> > >>> > > > > > > > more
>> > >>> > > > > > > > > >> detail.
>> > >>> > > > > > > > > >> >> > > Good
>> > >>> > > > > > > > > >> >> > > > > to know that you are willing to
>> > contribute
>> > >>> an
>> > >>> > > > > > > > implementation.
>> > >>> > > > > > > > > >> Maybe
>> > >>> > > > > > > > > >> >> > we
>> > >>> > > > > > > > > >> >> > > > > should file a separate JIRA for this?
>> > >>> > > > > > > > > >> >> > > > >
>> > >>> > > > > > > > > >> >> > > > > Ismael
>> > >>> > > > > > > > > >> >> > > > >
>> > >>> > > > > > > > > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM,
>> > >>> > > > > > > > > >> >> > > > > Rajini
>> > >>> > Sivaram <
>> > >>> > > > > > > > > >> >> > > > > rajinisivaram@googlemail.com> wrote:
>> > >>> > > > > > > > > >> >> > > > >
>> > >>> > > > > > > > > >> >> > > > > > SCRAM (Salted Challenge Response
>> > >>> > Authentication
>> > >>> > > > > > > > Mechanism)
>> > >>> > > > > > > > > >> is a
>> > >>> > > > > > > > > >> >> > > better
>> > >>> > > > > > > > > >> >> > > > > > mechanism than Digest-MD5. Java
>> > >>> > > > > > > > > >> >> > > > > > doesn't
>> > >>> come
>> > >>> > > > with a
>> > >>> > > > > > > > > built-in
>> > >>> > > > > > > > > >> SCRAM
>> > >>> > > > > > > > > >> >> > > > > > SaslServer or SaslClient, but I will
>> > >>> > > > > > > > > >> >> > > > > > be
>> > >>> happy
>> > >>> > > to
>> > >>> > > > > add
>> > >>> > > > > > > > > support
>> > >>> > > > > > > > > >> in
>> > >>> > > > > > > > > >> >> > Kafka
>> > >>> > > > > > > > > >> >> > > > > since
>> > >>> > > > > > > > > >> >> > > > > > it would be a useful mechanism to
>> > support
>> > >>> > > anyway.
>> > >>> > > > > > > > > >> >> > > > > > https://tools.ietf.org/html/rfc7677
>> > >>> > describes
>> > >>> > > > the
>> > >>> > > > > > > > protocol
>> > >>> > > > > > > > > >> for
>> > >>> > > > > > > > > >> >> > > > > > SCRAM-SHA-256.
>> > >>> > > > > > > > > >> >> > > > > >
>> > >>> > > > > > > > > >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun
>> > Rao <
>> > >>> > > > > > > > jun@confluent.io
>> > >>> > > > > > > > > >
>> > >>> > > > > > > > > >> wrote:
>> > >>> > > > > > > > > >> >> > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > Parth,
>> > >>> > > > > > > > > >> >> > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > Thanks for the explanation. A
>> > >>> > > > > > > > > >> >> > > > > > > couple
>> > of
>> > >>> > more
>> > >>> > > > > > > questions.
>> > >>> > > > > > > > > >> >> > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > 110. What does
>> > >>> > > > > > > > > >> >> > > > > > > getDelegationTokenAs
>> > >>> mean?
>> > >>> > > > > > > > > >> >> > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > 111. What's the typical rate of
>> > getting
>> > >>> and
>> > >>> > > > > > renewing
>> > >>> > > > > > > > > >> delegation
>> > >>> > > > > > > > > >> >> > > > tokens?
>> > >>> > > > > > > > > >> >> > > > > > > That may have an impact on whether
>> > they
>> > >>> > > should
>> > >>> > > > be
>> > >>> > > > > > > > > directed
>> > >>> > > > > > > > > >> to the
>> > >>> > > > > > > > > >> >> > > > > > > controller.
>> > >>> > > > > > > > > >> >> > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > Jun
>> > >>> > > > > > > > > >> >> > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM,
>> > parth
>> > >>> > > > > brahmbhatt <
>> > >>> > > > > > > > > >> >> > > > > > > brahmbhatt.parth@gmail.com> wrote:
>> > >>> > > > > > > > > >> >> > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > Hi Jun,
>> > >>> > > > > > > > > >> >> > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > Thanks for reviewing.
>> > >>> > > > > > > > > >> >> > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > * We could add a Cluster action
>> > >>> > > > > > > > > >> >> > > > > > > > to
>> > add
>> > >>> > acls
>> > >>> > > > on
>> > >>> > > > > > who
>> > >>> > > > > > > > can
>> > >>> > > > > > > > > >> request
>> > >>> > > > > > > > > >> >> > > > > > delegation
>> > >>> > > > > > > > > >> >> > > > > > > > tokens. I don't see the use case
>> > for
>> > >>> that
>> > >>> > > yet
>> > >>> > > > > but
>> > >>> > > > > > > > down
>> > >>> > > > > > > > > >> the line
>> > >>> > > > > > > > > >> >> > > > when
>> > >>> > > > > > > > > >> >> > > > > we
>> > >>> > > > > > > > > >> >> > > > > > > > start supporting
>> > getDelegationTokenAs
>> > >>> it
>> > >>> > > will
>> > >>> > > > > be
>> > >>> > > > > > > > > >> necessary.
>> > >>> > > > > > > > > >> >> > > > > > > > * Yes we recommend tokens to be
>> > only
>> > >>> > > > > > > used/distributed
>> > >>> > > > > > > > > >> over
>> > >>> > > > > > > > > >> >> > secure
>> > >>> > > > > > > > > >> >> > > > > > > channels.
>> > >>> > > > > > > > > >> >> > > > > > > > * Depending on what design we
>> > >>> > > > > > > > > >> >> > > > > > > > end
>> > up
>> > >>> > > choosing
>> > >>> > > > > > > > > >> Invalidation will
>> > >>> > > > > > > > > >> >> > > be
>> > >>> > > > > > > > > >> >> > > > > > > > responsibility of every broker
>> > >>> > > > > > > > > >> >> > > > > > > > or
>> > >>> > > controller.
>> > >>> > > > > > > > > >> >> > > > > > > > * I am not sure if I documented
>> > >>> somewhere
>> > >>> > > > that
>> > >>> > > > > > > > > >> invalidation
>> > >>> > > > > > > > > >> >> > will
>> > >>> > > > > > > > > >> >> > > > > > directly
>> > >>> > > > > > > > > >> >> > > > > > > > go through zookeeper but that is
>> > not
>> > >>> the
>> > >>> > > > > intent.
>> > >>> > > > > > > > > >> Invalidation
>> > >>> > > > > > > > > >> >> > > will
>> > >>> > > > > > > > > >> >> > > > > > either
>> > >>> > > > > > > > > >> >> > > > > > > > be request based or due to
>> > >>> expiration. No
>> > >>> > > > > direct
>> > >>> > > > > > > > > >> zookeeper
>> > >>> > > > > > > > > >> >> > > > > interaction
>> > >>> > > > > > > > > >> >> > > > > > > from
>> > >>> > > > > > > > > >> >> > > > > > > > any client.
>> > >>> > > > > > > > > >> >> > > > > > > > * "Broker also stores the
>> > >>> DelegationToken
>> > >>> > > > > without
>> > >>> > > > > > > the
>> > >>> > > > > > > > > >> hmac in
>> > >>> > > > > > > > > >> >> > the
>> > >>> > > > > > > > > >> >> > > > > > > > zookeeper." : Sorry about the
>> > >>> confusion.
>> > >>> > > The
>> > >>> > > > > sole
>> > >>> > > > > > > > > >> purpose of
>> > >>> > > > > > > > > >> >> > > > > zookeeper
>> > >>> > > > > > > > > >> >> > > > > > in
>> > >>> > > > > > > > > >> >> > > > > > > > this design is as distribution
>> > channel
>> > >>> > for
>> > >>> > > > > tokens
>> > >>> > > > > > > > > >> between all
>> > >>> > > > > > > > > >> >> > > > brokers
>> > >>> > > > > > > > > >> >> > > > > > > and a
>> > >>> > > > > > > > > >> >> > > > > > > > layer that ensures only tokens
>> > >>> > > > > > > > > >> >> > > > > > > > that
>> > >>> were
>> > >>> > > > > > generated
>> > >>> > > > > > > by
>> > >>> > > > > > > > > >> making a
>> > >>> > > > > > > > > >> >> > > > > request
>> > >>> > > > > > > > > >> >> > > > > > > to a
>> > >>> > > > > > > > > >> >> > > > > > > > broker will be accepted (more on
>> > this
>> > >>> in
>> > >>> > > > second
>> > >>> > > > > > > > > >> paragraph). The
>> > >>> > > > > > > > > >> >> > > > token
>> > >>> > > > > > > > > >> >> > > > > > > > consists of few elements (owner,
>> > >>> renewer,
>> > >>> > > > uuid
>> > >>> > > > > ,
>> > >>> > > > > > > > > >> expiration,
>> > >>> > > > > > > > > >> >> > > hmac)
>> > >>> > > > > > > > > >> >> > > > ,
>> > >>> > > > > > > > > >> >> > > > > > one
>> > >>> > > > > > > > > >> >> > > > > > > of
>> > >>> > > > > > > > > >> >> > > > > > > > which is the finally generated
>> > >>> > > > > > > > > >> >> > > > > > > > hmac
>> > >>> but
>> > >>> > > hmac
>> > >>> > > > it
>> > >>> > > > > > > self
>> > >>> > > > > > > > is
>> > >>> > > > > > > > > >> >> > derivable
>> > >>> > > > > > > > > >> >> > > > if
>> > >>> > > > > > > > > >> >> > > > > > you
>> > >>> > > > > > > > > >> >> > > > > > > > have all the other elements of
>> > >>> > > > > > > > > >> >> > > > > > > > the
>> > >>> token
>> > >>> > +
>> > >>> > > > > secret
>> > >>> > > > > > > key
>> > >>> > > > > > > > > to
>> > >>> > > > > > > > > >> >> > generate
>> > >>> > > > > > > > > >> >> > > > > hmac.
>> > >>> > > > > > > > > >> >> > > > > > > > Given zookeeper does not provide
>> > SSL
>> > >>> > > support
>> > >>> > > > we
>> > >>> > > > > > do
>> > >>> > > > > > > > not
>> > >>> > > > > > > > > >> want the
>> > >>> > > > > > > > > >> >> > > > > entire
>> > >>> > > > > > > > > >> >> > > > > > > > token to be wire transferred to
>> > >>> zookeeper
>> > >>> > > as
>> > >>> > > > > that
>> > >>> > > > > > > > will
>> > >>> > > > > > > > > >> be an
>> > >>> > > > > > > > > >> >> > > > insecure
>> > >>> > > > > > > > > >> >> > > > > > > wire
>> > >>> > > > > > > > > >> >> > > > > > > > transfer. Instead we only store
>> > >>> > > > > > > > > >> >> > > > > > > > all
>> > >>> the
>> > >>> > > other
>> > >>> > > > > > > > elements
>> > >>> > > > > > > > > >> of a
>> > >>> > > > > > > > > >> >> > > > > delegation
>> > >>> > > > > > > > > >> >> > > > > > > > tokens. Brokers can read these
>> > >>> elements
>> > >>> > and
>> > >>> > > > > > because
>> > >>> > > > > > > > > they
>> > >>> > > > > > > > > >> also
>> > >>> > > > > > > > > >> >> > > have
>> > >>> > > > > > > > > >> >> > > > > > access
>> > >>> > > > > > > > > >> >> > > > > > > > to secret key they will be able
>> > >>> > > > > > > > > >> >> > > > > > > > to
>> > >>> > generate
>> > >>> > > > > hmac
>> > >>> > > > > > on
>> > >>> > > > > > > > > >> their end.
>> > >>> > > > > > > > > >> >> > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > One of the alternative proposed
>> > >>> > > > > > > > > >> >> > > > > > > > is
>> > to
>> > >>> > avoid
>> > >>> > > > > > > zookeeper
>> > >>> > > > > > > > > >> >> > > altogether. A
>> > >>> > > > > > > > > >> >> > > > > > > Client
>> > >>> > > > > > > > > >> >> > > > > > > > will call broker with required
>> > >>> > information
>> > >>> > > > > > (owner,
>> > >>> > > > > > > > > >> renwer,
>> > >>> > > > > > > > > >> >> > > > > expiration)
>> > >>> > > > > > > > > >> >> > > > > > > and
>> > >>> > > > > > > > > >> >> > > > > > > > get back (signed hmac, uuid).
>> > Broker
>> > >>> > won't
>> > >>> > > > > store
>> > >>> > > > > > > this
>> > >>> > > > > > > > > in
>> > >>> > > > > > > > > >> >> > > zookeeper.
>> > >>> > > > > > > > > >> >> > > > > > From
>> > >>> > > > > > > > > >> >> > > > > > > > this point a client can contact
>> > >>> > > > > > > > > >> >> > > > > > > > any
>> > >>> > broker
>> > >>> > > > with
>> > >>> > > > > > all
>> > >>> > > > > > > > the
>> > >>> > > > > > > > > >> >> > > delegation
>> > >>> > > > > > > > > >> >> > > > > > token
>> > >>> > > > > > > > > >> >> > > > > > > > info (owner, rewner, expiration,
>> > hmac,
>> > >>> > > uuid)
>> > >>> > > > > the
>> > >>> > > > > > > > borker
>> > >>> > > > > > > > > >> will
>> > >>> > > > > > > > > >> >> > > > > regenerate
>> > >>> > > > > > > > > >> >> > > > > > > the
>> > >>> > > > > > > > > >> >> > > > > > > > hmac and as long as it matches
>> > >>> > > > > > > > > >> >> > > > > > > > with
>> > >>> hmac
>> > >>> > > > > > presented
>> > >>> > > > > > > by
>> > >>> > > > > > > > > >> client ,
>> > >>> > > > > > > > > >> >> > > > broker
>> > >>> > > > > > > > > >> >> > > > > > > will
>> > >>> > > > > > > > > >> >> > > > > > > > allow the request to
>> > >>> > > > > > > > > >> >> > > > > > > > authenticate.
>> > >>> Only
>> > >>> > > > > problem
>> > >>> > > > > > > with
>> > >>> > > > > > > > > >> this
>> > >>> > > > > > > > > >> >> > > approach
>> > >>> > > > > > > > > >> >> > > > > is
>> > >>> > > > > > > > > >> >> > > > > > if
>> > >>> > > > > > > > > >> >> > > > > > > > the secret key is compromised
>> > >>> > > > > > > > > >> >> > > > > > > > any
>> > >>> client
>> > >>> > > can
>> > >>> > > > > now
>> > >>> > > > > > > > > generate
>> > >>> > > > > > > > > >> >> > random
>> > >>> > > > > > > > > >> >> > > > > tokens
>> > >>> > > > > > > > > >> >> > > > > > > and
>> > >>> > > > > > > > > >> >> > > > > > > > they will still be able to
>> > >>> authenticate
>> > >>> > as
>> > >>> > > > any
>> > >>> > > > > > user
>> > >>> > > > > > > > > they
>> > >>> > > > > > > > > >> like.
>> > >>> > > > > > > > > >> >> > > with
>> > >>> > > > > > > > > >> >> > > > > > > > zookeeper we guarantee that only
>> > >>> tokens
>> > >>> > > > > acquired
>> > >>> > > > > > > via
>> > >>> > > > > > > > a
>> > >>> > > > > > > > > >> broker
>> > >>> > > > > > > > > >> >> > > > (using
>> > >>> > > > > > > > > >> >> > > > > > some
>> > >>> > > > > > > > > >> >> > > > > > > > auth scheme other than
>> > >>> > > > > > > > > >> >> > > > > > > > delegation
>> > >>> token)
>> > >>> > > will
>> > >>> > > > > be
>> > >>> > > > > > > > > >> accepted. We
>> > >>> > > > > > > > > >> >> > > need
>> > >>> > > > > > > > > >> >> > > > to
>> > >>> > > > > > > > > >> >> > > > > > > > discuss which proposal makes
>> > >>> > > > > > > > > >> >> > > > > > > > more
>> > >>> sense
>> > >>> > and
>> > >>> > > > we
>> > >>> > > > > > can
>> > >>> > > > > > > go
>> > >>> > > > > > > > > >> over it
>> > >>> > > > > > > > > >> >> > in
>> > >>> > > > > > > > > >> >> > > > > > > tomorrow's
>> > >>> > > > > > > > > >> >> > > > > > > > meeting.
>> > >>> > > > > > > > > >> >> > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > Also, can you forward the invite
>> > >>> > > > > > > > > >> >> > > > > > > > to
>> > >>> me?
>> > >>> > > > > > > > > >> >> > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > Thanks
>> > >>> > > > > > > > > >> >> > > > > > > > Parth
>> > >>> > > > > > > > > >> >> > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > On Mon, May 23, 2016 at 10:35
>> > >>> > > > > > > > > >> >> > > > > > > > AM,
>> > Jun
>> > >>> > Rao <
>> > >>> > > > > > > > > >> jun@confluent.io>
>> > >>> > > > > > > > > >> >> > > > wrote:
>> > >>> > > > > > > > > >> >> > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > Thanks for the KIP. A few
>> > comments.
>> > >>> > > > > > > > > >> >> > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > 100. This potentially can be
>> > useful
>> > >>> for
>> > >>> > > > Kafka
>> > >>> > > > > > > > Connect
>> > >>> > > > > > > > > >> and
>> > >>> > > > > > > > > >> >> > Kafka
>> > >>> > > > > > > > > >> >> > > > > rest
>> > >>> > > > > > > > > >> >> > > > > > > > proxy
>> > >>> > > > > > > > > >> >> > > > > > > > > where a worker agent will need
>> > >>> > > > > > > > > >> >> > > > > > > > > to
>> > >>> run a
>> > >>> > > > task
>> > >>> > > > > on
>> > >>> > > > > > > > > behalf
>> > >>> > > > > > > > > >> of a
>> > >>> > > > > > > > > >> >> > > > client.
>> > >>> > > > > > > > > >> >> > > > > > We
>> > >>> > > > > > > > > >> >> > > > > > > > will
>> > >>> > > > > > > > > >> >> > > > > > > > > likely need to change how
>> > >>> > > > > > > > > >> >> > > > > > > > > those
>> > >>> > services
>> > >>> > > > use
>> > >>> > > > > > > Kafka
>> > >>> > > > > > > > > >> clients
>> > >>> > > > > > > > > >> >> > > > > > > > > (producer/consumer). Instead
>> > >>> > > > > > > > > >> >> > > > > > > > > of a
>> > >>> > shared
>> > >>> > > > > client
>> > >>> > > > > > > per
>> > >>> > > > > > > > > >> worker,
>> > >>> > > > > > > > > >> >> > we
>> > >>> > > > > > > > > >> >> > > > will
>> > >>> > > > > > > > > >> >> > > > > > > need
>> > >>> > > > > > > > > >> >> > > > > > > > a
>> > >>> > > > > > > > > >> >> > > > > > > > > client per user task since the
>> > >>> > > > authentication
>> > >>> > > > > > > > happens
>> > >>> > > > > > > > > >> at the
>> > >>> > > > > > > > > >> >> > > > > > connection
>> > >>> > > > > > > > > >> >> > > > > > > > > level. For Kafka Connect, the
>> > >>> renewer
>> > >>> > > will
>> > >>> > > > be
>> > >>> > > > > > the
>> > >>> > > > > > > > > >> workers.
>> > >>> > > > > > > > > >> >> > So,
>> > >>> > > > > > > > > >> >> > > we
>> > >>> > > > > > > > > >> >> > > > > > > > probably
>> > >>> > > > > > > > > >> >> > > > > > > > > need to allow multiple
>> > >>> > > > > > > > > >> >> > > > > > > > > renewers.
>> > For
>> > >>> > > Kafka
>> > >>> > > > > rest
>> > >>> > > > > > > > > proxy,
>> > >>> > > > > > > > > >> the
>> > >>> > > > > > > > > >> >> > > > renewer
>> > >>> > > > > > > > > >> >> > > > > > can
>> > >>> > > > > > > > > >> >> > > > > > > > > probably just be the creator
>> > >>> > > > > > > > > >> >> > > > > > > > > of
>> > the
>> > >>> > > token.
>> > >>> > > > > > > > > >> >> > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > 101. Do we need new acl on who
>> > can
>> > >>> > > request
>> > >>> > > > > > > > delegation
>> > >>> > > > > > > > > >> tokens?
>> > >>> > > > > > > > > >> >> > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > 102. Do we recommend people to
>> > send
>> > >>> > > > > delegation
>> > >>> > > > > > > > tokens
>> > >>> > > > > > > > > >> in an
>> > >>> > > > > > > > > >> >> > > > > encrypted
>> > >>> > > > > > > > > >> >> > > > > > > > > channel?
>> > >>> > > > > > > > > >> >> > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > 103. Who is responsible for
>> > expiring
>> > >>> > > > tokens,
>> > >>> > > > > > > every
>> > >>> > > > > > > > > >> broker?
>> > >>> > > > > > > > > >> >> > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > 104. For invalidating tokens,
>> > would
>> > >>> it
>> > >>> > be
>> > >>> > > > > > better
>> > >>> > > > > > > to
>> > >>> > > > > > > > > do
>> > >>> > > > > > > > > >> it in
>> > >>> > > > > > > > > >> >> > a
>> > >>> > > > > > > > > >> >> > > > > > request
>> > >>> > > > > > > > > >> >> > > > > > > > > instead of going to ZK
>> > >>> > > > > > > > > >> >> > > > > > > > > directly?
>> > >>> > > > > > > > > >> >> > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > 105. The terminology of client
>> > >>> > > > > > > > > >> >> > > > > > > > > in
>> > >>> the
>> > >>> > > wiki
>> > >>> > > > > > > > sometimes
>> > >>> > > > > > > > > >> refers
>> > >>> > > > > > > > > >> >> > to
>> > >>> > > > > > > > > >> >> > > > the
>> > >>> > > > > > > > > >> >> > > > > > end
>> > >>> > > > > > > > > >> >> > > > > > > > > client and some other times
>> > refers
>> > >>> to
>> > >>> > the
>> > >>> > > > > > client
>> > >>> > > > > > > > > using
>> > >>> > > > > > > > > >> the
>> > >>> > > > > > > > > >> >> > > > > delegation
>> > >>> > > > > > > > > >> >> > > > > > > > > tokens. It would be useful to
>> > >>> > distinguish
>> > >>> > > > > > between
>> > >>> > > > > > > > the
>> > >>> > > > > > > > > >> two.
>> > >>> > > > > > > > > >> >> > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > 106. Could you explain the
>> > sentence
>> > >>> > > "Broker
>> > >>> > > > > > also
>> > >>> > > > > > > > > >> stores the
>> > >>> > > > > > > > > >> >> > > > > > > > DelegationToken
>> > >>> > > > > > > > > >> >> > > > > > > > > without the hmac in the
>> > zookeeper."
>> > >>> a
>> > >>> > bit
>> > >>> > > > > > more? I
>> > >>> > > > > > > > > >> thought the
>> > >>> > > > > > > > > >> >> > > > > > > delegation
>> > >>> > > > > > > > > >> >> > > > > > > > > token is the hmac.
>> > >>> > > > > > > > > >> >> > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > Thanks,
>> > >>> > > > > > > > > >> >> > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > Jun
>> > >>> > > > > > > > > >> >> > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > On Mon, May 23, 2016 at 9:22
>> > >>> > > > > > > > > >> >> > > > > > > > > AM,
>> > Jun
>> > >>> > Rao
>> > >>> > > <
>> > >>> > > > > > > > > >> jun@confluent.io>
>> > >>> > > > > > > > > >> >> > > > wrote:
>> > >>> > > > > > > > > >> >> > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > > Hi, Harsha,
>> > >>> > > > > > > > > >> >> > > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > > Just sent out a KIP meeting
>> > >>> invite.
>> > >>> > We
>> > >>> > > > can
>> > >>> > > > > > > > discuss
>> > >>> > > > > > > > > >> this in
>> > >>> > > > > > > > > >> >> > > the
>> > >>> > > > > > > > > >> >> > > > > > > meeting
>> > >>> > > > > > > > > >> >> > > > > > > > > > tomorrow.
>> > >>> > > > > > > > > >> >> > > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > > Thanks,
>> > >>> > > > > > > > > >> >> > > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > > Jun
>> > >>> > > > > > > > > >> >> > > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47
>> > AM,
>> > >>> > > Harsha <
>> > >>> > > > > > > > > >> kafka@harsha.io>
>> > >>> > > > > > > > > >> >> > > > wrote:
>> > >>> > > > > > > > > >> >> > > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > >> Hi All,
>> > >>> > > > > > > > > >> >> > > > > > > > > >>            Can we have a
>> > >>> > > > > > > > > >> >> > > > > > > > > >> KIP
>> > >>> meeting
>> > >>> > > > > around
>> > >>> > > > > > > > this.
>> > >>> > > > > > > > > >> The KIP
>> > >>> > > > > > > > > >> >> > is
>> > >>> > > > > > > > > >> >> > > > up
>> > >>> > > > > > > > > >> >> > > > > > for
>> > >>> > > > > > > > > >> >> > > > > > > > > >>            sometime and if
>> > there
>> > >>> are
>> > >>> > > any
>> > >>> > > > > > > > questions
>> > >>> > > > > > > > > >> lets
>> > >>> > > > > > > > > >> >> > > > quickly
>> > >>> > > > > > > > > >> >> > > > > > hash
>> > >>> > > > > > > > > >> >> > > > > > > > out
>> > >>> > > > > > > > > >> >> > > > > > > > > >>            details.
>> > >>> > > > > > > > > >> >> > > > > > > > > >>
>> > >>> > > > > > > > > >> >> > > > > > > > > >> Thanks,
>> > >>> > > > > > > > > >> >> > > > > > > > > >> Harsha
>> > >>> > > > > > > > > >> >> > > > > > > > > >>
>> > >>> > > > > > > > > >> >> > > > > > > > > >> On Thu, May 19, 2016, at
>> > >>> > > > > > > > > >> >> > > > > > > > > >> 08:40
>> > >>> AM,
>> > >>> > > parth
>> > >>> > > > > > > > > brahmbhatt
>> > >>> > > > > > > > > >> wrote:
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > That is what the hadoop
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > echo
>> > >>> > system
>> > >>> > > > uses
>> > >>> > > > > > so
>> > >>> > > > > > > no
>> > >>> > > > > > > > > >> good
>> > >>> > > > > > > > > >> >> > reason
>> > >>> > > > > > > > > >> >> > > > > > really.
>> > >>> > > > > > > > > >> >> > > > > > > > We
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > could
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > change it to whatever is
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > the
>> > >>> > newest
>> > >>> > > > > > > > recommended
>> > >>> > > > > > > > > >> standard
>> > >>> > > > > > > > > >> >> > > is.
>> > >>> > > > > > > > > >> >> > > > > > > > > >> >
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > Thanks
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > Parth
>> > >>> > > > > > > > > >> >> > > > > > > > > >> >
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > On Thu, May 19, 2016 at
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > 3:33
>> > >>> AM,
>> > >>> > > > Ismael
>> > >>> > > > > > > Juma <
>> > >>> > > > > > > > > >> >> > > > > ismael@juma.me.uk
>> > >>> > > > > > > > > >> >> > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > wrote:
>> > >>> > > > > > > > > >> >> > > > > > > > > >> >
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > Hi Parth,
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > Thanks for the KIP. I
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > only
>> > >>> > started
>> > >>> > > > > > > reviewing
>> > >>> > > > > > > > > >> this and
>> > >>> > > > > > > > > >> >> > > may
>> > >>> > > > > > > > > >> >> > > > > have
>> > >>> > > > > > > > > >> >> > > > > > > > > >> additional
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > questions later. The
>> > >>> immediate
>> > >>> > > > > question
>> > >>> > > > > > > that
>> > >>> > > > > > > > > >> came to
>> > >>> > > > > > > > > >> >> > > mind
>> > >>> > > > > > > > > >> >> > > > is
>> > >>> > > > > > > > > >> >> > > > > > our
>> > >>> > > > > > > > > >> >> > > > > > > > > >> choice of
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > "DIGEST-MD5" even
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > though
>> > it's
>> > >>> > > marked
>> > >>> > > > > as
>> > >>> > > > > > > > > >> OBSOLETE in
>> > >>> > > > > > > > > >> >> > the
>> > >>> > > > > > > > > >> >> > > > IANA
>> > >>> > > > > > > > > >> >> > > > > > > > > Registry
>> > >>> > > > > > > > > >> >> > > > > > > > > >> of
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > SASL mechanisms and the
>> > >>> original
>> > >>> > > RFC
>> > >>> > > > > > > (2831)
>> > >>> > > > > > > > > has
>> > >>> > > > > > > > > >> been
>> > >>> > > > > > > > > >> >> > > moved
>> > >>> > > > > > > > > >> >> > > > > to
>> > >>> > > > > > > > > >> >> > > > > > > > > Historic
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > status:
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
>> > https://tools.ietf.org/html/
>> > >>> > > rfc6331
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
>> > >>> > > > > > > > > >> >> > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > >
>> > >>> > > > > > > > > >> >> > >
>> > >>> > > > > > > > > >>
>> > >>> > > > > > >
>> > >>> > > > http://www.iana.org/assignments/sasl-mechanisms/sasl-
>> > >>> mechanisms.xhtml
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > What is the reasoning
>> > behind
>> > >>> > that
>> > >>> > > > > > choice?
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > Thanks,
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > Ismael
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > On Fri, May 13, 2016 at
>> > 11:29
>> > >>> > PM,
>> > >>> > > > Gwen
>> > >>> > > > > > > > > Shapira <
>> > >>> > > > > > > > > >> >> > > > > > > gwen@confluent.io
>> > >>> > > > > > > > > >> >> > > > > > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > >> wrote:
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > Also comments inline
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > :)
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > * I want to
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > emphasize
>> > >>> that
>> > >>> > > even
>> > >>> > > > > > though
>> > >>> > > > > > > > > >> delegation
>> > >>> > > > > > > > > >> >> > > > tokens
>> > >>> > > > > > > > > >> >> > > > > > > are a
>> > >>> > > > > > > > > >> >> > > > > > > > > >> Hadoop
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > innovation, I feel
>> > very
>> > >>> > > strongly
>> > >>> > > > > > about
>> > >>> > > > > > > > not
>> > >>> > > > > > > > > >> adding
>> > >>> > > > > > > > > >> >> > > > > > dependency
>> > >>> > > > > > > > > >> >> > > > > > > > on
>> > >>> > > > > > > > > >> >> > > > > > > > > >> Hadoop
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > when implementing
>> > >>> delegation
>> > >>> > > > > tokens
>> > >>> > > > > > > for
>> > >>> > > > > > > > > >> Kafka. The
>> > >>> > > > > > > > > >> >> > > KIP
>> > >>> > > > > > > > > >> >> > > > > > > doesn't
>> > >>> > > > > > > > > >> >> > > > > > > > > >> imply
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > such dependency,
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > but
>> > if
>> > >>> you
>> > >>> > > can
>> > >>> > > > > > > > clarify...
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *No hadoop
>> > dependency.*
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > Yay! Just add this to
>> > the
>> > >>> KIP
>> > >>> > so
>> > >>> > > > no
>> > >>> > > > > > one
>> > >>> > > > > > > > will
>> > >>> > > > > > > > > >> read
>> > >>> > > > > > > > > >> >> > the
>> > >>> > > > > > > > > >> >> > > > KIP
>> > >>> > > > > > > > > >> >> > > > > > and
>> > >>> > > > > > > > > >> >> > > > > > > > > panic
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > three weeks before
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > the
>> > next
>> > >>> > > > > release...
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > * Can we get
>> > delegation
>> > >>> > token
>> > >>> > > at
>> > >>> > > > > any
>> > >>> > > > > > > > time
>> > >>> > > > > > > > > >> after
>> > >>> > > > > > > > > >> >> > > > > > > > authenticating?
>> > >>> > > > > > > > > >> >> > > > > > > > > >> only
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > immediately after?
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *As long as you are
>> > >>> > > > authenticated
>> > >>> > > > > > you
>> > >>> > > > > > > > can
>> > >>> > > > > > > > > >> get
>> > >>> > > > > > > > > >> >> > > > delegation
>> > >>> > > > > > > > > >> >> > > > > > > > tokens.
>> > >>> > > > > > > > > >> >> > > > > > > > > >> We
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > need
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > to
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > discuss if a client
>> > >>> > > > authenticated
>> > >>> > > > > > > using
>> > >>> > > > > > > > > >> delegation
>> > >>> > > > > > > > > >> >> > > > > token,
>> > >>> > > > > > > > > >> >> > > > > > > can
>> > >>> > > > > > > > > >> >> > > > > > > > > also
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > acquire
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > delegation token
>> > again or
>> > >>> > not.
>> > >>> > > > > Also
>> > >>> > > > > > > > there
>> > >>> > > > > > > > > >> is the
>> > >>> > > > > > > > > >> >> > > > > question
>> > >>> > > > > > > > > >> >> > > > > > of
>> > >>> > > > > > > > > >> >> > > > > > > > do
>> > >>> > > > > > > > > >> >> > > > > > > > > we
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > allow
>> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > anyone to acquire
>> > >>> delegation
>> > >>> > > > token
>> > >>> > > > > > or
>> > >>> > > > > > > we
>> > >>> > > > > > > > > >> want
>> > >>> > > > > > > > > >> >> > > specific
>> > >>> > > > > > > > > >>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Harsha Chintalapani <ka...@harsha.io>]

The only pending update for the KIP is to write up the protocol changes
like we've it KIP-4. I'll update the wiki.

On Thu, Sep 15, 2016 at 4:27 PM Ashish Singh <as...@cloudera.com> wrote:

> I think we decided to not support secret rotation, I guess this can be
> stated clearly on the KIP. Also, more details on how clients will perform
> token distribution and how CLI will look like will be helpful.
>
> On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <gw...@confluent.io> wrote:
>
> > Hi Guys,
> >
> > This discussion was dead for a while. Are there still contentious
> > points? If not, why are there no votes?
> >
> > On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <ju...@confluent.io> wrote:
> > > Ashish,
> > >
> > > Yes, I will send out a KIP invite for next week to discuss KIP-48 and
> > other
> > > remaining KIPs.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh <as...@cloudera.com>
> > wrote:
> > >
> > >> Thanks Harsha!
> > >>
> > >> Jun, can we add KIP-48 to next KIP hangout's agenda. Also, we did not
> > >> actually make a call on when we should have next KIP call. As there
> are
> > a
> > >> few outstanding KIPs that could not be discussed this week, can we
> have
> > a
> > >> KIP hangout call next week?
> > >>
> > >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha Chintalapani <kafka@harsha.io
> >
> > >> wrote:
> > >>
> > >>> Ashish,
> > >>>         Yes we are working on it. Lets discuss in the next KIP
> meeting.
> > >>> I'll join.
> > >>> -Harsha
> > >>>
> > >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh <as...@cloudera.com>
> > >>> wrote:
> > >>>
> > >>> > Hello Harsha,
> > >>> >
> > >>> > Are you still working on this? Wondering if we can discuss this in
> > next
> > >>> KIP
> > >>> > meeting, if you can join.
> > >>> >
> > >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha Chintalapani <
> > kafka@harsha.io>
> > >>> > wrote:
> > >>> >
> > >>> > > Hi Grant,
> > >>> > >           We are working on it. Will add the details to KIP about
> > the
> > >>> > > request protocol.
> > >>> > >
> > >>> > > Thanks,
> > >>> > > Harsha
> > >>> > >
> > >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant Henke <ghenke@cloudera.com
> >
> > >>> wrote:
> > >>> > >
> > >>> > > > Hi Parth,
> > >>> > > >
> > >>> > > > Are you still working on this? If you need any help please
> don't
> > >>> > hesitate
> > >>> > > > to ask.
> > >>> > > >
> > >>> > > > Thanks,
> > >>> > > > Grant
> > >>> > > >
> > >>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <ju...@confluent.io>
> > wrote:
> > >>> > > >
> > >>> > > > > Parth,
> > >>> > > > >
> > >>> > > > > Thanks for the reply.
> > >>> > > > >
> > >>> > > > > It makes sense to only allow the renewal by users that
> > >>> authenticated
> > >>> > > > using
> > >>> > > > > *non* delegation token mechanism. Then, should we make the
> > >>> renewal a
> > >>> > > > list?
> > >>> > > > > For example, in the case of rest proxy, it will be useful for
> > >>> every
> > >>> > > > > instance of rest proxy to be able to renew the tokens.
> > >>> > > > >
> > >>> > > > > It would be clearer if we can document the request protocol
> > like
> > >>> > > > >
> > >>> > > > >
> > >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >>> > > 4+-+Command+line+and+centralized+administrative+operations#KIP-4-
> > >>> > > Commandlineandcentralizedadministrativeoperations-
> > >>> > > CreateTopicsRequest(KAFKA-2945):(VotedandPlannedforin0.10.1.0)
> > >>> > > > > .
> > >>> > > > >
> > >>> > > > > It would also be useful to document the client APIs.
> > >>> > > > >
> > >>> > > > > Thanks,
> > >>> > > > >
> > >>> > > > > Jun
> > >>> > > > >
> > >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM, parth brahmbhatt <
> > >>> > > > > brahmbhatt.parth@gmail.com> wrote:
> > >>> > > > >
> > >>> > > > > > Hi,
> > >>> > > > > >
> > >>> > > > > > I am suggesting that we will only allow the renewal by
> users
> > >>> that
> > >>> > > > > > authenticated using *non* delegation token mechanism. For
> > >>> example,
> > >>> > If
> > >>> > > > > user
> > >>> > > > > > Alice authenticated using kerberos and requested delegation
> > >>> tokens,
> > >>> > > > only
> > >>> > > > > > user Alice authenticated via non delegation token mechanism
> > can
> > >>> > > renew.
> > >>> > > > > > Clients that have  access to delegation tokens can not
> issue
> > >>> > renewal
> > >>> > > > > > request for renewing their own token and this is primarily
> > >>> > important
> > >>> > > to
> > >>> > > > > > reduce the time window for which a compromised token will
> be
> > >>> valid.
> > >>> > > > > >
> > >>> > > > > > To clarify, Yes any authenticated user can request
> delegation
> > >>> > tokens
> > >>> > > > but
> > >>> > > > > > even here I would recommend to avoid creating a chain
> where a
> > >>> > client
> > >>> > > > > > authenticated via delegation token request for more
> > delegation
> > >>> > > tokens.
> > >>> > > > > > Basically anyone can request delegation token, as long as
> > they
> > >>> > > > > authenticate
> > >>> > > > > > via a non delegation token mechanism.
> > >>> > > > > >
> > >>> > > > > > Aren't classes listed here
> > >>> > > > > > <
> > >>> > > > > >
> > >>> > > > >
> > >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >>> > > 48+Delegation+token+support+for+Kafka#KIP-48Delegationtokens
> > >>> upportforKaf
> > >>> > > ka-PublicInterfaces
> > >>> > > > > > >
> > >>> > > > > > sufficient?
> > >>> > > > > >
> > >>> > > > > > Thanks
> > >>> > > > > > Parth
> > >>> > > > > >
> > >>> > > > > >
> > >>> > > > > >
> > >>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao <jun@confluent.io
> >
> > >>> wrote:
> > >>> > > > > >
> > >>> > > > > > > Parth,
> > >>> > > > > > >
> > >>> > > > > > > Thanks for the reply. A couple of comments inline below.
> > >>> > > > > > >
> > >>> > > > > > > On Tue, Jun 21, 2016 at 10:36 AM, parth brahmbhatt <
> > >>> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > >>> > > > > > >
> > >>> > > > > > > > 1. Who / how are tokens renewed? By original requester
> > >>> only? or
> > >>> > > > using
> > >>> > > > > > > > Kerberos
> > >>> > > > > > > > auth only?
> > >>> > > > > > > > My recommendation is to do this only using Kerberos
> auth
> > and
> > >>> > only
> > >>> > > > > threw
> > >>> > > > > > > the
> > >>> > > > > > > > renewer specified during the acquisition request.
> > >>> > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > > Hmm, not sure that I follow this. Are you saying that any
> > >>> client
> > >>> > > > > > > authenticated with the delegation token can renew, i.e.
> > there
> > >>> is
> > >>> > no
> > >>> > > > > > renewer
> > >>> > > > > > > needed?
> > >>> > > > > > >
> > >>> > > > > > > Also, just to be clear, any authenticated client (either
> > >>> through
> > >>> > > SASL
> > >>> > > > > or
> > >>> > > > > > > SSL) can request a delegation token for the authenticated
> > >>> user,
> > >>> > > > right?
> > >>> > > > > > >
> > >>> > > > > > >
> > >>> > > > > > > > 2. Are tokens stored on each broker or in ZK?
> > >>> > > > > > > > My recommendation is still to store in ZK or not store
> > them
> > >>> at
> > >>> > > all.
> > >>> > > > > The
> > >>> > > > > > > > whole controller based distribution is too much
> overhead
> > >>> with
> > >>> > not
> > >>> > > > > much
> > >>> > > > > > to
> > >>> > > > > > > > achieve.
> > >>> > > > > > > >
> > >>> > > > > > > > 3. How are tokens invalidated / expired?
> > >>> > > > > > > > Either by expiration time out or through an explicit
> > >>> request to
> > >>> > > > > > > invalidate.
> > >>> > > > > > > >
> > >>> > > > > > > > 4. Which encryption algorithm is used?
> > >>> > > > > > > > SCRAM
> > >>> > > > > > > >
> > >>> > > > > > > > 5. What is the impersonation proposal (it wasn't in the
> > KIP
> > >>> but
> > >>> > > was
> > >>> > > > > > > > discussed
> > >>> > > > > > > > in this thread)?
> > >>> > > > > > > > There is no imperonation proposal. I tried and
> explained
> > how
> > >>> > its
> > >>> > > a
> > >>> > > > > > > > different problem and why its not really necessary to
> > >>> discuss
> > >>> > > that
> > >>> > > > as
> > >>> > > > > > > part
> > >>> > > > > > > > of this KIP.  This KIP will not support any
> > impersonation,
> > >>> it
> > >>> > > will
> > >>> > > > > just
> > >>> > > > > > > be
> > >>> > > > > > > > another way to authenticate.
> > >>> > > > > > > >
> > >>> > > > > > > > 6. Do we need new ACLs, if so - for what actions?
> > >>> > > > > > > > We do not need new ACLs.
> > >>> > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > > Could we document the format of the new request/response
> > and
> > >>> > their
> > >>> > > > > > > associated Resource and Operation for ACL?
> > >>> > > > > > >
> > >>> > > > > > >
> > >>> > > > > > > > 7. How would the delegation token be configured in the
> > >>> client?
> > >>> > > > > > > > Should be through config. I wasn't planning on
> supporting
> > >>> JAAS
> > >>> > > for
> > >>> > > > > > > tokens.
> > >>> > > > > > > > I don't believe hadoop does this either.
> > >>> > > > > > > >
> > >>> > > > > > > > Thanks
> > >>> > > > > > > > Parth
> > >>> > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <
> > jun@confluent.io>
> > >>> > > wrote:
> > >>> > > > > > > >
> > >>> > > > > > > > > Harsha,
> > >>> > > > > > > > >
> > >>> > > > > > > > > Another question.
> > >>> > > > > > > > >
> > >>> > > > > > > > > 9. How would the delegation token be configured in
> the
> > >>> > client?
> > >>> > > > The
> > >>> > > > > > > > standard
> > >>> > > > > > > > > way is to do this through JAAS. However, we will need
> > to
> > >>> > think
> > >>> > > > > > through
> > >>> > > > > > > if
> > >>> > > > > > > > > this is convenient in a shared environment. For
> > example,
> > >>> > when a
> > >>> > > > new
> > >>> > > > > > > task
> > >>> > > > > > > > is
> > >>> > > > > > > > > added to a Storm worker node, do we need to
> dynamically
> > >>> add a
> > >>> > > new
> > >>> > > > > > > section
> > >>> > > > > > > > > in the JAAS file? It may be more convenient if we can
> > >>> pass in
> > >>> > > the
> > >>> > > > > > token
> > >>> > > > > > > > > through the config directly w/o going through JAAS.
> > >>> > > > > > > > >
> > >>> > > > > > > > > Are you or Parth still actively working on this KIP?
> > >>> > > > > > > > >
> > >>> > > > > > > > > Thanks,
> > >>> > > > > > > > >
> > >>> > > > > > > > > Jun
> > >>> > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <
> > >>> jun@confluent.io>
> > >>> > > > wrote:
> > >>> > > > > > > > >
> > >>> > > > > > > > > > Just to add on that list.
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > 2. It would be good to document the format of the
> > data
> > >>> > stored
> > >>> > > > in
> > >>> > > > > > ZK.
> > >>> > > > > > > > > > 7. Earlier, there was a discussion on whether the
> > tokens
> > >>> > > should
> > >>> > > > > be
> > >>> > > > > > > > > > propagated through ZK like config/acl/quota, or
> > through
> > >>> the
> > >>> > > > > > > controller.
> > >>> > > > > > > > > > Currently, the controller is only designed for
> > >>> propagating
> > >>> > > > topic
> > >>> > > > > > > > > metadata,
> > >>> > > > > > > > > > but not other data.
> > >>> > > > > > > > > > 8. Should we use SCRAM to send the token instead of
> > >>> > > DIGEST-MD5
> > >>> > > > > > since
> > >>> > > > > > > > it's
> > >>> > > > > > > > > > deprecated?
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > Also, the images in the wiki seem broken.
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > Thanks,
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > Jun
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <
> > >>> > > > > gwen@confluent.io>
> > >>> > > > > > > > > wrote:
> > >>> > > > > > > > > >
> > >>> > > > > > > > > >> From what I can see, remaining questions are:
> > >>> > > > > > > > > >>
> > >>> > > > > > > > > >> 1. Who / how are tokens renewed? By original
> > requester
> > >>> > only?
> > >>> > > > or
> > >>> > > > > > > using
> > >>> > > > > > > > > >> Kerberos auth only?
> > >>> > > > > > > > > >> 2. Are tokens stored on each broker or in ZK?
> > >>> > > > > > > > > >> 3. How are tokens invalidated / expired?
> > >>> > > > > > > > > >> 4. Which encryption algorithm is used?
> > >>> > > > > > > > > >> 5. What is the impersonation proposal (it wasn't
> in
> > the
> > >>> > KIP
> > >>> > > > but
> > >>> > > > > > was
> > >>> > > > > > > > > >> discussed in this thread)?
> > >>> > > > > > > > > >> 6. Do we need new ACLs, if so - for what actions?
> > >>> > > > > > > > > >>
> > >>> > > > > > > > > >> Gwen
> > >>> > > > > > > > > >>
> > >>> > > > > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <
> > >>> kafka@harsha.io>
> > >>> > > > wrote:
> > >>> > > > > > > > > >> > Jun & Ismael,
> > >>> > > > > > > > > >> >                          Unfortunately I
> couldn't
> > >>> attend
> > >>> > > the
> > >>> > > > > KIP
> > >>> > > > > > > > > meeting
> > >>> > > > > > > > > >> >                          when delegation tokens
> > >>> > discussed.
> > >>> > > > > > > > Appreciate
> > >>> > > > > > > > > if
> > >>> > > > > > > > > >> >                          you can update the
> > thread if
> > >>> > you
> > >>> > > > have
> > >>> > > > > > any
> > >>> > > > > > > > > >> >                          further questions.
> > >>> > > > > > > > > >> > Thanks,
> > >>> > > > > > > > > >> > Harsha
> > >>> > > > > > > > > >> >
> > >>> > > > > > > > > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei
> > wrote:
> > >>> > > > > > > > > >> >> It seems that the links to images in the KIP
> are
> > >>> > broken.
> > >>> > > > > > > > > >> >>
> > >>> > > > > > > > > >> >> Liquan
> > >>> > > > > > > > > >> >>
> > >>> > > > > > > > > >> >> On Tue, May 24, 2016 at 9:33 AM, parth
> > brahmbhatt <
> > >>> > > > > > > > > >> >> brahmbhatt.parth@gmail.com> wrote:
> > >>> > > > > > > > > >> >>
> > >>> > > > > > > > > >> >> > 110. What does getDelegationTokenAs mean?
> > >>> > > > > > > > > >> >> > In the current proposal we only allow a user
> to
> > >>> get
> > >>> > > > > > delegation
> > >>> > > > > > > > > token
> > >>> > > > > > > > > >> for
> > >>> > > > > > > > > >> >> > the identity that it authenticated as using
> > >>> another
> > >>> > > > > > mechanism,
> > >>> > > > > > > > i.e.
> > >>> > > > > > > > > >> A user
> > >>> > > > > > > > > >> >> > that authenticate using a keytab for
> principal
> > >>> > > > > > > user1@EXAMPLE.COM
> > >>> > > > > > > > > >> will get
> > >>> > > > > > > > > >> >> > delegation tokens for that user only. In
> > future I
> > >>> > think
> > >>> > > > we
> > >>> > > > > > will
> > >>> > > > > > > > > have
> > >>> > > > > > > > > >> to
> > >>> > > > > > > > > >> >> > extend support such that we allow some set of
> > >>> users (
> > >>> > > > > > > > > >> >> > kafka-rest-user@EXAMPLE.COM,
> > >>> > storm-nimbus@EXAMPLE.COM)
> > >>> > > > to
> > >>> > > > > > > > acquire
> > >>> > > > > > > > > >> >> > delegation tokens on behalf of other users
> > whose
> > >>> > > identity
> > >>> > > > > > they
> > >>> > > > > > > > have
> > >>> > > > > > > > > >> >> > verified independently.  Kafka brokers will
> > have
> > >>> ACLs
> > >>> > > to
> > >>> > > > > > > control
> > >>> > > > > > > > > >> which
> > >>> > > > > > > > > >> >> > users are allowed to impersonate other users
> > and
> > >>> get
> > >>> > > > tokens
> > >>> > > > > > on
> > >>> > > > > > > > > >> behalf of
> > >>> > > > > > > > > >> >> > them. Overall Impersonation is a whole
> > different
> > >>> > > problem
> > >>> > > > in
> > >>> > > > > > my
> > >>> > > > > > > > > >> opinion and
> > >>> > > > > > > > > >> >> > I think we can tackle it in separate KIP.
> > >>> > > > > > > > > >> >> >
> > >>> > > > > > > > > >> >> > 111. What's the typical rate of getting and
> > >>> renewing
> > >>> > > > > > delegation
> > >>> > > > > > > > > >> tokens?
> > >>> > > > > > > > > >> >> > Typically this should be very very low, 1
> > request
> > >>> per
> > >>> > > > > minute
> > >>> > > > > > > is a
> > >>> > > > > > > > > >> >> > relatively high estimate. However it depends
> on
> > >>> the
> > >>> > > token
> > >>> > > > > > > > > >> expiration. I am
> > >>> > > > > > > > > >> >> > less worried about the extra load it puts on
> > >>> > controller
> > >>> > > > vs
> > >>> > > > > > the
> > >>> > > > > > > > > added
> > >>> > > > > > > > > >> >> > complexity and the value it offers.
> > >>> > > > > > > > > >> >> >
> > >>> > > > > > > > > >> >> > Thanks
> > >>> > > > > > > > > >> >> > Parth
> > >>> > > > > > > > > >> >> >
> > >>> > > > > > > > > >> >> >
> > >>> > > > > > > > > >> >> >
> > >>> > > > > > > > > >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma
> <
> > >>> > > > > > > ismael@juma.me.uk>
> > >>> > > > > > > > > >> wrote:
> > >>> > > > > > > > > >> >> >
> > >>> > > > > > > > > >> >> > > Thanks Rajini. It would probably require a
> > >>> separate
> > >>> > > KIP
> > >>> > > > > as
> > >>> > > > > > it
> > >>> > > > > > > > > will
> > >>> > > > > > > > > >> >> > > introduce user visible changes. We could
> also
> > >>> > update
> > >>> > > > > KIP-48
> > >>> > > > > > > to
> > >>> > > > > > > > > >> have this
> > >>> > > > > > > > > >> >> > > information, but it seems cleaner to do it
> > >>> > > separately.
> > >>> > > > We
> > >>> > > > > > can
> > >>> > > > > > > > > >> discuss
> > >>> > > > > > > > > >> >> > that
> > >>> > > > > > > > > >> >> > > in the KIP call today.
> > >>> > > > > > > > > >> >> > >
> > >>> > > > > > > > > >> >> > > Ismael
> > >>> > > > > > > > > >> >> > >
> > >>> > > > > > > > > >> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini
> > Sivaram
> > >>> <
> > >>> > > > > > > > > >> >> > > rajinisivaram@googlemail.com> wrote:
> > >>> > > > > > > > > >> >> > >
> > >>> > > > > > > > > >> >> > > > Ismael,
> > >>> > > > > > > > > >> >> > > >
> > >>> > > > > > > > > >> >> > > > I have created a JIRA (
> > >>> > > > > > > > > >> >> > https://issues.apache.org/
> > jira/browse/KAFKA-3751)
> > >>> > > > > > > > > >> >> > > > for adding SCRAM as a SASL mechanism.
> Would
> > >>> that
> > >>> > > need
> > >>> > > > > > > another
> > >>> > > > > > > > > >> KIP? If
> > >>> > > > > > > > > >> >> > > > KIP-48 will use this mechanism, can this
> > just
> > >>> be
> > >>> > a
> > >>> > > > JIRA
> > >>> > > > > > > that
> > >>> > > > > > > > > gets
> > >>> > > > > > > > > >> >> > > reviewed
> > >>> > > > > > > > > >> >> > > > when the PR is ready?
> > >>> > > > > > > > > >> >> > > >
> > >>> > > > > > > > > >> >> > > > Thank you,
> > >>> > > > > > > > > >> >> > > >
> > >>> > > > > > > > > >> >> > > > Rajini
> > >>> > > > > > > > > >> >> > > >
> > >>> > > > > > > > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael
> > Juma <
> > >>> > > > > > > > > ismael@juma.me.uk>
> > >>> > > > > > > > > >> >> > wrote:
> > >>> > > > > > > > > >> >> > > >
> > >>> > > > > > > > > >> >> > > > > Thanks Rajini, SCRAM seems like a good
> > >>> > candidate.
> > >>> > > > > > > > > >> >> > > > >
> > >>> > > > > > > > > >> >> > > > > Gwen had independently mentioned this
> as
> > a
> > >>> SASL
> > >>> > > > > > mechanism
> > >>> > > > > > > > > that
> > >>> > > > > > > > > >> might
> > >>> > > > > > > > > >> >> > be
> > >>> > > > > > > > > >> >> > > > > useful for Kafka and I have been
> meaning
> > to
> > >>> > check
> > >>> > > > it
> > >>> > > > > in
> > >>> > > > > > > > more
> > >>> > > > > > > > > >> detail.
> > >>> > > > > > > > > >> >> > > Good
> > >>> > > > > > > > > >> >> > > > > to know that you are willing to
> > contribute
> > >>> an
> > >>> > > > > > > > implementation.
> > >>> > > > > > > > > >> Maybe
> > >>> > > > > > > > > >> >> > we
> > >>> > > > > > > > > >> >> > > > > should file a separate JIRA for this?
> > >>> > > > > > > > > >> >> > > > >
> > >>> > > > > > > > > >> >> > > > > Ismael
> > >>> > > > > > > > > >> >> > > > >
> > >>> > > > > > > > > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini
> > >>> > Sivaram <
> > >>> > > > > > > > > >> >> > > > > rajinisivaram@googlemail.com> wrote:
> > >>> > > > > > > > > >> >> > > > >
> > >>> > > > > > > > > >> >> > > > > > SCRAM (Salted Challenge Response
> > >>> > Authentication
> > >>> > > > > > > > Mechanism)
> > >>> > > > > > > > > >> is a
> > >>> > > > > > > > > >> >> > > better
> > >>> > > > > > > > > >> >> > > > > > mechanism than Digest-MD5. Java
> doesn't
> > >>> come
> > >>> > > > with a
> > >>> > > > > > > > > built-in
> > >>> > > > > > > > > >> SCRAM
> > >>> > > > > > > > > >> >> > > > > > SaslServer or SaslClient, but I will
> be
> > >>> happy
> > >>> > > to
> > >>> > > > > add
> > >>> > > > > > > > > support
> > >>> > > > > > > > > >> in
> > >>> > > > > > > > > >> >> > Kafka
> > >>> > > > > > > > > >> >> > > > > since
> > >>> > > > > > > > > >> >> > > > > > it would be a useful mechanism to
> > support
> > >>> > > anyway.
> > >>> > > > > > > > > >> >> > > > > > https://tools.ietf.org/html/rfc7677
> > >>> > describes
> > >>> > > > the
> > >>> > > > > > > > protocol
> > >>> > > > > > > > > >> for
> > >>> > > > > > > > > >> >> > > > > > SCRAM-SHA-256.
> > >>> > > > > > > > > >> >> > > > > >
> > >>> > > > > > > > > >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun
> > Rao <
> > >>> > > > > > > > jun@confluent.io
> > >>> > > > > > > > > >
> > >>> > > > > > > > > >> wrote:
> > >>> > > > > > > > > >> >> > > > > >
> > >>> > > > > > > > > >> >> > > > > > > Parth,
> > >>> > > > > > > > > >> >> > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > Thanks for the explanation. A
> couple
> > of
> > >>> > more
> > >>> > > > > > > questions.
> > >>> > > > > > > > > >> >> > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > 110. What does getDelegationTokenAs
> > >>> mean?
> > >>> > > > > > > > > >> >> > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > 111. What's the typical rate of
> > getting
> > >>> and
> > >>> > > > > > renewing
> > >>> > > > > > > > > >> delegation
> > >>> > > > > > > > > >> >> > > > tokens?
> > >>> > > > > > > > > >> >> > > > > > > That may have an impact on whether
> > they
> > >>> > > should
> > >>> > > > be
> > >>> > > > > > > > > directed
> > >>> > > > > > > > > >> to the
> > >>> > > > > > > > > >> >> > > > > > > controller.
> > >>> > > > > > > > > >> >> > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > Jun
> > >>> > > > > > > > > >> >> > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM,
> > parth
> > >>> > > > > brahmbhatt <
> > >>> > > > > > > > > >> >> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > >>> > > > > > > > > >> >> > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > Hi Jun,
> > >>> > > > > > > > > >> >> > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > Thanks for reviewing.
> > >>> > > > > > > > > >> >> > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > * We could add a Cluster action
> to
> > add
> > >>> > acls
> > >>> > > > on
> > >>> > > > > > who
> > >>> > > > > > > > can
> > >>> > > > > > > > > >> request
> > >>> > > > > > > > > >> >> > > > > > delegation
> > >>> > > > > > > > > >> >> > > > > > > > tokens. I don't see the use case
> > for
> > >>> that
> > >>> > > yet
> > >>> > > > > but
> > >>> > > > > > > > down
> > >>> > > > > > > > > >> the line
> > >>> > > > > > > > > >> >> > > > when
> > >>> > > > > > > > > >> >> > > > > we
> > >>> > > > > > > > > >> >> > > > > > > > start supporting
> > getDelegationTokenAs
> > >>> it
> > >>> > > will
> > >>> > > > > be
> > >>> > > > > > > > > >> necessary.
> > >>> > > > > > > > > >> >> > > > > > > > * Yes we recommend tokens to be
> > only
> > >>> > > > > > > used/distributed
> > >>> > > > > > > > > >> over
> > >>> > > > > > > > > >> >> > secure
> > >>> > > > > > > > > >> >> > > > > > > channels.
> > >>> > > > > > > > > >> >> > > > > > > > * Depending on what design we end
> > up
> > >>> > > choosing
> > >>> > > > > > > > > >> Invalidation will
> > >>> > > > > > > > > >> >> > > be
> > >>> > > > > > > > > >> >> > > > > > > > responsibility of every broker or
> > >>> > > controller.
> > >>> > > > > > > > > >> >> > > > > > > > * I am not sure if I documented
> > >>> somewhere
> > >>> > > > that
> > >>> > > > > > > > > >> invalidation
> > >>> > > > > > > > > >> >> > will
> > >>> > > > > > > > > >> >> > > > > > directly
> > >>> > > > > > > > > >> >> > > > > > > > go through zookeeper but that is
> > not
> > >>> the
> > >>> > > > > intent.
> > >>> > > > > > > > > >> Invalidation
> > >>> > > > > > > > > >> >> > > will
> > >>> > > > > > > > > >> >> > > > > > either
> > >>> > > > > > > > > >> >> > > > > > > > be request based or due to
> > >>> expiration. No
> > >>> > > > > direct
> > >>> > > > > > > > > >> zookeeper
> > >>> > > > > > > > > >> >> > > > > interaction
> > >>> > > > > > > > > >> >> > > > > > > from
> > >>> > > > > > > > > >> >> > > > > > > > any client.
> > >>> > > > > > > > > >> >> > > > > > > > * "Broker also stores the
> > >>> DelegationToken
> > >>> > > > > without
> > >>> > > > > > > the
> > >>> > > > > > > > > >> hmac in
> > >>> > > > > > > > > >> >> > the
> > >>> > > > > > > > > >> >> > > > > > > > zookeeper." : Sorry about the
> > >>> confusion.
> > >>> > > The
> > >>> > > > > sole
> > >>> > > > > > > > > >> purpose of
> > >>> > > > > > > > > >> >> > > > > zookeeper
> > >>> > > > > > > > > >> >> > > > > > in
> > >>> > > > > > > > > >> >> > > > > > > > this design is as distribution
> > channel
> > >>> > for
> > >>> > > > > tokens
> > >>> > > > > > > > > >> between all
> > >>> > > > > > > > > >> >> > > > brokers
> > >>> > > > > > > > > >> >> > > > > > > and a
> > >>> > > > > > > > > >> >> > > > > > > > layer that ensures only tokens
> that
> > >>> were
> > >>> > > > > > generated
> > >>> > > > > > > by
> > >>> > > > > > > > > >> making a
> > >>> > > > > > > > > >> >> > > > > request
> > >>> > > > > > > > > >> >> > > > > > > to a
> > >>> > > > > > > > > >> >> > > > > > > > broker will be accepted (more on
> > this
> > >>> in
> > >>> > > > second
> > >>> > > > > > > > > >> paragraph). The
> > >>> > > > > > > > > >> >> > > > token
> > >>> > > > > > > > > >> >> > > > > > > > consists of few elements (owner,
> > >>> renewer,
> > >>> > > > uuid
> > >>> > > > > ,
> > >>> > > > > > > > > >> expiration,
> > >>> > > > > > > > > >> >> > > hmac)
> > >>> > > > > > > > > >> >> > > > ,
> > >>> > > > > > > > > >> >> > > > > > one
> > >>> > > > > > > > > >> >> > > > > > > of
> > >>> > > > > > > > > >> >> > > > > > > > which is the finally generated
> hmac
> > >>> but
> > >>> > > hmac
> > >>> > > > it
> > >>> > > > > > > self
> > >>> > > > > > > > is
> > >>> > > > > > > > > >> >> > derivable
> > >>> > > > > > > > > >> >> > > > if
> > >>> > > > > > > > > >> >> > > > > > you
> > >>> > > > > > > > > >> >> > > > > > > > have all the other elements of
> the
> > >>> token
> > >>> > +
> > >>> > > > > secret
> > >>> > > > > > > key
> > >>> > > > > > > > > to
> > >>> > > > > > > > > >> >> > generate
> > >>> > > > > > > > > >> >> > > > > hmac.
> > >>> > > > > > > > > >> >> > > > > > > > Given zookeeper does not provide
> > SSL
> > >>> > > support
> > >>> > > > we
> > >>> > > > > > do
> > >>> > > > > > > > not
> > >>> > > > > > > > > >> want the
> > >>> > > > > > > > > >> >> > > > > entire
> > >>> > > > > > > > > >> >> > > > > > > > token to be wire transferred to
> > >>> zookeeper
> > >>> > > as
> > >>> > > > > that
> > >>> > > > > > > > will
> > >>> > > > > > > > > >> be an
> > >>> > > > > > > > > >> >> > > > insecure
> > >>> > > > > > > > > >> >> > > > > > > wire
> > >>> > > > > > > > > >> >> > > > > > > > transfer. Instead we only store
> all
> > >>> the
> > >>> > > other
> > >>> > > > > > > > elements
> > >>> > > > > > > > > >> of a
> > >>> > > > > > > > > >> >> > > > > delegation
> > >>> > > > > > > > > >> >> > > > > > > > tokens. Brokers can read these
> > >>> elements
> > >>> > and
> > >>> > > > > > because
> > >>> > > > > > > > > they
> > >>> > > > > > > > > >> also
> > >>> > > > > > > > > >> >> > > have
> > >>> > > > > > > > > >> >> > > > > > access
> > >>> > > > > > > > > >> >> > > > > > > > to secret key they will be able
> to
> > >>> > generate
> > >>> > > > > hmac
> > >>> > > > > > on
> > >>> > > > > > > > > >> their end.
> > >>> > > > > > > > > >> >> > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > One of the alternative proposed
> is
> > to
> > >>> > avoid
> > >>> > > > > > > zookeeper
> > >>> > > > > > > > > >> >> > > altogether. A
> > >>> > > > > > > > > >> >> > > > > > > Client
> > >>> > > > > > > > > >> >> > > > > > > > will call broker with required
> > >>> > information
> > >>> > > > > > (owner,
> > >>> > > > > > > > > >> renwer,
> > >>> > > > > > > > > >> >> > > > > expiration)
> > >>> > > > > > > > > >> >> > > > > > > and
> > >>> > > > > > > > > >> >> > > > > > > > get back (signed hmac, uuid).
> > Broker
> > >>> > won't
> > >>> > > > > store
> > >>> > > > > > > this
> > >>> > > > > > > > > in
> > >>> > > > > > > > > >> >> > > zookeeper.
> > >>> > > > > > > > > >> >> > > > > > From
> > >>> > > > > > > > > >> >> > > > > > > > this point a client can contact
> any
> > >>> > broker
> > >>> > > > with
> > >>> > > > > > all
> > >>> > > > > > > > the
> > >>> > > > > > > > > >> >> > > delegation
> > >>> > > > > > > > > >> >> > > > > > token
> > >>> > > > > > > > > >> >> > > > > > > > info (owner, rewner, expiration,
> > hmac,
> > >>> > > uuid)
> > >>> > > > > the
> > >>> > > > > > > > borker
> > >>> > > > > > > > > >> will
> > >>> > > > > > > > > >> >> > > > > regenerate
> > >>> > > > > > > > > >> >> > > > > > > the
> > >>> > > > > > > > > >> >> > > > > > > > hmac and as long as it matches
> with
> > >>> hmac
> > >>> > > > > > presented
> > >>> > > > > > > by
> > >>> > > > > > > > > >> client ,
> > >>> > > > > > > > > >> >> > > > broker
> > >>> > > > > > > > > >> >> > > > > > > will
> > >>> > > > > > > > > >> >> > > > > > > > allow the request to
> authenticate.
> > >>> Only
> > >>> > > > > problem
> > >>> > > > > > > with
> > >>> > > > > > > > > >> this
> > >>> > > > > > > > > >> >> > > approach
> > >>> > > > > > > > > >> >> > > > > is
> > >>> > > > > > > > > >> >> > > > > > if
> > >>> > > > > > > > > >> >> > > > > > > > the secret key is compromised any
> > >>> client
> > >>> > > can
> > >>> > > > > now
> > >>> > > > > > > > > generate
> > >>> > > > > > > > > >> >> > random
> > >>> > > > > > > > > >> >> > > > > tokens
> > >>> > > > > > > > > >> >> > > > > > > and
> > >>> > > > > > > > > >> >> > > > > > > > they will still be able to
> > >>> authenticate
> > >>> > as
> > >>> > > > any
> > >>> > > > > > user
> > >>> > > > > > > > > they
> > >>> > > > > > > > > >> like.
> > >>> > > > > > > > > >> >> > > with
> > >>> > > > > > > > > >> >> > > > > > > > zookeeper we guarantee that only
> > >>> tokens
> > >>> > > > > acquired
> > >>> > > > > > > via
> > >>> > > > > > > > a
> > >>> > > > > > > > > >> broker
> > >>> > > > > > > > > >> >> > > > (using
> > >>> > > > > > > > > >> >> > > > > > some
> > >>> > > > > > > > > >> >> > > > > > > > auth scheme other than delegation
> > >>> token)
> > >>> > > will
> > >>> > > > > be
> > >>> > > > > > > > > >> accepted. We
> > >>> > > > > > > > > >> >> > > need
> > >>> > > > > > > > > >> >> > > > to
> > >>> > > > > > > > > >> >> > > > > > > > discuss which proposal makes more
> > >>> sense
> > >>> > and
> > >>> > > > we
> > >>> > > > > > can
> > >>> > > > > > > go
> > >>> > > > > > > > > >> over it
> > >>> > > > > > > > > >> >> > in
> > >>> > > > > > > > > >> >> > > > > > > tomorrow's
> > >>> > > > > > > > > >> >> > > > > > > > meeting.
> > >>> > > > > > > > > >> >> > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > Also, can you forward the invite
> to
> > >>> me?
> > >>> > > > > > > > > >> >> > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > Thanks
> > >>> > > > > > > > > >> >> > > > > > > > Parth
> > >>> > > > > > > > > >> >> > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM,
> > Jun
> > >>> > Rao <
> > >>> > > > > > > > > >> jun@confluent.io>
> > >>> > > > > > > > > >> >> > > > wrote:
> > >>> > > > > > > > > >> >> > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > Thanks for the KIP. A few
> > comments.
> > >>> > > > > > > > > >> >> > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > 100. This potentially can be
> > useful
> > >>> for
> > >>> > > > Kafka
> > >>> > > > > > > > Connect
> > >>> > > > > > > > > >> and
> > >>> > > > > > > > > >> >> > Kafka
> > >>> > > > > > > > > >> >> > > > > rest
> > >>> > > > > > > > > >> >> > > > > > > > proxy
> > >>> > > > > > > > > >> >> > > > > > > > > where a worker agent will need
> to
> > >>> run a
> > >>> > > > task
> > >>> > > > > on
> > >>> > > > > > > > > behalf
> > >>> > > > > > > > > >> of a
> > >>> > > > > > > > > >> >> > > > client.
> > >>> > > > > > > > > >> >> > > > > > We
> > >>> > > > > > > > > >> >> > > > > > > > will
> > >>> > > > > > > > > >> >> > > > > > > > > likely need to change how those
> > >>> > services
> > >>> > > > use
> > >>> > > > > > > Kafka
> > >>> > > > > > > > > >> clients
> > >>> > > > > > > > > >> >> > > > > > > > > (producer/consumer). Instead
> of a
> > >>> > shared
> > >>> > > > > client
> > >>> > > > > > > per
> > >>> > > > > > > > > >> worker,
> > >>> > > > > > > > > >> >> > we
> > >>> > > > > > > > > >> >> > > > will
> > >>> > > > > > > > > >> >> > > > > > > need
> > >>> > > > > > > > > >> >> > > > > > > > a
> > >>> > > > > > > > > >> >> > > > > > > > > client per user task since the
> > >>> > > > authentication
> > >>> > > > > > > > happens
> > >>> > > > > > > > > >> at the
> > >>> > > > > > > > > >> >> > > > > > connection
> > >>> > > > > > > > > >> >> > > > > > > > > level. For Kafka Connect, the
> > >>> renewer
> > >>> > > will
> > >>> > > > be
> > >>> > > > > > the
> > >>> > > > > > > > > >> workers.
> > >>> > > > > > > > > >> >> > So,
> > >>> > > > > > > > > >> >> > > we
> > >>> > > > > > > > > >> >> > > > > > > > probably
> > >>> > > > > > > > > >> >> > > > > > > > > need to allow multiple
> renewers.
> > For
> > >>> > > Kafka
> > >>> > > > > rest
> > >>> > > > > > > > > proxy,
> > >>> > > > > > > > > >> the
> > >>> > > > > > > > > >> >> > > > renewer
> > >>> > > > > > > > > >> >> > > > > > can
> > >>> > > > > > > > > >> >> > > > > > > > > probably just be the creator of
> > the
> > >>> > > token.
> > >>> > > > > > > > > >> >> > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > 101. Do we need new acl on who
> > can
> > >>> > > request
> > >>> > > > > > > > delegation
> > >>> > > > > > > > > >> tokens?
> > >>> > > > > > > > > >> >> > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > 102. Do we recommend people to
> > send
> > >>> > > > > delegation
> > >>> > > > > > > > tokens
> > >>> > > > > > > > > >> in an
> > >>> > > > > > > > > >> >> > > > > encrypted
> > >>> > > > > > > > > >> >> > > > > > > > > channel?
> > >>> > > > > > > > > >> >> > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > 103. Who is responsible for
> > expiring
> > >>> > > > tokens,
> > >>> > > > > > > every
> > >>> > > > > > > > > >> broker?
> > >>> > > > > > > > > >> >> > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > 104. For invalidating tokens,
> > would
> > >>> it
> > >>> > be
> > >>> > > > > > better
> > >>> > > > > > > to
> > >>> > > > > > > > > do
> > >>> > > > > > > > > >> it in
> > >>> > > > > > > > > >> >> > a
> > >>> > > > > > > > > >> >> > > > > > request
> > >>> > > > > > > > > >> >> > > > > > > > > instead of going to ZK
> directly?
> > >>> > > > > > > > > >> >> > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > 105. The terminology of client
> in
> > >>> the
> > >>> > > wiki
> > >>> > > > > > > > sometimes
> > >>> > > > > > > > > >> refers
> > >>> > > > > > > > > >> >> > to
> > >>> > > > > > > > > >> >> > > > the
> > >>> > > > > > > > > >> >> > > > > > end
> > >>> > > > > > > > > >> >> > > > > > > > > client and some other times
> > refers
> > >>> to
> > >>> > the
> > >>> > > > > > client
> > >>> > > > > > > > > using
> > >>> > > > > > > > > >> the
> > >>> > > > > > > > > >> >> > > > > delegation
> > >>> > > > > > > > > >> >> > > > > > > > > tokens. It would be useful to
> > >>> > distinguish
> > >>> > > > > > between
> > >>> > > > > > > > the
> > >>> > > > > > > > > >> two.
> > >>> > > > > > > > > >> >> > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > 106. Could you explain the
> > sentence
> > >>> > > "Broker
> > >>> > > > > > also
> > >>> > > > > > > > > >> stores the
> > >>> > > > > > > > > >> >> > > > > > > > DelegationToken
> > >>> > > > > > > > > >> >> > > > > > > > > without the hmac in the
> > zookeeper."
> > >>> a
> > >>> > bit
> > >>> > > > > > more? I
> > >>> > > > > > > > > >> thought the
> > >>> > > > > > > > > >> >> > > > > > > delegation
> > >>> > > > > > > > > >> >> > > > > > > > > token is the hmac.
> > >>> > > > > > > > > >> >> > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > Thanks,
> > >>> > > > > > > > > >> >> > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > Jun
> > >>> > > > > > > > > >> >> > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > On Mon, May 23, 2016 at 9:22
> AM,
> > Jun
> > >>> > Rao
> > >>> > > <
> > >>> > > > > > > > > >> jun@confluent.io>
> > >>> > > > > > > > > >> >> > > > wrote:
> > >>> > > > > > > > > >> >> > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > > Hi, Harsha,
> > >>> > > > > > > > > >> >> > > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > > Just sent out a KIP meeting
> > >>> invite.
> > >>> > We
> > >>> > > > can
> > >>> > > > > > > > discuss
> > >>> > > > > > > > > >> this in
> > >>> > > > > > > > > >> >> > > the
> > >>> > > > > > > > > >> >> > > > > > > meeting
> > >>> > > > > > > > > >> >> > > > > > > > > > tomorrow.
> > >>> > > > > > > > > >> >> > > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > > Thanks,
> > >>> > > > > > > > > >> >> > > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > > Jun
> > >>> > > > > > > > > >> >> > > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47
> > AM,
> > >>> > > Harsha <
> > >>> > > > > > > > > >> kafka@harsha.io>
> > >>> > > > > > > > > >> >> > > > wrote:
> > >>> > > > > > > > > >> >> > > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > >> Hi All,
> > >>> > > > > > > > > >> >> > > > > > > > > >>            Can we have a KIP
> > >>> meeting
> > >>> > > > > around
> > >>> > > > > > > > this.
> > >>> > > > > > > > > >> The KIP
> > >>> > > > > > > > > >> >> > is
> > >>> > > > > > > > > >> >> > > > up
> > >>> > > > > > > > > >> >> > > > > > for
> > >>> > > > > > > > > >> >> > > > > > > > > >>            sometime and if
> > there
> > >>> are
> > >>> > > any
> > >>> > > > > > > > questions
> > >>> > > > > > > > > >> lets
> > >>> > > > > > > > > >> >> > > > quickly
> > >>> > > > > > > > > >> >> > > > > > hash
> > >>> > > > > > > > > >> >> > > > > > > > out
> > >>> > > > > > > > > >> >> > > > > > > > > >>            details.
> > >>> > > > > > > > > >> >> > > > > > > > > >>
> > >>> > > > > > > > > >> >> > > > > > > > > >> Thanks,
> > >>> > > > > > > > > >> >> > > > > > > > > >> Harsha
> > >>> > > > > > > > > >> >> > > > > > > > > >>
> > >>> > > > > > > > > >> >> > > > > > > > > >> On Thu, May 19, 2016, at
> 08:40
> > >>> AM,
> > >>> > > parth
> > >>> > > > > > > > > brahmbhatt
> > >>> > > > > > > > > >> wrote:
> > >>> > > > > > > > > >> >> > > > > > > > > >> > That is what the hadoop
> echo
> > >>> > system
> > >>> > > > uses
> > >>> > > > > > so
> > >>> > > > > > > no
> > >>> > > > > > > > > >> good
> > >>> > > > > > > > > >> >> > reason
> > >>> > > > > > > > > >> >> > > > > > really.
> > >>> > > > > > > > > >> >> > > > > > > > We
> > >>> > > > > > > > > >> >> > > > > > > > > >> > could
> > >>> > > > > > > > > >> >> > > > > > > > > >> > change it to whatever is
> the
> > >>> > newest
> > >>> > > > > > > > recommended
> > >>> > > > > > > > > >> standard
> > >>> > > > > > > > > >> >> > > is.
> > >>> > > > > > > > > >> >> > > > > > > > > >> >
> > >>> > > > > > > > > >> >> > > > > > > > > >> > Thanks
> > >>> > > > > > > > > >> >> > > > > > > > > >> > Parth
> > >>> > > > > > > > > >> >> > > > > > > > > >> >
> > >>> > > > > > > > > >> >> > > > > > > > > >> > On Thu, May 19, 2016 at
> 3:33
> > >>> AM,
> > >>> > > > Ismael
> > >>> > > > > > > Juma <
> > >>> > > > > > > > > >> >> > > > > ismael@juma.me.uk
> > >>> > > > > > > > > >> >> > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > wrote:
> > >>> > > > > > > > > >> >> > > > > > > > > >> >
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > Hi Parth,
> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > Thanks for the KIP. I
> only
> > >>> > started
> > >>> > > > > > > reviewing
> > >>> > > > > > > > > >> this and
> > >>> > > > > > > > > >> >> > > may
> > >>> > > > > > > > > >> >> > > > > have
> > >>> > > > > > > > > >> >> > > > > > > > > >> additional
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > questions later. The
> > >>> immediate
> > >>> > > > > question
> > >>> > > > > > > that
> > >>> > > > > > > > > >> came to
> > >>> > > > > > > > > >> >> > > mind
> > >>> > > > > > > > > >> >> > > > is
> > >>> > > > > > > > > >> >> > > > > > our
> > >>> > > > > > > > > >> >> > > > > > > > > >> choice of
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > "DIGEST-MD5" even though
> > it's
> > >>> > > marked
> > >>> > > > > as
> > >>> > > > > > > > > >> OBSOLETE in
> > >>> > > > > > > > > >> >> > the
> > >>> > > > > > > > > >> >> > > > IANA
> > >>> > > > > > > > > >> >> > > > > > > > > Registry
> > >>> > > > > > > > > >> >> > > > > > > > > >> of
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > SASL mechanisms and the
> > >>> original
> > >>> > > RFC
> > >>> > > > > > > (2831)
> > >>> > > > > > > > > has
> > >>> > > > > > > > > >> been
> > >>> > > > > > > > > >> >> > > moved
> > >>> > > > > > > > > >> >> > > > > to
> > >>> > > > > > > > > >> >> > > > > > > > > Historic
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > status:
> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> > https://tools.ietf.org/html/
> > >>> > > rfc6331
> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> > >>> > > > > > > > > >> >> > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > >
> > >>> > > > > > > > > >> >> > >
> > >>> > > > > > > > > >>
> > >>> > > > > > >
> > >>> > > > http://www.iana.org/assignments/sasl-mechanisms/sasl-
> > >>> mechanisms.xhtml
> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > What is the reasoning
> > behind
> > >>> > that
> > >>> > > > > > choice?
> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > Thanks,
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > Ismael
> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > On Fri, May 13, 2016 at
> > 11:29
> > >>> > PM,
> > >>> > > > Gwen
> > >>> > > > > > > > > Shapira <
> > >>> > > > > > > > > >> >> > > > > > > gwen@confluent.io
> > >>> > > > > > > > > >> >> > > > > > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > >> wrote:
> > >>> > > > > > > > > >> >> > > > > > > > > >> > >
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > Also comments inline
> :)
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > * I want to
> emphasize
> > >>> that
> > >>> > > even
> > >>> > > > > > though
> > >>> > > > > > > > > >> delegation
> > >>> > > > > > > > > >> >> > > > tokens
> > >>> > > > > > > > > >> >> > > > > > > are a
> > >>> > > > > > > > > >> >> > > > > > > > > >> Hadoop
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > innovation, I feel
> > very
> > >>> > > strongly
> > >>> > > > > > about
> > >>> > > > > > > > not
> > >>> > > > > > > > > >> adding
> > >>> > > > > > > > > >> >> > > > > > dependency
> > >>> > > > > > > > > >> >> > > > > > > > on
> > >>> > > > > > > > > >> >> > > > > > > > > >> Hadoop
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > when implementing
> > >>> delegation
> > >>> > > > > tokens
> > >>> > > > > > > for
> > >>> > > > > > > > > >> Kafka. The
> > >>> > > > > > > > > >> >> > > KIP
> > >>> > > > > > > > > >> >> > > > > > > doesn't
> > >>> > > > > > > > > >> >> > > > > > > > > >> imply
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > such dependency, but
> > if
> > >>> you
> > >>> > > can
> > >>> > > > > > > > clarify...
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *No hadoop
> > dependency.*
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > Yay! Just add this to
> > the
> > >>> KIP
> > >>> > so
> > >>> > > > no
> > >>> > > > > > one
> > >>> > > > > > > > will
> > >>> > > > > > > > > >> read
> > >>> > > > > > > > > >> >> > the
> > >>> > > > > > > > > >> >> > > > KIP
> > >>> > > > > > > > > >> >> > > > > > and
> > >>> > > > > > > > > >> >> > > > > > > > > panic
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > three weeks before the
> > next
> > >>> > > > > release...
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > * Can we get
> > delegation
> > >>> > token
> > >>> > > at
> > >>> > > > > any
> > >>> > > > > > > > time
> > >>> > > > > > > > > >> after
> > >>> > > > > > > > > >> >> > > > > > > > authenticating?
> > >>> > > > > > > > > >> >> > > > > > > > > >> only
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > immediately after?
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *As long as you are
> > >>> > > > authenticated
> > >>> > > > > > you
> > >>> > > > > > > > can
> > >>> > > > > > > > > >> get
> > >>> > > > > > > > > >> >> > > > delegation
> > >>> > > > > > > > > >> >> > > > > > > > tokens.
> > >>> > > > > > > > > >> >> > > > > > > > > >> We
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > need
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > to
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > discuss if a client
> > >>> > > > authenticated
> > >>> > > > > > > using
> > >>> > > > > > > > > >> delegation
> > >>> > > > > > > > > >> >> > > > > token,
> > >>> > > > > > > > > >> >> > > > > > > can
> > >>> > > > > > > > > >> >> > > > > > > > > also
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > acquire
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > delegation token
> > again or
> > >>> > not.
> > >>> > > > > Also
> > >>> > > > > > > > there
> > >>> > > > > > > > > >> is the
> > >>> > > > > > > > > >> >> > > > > question
> > >>> > > > > > > > > >> >> > > > > > of
> > >>> > > > > > > > > >> >> > > > > > > > do
> > >>> > > > > > > > > >> >> > > > > > > > > we
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > allow
> > >>> > > > > > > > > >> >> > > > > > > > > >> > > > > anyone to acquire
> > >>> delegation
> > >>> > > > token
> > >>> > > > > > or
> > >>> > > > > > > we
> > >>> > > > > > > > > >> want
> > >>> > > > > > > > > >> >> > > specific
> > >>> > > > > > > > > >>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Ashish Singh <as...@cloudera.com>]

I think we decided to not support secret rotation, I guess this can be
stated clearly on the KIP. Also, more details on how clients will perform
token distribution and how CLI will look like will be helpful.

On Thu, Sep 15, 2016 at 3:20 PM, Gwen Shapira <gw...@confluent.io> wrote:

> Hi Guys,
>
> This discussion was dead for a while. Are there still contentious
> points? If not, why are there no votes?
>
> On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <ju...@confluent.io> wrote:
> > Ashish,
> >
> > Yes, I will send out a KIP invite for next week to discuss KIP-48 and
> other
> > remaining KIPs.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh <as...@cloudera.com>
> wrote:
> >
> >> Thanks Harsha!
> >>
> >> Jun, can we add KIP-48 to next KIP hangout's agenda. Also, we did not
> >> actually make a call on when we should have next KIP call. As there are
> a
> >> few outstanding KIPs that could not be discussed this week, can we have
> a
> >> KIP hangout call next week?
> >>
> >> On Tue, Aug 23, 2016 at 1:10 PM, Harsha Chintalapani <ka...@harsha.io>
> >> wrote:
> >>
> >>> Ashish,
> >>>         Yes we are working on it. Lets discuss in the next KIP meeting.
> >>> I'll join.
> >>> -Harsha
> >>>
> >>> On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh <as...@cloudera.com>
> >>> wrote:
> >>>
> >>> > Hello Harsha,
> >>> >
> >>> > Are you still working on this? Wondering if we can discuss this in
> next
> >>> KIP
> >>> > meeting, if you can join.
> >>> >
> >>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha Chintalapani <
> kafka@harsha.io>
> >>> > wrote:
> >>> >
> >>> > > Hi Grant,
> >>> > >           We are working on it. Will add the details to KIP about
> the
> >>> > > request protocol.
> >>> > >
> >>> > > Thanks,
> >>> > > Harsha
> >>> > >
> >>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant Henke <gh...@cloudera.com>
> >>> wrote:
> >>> > >
> >>> > > > Hi Parth,
> >>> > > >
> >>> > > > Are you still working on this? If you need any help please don't
> >>> > hesitate
> >>> > > > to ask.
> >>> > > >
> >>> > > > Thanks,
> >>> > > > Grant
> >>> > > >
> >>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <ju...@confluent.io>
> wrote:
> >>> > > >
> >>> > > > > Parth,
> >>> > > > >
> >>> > > > > Thanks for the reply.
> >>> > > > >
> >>> > > > > It makes sense to only allow the renewal by users that
> >>> authenticated
> >>> > > > using
> >>> > > > > *non* delegation token mechanism. Then, should we make the
> >>> renewal a
> >>> > > > list?
> >>> > > > > For example, in the case of rest proxy, it will be useful for
> >>> every
> >>> > > > > instance of rest proxy to be able to renew the tokens.
> >>> > > > >
> >>> > > > > It would be clearer if we can document the request protocol
> like
> >>> > > > >
> >>> > > > >
> >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> > > 4+-+Command+line+and+centralized+administrative+operations#KIP-4-
> >>> > > Commandlineandcentralizedadministrativeoperations-
> >>> > > CreateTopicsRequest(KAFKA-2945):(VotedandPlannedforin0.10.1.0)
> >>> > > > > .
> >>> > > > >
> >>> > > > > It would also be useful to document the client APIs.
> >>> > > > >
> >>> > > > > Thanks,
> >>> > > > >
> >>> > > > > Jun
> >>> > > > >
> >>> > > > > On Tue, Jun 28, 2016 at 2:55 PM, parth brahmbhatt <
> >>> > > > > brahmbhatt.parth@gmail.com> wrote:
> >>> > > > >
> >>> > > > > > Hi,
> >>> > > > > >
> >>> > > > > > I am suggesting that we will only allow the renewal by users
> >>> that
> >>> > > > > > authenticated using *non* delegation token mechanism. For
> >>> example,
> >>> > If
> >>> > > > > user
> >>> > > > > > Alice authenticated using kerberos and requested delegation
> >>> tokens,
> >>> > > > only
> >>> > > > > > user Alice authenticated via non delegation token mechanism
> can
> >>> > > renew.
> >>> > > > > > Clients that have  access to delegation tokens can not issue
> >>> > renewal
> >>> > > > > > request for renewing their own token and this is primarily
> >>> > important
> >>> > > to
> >>> > > > > > reduce the time window for which a compromised token will be
> >>> valid.
> >>> > > > > >
> >>> > > > > > To clarify, Yes any authenticated user can request delegation
> >>> > tokens
> >>> > > > but
> >>> > > > > > even here I would recommend to avoid creating a chain where a
> >>> > client
> >>> > > > > > authenticated via delegation token request for more
> delegation
> >>> > > tokens.
> >>> > > > > > Basically anyone can request delegation token, as long as
> they
> >>> > > > > authenticate
> >>> > > > > > via a non delegation token mechanism.
> >>> > > > > >
> >>> > > > > > Aren't classes listed here
> >>> > > > > > <
> >>> > > > > >
> >>> > > > >
> >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> > > 48+Delegation+token+support+for+Kafka#KIP-48Delegationtokens
> >>> upportforKaf
> >>> > > ka-PublicInterfaces
> >>> > > > > > >
> >>> > > > > > sufficient?
> >>> > > > > >
> >>> > > > > > Thanks
> >>> > > > > > Parth
> >>> > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao <ju...@confluent.io>
> >>> wrote:
> >>> > > > > >
> >>> > > > > > > Parth,
> >>> > > > > > >
> >>> > > > > > > Thanks for the reply. A couple of comments inline below.
> >>> > > > > > >
> >>> > > > > > > On Tue, Jun 21, 2016 at 10:36 AM, parth brahmbhatt <
> >>> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> >>> > > > > > >
> >>> > > > > > > > 1. Who / how are tokens renewed? By original requester
> >>> only? or
> >>> > > > using
> >>> > > > > > > > Kerberos
> >>> > > > > > > > auth only?
> >>> > > > > > > > My recommendation is to do this only using Kerberos auth
> and
> >>> > only
> >>> > > > > threw
> >>> > > > > > > the
> >>> > > > > > > > renewer specified during the acquisition request.
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > Hmm, not sure that I follow this. Are you saying that any
> >>> client
> >>> > > > > > > authenticated with the delegation token can renew, i.e.
> there
> >>> is
> >>> > no
> >>> > > > > > renewer
> >>> > > > > > > needed?
> >>> > > > > > >
> >>> > > > > > > Also, just to be clear, any authenticated client (either
> >>> through
> >>> > > SASL
> >>> > > > > or
> >>> > > > > > > SSL) can request a delegation token for the authenticated
> >>> user,
> >>> > > > right?
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > > > 2. Are tokens stored on each broker or in ZK?
> >>> > > > > > > > My recommendation is still to store in ZK or not store
> them
> >>> at
> >>> > > all.
> >>> > > > > The
> >>> > > > > > > > whole controller based distribution is too much overhead
> >>> with
> >>> > not
> >>> > > > > much
> >>> > > > > > to
> >>> > > > > > > > achieve.
> >>> > > > > > > >
> >>> > > > > > > > 3. How are tokens invalidated / expired?
> >>> > > > > > > > Either by expiration time out or through an explicit
> >>> request to
> >>> > > > > > > invalidate.
> >>> > > > > > > >
> >>> > > > > > > > 4. Which encryption algorithm is used?
> >>> > > > > > > > SCRAM
> >>> > > > > > > >
> >>> > > > > > > > 5. What is the impersonation proposal (it wasn't in the
> KIP
> >>> but
> >>> > > was
> >>> > > > > > > > discussed
> >>> > > > > > > > in this thread)?
> >>> > > > > > > > There is no imperonation proposal. I tried and explained
> how
> >>> > its
> >>> > > a
> >>> > > > > > > > different problem and why its not really necessary to
> >>> discuss
> >>> > > that
> >>> > > > as
> >>> > > > > > > part
> >>> > > > > > > > of this KIP.  This KIP will not support any
> impersonation,
> >>> it
> >>> > > will
> >>> > > > > just
> >>> > > > > > > be
> >>> > > > > > > > another way to authenticate.
> >>> > > > > > > >
> >>> > > > > > > > 6. Do we need new ACLs, if so - for what actions?
> >>> > > > > > > > We do not need new ACLs.
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > Could we document the format of the new request/response
> and
> >>> > their
> >>> > > > > > > associated Resource and Operation for ACL?
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > > > 7. How would the delegation token be configured in the
> >>> client?
> >>> > > > > > > > Should be through config. I wasn't planning on supporting
> >>> JAAS
> >>> > > for
> >>> > > > > > > tokens.
> >>> > > > > > > > I don't believe hadoop does this either.
> >>> > > > > > > >
> >>> > > > > > > > Thanks
> >>> > > > > > > > Parth
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <
> jun@confluent.io>
> >>> > > wrote:
> >>> > > > > > > >
> >>> > > > > > > > > Harsha,
> >>> > > > > > > > >
> >>> > > > > > > > > Another question.
> >>> > > > > > > > >
> >>> > > > > > > > > 9. How would the delegation token be configured in the
> >>> > client?
> >>> > > > The
> >>> > > > > > > > standard
> >>> > > > > > > > > way is to do this through JAAS. However, we will need
> to
> >>> > think
> >>> > > > > > through
> >>> > > > > > > if
> >>> > > > > > > > > this is convenient in a shared environment. For
> example,
> >>> > when a
> >>> > > > new
> >>> > > > > > > task
> >>> > > > > > > > is
> >>> > > > > > > > > added to a Storm worker node, do we need to dynamically
> >>> add a
> >>> > > new
> >>> > > > > > > section
> >>> > > > > > > > > in the JAAS file? It may be more convenient if we can
> >>> pass in
> >>> > > the
> >>> > > > > > token
> >>> > > > > > > > > through the config directly w/o going through JAAS.
> >>> > > > > > > > >
> >>> > > > > > > > > Are you or Parth still actively working on this KIP?
> >>> > > > > > > > >
> >>> > > > > > > > > Thanks,
> >>> > > > > > > > >
> >>> > > > > > > > > Jun
> >>> > > > > > > > >
> >>> > > > > > > > >
> >>> > > > > > > > >
> >>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <
> >>> jun@confluent.io>
> >>> > > > wrote:
> >>> > > > > > > > >
> >>> > > > > > > > > > Just to add on that list.
> >>> > > > > > > > > >
> >>> > > > > > > > > > 2. It would be good to document the format of the
> data
> >>> > stored
> >>> > > > in
> >>> > > > > > ZK.
> >>> > > > > > > > > > 7. Earlier, there was a discussion on whether the
> tokens
> >>> > > should
> >>> > > > > be
> >>> > > > > > > > > > propagated through ZK like config/acl/quota, or
> through
> >>> the
> >>> > > > > > > controller.
> >>> > > > > > > > > > Currently, the controller is only designed for
> >>> propagating
> >>> > > > topic
> >>> > > > > > > > > metadata,
> >>> > > > > > > > > > but not other data.
> >>> > > > > > > > > > 8. Should we use SCRAM to send the token instead of
> >>> > > DIGEST-MD5
> >>> > > > > > since
> >>> > > > > > > > it's
> >>> > > > > > > > > > deprecated?
> >>> > > > > > > > > >
> >>> > > > > > > > > > Also, the images in the wiki seem broken.
> >>> > > > > > > > > >
> >>> > > > > > > > > > Thanks,
> >>> > > > > > > > > >
> >>> > > > > > > > > > Jun
> >>> > > > > > > > > >
> >>> > > > > > > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <
> >>> > > > > gwen@confluent.io>
> >>> > > > > > > > > wrote:
> >>> > > > > > > > > >
> >>> > > > > > > > > >> From what I can see, remaining questions are:
> >>> > > > > > > > > >>
> >>> > > > > > > > > >> 1. Who / how are tokens renewed? By original
> requester
> >>> > only?
> >>> > > > or
> >>> > > > > > > using
> >>> > > > > > > > > >> Kerberos auth only?
> >>> > > > > > > > > >> 2. Are tokens stored on each broker or in ZK?
> >>> > > > > > > > > >> 3. How are tokens invalidated / expired?
> >>> > > > > > > > > >> 4. Which encryption algorithm is used?
> >>> > > > > > > > > >> 5. What is the impersonation proposal (it wasn't in
> the
> >>> > KIP
> >>> > > > but
> >>> > > > > > was
> >>> > > > > > > > > >> discussed in this thread)?
> >>> > > > > > > > > >> 6. Do we need new ACLs, if so - for what actions?
> >>> > > > > > > > > >>
> >>> > > > > > > > > >> Gwen
> >>> > > > > > > > > >>
> >>> > > > > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <
> >>> kafka@harsha.io>
> >>> > > > wrote:
> >>> > > > > > > > > >> > Jun & Ismael,
> >>> > > > > > > > > >> >                          Unfortunately I couldn't
> >>> attend
> >>> > > the
> >>> > > > > KIP
> >>> > > > > > > > > meeting
> >>> > > > > > > > > >> >                          when delegation tokens
> >>> > discussed.
> >>> > > > > > > > Appreciate
> >>> > > > > > > > > if
> >>> > > > > > > > > >> >                          you can update the
> thread if
> >>> > you
> >>> > > > have
> >>> > > > > > any
> >>> > > > > > > > > >> >                          further questions.
> >>> > > > > > > > > >> > Thanks,
> >>> > > > > > > > > >> > Harsha
> >>> > > > > > > > > >> >
> >>> > > > > > > > > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei
> wrote:
> >>> > > > > > > > > >> >> It seems that the links to images in the KIP are
> >>> > broken.
> >>> > > > > > > > > >> >>
> >>> > > > > > > > > >> >> Liquan
> >>> > > > > > > > > >> >>
> >>> > > > > > > > > >> >> On Tue, May 24, 2016 at 9:33 AM, parth
> brahmbhatt <
> >>> > > > > > > > > >> >> brahmbhatt.parth@gmail.com> wrote:
> >>> > > > > > > > > >> >>
> >>> > > > > > > > > >> >> > 110. What does getDelegationTokenAs mean?
> >>> > > > > > > > > >> >> > In the current proposal we only allow a user to
> >>> get
> >>> > > > > > delegation
> >>> > > > > > > > > token
> >>> > > > > > > > > >> for
> >>> > > > > > > > > >> >> > the identity that it authenticated as using
> >>> another
> >>> > > > > > mechanism,
> >>> > > > > > > > i.e.
> >>> > > > > > > > > >> A user
> >>> > > > > > > > > >> >> > that authenticate using a keytab for principal
> >>> > > > > > > user1@EXAMPLE.COM
> >>> > > > > > > > > >> will get
> >>> > > > > > > > > >> >> > delegation tokens for that user only. In
> future I
> >>> > think
> >>> > > > we
> >>> > > > > > will
> >>> > > > > > > > > have
> >>> > > > > > > > > >> to
> >>> > > > > > > > > >> >> > extend support such that we allow some set of
> >>> users (
> >>> > > > > > > > > >> >> > kafka-rest-user@EXAMPLE.COM,
> >>> > storm-nimbus@EXAMPLE.COM)
> >>> > > > to
> >>> > > > > > > > acquire
> >>> > > > > > > > > >> >> > delegation tokens on behalf of other users
> whose
> >>> > > identity
> >>> > > > > > they
> >>> > > > > > > > have
> >>> > > > > > > > > >> >> > verified independently.  Kafka brokers will
> have
> >>> ACLs
> >>> > > to
> >>> > > > > > > control
> >>> > > > > > > > > >> which
> >>> > > > > > > > > >> >> > users are allowed to impersonate other users
> and
> >>> get
> >>> > > > tokens
> >>> > > > > > on
> >>> > > > > > > > > >> behalf of
> >>> > > > > > > > > >> >> > them. Overall Impersonation is a whole
> different
> >>> > > problem
> >>> > > > in
> >>> > > > > > my
> >>> > > > > > > > > >> opinion and
> >>> > > > > > > > > >> >> > I think we can tackle it in separate KIP.
> >>> > > > > > > > > >> >> >
> >>> > > > > > > > > >> >> > 111. What's the typical rate of getting and
> >>> renewing
> >>> > > > > > delegation
> >>> > > > > > > > > >> tokens?
> >>> > > > > > > > > >> >> > Typically this should be very very low, 1
> request
> >>> per
> >>> > > > > minute
> >>> > > > > > > is a
> >>> > > > > > > > > >> >> > relatively high estimate. However it depends on
> >>> the
> >>> > > token
> >>> > > > > > > > > >> expiration. I am
> >>> > > > > > > > > >> >> > less worried about the extra load it puts on
> >>> > controller
> >>> > > > vs
> >>> > > > > > the
> >>> > > > > > > > > added
> >>> > > > > > > > > >> >> > complexity and the value it offers.
> >>> > > > > > > > > >> >> >
> >>> > > > > > > > > >> >> > Thanks
> >>> > > > > > > > > >> >> > Parth
> >>> > > > > > > > > >> >> >
> >>> > > > > > > > > >> >> >
> >>> > > > > > > > > >> >> >
> >>> > > > > > > > > >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <
> >>> > > > > > > ismael@juma.me.uk>
> >>> > > > > > > > > >> wrote:
> >>> > > > > > > > > >> >> >
> >>> > > > > > > > > >> >> > > Thanks Rajini. It would probably require a
> >>> separate
> >>> > > KIP
> >>> > > > > as
> >>> > > > > > it
> >>> > > > > > > > > will
> >>> > > > > > > > > >> >> > > introduce user visible changes. We could also
> >>> > update
> >>> > > > > KIP-48
> >>> > > > > > > to
> >>> > > > > > > > > >> have this
> >>> > > > > > > > > >> >> > > information, but it seems cleaner to do it
> >>> > > separately.
> >>> > > > We
> >>> > > > > > can
> >>> > > > > > > > > >> discuss
> >>> > > > > > > > > >> >> > that
> >>> > > > > > > > > >> >> > > in the KIP call today.
> >>> > > > > > > > > >> >> > >
> >>> > > > > > > > > >> >> > > Ismael
> >>> > > > > > > > > >> >> > >
> >>> > > > > > > > > >> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini
> Sivaram
> >>> <
> >>> > > > > > > > > >> >> > > rajinisivaram@googlemail.com> wrote:
> >>> > > > > > > > > >> >> > >
> >>> > > > > > > > > >> >> > > > Ismael,
> >>> > > > > > > > > >> >> > > >
> >>> > > > > > > > > >> >> > > > I have created a JIRA (
> >>> > > > > > > > > >> >> > https://issues.apache.org/
> jira/browse/KAFKA-3751)
> >>> > > > > > > > > >> >> > > > for adding SCRAM as a SASL mechanism. Would
> >>> that
> >>> > > need
> >>> > > > > > > another
> >>> > > > > > > > > >> KIP? If
> >>> > > > > > > > > >> >> > > > KIP-48 will use this mechanism, can this
> just
> >>> be
> >>> > a
> >>> > > > JIRA
> >>> > > > > > > that
> >>> > > > > > > > > gets
> >>> > > > > > > > > >> >> > > reviewed
> >>> > > > > > > > > >> >> > > > when the PR is ready?
> >>> > > > > > > > > >> >> > > >
> >>> > > > > > > > > >> >> > > > Thank you,
> >>> > > > > > > > > >> >> > > >
> >>> > > > > > > > > >> >> > > > Rajini
> >>> > > > > > > > > >> >> > > >
> >>> > > > > > > > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael
> Juma <
> >>> > > > > > > > > ismael@juma.me.uk>
> >>> > > > > > > > > >> >> > wrote:
> >>> > > > > > > > > >> >> > > >
> >>> > > > > > > > > >> >> > > > > Thanks Rajini, SCRAM seems like a good
> >>> > candidate.
> >>> > > > > > > > > >> >> > > > >
> >>> > > > > > > > > >> >> > > > > Gwen had independently mentioned this as
> a
> >>> SASL
> >>> > > > > > mechanism
> >>> > > > > > > > > that
> >>> > > > > > > > > >> might
> >>> > > > > > > > > >> >> > be
> >>> > > > > > > > > >> >> > > > > useful for Kafka and I have been meaning
> to
> >>> > check
> >>> > > > it
> >>> > > > > in
> >>> > > > > > > > more
> >>> > > > > > > > > >> detail.
> >>> > > > > > > > > >> >> > > Good
> >>> > > > > > > > > >> >> > > > > to know that you are willing to
> contribute
> >>> an
> >>> > > > > > > > implementation.
> >>> > > > > > > > > >> Maybe
> >>> > > > > > > > > >> >> > we
> >>> > > > > > > > > >> >> > > > > should file a separate JIRA for this?
> >>> > > > > > > > > >> >> > > > >
> >>> > > > > > > > > >> >> > > > > Ismael
> >>> > > > > > > > > >> >> > > > >
> >>> > > > > > > > > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini
> >>> > Sivaram <
> >>> > > > > > > > > >> >> > > > > rajinisivaram@googlemail.com> wrote:
> >>> > > > > > > > > >> >> > > > >
> >>> > > > > > > > > >> >> > > > > > SCRAM (Salted Challenge Response
> >>> > Authentication
> >>> > > > > > > > Mechanism)
> >>> > > > > > > > > >> is a
> >>> > > > > > > > > >> >> > > better
> >>> > > > > > > > > >> >> > > > > > mechanism than Digest-MD5. Java doesn't
> >>> come
> >>> > > > with a
> >>> > > > > > > > > built-in
> >>> > > > > > > > > >> SCRAM
> >>> > > > > > > > > >> >> > > > > > SaslServer or SaslClient, but I will be
> >>> happy
> >>> > > to
> >>> > > > > add
> >>> > > > > > > > > support
> >>> > > > > > > > > >> in
> >>> > > > > > > > > >> >> > Kafka
> >>> > > > > > > > > >> >> > > > > since
> >>> > > > > > > > > >> >> > > > > > it would be a useful mechanism to
> support
> >>> > > anyway.
> >>> > > > > > > > > >> >> > > > > > https://tools.ietf.org/html/rfc7677
> >>> > describes
> >>> > > > the
> >>> > > > > > > > protocol
> >>> > > > > > > > > >> for
> >>> > > > > > > > > >> >> > > > > > SCRAM-SHA-256.
> >>> > > > > > > > > >> >> > > > > >
> >>> > > > > > > > > >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun
> Rao <
> >>> > > > > > > > jun@confluent.io
> >>> > > > > > > > > >
> >>> > > > > > > > > >> wrote:
> >>> > > > > > > > > >> >> > > > > >
> >>> > > > > > > > > >> >> > > > > > > Parth,
> >>> > > > > > > > > >> >> > > > > > >
> >>> > > > > > > > > >> >> > > > > > > Thanks for the explanation. A couple
> of
> >>> > more
> >>> > > > > > > questions.
> >>> > > > > > > > > >> >> > > > > > >
> >>> > > > > > > > > >> >> > > > > > > 110. What does getDelegationTokenAs
> >>> mean?
> >>> > > > > > > > > >> >> > > > > > >
> >>> > > > > > > > > >> >> > > > > > > 111. What's the typical rate of
> getting
> >>> and
> >>> > > > > > renewing
> >>> > > > > > > > > >> delegation
> >>> > > > > > > > > >> >> > > > tokens?
> >>> > > > > > > > > >> >> > > > > > > That may have an impact on whether
> they
> >>> > > should
> >>> > > > be
> >>> > > > > > > > > directed
> >>> > > > > > > > > >> to the
> >>> > > > > > > > > >> >> > > > > > > controller.
> >>> > > > > > > > > >> >> > > > > > >
> >>> > > > > > > > > >> >> > > > > > > Jun
> >>> > > > > > > > > >> >> > > > > > >
> >>> > > > > > > > > >> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM,
> parth
> >>> > > > > brahmbhatt <
> >>> > > > > > > > > >> >> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> >>> > > > > > > > > >> >> > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > Hi Jun,
> >>> > > > > > > > > >> >> > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > Thanks for reviewing.
> >>> > > > > > > > > >> >> > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > * We could add a Cluster action to
> add
> >>> > acls
> >>> > > > on
> >>> > > > > > who
> >>> > > > > > > > can
> >>> > > > > > > > > >> request
> >>> > > > > > > > > >> >> > > > > > delegation
> >>> > > > > > > > > >> >> > > > > > > > tokens. I don't see the use case
> for
> >>> that
> >>> > > yet
> >>> > > > > but
> >>> > > > > > > > down
> >>> > > > > > > > > >> the line
> >>> > > > > > > > > >> >> > > > when
> >>> > > > > > > > > >> >> > > > > we
> >>> > > > > > > > > >> >> > > > > > > > start supporting
> getDelegationTokenAs
> >>> it
> >>> > > will
> >>> > > > > be
> >>> > > > > > > > > >> necessary.
> >>> > > > > > > > > >> >> > > > > > > > * Yes we recommend tokens to be
> only
> >>> > > > > > > used/distributed
> >>> > > > > > > > > >> over
> >>> > > > > > > > > >> >> > secure
> >>> > > > > > > > > >> >> > > > > > > channels.
> >>> > > > > > > > > >> >> > > > > > > > * Depending on what design we end
> up
> >>> > > choosing
> >>> > > > > > > > > >> Invalidation will
> >>> > > > > > > > > >> >> > > be
> >>> > > > > > > > > >> >> > > > > > > > responsibility of every broker or
> >>> > > controller.
> >>> > > > > > > > > >> >> > > > > > > > * I am not sure if I documented
> >>> somewhere
> >>> > > > that
> >>> > > > > > > > > >> invalidation
> >>> > > > > > > > > >> >> > will
> >>> > > > > > > > > >> >> > > > > > directly
> >>> > > > > > > > > >> >> > > > > > > > go through zookeeper but that is
> not
> >>> the
> >>> > > > > intent.
> >>> > > > > > > > > >> Invalidation
> >>> > > > > > > > > >> >> > > will
> >>> > > > > > > > > >> >> > > > > > either
> >>> > > > > > > > > >> >> > > > > > > > be request based or due to
> >>> expiration. No
> >>> > > > > direct
> >>> > > > > > > > > >> zookeeper
> >>> > > > > > > > > >> >> > > > > interaction
> >>> > > > > > > > > >> >> > > > > > > from
> >>> > > > > > > > > >> >> > > > > > > > any client.
> >>> > > > > > > > > >> >> > > > > > > > * "Broker also stores the
> >>> DelegationToken
> >>> > > > > without
> >>> > > > > > > the
> >>> > > > > > > > > >> hmac in
> >>> > > > > > > > > >> >> > the
> >>> > > > > > > > > >> >> > > > > > > > zookeeper." : Sorry about the
> >>> confusion.
> >>> > > The
> >>> > > > > sole
> >>> > > > > > > > > >> purpose of
> >>> > > > > > > > > >> >> > > > > zookeeper
> >>> > > > > > > > > >> >> > > > > > in
> >>> > > > > > > > > >> >> > > > > > > > this design is as distribution
> channel
> >>> > for
> >>> > > > > tokens
> >>> > > > > > > > > >> between all
> >>> > > > > > > > > >> >> > > > brokers
> >>> > > > > > > > > >> >> > > > > > > and a
> >>> > > > > > > > > >> >> > > > > > > > layer that ensures only tokens that
> >>> were
> >>> > > > > > generated
> >>> > > > > > > by
> >>> > > > > > > > > >> making a
> >>> > > > > > > > > >> >> > > > > request
> >>> > > > > > > > > >> >> > > > > > > to a
> >>> > > > > > > > > >> >> > > > > > > > broker will be accepted (more on
> this
> >>> in
> >>> > > > second
> >>> > > > > > > > > >> paragraph). The
> >>> > > > > > > > > >> >> > > > token
> >>> > > > > > > > > >> >> > > > > > > > consists of few elements (owner,
> >>> renewer,
> >>> > > > uuid
> >>> > > > > ,
> >>> > > > > > > > > >> expiration,
> >>> > > > > > > > > >> >> > > hmac)
> >>> > > > > > > > > >> >> > > > ,
> >>> > > > > > > > > >> >> > > > > > one
> >>> > > > > > > > > >> >> > > > > > > of
> >>> > > > > > > > > >> >> > > > > > > > which is the finally generated hmac
> >>> but
> >>> > > hmac
> >>> > > > it
> >>> > > > > > > self
> >>> > > > > > > > is
> >>> > > > > > > > > >> >> > derivable
> >>> > > > > > > > > >> >> > > > if
> >>> > > > > > > > > >> >> > > > > > you
> >>> > > > > > > > > >> >> > > > > > > > have all the other elements of the
> >>> token
> >>> > +
> >>> > > > > secret
> >>> > > > > > > key
> >>> > > > > > > > > to
> >>> > > > > > > > > >> >> > generate
> >>> > > > > > > > > >> >> > > > > hmac.
> >>> > > > > > > > > >> >> > > > > > > > Given zookeeper does not provide
> SSL
> >>> > > support
> >>> > > > we
> >>> > > > > > do
> >>> > > > > > > > not
> >>> > > > > > > > > >> want the
> >>> > > > > > > > > >> >> > > > > entire
> >>> > > > > > > > > >> >> > > > > > > > token to be wire transferred to
> >>> zookeeper
> >>> > > as
> >>> > > > > that
> >>> > > > > > > > will
> >>> > > > > > > > > >> be an
> >>> > > > > > > > > >> >> > > > insecure
> >>> > > > > > > > > >> >> > > > > > > wire
> >>> > > > > > > > > >> >> > > > > > > > transfer. Instead we only store all
> >>> the
> >>> > > other
> >>> > > > > > > > elements
> >>> > > > > > > > > >> of a
> >>> > > > > > > > > >> >> > > > > delegation
> >>> > > > > > > > > >> >> > > > > > > > tokens. Brokers can read these
> >>> elements
> >>> > and
> >>> > > > > > because
> >>> > > > > > > > > they
> >>> > > > > > > > > >> also
> >>> > > > > > > > > >> >> > > have
> >>> > > > > > > > > >> >> > > > > > access
> >>> > > > > > > > > >> >> > > > > > > > to secret key they will be able to
> >>> > generate
> >>> > > > > hmac
> >>> > > > > > on
> >>> > > > > > > > > >> their end.
> >>> > > > > > > > > >> >> > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > One of the alternative proposed is
> to
> >>> > avoid
> >>> > > > > > > zookeeper
> >>> > > > > > > > > >> >> > > altogether. A
> >>> > > > > > > > > >> >> > > > > > > Client
> >>> > > > > > > > > >> >> > > > > > > > will call broker with required
> >>> > information
> >>> > > > > > (owner,
> >>> > > > > > > > > >> renwer,
> >>> > > > > > > > > >> >> > > > > expiration)
> >>> > > > > > > > > >> >> > > > > > > and
> >>> > > > > > > > > >> >> > > > > > > > get back (signed hmac, uuid).
> Broker
> >>> > won't
> >>> > > > > store
> >>> > > > > > > this
> >>> > > > > > > > > in
> >>> > > > > > > > > >> >> > > zookeeper.
> >>> > > > > > > > > >> >> > > > > > From
> >>> > > > > > > > > >> >> > > > > > > > this point a client can contact any
> >>> > broker
> >>> > > > with
> >>> > > > > > all
> >>> > > > > > > > the
> >>> > > > > > > > > >> >> > > delegation
> >>> > > > > > > > > >> >> > > > > > token
> >>> > > > > > > > > >> >> > > > > > > > info (owner, rewner, expiration,
> hmac,
> >>> > > uuid)
> >>> > > > > the
> >>> > > > > > > > borker
> >>> > > > > > > > > >> will
> >>> > > > > > > > > >> >> > > > > regenerate
> >>> > > > > > > > > >> >> > > > > > > the
> >>> > > > > > > > > >> >> > > > > > > > hmac and as long as it matches with
> >>> hmac
> >>> > > > > > presented
> >>> > > > > > > by
> >>> > > > > > > > > >> client ,
> >>> > > > > > > > > >> >> > > > broker
> >>> > > > > > > > > >> >> > > > > > > will
> >>> > > > > > > > > >> >> > > > > > > > allow the request to authenticate.
> >>> Only
> >>> > > > > problem
> >>> > > > > > > with
> >>> > > > > > > > > >> this
> >>> > > > > > > > > >> >> > > approach
> >>> > > > > > > > > >> >> > > > > is
> >>> > > > > > > > > >> >> > > > > > if
> >>> > > > > > > > > >> >> > > > > > > > the secret key is compromised any
> >>> client
> >>> > > can
> >>> > > > > now
> >>> > > > > > > > > generate
> >>> > > > > > > > > >> >> > random
> >>> > > > > > > > > >> >> > > > > tokens
> >>> > > > > > > > > >> >> > > > > > > and
> >>> > > > > > > > > >> >> > > > > > > > they will still be able to
> >>> authenticate
> >>> > as
> >>> > > > any
> >>> > > > > > user
> >>> > > > > > > > > they
> >>> > > > > > > > > >> like.
> >>> > > > > > > > > >> >> > > with
> >>> > > > > > > > > >> >> > > > > > > > zookeeper we guarantee that only
> >>> tokens
> >>> > > > > acquired
> >>> > > > > > > via
> >>> > > > > > > > a
> >>> > > > > > > > > >> broker
> >>> > > > > > > > > >> >> > > > (using
> >>> > > > > > > > > >> >> > > > > > some
> >>> > > > > > > > > >> >> > > > > > > > auth scheme other than delegation
> >>> token)
> >>> > > will
> >>> > > > > be
> >>> > > > > > > > > >> accepted. We
> >>> > > > > > > > > >> >> > > need
> >>> > > > > > > > > >> >> > > > to
> >>> > > > > > > > > >> >> > > > > > > > discuss which proposal makes more
> >>> sense
> >>> > and
> >>> > > > we
> >>> > > > > > can
> >>> > > > > > > go
> >>> > > > > > > > > >> over it
> >>> > > > > > > > > >> >> > in
> >>> > > > > > > > > >> >> > > > > > > tomorrow's
> >>> > > > > > > > > >> >> > > > > > > > meeting.
> >>> > > > > > > > > >> >> > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > Also, can you forward the invite to
> >>> me?
> >>> > > > > > > > > >> >> > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > Thanks
> >>> > > > > > > > > >> >> > > > > > > > Parth
> >>> > > > > > > > > >> >> > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM,
> Jun
> >>> > Rao <
> >>> > > > > > > > > >> jun@confluent.io>
> >>> > > > > > > > > >> >> > > > wrote:
> >>> > > > > > > > > >> >> > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > Thanks for the KIP. A few
> comments.
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > 100. This potentially can be
> useful
> >>> for
> >>> > > > Kafka
> >>> > > > > > > > Connect
> >>> > > > > > > > > >> and
> >>> > > > > > > > > >> >> > Kafka
> >>> > > > > > > > > >> >> > > > > rest
> >>> > > > > > > > > >> >> > > > > > > > proxy
> >>> > > > > > > > > >> >> > > > > > > > > where a worker agent will need to
> >>> run a
> >>> > > > task
> >>> > > > > on
> >>> > > > > > > > > behalf
> >>> > > > > > > > > >> of a
> >>> > > > > > > > > >> >> > > > client.
> >>> > > > > > > > > >> >> > > > > > We
> >>> > > > > > > > > >> >> > > > > > > > will
> >>> > > > > > > > > >> >> > > > > > > > > likely need to change how those
> >>> > services
> >>> > > > use
> >>> > > > > > > Kafka
> >>> > > > > > > > > >> clients
> >>> > > > > > > > > >> >> > > > > > > > > (producer/consumer). Instead of a
> >>> > shared
> >>> > > > > client
> >>> > > > > > > per
> >>> > > > > > > > > >> worker,
> >>> > > > > > > > > >> >> > we
> >>> > > > > > > > > >> >> > > > will
> >>> > > > > > > > > >> >> > > > > > > need
> >>> > > > > > > > > >> >> > > > > > > > a
> >>> > > > > > > > > >> >> > > > > > > > > client per user task since the
> >>> > > > authentication
> >>> > > > > > > > happens
> >>> > > > > > > > > >> at the
> >>> > > > > > > > > >> >> > > > > > connection
> >>> > > > > > > > > >> >> > > > > > > > > level. For Kafka Connect, the
> >>> renewer
> >>> > > will
> >>> > > > be
> >>> > > > > > the
> >>> > > > > > > > > >> workers.
> >>> > > > > > > > > >> >> > So,
> >>> > > > > > > > > >> >> > > we
> >>> > > > > > > > > >> >> > > > > > > > probably
> >>> > > > > > > > > >> >> > > > > > > > > need to allow multiple renewers.
> For
> >>> > > Kafka
> >>> > > > > rest
> >>> > > > > > > > > proxy,
> >>> > > > > > > > > >> the
> >>> > > > > > > > > >> >> > > > renewer
> >>> > > > > > > > > >> >> > > > > > can
> >>> > > > > > > > > >> >> > > > > > > > > probably just be the creator of
> the
> >>> > > token.
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > 101. Do we need new acl on who
> can
> >>> > > request
> >>> > > > > > > > delegation
> >>> > > > > > > > > >> tokens?
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > 102. Do we recommend people to
> send
> >>> > > > > delegation
> >>> > > > > > > > tokens
> >>> > > > > > > > > >> in an
> >>> > > > > > > > > >> >> > > > > encrypted
> >>> > > > > > > > > >> >> > > > > > > > > channel?
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > 103. Who is responsible for
> expiring
> >>> > > > tokens,
> >>> > > > > > > every
> >>> > > > > > > > > >> broker?
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > 104. For invalidating tokens,
> would
> >>> it
> >>> > be
> >>> > > > > > better
> >>> > > > > > > to
> >>> > > > > > > > > do
> >>> > > > > > > > > >> it in
> >>> > > > > > > > > >> >> > a
> >>> > > > > > > > > >> >> > > > > > request
> >>> > > > > > > > > >> >> > > > > > > > > instead of going to ZK directly?
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > 105. The terminology of client in
> >>> the
> >>> > > wiki
> >>> > > > > > > > sometimes
> >>> > > > > > > > > >> refers
> >>> > > > > > > > > >> >> > to
> >>> > > > > > > > > >> >> > > > the
> >>> > > > > > > > > >> >> > > > > > end
> >>> > > > > > > > > >> >> > > > > > > > > client and some other times
> refers
> >>> to
> >>> > the
> >>> > > > > > client
> >>> > > > > > > > > using
> >>> > > > > > > > > >> the
> >>> > > > > > > > > >> >> > > > > delegation
> >>> > > > > > > > > >> >> > > > > > > > > tokens. It would be useful to
> >>> > distinguish
> >>> > > > > > between
> >>> > > > > > > > the
> >>> > > > > > > > > >> two.
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > 106. Could you explain the
> sentence
> >>> > > "Broker
> >>> > > > > > also
> >>> > > > > > > > > >> stores the
> >>> > > > > > > > > >> >> > > > > > > > DelegationToken
> >>> > > > > > > > > >> >> > > > > > > > > without the hmac in the
> zookeeper."
> >>> a
> >>> > bit
> >>> > > > > > more? I
> >>> > > > > > > > > >> thought the
> >>> > > > > > > > > >> >> > > > > > > delegation
> >>> > > > > > > > > >> >> > > > > > > > > token is the hmac.
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > Thanks,
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > Jun
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM,
> Jun
> >>> > Rao
> >>> > > <
> >>> > > > > > > > > >> jun@confluent.io>
> >>> > > > > > > > > >> >> > > > wrote:
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > > Hi, Harsha,
> >>> > > > > > > > > >> >> > > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > > Just sent out a KIP meeting
> >>> invite.
> >>> > We
> >>> > > > can
> >>> > > > > > > > discuss
> >>> > > > > > > > > >> this in
> >>> > > > > > > > > >> >> > > the
> >>> > > > > > > > > >> >> > > > > > > meeting
> >>> > > > > > > > > >> >> > > > > > > > > > tomorrow.
> >>> > > > > > > > > >> >> > > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > > Thanks,
> >>> > > > > > > > > >> >> > > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > > Jun
> >>> > > > > > > > > >> >> > > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47
> AM,
> >>> > > Harsha <
> >>> > > > > > > > > >> kafka@harsha.io>
> >>> > > > > > > > > >> >> > > > wrote:
> >>> > > > > > > > > >> >> > > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> Hi All,
> >>> > > > > > > > > >> >> > > > > > > > > >>            Can we have a KIP
> >>> meeting
> >>> > > > > around
> >>> > > > > > > > this.
> >>> > > > > > > > > >> The KIP
> >>> > > > > > > > > >> >> > is
> >>> > > > > > > > > >> >> > > > up
> >>> > > > > > > > > >> >> > > > > > for
> >>> > > > > > > > > >> >> > > > > > > > > >>            sometime and if
> there
> >>> are
> >>> > > any
> >>> > > > > > > > questions
> >>> > > > > > > > > >> lets
> >>> > > > > > > > > >> >> > > > quickly
> >>> > > > > > > > > >> >> > > > > > hash
> >>> > > > > > > > > >> >> > > > > > > > out
> >>> > > > > > > > > >> >> > > > > > > > > >>            details.
> >>> > > > > > > > > >> >> > > > > > > > > >>
> >>> > > > > > > > > >> >> > > > > > > > > >> Thanks,
> >>> > > > > > > > > >> >> > > > > > > > > >> Harsha
> >>> > > > > > > > > >> >> > > > > > > > > >>
> >>> > > > > > > > > >> >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40
> >>> AM,
> >>> > > parth
> >>> > > > > > > > > brahmbhatt
> >>> > > > > > > > > >> wrote:
> >>> > > > > > > > > >> >> > > > > > > > > >> > That is what the hadoop echo
> >>> > system
> >>> > > > uses
> >>> > > > > > so
> >>> > > > > > > no
> >>> > > > > > > > > >> good
> >>> > > > > > > > > >> >> > reason
> >>> > > > > > > > > >> >> > > > > > really.
> >>> > > > > > > > > >> >> > > > > > > > We
> >>> > > > > > > > > >> >> > > > > > > > > >> > could
> >>> > > > > > > > > >> >> > > > > > > > > >> > change it to whatever is the
> >>> > newest
> >>> > > > > > > > recommended
> >>> > > > > > > > > >> standard
> >>> > > > > > > > > >> >> > > is.
> >>> > > > > > > > > >> >> > > > > > > > > >> >
> >>> > > > > > > > > >> >> > > > > > > > > >> > Thanks
> >>> > > > > > > > > >> >> > > > > > > > > >> > Parth
> >>> > > > > > > > > >> >> > > > > > > > > >> >
> >>> > > > > > > > > >> >> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33
> >>> AM,
> >>> > > > Ismael
> >>> > > > > > > Juma <
> >>> > > > > > > > > >> >> > > > > ismael@juma.me.uk
> >>> > > > > > > > > >> >> > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > wrote:
> >>> > > > > > > > > >> >> > > > > > > > > >> >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > Hi Parth,
> >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > Thanks for the KIP. I only
> >>> > started
> >>> > > > > > > reviewing
> >>> > > > > > > > > >> this and
> >>> > > > > > > > > >> >> > > may
> >>> > > > > > > > > >> >> > > > > have
> >>> > > > > > > > > >> >> > > > > > > > > >> additional
> >>> > > > > > > > > >> >> > > > > > > > > >> > > questions later. The
> >>> immediate
> >>> > > > > question
> >>> > > > > > > that
> >>> > > > > > > > > >> came to
> >>> > > > > > > > > >> >> > > mind
> >>> > > > > > > > > >> >> > > > is
> >>> > > > > > > > > >> >> > > > > > our
> >>> > > > > > > > > >> >> > > > > > > > > >> choice of
> >>> > > > > > > > > >> >> > > > > > > > > >> > > "DIGEST-MD5" even though
> it's
> >>> > > marked
> >>> > > > > as
> >>> > > > > > > > > >> OBSOLETE in
> >>> > > > > > > > > >> >> > the
> >>> > > > > > > > > >> >> > > > IANA
> >>> > > > > > > > > >> >> > > > > > > > > Registry
> >>> > > > > > > > > >> >> > > > > > > > > >> of
> >>> > > > > > > > > >> >> > > > > > > > > >> > > SASL mechanisms and the
> >>> original
> >>> > > RFC
> >>> > > > > > > (2831)
> >>> > > > > > > > > has
> >>> > > > > > > > > >> been
> >>> > > > > > > > > >> >> > > moved
> >>> > > > > > > > > >> >> > > > > to
> >>> > > > > > > > > >> >> > > > > > > > > Historic
> >>> > > > > > > > > >> >> > > > > > > > > >> > > status:
> >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > >
> https://tools.ietf.org/html/
> >>> > > rfc6331
> >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > >
> >>> > > > > > > > > >> >> > >
> >>> > > > > > > > > >>
> >>> > > > > > >
> >>> > > > http://www.iana.org/assignments/sasl-mechanisms/sasl-
> >>> mechanisms.xhtml
> >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > What is the reasoning
> behind
> >>> > that
> >>> > > > > > choice?
> >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > Thanks,
> >>> > > > > > > > > >> >> > > > > > > > > >> > > Ismael
> >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > On Fri, May 13, 2016 at
> 11:29
> >>> > PM,
> >>> > > > Gwen
> >>> > > > > > > > > Shapira <
> >>> > > > > > > > > >> >> > > > > > > gwen@confluent.io
> >>> > > > > > > > > >> >> > > > > > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> wrote:
> >>> > > > > > > > > >> >> > > > > > > > > >> > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > Also comments inline :)
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > * I want to emphasize
> >>> that
> >>> > > even
> >>> > > > > > though
> >>> > > > > > > > > >> delegation
> >>> > > > > > > > > >> >> > > > tokens
> >>> > > > > > > > > >> >> > > > > > > are a
> >>> > > > > > > > > >> >> > > > > > > > > >> Hadoop
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > innovation, I feel
> very
> >>> > > strongly
> >>> > > > > > about
> >>> > > > > > > > not
> >>> > > > > > > > > >> adding
> >>> > > > > > > > > >> >> > > > > > dependency
> >>> > > > > > > > > >> >> > > > > > > > on
> >>> > > > > > > > > >> >> > > > > > > > > >> Hadoop
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > when implementing
> >>> delegation
> >>> > > > > tokens
> >>> > > > > > > for
> >>> > > > > > > > > >> Kafka. The
> >>> > > > > > > > > >> >> > > KIP
> >>> > > > > > > > > >> >> > > > > > > doesn't
> >>> > > > > > > > > >> >> > > > > > > > > >> imply
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > such dependency, but
> if
> >>> you
> >>> > > can
> >>> > > > > > > > clarify...
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *No hadoop
> dependency.*
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > Yay! Just add this to
> the
> >>> KIP
> >>> > so
> >>> > > > no
> >>> > > > > > one
> >>> > > > > > > > will
> >>> > > > > > > > > >> read
> >>> > > > > > > > > >> >> > the
> >>> > > > > > > > > >> >> > > > KIP
> >>> > > > > > > > > >> >> > > > > > and
> >>> > > > > > > > > >> >> > > > > > > > > panic
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > three weeks before the
> next
> >>> > > > > release...
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > * Can we get
> delegation
> >>> > token
> >>> > > at
> >>> > > > > any
> >>> > > > > > > > time
> >>> > > > > > > > > >> after
> >>> > > > > > > > > >> >> > > > > > > > authenticating?
> >>> > > > > > > > > >> >> > > > > > > > > >> only
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > immediately after?
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *As long as you are
> >>> > > > authenticated
> >>> > > > > > you
> >>> > > > > > > > can
> >>> > > > > > > > > >> get
> >>> > > > > > > > > >> >> > > > delegation
> >>> > > > > > > > > >> >> > > > > > > > tokens.
> >>> > > > > > > > > >> >> > > > > > > > > >> We
> >>> > > > > > > > > >> >> > > > > > > > > >> > > need
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > to
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > discuss if a client
> >>> > > > authenticated
> >>> > > > > > > using
> >>> > > > > > > > > >> delegation
> >>> > > > > > > > > >> >> > > > > token,
> >>> > > > > > > > > >> >> > > > > > > can
> >>> > > > > > > > > >> >> > > > > > > > > also
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > acquire
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > delegation token
> again or
> >>> > not.
> >>> > > > > Also
> >>> > > > > > > > there
> >>> > > > > > > > > >> is the
> >>> > > > > > > > > >> >> > > > > question
> >>> > > > > > > > > >> >> > > > > > of
> >>> > > > > > > > > >> >> > > > > > > > do
> >>> > > > > > > > > >> >> > > > > > > > > we
> >>> > > > > > > > > >> >> > > > > > > > > >> > > allow
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > anyone to acquire
> >>> delegation
> >>> > > > token
> >>> > > > > > or
> >>> > > > > > > we
> >>> > > > > > > > > >> want
> >>> > > > > > > > > >> >> > > specific
> >>> > > > > > > > > >> >> > > > > > ACLs
> >>> > > > > > > > > >> >> > > > > > > (I
> >>> > > > > > > > > >> >> > > > > > > > > >> think
> >>> > > > > > > > > >> >> > > > > > > > > >> > > its
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > an
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > overkill.)*
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > I agree that ACLs is an
> >>> > > overkill.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > I think we are debating
> two
> >>> > > > options:
> >>> > > > > > > > Either
> >>> > > > > > > > > >> require
> >>> > > > > > > > > >> >> > > > > Kerberos
> >>> > > > > > > > > >> >> > > > > > > > auth
> >>> > > > > > > > > >> >> > > > > > > > > >> for
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > renewal or require
> >>> non-owners
> >>> > to
> >>> > > > > > renew.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > I *think* the latter is
> >>> > simpler
> >>> > > > (it
> >>> > > > > > > > > basically
> >>> > > > > > > > > >> >> > require
> >>> > > > > > > > > >> >> > > a
> >>> > > > > > > > > >> >> > > > > "job
> >>> > > > > > > > > >> >> > > > > > > > > master"
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > to take responsibility
> for
> >>> the
> >>> > > > > > renewal,
> >>> > > > > > > it
> >>> > > > > > > > > >> will have
> >>> > > > > > > > > >> >> > > its
> >>> > > > > > > > > >> >> > > > > own
> >>> > > > > > > > > >> >> > > > > > > > > >> identity
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > anyway and I think this
> is
> >>> the
> >>> > > > > correct
> >>> > > > > > > > > design
> >>> > > > > > > > > >> >> > pattern
> >>> > > > > > > > > >> >> > > > > > anyway.
> >>> > > > > > > > > >> >> > > > > > > > For
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > storm, I'd expect
> Nimbus to
> >>> > > > > coordinate
> >>> > > > > > > > > >> renewals?),
> >>> > > > > > > > > >> >> > but
> >>> > > > > > > > > >> >> > > > it
> >>> > > > > > > > > >> >> > > > > is
> >>> > > > > > > > > >> >> > > > > > > > hard
> >>> > > > > > > > > >> >> > > > > > > > > to
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > debate simplicity
> without
> >>> > > looking
> >>> > > > at
> >>> > > > > > the
> >>> > > > > > > > > code
> >>> > > > > > > > > >> >> > changes
> >>> > > > > > > > > >> >> > > > > > > required.
> >>> > > > > > > > > >> >> > > > > > > > If
> >>> > > > > > > > > >> >> > > > > > > > > >> you
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > have a draft of how the
> >>> > "require
> >>> > > > > > > Kerberos"
> >>> > > > > > > > > >> will look
> >>> > > > > > > > > >> >> > > in
> >>> > > > > > > > > >> >> > > > > > Kafka
> >>> > > > > > > > > >> >> > > > > > > > > code,
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > I'll be happy to take a
> >>> look.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > * My understanding is
> >>> that
> >>> > > > tokens
> >>> > > > > > will
> >>> > > > > > > > > >> propagate
> >>> > > > > > > > > >> >> > via
> >>> > > > > > > > > >> >> > > > ZK
> >>> > > > > > > > > >> >> > > > > > but
> >>> > > > > > > > > >> >> > > > > > > > > >> without
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > additional changes to
> >>> > > > > UpdateMetadata
> >>> > > > > > > > > >> protocol,
> >>> > > > > > > > > >> >> > > > correct?
> >>> > > > > > > > > >> >> > > > > > > > Clients
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > currently don't retry
> on
> >>> > SASL
> >>> > > > auth
> >>> > > > > > > > failure
> >>> > > > > > > > > >> (IIRC),
> >>> > > > > > > > > >> >> > > but
> >>> > > > > > > > > >> >> > > > > > since
> >>> > > > > > > > > >> >> > > > > > > > the
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > tokens propagate
> between
> >>> > > brokers
> >>> > > > > > > asynch,
> >>> > > > > > > > > we
> >>> > > > > > > > > >> will
> >>> > > > > > > > > >> >> > > need
> >>> > > > > > > > > >> >> > > > to
> >>> > > > > > > > > >> >> > > > > > > > retry a
> >>> > > > > > > > > >> >> > > > > > > > > >> bit
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > to avoid clients
> failing
> >>> > auth
> >>> > > > due
> >>> > > > > to
> >>> > > > > > > > > timing
> >>> > > > > > > > > >> >> > issues.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *I am considering 2
> >>> > > alternatives
> >>> > > > > > right
> >>> > > > > > > > > now.
> >>> > > > > > > > > >> The
> >>> > > > > > > > > >> >> > > > current
> >>> > > > > > > > > >> >> > > > > > > > > documented
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > approach
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > is zookeeper based
> and it
> >>> > does
> >>> > > > not
> >>> > > > > > > > require
> >>> > > > > > > > > >> any
> >>> > > > > > > > > >> >> > > changes
> >>> > > > > > > > > >> >> > > > > to
> >>> > > > > > > > > >> >> > > > > > > > > >> > > UpdateMetadata
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > protocol. An
> alternative
> >>> > > > approach
> >>> > > > > > can
> >>> > > > > > > > > remove
> >>> > > > > > > > > >> >> > > zookeeper
> >>> > > > > > > > > >> >> > > > > > > > > dependency
> >>> > > > > > > > > >> >> > > > > > > > > >> as
> >>> > > > > > > > > >> >> > > > > > > > > >> > > well
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > but we can discuss
> that
> >>> in
> >>> > KIP
> >>> > > > > > > > discussion
> >>> > > > > > > > > >> call.*
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > Oooh! Sounds
> interesting.
> >>> Do
> >>> > you
> >>> > > > > want
> >>> > > > > > to
> >>> > > > > > > > > ping
> >>> > > > > > > > > >> Jun to
> >>> > > > > > > > > >> >> > > > > > arrange a
> >>> > > > > > > > > >> >> > > > > > > > > call?
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > * I liked Ashish's
> >>> > suggestion
> >>> > > of
> >>> > > > > > > having
> >>> > > > > > > > > >> just the
> >>> > > > > > > > > >> >> > > > > > controller
> >>> > > > > > > > > >> >> > > > > > > > > issue
> >>> > > > > > > > > >> >> > > > > > > > > >> the
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > delegation tokens, to
> >>> avoid
> >>> > > > > syncing
> >>> > > > > > a
> >>> > > > > > > > > shared
> >>> > > > > > > > > >> >> > secret.
> >>> > > > > > > > > >> >> > > > Not
> >>> > > > > > > > > >> >> > > > > > > sure
> >>> > > > > > > > > >> >> > > > > > > > if
> >>> > > > > > > > > >> >> > > > > > > > > >> we
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > want to continue the
> >>> > > discussion
> >>> > > > > here
> >>> > > > > > > or
> >>> > > > > > > > on
> >>> > > > > > > > > >> the
> >>> > > > > > > > > >> >> > > wiki. I
> >>> > > > > > > > > >> >> > > > > > think
> >>> > > > > > > > > >> >> > > > > > > > > that
> >>> > > > > > > > > >> >> > > > > > > > > >> we
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > can decouple the
> problem
> >>> of
> >>> > > > "token
> >>> > > > > > > > > >> distribution"
> >>> > > > > > > > > >> >> > > from
> >>> > > > > > > > > >> >> > > > > > > "shared
> >>> > > > > > > > > >> >> > > > > > > > > >> secret
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > distribution" and use
> the
> >>> > > > > controller
> >>> > > > > > > as
> >>> > > > > > > > > the
> >>> > > > > > > > > >> only
> >>> > > > > > > > > >> >> > > token
> >>> > > > > > > > > >> >> > > > > > > > generator
> >>> > > > > > > > > >> >> > > > > > > > > >> to
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > solve the second
> issue,
> >>> > while
> >>> > > > > still
> >>> > > > > > > > using
> >>> > > > > > > > > >> ZK async
> >>> > > > > > > > > >> >> > > to
> >>> > > > > > > > > >> >> > > > > > > > distribute
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > tokens.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *As mentioned in the
> >>> > previous
> >>> > > > > Email
> >>> > > > > > I
> >>> > > > > > > am
> >>> > > > > > > > > >> fine with
> >>> > > > > > > > > >> >> > > > that
> >>> > > > > > > > > >> >> > > > > > > > approach
> >>> > > > > > > > > >> >> > > > > > > > > >> as
> >>> > > > > > > > > >> >> > > > > > > > > >> > > long
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > as
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > we agree that the
> extra
> >>> > > > complexity
> >>> > > > > > of
> >>> > > > > > > > > >> >> > > adding/updating
> >>> > > > > > > > > >> >> > > > > APIS
> >>> > > > > > > > > >> >> > > > > > > > adds
> >>> > > > > > > > > >> >> > > > > > > > > >> enough
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > value. The advantage
> with
> >>> > the
> >>> > > > > > > controller
> >>> > > > > > > > > >> approach
> >>> > > > > > > > > >> >> > is
> >>> > > > > > > > > >> >> > > > > > secret
> >>> > > > > > > > > >> >> > > > > > > > > >> rotation
> >>> > > > > > > > > >> >> > > > > > > > > >> > > can
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > be
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > automated,frequent and
> >>> would
> >>> > > not
> >>> > > > > > > require
> >>> > > > > > > > > >> >> > > deployment. *
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > Can you detail the extra
> >>> > > > complexity
> >>> > > > > > (or
> >>> > > > > > > > > point
> >>> > > > > > > > > >> me to
> >>> > > > > > > > > >> >> > > the
> >>> > > > > > > > > >> >> > > > > > email
> >>> > > > > > > > > >> >> > > > > > > I
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > missed?) - which Apis
> are
> >>> > > > required?
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > As far as I can tell,
> >>> clients
> >>> > > can
> >>> > > > > > > already
> >>> > > > > > > > > >> find the
> >>> > > > > > > > > >> >> > > > > > controller
> >>> > > > > > > > > >> >> > > > > > > > from
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > metadata. I'm a bit more
> >>> > > concerned
> >>> > > > > > about
> >>> > > > > > > > > >> controller
> >>> > > > > > > > > >> >> > > > load.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > * While I like the
> idea
> >>> of
> >>> > > > forcing
> >>> > > > > > > > > kerberos
> >>> > > > > > > > > >> auth
> >>> > > > > > > > > >> >> > for
> >>> > > > > > > > > >> >> > > > > > > renwal, I
> >>> > > > > > > > > >> >> > > > > > > > > >> think
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > it mixes the transport
> >>> layer
> >>> > > the
> >>> > > > > the
> >>> > > > > > > > > request
> >>> > > > > > > > > >> >> > content
> >>> > > > > > > > > >> >> > > > in
> >>> > > > > > > > > >> >> > > > > a
> >>> > > > > > > > > >> >> > > > > > > > pretty
> >>> > > > > > > > > >> >> > > > > > > > > >> ugly
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > way. Perhaps limiting
> >>> > renewer
> >>> > > to
> >>> > > > > > > > non-owner
> >>> > > > > > > > > >> is
> >>> > > > > > > > > >> >> > > better.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *I feel this is a
> >>> necessary
> >>> > > > evil.
> >>> > > > > > > While
> >>> > > > > > > > > >> this will
> >>> > > > > > > > > >> >> > > make
> >>> > > > > > > > > >> >> > > > > the
> >>> > > > > > > > > >> >> > > > > > > > kafka
> >>> > > > > > > > > >> >> > > > > > > > > >> code
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > pretty straight
> forward ,
> >>> > > > forcing
> >>> > > > > > > > renewer
> >>> > > > > > > > > >> to
> >>> > > > > > > > > >> >> > > > non-owner
> >>> > > > > > > > > >> >> > > > > > > pushes
> >>> > > > > > > > > >> >> > > > > > > > > >> the code
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > ugliness to client and
> >>> makes
> >>> > > it
> >>> > > > > even
> >>> > > > > > > > > harder
> >>> > > > > > > > > >> to
> >>> > > > > > > > > >> >> > > > > > integrate.  *
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > As mentioned before, I
> >>> don't
> >>> > > think
> >>> > > > > the
> >>> > > > > > > > > >> "renewal by
> >>> > > > > > > > > >> >> > > > other"
> >>> > > > > > > > > >> >> > > > > > > > approach
> >>> > > > > > > > > >> >> > > > > > > > > >> is
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > that ugly for the
> clients
> >>> we
> >>> > > > expect
> >>> > > > > to
> >>> > > > > > > use
> >>> > > > > > > > > >> >> > delegation
> >>> > > > > > > > > >> >> > > > > tokens
> >>> > > > > > > > > >> >> > > > > > > > since
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > they will have an
> >>> app-master
> >>> > of
> >>> > > > some
> >>> > > > > > > sort
> >>> > > > > > > > > who
> >>> > > > > > > > > >> >> > > requested
> >>> > > > > > > > > >> >> > > > > the
> >>> > > > > > > > > >> >> > > > > > > > token
> >>> > > > > > > > > >> >> > > > > > > > > to
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > begin with.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > The response for my
> >>> question
> >>> > > on
> >>> > > > > how
> >>> > > > > > > > > multiple
> >>> > > > > > > > > >> >> > > > identities
> >>> > > > > > > > > >> >> > > > > > will
> >>> > > > > > > > > >> >> > > > > > > > be
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > handled wasn't super
> >>> clear
> >>> > to
> >>> > > > me -
> >>> > > > > > > > AFAIK,
> >>> > > > > > > > > >> we don't
> >>> > > > > > > > > >> >> > > > > > > > authenticate
> >>> > > > > > > > > >> >> > > > > > > > > >> each
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > request, we
> authenticate
> >>> > > > > > connections.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > *We authenticate
> >>> > connections,
> >>> > > > and
> >>> > > > > > only
> >>> > > > > > > > > when
> >>> > > > > > > > > >> they
> >>> > > > > > > > > >> >> > are
> >>> > > > > > > > > >> >> > > > > being
> >>> > > > > > > > > >> >> > > > > > > > > >> established.
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > Let
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > me try to phrase this
> as
> >>> a
> >>> > > > > question,
> >>> > > > > > > in
> >>> > > > > > > > > >> absence of
> >>> > > > > > > > > >> >> > > > > > > delegation
> >>> > > > > > > > > >> >> > > > > > > > > >> tokens if
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > we
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > had to support the use
> >>> case
> >>> > > > using
> >>> > > > > > user
> >>> > > > > > > > > >> TGT's how
> >>> > > > > > > > > >> >> > > would
> >>> > > > > > > > > >> >> > > > > we
> >>> > > > > > > > > >> >> > > > > > do
> >>> > > > > > > > > >> >> > > > > > > > it?
> >>> > > > > > > > > >> >> > > > > > > > > >> My
> >>> > > > > > > > > >> >> > > > > > > > > >> > > point
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > was it would be no
> >>> different
> >>> > > > with
> >>> > > > > > > > > delegation
> >>> > > > > > > > > >> >> > tokens.
> >>> > > > > > > > > >> >> > > > The
> >>> > > > > > > > > >> >> > > > > > use
> >>> > > > > > > > > >> >> > > > > > > > > case
> >>> > > > > > > > > >> >> > > > > > > > > >> you
> >>> > > > > > > > > >> >> > > > > > > > > >> > > are
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > > describing seems more
> >>> like
> >>> > > > > > > > impersonation.*
> >>> > > > > > > > > >> >> > > > > > > > > >> > > >
> >>> > > > > > > > > >> >> > > > > > > > > >> > > > Yeah, I thought that
> one of
> >>> > the
> >>> > > > > things
> >>> > > > > > > > that
> >>> > > > > > > > > >> >> > del
> >>>
> >>
> >>
> >>
> >> --
> >>
> >> Regards,
> >> Ashish
> >>
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Gwen Shapira <gw...@confluent.io>]

Hi Guys,

This discussion was dead for a while. Are there still contentious
points? If not, why are there no votes?

On Tue, Aug 23, 2016 at 1:26 PM, Jun Rao <ju...@confluent.io> wrote:
> Ashish,
>
> Yes, I will send out a KIP invite for next week to discuss KIP-48 and other
> remaining KIPs.
>
> Thanks,
>
> Jun
>
> On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh <as...@cloudera.com> wrote:
>
>> Thanks Harsha!
>>
>> Jun, can we add KIP-48 to next KIP hangout's agenda. Also, we did not
>> actually make a call on when we should have next KIP call. As there are a
>> few outstanding KIPs that could not be discussed this week, can we have a
>> KIP hangout call next week?
>>
>> On Tue, Aug 23, 2016 at 1:10 PM, Harsha Chintalapani <ka...@harsha.io>
>> wrote:
>>
>>> Ashish,
>>>         Yes we are working on it. Lets discuss in the next KIP meeting.
>>> I'll join.
>>> -Harsha
>>>
>>> On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh <as...@cloudera.com>
>>> wrote:
>>>
>>> > Hello Harsha,
>>> >
>>> > Are you still working on this? Wondering if we can discuss this in next
>>> KIP
>>> > meeting, if you can join.
>>> >
>>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha Chintalapani <ka...@harsha.io>
>>> > wrote:
>>> >
>>> > > Hi Grant,
>>> > >           We are working on it. Will add the details to KIP about the
>>> > > request protocol.
>>> > >
>>> > > Thanks,
>>> > > Harsha
>>> > >
>>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant Henke <gh...@cloudera.com>
>>> wrote:
>>> > >
>>> > > > Hi Parth,
>>> > > >
>>> > > > Are you still working on this? If you need any help please don't
>>> > hesitate
>>> > > > to ask.
>>> > > >
>>> > > > Thanks,
>>> > > > Grant
>>> > > >
>>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <ju...@confluent.io> wrote:
>>> > > >
>>> > > > > Parth,
>>> > > > >
>>> > > > > Thanks for the reply.
>>> > > > >
>>> > > > > It makes sense to only allow the renewal by users that
>>> authenticated
>>> > > > using
>>> > > > > *non* delegation token mechanism. Then, should we make the
>>> renewal a
>>> > > > list?
>>> > > > > For example, in the case of rest proxy, it will be useful for
>>> every
>>> > > > > instance of rest proxy to be able to renew the tokens.
>>> > > > >
>>> > > > > It would be clearer if we can document the request protocol like
>>> > > > >
>>> > > > >
>>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> > > 4+-+Command+line+and+centralized+administrative+operations#KIP-4-
>>> > > Commandlineandcentralizedadministrativeoperations-
>>> > > CreateTopicsRequest(KAFKA-2945):(VotedandPlannedforin0.10.1.0)
>>> > > > > .
>>> > > > >
>>> > > > > It would also be useful to document the client APIs.
>>> > > > >
>>> > > > > Thanks,
>>> > > > >
>>> > > > > Jun
>>> > > > >
>>> > > > > On Tue, Jun 28, 2016 at 2:55 PM, parth brahmbhatt <
>>> > > > > brahmbhatt.parth@gmail.com> wrote:
>>> > > > >
>>> > > > > > Hi,
>>> > > > > >
>>> > > > > > I am suggesting that we will only allow the renewal by users
>>> that
>>> > > > > > authenticated using *non* delegation token mechanism. For
>>> example,
>>> > If
>>> > > > > user
>>> > > > > > Alice authenticated using kerberos and requested delegation
>>> tokens,
>>> > > > only
>>> > > > > > user Alice authenticated via non delegation token mechanism can
>>> > > renew.
>>> > > > > > Clients that have  access to delegation tokens can not issue
>>> > renewal
>>> > > > > > request for renewing their own token and this is primarily
>>> > important
>>> > > to
>>> > > > > > reduce the time window for which a compromised token will be
>>> valid.
>>> > > > > >
>>> > > > > > To clarify, Yes any authenticated user can request delegation
>>> > tokens
>>> > > > but
>>> > > > > > even here I would recommend to avoid creating a chain where a
>>> > client
>>> > > > > > authenticated via delegation token request for more delegation
>>> > > tokens.
>>> > > > > > Basically anyone can request delegation token, as long as they
>>> > > > > authenticate
>>> > > > > > via a non delegation token mechanism.
>>> > > > > >
>>> > > > > > Aren't classes listed here
>>> > > > > > <
>>> > > > > >
>>> > > > >
>>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> > > 48+Delegation+token+support+for+Kafka#KIP-48Delegationtokens
>>> upportforKaf
>>> > > ka-PublicInterfaces
>>> > > > > > >
>>> > > > > > sufficient?
>>> > > > > >
>>> > > > > > Thanks
>>> > > > > > Parth
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao <ju...@confluent.io>
>>> wrote:
>>> > > > > >
>>> > > > > > > Parth,
>>> > > > > > >
>>> > > > > > > Thanks for the reply. A couple of comments inline below.
>>> > > > > > >
>>> > > > > > > On Tue, Jun 21, 2016 at 10:36 AM, parth brahmbhatt <
>>> > > > > > > brahmbhatt.parth@gmail.com> wrote:
>>> > > > > > >
>>> > > > > > > > 1. Who / how are tokens renewed? By original requester
>>> only? or
>>> > > > using
>>> > > > > > > > Kerberos
>>> > > > > > > > auth only?
>>> > > > > > > > My recommendation is to do this only using Kerberos auth and
>>> > only
>>> > > > > threw
>>> > > > > > > the
>>> > > > > > > > renewer specified during the acquisition request.
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > Hmm, not sure that I follow this. Are you saying that any
>>> client
>>> > > > > > > authenticated with the delegation token can renew, i.e. there
>>> is
>>> > no
>>> > > > > > renewer
>>> > > > > > > needed?
>>> > > > > > >
>>> > > > > > > Also, just to be clear, any authenticated client (either
>>> through
>>> > > SASL
>>> > > > > or
>>> > > > > > > SSL) can request a delegation token for the authenticated
>>> user,
>>> > > > right?
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > > 2. Are tokens stored on each broker or in ZK?
>>> > > > > > > > My recommendation is still to store in ZK or not store them
>>> at
>>> > > all.
>>> > > > > The
>>> > > > > > > > whole controller based distribution is too much overhead
>>> with
>>> > not
>>> > > > > much
>>> > > > > > to
>>> > > > > > > > achieve.
>>> > > > > > > >
>>> > > > > > > > 3. How are tokens invalidated / expired?
>>> > > > > > > > Either by expiration time out or through an explicit
>>> request to
>>> > > > > > > invalidate.
>>> > > > > > > >
>>> > > > > > > > 4. Which encryption algorithm is used?
>>> > > > > > > > SCRAM
>>> > > > > > > >
>>> > > > > > > > 5. What is the impersonation proposal (it wasn't in the KIP
>>> but
>>> > > was
>>> > > > > > > > discussed
>>> > > > > > > > in this thread)?
>>> > > > > > > > There is no imperonation proposal. I tried and explained how
>>> > its
>>> > > a
>>> > > > > > > > different problem and why its not really necessary to
>>> discuss
>>> > > that
>>> > > > as
>>> > > > > > > part
>>> > > > > > > > of this KIP.  This KIP will not support any impersonation,
>>> it
>>> > > will
>>> > > > > just
>>> > > > > > > be
>>> > > > > > > > another way to authenticate.
>>> > > > > > > >
>>> > > > > > > > 6. Do we need new ACLs, if so - for what actions?
>>> > > > > > > > We do not need new ACLs.
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > Could we document the format of the new request/response and
>>> > their
>>> > > > > > > associated Resource and Operation for ACL?
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > > 7. How would the delegation token be configured in the
>>> client?
>>> > > > > > > > Should be through config. I wasn't planning on supporting
>>> JAAS
>>> > > for
>>> > > > > > > tokens.
>>> > > > > > > > I don't believe hadoop does this either.
>>> > > > > > > >
>>> > > > > > > > Thanks
>>> > > > > > > > Parth
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <ju...@confluent.io>
>>> > > wrote:
>>> > > > > > > >
>>> > > > > > > > > Harsha,
>>> > > > > > > > >
>>> > > > > > > > > Another question.
>>> > > > > > > > >
>>> > > > > > > > > 9. How would the delegation token be configured in the
>>> > client?
>>> > > > The
>>> > > > > > > > standard
>>> > > > > > > > > way is to do this through JAAS. However, we will need to
>>> > think
>>> > > > > > through
>>> > > > > > > if
>>> > > > > > > > > this is convenient in a shared environment. For example,
>>> > when a
>>> > > > new
>>> > > > > > > task
>>> > > > > > > > is
>>> > > > > > > > > added to a Storm worker node, do we need to dynamically
>>> add a
>>> > > new
>>> > > > > > > section
>>> > > > > > > > > in the JAAS file? It may be more convenient if we can
>>> pass in
>>> > > the
>>> > > > > > token
>>> > > > > > > > > through the config directly w/o going through JAAS.
>>> > > > > > > > >
>>> > > > > > > > > Are you or Parth still actively working on this KIP?
>>> > > > > > > > >
>>> > > > > > > > > Thanks,
>>> > > > > > > > >
>>> > > > > > > > > Jun
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <
>>> jun@confluent.io>
>>> > > > wrote:
>>> > > > > > > > >
>>> > > > > > > > > > Just to add on that list.
>>> > > > > > > > > >
>>> > > > > > > > > > 2. It would be good to document the format of the data
>>> > stored
>>> > > > in
>>> > > > > > ZK.
>>> > > > > > > > > > 7. Earlier, there was a discussion on whether the tokens
>>> > > should
>>> > > > > be
>>> > > > > > > > > > propagated through ZK like config/acl/quota, or through
>>> the
>>> > > > > > > controller.
>>> > > > > > > > > > Currently, the controller is only designed for
>>> propagating
>>> > > > topic
>>> > > > > > > > > metadata,
>>> > > > > > > > > > but not other data.
>>> > > > > > > > > > 8. Should we use SCRAM to send the token instead of
>>> > > DIGEST-MD5
>>> > > > > > since
>>> > > > > > > > it's
>>> > > > > > > > > > deprecated?
>>> > > > > > > > > >
>>> > > > > > > > > > Also, the images in the wiki seem broken.
>>> > > > > > > > > >
>>> > > > > > > > > > Thanks,
>>> > > > > > > > > >
>>> > > > > > > > > > Jun
>>> > > > > > > > > >
>>> > > > > > > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <
>>> > > > > gwen@confluent.io>
>>> > > > > > > > > wrote:
>>> > > > > > > > > >
>>> > > > > > > > > >> From what I can see, remaining questions are:
>>> > > > > > > > > >>
>>> > > > > > > > > >> 1. Who / how are tokens renewed? By original requester
>>> > only?
>>> > > > or
>>> > > > > > > using
>>> > > > > > > > > >> Kerberos auth only?
>>> > > > > > > > > >> 2. Are tokens stored on each broker or in ZK?
>>> > > > > > > > > >> 3. How are tokens invalidated / expired?
>>> > > > > > > > > >> 4. Which encryption algorithm is used?
>>> > > > > > > > > >> 5. What is the impersonation proposal (it wasn't in the
>>> > KIP
>>> > > > but
>>> > > > > > was
>>> > > > > > > > > >> discussed in this thread)?
>>> > > > > > > > > >> 6. Do we need new ACLs, if so - for what actions?
>>> > > > > > > > > >>
>>> > > > > > > > > >> Gwen
>>> > > > > > > > > >>
>>> > > > > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <
>>> kafka@harsha.io>
>>> > > > wrote:
>>> > > > > > > > > >> > Jun & Ismael,
>>> > > > > > > > > >> >                          Unfortunately I couldn't
>>> attend
>>> > > the
>>> > > > > KIP
>>> > > > > > > > > meeting
>>> > > > > > > > > >> >                          when delegation tokens
>>> > discussed.
>>> > > > > > > > Appreciate
>>> > > > > > > > > if
>>> > > > > > > > > >> >                          you can update the thread if
>>> > you
>>> > > > have
>>> > > > > > any
>>> > > > > > > > > >> >                          further questions.
>>> > > > > > > > > >> > Thanks,
>>> > > > > > > > > >> > Harsha
>>> > > > > > > > > >> >
>>> > > > > > > > > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
>>> > > > > > > > > >> >> It seems that the links to images in the KIP are
>>> > broken.
>>> > > > > > > > > >> >>
>>> > > > > > > > > >> >> Liquan
>>> > > > > > > > > >> >>
>>> > > > > > > > > >> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
>>> > > > > > > > > >> >> brahmbhatt.parth@gmail.com> wrote:
>>> > > > > > > > > >> >>
>>> > > > > > > > > >> >> > 110. What does getDelegationTokenAs mean?
>>> > > > > > > > > >> >> > In the current proposal we only allow a user to
>>> get
>>> > > > > > delegation
>>> > > > > > > > > token
>>> > > > > > > > > >> for
>>> > > > > > > > > >> >> > the identity that it authenticated as using
>>> another
>>> > > > > > mechanism,
>>> > > > > > > > i.e.
>>> > > > > > > > > >> A user
>>> > > > > > > > > >> >> > that authenticate using a keytab for principal
>>> > > > > > > user1@EXAMPLE.COM
>>> > > > > > > > > >> will get
>>> > > > > > > > > >> >> > delegation tokens for that user only. In future I
>>> > think
>>> > > > we
>>> > > > > > will
>>> > > > > > > > > have
>>> > > > > > > > > >> to
>>> > > > > > > > > >> >> > extend support such that we allow some set of
>>> users (
>>> > > > > > > > > >> >> > kafka-rest-user@EXAMPLE.COM,
>>> > storm-nimbus@EXAMPLE.COM)
>>> > > > to
>>> > > > > > > > acquire
>>> > > > > > > > > >> >> > delegation tokens on behalf of other users whose
>>> > > identity
>>> > > > > > they
>>> > > > > > > > have
>>> > > > > > > > > >> >> > verified independently.  Kafka brokers will have
>>> ACLs
>>> > > to
>>> > > > > > > control
>>> > > > > > > > > >> which
>>> > > > > > > > > >> >> > users are allowed to impersonate other users and
>>> get
>>> > > > tokens
>>> > > > > > on
>>> > > > > > > > > >> behalf of
>>> > > > > > > > > >> >> > them. Overall Impersonation is a whole different
>>> > > problem
>>> > > > in
>>> > > > > > my
>>> > > > > > > > > >> opinion and
>>> > > > > > > > > >> >> > I think we can tackle it in separate KIP.
>>> > > > > > > > > >> >> >
>>> > > > > > > > > >> >> > 111. What's the typical rate of getting and
>>> renewing
>>> > > > > > delegation
>>> > > > > > > > > >> tokens?
>>> > > > > > > > > >> >> > Typically this should be very very low, 1 request
>>> per
>>> > > > > minute
>>> > > > > > > is a
>>> > > > > > > > > >> >> > relatively high estimate. However it depends on
>>> the
>>> > > token
>>> > > > > > > > > >> expiration. I am
>>> > > > > > > > > >> >> > less worried about the extra load it puts on
>>> > controller
>>> > > > vs
>>> > > > > > the
>>> > > > > > > > > added
>>> > > > > > > > > >> >> > complexity and the value it offers.
>>> > > > > > > > > >> >> >
>>> > > > > > > > > >> >> > Thanks
>>> > > > > > > > > >> >> > Parth
>>> > > > > > > > > >> >> >
>>> > > > > > > > > >> >> >
>>> > > > > > > > > >> >> >
>>> > > > > > > > > >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <
>>> > > > > > > ismael@juma.me.uk>
>>> > > > > > > > > >> wrote:
>>> > > > > > > > > >> >> >
>>> > > > > > > > > >> >> > > Thanks Rajini. It would probably require a
>>> separate
>>> > > KIP
>>> > > > > as
>>> > > > > > it
>>> > > > > > > > > will
>>> > > > > > > > > >> >> > > introduce user visible changes. We could also
>>> > update
>>> > > > > KIP-48
>>> > > > > > > to
>>> > > > > > > > > >> have this
>>> > > > > > > > > >> >> > > information, but it seems cleaner to do it
>>> > > separately.
>>> > > > We
>>> > > > > > can
>>> > > > > > > > > >> discuss
>>> > > > > > > > > >> >> > that
>>> > > > > > > > > >> >> > > in the KIP call today.
>>> > > > > > > > > >> >> > >
>>> > > > > > > > > >> >> > > Ismael
>>> > > > > > > > > >> >> > >
>>> > > > > > > > > >> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram
>>> <
>>> > > > > > > > > >> >> > > rajinisivaram@googlemail.com> wrote:
>>> > > > > > > > > >> >> > >
>>> > > > > > > > > >> >> > > > Ismael,
>>> > > > > > > > > >> >> > > >
>>> > > > > > > > > >> >> > > > I have created a JIRA (
>>> > > > > > > > > >> >> > https://issues.apache.org/jira/browse/KAFKA-3751)
>>> > > > > > > > > >> >> > > > for adding SCRAM as a SASL mechanism. Would
>>> that
>>> > > need
>>> > > > > > > another
>>> > > > > > > > > >> KIP? If
>>> > > > > > > > > >> >> > > > KIP-48 will use this mechanism, can this just
>>> be
>>> > a
>>> > > > JIRA
>>> > > > > > > that
>>> > > > > > > > > gets
>>> > > > > > > > > >> >> > > reviewed
>>> > > > > > > > > >> >> > > > when the PR is ready?
>>> > > > > > > > > >> >> > > >
>>> > > > > > > > > >> >> > > > Thank you,
>>> > > > > > > > > >> >> > > >
>>> > > > > > > > > >> >> > > > Rajini
>>> > > > > > > > > >> >> > > >
>>> > > > > > > > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <
>>> > > > > > > > > ismael@juma.me.uk>
>>> > > > > > > > > >> >> > wrote:
>>> > > > > > > > > >> >> > > >
>>> > > > > > > > > >> >> > > > > Thanks Rajini, SCRAM seems like a good
>>> > candidate.
>>> > > > > > > > > >> >> > > > >
>>> > > > > > > > > >> >> > > > > Gwen had independently mentioned this as a
>>> SASL
>>> > > > > > mechanism
>>> > > > > > > > > that
>>> > > > > > > > > >> might
>>> > > > > > > > > >> >> > be
>>> > > > > > > > > >> >> > > > > useful for Kafka and I have been meaning to
>>> > check
>>> > > > it
>>> > > > > in
>>> > > > > > > > more
>>> > > > > > > > > >> detail.
>>> > > > > > > > > >> >> > > Good
>>> > > > > > > > > >> >> > > > > to know that you are willing to contribute
>>> an
>>> > > > > > > > implementation.
>>> > > > > > > > > >> Maybe
>>> > > > > > > > > >> >> > we
>>> > > > > > > > > >> >> > > > > should file a separate JIRA for this?
>>> > > > > > > > > >> >> > > > >
>>> > > > > > > > > >> >> > > > > Ismael
>>> > > > > > > > > >> >> > > > >
>>> > > > > > > > > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini
>>> > Sivaram <
>>> > > > > > > > > >> >> > > > > rajinisivaram@googlemail.com> wrote:
>>> > > > > > > > > >> >> > > > >
>>> > > > > > > > > >> >> > > > > > SCRAM (Salted Challenge Response
>>> > Authentication
>>> > > > > > > > Mechanism)
>>> > > > > > > > > >> is a
>>> > > > > > > > > >> >> > > better
>>> > > > > > > > > >> >> > > > > > mechanism than Digest-MD5. Java doesn't
>>> come
>>> > > > with a
>>> > > > > > > > > built-in
>>> > > > > > > > > >> SCRAM
>>> > > > > > > > > >> >> > > > > > SaslServer or SaslClient, but I will be
>>> happy
>>> > > to
>>> > > > > add
>>> > > > > > > > > support
>>> > > > > > > > > >> in
>>> > > > > > > > > >> >> > Kafka
>>> > > > > > > > > >> >> > > > > since
>>> > > > > > > > > >> >> > > > > > it would be a useful mechanism to support
>>> > > anyway.
>>> > > > > > > > > >> >> > > > > > https://tools.ietf.org/html/rfc7677
>>> > describes
>>> > > > the
>>> > > > > > > > protocol
>>> > > > > > > > > >> for
>>> > > > > > > > > >> >> > > > > > SCRAM-SHA-256.
>>> > > > > > > > > >> >> > > > > >
>>> > > > > > > > > >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <
>>> > > > > > > > jun@confluent.io
>>> > > > > > > > > >
>>> > > > > > > > > >> wrote:
>>> > > > > > > > > >> >> > > > > >
>>> > > > > > > > > >> >> > > > > > > Parth,
>>> > > > > > > > > >> >> > > > > > >
>>> > > > > > > > > >> >> > > > > > > Thanks for the explanation. A couple of
>>> > more
>>> > > > > > > questions.
>>> > > > > > > > > >> >> > > > > > >
>>> > > > > > > > > >> >> > > > > > > 110. What does getDelegationTokenAs
>>> mean?
>>> > > > > > > > > >> >> > > > > > >
>>> > > > > > > > > >> >> > > > > > > 111. What's the typical rate of getting
>>> and
>>> > > > > > renewing
>>> > > > > > > > > >> delegation
>>> > > > > > > > > >> >> > > > tokens?
>>> > > > > > > > > >> >> > > > > > > That may have an impact on whether they
>>> > > should
>>> > > > be
>>> > > > > > > > > directed
>>> > > > > > > > > >> to the
>>> > > > > > > > > >> >> > > > > > > controller.
>>> > > > > > > > > >> >> > > > > > >
>>> > > > > > > > > >> >> > > > > > > Jun
>>> > > > > > > > > >> >> > > > > > >
>>> > > > > > > > > >> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth
>>> > > > > brahmbhatt <
>>> > > > > > > > > >> >> > > > > > > brahmbhatt.parth@gmail.com> wrote:
>>> > > > > > > > > >> >> > > > > > >
>>> > > > > > > > > >> >> > > > > > > > Hi Jun,
>>> > > > > > > > > >> >> > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > Thanks for reviewing.
>>> > > > > > > > > >> >> > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > * We could add a Cluster action to add
>>> > acls
>>> > > > on
>>> > > > > > who
>>> > > > > > > > can
>>> > > > > > > > > >> request
>>> > > > > > > > > >> >> > > > > > delegation
>>> > > > > > > > > >> >> > > > > > > > tokens. I don't see the use case for
>>> that
>>> > > yet
>>> > > > > but
>>> > > > > > > > down
>>> > > > > > > > > >> the line
>>> > > > > > > > > >> >> > > > when
>>> > > > > > > > > >> >> > > > > we
>>> > > > > > > > > >> >> > > > > > > > start supporting getDelegationTokenAs
>>> it
>>> > > will
>>> > > > > be
>>> > > > > > > > > >> necessary.
>>> > > > > > > > > >> >> > > > > > > > * Yes we recommend tokens to be only
>>> > > > > > > used/distributed
>>> > > > > > > > > >> over
>>> > > > > > > > > >> >> > secure
>>> > > > > > > > > >> >> > > > > > > channels.
>>> > > > > > > > > >> >> > > > > > > > * Depending on what design we end up
>>> > > choosing
>>> > > > > > > > > >> Invalidation will
>>> > > > > > > > > >> >> > > be
>>> > > > > > > > > >> >> > > > > > > > responsibility of every broker or
>>> > > controller.
>>> > > > > > > > > >> >> > > > > > > > * I am not sure if I documented
>>> somewhere
>>> > > > that
>>> > > > > > > > > >> invalidation
>>> > > > > > > > > >> >> > will
>>> > > > > > > > > >> >> > > > > > directly
>>> > > > > > > > > >> >> > > > > > > > go through zookeeper but that is not
>>> the
>>> > > > > intent.
>>> > > > > > > > > >> Invalidation
>>> > > > > > > > > >> >> > > will
>>> > > > > > > > > >> >> > > > > > either
>>> > > > > > > > > >> >> > > > > > > > be request based or due to
>>> expiration. No
>>> > > > > direct
>>> > > > > > > > > >> zookeeper
>>> > > > > > > > > >> >> > > > > interaction
>>> > > > > > > > > >> >> > > > > > > from
>>> > > > > > > > > >> >> > > > > > > > any client.
>>> > > > > > > > > >> >> > > > > > > > * "Broker also stores the
>>> DelegationToken
>>> > > > > without
>>> > > > > > > the
>>> > > > > > > > > >> hmac in
>>> > > > > > > > > >> >> > the
>>> > > > > > > > > >> >> > > > > > > > zookeeper." : Sorry about the
>>> confusion.
>>> > > The
>>> > > > > sole
>>> > > > > > > > > >> purpose of
>>> > > > > > > > > >> >> > > > > zookeeper
>>> > > > > > > > > >> >> > > > > > in
>>> > > > > > > > > >> >> > > > > > > > this design is as distribution channel
>>> > for
>>> > > > > tokens
>>> > > > > > > > > >> between all
>>> > > > > > > > > >> >> > > > brokers
>>> > > > > > > > > >> >> > > > > > > and a
>>> > > > > > > > > >> >> > > > > > > > layer that ensures only tokens that
>>> were
>>> > > > > > generated
>>> > > > > > > by
>>> > > > > > > > > >> making a
>>> > > > > > > > > >> >> > > > > request
>>> > > > > > > > > >> >> > > > > > > to a
>>> > > > > > > > > >> >> > > > > > > > broker will be accepted (more on this
>>> in
>>> > > > second
>>> > > > > > > > > >> paragraph). The
>>> > > > > > > > > >> >> > > > token
>>> > > > > > > > > >> >> > > > > > > > consists of few elements (owner,
>>> renewer,
>>> > > > uuid
>>> > > > > ,
>>> > > > > > > > > >> expiration,
>>> > > > > > > > > >> >> > > hmac)
>>> > > > > > > > > >> >> > > > ,
>>> > > > > > > > > >> >> > > > > > one
>>> > > > > > > > > >> >> > > > > > > of
>>> > > > > > > > > >> >> > > > > > > > which is the finally generated hmac
>>> but
>>> > > hmac
>>> > > > it
>>> > > > > > > self
>>> > > > > > > > is
>>> > > > > > > > > >> >> > derivable
>>> > > > > > > > > >> >> > > > if
>>> > > > > > > > > >> >> > > > > > you
>>> > > > > > > > > >> >> > > > > > > > have all the other elements of the
>>> token
>>> > +
>>> > > > > secret
>>> > > > > > > key
>>> > > > > > > > > to
>>> > > > > > > > > >> >> > generate
>>> > > > > > > > > >> >> > > > > hmac.
>>> > > > > > > > > >> >> > > > > > > > Given zookeeper does not provide SSL
>>> > > support
>>> > > > we
>>> > > > > > do
>>> > > > > > > > not
>>> > > > > > > > > >> want the
>>> > > > > > > > > >> >> > > > > entire
>>> > > > > > > > > >> >> > > > > > > > token to be wire transferred to
>>> zookeeper
>>> > > as
>>> > > > > that
>>> > > > > > > > will
>>> > > > > > > > > >> be an
>>> > > > > > > > > >> >> > > > insecure
>>> > > > > > > > > >> >> > > > > > > wire
>>> > > > > > > > > >> >> > > > > > > > transfer. Instead we only store all
>>> the
>>> > > other
>>> > > > > > > > elements
>>> > > > > > > > > >> of a
>>> > > > > > > > > >> >> > > > > delegation
>>> > > > > > > > > >> >> > > > > > > > tokens. Brokers can read these
>>> elements
>>> > and
>>> > > > > > because
>>> > > > > > > > > they
>>> > > > > > > > > >> also
>>> > > > > > > > > >> >> > > have
>>> > > > > > > > > >> >> > > > > > access
>>> > > > > > > > > >> >> > > > > > > > to secret key they will be able to
>>> > generate
>>> > > > > hmac
>>> > > > > > on
>>> > > > > > > > > >> their end.
>>> > > > > > > > > >> >> > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > One of the alternative proposed is to
>>> > avoid
>>> > > > > > > zookeeper
>>> > > > > > > > > >> >> > > altogether. A
>>> > > > > > > > > >> >> > > > > > > Client
>>> > > > > > > > > >> >> > > > > > > > will call broker with required
>>> > information
>>> > > > > > (owner,
>>> > > > > > > > > >> renwer,
>>> > > > > > > > > >> >> > > > > expiration)
>>> > > > > > > > > >> >> > > > > > > and
>>> > > > > > > > > >> >> > > > > > > > get back (signed hmac, uuid). Broker
>>> > won't
>>> > > > > store
>>> > > > > > > this
>>> > > > > > > > > in
>>> > > > > > > > > >> >> > > zookeeper.
>>> > > > > > > > > >> >> > > > > > From
>>> > > > > > > > > >> >> > > > > > > > this point a client can contact any
>>> > broker
>>> > > > with
>>> > > > > > all
>>> > > > > > > > the
>>> > > > > > > > > >> >> > > delegation
>>> > > > > > > > > >> >> > > > > > token
>>> > > > > > > > > >> >> > > > > > > > info (owner, rewner, expiration, hmac,
>>> > > uuid)
>>> > > > > the
>>> > > > > > > > borker
>>> > > > > > > > > >> will
>>> > > > > > > > > >> >> > > > > regenerate
>>> > > > > > > > > >> >> > > > > > > the
>>> > > > > > > > > >> >> > > > > > > > hmac and as long as it matches with
>>> hmac
>>> > > > > > presented
>>> > > > > > > by
>>> > > > > > > > > >> client ,
>>> > > > > > > > > >> >> > > > broker
>>> > > > > > > > > >> >> > > > > > > will
>>> > > > > > > > > >> >> > > > > > > > allow the request to authenticate.
>>> Only
>>> > > > > problem
>>> > > > > > > with
>>> > > > > > > > > >> this
>>> > > > > > > > > >> >> > > approach
>>> > > > > > > > > >> >> > > > > is
>>> > > > > > > > > >> >> > > > > > if
>>> > > > > > > > > >> >> > > > > > > > the secret key is compromised any
>>> client
>>> > > can
>>> > > > > now
>>> > > > > > > > > generate
>>> > > > > > > > > >> >> > random
>>> > > > > > > > > >> >> > > > > tokens
>>> > > > > > > > > >> >> > > > > > > and
>>> > > > > > > > > >> >> > > > > > > > they will still be able to
>>> authenticate
>>> > as
>>> > > > any
>>> > > > > > user
>>> > > > > > > > > they
>>> > > > > > > > > >> like.
>>> > > > > > > > > >> >> > > with
>>> > > > > > > > > >> >> > > > > > > > zookeeper we guarantee that only
>>> tokens
>>> > > > > acquired
>>> > > > > > > via
>>> > > > > > > > a
>>> > > > > > > > > >> broker
>>> > > > > > > > > >> >> > > > (using
>>> > > > > > > > > >> >> > > > > > some
>>> > > > > > > > > >> >> > > > > > > > auth scheme other than delegation
>>> token)
>>> > > will
>>> > > > > be
>>> > > > > > > > > >> accepted. We
>>> > > > > > > > > >> >> > > need
>>> > > > > > > > > >> >> > > > to
>>> > > > > > > > > >> >> > > > > > > > discuss which proposal makes more
>>> sense
>>> > and
>>> > > > we
>>> > > > > > can
>>> > > > > > > go
>>> > > > > > > > > >> over it
>>> > > > > > > > > >> >> > in
>>> > > > > > > > > >> >> > > > > > > tomorrow's
>>> > > > > > > > > >> >> > > > > > > > meeting.
>>> > > > > > > > > >> >> > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > Also, can you forward the invite to
>>> me?
>>> > > > > > > > > >> >> > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > Thanks
>>> > > > > > > > > >> >> > > > > > > > Parth
>>> > > > > > > > > >> >> > > > > > > >
>>> > > > > > > > > >> >> > > > > > > >
>>> > > > > > > > > >> >> > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun
>>> > Rao <
>>> > > > > > > > > >> jun@confluent.io>
>>> > > > > > > > > >> >> > > > wrote:
>>> > > > > > > > > >> >> > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > Thanks for the KIP. A few comments.
>>> > > > > > > > > >> >> > > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > 100. This potentially can be useful
>>> for
>>> > > > Kafka
>>> > > > > > > > Connect
>>> > > > > > > > > >> and
>>> > > > > > > > > >> >> > Kafka
>>> > > > > > > > > >> >> > > > > rest
>>> > > > > > > > > >> >> > > > > > > > proxy
>>> > > > > > > > > >> >> > > > > > > > > where a worker agent will need to
>>> run a
>>> > > > task
>>> > > > > on
>>> > > > > > > > > behalf
>>> > > > > > > > > >> of a
>>> > > > > > > > > >> >> > > > client.
>>> > > > > > > > > >> >> > > > > > We
>>> > > > > > > > > >> >> > > > > > > > will
>>> > > > > > > > > >> >> > > > > > > > > likely need to change how those
>>> > services
>>> > > > use
>>> > > > > > > Kafka
>>> > > > > > > > > >> clients
>>> > > > > > > > > >> >> > > > > > > > > (producer/consumer). Instead of a
>>> > shared
>>> > > > > client
>>> > > > > > > per
>>> > > > > > > > > >> worker,
>>> > > > > > > > > >> >> > we
>>> > > > > > > > > >> >> > > > will
>>> > > > > > > > > >> >> > > > > > > need
>>> > > > > > > > > >> >> > > > > > > > a
>>> > > > > > > > > >> >> > > > > > > > > client per user task since the
>>> > > > authentication
>>> > > > > > > > happens
>>> > > > > > > > > >> at the
>>> > > > > > > > > >> >> > > > > > connection
>>> > > > > > > > > >> >> > > > > > > > > level. For Kafka Connect, the
>>> renewer
>>> > > will
>>> > > > be
>>> > > > > > the
>>> > > > > > > > > >> workers.
>>> > > > > > > > > >> >> > So,
>>> > > > > > > > > >> >> > > we
>>> > > > > > > > > >> >> > > > > > > > probably
>>> > > > > > > > > >> >> > > > > > > > > need to allow multiple renewers. For
>>> > > Kafka
>>> > > > > rest
>>> > > > > > > > > proxy,
>>> > > > > > > > > >> the
>>> > > > > > > > > >> >> > > > renewer
>>> > > > > > > > > >> >> > > > > > can
>>> > > > > > > > > >> >> > > > > > > > > probably just be the creator of the
>>> > > token.
>>> > > > > > > > > >> >> > > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > 101. Do we need new acl on who can
>>> > > request
>>> > > > > > > > delegation
>>> > > > > > > > > >> tokens?
>>> > > > > > > > > >> >> > > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > 102. Do we recommend people to send
>>> > > > > delegation
>>> > > > > > > > tokens
>>> > > > > > > > > >> in an
>>> > > > > > > > > >> >> > > > > encrypted
>>> > > > > > > > > >> >> > > > > > > > > channel?
>>> > > > > > > > > >> >> > > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > 103. Who is responsible for expiring
>>> > > > tokens,
>>> > > > > > > every
>>> > > > > > > > > >> broker?
>>> > > > > > > > > >> >> > > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > 104. For invalidating tokens, would
>>> it
>>> > be
>>> > > > > > better
>>> > > > > > > to
>>> > > > > > > > > do
>>> > > > > > > > > >> it in
>>> > > > > > > > > >> >> > a
>>> > > > > > > > > >> >> > > > > > request
>>> > > > > > > > > >> >> > > > > > > > > instead of going to ZK directly?
>>> > > > > > > > > >> >> > > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > 105. The terminology of client in
>>> the
>>> > > wiki
>>> > > > > > > > sometimes
>>> > > > > > > > > >> refers
>>> > > > > > > > > >> >> > to
>>> > > > > > > > > >> >> > > > the
>>> > > > > > > > > >> >> > > > > > end
>>> > > > > > > > > >> >> > > > > > > > > client and some other times refers
>>> to
>>> > the
>>> > > > > > client
>>> > > > > > > > > using
>>> > > > > > > > > >> the
>>> > > > > > > > > >> >> > > > > delegation
>>> > > > > > > > > >> >> > > > > > > > > tokens. It would be useful to
>>> > distinguish
>>> > > > > > between
>>> > > > > > > > the
>>> > > > > > > > > >> two.
>>> > > > > > > > > >> >> > > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > 106. Could you explain the sentence
>>> > > "Broker
>>> > > > > > also
>>> > > > > > > > > >> stores the
>>> > > > > > > > > >> >> > > > > > > > DelegationToken
>>> > > > > > > > > >> >> > > > > > > > > without the hmac in the zookeeper."
>>> a
>>> > bit
>>> > > > > > more? I
>>> > > > > > > > > >> thought the
>>> > > > > > > > > >> >> > > > > > > delegation
>>> > > > > > > > > >> >> > > > > > > > > token is the hmac.
>>> > > > > > > > > >> >> > > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > Thanks,
>>> > > > > > > > > >> >> > > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > Jun
>>> > > > > > > > > >> >> > > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun
>>> > Rao
>>> > > <
>>> > > > > > > > > >> jun@confluent.io>
>>> > > > > > > > > >> >> > > > wrote:
>>> > > > > > > > > >> >> > > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > > Hi, Harsha,
>>> > > > > > > > > >> >> > > > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > > Just sent out a KIP meeting
>>> invite.
>>> > We
>>> > > > can
>>> > > > > > > > discuss
>>> > > > > > > > > >> this in
>>> > > > > > > > > >> >> > > the
>>> > > > > > > > > >> >> > > > > > > meeting
>>> > > > > > > > > >> >> > > > > > > > > > tomorrow.
>>> > > > > > > > > >> >> > > > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > > Thanks,
>>> > > > > > > > > >> >> > > > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > > Jun
>>> > > > > > > > > >> >> > > > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM,
>>> > > Harsha <
>>> > > > > > > > > >> kafka@harsha.io>
>>> > > > > > > > > >> >> > > > wrote:
>>> > > > > > > > > >> >> > > > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > >> Hi All,
>>> > > > > > > > > >> >> > > > > > > > > >>            Can we have a KIP
>>> meeting
>>> > > > > around
>>> > > > > > > > this.
>>> > > > > > > > > >> The KIP
>>> > > > > > > > > >> >> > is
>>> > > > > > > > > >> >> > > > up
>>> > > > > > > > > >> >> > > > > > for
>>> > > > > > > > > >> >> > > > > > > > > >>            sometime and if there
>>> are
>>> > > any
>>> > > > > > > > questions
>>> > > > > > > > > >> lets
>>> > > > > > > > > >> >> > > > quickly
>>> > > > > > > > > >> >> > > > > > hash
>>> > > > > > > > > >> >> > > > > > > > out
>>> > > > > > > > > >> >> > > > > > > > > >>            details.
>>> > > > > > > > > >> >> > > > > > > > > >>
>>> > > > > > > > > >> >> > > > > > > > > >> Thanks,
>>> > > > > > > > > >> >> > > > > > > > > >> Harsha
>>> > > > > > > > > >> >> > > > > > > > > >>
>>> > > > > > > > > >> >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40
>>> AM,
>>> > > parth
>>> > > > > > > > > brahmbhatt
>>> > > > > > > > > >> wrote:
>>> > > > > > > > > >> >> > > > > > > > > >> > That is what the hadoop echo
>>> > system
>>> > > > uses
>>> > > > > > so
>>> > > > > > > no
>>> > > > > > > > > >> good
>>> > > > > > > > > >> >> > reason
>>> > > > > > > > > >> >> > > > > > really.
>>> > > > > > > > > >> >> > > > > > > > We
>>> > > > > > > > > >> >> > > > > > > > > >> > could
>>> > > > > > > > > >> >> > > > > > > > > >> > change it to whatever is the
>>> > newest
>>> > > > > > > > recommended
>>> > > > > > > > > >> standard
>>> > > > > > > > > >> >> > > is.
>>> > > > > > > > > >> >> > > > > > > > > >> >
>>> > > > > > > > > >> >> > > > > > > > > >> > Thanks
>>> > > > > > > > > >> >> > > > > > > > > >> > Parth
>>> > > > > > > > > >> >> > > > > > > > > >> >
>>> > > > > > > > > >> >> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33
>>> AM,
>>> > > > Ismael
>>> > > > > > > Juma <
>>> > > > > > > > > >> >> > > > > ismael@juma.me.uk
>>> > > > > > > > > >> >> > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > wrote:
>>> > > > > > > > > >> >> > > > > > > > > >> >
>>> > > > > > > > > >> >> > > > > > > > > >> > > Hi Parth,
>>> > > > > > > > > >> >> > > > > > > > > >> > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > Thanks for the KIP. I only
>>> > started
>>> > > > > > > reviewing
>>> > > > > > > > > >> this and
>>> > > > > > > > > >> >> > > may
>>> > > > > > > > > >> >> > > > > have
>>> > > > > > > > > >> >> > > > > > > > > >> additional
>>> > > > > > > > > >> >> > > > > > > > > >> > > questions later. The
>>> immediate
>>> > > > > question
>>> > > > > > > that
>>> > > > > > > > > >> came to
>>> > > > > > > > > >> >> > > mind
>>> > > > > > > > > >> >> > > > is
>>> > > > > > > > > >> >> > > > > > our
>>> > > > > > > > > >> >> > > > > > > > > >> choice of
>>> > > > > > > > > >> >> > > > > > > > > >> > > "DIGEST-MD5" even though it's
>>> > > marked
>>> > > > > as
>>> > > > > > > > > >> OBSOLETE in
>>> > > > > > > > > >> >> > the
>>> > > > > > > > > >> >> > > > IANA
>>> > > > > > > > > >> >> > > > > > > > > Registry
>>> > > > > > > > > >> >> > > > > > > > > >> of
>>> > > > > > > > > >> >> > > > > > > > > >> > > SASL mechanisms and the
>>> original
>>> > > RFC
>>> > > > > > > (2831)
>>> > > > > > > > > has
>>> > > > > > > > > >> been
>>> > > > > > > > > >> >> > > moved
>>> > > > > > > > > >> >> > > > > to
>>> > > > > > > > > >> >> > > > > > > > > Historic
>>> > > > > > > > > >> >> > > > > > > > > >> > > status:
>>> > > > > > > > > >> >> > > > > > > > > >> > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > https://tools.ietf.org/html/
>>> > > rfc6331
>>> > > > > > > > > >> >> > > > > > > > > >> > >
>>> > > > > > > > > >> >> > > > > > > > >
>>> > > > > > > > > >> >> > > > > >
>>> > > > > > > > > >> >> > >
>>> > > > > > > > > >>
>>> > > > > > >
>>> > > > http://www.iana.org/assignments/sasl-mechanisms/sasl-
>>> mechanisms.xhtml
>>> > > > > > > > > >> >> > > > > > > > > >> > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > What is the reasoning behind
>>> > that
>>> > > > > > choice?
>>> > > > > > > > > >> >> > > > > > > > > >> > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > Thanks,
>>> > > > > > > > > >> >> > > > > > > > > >> > > Ismael
>>> > > > > > > > > >> >> > > > > > > > > >> > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29
>>> > PM,
>>> > > > Gwen
>>> > > > > > > > > Shapira <
>>> > > > > > > > > >> >> > > > > > > gwen@confluent.io
>>> > > > > > > > > >> >> > > > > > > > >
>>> > > > > > > > > >> >> > > > > > > > > >> wrote:
>>> > > > > > > > > >> >> > > > > > > > > >> > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > Also comments inline :)
>>> > > > > > > > > >> >> > > > > > > > > >> > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > * I want to emphasize
>>> that
>>> > > even
>>> > > > > > though
>>> > > > > > > > > >> delegation
>>> > > > > > > > > >> >> > > > tokens
>>> > > > > > > > > >> >> > > > > > > are a
>>> > > > > > > > > >> >> > > > > > > > > >> Hadoop
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > innovation, I feel very
>>> > > strongly
>>> > > > > > about
>>> > > > > > > > not
>>> > > > > > > > > >> adding
>>> > > > > > > > > >> >> > > > > > dependency
>>> > > > > > > > > >> >> > > > > > > > on
>>> > > > > > > > > >> >> > > > > > > > > >> Hadoop
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > when implementing
>>> delegation
>>> > > > > tokens
>>> > > > > > > for
>>> > > > > > > > > >> Kafka. The
>>> > > > > > > > > >> >> > > KIP
>>> > > > > > > > > >> >> > > > > > > doesn't
>>> > > > > > > > > >> >> > > > > > > > > >> imply
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > such dependency, but if
>>> you
>>> > > can
>>> > > > > > > > clarify...
>>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > *No hadoop dependency.*
>>> > > > > > > > > >> >> > > > > > > > > >> > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > Yay! Just add this to the
>>> KIP
>>> > so
>>> > > > no
>>> > > > > > one
>>> > > > > > > > will
>>> > > > > > > > > >> read
>>> > > > > > > > > >> >> > the
>>> > > > > > > > > >> >> > > > KIP
>>> > > > > > > > > >> >> > > > > > and
>>> > > > > > > > > >> >> > > > > > > > > panic
>>> > > > > > > > > >> >> > > > > > > > > >> > > > three weeks before the next
>>> > > > > release...
>>> > > > > > > > > >> >> > > > > > > > > >> > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > * Can we get delegation
>>> > token
>>> > > at
>>> > > > > any
>>> > > > > > > > time
>>> > > > > > > > > >> after
>>> > > > > > > > > >> >> > > > > > > > authenticating?
>>> > > > > > > > > >> >> > > > > > > > > >> only
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > immediately after?
>>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > *As long as you are
>>> > > > authenticated
>>> > > > > > you
>>> > > > > > > > can
>>> > > > > > > > > >> get
>>> > > > > > > > > >> >> > > > delegation
>>> > > > > > > > > >> >> > > > > > > > tokens.
>>> > > > > > > > > >> >> > > > > > > > > >> We
>>> > > > > > > > > >> >> > > > > > > > > >> > > need
>>> > > > > > > > > >> >> > > > > > > > > >> > > > to
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > discuss if a client
>>> > > > authenticated
>>> > > > > > > using
>>> > > > > > > > > >> delegation
>>> > > > > > > > > >> >> > > > > token,
>>> > > > > > > > > >> >> > > > > > > can
>>> > > > > > > > > >> >> > > > > > > > > also
>>> > > > > > > > > >> >> > > > > > > > > >> > > > acquire
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > delegation token again or
>>> > not.
>>> > > > > Also
>>> > > > > > > > there
>>> > > > > > > > > >> is the
>>> > > > > > > > > >> >> > > > > question
>>> > > > > > > > > >> >> > > > > > of
>>> > > > > > > > > >> >> > > > > > > > do
>>> > > > > > > > > >> >> > > > > > > > > we
>>> > > > > > > > > >> >> > > > > > > > > >> > > allow
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > anyone to acquire
>>> delegation
>>> > > > token
>>> > > > > > or
>>> > > > > > > we
>>> > > > > > > > > >> want
>>> > > > > > > > > >> >> > > specific
>>> > > > > > > > > >> >> > > > > > ACLs
>>> > > > > > > > > >> >> > > > > > > (I
>>> > > > > > > > > >> >> > > > > > > > > >> think
>>> > > > > > > > > >> >> > > > > > > > > >> > > its
>>> > > > > > > > > >> >> > > > > > > > > >> > > > an
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > overkill.)*
>>> > > > > > > > > >> >> > > > > > > > > >> > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > I agree that ACLs is an
>>> > > overkill.
>>> > > > > > > > > >> >> > > > > > > > > >> > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > I think we are debating two
>>> > > > options:
>>> > > > > > > > Either
>>> > > > > > > > > >> require
>>> > > > > > > > > >> >> > > > > Kerberos
>>> > > > > > > > > >> >> > > > > > > > auth
>>> > > > > > > > > >> >> > > > > > > > > >> for
>>> > > > > > > > > >> >> > > > > > > > > >> > > > renewal or require
>>> non-owners
>>> > to
>>> > > > > > renew.
>>> > > > > > > > > >> >> > > > > > > > > >> > > > I *think* the latter is
>>> > simpler
>>> > > > (it
>>> > > > > > > > > basically
>>> > > > > > > > > >> >> > require
>>> > > > > > > > > >> >> > > a
>>> > > > > > > > > >> >> > > > > "job
>>> > > > > > > > > >> >> > > > > > > > > master"
>>> > > > > > > > > >> >> > > > > > > > > >> > > > to take responsibility for
>>> the
>>> > > > > > renewal,
>>> > > > > > > it
>>> > > > > > > > > >> will have
>>> > > > > > > > > >> >> > > its
>>> > > > > > > > > >> >> > > > > own
>>> > > > > > > > > >> >> > > > > > > > > >> identity
>>> > > > > > > > > >> >> > > > > > > > > >> > > > anyway and I think this is
>>> the
>>> > > > > correct
>>> > > > > > > > > design
>>> > > > > > > > > >> >> > pattern
>>> > > > > > > > > >> >> > > > > > anyway.
>>> > > > > > > > > >> >> > > > > > > > For
>>> > > > > > > > > >> >> > > > > > > > > >> > > > storm, I'd expect Nimbus to
>>> > > > > coordinate
>>> > > > > > > > > >> renewals?),
>>> > > > > > > > > >> >> > but
>>> > > > > > > > > >> >> > > > it
>>> > > > > > > > > >> >> > > > > is
>>> > > > > > > > > >> >> > > > > > > > hard
>>> > > > > > > > > >> >> > > > > > > > > to
>>> > > > > > > > > >> >> > > > > > > > > >> > > > debate simplicity without
>>> > > looking
>>> > > > at
>>> > > > > > the
>>> > > > > > > > > code
>>> > > > > > > > > >> >> > changes
>>> > > > > > > > > >> >> > > > > > > required.
>>> > > > > > > > > >> >> > > > > > > > If
>>> > > > > > > > > >> >> > > > > > > > > >> you
>>> > > > > > > > > >> >> > > > > > > > > >> > > > have a draft of how the
>>> > "require
>>> > > > > > > Kerberos"
>>> > > > > > > > > >> will look
>>> > > > > > > > > >> >> > > in
>>> > > > > > > > > >> >> > > > > > Kafka
>>> > > > > > > > > >> >> > > > > > > > > code,
>>> > > > > > > > > >> >> > > > > > > > > >> > > > I'll be happy to take a
>>> look.
>>> > > > > > > > > >> >> > > > > > > > > >> > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > * My understanding is
>>> that
>>> > > > tokens
>>> > > > > > will
>>> > > > > > > > > >> propagate
>>> > > > > > > > > >> >> > via
>>> > > > > > > > > >> >> > > > ZK
>>> > > > > > > > > >> >> > > > > > but
>>> > > > > > > > > >> >> > > > > > > > > >> without
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > additional changes to
>>> > > > > UpdateMetadata
>>> > > > > > > > > >> protocol,
>>> > > > > > > > > >> >> > > > correct?
>>> > > > > > > > > >> >> > > > > > > > Clients
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > currently don't retry on
>>> > SASL
>>> > > > auth
>>> > > > > > > > failure
>>> > > > > > > > > >> (IIRC),
>>> > > > > > > > > >> >> > > but
>>> > > > > > > > > >> >> > > > > > since
>>> > > > > > > > > >> >> > > > > > > > the
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > tokens propagate between
>>> > > brokers
>>> > > > > > > asynch,
>>> > > > > > > > > we
>>> > > > > > > > > >> will
>>> > > > > > > > > >> >> > > need
>>> > > > > > > > > >> >> > > > to
>>> > > > > > > > > >> >> > > > > > > > retry a
>>> > > > > > > > > >> >> > > > > > > > > >> bit
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > to avoid clients failing
>>> > auth
>>> > > > due
>>> > > > > to
>>> > > > > > > > > timing
>>> > > > > > > > > >> >> > issues.
>>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > *I am considering 2
>>> > > alternatives
>>> > > > > > right
>>> > > > > > > > > now.
>>> > > > > > > > > >> The
>>> > > > > > > > > >> >> > > > current
>>> > > > > > > > > >> >> > > > > > > > > documented
>>> > > > > > > > > >> >> > > > > > > > > >> > > > approach
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > is zookeeper based and it
>>> > does
>>> > > > not
>>> > > > > > > > require
>>> > > > > > > > > >> any
>>> > > > > > > > > >> >> > > changes
>>> > > > > > > > > >> >> > > > > to
>>> > > > > > > > > >> >> > > > > > > > > >> > > UpdateMetadata
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > protocol. An alternative
>>> > > > approach
>>> > > > > > can
>>> > > > > > > > > remove
>>> > > > > > > > > >> >> > > zookeeper
>>> > > > > > > > > >> >> > > > > > > > > dependency
>>> > > > > > > > > >> >> > > > > > > > > >> as
>>> > > > > > > > > >> >> > > > > > > > > >> > > well
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > but we can discuss that
>>> in
>>> > KIP
>>> > > > > > > > discussion
>>> > > > > > > > > >> call.*
>>> > > > > > > > > >> >> > > > > > > > > >> > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > Oooh! Sounds interesting.
>>> Do
>>> > you
>>> > > > > want
>>> > > > > > to
>>> > > > > > > > > ping
>>> > > > > > > > > >> Jun to
>>> > > > > > > > > >> >> > > > > > arrange a
>>> > > > > > > > > >> >> > > > > > > > > call?
>>> > > > > > > > > >> >> > > > > > > > > >> > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > * I liked Ashish's
>>> > suggestion
>>> > > of
>>> > > > > > > having
>>> > > > > > > > > >> just the
>>> > > > > > > > > >> >> > > > > > controller
>>> > > > > > > > > >> >> > > > > > > > > issue
>>> > > > > > > > > >> >> > > > > > > > > >> the
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > delegation tokens, to
>>> avoid
>>> > > > > syncing
>>> > > > > > a
>>> > > > > > > > > shared
>>> > > > > > > > > >> >> > secret.
>>> > > > > > > > > >> >> > > > Not
>>> > > > > > > > > >> >> > > > > > > sure
>>> > > > > > > > > >> >> > > > > > > > if
>>> > > > > > > > > >> >> > > > > > > > > >> we
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > want to continue the
>>> > > discussion
>>> > > > > here
>>> > > > > > > or
>>> > > > > > > > on
>>> > > > > > > > > >> the
>>> > > > > > > > > >> >> > > wiki. I
>>> > > > > > > > > >> >> > > > > > think
>>> > > > > > > > > >> >> > > > > > > > > that
>>> > > > > > > > > >> >> > > > > > > > > >> we
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > can decouple the problem
>>> of
>>> > > > "token
>>> > > > > > > > > >> distribution"
>>> > > > > > > > > >> >> > > from
>>> > > > > > > > > >> >> > > > > > > "shared
>>> > > > > > > > > >> >> > > > > > > > > >> secret
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > distribution" and use the
>>> > > > > controller
>>> > > > > > > as
>>> > > > > > > > > the
>>> > > > > > > > > >> only
>>> > > > > > > > > >> >> > > token
>>> > > > > > > > > >> >> > > > > > > > generator
>>> > > > > > > > > >> >> > > > > > > > > >> to
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > solve the second issue,
>>> > while
>>> > > > > still
>>> > > > > > > > using
>>> > > > > > > > > >> ZK async
>>> > > > > > > > > >> >> > > to
>>> > > > > > > > > >> >> > > > > > > > distribute
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > tokens.
>>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > *As mentioned in the
>>> > previous
>>> > > > > Email
>>> > > > > > I
>>> > > > > > > am
>>> > > > > > > > > >> fine with
>>> > > > > > > > > >> >> > > > that
>>> > > > > > > > > >> >> > > > > > > > approach
>>> > > > > > > > > >> >> > > > > > > > > >> as
>>> > > > > > > > > >> >> > > > > > > > > >> > > long
>>> > > > > > > > > >> >> > > > > > > > > >> > > > as
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > we agree that the extra
>>> > > > complexity
>>> > > > > > of
>>> > > > > > > > > >> >> > > adding/updating
>>> > > > > > > > > >> >> > > > > APIS
>>> > > > > > > > > >> >> > > > > > > > adds
>>> > > > > > > > > >> >> > > > > > > > > >> enough
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > value. The advantage with
>>> > the
>>> > > > > > > controller
>>> > > > > > > > > >> approach
>>> > > > > > > > > >> >> > is
>>> > > > > > > > > >> >> > > > > > secret
>>> > > > > > > > > >> >> > > > > > > > > >> rotation
>>> > > > > > > > > >> >> > > > > > > > > >> > > can
>>> > > > > > > > > >> >> > > > > > > > > >> > > > be
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > automated,frequent and
>>> would
>>> > > not
>>> > > > > > > require
>>> > > > > > > > > >> >> > > deployment. *
>>> > > > > > > > > >> >> > > > > > > > > >> > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > Can you detail the extra
>>> > > > complexity
>>> > > > > > (or
>>> > > > > > > > > point
>>> > > > > > > > > >> me to
>>> > > > > > > > > >> >> > > the
>>> > > > > > > > > >> >> > > > > > email
>>> > > > > > > > > >> >> > > > > > > I
>>> > > > > > > > > >> >> > > > > > > > > >> > > > missed?) - which Apis are
>>> > > > required?
>>> > > > > > > > > >> >> > > > > > > > > >> > > > As far as I can tell,
>>> clients
>>> > > can
>>> > > > > > > already
>>> > > > > > > > > >> find the
>>> > > > > > > > > >> >> > > > > > controller
>>> > > > > > > > > >> >> > > > > > > > from
>>> > > > > > > > > >> >> > > > > > > > > >> > > > metadata. I'm a bit more
>>> > > concerned
>>> > > > > > about
>>> > > > > > > > > >> controller
>>> > > > > > > > > >> >> > > > load.
>>> > > > > > > > > >> >> > > > > > > > > >> > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > * While I like the idea
>>> of
>>> > > > forcing
>>> > > > > > > > > kerberos
>>> > > > > > > > > >> auth
>>> > > > > > > > > >> >> > for
>>> > > > > > > > > >> >> > > > > > > renwal, I
>>> > > > > > > > > >> >> > > > > > > > > >> think
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > it mixes the transport
>>> layer
>>> > > the
>>> > > > > the
>>> > > > > > > > > request
>>> > > > > > > > > >> >> > content
>>> > > > > > > > > >> >> > > > in
>>> > > > > > > > > >> >> > > > > a
>>> > > > > > > > > >> >> > > > > > > > pretty
>>> > > > > > > > > >> >> > > > > > > > > >> ugly
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > way. Perhaps limiting
>>> > renewer
>>> > > to
>>> > > > > > > > non-owner
>>> > > > > > > > > >> is
>>> > > > > > > > > >> >> > > better.
>>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > *I feel this is a
>>> necessary
>>> > > > evil.
>>> > > > > > > While
>>> > > > > > > > > >> this will
>>> > > > > > > > > >> >> > > make
>>> > > > > > > > > >> >> > > > > the
>>> > > > > > > > > >> >> > > > > > > > kafka
>>> > > > > > > > > >> >> > > > > > > > > >> code
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > pretty straight forward ,
>>> > > > forcing
>>> > > > > > > > renewer
>>> > > > > > > > > >> to
>>> > > > > > > > > >> >> > > > non-owner
>>> > > > > > > > > >> >> > > > > > > pushes
>>> > > > > > > > > >> >> > > > > > > > > >> the code
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > ugliness to client and
>>> makes
>>> > > it
>>> > > > > even
>>> > > > > > > > > harder
>>> > > > > > > > > >> to
>>> > > > > > > > > >> >> > > > > > integrate.  *
>>> > > > > > > > > >> >> > > > > > > > > >> > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > As mentioned before, I
>>> don't
>>> > > think
>>> > > > > the
>>> > > > > > > > > >> "renewal by
>>> > > > > > > > > >> >> > > > other"
>>> > > > > > > > > >> >> > > > > > > > approach
>>> > > > > > > > > >> >> > > > > > > > > >> is
>>> > > > > > > > > >> >> > > > > > > > > >> > > > that ugly for the clients
>>> we
>>> > > > expect
>>> > > > > to
>>> > > > > > > use
>>> > > > > > > > > >> >> > delegation
>>> > > > > > > > > >> >> > > > > tokens
>>> > > > > > > > > >> >> > > > > > > > since
>>> > > > > > > > > >> >> > > > > > > > > >> > > > they will have an
>>> app-master
>>> > of
>>> > > > some
>>> > > > > > > sort
>>> > > > > > > > > who
>>> > > > > > > > > >> >> > > requested
>>> > > > > > > > > >> >> > > > > the
>>> > > > > > > > > >> >> > > > > > > > token
>>> > > > > > > > > >> >> > > > > > > > > to
>>> > > > > > > > > >> >> > > > > > > > > >> > > > begin with.
>>> > > > > > > > > >> >> > > > > > > > > >> > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > The response for my
>>> question
>>> > > on
>>> > > > > how
>>> > > > > > > > > multiple
>>> > > > > > > > > >> >> > > > identities
>>> > > > > > > > > >> >> > > > > > will
>>> > > > > > > > > >> >> > > > > > > > be
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > handled wasn't super
>>> clear
>>> > to
>>> > > > me -
>>> > > > > > > > AFAIK,
>>> > > > > > > > > >> we don't
>>> > > > > > > > > >> >> > > > > > > > authenticate
>>> > > > > > > > > >> >> > > > > > > > > >> each
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > request, we authenticate
>>> > > > > > connections.
>>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > *We authenticate
>>> > connections,
>>> > > > and
>>> > > > > > only
>>> > > > > > > > > when
>>> > > > > > > > > >> they
>>> > > > > > > > > >> >> > are
>>> > > > > > > > > >> >> > > > > being
>>> > > > > > > > > >> >> > > > > > > > > >> established.
>>> > > > > > > > > >> >> > > > > > > > > >> > > > Let
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > me try to phrase this as
>>> a
>>> > > > > question,
>>> > > > > > > in
>>> > > > > > > > > >> absence of
>>> > > > > > > > > >> >> > > > > > > delegation
>>> > > > > > > > > >> >> > > > > > > > > >> tokens if
>>> > > > > > > > > >> >> > > > > > > > > >> > > > we
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > had to support the use
>>> case
>>> > > > using
>>> > > > > > user
>>> > > > > > > > > >> TGT's how
>>> > > > > > > > > >> >> > > would
>>> > > > > > > > > >> >> > > > > we
>>> > > > > > > > > >> >> > > > > > do
>>> > > > > > > > > >> >> > > > > > > > it?
>>> > > > > > > > > >> >> > > > > > > > > >> My
>>> > > > > > > > > >> >> > > > > > > > > >> > > point
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > was it would be no
>>> different
>>> > > > with
>>> > > > > > > > > delegation
>>> > > > > > > > > >> >> > tokens.
>>> > > > > > > > > >> >> > > > The
>>> > > > > > > > > >> >> > > > > > use
>>> > > > > > > > > >> >> > > > > > > > > case
>>> > > > > > > > > >> >> > > > > > > > > >> you
>>> > > > > > > > > >> >> > > > > > > > > >> > > are
>>> > > > > > > > > >> >> > > > > > > > > >> > > > > describing seems more
>>> like
>>> > > > > > > > impersonation.*
>>> > > > > > > > > >> >> > > > > > > > > >> > > >
>>> > > > > > > > > >> >> > > > > > > > > >> > > > Yeah, I thought that one of
>>> > the
>>> > > > > things
>>> > > > > > > > that
>>> > > > > > > > > >> >> > del
>>>
>>
>>
>>
>> --
>>
>> Regards,
>> Ashish
>>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Jun Rao <ju...@confluent.io>]

Ashish,

Yes, I will send out a KIP invite for next week to discuss KIP-48 and other
remaining KIPs.

Thanks,

Jun

On Tue, Aug 23, 2016 at 1:22 PM, Ashish Singh <as...@cloudera.com> wrote:

> Thanks Harsha!
>
> Jun, can we add KIP-48 to next KIP hangout's agenda. Also, we did not
> actually make a call on when we should have next KIP call. As there are a
> few outstanding KIPs that could not be discussed this week, can we have a
> KIP hangout call next week?
>
> On Tue, Aug 23, 2016 at 1:10 PM, Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
>> Ashish,
>>         Yes we are working on it. Lets discuss in the next KIP meeting.
>> I'll join.
>> -Harsha
>>
>> On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh <as...@cloudera.com>
>> wrote:
>>
>> > Hello Harsha,
>> >
>> > Are you still working on this? Wondering if we can discuss this in next
>> KIP
>> > meeting, if you can join.
>> >
>> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha Chintalapani <ka...@harsha.io>
>> > wrote:
>> >
>> > > Hi Grant,
>> > >           We are working on it. Will add the details to KIP about the
>> > > request protocol.
>> > >
>> > > Thanks,
>> > > Harsha
>> > >
>> > > On Mon, Jul 18, 2016 at 6:50 AM Grant Henke <gh...@cloudera.com>
>> wrote:
>> > >
>> > > > Hi Parth,
>> > > >
>> > > > Are you still working on this? If you need any help please don't
>> > hesitate
>> > > > to ask.
>> > > >
>> > > > Thanks,
>> > > > Grant
>> > > >
>> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <ju...@confluent.io> wrote:
>> > > >
>> > > > > Parth,
>> > > > >
>> > > > > Thanks for the reply.
>> > > > >
>> > > > > It makes sense to only allow the renewal by users that
>> authenticated
>> > > > using
>> > > > > *non* delegation token mechanism. Then, should we make the
>> renewal a
>> > > > list?
>> > > > > For example, in the case of rest proxy, it will be useful for
>> every
>> > > > > instance of rest proxy to be able to renew the tokens.
>> > > > >
>> > > > > It would be clearer if we can document the request protocol like
>> > > > >
>> > > > >
>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > 4+-+Command+line+and+centralized+administrative+operations#KIP-4-
>> > > Commandlineandcentralizedadministrativeoperations-
>> > > CreateTopicsRequest(KAFKA-2945):(VotedandPlannedforin0.10.1.0)
>> > > > > .
>> > > > >
>> > > > > It would also be useful to document the client APIs.
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > Jun
>> > > > >
>> > > > > On Tue, Jun 28, 2016 at 2:55 PM, parth brahmbhatt <
>> > > > > brahmbhatt.parth@gmail.com> wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > I am suggesting that we will only allow the renewal by users
>> that
>> > > > > > authenticated using *non* delegation token mechanism. For
>> example,
>> > If
>> > > > > user
>> > > > > > Alice authenticated using kerberos and requested delegation
>> tokens,
>> > > > only
>> > > > > > user Alice authenticated via non delegation token mechanism can
>> > > renew.
>> > > > > > Clients that have  access to delegation tokens can not issue
>> > renewal
>> > > > > > request for renewing their own token and this is primarily
>> > important
>> > > to
>> > > > > > reduce the time window for which a compromised token will be
>> valid.
>> > > > > >
>> > > > > > To clarify, Yes any authenticated user can request delegation
>> > tokens
>> > > > but
>> > > > > > even here I would recommend to avoid creating a chain where a
>> > client
>> > > > > > authenticated via delegation token request for more delegation
>> > > tokens.
>> > > > > > Basically anyone can request delegation token, as long as they
>> > > > > authenticate
>> > > > > > via a non delegation token mechanism.
>> > > > > >
>> > > > > > Aren't classes listed here
>> > > > > > <
>> > > > > >
>> > > > >
>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > 48+Delegation+token+support+for+Kafka#KIP-48Delegationtokens
>> upportforKaf
>> > > ka-PublicInterfaces
>> > > > > > >
>> > > > > > sufficient?
>> > > > > >
>> > > > > > Thanks
>> > > > > > Parth
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao <ju...@confluent.io>
>> wrote:
>> > > > > >
>> > > > > > > Parth,
>> > > > > > >
>> > > > > > > Thanks for the reply. A couple of comments inline below.
>> > > > > > >
>> > > > > > > On Tue, Jun 21, 2016 at 10:36 AM, parth brahmbhatt <
>> > > > > > > brahmbhatt.parth@gmail.com> wrote:
>> > > > > > >
>> > > > > > > > 1. Who / how are tokens renewed? By original requester
>> only? or
>> > > > using
>> > > > > > > > Kerberos
>> > > > > > > > auth only?
>> > > > > > > > My recommendation is to do this only using Kerberos auth and
>> > only
>> > > > > threw
>> > > > > > > the
>> > > > > > > > renewer specified during the acquisition request.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > Hmm, not sure that I follow this. Are you saying that any
>> client
>> > > > > > > authenticated with the delegation token can renew, i.e. there
>> is
>> > no
>> > > > > > renewer
>> > > > > > > needed?
>> > > > > > >
>> > > > > > > Also, just to be clear, any authenticated client (either
>> through
>> > > SASL
>> > > > > or
>> > > > > > > SSL) can request a delegation token for the authenticated
>> user,
>> > > > right?
>> > > > > > >
>> > > > > > >
>> > > > > > > > 2. Are tokens stored on each broker or in ZK?
>> > > > > > > > My recommendation is still to store in ZK or not store them
>> at
>> > > all.
>> > > > > The
>> > > > > > > > whole controller based distribution is too much overhead
>> with
>> > not
>> > > > > much
>> > > > > > to
>> > > > > > > > achieve.
>> > > > > > > >
>> > > > > > > > 3. How are tokens invalidated / expired?
>> > > > > > > > Either by expiration time out or through an explicit
>> request to
>> > > > > > > invalidate.
>> > > > > > > >
>> > > > > > > > 4. Which encryption algorithm is used?
>> > > > > > > > SCRAM
>> > > > > > > >
>> > > > > > > > 5. What is the impersonation proposal (it wasn't in the KIP
>> but
>> > > was
>> > > > > > > > discussed
>> > > > > > > > in this thread)?
>> > > > > > > > There is no imperonation proposal. I tried and explained how
>> > its
>> > > a
>> > > > > > > > different problem and why its not really necessary to
>> discuss
>> > > that
>> > > > as
>> > > > > > > part
>> > > > > > > > of this KIP.  This KIP will not support any impersonation,
>> it
>> > > will
>> > > > > just
>> > > > > > > be
>> > > > > > > > another way to authenticate.
>> > > > > > > >
>> > > > > > > > 6. Do we need new ACLs, if so - for what actions?
>> > > > > > > > We do not need new ACLs.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > Could we document the format of the new request/response and
>> > their
>> > > > > > > associated Resource and Operation for ACL?
>> > > > > > >
>> > > > > > >
>> > > > > > > > 7. How would the delegation token be configured in the
>> client?
>> > > > > > > > Should be through config. I wasn't planning on supporting
>> JAAS
>> > > for
>> > > > > > > tokens.
>> > > > > > > > I don't believe hadoop does this either.
>> > > > > > > >
>> > > > > > > > Thanks
>> > > > > > > > Parth
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <ju...@confluent.io>
>> > > wrote:
>> > > > > > > >
>> > > > > > > > > Harsha,
>> > > > > > > > >
>> > > > > > > > > Another question.
>> > > > > > > > >
>> > > > > > > > > 9. How would the delegation token be configured in the
>> > client?
>> > > > The
>> > > > > > > > standard
>> > > > > > > > > way is to do this through JAAS. However, we will need to
>> > think
>> > > > > > through
>> > > > > > > if
>> > > > > > > > > this is convenient in a shared environment. For example,
>> > when a
>> > > > new
>> > > > > > > task
>> > > > > > > > is
>> > > > > > > > > added to a Storm worker node, do we need to dynamically
>> add a
>> > > new
>> > > > > > > section
>> > > > > > > > > in the JAAS file? It may be more convenient if we can
>> pass in
>> > > the
>> > > > > > token
>> > > > > > > > > through the config directly w/o going through JAAS.
>> > > > > > > > >
>> > > > > > > > > Are you or Parth still actively working on this KIP?
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > >
>> > > > > > > > > Jun
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <
>> jun@confluent.io>
>> > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Just to add on that list.
>> > > > > > > > > >
>> > > > > > > > > > 2. It would be good to document the format of the data
>> > stored
>> > > > in
>> > > > > > ZK.
>> > > > > > > > > > 7. Earlier, there was a discussion on whether the tokens
>> > > should
>> > > > > be
>> > > > > > > > > > propagated through ZK like config/acl/quota, or through
>> the
>> > > > > > > controller.
>> > > > > > > > > > Currently, the controller is only designed for
>> propagating
>> > > > topic
>> > > > > > > > > metadata,
>> > > > > > > > > > but not other data.
>> > > > > > > > > > 8. Should we use SCRAM to send the token instead of
>> > > DIGEST-MD5
>> > > > > > since
>> > > > > > > > it's
>> > > > > > > > > > deprecated?
>> > > > > > > > > >
>> > > > > > > > > > Also, the images in the wiki seem broken.
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > >
>> > > > > > > > > > Jun
>> > > > > > > > > >
>> > > > > > > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <
>> > > > > gwen@confluent.io>
>> > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > >> From what I can see, remaining questions are:
>> > > > > > > > > >>
>> > > > > > > > > >> 1. Who / how are tokens renewed? By original requester
>> > only?
>> > > > or
>> > > > > > > using
>> > > > > > > > > >> Kerberos auth only?
>> > > > > > > > > >> 2. Are tokens stored on each broker or in ZK?
>> > > > > > > > > >> 3. How are tokens invalidated / expired?
>> > > > > > > > > >> 4. Which encryption algorithm is used?
>> > > > > > > > > >> 5. What is the impersonation proposal (it wasn't in the
>> > KIP
>> > > > but
>> > > > > > was
>> > > > > > > > > >> discussed in this thread)?
>> > > > > > > > > >> 6. Do we need new ACLs, if so - for what actions?
>> > > > > > > > > >>
>> > > > > > > > > >> Gwen
>> > > > > > > > > >>
>> > > > > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <
>> kafka@harsha.io>
>> > > > wrote:
>> > > > > > > > > >> > Jun & Ismael,
>> > > > > > > > > >> >                          Unfortunately I couldn't
>> attend
>> > > the
>> > > > > KIP
>> > > > > > > > > meeting
>> > > > > > > > > >> >                          when delegation tokens
>> > discussed.
>> > > > > > > > Appreciate
>> > > > > > > > > if
>> > > > > > > > > >> >                          you can update the thread if
>> > you
>> > > > have
>> > > > > > any
>> > > > > > > > > >> >                          further questions.
>> > > > > > > > > >> > Thanks,
>> > > > > > > > > >> > Harsha
>> > > > > > > > > >> >
>> > > > > > > > > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
>> > > > > > > > > >> >> It seems that the links to images in the KIP are
>> > broken.
>> > > > > > > > > >> >>
>> > > > > > > > > >> >> Liquan
>> > > > > > > > > >> >>
>> > > > > > > > > >> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
>> > > > > > > > > >> >> brahmbhatt.parth@gmail.com> wrote:
>> > > > > > > > > >> >>
>> > > > > > > > > >> >> > 110. What does getDelegationTokenAs mean?
>> > > > > > > > > >> >> > In the current proposal we only allow a user to
>> get
>> > > > > > delegation
>> > > > > > > > > token
>> > > > > > > > > >> for
>> > > > > > > > > >> >> > the identity that it authenticated as using
>> another
>> > > > > > mechanism,
>> > > > > > > > i.e.
>> > > > > > > > > >> A user
>> > > > > > > > > >> >> > that authenticate using a keytab for principal
>> > > > > > > user1@EXAMPLE.COM
>> > > > > > > > > >> will get
>> > > > > > > > > >> >> > delegation tokens for that user only. In future I
>> > think
>> > > > we
>> > > > > > will
>> > > > > > > > > have
>> > > > > > > > > >> to
>> > > > > > > > > >> >> > extend support such that we allow some set of
>> users (
>> > > > > > > > > >> >> > kafka-rest-user@EXAMPLE.COM,
>> > storm-nimbus@EXAMPLE.COM)
>> > > > to
>> > > > > > > > acquire
>> > > > > > > > > >> >> > delegation tokens on behalf of other users whose
>> > > identity
>> > > > > > they
>> > > > > > > > have
>> > > > > > > > > >> >> > verified independently.  Kafka brokers will have
>> ACLs
>> > > to
>> > > > > > > control
>> > > > > > > > > >> which
>> > > > > > > > > >> >> > users are allowed to impersonate other users and
>> get
>> > > > tokens
>> > > > > > on
>> > > > > > > > > >> behalf of
>> > > > > > > > > >> >> > them. Overall Impersonation is a whole different
>> > > problem
>> > > > in
>> > > > > > my
>> > > > > > > > > >> opinion and
>> > > > > > > > > >> >> > I think we can tackle it in separate KIP.
>> > > > > > > > > >> >> >
>> > > > > > > > > >> >> > 111. What's the typical rate of getting and
>> renewing
>> > > > > > delegation
>> > > > > > > > > >> tokens?
>> > > > > > > > > >> >> > Typically this should be very very low, 1 request
>> per
>> > > > > minute
>> > > > > > > is a
>> > > > > > > > > >> >> > relatively high estimate. However it depends on
>> the
>> > > token
>> > > > > > > > > >> expiration. I am
>> > > > > > > > > >> >> > less worried about the extra load it puts on
>> > controller
>> > > > vs
>> > > > > > the
>> > > > > > > > > added
>> > > > > > > > > >> >> > complexity and the value it offers.
>> > > > > > > > > >> >> >
>> > > > > > > > > >> >> > Thanks
>> > > > > > > > > >> >> > Parth
>> > > > > > > > > >> >> >
>> > > > > > > > > >> >> >
>> > > > > > > > > >> >> >
>> > > > > > > > > >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <
>> > > > > > > ismael@juma.me.uk>
>> > > > > > > > > >> wrote:
>> > > > > > > > > >> >> >
>> > > > > > > > > >> >> > > Thanks Rajini. It would probably require a
>> separate
>> > > KIP
>> > > > > as
>> > > > > > it
>> > > > > > > > > will
>> > > > > > > > > >> >> > > introduce user visible changes. We could also
>> > update
>> > > > > KIP-48
>> > > > > > > to
>> > > > > > > > > >> have this
>> > > > > > > > > >> >> > > information, but it seems cleaner to do it
>> > > separately.
>> > > > We
>> > > > > > can
>> > > > > > > > > >> discuss
>> > > > > > > > > >> >> > that
>> > > > > > > > > >> >> > > in the KIP call today.
>> > > > > > > > > >> >> > >
>> > > > > > > > > >> >> > > Ismael
>> > > > > > > > > >> >> > >
>> > > > > > > > > >> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram
>> <
>> > > > > > > > > >> >> > > rajinisivaram@googlemail.com> wrote:
>> > > > > > > > > >> >> > >
>> > > > > > > > > >> >> > > > Ismael,
>> > > > > > > > > >> >> > > >
>> > > > > > > > > >> >> > > > I have created a JIRA (
>> > > > > > > > > >> >> > https://issues.apache.org/jira/browse/KAFKA-3751)
>> > > > > > > > > >> >> > > > for adding SCRAM as a SASL mechanism. Would
>> that
>> > > need
>> > > > > > > another
>> > > > > > > > > >> KIP? If
>> > > > > > > > > >> >> > > > KIP-48 will use this mechanism, can this just
>> be
>> > a
>> > > > JIRA
>> > > > > > > that
>> > > > > > > > > gets
>> > > > > > > > > >> >> > > reviewed
>> > > > > > > > > >> >> > > > when the PR is ready?
>> > > > > > > > > >> >> > > >
>> > > > > > > > > >> >> > > > Thank you,
>> > > > > > > > > >> >> > > >
>> > > > > > > > > >> >> > > > Rajini
>> > > > > > > > > >> >> > > >
>> > > > > > > > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <
>> > > > > > > > > ismael@juma.me.uk>
>> > > > > > > > > >> >> > wrote:
>> > > > > > > > > >> >> > > >
>> > > > > > > > > >> >> > > > > Thanks Rajini, SCRAM seems like a good
>> > candidate.
>> > > > > > > > > >> >> > > > >
>> > > > > > > > > >> >> > > > > Gwen had independently mentioned this as a
>> SASL
>> > > > > > mechanism
>> > > > > > > > > that
>> > > > > > > > > >> might
>> > > > > > > > > >> >> > be
>> > > > > > > > > >> >> > > > > useful for Kafka and I have been meaning to
>> > check
>> > > > it
>> > > > > in
>> > > > > > > > more
>> > > > > > > > > >> detail.
>> > > > > > > > > >> >> > > Good
>> > > > > > > > > >> >> > > > > to know that you are willing to contribute
>> an
>> > > > > > > > implementation.
>> > > > > > > > > >> Maybe
>> > > > > > > > > >> >> > we
>> > > > > > > > > >> >> > > > > should file a separate JIRA for this?
>> > > > > > > > > >> >> > > > >
>> > > > > > > > > >> >> > > > > Ismael
>> > > > > > > > > >> >> > > > >
>> > > > > > > > > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini
>> > Sivaram <
>> > > > > > > > > >> >> > > > > rajinisivaram@googlemail.com> wrote:
>> > > > > > > > > >> >> > > > >
>> > > > > > > > > >> >> > > > > > SCRAM (Salted Challenge Response
>> > Authentication
>> > > > > > > > Mechanism)
>> > > > > > > > > >> is a
>> > > > > > > > > >> >> > > better
>> > > > > > > > > >> >> > > > > > mechanism than Digest-MD5. Java doesn't
>> come
>> > > > with a
>> > > > > > > > > built-in
>> > > > > > > > > >> SCRAM
>> > > > > > > > > >> >> > > > > > SaslServer or SaslClient, but I will be
>> happy
>> > > to
>> > > > > add
>> > > > > > > > > support
>> > > > > > > > > >> in
>> > > > > > > > > >> >> > Kafka
>> > > > > > > > > >> >> > > > > since
>> > > > > > > > > >> >> > > > > > it would be a useful mechanism to support
>> > > anyway.
>> > > > > > > > > >> >> > > > > > https://tools.ietf.org/html/rfc7677
>> > describes
>> > > > the
>> > > > > > > > protocol
>> > > > > > > > > >> for
>> > > > > > > > > >> >> > > > > > SCRAM-SHA-256.
>> > > > > > > > > >> >> > > > > >
>> > > > > > > > > >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <
>> > > > > > > > jun@confluent.io
>> > > > > > > > > >
>> > > > > > > > > >> wrote:
>> > > > > > > > > >> >> > > > > >
>> > > > > > > > > >> >> > > > > > > Parth,
>> > > > > > > > > >> >> > > > > > >
>> > > > > > > > > >> >> > > > > > > Thanks for the explanation. A couple of
>> > more
>> > > > > > > questions.
>> > > > > > > > > >> >> > > > > > >
>> > > > > > > > > >> >> > > > > > > 110. What does getDelegationTokenAs
>> mean?
>> > > > > > > > > >> >> > > > > > >
>> > > > > > > > > >> >> > > > > > > 111. What's the typical rate of getting
>> and
>> > > > > > renewing
>> > > > > > > > > >> delegation
>> > > > > > > > > >> >> > > > tokens?
>> > > > > > > > > >> >> > > > > > > That may have an impact on whether they
>> > > should
>> > > > be
>> > > > > > > > > directed
>> > > > > > > > > >> to the
>> > > > > > > > > >> >> > > > > > > controller.
>> > > > > > > > > >> >> > > > > > >
>> > > > > > > > > >> >> > > > > > > Jun
>> > > > > > > > > >> >> > > > > > >
>> > > > > > > > > >> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth
>> > > > > brahmbhatt <
>> > > > > > > > > >> >> > > > > > > brahmbhatt.parth@gmail.com> wrote:
>> > > > > > > > > >> >> > > > > > >
>> > > > > > > > > >> >> > > > > > > > Hi Jun,
>> > > > > > > > > >> >> > > > > > > >
>> > > > > > > > > >> >> > > > > > > > Thanks for reviewing.
>> > > > > > > > > >> >> > > > > > > >
>> > > > > > > > > >> >> > > > > > > > * We could add a Cluster action to add
>> > acls
>> > > > on
>> > > > > > who
>> > > > > > > > can
>> > > > > > > > > >> request
>> > > > > > > > > >> >> > > > > > delegation
>> > > > > > > > > >> >> > > > > > > > tokens. I don't see the use case for
>> that
>> > > yet
>> > > > > but
>> > > > > > > > down
>> > > > > > > > > >> the line
>> > > > > > > > > >> >> > > > when
>> > > > > > > > > >> >> > > > > we
>> > > > > > > > > >> >> > > > > > > > start supporting getDelegationTokenAs
>> it
>> > > will
>> > > > > be
>> > > > > > > > > >> necessary.
>> > > > > > > > > >> >> > > > > > > > * Yes we recommend tokens to be only
>> > > > > > > used/distributed
>> > > > > > > > > >> over
>> > > > > > > > > >> >> > secure
>> > > > > > > > > >> >> > > > > > > channels.
>> > > > > > > > > >> >> > > > > > > > * Depending on what design we end up
>> > > choosing
>> > > > > > > > > >> Invalidation will
>> > > > > > > > > >> >> > > be
>> > > > > > > > > >> >> > > > > > > > responsibility of every broker or
>> > > controller.
>> > > > > > > > > >> >> > > > > > > > * I am not sure if I documented
>> somewhere
>> > > > that
>> > > > > > > > > >> invalidation
>> > > > > > > > > >> >> > will
>> > > > > > > > > >> >> > > > > > directly
>> > > > > > > > > >> >> > > > > > > > go through zookeeper but that is not
>> the
>> > > > > intent.
>> > > > > > > > > >> Invalidation
>> > > > > > > > > >> >> > > will
>> > > > > > > > > >> >> > > > > > either
>> > > > > > > > > >> >> > > > > > > > be request based or due to
>> expiration. No
>> > > > > direct
>> > > > > > > > > >> zookeeper
>> > > > > > > > > >> >> > > > > interaction
>> > > > > > > > > >> >> > > > > > > from
>> > > > > > > > > >> >> > > > > > > > any client.
>> > > > > > > > > >> >> > > > > > > > * "Broker also stores the
>> DelegationToken
>> > > > > without
>> > > > > > > the
>> > > > > > > > > >> hmac in
>> > > > > > > > > >> >> > the
>> > > > > > > > > >> >> > > > > > > > zookeeper." : Sorry about the
>> confusion.
>> > > The
>> > > > > sole
>> > > > > > > > > >> purpose of
>> > > > > > > > > >> >> > > > > zookeeper
>> > > > > > > > > >> >> > > > > > in
>> > > > > > > > > >> >> > > > > > > > this design is as distribution channel
>> > for
>> > > > > tokens
>> > > > > > > > > >> between all
>> > > > > > > > > >> >> > > > brokers
>> > > > > > > > > >> >> > > > > > > and a
>> > > > > > > > > >> >> > > > > > > > layer that ensures only tokens that
>> were
>> > > > > > generated
>> > > > > > > by
>> > > > > > > > > >> making a
>> > > > > > > > > >> >> > > > > request
>> > > > > > > > > >> >> > > > > > > to a
>> > > > > > > > > >> >> > > > > > > > broker will be accepted (more on this
>> in
>> > > > second
>> > > > > > > > > >> paragraph). The
>> > > > > > > > > >> >> > > > token
>> > > > > > > > > >> >> > > > > > > > consists of few elements (owner,
>> renewer,
>> > > > uuid
>> > > > > ,
>> > > > > > > > > >> expiration,
>> > > > > > > > > >> >> > > hmac)
>> > > > > > > > > >> >> > > > ,
>> > > > > > > > > >> >> > > > > > one
>> > > > > > > > > >> >> > > > > > > of
>> > > > > > > > > >> >> > > > > > > > which is the finally generated hmac
>> but
>> > > hmac
>> > > > it
>> > > > > > > self
>> > > > > > > > is
>> > > > > > > > > >> >> > derivable
>> > > > > > > > > >> >> > > > if
>> > > > > > > > > >> >> > > > > > you
>> > > > > > > > > >> >> > > > > > > > have all the other elements of the
>> token
>> > +
>> > > > > secret
>> > > > > > > key
>> > > > > > > > > to
>> > > > > > > > > >> >> > generate
>> > > > > > > > > >> >> > > > > hmac.
>> > > > > > > > > >> >> > > > > > > > Given zookeeper does not provide SSL
>> > > support
>> > > > we
>> > > > > > do
>> > > > > > > > not
>> > > > > > > > > >> want the
>> > > > > > > > > >> >> > > > > entire
>> > > > > > > > > >> >> > > > > > > > token to be wire transferred to
>> zookeeper
>> > > as
>> > > > > that
>> > > > > > > > will
>> > > > > > > > > >> be an
>> > > > > > > > > >> >> > > > insecure
>> > > > > > > > > >> >> > > > > > > wire
>> > > > > > > > > >> >> > > > > > > > transfer. Instead we only store all
>> the
>> > > other
>> > > > > > > > elements
>> > > > > > > > > >> of a
>> > > > > > > > > >> >> > > > > delegation
>> > > > > > > > > >> >> > > > > > > > tokens. Brokers can read these
>> elements
>> > and
>> > > > > > because
>> > > > > > > > > they
>> > > > > > > > > >> also
>> > > > > > > > > >> >> > > have
>> > > > > > > > > >> >> > > > > > access
>> > > > > > > > > >> >> > > > > > > > to secret key they will be able to
>> > generate
>> > > > > hmac
>> > > > > > on
>> > > > > > > > > >> their end.
>> > > > > > > > > >> >> > > > > > > >
>> > > > > > > > > >> >> > > > > > > > One of the alternative proposed is to
>> > avoid
>> > > > > > > zookeeper
>> > > > > > > > > >> >> > > altogether. A
>> > > > > > > > > >> >> > > > > > > Client
>> > > > > > > > > >> >> > > > > > > > will call broker with required
>> > information
>> > > > > > (owner,
>> > > > > > > > > >> renwer,
>> > > > > > > > > >> >> > > > > expiration)
>> > > > > > > > > >> >> > > > > > > and
>> > > > > > > > > >> >> > > > > > > > get back (signed hmac, uuid). Broker
>> > won't
>> > > > > store
>> > > > > > > this
>> > > > > > > > > in
>> > > > > > > > > >> >> > > zookeeper.
>> > > > > > > > > >> >> > > > > > From
>> > > > > > > > > >> >> > > > > > > > this point a client can contact any
>> > broker
>> > > > with
>> > > > > > all
>> > > > > > > > the
>> > > > > > > > > >> >> > > delegation
>> > > > > > > > > >> >> > > > > > token
>> > > > > > > > > >> >> > > > > > > > info (owner, rewner, expiration, hmac,
>> > > uuid)
>> > > > > the
>> > > > > > > > borker
>> > > > > > > > > >> will
>> > > > > > > > > >> >> > > > > regenerate
>> > > > > > > > > >> >> > > > > > > the
>> > > > > > > > > >> >> > > > > > > > hmac and as long as it matches with
>> hmac
>> > > > > > presented
>> > > > > > > by
>> > > > > > > > > >> client ,
>> > > > > > > > > >> >> > > > broker
>> > > > > > > > > >> >> > > > > > > will
>> > > > > > > > > >> >> > > > > > > > allow the request to authenticate.
>> Only
>> > > > > problem
>> > > > > > > with
>> > > > > > > > > >> this
>> > > > > > > > > >> >> > > approach
>> > > > > > > > > >> >> > > > > is
>> > > > > > > > > >> >> > > > > > if
>> > > > > > > > > >> >> > > > > > > > the secret key is compromised any
>> client
>> > > can
>> > > > > now
>> > > > > > > > > generate
>> > > > > > > > > >> >> > random
>> > > > > > > > > >> >> > > > > tokens
>> > > > > > > > > >> >> > > > > > > and
>> > > > > > > > > >> >> > > > > > > > they will still be able to
>> authenticate
>> > as
>> > > > any
>> > > > > > user
>> > > > > > > > > they
>> > > > > > > > > >> like.
>> > > > > > > > > >> >> > > with
>> > > > > > > > > >> >> > > > > > > > zookeeper we guarantee that only
>> tokens
>> > > > > acquired
>> > > > > > > via
>> > > > > > > > a
>> > > > > > > > > >> broker
>> > > > > > > > > >> >> > > > (using
>> > > > > > > > > >> >> > > > > > some
>> > > > > > > > > >> >> > > > > > > > auth scheme other than delegation
>> token)
>> > > will
>> > > > > be
>> > > > > > > > > >> accepted. We
>> > > > > > > > > >> >> > > need
>> > > > > > > > > >> >> > > > to
>> > > > > > > > > >> >> > > > > > > > discuss which proposal makes more
>> sense
>> > and
>> > > > we
>> > > > > > can
>> > > > > > > go
>> > > > > > > > > >> over it
>> > > > > > > > > >> >> > in
>> > > > > > > > > >> >> > > > > > > tomorrow's
>> > > > > > > > > >> >> > > > > > > > meeting.
>> > > > > > > > > >> >> > > > > > > >
>> > > > > > > > > >> >> > > > > > > > Also, can you forward the invite to
>> me?
>> > > > > > > > > >> >> > > > > > > >
>> > > > > > > > > >> >> > > > > > > > Thanks
>> > > > > > > > > >> >> > > > > > > > Parth
>> > > > > > > > > >> >> > > > > > > >
>> > > > > > > > > >> >> > > > > > > >
>> > > > > > > > > >> >> > > > > > > >
>> > > > > > > > > >> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun
>> > Rao <
>> > > > > > > > > >> jun@confluent.io>
>> > > > > > > > > >> >> > > > wrote:
>> > > > > > > > > >> >> > > > > > > >
>> > > > > > > > > >> >> > > > > > > > > Thanks for the KIP. A few comments.
>> > > > > > > > > >> >> > > > > > > > >
>> > > > > > > > > >> >> > > > > > > > > 100. This potentially can be useful
>> for
>> > > > Kafka
>> > > > > > > > Connect
>> > > > > > > > > >> and
>> > > > > > > > > >> >> > Kafka
>> > > > > > > > > >> >> > > > > rest
>> > > > > > > > > >> >> > > > > > > > proxy
>> > > > > > > > > >> >> > > > > > > > > where a worker agent will need to
>> run a
>> > > > task
>> > > > > on
>> > > > > > > > > behalf
>> > > > > > > > > >> of a
>> > > > > > > > > >> >> > > > client.
>> > > > > > > > > >> >> > > > > > We
>> > > > > > > > > >> >> > > > > > > > will
>> > > > > > > > > >> >> > > > > > > > > likely need to change how those
>> > services
>> > > > use
>> > > > > > > Kafka
>> > > > > > > > > >> clients
>> > > > > > > > > >> >> > > > > > > > > (producer/consumer). Instead of a
>> > shared
>> > > > > client
>> > > > > > > per
>> > > > > > > > > >> worker,
>> > > > > > > > > >> >> > we
>> > > > > > > > > >> >> > > > will
>> > > > > > > > > >> >> > > > > > > need
>> > > > > > > > > >> >> > > > > > > > a
>> > > > > > > > > >> >> > > > > > > > > client per user task since the
>> > > > authentication
>> > > > > > > > happens
>> > > > > > > > > >> at the
>> > > > > > > > > >> >> > > > > > connection
>> > > > > > > > > >> >> > > > > > > > > level. For Kafka Connect, the
>> renewer
>> > > will
>> > > > be
>> > > > > > the
>> > > > > > > > > >> workers.
>> > > > > > > > > >> >> > So,
>> > > > > > > > > >> >> > > we
>> > > > > > > > > >> >> > > > > > > > probably
>> > > > > > > > > >> >> > > > > > > > > need to allow multiple renewers. For
>> > > Kafka
>> > > > > rest
>> > > > > > > > > proxy,
>> > > > > > > > > >> the
>> > > > > > > > > >> >> > > > renewer
>> > > > > > > > > >> >> > > > > > can
>> > > > > > > > > >> >> > > > > > > > > probably just be the creator of the
>> > > token.
>> > > > > > > > > >> >> > > > > > > > >
>> > > > > > > > > >> >> > > > > > > > > 101. Do we need new acl on who can
>> > > request
>> > > > > > > > delegation
>> > > > > > > > > >> tokens?
>> > > > > > > > > >> >> > > > > > > > >
>> > > > > > > > > >> >> > > > > > > > > 102. Do we recommend people to send
>> > > > > delegation
>> > > > > > > > tokens
>> > > > > > > > > >> in an
>> > > > > > > > > >> >> > > > > encrypted
>> > > > > > > > > >> >> > > > > > > > > channel?
>> > > > > > > > > >> >> > > > > > > > >
>> > > > > > > > > >> >> > > > > > > > > 103. Who is responsible for expiring
>> > > > tokens,
>> > > > > > > every
>> > > > > > > > > >> broker?
>> > > > > > > > > >> >> > > > > > > > >
>> > > > > > > > > >> >> > > > > > > > > 104. For invalidating tokens, would
>> it
>> > be
>> > > > > > better
>> > > > > > > to
>> > > > > > > > > do
>> > > > > > > > > >> it in
>> > > > > > > > > >> >> > a
>> > > > > > > > > >> >> > > > > > request
>> > > > > > > > > >> >> > > > > > > > > instead of going to ZK directly?
>> > > > > > > > > >> >> > > > > > > > >
>> > > > > > > > > >> >> > > > > > > > > 105. The terminology of client in
>> the
>> > > wiki
>> > > > > > > > sometimes
>> > > > > > > > > >> refers
>> > > > > > > > > >> >> > to
>> > > > > > > > > >> >> > > > the
>> > > > > > > > > >> >> > > > > > end
>> > > > > > > > > >> >> > > > > > > > > client and some other times refers
>> to
>> > the
>> > > > > > client
>> > > > > > > > > using
>> > > > > > > > > >> the
>> > > > > > > > > >> >> > > > > delegation
>> > > > > > > > > >> >> > > > > > > > > tokens. It would be useful to
>> > distinguish
>> > > > > > between
>> > > > > > > > the
>> > > > > > > > > >> two.
>> > > > > > > > > >> >> > > > > > > > >
>> > > > > > > > > >> >> > > > > > > > > 106. Could you explain the sentence
>> > > "Broker
>> > > > > > also
>> > > > > > > > > >> stores the
>> > > > > > > > > >> >> > > > > > > > DelegationToken
>> > > > > > > > > >> >> > > > > > > > > without the hmac in the zookeeper."
>> a
>> > bit
>> > > > > > more? I
>> > > > > > > > > >> thought the
>> > > > > > > > > >> >> > > > > > > delegation
>> > > > > > > > > >> >> > > > > > > > > token is the hmac.
>> > > > > > > > > >> >> > > > > > > > >
>> > > > > > > > > >> >> > > > > > > > > Thanks,
>> > > > > > > > > >> >> > > > > > > > >
>> > > > > > > > > >> >> > > > > > > > > Jun
>> > > > > > > > > >> >> > > > > > > > >
>> > > > > > > > > >> >> > > > > > > > >
>> > > > > > > > > >> >> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun
>> > Rao
>> > > <
>> > > > > > > > > >> jun@confluent.io>
>> > > > > > > > > >> >> > > > wrote:
>> > > > > > > > > >> >> > > > > > > > >
>> > > > > > > > > >> >> > > > > > > > > > Hi, Harsha,
>> > > > > > > > > >> >> > > > > > > > > >
>> > > > > > > > > >> >> > > > > > > > > > Just sent out a KIP meeting
>> invite.
>> > We
>> > > > can
>> > > > > > > > discuss
>> > > > > > > > > >> this in
>> > > > > > > > > >> >> > > the
>> > > > > > > > > >> >> > > > > > > meeting
>> > > > > > > > > >> >> > > > > > > > > > tomorrow.
>> > > > > > > > > >> >> > > > > > > > > >
>> > > > > > > > > >> >> > > > > > > > > > Thanks,
>> > > > > > > > > >> >> > > > > > > > > >
>> > > > > > > > > >> >> > > > > > > > > > Jun
>> > > > > > > > > >> >> > > > > > > > > >
>> > > > > > > > > >> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM,
>> > > Harsha <
>> > > > > > > > > >> kafka@harsha.io>
>> > > > > > > > > >> >> > > > wrote:
>> > > > > > > > > >> >> > > > > > > > > >
>> > > > > > > > > >> >> > > > > > > > > >> Hi All,
>> > > > > > > > > >> >> > > > > > > > > >>            Can we have a KIP
>> meeting
>> > > > > around
>> > > > > > > > this.
>> > > > > > > > > >> The KIP
>> > > > > > > > > >> >> > is
>> > > > > > > > > >> >> > > > up
>> > > > > > > > > >> >> > > > > > for
>> > > > > > > > > >> >> > > > > > > > > >>            sometime and if there
>> are
>> > > any
>> > > > > > > > questions
>> > > > > > > > > >> lets
>> > > > > > > > > >> >> > > > quickly
>> > > > > > > > > >> >> > > > > > hash
>> > > > > > > > > >> >> > > > > > > > out
>> > > > > > > > > >> >> > > > > > > > > >>            details.
>> > > > > > > > > >> >> > > > > > > > > >>
>> > > > > > > > > >> >> > > > > > > > > >> Thanks,
>> > > > > > > > > >> >> > > > > > > > > >> Harsha
>> > > > > > > > > >> >> > > > > > > > > >>
>> > > > > > > > > >> >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40
>> AM,
>> > > parth
>> > > > > > > > > brahmbhatt
>> > > > > > > > > >> wrote:
>> > > > > > > > > >> >> > > > > > > > > >> > That is what the hadoop echo
>> > system
>> > > > uses
>> > > > > > so
>> > > > > > > no
>> > > > > > > > > >> good
>> > > > > > > > > >> >> > reason
>> > > > > > > > > >> >> > > > > > really.
>> > > > > > > > > >> >> > > > > > > > We
>> > > > > > > > > >> >> > > > > > > > > >> > could
>> > > > > > > > > >> >> > > > > > > > > >> > change it to whatever is the
>> > newest
>> > > > > > > > recommended
>> > > > > > > > > >> standard
>> > > > > > > > > >> >> > > is.
>> > > > > > > > > >> >> > > > > > > > > >> >
>> > > > > > > > > >> >> > > > > > > > > >> > Thanks
>> > > > > > > > > >> >> > > > > > > > > >> > Parth
>> > > > > > > > > >> >> > > > > > > > > >> >
>> > > > > > > > > >> >> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33
>> AM,
>> > > > Ismael
>> > > > > > > Juma <
>> > > > > > > > > >> >> > > > > ismael@juma.me.uk
>> > > > > > > > > >> >> > > > > > >
>> > > > > > > > > >> >> > > > > > > > > wrote:
>> > > > > > > > > >> >> > > > > > > > > >> >
>> > > > > > > > > >> >> > > > > > > > > >> > > Hi Parth,
>> > > > > > > > > >> >> > > > > > > > > >> > >
>> > > > > > > > > >> >> > > > > > > > > >> > > Thanks for the KIP. I only
>> > started
>> > > > > > > reviewing
>> > > > > > > > > >> this and
>> > > > > > > > > >> >> > > may
>> > > > > > > > > >> >> > > > > have
>> > > > > > > > > >> >> > > > > > > > > >> additional
>> > > > > > > > > >> >> > > > > > > > > >> > > questions later. The
>> immediate
>> > > > > question
>> > > > > > > that
>> > > > > > > > > >> came to
>> > > > > > > > > >> >> > > mind
>> > > > > > > > > >> >> > > > is
>> > > > > > > > > >> >> > > > > > our
>> > > > > > > > > >> >> > > > > > > > > >> choice of
>> > > > > > > > > >> >> > > > > > > > > >> > > "DIGEST-MD5" even though it's
>> > > marked
>> > > > > as
>> > > > > > > > > >> OBSOLETE in
>> > > > > > > > > >> >> > the
>> > > > > > > > > >> >> > > > IANA
>> > > > > > > > > >> >> > > > > > > > > Registry
>> > > > > > > > > >> >> > > > > > > > > >> of
>> > > > > > > > > >> >> > > > > > > > > >> > > SASL mechanisms and the
>> original
>> > > RFC
>> > > > > > > (2831)
>> > > > > > > > > has
>> > > > > > > > > >> been
>> > > > > > > > > >> >> > > moved
>> > > > > > > > > >> >> > > > > to
>> > > > > > > > > >> >> > > > > > > > > Historic
>> > > > > > > > > >> >> > > > > > > > > >> > > status:
>> > > > > > > > > >> >> > > > > > > > > >> > >
>> > > > > > > > > >> >> > > > > > > > > >> > > https://tools.ietf.org/html/
>> > > rfc6331
>> > > > > > > > > >> >> > > > > > > > > >> > >
>> > > > > > > > > >> >> > > > > > > > >
>> > > > > > > > > >> >> > > > > >
>> > > > > > > > > >> >> > >
>> > > > > > > > > >>
>> > > > > > >
>> > > > http://www.iana.org/assignments/sasl-mechanisms/sasl-
>> mechanisms.xhtml
>> > > > > > > > > >> >> > > > > > > > > >> > >
>> > > > > > > > > >> >> > > > > > > > > >> > > What is the reasoning behind
>> > that
>> > > > > > choice?
>> > > > > > > > > >> >> > > > > > > > > >> > >
>> > > > > > > > > >> >> > > > > > > > > >> > > Thanks,
>> > > > > > > > > >> >> > > > > > > > > >> > > Ismael
>> > > > > > > > > >> >> > > > > > > > > >> > >
>> > > > > > > > > >> >> > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29
>> > PM,
>> > > > Gwen
>> > > > > > > > > Shapira <
>> > > > > > > > > >> >> > > > > > > gwen@confluent.io
>> > > > > > > > > >> >> > > > > > > > >
>> > > > > > > > > >> >> > > > > > > > > >> wrote:
>> > > > > > > > > >> >> > > > > > > > > >> > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > Also comments inline :)
>> > > > > > > > > >> >> > > > > > > > > >> > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > > * I want to emphasize
>> that
>> > > even
>> > > > > > though
>> > > > > > > > > >> delegation
>> > > > > > > > > >> >> > > > tokens
>> > > > > > > > > >> >> > > > > > > are a
>> > > > > > > > > >> >> > > > > > > > > >> Hadoop
>> > > > > > > > > >> >> > > > > > > > > >> > > > > innovation, I feel very
>> > > strongly
>> > > > > > about
>> > > > > > > > not
>> > > > > > > > > >> adding
>> > > > > > > > > >> >> > > > > > dependency
>> > > > > > > > > >> >> > > > > > > > on
>> > > > > > > > > >> >> > > > > > > > > >> Hadoop
>> > > > > > > > > >> >> > > > > > > > > >> > > > > when implementing
>> delegation
>> > > > > tokens
>> > > > > > > for
>> > > > > > > > > >> Kafka. The
>> > > > > > > > > >> >> > > KIP
>> > > > > > > > > >> >> > > > > > > doesn't
>> > > > > > > > > >> >> > > > > > > > > >> imply
>> > > > > > > > > >> >> > > > > > > > > >> > > > > such dependency, but if
>> you
>> > > can
>> > > > > > > > clarify...
>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > > *No hadoop dependency.*
>> > > > > > > > > >> >> > > > > > > > > >> > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > Yay! Just add this to the
>> KIP
>> > so
>> > > > no
>> > > > > > one
>> > > > > > > > will
>> > > > > > > > > >> read
>> > > > > > > > > >> >> > the
>> > > > > > > > > >> >> > > > KIP
>> > > > > > > > > >> >> > > > > > and
>> > > > > > > > > >> >> > > > > > > > > panic
>> > > > > > > > > >> >> > > > > > > > > >> > > > three weeks before the next
>> > > > > release...
>> > > > > > > > > >> >> > > > > > > > > >> > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > > * Can we get delegation
>> > token
>> > > at
>> > > > > any
>> > > > > > > > time
>> > > > > > > > > >> after
>> > > > > > > > > >> >> > > > > > > > authenticating?
>> > > > > > > > > >> >> > > > > > > > > >> only
>> > > > > > > > > >> >> > > > > > > > > >> > > > > immediately after?
>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > > *As long as you are
>> > > > authenticated
>> > > > > > you
>> > > > > > > > can
>> > > > > > > > > >> get
>> > > > > > > > > >> >> > > > delegation
>> > > > > > > > > >> >> > > > > > > > tokens.
>> > > > > > > > > >> >> > > > > > > > > >> We
>> > > > > > > > > >> >> > > > > > > > > >> > > need
>> > > > > > > > > >> >> > > > > > > > > >> > > > to
>> > > > > > > > > >> >> > > > > > > > > >> > > > > discuss if a client
>> > > > authenticated
>> > > > > > > using
>> > > > > > > > > >> delegation
>> > > > > > > > > >> >> > > > > token,
>> > > > > > > > > >> >> > > > > > > can
>> > > > > > > > > >> >> > > > > > > > > also
>> > > > > > > > > >> >> > > > > > > > > >> > > > acquire
>> > > > > > > > > >> >> > > > > > > > > >> > > > > delegation token again or
>> > not.
>> > > > > Also
>> > > > > > > > there
>> > > > > > > > > >> is the
>> > > > > > > > > >> >> > > > > question
>> > > > > > > > > >> >> > > > > > of
>> > > > > > > > > >> >> > > > > > > > do
>> > > > > > > > > >> >> > > > > > > > > we
>> > > > > > > > > >> >> > > > > > > > > >> > > allow
>> > > > > > > > > >> >> > > > > > > > > >> > > > > anyone to acquire
>> delegation
>> > > > token
>> > > > > > or
>> > > > > > > we
>> > > > > > > > > >> want
>> > > > > > > > > >> >> > > specific
>> > > > > > > > > >> >> > > > > > ACLs
>> > > > > > > > > >> >> > > > > > > (I
>> > > > > > > > > >> >> > > > > > > > > >> think
>> > > > > > > > > >> >> > > > > > > > > >> > > its
>> > > > > > > > > >> >> > > > > > > > > >> > > > an
>> > > > > > > > > >> >> > > > > > > > > >> > > > > overkill.)*
>> > > > > > > > > >> >> > > > > > > > > >> > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > I agree that ACLs is an
>> > > overkill.
>> > > > > > > > > >> >> > > > > > > > > >> > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > I think we are debating two
>> > > > options:
>> > > > > > > > Either
>> > > > > > > > > >> require
>> > > > > > > > > >> >> > > > > Kerberos
>> > > > > > > > > >> >> > > > > > > > auth
>> > > > > > > > > >> >> > > > > > > > > >> for
>> > > > > > > > > >> >> > > > > > > > > >> > > > renewal or require
>> non-owners
>> > to
>> > > > > > renew.
>> > > > > > > > > >> >> > > > > > > > > >> > > > I *think* the latter is
>> > simpler
>> > > > (it
>> > > > > > > > > basically
>> > > > > > > > > >> >> > require
>> > > > > > > > > >> >> > > a
>> > > > > > > > > >> >> > > > > "job
>> > > > > > > > > >> >> > > > > > > > > master"
>> > > > > > > > > >> >> > > > > > > > > >> > > > to take responsibility for
>> the
>> > > > > > renewal,
>> > > > > > > it
>> > > > > > > > > >> will have
>> > > > > > > > > >> >> > > its
>> > > > > > > > > >> >> > > > > own
>> > > > > > > > > >> >> > > > > > > > > >> identity
>> > > > > > > > > >> >> > > > > > > > > >> > > > anyway and I think this is
>> the
>> > > > > correct
>> > > > > > > > > design
>> > > > > > > > > >> >> > pattern
>> > > > > > > > > >> >> > > > > > anyway.
>> > > > > > > > > >> >> > > > > > > > For
>> > > > > > > > > >> >> > > > > > > > > >> > > > storm, I'd expect Nimbus to
>> > > > > coordinate
>> > > > > > > > > >> renewals?),
>> > > > > > > > > >> >> > but
>> > > > > > > > > >> >> > > > it
>> > > > > > > > > >> >> > > > > is
>> > > > > > > > > >> >> > > > > > > > hard
>> > > > > > > > > >> >> > > > > > > > > to
>> > > > > > > > > >> >> > > > > > > > > >> > > > debate simplicity without
>> > > looking
>> > > > at
>> > > > > > the
>> > > > > > > > > code
>> > > > > > > > > >> >> > changes
>> > > > > > > > > >> >> > > > > > > required.
>> > > > > > > > > >> >> > > > > > > > If
>> > > > > > > > > >> >> > > > > > > > > >> you
>> > > > > > > > > >> >> > > > > > > > > >> > > > have a draft of how the
>> > "require
>> > > > > > > Kerberos"
>> > > > > > > > > >> will look
>> > > > > > > > > >> >> > > in
>> > > > > > > > > >> >> > > > > > Kafka
>> > > > > > > > > >> >> > > > > > > > > code,
>> > > > > > > > > >> >> > > > > > > > > >> > > > I'll be happy to take a
>> look.
>> > > > > > > > > >> >> > > > > > > > > >> > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > > * My understanding is
>> that
>> > > > tokens
>> > > > > > will
>> > > > > > > > > >> propagate
>> > > > > > > > > >> >> > via
>> > > > > > > > > >> >> > > > ZK
>> > > > > > > > > >> >> > > > > > but
>> > > > > > > > > >> >> > > > > > > > > >> without
>> > > > > > > > > >> >> > > > > > > > > >> > > > > additional changes to
>> > > > > UpdateMetadata
>> > > > > > > > > >> protocol,
>> > > > > > > > > >> >> > > > correct?
>> > > > > > > > > >> >> > > > > > > > Clients
>> > > > > > > > > >> >> > > > > > > > > >> > > > > currently don't retry on
>> > SASL
>> > > > auth
>> > > > > > > > failure
>> > > > > > > > > >> (IIRC),
>> > > > > > > > > >> >> > > but
>> > > > > > > > > >> >> > > > > > since
>> > > > > > > > > >> >> > > > > > > > the
>> > > > > > > > > >> >> > > > > > > > > >> > > > > tokens propagate between
>> > > brokers
>> > > > > > > asynch,
>> > > > > > > > > we
>> > > > > > > > > >> will
>> > > > > > > > > >> >> > > need
>> > > > > > > > > >> >> > > > to
>> > > > > > > > > >> >> > > > > > > > retry a
>> > > > > > > > > >> >> > > > > > > > > >> bit
>> > > > > > > > > >> >> > > > > > > > > >> > > > > to avoid clients failing
>> > auth
>> > > > due
>> > > > > to
>> > > > > > > > > timing
>> > > > > > > > > >> >> > issues.
>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > > *I am considering 2
>> > > alternatives
>> > > > > > right
>> > > > > > > > > now.
>> > > > > > > > > >> The
>> > > > > > > > > >> >> > > > current
>> > > > > > > > > >> >> > > > > > > > > documented
>> > > > > > > > > >> >> > > > > > > > > >> > > > approach
>> > > > > > > > > >> >> > > > > > > > > >> > > > > is zookeeper based and it
>> > does
>> > > > not
>> > > > > > > > require
>> > > > > > > > > >> any
>> > > > > > > > > >> >> > > changes
>> > > > > > > > > >> >> > > > > to
>> > > > > > > > > >> >> > > > > > > > > >> > > UpdateMetadata
>> > > > > > > > > >> >> > > > > > > > > >> > > > > protocol. An alternative
>> > > > approach
>> > > > > > can
>> > > > > > > > > remove
>> > > > > > > > > >> >> > > zookeeper
>> > > > > > > > > >> >> > > > > > > > > dependency
>> > > > > > > > > >> >> > > > > > > > > >> as
>> > > > > > > > > >> >> > > > > > > > > >> > > well
>> > > > > > > > > >> >> > > > > > > > > >> > > > > but we can discuss that
>> in
>> > KIP
>> > > > > > > > discussion
>> > > > > > > > > >> call.*
>> > > > > > > > > >> >> > > > > > > > > >> > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > Oooh! Sounds interesting.
>> Do
>> > you
>> > > > > want
>> > > > > > to
>> > > > > > > > > ping
>> > > > > > > > > >> Jun to
>> > > > > > > > > >> >> > > > > > arrange a
>> > > > > > > > > >> >> > > > > > > > > call?
>> > > > > > > > > >> >> > > > > > > > > >> > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > > * I liked Ashish's
>> > suggestion
>> > > of
>> > > > > > > having
>> > > > > > > > > >> just the
>> > > > > > > > > >> >> > > > > > controller
>> > > > > > > > > >> >> > > > > > > > > issue
>> > > > > > > > > >> >> > > > > > > > > >> the
>> > > > > > > > > >> >> > > > > > > > > >> > > > > delegation tokens, to
>> avoid
>> > > > > syncing
>> > > > > > a
>> > > > > > > > > shared
>> > > > > > > > > >> >> > secret.
>> > > > > > > > > >> >> > > > Not
>> > > > > > > > > >> >> > > > > > > sure
>> > > > > > > > > >> >> > > > > > > > if
>> > > > > > > > > >> >> > > > > > > > > >> we
>> > > > > > > > > >> >> > > > > > > > > >> > > > > want to continue the
>> > > discussion
>> > > > > here
>> > > > > > > or
>> > > > > > > > on
>> > > > > > > > > >> the
>> > > > > > > > > >> >> > > wiki. I
>> > > > > > > > > >> >> > > > > > think
>> > > > > > > > > >> >> > > > > > > > > that
>> > > > > > > > > >> >> > > > > > > > > >> we
>> > > > > > > > > >> >> > > > > > > > > >> > > > > can decouple the problem
>> of
>> > > > "token
>> > > > > > > > > >> distribution"
>> > > > > > > > > >> >> > > from
>> > > > > > > > > >> >> > > > > > > "shared
>> > > > > > > > > >> >> > > > > > > > > >> secret
>> > > > > > > > > >> >> > > > > > > > > >> > > > > distribution" and use the
>> > > > > controller
>> > > > > > > as
>> > > > > > > > > the
>> > > > > > > > > >> only
>> > > > > > > > > >> >> > > token
>> > > > > > > > > >> >> > > > > > > > generator
>> > > > > > > > > >> >> > > > > > > > > >> to
>> > > > > > > > > >> >> > > > > > > > > >> > > > > solve the second issue,
>> > while
>> > > > > still
>> > > > > > > > using
>> > > > > > > > > >> ZK async
>> > > > > > > > > >> >> > > to
>> > > > > > > > > >> >> > > > > > > > distribute
>> > > > > > > > > >> >> > > > > > > > > >> > > > > tokens.
>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > > *As mentioned in the
>> > previous
>> > > > > Email
>> > > > > > I
>> > > > > > > am
>> > > > > > > > > >> fine with
>> > > > > > > > > >> >> > > > that
>> > > > > > > > > >> >> > > > > > > > approach
>> > > > > > > > > >> >> > > > > > > > > >> as
>> > > > > > > > > >> >> > > > > > > > > >> > > long
>> > > > > > > > > >> >> > > > > > > > > >> > > > as
>> > > > > > > > > >> >> > > > > > > > > >> > > > > we agree that the extra
>> > > > complexity
>> > > > > > of
>> > > > > > > > > >> >> > > adding/updating
>> > > > > > > > > >> >> > > > > APIS
>> > > > > > > > > >> >> > > > > > > > adds
>> > > > > > > > > >> >> > > > > > > > > >> enough
>> > > > > > > > > >> >> > > > > > > > > >> > > > > value. The advantage with
>> > the
>> > > > > > > controller
>> > > > > > > > > >> approach
>> > > > > > > > > >> >> > is
>> > > > > > > > > >> >> > > > > > secret
>> > > > > > > > > >> >> > > > > > > > > >> rotation
>> > > > > > > > > >> >> > > > > > > > > >> > > can
>> > > > > > > > > >> >> > > > > > > > > >> > > > be
>> > > > > > > > > >> >> > > > > > > > > >> > > > > automated,frequent and
>> would
>> > > not
>> > > > > > > require
>> > > > > > > > > >> >> > > deployment. *
>> > > > > > > > > >> >> > > > > > > > > >> > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > Can you detail the extra
>> > > > complexity
>> > > > > > (or
>> > > > > > > > > point
>> > > > > > > > > >> me to
>> > > > > > > > > >> >> > > the
>> > > > > > > > > >> >> > > > > > email
>> > > > > > > > > >> >> > > > > > > I
>> > > > > > > > > >> >> > > > > > > > > >> > > > missed?) - which Apis are
>> > > > required?
>> > > > > > > > > >> >> > > > > > > > > >> > > > As far as I can tell,
>> clients
>> > > can
>> > > > > > > already
>> > > > > > > > > >> find the
>> > > > > > > > > >> >> > > > > > controller
>> > > > > > > > > >> >> > > > > > > > from
>> > > > > > > > > >> >> > > > > > > > > >> > > > metadata. I'm a bit more
>> > > concerned
>> > > > > > about
>> > > > > > > > > >> controller
>> > > > > > > > > >> >> > > > load.
>> > > > > > > > > >> >> > > > > > > > > >> > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > > * While I like the idea
>> of
>> > > > forcing
>> > > > > > > > > kerberos
>> > > > > > > > > >> auth
>> > > > > > > > > >> >> > for
>> > > > > > > > > >> >> > > > > > > renwal, I
>> > > > > > > > > >> >> > > > > > > > > >> think
>> > > > > > > > > >> >> > > > > > > > > >> > > > > it mixes the transport
>> layer
>> > > the
>> > > > > the
>> > > > > > > > > request
>> > > > > > > > > >> >> > content
>> > > > > > > > > >> >> > > > in
>> > > > > > > > > >> >> > > > > a
>> > > > > > > > > >> >> > > > > > > > pretty
>> > > > > > > > > >> >> > > > > > > > > >> ugly
>> > > > > > > > > >> >> > > > > > > > > >> > > > > way. Perhaps limiting
>> > renewer
>> > > to
>> > > > > > > > non-owner
>> > > > > > > > > >> is
>> > > > > > > > > >> >> > > better.
>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > > *I feel this is a
>> necessary
>> > > > evil.
>> > > > > > > While
>> > > > > > > > > >> this will
>> > > > > > > > > >> >> > > make
>> > > > > > > > > >> >> > > > > the
>> > > > > > > > > >> >> > > > > > > > kafka
>> > > > > > > > > >> >> > > > > > > > > >> code
>> > > > > > > > > >> >> > > > > > > > > >> > > > > pretty straight forward ,
>> > > > forcing
>> > > > > > > > renewer
>> > > > > > > > > >> to
>> > > > > > > > > >> >> > > > non-owner
>> > > > > > > > > >> >> > > > > > > pushes
>> > > > > > > > > >> >> > > > > > > > > >> the code
>> > > > > > > > > >> >> > > > > > > > > >> > > > > ugliness to client and
>> makes
>> > > it
>> > > > > even
>> > > > > > > > > harder
>> > > > > > > > > >> to
>> > > > > > > > > >> >> > > > > > integrate.  *
>> > > > > > > > > >> >> > > > > > > > > >> > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > As mentioned before, I
>> don't
>> > > think
>> > > > > the
>> > > > > > > > > >> "renewal by
>> > > > > > > > > >> >> > > > other"
>> > > > > > > > > >> >> > > > > > > > approach
>> > > > > > > > > >> >> > > > > > > > > >> is
>> > > > > > > > > >> >> > > > > > > > > >> > > > that ugly for the clients
>> we
>> > > > expect
>> > > > > to
>> > > > > > > use
>> > > > > > > > > >> >> > delegation
>> > > > > > > > > >> >> > > > > tokens
>> > > > > > > > > >> >> > > > > > > > since
>> > > > > > > > > >> >> > > > > > > > > >> > > > they will have an
>> app-master
>> > of
>> > > > some
>> > > > > > > sort
>> > > > > > > > > who
>> > > > > > > > > >> >> > > requested
>> > > > > > > > > >> >> > > > > the
>> > > > > > > > > >> >> > > > > > > > token
>> > > > > > > > > >> >> > > > > > > > > to
>> > > > > > > > > >> >> > > > > > > > > >> > > > begin with.
>> > > > > > > > > >> >> > > > > > > > > >> > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > > The response for my
>> question
>> > > on
>> > > > > how
>> > > > > > > > > multiple
>> > > > > > > > > >> >> > > > identities
>> > > > > > > > > >> >> > > > > > will
>> > > > > > > > > >> >> > > > > > > > be
>> > > > > > > > > >> >> > > > > > > > > >> > > > > handled wasn't super
>> clear
>> > to
>> > > > me -
>> > > > > > > > AFAIK,
>> > > > > > > > > >> we don't
>> > > > > > > > > >> >> > > > > > > > authenticate
>> > > > > > > > > >> >> > > > > > > > > >> each
>> > > > > > > > > >> >> > > > > > > > > >> > > > > request, we authenticate
>> > > > > > connections.
>> > > > > > > > > >> >> > > > > > > > > >> > > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > > *We authenticate
>> > connections,
>> > > > and
>> > > > > > only
>> > > > > > > > > when
>> > > > > > > > > >> they
>> > > > > > > > > >> >> > are
>> > > > > > > > > >> >> > > > > being
>> > > > > > > > > >> >> > > > > > > > > >> established.
>> > > > > > > > > >> >> > > > > > > > > >> > > > Let
>> > > > > > > > > >> >> > > > > > > > > >> > > > > me try to phrase this as
>> a
>> > > > > question,
>> > > > > > > in
>> > > > > > > > > >> absence of
>> > > > > > > > > >> >> > > > > > > delegation
>> > > > > > > > > >> >> > > > > > > > > >> tokens if
>> > > > > > > > > >> >> > > > > > > > > >> > > > we
>> > > > > > > > > >> >> > > > > > > > > >> > > > > had to support the use
>> case
>> > > > using
>> > > > > > user
>> > > > > > > > > >> TGT's how
>> > > > > > > > > >> >> > > would
>> > > > > > > > > >> >> > > > > we
>> > > > > > > > > >> >> > > > > > do
>> > > > > > > > > >> >> > > > > > > > it?
>> > > > > > > > > >> >> > > > > > > > > >> My
>> > > > > > > > > >> >> > > > > > > > > >> > > point
>> > > > > > > > > >> >> > > > > > > > > >> > > > > was it would be no
>> different
>> > > > with
>> > > > > > > > > delegation
>> > > > > > > > > >> >> > tokens.
>> > > > > > > > > >> >> > > > The
>> > > > > > > > > >> >> > > > > > use
>> > > > > > > > > >> >> > > > > > > > > case
>> > > > > > > > > >> >> > > > > > > > > >> you
>> > > > > > > > > >> >> > > > > > > > > >> > > are
>> > > > > > > > > >> >> > > > > > > > > >> > > > > describing seems more
>> like
>> > > > > > > > impersonation.*
>> > > > > > > > > >> >> > > > > > > > > >> > > >
>> > > > > > > > > >> >> > > > > > > > > >> > > > Yeah, I thought that one of
>> > the
>> > > > > things
>> > > > > > > > that
>> > > > > > > > > >> >> > del
>>
>
>
>
> --
>
> Regards,
> Ashish
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Ashish Singh <as...@cloudera.com>]

Thanks Harsha!

Jun, can we add KIP-48 to next KIP hangout's agenda. Also, we did not
actually make a call on when we should have next KIP call. As there are a
few outstanding KIPs that could not be discussed this week, can we have a
KIP hangout call next week?

On Tue, Aug 23, 2016 at 1:10 PM, Harsha Chintalapani <ka...@harsha.io>
wrote:

> Ashish,
>         Yes we are working on it. Lets discuss in the next KIP meeting.
> I'll join.
> -Harsha
>
> On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh <as...@cloudera.com> wrote:
>
> > Hello Harsha,
> >
> > Are you still working on this? Wondering if we can discuss this in next
> KIP
> > meeting, if you can join.
> >
> > On Mon, Jul 18, 2016 at 9:51 AM, Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > > Hi Grant,
> > >           We are working on it. Will add the details to KIP about the
> > > request protocol.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Mon, Jul 18, 2016 at 6:50 AM Grant Henke <gh...@cloudera.com>
> wrote:
> > >
> > > > Hi Parth,
> > > >
> > > > Are you still working on this? If you need any help please don't
> > hesitate
> > > > to ask.
> > > >
> > > > Thanks,
> > > > Grant
> > > >
> > > > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <ju...@confluent.io> wrote:
> > > >
> > > > > Parth,
> > > > >
> > > > > Thanks for the reply.
> > > > >
> > > > > It makes sense to only allow the renewal by users that
> authenticated
> > > > using
> > > > > *non* delegation token mechanism. Then, should we make the renewal
> a
> > > > list?
> > > > > For example, in the case of rest proxy, it will be useful for every
> > > > > instance of rest proxy to be able to renew the tokens.
> > > > >
> > > > > It would be clearer if we can document the request protocol like
> > > > >
> > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 4+-+Command+line+and+centralized+administrative+operations#KIP-4-
> > > Commandlineandcentralizedadministrativeoperations-
> > > CreateTopicsRequest(KAFKA-2945):(VotedandPlannedforin0.10.1.0)
> > > > > .
> > > > >
> > > > > It would also be useful to document the client APIs.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Tue, Jun 28, 2016 at 2:55 PM, parth brahmbhatt <
> > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I am suggesting that we will only allow the renewal by users that
> > > > > > authenticated using *non* delegation token mechanism. For
> example,
> > If
> > > > > user
> > > > > > Alice authenticated using kerberos and requested delegation
> tokens,
> > > > only
> > > > > > user Alice authenticated via non delegation token mechanism can
> > > renew.
> > > > > > Clients that have  access to delegation tokens can not issue
> > renewal
> > > > > > request for renewing their own token and this is primarily
> > important
> > > to
> > > > > > reduce the time window for which a compromised token will be
> valid.
> > > > > >
> > > > > > To clarify, Yes any authenticated user can request delegation
> > tokens
> > > > but
> > > > > > even here I would recommend to avoid creating a chain where a
> > client
> > > > > > authenticated via delegation token request for more delegation
> > > tokens.
> > > > > > Basically anyone can request delegation token, as long as they
> > > > > authenticate
> > > > > > via a non delegation token mechanism.
> > > > > >
> > > > > > Aren't classes listed here
> > > > > > <
> > > > > >
> > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 48+Delegation+token+support+for+Kafka#KIP-
> 48DelegationtokensupportforKaf
> > > ka-PublicInterfaces
> > > > > > >
> > > > > > sufficient?
> > > > > >
> > > > > > Thanks
> > > > > > Parth
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao <ju...@confluent.io>
> wrote:
> > > > > >
> > > > > > > Parth,
> > > > > > >
> > > > > > > Thanks for the reply. A couple of comments inline below.
> > > > > > >
> > > > > > > On Tue, Jun 21, 2016 at 10:36 AM, parth brahmbhatt <
> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > > > >
> > > > > > > > 1. Who / how are tokens renewed? By original requester only?
> or
> > > > using
> > > > > > > > Kerberos
> > > > > > > > auth only?
> > > > > > > > My recommendation is to do this only using Kerberos auth and
> > only
> > > > > threw
> > > > > > > the
> > > > > > > > renewer specified during the acquisition request.
> > > > > > > >
> > > > > > > >
> > > > > > > Hmm, not sure that I follow this. Are you saying that any
> client
> > > > > > > authenticated with the delegation token can renew, i.e. there
> is
> > no
> > > > > > renewer
> > > > > > > needed?
> > > > > > >
> > > > > > > Also, just to be clear, any authenticated client (either
> through
> > > SASL
> > > > > or
> > > > > > > SSL) can request a delegation token for the authenticated user,
> > > > right?
> > > > > > >
> > > > > > >
> > > > > > > > 2. Are tokens stored on each broker or in ZK?
> > > > > > > > My recommendation is still to store in ZK or not store them
> at
> > > all.
> > > > > The
> > > > > > > > whole controller based distribution is too much overhead with
> > not
> > > > > much
> > > > > > to
> > > > > > > > achieve.
> > > > > > > >
> > > > > > > > 3. How are tokens invalidated / expired?
> > > > > > > > Either by expiration time out or through an explicit request
> to
> > > > > > > invalidate.
> > > > > > > >
> > > > > > > > 4. Which encryption algorithm is used?
> > > > > > > > SCRAM
> > > > > > > >
> > > > > > > > 5. What is the impersonation proposal (it wasn't in the KIP
> but
> > > was
> > > > > > > > discussed
> > > > > > > > in this thread)?
> > > > > > > > There is no imperonation proposal. I tried and explained how
> > its
> > > a
> > > > > > > > different problem and why its not really necessary to discuss
> > > that
> > > > as
> > > > > > > part
> > > > > > > > of this KIP.  This KIP will not support any impersonation, it
> > > will
> > > > > just
> > > > > > > be
> > > > > > > > another way to authenticate.
> > > > > > > >
> > > > > > > > 6. Do we need new ACLs, if so - for what actions?
> > > > > > > > We do not need new ACLs.
> > > > > > > >
> > > > > > > >
> > > > > > > Could we document the format of the new request/response and
> > their
> > > > > > > associated Resource and Operation for ACL?
> > > > > > >
> > > > > > >
> > > > > > > > 7. How would the delegation token be configured in the
> client?
> > > > > > > > Should be through config. I wasn't planning on supporting
> JAAS
> > > for
> > > > > > > tokens.
> > > > > > > > I don't believe hadoop does this either.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Parth
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <ju...@confluent.io>
> > > wrote:
> > > > > > > >
> > > > > > > > > Harsha,
> > > > > > > > >
> > > > > > > > > Another question.
> > > > > > > > >
> > > > > > > > > 9. How would the delegation token be configured in the
> > client?
> > > > The
> > > > > > > > standard
> > > > > > > > > way is to do this through JAAS. However, we will need to
> > think
> > > > > > through
> > > > > > > if
> > > > > > > > > this is convenient in a shared environment. For example,
> > when a
> > > > new
> > > > > > > task
> > > > > > > > is
> > > > > > > > > added to a Storm worker node, do we need to dynamically
> add a
> > > new
> > > > > > > section
> > > > > > > > > in the JAAS file? It may be more convenient if we can pass
> in
> > > the
> > > > > > token
> > > > > > > > > through the config directly w/o going through JAAS.
> > > > > > > > >
> > > > > > > > > Are you or Parth still actively working on this KIP?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <jun@confluent.io
> >
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > Just to add on that list.
> > > > > > > > > >
> > > > > > > > > > 2. It would be good to document the format of the data
> > stored
> > > > in
> > > > > > ZK.
> > > > > > > > > > 7. Earlier, there was a discussion on whether the tokens
> > > should
> > > > > be
> > > > > > > > > > propagated through ZK like config/acl/quota, or through
> the
> > > > > > > controller.
> > > > > > > > > > Currently, the controller is only designed for
> propagating
> > > > topic
> > > > > > > > > metadata,
> > > > > > > > > > but not other data.
> > > > > > > > > > 8. Should we use SCRAM to send the token instead of
> > > DIGEST-MD5
> > > > > > since
> > > > > > > > it's
> > > > > > > > > > deprecated?
> > > > > > > > > >
> > > > > > > > > > Also, the images in the wiki seem broken.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <
> > > > > gwen@confluent.io>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> From what I can see, remaining questions are:
> > > > > > > > > >>
> > > > > > > > > >> 1. Who / how are tokens renewed? By original requester
> > only?
> > > > or
> > > > > > > using
> > > > > > > > > >> Kerberos auth only?
> > > > > > > > > >> 2. Are tokens stored on each broker or in ZK?
> > > > > > > > > >> 3. How are tokens invalidated / expired?
> > > > > > > > > >> 4. Which encryption algorithm is used?
> > > > > > > > > >> 5. What is the impersonation proposal (it wasn't in the
> > KIP
> > > > but
> > > > > > was
> > > > > > > > > >> discussed in this thread)?
> > > > > > > > > >> 6. Do we need new ACLs, if so - for what actions?
> > > > > > > > > >>
> > > > > > > > > >> Gwen
> > > > > > > > > >>
> > > > > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <kafka@harsha.io
> >
> > > > wrote:
> > > > > > > > > >> > Jun & Ismael,
> > > > > > > > > >> >                          Unfortunately I couldn't
> attend
> > > the
> > > > > KIP
> > > > > > > > > meeting
> > > > > > > > > >> >                          when delegation tokens
> > discussed.
> > > > > > > > Appreciate
> > > > > > > > > if
> > > > > > > > > >> >                          you can update the thread if
> > you
> > > > have
> > > > > > any
> > > > > > > > > >> >                          further questions.
> > > > > > > > > >> > Thanks,
> > > > > > > > > >> > Harsha
> > > > > > > > > >> >
> > > > > > > > > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
> > > > > > > > > >> >> It seems that the links to images in the KIP are
> > broken.
> > > > > > > > > >> >>
> > > > > > > > > >> >> Liquan
> > > > > > > > > >> >>
> > > > > > > > > >> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
> > > > > > > > > >> >> brahmbhatt.parth@gmail.com> wrote:
> > > > > > > > > >> >>
> > > > > > > > > >> >> > 110. What does getDelegationTokenAs mean?
> > > > > > > > > >> >> > In the current proposal we only allow a user to get
> > > > > > delegation
> > > > > > > > > token
> > > > > > > > > >> for
> > > > > > > > > >> >> > the identity that it authenticated as using another
> > > > > > mechanism,
> > > > > > > > i.e.
> > > > > > > > > >> A user
> > > > > > > > > >> >> > that authenticate using a keytab for principal
> > > > > > > user1@EXAMPLE.COM
> > > > > > > > > >> will get
> > > > > > > > > >> >> > delegation tokens for that user only. In future I
> > think
> > > > we
> > > > > > will
> > > > > > > > > have
> > > > > > > > > >> to
> > > > > > > > > >> >> > extend support such that we allow some set of
> users (
> > > > > > > > > >> >> > kafka-rest-user@EXAMPLE.COM,
> > storm-nimbus@EXAMPLE.COM)
> > > > to
> > > > > > > > acquire
> > > > > > > > > >> >> > delegation tokens on behalf of other users whose
> > > identity
> > > > > > they
> > > > > > > > have
> > > > > > > > > >> >> > verified independently.  Kafka brokers will have
> ACLs
> > > to
> > > > > > > control
> > > > > > > > > >> which
> > > > > > > > > >> >> > users are allowed to impersonate other users and
> get
> > > > tokens
> > > > > > on
> > > > > > > > > >> behalf of
> > > > > > > > > >> >> > them. Overall Impersonation is a whole different
> > > problem
> > > > in
> > > > > > my
> > > > > > > > > >> opinion and
> > > > > > > > > >> >> > I think we can tackle it in separate KIP.
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > 111. What's the typical rate of getting and
> renewing
> > > > > > delegation
> > > > > > > > > >> tokens?
> > > > > > > > > >> >> > Typically this should be very very low, 1 request
> per
> > > > > minute
> > > > > > > is a
> > > > > > > > > >> >> > relatively high estimate. However it depends on the
> > > token
> > > > > > > > > >> expiration. I am
> > > > > > > > > >> >> > less worried about the extra load it puts on
> > controller
> > > > vs
> > > > > > the
> > > > > > > > > added
> > > > > > > > > >> >> > complexity and the value it offers.
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > Thanks
> > > > > > > > > >> >> > Parth
> > > > > > > > > >> >> >
> > > > > > > > > >> >> >
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <
> > > > > > > ismael@juma.me.uk>
> > > > > > > > > >> wrote:
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > > Thanks Rajini. It would probably require a
> separate
> > > KIP
> > > > > as
> > > > > > it
> > > > > > > > > will
> > > > > > > > > >> >> > > introduce user visible changes. We could also
> > update
> > > > > KIP-48
> > > > > > > to
> > > > > > > > > >> have this
> > > > > > > > > >> >> > > information, but it seems cleaner to do it
> > > separately.
> > > > We
> > > > > > can
> > > > > > > > > >> discuss
> > > > > > > > > >> >> > that
> > > > > > > > > >> >> > > in the KIP call today.
> > > > > > > > > >> >> > >
> > > > > > > > > >> >> > > Ismael
> > > > > > > > > >> >> > >
> > > > > > > > > >> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <
> > > > > > > > > >> >> > > rajinisivaram@googlemail.com> wrote:
> > > > > > > > > >> >> > >
> > > > > > > > > >> >> > > > Ismael,
> > > > > > > > > >> >> > > >
> > > > > > > > > >> >> > > > I have created a JIRA (
> > > > > > > > > >> >> > https://issues.apache.org/jira/browse/KAFKA-3751)
> > > > > > > > > >> >> > > > for adding SCRAM as a SASL mechanism. Would
> that
> > > need
> > > > > > > another
> > > > > > > > > >> KIP? If
> > > > > > > > > >> >> > > > KIP-48 will use this mechanism, can this just
> be
> > a
> > > > JIRA
> > > > > > > that
> > > > > > > > > gets
> > > > > > > > > >> >> > > reviewed
> > > > > > > > > >> >> > > > when the PR is ready?
> > > > > > > > > >> >> > > >
> > > > > > > > > >> >> > > > Thank you,
> > > > > > > > > >> >> > > >
> > > > > > > > > >> >> > > > Rajini
> > > > > > > > > >> >> > > >
> > > > > > > > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <
> > > > > > > > > ismael@juma.me.uk>
> > > > > > > > > >> >> > wrote:
> > > > > > > > > >> >> > > >
> > > > > > > > > >> >> > > > > Thanks Rajini, SCRAM seems like a good
> > candidate.
> > > > > > > > > >> >> > > > >
> > > > > > > > > >> >> > > > > Gwen had independently mentioned this as a
> SASL
> > > > > > mechanism
> > > > > > > > > that
> > > > > > > > > >> might
> > > > > > > > > >> >> > be
> > > > > > > > > >> >> > > > > useful for Kafka and I have been meaning to
> > check
> > > > it
> > > > > in
> > > > > > > > more
> > > > > > > > > >> detail.
> > > > > > > > > >> >> > > Good
> > > > > > > > > >> >> > > > > to know that you are willing to contribute an
> > > > > > > > implementation.
> > > > > > > > > >> Maybe
> > > > > > > > > >> >> > we
> > > > > > > > > >> >> > > > > should file a separate JIRA for this?
> > > > > > > > > >> >> > > > >
> > > > > > > > > >> >> > > > > Ismael
> > > > > > > > > >> >> > > > >
> > > > > > > > > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini
> > Sivaram <
> > > > > > > > > >> >> > > > > rajinisivaram@googlemail.com> wrote:
> > > > > > > > > >> >> > > > >
> > > > > > > > > >> >> > > > > > SCRAM (Salted Challenge Response
> > Authentication
> > > > > > > > Mechanism)
> > > > > > > > > >> is a
> > > > > > > > > >> >> > > better
> > > > > > > > > >> >> > > > > > mechanism than Digest-MD5. Java doesn't
> come
> > > > with a
> > > > > > > > > built-in
> > > > > > > > > >> SCRAM
> > > > > > > > > >> >> > > > > > SaslServer or SaslClient, but I will be
> happy
> > > to
> > > > > add
> > > > > > > > > support
> > > > > > > > > >> in
> > > > > > > > > >> >> > Kafka
> > > > > > > > > >> >> > > > > since
> > > > > > > > > >> >> > > > > > it would be a useful mechanism to support
> > > anyway.
> > > > > > > > > >> >> > > > > > https://tools.ietf.org/html/rfc7677
> > describes
> > > > the
> > > > > > > > protocol
> > > > > > > > > >> for
> > > > > > > > > >> >> > > > > > SCRAM-SHA-256.
> > > > > > > > > >> >> > > > > >
> > > > > > > > > >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <
> > > > > > > > jun@confluent.io
> > > > > > > > > >
> > > > > > > > > >> wrote:
> > > > > > > > > >> >> > > > > >
> > > > > > > > > >> >> > > > > > > Parth,
> > > > > > > > > >> >> > > > > > >
> > > > > > > > > >> >> > > > > > > Thanks for the explanation. A couple of
> > more
> > > > > > > questions.
> > > > > > > > > >> >> > > > > > >
> > > > > > > > > >> >> > > > > > > 110. What does getDelegationTokenAs mean?
> > > > > > > > > >> >> > > > > > >
> > > > > > > > > >> >> > > > > > > 111. What's the typical rate of getting
> and
> > > > > > renewing
> > > > > > > > > >> delegation
> > > > > > > > > >> >> > > > tokens?
> > > > > > > > > >> >> > > > > > > That may have an impact on whether they
> > > should
> > > > be
> > > > > > > > > directed
> > > > > > > > > >> to the
> > > > > > > > > >> >> > > > > > > controller.
> > > > > > > > > >> >> > > > > > >
> > > > > > > > > >> >> > > > > > > Jun
> > > > > > > > > >> >> > > > > > >
> > > > > > > > > >> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth
> > > > > brahmbhatt <
> > > > > > > > > >> >> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > > > > > > >> >> > > > > > >
> > > > > > > > > >> >> > > > > > > > Hi Jun,
> > > > > > > > > >> >> > > > > > > >
> > > > > > > > > >> >> > > > > > > > Thanks for reviewing.
> > > > > > > > > >> >> > > > > > > >
> > > > > > > > > >> >> > > > > > > > * We could add a Cluster action to add
> > acls
> > > > on
> > > > > > who
> > > > > > > > can
> > > > > > > > > >> request
> > > > > > > > > >> >> > > > > > delegation
> > > > > > > > > >> >> > > > > > > > tokens. I don't see the use case for
> that
> > > yet
> > > > > but
> > > > > > > > down
> > > > > > > > > >> the line
> > > > > > > > > >> >> > > > when
> > > > > > > > > >> >> > > > > we
> > > > > > > > > >> >> > > > > > > > start supporting getDelegationTokenAs
> it
> > > will
> > > > > be
> > > > > > > > > >> necessary.
> > > > > > > > > >> >> > > > > > > > * Yes we recommend tokens to be only
> > > > > > > used/distributed
> > > > > > > > > >> over
> > > > > > > > > >> >> > secure
> > > > > > > > > >> >> > > > > > > channels.
> > > > > > > > > >> >> > > > > > > > * Depending on what design we end up
> > > choosing
> > > > > > > > > >> Invalidation will
> > > > > > > > > >> >> > > be
> > > > > > > > > >> >> > > > > > > > responsibility of every broker or
> > > controller.
> > > > > > > > > >> >> > > > > > > > * I am not sure if I documented
> somewhere
> > > > that
> > > > > > > > > >> invalidation
> > > > > > > > > >> >> > will
> > > > > > > > > >> >> > > > > > directly
> > > > > > > > > >> >> > > > > > > > go through zookeeper but that is not
> the
> > > > > intent.
> > > > > > > > > >> Invalidation
> > > > > > > > > >> >> > > will
> > > > > > > > > >> >> > > > > > either
> > > > > > > > > >> >> > > > > > > > be request based or due to expiration.
> No
> > > > > direct
> > > > > > > > > >> zookeeper
> > > > > > > > > >> >> > > > > interaction
> > > > > > > > > >> >> > > > > > > from
> > > > > > > > > >> >> > > > > > > > any client.
> > > > > > > > > >> >> > > > > > > > * "Broker also stores the
> DelegationToken
> > > > > without
> > > > > > > the
> > > > > > > > > >> hmac in
> > > > > > > > > >> >> > the
> > > > > > > > > >> >> > > > > > > > zookeeper." : Sorry about the
> confusion.
> > > The
> > > > > sole
> > > > > > > > > >> purpose of
> > > > > > > > > >> >> > > > > zookeeper
> > > > > > > > > >> >> > > > > > in
> > > > > > > > > >> >> > > > > > > > this design is as distribution channel
> > for
> > > > > tokens
> > > > > > > > > >> between all
> > > > > > > > > >> >> > > > brokers
> > > > > > > > > >> >> > > > > > > and a
> > > > > > > > > >> >> > > > > > > > layer that ensures only tokens that
> were
> > > > > > generated
> > > > > > > by
> > > > > > > > > >> making a
> > > > > > > > > >> >> > > > > request
> > > > > > > > > >> >> > > > > > > to a
> > > > > > > > > >> >> > > > > > > > broker will be accepted (more on this
> in
> > > > second
> > > > > > > > > >> paragraph). The
> > > > > > > > > >> >> > > > token
> > > > > > > > > >> >> > > > > > > > consists of few elements (owner,
> renewer,
> > > > uuid
> > > > > ,
> > > > > > > > > >> expiration,
> > > > > > > > > >> >> > > hmac)
> > > > > > > > > >> >> > > > ,
> > > > > > > > > >> >> > > > > > one
> > > > > > > > > >> >> > > > > > > of
> > > > > > > > > >> >> > > > > > > > which is the finally generated hmac but
> > > hmac
> > > > it
> > > > > > > self
> > > > > > > > is
> > > > > > > > > >> >> > derivable
> > > > > > > > > >> >> > > > if
> > > > > > > > > >> >> > > > > > you
> > > > > > > > > >> >> > > > > > > > have all the other elements of the
> token
> > +
> > > > > secret
> > > > > > > key
> > > > > > > > > to
> > > > > > > > > >> >> > generate
> > > > > > > > > >> >> > > > > hmac.
> > > > > > > > > >> >> > > > > > > > Given zookeeper does not provide SSL
> > > support
> > > > we
> > > > > > do
> > > > > > > > not
> > > > > > > > > >> want the
> > > > > > > > > >> >> > > > > entire
> > > > > > > > > >> >> > > > > > > > token to be wire transferred to
> zookeeper
> > > as
> > > > > that
> > > > > > > > will
> > > > > > > > > >> be an
> > > > > > > > > >> >> > > > insecure
> > > > > > > > > >> >> > > > > > > wire
> > > > > > > > > >> >> > > > > > > > transfer. Instead we only store all the
> > > other
> > > > > > > > elements
> > > > > > > > > >> of a
> > > > > > > > > >> >> > > > > delegation
> > > > > > > > > >> >> > > > > > > > tokens. Brokers can read these elements
> > and
> > > > > > because
> > > > > > > > > they
> > > > > > > > > >> also
> > > > > > > > > >> >> > > have
> > > > > > > > > >> >> > > > > > access
> > > > > > > > > >> >> > > > > > > > to secret key they will be able to
> > generate
> > > > > hmac
> > > > > > on
> > > > > > > > > >> their end.
> > > > > > > > > >> >> > > > > > > >
> > > > > > > > > >> >> > > > > > > > One of the alternative proposed is to
> > avoid
> > > > > > > zookeeper
> > > > > > > > > >> >> > > altogether. A
> > > > > > > > > >> >> > > > > > > Client
> > > > > > > > > >> >> > > > > > > > will call broker with required
> > information
> > > > > > (owner,
> > > > > > > > > >> renwer,
> > > > > > > > > >> >> > > > > expiration)
> > > > > > > > > >> >> > > > > > > and
> > > > > > > > > >> >> > > > > > > > get back (signed hmac, uuid). Broker
> > won't
> > > > > store
> > > > > > > this
> > > > > > > > > in
> > > > > > > > > >> >> > > zookeeper.
> > > > > > > > > >> >> > > > > > From
> > > > > > > > > >> >> > > > > > > > this point a client can contact any
> > broker
> > > > with
> > > > > > all
> > > > > > > > the
> > > > > > > > > >> >> > > delegation
> > > > > > > > > >> >> > > > > > token
> > > > > > > > > >> >> > > > > > > > info (owner, rewner, expiration, hmac,
> > > uuid)
> > > > > the
> > > > > > > > borker
> > > > > > > > > >> will
> > > > > > > > > >> >> > > > > regenerate
> > > > > > > > > >> >> > > > > > > the
> > > > > > > > > >> >> > > > > > > > hmac and as long as it matches with
> hmac
> > > > > > presented
> > > > > > > by
> > > > > > > > > >> client ,
> > > > > > > > > >> >> > > > broker
> > > > > > > > > >> >> > > > > > > will
> > > > > > > > > >> >> > > > > > > > allow the request to authenticate.
> Only
> > > > > problem
> > > > > > > with
> > > > > > > > > >> this
> > > > > > > > > >> >> > > approach
> > > > > > > > > >> >> > > > > is
> > > > > > > > > >> >> > > > > > if
> > > > > > > > > >> >> > > > > > > > the secret key is compromised any
> client
> > > can
> > > > > now
> > > > > > > > > generate
> > > > > > > > > >> >> > random
> > > > > > > > > >> >> > > > > tokens
> > > > > > > > > >> >> > > > > > > and
> > > > > > > > > >> >> > > > > > > > they will still be able to authenticate
> > as
> > > > any
> > > > > > user
> > > > > > > > > they
> > > > > > > > > >> like.
> > > > > > > > > >> >> > > with
> > > > > > > > > >> >> > > > > > > > zookeeper we guarantee that only tokens
> > > > > acquired
> > > > > > > via
> > > > > > > > a
> > > > > > > > > >> broker
> > > > > > > > > >> >> > > > (using
> > > > > > > > > >> >> > > > > > some
> > > > > > > > > >> >> > > > > > > > auth scheme other than delegation
> token)
> > > will
> > > > > be
> > > > > > > > > >> accepted. We
> > > > > > > > > >> >> > > need
> > > > > > > > > >> >> > > > to
> > > > > > > > > >> >> > > > > > > > discuss which proposal makes more sense
> > and
> > > > we
> > > > > > can
> > > > > > > go
> > > > > > > > > >> over it
> > > > > > > > > >> >> > in
> > > > > > > > > >> >> > > > > > > tomorrow's
> > > > > > > > > >> >> > > > > > > > meeting.
> > > > > > > > > >> >> > > > > > > >
> > > > > > > > > >> >> > > > > > > > Also, can you forward the invite to me?
> > > > > > > > > >> >> > > > > > > >
> > > > > > > > > >> >> > > > > > > > Thanks
> > > > > > > > > >> >> > > > > > > > Parth
> > > > > > > > > >> >> > > > > > > >
> > > > > > > > > >> >> > > > > > > >
> > > > > > > > > >> >> > > > > > > >
> > > > > > > > > >> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun
> > Rao <
> > > > > > > > > >> jun@confluent.io>
> > > > > > > > > >> >> > > > wrote:
> > > > > > > > > >> >> > > > > > > >
> > > > > > > > > >> >> > > > > > > > > Thanks for the KIP. A few comments.
> > > > > > > > > >> >> > > > > > > > >
> > > > > > > > > >> >> > > > > > > > > 100. This potentially can be useful
> for
> > > > Kafka
> > > > > > > > Connect
> > > > > > > > > >> and
> > > > > > > > > >> >> > Kafka
> > > > > > > > > >> >> > > > > rest
> > > > > > > > > >> >> > > > > > > > proxy
> > > > > > > > > >> >> > > > > > > > > where a worker agent will need to
> run a
> > > > task
> > > > > on
> > > > > > > > > behalf
> > > > > > > > > >> of a
> > > > > > > > > >> >> > > > client.
> > > > > > > > > >> >> > > > > > We
> > > > > > > > > >> >> > > > > > > > will
> > > > > > > > > >> >> > > > > > > > > likely need to change how those
> > services
> > > > use
> > > > > > > Kafka
> > > > > > > > > >> clients
> > > > > > > > > >> >> > > > > > > > > (producer/consumer). Instead of a
> > shared
> > > > > client
> > > > > > > per
> > > > > > > > > >> worker,
> > > > > > > > > >> >> > we
> > > > > > > > > >> >> > > > will
> > > > > > > > > >> >> > > > > > > need
> > > > > > > > > >> >> > > > > > > > a
> > > > > > > > > >> >> > > > > > > > > client per user task since the
> > > > authentication
> > > > > > > > happens
> > > > > > > > > >> at the
> > > > > > > > > >> >> > > > > > connection
> > > > > > > > > >> >> > > > > > > > > level. For Kafka Connect, the renewer
> > > will
> > > > be
> > > > > > the
> > > > > > > > > >> workers.
> > > > > > > > > >> >> > So,
> > > > > > > > > >> >> > > we
> > > > > > > > > >> >> > > > > > > > probably
> > > > > > > > > >> >> > > > > > > > > need to allow multiple renewers. For
> > > Kafka
> > > > > rest
> > > > > > > > > proxy,
> > > > > > > > > >> the
> > > > > > > > > >> >> > > > renewer
> > > > > > > > > >> >> > > > > > can
> > > > > > > > > >> >> > > > > > > > > probably just be the creator of the
> > > token.
> > > > > > > > > >> >> > > > > > > > >
> > > > > > > > > >> >> > > > > > > > > 101. Do we need new acl on who can
> > > request
> > > > > > > > delegation
> > > > > > > > > >> tokens?
> > > > > > > > > >> >> > > > > > > > >
> > > > > > > > > >> >> > > > > > > > > 102. Do we recommend people to send
> > > > > delegation
> > > > > > > > tokens
> > > > > > > > > >> in an
> > > > > > > > > >> >> > > > > encrypted
> > > > > > > > > >> >> > > > > > > > > channel?
> > > > > > > > > >> >> > > > > > > > >
> > > > > > > > > >> >> > > > > > > > > 103. Who is responsible for expiring
> > > > tokens,
> > > > > > > every
> > > > > > > > > >> broker?
> > > > > > > > > >> >> > > > > > > > >
> > > > > > > > > >> >> > > > > > > > > 104. For invalidating tokens, would
> it
> > be
> > > > > > better
> > > > > > > to
> > > > > > > > > do
> > > > > > > > > >> it in
> > > > > > > > > >> >> > a
> > > > > > > > > >> >> > > > > > request
> > > > > > > > > >> >> > > > > > > > > instead of going to ZK directly?
> > > > > > > > > >> >> > > > > > > > >
> > > > > > > > > >> >> > > > > > > > > 105. The terminology of client in the
> > > wiki
> > > > > > > > sometimes
> > > > > > > > > >> refers
> > > > > > > > > >> >> > to
> > > > > > > > > >> >> > > > the
> > > > > > > > > >> >> > > > > > end
> > > > > > > > > >> >> > > > > > > > > client and some other times refers to
> > the
> > > > > > client
> > > > > > > > > using
> > > > > > > > > >> the
> > > > > > > > > >> >> > > > > delegation
> > > > > > > > > >> >> > > > > > > > > tokens. It would be useful to
> > distinguish
> > > > > > between
> > > > > > > > the
> > > > > > > > > >> two.
> > > > > > > > > >> >> > > > > > > > >
> > > > > > > > > >> >> > > > > > > > > 106. Could you explain the sentence
> > > "Broker
> > > > > > also
> > > > > > > > > >> stores the
> > > > > > > > > >> >> > > > > > > > DelegationToken
> > > > > > > > > >> >> > > > > > > > > without the hmac in the zookeeper." a
> > bit
> > > > > > more? I
> > > > > > > > > >> thought the
> > > > > > > > > >> >> > > > > > > delegation
> > > > > > > > > >> >> > > > > > > > > token is the hmac.
> > > > > > > > > >> >> > > > > > > > >
> > > > > > > > > >> >> > > > > > > > > Thanks,
> > > > > > > > > >> >> > > > > > > > >
> > > > > > > > > >> >> > > > > > > > > Jun
> > > > > > > > > >> >> > > > > > > > >
> > > > > > > > > >> >> > > > > > > > >
> > > > > > > > > >> >> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun
> > Rao
> > > <
> > > > > > > > > >> jun@confluent.io>
> > > > > > > > > >> >> > > > wrote:
> > > > > > > > > >> >> > > > > > > > >
> > > > > > > > > >> >> > > > > > > > > > Hi, Harsha,
> > > > > > > > > >> >> > > > > > > > > >
> > > > > > > > > >> >> > > > > > > > > > Just sent out a KIP meeting invite.
> > We
> > > > can
> > > > > > > > discuss
> > > > > > > > > >> this in
> > > > > > > > > >> >> > > the
> > > > > > > > > >> >> > > > > > > meeting
> > > > > > > > > >> >> > > > > > > > > > tomorrow.
> > > > > > > > > >> >> > > > > > > > > >
> > > > > > > > > >> >> > > > > > > > > > Thanks,
> > > > > > > > > >> >> > > > > > > > > >
> > > > > > > > > >> >> > > > > > > > > > Jun
> > > > > > > > > >> >> > > > > > > > > >
> > > > > > > > > >> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM,
> > > Harsha <
> > > > > > > > > >> kafka@harsha.io>
> > > > > > > > > >> >> > > > wrote:
> > > > > > > > > >> >> > > > > > > > > >
> > > > > > > > > >> >> > > > > > > > > >> Hi All,
> > > > > > > > > >> >> > > > > > > > > >>            Can we have a KIP
> meeting
> > > > > around
> > > > > > > > this.
> > > > > > > > > >> The KIP
> > > > > > > > > >> >> > is
> > > > > > > > > >> >> > > > up
> > > > > > > > > >> >> > > > > > for
> > > > > > > > > >> >> > > > > > > > > >>            sometime and if there
> are
> > > any
> > > > > > > > questions
> > > > > > > > > >> lets
> > > > > > > > > >> >> > > > quickly
> > > > > > > > > >> >> > > > > > hash
> > > > > > > > > >> >> > > > > > > > out
> > > > > > > > > >> >> > > > > > > > > >>            details.
> > > > > > > > > >> >> > > > > > > > > >>
> > > > > > > > > >> >> > > > > > > > > >> Thanks,
> > > > > > > > > >> >> > > > > > > > > >> Harsha
> > > > > > > > > >> >> > > > > > > > > >>
> > > > > > > > > >> >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40 AM,
> > > parth
> > > > > > > > > brahmbhatt
> > > > > > > > > >> wrote:
> > > > > > > > > >> >> > > > > > > > > >> > That is what the hadoop echo
> > system
> > > > uses
> > > > > > so
> > > > > > > no
> > > > > > > > > >> good
> > > > > > > > > >> >> > reason
> > > > > > > > > >> >> > > > > > really.
> > > > > > > > > >> >> > > > > > > > We
> > > > > > > > > >> >> > > > > > > > > >> > could
> > > > > > > > > >> >> > > > > > > > > >> > change it to whatever is the
> > newest
> > > > > > > > recommended
> > > > > > > > > >> standard
> > > > > > > > > >> >> > > is.
> > > > > > > > > >> >> > > > > > > > > >> >
> > > > > > > > > >> >> > > > > > > > > >> > Thanks
> > > > > > > > > >> >> > > > > > > > > >> > Parth
> > > > > > > > > >> >> > > > > > > > > >> >
> > > > > > > > > >> >> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM,
> > > > Ismael
> > > > > > > Juma <
> > > > > > > > > >> >> > > > > ismael@juma.me.uk
> > > > > > > > > >> >> > > > > > >
> > > > > > > > > >> >> > > > > > > > > wrote:
> > > > > > > > > >> >> > > > > > > > > >> >
> > > > > > > > > >> >> > > > > > > > > >> > > Hi Parth,
> > > > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > > > >> >> > > > > > > > > >> > > Thanks for the KIP. I only
> > started
> > > > > > > reviewing
> > > > > > > > > >> this and
> > > > > > > > > >> >> > > may
> > > > > > > > > >> >> > > > > have
> > > > > > > > > >> >> > > > > > > > > >> additional
> > > > > > > > > >> >> > > > > > > > > >> > > questions later. The immediate
> > > > > question
> > > > > > > that
> > > > > > > > > >> came to
> > > > > > > > > >> >> > > mind
> > > > > > > > > >> >> > > > is
> > > > > > > > > >> >> > > > > > our
> > > > > > > > > >> >> > > > > > > > > >> choice of
> > > > > > > > > >> >> > > > > > > > > >> > > "DIGEST-MD5" even though it's
> > > marked
> > > > > as
> > > > > > > > > >> OBSOLETE in
> > > > > > > > > >> >> > the
> > > > > > > > > >> >> > > > IANA
> > > > > > > > > >> >> > > > > > > > > Registry
> > > > > > > > > >> >> > > > > > > > > >> of
> > > > > > > > > >> >> > > > > > > > > >> > > SASL mechanisms and the
> original
> > > RFC
> > > > > > > (2831)
> > > > > > > > > has
> > > > > > > > > >> been
> > > > > > > > > >> >> > > moved
> > > > > > > > > >> >> > > > > to
> > > > > > > > > >> >> > > > > > > > > Historic
> > > > > > > > > >> >> > > > > > > > > >> > > status:
> > > > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > > > >> >> > > > > > > > > >> > > https://tools.ietf.org/html/
> > > rfc6331
> > > > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > > > >> >> > > > > > > > >
> > > > > > > > > >> >> > > > > >
> > > > > > > > > >> >> > >
> > > > > > > > > >>
> > > > > > >
> > > > http://www.iana.org/assignments/sasl-mechanisms/
> sasl-mechanisms.xhtml
> > > > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > > > >> >> > > > > > > > > >> > > What is the reasoning behind
> > that
> > > > > > choice?
> > > > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > > > >> >> > > > > > > > > >> > > Thanks,
> > > > > > > > > >> >> > > > > > > > > >> > > Ismael
> > > > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > > > >> >> > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29
> > PM,
> > > > Gwen
> > > > > > > > > Shapira <
> > > > > > > > > >> >> > > > > > > gwen@confluent.io
> > > > > > > > > >> >> > > > > > > > >
> > > > > > > > > >> >> > > > > > > > > >> wrote:
> > > > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > > > >> >> > > > > > > > > >> > > > Also comments inline :)
> > > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > > * I want to emphasize that
> > > even
> > > > > > though
> > > > > > > > > >> delegation
> > > > > > > > > >> >> > > > tokens
> > > > > > > > > >> >> > > > > > > are a
> > > > > > > > > >> >> > > > > > > > > >> Hadoop
> > > > > > > > > >> >> > > > > > > > > >> > > > > innovation, I feel very
> > > strongly
> > > > > > about
> > > > > > > > not
> > > > > > > > > >> adding
> > > > > > > > > >> >> > > > > > dependency
> > > > > > > > > >> >> > > > > > > > on
> > > > > > > > > >> >> > > > > > > > > >> Hadoop
> > > > > > > > > >> >> > > > > > > > > >> > > > > when implementing
> delegation
> > > > > tokens
> > > > > > > for
> > > > > > > > > >> Kafka. The
> > > > > > > > > >> >> > > KIP
> > > > > > > > > >> >> > > > > > > doesn't
> > > > > > > > > >> >> > > > > > > > > >> imply
> > > > > > > > > >> >> > > > > > > > > >> > > > > such dependency, but if
> you
> > > can
> > > > > > > > clarify...
> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > > *No hadoop dependency.*
> > > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > Yay! Just add this to the
> KIP
> > so
> > > > no
> > > > > > one
> > > > > > > > will
> > > > > > > > > >> read
> > > > > > > > > >> >> > the
> > > > > > > > > >> >> > > > KIP
> > > > > > > > > >> >> > > > > > and
> > > > > > > > > >> >> > > > > > > > > panic
> > > > > > > > > >> >> > > > > > > > > >> > > > three weeks before the next
> > > > > release...
> > > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > > * Can we get delegation
> > token
> > > at
> > > > > any
> > > > > > > > time
> > > > > > > > > >> after
> > > > > > > > > >> >> > > > > > > > authenticating?
> > > > > > > > > >> >> > > > > > > > > >> only
> > > > > > > > > >> >> > > > > > > > > >> > > > > immediately after?
> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > > *As long as you are
> > > > authenticated
> > > > > > you
> > > > > > > > can
> > > > > > > > > >> get
> > > > > > > > > >> >> > > > delegation
> > > > > > > > > >> >> > > > > > > > tokens.
> > > > > > > > > >> >> > > > > > > > > >> We
> > > > > > > > > >> >> > > > > > > > > >> > > need
> > > > > > > > > >> >> > > > > > > > > >> > > > to
> > > > > > > > > >> >> > > > > > > > > >> > > > > discuss if a client
> > > > authenticated
> > > > > > > using
> > > > > > > > > >> delegation
> > > > > > > > > >> >> > > > > token,
> > > > > > > > > >> >> > > > > > > can
> > > > > > > > > >> >> > > > > > > > > also
> > > > > > > > > >> >> > > > > > > > > >> > > > acquire
> > > > > > > > > >> >> > > > > > > > > >> > > > > delegation token again or
> > not.
> > > > > Also
> > > > > > > > there
> > > > > > > > > >> is the
> > > > > > > > > >> >> > > > > question
> > > > > > > > > >> >> > > > > > of
> > > > > > > > > >> >> > > > > > > > do
> > > > > > > > > >> >> > > > > > > > > we
> > > > > > > > > >> >> > > > > > > > > >> > > allow
> > > > > > > > > >> >> > > > > > > > > >> > > > > anyone to acquire
> delegation
> > > > token
> > > > > > or
> > > > > > > we
> > > > > > > > > >> want
> > > > > > > > > >> >> > > specific
> > > > > > > > > >> >> > > > > > ACLs
> > > > > > > > > >> >> > > > > > > (I
> > > > > > > > > >> >> > > > > > > > > >> think
> > > > > > > > > >> >> > > > > > > > > >> > > its
> > > > > > > > > >> >> > > > > > > > > >> > > > an
> > > > > > > > > >> >> > > > > > > > > >> > > > > overkill.)*
> > > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > I agree that ACLs is an
> > > overkill.
> > > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > I think we are debating two
> > > > options:
> > > > > > > > Either
> > > > > > > > > >> require
> > > > > > > > > >> >> > > > > Kerberos
> > > > > > > > > >> >> > > > > > > > auth
> > > > > > > > > >> >> > > > > > > > > >> for
> > > > > > > > > >> >> > > > > > > > > >> > > > renewal or require
> non-owners
> > to
> > > > > > renew.
> > > > > > > > > >> >> > > > > > > > > >> > > > I *think* the latter is
> > simpler
> > > > (it
> > > > > > > > > basically
> > > > > > > > > >> >> > require
> > > > > > > > > >> >> > > a
> > > > > > > > > >> >> > > > > "job
> > > > > > > > > >> >> > > > > > > > > master"
> > > > > > > > > >> >> > > > > > > > > >> > > > to take responsibility for
> the
> > > > > > renewal,
> > > > > > > it
> > > > > > > > > >> will have
> > > > > > > > > >> >> > > its
> > > > > > > > > >> >> > > > > own
> > > > > > > > > >> >> > > > > > > > > >> identity
> > > > > > > > > >> >> > > > > > > > > >> > > > anyway and I think this is
> the
> > > > > correct
> > > > > > > > > design
> > > > > > > > > >> >> > pattern
> > > > > > > > > >> >> > > > > > anyway.
> > > > > > > > > >> >> > > > > > > > For
> > > > > > > > > >> >> > > > > > > > > >> > > > storm, I'd expect Nimbus to
> > > > > coordinate
> > > > > > > > > >> renewals?),
> > > > > > > > > >> >> > but
> > > > > > > > > >> >> > > > it
> > > > > > > > > >> >> > > > > is
> > > > > > > > > >> >> > > > > > > > hard
> > > > > > > > > >> >> > > > > > > > > to
> > > > > > > > > >> >> > > > > > > > > >> > > > debate simplicity without
> > > looking
> > > > at
> > > > > > the
> > > > > > > > > code
> > > > > > > > > >> >> > changes
> > > > > > > > > >> >> > > > > > > required.
> > > > > > > > > >> >> > > > > > > > If
> > > > > > > > > >> >> > > > > > > > > >> you
> > > > > > > > > >> >> > > > > > > > > >> > > > have a draft of how the
> > "require
> > > > > > > Kerberos"
> > > > > > > > > >> will look
> > > > > > > > > >> >> > > in
> > > > > > > > > >> >> > > > > > Kafka
> > > > > > > > > >> >> > > > > > > > > code,
> > > > > > > > > >> >> > > > > > > > > >> > > > I'll be happy to take a
> look.
> > > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > > * My understanding is that
> > > > tokens
> > > > > > will
> > > > > > > > > >> propagate
> > > > > > > > > >> >> > via
> > > > > > > > > >> >> > > > ZK
> > > > > > > > > >> >> > > > > > but
> > > > > > > > > >> >> > > > > > > > > >> without
> > > > > > > > > >> >> > > > > > > > > >> > > > > additional changes to
> > > > > UpdateMetadata
> > > > > > > > > >> protocol,
> > > > > > > > > >> >> > > > correct?
> > > > > > > > > >> >> > > > > > > > Clients
> > > > > > > > > >> >> > > > > > > > > >> > > > > currently don't retry on
> > SASL
> > > > auth
> > > > > > > > failure
> > > > > > > > > >> (IIRC),
> > > > > > > > > >> >> > > but
> > > > > > > > > >> >> > > > > > since
> > > > > > > > > >> >> > > > > > > > the
> > > > > > > > > >> >> > > > > > > > > >> > > > > tokens propagate between
> > > brokers
> > > > > > > asynch,
> > > > > > > > > we
> > > > > > > > > >> will
> > > > > > > > > >> >> > > need
> > > > > > > > > >> >> > > > to
> > > > > > > > > >> >> > > > > > > > retry a
> > > > > > > > > >> >> > > > > > > > > >> bit
> > > > > > > > > >> >> > > > > > > > > >> > > > > to avoid clients failing
> > auth
> > > > due
> > > > > to
> > > > > > > > > timing
> > > > > > > > > >> >> > issues.
> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > > *I am considering 2
> > > alternatives
> > > > > > right
> > > > > > > > > now.
> > > > > > > > > >> The
> > > > > > > > > >> >> > > > current
> > > > > > > > > >> >> > > > > > > > > documented
> > > > > > > > > >> >> > > > > > > > > >> > > > approach
> > > > > > > > > >> >> > > > > > > > > >> > > > > is zookeeper based and it
> > does
> > > > not
> > > > > > > > require
> > > > > > > > > >> any
> > > > > > > > > >> >> > > changes
> > > > > > > > > >> >> > > > > to
> > > > > > > > > >> >> > > > > > > > > >> > > UpdateMetadata
> > > > > > > > > >> >> > > > > > > > > >> > > > > protocol. An alternative
> > > > approach
> > > > > > can
> > > > > > > > > remove
> > > > > > > > > >> >> > > zookeeper
> > > > > > > > > >> >> > > > > > > > > dependency
> > > > > > > > > >> >> > > > > > > > > >> as
> > > > > > > > > >> >> > > > > > > > > >> > > well
> > > > > > > > > >> >> > > > > > > > > >> > > > > but we can discuss that in
> > KIP
> > > > > > > > discussion
> > > > > > > > > >> call.*
> > > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > Oooh! Sounds interesting. Do
> > you
> > > > > want
> > > > > > to
> > > > > > > > > ping
> > > > > > > > > >> Jun to
> > > > > > > > > >> >> > > > > > arrange a
> > > > > > > > > >> >> > > > > > > > > call?
> > > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > > * I liked Ashish's
> > suggestion
> > > of
> > > > > > > having
> > > > > > > > > >> just the
> > > > > > > > > >> >> > > > > > controller
> > > > > > > > > >> >> > > > > > > > > issue
> > > > > > > > > >> >> > > > > > > > > >> the
> > > > > > > > > >> >> > > > > > > > > >> > > > > delegation tokens, to
> avoid
> > > > > syncing
> > > > > > a
> > > > > > > > > shared
> > > > > > > > > >> >> > secret.
> > > > > > > > > >> >> > > > Not
> > > > > > > > > >> >> > > > > > > sure
> > > > > > > > > >> >> > > > > > > > if
> > > > > > > > > >> >> > > > > > > > > >> we
> > > > > > > > > >> >> > > > > > > > > >> > > > > want to continue the
> > > discussion
> > > > > here
> > > > > > > or
> > > > > > > > on
> > > > > > > > > >> the
> > > > > > > > > >> >> > > wiki. I
> > > > > > > > > >> >> > > > > > think
> > > > > > > > > >> >> > > > > > > > > that
> > > > > > > > > >> >> > > > > > > > > >> we
> > > > > > > > > >> >> > > > > > > > > >> > > > > can decouple the problem
> of
> > > > "token
> > > > > > > > > >> distribution"
> > > > > > > > > >> >> > > from
> > > > > > > > > >> >> > > > > > > "shared
> > > > > > > > > >> >> > > > > > > > > >> secret
> > > > > > > > > >> >> > > > > > > > > >> > > > > distribution" and use the
> > > > > controller
> > > > > > > as
> > > > > > > > > the
> > > > > > > > > >> only
> > > > > > > > > >> >> > > token
> > > > > > > > > >> >> > > > > > > > generator
> > > > > > > > > >> >> > > > > > > > > >> to
> > > > > > > > > >> >> > > > > > > > > >> > > > > solve the second issue,
> > while
> > > > > still
> > > > > > > > using
> > > > > > > > > >> ZK async
> > > > > > > > > >> >> > > to
> > > > > > > > > >> >> > > > > > > > distribute
> > > > > > > > > >> >> > > > > > > > > >> > > > > tokens.
> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > > *As mentioned in the
> > previous
> > > > > Email
> > > > > > I
> > > > > > > am
> > > > > > > > > >> fine with
> > > > > > > > > >> >> > > > that
> > > > > > > > > >> >> > > > > > > > approach
> > > > > > > > > >> >> > > > > > > > > >> as
> > > > > > > > > >> >> > > > > > > > > >> > > long
> > > > > > > > > >> >> > > > > > > > > >> > > > as
> > > > > > > > > >> >> > > > > > > > > >> > > > > we agree that the extra
> > > > complexity
> > > > > > of
> > > > > > > > > >> >> > > adding/updating
> > > > > > > > > >> >> > > > > APIS
> > > > > > > > > >> >> > > > > > > > adds
> > > > > > > > > >> >> > > > > > > > > >> enough
> > > > > > > > > >> >> > > > > > > > > >> > > > > value. The advantage with
> > the
> > > > > > > controller
> > > > > > > > > >> approach
> > > > > > > > > >> >> > is
> > > > > > > > > >> >> > > > > > secret
> > > > > > > > > >> >> > > > > > > > > >> rotation
> > > > > > > > > >> >> > > > > > > > > >> > > can
> > > > > > > > > >> >> > > > > > > > > >> > > > be
> > > > > > > > > >> >> > > > > > > > > >> > > > > automated,frequent and
> would
> > > not
> > > > > > > require
> > > > > > > > > >> >> > > deployment. *
> > > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > Can you detail the extra
> > > > complexity
> > > > > > (or
> > > > > > > > > point
> > > > > > > > > >> me to
> > > > > > > > > >> >> > > the
> > > > > > > > > >> >> > > > > > email
> > > > > > > > > >> >> > > > > > > I
> > > > > > > > > >> >> > > > > > > > > >> > > > missed?) - which Apis are
> > > > required?
> > > > > > > > > >> >> > > > > > > > > >> > > > As far as I can tell,
> clients
> > > can
> > > > > > > already
> > > > > > > > > >> find the
> > > > > > > > > >> >> > > > > > controller
> > > > > > > > > >> >> > > > > > > > from
> > > > > > > > > >> >> > > > > > > > > >> > > > metadata. I'm a bit more
> > > concerned
> > > > > > about
> > > > > > > > > >> controller
> > > > > > > > > >> >> > > > load.
> > > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > > * While I like the idea of
> > > > forcing
> > > > > > > > > kerberos
> > > > > > > > > >> auth
> > > > > > > > > >> >> > for
> > > > > > > > > >> >> > > > > > > renwal, I
> > > > > > > > > >> >> > > > > > > > > >> think
> > > > > > > > > >> >> > > > > > > > > >> > > > > it mixes the transport
> layer
> > > the
> > > > > the
> > > > > > > > > request
> > > > > > > > > >> >> > content
> > > > > > > > > >> >> > > > in
> > > > > > > > > >> >> > > > > a
> > > > > > > > > >> >> > > > > > > > pretty
> > > > > > > > > >> >> > > > > > > > > >> ugly
> > > > > > > > > >> >> > > > > > > > > >> > > > > way. Perhaps limiting
> > renewer
> > > to
> > > > > > > > non-owner
> > > > > > > > > >> is
> > > > > > > > > >> >> > > better.
> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > > *I feel this is a
> necessary
> > > > evil.
> > > > > > > While
> > > > > > > > > >> this will
> > > > > > > > > >> >> > > make
> > > > > > > > > >> >> > > > > the
> > > > > > > > > >> >> > > > > > > > kafka
> > > > > > > > > >> >> > > > > > > > > >> code
> > > > > > > > > >> >> > > > > > > > > >> > > > > pretty straight forward ,
> > > > forcing
> > > > > > > > renewer
> > > > > > > > > >> to
> > > > > > > > > >> >> > > > non-owner
> > > > > > > > > >> >> > > > > > > pushes
> > > > > > > > > >> >> > > > > > > > > >> the code
> > > > > > > > > >> >> > > > > > > > > >> > > > > ugliness to client and
> makes
> > > it
> > > > > even
> > > > > > > > > harder
> > > > > > > > > >> to
> > > > > > > > > >> >> > > > > > integrate.  *
> > > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > As mentioned before, I don't
> > > think
> > > > > the
> > > > > > > > > >> "renewal by
> > > > > > > > > >> >> > > > other"
> > > > > > > > > >> >> > > > > > > > approach
> > > > > > > > > >> >> > > > > > > > > >> is
> > > > > > > > > >> >> > > > > > > > > >> > > > that ugly for the clients we
> > > > expect
> > > > > to
> > > > > > > use
> > > > > > > > > >> >> > delegation
> > > > > > > > > >> >> > > > > tokens
> > > > > > > > > >> >> > > > > > > > since
> > > > > > > > > >> >> > > > > > > > > >> > > > they will have an app-master
> > of
> > > > some
> > > > > > > sort
> > > > > > > > > who
> > > > > > > > > >> >> > > requested
> > > > > > > > > >> >> > > > > the
> > > > > > > > > >> >> > > > > > > > token
> > > > > > > > > >> >> > > > > > > > > to
> > > > > > > > > >> >> > > > > > > > > >> > > > begin with.
> > > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > > The response for my
> question
> > > on
> > > > > how
> > > > > > > > > multiple
> > > > > > > > > >> >> > > > identities
> > > > > > > > > >> >> > > > > > will
> > > > > > > > > >> >> > > > > > > > be
> > > > > > > > > >> >> > > > > > > > > >> > > > > handled wasn't super clear
> > to
> > > > me -
> > > > > > > > AFAIK,
> > > > > > > > > >> we don't
> > > > > > > > > >> >> > > > > > > > authenticate
> > > > > > > > > >> >> > > > > > > > > >> each
> > > > > > > > > >> >> > > > > > > > > >> > > > > request, we authenticate
> > > > > > connections.
> > > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > > *We authenticate
> > connections,
> > > > and
> > > > > > only
> > > > > > > > > when
> > > > > > > > > >> they
> > > > > > > > > >> >> > are
> > > > > > > > > >> >> > > > > being
> > > > > > > > > >> >> > > > > > > > > >> established.
> > > > > > > > > >> >> > > > > > > > > >> > > > Let
> > > > > > > > > >> >> > > > > > > > > >> > > > > me try to phrase this as a
> > > > > question,
> > > > > > > in
> > > > > > > > > >> absence of
> > > > > > > > > >> >> > > > > > > delegation
> > > > > > > > > >> >> > > > > > > > > >> tokens if
> > > > > > > > > >> >> > > > > > > > > >> > > > we
> > > > > > > > > >> >> > > > > > > > > >> > > > > had to support the use
> case
> > > > using
> > > > > > user
> > > > > > > > > >> TGT's how
> > > > > > > > > >> >> > > would
> > > > > > > > > >> >> > > > > we
> > > > > > > > > >> >> > > > > > do
> > > > > > > > > >> >> > > > > > > > it?
> > > > > > > > > >> >> > > > > > > > > >> My
> > > > > > > > > >> >> > > > > > > > > >> > > point
> > > > > > > > > >> >> > > > > > > > > >> > > > > was it would be no
> different
> > > > with
> > > > > > > > > delegation
> > > > > > > > > >> >> > tokens.
> > > > > > > > > >> >> > > > The
> > > > > > > > > >> >> > > > > > use
> > > > > > > > > >> >> > > > > > > > > case
> > > > > > > > > >> >> > > > > > > > > >> you
> > > > > > > > > >> >> > > > > > > > > >> > > are
> > > > > > > > > >> >> > > > > > > > > >> > > > > describing seems more like
> > > > > > > > impersonation.*
> > > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > > >> >> > > > > > > > > >> > > > Yeah, I thought that one of
> > the
> > > > > things
> > > > > > > > that
> > > > > > > > > >> >> > del
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Harsha Chintalapani <ka...@harsha.io>]

Ashish,
        Yes we are working on it. Lets discuss in the next KIP meeting.
I'll join.
-Harsha

On Tue, Aug 23, 2016 at 12:07 PM Ashish Singh <as...@cloudera.com> wrote:

> Hello Harsha,
>
> Are you still working on this? Wondering if we can discuss this in next KIP
> meeting, if you can join.
>
> On Mon, Jul 18, 2016 at 9:51 AM, Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> > Hi Grant,
> >           We are working on it. Will add the details to KIP about the
> > request protocol.
> >
> > Thanks,
> > Harsha
> >
> > On Mon, Jul 18, 2016 at 6:50 AM Grant Henke <gh...@cloudera.com> wrote:
> >
> > > Hi Parth,
> > >
> > > Are you still working on this? If you need any help please don't
> hesitate
> > > to ask.
> > >
> > > Thanks,
> > > Grant
> > >
> > > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Parth,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > It makes sense to only allow the renewal by users that authenticated
> > > using
> > > > *non* delegation token mechanism. Then, should we make the renewal a
> > > list?
> > > > For example, in the case of rest proxy, it will be useful for every
> > > > instance of rest proxy to be able to renew the tokens.
> > > >
> > > > It would be clearer if we can document the request protocol like
> > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 4+-+Command+line+and+centralized+administrative+operations#KIP-4-
> > Commandlineandcentralizedadministrativeoperations-
> > CreateTopicsRequest(KAFKA-2945):(VotedandPlannedforin0.10.1.0)
> > > > .
> > > >
> > > > It would also be useful to document the client APIs.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Jun 28, 2016 at 2:55 PM, parth brahmbhatt <
> > > > brahmbhatt.parth@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I am suggesting that we will only allow the renewal by users that
> > > > > authenticated using *non* delegation token mechanism. For example,
> If
> > > > user
> > > > > Alice authenticated using kerberos and requested delegation tokens,
> > > only
> > > > > user Alice authenticated via non delegation token mechanism can
> > renew.
> > > > > Clients that have  access to delegation tokens can not issue
> renewal
> > > > > request for renewing their own token and this is primarily
> important
> > to
> > > > > reduce the time window for which a compromised token will be valid.
> > > > >
> > > > > To clarify, Yes any authenticated user can request delegation
> tokens
> > > but
> > > > > even here I would recommend to avoid creating a chain where a
> client
> > > > > authenticated via delegation token request for more delegation
> > tokens.
> > > > > Basically anyone can request delegation token, as long as they
> > > > authenticate
> > > > > via a non delegation token mechanism.
> > > > >
> > > > > Aren't classes listed here
> > > > > <
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKaf
> > ka-PublicInterfaces
> > > > > >
> > > > > sufficient?
> > > > >
> > > > > Thanks
> > > > > Parth
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao <ju...@confluent.io> wrote:
> > > > >
> > > > > > Parth,
> > > > > >
> > > > > > Thanks for the reply. A couple of comments inline below.
> > > > > >
> > > > > > On Tue, Jun 21, 2016 at 10:36 AM, parth brahmbhatt <
> > > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > > >
> > > > > > > 1. Who / how are tokens renewed? By original requester only? or
> > > using
> > > > > > > Kerberos
> > > > > > > auth only?
> > > > > > > My recommendation is to do this only using Kerberos auth and
> only
> > > > threw
> > > > > > the
> > > > > > > renewer specified during the acquisition request.
> > > > > > >
> > > > > > >
> > > > > > Hmm, not sure that I follow this. Are you saying that any client
> > > > > > authenticated with the delegation token can renew, i.e. there is
> no
> > > > > renewer
> > > > > > needed?
> > > > > >
> > > > > > Also, just to be clear, any authenticated client (either through
> > SASL
> > > > or
> > > > > > SSL) can request a delegation token for the authenticated user,
> > > right?
> > > > > >
> > > > > >
> > > > > > > 2. Are tokens stored on each broker or in ZK?
> > > > > > > My recommendation is still to store in ZK or not store them at
> > all.
> > > > The
> > > > > > > whole controller based distribution is too much overhead with
> not
> > > > much
> > > > > to
> > > > > > > achieve.
> > > > > > >
> > > > > > > 3. How are tokens invalidated / expired?
> > > > > > > Either by expiration time out or through an explicit request to
> > > > > > invalidate.
> > > > > > >
> > > > > > > 4. Which encryption algorithm is used?
> > > > > > > SCRAM
> > > > > > >
> > > > > > > 5. What is the impersonation proposal (it wasn't in the KIP but
> > was
> > > > > > > discussed
> > > > > > > in this thread)?
> > > > > > > There is no imperonation proposal. I tried and explained how
> its
> > a
> > > > > > > different problem and why its not really necessary to discuss
> > that
> > > as
> > > > > > part
> > > > > > > of this KIP.  This KIP will not support any impersonation, it
> > will
> > > > just
> > > > > > be
> > > > > > > another way to authenticate.
> > > > > > >
> > > > > > > 6. Do we need new ACLs, if so - for what actions?
> > > > > > > We do not need new ACLs.
> > > > > > >
> > > > > > >
> > > > > > Could we document the format of the new request/response and
> their
> > > > > > associated Resource and Operation for ACL?
> > > > > >
> > > > > >
> > > > > > > 7. How would the delegation token be configured in the client?
> > > > > > > Should be through config. I wasn't planning on supporting JAAS
> > for
> > > > > > tokens.
> > > > > > > I don't believe hadoop does this either.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Parth
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <ju...@confluent.io>
> > wrote:
> > > > > > >
> > > > > > > > Harsha,
> > > > > > > >
> > > > > > > > Another question.
> > > > > > > >
> > > > > > > > 9. How would the delegation token be configured in the
> client?
> > > The
> > > > > > > standard
> > > > > > > > way is to do this through JAAS. However, we will need to
> think
> > > > > through
> > > > > > if
> > > > > > > > this is convenient in a shared environment. For example,
> when a
> > > new
> > > > > > task
> > > > > > > is
> > > > > > > > added to a Storm worker node, do we need to dynamically add a
> > new
> > > > > > section
> > > > > > > > in the JAAS file? It may be more convenient if we can pass in
> > the
> > > > > token
> > > > > > > > through the config directly w/o going through JAAS.
> > > > > > > >
> > > > > > > > Are you or Parth still actively working on this KIP?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <ju...@confluent.io>
> > > wrote:
> > > > > > > >
> > > > > > > > > Just to add on that list.
> > > > > > > > >
> > > > > > > > > 2. It would be good to document the format of the data
> stored
> > > in
> > > > > ZK.
> > > > > > > > > 7. Earlier, there was a discussion on whether the tokens
> > should
> > > > be
> > > > > > > > > propagated through ZK like config/acl/quota, or through the
> > > > > > controller.
> > > > > > > > > Currently, the controller is only designed for propagating
> > > topic
> > > > > > > > metadata,
> > > > > > > > > but not other data.
> > > > > > > > > 8. Should we use SCRAM to send the token instead of
> > DIGEST-MD5
> > > > > since
> > > > > > > it's
> > > > > > > > > deprecated?
> > > > > > > > >
> > > > > > > > > Also, the images in the wiki seem broken.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <
> > > > gwen@confluent.io>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> From what I can see, remaining questions are:
> > > > > > > > >>
> > > > > > > > >> 1. Who / how are tokens renewed? By original requester
> only?
> > > or
> > > > > > using
> > > > > > > > >> Kerberos auth only?
> > > > > > > > >> 2. Are tokens stored on each broker or in ZK?
> > > > > > > > >> 3. How are tokens invalidated / expired?
> > > > > > > > >> 4. Which encryption algorithm is used?
> > > > > > > > >> 5. What is the impersonation proposal (it wasn't in the
> KIP
> > > but
> > > > > was
> > > > > > > > >> discussed in this thread)?
> > > > > > > > >> 6. Do we need new ACLs, if so - for what actions?
> > > > > > > > >>
> > > > > > > > >> Gwen
> > > > > > > > >>
> > > > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <ka...@harsha.io>
> > > wrote:
> > > > > > > > >> > Jun & Ismael,
> > > > > > > > >> >                          Unfortunately I couldn't attend
> > the
> > > > KIP
> > > > > > > > meeting
> > > > > > > > >> >                          when delegation tokens
> discussed.
> > > > > > > Appreciate
> > > > > > > > if
> > > > > > > > >> >                          you can update the thread if
> you
> > > have
> > > > > any
> > > > > > > > >> >                          further questions.
> > > > > > > > >> > Thanks,
> > > > > > > > >> > Harsha
> > > > > > > > >> >
> > > > > > > > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
> > > > > > > > >> >> It seems that the links to images in the KIP are
> broken.
> > > > > > > > >> >>
> > > > > > > > >> >> Liquan
> > > > > > > > >> >>
> > > > > > > > >> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
> > > > > > > > >> >> brahmbhatt.parth@gmail.com> wrote:
> > > > > > > > >> >>
> > > > > > > > >> >> > 110. What does getDelegationTokenAs mean?
> > > > > > > > >> >> > In the current proposal we only allow a user to get
> > > > > delegation
> > > > > > > > token
> > > > > > > > >> for
> > > > > > > > >> >> > the identity that it authenticated as using another
> > > > > mechanism,
> > > > > > > i.e.
> > > > > > > > >> A user
> > > > > > > > >> >> > that authenticate using a keytab for principal
> > > > > > user1@EXAMPLE.COM
> > > > > > > > >> will get
> > > > > > > > >> >> > delegation tokens for that user only. In future I
> think
> > > we
> > > > > will
> > > > > > > > have
> > > > > > > > >> to
> > > > > > > > >> >> > extend support such that we allow some set of users (
> > > > > > > > >> >> > kafka-rest-user@EXAMPLE.COM,
> storm-nimbus@EXAMPLE.COM)
> > > to
> > > > > > > acquire
> > > > > > > > >> >> > delegation tokens on behalf of other users whose
> > identity
> > > > > they
> > > > > > > have
> > > > > > > > >> >> > verified independently.  Kafka brokers will have ACLs
> > to
> > > > > > control
> > > > > > > > >> which
> > > > > > > > >> >> > users are allowed to impersonate other users and get
> > > tokens
> > > > > on
> > > > > > > > >> behalf of
> > > > > > > > >> >> > them. Overall Impersonation is a whole different
> > problem
> > > in
> > > > > my
> > > > > > > > >> opinion and
> > > > > > > > >> >> > I think we can tackle it in separate KIP.
> > > > > > > > >> >> >
> > > > > > > > >> >> > 111. What's the typical rate of getting and renewing
> > > > > delegation
> > > > > > > > >> tokens?
> > > > > > > > >> >> > Typically this should be very very low, 1 request per
> > > > minute
> > > > > > is a
> > > > > > > > >> >> > relatively high estimate. However it depends on the
> > token
> > > > > > > > >> expiration. I am
> > > > > > > > >> >> > less worried about the extra load it puts on
> controller
> > > vs
> > > > > the
> > > > > > > > added
> > > > > > > > >> >> > complexity and the value it offers.
> > > > > > > > >> >> >
> > > > > > > > >> >> > Thanks
> > > > > > > > >> >> > Parth
> > > > > > > > >> >> >
> > > > > > > > >> >> >
> > > > > > > > >> >> >
> > > > > > > > >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <
> > > > > > ismael@juma.me.uk>
> > > > > > > > >> wrote:
> > > > > > > > >> >> >
> > > > > > > > >> >> > > Thanks Rajini. It would probably require a separate
> > KIP
> > > > as
> > > > > it
> > > > > > > > will
> > > > > > > > >> >> > > introduce user visible changes. We could also
> update
> > > > KIP-48
> > > > > > to
> > > > > > > > >> have this
> > > > > > > > >> >> > > information, but it seems cleaner to do it
> > separately.
> > > We
> > > > > can
> > > > > > > > >> discuss
> > > > > > > > >> >> > that
> > > > > > > > >> >> > > in the KIP call today.
> > > > > > > > >> >> > >
> > > > > > > > >> >> > > Ismael
> > > > > > > > >> >> > >
> > > > > > > > >> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <
> > > > > > > > >> >> > > rajinisivaram@googlemail.com> wrote:
> > > > > > > > >> >> > >
> > > > > > > > >> >> > > > Ismael,
> > > > > > > > >> >> > > >
> > > > > > > > >> >> > > > I have created a JIRA (
> > > > > > > > >> >> > https://issues.apache.org/jira/browse/KAFKA-3751)
> > > > > > > > >> >> > > > for adding SCRAM as a SASL mechanism. Would that
> > need
> > > > > > another
> > > > > > > > >> KIP? If
> > > > > > > > >> >> > > > KIP-48 will use this mechanism, can this just be
> a
> > > JIRA
> > > > > > that
> > > > > > > > gets
> > > > > > > > >> >> > > reviewed
> > > > > > > > >> >> > > > when the PR is ready?
> > > > > > > > >> >> > > >
> > > > > > > > >> >> > > > Thank you,
> > > > > > > > >> >> > > >
> > > > > > > > >> >> > > > Rajini
> > > > > > > > >> >> > > >
> > > > > > > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <
> > > > > > > > ismael@juma.me.uk>
> > > > > > > > >> >> > wrote:
> > > > > > > > >> >> > > >
> > > > > > > > >> >> > > > > Thanks Rajini, SCRAM seems like a good
> candidate.
> > > > > > > > >> >> > > > >
> > > > > > > > >> >> > > > > Gwen had independently mentioned this as a SASL
> > > > > mechanism
> > > > > > > > that
> > > > > > > > >> might
> > > > > > > > >> >> > be
> > > > > > > > >> >> > > > > useful for Kafka and I have been meaning to
> check
> > > it
> > > > in
> > > > > > > more
> > > > > > > > >> detail.
> > > > > > > > >> >> > > Good
> > > > > > > > >> >> > > > > to know that you are willing to contribute an
> > > > > > > implementation.
> > > > > > > > >> Maybe
> > > > > > > > >> >> > we
> > > > > > > > >> >> > > > > should file a separate JIRA for this?
> > > > > > > > >> >> > > > >
> > > > > > > > >> >> > > > > Ismael
> > > > > > > > >> >> > > > >
> > > > > > > > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini
> Sivaram <
> > > > > > > > >> >> > > > > rajinisivaram@googlemail.com> wrote:
> > > > > > > > >> >> > > > >
> > > > > > > > >> >> > > > > > SCRAM (Salted Challenge Response
> Authentication
> > > > > > > Mechanism)
> > > > > > > > >> is a
> > > > > > > > >> >> > > better
> > > > > > > > >> >> > > > > > mechanism than Digest-MD5. Java doesn't come
> > > with a
> > > > > > > > built-in
> > > > > > > > >> SCRAM
> > > > > > > > >> >> > > > > > SaslServer or SaslClient, but I will be happy
> > to
> > > > add
> > > > > > > > support
> > > > > > > > >> in
> > > > > > > > >> >> > Kafka
> > > > > > > > >> >> > > > > since
> > > > > > > > >> >> > > > > > it would be a useful mechanism to support
> > anyway.
> > > > > > > > >> >> > > > > > https://tools.ietf.org/html/rfc7677
> describes
> > > the
> > > > > > > protocol
> > > > > > > > >> for
> > > > > > > > >> >> > > > > > SCRAM-SHA-256.
> > > > > > > > >> >> > > > > >
> > > > > > > > >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <
> > > > > > > jun@confluent.io
> > > > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >> >> > > > > >
> > > > > > > > >> >> > > > > > > Parth,
> > > > > > > > >> >> > > > > > >
> > > > > > > > >> >> > > > > > > Thanks for the explanation. A couple of
> more
> > > > > > questions.
> > > > > > > > >> >> > > > > > >
> > > > > > > > >> >> > > > > > > 110. What does getDelegationTokenAs mean?
> > > > > > > > >> >> > > > > > >
> > > > > > > > >> >> > > > > > > 111. What's the typical rate of getting and
> > > > > renewing
> > > > > > > > >> delegation
> > > > > > > > >> >> > > > tokens?
> > > > > > > > >> >> > > > > > > That may have an impact on whether they
> > should
> > > be
> > > > > > > > directed
> > > > > > > > >> to the
> > > > > > > > >> >> > > > > > > controller.
> > > > > > > > >> >> > > > > > >
> > > > > > > > >> >> > > > > > > Jun
> > > > > > > > >> >> > > > > > >
> > > > > > > > >> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth
> > > > brahmbhatt <
> > > > > > > > >> >> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > > > > > >> >> > > > > > >
> > > > > > > > >> >> > > > > > > > Hi Jun,
> > > > > > > > >> >> > > > > > > >
> > > > > > > > >> >> > > > > > > > Thanks for reviewing.
> > > > > > > > >> >> > > > > > > >
> > > > > > > > >> >> > > > > > > > * We could add a Cluster action to add
> acls
> > > on
> > > > > who
> > > > > > > can
> > > > > > > > >> request
> > > > > > > > >> >> > > > > > delegation
> > > > > > > > >> >> > > > > > > > tokens. I don't see the use case for that
> > yet
> > > > but
> > > > > > > down
> > > > > > > > >> the line
> > > > > > > > >> >> > > > when
> > > > > > > > >> >> > > > > we
> > > > > > > > >> >> > > > > > > > start supporting getDelegationTokenAs it
> > will
> > > > be
> > > > > > > > >> necessary.
> > > > > > > > >> >> > > > > > > > * Yes we recommend tokens to be only
> > > > > > used/distributed
> > > > > > > > >> over
> > > > > > > > >> >> > secure
> > > > > > > > >> >> > > > > > > channels.
> > > > > > > > >> >> > > > > > > > * Depending on what design we end up
> > choosing
> > > > > > > > >> Invalidation will
> > > > > > > > >> >> > > be
> > > > > > > > >> >> > > > > > > > responsibility of every broker or
> > controller.
> > > > > > > > >> >> > > > > > > > * I am not sure if I documented somewhere
> > > that
> > > > > > > > >> invalidation
> > > > > > > > >> >> > will
> > > > > > > > >> >> > > > > > directly
> > > > > > > > >> >> > > > > > > > go through zookeeper but that is not the
> > > > intent.
> > > > > > > > >> Invalidation
> > > > > > > > >> >> > > will
> > > > > > > > >> >> > > > > > either
> > > > > > > > >> >> > > > > > > > be request based or due to expiration. No
> > > > direct
> > > > > > > > >> zookeeper
> > > > > > > > >> >> > > > > interaction
> > > > > > > > >> >> > > > > > > from
> > > > > > > > >> >> > > > > > > > any client.
> > > > > > > > >> >> > > > > > > > * "Broker also stores the DelegationToken
> > > > without
> > > > > > the
> > > > > > > > >> hmac in
> > > > > > > > >> >> > the
> > > > > > > > >> >> > > > > > > > zookeeper." : Sorry about the confusion.
> > The
> > > > sole
> > > > > > > > >> purpose of
> > > > > > > > >> >> > > > > zookeeper
> > > > > > > > >> >> > > > > > in
> > > > > > > > >> >> > > > > > > > this design is as distribution channel
> for
> > > > tokens
> > > > > > > > >> between all
> > > > > > > > >> >> > > > brokers
> > > > > > > > >> >> > > > > > > and a
> > > > > > > > >> >> > > > > > > > layer that ensures only tokens that were
> > > > > generated
> > > > > > by
> > > > > > > > >> making a
> > > > > > > > >> >> > > > > request
> > > > > > > > >> >> > > > > > > to a
> > > > > > > > >> >> > > > > > > > broker will be accepted (more on this in
> > > second
> > > > > > > > >> paragraph). The
> > > > > > > > >> >> > > > token
> > > > > > > > >> >> > > > > > > > consists of few elements (owner, renewer,
> > > uuid
> > > > ,
> > > > > > > > >> expiration,
> > > > > > > > >> >> > > hmac)
> > > > > > > > >> >> > > > ,
> > > > > > > > >> >> > > > > > one
> > > > > > > > >> >> > > > > > > of
> > > > > > > > >> >> > > > > > > > which is the finally generated hmac but
> > hmac
> > > it
> > > > > > self
> > > > > > > is
> > > > > > > > >> >> > derivable
> > > > > > > > >> >> > > > if
> > > > > > > > >> >> > > > > > you
> > > > > > > > >> >> > > > > > > > have all the other elements of the token
> +
> > > > secret
> > > > > > key
> > > > > > > > to
> > > > > > > > >> >> > generate
> > > > > > > > >> >> > > > > hmac.
> > > > > > > > >> >> > > > > > > > Given zookeeper does not provide SSL
> > support
> > > we
> > > > > do
> > > > > > > not
> > > > > > > > >> want the
> > > > > > > > >> >> > > > > entire
> > > > > > > > >> >> > > > > > > > token to be wire transferred to zookeeper
> > as
> > > > that
> > > > > > > will
> > > > > > > > >> be an
> > > > > > > > >> >> > > > insecure
> > > > > > > > >> >> > > > > > > wire
> > > > > > > > >> >> > > > > > > > transfer. Instead we only store all the
> > other
> > > > > > > elements
> > > > > > > > >> of a
> > > > > > > > >> >> > > > > delegation
> > > > > > > > >> >> > > > > > > > tokens. Brokers can read these elements
> and
> > > > > because
> > > > > > > > they
> > > > > > > > >> also
> > > > > > > > >> >> > > have
> > > > > > > > >> >> > > > > > access
> > > > > > > > >> >> > > > > > > > to secret key they will be able to
> generate
> > > > hmac
> > > > > on
> > > > > > > > >> their end.
> > > > > > > > >> >> > > > > > > >
> > > > > > > > >> >> > > > > > > > One of the alternative proposed is to
> avoid
> > > > > > zookeeper
> > > > > > > > >> >> > > altogether. A
> > > > > > > > >> >> > > > > > > Client
> > > > > > > > >> >> > > > > > > > will call broker with required
> information
> > > > > (owner,
> > > > > > > > >> renwer,
> > > > > > > > >> >> > > > > expiration)
> > > > > > > > >> >> > > > > > > and
> > > > > > > > >> >> > > > > > > > get back (signed hmac, uuid). Broker
> won't
> > > > store
> > > > > > this
> > > > > > > > in
> > > > > > > > >> >> > > zookeeper.
> > > > > > > > >> >> > > > > > From
> > > > > > > > >> >> > > > > > > > this point a client can contact any
> broker
> > > with
> > > > > all
> > > > > > > the
> > > > > > > > >> >> > > delegation
> > > > > > > > >> >> > > > > > token
> > > > > > > > >> >> > > > > > > > info (owner, rewner, expiration, hmac,
> > uuid)
> > > > the
> > > > > > > borker
> > > > > > > > >> will
> > > > > > > > >> >> > > > > regenerate
> > > > > > > > >> >> > > > > > > the
> > > > > > > > >> >> > > > > > > > hmac and as long as it matches with hmac
> > > > > presented
> > > > > > by
> > > > > > > > >> client ,
> > > > > > > > >> >> > > > broker
> > > > > > > > >> >> > > > > > > will
> > > > > > > > >> >> > > > > > > > allow the request to authenticate.  Only
> > > > problem
> > > > > > with
> > > > > > > > >> this
> > > > > > > > >> >> > > approach
> > > > > > > > >> >> > > > > is
> > > > > > > > >> >> > > > > > if
> > > > > > > > >> >> > > > > > > > the secret key is compromised any client
> > can
> > > > now
> > > > > > > > generate
> > > > > > > > >> >> > random
> > > > > > > > >> >> > > > > tokens
> > > > > > > > >> >> > > > > > > and
> > > > > > > > >> >> > > > > > > > they will still be able to authenticate
> as
> > > any
> > > > > user
> > > > > > > > they
> > > > > > > > >> like.
> > > > > > > > >> >> > > with
> > > > > > > > >> >> > > > > > > > zookeeper we guarantee that only tokens
> > > > acquired
> > > > > > via
> > > > > > > a
> > > > > > > > >> broker
> > > > > > > > >> >> > > > (using
> > > > > > > > >> >> > > > > > some
> > > > > > > > >> >> > > > > > > > auth scheme other than delegation token)
> > will
> > > > be
> > > > > > > > >> accepted. We
> > > > > > > > >> >> > > need
> > > > > > > > >> >> > > > to
> > > > > > > > >> >> > > > > > > > discuss which proposal makes more sense
> and
> > > we
> > > > > can
> > > > > > go
> > > > > > > > >> over it
> > > > > > > > >> >> > in
> > > > > > > > >> >> > > > > > > tomorrow's
> > > > > > > > >> >> > > > > > > > meeting.
> > > > > > > > >> >> > > > > > > >
> > > > > > > > >> >> > > > > > > > Also, can you forward the invite to me?
> > > > > > > > >> >> > > > > > > >
> > > > > > > > >> >> > > > > > > > Thanks
> > > > > > > > >> >> > > > > > > > Parth
> > > > > > > > >> >> > > > > > > >
> > > > > > > > >> >> > > > > > > >
> > > > > > > > >> >> > > > > > > >
> > > > > > > > >> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun
> Rao <
> > > > > > > > >> jun@confluent.io>
> > > > > > > > >> >> > > > wrote:
> > > > > > > > >> >> > > > > > > >
> > > > > > > > >> >> > > > > > > > > Thanks for the KIP. A few comments.
> > > > > > > > >> >> > > > > > > > >
> > > > > > > > >> >> > > > > > > > > 100. This potentially can be useful for
> > > Kafka
> > > > > > > Connect
> > > > > > > > >> and
> > > > > > > > >> >> > Kafka
> > > > > > > > >> >> > > > > rest
> > > > > > > > >> >> > > > > > > > proxy
> > > > > > > > >> >> > > > > > > > > where a worker agent will need to run a
> > > task
> > > > on
> > > > > > > > behalf
> > > > > > > > >> of a
> > > > > > > > >> >> > > > client.
> > > > > > > > >> >> > > > > > We
> > > > > > > > >> >> > > > > > > > will
> > > > > > > > >> >> > > > > > > > > likely need to change how those
> services
> > > use
> > > > > > Kafka
> > > > > > > > >> clients
> > > > > > > > >> >> > > > > > > > > (producer/consumer). Instead of a
> shared
> > > > client
> > > > > > per
> > > > > > > > >> worker,
> > > > > > > > >> >> > we
> > > > > > > > >> >> > > > will
> > > > > > > > >> >> > > > > > > need
> > > > > > > > >> >> > > > > > > > a
> > > > > > > > >> >> > > > > > > > > client per user task since the
> > > authentication
> > > > > > > happens
> > > > > > > > >> at the
> > > > > > > > >> >> > > > > > connection
> > > > > > > > >> >> > > > > > > > > level. For Kafka Connect, the renewer
> > will
> > > be
> > > > > the
> > > > > > > > >> workers.
> > > > > > > > >> >> > So,
> > > > > > > > >> >> > > we
> > > > > > > > >> >> > > > > > > > probably
> > > > > > > > >> >> > > > > > > > > need to allow multiple renewers. For
> > Kafka
> > > > rest
> > > > > > > > proxy,
> > > > > > > > >> the
> > > > > > > > >> >> > > > renewer
> > > > > > > > >> >> > > > > > can
> > > > > > > > >> >> > > > > > > > > probably just be the creator of the
> > token.
> > > > > > > > >> >> > > > > > > > >
> > > > > > > > >> >> > > > > > > > > 101. Do we need new acl on who can
> > request
> > > > > > > delegation
> > > > > > > > >> tokens?
> > > > > > > > >> >> > > > > > > > >
> > > > > > > > >> >> > > > > > > > > 102. Do we recommend people to send
> > > > delegation
> > > > > > > tokens
> > > > > > > > >> in an
> > > > > > > > >> >> > > > > encrypted
> > > > > > > > >> >> > > > > > > > > channel?
> > > > > > > > >> >> > > > > > > > >
> > > > > > > > >> >> > > > > > > > > 103. Who is responsible for expiring
> > > tokens,
> > > > > > every
> > > > > > > > >> broker?
> > > > > > > > >> >> > > > > > > > >
> > > > > > > > >> >> > > > > > > > > 104. For invalidating tokens, would it
> be
> > > > > better
> > > > > > to
> > > > > > > > do
> > > > > > > > >> it in
> > > > > > > > >> >> > a
> > > > > > > > >> >> > > > > > request
> > > > > > > > >> >> > > > > > > > > instead of going to ZK directly?
> > > > > > > > >> >> > > > > > > > >
> > > > > > > > >> >> > > > > > > > > 105. The terminology of client in the
> > wiki
> > > > > > > sometimes
> > > > > > > > >> refers
> > > > > > > > >> >> > to
> > > > > > > > >> >> > > > the
> > > > > > > > >> >> > > > > > end
> > > > > > > > >> >> > > > > > > > > client and some other times refers to
> the
> > > > > client
> > > > > > > > using
> > > > > > > > >> the
> > > > > > > > >> >> > > > > delegation
> > > > > > > > >> >> > > > > > > > > tokens. It would be useful to
> distinguish
> > > > > between
> > > > > > > the
> > > > > > > > >> two.
> > > > > > > > >> >> > > > > > > > >
> > > > > > > > >> >> > > > > > > > > 106. Could you explain the sentence
> > "Broker
> > > > > also
> > > > > > > > >> stores the
> > > > > > > > >> >> > > > > > > > DelegationToken
> > > > > > > > >> >> > > > > > > > > without the hmac in the zookeeper." a
> bit
> > > > > more? I
> > > > > > > > >> thought the
> > > > > > > > >> >> > > > > > > delegation
> > > > > > > > >> >> > > > > > > > > token is the hmac.
> > > > > > > > >> >> > > > > > > > >
> > > > > > > > >> >> > > > > > > > > Thanks,
> > > > > > > > >> >> > > > > > > > >
> > > > > > > > >> >> > > > > > > > > Jun
> > > > > > > > >> >> > > > > > > > >
> > > > > > > > >> >> > > > > > > > >
> > > > > > > > >> >> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun
> Rao
> > <
> > > > > > > > >> jun@confluent.io>
> > > > > > > > >> >> > > > wrote:
> > > > > > > > >> >> > > > > > > > >
> > > > > > > > >> >> > > > > > > > > > Hi, Harsha,
> > > > > > > > >> >> > > > > > > > > >
> > > > > > > > >> >> > > > > > > > > > Just sent out a KIP meeting invite.
> We
> > > can
> > > > > > > discuss
> > > > > > > > >> this in
> > > > > > > > >> >> > > the
> > > > > > > > >> >> > > > > > > meeting
> > > > > > > > >> >> > > > > > > > > > tomorrow.
> > > > > > > > >> >> > > > > > > > > >
> > > > > > > > >> >> > > > > > > > > > Thanks,
> > > > > > > > >> >> > > > > > > > > >
> > > > > > > > >> >> > > > > > > > > > Jun
> > > > > > > > >> >> > > > > > > > > >
> > > > > > > > >> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM,
> > Harsha <
> > > > > > > > >> kafka@harsha.io>
> > > > > > > > >> >> > > > wrote:
> > > > > > > > >> >> > > > > > > > > >
> > > > > > > > >> >> > > > > > > > > >> Hi All,
> > > > > > > > >> >> > > > > > > > > >>            Can we have a KIP meeting
> > > > around
> > > > > > > this.
> > > > > > > > >> The KIP
> > > > > > > > >> >> > is
> > > > > > > > >> >> > > > up
> > > > > > > > >> >> > > > > > for
> > > > > > > > >> >> > > > > > > > > >>            sometime and if there are
> > any
> > > > > > > questions
> > > > > > > > >> lets
> > > > > > > > >> >> > > > quickly
> > > > > > > > >> >> > > > > > hash
> > > > > > > > >> >> > > > > > > > out
> > > > > > > > >> >> > > > > > > > > >>            details.
> > > > > > > > >> >> > > > > > > > > >>
> > > > > > > > >> >> > > > > > > > > >> Thanks,
> > > > > > > > >> >> > > > > > > > > >> Harsha
> > > > > > > > >> >> > > > > > > > > >>
> > > > > > > > >> >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40 AM,
> > parth
> > > > > > > > brahmbhatt
> > > > > > > > >> wrote:
> > > > > > > > >> >> > > > > > > > > >> > That is what the hadoop echo
> system
> > > uses
> > > > > so
> > > > > > no
> > > > > > > > >> good
> > > > > > > > >> >> > reason
> > > > > > > > >> >> > > > > > really.
> > > > > > > > >> >> > > > > > > > We
> > > > > > > > >> >> > > > > > > > > >> > could
> > > > > > > > >> >> > > > > > > > > >> > change it to whatever is the
> newest
> > > > > > > recommended
> > > > > > > > >> standard
> > > > > > > > >> >> > > is.
> > > > > > > > >> >> > > > > > > > > >> >
> > > > > > > > >> >> > > > > > > > > >> > Thanks
> > > > > > > > >> >> > > > > > > > > >> > Parth
> > > > > > > > >> >> > > > > > > > > >> >
> > > > > > > > >> >> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM,
> > > Ismael
> > > > > > Juma <
> > > > > > > > >> >> > > > > ismael@juma.me.uk
> > > > > > > > >> >> > > > > > >
> > > > > > > > >> >> > > > > > > > > wrote:
> > > > > > > > >> >> > > > > > > > > >> >
> > > > > > > > >> >> > > > > > > > > >> > > Hi Parth,
> > > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > > >> >> > > > > > > > > >> > > Thanks for the KIP. I only
> started
> > > > > > reviewing
> > > > > > > > >> this and
> > > > > > > > >> >> > > may
> > > > > > > > >> >> > > > > have
> > > > > > > > >> >> > > > > > > > > >> additional
> > > > > > > > >> >> > > > > > > > > >> > > questions later. The immediate
> > > > question
> > > > > > that
> > > > > > > > >> came to
> > > > > > > > >> >> > > mind
> > > > > > > > >> >> > > > is
> > > > > > > > >> >> > > > > > our
> > > > > > > > >> >> > > > > > > > > >> choice of
> > > > > > > > >> >> > > > > > > > > >> > > "DIGEST-MD5" even though it's
> > marked
> > > > as
> > > > > > > > >> OBSOLETE in
> > > > > > > > >> >> > the
> > > > > > > > >> >> > > > IANA
> > > > > > > > >> >> > > > > > > > > Registry
> > > > > > > > >> >> > > > > > > > > >> of
> > > > > > > > >> >> > > > > > > > > >> > > SASL mechanisms and the original
> > RFC
> > > > > > (2831)
> > > > > > > > has
> > > > > > > > >> been
> > > > > > > > >> >> > > moved
> > > > > > > > >> >> > > > > to
> > > > > > > > >> >> > > > > > > > > Historic
> > > > > > > > >> >> > > > > > > > > >> > > status:
> > > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > > >> >> > > > > > > > > >> > > https://tools.ietf.org/html/
> > rfc6331
> > > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > > >> >> > > > > > > > >
> > > > > > > > >> >> > > > > >
> > > > > > > > >> >> > >
> > > > > > > > >>
> > > > > >
> > > http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> > > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > > >> >> > > > > > > > > >> > > What is the reasoning behind
> that
> > > > > choice?
> > > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > > >> >> > > > > > > > > >> > > Thanks,
> > > > > > > > >> >> > > > > > > > > >> > > Ismael
> > > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > > >> >> > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29
> PM,
> > > Gwen
> > > > > > > > Shapira <
> > > > > > > > >> >> > > > > > > gwen@confluent.io
> > > > > > > > >> >> > > > > > > > >
> > > > > > > > >> >> > > > > > > > > >> wrote:
> > > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > > >> >> > > > > > > > > >> > > > Also comments inline :)
> > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > >> >> > > > > > > > > >> > > > > * I want to emphasize that
> > even
> > > > > though
> > > > > > > > >> delegation
> > > > > > > > >> >> > > > tokens
> > > > > > > > >> >> > > > > > > are a
> > > > > > > > >> >> > > > > > > > > >> Hadoop
> > > > > > > > >> >> > > > > > > > > >> > > > > innovation, I feel very
> > strongly
> > > > > about
> > > > > > > not
> > > > > > > > >> adding
> > > > > > > > >> >> > > > > > dependency
> > > > > > > > >> >> > > > > > > > on
> > > > > > > > >> >> > > > > > > > > >> Hadoop
> > > > > > > > >> >> > > > > > > > > >> > > > > when implementing delegation
> > > > tokens
> > > > > > for
> > > > > > > > >> Kafka. The
> > > > > > > > >> >> > > KIP
> > > > > > > > >> >> > > > > > > doesn't
> > > > > > > > >> >> > > > > > > > > >> imply
> > > > > > > > >> >> > > > > > > > > >> > > > > such dependency, but if you
> > can
> > > > > > > clarify...
> > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > >> >> > > > > > > > > >> > > > > *No hadoop dependency.*
> > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > >> >> > > > > > > > > >> > > > Yay! Just add this to the KIP
> so
> > > no
> > > > > one
> > > > > > > will
> > > > > > > > >> read
> > > > > > > > >> >> > the
> > > > > > > > >> >> > > > KIP
> > > > > > > > >> >> > > > > > and
> > > > > > > > >> >> > > > > > > > > panic
> > > > > > > > >> >> > > > > > > > > >> > > > three weeks before the next
> > > > release...
> > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > >> >> > > > > > > > > >> > > > > * Can we get delegation
> token
> > at
> > > > any
> > > > > > > time
> > > > > > > > >> after
> > > > > > > > >> >> > > > > > > > authenticating?
> > > > > > > > >> >> > > > > > > > > >> only
> > > > > > > > >> >> > > > > > > > > >> > > > > immediately after?
> > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > >> >> > > > > > > > > >> > > > > *As long as you are
> > > authenticated
> > > > > you
> > > > > > > can
> > > > > > > > >> get
> > > > > > > > >> >> > > > delegation
> > > > > > > > >> >> > > > > > > > tokens.
> > > > > > > > >> >> > > > > > > > > >> We
> > > > > > > > >> >> > > > > > > > > >> > > need
> > > > > > > > >> >> > > > > > > > > >> > > > to
> > > > > > > > >> >> > > > > > > > > >> > > > > discuss if a client
> > > authenticated
> > > > > > using
> > > > > > > > >> delegation
> > > > > > > > >> >> > > > > token,
> > > > > > > > >> >> > > > > > > can
> > > > > > > > >> >> > > > > > > > > also
> > > > > > > > >> >> > > > > > > > > >> > > > acquire
> > > > > > > > >> >> > > > > > > > > >> > > > > delegation token again or
> not.
> > > > Also
> > > > > > > there
> > > > > > > > >> is the
> > > > > > > > >> >> > > > > question
> > > > > > > > >> >> > > > > > of
> > > > > > > > >> >> > > > > > > > do
> > > > > > > > >> >> > > > > > > > > we
> > > > > > > > >> >> > > > > > > > > >> > > allow
> > > > > > > > >> >> > > > > > > > > >> > > > > anyone to acquire delegation
> > > token
> > > > > or
> > > > > > we
> > > > > > > > >> want
> > > > > > > > >> >> > > specific
> > > > > > > > >> >> > > > > > ACLs
> > > > > > > > >> >> > > > > > > (I
> > > > > > > > >> >> > > > > > > > > >> think
> > > > > > > > >> >> > > > > > > > > >> > > its
> > > > > > > > >> >> > > > > > > > > >> > > > an
> > > > > > > > >> >> > > > > > > > > >> > > > > overkill.)*
> > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > >> >> > > > > > > > > >> > > > I agree that ACLs is an
> > overkill.
> > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > >> >> > > > > > > > > >> > > > I think we are debating two
> > > options:
> > > > > > > Either
> > > > > > > > >> require
> > > > > > > > >> >> > > > > Kerberos
> > > > > > > > >> >> > > > > > > > auth
> > > > > > > > >> >> > > > > > > > > >> for
> > > > > > > > >> >> > > > > > > > > >> > > > renewal or require non-owners
> to
> > > > > renew.
> > > > > > > > >> >> > > > > > > > > >> > > > I *think* the latter is
> simpler
> > > (it
> > > > > > > > basically
> > > > > > > > >> >> > require
> > > > > > > > >> >> > > a
> > > > > > > > >> >> > > > > "job
> > > > > > > > >> >> > > > > > > > > master"
> > > > > > > > >> >> > > > > > > > > >> > > > to take responsibility for the
> > > > > renewal,
> > > > > > it
> > > > > > > > >> will have
> > > > > > > > >> >> > > its
> > > > > > > > >> >> > > > > own
> > > > > > > > >> >> > > > > > > > > >> identity
> > > > > > > > >> >> > > > > > > > > >> > > > anyway and I think this is the
> > > > correct
> > > > > > > > design
> > > > > > > > >> >> > pattern
> > > > > > > > >> >> > > > > > anyway.
> > > > > > > > >> >> > > > > > > > For
> > > > > > > > >> >> > > > > > > > > >> > > > storm, I'd expect Nimbus to
> > > > coordinate
> > > > > > > > >> renewals?),
> > > > > > > > >> >> > but
> > > > > > > > >> >> > > > it
> > > > > > > > >> >> > > > > is
> > > > > > > > >> >> > > > > > > > hard
> > > > > > > > >> >> > > > > > > > > to
> > > > > > > > >> >> > > > > > > > > >> > > > debate simplicity without
> > looking
> > > at
> > > > > the
> > > > > > > > code
> > > > > > > > >> >> > changes
> > > > > > > > >> >> > > > > > > required.
> > > > > > > > >> >> > > > > > > > If
> > > > > > > > >> >> > > > > > > > > >> you
> > > > > > > > >> >> > > > > > > > > >> > > > have a draft of how the
> "require
> > > > > > Kerberos"
> > > > > > > > >> will look
> > > > > > > > >> >> > > in
> > > > > > > > >> >> > > > > > Kafka
> > > > > > > > >> >> > > > > > > > > code,
> > > > > > > > >> >> > > > > > > > > >> > > > I'll be happy to take a look.
> > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > >> >> > > > > > > > > >> > > > > * My understanding is that
> > > tokens
> > > > > will
> > > > > > > > >> propagate
> > > > > > > > >> >> > via
> > > > > > > > >> >> > > > ZK
> > > > > > > > >> >> > > > > > but
> > > > > > > > >> >> > > > > > > > > >> without
> > > > > > > > >> >> > > > > > > > > >> > > > > additional changes to
> > > > UpdateMetadata
> > > > > > > > >> protocol,
> > > > > > > > >> >> > > > correct?
> > > > > > > > >> >> > > > > > > > Clients
> > > > > > > > >> >> > > > > > > > > >> > > > > currently don't retry on
> SASL
> > > auth
> > > > > > > failure
> > > > > > > > >> (IIRC),
> > > > > > > > >> >> > > but
> > > > > > > > >> >> > > > > > since
> > > > > > > > >> >> > > > > > > > the
> > > > > > > > >> >> > > > > > > > > >> > > > > tokens propagate between
> > brokers
> > > > > > asynch,
> > > > > > > > we
> > > > > > > > >> will
> > > > > > > > >> >> > > need
> > > > > > > > >> >> > > > to
> > > > > > > > >> >> > > > > > > > retry a
> > > > > > > > >> >> > > > > > > > > >> bit
> > > > > > > > >> >> > > > > > > > > >> > > > > to avoid clients failing
> auth
> > > due
> > > > to
> > > > > > > > timing
> > > > > > > > >> >> > issues.
> > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > >> >> > > > > > > > > >> > > > > *I am considering 2
> > alternatives
> > > > > right
> > > > > > > > now.
> > > > > > > > >> The
> > > > > > > > >> >> > > > current
> > > > > > > > >> >> > > > > > > > > documented
> > > > > > > > >> >> > > > > > > > > >> > > > approach
> > > > > > > > >> >> > > > > > > > > >> > > > > is zookeeper based and it
> does
> > > not
> > > > > > > require
> > > > > > > > >> any
> > > > > > > > >> >> > > changes
> > > > > > > > >> >> > > > > to
> > > > > > > > >> >> > > > > > > > > >> > > UpdateMetadata
> > > > > > > > >> >> > > > > > > > > >> > > > > protocol. An alternative
> > > approach
> > > > > can
> > > > > > > > remove
> > > > > > > > >> >> > > zookeeper
> > > > > > > > >> >> > > > > > > > > dependency
> > > > > > > > >> >> > > > > > > > > >> as
> > > > > > > > >> >> > > > > > > > > >> > > well
> > > > > > > > >> >> > > > > > > > > >> > > > > but we can discuss that in
> KIP
> > > > > > > discussion
> > > > > > > > >> call.*
> > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > >> >> > > > > > > > > >> > > > Oooh! Sounds interesting. Do
> you
> > > > want
> > > > > to
> > > > > > > > ping
> > > > > > > > >> Jun to
> > > > > > > > >> >> > > > > > arrange a
> > > > > > > > >> >> > > > > > > > > call?
> > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > >> >> > > > > > > > > >> > > > > * I liked Ashish's
> suggestion
> > of
> > > > > > having
> > > > > > > > >> just the
> > > > > > > > >> >> > > > > > controller
> > > > > > > > >> >> > > > > > > > > issue
> > > > > > > > >> >> > > > > > > > > >> the
> > > > > > > > >> >> > > > > > > > > >> > > > > delegation tokens, to avoid
> > > > syncing
> > > > > a
> > > > > > > > shared
> > > > > > > > >> >> > secret.
> > > > > > > > >> >> > > > Not
> > > > > > > > >> >> > > > > > > sure
> > > > > > > > >> >> > > > > > > > if
> > > > > > > > >> >> > > > > > > > > >> we
> > > > > > > > >> >> > > > > > > > > >> > > > > want to continue the
> > discussion
> > > > here
> > > > > > or
> > > > > > > on
> > > > > > > > >> the
> > > > > > > > >> >> > > wiki. I
> > > > > > > > >> >> > > > > > think
> > > > > > > > >> >> > > > > > > > > that
> > > > > > > > >> >> > > > > > > > > >> we
> > > > > > > > >> >> > > > > > > > > >> > > > > can decouple the problem of
> > > "token
> > > > > > > > >> distribution"
> > > > > > > > >> >> > > from
> > > > > > > > >> >> > > > > > > "shared
> > > > > > > > >> >> > > > > > > > > >> secret
> > > > > > > > >> >> > > > > > > > > >> > > > > distribution" and use the
> > > > controller
> > > > > > as
> > > > > > > > the
> > > > > > > > >> only
> > > > > > > > >> >> > > token
> > > > > > > > >> >> > > > > > > > generator
> > > > > > > > >> >> > > > > > > > > >> to
> > > > > > > > >> >> > > > > > > > > >> > > > > solve the second issue,
> while
> > > > still
> > > > > > > using
> > > > > > > > >> ZK async
> > > > > > > > >> >> > > to
> > > > > > > > >> >> > > > > > > > distribute
> > > > > > > > >> >> > > > > > > > > >> > > > > tokens.
> > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > >> >> > > > > > > > > >> > > > > *As mentioned in the
> previous
> > > > Email
> > > > > I
> > > > > > am
> > > > > > > > >> fine with
> > > > > > > > >> >> > > > that
> > > > > > > > >> >> > > > > > > > approach
> > > > > > > > >> >> > > > > > > > > >> as
> > > > > > > > >> >> > > > > > > > > >> > > long
> > > > > > > > >> >> > > > > > > > > >> > > > as
> > > > > > > > >> >> > > > > > > > > >> > > > > we agree that the extra
> > > complexity
> > > > > of
> > > > > > > > >> >> > > adding/updating
> > > > > > > > >> >> > > > > APIS
> > > > > > > > >> >> > > > > > > > adds
> > > > > > > > >> >> > > > > > > > > >> enough
> > > > > > > > >> >> > > > > > > > > >> > > > > value. The advantage with
> the
> > > > > > controller
> > > > > > > > >> approach
> > > > > > > > >> >> > is
> > > > > > > > >> >> > > > > > secret
> > > > > > > > >> >> > > > > > > > > >> rotation
> > > > > > > > >> >> > > > > > > > > >> > > can
> > > > > > > > >> >> > > > > > > > > >> > > > be
> > > > > > > > >> >> > > > > > > > > >> > > > > automated,frequent and would
> > not
> > > > > > require
> > > > > > > > >> >> > > deployment. *
> > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > >> >> > > > > > > > > >> > > > Can you detail the extra
> > > complexity
> > > > > (or
> > > > > > > > point
> > > > > > > > >> me to
> > > > > > > > >> >> > > the
> > > > > > > > >> >> > > > > > email
> > > > > > > > >> >> > > > > > > I
> > > > > > > > >> >> > > > > > > > > >> > > > missed?) - which Apis are
> > > required?
> > > > > > > > >> >> > > > > > > > > >> > > > As far as I can tell, clients
> > can
> > > > > > already
> > > > > > > > >> find the
> > > > > > > > >> >> > > > > > controller
> > > > > > > > >> >> > > > > > > > from
> > > > > > > > >> >> > > > > > > > > >> > > > metadata. I'm a bit more
> > concerned
> > > > > about
> > > > > > > > >> controller
> > > > > > > > >> >> > > > load.
> > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > >> >> > > > > > > > > >> > > > > * While I like the idea of
> > > forcing
> > > > > > > > kerberos
> > > > > > > > >> auth
> > > > > > > > >> >> > for
> > > > > > > > >> >> > > > > > > renwal, I
> > > > > > > > >> >> > > > > > > > > >> think
> > > > > > > > >> >> > > > > > > > > >> > > > > it mixes the transport layer
> > the
> > > > the
> > > > > > > > request
> > > > > > > > >> >> > content
> > > > > > > > >> >> > > > in
> > > > > > > > >> >> > > > > a
> > > > > > > > >> >> > > > > > > > pretty
> > > > > > > > >> >> > > > > > > > > >> ugly
> > > > > > > > >> >> > > > > > > > > >> > > > > way. Perhaps limiting
> renewer
> > to
> > > > > > > non-owner
> > > > > > > > >> is
> > > > > > > > >> >> > > better.
> > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > >> >> > > > > > > > > >> > > > > *I feel this is a necessary
> > > evil.
> > > > > > While
> > > > > > > > >> this will
> > > > > > > > >> >> > > make
> > > > > > > > >> >> > > > > the
> > > > > > > > >> >> > > > > > > > kafka
> > > > > > > > >> >> > > > > > > > > >> code
> > > > > > > > >> >> > > > > > > > > >> > > > > pretty straight forward ,
> > > forcing
> > > > > > > renewer
> > > > > > > > >> to
> > > > > > > > >> >> > > > non-owner
> > > > > > > > >> >> > > > > > > pushes
> > > > > > > > >> >> > > > > > > > > >> the code
> > > > > > > > >> >> > > > > > > > > >> > > > > ugliness to client and makes
> > it
> > > > even
> > > > > > > > harder
> > > > > > > > >> to
> > > > > > > > >> >> > > > > > integrate.  *
> > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > >> >> > > > > > > > > >> > > > As mentioned before, I don't
> > think
> > > > the
> > > > > > > > >> "renewal by
> > > > > > > > >> >> > > > other"
> > > > > > > > >> >> > > > > > > > approach
> > > > > > > > >> >> > > > > > > > > >> is
> > > > > > > > >> >> > > > > > > > > >> > > > that ugly for the clients we
> > > expect
> > > > to
> > > > > > use
> > > > > > > > >> >> > delegation
> > > > > > > > >> >> > > > > tokens
> > > > > > > > >> >> > > > > > > > since
> > > > > > > > >> >> > > > > > > > > >> > > > they will have an app-master
> of
> > > some
> > > > > > sort
> > > > > > > > who
> > > > > > > > >> >> > > requested
> > > > > > > > >> >> > > > > the
> > > > > > > > >> >> > > > > > > > token
> > > > > > > > >> >> > > > > > > > > to
> > > > > > > > >> >> > > > > > > > > >> > > > begin with.
> > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > >> >> > > > > > > > > >> > > > > The response for my question
> > on
> > > > how
> > > > > > > > multiple
> > > > > > > > >> >> > > > identities
> > > > > > > > >> >> > > > > > will
> > > > > > > > >> >> > > > > > > > be
> > > > > > > > >> >> > > > > > > > > >> > > > > handled wasn't super clear
> to
> > > me -
> > > > > > > AFAIK,
> > > > > > > > >> we don't
> > > > > > > > >> >> > > > > > > > authenticate
> > > > > > > > >> >> > > > > > > > > >> each
> > > > > > > > >> >> > > > > > > > > >> > > > > request, we authenticate
> > > > > connections.
> > > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > > >> >> > > > > > > > > >> > > > > *We authenticate
> connections,
> > > and
> > > > > only
> > > > > > > > when
> > > > > > > > >> they
> > > > > > > > >> >> > are
> > > > > > > > >> >> > > > > being
> > > > > > > > >> >> > > > > > > > > >> established.
> > > > > > > > >> >> > > > > > > > > >> > > > Let
> > > > > > > > >> >> > > > > > > > > >> > > > > me try to phrase this as a
> > > > question,
> > > > > > in
> > > > > > > > >> absence of
> > > > > > > > >> >> > > > > > > delegation
> > > > > > > > >> >> > > > > > > > > >> tokens if
> > > > > > > > >> >> > > > > > > > > >> > > > we
> > > > > > > > >> >> > > > > > > > > >> > > > > had to support the use case
> > > using
> > > > > user
> > > > > > > > >> TGT's how
> > > > > > > > >> >> > > would
> > > > > > > > >> >> > > > > we
> > > > > > > > >> >> > > > > > do
> > > > > > > > >> >> > > > > > > > it?
> > > > > > > > >> >> > > > > > > > > >> My
> > > > > > > > >> >> > > > > > > > > >> > > point
> > > > > > > > >> >> > > > > > > > > >> > > > > was it would be no different
> > > with
> > > > > > > > delegation
> > > > > > > > >> >> > tokens.
> > > > > > > > >> >> > > > The
> > > > > > > > >> >> > > > > > use
> > > > > > > > >> >> > > > > > > > > case
> > > > > > > > >> >> > > > > > > > > >> you
> > > > > > > > >> >> > > > > > > > > >> > > are
> > > > > > > > >> >> > > > > > > > > >> > > > > describing seems more like
> > > > > > > impersonation.*
> > > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > > >> >> > > > > > > > > >> > > > Yeah, I thought that one of
> the
> > > > things
> > > > > > > that
> > > > > > > > >> >> > del

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Ashish Singh <as...@cloudera.com>]

Hello Harsha,

Are you still working on this? Wondering if we can discuss this in next KIP
meeting, if you can join.

On Mon, Jul 18, 2016 at 9:51 AM, Harsha Chintalapani <ka...@harsha.io>
wrote:

> Hi Grant,
>           We are working on it. Will add the details to KIP about the
> request protocol.
>
> Thanks,
> Harsha
>
> On Mon, Jul 18, 2016 at 6:50 AM Grant Henke <gh...@cloudera.com> wrote:
>
> > Hi Parth,
> >
> > Are you still working on this? If you need any help please don't hesitate
> > to ask.
> >
> > Thanks,
> > Grant
> >
> > On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Parth,
> > >
> > > Thanks for the reply.
> > >
> > > It makes sense to only allow the renewal by users that authenticated
> > using
> > > *non* delegation token mechanism. Then, should we make the renewal a
> > list?
> > > For example, in the case of rest proxy, it will be useful for every
> > > instance of rest proxy to be able to renew the tokens.
> > >
> > > It would be clearer if we can document the request protocol like
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 4+-+Command+line+and+centralized+administrative+operations#KIP-4-
> Commandlineandcentralizedadministrativeoperations-
> CreateTopicsRequest(KAFKA-2945):(VotedandPlannedforin0.10.1.0)
> > > .
> > >
> > > It would also be useful to document the client APIs.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Jun 28, 2016 at 2:55 PM, parth brahmbhatt <
> > > brahmbhatt.parth@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I am suggesting that we will only allow the renewal by users that
> > > > authenticated using *non* delegation token mechanism. For example, If
> > > user
> > > > Alice authenticated using kerberos and requested delegation tokens,
> > only
> > > > user Alice authenticated via non delegation token mechanism can
> renew.
> > > > Clients that have  access to delegation tokens can not issue renewal
> > > > request for renewing their own token and this is primarily important
> to
> > > > reduce the time window for which a compromised token will be valid.
> > > >
> > > > To clarify, Yes any authenticated user can request delegation tokens
> > but
> > > > even here I would recommend to avoid creating a chain where a client
> > > > authenticated via delegation token request for more delegation
> tokens.
> > > > Basically anyone can request delegation token, as long as they
> > > authenticate
> > > > via a non delegation token mechanism.
> > > >
> > > > Aren't classes listed here
> > > > <
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKaf
> ka-PublicInterfaces
> > > > >
> > > > sufficient?
> > > >
> > > > Thanks
> > > > Parth
> > > >
> > > >
> > > >
> > > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao <ju...@confluent.io> wrote:
> > > >
> > > > > Parth,
> > > > >
> > > > > Thanks for the reply. A couple of comments inline below.
> > > > >
> > > > > On Tue, Jun 21, 2016 at 10:36 AM, parth brahmbhatt <
> > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > >
> > > > > > 1. Who / how are tokens renewed? By original requester only? or
> > using
> > > > > > Kerberos
> > > > > > auth only?
> > > > > > My recommendation is to do this only using Kerberos auth and only
> > > threw
> > > > > the
> > > > > > renewer specified during the acquisition request.
> > > > > >
> > > > > >
> > > > > Hmm, not sure that I follow this. Are you saying that any client
> > > > > authenticated with the delegation token can renew, i.e. there is no
> > > > renewer
> > > > > needed?
> > > > >
> > > > > Also, just to be clear, any authenticated client (either through
> SASL
> > > or
> > > > > SSL) can request a delegation token for the authenticated user,
> > right?
> > > > >
> > > > >
> > > > > > 2. Are tokens stored on each broker or in ZK?
> > > > > > My recommendation is still to store in ZK or not store them at
> all.
> > > The
> > > > > > whole controller based distribution is too much overhead with not
> > > much
> > > > to
> > > > > > achieve.
> > > > > >
> > > > > > 3. How are tokens invalidated / expired?
> > > > > > Either by expiration time out or through an explicit request to
> > > > > invalidate.
> > > > > >
> > > > > > 4. Which encryption algorithm is used?
> > > > > > SCRAM
> > > > > >
> > > > > > 5. What is the impersonation proposal (it wasn't in the KIP but
> was
> > > > > > discussed
> > > > > > in this thread)?
> > > > > > There is no imperonation proposal. I tried and explained how its
> a
> > > > > > different problem and why its not really necessary to discuss
> that
> > as
> > > > > part
> > > > > > of this KIP.  This KIP will not support any impersonation, it
> will
> > > just
> > > > > be
> > > > > > another way to authenticate.
> > > > > >
> > > > > > 6. Do we need new ACLs, if so - for what actions?
> > > > > > We do not need new ACLs.
> > > > > >
> > > > > >
> > > > > Could we document the format of the new request/response and their
> > > > > associated Resource and Operation for ACL?
> > > > >
> > > > >
> > > > > > 7. How would the delegation token be configured in the client?
> > > > > > Should be through config. I wasn't planning on supporting JAAS
> for
> > > > > tokens.
> > > > > > I don't believe hadoop does this either.
> > > > > >
> > > > > > Thanks
> > > > > > Parth
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <ju...@confluent.io>
> wrote:
> > > > > >
> > > > > > > Harsha,
> > > > > > >
> > > > > > > Another question.
> > > > > > >
> > > > > > > 9. How would the delegation token be configured in the client?
> > The
> > > > > > standard
> > > > > > > way is to do this through JAAS. However, we will need to think
> > > > through
> > > > > if
> > > > > > > this is convenient in a shared environment. For example, when a
> > new
> > > > > task
> > > > > > is
> > > > > > > added to a Storm worker node, do we need to dynamically add a
> new
> > > > > section
> > > > > > > in the JAAS file? It may be more convenient if we can pass in
> the
> > > > token
> > > > > > > through the config directly w/o going through JAAS.
> > > > > > >
> > > > > > > Are you or Parth still actively working on this KIP?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <ju...@confluent.io>
> > wrote:
> > > > > > >
> > > > > > > > Just to add on that list.
> > > > > > > >
> > > > > > > > 2. It would be good to document the format of the data stored
> > in
> > > > ZK.
> > > > > > > > 7. Earlier, there was a discussion on whether the tokens
> should
> > > be
> > > > > > > > propagated through ZK like config/acl/quota, or through the
> > > > > controller.
> > > > > > > > Currently, the controller is only designed for propagating
> > topic
> > > > > > > metadata,
> > > > > > > > but not other data.
> > > > > > > > 8. Should we use SCRAM to send the token instead of
> DIGEST-MD5
> > > > since
> > > > > > it's
> > > > > > > > deprecated?
> > > > > > > >
> > > > > > > > Also, the images in the wiki seem broken.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <
> > > gwen@confluent.io>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >> From what I can see, remaining questions are:
> > > > > > > >>
> > > > > > > >> 1. Who / how are tokens renewed? By original requester only?
> > or
> > > > > using
> > > > > > > >> Kerberos auth only?
> > > > > > > >> 2. Are tokens stored on each broker or in ZK?
> > > > > > > >> 3. How are tokens invalidated / expired?
> > > > > > > >> 4. Which encryption algorithm is used?
> > > > > > > >> 5. What is the impersonation proposal (it wasn't in the KIP
> > but
> > > > was
> > > > > > > >> discussed in this thread)?
> > > > > > > >> 6. Do we need new ACLs, if so - for what actions?
> > > > > > > >>
> > > > > > > >> Gwen
> > > > > > > >>
> > > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <ka...@harsha.io>
> > wrote:
> > > > > > > >> > Jun & Ismael,
> > > > > > > >> >                          Unfortunately I couldn't attend
> the
> > > KIP
> > > > > > > meeting
> > > > > > > >> >                          when delegation tokens discussed.
> > > > > > Appreciate
> > > > > > > if
> > > > > > > >> >                          you can update the thread if you
> > have
> > > > any
> > > > > > > >> >                          further questions.
> > > > > > > >> > Thanks,
> > > > > > > >> > Harsha
> > > > > > > >> >
> > > > > > > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
> > > > > > > >> >> It seems that the links to images in the KIP are broken.
> > > > > > > >> >>
> > > > > > > >> >> Liquan
> > > > > > > >> >>
> > > > > > > >> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
> > > > > > > >> >> brahmbhatt.parth@gmail.com> wrote:
> > > > > > > >> >>
> > > > > > > >> >> > 110. What does getDelegationTokenAs mean?
> > > > > > > >> >> > In the current proposal we only allow a user to get
> > > > delegation
> > > > > > > token
> > > > > > > >> for
> > > > > > > >> >> > the identity that it authenticated as using another
> > > > mechanism,
> > > > > > i.e.
> > > > > > > >> A user
> > > > > > > >> >> > that authenticate using a keytab for principal
> > > > > user1@EXAMPLE.COM
> > > > > > > >> will get
> > > > > > > >> >> > delegation tokens for that user only. In future I think
> > we
> > > > will
> > > > > > > have
> > > > > > > >> to
> > > > > > > >> >> > extend support such that we allow some set of users (
> > > > > > > >> >> > kafka-rest-user@EXAMPLE.COM, storm-nimbus@EXAMPLE.COM)
> > to
> > > > > > acquire
> > > > > > > >> >> > delegation tokens on behalf of other users whose
> identity
> > > > they
> > > > > > have
> > > > > > > >> >> > verified independently.  Kafka brokers will have ACLs
> to
> > > > > control
> > > > > > > >> which
> > > > > > > >> >> > users are allowed to impersonate other users and get
> > tokens
> > > > on
> > > > > > > >> behalf of
> > > > > > > >> >> > them. Overall Impersonation is a whole different
> problem
> > in
> > > > my
> > > > > > > >> opinion and
> > > > > > > >> >> > I think we can tackle it in separate KIP.
> > > > > > > >> >> >
> > > > > > > >> >> > 111. What's the typical rate of getting and renewing
> > > > delegation
> > > > > > > >> tokens?
> > > > > > > >> >> > Typically this should be very very low, 1 request per
> > > minute
> > > > > is a
> > > > > > > >> >> > relatively high estimate. However it depends on the
> token
> > > > > > > >> expiration. I am
> > > > > > > >> >> > less worried about the extra load it puts on controller
> > vs
> > > > the
> > > > > > > added
> > > > > > > >> >> > complexity and the value it offers.
> > > > > > > >> >> >
> > > > > > > >> >> > Thanks
> > > > > > > >> >> > Parth
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <
> > > > > ismael@juma.me.uk>
> > > > > > > >> wrote:
> > > > > > > >> >> >
> > > > > > > >> >> > > Thanks Rajini. It would probably require a separate
> KIP
> > > as
> > > > it
> > > > > > > will
> > > > > > > >> >> > > introduce user visible changes. We could also update
> > > KIP-48
> > > > > to
> > > > > > > >> have this
> > > > > > > >> >> > > information, but it seems cleaner to do it
> separately.
> > We
> > > > can
> > > > > > > >> discuss
> > > > > > > >> >> > that
> > > > > > > >> >> > > in the KIP call today.
> > > > > > > >> >> > >
> > > > > > > >> >> > > Ismael
> > > > > > > >> >> > >
> > > > > > > >> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <
> > > > > > > >> >> > > rajinisivaram@googlemail.com> wrote:
> > > > > > > >> >> > >
> > > > > > > >> >> > > > Ismael,
> > > > > > > >> >> > > >
> > > > > > > >> >> > > > I have created a JIRA (
> > > > > > > >> >> > https://issues.apache.org/jira/browse/KAFKA-3751)
> > > > > > > >> >> > > > for adding SCRAM as a SASL mechanism. Would that
> need
> > > > > another
> > > > > > > >> KIP? If
> > > > > > > >> >> > > > KIP-48 will use this mechanism, can this just be a
> > JIRA
> > > > > that
> > > > > > > gets
> > > > > > > >> >> > > reviewed
> > > > > > > >> >> > > > when the PR is ready?
> > > > > > > >> >> > > >
> > > > > > > >> >> > > > Thank you,
> > > > > > > >> >> > > >
> > > > > > > >> >> > > > Rajini
> > > > > > > >> >> > > >
> > > > > > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <
> > > > > > > ismael@juma.me.uk>
> > > > > > > >> >> > wrote:
> > > > > > > >> >> > > >
> > > > > > > >> >> > > > > Thanks Rajini, SCRAM seems like a good candidate.
> > > > > > > >> >> > > > >
> > > > > > > >> >> > > > > Gwen had independently mentioned this as a SASL
> > > > mechanism
> > > > > > > that
> > > > > > > >> might
> > > > > > > >> >> > be
> > > > > > > >> >> > > > > useful for Kafka and I have been meaning to check
> > it
> > > in
> > > > > > more
> > > > > > > >> detail.
> > > > > > > >> >> > > Good
> > > > > > > >> >> > > > > to know that you are willing to contribute an
> > > > > > implementation.
> > > > > > > >> Maybe
> > > > > > > >> >> > we
> > > > > > > >> >> > > > > should file a separate JIRA for this?
> > > > > > > >> >> > > > >
> > > > > > > >> >> > > > > Ismael
> > > > > > > >> >> > > > >
> > > > > > > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram <
> > > > > > > >> >> > > > > rajinisivaram@googlemail.com> wrote:
> > > > > > > >> >> > > > >
> > > > > > > >> >> > > > > > SCRAM (Salted Challenge Response Authentication
> > > > > > Mechanism)
> > > > > > > >> is a
> > > > > > > >> >> > > better
> > > > > > > >> >> > > > > > mechanism than Digest-MD5. Java doesn't come
> > with a
> > > > > > > built-in
> > > > > > > >> SCRAM
> > > > > > > >> >> > > > > > SaslServer or SaslClient, but I will be happy
> to
> > > add
> > > > > > > support
> > > > > > > >> in
> > > > > > > >> >> > Kafka
> > > > > > > >> >> > > > > since
> > > > > > > >> >> > > > > > it would be a useful mechanism to support
> anyway.
> > > > > > > >> >> > > > > > https://tools.ietf.org/html/rfc7677 describes
> > the
> > > > > > protocol
> > > > > > > >> for
> > > > > > > >> >> > > > > > SCRAM-SHA-256.
> > > > > > > >> >> > > > > >
> > > > > > > >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <
> > > > > > jun@confluent.io
> > > > > > > >
> > > > > > > >> wrote:
> > > > > > > >> >> > > > > >
> > > > > > > >> >> > > > > > > Parth,
> > > > > > > >> >> > > > > > >
> > > > > > > >> >> > > > > > > Thanks for the explanation. A couple of more
> > > > > questions.
> > > > > > > >> >> > > > > > >
> > > > > > > >> >> > > > > > > 110. What does getDelegationTokenAs mean?
> > > > > > > >> >> > > > > > >
> > > > > > > >> >> > > > > > > 111. What's the typical rate of getting and
> > > > renewing
> > > > > > > >> delegation
> > > > > > > >> >> > > > tokens?
> > > > > > > >> >> > > > > > > That may have an impact on whether they
> should
> > be
> > > > > > > directed
> > > > > > > >> to the
> > > > > > > >> >> > > > > > > controller.
> > > > > > > >> >> > > > > > >
> > > > > > > >> >> > > > > > > Jun
> > > > > > > >> >> > > > > > >
> > > > > > > >> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth
> > > brahmbhatt <
> > > > > > > >> >> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > > > > >> >> > > > > > >
> > > > > > > >> >> > > > > > > > Hi Jun,
> > > > > > > >> >> > > > > > > >
> > > > > > > >> >> > > > > > > > Thanks for reviewing.
> > > > > > > >> >> > > > > > > >
> > > > > > > >> >> > > > > > > > * We could add a Cluster action to add acls
> > on
> > > > who
> > > > > > can
> > > > > > > >> request
> > > > > > > >> >> > > > > > delegation
> > > > > > > >> >> > > > > > > > tokens. I don't see the use case for that
> yet
> > > but
> > > > > > down
> > > > > > > >> the line
> > > > > > > >> >> > > > when
> > > > > > > >> >> > > > > we
> > > > > > > >> >> > > > > > > > start supporting getDelegationTokenAs it
> will
> > > be
> > > > > > > >> necessary.
> > > > > > > >> >> > > > > > > > * Yes we recommend tokens to be only
> > > > > used/distributed
> > > > > > > >> over
> > > > > > > >> >> > secure
> > > > > > > >> >> > > > > > > channels.
> > > > > > > >> >> > > > > > > > * Depending on what design we end up
> choosing
> > > > > > > >> Invalidation will
> > > > > > > >> >> > > be
> > > > > > > >> >> > > > > > > > responsibility of every broker or
> controller.
> > > > > > > >> >> > > > > > > > * I am not sure if I documented somewhere
> > that
> > > > > > > >> invalidation
> > > > > > > >> >> > will
> > > > > > > >> >> > > > > > directly
> > > > > > > >> >> > > > > > > > go through zookeeper but that is not the
> > > intent.
> > > > > > > >> Invalidation
> > > > > > > >> >> > > will
> > > > > > > >> >> > > > > > either
> > > > > > > >> >> > > > > > > > be request based or due to expiration. No
> > > direct
> > > > > > > >> zookeeper
> > > > > > > >> >> > > > > interaction
> > > > > > > >> >> > > > > > > from
> > > > > > > >> >> > > > > > > > any client.
> > > > > > > >> >> > > > > > > > * "Broker also stores the DelegationToken
> > > without
> > > > > the
> > > > > > > >> hmac in
> > > > > > > >> >> > the
> > > > > > > >> >> > > > > > > > zookeeper." : Sorry about the confusion.
> The
> > > sole
> > > > > > > >> purpose of
> > > > > > > >> >> > > > > zookeeper
> > > > > > > >> >> > > > > > in
> > > > > > > >> >> > > > > > > > this design is as distribution channel for
> > > tokens
> > > > > > > >> between all
> > > > > > > >> >> > > > brokers
> > > > > > > >> >> > > > > > > and a
> > > > > > > >> >> > > > > > > > layer that ensures only tokens that were
> > > > generated
> > > > > by
> > > > > > > >> making a
> > > > > > > >> >> > > > > request
> > > > > > > >> >> > > > > > > to a
> > > > > > > >> >> > > > > > > > broker will be accepted (more on this in
> > second
> > > > > > > >> paragraph). The
> > > > > > > >> >> > > > token
> > > > > > > >> >> > > > > > > > consists of few elements (owner, renewer,
> > uuid
> > > ,
> > > > > > > >> expiration,
> > > > > > > >> >> > > hmac)
> > > > > > > >> >> > > > ,
> > > > > > > >> >> > > > > > one
> > > > > > > >> >> > > > > > > of
> > > > > > > >> >> > > > > > > > which is the finally generated hmac but
> hmac
> > it
> > > > > self
> > > > > > is
> > > > > > > >> >> > derivable
> > > > > > > >> >> > > > if
> > > > > > > >> >> > > > > > you
> > > > > > > >> >> > > > > > > > have all the other elements of the token +
> > > secret
> > > > > key
> > > > > > > to
> > > > > > > >> >> > generate
> > > > > > > >> >> > > > > hmac.
> > > > > > > >> >> > > > > > > > Given zookeeper does not provide SSL
> support
> > we
> > > > do
> > > > > > not
> > > > > > > >> want the
> > > > > > > >> >> > > > > entire
> > > > > > > >> >> > > > > > > > token to be wire transferred to zookeeper
> as
> > > that
> > > > > > will
> > > > > > > >> be an
> > > > > > > >> >> > > > insecure
> > > > > > > >> >> > > > > > > wire
> > > > > > > >> >> > > > > > > > transfer. Instead we only store all the
> other
> > > > > > elements
> > > > > > > >> of a
> > > > > > > >> >> > > > > delegation
> > > > > > > >> >> > > > > > > > tokens. Brokers can read these elements and
> > > > because
> > > > > > > they
> > > > > > > >> also
> > > > > > > >> >> > > have
> > > > > > > >> >> > > > > > access
> > > > > > > >> >> > > > > > > > to secret key they will be able to generate
> > > hmac
> > > > on
> > > > > > > >> their end.
> > > > > > > >> >> > > > > > > >
> > > > > > > >> >> > > > > > > > One of the alternative proposed is to avoid
> > > > > zookeeper
> > > > > > > >> >> > > altogether. A
> > > > > > > >> >> > > > > > > Client
> > > > > > > >> >> > > > > > > > will call broker with required information
> > > > (owner,
> > > > > > > >> renwer,
> > > > > > > >> >> > > > > expiration)
> > > > > > > >> >> > > > > > > and
> > > > > > > >> >> > > > > > > > get back (signed hmac, uuid). Broker won't
> > > store
> > > > > this
> > > > > > > in
> > > > > > > >> >> > > zookeeper.
> > > > > > > >> >> > > > > > From
> > > > > > > >> >> > > > > > > > this point a client can contact any broker
> > with
> > > > all
> > > > > > the
> > > > > > > >> >> > > delegation
> > > > > > > >> >> > > > > > token
> > > > > > > >> >> > > > > > > > info (owner, rewner, expiration, hmac,
> uuid)
> > > the
> > > > > > borker
> > > > > > > >> will
> > > > > > > >> >> > > > > regenerate
> > > > > > > >> >> > > > > > > the
> > > > > > > >> >> > > > > > > > hmac and as long as it matches with hmac
> > > > presented
> > > > > by
> > > > > > > >> client ,
> > > > > > > >> >> > > > broker
> > > > > > > >> >> > > > > > > will
> > > > > > > >> >> > > > > > > > allow the request to authenticate.  Only
> > > problem
> > > > > with
> > > > > > > >> this
> > > > > > > >> >> > > approach
> > > > > > > >> >> > > > > is
> > > > > > > >> >> > > > > > if
> > > > > > > >> >> > > > > > > > the secret key is compromised any client
> can
> > > now
> > > > > > > generate
> > > > > > > >> >> > random
> > > > > > > >> >> > > > > tokens
> > > > > > > >> >> > > > > > > and
> > > > > > > >> >> > > > > > > > they will still be able to authenticate as
> > any
> > > > user
> > > > > > > they
> > > > > > > >> like.
> > > > > > > >> >> > > with
> > > > > > > >> >> > > > > > > > zookeeper we guarantee that only tokens
> > > acquired
> > > > > via
> > > > > > a
> > > > > > > >> broker
> > > > > > > >> >> > > > (using
> > > > > > > >> >> > > > > > some
> > > > > > > >> >> > > > > > > > auth scheme other than delegation token)
> will
> > > be
> > > > > > > >> accepted. We
> > > > > > > >> >> > > need
> > > > > > > >> >> > > > to
> > > > > > > >> >> > > > > > > > discuss which proposal makes more sense and
> > we
> > > > can
> > > > > go
> > > > > > > >> over it
> > > > > > > >> >> > in
> > > > > > > >> >> > > > > > > tomorrow's
> > > > > > > >> >> > > > > > > > meeting.
> > > > > > > >> >> > > > > > > >
> > > > > > > >> >> > > > > > > > Also, can you forward the invite to me?
> > > > > > > >> >> > > > > > > >
> > > > > > > >> >> > > > > > > > Thanks
> > > > > > > >> >> > > > > > > > Parth
> > > > > > > >> >> > > > > > > >
> > > > > > > >> >> > > > > > > >
> > > > > > > >> >> > > > > > > >
> > > > > > > >> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <
> > > > > > > >> jun@confluent.io>
> > > > > > > >> >> > > > wrote:
> > > > > > > >> >> > > > > > > >
> > > > > > > >> >> > > > > > > > > Thanks for the KIP. A few comments.
> > > > > > > >> >> > > > > > > > >
> > > > > > > >> >> > > > > > > > > 100. This potentially can be useful for
> > Kafka
> > > > > > Connect
> > > > > > > >> and
> > > > > > > >> >> > Kafka
> > > > > > > >> >> > > > > rest
> > > > > > > >> >> > > > > > > > proxy
> > > > > > > >> >> > > > > > > > > where a worker agent will need to run a
> > task
> > > on
> > > > > > > behalf
> > > > > > > >> of a
> > > > > > > >> >> > > > client.
> > > > > > > >> >> > > > > > We
> > > > > > > >> >> > > > > > > > will
> > > > > > > >> >> > > > > > > > > likely need to change how those services
> > use
> > > > > Kafka
> > > > > > > >> clients
> > > > > > > >> >> > > > > > > > > (producer/consumer). Instead of a shared
> > > client
> > > > > per
> > > > > > > >> worker,
> > > > > > > >> >> > we
> > > > > > > >> >> > > > will
> > > > > > > >> >> > > > > > > need
> > > > > > > >> >> > > > > > > > a
> > > > > > > >> >> > > > > > > > > client per user task since the
> > authentication
> > > > > > happens
> > > > > > > >> at the
> > > > > > > >> >> > > > > > connection
> > > > > > > >> >> > > > > > > > > level. For Kafka Connect, the renewer
> will
> > be
> > > > the
> > > > > > > >> workers.
> > > > > > > >> >> > So,
> > > > > > > >> >> > > we
> > > > > > > >> >> > > > > > > > probably
> > > > > > > >> >> > > > > > > > > need to allow multiple renewers. For
> Kafka
> > > rest
> > > > > > > proxy,
> > > > > > > >> the
> > > > > > > >> >> > > > renewer
> > > > > > > >> >> > > > > > can
> > > > > > > >> >> > > > > > > > > probably just be the creator of the
> token.
> > > > > > > >> >> > > > > > > > >
> > > > > > > >> >> > > > > > > > > 101. Do we need new acl on who can
> request
> > > > > > delegation
> > > > > > > >> tokens?
> > > > > > > >> >> > > > > > > > >
> > > > > > > >> >> > > > > > > > > 102. Do we recommend people to send
> > > delegation
> > > > > > tokens
> > > > > > > >> in an
> > > > > > > >> >> > > > > encrypted
> > > > > > > >> >> > > > > > > > > channel?
> > > > > > > >> >> > > > > > > > >
> > > > > > > >> >> > > > > > > > > 103. Who is responsible for expiring
> > tokens,
> > > > > every
> > > > > > > >> broker?
> > > > > > > >> >> > > > > > > > >
> > > > > > > >> >> > > > > > > > > 104. For invalidating tokens, would it be
> > > > better
> > > > > to
> > > > > > > do
> > > > > > > >> it in
> > > > > > > >> >> > a
> > > > > > > >> >> > > > > > request
> > > > > > > >> >> > > > > > > > > instead of going to ZK directly?
> > > > > > > >> >> > > > > > > > >
> > > > > > > >> >> > > > > > > > > 105. The terminology of client in the
> wiki
> > > > > > sometimes
> > > > > > > >> refers
> > > > > > > >> >> > to
> > > > > > > >> >> > > > the
> > > > > > > >> >> > > > > > end
> > > > > > > >> >> > > > > > > > > client and some other times refers to the
> > > > client
> > > > > > > using
> > > > > > > >> the
> > > > > > > >> >> > > > > delegation
> > > > > > > >> >> > > > > > > > > tokens. It would be useful to distinguish
> > > > between
> > > > > > the
> > > > > > > >> two.
> > > > > > > >> >> > > > > > > > >
> > > > > > > >> >> > > > > > > > > 106. Could you explain the sentence
> "Broker
> > > > also
> > > > > > > >> stores the
> > > > > > > >> >> > > > > > > > DelegationToken
> > > > > > > >> >> > > > > > > > > without the hmac in the zookeeper." a bit
> > > > more? I
> > > > > > > >> thought the
> > > > > > > >> >> > > > > > > delegation
> > > > > > > >> >> > > > > > > > > token is the hmac.
> > > > > > > >> >> > > > > > > > >
> > > > > > > >> >> > > > > > > > > Thanks,
> > > > > > > >> >> > > > > > > > >
> > > > > > > >> >> > > > > > > > > Jun
> > > > > > > >> >> > > > > > > > >
> > > > > > > >> >> > > > > > > > >
> > > > > > > >> >> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao
> <
> > > > > > > >> jun@confluent.io>
> > > > > > > >> >> > > > wrote:
> > > > > > > >> >> > > > > > > > >
> > > > > > > >> >> > > > > > > > > > Hi, Harsha,
> > > > > > > >> >> > > > > > > > > >
> > > > > > > >> >> > > > > > > > > > Just sent out a KIP meeting invite. We
> > can
> > > > > > discuss
> > > > > > > >> this in
> > > > > > > >> >> > > the
> > > > > > > >> >> > > > > > > meeting
> > > > > > > >> >> > > > > > > > > > tomorrow.
> > > > > > > >> >> > > > > > > > > >
> > > > > > > >> >> > > > > > > > > > Thanks,
> > > > > > > >> >> > > > > > > > > >
> > > > > > > >> >> > > > > > > > > > Jun
> > > > > > > >> >> > > > > > > > > >
> > > > > > > >> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM,
> Harsha <
> > > > > > > >> kafka@harsha.io>
> > > > > > > >> >> > > > wrote:
> > > > > > > >> >> > > > > > > > > >
> > > > > > > >> >> > > > > > > > > >> Hi All,
> > > > > > > >> >> > > > > > > > > >>            Can we have a KIP meeting
> > > around
> > > > > > this.
> > > > > > > >> The KIP
> > > > > > > >> >> > is
> > > > > > > >> >> > > > up
> > > > > > > >> >> > > > > > for
> > > > > > > >> >> > > > > > > > > >>            sometime and if there are
> any
> > > > > > questions
> > > > > > > >> lets
> > > > > > > >> >> > > > quickly
> > > > > > > >> >> > > > > > hash
> > > > > > > >> >> > > > > > > > out
> > > > > > > >> >> > > > > > > > > >>            details.
> > > > > > > >> >> > > > > > > > > >>
> > > > > > > >> >> > > > > > > > > >> Thanks,
> > > > > > > >> >> > > > > > > > > >> Harsha
> > > > > > > >> >> > > > > > > > > >>
> > > > > > > >> >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40 AM,
> parth
> > > > > > > brahmbhatt
> > > > > > > >> wrote:
> > > > > > > >> >> > > > > > > > > >> > That is what the hadoop echo system
> > uses
> > > > so
> > > > > no
> > > > > > > >> good
> > > > > > > >> >> > reason
> > > > > > > >> >> > > > > > really.
> > > > > > > >> >> > > > > > > > We
> > > > > > > >> >> > > > > > > > > >> > could
> > > > > > > >> >> > > > > > > > > >> > change it to whatever is the newest
> > > > > > recommended
> > > > > > > >> standard
> > > > > > > >> >> > > is.
> > > > > > > >> >> > > > > > > > > >> >
> > > > > > > >> >> > > > > > > > > >> > Thanks
> > > > > > > >> >> > > > > > > > > >> > Parth
> > > > > > > >> >> > > > > > > > > >> >
> > > > > > > >> >> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM,
> > Ismael
> > > > > Juma <
> > > > > > > >> >> > > > > ismael@juma.me.uk
> > > > > > > >> >> > > > > > >
> > > > > > > >> >> > > > > > > > > wrote:
> > > > > > > >> >> > > > > > > > > >> >
> > > > > > > >> >> > > > > > > > > >> > > Hi Parth,
> > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > >> >> > > > > > > > > >> > > Thanks for the KIP. I only started
> > > > > reviewing
> > > > > > > >> this and
> > > > > > > >> >> > > may
> > > > > > > >> >> > > > > have
> > > > > > > >> >> > > > > > > > > >> additional
> > > > > > > >> >> > > > > > > > > >> > > questions later. The immediate
> > > question
> > > > > that
> > > > > > > >> came to
> > > > > > > >> >> > > mind
> > > > > > > >> >> > > > is
> > > > > > > >> >> > > > > > our
> > > > > > > >> >> > > > > > > > > >> choice of
> > > > > > > >> >> > > > > > > > > >> > > "DIGEST-MD5" even though it's
> marked
> > > as
> > > > > > > >> OBSOLETE in
> > > > > > > >> >> > the
> > > > > > > >> >> > > > IANA
> > > > > > > >> >> > > > > > > > > Registry
> > > > > > > >> >> > > > > > > > > >> of
> > > > > > > >> >> > > > > > > > > >> > > SASL mechanisms and the original
> RFC
> > > > > (2831)
> > > > > > > has
> > > > > > > >> been
> > > > > > > >> >> > > moved
> > > > > > > >> >> > > > > to
> > > > > > > >> >> > > > > > > > > Historic
> > > > > > > >> >> > > > > > > > > >> > > status:
> > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > >> >> > > > > > > > > >> > > https://tools.ietf.org/html/
> rfc6331
> > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > >> >> > > > > > > > >
> > > > > > > >> >> > > > > >
> > > > > > > >> >> > >
> > > > > > > >>
> > > > >
> > http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > >> >> > > > > > > > > >> > > What is the reasoning behind that
> > > > choice?
> > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > >> >> > > > > > > > > >> > > Thanks,
> > > > > > > >> >> > > > > > > > > >> > > Ismael
> > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > >> >> > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29 PM,
> > Gwen
> > > > > > > Shapira <
> > > > > > > >> >> > > > > > > gwen@confluent.io
> > > > > > > >> >> > > > > > > > >
> > > > > > > >> >> > > > > > > > > >> wrote:
> > > > > > > >> >> > > > > > > > > >> > >
> > > > > > > >> >> > > > > > > > > >> > > > Also comments inline :)
> > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > >> >> > > > > > > > > >> > > > > * I want to emphasize that
> even
> > > > though
> > > > > > > >> delegation
> > > > > > > >> >> > > > tokens
> > > > > > > >> >> > > > > > > are a
> > > > > > > >> >> > > > > > > > > >> Hadoop
> > > > > > > >> >> > > > > > > > > >> > > > > innovation, I feel very
> strongly
> > > > about
> > > > > > not
> > > > > > > >> adding
> > > > > > > >> >> > > > > > dependency
> > > > > > > >> >> > > > > > > > on
> > > > > > > >> >> > > > > > > > > >> Hadoop
> > > > > > > >> >> > > > > > > > > >> > > > > when implementing delegation
> > > tokens
> > > > > for
> > > > > > > >> Kafka. The
> > > > > > > >> >> > > KIP
> > > > > > > >> >> > > > > > > doesn't
> > > > > > > >> >> > > > > > > > > >> imply
> > > > > > > >> >> > > > > > > > > >> > > > > such dependency, but if you
> can
> > > > > > clarify...
> > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > >> >> > > > > > > > > >> > > > > *No hadoop dependency.*
> > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > >> >> > > > > > > > > >> > > > Yay! Just add this to the KIP so
> > no
> > > > one
> > > > > > will
> > > > > > > >> read
> > > > > > > >> >> > the
> > > > > > > >> >> > > > KIP
> > > > > > > >> >> > > > > > and
> > > > > > > >> >> > > > > > > > > panic
> > > > > > > >> >> > > > > > > > > >> > > > three weeks before the next
> > > release...
> > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > >> >> > > > > > > > > >> > > > > * Can we get delegation token
> at
> > > any
> > > > > > time
> > > > > > > >> after
> > > > > > > >> >> > > > > > > > authenticating?
> > > > > > > >> >> > > > > > > > > >> only
> > > > > > > >> >> > > > > > > > > >> > > > > immediately after?
> > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > >> >> > > > > > > > > >> > > > > *As long as you are
> > authenticated
> > > > you
> > > > > > can
> > > > > > > >> get
> > > > > > > >> >> > > > delegation
> > > > > > > >> >> > > > > > > > tokens.
> > > > > > > >> >> > > > > > > > > >> We
> > > > > > > >> >> > > > > > > > > >> > > need
> > > > > > > >> >> > > > > > > > > >> > > > to
> > > > > > > >> >> > > > > > > > > >> > > > > discuss if a client
> > authenticated
> > > > > using
> > > > > > > >> delegation
> > > > > > > >> >> > > > > token,
> > > > > > > >> >> > > > > > > can
> > > > > > > >> >> > > > > > > > > also
> > > > > > > >> >> > > > > > > > > >> > > > acquire
> > > > > > > >> >> > > > > > > > > >> > > > > delegation token again or not.
> > > Also
> > > > > > there
> > > > > > > >> is the
> > > > > > > >> >> > > > > question
> > > > > > > >> >> > > > > > of
> > > > > > > >> >> > > > > > > > do
> > > > > > > >> >> > > > > > > > > we
> > > > > > > >> >> > > > > > > > > >> > > allow
> > > > > > > >> >> > > > > > > > > >> > > > > anyone to acquire delegation
> > token
> > > > or
> > > > > we
> > > > > > > >> want
> > > > > > > >> >> > > specific
> > > > > > > >> >> > > > > > ACLs
> > > > > > > >> >> > > > > > > (I
> > > > > > > >> >> > > > > > > > > >> think
> > > > > > > >> >> > > > > > > > > >> > > its
> > > > > > > >> >> > > > > > > > > >> > > > an
> > > > > > > >> >> > > > > > > > > >> > > > > overkill.)*
> > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > >> >> > > > > > > > > >> > > > I agree that ACLs is an
> overkill.
> > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > >> >> > > > > > > > > >> > > > I think we are debating two
> > options:
> > > > > > Either
> > > > > > > >> require
> > > > > > > >> >> > > > > Kerberos
> > > > > > > >> >> > > > > > > > auth
> > > > > > > >> >> > > > > > > > > >> for
> > > > > > > >> >> > > > > > > > > >> > > > renewal or require non-owners to
> > > > renew.
> > > > > > > >> >> > > > > > > > > >> > > > I *think* the latter is simpler
> > (it
> > > > > > > basically
> > > > > > > >> >> > require
> > > > > > > >> >> > > a
> > > > > > > >> >> > > > > "job
> > > > > > > >> >> > > > > > > > > master"
> > > > > > > >> >> > > > > > > > > >> > > > to take responsibility for the
> > > > renewal,
> > > > > it
> > > > > > > >> will have
> > > > > > > >> >> > > its
> > > > > > > >> >> > > > > own
> > > > > > > >> >> > > > > > > > > >> identity
> > > > > > > >> >> > > > > > > > > >> > > > anyway and I think this is the
> > > correct
> > > > > > > design
> > > > > > > >> >> > pattern
> > > > > > > >> >> > > > > > anyway.
> > > > > > > >> >> > > > > > > > For
> > > > > > > >> >> > > > > > > > > >> > > > storm, I'd expect Nimbus to
> > > coordinate
> > > > > > > >> renewals?),
> > > > > > > >> >> > but
> > > > > > > >> >> > > > it
> > > > > > > >> >> > > > > is
> > > > > > > >> >> > > > > > > > hard
> > > > > > > >> >> > > > > > > > > to
> > > > > > > >> >> > > > > > > > > >> > > > debate simplicity without
> looking
> > at
> > > > the
> > > > > > > code
> > > > > > > >> >> > changes
> > > > > > > >> >> > > > > > > required.
> > > > > > > >> >> > > > > > > > If
> > > > > > > >> >> > > > > > > > > >> you
> > > > > > > >> >> > > > > > > > > >> > > > have a draft of how the "require
> > > > > Kerberos"
> > > > > > > >> will look
> > > > > > > >> >> > > in
> > > > > > > >> >> > > > > > Kafka
> > > > > > > >> >> > > > > > > > > code,
> > > > > > > >> >> > > > > > > > > >> > > > I'll be happy to take a look.
> > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > >> >> > > > > > > > > >> > > > > * My understanding is that
> > tokens
> > > > will
> > > > > > > >> propagate
> > > > > > > >> >> > via
> > > > > > > >> >> > > > ZK
> > > > > > > >> >> > > > > > but
> > > > > > > >> >> > > > > > > > > >> without
> > > > > > > >> >> > > > > > > > > >> > > > > additional changes to
> > > UpdateMetadata
> > > > > > > >> protocol,
> > > > > > > >> >> > > > correct?
> > > > > > > >> >> > > > > > > > Clients
> > > > > > > >> >> > > > > > > > > >> > > > > currently don't retry on SASL
> > auth
> > > > > > failure
> > > > > > > >> (IIRC),
> > > > > > > >> >> > > but
> > > > > > > >> >> > > > > > since
> > > > > > > >> >> > > > > > > > the
> > > > > > > >> >> > > > > > > > > >> > > > > tokens propagate between
> brokers
> > > > > asynch,
> > > > > > > we
> > > > > > > >> will
> > > > > > > >> >> > > need
> > > > > > > >> >> > > > to
> > > > > > > >> >> > > > > > > > retry a
> > > > > > > >> >> > > > > > > > > >> bit
> > > > > > > >> >> > > > > > > > > >> > > > > to avoid clients failing auth
> > due
> > > to
> > > > > > > timing
> > > > > > > >> >> > issues.
> > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > >> >> > > > > > > > > >> > > > > *I am considering 2
> alternatives
> > > > right
> > > > > > > now.
> > > > > > > >> The
> > > > > > > >> >> > > > current
> > > > > > > >> >> > > > > > > > > documented
> > > > > > > >> >> > > > > > > > > >> > > > approach
> > > > > > > >> >> > > > > > > > > >> > > > > is zookeeper based and it does
> > not
> > > > > > require
> > > > > > > >> any
> > > > > > > >> >> > > changes
> > > > > > > >> >> > > > > to
> > > > > > > >> >> > > > > > > > > >> > > UpdateMetadata
> > > > > > > >> >> > > > > > > > > >> > > > > protocol. An alternative
> > approach
> > > > can
> > > > > > > remove
> > > > > > > >> >> > > zookeeper
> > > > > > > >> >> > > > > > > > > dependency
> > > > > > > >> >> > > > > > > > > >> as
> > > > > > > >> >> > > > > > > > > >> > > well
> > > > > > > >> >> > > > > > > > > >> > > > > but we can discuss that in KIP
> > > > > > discussion
> > > > > > > >> call.*
> > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > >> >> > > > > > > > > >> > > > Oooh! Sounds interesting. Do you
> > > want
> > > > to
> > > > > > > ping
> > > > > > > >> Jun to
> > > > > > > >> >> > > > > > arrange a
> > > > > > > >> >> > > > > > > > > call?
> > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > >> >> > > > > > > > > >> > > > > * I liked Ashish's suggestion
> of
> > > > > having
> > > > > > > >> just the
> > > > > > > >> >> > > > > > controller
> > > > > > > >> >> > > > > > > > > issue
> > > > > > > >> >> > > > > > > > > >> the
> > > > > > > >> >> > > > > > > > > >> > > > > delegation tokens, to avoid
> > > syncing
> > > > a
> > > > > > > shared
> > > > > > > >> >> > secret.
> > > > > > > >> >> > > > Not
> > > > > > > >> >> > > > > > > sure
> > > > > > > >> >> > > > > > > > if
> > > > > > > >> >> > > > > > > > > >> we
> > > > > > > >> >> > > > > > > > > >> > > > > want to continue the
> discussion
> > > here
> > > > > or
> > > > > > on
> > > > > > > >> the
> > > > > > > >> >> > > wiki. I
> > > > > > > >> >> > > > > > think
> > > > > > > >> >> > > > > > > > > that
> > > > > > > >> >> > > > > > > > > >> we
> > > > > > > >> >> > > > > > > > > >> > > > > can decouple the problem of
> > "token
> > > > > > > >> distribution"
> > > > > > > >> >> > > from
> > > > > > > >> >> > > > > > > "shared
> > > > > > > >> >> > > > > > > > > >> secret
> > > > > > > >> >> > > > > > > > > >> > > > > distribution" and use the
> > > controller
> > > > > as
> > > > > > > the
> > > > > > > >> only
> > > > > > > >> >> > > token
> > > > > > > >> >> > > > > > > > generator
> > > > > > > >> >> > > > > > > > > >> to
> > > > > > > >> >> > > > > > > > > >> > > > > solve the second issue, while
> > > still
> > > > > > using
> > > > > > > >> ZK async
> > > > > > > >> >> > > to
> > > > > > > >> >> > > > > > > > distribute
> > > > > > > >> >> > > > > > > > > >> > > > > tokens.
> > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > >> >> > > > > > > > > >> > > > > *As mentioned in the previous
> > > Email
> > > > I
> > > > > am
> > > > > > > >> fine with
> > > > > > > >> >> > > > that
> > > > > > > >> >> > > > > > > > approach
> > > > > > > >> >> > > > > > > > > >> as
> > > > > > > >> >> > > > > > > > > >> > > long
> > > > > > > >> >> > > > > > > > > >> > > > as
> > > > > > > >> >> > > > > > > > > >> > > > > we agree that the extra
> > complexity
> > > > of
> > > > > > > >> >> > > adding/updating
> > > > > > > >> >> > > > > APIS
> > > > > > > >> >> > > > > > > > adds
> > > > > > > >> >> > > > > > > > > >> enough
> > > > > > > >> >> > > > > > > > > >> > > > > value. The advantage with the
> > > > > controller
> > > > > > > >> approach
> > > > > > > >> >> > is
> > > > > > > >> >> > > > > > secret
> > > > > > > >> >> > > > > > > > > >> rotation
> > > > > > > >> >> > > > > > > > > >> > > can
> > > > > > > >> >> > > > > > > > > >> > > > be
> > > > > > > >> >> > > > > > > > > >> > > > > automated,frequent and would
> not
> > > > > require
> > > > > > > >> >> > > deployment. *
> > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > >> >> > > > > > > > > >> > > > Can you detail the extra
> > complexity
> > > > (or
> > > > > > > point
> > > > > > > >> me to
> > > > > > > >> >> > > the
> > > > > > > >> >> > > > > > email
> > > > > > > >> >> > > > > > > I
> > > > > > > >> >> > > > > > > > > >> > > > missed?) - which Apis are
> > required?
> > > > > > > >> >> > > > > > > > > >> > > > As far as I can tell, clients
> can
> > > > > already
> > > > > > > >> find the
> > > > > > > >> >> > > > > > controller
> > > > > > > >> >> > > > > > > > from
> > > > > > > >> >> > > > > > > > > >> > > > metadata. I'm a bit more
> concerned
> > > > about
> > > > > > > >> controller
> > > > > > > >> >> > > > load.
> > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > >> >> > > > > > > > > >> > > > > * While I like the idea of
> > forcing
> > > > > > > kerberos
> > > > > > > >> auth
> > > > > > > >> >> > for
> > > > > > > >> >> > > > > > > renwal, I
> > > > > > > >> >> > > > > > > > > >> think
> > > > > > > >> >> > > > > > > > > >> > > > > it mixes the transport layer
> the
> > > the
> > > > > > > request
> > > > > > > >> >> > content
> > > > > > > >> >> > > > in
> > > > > > > >> >> > > > > a
> > > > > > > >> >> > > > > > > > pretty
> > > > > > > >> >> > > > > > > > > >> ugly
> > > > > > > >> >> > > > > > > > > >> > > > > way. Perhaps limiting renewer
> to
> > > > > > non-owner
> > > > > > > >> is
> > > > > > > >> >> > > better.
> > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > >> >> > > > > > > > > >> > > > > *I feel this is a necessary
> > evil.
> > > > > While
> > > > > > > >> this will
> > > > > > > >> >> > > make
> > > > > > > >> >> > > > > the
> > > > > > > >> >> > > > > > > > kafka
> > > > > > > >> >> > > > > > > > > >> code
> > > > > > > >> >> > > > > > > > > >> > > > > pretty straight forward ,
> > forcing
> > > > > > renewer
> > > > > > > >> to
> > > > > > > >> >> > > > non-owner
> > > > > > > >> >> > > > > > > pushes
> > > > > > > >> >> > > > > > > > > >> the code
> > > > > > > >> >> > > > > > > > > >> > > > > ugliness to client and makes
> it
> > > even
> > > > > > > harder
> > > > > > > >> to
> > > > > > > >> >> > > > > > integrate.  *
> > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > >> >> > > > > > > > > >> > > > As mentioned before, I don't
> think
> > > the
> > > > > > > >> "renewal by
> > > > > > > >> >> > > > other"
> > > > > > > >> >> > > > > > > > approach
> > > > > > > >> >> > > > > > > > > >> is
> > > > > > > >> >> > > > > > > > > >> > > > that ugly for the clients we
> > expect
> > > to
> > > > > use
> > > > > > > >> >> > delegation
> > > > > > > >> >> > > > > tokens
> > > > > > > >> >> > > > > > > > since
> > > > > > > >> >> > > > > > > > > >> > > > they will have an app-master of
> > some
> > > > > sort
> > > > > > > who
> > > > > > > >> >> > > requested
> > > > > > > >> >> > > > > the
> > > > > > > >> >> > > > > > > > token
> > > > > > > >> >> > > > > > > > > to
> > > > > > > >> >> > > > > > > > > >> > > > begin with.
> > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > >> >> > > > > > > > > >> > > > > The response for my question
> on
> > > how
> > > > > > > multiple
> > > > > > > >> >> > > > identities
> > > > > > > >> >> > > > > > will
> > > > > > > >> >> > > > > > > > be
> > > > > > > >> >> > > > > > > > > >> > > > > handled wasn't super clear to
> > me -
> > > > > > AFAIK,
> > > > > > > >> we don't
> > > > > > > >> >> > > > > > > > authenticate
> > > > > > > >> >> > > > > > > > > >> each
> > > > > > > >> >> > > > > > > > > >> > > > > request, we authenticate
> > > > connections.
> > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > >> >> > > > > > > > > >> > > > > *We authenticate connections,
> > and
> > > > only
> > > > > > > when
> > > > > > > >> they
> > > > > > > >> >> > are
> > > > > > > >> >> > > > > being
> > > > > > > >> >> > > > > > > > > >> established.
> > > > > > > >> >> > > > > > > > > >> > > > Let
> > > > > > > >> >> > > > > > > > > >> > > > > me try to phrase this as a
> > > question,
> > > > > in
> > > > > > > >> absence of
> > > > > > > >> >> > > > > > > delegation
> > > > > > > >> >> > > > > > > > > >> tokens if
> > > > > > > >> >> > > > > > > > > >> > > > we
> > > > > > > >> >> > > > > > > > > >> > > > > had to support the use case
> > using
> > > > user
> > > > > > > >> TGT's how
> > > > > > > >> >> > > would
> > > > > > > >> >> > > > > we
> > > > > > > >> >> > > > > > do
> > > > > > > >> >> > > > > > > > it?
> > > > > > > >> >> > > > > > > > > >> My
> > > > > > > >> >> > > > > > > > > >> > > point
> > > > > > > >> >> > > > > > > > > >> > > > > was it would be no different
> > with
> > > > > > > delegation
> > > > > > > >> >> > tokens.
> > > > > > > >> >> > > > The
> > > > > > > >> >> > > > > > use
> > > > > > > >> >> > > > > > > > > case
> > > > > > > >> >> > > > > > > > > >> you
> > > > > > > >> >> > > > > > > > > >> > > are
> > > > > > > >> >> > > > > > > > > >> > > > > describing seems more like
> > > > > > impersonation.*
> > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > >> >> > > > > > > > > >> > > > Yeah, I thought that one of the
> > > things
> > > > > > that
> > > > > > > >> >> > delegation
> > > > > > > >> >> > > > > > tokens
> > > > > > > >> >> > > > > > > > > >> handled.
> > > > > > > >> >> > > > > > > > > >> > > > Maybe I got it wrong :)
> > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > >> >> > > > > > > > > >> > > > Thanks for the detailed answers.
> > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > >> >> > > > > > > > > >> > > > Gwen
> > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > >> >> > > > > > > > > >> > > >
> > > > > > > >> >> > > > > > > > > >> > > > > Thanks
> > > > > > > >> >> > > > > > > > > >> > > > > Parth
> > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > >> >> > > > > > > > > >> > > > > On Fri, May 13, 2016 at 12:19
> > AM,
> > > > Gwen
> > > > > > > >> Shapira <
> > > > > > > >> >> > > > > > > > > gwen@confluent.io
> > > > > > > >> >> > > > > > > > > >> >
> > > > > > > >> >> > > > > > > > > >> > > > wrote:
> > > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > > >> >> > > > > > > > > >> > > > >> Hi Parth and Harsha,
> > > > > > > >> >> > > > > > > > > >> > > > >>
> > > > > > > >> >> > > > > > > > > >> > > > >> Few more comments:
> > > > > > > >> >> > > > > > > > > >> > > > >>
> > > > > > > >> >> > > > > > > > > >> > > > >> * The API / RequestResponse
> > > section
> > > > > > > >> doesn't seem
> > > > > > > >> >> > to
> > > > > > > >> >> > > > > have
> > > > > > > >> >> > > > > > > good
> > > > > > > >> >> > > > > > > > > >> > > > >> description of the changes to
> > the
> > > > > Kafka
> > > > > > > >> Protocol.
> > > > > > > >> >> > > > > Sounds
> > > > > > > >> >> > > > > > > like
> > > > > > > >> >> > > > > > > > > >> you are
> > > > > > > >> >> > > > > > > > > >> > > > >> proposing new
> > > > DelegationTokenRequest
> > > > > > and
> > > > > > > >> >> > > > > > RenewTokenRequest
> > > > > > > >> >> > > > > > > > (and
> > > > > > > >> >> > > > > > > > > >> > > > >> matching responses), without
> > > > > detailing
> > > > > > > the
> > > > > > > >> >> > contents
> > > > > > > >> >> > > > of
> > > > > > > >> >> > > > > > the
> > > > > > > >> >> > > > > > > > > >> requests
> > > > > > > >> >> > > > > > > > > >> > > > >> and responses? Or rather, you
> > > show
> > > > > the
> > > > > > > >> class
> > > > > > > >> >> > > > interface,
> > > > > > > >> >> > > > > > but
> > > > > > > >> >> > > > > > > > not
> > > > > > > >> >> > > > > > > > > >> the
> > > > > > > >> >> > > > > > > > > >> > > > >> underlying protocol. This
> could
> > > be
> > > > > seen
> > > > > > > as
> > > > > > > >> an
> > > > > > > >> >> > > > > > > implementation
> > > > > > > >> >> > > > > > > > > >> detail,
> > > > > > > >> >> > > > > > > > > >> > > > >> but since the binary protocol
> > is
> > > > what
> > > > > > we
> > > > > > > >> provide
> > > > > > > >> >> > to
> > > > > > > >> >> > > > > > > non-Java
> > > > > > > >> >> > > > > > > > > >> clients,
> > > > > > > >> >> > > > > > > > > >> > > > >> we need to show the changes
> > > there.
> > > > > > > >> >> > > > > > > > > >> > > > >>
> > > > > > > >> >> > > > > > > > > >> > > > >> * getDelegationToken sounds
> > like
> > > > > > > >> >> > > > > > > > delegationTokenRequestHandler?
> > > > > > > >> >> > > > > > > > > >> Is it
> > > > > > > >> >> > > > > > > > > >> > > > >> planned to be part of
> KafkaApi?
> > > or
> > > > > > > Client?
> > > > > > > >> Its
> > > > > > > >> >> > > > > unclear...
> > > > > > > >> >> > > > > > > > > >> > > > >>
> > > > > > > >> >> > > > > > > > > >> > > > >> * I want to emphasize that
> even
> > > > > though
> > > > > > > >> delegation
> > > > > > > >> >> > > > > tokens
> > > > > > > >> >> > > > > > > are
> > > > > > > >> >> > > > > > > > a
> > > > > > > >> >> > > > > > > > > >> Hadoop
> > > > > > > >> >> > > > > > > > > >> > > > >> innovation, I feel very
> > strongly
> > > > > about
> > > > > > > not
> > > > > > > >> adding
> > > > > > > >> >> > > > > > > dependency
> > > > > > > >> >> > > > > > > > on
> > > > > > > >> >> > > > > > > > > >> Hadoop
> > > > > > > >> >> > > > > > > > > >> > > > >> when implementing delegation
> > > tokens
> > > > > for
> > > > > > > >> Kafka.
> > > > > > > >> >> > The
> > > > > > > >> >> > > > KIP
> > > > > > > >> >> > > > > > > > doesn't
> > > > > > > >> >> > > > > > > > > >> imply
> > > > > > > >> >> > > > > > > > > >> > > > >> such dependency, but if you
> can
> > > > > > > clarify...
> > > > > > > >> >> > > > > > > > > >> > > > >>
> > > > > > > >> >> > > > > > > > > >> > > > >> * Can we get delegation token
> > at
> > > > any
> > > > > > time
> > > > > > > >> after
> > > > > > > >> >> > > > > > > > authenticating?
> > > > > > > >> >> > > > > > > > > >> only
> > > > > > > >> >> > > > > > > > > >> > > > >> immediately after?
> > > > > > > >> >> > > > > > > > > >> > > > >>
> > > > > > > >> >> > > > > > > > > >> > > > >> * My understanding is that
> > tokens
> > > > > will
> > > > > > > >> propagate
> > > > > > > >> >> > > via
> > > > > > > >> >> > > > ZK
> > > > > > > >> >> > > > > > but
> > > > > > > >> >> > > > > > > > > >> without
> > > > > > > >> >> > > > > > > > > >> > > > >> additional changes to
> > > > UpdateMetadata
> > > > > > > >> protocol,
> > > > > > > >> >> > > > correct?
> > > > > > > >> >> > > > > > > > Clients
> > > > > > > >> >> > > > > > > > > >> > > > >> currently don't retry on SASL
> > > auth
> > > > > > > failure
> > > > > > > >> >> > (IIRC),
> > > > > > > >> >> > > > but
> > > > > > > >> >> > > > > > > since
> > > > > > > >> >> > > > > > > > > the
> > > > > > > >> >> > > > > > > > > >> > > > >> tokens propagate between
> > brokers
> > > > > > asynch,
> > > > > > > >> we will
> > > > > > > >> >> > > need
> > > > > > > >> >> > > > > to
> > > > > > > >> >> > > > > > > > retry
> > > > > > > >> >> > > > > > > > > a
> > > > > > > >> >> > > > > > > > > >> bit
> > > > > > > >> >> > > > > > > > > >> > > > >> to avoid clients failing auth
> > due
> > > > to
> > > > > > > timing
> > > > > > > >> >> > issues.
> > > > > > > >> >> > > > > > > > > >> > > > >>
> > > > > > > >> >> > > > > > > > > >> > > > >> * Strongly agreeing on
> clients
> > > not
> > > > > > > >> touching ZK
> > > > > > > >> >> > > > directly
> > > > > > > >> >> > > > > > :)
> > > > > > > >> >> > > > > > > > > >> > > > >>
> > > > > > > >> >> > > > > > > > > >> > > > >> * I liked Ashish's suggestion
> > of
> > > > > having
> > > > > > > >> just the
> > > > > > > >> >> > > > > > controller
> > > > > > > >> >> > > > > > > > > >> issue the
> > > > > > > >> >> > > > > > > > > >> > > > >> delegation tokens, to avoid
> > > > syncing a
> > > > > > > >> shared
> > > > > > > >> >> > > secret.
> > > > > > > >> >> > > > > Not
> > > > > > > >> >> > > > > > > sure
> > > > > > > >> >> > > > > > > > > if
> > > > > > > >> >> > > > > > > > > >> we
> > > > > > > >> >> > > > > > > > > >> > > > >> want to continue the
> discussion
> > > > here
> > > > > or
> > > > > > > on
> > > > > > > >> the
> > > > > > > >> >> > > wiki.
> > > > > > > >> >> > >
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Harsha Chintalapani <ka...@harsha.io>]

Hi Grant,
          We are working on it. Will add the details to KIP about the
request protocol.

Thanks,
Harsha

On Mon, Jul 18, 2016 at 6:50 AM Grant Henke <gh...@cloudera.com> wrote:

> Hi Parth,
>
> Are you still working on this? If you need any help please don't hesitate
> to ask.
>
> Thanks,
> Grant
>
> On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <ju...@confluent.io> wrote:
>
> > Parth,
> >
> > Thanks for the reply.
> >
> > It makes sense to only allow the renewal by users that authenticated
> using
> > *non* delegation token mechanism. Then, should we make the renewal a
> list?
> > For example, in the case of rest proxy, it will be useful for every
> > instance of rest proxy to be able to renew the tokens.
> >
> > It would be clearer if we can document the request protocol like
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945):(VotedandPlannedforin0.10.1.0)
> > .
> >
> > It would also be useful to document the client APIs.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Jun 28, 2016 at 2:55 PM, parth brahmbhatt <
> > brahmbhatt.parth@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I am suggesting that we will only allow the renewal by users that
> > > authenticated using *non* delegation token mechanism. For example, If
> > user
> > > Alice authenticated using kerberos and requested delegation tokens,
> only
> > > user Alice authenticated via non delegation token mechanism can renew.
> > > Clients that have  access to delegation tokens can not issue renewal
> > > request for renewing their own token and this is primarily important to
> > > reduce the time window for which a compromised token will be valid.
> > >
> > > To clarify, Yes any authenticated user can request delegation tokens
> but
> > > even here I would recommend to avoid creating a chain where a client
> > > authenticated via delegation token request for more delegation tokens.
> > > Basically anyone can request delegation token, as long as they
> > authenticate
> > > via a non delegation token mechanism.
> > >
> > > Aren't classes listed here
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-PublicInterfaces
> > > >
> > > sufficient?
> > >
> > > Thanks
> > > Parth
> > >
> > >
> > >
> > > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Parth,
> > > >
> > > > Thanks for the reply. A couple of comments inline below.
> > > >
> > > > On Tue, Jun 21, 2016 at 10:36 AM, parth brahmbhatt <
> > > > brahmbhatt.parth@gmail.com> wrote:
> > > >
> > > > > 1. Who / how are tokens renewed? By original requester only? or
> using
> > > > > Kerberos
> > > > > auth only?
> > > > > My recommendation is to do this only using Kerberos auth and only
> > threw
> > > > the
> > > > > renewer specified during the acquisition request.
> > > > >
> > > > >
> > > > Hmm, not sure that I follow this. Are you saying that any client
> > > > authenticated with the delegation token can renew, i.e. there is no
> > > renewer
> > > > needed?
> > > >
> > > > Also, just to be clear, any authenticated client (either through SASL
> > or
> > > > SSL) can request a delegation token for the authenticated user,
> right?
> > > >
> > > >
> > > > > 2. Are tokens stored on each broker or in ZK?
> > > > > My recommendation is still to store in ZK or not store them at all.
> > The
> > > > > whole controller based distribution is too much overhead with not
> > much
> > > to
> > > > > achieve.
> > > > >
> > > > > 3. How are tokens invalidated / expired?
> > > > > Either by expiration time out or through an explicit request to
> > > > invalidate.
> > > > >
> > > > > 4. Which encryption algorithm is used?
> > > > > SCRAM
> > > > >
> > > > > 5. What is the impersonation proposal (it wasn't in the KIP but was
> > > > > discussed
> > > > > in this thread)?
> > > > > There is no imperonation proposal. I tried and explained how its a
> > > > > different problem and why its not really necessary to discuss that
> as
> > > > part
> > > > > of this KIP.  This KIP will not support any impersonation, it will
> > just
> > > > be
> > > > > another way to authenticate.
> > > > >
> > > > > 6. Do we need new ACLs, if so - for what actions?
> > > > > We do not need new ACLs.
> > > > >
> > > > >
> > > > Could we document the format of the new request/response and their
> > > > associated Resource and Operation for ACL?
> > > >
> > > >
> > > > > 7. How would the delegation token be configured in the client?
> > > > > Should be through config. I wasn't planning on supporting JAAS for
> > > > tokens.
> > > > > I don't believe hadoop does this either.
> > > > >
> > > > > Thanks
> > > > > Parth
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <ju...@confluent.io> wrote:
> > > > >
> > > > > > Harsha,
> > > > > >
> > > > > > Another question.
> > > > > >
> > > > > > 9. How would the delegation token be configured in the client?
> The
> > > > > standard
> > > > > > way is to do this through JAAS. However, we will need to think
> > > through
> > > > if
> > > > > > this is convenient in a shared environment. For example, when a
> new
> > > > task
> > > > > is
> > > > > > added to a Storm worker node, do we need to dynamically add a new
> > > > section
> > > > > > in the JAAS file? It may be more convenient if we can pass in the
> > > token
> > > > > > through the config directly w/o going through JAAS.
> > > > > >
> > > > > > Are you or Parth still actively working on this KIP?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <ju...@confluent.io>
> wrote:
> > > > > >
> > > > > > > Just to add on that list.
> > > > > > >
> > > > > > > 2. It would be good to document the format of the data stored
> in
> > > ZK.
> > > > > > > 7. Earlier, there was a discussion on whether the tokens should
> > be
> > > > > > > propagated through ZK like config/acl/quota, or through the
> > > > controller.
> > > > > > > Currently, the controller is only designed for propagating
> topic
> > > > > > metadata,
> > > > > > > but not other data.
> > > > > > > 8. Should we use SCRAM to send the token instead of DIGEST-MD5
> > > since
> > > > > it's
> > > > > > > deprecated?
> > > > > > >
> > > > > > > Also, the images in the wiki seem broken.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <
> > gwen@confluent.io>
> > > > > > wrote:
> > > > > > >
> > > > > > >> From what I can see, remaining questions are:
> > > > > > >>
> > > > > > >> 1. Who / how are tokens renewed? By original requester only?
> or
> > > > using
> > > > > > >> Kerberos auth only?
> > > > > > >> 2. Are tokens stored on each broker or in ZK?
> > > > > > >> 3. How are tokens invalidated / expired?
> > > > > > >> 4. Which encryption algorithm is used?
> > > > > > >> 5. What is the impersonation proposal (it wasn't in the KIP
> but
> > > was
> > > > > > >> discussed in this thread)?
> > > > > > >> 6. Do we need new ACLs, if so - for what actions?
> > > > > > >>
> > > > > > >> Gwen
> > > > > > >>
> > > > > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <ka...@harsha.io>
> wrote:
> > > > > > >> > Jun & Ismael,
> > > > > > >> >                          Unfortunately I couldn't attend the
> > KIP
> > > > > > meeting
> > > > > > >> >                          when delegation tokens discussed.
> > > > > Appreciate
> > > > > > if
> > > > > > >> >                          you can update the thread if you
> have
> > > any
> > > > > > >> >                          further questions.
> > > > > > >> > Thanks,
> > > > > > >> > Harsha
> > > > > > >> >
> > > > > > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
> > > > > > >> >> It seems that the links to images in the KIP are broken.
> > > > > > >> >>
> > > > > > >> >> Liquan
> > > > > > >> >>
> > > > > > >> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
> > > > > > >> >> brahmbhatt.parth@gmail.com> wrote:
> > > > > > >> >>
> > > > > > >> >> > 110. What does getDelegationTokenAs mean?
> > > > > > >> >> > In the current proposal we only allow a user to get
> > > delegation
> > > > > > token
> > > > > > >> for
> > > > > > >> >> > the identity that it authenticated as using another
> > > mechanism,
> > > > > i.e.
> > > > > > >> A user
> > > > > > >> >> > that authenticate using a keytab for principal
> > > > user1@EXAMPLE.COM
> > > > > > >> will get
> > > > > > >> >> > delegation tokens for that user only. In future I think
> we
> > > will
> > > > > > have
> > > > > > >> to
> > > > > > >> >> > extend support such that we allow some set of users (
> > > > > > >> >> > kafka-rest-user@EXAMPLE.COM, storm-nimbus@EXAMPLE.COM)
> to
> > > > > acquire
> > > > > > >> >> > delegation tokens on behalf of other users whose identity
> > > they
> > > > > have
> > > > > > >> >> > verified independently.  Kafka brokers will have ACLs to
> > > > control
> > > > > > >> which
> > > > > > >> >> > users are allowed to impersonate other users and get
> tokens
> > > on
> > > > > > >> behalf of
> > > > > > >> >> > them. Overall Impersonation is a whole different problem
> in
> > > my
> > > > > > >> opinion and
> > > > > > >> >> > I think we can tackle it in separate KIP.
> > > > > > >> >> >
> > > > > > >> >> > 111. What's the typical rate of getting and renewing
> > > delegation
> > > > > > >> tokens?
> > > > > > >> >> > Typically this should be very very low, 1 request per
> > minute
> > > > is a
> > > > > > >> >> > relatively high estimate. However it depends on the token
> > > > > > >> expiration. I am
> > > > > > >> >> > less worried about the extra load it puts on controller
> vs
> > > the
> > > > > > added
> > > > > > >> >> > complexity and the value it offers.
> > > > > > >> >> >
> > > > > > >> >> > Thanks
> > > > > > >> >> > Parth
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <
> > > > ismael@juma.me.uk>
> > > > > > >> wrote:
> > > > > > >> >> >
> > > > > > >> >> > > Thanks Rajini. It would probably require a separate KIP
> > as
> > > it
> > > > > > will
> > > > > > >> >> > > introduce user visible changes. We could also update
> > KIP-48
> > > > to
> > > > > > >> have this
> > > > > > >> >> > > information, but it seems cleaner to do it separately.
> We
> > > can
> > > > > > >> discuss
> > > > > > >> >> > that
> > > > > > >> >> > > in the KIP call today.
> > > > > > >> >> > >
> > > > > > >> >> > > Ismael
> > > > > > >> >> > >
> > > > > > >> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <
> > > > > > >> >> > > rajinisivaram@googlemail.com> wrote:
> > > > > > >> >> > >
> > > > > > >> >> > > > Ismael,
> > > > > > >> >> > > >
> > > > > > >> >> > > > I have created a JIRA (
> > > > > > >> >> > https://issues.apache.org/jira/browse/KAFKA-3751)
> > > > > > >> >> > > > for adding SCRAM as a SASL mechanism. Would that need
> > > > another
> > > > > > >> KIP? If
> > > > > > >> >> > > > KIP-48 will use this mechanism, can this just be a
> JIRA
> > > > that
> > > > > > gets
> > > > > > >> >> > > reviewed
> > > > > > >> >> > > > when the PR is ready?
> > > > > > >> >> > > >
> > > > > > >> >> > > > Thank you,
> > > > > > >> >> > > >
> > > > > > >> >> > > > Rajini
> > > > > > >> >> > > >
> > > > > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <
> > > > > > ismael@juma.me.uk>
> > > > > > >> >> > wrote:
> > > > > > >> >> > > >
> > > > > > >> >> > > > > Thanks Rajini, SCRAM seems like a good candidate.
> > > > > > >> >> > > > >
> > > > > > >> >> > > > > Gwen had independently mentioned this as a SASL
> > > mechanism
> > > > > > that
> > > > > > >> might
> > > > > > >> >> > be
> > > > > > >> >> > > > > useful for Kafka and I have been meaning to check
> it
> > in
> > > > > more
> > > > > > >> detail.
> > > > > > >> >> > > Good
> > > > > > >> >> > > > > to know that you are willing to contribute an
> > > > > implementation.
> > > > > > >> Maybe
> > > > > > >> >> > we
> > > > > > >> >> > > > > should file a separate JIRA for this?
> > > > > > >> >> > > > >
> > > > > > >> >> > > > > Ismael
> > > > > > >> >> > > > >
> > > > > > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram <
> > > > > > >> >> > > > > rajinisivaram@googlemail.com> wrote:
> > > > > > >> >> > > > >
> > > > > > >> >> > > > > > SCRAM (Salted Challenge Response Authentication
> > > > > Mechanism)
> > > > > > >> is a
> > > > > > >> >> > > better
> > > > > > >> >> > > > > > mechanism than Digest-MD5. Java doesn't come
> with a
> > > > > > built-in
> > > > > > >> SCRAM
> > > > > > >> >> > > > > > SaslServer or SaslClient, but I will be happy to
> > add
> > > > > > support
> > > > > > >> in
> > > > > > >> >> > Kafka
> > > > > > >> >> > > > > since
> > > > > > >> >> > > > > > it would be a useful mechanism to support anyway.
> > > > > > >> >> > > > > > https://tools.ietf.org/html/rfc7677 describes
> the
> > > > > protocol
> > > > > > >> for
> > > > > > >> >> > > > > > SCRAM-SHA-256.
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <
> > > > > jun@confluent.io
> > > > > > >
> > > > > > >> wrote:
> > > > > > >> >> > > > > >
> > > > > > >> >> > > > > > > Parth,
> > > > > > >> >> > > > > > >
> > > > > > >> >> > > > > > > Thanks for the explanation. A couple of more
> > > > questions.
> > > > > > >> >> > > > > > >
> > > > > > >> >> > > > > > > 110. What does getDelegationTokenAs mean?
> > > > > > >> >> > > > > > >
> > > > > > >> >> > > > > > > 111. What's the typical rate of getting and
> > > renewing
> > > > > > >> delegation
> > > > > > >> >> > > > tokens?
> > > > > > >> >> > > > > > > That may have an impact on whether they should
> be
> > > > > > directed
> > > > > > >> to the
> > > > > > >> >> > > > > > > controller.
> > > > > > >> >> > > > > > >
> > > > > > >> >> > > > > > > Jun
> > > > > > >> >> > > > > > >
> > > > > > >> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth
> > brahmbhatt <
> > > > > > >> >> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > > > >> >> > > > > > >
> > > > > > >> >> > > > > > > > Hi Jun,
> > > > > > >> >> > > > > > > >
> > > > > > >> >> > > > > > > > Thanks for reviewing.
> > > > > > >> >> > > > > > > >
> > > > > > >> >> > > > > > > > * We could add a Cluster action to add acls
> on
> > > who
> > > > > can
> > > > > > >> request
> > > > > > >> >> > > > > > delegation
> > > > > > >> >> > > > > > > > tokens. I don't see the use case for that yet
> > but
> > > > > down
> > > > > > >> the line
> > > > > > >> >> > > > when
> > > > > > >> >> > > > > we
> > > > > > >> >> > > > > > > > start supporting getDelegationTokenAs it will
> > be
> > > > > > >> necessary.
> > > > > > >> >> > > > > > > > * Yes we recommend tokens to be only
> > > > used/distributed
> > > > > > >> over
> > > > > > >> >> > secure
> > > > > > >> >> > > > > > > channels.
> > > > > > >> >> > > > > > > > * Depending on what design we end up choosing
> > > > > > >> Invalidation will
> > > > > > >> >> > > be
> > > > > > >> >> > > > > > > > responsibility of every broker or controller.
> > > > > > >> >> > > > > > > > * I am not sure if I documented somewhere
> that
> > > > > > >> invalidation
> > > > > > >> >> > will
> > > > > > >> >> > > > > > directly
> > > > > > >> >> > > > > > > > go through zookeeper but that is not the
> > intent.
> > > > > > >> Invalidation
> > > > > > >> >> > > will
> > > > > > >> >> > > > > > either
> > > > > > >> >> > > > > > > > be request based or due to expiration. No
> > direct
> > > > > > >> zookeeper
> > > > > > >> >> > > > > interaction
> > > > > > >> >> > > > > > > from
> > > > > > >> >> > > > > > > > any client.
> > > > > > >> >> > > > > > > > * "Broker also stores the DelegationToken
> > without
> > > > the
> > > > > > >> hmac in
> > > > > > >> >> > the
> > > > > > >> >> > > > > > > > zookeeper." : Sorry about the confusion. The
> > sole
> > > > > > >> purpose of
> > > > > > >> >> > > > > zookeeper
> > > > > > >> >> > > > > > in
> > > > > > >> >> > > > > > > > this design is as distribution channel for
> > tokens
> > > > > > >> between all
> > > > > > >> >> > > > brokers
> > > > > > >> >> > > > > > > and a
> > > > > > >> >> > > > > > > > layer that ensures only tokens that were
> > > generated
> > > > by
> > > > > > >> making a
> > > > > > >> >> > > > > request
> > > > > > >> >> > > > > > > to a
> > > > > > >> >> > > > > > > > broker will be accepted (more on this in
> second
> > > > > > >> paragraph). The
> > > > > > >> >> > > > token
> > > > > > >> >> > > > > > > > consists of few elements (owner, renewer,
> uuid
> > ,
> > > > > > >> expiration,
> > > > > > >> >> > > hmac)
> > > > > > >> >> > > > ,
> > > > > > >> >> > > > > > one
> > > > > > >> >> > > > > > > of
> > > > > > >> >> > > > > > > > which is the finally generated hmac but hmac
> it
> > > > self
> > > > > is
> > > > > > >> >> > derivable
> > > > > > >> >> > > > if
> > > > > > >> >> > > > > > you
> > > > > > >> >> > > > > > > > have all the other elements of the token +
> > secret
> > > > key
> > > > > > to
> > > > > > >> >> > generate
> > > > > > >> >> > > > > hmac.
> > > > > > >> >> > > > > > > > Given zookeeper does not provide SSL support
> we
> > > do
> > > > > not
> > > > > > >> want the
> > > > > > >> >> > > > > entire
> > > > > > >> >> > > > > > > > token to be wire transferred to zookeeper as
> > that
> > > > > will
> > > > > > >> be an
> > > > > > >> >> > > > insecure
> > > > > > >> >> > > > > > > wire
> > > > > > >> >> > > > > > > > transfer. Instead we only store all the other
> > > > > elements
> > > > > > >> of a
> > > > > > >> >> > > > > delegation
> > > > > > >> >> > > > > > > > tokens. Brokers can read these elements and
> > > because
> > > > > > they
> > > > > > >> also
> > > > > > >> >> > > have
> > > > > > >> >> > > > > > access
> > > > > > >> >> > > > > > > > to secret key they will be able to generate
> > hmac
> > > on
> > > > > > >> their end.
> > > > > > >> >> > > > > > > >
> > > > > > >> >> > > > > > > > One of the alternative proposed is to avoid
> > > > zookeeper
> > > > > > >> >> > > altogether. A
> > > > > > >> >> > > > > > > Client
> > > > > > >> >> > > > > > > > will call broker with required information
> > > (owner,
> > > > > > >> renwer,
> > > > > > >> >> > > > > expiration)
> > > > > > >> >> > > > > > > and
> > > > > > >> >> > > > > > > > get back (signed hmac, uuid). Broker won't
> > store
> > > > this
> > > > > > in
> > > > > > >> >> > > zookeeper.
> > > > > > >> >> > > > > > From
> > > > > > >> >> > > > > > > > this point a client can contact any broker
> with
> > > all
> > > > > the
> > > > > > >> >> > > delegation
> > > > > > >> >> > > > > > token
> > > > > > >> >> > > > > > > > info (owner, rewner, expiration, hmac, uuid)
> > the
> > > > > borker
> > > > > > >> will
> > > > > > >> >> > > > > regenerate
> > > > > > >> >> > > > > > > the
> > > > > > >> >> > > > > > > > hmac and as long as it matches with hmac
> > > presented
> > > > by
> > > > > > >> client ,
> > > > > > >> >> > > > broker
> > > > > > >> >> > > > > > > will
> > > > > > >> >> > > > > > > > allow the request to authenticate.  Only
> > problem
> > > > with
> > > > > > >> this
> > > > > > >> >> > > approach
> > > > > > >> >> > > > > is
> > > > > > >> >> > > > > > if
> > > > > > >> >> > > > > > > > the secret key is compromised any client can
> > now
> > > > > > generate
> > > > > > >> >> > random
> > > > > > >> >> > > > > tokens
> > > > > > >> >> > > > > > > and
> > > > > > >> >> > > > > > > > they will still be able to authenticate as
> any
> > > user
> > > > > > they
> > > > > > >> like.
> > > > > > >> >> > > with
> > > > > > >> >> > > > > > > > zookeeper we guarantee that only tokens
> > acquired
> > > > via
> > > > > a
> > > > > > >> broker
> > > > > > >> >> > > > (using
> > > > > > >> >> > > > > > some
> > > > > > >> >> > > > > > > > auth scheme other than delegation token) will
> > be
> > > > > > >> accepted. We
> > > > > > >> >> > > need
> > > > > > >> >> > > > to
> > > > > > >> >> > > > > > > > discuss which proposal makes more sense and
> we
> > > can
> > > > go
> > > > > > >> over it
> > > > > > >> >> > in
> > > > > > >> >> > > > > > > tomorrow's
> > > > > > >> >> > > > > > > > meeting.
> > > > > > >> >> > > > > > > >
> > > > > > >> >> > > > > > > > Also, can you forward the invite to me?
> > > > > > >> >> > > > > > > >
> > > > > > >> >> > > > > > > > Thanks
> > > > > > >> >> > > > > > > > Parth
> > > > > > >> >> > > > > > > >
> > > > > > >> >> > > > > > > >
> > > > > > >> >> > > > > > > >
> > > > > > >> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <
> > > > > > >> jun@confluent.io>
> > > > > > >> >> > > > wrote:
> > > > > > >> >> > > > > > > >
> > > > > > >> >> > > > > > > > > Thanks for the KIP. A few comments.
> > > > > > >> >> > > > > > > > >
> > > > > > >> >> > > > > > > > > 100. This potentially can be useful for
> Kafka
> > > > > Connect
> > > > > > >> and
> > > > > > >> >> > Kafka
> > > > > > >> >> > > > > rest
> > > > > > >> >> > > > > > > > proxy
> > > > > > >> >> > > > > > > > > where a worker agent will need to run a
> task
> > on
> > > > > > behalf
> > > > > > >> of a
> > > > > > >> >> > > > client.
> > > > > > >> >> > > > > > We
> > > > > > >> >> > > > > > > > will
> > > > > > >> >> > > > > > > > > likely need to change how those services
> use
> > > > Kafka
> > > > > > >> clients
> > > > > > >> >> > > > > > > > > (producer/consumer). Instead of a shared
> > client
> > > > per
> > > > > > >> worker,
> > > > > > >> >> > we
> > > > > > >> >> > > > will
> > > > > > >> >> > > > > > > need
> > > > > > >> >> > > > > > > > a
> > > > > > >> >> > > > > > > > > client per user task since the
> authentication
> > > > > happens
> > > > > > >> at the
> > > > > > >> >> > > > > > connection
> > > > > > >> >> > > > > > > > > level. For Kafka Connect, the renewer will
> be
> > > the
> > > > > > >> workers.
> > > > > > >> >> > So,
> > > > > > >> >> > > we
> > > > > > >> >> > > > > > > > probably
> > > > > > >> >> > > > > > > > > need to allow multiple renewers. For Kafka
> > rest
> > > > > > proxy,
> > > > > > >> the
> > > > > > >> >> > > > renewer
> > > > > > >> >> > > > > > can
> > > > > > >> >> > > > > > > > > probably just be the creator of the token.
> > > > > > >> >> > > > > > > > >
> > > > > > >> >> > > > > > > > > 101. Do we need new acl on who can request
> > > > > delegation
> > > > > > >> tokens?
> > > > > > >> >> > > > > > > > >
> > > > > > >> >> > > > > > > > > 102. Do we recommend people to send
> > delegation
> > > > > tokens
> > > > > > >> in an
> > > > > > >> >> > > > > encrypted
> > > > > > >> >> > > > > > > > > channel?
> > > > > > >> >> > > > > > > > >
> > > > > > >> >> > > > > > > > > 103. Who is responsible for expiring
> tokens,
> > > > every
> > > > > > >> broker?
> > > > > > >> >> > > > > > > > >
> > > > > > >> >> > > > > > > > > 104. For invalidating tokens, would it be
> > > better
> > > > to
> > > > > > do
> > > > > > >> it in
> > > > > > >> >> > a
> > > > > > >> >> > > > > > request
> > > > > > >> >> > > > > > > > > instead of going to ZK directly?
> > > > > > >> >> > > > > > > > >
> > > > > > >> >> > > > > > > > > 105. The terminology of client in the wiki
> > > > > sometimes
> > > > > > >> refers
> > > > > > >> >> > to
> > > > > > >> >> > > > the
> > > > > > >> >> > > > > > end
> > > > > > >> >> > > > > > > > > client and some other times refers to the
> > > client
> > > > > > using
> > > > > > >> the
> > > > > > >> >> > > > > delegation
> > > > > > >> >> > > > > > > > > tokens. It would be useful to distinguish
> > > between
> > > > > the
> > > > > > >> two.
> > > > > > >> >> > > > > > > > >
> > > > > > >> >> > > > > > > > > 106. Could you explain the sentence "Broker
> > > also
> > > > > > >> stores the
> > > > > > >> >> > > > > > > > DelegationToken
> > > > > > >> >> > > > > > > > > without the hmac in the zookeeper." a bit
> > > more? I
> > > > > > >> thought the
> > > > > > >> >> > > > > > > delegation
> > > > > > >> >> > > > > > > > > token is the hmac.
> > > > > > >> >> > > > > > > > >
> > > > > > >> >> > > > > > > > > Thanks,
> > > > > > >> >> > > > > > > > >
> > > > > > >> >> > > > > > > > > Jun
> > > > > > >> >> > > > > > > > >
> > > > > > >> >> > > > > > > > >
> > > > > > >> >> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <
> > > > > > >> jun@confluent.io>
> > > > > > >> >> > > > wrote:
> > > > > > >> >> > > > > > > > >
> > > > > > >> >> > > > > > > > > > Hi, Harsha,
> > > > > > >> >> > > > > > > > > >
> > > > > > >> >> > > > > > > > > > Just sent out a KIP meeting invite. We
> can
> > > > > discuss
> > > > > > >> this in
> > > > > > >> >> > > the
> > > > > > >> >> > > > > > > meeting
> > > > > > >> >> > > > > > > > > > tomorrow.
> > > > > > >> >> > > > > > > > > >
> > > > > > >> >> > > > > > > > > > Thanks,
> > > > > > >> >> > > > > > > > > >
> > > > > > >> >> > > > > > > > > > Jun
> > > > > > >> >> > > > > > > > > >
> > > > > > >> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <
> > > > > > >> kafka@harsha.io>
> > > > > > >> >> > > > wrote:
> > > > > > >> >> > > > > > > > > >
> > > > > > >> >> > > > > > > > > >> Hi All,
> > > > > > >> >> > > > > > > > > >>            Can we have a KIP meeting
> > around
> > > > > this.
> > > > > > >> The KIP
> > > > > > >> >> > is
> > > > > > >> >> > > > up
> > > > > > >> >> > > > > > for
> > > > > > >> >> > > > > > > > > >>            sometime and if there are any
> > > > > questions
> > > > > > >> lets
> > > > > > >> >> > > > quickly
> > > > > > >> >> > > > > > hash
> > > > > > >> >> > > > > > > > out
> > > > > > >> >> > > > > > > > > >>            details.
> > > > > > >> >> > > > > > > > > >>
> > > > > > >> >> > > > > > > > > >> Thanks,
> > > > > > >> >> > > > > > > > > >> Harsha
> > > > > > >> >> > > > > > > > > >>
> > > > > > >> >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40 AM, parth
> > > > > > brahmbhatt
> > > > > > >> wrote:
> > > > > > >> >> > > > > > > > > >> > That is what the hadoop echo system
> uses
> > > so
> > > > no
> > > > > > >> good
> > > > > > >> >> > reason
> > > > > > >> >> > > > > > really.
> > > > > > >> >> > > > > > > > We
> > > > > > >> >> > > > > > > > > >> > could
> > > > > > >> >> > > > > > > > > >> > change it to whatever is the newest
> > > > > recommended
> > > > > > >> standard
> > > > > > >> >> > > is.
> > > > > > >> >> > > > > > > > > >> >
> > > > > > >> >> > > > > > > > > >> > Thanks
> > > > > > >> >> > > > > > > > > >> > Parth
> > > > > > >> >> > > > > > > > > >> >
> > > > > > >> >> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM,
> Ismael
> > > > Juma <
> > > > > > >> >> > > > > ismael@juma.me.uk
> > > > > > >> >> > > > > > >
> > > > > > >> >> > > > > > > > > wrote:
> > > > > > >> >> > > > > > > > > >> >
> > > > > > >> >> > > > > > > > > >> > > Hi Parth,
> > > > > > >> >> > > > > > > > > >> > >
> > > > > > >> >> > > > > > > > > >> > > Thanks for the KIP. I only started
> > > > reviewing
> > > > > > >> this and
> > > > > > >> >> > > may
> > > > > > >> >> > > > > have
> > > > > > >> >> > > > > > > > > >> additional
> > > > > > >> >> > > > > > > > > >> > > questions later. The immediate
> > question
> > > > that
> > > > > > >> came to
> > > > > > >> >> > > mind
> > > > > > >> >> > > > is
> > > > > > >> >> > > > > > our
> > > > > > >> >> > > > > > > > > >> choice of
> > > > > > >> >> > > > > > > > > >> > > "DIGEST-MD5" even though it's marked
> > as
> > > > > > >> OBSOLETE in
> > > > > > >> >> > the
> > > > > > >> >> > > > IANA
> > > > > > >> >> > > > > > > > > Registry
> > > > > > >> >> > > > > > > > > >> of
> > > > > > >> >> > > > > > > > > >> > > SASL mechanisms and the original RFC
> > > > (2831)
> > > > > > has
> > > > > > >> been
> > > > > > >> >> > > moved
> > > > > > >> >> > > > > to
> > > > > > >> >> > > > > > > > > Historic
> > > > > > >> >> > > > > > > > > >> > > status:
> > > > > > >> >> > > > > > > > > >> > >
> > > > > > >> >> > > > > > > > > >> > > https://tools.ietf.org/html/rfc6331
> > > > > > >> >> > > > > > > > > >> > >
> > > > > > >> >> > > > > > > > >
> > > > > > >> >> > > > > >
> > > > > > >> >> > >
> > > > > > >>
> > > >
> http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> > > > > > >> >> > > > > > > > > >> > >
> > > > > > >> >> > > > > > > > > >> > > What is the reasoning behind that
> > > choice?
> > > > > > >> >> > > > > > > > > >> > >
> > > > > > >> >> > > > > > > > > >> > > Thanks,
> > > > > > >> >> > > > > > > > > >> > > Ismael
> > > > > > >> >> > > > > > > > > >> > >
> > > > > > >> >> > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29 PM,
> Gwen
> > > > > > Shapira <
> > > > > > >> >> > > > > > > gwen@confluent.io
> > > > > > >> >> > > > > > > > >
> > > > > > >> >> > > > > > > > > >> wrote:
> > > > > > >> >> > > > > > > > > >> > >
> > > > > > >> >> > > > > > > > > >> > > > Also comments inline :)
> > > > > > >> >> > > > > > > > > >> > > >
> > > > > > >> >> > > > > > > > > >> > > > > * I want to emphasize that even
> > > though
> > > > > > >> delegation
> > > > > > >> >> > > > tokens
> > > > > > >> >> > > > > > > are a
> > > > > > >> >> > > > > > > > > >> Hadoop
> > > > > > >> >> > > > > > > > > >> > > > > innovation, I feel very strongly
> > > about
> > > > > not
> > > > > > >> adding
> > > > > > >> >> > > > > > dependency
> > > > > > >> >> > > > > > > > on
> > > > > > >> >> > > > > > > > > >> Hadoop
> > > > > > >> >> > > > > > > > > >> > > > > when implementing delegation
> > tokens
> > > > for
> > > > > > >> Kafka. The
> > > > > > >> >> > > KIP
> > > > > > >> >> > > > > > > doesn't
> > > > > > >> >> > > > > > > > > >> imply
> > > > > > >> >> > > > > > > > > >> > > > > such dependency, but if you can
> > > > > clarify...
> > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > >> >> > > > > > > > > >> > > > > *No hadoop dependency.*
> > > > > > >> >> > > > > > > > > >> > > >
> > > > > > >> >> > > > > > > > > >> > > > Yay! Just add this to the KIP so
> no
> > > one
> > > > > will
> > > > > > >> read
> > > > > > >> >> > the
> > > > > > >> >> > > > KIP
> > > > > > >> >> > > > > > and
> > > > > > >> >> > > > > > > > > panic
> > > > > > >> >> > > > > > > > > >> > > > three weeks before the next
> > release...
> > > > > > >> >> > > > > > > > > >> > > >
> > > > > > >> >> > > > > > > > > >> > > > > * Can we get delegation token at
> > any
> > > > > time
> > > > > > >> after
> > > > > > >> >> > > > > > > > authenticating?
> > > > > > >> >> > > > > > > > > >> only
> > > > > > >> >> > > > > > > > > >> > > > > immediately after?
> > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > >> >> > > > > > > > > >> > > > > *As long as you are
> authenticated
> > > you
> > > > > can
> > > > > > >> get
> > > > > > >> >> > > > delegation
> > > > > > >> >> > > > > > > > tokens.
> > > > > > >> >> > > > > > > > > >> We
> > > > > > >> >> > > > > > > > > >> > > need
> > > > > > >> >> > > > > > > > > >> > > > to
> > > > > > >> >> > > > > > > > > >> > > > > discuss if a client
> authenticated
> > > > using
> > > > > > >> delegation
> > > > > > >> >> > > > > token,
> > > > > > >> >> > > > > > > can
> > > > > > >> >> > > > > > > > > also
> > > > > > >> >> > > > > > > > > >> > > > acquire
> > > > > > >> >> > > > > > > > > >> > > > > delegation token again or not.
> > Also
> > > > > there
> > > > > > >> is the
> > > > > > >> >> > > > > question
> > > > > > >> >> > > > > > of
> > > > > > >> >> > > > > > > > do
> > > > > > >> >> > > > > > > > > we
> > > > > > >> >> > > > > > > > > >> > > allow
> > > > > > >> >> > > > > > > > > >> > > > > anyone to acquire delegation
> token
> > > or
> > > > we
> > > > > > >> want
> > > > > > >> >> > > specific
> > > > > > >> >> > > > > > ACLs
> > > > > > >> >> > > > > > > (I
> > > > > > >> >> > > > > > > > > >> think
> > > > > > >> >> > > > > > > > > >> > > its
> > > > > > >> >> > > > > > > > > >> > > > an
> > > > > > >> >> > > > > > > > > >> > > > > overkill.)*
> > > > > > >> >> > > > > > > > > >> > > >
> > > > > > >> >> > > > > > > > > >> > > > I agree that ACLs is an overkill.
> > > > > > >> >> > > > > > > > > >> > > >
> > > > > > >> >> > > > > > > > > >> > > > I think we are debating two
> options:
> > > > > Either
> > > > > > >> require
> > > > > > >> >> > > > > Kerberos
> > > > > > >> >> > > > > > > > auth
> > > > > > >> >> > > > > > > > > >> for
> > > > > > >> >> > > > > > > > > >> > > > renewal or require non-owners to
> > > renew.
> > > > > > >> >> > > > > > > > > >> > > > I *think* the latter is simpler
> (it
> > > > > > basically
> > > > > > >> >> > require
> > > > > > >> >> > > a
> > > > > > >> >> > > > > "job
> > > > > > >> >> > > > > > > > > master"
> > > > > > >> >> > > > > > > > > >> > > > to take responsibility for the
> > > renewal,
> > > > it
> > > > > > >> will have
> > > > > > >> >> > > its
> > > > > > >> >> > > > > own
> > > > > > >> >> > > > > > > > > >> identity
> > > > > > >> >> > > > > > > > > >> > > > anyway and I think this is the
> > correct
> > > > > > design
> > > > > > >> >> > pattern
> > > > > > >> >> > > > > > anyway.
> > > > > > >> >> > > > > > > > For
> > > > > > >> >> > > > > > > > > >> > > > storm, I'd expect Nimbus to
> > coordinate
> > > > > > >> renewals?),
> > > > > > >> >> > but
> > > > > > >> >> > > > it
> > > > > > >> >> > > > > is
> > > > > > >> >> > > > > > > > hard
> > > > > > >> >> > > > > > > > > to
> > > > > > >> >> > > > > > > > > >> > > > debate simplicity without looking
> at
> > > the
> > > > > > code
> > > > > > >> >> > changes
> > > > > > >> >> > > > > > > required.
> > > > > > >> >> > > > > > > > If
> > > > > > >> >> > > > > > > > > >> you
> > > > > > >> >> > > > > > > > > >> > > > have a draft of how the "require
> > > > Kerberos"
> > > > > > >> will look
> > > > > > >> >> > > in
> > > > > > >> >> > > > > > Kafka
> > > > > > >> >> > > > > > > > > code,
> > > > > > >> >> > > > > > > > > >> > > > I'll be happy to take a look.
> > > > > > >> >> > > > > > > > > >> > > >
> > > > > > >> >> > > > > > > > > >> > > > > * My understanding is that
> tokens
> > > will
> > > > > > >> propagate
> > > > > > >> >> > via
> > > > > > >> >> > > > ZK
> > > > > > >> >> > > > > > but
> > > > > > >> >> > > > > > > > > >> without
> > > > > > >> >> > > > > > > > > >> > > > > additional changes to
> > UpdateMetadata
> > > > > > >> protocol,
> > > > > > >> >> > > > correct?
> > > > > > >> >> > > > > > > > Clients
> > > > > > >> >> > > > > > > > > >> > > > > currently don't retry on SASL
> auth
> > > > > failure
> > > > > > >> (IIRC),
> > > > > > >> >> > > but
> > > > > > >> >> > > > > > since
> > > > > > >> >> > > > > > > > the
> > > > > > >> >> > > > > > > > > >> > > > > tokens propagate between brokers
> > > > asynch,
> > > > > > we
> > > > > > >> will
> > > > > > >> >> > > need
> > > > > > >> >> > > > to
> > > > > > >> >> > > > > > > > retry a
> > > > > > >> >> > > > > > > > > >> bit
> > > > > > >> >> > > > > > > > > >> > > > > to avoid clients failing auth
> due
> > to
> > > > > > timing
> > > > > > >> >> > issues.
> > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > >> >> > > > > > > > > >> > > > > *I am considering 2 alternatives
> > > right
> > > > > > now.
> > > > > > >> The
> > > > > > >> >> > > > current
> > > > > > >> >> > > > > > > > > documented
> > > > > > >> >> > > > > > > > > >> > > > approach
> > > > > > >> >> > > > > > > > > >> > > > > is zookeeper based and it does
> not
> > > > > require
> > > > > > >> any
> > > > > > >> >> > > changes
> > > > > > >> >> > > > > to
> > > > > > >> >> > > > > > > > > >> > > UpdateMetadata
> > > > > > >> >> > > > > > > > > >> > > > > protocol. An alternative
> approach
> > > can
> > > > > > remove
> > > > > > >> >> > > zookeeper
> > > > > > >> >> > > > > > > > > dependency
> > > > > > >> >> > > > > > > > > >> as
> > > > > > >> >> > > > > > > > > >> > > well
> > > > > > >> >> > > > > > > > > >> > > > > but we can discuss that in KIP
> > > > > discussion
> > > > > > >> call.*
> > > > > > >> >> > > > > > > > > >> > > >
> > > > > > >> >> > > > > > > > > >> > > > Oooh! Sounds interesting. Do you
> > want
> > > to
> > > > > > ping
> > > > > > >> Jun to
> > > > > > >> >> > > > > > arrange a
> > > > > > >> >> > > > > > > > > call?
> > > > > > >> >> > > > > > > > > >> > > >
> > > > > > >> >> > > > > > > > > >> > > > > * I liked Ashish's suggestion of
> > > > having
> > > > > > >> just the
> > > > > > >> >> > > > > > controller
> > > > > > >> >> > > > > > > > > issue
> > > > > > >> >> > > > > > > > > >> the
> > > > > > >> >> > > > > > > > > >> > > > > delegation tokens, to avoid
> > syncing
> > > a
> > > > > > shared
> > > > > > >> >> > secret.
> > > > > > >> >> > > > Not
> > > > > > >> >> > > > > > > sure
> > > > > > >> >> > > > > > > > if
> > > > > > >> >> > > > > > > > > >> we
> > > > > > >> >> > > > > > > > > >> > > > > want to continue the discussion
> > here
> > > > or
> > > > > on
> > > > > > >> the
> > > > > > >> >> > > wiki. I
> > > > > > >> >> > > > > > think
> > > > > > >> >> > > > > > > > > that
> > > > > > >> >> > > > > > > > > >> we
> > > > > > >> >> > > > > > > > > >> > > > > can decouple the problem of
> "token
> > > > > > >> distribution"
> > > > > > >> >> > > from
> > > > > > >> >> > > > > > > "shared
> > > > > > >> >> > > > > > > > > >> secret
> > > > > > >> >> > > > > > > > > >> > > > > distribution" and use the
> > controller
> > > > as
> > > > > > the
> > > > > > >> only
> > > > > > >> >> > > token
> > > > > > >> >> > > > > > > > generator
> > > > > > >> >> > > > > > > > > >> to
> > > > > > >> >> > > > > > > > > >> > > > > solve the second issue, while
> > still
> > > > > using
> > > > > > >> ZK async
> > > > > > >> >> > > to
> > > > > > >> >> > > > > > > > distribute
> > > > > > >> >> > > > > > > > > >> > > > > tokens.
> > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > >> >> > > > > > > > > >> > > > > *As mentioned in the previous
> > Email
> > > I
> > > > am
> > > > > > >> fine with
> > > > > > >> >> > > > that
> > > > > > >> >> > > > > > > > approach
> > > > > > >> >> > > > > > > > > >> as
> > > > > > >> >> > > > > > > > > >> > > long
> > > > > > >> >> > > > > > > > > >> > > > as
> > > > > > >> >> > > > > > > > > >> > > > > we agree that the extra
> complexity
> > > of
> > > > > > >> >> > > adding/updating
> > > > > > >> >> > > > > APIS
> > > > > > >> >> > > > > > > > adds
> > > > > > >> >> > > > > > > > > >> enough
> > > > > > >> >> > > > > > > > > >> > > > > value. The advantage with the
> > > > controller
> > > > > > >> approach
> > > > > > >> >> > is
> > > > > > >> >> > > > > > secret
> > > > > > >> >> > > > > > > > > >> rotation
> > > > > > >> >> > > > > > > > > >> > > can
> > > > > > >> >> > > > > > > > > >> > > > be
> > > > > > >> >> > > > > > > > > >> > > > > automated,frequent and would not
> > > > require
> > > > > > >> >> > > deployment. *
> > > > > > >> >> > > > > > > > > >> > > >
> > > > > > >> >> > > > > > > > > >> > > > Can you detail the extra
> complexity
> > > (or
> > > > > > point
> > > > > > >> me to
> > > > > > >> >> > > the
> > > > > > >> >> > > > > > email
> > > > > > >> >> > > > > > > I
> > > > > > >> >> > > > > > > > > >> > > > missed?) - which Apis are
> required?
> > > > > > >> >> > > > > > > > > >> > > > As far as I can tell, clients can
> > > > already
> > > > > > >> find the
> > > > > > >> >> > > > > > controller
> > > > > > >> >> > > > > > > > from
> > > > > > >> >> > > > > > > > > >> > > > metadata. I'm a bit more concerned
> > > about
> > > > > > >> controller
> > > > > > >> >> > > > load.
> > > > > > >> >> > > > > > > > > >> > > >
> > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > >> >> > > > > > > > > >> > > > > * While I like the idea of
> forcing
> > > > > > kerberos
> > > > > > >> auth
> > > > > > >> >> > for
> > > > > > >> >> > > > > > > renwal, I
> > > > > > >> >> > > > > > > > > >> think
> > > > > > >> >> > > > > > > > > >> > > > > it mixes the transport layer the
> > the
> > > > > > request
> > > > > > >> >> > content
> > > > > > >> >> > > > in
> > > > > > >> >> > > > > a
> > > > > > >> >> > > > > > > > pretty
> > > > > > >> >> > > > > > > > > >> ugly
> > > > > > >> >> > > > > > > > > >> > > > > way. Perhaps limiting renewer to
> > > > > non-owner
> > > > > > >> is
> > > > > > >> >> > > better.
> > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > >> >> > > > > > > > > >> > > > > *I feel this is a necessary
> evil.
> > > > While
> > > > > > >> this will
> > > > > > >> >> > > make
> > > > > > >> >> > > > > the
> > > > > > >> >> > > > > > > > kafka
> > > > > > >> >> > > > > > > > > >> code
> > > > > > >> >> > > > > > > > > >> > > > > pretty straight forward ,
> forcing
> > > > > renewer
> > > > > > >> to
> > > > > > >> >> > > > non-owner
> > > > > > >> >> > > > > > > pushes
> > > > > > >> >> > > > > > > > > >> the code
> > > > > > >> >> > > > > > > > > >> > > > > ugliness to client and makes it
> > even
> > > > > > harder
> > > > > > >> to
> > > > > > >> >> > > > > > integrate.  *
> > > > > > >> >> > > > > > > > > >> > > >
> > > > > > >> >> > > > > > > > > >> > > > As mentioned before, I don't think
> > the
> > > > > > >> "renewal by
> > > > > > >> >> > > > other"
> > > > > > >> >> > > > > > > > approach
> > > > > > >> >> > > > > > > > > >> is
> > > > > > >> >> > > > > > > > > >> > > > that ugly for the clients we
> expect
> > to
> > > > use
> > > > > > >> >> > delegation
> > > > > > >> >> > > > > tokens
> > > > > > >> >> > > > > > > > since
> > > > > > >> >> > > > > > > > > >> > > > they will have an app-master of
> some
> > > > sort
> > > > > > who
> > > > > > >> >> > > requested
> > > > > > >> >> > > > > the
> > > > > > >> >> > > > > > > > token
> > > > > > >> >> > > > > > > > > to
> > > > > > >> >> > > > > > > > > >> > > > begin with.
> > > > > > >> >> > > > > > > > > >> > > >
> > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > >> >> > > > > > > > > >> > > > > The response for my question on
> > how
> > > > > > multiple
> > > > > > >> >> > > > identities
> > > > > > >> >> > > > > > will
> > > > > > >> >> > > > > > > > be
> > > > > > >> >> > > > > > > > > >> > > > > handled wasn't super clear to
> me -
> > > > > AFAIK,
> > > > > > >> we don't
> > > > > > >> >> > > > > > > > authenticate
> > > > > > >> >> > > > > > > > > >> each
> > > > > > >> >> > > > > > > > > >> > > > > request, we authenticate
> > > connections.
> > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > >> >> > > > > > > > > >> > > > > *We authenticate connections,
> and
> > > only
> > > > > > when
> > > > > > >> they
> > > > > > >> >> > are
> > > > > > >> >> > > > > being
> > > > > > >> >> > > > > > > > > >> established.
> > > > > > >> >> > > > > > > > > >> > > > Let
> > > > > > >> >> > > > > > > > > >> > > > > me try to phrase this as a
> > question,
> > > > in
> > > > > > >> absence of
> > > > > > >> >> > > > > > > delegation
> > > > > > >> >> > > > > > > > > >> tokens if
> > > > > > >> >> > > > > > > > > >> > > > we
> > > > > > >> >> > > > > > > > > >> > > > > had to support the use case
> using
> > > user
> > > > > > >> TGT's how
> > > > > > >> >> > > would
> > > > > > >> >> > > > > we
> > > > > > >> >> > > > > > do
> > > > > > >> >> > > > > > > > it?
> > > > > > >> >> > > > > > > > > >> My
> > > > > > >> >> > > > > > > > > >> > > point
> > > > > > >> >> > > > > > > > > >> > > > > was it would be no different
> with
> > > > > > delegation
> > > > > > >> >> > tokens.
> > > > > > >> >> > > > The
> > > > > > >> >> > > > > > use
> > > > > > >> >> > > > > > > > > case
> > > > > > >> >> > > > > > > > > >> you
> > > > > > >> >> > > > > > > > > >> > > are
> > > > > > >> >> > > > > > > > > >> > > > > describing seems more like
> > > > > impersonation.*
> > > > > > >> >> > > > > > > > > >> > > >
> > > > > > >> >> > > > > > > > > >> > > > Yeah, I thought that one of the
> > things
> > > > > that
> > > > > > >> >> > delegation
> > > > > > >> >> > > > > > tokens
> > > > > > >> >> > > > > > > > > >> handled.
> > > > > > >> >> > > > > > > > > >> > > > Maybe I got it wrong :)
> > > > > > >> >> > > > > > > > > >> > > >
> > > > > > >> >> > > > > > > > > >> > > > Thanks for the detailed answers.
> > > > > > >> >> > > > > > > > > >> > > >
> > > > > > >> >> > > > > > > > > >> > > > Gwen
> > > > > > >> >> > > > > > > > > >> > > >
> > > > > > >> >> > > > > > > > > >> > > >
> > > > > > >> >> > > > > > > > > >> > > > > Thanks
> > > > > > >> >> > > > > > > > > >> > > > > Parth
> > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > >> >> > > > > > > > > >> > > > > On Fri, May 13, 2016 at 12:19
> AM,
> > > Gwen
> > > > > > >> Shapira <
> > > > > > >> >> > > > > > > > > gwen@confluent.io
> > > > > > >> >> > > > > > > > > >> >
> > > > > > >> >> > > > > > > > > >> > > > wrote:
> > > > > > >> >> > > > > > > > > >> > > > >
> > > > > > >> >> > > > > > > > > >> > > > >> Hi Parth and Harsha,
> > > > > > >> >> > > > > > > > > >> > > > >>
> > > > > > >> >> > > > > > > > > >> > > > >> Few more comments:
> > > > > > >> >> > > > > > > > > >> > > > >>
> > > > > > >> >> > > > > > > > > >> > > > >> * The API / RequestResponse
> > section
> > > > > > >> doesn't seem
> > > > > > >> >> > to
> > > > > > >> >> > > > > have
> > > > > > >> >> > > > > > > good
> > > > > > >> >> > > > > > > > > >> > > > >> description of the changes to
> the
> > > > Kafka
> > > > > > >> Protocol.
> > > > > > >> >> > > > > Sounds
> > > > > > >> >> > > > > > > like
> > > > > > >> >> > > > > > > > > >> you are
> > > > > > >> >> > > > > > > > > >> > > > >> proposing new
> > > DelegationTokenRequest
> > > > > and
> > > > > > >> >> > > > > > RenewTokenRequest
> > > > > > >> >> > > > > > > > (and
> > > > > > >> >> > > > > > > > > >> > > > >> matching responses), without
> > > > detailing
> > > > > > the
> > > > > > >> >> > contents
> > > > > > >> >> > > > of
> > > > > > >> >> > > > > > the
> > > > > > >> >> > > > > > > > > >> requests
> > > > > > >> >> > > > > > > > > >> > > > >> and responses? Or rather, you
> > show
> > > > the
> > > > > > >> class
> > > > > > >> >> > > > interface,
> > > > > > >> >> > > > > > but
> > > > > > >> >> > > > > > > > not
> > > > > > >> >> > > > > > > > > >> the
> > > > > > >> >> > > > > > > > > >> > > > >> underlying protocol. This could
> > be
> > > > seen
> > > > > > as
> > > > > > >> an
> > > > > > >> >> > > > > > > implementation
> > > > > > >> >> > > > > > > > > >> detail,
> > > > > > >> >> > > > > > > > > >> > > > >> but since the binary protocol
> is
> > > what
> > > > > we
> > > > > > >> provide
> > > > > > >> >> > to
> > > > > > >> >> > > > > > > non-Java
> > > > > > >> >> > > > > > > > > >> clients,
> > > > > > >> >> > > > > > > > > >> > > > >> we need to show the changes
> > there.
> > > > > > >> >> > > > > > > > > >> > > > >>
> > > > > > >> >> > > > > > > > > >> > > > >> * getDelegationToken sounds
> like
> > > > > > >> >> > > > > > > > delegationTokenRequestHandler?
> > > > > > >> >> > > > > > > > > >> Is it
> > > > > > >> >> > > > > > > > > >> > > > >> planned to be part of KafkaApi?
> > or
> > > > > > Client?
> > > > > > >> Its
> > > > > > >> >> > > > > unclear...
> > > > > > >> >> > > > > > > > > >> > > > >>
> > > > > > >> >> > > > > > > > > >> > > > >> * I want to emphasize that even
> > > > though
> > > > > > >> delegation
> > > > > > >> >> > > > > tokens
> > > > > > >> >> > > > > > > are
> > > > > > >> >> > > > > > > > a
> > > > > > >> >> > > > > > > > > >> Hadoop
> > > > > > >> >> > > > > > > > > >> > > > >> innovation, I feel very
> strongly
> > > > about
> > > > > > not
> > > > > > >> adding
> > > > > > >> >> > > > > > > dependency
> > > > > > >> >> > > > > > > > on
> > > > > > >> >> > > > > > > > > >> Hadoop
> > > > > > >> >> > > > > > > > > >> > > > >> when implementing delegation
> > tokens
> > > > for
> > > > > > >> Kafka.
> > > > > > >> >> > The
> > > > > > >> >> > > > KIP
> > > > > > >> >> > > > > > > > doesn't
> > > > > > >> >> > > > > > > > > >> imply
> > > > > > >> >> > > > > > > > > >> > > > >> such dependency, but if you can
> > > > > > clarify...
> > > > > > >> >> > > > > > > > > >> > > > >>
> > > > > > >> >> > > > > > > > > >> > > > >> * Can we get delegation token
> at
> > > any
> > > > > time
> > > > > > >> after
> > > > > > >> >> > > > > > > > authenticating?
> > > > > > >> >> > > > > > > > > >> only
> > > > > > >> >> > > > > > > > > >> > > > >> immediately after?
> > > > > > >> >> > > > > > > > > >> > > > >>
> > > > > > >> >> > > > > > > > > >> > > > >> * My understanding is that
> tokens
> > > > will
> > > > > > >> propagate
> > > > > > >> >> > > via
> > > > > > >> >> > > > ZK
> > > > > > >> >> > > > > > but
> > > > > > >> >> > > > > > > > > >> without
> > > > > > >> >> > > > > > > > > >> > > > >> additional changes to
> > > UpdateMetadata
> > > > > > >> protocol,
> > > > > > >> >> > > > correct?
> > > > > > >> >> > > > > > > > Clients
> > > > > > >> >> > > > > > > > > >> > > > >> currently don't retry on SASL
> > auth
> > > > > > failure
> > > > > > >> >> > (IIRC),
> > > > > > >> >> > > > but
> > > > > > >> >> > > > > > > since
> > > > > > >> >> > > > > > > > > the
> > > > > > >> >> > > > > > > > > >> > > > >> tokens propagate between
> brokers
> > > > > asynch,
> > > > > > >> we will
> > > > > > >> >> > > need
> > > > > > >> >> > > > > to
> > > > > > >> >> > > > > > > > retry
> > > > > > >> >> > > > > > > > > a
> > > > > > >> >> > > > > > > > > >> bit
> > > > > > >> >> > > > > > > > > >> > > > >> to avoid clients failing auth
> due
> > > to
> > > > > > timing
> > > > > > >> >> > issues.
> > > > > > >> >> > > > > > > > > >> > > > >>
> > > > > > >> >> > > > > > > > > >> > > > >> * Strongly agreeing on clients
> > not
> > > > > > >> touching ZK
> > > > > > >> >> > > > directly
> > > > > > >> >> > > > > > :)
> > > > > > >> >> > > > > > > > > >> > > > >>
> > > > > > >> >> > > > > > > > > >> > > > >> * I liked Ashish's suggestion
> of
> > > > having
> > > > > > >> just the
> > > > > > >> >> > > > > > controller
> > > > > > >> >> > > > > > > > > >> issue the
> > > > > > >> >> > > > > > > > > >> > > > >> delegation tokens, to avoid
> > > syncing a
> > > > > > >> shared
> > > > > > >> >> > > secret.
> > > > > > >> >> > > > > Not
> > > > > > >> >> > > > > > > sure
> > > > > > >> >> > > > > > > > > if
> > > > > > >> >> > > > > > > > > >> we
> > > > > > >> >> > > > > > > > > >> > > > >> want to continue the discussion
> > > here
> > > > or
> > > > > > on
> > > > > > >> the
> > > > > > >> >> > > wiki.
> > > > > > >> >> > >

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Grant Henke <gh...@cloudera.com>]

Hi Parth,

Are you still working on this? If you need any help please don't hesitate
to ask.

Thanks,
Grant

On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao <ju...@confluent.io> wrote:

> Parth,
>
> Thanks for the reply.
>
> It makes sense to only allow the renewal by users that authenticated using
> *non* delegation token mechanism. Then, should we make the renewal a list?
> For example, in the case of rest proxy, it will be useful for every
> instance of rest proxy to be able to renew the tokens.
>
> It would be clearer if we can document the request protocol like
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945):(VotedandPlannedforin0.10.1.0)
> .
>
> It would also be useful to document the client APIs.
>
> Thanks,
>
> Jun
>
> On Tue, Jun 28, 2016 at 2:55 PM, parth brahmbhatt <
> brahmbhatt.parth@gmail.com> wrote:
>
> > Hi,
> >
> > I am suggesting that we will only allow the renewal by users that
> > authenticated using *non* delegation token mechanism. For example, If
> user
> > Alice authenticated using kerberos and requested delegation tokens, only
> > user Alice authenticated via non delegation token mechanism can renew.
> > Clients that have  access to delegation tokens can not issue renewal
> > request for renewing their own token and this is primarily important to
> > reduce the time window for which a compromised token will be valid.
> >
> > To clarify, Yes any authenticated user can request delegation tokens but
> > even here I would recommend to avoid creating a chain where a client
> > authenticated via delegation token request for more delegation tokens.
> > Basically anyone can request delegation token, as long as they
> authenticate
> > via a non delegation token mechanism.
> >
> > Aren't classes listed here
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-PublicInterfaces
> > >
> > sufficient?
> >
> > Thanks
> > Parth
> >
> >
> >
> > On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Parth,
> > >
> > > Thanks for the reply. A couple of comments inline below.
> > >
> > > On Tue, Jun 21, 2016 at 10:36 AM, parth brahmbhatt <
> > > brahmbhatt.parth@gmail.com> wrote:
> > >
> > > > 1. Who / how are tokens renewed? By original requester only? or using
> > > > Kerberos
> > > > auth only?
> > > > My recommendation is to do this only using Kerberos auth and only
> threw
> > > the
> > > > renewer specified during the acquisition request.
> > > >
> > > >
> > > Hmm, not sure that I follow this. Are you saying that any client
> > > authenticated with the delegation token can renew, i.e. there is no
> > renewer
> > > needed?
> > >
> > > Also, just to be clear, any authenticated client (either through SASL
> or
> > > SSL) can request a delegation token for the authenticated user, right?
> > >
> > >
> > > > 2. Are tokens stored on each broker or in ZK?
> > > > My recommendation is still to store in ZK or not store them at all.
> The
> > > > whole controller based distribution is too much overhead with not
> much
> > to
> > > > achieve.
> > > >
> > > > 3. How are tokens invalidated / expired?
> > > > Either by expiration time out or through an explicit request to
> > > invalidate.
> > > >
> > > > 4. Which encryption algorithm is used?
> > > > SCRAM
> > > >
> > > > 5. What is the impersonation proposal (it wasn't in the KIP but was
> > > > discussed
> > > > in this thread)?
> > > > There is no imperonation proposal. I tried and explained how its a
> > > > different problem and why its not really necessary to discuss that as
> > > part
> > > > of this KIP.  This KIP will not support any impersonation, it will
> just
> > > be
> > > > another way to authenticate.
> > > >
> > > > 6. Do we need new ACLs, if so - for what actions?
> > > > We do not need new ACLs.
> > > >
> > > >
> > > Could we document the format of the new request/response and their
> > > associated Resource and Operation for ACL?
> > >
> > >
> > > > 7. How would the delegation token be configured in the client?
> > > > Should be through config. I wasn't planning on supporting JAAS for
> > > tokens.
> > > > I don't believe hadoop does this either.
> > > >
> > > > Thanks
> > > > Parth
> > > >
> > > >
> > > >
> > > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <ju...@confluent.io> wrote:
> > > >
> > > > > Harsha,
> > > > >
> > > > > Another question.
> > > > >
> > > > > 9. How would the delegation token be configured in the client? The
> > > > standard
> > > > > way is to do this through JAAS. However, we will need to think
> > through
> > > if
> > > > > this is convenient in a shared environment. For example, when a new
> > > task
> > > > is
> > > > > added to a Storm worker node, do we need to dynamically add a new
> > > section
> > > > > in the JAAS file? It may be more convenient if we can pass in the
> > token
> > > > > through the config directly w/o going through JAAS.
> > > > >
> > > > > Are you or Parth still actively working on this KIP?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <ju...@confluent.io> wrote:
> > > > >
> > > > > > Just to add on that list.
> > > > > >
> > > > > > 2. It would be good to document the format of the data stored in
> > ZK.
> > > > > > 7. Earlier, there was a discussion on whether the tokens should
> be
> > > > > > propagated through ZK like config/acl/quota, or through the
> > > controller.
> > > > > > Currently, the controller is only designed for propagating topic
> > > > > metadata,
> > > > > > but not other data.
> > > > > > 8. Should we use SCRAM to send the token instead of DIGEST-MD5
> > since
> > > > it's
> > > > > > deprecated?
> > > > > >
> > > > > > Also, the images in the wiki seem broken.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <
> gwen@confluent.io>
> > > > > wrote:
> > > > > >
> > > > > >> From what I can see, remaining questions are:
> > > > > >>
> > > > > >> 1. Who / how are tokens renewed? By original requester only? or
> > > using
> > > > > >> Kerberos auth only?
> > > > > >> 2. Are tokens stored on each broker or in ZK?
> > > > > >> 3. How are tokens invalidated / expired?
> > > > > >> 4. Which encryption algorithm is used?
> > > > > >> 5. What is the impersonation proposal (it wasn't in the KIP but
> > was
> > > > > >> discussed in this thread)?
> > > > > >> 6. Do we need new ACLs, if so - for what actions?
> > > > > >>
> > > > > >> Gwen
> > > > > >>
> > > > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <ka...@harsha.io> wrote:
> > > > > >> > Jun & Ismael,
> > > > > >> >                          Unfortunately I couldn't attend the
> KIP
> > > > > meeting
> > > > > >> >                          when delegation tokens discussed.
> > > > Appreciate
> > > > > if
> > > > > >> >                          you can update the thread if you have
> > any
> > > > > >> >                          further questions.
> > > > > >> > Thanks,
> > > > > >> > Harsha
> > > > > >> >
> > > > > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
> > > > > >> >> It seems that the links to images in the KIP are broken.
> > > > > >> >>
> > > > > >> >> Liquan
> > > > > >> >>
> > > > > >> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
> > > > > >> >> brahmbhatt.parth@gmail.com> wrote:
> > > > > >> >>
> > > > > >> >> > 110. What does getDelegationTokenAs mean?
> > > > > >> >> > In the current proposal we only allow a user to get
> > delegation
> > > > > token
> > > > > >> for
> > > > > >> >> > the identity that it authenticated as using another
> > mechanism,
> > > > i.e.
> > > > > >> A user
> > > > > >> >> > that authenticate using a keytab for principal
> > > user1@EXAMPLE.COM
> > > > > >> will get
> > > > > >> >> > delegation tokens for that user only. In future I think we
> > will
> > > > > have
> > > > > >> to
> > > > > >> >> > extend support such that we allow some set of users (
> > > > > >> >> > kafka-rest-user@EXAMPLE.COM, storm-nimbus@EXAMPLE.COM) to
> > > > acquire
> > > > > >> >> > delegation tokens on behalf of other users whose identity
> > they
> > > > have
> > > > > >> >> > verified independently.  Kafka brokers will have ACLs to
> > > control
> > > > > >> which
> > > > > >> >> > users are allowed to impersonate other users and get tokens
> > on
> > > > > >> behalf of
> > > > > >> >> > them. Overall Impersonation is a whole different problem in
> > my
> > > > > >> opinion and
> > > > > >> >> > I think we can tackle it in separate KIP.
> > > > > >> >> >
> > > > > >> >> > 111. What's the typical rate of getting and renewing
> > delegation
> > > > > >> tokens?
> > > > > >> >> > Typically this should be very very low, 1 request per
> minute
> > > is a
> > > > > >> >> > relatively high estimate. However it depends on the token
> > > > > >> expiration. I am
> > > > > >> >> > less worried about the extra load it puts on controller vs
> > the
> > > > > added
> > > > > >> >> > complexity and the value it offers.
> > > > > >> >> >
> > > > > >> >> > Thanks
> > > > > >> >> > Parth
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <
> > > ismael@juma.me.uk>
> > > > > >> wrote:
> > > > > >> >> >
> > > > > >> >> > > Thanks Rajini. It would probably require a separate KIP
> as
> > it
> > > > > will
> > > > > >> >> > > introduce user visible changes. We could also update
> KIP-48
> > > to
> > > > > >> have this
> > > > > >> >> > > information, but it seems cleaner to do it separately. We
> > can
> > > > > >> discuss
> > > > > >> >> > that
> > > > > >> >> > > in the KIP call today.
> > > > > >> >> > >
> > > > > >> >> > > Ismael
> > > > > >> >> > >
> > > > > >> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <
> > > > > >> >> > > rajinisivaram@googlemail.com> wrote:
> > > > > >> >> > >
> > > > > >> >> > > > Ismael,
> > > > > >> >> > > >
> > > > > >> >> > > > I have created a JIRA (
> > > > > >> >> > https://issues.apache.org/jira/browse/KAFKA-3751)
> > > > > >> >> > > > for adding SCRAM as a SASL mechanism. Would that need
> > > another
> > > > > >> KIP? If
> > > > > >> >> > > > KIP-48 will use this mechanism, can this just be a JIRA
> > > that
> > > > > gets
> > > > > >> >> > > reviewed
> > > > > >> >> > > > when the PR is ready?
> > > > > >> >> > > >
> > > > > >> >> > > > Thank you,
> > > > > >> >> > > >
> > > > > >> >> > > > Rajini
> > > > > >> >> > > >
> > > > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <
> > > > > ismael@juma.me.uk>
> > > > > >> >> > wrote:
> > > > > >> >> > > >
> > > > > >> >> > > > > Thanks Rajini, SCRAM seems like a good candidate.
> > > > > >> >> > > > >
> > > > > >> >> > > > > Gwen had independently mentioned this as a SASL
> > mechanism
> > > > > that
> > > > > >> might
> > > > > >> >> > be
> > > > > >> >> > > > > useful for Kafka and I have been meaning to check it
> in
> > > > more
> > > > > >> detail.
> > > > > >> >> > > Good
> > > > > >> >> > > > > to know that you are willing to contribute an
> > > > implementation.
> > > > > >> Maybe
> > > > > >> >> > we
> > > > > >> >> > > > > should file a separate JIRA for this?
> > > > > >> >> > > > >
> > > > > >> >> > > > > Ismael
> > > > > >> >> > > > >
> > > > > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram <
> > > > > >> >> > > > > rajinisivaram@googlemail.com> wrote:
> > > > > >> >> > > > >
> > > > > >> >> > > > > > SCRAM (Salted Challenge Response Authentication
> > > > Mechanism)
> > > > > >> is a
> > > > > >> >> > > better
> > > > > >> >> > > > > > mechanism than Digest-MD5. Java doesn't come with a
> > > > > built-in
> > > > > >> SCRAM
> > > > > >> >> > > > > > SaslServer or SaslClient, but I will be happy to
> add
> > > > > support
> > > > > >> in
> > > > > >> >> > Kafka
> > > > > >> >> > > > > since
> > > > > >> >> > > > > > it would be a useful mechanism to support anyway.
> > > > > >> >> > > > > > https://tools.ietf.org/html/rfc7677 describes the
> > > > protocol
> > > > > >> for
> > > > > >> >> > > > > > SCRAM-SHA-256.
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <
> > > > jun@confluent.io
> > > > > >
> > > > > >> wrote:
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > > Parth,
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > > Thanks for the explanation. A couple of more
> > > questions.
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > > 110. What does getDelegationTokenAs mean?
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > > 111. What's the typical rate of getting and
> > renewing
> > > > > >> delegation
> > > > > >> >> > > > tokens?
> > > > > >> >> > > > > > > That may have an impact on whether they should be
> > > > > directed
> > > > > >> to the
> > > > > >> >> > > > > > > controller.
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > > Jun
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth
> brahmbhatt <
> > > > > >> >> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > > > Hi Jun,
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > > Thanks for reviewing.
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > > * We could add a Cluster action to add acls on
> > who
> > > > can
> > > > > >> request
> > > > > >> >> > > > > > delegation
> > > > > >> >> > > > > > > > tokens. I don't see the use case for that yet
> but
> > > > down
> > > > > >> the line
> > > > > >> >> > > > when
> > > > > >> >> > > > > we
> > > > > >> >> > > > > > > > start supporting getDelegationTokenAs it will
> be
> > > > > >> necessary.
> > > > > >> >> > > > > > > > * Yes we recommend tokens to be only
> > > used/distributed
> > > > > >> over
> > > > > >> >> > secure
> > > > > >> >> > > > > > > channels.
> > > > > >> >> > > > > > > > * Depending on what design we end up choosing
> > > > > >> Invalidation will
> > > > > >> >> > > be
> > > > > >> >> > > > > > > > responsibility of every broker or controller.
> > > > > >> >> > > > > > > > * I am not sure if I documented somewhere that
> > > > > >> invalidation
> > > > > >> >> > will
> > > > > >> >> > > > > > directly
> > > > > >> >> > > > > > > > go through zookeeper but that is not the
> intent.
> > > > > >> Invalidation
> > > > > >> >> > > will
> > > > > >> >> > > > > > either
> > > > > >> >> > > > > > > > be request based or due to expiration. No
> direct
> > > > > >> zookeeper
> > > > > >> >> > > > > interaction
> > > > > >> >> > > > > > > from
> > > > > >> >> > > > > > > > any client.
> > > > > >> >> > > > > > > > * "Broker also stores the DelegationToken
> without
> > > the
> > > > > >> hmac in
> > > > > >> >> > the
> > > > > >> >> > > > > > > > zookeeper." : Sorry about the confusion. The
> sole
> > > > > >> purpose of
> > > > > >> >> > > > > zookeeper
> > > > > >> >> > > > > > in
> > > > > >> >> > > > > > > > this design is as distribution channel for
> tokens
> > > > > >> between all
> > > > > >> >> > > > brokers
> > > > > >> >> > > > > > > and a
> > > > > >> >> > > > > > > > layer that ensures only tokens that were
> > generated
> > > by
> > > > > >> making a
> > > > > >> >> > > > > request
> > > > > >> >> > > > > > > to a
> > > > > >> >> > > > > > > > broker will be accepted (more on this in second
> > > > > >> paragraph). The
> > > > > >> >> > > > token
> > > > > >> >> > > > > > > > consists of few elements (owner, renewer, uuid
> ,
> > > > > >> expiration,
> > > > > >> >> > > hmac)
> > > > > >> >> > > > ,
> > > > > >> >> > > > > > one
> > > > > >> >> > > > > > > of
> > > > > >> >> > > > > > > > which is the finally generated hmac but hmac it
> > > self
> > > > is
> > > > > >> >> > derivable
> > > > > >> >> > > > if
> > > > > >> >> > > > > > you
> > > > > >> >> > > > > > > > have all the other elements of the token +
> secret
> > > key
> > > > > to
> > > > > >> >> > generate
> > > > > >> >> > > > > hmac.
> > > > > >> >> > > > > > > > Given zookeeper does not provide SSL support we
> > do
> > > > not
> > > > > >> want the
> > > > > >> >> > > > > entire
> > > > > >> >> > > > > > > > token to be wire transferred to zookeeper as
> that
> > > > will
> > > > > >> be an
> > > > > >> >> > > > insecure
> > > > > >> >> > > > > > > wire
> > > > > >> >> > > > > > > > transfer. Instead we only store all the other
> > > > elements
> > > > > >> of a
> > > > > >> >> > > > > delegation
> > > > > >> >> > > > > > > > tokens. Brokers can read these elements and
> > because
> > > > > they
> > > > > >> also
> > > > > >> >> > > have
> > > > > >> >> > > > > > access
> > > > > >> >> > > > > > > > to secret key they will be able to generate
> hmac
> > on
> > > > > >> their end.
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > > One of the alternative proposed is to avoid
> > > zookeeper
> > > > > >> >> > > altogether. A
> > > > > >> >> > > > > > > Client
> > > > > >> >> > > > > > > > will call broker with required information
> > (owner,
> > > > > >> renwer,
> > > > > >> >> > > > > expiration)
> > > > > >> >> > > > > > > and
> > > > > >> >> > > > > > > > get back (signed hmac, uuid). Broker won't
> store
> > > this
> > > > > in
> > > > > >> >> > > zookeeper.
> > > > > >> >> > > > > > From
> > > > > >> >> > > > > > > > this point a client can contact any broker with
> > all
> > > > the
> > > > > >> >> > > delegation
> > > > > >> >> > > > > > token
> > > > > >> >> > > > > > > > info (owner, rewner, expiration, hmac, uuid)
> the
> > > > borker
> > > > > >> will
> > > > > >> >> > > > > regenerate
> > > > > >> >> > > > > > > the
> > > > > >> >> > > > > > > > hmac and as long as it matches with hmac
> > presented
> > > by
> > > > > >> client ,
> > > > > >> >> > > > broker
> > > > > >> >> > > > > > > will
> > > > > >> >> > > > > > > > allow the request to authenticate.  Only
> problem
> > > with
> > > > > >> this
> > > > > >> >> > > approach
> > > > > >> >> > > > > is
> > > > > >> >> > > > > > if
> > > > > >> >> > > > > > > > the secret key is compromised any client can
> now
> > > > > generate
> > > > > >> >> > random
> > > > > >> >> > > > > tokens
> > > > > >> >> > > > > > > and
> > > > > >> >> > > > > > > > they will still be able to authenticate as any
> > user
> > > > > they
> > > > > >> like.
> > > > > >> >> > > with
> > > > > >> >> > > > > > > > zookeeper we guarantee that only tokens
> acquired
> > > via
> > > > a
> > > > > >> broker
> > > > > >> >> > > > (using
> > > > > >> >> > > > > > some
> > > > > >> >> > > > > > > > auth scheme other than delegation token) will
> be
> > > > > >> accepted. We
> > > > > >> >> > > need
> > > > > >> >> > > > to
> > > > > >> >> > > > > > > > discuss which proposal makes more sense and we
> > can
> > > go
> > > > > >> over it
> > > > > >> >> > in
> > > > > >> >> > > > > > > tomorrow's
> > > > > >> >> > > > > > > > meeting.
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > > Also, can you forward the invite to me?
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > > Thanks
> > > > > >> >> > > > > > > > Parth
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <
> > > > > >> jun@confluent.io>
> > > > > >> >> > > > wrote:
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > > > > Thanks for the KIP. A few comments.
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > 100. This potentially can be useful for Kafka
> > > > Connect
> > > > > >> and
> > > > > >> >> > Kafka
> > > > > >> >> > > > > rest
> > > > > >> >> > > > > > > > proxy
> > > > > >> >> > > > > > > > > where a worker agent will need to run a task
> on
> > > > > behalf
> > > > > >> of a
> > > > > >> >> > > > client.
> > > > > >> >> > > > > > We
> > > > > >> >> > > > > > > > will
> > > > > >> >> > > > > > > > > likely need to change how those services use
> > > Kafka
> > > > > >> clients
> > > > > >> >> > > > > > > > > (producer/consumer). Instead of a shared
> client
> > > per
> > > > > >> worker,
> > > > > >> >> > we
> > > > > >> >> > > > will
> > > > > >> >> > > > > > > need
> > > > > >> >> > > > > > > > a
> > > > > >> >> > > > > > > > > client per user task since the authentication
> > > > happens
> > > > > >> at the
> > > > > >> >> > > > > > connection
> > > > > >> >> > > > > > > > > level. For Kafka Connect, the renewer will be
> > the
> > > > > >> workers.
> > > > > >> >> > So,
> > > > > >> >> > > we
> > > > > >> >> > > > > > > > probably
> > > > > >> >> > > > > > > > > need to allow multiple renewers. For Kafka
> rest
> > > > > proxy,
> > > > > >> the
> > > > > >> >> > > > renewer
> > > > > >> >> > > > > > can
> > > > > >> >> > > > > > > > > probably just be the creator of the token.
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > 101. Do we need new acl on who can request
> > > > delegation
> > > > > >> tokens?
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > 102. Do we recommend people to send
> delegation
> > > > tokens
> > > > > >> in an
> > > > > >> >> > > > > encrypted
> > > > > >> >> > > > > > > > > channel?
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > 103. Who is responsible for expiring tokens,
> > > every
> > > > > >> broker?
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > 104. For invalidating tokens, would it be
> > better
> > > to
> > > > > do
> > > > > >> it in
> > > > > >> >> > a
> > > > > >> >> > > > > > request
> > > > > >> >> > > > > > > > > instead of going to ZK directly?
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > 105. The terminology of client in the wiki
> > > > sometimes
> > > > > >> refers
> > > > > >> >> > to
> > > > > >> >> > > > the
> > > > > >> >> > > > > > end
> > > > > >> >> > > > > > > > > client and some other times refers to the
> > client
> > > > > using
> > > > > >> the
> > > > > >> >> > > > > delegation
> > > > > >> >> > > > > > > > > tokens. It would be useful to distinguish
> > between
> > > > the
> > > > > >> two.
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > 106. Could you explain the sentence "Broker
> > also
> > > > > >> stores the
> > > > > >> >> > > > > > > > DelegationToken
> > > > > >> >> > > > > > > > > without the hmac in the zookeeper." a bit
> > more? I
> > > > > >> thought the
> > > > > >> >> > > > > > > delegation
> > > > > >> >> > > > > > > > > token is the hmac.
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > Thanks,
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > Jun
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <
> > > > > >> jun@confluent.io>
> > > > > >> >> > > > wrote:
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > > Hi, Harsha,
> > > > > >> >> > > > > > > > > >
> > > > > >> >> > > > > > > > > > Just sent out a KIP meeting invite. We can
> > > > discuss
> > > > > >> this in
> > > > > >> >> > > the
> > > > > >> >> > > > > > > meeting
> > > > > >> >> > > > > > > > > > tomorrow.
> > > > > >> >> > > > > > > > > >
> > > > > >> >> > > > > > > > > > Thanks,
> > > > > >> >> > > > > > > > > >
> > > > > >> >> > > > > > > > > > Jun
> > > > > >> >> > > > > > > > > >
> > > > > >> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <
> > > > > >> kafka@harsha.io>
> > > > > >> >> > > > wrote:
> > > > > >> >> > > > > > > > > >
> > > > > >> >> > > > > > > > > >> Hi All,
> > > > > >> >> > > > > > > > > >>            Can we have a KIP meeting
> around
> > > > this.
> > > > > >> The KIP
> > > > > >> >> > is
> > > > > >> >> > > > up
> > > > > >> >> > > > > > for
> > > > > >> >> > > > > > > > > >>            sometime and if there are any
> > > > questions
> > > > > >> lets
> > > > > >> >> > > > quickly
> > > > > >> >> > > > > > hash
> > > > > >> >> > > > > > > > out
> > > > > >> >> > > > > > > > > >>            details.
> > > > > >> >> > > > > > > > > >>
> > > > > >> >> > > > > > > > > >> Thanks,
> > > > > >> >> > > > > > > > > >> Harsha
> > > > > >> >> > > > > > > > > >>
> > > > > >> >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40 AM, parth
> > > > > brahmbhatt
> > > > > >> wrote:
> > > > > >> >> > > > > > > > > >> > That is what the hadoop echo system uses
> > so
> > > no
> > > > > >> good
> > > > > >> >> > reason
> > > > > >> >> > > > > > really.
> > > > > >> >> > > > > > > > We
> > > > > >> >> > > > > > > > > >> > could
> > > > > >> >> > > > > > > > > >> > change it to whatever is the newest
> > > > recommended
> > > > > >> standard
> > > > > >> >> > > is.
> > > > > >> >> > > > > > > > > >> >
> > > > > >> >> > > > > > > > > >> > Thanks
> > > > > >> >> > > > > > > > > >> > Parth
> > > > > >> >> > > > > > > > > >> >
> > > > > >> >> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael
> > > Juma <
> > > > > >> >> > > > > ismael@juma.me.uk
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > > > > wrote:
> > > > > >> >> > > > > > > > > >> >
> > > > > >> >> > > > > > > > > >> > > Hi Parth,
> > > > > >> >> > > > > > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > Thanks for the KIP. I only started
> > > reviewing
> > > > > >> this and
> > > > > >> >> > > may
> > > > > >> >> > > > > have
> > > > > >> >> > > > > > > > > >> additional
> > > > > >> >> > > > > > > > > >> > > questions later. The immediate
> question
> > > that
> > > > > >> came to
> > > > > >> >> > > mind
> > > > > >> >> > > > is
> > > > > >> >> > > > > > our
> > > > > >> >> > > > > > > > > >> choice of
> > > > > >> >> > > > > > > > > >> > > "DIGEST-MD5" even though it's marked
> as
> > > > > >> OBSOLETE in
> > > > > >> >> > the
> > > > > >> >> > > > IANA
> > > > > >> >> > > > > > > > > Registry
> > > > > >> >> > > > > > > > > >> of
> > > > > >> >> > > > > > > > > >> > > SASL mechanisms and the original RFC
> > > (2831)
> > > > > has
> > > > > >> been
> > > > > >> >> > > moved
> > > > > >> >> > > > > to
> > > > > >> >> > > > > > > > > Historic
> > > > > >> >> > > > > > > > > >> > > status:
> > > > > >> >> > > > > > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > https://tools.ietf.org/html/rfc6331
> > > > > >> >> > > > > > > > > >> > >
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > >
> > > > > >> >> > >
> > > > > >>
> > > http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> > > > > >> >> > > > > > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > What is the reasoning behind that
> > choice?
> > > > > >> >> > > > > > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > Thanks,
> > > > > >> >> > > > > > > > > >> > > Ismael
> > > > > >> >> > > > > > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen
> > > > > Shapira <
> > > > > >> >> > > > > > > gwen@confluent.io
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > >> wrote:
> > > > > >> >> > > > > > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > Also comments inline :)
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > > > > * I want to emphasize that even
> > though
> > > > > >> delegation
> > > > > >> >> > > > tokens
> > > > > >> >> > > > > > > are a
> > > > > >> >> > > > > > > > > >> Hadoop
> > > > > >> >> > > > > > > > > >> > > > > innovation, I feel very strongly
> > about
> > > > not
> > > > > >> adding
> > > > > >> >> > > > > > dependency
> > > > > >> >> > > > > > > > on
> > > > > >> >> > > > > > > > > >> Hadoop
> > > > > >> >> > > > > > > > > >> > > > > when implementing delegation
> tokens
> > > for
> > > > > >> Kafka. The
> > > > > >> >> > > KIP
> > > > > >> >> > > > > > > doesn't
> > > > > >> >> > > > > > > > > >> imply
> > > > > >> >> > > > > > > > > >> > > > > such dependency, but if you can
> > > > clarify...
> > > > > >> >> > > > > > > > > >> > > > >
> > > > > >> >> > > > > > > > > >> > > > >
> > > > > >> >> > > > > > > > > >> > > > > *No hadoop dependency.*
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > > > Yay! Just add this to the KIP so no
> > one
> > > > will
> > > > > >> read
> > > > > >> >> > the
> > > > > >> >> > > > KIP
> > > > > >> >> > > > > > and
> > > > > >> >> > > > > > > > > panic
> > > > > >> >> > > > > > > > > >> > > > three weeks before the next
> release...
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > > > > * Can we get delegation token at
> any
> > > > time
> > > > > >> after
> > > > > >> >> > > > > > > > authenticating?
> > > > > >> >> > > > > > > > > >> only
> > > > > >> >> > > > > > > > > >> > > > > immediately after?
> > > > > >> >> > > > > > > > > >> > > > >
> > > > > >> >> > > > > > > > > >> > > > >
> > > > > >> >> > > > > > > > > >> > > > > *As long as you are authenticated
> > you
> > > > can
> > > > > >> get
> > > > > >> >> > > > delegation
> > > > > >> >> > > > > > > > tokens.
> > > > > >> >> > > > > > > > > >> We
> > > > > >> >> > > > > > > > > >> > > need
> > > > > >> >> > > > > > > > > >> > > > to
> > > > > >> >> > > > > > > > > >> > > > > discuss if a client authenticated
> > > using
> > > > > >> delegation
> > > > > >> >> > > > > token,
> > > > > >> >> > > > > > > can
> > > > > >> >> > > > > > > > > also
> > > > > >> >> > > > > > > > > >> > > > acquire
> > > > > >> >> > > > > > > > > >> > > > > delegation token again or not.
> Also
> > > > there
> > > > > >> is the
> > > > > >> >> > > > > question
> > > > > >> >> > > > > > of
> > > > > >> >> > > > > > > > do
> > > > > >> >> > > > > > > > > we
> > > > > >> >> > > > > > > > > >> > > allow
> > > > > >> >> > > > > > > > > >> > > > > anyone to acquire delegation token
> > or
> > > we
> > > > > >> want
> > > > > >> >> > > specific
> > > > > >> >> > > > > > ACLs
> > > > > >> >> > > > > > > (I
> > > > > >> >> > > > > > > > > >> think
> > > > > >> >> > > > > > > > > >> > > its
> > > > > >> >> > > > > > > > > >> > > > an
> > > > > >> >> > > > > > > > > >> > > > > overkill.)*
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > > > I agree that ACLs is an overkill.
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > > > I think we are debating two options:
> > > > Either
> > > > > >> require
> > > > > >> >> > > > > Kerberos
> > > > > >> >> > > > > > > > auth
> > > > > >> >> > > > > > > > > >> for
> > > > > >> >> > > > > > > > > >> > > > renewal or require non-owners to
> > renew.
> > > > > >> >> > > > > > > > > >> > > > I *think* the latter is simpler (it
> > > > > basically
> > > > > >> >> > require
> > > > > >> >> > > a
> > > > > >> >> > > > > "job
> > > > > >> >> > > > > > > > > master"
> > > > > >> >> > > > > > > > > >> > > > to take responsibility for the
> > renewal,
> > > it
> > > > > >> will have
> > > > > >> >> > > its
> > > > > >> >> > > > > own
> > > > > >> >> > > > > > > > > >> identity
> > > > > >> >> > > > > > > > > >> > > > anyway and I think this is the
> correct
> > > > > design
> > > > > >> >> > pattern
> > > > > >> >> > > > > > anyway.
> > > > > >> >> > > > > > > > For
> > > > > >> >> > > > > > > > > >> > > > storm, I'd expect Nimbus to
> coordinate
> > > > > >> renewals?),
> > > > > >> >> > but
> > > > > >> >> > > > it
> > > > > >> >> > > > > is
> > > > > >> >> > > > > > > > hard
> > > > > >> >> > > > > > > > > to
> > > > > >> >> > > > > > > > > >> > > > debate simplicity without looking at
> > the
> > > > > code
> > > > > >> >> > changes
> > > > > >> >> > > > > > > required.
> > > > > >> >> > > > > > > > If
> > > > > >> >> > > > > > > > > >> you
> > > > > >> >> > > > > > > > > >> > > > have a draft of how the "require
> > > Kerberos"
> > > > > >> will look
> > > > > >> >> > > in
> > > > > >> >> > > > > > Kafka
> > > > > >> >> > > > > > > > > code,
> > > > > >> >> > > > > > > > > >> > > > I'll be happy to take a look.
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > > > > * My understanding is that tokens
> > will
> > > > > >> propagate
> > > > > >> >> > via
> > > > > >> >> > > > ZK
> > > > > >> >> > > > > > but
> > > > > >> >> > > > > > > > > >> without
> > > > > >> >> > > > > > > > > >> > > > > additional changes to
> UpdateMetadata
> > > > > >> protocol,
> > > > > >> >> > > > correct?
> > > > > >> >> > > > > > > > Clients
> > > > > >> >> > > > > > > > > >> > > > > currently don't retry on SASL auth
> > > > failure
> > > > > >> (IIRC),
> > > > > >> >> > > but
> > > > > >> >> > > > > > since
> > > > > >> >> > > > > > > > the
> > > > > >> >> > > > > > > > > >> > > > > tokens propagate between brokers
> > > asynch,
> > > > > we
> > > > > >> will
> > > > > >> >> > > need
> > > > > >> >> > > > to
> > > > > >> >> > > > > > > > retry a
> > > > > >> >> > > > > > > > > >> bit
> > > > > >> >> > > > > > > > > >> > > > > to avoid clients failing auth due
> to
> > > > > timing
> > > > > >> >> > issues.
> > > > > >> >> > > > > > > > > >> > > > >
> > > > > >> >> > > > > > > > > >> > > > > *I am considering 2 alternatives
> > right
> > > > > now.
> > > > > >> The
> > > > > >> >> > > > current
> > > > > >> >> > > > > > > > > documented
> > > > > >> >> > > > > > > > > >> > > > approach
> > > > > >> >> > > > > > > > > >> > > > > is zookeeper based and it does not
> > > > require
> > > > > >> any
> > > > > >> >> > > changes
> > > > > >> >> > > > > to
> > > > > >> >> > > > > > > > > >> > > UpdateMetadata
> > > > > >> >> > > > > > > > > >> > > > > protocol. An alternative approach
> > can
> > > > > remove
> > > > > >> >> > > zookeeper
> > > > > >> >> > > > > > > > > dependency
> > > > > >> >> > > > > > > > > >> as
> > > > > >> >> > > > > > > > > >> > > well
> > > > > >> >> > > > > > > > > >> > > > > but we can discuss that in KIP
> > > > discussion
> > > > > >> call.*
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > > > Oooh! Sounds interesting. Do you
> want
> > to
> > > > > ping
> > > > > >> Jun to
> > > > > >> >> > > > > > arrange a
> > > > > >> >> > > > > > > > > call?
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > > > > * I liked Ashish's suggestion of
> > > having
> > > > > >> just the
> > > > > >> >> > > > > > controller
> > > > > >> >> > > > > > > > > issue
> > > > > >> >> > > > > > > > > >> the
> > > > > >> >> > > > > > > > > >> > > > > delegation tokens, to avoid
> syncing
> > a
> > > > > shared
> > > > > >> >> > secret.
> > > > > >> >> > > > Not
> > > > > >> >> > > > > > > sure
> > > > > >> >> > > > > > > > if
> > > > > >> >> > > > > > > > > >> we
> > > > > >> >> > > > > > > > > >> > > > > want to continue the discussion
> here
> > > or
> > > > on
> > > > > >> the
> > > > > >> >> > > wiki. I
> > > > > >> >> > > > > > think
> > > > > >> >> > > > > > > > > that
> > > > > >> >> > > > > > > > > >> we
> > > > > >> >> > > > > > > > > >> > > > > can decouple the problem of "token
> > > > > >> distribution"
> > > > > >> >> > > from
> > > > > >> >> > > > > > > "shared
> > > > > >> >> > > > > > > > > >> secret
> > > > > >> >> > > > > > > > > >> > > > > distribution" and use the
> controller
> > > as
> > > > > the
> > > > > >> only
> > > > > >> >> > > token
> > > > > >> >> > > > > > > > generator
> > > > > >> >> > > > > > > > > >> to
> > > > > >> >> > > > > > > > > >> > > > > solve the second issue, while
> still
> > > > using
> > > > > >> ZK async
> > > > > >> >> > > to
> > > > > >> >> > > > > > > > distribute
> > > > > >> >> > > > > > > > > >> > > > > tokens.
> > > > > >> >> > > > > > > > > >> > > > >
> > > > > >> >> > > > > > > > > >> > > > >
> > > > > >> >> > > > > > > > > >> > > > > *As mentioned in the previous
> Email
> > I
> > > am
> > > > > >> fine with
> > > > > >> >> > > > that
> > > > > >> >> > > > > > > > approach
> > > > > >> >> > > > > > > > > >> as
> > > > > >> >> > > > > > > > > >> > > long
> > > > > >> >> > > > > > > > > >> > > > as
> > > > > >> >> > > > > > > > > >> > > > > we agree that the extra complexity
> > of
> > > > > >> >> > > adding/updating
> > > > > >> >> > > > > APIS
> > > > > >> >> > > > > > > > adds
> > > > > >> >> > > > > > > > > >> enough
> > > > > >> >> > > > > > > > > >> > > > > value. The advantage with the
> > > controller
> > > > > >> approach
> > > > > >> >> > is
> > > > > >> >> > > > > > secret
> > > > > >> >> > > > > > > > > >> rotation
> > > > > >> >> > > > > > > > > >> > > can
> > > > > >> >> > > > > > > > > >> > > > be
> > > > > >> >> > > > > > > > > >> > > > > automated,frequent and would not
> > > require
> > > > > >> >> > > deployment. *
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > > > Can you detail the extra complexity
> > (or
> > > > > point
> > > > > >> me to
> > > > > >> >> > > the
> > > > > >> >> > > > > > email
> > > > > >> >> > > > > > > I
> > > > > >> >> > > > > > > > > >> > > > missed?) - which Apis are required?
> > > > > >> >> > > > > > > > > >> > > > As far as I can tell, clients can
> > > already
> > > > > >> find the
> > > > > >> >> > > > > > controller
> > > > > >> >> > > > > > > > from
> > > > > >> >> > > > > > > > > >> > > > metadata. I'm a bit more concerned
> > about
> > > > > >> controller
> > > > > >> >> > > > load.
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > > > >
> > > > > >> >> > > > > > > > > >> > > > > * While I like the idea of forcing
> > > > > kerberos
> > > > > >> auth
> > > > > >> >> > for
> > > > > >> >> > > > > > > renwal, I
> > > > > >> >> > > > > > > > > >> think
> > > > > >> >> > > > > > > > > >> > > > > it mixes the transport layer the
> the
> > > > > request
> > > > > >> >> > content
> > > > > >> >> > > > in
> > > > > >> >> > > > > a
> > > > > >> >> > > > > > > > pretty
> > > > > >> >> > > > > > > > > >> ugly
> > > > > >> >> > > > > > > > > >> > > > > way. Perhaps limiting renewer to
> > > > non-owner
> > > > > >> is
> > > > > >> >> > > better.
> > > > > >> >> > > > > > > > > >> > > > >
> > > > > >> >> > > > > > > > > >> > > > > *I feel this is a necessary evil.
> > > While
> > > > > >> this will
> > > > > >> >> > > make
> > > > > >> >> > > > > the
> > > > > >> >> > > > > > > > kafka
> > > > > >> >> > > > > > > > > >> code
> > > > > >> >> > > > > > > > > >> > > > > pretty straight forward , forcing
> > > > renewer
> > > > > >> to
> > > > > >> >> > > > non-owner
> > > > > >> >> > > > > > > pushes
> > > > > >> >> > > > > > > > > >> the code
> > > > > >> >> > > > > > > > > >> > > > > ugliness to client and makes it
> even
> > > > > harder
> > > > > >> to
> > > > > >> >> > > > > > integrate.  *
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > > > As mentioned before, I don't think
> the
> > > > > >> "renewal by
> > > > > >> >> > > > other"
> > > > > >> >> > > > > > > > approach
> > > > > >> >> > > > > > > > > >> is
> > > > > >> >> > > > > > > > > >> > > > that ugly for the clients we expect
> to
> > > use
> > > > > >> >> > delegation
> > > > > >> >> > > > > tokens
> > > > > >> >> > > > > > > > since
> > > > > >> >> > > > > > > > > >> > > > they will have an app-master of some
> > > sort
> > > > > who
> > > > > >> >> > > requested
> > > > > >> >> > > > > the
> > > > > >> >> > > > > > > > token
> > > > > >> >> > > > > > > > > to
> > > > > >> >> > > > > > > > > >> > > > begin with.
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > > > >
> > > > > >> >> > > > > > > > > >> > > > > The response for my question on
> how
> > > > > multiple
> > > > > >> >> > > > identities
> > > > > >> >> > > > > > will
> > > > > >> >> > > > > > > > be
> > > > > >> >> > > > > > > > > >> > > > > handled wasn't super clear to me -
> > > > AFAIK,
> > > > > >> we don't
> > > > > >> >> > > > > > > > authenticate
> > > > > >> >> > > > > > > > > >> each
> > > > > >> >> > > > > > > > > >> > > > > request, we authenticate
> > connections.
> > > > > >> >> > > > > > > > > >> > > > >
> > > > > >> >> > > > > > > > > >> > > > > *We authenticate connections, and
> > only
> > > > > when
> > > > > >> they
> > > > > >> >> > are
> > > > > >> >> > > > > being
> > > > > >> >> > > > > > > > > >> established.
> > > > > >> >> > > > > > > > > >> > > > Let
> > > > > >> >> > > > > > > > > >> > > > > me try to phrase this as a
> question,
> > > in
> > > > > >> absence of
> > > > > >> >> > > > > > > delegation
> > > > > >> >> > > > > > > > > >> tokens if
> > > > > >> >> > > > > > > > > >> > > > we
> > > > > >> >> > > > > > > > > >> > > > > had to support the use case using
> > user
> > > > > >> TGT's how
> > > > > >> >> > > would
> > > > > >> >> > > > > we
> > > > > >> >> > > > > > do
> > > > > >> >> > > > > > > > it?
> > > > > >> >> > > > > > > > > >> My
> > > > > >> >> > > > > > > > > >> > > point
> > > > > >> >> > > > > > > > > >> > > > > was it would be no different with
> > > > > delegation
> > > > > >> >> > tokens.
> > > > > >> >> > > > The
> > > > > >> >> > > > > > use
> > > > > >> >> > > > > > > > > case
> > > > > >> >> > > > > > > > > >> you
> > > > > >> >> > > > > > > > > >> > > are
> > > > > >> >> > > > > > > > > >> > > > > describing seems more like
> > > > impersonation.*
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > > > Yeah, I thought that one of the
> things
> > > > that
> > > > > >> >> > delegation
> > > > > >> >> > > > > > tokens
> > > > > >> >> > > > > > > > > >> handled.
> > > > > >> >> > > > > > > > > >> > > > Maybe I got it wrong :)
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > > > Thanks for the detailed answers.
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > > > Gwen
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > > > > Thanks
> > > > > >> >> > > > > > > > > >> > > > > Parth
> > > > > >> >> > > > > > > > > >> > > > >
> > > > > >> >> > > > > > > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM,
> > Gwen
> > > > > >> Shapira <
> > > > > >> >> > > > > > > > > gwen@confluent.io
> > > > > >> >> > > > > > > > > >> >
> > > > > >> >> > > > > > > > > >> > > > wrote:
> > > > > >> >> > > > > > > > > >> > > > >
> > > > > >> >> > > > > > > > > >> > > > >> Hi Parth and Harsha,
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > > >> Few more comments:
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > > >> * The API / RequestResponse
> section
> > > > > >> doesn't seem
> > > > > >> >> > to
> > > > > >> >> > > > > have
> > > > > >> >> > > > > > > good
> > > > > >> >> > > > > > > > > >> > > > >> description of the changes to the
> > > Kafka
> > > > > >> Protocol.
> > > > > >> >> > > > > Sounds
> > > > > >> >> > > > > > > like
> > > > > >> >> > > > > > > > > >> you are
> > > > > >> >> > > > > > > > > >> > > > >> proposing new
> > DelegationTokenRequest
> > > > and
> > > > > >> >> > > > > > RenewTokenRequest
> > > > > >> >> > > > > > > > (and
> > > > > >> >> > > > > > > > > >> > > > >> matching responses), without
> > > detailing
> > > > > the
> > > > > >> >> > contents
> > > > > >> >> > > > of
> > > > > >> >> > > > > > the
> > > > > >> >> > > > > > > > > >> requests
> > > > > >> >> > > > > > > > > >> > > > >> and responses? Or rather, you
> show
> > > the
> > > > > >> class
> > > > > >> >> > > > interface,
> > > > > >> >> > > > > > but
> > > > > >> >> > > > > > > > not
> > > > > >> >> > > > > > > > > >> the
> > > > > >> >> > > > > > > > > >> > > > >> underlying protocol. This could
> be
> > > seen
> > > > > as
> > > > > >> an
> > > > > >> >> > > > > > > implementation
> > > > > >> >> > > > > > > > > >> detail,
> > > > > >> >> > > > > > > > > >> > > > >> but since the binary protocol is
> > what
> > > > we
> > > > > >> provide
> > > > > >> >> > to
> > > > > >> >> > > > > > > non-Java
> > > > > >> >> > > > > > > > > >> clients,
> > > > > >> >> > > > > > > > > >> > > > >> we need to show the changes
> there.
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > > >> * getDelegationToken sounds like
> > > > > >> >> > > > > > > > delegationTokenRequestHandler?
> > > > > >> >> > > > > > > > > >> Is it
> > > > > >> >> > > > > > > > > >> > > > >> planned to be part of KafkaApi?
> or
> > > > > Client?
> > > > > >> Its
> > > > > >> >> > > > > unclear...
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > > >> * I want to emphasize that even
> > > though
> > > > > >> delegation
> > > > > >> >> > > > > tokens
> > > > > >> >> > > > > > > are
> > > > > >> >> > > > > > > > a
> > > > > >> >> > > > > > > > > >> Hadoop
> > > > > >> >> > > > > > > > > >> > > > >> innovation, I feel very strongly
> > > about
> > > > > not
> > > > > >> adding
> > > > > >> >> > > > > > > dependency
> > > > > >> >> > > > > > > > on
> > > > > >> >> > > > > > > > > >> Hadoop
> > > > > >> >> > > > > > > > > >> > > > >> when implementing delegation
> tokens
> > > for
> > > > > >> Kafka.
> > > > > >> >> > The
> > > > > >> >> > > > KIP
> > > > > >> >> > > > > > > > doesn't
> > > > > >> >> > > > > > > > > >> imply
> > > > > >> >> > > > > > > > > >> > > > >> such dependency, but if you can
> > > > > clarify...
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > > >> * Can we get delegation token at
> > any
> > > > time
> > > > > >> after
> > > > > >> >> > > > > > > > authenticating?
> > > > > >> >> > > > > > > > > >> only
> > > > > >> >> > > > > > > > > >> > > > >> immediately after?
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > > >> * My understanding is that tokens
> > > will
> > > > > >> propagate
> > > > > >> >> > > via
> > > > > >> >> > > > ZK
> > > > > >> >> > > > > > but
> > > > > >> >> > > > > > > > > >> without
> > > > > >> >> > > > > > > > > >> > > > >> additional changes to
> > UpdateMetadata
> > > > > >> protocol,
> > > > > >> >> > > > correct?
> > > > > >> >> > > > > > > > Clients
> > > > > >> >> > > > > > > > > >> > > > >> currently don't retry on SASL
> auth
> > > > > failure
> > > > > >> >> > (IIRC),
> > > > > >> >> > > > but
> > > > > >> >> > > > > > > since
> > > > > >> >> > > > > > > > > the
> > > > > >> >> > > > > > > > > >> > > > >> tokens propagate between brokers
> > > > asynch,
> > > > > >> we will
> > > > > >> >> > > need
> > > > > >> >> > > > > to
> > > > > >> >> > > > > > > > retry
> > > > > >> >> > > > > > > > > a
> > > > > >> >> > > > > > > > > >> bit
> > > > > >> >> > > > > > > > > >> > > > >> to avoid clients failing auth due
> > to
> > > > > timing
> > > > > >> >> > issues.
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > > >> * Strongly agreeing on clients
> not
> > > > > >> touching ZK
> > > > > >> >> > > > directly
> > > > > >> >> > > > > > :)
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > > >> * I liked Ashish's suggestion of
> > > having
> > > > > >> just the
> > > > > >> >> > > > > > controller
> > > > > >> >> > > > > > > > > >> issue the
> > > > > >> >> > > > > > > > > >> > > > >> delegation tokens, to avoid
> > syncing a
> > > > > >> shared
> > > > > >> >> > > secret.
> > > > > >> >> > > > > Not
> > > > > >> >> > > > > > > sure
> > > > > >> >> > > > > > > > > if
> > > > > >> >> > > > > > > > > >> we
> > > > > >> >> > > > > > > > > >> > > > >> want to continue the discussion
> > here
> > > or
> > > > > on
> > > > > >> the
> > > > > >> >> > > wiki.
> > > > > >> >> > > > I
> > > > > >> >> > > > > > > think
> > > > > >> >> > > > > > > > > >> that we
> > > > > >> >> > > > > > > > > >> > > > >> can decouple the problem of
> "token
> > > > > >> distribution"
> > > > > >> >> > > from
> > > > > >> >> > > > > > > "shared
> > > > > >> >> > > > > > > > > >> secret
> > > > > >> >> > > > > > > > > >> > > > >> distribution" and use the
> > controller
> > > as
> > > > > >> the only
> > > > > >> >> > > > token
> > > > > >> >> > > > > > > > > generator
> > > > > >> >> > > > > > > > > >> to
> > > > > >> >> > > > > > > > > >> > > > >> solve the second issue, while
> still
> > > > using
> > > > > >> ZK
> > > > > >> >> > async
> > > > > >> >> > > to
> > > > > >> >> > > > > > > > > distribute
> > > > > >> >> > > > > > > > > >> > > > >> tokens.
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > > >> * I am also uncomfortable with
> > > infinite
> > > > > >> lifetime
> > > > > >> >> > of
> > > > > >> >> > > > > > tokens
> > > > > >> >> > > > > > > > (and
> > > > > >> >> > > > > > > > > >> hoped
> > > > > >> >> > > > > > > > > >> > > > >> to hear from others in the
> > > community) -
> > > > > but
> > > > > >> >> > having
> > > > > >> >> > > > the
> > > > > >> >> > > > > > > option
> > > > > >> >> > > > > > > > > and
> > > > > >> >> > > > > > > > > >> > > > >> documenting it as less secure,
> > allows
> > > > > >> users to
> > > > > >> >> > > > > configure
> > > > > >> >> > > > > > > > their
> > > > > >> >> > > > > > > > > >> system
> > > > > >> >> > > > > > > > > >> > > > >> as the wish.
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > > >> * While I like the idea of
> forcing
> > > > > >> kerberos auth
> > > > > >> >> > > for
> > > > > >> >> > > > > > > renwal,
> > > > > >> >> > > > > > > > I
> > > > > >> >> > > > > > > > > >> think
> > > > > >> >> > > > > > > > > >> > > > >> it mixes the transport layer the
> > the
> > > > > >> request
> > > > > >> >> > > content
> > > > > >> >> > > > > in a
> > > > > >> >> > > > > > > > > pretty
> > > > > >> >> > > > > > > > > >> ugly
> > > > > >> >> > > > > > > > > >> > > > >> way. Perhaps limiting renewer to
> > > > > non-owner
> > > > > >> is
> > > > > >> >> > > better.
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > > >> Things I'd still like to see:
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > > >> * More detailed explanation on
> what
> > > we
> > > > > >> plan to do
> > > > > >> >> > > for
> > > > > >> >> > > > > the
> > > > > >> >> > > > > > > > java
> > > > > >> >> > > > > > > > > >> clients
> > > > > >> >> > > > > > > > > >> > > > >> specifically - new configuration?
> > new
> > > > > APIs?
> > > > > >> >> > > > > > > > > >> > > > >> The response for my question on
> how
> > > > > >> multiple
> > > > > >> >> > > > identities
> > > > > >> >> > > > > > > will
> > > > > >> >> > > > > > > > be
> > > > > >> >> > > > > > > > > >> > > > >> handled wasn't super clear to me
> -
> > > > AFAIK,
> > > > > >> we
> > > > > >> >> > don't
> > > > > >> >> > > > > > > > authenticate
> > > > > >> >> > > > > > > > > >> each
> > > > > >> >> > > > > > > > > >> > > > >> request, we authenticate
> > connections.
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > > >> * Alternatives: Delegation tokens
> > are
> > > > > only
> > > > > >> used
> > > > > >> >> > in
> > > > > >> >> > > > the
> > > > > >> >> > > > > > > Hadoop
> > > > > >> >> > > > > > > > > >> > > > >> ecosystem. I'm wondering if there
> > are
> > > > > >> >> > alternatives
> > > > > >> >> > > in
> > > > > >> >> > > > > > other
> > > > > >> >> > > > > > > > > >> ecosystems
> > > > > >> >> > > > > > > > > >> > > > >> (Mesos? Tachyon? Cassandra?) and
> > > > whether
> > > > > >> there
> > > > > >> >> > are
> > > > > >> >> > > > some
> > > > > >> >> > > > > > > > > >> advantages
> > > > > >> >> > > > > > > > > >> > > > >> there.
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > > >> Gwen
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM,
> > > > Harsha <
> > > > > >> >> > > > > kafka@harsha.io
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > > > > > wrote:
> > > > > >> >> > > > > > > > > >> > > > >> > Hi Gwen,
> > > > > >> >> > > > > > > > > >> > > > >> >            Can you look at
> > Parth's
> > > > last
> > > > > >> reply.
> > > > > >> >> > > Does
> > > > > >> >> > > > > it
> > > > > >> >> > > > > > > > answer
> > > > > >> >> > > > > > > > > >> your
> > > > > >> >> > > > > > > > > >> > > > >> >            concerns.
> > > > > >> >> > > > > > > > > >> > > > >> >
> > > > > >> >> > > > > > > > > >> > > > >> > Thanks,
> > > > > >> >> > > > > > > > > >> > > > >> > Harsha
> > > > > >> >> > > > > > > > > >> > > > >> >
> > > > > >> >> > > > > > > > > >> > > > >> > On Wed, May 4, 2016, at 09:25
> AM,
> > > > parth
> > > > > >> >> > > brahmbhatt
> > > > > >> >> > > > > > wrote:
> > > > > >> >> > > > > > > > > >> > > > >> >> Thanks for reviewing Gwen. The
> > > wiki
> > > > > >> already
> > > > > >> >> > has
> > > > > >> >> > > > > > details
> > > > > >> >> > > > > > > on
> > > > > >> >> > > > > > > > > >> token
> > > > > >> >> > > > > > > > > >> > > > >> >> expiration
> > > > > >> >> > > > > > > > > >> > > > >> >> under token acquisition
> process
> > > > > >> >> > > > > > > > > >> > > > >> >> <
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > >
> > > > > >> >> > > > > > > > > >>
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > >
> > > > > >> >> > > > >
> > > > > >> >> > > >
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> > > > > >> >> > > > > > > > > >> > > > >> >.
> > > > > >> >> > > > > > > > > >> > > > >> >> Current proposal is that
> tokens
> > > will
> > > > > >> expire
> > > > > >> >> > > based
> > > > > >> >> > > > > on a
> > > > > >> >> > > > > > > > > server
> > > > > >> >> > > > > > > > > >> side
> > > > > >> >> > > > > > > > > >> > > > >> >> configuration (default 24
> hours)
> > > > > unless
> > > > > >> >> > renewed.
> > > > > >> >> > > > > > Renewal
> > > > > >> >> > > > > > > > is
> > > > > >> >> > > > > > > > > >> only
> > > > > >> >> > > > > > > > > >> > > > allowed
> > > > > >> >> > > > > > > > > >> > > > >> >> until the max life time of
> > token.
> > > > > >> >> > Alternatively
> > > > > >> >> > > we
> > > > > >> >> > > > > > could
> > > > > >> >> > > > > > > > > also
> > > > > >> >> > > > > > > > > >> make
> > > > > >> >> > > > > > > > > >> > > > that
> > > > > >> >> > > > > > > > > >> > > > >> >> an
> > > > > >> >> > > > > > > > > >> > > > >> >> optional param and the server
> > side
> > > > > >> default can
> > > > > >> >> > > > serve
> > > > > >> >> > > > > > as
> > > > > >> >> > > > > > > > the
> > > > > >> >> > > > > > > > > >> upper
> > > > > >> >> > > > > > > > > >> > > > bound.
> > > > > >> >> > > > > > > > > >> > > > >> >>
> > > > > >> >> > > > > > > > > >> > > > >> >> To your second point it will
> be
> > > done
> > > > > >> exactly
> > > > > >> >> > the
> > > > > >> >> > > > > same
> > > > > >> >> > > > > > > way
> > > > > >> >> > > > > > > > we
> > > > > >> >> > > > > > > > > >> would
> > > > > >> >> > > > > > > > > >> > > > >> >> support
> > > > > >> >> > > > > > > > > >> > > > >> >> multiple keytabs. The calling
> > > client
> > > > > >> will have
> > > > > >> >> > > to
> > > > > >> >> > > > > put
> > > > > >> >> > > > > > > the
> > > > > >> >> > > > > > > > > >> tokens it
> > > > > >> >> > > > > > > > > >> > > > >> wants
> > > > > >> >> > > > > > > > > >> > > > >> >> to use in the subject instance
> > and
> > > > > call
> > > > > >> >> > > > > > produce/consume
> > > > > >> >> > > > > > > > > inside
> > > > > >> >> > > > > > > > > >> > > > >> >> subject.doas. Each caller will
> > > have
> > > > to
> > > > > >> keep
> > > > > >> >> > > track
> > > > > >> >> > > > of
> > > > > >> >> > > > > > its
> > > > > >> >> > > > > > > > own
> > > > > >> >> > > > > > > > > >> > > > subject. I
> > > > > >> >> > > > > > > > > >> > > > >> >> will have to look at the code
> to
> > > see
> > > > > if
> > > > > >> we
> > > > > >> >> > > support
> > > > > >> >> > > > > > this
> > > > > >> >> > > > > > > > > >> feature
> > > > > >> >> > > > > > > > > >> > > right
> > > > > >> >> > > > > > > > > >> > > > >> now
> > > > > >> >> > > > > > > > > >> > > > >> >> but my understanding is
> > delegation
> > > > > token
> > > > > >> >> > > shouldn't
> > > > > >> >> > > > > > need
> > > > > >> >> > > > > > > > any
> > > > > >> >> > > > > > > > > >> special
> > > > > >> >> > > > > > > > > >> > > > >> >> treatment as its just another
> > type
> > > > of
> > > > > >> >> > Credential
> > > > > >> >> > > > in
> > > > > >> >> > > > > > the
> > > > > >> >> > > > > > > > > >> subject.
> > > > > >> >> > > > > > > > > >> > > > >> >>
> > > > > >> >> > > > > > > > > >> > > > >> >> I would also like to know what
> > is
> > > > your
> > > > > >> opinion
> > > > > >> >> > > > about
> > > > > >> >> > > > > > > > > infinite
> > > > > >> >> > > > > > > > > >> > > renewal
> > > > > >> >> > > > > > > > > >> > > > >> (my
> > > > > >> >> > > > > > > > > >> > > > >> >> recommendation is to not
> support
> > > > > this),
> > > > > >> tokens
> > > > > >> >> > > > > > renewing
> > > > > >> >> > > > > > > > them
> > > > > >> >> > > > > > > > > >> > > self(my
> > > > > >> >> > > > > > > > > >> > > > >> >> recommendation is to not
> support
> > > > this)
> > > > > >> and
> > > > > >> >> > most
> > > > > >> >> > > > > > > > importantly
> > > > > >> >> > > > > > > > > >> your
> > > > > >> >> > > > > > > > > >> > > > choice
> > > > > >> >> > > > > > > > > >> > > > >> >> between the alternatives
> listed
> > on
> > > > > this
> > > > > >> thread
> > > > > >> >> > > > > > > > > >> > > > >> >> <
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > >
> > > > > >> >> > > > > > > > > >>
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > >
> > > > > >> >> > > > >
> > > > > >> >> > > >
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> > > > > >> >> > > > > > > > > >> > > > >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> ( I am leaning towards the
> > > > > >> alternative-2 minus
> > > > > >> >> > > > > > > controller
> > > > > >> >> > > > > > > > > >> > > > distributing
> > > > > >> >> > > > > > > > > >> > > > >> >> secret). Thanks again for
> > > reviewing.
> > > > > >> >> > > > > > > > > >> > > > >> >>
> > > > > >> >> > > > > > > > > >> > > > >> >> Thanks
> > > > > >> >> > > > > > > > > >> > > > >> >> Parth
> > > > > >> >> > > > > > > > > >> > > > >> >>
> > > > > >> >> > > > > > > > > >> > > > >> >>
> > > > > >> >> > > > > > > > > >> > > > >> >>
> > > > > >> >> > > > > > > > > >> > > > >> >> On Wed, May 4, 2016 at 6:17
> AM,
> > > Gwen
> > > > > >> Shapira <
> > > > > >> >> > > > > > > > > >> gwen@confluent.io>
> > > > > >> >> > > > > > > > > >> > > > wrote:
> > > > > >> >> > > > > > > > > >> > > > >> >>
> > > > > >> >> > > > > > > > > >> > > > >> >> > Harsha,
> > > > > >> >> > > > > > > > > >> > > > >> >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > I was thinking of the Rest
> > > Proxy.
> > > > I
> > > > > >> didn't
> > > > > >> >> > see
> > > > > >> >> > > > > your
> > > > > >> >> > > > > > > > design
> > > > > >> >> > > > > > > > > >> yet,
> > > > > >> >> > > > > > > > > >> > > > but in
> > > > > >> >> > > > > > > > > >> > > > >> >> > our proxy, we have a set of
> > > > > >> producers, which
> > > > > >> >> > > > will
> > > > > >> >> > > > > > > serve
> > > > > >> >> > > > > > > > > >> multiple
> > > > > >> >> > > > > > > > > >> > > > users
> > > > > >> >> > > > > > > > > >> > > > >> >> > going through the proxy.
> Since
> > > > these
> > > > > >> users
> > > > > >> >> > > will
> > > > > >> >> > > > > have
> > > > > >> >> > > > > > > > > >> different
> > > > > >> >> > > > > > > > > >> > > > >> >> > privileges, they'll need to
> > > > > >> authenticate
> > > > > >> >> > > > > separately,
> > > > > >> >> > > > > > > and
> > > > > >> >> > > > > > > > > >> can't
> > > > > >> >> > > > > > > > > >> > > > share a
> > > > > >> >> > > > > > > > > >> > > > >> >> > token.
> > > > > >> >> > > > > > > > > >> > > > >> >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > Am I missing anything?
> > > > > >> >> > > > > > > > > >> > > > >> >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > Gwen
> > > > > >> >> > > > > > > > > >> > > > >> >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11
> > PM,
> > > > > >> Harsha <
> > > > > >> >> > > > > > > kafka@harsha.io
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > > > >> wrote:
> > > > > >> >> > > > > > > > > >> > > > >> >> > > Gwen,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >            On your second
> > > point.
> > > > > >> Can you
> > > > > >> >> > > > > describe
> > > > > >> >> > > > > > a
> > > > > >> >> > > > > > > > > >> usecase
> > > > > >> >> > > > > > > > > >> > > where
> > > > > >> >> > > > > > > > > >> > > > >> >> > >            mutliple
> clients
> > > > ended
> > > > > up
> > > > > >> >> > > sharing a
> > > > > >> >> > > > > > > > producer
> > > > > >> >> > > > > > > > > >> and
> > > > > >> >> > > > > > > > > >> > > even
> > > > > >> >> > > > > > > > > >> > > > if
> > > > > >> >> > > > > > > > > >> > > > >> they
> > > > > >> >> > > > > > > > > >> > > > >> >> > >            do why can't
> they
> > > not
> > > > > use
> > > > > >> >> > single
> > > > > >> >> > > > > token
> > > > > >> >> > > > > > > that
> > > > > >> >> > > > > > > > > >> producer
> > > > > >> >> > > > > > > > > >> > > > >> >> > >            captures. Why
> > would
> > > > we
> > > > > >> need
> > > > > >> >> > > > multiple
> > > > > >> >> > > > > > > > clients
> > > > > >> >> > > > > > > > > >> with
> > > > > >> >> > > > > > > > > >> > > > >> different
> > > > > >> >> > > > > > > > > >> > > > >> >> > >            tokens sharing
> a
> > > > single
> > > > > >> >> > instance
> > > > > >> >> > > of
> > > > > >> >> > > > > > > > producer.
> > > > > >> >> > > > > > > > > >> Also
> > > > > >> >> > > > > > > > > >> > > in
> > > > > >> >> > > > > > > > > >> > > > >> this
> > > > > >> >> > > > > > > > > >> > > > >> >> > >            case other
> > clients
> > > > have
> > > > > >> access
> > > > > >> >> > > all
> > > > > >> >> > > > > the
> > > > > >> >> > > > > > > > tokens
> > > > > >> >> > > > > > > > > >> no?
> > > > > >> >> > > > > > > > > >> > > > >> >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > > Thanks,
> > > > > >> >> > > > > > > > > >> > > > >> >> > > Harsha
> > > > > >> >> > > > > > > > > >> > > > >> >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > > On Tue, May 3, 2016, at
> > 11:49
> > > > AM,
> > > > > >> Gwen
> > > > > >> >> > > Shapira
> > > > > >> >> > > > > > > wrote:
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> Sorry for the delay:
> > > > > >> >> > > > > > > > > >> > > > >> >> > >>
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> Two questions that we
> > didn't
> > > > see
> > > > > >> in the
> > > > > >> >> > > wiki:
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> 1. Is there an expiration
> > for
> > > > > >> delegation
> > > > > >> >> > > > > tokens?
> > > > > >> >> > > > > > > > > >> Renewal? How
> > > > > >> >> > > > > > > > > >> > > > do we
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> revoke them?
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> 2. If we want to use
> > > delegation
> > > > > >> tokens
> > > > > >> >> > for
> > > > > >> >> > > > > > "do-as"
> > > > > >> >> > > > > > > > > (say,
> > > > > >> >> > > > > > > > > >> > > submit
> > > > > >> >> > > > > > > > > >> > > > >> Storm
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> job as my user), we will
> > > need a
> > > > > >> producer
> > > > > >> >> > > for
> > > > > >> >> > > > > > every
> > > > > >> >> > > > > > > > job
> > > > > >> >> > > > > > > > > >> (we
> > > > > >> >> > > > > > > > > >> > > can't
> > > > > >> >> > > > > > > > > >> > > > >> share
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> them between multiple
> jobs
> > > > > running
> > > > > >> on
> > > > > >> >> > same
> > > > > >> >> > > > > node),
> > > > > >> >> > > > > > > > since
> > > > > >> >> > > > > > > > > >> we
> > > > > >> >> > > > > > > > > >> > > only
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> authenticate when
> > connecting.
> > > > Is
> > > > > >> there a
> > > > > >> >> > > plan
> > > > > >> >> > > > > to
> > > > > >> >> > > > > > > > change
> > > > > >> >> > > > > > > > > >> this
> > > > > >> >> > > > > > > > > >> > > for
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> delegation tokens, in
> order
> > > to
> > > > > >> allow
> > > > > >> >> > > multiple
> > > > > >> >> > > > > > users
> > > > > >> >> > > > > > > > > with
> > > > > >> >> > > > > > > > > >> > > > different
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> tokens to share a client?
> > > > > >> >> > > > > > > > > >> > > > >> >> > >>
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> Gwen
> > > > > >> >> > > > > > > > > >> > > > >> >> > >>
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> On Tue, May 3, 2016 at
> 9:12
> > > AM,
> > > > > >> parth
> > > > > >> >> > > > > brahmbhatt
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> <
> > brahmbhatt.parth@gmail.com>
> > > > > >> wrote:
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> > Bumping this up one
> more
> > > > time,
> > > > > >> can
> > > > > >> >> > other
> > > > > >> >> > > > > > > committers
> > > > > >> >> > > > > > > > > >> review?
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> > Thanks
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> > Parth
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at
> > > 9:07
> > > > > AM,
> > > > > >> >> > Harsha <
> > > > > >> >> > > > > > > > > >> kafka@harsha.io>
> > > > > >> >> > > > > > > > > >> > > > wrote:
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> Parth,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >>           Overall
> > current
> > > > > >> design looks
> > > > > >> >> > > > good
> > > > > >> >> > > > > to
> > > > > >> >> > > > > > > > me. I
> > > > > >> >> > > > > > > > > >> am +1
> > > > > >> >> > > > > > > > > >> > > on
> > > > > >> >> > > > > > > > > >> > > > >> the
> > > > > >> >> > > > > > > > > >> > > > >> >> > KIP.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> Gwen , Jun can you
> > review
> > > > this
> > > > > >> as
> > > > > >> >> > well.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> -Harsha
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016,
> at
> > > > 09:57
> > > > > >> AM,
> > > > > >> >> > parth
> > > > > >> >> > > > > > > > brahmbhatt
> > > > > >> >> > > > > > > > > >> wrote:
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks for review
> > > > Jitendra.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > I don't like the
> idea
> > of
> > > > > >> infinite
> > > > > >> >> > > > lifetime
> > > > > >> >> > > > > > > but I
> > > > > >> >> > > > > > > > > >> see the
> > > > > >> >> > > > > > > > > >> > > > >> Streaming
> > > > > >> >> > > > > > > > > >> > > > >> >> > use
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > case. Even for
> > Streaming
> > > > use
> > > > > >> case I
> > > > > >> >> > > was
> > > > > >> >> > > > > > hoping
> > > > > >> >> > > > > > > > > >> there will
> > > > > >> >> > > > > > > > > >> > > > be
> > > > > >> >> > > > > > > > > >> > > > >> some
> > > > > >> >> > > > > > > > > >> > > > >> >> > notion
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > of
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > master/driver that
> can
> > > get
> > > > > new
> > > > > >> >> > > > delegation
> > > > > >> >> > > > > > > tokens
> > > > > >> >> > > > > > > > > at
> > > > > >> >> > > > > > > > > >> fixed
> > > > > >> >> > > > > > > > > >> > > > >> interval
> > > > > >> >> > > > > > > > > >> > > > >> >> > and
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > distribute to
> workers.
> > > If
> > > > > >> that is
> > > > > >> >> > not
> > > > > >> >> > > > the
> > > > > >> >> > > > > > case
> > > > > >> >> > > > > > > > for
> > > > > >> >> > > > > > > > > >> we can
> > > > > >> >> > > > > > > > > >> > > > >> discuss
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > delegation tokens
> > > renewing
> > > > > >> them self
> > > > > >> >> > > and
> > > > > >> >> > > > > the
> > > > > >> >> > > > > > > > > >> security
> > > > > >> >> > > > > > > > > >> > > > >> implications
> > > > > >> >> > > > > > > > > >> > > > >> >> > of the
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > same.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > I did not want
> clients
> > > to
> > > > > >> fetch
> > > > > >> >> > tokens
> > > > > >> >> > > > > from
> > > > > >> >> > > > > > > > > >> zookeeper,
> > > > > >> >> > > > > > > > > >> > > > >> overall I
> > > > > >> >> > > > > > > > > >> > > > >> >> > think
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > its
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > better if clients
> > don't
> > > > rely
> > > > > >> on our
> > > > > >> >> > > > > metadata
> > > > > >> >> > > > > > > > store
> > > > > >> >> > > > > > > > > >> and I
> > > > > >> >> > > > > > > > > >> > > > >> think we
> > > > > >> >> > > > > > > > > >> > > > >> >> > are
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > moving in that
> > direction
> > > > > with
> > > > > >> all
> > > > > >> >> > the
> > > > > >> >> > > > > KIP-4
> > > > > >> >> > > > > > > > > >> improvements.
> > > > > >> >> > > > > > > > > >> > > > I
> > > > > >> >> > > > > > > > > >> > > > >> chose
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as in this
> > > case
> > > > > the
> > > > > >> client
> > > > > >> >> > > > will
> > > > > >> >> > > > > > > still
> > > > > >> >> > > > > > > > > >> just talk
> > > > > >> >> > > > > > > > > >> > > > to
> > > > > >> >> > > > > > > > > >> > > > >> >> > broker , its
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > the brokers that
> will
> > > use
> > > > > >> zookeeper
> > > > > >> >> > > > which
> > > > > >> >> > > > > we
> > > > > >> >> > > > > > > > > >> already do
> > > > > >> >> > > > > > > > > >> > > > for a
> > > > > >> >> > > > > > > > > >> > > > >> lot
> > > > > >> >> > > > > > > > > >> > > > >> >> > of
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > other
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > usecases + ease of
> > > > > >> development + and
> > > > > >> >> > > the
> > > > > >> >> > > > > > > ability
> > > > > >> >> > > > > > > > > so
> > > > > >> >> > > > > > > > > >> > > tokens
> > > > > >> >> > > > > > > > > >> > > > >> will
> > > > > >> >> > > > > > > > > >> > > > >> >> > survive
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > even a rolling
> > > > > restart/cluster
> > > > > >> >> > > failure.
> > > > > >> >> > > > > if a
> > > > > >> >> > > > > > > > > >> majority
> > > > > >> >> > > > > > > > > >> > > > agrees
> > > > > >> >> > > > > > > > > >> > > > >> the
> > > > > >> >> > > > > > > > > >> > > > >> >> > added
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > complexity to have
> > > > > controller
> > > > > >> >> > > forwarding
> > > > > >> >> > > > > > keys
> > > > > >> >> > > > > > > to
> > > > > >> >> > > > > > > > > all
> > > > > >> >> > > > > > > > > >> > > > broker is
> > > > > >> >> > > > > > > > > >> > > > >> >> > justified
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > as
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > it provides tighter
> > > > security
> > > > > >> , I am
> > > > > >> >> > > fine
> > > > > >> >> > > > > > with
> > > > > >> >> > > > > > > > that
> > > > > >> >> > > > > > > > > >> option
> > > > > >> >> > > > > > > > > >> > > > too.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > Given zookeeper does
> > not
> > > > > >> support SSL
> > > > > >> >> > > we
> > > > > >> >> > > > > can
> > > > > >> >> > > > > > > not
> > > > > >> >> > > > > > > > > >> store
> > > > > >> >> > > > > > > > > >> > > > master
> > > > > >> >> > > > > > > > > >> > > > >> keys
> > > > > >> >> > > > > > > > > >> > > > >> >> > in
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as master
> > keys
> > > > > will
> > > > > >> be
> > > > > >> >> > > exposed
> > > > > >> >> > > > > on
> > > > > >> >> > > > > > > > wire.
> > > > > >> >> > > > > > > > > To
> > > > > >> >> > > > > > > > > >> > > > support
> > > > > >> >> > > > > > > > > >> > > > >> >> > rotation
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > without affecting
> > > current
> > > > > >> clients is
> > > > > >> >> > > > > > > something I
> > > > > >> >> > > > > > > > > >> need to
> > > > > >> >> > > > > > > > > >> > > > put
> > > > > >> >> > > > > > > > > >> > > > >> more
> > > > > >> >> > > > > > > > > >> > > > >> >> > thought
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > in. My current
> > proposal
> > > > > >> assumes the
> > > > > >> >> > > > > rotation
> > > > > >> >> > > > > > > > will
> > > > > >> >> > > > > > > > > >> > > > invalidate
> > > > > >> >> > > > > > > > > >> > > > >> all
> > > > > >> >> > > > > > > > > >> > > > >> >> > current
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > tokens.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > I request committers
> > to
> > > > also
> > > > > >> review
> > > > > >> >> > > and
> > > > > >> >> > > > > post
> > > > > >> >> > > > > > > > their
> > > > > >> >> > > > > > > > > >> > > comments
> > > > > >> >> > > > > > > > > >> > > > >> so we
> > > > > >> >> > > > > > > > > >> > > > >> >> > can
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > make
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > progress on this
> KIP.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > Parth
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016
> > at
> > > > 8:39
> > > > > >> AM,
> > > > > >> >> > > Ashish
> > > > > >> >> > > > > > Singh
> > > > > >> >> > > > > > > <
> > > > > >> >> > > > > > > > > >> > > > >> asingh@cloudera.com
> > > > > >> >> > > > > > > > > >> > > > >> >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > wrote:
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > On Mon, Apr 18,
> 2016
> > > at
> > > > > >> 11:26 AM,
> > > > > >> >> > > > > Harsha <
> > > > > >> >> > > > > > > > > >> > > > kafka@harsha.io>
> > > > > >> >> > > > > > > > > >> > > > >> >> > wrote:
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Unifying the two
> > > > > >> discussion
> > > > > >> >> > > threads
> > > > > >> >> > > > on
> > > > > >> >> > > > > > > this
> > > > > >> >> > > > > > > > > KIP.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Here is the
> > response
> > > > > from
> > > > > >> >> > Jitendra
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > "The need for a
> > > large
> > > > > >> number of
> > > > > >> >> > > > > clients
> > > > > >> >> > > > > > > that
> > > > > >> >> > > > > > > > > are
> > > > > >> >> > > > > > > > > >> > > > running
> > > > > >> >> > > > > > > > > >> > > > >> all
> > > > > >> >> > > > > > > > > >> > > > >> >> > over the
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > cluster that
> > > > > authenticate
> > > > > >> with
> > > > > >> >> > > Kafka
> > > > > >> >> > > > > > > > brokers,
> > > > > >> >> > > > > > > > > >> is very
> > > > > >> >> > > > > > > > > >> > > > >> similar
> > > > > >> >> > > > > > > > > >> > > > >> >> > to the
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Hadoop use case
> of
> > > > large
> > > > > >> number
> > > > > >> >> > of
> > > > > >> >> > > > > tasks
> > > > > >> >> > > > > > > > > running
> > > > > >> >> > > > > > > > > >> > > across
> > > > > >> >> > > > > > > > > >> > > > >> the
> > > > > >> >> > > > > > > > > >> > > > >> >> > cluster
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> that
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need
> > authentication
> > > to
> > > > > >> Hdfs
> > > > > >> >> > > > Namenode.
> > > > > >> >> > > > > > > > > >> Therefore, the
> > > > > >> >> > > > > > > > > >> > > > >> >> > delegation token
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > approach does
> seem
> > > > like
> > > > > a
> > > > > >> good
> > > > > >> >> > fit
> > > > > >> >> > > > for
> > > > > >> >> > > > > > > this
> > > > > >> >> > > > > > > > > use
> > > > > >> >> > > > > > > > > >> case
> > > > > >> >> > > > > > > > > >> > > > as we
> > > > > >> >> > > > > > > > > >> > > > >> >> > have seen
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> it
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > working at large
> > > scale
> > > > > in
> > > > > >> HDFS
> > > > > >> >> > and
> > > > > >> >> > > > > YARN.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >   The proposed
> > > design
> > > > is
> > > > > >> very
> > > > > >> >> > much
> > > > > >> >> > > > > > inline
> > > > > >> >> > > > > > > > with
> > > > > >> >> > > > > > > > > >> Hadoop
> > > > > >> >> > > > > > > > > >> > > > >> >> > approach. A few
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >   comments:
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 1) Why do you
> guys
> > > > want
> > > > > >> to allow
> > > > > >> >> > > > > > infinite
> > > > > >> >> > > > > > > > > >> renewable
> > > > > >> >> > > > > > > > > >> > > > >> lifetime
> > > > > >> >> > > > > > > > > >> > > > >> >> > for a
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > token? HDFS
> > > restricts
> > > > a
> > > > > >> token
> > > > > >> >> > to a
> > > > > >> >> > > > max
> > > > > >> >> > > > > > > life
> > > > > >> >> > > > > > > > > time
> > > > > >> >> > > > > > > > > >> > > > (default
> > > > > >> >> > > > > > > > > >> > > > >> 7
> > > > > >> >> > > > > > > > > >> > > > >> >> > days).  A
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > token's
> > > vulnerability
> > > > is
> > > > > >> >> > believed
> > > > > >> >> > > to
> > > > > >> >> > > > > > > > increase
> > > > > >> >> > > > > > > > > >> with
> > > > > >> >> > > > > > > > > >> > > > time.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > I agree that
> having
> > > > > infinite
> > > > > >> >> > > lifetime
> > > > > >> >> > > > > > might
> > > > > >> >> > > > > > > > not
> > > > > >> >> > > > > > > > > >> be the
> > > > > >> >> > > > > > > > > >> > > > best
> > > > > >> >> > > > > > > > > >> > > > >> idea.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 2) As I
> understand
> > > the
> > > > > >> tokens
> > > > > >> >> > are
> > > > > >> >> > > > > stored
> > > > > >> >> > > > > > > in
> > > > > >> >> > > > > > > > > >> zookeeper
> > > > > >> >> > > > > > > > > >> > > > as
> > > > > >> >> > > > > > > > > >> > > > >> well,
> > > > > >> >> > > > > > > > > >> > > > >> >> > and
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> can
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > be updated
> there.
> > > This
> > > > > is
> > > > > >> clever
> > > > > >> >> > > as
> > > > > >> >> > > > it
> > > > > >> >> > > > > > can
> > > > > >> >> > > > > > > > > allow
> > > > > >> >> > > > > > > > > >> > > > >> replacing the
> > > > > >> >> > > > > > > > > >> > > > >> >> > tokens
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > once they run
> out
> > of
> > > > max
> > > > > >> life
> > > > > >> >> > > time,
> > > > > >> >> > > > > and
> > > > > >> >> > > > > > > > > clients
> > > > > >> >> > > > > > > > > >> can
> > > > > >> >> > > > > > > > > >> > > > >> download
> > > > > >> >> > > > > > > > > >> > > > >> >> > new
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> tokens
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from zookeeper.
> It
> > > > > >> shouldn't be
> > > > > >> >> > a
> > > > > >> >> > > > big
> > > > > >> >> > > > > > load
> > > > > >> >> > > > > > > > on
> > > > > >> >> > > > > > > > > >> > > zookeeper
> > > > > >> >> > > > > > > > > >> > > > >> as a
> > > > > >> >> > > > > > > > > >> > > > >> >> > client
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> will
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need to get a
> new
> > > > token
> > > > > >> once in
> > > > > >> >> > > > > several
> > > > > >> >> > > > > > > > days.
> > > > > >> >> > > > > > > > > >> In this
> > > > > >> >> > > > > > > > > >> > > > >> approach
> > > > > >> >> > > > > > > > > >> > > > >> >> > you
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> don't
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need infinite
> > > lifetime
> > > > > on
> > > > > >> the
> > > > > >> >> > > token
> > > > > >> >> > > > > even
> > > > > >> >> > > > > > > for
> > > > > >> >> > > > > > > > > >> long
> > > > > >> >> > > > > > > > > >> > > > running
> > > > > >> >> > > > > > > > > >> > > > >> >> > clients.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 3) The token
> > > password
> > > > > are
> > > > > >> >> > > generated
> > > > > >> >> > > > > > using
> > > > > >> >> > > > > > > a
> > > > > >> >> > > > > > > > > >> master
> > > > > >> >> > > > > > > > > >> > > key.
> > > > > >> >> > > > > > > > > >> > > > >> The
> > > > > >> >> > > > > > > > > >> > > > >> >> > master
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> key
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > should also be
> > > > > >> periodically
> > > > > >> >> > > changed.
> > > > > >> >> > > > > In
> > > > > >> >> > > > > > > > > Hadoop,
> > > > > >> >> > > > > > > > > >> the
> > > > > >> >> > > > > > > > > >> > > > >> default
> > > > > >> >> > > > > > > > > >> > > > >> >> > renewal
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > period is 1
> day.?
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > IIUC, this will
> > > require
> > > > > >> brokers
> > > > > >> >> > > > > > maintaining
> > > > > >> >> > > > > > > a
> > > > > >> >> > > > > > > > > >> list of X
> > > > > >> >> > > > > > > > > >> > > > most
> > > > > >> >> > > > > > > > > >> > > > >> >> > recent
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> master
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > keys. This list
> will
> > > > have
> > > > > >> to be
> > > > > >> >> > > > > persisted
> > > > > >> >> > > > > > > > > >> somewhere, as
> > > > > >> >> > > > > > > > > >> > > > if a
> > > > > >> >> > > > > > > > > >> > > > >> >> > broker
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> goes
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > down it will have
> to
> > > get
> > > > > >> that list
> > > > > >> >> > > > again
> > > > > >> >> > > > > > and
> > > > > >> >> > > > > > > > > >> storing
> > > > > >> >> > > > > > > > > >> > > > master
> > > > > >> >> > > > > > > > > >> > > > >> keys
> > > > > >> >> > > > > > > > > >> > > > >> >> > on ZK
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> is
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > not the best idea.
> > > > > However,
> > > > > >> if a
> > > > > >> >> > > > broker
> > > > > >> >> > > > > > goes
> > > > > >> >> > > > > > > > > down
> > > > > >> >> > > > > > > > > >> then
> > > > > >> >> > > > > > > > > >> > > we
> > > > > >> >> > > > > > > > > >> > > > >> have
> > > > > >> >> > > > > > > > > >> > > > >> >> > much
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> bigger
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > issue to deal with
> > and
> > > > > >> client can
> > > > > >> >> > > > always
> > > > > >> >> > > > > > > > > >> > > re-authenticate
> > > > > >> >> > > > > > > > > >> > > > is
> > > > > >> >> > > > > > > > > >> > > > >> such
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> events.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > Did you happen to
> > > take a
> > > > > >> look at
> > > > > >> >> > > other
> > > > > >> >> > > > > > > > > >> alternatives
> > > > > >> >> > > > > > > > > >> > > this
> > > > > >> >> > > > > > > > > >> > > > >> list has
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > suggested?
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Thanks for a
> > > thorough
> > > > > >> proposal,
> > > > > >> >> > > > great
> > > > > >> >> > > > > > > work!"
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7,
> > 2016,
> > > > at
> > > > > >> 10:28
> > > > > >> >> > PM,
> > > > > >> >> > > > Gwen
> > > > > >> >> > > > > > > > Shapira
> > > > > >> >> > > > > > > > > >> wrote:
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > Makes sense to
> > me.
> > > > > >> Thanks!
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7,
> > > 2016
> > > > at
> > > > > >> 9:25
> > > > > >> >> > PM,
> > > > > >> >> > > > > > Harsha <
> > > > > >> >> > > > > > > > > >> > > > kafka@harsha.io
> > > > > >> >> > > > > > > > > >> > > > >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > wrote:
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > It doesn't
> > need
> > > > any
> > > > > >> release
> > > > > >> >> > > > > vehicle
> > > > > >> >> > > > > > > but
> > > > > >> >> > > > > > > > > >> still the
> > > > > >> >> > > > > > > > > >> > > > >> work can
> > > > > >> >> > > > > > > > > >> > > > >> >> > move
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > forward.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > If anyone is
> > > > > >> interested in
> > > > > >> >> > the
> > > > > >> >> > > > KIP
> > > > > >> >> > > > > > > > please
> > > > > >> >> > > > > > > > > >> do the
> > > > > >> >> > > > > > > > > >> > > > >> review and
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> provide
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > the
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > comments.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > -Harsha
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar
> 7,
> > > > 2016,
> > > > > >> at
> > > > > >> >> > 04:59
> > > > > >> >> > > > PM,
> > > > > >> >> > > > > > > Ismael
> > > > > >> >> > > > > > > > > >> Juma
> > > > > >> >> > > > > > > > > >> > > > wrote:
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> I agree
> that
> > it
> > > > > >> would be
> > > > > >> >> > good
> > > > > >> >> > > > to
> > > > > >> >> > > > > > have
> > > > > >> >> > > > > > > > > more
> > > > > >> >> > > > > > > > > >> time
> > > > > >> >> > > > > > > > > >> > > to
> > > > > >> >> > > > > > > > > >> > > > >> review
> > > > > >> >> > > > > > > > > >> > > > >> >> > and
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > discuss
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> KIP-48.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> Ismael
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar
> > 8,
> > > > 2016
> > > > > >> at
> > > > > >> >> > 12:55
> > > > > >> >> > > > AM,
> > > > > >> >> > > > > > Gwen
> > > > > >> >> > > > > > > > > >> Shapira <
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> gwen@confluent.io>
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Hi Team,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Since
> > KIP-48
> > > > > >> depends on
> > > > > >> >> > > > KIP-43,
> > > > > >> >> > > > > > > which
> > > > > >> >> > > > > > > > > is
> > > > > >> >> > > > > > > > > >> > > > already a
> > > > > >> >> > > > > > > > > >> > > > >> bit
> > > > > >> >> > > > > > > > > >> > > > >> >> > of a
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> risk
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > for
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > the next
> > > > release
> > > > > -
> > > > > >> any
> > > > > >> >> > > chance
> > > > > >> >> > > > > we
> > > > > >> >> > > > > > > can
> > > > > >> >> > > > > > > > > >> delay
> > > > > >> >> > > > > > > > > >> > > > >> delegation
> > > > > >> >> > > > > > > > > >> > > > >> >> > tokens
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> to
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Kafka
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > 0.10.1?
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > With the
> > > > > community
> > > > > >> >> > working
> > > > > >> >> > > > on a
> > > > > >> >> > > > > > > > release
> > > > > >> >> > > > > > > > > >> every
> > > > > >> >> > > > > > > > > >> > > 3
> > > > > >> >> > > > > > > > > >> > > > >> month,
> > > > > >> >> > > > > > > > > >> > > > >> >> > this
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> is not
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a huge
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > delay.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Gwen
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > On Fri,
> Feb
> > > 26,
> > > > > >> 2016 at
> > > > > >> >> > > 5:11
> > > > > >> >> > > > > PM,
> > > > > >> >> > > > > > > > Ashish
> > > > > >> >> > > > > > > > > >> Singh
> > > > > >> >> > > > > > > > > >> > > <
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> asingh@cloudera.com
> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Parth,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Thanks
> > > again
> > > > > for
> > > > > >> the
> > > > > >> >> > > > awesome
> > > > > >> >> > > > > > > write
> > > > > >> >> > > > > > > > > up.
> > > > > >> >> > > > > > > > > >> > > > Following
> > > > > >> >> > > > > > > > > >> > > > >> our
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> discussion
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from the
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I
> > > think
> > > > > it
> > > > > >> will
> > > > > >> >> > be
> > > > > >> >> > > > > easier
> > > > > >> >> > > > > > > to
> > > > > >> >> > > > > > > > > >> compare
> > > > > >> >> > > > > > > > > >> > > > >> various
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> alternatives
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > if they
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > are
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > listed
> > > > > together.
> > > > > >> I am
> > > > > >> >> > > > stating
> > > > > >> >> > > > > > > > below a
> > > > > >> >> > > > > > > > > >> few
> > > > > >> >> > > > > > > > > >> > > > >> >> > alternatives along
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > with
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a the
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > current
> > > > > proposal.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> (Current
> > > > > >> proposal)
> > > > > >> >> > Store
> > > > > >> >> > > > > > > Delegation
> > > > > >> >> > > > > > > > > >> Token,
> > > > > >> >> > > > > > > > > >> > > DT,
> > > > > >> >> > > > > > > > > >> > > > >> on ZK.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1.
> > > Client
> > > > > >> >> > > authenticates
> > > > > >> >> > > > > > with a
> > > > > >> >> > > > > > > > > >> broker.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2.
> > Once
> > > a
> > > > > >> client is
> > > > > >> >> > > > > > > > authenticated,
> > > > > >> >> > > > > > > > > >> it
> > > > > >> >> > > > > > > > > >> > > will
> > > > > >> >> > > > > > > > > >> > > > >> make a
> > > > > >> >> > > > > > > > > >> > > > >> >> > broker
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> side
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> issue
> > a
> > > > > >> delegation
> > > > > >> >> > > > token.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3.
> The
> > > > > broker
> > > > > >> >> > > generates
> > > > > >> >> > > > a
> > > > > >> >> > > > > > > shared
> > > > > >> >> > > > > > > > > >> secret
> > > > > >> >> > > > > > > > > >> > > > based
> > > > > >> >> > > > > > > > > >> > > > >> on
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > HMAC-SHA256(a
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> Password/Secret
> > > > > >> >> > shared
> > > > > >> >> > > > > > between
> > > > > >> >> > > > > > > > all
> > > > > >> >> > > > > > > > > >> > > brokers,
> > > > > >> >> > > > > > > > > >> > > > >> >> > randomly
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > generated
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > tokenId).
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4.
> > > Broker
> > > > > >> stores
> > > > > >> >> > this
> > > > > >> >> > > > > token
> > > > > >> >> > > > > > in
> > > > > >> >> > > > > > > > its
> > > > > >> >> > > > > > > > > >> in
> > > > > >> >> > > > > > > > > >> > > > memory
> > > > > >> >> > > > > > > > > >> > > > >> cache.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> Broker
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > also stores
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the
> > > > > >> DelegationToken
> > > > > >> >> > > > > without
> > > > > >> >> > > > > > > the
> > > > > >> >> > > > > > > > > >> hmac in
> > > > > >> >> > > > > > > > > >> > > the
> > > > > >> >> > > > > > > > > >> > > > >> >> > zookeeper.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5.
> All
> > > > > >> brokers will
> > > > > >> >> > > > have a
> > > > > >> >> > > > > > > cache
> > > > > >> >> > > > > > > > > >> backed
> > > > > >> >> > > > > > > > > >> > > by
> > > > > >> >> > > > > > > > > >> > > > >> >> > zookeeper so
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> they
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > will all
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    get
> > > > notified
> > > > > >> >> > whenever
> > > > > >> >> > > a
> > > > > >> >> > > > > new
> > > > > >> >> > > > > > > > token
> > > > > >> >> > > > > > > > > is
> > > > > >> >> > > > > > > > > >> > > > >> generated and
> > > > > >> >> > > > > > > > > >> > > > >> >> > they
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> will
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > update
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> local
> > > > cache
> > > > > >> whenever
> > > > > >> >> > > > token
> > > > > >> >> > > > > > > state
> > > > > >> >> > > > > > > > > >> changes.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6.
> > > Broker
> > > > > >> returns
> > > > > >> >> > the
> > > > > >> >> > > > > token
> > > > > >> >> > > > > > to
> > > > > >> >> > > > > > > > > >> Client.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> Probable
> > > > issues
> > > > > >> and
> > > > > >> >> > fixes
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1.
> > > > Probable
> > > > > >> race
> > > > > >> >> > > > > condition,
> > > > > >> >> > > > > > > > client
> > > > > >> >> > > > > > > > > >> tries
> > > > > >> >> > > > > > > > > >> > > to
> > > > > >> >> > > > > > > > > >> > > > >> >> > authenticate
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> with
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a broker
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    that
> > is
> > > > yet
> > > > > >> to be
> > > > > >> >> > > > updated
> > > > > >> >> > > > > > with
> > > > > >> >> > > > > > > > the
> > > > > >> >> > > > > > > > > >> newly
> > > > > >> >> > > > > > > > > >> > > > >> generated
> > > > > >> >> > > > > > > > > >> > > > >> >> > DT.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> This
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > can
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > probably
> be
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> dealt
> > > with
> > > > > >> making
> > > > > >> >> > > > > dtRequest
> > > > > >> >> > > > > > > > block
> > > > > >> >> > > > > > > > > >> until
> > > > > >> >> > > > > > > > > >> > > all
> > > > > >> >> > > > > > > > > >> > > > >> >> > brokers have
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > updated
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their DT
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> cache.
> > > Zk
> > > > > >> barrier or
> > > > > >> >> > > > > similar
> > > > > >> >> > > > > > > > > >> mechanism
> > > > > >> >> > > > > > > > > >> > > can
> > > > > >> >> > > > > > > > > >> > > > be
> > > > > >> >> > > > > > > > > >> > > > >> used.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> However,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > all such
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > mechanisms
> > > > > >> will
> > > > > >> >> > > increase
> > > > > >> >> > > > > > > > > complexity.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2.
> > > Using a
> > > > > >> static
> > > > > >> >> > > secret
> > > > > >> >> > > > > key
> > > > > >> >> > > > > > > > from
> > > > > >> >> > > > > > > > > >> config
> > > > > >> >> > > > > > > > > >> > > > >> file. Will
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> require
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > yet
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > another
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> config
> > > and
> > > > > >> uses a
> > > > > >> >> > > static
> > > > > >> >> > > > > > > secret
> > > > > >> >> > > > > > > > > >> key. It
> > > > > >> >> > > > > > > > > >> > > is
> > > > > >> >> > > > > > > > > >> > > > >> advised
> > > > > >> >> > > > > > > > > >> > > > >> >> > to
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> rotate
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > secret
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > keys
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > periodically.
> > > > > >> This
> > > > > >> >> > can
> > > > > >> >> > > > be
> > > > > >> >> > > > > > > > avoided
> > > > > >> >> > > > > > > > > >> with
> > > > > >> >> > > > > > > > > >> > > > >> controller
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> generating
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > secretKey
> > and
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > passing
> > > to
> > > > > >> brokers
> > > > > >> >> > > > > > > periodically.
> > > > > >> >> > > > > > > > > >> However,
> > > > > >> >> > > > > > > > > >> > > > >> this will
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> require
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > brokers to
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > maintain
> > > > > >> certain
> > > > > >> >> > > counts
> > > > > >> >> > > > of
> > > > > >> >> > > > > > > > > >> secretKeys.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > (Alternative
> > > > 1)
> > > > > >> Have
> > > > > >> >> > > > > controller
> > > > > >> >> > > > > > > > > >> generate
> > > > > >> >> > > > > > > > > >> > > > >> delegation
> > > > > >> >> > > > > > > > > >> > > > >> >> > token.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1.
> > > Client
> > > > > >> >> > > authenticates
> > > > > >> >> > > > > > with a
> > > > > >> >> > > > > > > > > >> broker.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2.
> > Once
> > > a
> > > > > >> client is
> > > > > >> >> > > > > > > > authenticated,
> > > > > >> >> > > > > > > > > >> it
> > > > > >> >> > > > > > > > > >> > > will
> > > > > >> >> > > > > > > > > >> > > > >> make a
> > > > > >> >> > > > > > > > > >> > > > >> >> > broker
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> side
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> issue
> > a
> > > > > >> delegation
> > > > > >> >> > > > token.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3.
> > > Broker
> > > > > >> forwards
> > > > > >> >> > the
> > > > > >> >> > > > > > request
> > > > > >> >> > > > > > > > to
> > > > > >> >> > > > > > > > > >> > > > controller.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4.
> > > > > Controller
> > > > > >> >> > > generates
> > > > > >> >> > > > a
> > > > > >> >> > > > > DT
> > > > > >> >> > > > > > > and
> > > > > >> >> > > > > > > > > >> > > broadcasts
> > > > > >> >> > > > > > > > > >> > > > >> to all
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> brokers.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5.
> > > Broker
> > > > > >> stores
> > > > > >> >> > this
> > > > > >> >> > > > > token
> > > > > >> >> > > > > > in
> > > > > >> >> > > > > > > > its
> > > > > >> >> > > > > > > > > >> memory
> > > > > >> >> > > > > > > > > >> > > > >> cache.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6.
> > > > > Controller
> > > > > >> >> > responds
> > > > > >> >> > > > to
> > > > > >> >> > > > > > > > broker’s
> > > > > >> >> > > > > > > > > >> DT
> > > > > >> >> > > > > > > > > >> > > req.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    7.
> > > Broker
> > > > > >> returns
> > > > > >> >> > the
> > > > > >> >> > > > > token
> > > > > >> >> > > > > > to
> > > > > >> >> > > > > > > > > >> Client.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> Probable
> > > > issues
> > > > > >> and
> > > > > >> >> > fixes
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1.
> We
> > > will
> > > > > >> have to
> > > > > >> >> > add
> > > > > >> >> > > > new
> > > > > >> >> > > > > > > APIs
> > > > > >> >> > > > > > > > to
> > > > > >> >> > > > > > > > > >> > > support
> > > > > >> >> > > > > > > > > >> > > > >> >> > controller
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> pushing
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > to
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > brokers
> > > on
> > > > > >> top of
> > > > > >> >> > the
> > > > > >> >> > > > > > minimal
> > > > > >> >> > > > > > > > APIs
> > > > > >> >> > > > > > > > > >> that
> > > > > >> >> > > > > > > > > >> > > are
> > > > > >> >> > > > > > > > > >> > > > >> >> > currently
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > proposed.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2.
> We
> > > will
> > > > > >> also have
> > > > > >> >> > > to
> > > > > >> >> > > > > add
> > > > > >> >> > > > > > > APIs
> > > > > >> >> > > > > > > > > to
> > > > > >> >> > > > > > > > > >> > > support
> > > > > >> >> > > > > > > > > >> > > > >> the
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> bootstrapping
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > case,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > i.e,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> when a
> > > new
> > > > > >> broker
> > > > > >> >> > > comes
> > > > > >> >> > > > up
> > > > > >> >> > > > > > it
> > > > > >> >> > > > > > > > will
> > > > > >> >> > > > > > > > > >> have
> > > > > >> >> > > > > > > > > >> > > to
> > > > > >> >> > > > > > > > > >> > > > >> get all
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> delegation
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > from
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the
> > > > > >> controller.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3.
> In
> > > > > >> catastrophic
> > > > > >> >> > > > > failures
> > > > > >> >> > > > > > > > where
> > > > > >> >> > > > > > > > > >> all
> > > > > >> >> > > > > > > > > >> > > > brokers
> > > > > >> >> > > > > > > > > >> > > > >> go
> > > > > >> >> > > > > > > > > >> > > > >> >> > down,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> the
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be
> > lost
> > > > even
> > > > > >> if
> > > > > >> >> > > servers
> > > > > >> >> > > > > are
> > > > > >> >> > > > > > > > > >> restarted as
> > > > > >> >> > > > > > > > > >> > > > >> tokens
> > > > > >> >> > > > > > > > > >> > > > >> >> > are not
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If
> > this
> > > > > >> happens,
> > > > > >> >> > then
> > > > > >> >> > > > > there
> > > > > >> >> > > > > > > are
> > > > > >> >> > > > > > > > > more
> > > > > >> >> > > > > > > > > >> > > > important
> > > > > >> >> > > > > > > > > >> > > > >> >> > things to
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > worry
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > about
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> maybe
> > it
> > > > is
> > > > > >> better
> > > > > >> >> > to
> > > > > >> >> > > > > > > > > >> re-authenticate.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > (Alternative
> > > > 2)
> > > > > >> Do not
> > > > > >> >> > > > > > distribute
> > > > > >> >> > > > > > > > DT
> > > > > >> >> > > > > > > > > to
> > > > > >> >> > > > > > > > > >> > > other
> > > > > >> >> > > > > > > > > >> > > > >> brokers
> > > > > >> >> > > > > > > > > >> > > > >> >> > at
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> all.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1.
> > > Client
> > > > > >> >> > > authenticates
> > > > > >> >> > > > > > with a
> > > > > >> >> > > > > > > > > >> broker.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2.
> > Once
> > > a
> > > > > >> client is
> > > > > >> >> > > > > > > > authenticated,
> > > > > >> >> > > > > > > > > >> it
> > > > > >> >> > > > > > > > > >> > > will
> > > > > >> >> > > > > > > > > >> > > > >> make a
> > > > > >> >> > > > > > > > > >> > > > >> >> > broker
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> side
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> issue
> > a
> > > > > >> delegation
> > > > > >> >> > > > token.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3.
> The
> > > > > broker
> > > > > >> >> > > generates
> > > > > >> >> > > > DT
> > > > > >> >> > > > > > of
> > > > > >> >> > > > > > > > > form,
> > > > > >> >> > > > > > > > > >> > > [hmac +
> > > > > >> >> > > > > > > > > >> > > > >> (owner,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> renewer,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > maxLifeTime,
> > > > > >> id,
> > > > > >> >> > hmac,
> > > > > >> >> > > > > > > > > >> expirationTime)]
> > > > > >> >> > > > > > > > > >> > > and
> > > > > >> >> > > > > > > > > >> > > > >> passes
> > > > > >> >> > > > > > > > > >> > > > >> >> > back
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> this
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > DT to
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > client.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    hmac
> > is
> > > > > >> generated
> > > > > >> >> > via
> > > > > >> >> > > > > > > > > >> {HMAC-SHA256(owner,
> > > > > >> >> > > > > > > > > >> > > > >> renewer,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > maxLifeTime, id,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> expirationTime)
> > > > > >> >> > using
> > > > > >> >> > > > > > > > SecretKey}.
> > > > > >> >> > > > > > > > > >> Note
> > > > > >> >> > > > > > > > > >> > > that
> > > > > >> >> > > > > > > > > >> > > > >> all
> > > > > >> >> > > > > > > > > >> > > > >> >> > brokers
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> have
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > this
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> SecretKey.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4.
> > > Client
> > > > > >> then goes
> > > > > >> >> > to
> > > > > >> >> > > > any
> > > > > >> >> > > > > > > > broker
> > > > > >> >> > > > > > > > > >> and to
> > > > > >> >> > > > > > > > > >> > > > >> >> > authenticate
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> sends
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > the DT.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> Broker
> > > > > >> recalculates
> > > > > >> >> > > hmac
> > > > > >> >> > > > > > using
> > > > > >> >> > > > > > > > > >> (owner,
> > > > > >> >> > > > > > > > > >> > > > >> renewer,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> maxLifeTime,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > id, hmac,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> expirationTime) info
> > > > > >> >> > > > from
> > > > > >> >> > > > > DT
> > > > > >> >> > > > > > > and
> > > > > >> >> > > > > > > > > its
> > > > > >> >> > > > > > > > > >> > > > >> SecretKey. If
> > > > > >> >> > > > > > > > > >> > > > >> >> > it
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> matches
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > with
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac of
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    DT,
> > > client
> > > > > is
> > > > > >> >> > > > > authenticated.
> > > > > >> >> > > > > > > > Yes,
> > > > > >> >> > > > > > > > > >> it will
> > > > > >> >> > > > > > > > > >> > > > do
> > > > > >> >> > > > > > > > > >> > > > >> other
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> obvious
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > checks of
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > timestamp
> > > > > >> expiry and
> > > > > >> >> > > > such.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Note
> that
> > > > > secret
> > > > > >> key
> > > > > >> >> > will
> > > > > >> >> > > > be
> > > > > >> >> > > > > > > > > generated
> > > > > >> >> > > > > > > > > >> by
> > > > > >> >> > > > > > > > > >> > > > >> controller
> > > > > >> >> > > > > > > > > >> > > > >> >> > and
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> passed
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > to
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > brokers
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > periodically.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> Probable
> > > > issues
> > > > > >> and
> > > > > >> >> > fixes
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1.
> How
> > > to
> > > > > >> delete a
> > > > > >> >> > DT?
> > > > > >> >> > > > > Yes,
> > > > > >> >> > > > > > > that
> > > > > >> >> > > > > > > > > is
> > > > > >> >> > > > > > > > > >> a
> > > > > >> >> > > > > > > > > >> > > > downside
> > > > > >> >> > > > > > > > > >> > > > >> >> > here.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> However,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > this can
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be
> > > handled
> > > > > >> with
> > > > > >> >> > > brokers
> > > > > >> >> > > > > > > > > maintaining
> > > > > >> >> > > > > > > > > >> a
> > > > > >> >> > > > > > > > > >> > > > >> blacklist of
> > > > > >> >> > > > > > > > > >> > > > >> >> > DTs,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> DTs
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from this
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > list
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    can
> be
> > > > > >> removed after
> > > > > >> >> > > > > expiry.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2.
> In
> > > > > >> catastrophic
> > > > > >> >> > > > > failures
> > > > > >> >> > > > > > > > where
> > > > > >> >> > > > > > > > > >> all
> > > > > >> >> > > > > > > > > >> > > > brokers
> > > > > >> >> > > > > > > > > >> > > > >> go
> > > > > >> >> > > > > > > > > >> > > > >> >> > down,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> the
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be
> > lost
> > > > even
> > > > > >> if
> > > > > >> >> > > servers
> > > > > >> >> > > > > are
> > > > > >> >> > > > > > > > > >> restarted as
> > > > > >> >> > > > > > > > > >> > > > >> tokens
> > > > > >> >> > > > > > > > > >> > > > >> >> > are not
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If
> > this
> > > > > >> happens,
> > > > > >> >> > then
> > > > > >> >> > > > > there
> > > > > >> >> > > > > > > are
> > > > > >> >> > > > > > > > > more
> > > > > >> >> > > > > > > > > >> > > > important
> > > > > >> >> > > > > > > > > >> > > > >> >> > things to
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > worry
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > about
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> maybe
> > it
> > > > is
> > > > > >> better
> > > > > >> >> > to
> > > > > >> >> > > > > > > > > >> re-authenticate.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > On Fri,
> > Feb
> > > > 26,
> > > > > >> 2016 at
> > > > > >> >> > > > 1:58
> > > > > >> >> > > > > > PM,
> > > > > >> >> > > > > > > > > Parth
> > > > > >> >> > > > > > > > > >> > > > >> Brahmbhatt <
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > pbrahmbhatt@hortonworks.com>
> > > > > >> >> > > > > > > > wrote:
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Hi,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> I have
> > > filed
> > > > > >> KIP-48 so
> > > > > >> >> > > we
> > > > > >> >> > > > > can
> > > > > >> >> > > > > > > > offer
> > > > > >> >> > > > > > > > > >> hadoop
> > > > > >> >> > > > > > > > > >> > > > like
> > > > > >> >> > > > > > > > > >> > > > >> >> > delegation
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens in
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> kafka.
> > You
> > > > can
> > > > > >> review
> > > > > >> >> > > the
> > > > > >> >> > > > > > design
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > > > > >> >> > > > > > > > > >> > > > >> >> >
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > >
> > > > > >> >> > > > > > > > > >>
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > >
> > > > > >> >> > > > >
> > > > > >> >> > > >
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > .
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> This
> KIP
> > > > > >> depends on
> > > > > >> >> > > KIP-43
> > > > > >> >> > > > > and
> > > > > >> >> > > > > > > we
> > > > > >> >> > > > > > > > > >> have also
> > > > > >> >> > > > > > > > > >> > > > >> >> > discussed an
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > alternative to
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> proposed
> > > > > design
> > > > > >> here<
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > > > > >> >> > > > > > > > > >> > > > >> >> >
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > >
> > > > > >> >> > > > > > > > > >>
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > >
> > > > > >> >> > > > >
> > > > > >> >> > > >
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >.
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Thanks
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Parth
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > --
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> Regards,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Ashish
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > --
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > Regards,
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > Ashish
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > > > > >> >> > > > > > > > > >> > > > >> >> >
> > > > > >> >> > > > > > > > > >> > > > >>
> > > > > >> >> > > > > > > > > >> > > >
> > > > > >> >> > > > > > > > > >> > >
> > > > > >> >> > > > > > > > > >>
> > > > > >> >> > > > > > > > > >>
> > > > > >> >> > > > > > > > > >
> > > > > >> >> > > > > > > > >
> > > > > >> >> > > > > > > >
> > > > > >> >> > > > > > >
> > > > > >> >> > > > > >
> > > > > >> >> > > > > >
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > --
> > > > > >> >> > > > > > Regards,
> > > > > >> >> > > > > >
> > > > > >> >> > > > > > Rajini
> > > > > >> >> > > > > >
> > > > > >> >> > > > >
> > > > > >> >> > > >
> > > > > >> >> > > >
> > > > > >> >> > > >
> > > > > >> >> > > > --
> > > > > >> >> > > > Regards,
> > > > > >> >> > > >
> > > > > >> >> > > > Rajini
> > > > > >> >> > > >
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> --
> > > > > >> >> Liquan Pei
> > > > > >> >> Software Engineer, Confluent Inc
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>



-- 
Grant Henke
Software Engineer | Cloudera
grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Jun Rao <ju...@confluent.io>]

Parth,

Thanks for the reply.

It makes sense to only allow the renewal by users that authenticated using
*non* delegation token mechanism. Then, should we make the renewal a list?
For example, in the case of rest proxy, it will be useful for every
instance of rest proxy to be able to renew the tokens.

It would be clearer if we can document the request protocol like
https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945):(VotedandPlannedforin0.10.1.0)
.

It would also be useful to document the client APIs.

Thanks,

Jun

On Tue, Jun 28, 2016 at 2:55 PM, parth brahmbhatt <
brahmbhatt.parth@gmail.com> wrote:

> Hi,
>
> I am suggesting that we will only allow the renewal by users that
> authenticated using *non* delegation token mechanism. For example, If user
> Alice authenticated using kerberos and requested delegation tokens, only
> user Alice authenticated via non delegation token mechanism can renew.
> Clients that have  access to delegation tokens can not issue renewal
> request for renewing their own token and this is primarily important to
> reduce the time window for which a compromised token will be valid.
>
> To clarify, Yes any authenticated user can request delegation tokens but
> even here I would recommend to avoid creating a chain where a client
> authenticated via delegation token request for more delegation tokens.
> Basically anyone can request delegation token, as long as they authenticate
> via a non delegation token mechanism.
>
> Aren't classes listed here
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-PublicInterfaces
> >
> sufficient?
>
> Thanks
> Parth
>
>
>
> On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao <ju...@confluent.io> wrote:
>
> > Parth,
> >
> > Thanks for the reply. A couple of comments inline below.
> >
> > On Tue, Jun 21, 2016 at 10:36 AM, parth brahmbhatt <
> > brahmbhatt.parth@gmail.com> wrote:
> >
> > > 1. Who / how are tokens renewed? By original requester only? or using
> > > Kerberos
> > > auth only?
> > > My recommendation is to do this only using Kerberos auth and only threw
> > the
> > > renewer specified during the acquisition request.
> > >
> > >
> > Hmm, not sure that I follow this. Are you saying that any client
> > authenticated with the delegation token can renew, i.e. there is no
> renewer
> > needed?
> >
> > Also, just to be clear, any authenticated client (either through SASL or
> > SSL) can request a delegation token for the authenticated user, right?
> >
> >
> > > 2. Are tokens stored on each broker or in ZK?
> > > My recommendation is still to store in ZK or not store them at all. The
> > > whole controller based distribution is too much overhead with not much
> to
> > > achieve.
> > >
> > > 3. How are tokens invalidated / expired?
> > > Either by expiration time out or through an explicit request to
> > invalidate.
> > >
> > > 4. Which encryption algorithm is used?
> > > SCRAM
> > >
> > > 5. What is the impersonation proposal (it wasn't in the KIP but was
> > > discussed
> > > in this thread)?
> > > There is no imperonation proposal. I tried and explained how its a
> > > different problem and why its not really necessary to discuss that as
> > part
> > > of this KIP.  This KIP will not support any impersonation, it will just
> > be
> > > another way to authenticate.
> > >
> > > 6. Do we need new ACLs, if so - for what actions?
> > > We do not need new ACLs.
> > >
> > >
> > Could we document the format of the new request/response and their
> > associated Resource and Operation for ACL?
> >
> >
> > > 7. How would the delegation token be configured in the client?
> > > Should be through config. I wasn't planning on supporting JAAS for
> > tokens.
> > > I don't believe hadoop does this either.
> > >
> > > Thanks
> > > Parth
> > >
> > >
> > >
> > > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Harsha,
> > > >
> > > > Another question.
> > > >
> > > > 9. How would the delegation token be configured in the client? The
> > > standard
> > > > way is to do this through JAAS. However, we will need to think
> through
> > if
> > > > this is convenient in a shared environment. For example, when a new
> > task
> > > is
> > > > added to a Storm worker node, do we need to dynamically add a new
> > section
> > > > in the JAAS file? It may be more convenient if we can pass in the
> token
> > > > through the config directly w/o going through JAAS.
> > > >
> > > > Are you or Parth still actively working on this KIP?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > >
> > > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <ju...@confluent.io> wrote:
> > > >
> > > > > Just to add on that list.
> > > > >
> > > > > 2. It would be good to document the format of the data stored in
> ZK.
> > > > > 7. Earlier, there was a discussion on whether the tokens should be
> > > > > propagated through ZK like config/acl/quota, or through the
> > controller.
> > > > > Currently, the controller is only designed for propagating topic
> > > > metadata,
> > > > > but not other data.
> > > > > 8. Should we use SCRAM to send the token instead of DIGEST-MD5
> since
> > > it's
> > > > > deprecated?
> > > > >
> > > > > Also, the images in the wiki seem broken.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <gw...@confluent.io>
> > > > wrote:
> > > > >
> > > > >> From what I can see, remaining questions are:
> > > > >>
> > > > >> 1. Who / how are tokens renewed? By original requester only? or
> > using
> > > > >> Kerberos auth only?
> > > > >> 2. Are tokens stored on each broker or in ZK?
> > > > >> 3. How are tokens invalidated / expired?
> > > > >> 4. Which encryption algorithm is used?
> > > > >> 5. What is the impersonation proposal (it wasn't in the KIP but
> was
> > > > >> discussed in this thread)?
> > > > >> 6. Do we need new ACLs, if so - for what actions?
> > > > >>
> > > > >> Gwen
> > > > >>
> > > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <ka...@harsha.io> wrote:
> > > > >> > Jun & Ismael,
> > > > >> >                          Unfortunately I couldn't attend the KIP
> > > > meeting
> > > > >> >                          when delegation tokens discussed.
> > > Appreciate
> > > > if
> > > > >> >                          you can update the thread if you have
> any
> > > > >> >                          further questions.
> > > > >> > Thanks,
> > > > >> > Harsha
> > > > >> >
> > > > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
> > > > >> >> It seems that the links to images in the KIP are broken.
> > > > >> >>
> > > > >> >> Liquan
> > > > >> >>
> > > > >> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
> > > > >> >> brahmbhatt.parth@gmail.com> wrote:
> > > > >> >>
> > > > >> >> > 110. What does getDelegationTokenAs mean?
> > > > >> >> > In the current proposal we only allow a user to get
> delegation
> > > > token
> > > > >> for
> > > > >> >> > the identity that it authenticated as using another
> mechanism,
> > > i.e.
> > > > >> A user
> > > > >> >> > that authenticate using a keytab for principal
> > user1@EXAMPLE.COM
> > > > >> will get
> > > > >> >> > delegation tokens for that user only. In future I think we
> will
> > > > have
> > > > >> to
> > > > >> >> > extend support such that we allow some set of users (
> > > > >> >> > kafka-rest-user@EXAMPLE.COM, storm-nimbus@EXAMPLE.COM) to
> > > acquire
> > > > >> >> > delegation tokens on behalf of other users whose identity
> they
> > > have
> > > > >> >> > verified independently.  Kafka brokers will have ACLs to
> > control
> > > > >> which
> > > > >> >> > users are allowed to impersonate other users and get tokens
> on
> > > > >> behalf of
> > > > >> >> > them. Overall Impersonation is a whole different problem in
> my
> > > > >> opinion and
> > > > >> >> > I think we can tackle it in separate KIP.
> > > > >> >> >
> > > > >> >> > 111. What's the typical rate of getting and renewing
> delegation
> > > > >> tokens?
> > > > >> >> > Typically this should be very very low, 1 request per minute
> > is a
> > > > >> >> > relatively high estimate. However it depends on the token
> > > > >> expiration. I am
> > > > >> >> > less worried about the extra load it puts on controller vs
> the
> > > > added
> > > > >> >> > complexity and the value it offers.
> > > > >> >> >
> > > > >> >> > Thanks
> > > > >> >> > Parth
> > > > >> >> >
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <
> > ismael@juma.me.uk>
> > > > >> wrote:
> > > > >> >> >
> > > > >> >> > > Thanks Rajini. It would probably require a separate KIP as
> it
> > > > will
> > > > >> >> > > introduce user visible changes. We could also update KIP-48
> > to
> > > > >> have this
> > > > >> >> > > information, but it seems cleaner to do it separately. We
> can
> > > > >> discuss
> > > > >> >> > that
> > > > >> >> > > in the KIP call today.
> > > > >> >> > >
> > > > >> >> > > Ismael
> > > > >> >> > >
> > > > >> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <
> > > > >> >> > > rajinisivaram@googlemail.com> wrote:
> > > > >> >> > >
> > > > >> >> > > > Ismael,
> > > > >> >> > > >
> > > > >> >> > > > I have created a JIRA (
> > > > >> >> > https://issues.apache.org/jira/browse/KAFKA-3751)
> > > > >> >> > > > for adding SCRAM as a SASL mechanism. Would that need
> > another
> > > > >> KIP? If
> > > > >> >> > > > KIP-48 will use this mechanism, can this just be a JIRA
> > that
> > > > gets
> > > > >> >> > > reviewed
> > > > >> >> > > > when the PR is ready?
> > > > >> >> > > >
> > > > >> >> > > > Thank you,
> > > > >> >> > > >
> > > > >> >> > > > Rajini
> > > > >> >> > > >
> > > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <
> > > > ismael@juma.me.uk>
> > > > >> >> > wrote:
> > > > >> >> > > >
> > > > >> >> > > > > Thanks Rajini, SCRAM seems like a good candidate.
> > > > >> >> > > > >
> > > > >> >> > > > > Gwen had independently mentioned this as a SASL
> mechanism
> > > > that
> > > > >> might
> > > > >> >> > be
> > > > >> >> > > > > useful for Kafka and I have been meaning to check it in
> > > more
> > > > >> detail.
> > > > >> >> > > Good
> > > > >> >> > > > > to know that you are willing to contribute an
> > > implementation.
> > > > >> Maybe
> > > > >> >> > we
> > > > >> >> > > > > should file a separate JIRA for this?
> > > > >> >> > > > >
> > > > >> >> > > > > Ismael
> > > > >> >> > > > >
> > > > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram <
> > > > >> >> > > > > rajinisivaram@googlemail.com> wrote:
> > > > >> >> > > > >
> > > > >> >> > > > > > SCRAM (Salted Challenge Response Authentication
> > > Mechanism)
> > > > >> is a
> > > > >> >> > > better
> > > > >> >> > > > > > mechanism than Digest-MD5. Java doesn't come with a
> > > > built-in
> > > > >> SCRAM
> > > > >> >> > > > > > SaslServer or SaslClient, but I will be happy to add
> > > > support
> > > > >> in
> > > > >> >> > Kafka
> > > > >> >> > > > > since
> > > > >> >> > > > > > it would be a useful mechanism to support anyway.
> > > > >> >> > > > > > https://tools.ietf.org/html/rfc7677 describes the
> > > protocol
> > > > >> for
> > > > >> >> > > > > > SCRAM-SHA-256.
> > > > >> >> > > > > >
> > > > >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <
> > > jun@confluent.io
> > > > >
> > > > >> wrote:
> > > > >> >> > > > > >
> > > > >> >> > > > > > > Parth,
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > Thanks for the explanation. A couple of more
> > questions.
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > 110. What does getDelegationTokenAs mean?
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > 111. What's the typical rate of getting and
> renewing
> > > > >> delegation
> > > > >> >> > > > tokens?
> > > > >> >> > > > > > > That may have an impact on whether they should be
> > > > directed
> > > > >> to the
> > > > >> >> > > > > > > controller.
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > Jun
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt <
> > > > >> >> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > > Hi Jun,
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > Thanks for reviewing.
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > * We could add a Cluster action to add acls on
> who
> > > can
> > > > >> request
> > > > >> >> > > > > > delegation
> > > > >> >> > > > > > > > tokens. I don't see the use case for that yet but
> > > down
> > > > >> the line
> > > > >> >> > > > when
> > > > >> >> > > > > we
> > > > >> >> > > > > > > > start supporting getDelegationTokenAs it will be
> > > > >> necessary.
> > > > >> >> > > > > > > > * Yes we recommend tokens to be only
> > used/distributed
> > > > >> over
> > > > >> >> > secure
> > > > >> >> > > > > > > channels.
> > > > >> >> > > > > > > > * Depending on what design we end up choosing
> > > > >> Invalidation will
> > > > >> >> > > be
> > > > >> >> > > > > > > > responsibility of every broker or controller.
> > > > >> >> > > > > > > > * I am not sure if I documented somewhere that
> > > > >> invalidation
> > > > >> >> > will
> > > > >> >> > > > > > directly
> > > > >> >> > > > > > > > go through zookeeper but that is not the intent.
> > > > >> Invalidation
> > > > >> >> > > will
> > > > >> >> > > > > > either
> > > > >> >> > > > > > > > be request based or due to expiration. No direct
> > > > >> zookeeper
> > > > >> >> > > > > interaction
> > > > >> >> > > > > > > from
> > > > >> >> > > > > > > > any client.
> > > > >> >> > > > > > > > * "Broker also stores the DelegationToken without
> > the
> > > > >> hmac in
> > > > >> >> > the
> > > > >> >> > > > > > > > zookeeper." : Sorry about the confusion. The sole
> > > > >> purpose of
> > > > >> >> > > > > zookeeper
> > > > >> >> > > > > > in
> > > > >> >> > > > > > > > this design is as distribution channel for tokens
> > > > >> between all
> > > > >> >> > > > brokers
> > > > >> >> > > > > > > and a
> > > > >> >> > > > > > > > layer that ensures only tokens that were
> generated
> > by
> > > > >> making a
> > > > >> >> > > > > request
> > > > >> >> > > > > > > to a
> > > > >> >> > > > > > > > broker will be accepted (more on this in second
> > > > >> paragraph). The
> > > > >> >> > > > token
> > > > >> >> > > > > > > > consists of few elements (owner, renewer, uuid ,
> > > > >> expiration,
> > > > >> >> > > hmac)
> > > > >> >> > > > ,
> > > > >> >> > > > > > one
> > > > >> >> > > > > > > of
> > > > >> >> > > > > > > > which is the finally generated hmac but hmac it
> > self
> > > is
> > > > >> >> > derivable
> > > > >> >> > > > if
> > > > >> >> > > > > > you
> > > > >> >> > > > > > > > have all the other elements of the token + secret
> > key
> > > > to
> > > > >> >> > generate
> > > > >> >> > > > > hmac.
> > > > >> >> > > > > > > > Given zookeeper does not provide SSL support we
> do
> > > not
> > > > >> want the
> > > > >> >> > > > > entire
> > > > >> >> > > > > > > > token to be wire transferred to zookeeper as that
> > > will
> > > > >> be an
> > > > >> >> > > > insecure
> > > > >> >> > > > > > > wire
> > > > >> >> > > > > > > > transfer. Instead we only store all the other
> > > elements
> > > > >> of a
> > > > >> >> > > > > delegation
> > > > >> >> > > > > > > > tokens. Brokers can read these elements and
> because
> > > > they
> > > > >> also
> > > > >> >> > > have
> > > > >> >> > > > > > access
> > > > >> >> > > > > > > > to secret key they will be able to generate hmac
> on
> > > > >> their end.
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > One of the alternative proposed is to avoid
> > zookeeper
> > > > >> >> > > altogether. A
> > > > >> >> > > > > > > Client
> > > > >> >> > > > > > > > will call broker with required information
> (owner,
> > > > >> renwer,
> > > > >> >> > > > > expiration)
> > > > >> >> > > > > > > and
> > > > >> >> > > > > > > > get back (signed hmac, uuid). Broker won't store
> > this
> > > > in
> > > > >> >> > > zookeeper.
> > > > >> >> > > > > > From
> > > > >> >> > > > > > > > this point a client can contact any broker with
> all
> > > the
> > > > >> >> > > delegation
> > > > >> >> > > > > > token
> > > > >> >> > > > > > > > info (owner, rewner, expiration, hmac, uuid) the
> > > borker
> > > > >> will
> > > > >> >> > > > > regenerate
> > > > >> >> > > > > > > the
> > > > >> >> > > > > > > > hmac and as long as it matches with hmac
> presented
> > by
> > > > >> client ,
> > > > >> >> > > > broker
> > > > >> >> > > > > > > will
> > > > >> >> > > > > > > > allow the request to authenticate.  Only problem
> > with
> > > > >> this
> > > > >> >> > > approach
> > > > >> >> > > > > is
> > > > >> >> > > > > > if
> > > > >> >> > > > > > > > the secret key is compromised any client can now
> > > > generate
> > > > >> >> > random
> > > > >> >> > > > > tokens
> > > > >> >> > > > > > > and
> > > > >> >> > > > > > > > they will still be able to authenticate as any
> user
> > > > they
> > > > >> like.
> > > > >> >> > > with
> > > > >> >> > > > > > > > zookeeper we guarantee that only tokens acquired
> > via
> > > a
> > > > >> broker
> > > > >> >> > > > (using
> > > > >> >> > > > > > some
> > > > >> >> > > > > > > > auth scheme other than delegation token) will be
> > > > >> accepted. We
> > > > >> >> > > need
> > > > >> >> > > > to
> > > > >> >> > > > > > > > discuss which proposal makes more sense and we
> can
> > go
> > > > >> over it
> > > > >> >> > in
> > > > >> >> > > > > > > tomorrow's
> > > > >> >> > > > > > > > meeting.
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > Also, can you forward the invite to me?
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > Thanks
> > > > >> >> > > > > > > > Parth
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <
> > > > >> jun@confluent.io>
> > > > >> >> > > > wrote:
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > > Thanks for the KIP. A few comments.
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > 100. This potentially can be useful for Kafka
> > > Connect
> > > > >> and
> > > > >> >> > Kafka
> > > > >> >> > > > > rest
> > > > >> >> > > > > > > > proxy
> > > > >> >> > > > > > > > > where a worker agent will need to run a task on
> > > > behalf
> > > > >> of a
> > > > >> >> > > > client.
> > > > >> >> > > > > > We
> > > > >> >> > > > > > > > will
> > > > >> >> > > > > > > > > likely need to change how those services use
> > Kafka
> > > > >> clients
> > > > >> >> > > > > > > > > (producer/consumer). Instead of a shared client
> > per
> > > > >> worker,
> > > > >> >> > we
> > > > >> >> > > > will
> > > > >> >> > > > > > > need
> > > > >> >> > > > > > > > a
> > > > >> >> > > > > > > > > client per user task since the authentication
> > > happens
> > > > >> at the
> > > > >> >> > > > > > connection
> > > > >> >> > > > > > > > > level. For Kafka Connect, the renewer will be
> the
> > > > >> workers.
> > > > >> >> > So,
> > > > >> >> > > we
> > > > >> >> > > > > > > > probably
> > > > >> >> > > > > > > > > need to allow multiple renewers. For Kafka rest
> > > > proxy,
> > > > >> the
> > > > >> >> > > > renewer
> > > > >> >> > > > > > can
> > > > >> >> > > > > > > > > probably just be the creator of the token.
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > 101. Do we need new acl on who can request
> > > delegation
> > > > >> tokens?
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > 102. Do we recommend people to send delegation
> > > tokens
> > > > >> in an
> > > > >> >> > > > > encrypted
> > > > >> >> > > > > > > > > channel?
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > 103. Who is responsible for expiring tokens,
> > every
> > > > >> broker?
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > 104. For invalidating tokens, would it be
> better
> > to
> > > > do
> > > > >> it in
> > > > >> >> > a
> > > > >> >> > > > > > request
> > > > >> >> > > > > > > > > instead of going to ZK directly?
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > 105. The terminology of client in the wiki
> > > sometimes
> > > > >> refers
> > > > >> >> > to
> > > > >> >> > > > the
> > > > >> >> > > > > > end
> > > > >> >> > > > > > > > > client and some other times refers to the
> client
> > > > using
> > > > >> the
> > > > >> >> > > > > delegation
> > > > >> >> > > > > > > > > tokens. It would be useful to distinguish
> between
> > > the
> > > > >> two.
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > 106. Could you explain the sentence "Broker
> also
> > > > >> stores the
> > > > >> >> > > > > > > > DelegationToken
> > > > >> >> > > > > > > > > without the hmac in the zookeeper." a bit
> more? I
> > > > >> thought the
> > > > >> >> > > > > > > delegation
> > > > >> >> > > > > > > > > token is the hmac.
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > Thanks,
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > Jun
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <
> > > > >> jun@confluent.io>
> > > > >> >> > > > wrote:
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > > Hi, Harsha,
> > > > >> >> > > > > > > > > >
> > > > >> >> > > > > > > > > > Just sent out a KIP meeting invite. We can
> > > discuss
> > > > >> this in
> > > > >> >> > > the
> > > > >> >> > > > > > > meeting
> > > > >> >> > > > > > > > > > tomorrow.
> > > > >> >> > > > > > > > > >
> > > > >> >> > > > > > > > > > Thanks,
> > > > >> >> > > > > > > > > >
> > > > >> >> > > > > > > > > > Jun
> > > > >> >> > > > > > > > > >
> > > > >> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <
> > > > >> kafka@harsha.io>
> > > > >> >> > > > wrote:
> > > > >> >> > > > > > > > > >
> > > > >> >> > > > > > > > > >> Hi All,
> > > > >> >> > > > > > > > > >>            Can we have a KIP meeting around
> > > this.
> > > > >> The KIP
> > > > >> >> > is
> > > > >> >> > > > up
> > > > >> >> > > > > > for
> > > > >> >> > > > > > > > > >>            sometime and if there are any
> > > questions
> > > > >> lets
> > > > >> >> > > > quickly
> > > > >> >> > > > > > hash
> > > > >> >> > > > > > > > out
> > > > >> >> > > > > > > > > >>            details.
> > > > >> >> > > > > > > > > >>
> > > > >> >> > > > > > > > > >> Thanks,
> > > > >> >> > > > > > > > > >> Harsha
> > > > >> >> > > > > > > > > >>
> > > > >> >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40 AM, parth
> > > > brahmbhatt
> > > > >> wrote:
> > > > >> >> > > > > > > > > >> > That is what the hadoop echo system uses
> so
> > no
> > > > >> good
> > > > >> >> > reason
> > > > >> >> > > > > > really.
> > > > >> >> > > > > > > > We
> > > > >> >> > > > > > > > > >> > could
> > > > >> >> > > > > > > > > >> > change it to whatever is the newest
> > > recommended
> > > > >> standard
> > > > >> >> > > is.
> > > > >> >> > > > > > > > > >> >
> > > > >> >> > > > > > > > > >> > Thanks
> > > > >> >> > > > > > > > > >> > Parth
> > > > >> >> > > > > > > > > >> >
> > > > >> >> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael
> > Juma <
> > > > >> >> > > > > ismael@juma.me.uk
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > > > wrote:
> > > > >> >> > > > > > > > > >> >
> > > > >> >> > > > > > > > > >> > > Hi Parth,
> > > > >> >> > > > > > > > > >> > >
> > > > >> >> > > > > > > > > >> > > Thanks for the KIP. I only started
> > reviewing
> > > > >> this and
> > > > >> >> > > may
> > > > >> >> > > > > have
> > > > >> >> > > > > > > > > >> additional
> > > > >> >> > > > > > > > > >> > > questions later. The immediate question
> > that
> > > > >> came to
> > > > >> >> > > mind
> > > > >> >> > > > is
> > > > >> >> > > > > > our
> > > > >> >> > > > > > > > > >> choice of
> > > > >> >> > > > > > > > > >> > > "DIGEST-MD5" even though it's marked as
> > > > >> OBSOLETE in
> > > > >> >> > the
> > > > >> >> > > > IANA
> > > > >> >> > > > > > > > > Registry
> > > > >> >> > > > > > > > > >> of
> > > > >> >> > > > > > > > > >> > > SASL mechanisms and the original RFC
> > (2831)
> > > > has
> > > > >> been
> > > > >> >> > > moved
> > > > >> >> > > > > to
> > > > >> >> > > > > > > > > Historic
> > > > >> >> > > > > > > > > >> > > status:
> > > > >> >> > > > > > > > > >> > >
> > > > >> >> > > > > > > > > >> > > https://tools.ietf.org/html/rfc6331
> > > > >> >> > > > > > > > > >> > >
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > >
> > > > >> >> > >
> > > > >>
> > http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> > > > >> >> > > > > > > > > >> > >
> > > > >> >> > > > > > > > > >> > > What is the reasoning behind that
> choice?
> > > > >> >> > > > > > > > > >> > >
> > > > >> >> > > > > > > > > >> > > Thanks,
> > > > >> >> > > > > > > > > >> > > Ismael
> > > > >> >> > > > > > > > > >> > >
> > > > >> >> > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen
> > > > Shapira <
> > > > >> >> > > > > > > gwen@confluent.io
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > >> wrote:
> > > > >> >> > > > > > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > Also comments inline :)
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > > > > * I want to emphasize that even
> though
> > > > >> delegation
> > > > >> >> > > > tokens
> > > > >> >> > > > > > > are a
> > > > >> >> > > > > > > > > >> Hadoop
> > > > >> >> > > > > > > > > >> > > > > innovation, I feel very strongly
> about
> > > not
> > > > >> adding
> > > > >> >> > > > > > dependency
> > > > >> >> > > > > > > > on
> > > > >> >> > > > > > > > > >> Hadoop
> > > > >> >> > > > > > > > > >> > > > > when implementing delegation tokens
> > for
> > > > >> Kafka. The
> > > > >> >> > > KIP
> > > > >> >> > > > > > > doesn't
> > > > >> >> > > > > > > > > >> imply
> > > > >> >> > > > > > > > > >> > > > > such dependency, but if you can
> > > clarify...
> > > > >> >> > > > > > > > > >> > > > >
> > > > >> >> > > > > > > > > >> > > > >
> > > > >> >> > > > > > > > > >> > > > > *No hadoop dependency.*
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > > > Yay! Just add this to the KIP so no
> one
> > > will
> > > > >> read
> > > > >> >> > the
> > > > >> >> > > > KIP
> > > > >> >> > > > > > and
> > > > >> >> > > > > > > > > panic
> > > > >> >> > > > > > > > > >> > > > three weeks before the next release...
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > > > > * Can we get delegation token at any
> > > time
> > > > >> after
> > > > >> >> > > > > > > > authenticating?
> > > > >> >> > > > > > > > > >> only
> > > > >> >> > > > > > > > > >> > > > > immediately after?
> > > > >> >> > > > > > > > > >> > > > >
> > > > >> >> > > > > > > > > >> > > > >
> > > > >> >> > > > > > > > > >> > > > > *As long as you are authenticated
> you
> > > can
> > > > >> get
> > > > >> >> > > > delegation
> > > > >> >> > > > > > > > tokens.
> > > > >> >> > > > > > > > > >> We
> > > > >> >> > > > > > > > > >> > > need
> > > > >> >> > > > > > > > > >> > > > to
> > > > >> >> > > > > > > > > >> > > > > discuss if a client authenticated
> > using
> > > > >> delegation
> > > > >> >> > > > > token,
> > > > >> >> > > > > > > can
> > > > >> >> > > > > > > > > also
> > > > >> >> > > > > > > > > >> > > > acquire
> > > > >> >> > > > > > > > > >> > > > > delegation token again or not. Also
> > > there
> > > > >> is the
> > > > >> >> > > > > question
> > > > >> >> > > > > > of
> > > > >> >> > > > > > > > do
> > > > >> >> > > > > > > > > we
> > > > >> >> > > > > > > > > >> > > allow
> > > > >> >> > > > > > > > > >> > > > > anyone to acquire delegation token
> or
> > we
> > > > >> want
> > > > >> >> > > specific
> > > > >> >> > > > > > ACLs
> > > > >> >> > > > > > > (I
> > > > >> >> > > > > > > > > >> think
> > > > >> >> > > > > > > > > >> > > its
> > > > >> >> > > > > > > > > >> > > > an
> > > > >> >> > > > > > > > > >> > > > > overkill.)*
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > > > I agree that ACLs is an overkill.
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > > > I think we are debating two options:
> > > Either
> > > > >> require
> > > > >> >> > > > > Kerberos
> > > > >> >> > > > > > > > auth
> > > > >> >> > > > > > > > > >> for
> > > > >> >> > > > > > > > > >> > > > renewal or require non-owners to
> renew.
> > > > >> >> > > > > > > > > >> > > > I *think* the latter is simpler (it
> > > > basically
> > > > >> >> > require
> > > > >> >> > > a
> > > > >> >> > > > > "job
> > > > >> >> > > > > > > > > master"
> > > > >> >> > > > > > > > > >> > > > to take responsibility for the
> renewal,
> > it
> > > > >> will have
> > > > >> >> > > its
> > > > >> >> > > > > own
> > > > >> >> > > > > > > > > >> identity
> > > > >> >> > > > > > > > > >> > > > anyway and I think this is the correct
> > > > design
> > > > >> >> > pattern
> > > > >> >> > > > > > anyway.
> > > > >> >> > > > > > > > For
> > > > >> >> > > > > > > > > >> > > > storm, I'd expect Nimbus to coordinate
> > > > >> renewals?),
> > > > >> >> > but
> > > > >> >> > > > it
> > > > >> >> > > > > is
> > > > >> >> > > > > > > > hard
> > > > >> >> > > > > > > > > to
> > > > >> >> > > > > > > > > >> > > > debate simplicity without looking at
> the
> > > > code
> > > > >> >> > changes
> > > > >> >> > > > > > > required.
> > > > >> >> > > > > > > > If
> > > > >> >> > > > > > > > > >> you
> > > > >> >> > > > > > > > > >> > > > have a draft of how the "require
> > Kerberos"
> > > > >> will look
> > > > >> >> > > in
> > > > >> >> > > > > > Kafka
> > > > >> >> > > > > > > > > code,
> > > > >> >> > > > > > > > > >> > > > I'll be happy to take a look.
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > > > > * My understanding is that tokens
> will
> > > > >> propagate
> > > > >> >> > via
> > > > >> >> > > > ZK
> > > > >> >> > > > > > but
> > > > >> >> > > > > > > > > >> without
> > > > >> >> > > > > > > > > >> > > > > additional changes to UpdateMetadata
> > > > >> protocol,
> > > > >> >> > > > correct?
> > > > >> >> > > > > > > > Clients
> > > > >> >> > > > > > > > > >> > > > > currently don't retry on SASL auth
> > > failure
> > > > >> (IIRC),
> > > > >> >> > > but
> > > > >> >> > > > > > since
> > > > >> >> > > > > > > > the
> > > > >> >> > > > > > > > > >> > > > > tokens propagate between brokers
> > asynch,
> > > > we
> > > > >> will
> > > > >> >> > > need
> > > > >> >> > > > to
> > > > >> >> > > > > > > > retry a
> > > > >> >> > > > > > > > > >> bit
> > > > >> >> > > > > > > > > >> > > > > to avoid clients failing auth due to
> > > > timing
> > > > >> >> > issues.
> > > > >> >> > > > > > > > > >> > > > >
> > > > >> >> > > > > > > > > >> > > > > *I am considering 2 alternatives
> right
> > > > now.
> > > > >> The
> > > > >> >> > > > current
> > > > >> >> > > > > > > > > documented
> > > > >> >> > > > > > > > > >> > > > approach
> > > > >> >> > > > > > > > > >> > > > > is zookeeper based and it does not
> > > require
> > > > >> any
> > > > >> >> > > changes
> > > > >> >> > > > > to
> > > > >> >> > > > > > > > > >> > > UpdateMetadata
> > > > >> >> > > > > > > > > >> > > > > protocol. An alternative approach
> can
> > > > remove
> > > > >> >> > > zookeeper
> > > > >> >> > > > > > > > > dependency
> > > > >> >> > > > > > > > > >> as
> > > > >> >> > > > > > > > > >> > > well
> > > > >> >> > > > > > > > > >> > > > > but we can discuss that in KIP
> > > discussion
> > > > >> call.*
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > > > Oooh! Sounds interesting. Do you want
> to
> > > > ping
> > > > >> Jun to
> > > > >> >> > > > > > arrange a
> > > > >> >> > > > > > > > > call?
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > > > > * I liked Ashish's suggestion of
> > having
> > > > >> just the
> > > > >> >> > > > > > controller
> > > > >> >> > > > > > > > > issue
> > > > >> >> > > > > > > > > >> the
> > > > >> >> > > > > > > > > >> > > > > delegation tokens, to avoid syncing
> a
> > > > shared
> > > > >> >> > secret.
> > > > >> >> > > > Not
> > > > >> >> > > > > > > sure
> > > > >> >> > > > > > > > if
> > > > >> >> > > > > > > > > >> we
> > > > >> >> > > > > > > > > >> > > > > want to continue the discussion here
> > or
> > > on
> > > > >> the
> > > > >> >> > > wiki. I
> > > > >> >> > > > > > think
> > > > >> >> > > > > > > > > that
> > > > >> >> > > > > > > > > >> we
> > > > >> >> > > > > > > > > >> > > > > can decouple the problem of "token
> > > > >> distribution"
> > > > >> >> > > from
> > > > >> >> > > > > > > "shared
> > > > >> >> > > > > > > > > >> secret
> > > > >> >> > > > > > > > > >> > > > > distribution" and use the controller
> > as
> > > > the
> > > > >> only
> > > > >> >> > > token
> > > > >> >> > > > > > > > generator
> > > > >> >> > > > > > > > > >> to
> > > > >> >> > > > > > > > > >> > > > > solve the second issue, while still
> > > using
> > > > >> ZK async
> > > > >> >> > > to
> > > > >> >> > > > > > > > distribute
> > > > >> >> > > > > > > > > >> > > > > tokens.
> > > > >> >> > > > > > > > > >> > > > >
> > > > >> >> > > > > > > > > >> > > > >
> > > > >> >> > > > > > > > > >> > > > > *As mentioned in the previous Email
> I
> > am
> > > > >> fine with
> > > > >> >> > > > that
> > > > >> >> > > > > > > > approach
> > > > >> >> > > > > > > > > >> as
> > > > >> >> > > > > > > > > >> > > long
> > > > >> >> > > > > > > > > >> > > > as
> > > > >> >> > > > > > > > > >> > > > > we agree that the extra complexity
> of
> > > > >> >> > > adding/updating
> > > > >> >> > > > > APIS
> > > > >> >> > > > > > > > adds
> > > > >> >> > > > > > > > > >> enough
> > > > >> >> > > > > > > > > >> > > > > value. The advantage with the
> > controller
> > > > >> approach
> > > > >> >> > is
> > > > >> >> > > > > > secret
> > > > >> >> > > > > > > > > >> rotation
> > > > >> >> > > > > > > > > >> > > can
> > > > >> >> > > > > > > > > >> > > > be
> > > > >> >> > > > > > > > > >> > > > > automated,frequent and would not
> > require
> > > > >> >> > > deployment. *
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > > > Can you detail the extra complexity
> (or
> > > > point
> > > > >> me to
> > > > >> >> > > the
> > > > >> >> > > > > > email
> > > > >> >> > > > > > > I
> > > > >> >> > > > > > > > > >> > > > missed?) - which Apis are required?
> > > > >> >> > > > > > > > > >> > > > As far as I can tell, clients can
> > already
> > > > >> find the
> > > > >> >> > > > > > controller
> > > > >> >> > > > > > > > from
> > > > >> >> > > > > > > > > >> > > > metadata. I'm a bit more concerned
> about
> > > > >> controller
> > > > >> >> > > > load.
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > > > >
> > > > >> >> > > > > > > > > >> > > > > * While I like the idea of forcing
> > > > kerberos
> > > > >> auth
> > > > >> >> > for
> > > > >> >> > > > > > > renwal, I
> > > > >> >> > > > > > > > > >> think
> > > > >> >> > > > > > > > > >> > > > > it mixes the transport layer the the
> > > > request
> > > > >> >> > content
> > > > >> >> > > > in
> > > > >> >> > > > > a
> > > > >> >> > > > > > > > pretty
> > > > >> >> > > > > > > > > >> ugly
> > > > >> >> > > > > > > > > >> > > > > way. Perhaps limiting renewer to
> > > non-owner
> > > > >> is
> > > > >> >> > > better.
> > > > >> >> > > > > > > > > >> > > > >
> > > > >> >> > > > > > > > > >> > > > > *I feel this is a necessary evil.
> > While
> > > > >> this will
> > > > >> >> > > make
> > > > >> >> > > > > the
> > > > >> >> > > > > > > > kafka
> > > > >> >> > > > > > > > > >> code
> > > > >> >> > > > > > > > > >> > > > > pretty straight forward , forcing
> > > renewer
> > > > >> to
> > > > >> >> > > > non-owner
> > > > >> >> > > > > > > pushes
> > > > >> >> > > > > > > > > >> the code
> > > > >> >> > > > > > > > > >> > > > > ugliness to client and makes it even
> > > > harder
> > > > >> to
> > > > >> >> > > > > > integrate.  *
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > > > As mentioned before, I don't think the
> > > > >> "renewal by
> > > > >> >> > > > other"
> > > > >> >> > > > > > > > approach
> > > > >> >> > > > > > > > > >> is
> > > > >> >> > > > > > > > > >> > > > that ugly for the clients we expect to
> > use
> > > > >> >> > delegation
> > > > >> >> > > > > tokens
> > > > >> >> > > > > > > > since
> > > > >> >> > > > > > > > > >> > > > they will have an app-master of some
> > sort
> > > > who
> > > > >> >> > > requested
> > > > >> >> > > > > the
> > > > >> >> > > > > > > > token
> > > > >> >> > > > > > > > > to
> > > > >> >> > > > > > > > > >> > > > begin with.
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > > > >
> > > > >> >> > > > > > > > > >> > > > > The response for my question on how
> > > > multiple
> > > > >> >> > > > identities
> > > > >> >> > > > > > will
> > > > >> >> > > > > > > > be
> > > > >> >> > > > > > > > > >> > > > > handled wasn't super clear to me -
> > > AFAIK,
> > > > >> we don't
> > > > >> >> > > > > > > > authenticate
> > > > >> >> > > > > > > > > >> each
> > > > >> >> > > > > > > > > >> > > > > request, we authenticate
> connections.
> > > > >> >> > > > > > > > > >> > > > >
> > > > >> >> > > > > > > > > >> > > > > *We authenticate connections, and
> only
> > > > when
> > > > >> they
> > > > >> >> > are
> > > > >> >> > > > > being
> > > > >> >> > > > > > > > > >> established.
> > > > >> >> > > > > > > > > >> > > > Let
> > > > >> >> > > > > > > > > >> > > > > me try to phrase this as a question,
> > in
> > > > >> absence of
> > > > >> >> > > > > > > delegation
> > > > >> >> > > > > > > > > >> tokens if
> > > > >> >> > > > > > > > > >> > > > we
> > > > >> >> > > > > > > > > >> > > > > had to support the use case using
> user
> > > > >> TGT's how
> > > > >> >> > > would
> > > > >> >> > > > > we
> > > > >> >> > > > > > do
> > > > >> >> > > > > > > > it?
> > > > >> >> > > > > > > > > >> My
> > > > >> >> > > > > > > > > >> > > point
> > > > >> >> > > > > > > > > >> > > > > was it would be no different with
> > > > delegation
> > > > >> >> > tokens.
> > > > >> >> > > > The
> > > > >> >> > > > > > use
> > > > >> >> > > > > > > > > case
> > > > >> >> > > > > > > > > >> you
> > > > >> >> > > > > > > > > >> > > are
> > > > >> >> > > > > > > > > >> > > > > describing seems more like
> > > impersonation.*
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > > > Yeah, I thought that one of the things
> > > that
> > > > >> >> > delegation
> > > > >> >> > > > > > tokens
> > > > >> >> > > > > > > > > >> handled.
> > > > >> >> > > > > > > > > >> > > > Maybe I got it wrong :)
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > > > Thanks for the detailed answers.
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > > > Gwen
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > > > > Thanks
> > > > >> >> > > > > > > > > >> > > > > Parth
> > > > >> >> > > > > > > > > >> > > > >
> > > > >> >> > > > > > > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM,
> Gwen
> > > > >> Shapira <
> > > > >> >> > > > > > > > > gwen@confluent.io
> > > > >> >> > > > > > > > > >> >
> > > > >> >> > > > > > > > > >> > > > wrote:
> > > > >> >> > > > > > > > > >> > > > >
> > > > >> >> > > > > > > > > >> > > > >> Hi Parth and Harsha,
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > > >> Few more comments:
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > > >> * The API / RequestResponse section
> > > > >> doesn't seem
> > > > >> >> > to
> > > > >> >> > > > > have
> > > > >> >> > > > > > > good
> > > > >> >> > > > > > > > > >> > > > >> description of the changes to the
> > Kafka
> > > > >> Protocol.
> > > > >> >> > > > > Sounds
> > > > >> >> > > > > > > like
> > > > >> >> > > > > > > > > >> you are
> > > > >> >> > > > > > > > > >> > > > >> proposing new
> DelegationTokenRequest
> > > and
> > > > >> >> > > > > > RenewTokenRequest
> > > > >> >> > > > > > > > (and
> > > > >> >> > > > > > > > > >> > > > >> matching responses), without
> > detailing
> > > > the
> > > > >> >> > contents
> > > > >> >> > > > of
> > > > >> >> > > > > > the
> > > > >> >> > > > > > > > > >> requests
> > > > >> >> > > > > > > > > >> > > > >> and responses? Or rather, you show
> > the
> > > > >> class
> > > > >> >> > > > interface,
> > > > >> >> > > > > > but
> > > > >> >> > > > > > > > not
> > > > >> >> > > > > > > > > >> the
> > > > >> >> > > > > > > > > >> > > > >> underlying protocol. This could be
> > seen
> > > > as
> > > > >> an
> > > > >> >> > > > > > > implementation
> > > > >> >> > > > > > > > > >> detail,
> > > > >> >> > > > > > > > > >> > > > >> but since the binary protocol is
> what
> > > we
> > > > >> provide
> > > > >> >> > to
> > > > >> >> > > > > > > non-Java
> > > > >> >> > > > > > > > > >> clients,
> > > > >> >> > > > > > > > > >> > > > >> we need to show the changes there.
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > > >> * getDelegationToken sounds like
> > > > >> >> > > > > > > > delegationTokenRequestHandler?
> > > > >> >> > > > > > > > > >> Is it
> > > > >> >> > > > > > > > > >> > > > >> planned to be part of KafkaApi? or
> > > > Client?
> > > > >> Its
> > > > >> >> > > > > unclear...
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > > >> * I want to emphasize that even
> > though
> > > > >> delegation
> > > > >> >> > > > > tokens
> > > > >> >> > > > > > > are
> > > > >> >> > > > > > > > a
> > > > >> >> > > > > > > > > >> Hadoop
> > > > >> >> > > > > > > > > >> > > > >> innovation, I feel very strongly
> > about
> > > > not
> > > > >> adding
> > > > >> >> > > > > > > dependency
> > > > >> >> > > > > > > > on
> > > > >> >> > > > > > > > > >> Hadoop
> > > > >> >> > > > > > > > > >> > > > >> when implementing delegation tokens
> > for
> > > > >> Kafka.
> > > > >> >> > The
> > > > >> >> > > > KIP
> > > > >> >> > > > > > > > doesn't
> > > > >> >> > > > > > > > > >> imply
> > > > >> >> > > > > > > > > >> > > > >> such dependency, but if you can
> > > > clarify...
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > > >> * Can we get delegation token at
> any
> > > time
> > > > >> after
> > > > >> >> > > > > > > > authenticating?
> > > > >> >> > > > > > > > > >> only
> > > > >> >> > > > > > > > > >> > > > >> immediately after?
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > > >> * My understanding is that tokens
> > will
> > > > >> propagate
> > > > >> >> > > via
> > > > >> >> > > > ZK
> > > > >> >> > > > > > but
> > > > >> >> > > > > > > > > >> without
> > > > >> >> > > > > > > > > >> > > > >> additional changes to
> UpdateMetadata
> > > > >> protocol,
> > > > >> >> > > > correct?
> > > > >> >> > > > > > > > Clients
> > > > >> >> > > > > > > > > >> > > > >> currently don't retry on SASL auth
> > > > failure
> > > > >> >> > (IIRC),
> > > > >> >> > > > but
> > > > >> >> > > > > > > since
> > > > >> >> > > > > > > > > the
> > > > >> >> > > > > > > > > >> > > > >> tokens propagate between brokers
> > > asynch,
> > > > >> we will
> > > > >> >> > > need
> > > > >> >> > > > > to
> > > > >> >> > > > > > > > retry
> > > > >> >> > > > > > > > > a
> > > > >> >> > > > > > > > > >> bit
> > > > >> >> > > > > > > > > >> > > > >> to avoid clients failing auth due
> to
> > > > timing
> > > > >> >> > issues.
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > > >> * Strongly agreeing on clients not
> > > > >> touching ZK
> > > > >> >> > > > directly
> > > > >> >> > > > > > :)
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > > >> * I liked Ashish's suggestion of
> > having
> > > > >> just the
> > > > >> >> > > > > > controller
> > > > >> >> > > > > > > > > >> issue the
> > > > >> >> > > > > > > > > >> > > > >> delegation tokens, to avoid
> syncing a
> > > > >> shared
> > > > >> >> > > secret.
> > > > >> >> > > > > Not
> > > > >> >> > > > > > > sure
> > > > >> >> > > > > > > > > if
> > > > >> >> > > > > > > > > >> we
> > > > >> >> > > > > > > > > >> > > > >> want to continue the discussion
> here
> > or
> > > > on
> > > > >> the
> > > > >> >> > > wiki.
> > > > >> >> > > > I
> > > > >> >> > > > > > > think
> > > > >> >> > > > > > > > > >> that we
> > > > >> >> > > > > > > > > >> > > > >> can decouple the problem of "token
> > > > >> distribution"
> > > > >> >> > > from
> > > > >> >> > > > > > > "shared
> > > > >> >> > > > > > > > > >> secret
> > > > >> >> > > > > > > > > >> > > > >> distribution" and use the
> controller
> > as
> > > > >> the only
> > > > >> >> > > > token
> > > > >> >> > > > > > > > > generator
> > > > >> >> > > > > > > > > >> to
> > > > >> >> > > > > > > > > >> > > > >> solve the second issue, while still
> > > using
> > > > >> ZK
> > > > >> >> > async
> > > > >> >> > > to
> > > > >> >> > > > > > > > > distribute
> > > > >> >> > > > > > > > > >> > > > >> tokens.
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > > >> * I am also uncomfortable with
> > infinite
> > > > >> lifetime
> > > > >> >> > of
> > > > >> >> > > > > > tokens
> > > > >> >> > > > > > > > (and
> > > > >> >> > > > > > > > > >> hoped
> > > > >> >> > > > > > > > > >> > > > >> to hear from others in the
> > community) -
> > > > but
> > > > >> >> > having
> > > > >> >> > > > the
> > > > >> >> > > > > > > option
> > > > >> >> > > > > > > > > and
> > > > >> >> > > > > > > > > >> > > > >> documenting it as less secure,
> allows
> > > > >> users to
> > > > >> >> > > > > configure
> > > > >> >> > > > > > > > their
> > > > >> >> > > > > > > > > >> system
> > > > >> >> > > > > > > > > >> > > > >> as the wish.
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > > >> * While I like the idea of forcing
> > > > >> kerberos auth
> > > > >> >> > > for
> > > > >> >> > > > > > > renwal,
> > > > >> >> > > > > > > > I
> > > > >> >> > > > > > > > > >> think
> > > > >> >> > > > > > > > > >> > > > >> it mixes the transport layer the
> the
> > > > >> request
> > > > >> >> > > content
> > > > >> >> > > > > in a
> > > > >> >> > > > > > > > > pretty
> > > > >> >> > > > > > > > > >> ugly
> > > > >> >> > > > > > > > > >> > > > >> way. Perhaps limiting renewer to
> > > > non-owner
> > > > >> is
> > > > >> >> > > better.
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > > >> Things I'd still like to see:
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > > >> * More detailed explanation on what
> > we
> > > > >> plan to do
> > > > >> >> > > for
> > > > >> >> > > > > the
> > > > >> >> > > > > > > > java
> > > > >> >> > > > > > > > > >> clients
> > > > >> >> > > > > > > > > >> > > > >> specifically - new configuration?
> new
> > > > APIs?
> > > > >> >> > > > > > > > > >> > > > >> The response for my question on how
> > > > >> multiple
> > > > >> >> > > > identities
> > > > >> >> > > > > > > will
> > > > >> >> > > > > > > > be
> > > > >> >> > > > > > > > > >> > > > >> handled wasn't super clear to me -
> > > AFAIK,
> > > > >> we
> > > > >> >> > don't
> > > > >> >> > > > > > > > authenticate
> > > > >> >> > > > > > > > > >> each
> > > > >> >> > > > > > > > > >> > > > >> request, we authenticate
> connections.
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > > >> * Alternatives: Delegation tokens
> are
> > > > only
> > > > >> used
> > > > >> >> > in
> > > > >> >> > > > the
> > > > >> >> > > > > > > Hadoop
> > > > >> >> > > > > > > > > >> > > > >> ecosystem. I'm wondering if there
> are
> > > > >> >> > alternatives
> > > > >> >> > > in
> > > > >> >> > > > > > other
> > > > >> >> > > > > > > > > >> ecosystems
> > > > >> >> > > > > > > > > >> > > > >> (Mesos? Tachyon? Cassandra?) and
> > > whether
> > > > >> there
> > > > >> >> > are
> > > > >> >> > > > some
> > > > >> >> > > > > > > > > >> advantages
> > > > >> >> > > > > > > > > >> > > > >> there.
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > > >> Gwen
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM,
> > > Harsha <
> > > > >> >> > > > > kafka@harsha.io
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > > > wrote:
> > > > >> >> > > > > > > > > >> > > > >> > Hi Gwen,
> > > > >> >> > > > > > > > > >> > > > >> >            Can you look at
> Parth's
> > > last
> > > > >> reply.
> > > > >> >> > > Does
> > > > >> >> > > > > it
> > > > >> >> > > > > > > > answer
> > > > >> >> > > > > > > > > >> your
> > > > >> >> > > > > > > > > >> > > > >> >            concerns.
> > > > >> >> > > > > > > > > >> > > > >> >
> > > > >> >> > > > > > > > > >> > > > >> > Thanks,
> > > > >> >> > > > > > > > > >> > > > >> > Harsha
> > > > >> >> > > > > > > > > >> > > > >> >
> > > > >> >> > > > > > > > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM,
> > > parth
> > > > >> >> > > brahmbhatt
> > > > >> >> > > > > > wrote:
> > > > >> >> > > > > > > > > >> > > > >> >> Thanks for reviewing Gwen. The
> > wiki
> > > > >> already
> > > > >> >> > has
> > > > >> >> > > > > > details
> > > > >> >> > > > > > > on
> > > > >> >> > > > > > > > > >> token
> > > > >> >> > > > > > > > > >> > > > >> >> expiration
> > > > >> >> > > > > > > > > >> > > > >> >> under token acquisition process
> > > > >> >> > > > > > > > > >> > > > >> >> <
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > >
> > > > >> >> > > > > > > > > >>
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > >
> > > > >> >> > > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> > > > >> >> > > > > > > > > >> > > > >> >.
> > > > >> >> > > > > > > > > >> > > > >> >> Current proposal is that tokens
> > will
> > > > >> expire
> > > > >> >> > > based
> > > > >> >> > > > > on a
> > > > >> >> > > > > > > > > server
> > > > >> >> > > > > > > > > >> side
> > > > >> >> > > > > > > > > >> > > > >> >> configuration (default 24 hours)
> > > > unless
> > > > >> >> > renewed.
> > > > >> >> > > > > > Renewal
> > > > >> >> > > > > > > > is
> > > > >> >> > > > > > > > > >> only
> > > > >> >> > > > > > > > > >> > > > allowed
> > > > >> >> > > > > > > > > >> > > > >> >> until the max life time of
> token.
> > > > >> >> > Alternatively
> > > > >> >> > > we
> > > > >> >> > > > > > could
> > > > >> >> > > > > > > > > also
> > > > >> >> > > > > > > > > >> make
> > > > >> >> > > > > > > > > >> > > > that
> > > > >> >> > > > > > > > > >> > > > >> >> an
> > > > >> >> > > > > > > > > >> > > > >> >> optional param and the server
> side
> > > > >> default can
> > > > >> >> > > > serve
> > > > >> >> > > > > > as
> > > > >> >> > > > > > > > the
> > > > >> >> > > > > > > > > >> upper
> > > > >> >> > > > > > > > > >> > > > bound.
> > > > >> >> > > > > > > > > >> > > > >> >>
> > > > >> >> > > > > > > > > >> > > > >> >> To your second point it will be
> > done
> > > > >> exactly
> > > > >> >> > the
> > > > >> >> > > > > same
> > > > >> >> > > > > > > way
> > > > >> >> > > > > > > > we
> > > > >> >> > > > > > > > > >> would
> > > > >> >> > > > > > > > > >> > > > >> >> support
> > > > >> >> > > > > > > > > >> > > > >> >> multiple keytabs. The calling
> > client
> > > > >> will have
> > > > >> >> > > to
> > > > >> >> > > > > put
> > > > >> >> > > > > > > the
> > > > >> >> > > > > > > > > >> tokens it
> > > > >> >> > > > > > > > > >> > > > >> wants
> > > > >> >> > > > > > > > > >> > > > >> >> to use in the subject instance
> and
> > > > call
> > > > >> >> > > > > > produce/consume
> > > > >> >> > > > > > > > > inside
> > > > >> >> > > > > > > > > >> > > > >> >> subject.doas. Each caller will
> > have
> > > to
> > > > >> keep
> > > > >> >> > > track
> > > > >> >> > > > of
> > > > >> >> > > > > > its
> > > > >> >> > > > > > > > own
> > > > >> >> > > > > > > > > >> > > > subject. I
> > > > >> >> > > > > > > > > >> > > > >> >> will have to look at the code to
> > see
> > > > if
> > > > >> we
> > > > >> >> > > support
> > > > >> >> > > > > > this
> > > > >> >> > > > > > > > > >> feature
> > > > >> >> > > > > > > > > >> > > right
> > > > >> >> > > > > > > > > >> > > > >> now
> > > > >> >> > > > > > > > > >> > > > >> >> but my understanding is
> delegation
> > > > token
> > > > >> >> > > shouldn't
> > > > >> >> > > > > > need
> > > > >> >> > > > > > > > any
> > > > >> >> > > > > > > > > >> special
> > > > >> >> > > > > > > > > >> > > > >> >> treatment as its just another
> type
> > > of
> > > > >> >> > Credential
> > > > >> >> > > > in
> > > > >> >> > > > > > the
> > > > >> >> > > > > > > > > >> subject.
> > > > >> >> > > > > > > > > >> > > > >> >>
> > > > >> >> > > > > > > > > >> > > > >> >> I would also like to know what
> is
> > > your
> > > > >> opinion
> > > > >> >> > > > about
> > > > >> >> > > > > > > > > infinite
> > > > >> >> > > > > > > > > >> > > renewal
> > > > >> >> > > > > > > > > >> > > > >> (my
> > > > >> >> > > > > > > > > >> > > > >> >> recommendation is to not support
> > > > this),
> > > > >> tokens
> > > > >> >> > > > > > renewing
> > > > >> >> > > > > > > > them
> > > > >> >> > > > > > > > > >> > > self(my
> > > > >> >> > > > > > > > > >> > > > >> >> recommendation is to not support
> > > this)
> > > > >> and
> > > > >> >> > most
> > > > >> >> > > > > > > > importantly
> > > > >> >> > > > > > > > > >> your
> > > > >> >> > > > > > > > > >> > > > choice
> > > > >> >> > > > > > > > > >> > > > >> >> between the alternatives listed
> on
> > > > this
> > > > >> thread
> > > > >> >> > > > > > > > > >> > > > >> >> <
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > >
> > > > >> >> > > > > > > > > >>
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > >
> > > > >> >> > > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >>
> > > >
> > >
> >
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> > > > >> >> > > > > > > > > >> > > > >> >
> > > > >> >> > > > > > > > > >> > > > >> >> ( I am leaning towards the
> > > > >> alternative-2 minus
> > > > >> >> > > > > > > controller
> > > > >> >> > > > > > > > > >> > > > distributing
> > > > >> >> > > > > > > > > >> > > > >> >> secret). Thanks again for
> > reviewing.
> > > > >> >> > > > > > > > > >> > > > >> >>
> > > > >> >> > > > > > > > > >> > > > >> >> Thanks
> > > > >> >> > > > > > > > > >> > > > >> >> Parth
> > > > >> >> > > > > > > > > >> > > > >> >>
> > > > >> >> > > > > > > > > >> > > > >> >>
> > > > >> >> > > > > > > > > >> > > > >> >>
> > > > >> >> > > > > > > > > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM,
> > Gwen
> > > > >> Shapira <
> > > > >> >> > > > > > > > > >> gwen@confluent.io>
> > > > >> >> > > > > > > > > >> > > > wrote:
> > > > >> >> > > > > > > > > >> > > > >> >>
> > > > >> >> > > > > > > > > >> > > > >> >> > Harsha,
> > > > >> >> > > > > > > > > >> > > > >> >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > I was thinking of the Rest
> > Proxy.
> > > I
> > > > >> didn't
> > > > >> >> > see
> > > > >> >> > > > > your
> > > > >> >> > > > > > > > design
> > > > >> >> > > > > > > > > >> yet,
> > > > >> >> > > > > > > > > >> > > > but in
> > > > >> >> > > > > > > > > >> > > > >> >> > our proxy, we have a set of
> > > > >> producers, which
> > > > >> >> > > > will
> > > > >> >> > > > > > > serve
> > > > >> >> > > > > > > > > >> multiple
> > > > >> >> > > > > > > > > >> > > > users
> > > > >> >> > > > > > > > > >> > > > >> >> > going through the proxy. Since
> > > these
> > > > >> users
> > > > >> >> > > will
> > > > >> >> > > > > have
> > > > >> >> > > > > > > > > >> different
> > > > >> >> > > > > > > > > >> > > > >> >> > privileges, they'll need to
> > > > >> authenticate
> > > > >> >> > > > > separately,
> > > > >> >> > > > > > > and
> > > > >> >> > > > > > > > > >> can't
> > > > >> >> > > > > > > > > >> > > > share a
> > > > >> >> > > > > > > > > >> > > > >> >> > token.
> > > > >> >> > > > > > > > > >> > > > >> >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > Am I missing anything?
> > > > >> >> > > > > > > > > >> > > > >> >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > Gwen
> > > > >> >> > > > > > > > > >> > > > >> >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11
> PM,
> > > > >> Harsha <
> > > > >> >> > > > > > > kafka@harsha.io
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > >> wrote:
> > > > >> >> > > > > > > > > >> > > > >> >> > > Gwen,
> > > > >> >> > > > > > > > > >> > > > >> >> > >            On your second
> > point.
> > > > >> Can you
> > > > >> >> > > > > describe
> > > > >> >> > > > > > a
> > > > >> >> > > > > > > > > >> usecase
> > > > >> >> > > > > > > > > >> > > where
> > > > >> >> > > > > > > > > >> > > > >> >> > >            mutliple clients
> > > ended
> > > > up
> > > > >> >> > > sharing a
> > > > >> >> > > > > > > > producer
> > > > >> >> > > > > > > > > >> and
> > > > >> >> > > > > > > > > >> > > even
> > > > >> >> > > > > > > > > >> > > > if
> > > > >> >> > > > > > > > > >> > > > >> they
> > > > >> >> > > > > > > > > >> > > > >> >> > >            do why can't they
> > not
> > > > use
> > > > >> >> > single
> > > > >> >> > > > > token
> > > > >> >> > > > > > > that
> > > > >> >> > > > > > > > > >> producer
> > > > >> >> > > > > > > > > >> > > > >> >> > >            captures. Why
> would
> > > we
> > > > >> need
> > > > >> >> > > > multiple
> > > > >> >> > > > > > > > clients
> > > > >> >> > > > > > > > > >> with
> > > > >> >> > > > > > > > > >> > > > >> different
> > > > >> >> > > > > > > > > >> > > > >> >> > >            tokens sharing a
> > > single
> > > > >> >> > instance
> > > > >> >> > > of
> > > > >> >> > > > > > > > producer.
> > > > >> >> > > > > > > > > >> Also
> > > > >> >> > > > > > > > > >> > > in
> > > > >> >> > > > > > > > > >> > > > >> this
> > > > >> >> > > > > > > > > >> > > > >> >> > >            case other
> clients
> > > have
> > > > >> access
> > > > >> >> > > all
> > > > >> >> > > > > the
> > > > >> >> > > > > > > > tokens
> > > > >> >> > > > > > > > > >> no?
> > > > >> >> > > > > > > > > >> > > > >> >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > > Thanks,
> > > > >> >> > > > > > > > > >> > > > >> >> > > Harsha
> > > > >> >> > > > > > > > > >> > > > >> >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > > On Tue, May 3, 2016, at
> 11:49
> > > AM,
> > > > >> Gwen
> > > > >> >> > > Shapira
> > > > >> >> > > > > > > wrote:
> > > > >> >> > > > > > > > > >> > > > >> >> > >> Sorry for the delay:
> > > > >> >> > > > > > > > > >> > > > >> >> > >>
> > > > >> >> > > > > > > > > >> > > > >> >> > >> Two questions that we
> didn't
> > > see
> > > > >> in the
> > > > >> >> > > wiki:
> > > > >> >> > > > > > > > > >> > > > >> >> > >> 1. Is there an expiration
> for
> > > > >> delegation
> > > > >> >> > > > > tokens?
> > > > >> >> > > > > > > > > >> Renewal? How
> > > > >> >> > > > > > > > > >> > > > do we
> > > > >> >> > > > > > > > > >> > > > >> >> > >> revoke them?
> > > > >> >> > > > > > > > > >> > > > >> >> > >> 2. If we want to use
> > delegation
> > > > >> tokens
> > > > >> >> > for
> > > > >> >> > > > > > "do-as"
> > > > >> >> > > > > > > > > (say,
> > > > >> >> > > > > > > > > >> > > submit
> > > > >> >> > > > > > > > > >> > > > >> Storm
> > > > >> >> > > > > > > > > >> > > > >> >> > >> job as my user), we will
> > need a
> > > > >> producer
> > > > >> >> > > for
> > > > >> >> > > > > > every
> > > > >> >> > > > > > > > job
> > > > >> >> > > > > > > > > >> (we
> > > > >> >> > > > > > > > > >> > > can't
> > > > >> >> > > > > > > > > >> > > > >> share
> > > > >> >> > > > > > > > > >> > > > >> >> > >> them between multiple jobs
> > > > running
> > > > >> on
> > > > >> >> > same
> > > > >> >> > > > > node),
> > > > >> >> > > > > > > > since
> > > > >> >> > > > > > > > > >> we
> > > > >> >> > > > > > > > > >> > > only
> > > > >> >> > > > > > > > > >> > > > >> >> > >> authenticate when
> connecting.
> > > Is
> > > > >> there a
> > > > >> >> > > plan
> > > > >> >> > > > > to
> > > > >> >> > > > > > > > change
> > > > >> >> > > > > > > > > >> this
> > > > >> >> > > > > > > > > >> > > for
> > > > >> >> > > > > > > > > >> > > > >> >> > >> delegation tokens, in order
> > to
> > > > >> allow
> > > > >> >> > > multiple
> > > > >> >> > > > > > users
> > > > >> >> > > > > > > > > with
> > > > >> >> > > > > > > > > >> > > > different
> > > > >> >> > > > > > > > > >> > > > >> >> > >> tokens to share a client?
> > > > >> >> > > > > > > > > >> > > > >> >> > >>
> > > > >> >> > > > > > > > > >> > > > >> >> > >> Gwen
> > > > >> >> > > > > > > > > >> > > > >> >> > >>
> > > > >> >> > > > > > > > > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12
> > AM,
> > > > >> parth
> > > > >> >> > > > > brahmbhatt
> > > > >> >> > > > > > > > > >> > > > >> >> > >> <
> brahmbhatt.parth@gmail.com>
> > > > >> wrote:
> > > > >> >> > > > > > > > > >> > > > >> >> > >> > Bumping this up one more
> > > time,
> > > > >> can
> > > > >> >> > other
> > > > >> >> > > > > > > committers
> > > > >> >> > > > > > > > > >> review?
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> > Thanks
> > > > >> >> > > > > > > > > >> > > > >> >> > >> > Parth
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at
> > 9:07
> > > > AM,
> > > > >> >> > Harsha <
> > > > >> >> > > > > > > > > >> kafka@harsha.io>
> > > > >> >> > > > > > > > > >> > > > wrote:
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> Parth,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >>           Overall
> current
> > > > >> design looks
> > > > >> >> > > > good
> > > > >> >> > > > > to
> > > > >> >> > > > > > > > me. I
> > > > >> >> > > > > > > > > >> am +1
> > > > >> >> > > > > > > > > >> > > on
> > > > >> >> > > > > > > > > >> > > > >> the
> > > > >> >> > > > > > > > > >> > > > >> >> > KIP.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> Gwen , Jun can you
> review
> > > this
> > > > >> as
> > > > >> >> > well.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> -Harsha
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at
> > > 09:57
> > > > >> AM,
> > > > >> >> > parth
> > > > >> >> > > > > > > > brahmbhatt
> > > > >> >> > > > > > > > > >> wrote:
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks for review
> > > Jitendra.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > I don't like the idea
> of
> > > > >> infinite
> > > > >> >> > > > lifetime
> > > > >> >> > > > > > > but I
> > > > >> >> > > > > > > > > >> see the
> > > > >> >> > > > > > > > > >> > > > >> Streaming
> > > > >> >> > > > > > > > > >> > > > >> >> > use
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > case. Even for
> Streaming
> > > use
> > > > >> case I
> > > > >> >> > > was
> > > > >> >> > > > > > hoping
> > > > >> >> > > > > > > > > >> there will
> > > > >> >> > > > > > > > > >> > > > be
> > > > >> >> > > > > > > > > >> > > > >> some
> > > > >> >> > > > > > > > > >> > > > >> >> > notion
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > of
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > master/driver that can
> > get
> > > > new
> > > > >> >> > > > delegation
> > > > >> >> > > > > > > tokens
> > > > >> >> > > > > > > > > at
> > > > >> >> > > > > > > > > >> fixed
> > > > >> >> > > > > > > > > >> > > > >> interval
> > > > >> >> > > > > > > > > >> > > > >> >> > and
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > distribute to workers.
> > If
> > > > >> that is
> > > > >> >> > not
> > > > >> >> > > > the
> > > > >> >> > > > > > case
> > > > >> >> > > > > > > > for
> > > > >> >> > > > > > > > > >> we can
> > > > >> >> > > > > > > > > >> > > > >> discuss
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > delegation tokens
> > renewing
> > > > >> them self
> > > > >> >> > > and
> > > > >> >> > > > > the
> > > > >> >> > > > > > > > > >> security
> > > > >> >> > > > > > > > > >> > > > >> implications
> > > > >> >> > > > > > > > > >> > > > >> >> > of the
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > same.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > I did not want clients
> > to
> > > > >> fetch
> > > > >> >> > tokens
> > > > >> >> > > > > from
> > > > >> >> > > > > > > > > >> zookeeper,
> > > > >> >> > > > > > > > > >> > > > >> overall I
> > > > >> >> > > > > > > > > >> > > > >> >> > think
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > its
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > better if clients
> don't
> > > rely
> > > > >> on our
> > > > >> >> > > > > metadata
> > > > >> >> > > > > > > > store
> > > > >> >> > > > > > > > > >> and I
> > > > >> >> > > > > > > > > >> > > > >> think we
> > > > >> >> > > > > > > > > >> > > > >> >> > are
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > moving in that
> direction
> > > > with
> > > > >> all
> > > > >> >> > the
> > > > >> >> > > > > KIP-4
> > > > >> >> > > > > > > > > >> improvements.
> > > > >> >> > > > > > > > > >> > > > I
> > > > >> >> > > > > > > > > >> > > > >> chose
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as in this
> > case
> > > > the
> > > > >> client
> > > > >> >> > > > will
> > > > >> >> > > > > > > still
> > > > >> >> > > > > > > > > >> just talk
> > > > >> >> > > > > > > > > >> > > > to
> > > > >> >> > > > > > > > > >> > > > >> >> > broker , its
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > the brokers that will
> > use
> > > > >> zookeeper
> > > > >> >> > > > which
> > > > >> >> > > > > we
> > > > >> >> > > > > > > > > >> already do
> > > > >> >> > > > > > > > > >> > > > for a
> > > > >> >> > > > > > > > > >> > > > >> lot
> > > > >> >> > > > > > > > > >> > > > >> >> > of
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > other
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > usecases + ease of
> > > > >> development + and
> > > > >> >> > > the
> > > > >> >> > > > > > > ability
> > > > >> >> > > > > > > > > so
> > > > >> >> > > > > > > > > >> > > tokens
> > > > >> >> > > > > > > > > >> > > > >> will
> > > > >> >> > > > > > > > > >> > > > >> >> > survive
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > even a rolling
> > > > restart/cluster
> > > > >> >> > > failure.
> > > > >> >> > > > > if a
> > > > >> >> > > > > > > > > >> majority
> > > > >> >> > > > > > > > > >> > > > agrees
> > > > >> >> > > > > > > > > >> > > > >> the
> > > > >> >> > > > > > > > > >> > > > >> >> > added
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > complexity to have
> > > > controller
> > > > >> >> > > forwarding
> > > > >> >> > > > > > keys
> > > > >> >> > > > > > > to
> > > > >> >> > > > > > > > > all
> > > > >> >> > > > > > > > > >> > > > broker is
> > > > >> >> > > > > > > > > >> > > > >> >> > justified
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > as
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > it provides tighter
> > > security
> > > > >> , I am
> > > > >> >> > > fine
> > > > >> >> > > > > > with
> > > > >> >> > > > > > > > that
> > > > >> >> > > > > > > > > >> option
> > > > >> >> > > > > > > > > >> > > > too.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > Given zookeeper does
> not
> > > > >> support SSL
> > > > >> >> > > we
> > > > >> >> > > > > can
> > > > >> >> > > > > > > not
> > > > >> >> > > > > > > > > >> store
> > > > >> >> > > > > > > > > >> > > > master
> > > > >> >> > > > > > > > > >> > > > >> keys
> > > > >> >> > > > > > > > > >> > > > >> >> > in
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as master
> keys
> > > > will
> > > > >> be
> > > > >> >> > > exposed
> > > > >> >> > > > > on
> > > > >> >> > > > > > > > wire.
> > > > >> >> > > > > > > > > To
> > > > >> >> > > > > > > > > >> > > > support
> > > > >> >> > > > > > > > > >> > > > >> >> > rotation
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > without affecting
> > current
> > > > >> clients is
> > > > >> >> > > > > > > something I
> > > > >> >> > > > > > > > > >> need to
> > > > >> >> > > > > > > > > >> > > > put
> > > > >> >> > > > > > > > > >> > > > >> more
> > > > >> >> > > > > > > > > >> > > > >> >> > thought
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > in. My current
> proposal
> > > > >> assumes the
> > > > >> >> > > > > rotation
> > > > >> >> > > > > > > > will
> > > > >> >> > > > > > > > > >> > > > invalidate
> > > > >> >> > > > > > > > > >> > > > >> all
> > > > >> >> > > > > > > > > >> > > > >> >> > current
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > tokens.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > I request committers
> to
> > > also
> > > > >> review
> > > > >> >> > > and
> > > > >> >> > > > > post
> > > > >> >> > > > > > > > their
> > > > >> >> > > > > > > > > >> > > comments
> > > > >> >> > > > > > > > > >> > > > >> so we
> > > > >> >> > > > > > > > > >> > > > >> >> > can
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > make
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > progress on this KIP.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > Parth
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016
> at
> > > 8:39
> > > > >> AM,
> > > > >> >> > > Ashish
> > > > >> >> > > > > > Singh
> > > > >> >> > > > > > > <
> > > > >> >> > > > > > > > > >> > > > >> asingh@cloudera.com
> > > > >> >> > > > > > > > > >> > > > >> >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > wrote:
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016
> > at
> > > > >> 11:26 AM,
> > > > >> >> > > > > Harsha <
> > > > >> >> > > > > > > > > >> > > > kafka@harsha.io>
> > > > >> >> > > > > > > > > >> > > > >> >> > wrote:
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Unifying the two
> > > > >> discussion
> > > > >> >> > > threads
> > > > >> >> > > > on
> > > > >> >> > > > > > > this
> > > > >> >> > > > > > > > > KIP.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Here is the
> response
> > > > from
> > > > >> >> > Jitendra
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > "The need for a
> > large
> > > > >> number of
> > > > >> >> > > > > clients
> > > > >> >> > > > > > > that
> > > > >> >> > > > > > > > > are
> > > > >> >> > > > > > > > > >> > > > running
> > > > >> >> > > > > > > > > >> > > > >> all
> > > > >> >> > > > > > > > > >> > > > >> >> > over the
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > cluster that
> > > > authenticate
> > > > >> with
> > > > >> >> > > Kafka
> > > > >> >> > > > > > > > brokers,
> > > > >> >> > > > > > > > > >> is very
> > > > >> >> > > > > > > > > >> > > > >> similar
> > > > >> >> > > > > > > > > >> > > > >> >> > to the
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Hadoop use case of
> > > large
> > > > >> number
> > > > >> >> > of
> > > > >> >> > > > > tasks
> > > > >> >> > > > > > > > > running
> > > > >> >> > > > > > > > > >> > > across
> > > > >> >> > > > > > > > > >> > > > >> the
> > > > >> >> > > > > > > > > >> > > > >> >> > cluster
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> that
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need
> authentication
> > to
> > > > >> Hdfs
> > > > >> >> > > > Namenode.
> > > > >> >> > > > > > > > > >> Therefore, the
> > > > >> >> > > > > > > > > >> > > > >> >> > delegation token
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > approach does seem
> > > like
> > > > a
> > > > >> good
> > > > >> >> > fit
> > > > >> >> > > > for
> > > > >> >> > > > > > > this
> > > > >> >> > > > > > > > > use
> > > > >> >> > > > > > > > > >> case
> > > > >> >> > > > > > > > > >> > > > as we
> > > > >> >> > > > > > > > > >> > > > >> >> > have seen
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> it
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > working at large
> > scale
> > > > in
> > > > >> HDFS
> > > > >> >> > and
> > > > >> >> > > > > YARN.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >   The proposed
> > design
> > > is
> > > > >> very
> > > > >> >> > much
> > > > >> >> > > > > > inline
> > > > >> >> > > > > > > > with
> > > > >> >> > > > > > > > > >> Hadoop
> > > > >> >> > > > > > > > > >> > > > >> >> > approach. A few
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >   comments:
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 1) Why do you guys
> > > want
> > > > >> to allow
> > > > >> >> > > > > > infinite
> > > > >> >> > > > > > > > > >> renewable
> > > > >> >> > > > > > > > > >> > > > >> lifetime
> > > > >> >> > > > > > > > > >> > > > >> >> > for a
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > token? HDFS
> > restricts
> > > a
> > > > >> token
> > > > >> >> > to a
> > > > >> >> > > > max
> > > > >> >> > > > > > > life
> > > > >> >> > > > > > > > > time
> > > > >> >> > > > > > > > > >> > > > (default
> > > > >> >> > > > > > > > > >> > > > >> 7
> > > > >> >> > > > > > > > > >> > > > >> >> > days).  A
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > token's
> > vulnerability
> > > is
> > > > >> >> > believed
> > > > >> >> > > to
> > > > >> >> > > > > > > > increase
> > > > >> >> > > > > > > > > >> with
> > > > >> >> > > > > > > > > >> > > > time.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > I agree that having
> > > > infinite
> > > > >> >> > > lifetime
> > > > >> >> > > > > > might
> > > > >> >> > > > > > > > not
> > > > >> >> > > > > > > > > >> be the
> > > > >> >> > > > > > > > > >> > > > best
> > > > >> >> > > > > > > > > >> > > > >> idea.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 2) As I understand
> > the
> > > > >> tokens
> > > > >> >> > are
> > > > >> >> > > > > stored
> > > > >> >> > > > > > > in
> > > > >> >> > > > > > > > > >> zookeeper
> > > > >> >> > > > > > > > > >> > > > as
> > > > >> >> > > > > > > > > >> > > > >> well,
> > > > >> >> > > > > > > > > >> > > > >> >> > and
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> can
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > be updated there.
> > This
> > > > is
> > > > >> clever
> > > > >> >> > > as
> > > > >> >> > > > it
> > > > >> >> > > > > > can
> > > > >> >> > > > > > > > > allow
> > > > >> >> > > > > > > > > >> > > > >> replacing the
> > > > >> >> > > > > > > > > >> > > > >> >> > tokens
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > once they run out
> of
> > > max
> > > > >> life
> > > > >> >> > > time,
> > > > >> >> > > > > and
> > > > >> >> > > > > > > > > clients
> > > > >> >> > > > > > > > > >> can
> > > > >> >> > > > > > > > > >> > > > >> download
> > > > >> >> > > > > > > > > >> > > > >> >> > new
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> tokens
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from zookeeper. It
> > > > >> shouldn't be
> > > > >> >> > a
> > > > >> >> > > > big
> > > > >> >> > > > > > load
> > > > >> >> > > > > > > > on
> > > > >> >> > > > > > > > > >> > > zookeeper
> > > > >> >> > > > > > > > > >> > > > >> as a
> > > > >> >> > > > > > > > > >> > > > >> >> > client
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> will
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need to get a new
> > > token
> > > > >> once in
> > > > >> >> > > > > several
> > > > >> >> > > > > > > > days.
> > > > >> >> > > > > > > > > >> In this
> > > > >> >> > > > > > > > > >> > > > >> approach
> > > > >> >> > > > > > > > > >> > > > >> >> > you
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> don't
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need infinite
> > lifetime
> > > > on
> > > > >> the
> > > > >> >> > > token
> > > > >> >> > > > > even
> > > > >> >> > > > > > > for
> > > > >> >> > > > > > > > > >> long
> > > > >> >> > > > > > > > > >> > > > running
> > > > >> >> > > > > > > > > >> > > > >> >> > clients.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 3) The token
> > password
> > > > are
> > > > >> >> > > generated
> > > > >> >> > > > > > using
> > > > >> >> > > > > > > a
> > > > >> >> > > > > > > > > >> master
> > > > >> >> > > > > > > > > >> > > key.
> > > > >> >> > > > > > > > > >> > > > >> The
> > > > >> >> > > > > > > > > >> > > > >> >> > master
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> key
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > should also be
> > > > >> periodically
> > > > >> >> > > changed.
> > > > >> >> > > > > In
> > > > >> >> > > > > > > > > Hadoop,
> > > > >> >> > > > > > > > > >> the
> > > > >> >> > > > > > > > > >> > > > >> default
> > > > >> >> > > > > > > > > >> > > > >> >> > renewal
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > period is 1 day.?
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > IIUC, this will
> > require
> > > > >> brokers
> > > > >> >> > > > > > maintaining
> > > > >> >> > > > > > > a
> > > > >> >> > > > > > > > > >> list of X
> > > > >> >> > > > > > > > > >> > > > most
> > > > >> >> > > > > > > > > >> > > > >> >> > recent
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> master
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > keys. This list will
> > > have
> > > > >> to be
> > > > >> >> > > > > persisted
> > > > >> >> > > > > > > > > >> somewhere, as
> > > > >> >> > > > > > > > > >> > > > if a
> > > > >> >> > > > > > > > > >> > > > >> >> > broker
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> goes
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > down it will have to
> > get
> > > > >> that list
> > > > >> >> > > > again
> > > > >> >> > > > > > and
> > > > >> >> > > > > > > > > >> storing
> > > > >> >> > > > > > > > > >> > > > master
> > > > >> >> > > > > > > > > >> > > > >> keys
> > > > >> >> > > > > > > > > >> > > > >> >> > on ZK
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> is
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > not the best idea.
> > > > However,
> > > > >> if a
> > > > >> >> > > > broker
> > > > >> >> > > > > > goes
> > > > >> >> > > > > > > > > down
> > > > >> >> > > > > > > > > >> then
> > > > >> >> > > > > > > > > >> > > we
> > > > >> >> > > > > > > > > >> > > > >> have
> > > > >> >> > > > > > > > > >> > > > >> >> > much
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> bigger
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > issue to deal with
> and
> > > > >> client can
> > > > >> >> > > > always
> > > > >> >> > > > > > > > > >> > > re-authenticate
> > > > >> >> > > > > > > > > >> > > > is
> > > > >> >> > > > > > > > > >> > > > >> such
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> events.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > Did you happen to
> > take a
> > > > >> look at
> > > > >> >> > > other
> > > > >> >> > > > > > > > > >> alternatives
> > > > >> >> > > > > > > > > >> > > this
> > > > >> >> > > > > > > > > >> > > > >> list has
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > suggested?
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Thanks for a
> > thorough
> > > > >> proposal,
> > > > >> >> > > > great
> > > > >> >> > > > > > > work!"
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7,
> 2016,
> > > at
> > > > >> 10:28
> > > > >> >> > PM,
> > > > >> >> > > > Gwen
> > > > >> >> > > > > > > > Shapira
> > > > >> >> > > > > > > > > >> wrote:
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > Makes sense to
> me.
> > > > >> Thanks!
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7,
> > 2016
> > > at
> > > > >> 9:25
> > > > >> >> > PM,
> > > > >> >> > > > > > Harsha <
> > > > >> >> > > > > > > > > >> > > > kafka@harsha.io
> > > > >> >> > > > > > > > > >> > > > >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > wrote:
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > It doesn't
> need
> > > any
> > > > >> release
> > > > >> >> > > > > vehicle
> > > > >> >> > > > > > > but
> > > > >> >> > > > > > > > > >> still the
> > > > >> >> > > > > > > > > >> > > > >> work can
> > > > >> >> > > > > > > > > >> > > > >> >> > move
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > forward.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > If anyone is
> > > > >> interested in
> > > > >> >> > the
> > > > >> >> > > > KIP
> > > > >> >> > > > > > > > please
> > > > >> >> > > > > > > > > >> do the
> > > > >> >> > > > > > > > > >> > > > >> review and
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> provide
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > the
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > comments.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > -Harsha
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7,
> > > 2016,
> > > > >> at
> > > > >> >> > 04:59
> > > > >> >> > > > PM,
> > > > >> >> > > > > > > Ismael
> > > > >> >> > > > > > > > > >> Juma
> > > > >> >> > > > > > > > > >> > > > wrote:
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> I agree that
> it
> > > > >> would be
> > > > >> >> > good
> > > > >> >> > > > to
> > > > >> >> > > > > > have
> > > > >> >> > > > > > > > > more
> > > > >> >> > > > > > > > > >> time
> > > > >> >> > > > > > > > > >> > > to
> > > > >> >> > > > > > > > > >> > > > >> review
> > > > >> >> > > > > > > > > >> > > > >> >> > and
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > discuss
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> KIP-48.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> Ismael
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar
> 8,
> > > 2016
> > > > >> at
> > > > >> >> > 12:55
> > > > >> >> > > > AM,
> > > > >> >> > > > > > Gwen
> > > > >> >> > > > > > > > > >> Shapira <
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> gwen@confluent.io>
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Hi Team,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Since
> KIP-48
> > > > >> depends on
> > > > >> >> > > > KIP-43,
> > > > >> >> > > > > > > which
> > > > >> >> > > > > > > > > is
> > > > >> >> > > > > > > > > >> > > > already a
> > > > >> >> > > > > > > > > >> > > > >> bit
> > > > >> >> > > > > > > > > >> > > > >> >> > of a
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> risk
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > for
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > the next
> > > release
> > > > -
> > > > >> any
> > > > >> >> > > chance
> > > > >> >> > > > > we
> > > > >> >> > > > > > > can
> > > > >> >> > > > > > > > > >> delay
> > > > >> >> > > > > > > > > >> > > > >> delegation
> > > > >> >> > > > > > > > > >> > > > >> >> > tokens
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> to
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Kafka
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > 0.10.1?
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > With the
> > > > community
> > > > >> >> > working
> > > > >> >> > > > on a
> > > > >> >> > > > > > > > release
> > > > >> >> > > > > > > > > >> every
> > > > >> >> > > > > > > > > >> > > 3
> > > > >> >> > > > > > > > > >> > > > >> month,
> > > > >> >> > > > > > > > > >> > > > >> >> > this
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> is not
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a huge
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > delay.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Gwen
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb
> > 26,
> > > > >> 2016 at
> > > > >> >> > > 5:11
> > > > >> >> > > > > PM,
> > > > >> >> > > > > > > > Ashish
> > > > >> >> > > > > > > > > >> Singh
> > > > >> >> > > > > > > > > >> > > <
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > asingh@cloudera.com
> >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Parth,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Thanks
> > again
> > > > for
> > > > >> the
> > > > >> >> > > > awesome
> > > > >> >> > > > > > > write
> > > > >> >> > > > > > > > > up.
> > > > >> >> > > > > > > > > >> > > > Following
> > > > >> >> > > > > > > > > >> > > > >> our
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> discussion
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from the
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I
> > think
> > > > it
> > > > >> will
> > > > >> >> > be
> > > > >> >> > > > > easier
> > > > >> >> > > > > > > to
> > > > >> >> > > > > > > > > >> compare
> > > > >> >> > > > > > > > > >> > > > >> various
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> alternatives
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > if they
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > are
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > listed
> > > > together.
> > > > >> I am
> > > > >> >> > > > stating
> > > > >> >> > > > > > > > below a
> > > > >> >> > > > > > > > > >> few
> > > > >> >> > > > > > > > > >> > > > >> >> > alternatives along
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > with
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a the
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > current
> > > > proposal.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Current
> > > > >> proposal)
> > > > >> >> > Store
> > > > >> >> > > > > > > Delegation
> > > > >> >> > > > > > > > > >> Token,
> > > > >> >> > > > > > > > > >> > > DT,
> > > > >> >> > > > > > > > > >> > > > >> on ZK.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1.
> > Client
> > > > >> >> > > authenticates
> > > > >> >> > > > > > with a
> > > > >> >> > > > > > > > > >> broker.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2.
> Once
> > a
> > > > >> client is
> > > > >> >> > > > > > > > authenticated,
> > > > >> >> > > > > > > > > >> it
> > > > >> >> > > > > > > > > >> > > will
> > > > >> >> > > > > > > > > >> > > > >> make a
> > > > >> >> > > > > > > > > >> > > > >> >> > broker
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> side
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue
> a
> > > > >> delegation
> > > > >> >> > > > token.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The
> > > > broker
> > > > >> >> > > generates
> > > > >> >> > > > a
> > > > >> >> > > > > > > shared
> > > > >> >> > > > > > > > > >> secret
> > > > >> >> > > > > > > > > >> > > > based
> > > > >> >> > > > > > > > > >> > > > >> on
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > HMAC-SHA256(a
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> Password/Secret
> > > > >> >> > shared
> > > > >> >> > > > > > between
> > > > >> >> > > > > > > > all
> > > > >> >> > > > > > > > > >> > > brokers,
> > > > >> >> > > > > > > > > >> > > > >> >> > randomly
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > generated
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > tokenId).
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4.
> > Broker
> > > > >> stores
> > > > >> >> > this
> > > > >> >> > > > > token
> > > > >> >> > > > > > in
> > > > >> >> > > > > > > > its
> > > > >> >> > > > > > > > > >> in
> > > > >> >> > > > > > > > > >> > > > memory
> > > > >> >> > > > > > > > > >> > > > >> cache.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> Broker
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > also stores
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the
> > > > >> DelegationToken
> > > > >> >> > > > > without
> > > > >> >> > > > > > > the
> > > > >> >> > > > > > > > > >> hmac in
> > > > >> >> > > > > > > > > >> > > the
> > > > >> >> > > > > > > > > >> > > > >> >> > zookeeper.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. All
> > > > >> brokers will
> > > > >> >> > > > have a
> > > > >> >> > > > > > > cache
> > > > >> >> > > > > > > > > >> backed
> > > > >> >> > > > > > > > > >> > > by
> > > > >> >> > > > > > > > > >> > > > >> >> > zookeeper so
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> they
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > will all
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    get
> > > notified
> > > > >> >> > whenever
> > > > >> >> > > a
> > > > >> >> > > > > new
> > > > >> >> > > > > > > > token
> > > > >> >> > > > > > > > > is
> > > > >> >> > > > > > > > > >> > > > >> generated and
> > > > >> >> > > > > > > > > >> > > > >> >> > they
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> will
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > update
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    local
> > > cache
> > > > >> whenever
> > > > >> >> > > > token
> > > > >> >> > > > > > > state
> > > > >> >> > > > > > > > > >> changes.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6.
> > Broker
> > > > >> returns
> > > > >> >> > the
> > > > >> >> > > > > token
> > > > >> >> > > > > > to
> > > > >> >> > > > > > > > > >> Client.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable
> > > issues
> > > > >> and
> > > > >> >> > fixes
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1.
> > > Probable
> > > > >> race
> > > > >> >> > > > > condition,
> > > > >> >> > > > > > > > client
> > > > >> >> > > > > > > > > >> tries
> > > > >> >> > > > > > > > > >> > > to
> > > > >> >> > > > > > > > > >> > > > >> >> > authenticate
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> with
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a broker
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    that
> is
> > > yet
> > > > >> to be
> > > > >> >> > > > updated
> > > > >> >> > > > > > with
> > > > >> >> > > > > > > > the
> > > > >> >> > > > > > > > > >> newly
> > > > >> >> > > > > > > > > >> > > > >> generated
> > > > >> >> > > > > > > > > >> > > > >> >> > DT.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> This
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > can
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > probably be
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    dealt
> > with
> > > > >> making
> > > > >> >> > > > > dtRequest
> > > > >> >> > > > > > > > block
> > > > >> >> > > > > > > > > >> until
> > > > >> >> > > > > > > > > >> > > all
> > > > >> >> > > > > > > > > >> > > > >> >> > brokers have
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > updated
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their DT
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    cache.
> > Zk
> > > > >> barrier or
> > > > >> >> > > > > similar
> > > > >> >> > > > > > > > > >> mechanism
> > > > >> >> > > > > > > > > >> > > can
> > > > >> >> > > > > > > > > >> > > > be
> > > > >> >> > > > > > > > > >> > > > >> used.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> However,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > all such
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > mechanisms
> > > > >> will
> > > > >> >> > > increase
> > > > >> >> > > > > > > > > complexity.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2.
> > Using a
> > > > >> static
> > > > >> >> > > secret
> > > > >> >> > > > > key
> > > > >> >> > > > > > > > from
> > > > >> >> > > > > > > > > >> config
> > > > >> >> > > > > > > > > >> > > > >> file. Will
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> require
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > yet
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > another
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    config
> > and
> > > > >> uses a
> > > > >> >> > > static
> > > > >> >> > > > > > > secret
> > > > >> >> > > > > > > > > >> key. It
> > > > >> >> > > > > > > > > >> > > is
> > > > >> >> > > > > > > > > >> > > > >> advised
> > > > >> >> > > > > > > > > >> > > > >> >> > to
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> rotate
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > secret
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > keys
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > periodically.
> > > > >> This
> > > > >> >> > can
> > > > >> >> > > > be
> > > > >> >> > > > > > > > avoided
> > > > >> >> > > > > > > > > >> with
> > > > >> >> > > > > > > > > >> > > > >> controller
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> generating
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > secretKey
> and
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> passing
> > to
> > > > >> brokers
> > > > >> >> > > > > > > periodically.
> > > > >> >> > > > > > > > > >> However,
> > > > >> >> > > > > > > > > >> > > > >> this will
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> require
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > brokers to
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> maintain
> > > > >> certain
> > > > >> >> > > counts
> > > > >> >> > > > of
> > > > >> >> > > > > > > > > >> secretKeys.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > (Alternative
> > > 1)
> > > > >> Have
> > > > >> >> > > > > controller
> > > > >> >> > > > > > > > > >> generate
> > > > >> >> > > > > > > > > >> > > > >> delegation
> > > > >> >> > > > > > > > > >> > > > >> >> > token.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1.
> > Client
> > > > >> >> > > authenticates
> > > > >> >> > > > > > with a
> > > > >> >> > > > > > > > > >> broker.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2.
> Once
> > a
> > > > >> client is
> > > > >> >> > > > > > > > authenticated,
> > > > >> >> > > > > > > > > >> it
> > > > >> >> > > > > > > > > >> > > will
> > > > >> >> > > > > > > > > >> > > > >> make a
> > > > >> >> > > > > > > > > >> > > > >> >> > broker
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> side
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue
> a
> > > > >> delegation
> > > > >> >> > > > token.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3.
> > Broker
> > > > >> forwards
> > > > >> >> > the
> > > > >> >> > > > > > request
> > > > >> >> > > > > > > > to
> > > > >> >> > > > > > > > > >> > > > controller.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4.
> > > > Controller
> > > > >> >> > > generates
> > > > >> >> > > > a
> > > > >> >> > > > > DT
> > > > >> >> > > > > > > and
> > > > >> >> > > > > > > > > >> > > broadcasts
> > > > >> >> > > > > > > > > >> > > > >> to all
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> brokers.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5.
> > Broker
> > > > >> stores
> > > > >> >> > this
> > > > >> >> > > > > token
> > > > >> >> > > > > > in
> > > > >> >> > > > > > > > its
> > > > >> >> > > > > > > > > >> memory
> > > > >> >> > > > > > > > > >> > > > >> cache.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6.
> > > > Controller
> > > > >> >> > responds
> > > > >> >> > > > to
> > > > >> >> > > > > > > > broker’s
> > > > >> >> > > > > > > > > >> DT
> > > > >> >> > > > > > > > > >> > > req.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    7.
> > Broker
> > > > >> returns
> > > > >> >> > the
> > > > >> >> > > > > token
> > > > >> >> > > > > > to
> > > > >> >> > > > > > > > > >> Client.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable
> > > issues
> > > > >> and
> > > > >> >> > fixes
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. We
> > will
> > > > >> have to
> > > > >> >> > add
> > > > >> >> > > > new
> > > > >> >> > > > > > > APIs
> > > > >> >> > > > > > > > to
> > > > >> >> > > > > > > > > >> > > support
> > > > >> >> > > > > > > > > >> > > > >> >> > controller
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> pushing
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > to
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> brokers
> > on
> > > > >> top of
> > > > >> >> > the
> > > > >> >> > > > > > minimal
> > > > >> >> > > > > > > > APIs
> > > > >> >> > > > > > > > > >> that
> > > > >> >> > > > > > > > > >> > > are
> > > > >> >> > > > > > > > > >> > > > >> >> > currently
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > proposed.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. We
> > will
> > > > >> also have
> > > > >> >> > > to
> > > > >> >> > > > > add
> > > > >> >> > > > > > > APIs
> > > > >> >> > > > > > > > > to
> > > > >> >> > > > > > > > > >> > > support
> > > > >> >> > > > > > > > > >> > > > >> the
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> bootstrapping
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > case,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > i.e,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    when a
> > new
> > > > >> broker
> > > > >> >> > > comes
> > > > >> >> > > > up
> > > > >> >> > > > > > it
> > > > >> >> > > > > > > > will
> > > > >> >> > > > > > > > > >> have
> > > > >> >> > > > > > > > > >> > > to
> > > > >> >> > > > > > > > > >> > > > >> get all
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> delegation
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > from
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the
> > > > >> controller.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. In
> > > > >> catastrophic
> > > > >> >> > > > > failures
> > > > >> >> > > > > > > > where
> > > > >> >> > > > > > > > > >> all
> > > > >> >> > > > > > > > > >> > > > brokers
> > > > >> >> > > > > > > > > >> > > > >> go
> > > > >> >> > > > > > > > > >> > > > >> >> > down,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> the
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be
> lost
> > > even
> > > > >> if
> > > > >> >> > > servers
> > > > >> >> > > > > are
> > > > >> >> > > > > > > > > >> restarted as
> > > > >> >> > > > > > > > > >> > > > >> tokens
> > > > >> >> > > > > > > > > >> > > > >> >> > are not
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If
> this
> > > > >> happens,
> > > > >> >> > then
> > > > >> >> > > > > there
> > > > >> >> > > > > > > are
> > > > >> >> > > > > > > > > more
> > > > >> >> > > > > > > > > >> > > > important
> > > > >> >> > > > > > > > > >> > > > >> >> > things to
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > worry
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > about
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe
> it
> > > is
> > > > >> better
> > > > >> >> > to
> > > > >> >> > > > > > > > > >> re-authenticate.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > (Alternative
> > > 2)
> > > > >> Do not
> > > > >> >> > > > > > distribute
> > > > >> >> > > > > > > > DT
> > > > >> >> > > > > > > > > to
> > > > >> >> > > > > > > > > >> > > other
> > > > >> >> > > > > > > > > >> > > > >> brokers
> > > > >> >> > > > > > > > > >> > > > >> >> > at
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> all.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1.
> > Client
> > > > >> >> > > authenticates
> > > > >> >> > > > > > with a
> > > > >> >> > > > > > > > > >> broker.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2.
> Once
> > a
> > > > >> client is
> > > > >> >> > > > > > > > authenticated,
> > > > >> >> > > > > > > > > >> it
> > > > >> >> > > > > > > > > >> > > will
> > > > >> >> > > > > > > > > >> > > > >> make a
> > > > >> >> > > > > > > > > >> > > > >> >> > broker
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> side
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue
> a
> > > > >> delegation
> > > > >> >> > > > token.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The
> > > > broker
> > > > >> >> > > generates
> > > > >> >> > > > DT
> > > > >> >> > > > > > of
> > > > >> >> > > > > > > > > form,
> > > > >> >> > > > > > > > > >> > > [hmac +
> > > > >> >> > > > > > > > > >> > > > >> (owner,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> renewer,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > maxLifeTime,
> > > > >> id,
> > > > >> >> > hmac,
> > > > >> >> > > > > > > > > >> expirationTime)]
> > > > >> >> > > > > > > > > >> > > and
> > > > >> >> > > > > > > > > >> > > > >> passes
> > > > >> >> > > > > > > > > >> > > > >> >> > back
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> this
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > DT to
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > client.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    hmac
> is
> > > > >> generated
> > > > >> >> > via
> > > > >> >> > > > > > > > > >> {HMAC-SHA256(owner,
> > > > >> >> > > > > > > > > >> > > > >> renewer,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > maxLifeTime, id,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> expirationTime)
> > > > >> >> > using
> > > > >> >> > > > > > > > SecretKey}.
> > > > >> >> > > > > > > > > >> Note
> > > > >> >> > > > > > > > > >> > > that
> > > > >> >> > > > > > > > > >> > > > >> all
> > > > >> >> > > > > > > > > >> > > > >> >> > brokers
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> have
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > this
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > SecretKey.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4.
> > Client
> > > > >> then goes
> > > > >> >> > to
> > > > >> >> > > > any
> > > > >> >> > > > > > > > broker
> > > > >> >> > > > > > > > > >> and to
> > > > >> >> > > > > > > > > >> > > > >> >> > authenticate
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> sends
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > the DT.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    Broker
> > > > >> recalculates
> > > > >> >> > > hmac
> > > > >> >> > > > > > using
> > > > >> >> > > > > > > > > >> (owner,
> > > > >> >> > > > > > > > > >> > > > >> renewer,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> maxLifeTime,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > id, hmac,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> expirationTime) info
> > > > >> >> > > > from
> > > > >> >> > > > > DT
> > > > >> >> > > > > > > and
> > > > >> >> > > > > > > > > its
> > > > >> >> > > > > > > > > >> > > > >> SecretKey. If
> > > > >> >> > > > > > > > > >> > > > >> >> > it
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> matches
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > with
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac of
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    DT,
> > client
> > > > is
> > > > >> >> > > > > authenticated.
> > > > >> >> > > > > > > > Yes,
> > > > >> >> > > > > > > > > >> it will
> > > > >> >> > > > > > > > > >> > > > do
> > > > >> >> > > > > > > > > >> > > > >> other
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> obvious
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > checks of
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > timestamp
> > > > >> expiry and
> > > > >> >> > > > such.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Note that
> > > > secret
> > > > >> key
> > > > >> >> > will
> > > > >> >> > > > be
> > > > >> >> > > > > > > > > generated
> > > > >> >> > > > > > > > > >> by
> > > > >> >> > > > > > > > > >> > > > >> controller
> > > > >> >> > > > > > > > > >> > > > >> >> > and
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> passed
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > to
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > brokers
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > periodically.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable
> > > issues
> > > > >> and
> > > > >> >> > fixes
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. How
> > to
> > > > >> delete a
> > > > >> >> > DT?
> > > > >> >> > > > > Yes,
> > > > >> >> > > > > > > that
> > > > >> >> > > > > > > > > is
> > > > >> >> > > > > > > > > >> a
> > > > >> >> > > > > > > > > >> > > > downside
> > > > >> >> > > > > > > > > >> > > > >> >> > here.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> However,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > this can
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be
> > handled
> > > > >> with
> > > > >> >> > > brokers
> > > > >> >> > > > > > > > > maintaining
> > > > >> >> > > > > > > > > >> a
> > > > >> >> > > > > > > > > >> > > > >> blacklist of
> > > > >> >> > > > > > > > > >> > > > >> >> > DTs,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> DTs
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from this
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > list
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    can be
> > > > >> removed after
> > > > >> >> > > > > expiry.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. In
> > > > >> catastrophic
> > > > >> >> > > > > failures
> > > > >> >> > > > > > > > where
> > > > >> >> > > > > > > > > >> all
> > > > >> >> > > > > > > > > >> > > > brokers
> > > > >> >> > > > > > > > > >> > > > >> go
> > > > >> >> > > > > > > > > >> > > > >> >> > down,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> the
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be
> lost
> > > even
> > > > >> if
> > > > >> >> > > servers
> > > > >> >> > > > > are
> > > > >> >> > > > > > > > > >> restarted as
> > > > >> >> > > > > > > > > >> > > > >> tokens
> > > > >> >> > > > > > > > > >> > > > >> >> > are not
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If
> this
> > > > >> happens,
> > > > >> >> > then
> > > > >> >> > > > > there
> > > > >> >> > > > > > > are
> > > > >> >> > > > > > > > > more
> > > > >> >> > > > > > > > > >> > > > important
> > > > >> >> > > > > > > > > >> > > > >> >> > things to
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > worry
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > about
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe
> it
> > > is
> > > > >> better
> > > > >> >> > to
> > > > >> >> > > > > > > > > >> re-authenticate.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > On Fri,
> Feb
> > > 26,
> > > > >> 2016 at
> > > > >> >> > > > 1:58
> > > > >> >> > > > > > PM,
> > > > >> >> > > > > > > > > Parth
> > > > >> >> > > > > > > > > >> > > > >> Brahmbhatt <
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > pbrahmbhatt@hortonworks.com>
> > > > >> >> > > > > > > > wrote:
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Hi,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> I have
> > filed
> > > > >> KIP-48 so
> > > > >> >> > > we
> > > > >> >> > > > > can
> > > > >> >> > > > > > > > offer
> > > > >> >> > > > > > > > > >> hadoop
> > > > >> >> > > > > > > > > >> > > > like
> > > > >> >> > > > > > > > > >> > > > >> >> > delegation
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens in
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> kafka.
> You
> > > can
> > > > >> review
> > > > >> >> > > the
> > > > >> >> > > > > > design
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > > > >> >> > > > > > > > > >> > > > >> >> >
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > >
> > > > >> >> > > > > > > > > >>
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > >
> > > > >> >> > > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > .
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> This KIP
> > > > >> depends on
> > > > >> >> > > KIP-43
> > > > >> >> > > > > and
> > > > >> >> > > > > > > we
> > > > >> >> > > > > > > > > >> have also
> > > > >> >> > > > > > > > > >> > > > >> >> > discussed an
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > alternative to
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> proposed
> > > > design
> > > > >> here<
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > > > >> >> > > > > > > > > >> > > > >> >> >
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > >
> > > > >> >> > > > > > > > > >>
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > >
> > > > >> >> > > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >.
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Thanks
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Parth
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > --
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Regards,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Ashish
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > --
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > Regards,
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > Ashish
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > > > >> >> > > > > > > > > >> > > > >> >> >
> > > > >> >> > > > > > > > > >> > > > >>
> > > > >> >> > > > > > > > > >> > > >
> > > > >> >> > > > > > > > > >> > >
> > > > >> >> > > > > > > > > >>
> > > > >> >> > > > > > > > > >>
> > > > >> >> > > > > > > > > >
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > >
> > > > >> >> > > > > >
> > > > >> >> > > > > >
> > > > >> >> > > > > >
> > > > >> >> > > > > > --
> > > > >> >> > > > > > Regards,
> > > > >> >> > > > > >
> > > > >> >> > > > > > Rajini
> > > > >> >> > > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > > >
> > > > >> >> > > >
> > > > >> >> > > > --
> > > > >> >> > > > Regards,
> > > > >> >> > > >
> > > > >> >> > > > Rajini
> > > > >> >> > > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> --
> > > > >> >> Liquan Pei
> > > > >> >> Software Engineer, Confluent Inc
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[parth brahmbhatt <br...@gmail.com>]

Hi,

I am suggesting that we will only allow the renewal by users that
authenticated using *non* delegation token mechanism. For example, If user
Alice authenticated using kerberos and requested delegation tokens, only
user Alice authenticated via non delegation token mechanism can renew.
Clients that have  access to delegation tokens can not issue renewal
request for renewing their own token and this is primarily important to
reduce the time window for which a compromised token will be valid.

To clarify, Yes any authenticated user can request delegation tokens but
even here I would recommend to avoid creating a chain where a client
authenticated via delegation token request for more delegation tokens.
Basically anyone can request delegation token, as long as they authenticate
via a non delegation token mechanism.

Aren't classes listed here
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-PublicInterfaces>
sufficient?

Thanks
Parth



On Tue, Jun 21, 2016 at 4:33 PM, Jun Rao <ju...@confluent.io> wrote:

> Parth,
>
> Thanks for the reply. A couple of comments inline below.
>
> On Tue, Jun 21, 2016 at 10:36 AM, parth brahmbhatt <
> brahmbhatt.parth@gmail.com> wrote:
>
> > 1. Who / how are tokens renewed? By original requester only? or using
> > Kerberos
> > auth only?
> > My recommendation is to do this only using Kerberos auth and only threw
> the
> > renewer specified during the acquisition request.
> >
> >
> Hmm, not sure that I follow this. Are you saying that any client
> authenticated with the delegation token can renew, i.e. there is no renewer
> needed?
>
> Also, just to be clear, any authenticated client (either through SASL or
> SSL) can request a delegation token for the authenticated user, right?
>
>
> > 2. Are tokens stored on each broker or in ZK?
> > My recommendation is still to store in ZK or not store them at all. The
> > whole controller based distribution is too much overhead with not much to
> > achieve.
> >
> > 3. How are tokens invalidated / expired?
> > Either by expiration time out or through an explicit request to
> invalidate.
> >
> > 4. Which encryption algorithm is used?
> > SCRAM
> >
> > 5. What is the impersonation proposal (it wasn't in the KIP but was
> > discussed
> > in this thread)?
> > There is no imperonation proposal. I tried and explained how its a
> > different problem and why its not really necessary to discuss that as
> part
> > of this KIP.  This KIP will not support any impersonation, it will just
> be
> > another way to authenticate.
> >
> > 6. Do we need new ACLs, if so - for what actions?
> > We do not need new ACLs.
> >
> >
> Could we document the format of the new request/response and their
> associated Resource and Operation for ACL?
>
>
> > 7. How would the delegation token be configured in the client?
> > Should be through config. I wasn't planning on supporting JAAS for
> tokens.
> > I don't believe hadoop does this either.
> >
> > Thanks
> > Parth
> >
> >
> >
> > On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Harsha,
> > >
> > > Another question.
> > >
> > > 9. How would the delegation token be configured in the client? The
> > standard
> > > way is to do this through JAAS. However, we will need to think through
> if
> > > this is convenient in a shared environment. For example, when a new
> task
> > is
> > > added to a Storm worker node, do we need to dynamically add a new
> section
> > > in the JAAS file? It may be more convenient if we can pass in the token
> > > through the config directly w/o going through JAAS.
> > >
> > > Are you or Parth still actively working on this KIP?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > >
> > > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Just to add on that list.
> > > >
> > > > 2. It would be good to document the format of the data stored in ZK.
> > > > 7. Earlier, there was a discussion on whether the tokens should be
> > > > propagated through ZK like config/acl/quota, or through the
> controller.
> > > > Currently, the controller is only designed for propagating topic
> > > metadata,
> > > > but not other data.
> > > > 8. Should we use SCRAM to send the token instead of DIGEST-MD5 since
> > it's
> > > > deprecated?
> > > >
> > > > Also, the images in the wiki seem broken.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <gw...@confluent.io>
> > > wrote:
> > > >
> > > >> From what I can see, remaining questions are:
> > > >>
> > > >> 1. Who / how are tokens renewed? By original requester only? or
> using
> > > >> Kerberos auth only?
> > > >> 2. Are tokens stored on each broker or in ZK?
> > > >> 3. How are tokens invalidated / expired?
> > > >> 4. Which encryption algorithm is used?
> > > >> 5. What is the impersonation proposal (it wasn't in the KIP but was
> > > >> discussed in this thread)?
> > > >> 6. Do we need new ACLs, if so - for what actions?
> > > >>
> > > >> Gwen
> > > >>
> > > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <ka...@harsha.io> wrote:
> > > >> > Jun & Ismael,
> > > >> >                          Unfortunately I couldn't attend the KIP
> > > meeting
> > > >> >                          when delegation tokens discussed.
> > Appreciate
> > > if
> > > >> >                          you can update the thread if you have any
> > > >> >                          further questions.
> > > >> > Thanks,
> > > >> > Harsha
> > > >> >
> > > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
> > > >> >> It seems that the links to images in the KIP are broken.
> > > >> >>
> > > >> >> Liquan
> > > >> >>
> > > >> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
> > > >> >> brahmbhatt.parth@gmail.com> wrote:
> > > >> >>
> > > >> >> > 110. What does getDelegationTokenAs mean?
> > > >> >> > In the current proposal we only allow a user to get delegation
> > > token
> > > >> for
> > > >> >> > the identity that it authenticated as using another mechanism,
> > i.e.
> > > >> A user
> > > >> >> > that authenticate using a keytab for principal
> user1@EXAMPLE.COM
> > > >> will get
> > > >> >> > delegation tokens for that user only. In future I think we will
> > > have
> > > >> to
> > > >> >> > extend support such that we allow some set of users (
> > > >> >> > kafka-rest-user@EXAMPLE.COM, storm-nimbus@EXAMPLE.COM) to
> > acquire
> > > >> >> > delegation tokens on behalf of other users whose identity they
> > have
> > > >> >> > verified independently.  Kafka brokers will have ACLs to
> control
> > > >> which
> > > >> >> > users are allowed to impersonate other users and get tokens on
> > > >> behalf of
> > > >> >> > them. Overall Impersonation is a whole different problem in my
> > > >> opinion and
> > > >> >> > I think we can tackle it in separate KIP.
> > > >> >> >
> > > >> >> > 111. What's the typical rate of getting and renewing delegation
> > > >> tokens?
> > > >> >> > Typically this should be very very low, 1 request per minute
> is a
> > > >> >> > relatively high estimate. However it depends on the token
> > > >> expiration. I am
> > > >> >> > less worried about the extra load it puts on controller vs the
> > > added
> > > >> >> > complexity and the value it offers.
> > > >> >> >
> > > >> >> > Thanks
> > > >> >> > Parth
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <
> ismael@juma.me.uk>
> > > >> wrote:
> > > >> >> >
> > > >> >> > > Thanks Rajini. It would probably require a separate KIP as it
> > > will
> > > >> >> > > introduce user visible changes. We could also update KIP-48
> to
> > > >> have this
> > > >> >> > > information, but it seems cleaner to do it separately. We can
> > > >> discuss
> > > >> >> > that
> > > >> >> > > in the KIP call today.
> > > >> >> > >
> > > >> >> > > Ismael
> > > >> >> > >
> > > >> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <
> > > >> >> > > rajinisivaram@googlemail.com> wrote:
> > > >> >> > >
> > > >> >> > > > Ismael,
> > > >> >> > > >
> > > >> >> > > > I have created a JIRA (
> > > >> >> > https://issues.apache.org/jira/browse/KAFKA-3751)
> > > >> >> > > > for adding SCRAM as a SASL mechanism. Would that need
> another
> > > >> KIP? If
> > > >> >> > > > KIP-48 will use this mechanism, can this just be a JIRA
> that
> > > gets
> > > >> >> > > reviewed
> > > >> >> > > > when the PR is ready?
> > > >> >> > > >
> > > >> >> > > > Thank you,
> > > >> >> > > >
> > > >> >> > > > Rajini
> > > >> >> > > >
> > > >> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <
> > > ismael@juma.me.uk>
> > > >> >> > wrote:
> > > >> >> > > >
> > > >> >> > > > > Thanks Rajini, SCRAM seems like a good candidate.
> > > >> >> > > > >
> > > >> >> > > > > Gwen had independently mentioned this as a SASL mechanism
> > > that
> > > >> might
> > > >> >> > be
> > > >> >> > > > > useful for Kafka and I have been meaning to check it in
> > more
> > > >> detail.
> > > >> >> > > Good
> > > >> >> > > > > to know that you are willing to contribute an
> > implementation.
> > > >> Maybe
> > > >> >> > we
> > > >> >> > > > > should file a separate JIRA for this?
> > > >> >> > > > >
> > > >> >> > > > > Ismael
> > > >> >> > > > >
> > > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram <
> > > >> >> > > > > rajinisivaram@googlemail.com> wrote:
> > > >> >> > > > >
> > > >> >> > > > > > SCRAM (Salted Challenge Response Authentication
> > Mechanism)
> > > >> is a
> > > >> >> > > better
> > > >> >> > > > > > mechanism than Digest-MD5. Java doesn't come with a
> > > built-in
> > > >> SCRAM
> > > >> >> > > > > > SaslServer or SaslClient, but I will be happy to add
> > > support
> > > >> in
> > > >> >> > Kafka
> > > >> >> > > > > since
> > > >> >> > > > > > it would be a useful mechanism to support anyway.
> > > >> >> > > > > > https://tools.ietf.org/html/rfc7677 describes the
> > protocol
> > > >> for
> > > >> >> > > > > > SCRAM-SHA-256.
> > > >> >> > > > > >
> > > >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <
> > jun@confluent.io
> > > >
> > > >> wrote:
> > > >> >> > > > > >
> > > >> >> > > > > > > Parth,
> > > >> >> > > > > > >
> > > >> >> > > > > > > Thanks for the explanation. A couple of more
> questions.
> > > >> >> > > > > > >
> > > >> >> > > > > > > 110. What does getDelegationTokenAs mean?
> > > >> >> > > > > > >
> > > >> >> > > > > > > 111. What's the typical rate of getting and renewing
> > > >> delegation
> > > >> >> > > > tokens?
> > > >> >> > > > > > > That may have an impact on whether they should be
> > > directed
> > > >> to the
> > > >> >> > > > > > > controller.
> > > >> >> > > > > > >
> > > >> >> > > > > > > Jun
> > > >> >> > > > > > >
> > > >> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt <
> > > >> >> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > > >> >> > > > > > >
> > > >> >> > > > > > > > Hi Jun,
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > Thanks for reviewing.
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > * We could add a Cluster action to add acls on who
> > can
> > > >> request
> > > >> >> > > > > > delegation
> > > >> >> > > > > > > > tokens. I don't see the use case for that yet but
> > down
> > > >> the line
> > > >> >> > > > when
> > > >> >> > > > > we
> > > >> >> > > > > > > > start supporting getDelegationTokenAs it will be
> > > >> necessary.
> > > >> >> > > > > > > > * Yes we recommend tokens to be only
> used/distributed
> > > >> over
> > > >> >> > secure
> > > >> >> > > > > > > channels.
> > > >> >> > > > > > > > * Depending on what design we end up choosing
> > > >> Invalidation will
> > > >> >> > > be
> > > >> >> > > > > > > > responsibility of every broker or controller.
> > > >> >> > > > > > > > * I am not sure if I documented somewhere that
> > > >> invalidation
> > > >> >> > will
> > > >> >> > > > > > directly
> > > >> >> > > > > > > > go through zookeeper but that is not the intent.
> > > >> Invalidation
> > > >> >> > > will
> > > >> >> > > > > > either
> > > >> >> > > > > > > > be request based or due to expiration. No direct
> > > >> zookeeper
> > > >> >> > > > > interaction
> > > >> >> > > > > > > from
> > > >> >> > > > > > > > any client.
> > > >> >> > > > > > > > * "Broker also stores the DelegationToken without
> the
> > > >> hmac in
> > > >> >> > the
> > > >> >> > > > > > > > zookeeper." : Sorry about the confusion. The sole
> > > >> purpose of
> > > >> >> > > > > zookeeper
> > > >> >> > > > > > in
> > > >> >> > > > > > > > this design is as distribution channel for tokens
> > > >> between all
> > > >> >> > > > brokers
> > > >> >> > > > > > > and a
> > > >> >> > > > > > > > layer that ensures only tokens that were generated
> by
> > > >> making a
> > > >> >> > > > > request
> > > >> >> > > > > > > to a
> > > >> >> > > > > > > > broker will be accepted (more on this in second
> > > >> paragraph). The
> > > >> >> > > > token
> > > >> >> > > > > > > > consists of few elements (owner, renewer, uuid ,
> > > >> expiration,
> > > >> >> > > hmac)
> > > >> >> > > > ,
> > > >> >> > > > > > one
> > > >> >> > > > > > > of
> > > >> >> > > > > > > > which is the finally generated hmac but hmac it
> self
> > is
> > > >> >> > derivable
> > > >> >> > > > if
> > > >> >> > > > > > you
> > > >> >> > > > > > > > have all the other elements of the token + secret
> key
> > > to
> > > >> >> > generate
> > > >> >> > > > > hmac.
> > > >> >> > > > > > > > Given zookeeper does not provide SSL support we do
> > not
> > > >> want the
> > > >> >> > > > > entire
> > > >> >> > > > > > > > token to be wire transferred to zookeeper as that
> > will
> > > >> be an
> > > >> >> > > > insecure
> > > >> >> > > > > > > wire
> > > >> >> > > > > > > > transfer. Instead we only store all the other
> > elements
> > > >> of a
> > > >> >> > > > > delegation
> > > >> >> > > > > > > > tokens. Brokers can read these elements and because
> > > they
> > > >> also
> > > >> >> > > have
> > > >> >> > > > > > access
> > > >> >> > > > > > > > to secret key they will be able to generate hmac on
> > > >> their end.
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > One of the alternative proposed is to avoid
> zookeeper
> > > >> >> > > altogether. A
> > > >> >> > > > > > > Client
> > > >> >> > > > > > > > will call broker with required information (owner,
> > > >> renwer,
> > > >> >> > > > > expiration)
> > > >> >> > > > > > > and
> > > >> >> > > > > > > > get back (signed hmac, uuid). Broker won't store
> this
> > > in
> > > >> >> > > zookeeper.
> > > >> >> > > > > > From
> > > >> >> > > > > > > > this point a client can contact any broker with all
> > the
> > > >> >> > > delegation
> > > >> >> > > > > > token
> > > >> >> > > > > > > > info (owner, rewner, expiration, hmac, uuid) the
> > borker
> > > >> will
> > > >> >> > > > > regenerate
> > > >> >> > > > > > > the
> > > >> >> > > > > > > > hmac and as long as it matches with hmac presented
> by
> > > >> client ,
> > > >> >> > > > broker
> > > >> >> > > > > > > will
> > > >> >> > > > > > > > allow the request to authenticate.  Only problem
> with
> > > >> this
> > > >> >> > > approach
> > > >> >> > > > > is
> > > >> >> > > > > > if
> > > >> >> > > > > > > > the secret key is compromised any client can now
> > > generate
> > > >> >> > random
> > > >> >> > > > > tokens
> > > >> >> > > > > > > and
> > > >> >> > > > > > > > they will still be able to authenticate as any user
> > > they
> > > >> like.
> > > >> >> > > with
> > > >> >> > > > > > > > zookeeper we guarantee that only tokens acquired
> via
> > a
> > > >> broker
> > > >> >> > > > (using
> > > >> >> > > > > > some
> > > >> >> > > > > > > > auth scheme other than delegation token) will be
> > > >> accepted. We
> > > >> >> > > need
> > > >> >> > > > to
> > > >> >> > > > > > > > discuss which proposal makes more sense and we can
> go
> > > >> over it
> > > >> >> > in
> > > >> >> > > > > > > tomorrow's
> > > >> >> > > > > > > > meeting.
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > Also, can you forward the invite to me?
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > Thanks
> > > >> >> > > > > > > > Parth
> > > >> >> > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <
> > > >> jun@confluent.io>
> > > >> >> > > > wrote:
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > > Thanks for the KIP. A few comments.
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > 100. This potentially can be useful for Kafka
> > Connect
> > > >> and
> > > >> >> > Kafka
> > > >> >> > > > > rest
> > > >> >> > > > > > > > proxy
> > > >> >> > > > > > > > > where a worker agent will need to run a task on
> > > behalf
> > > >> of a
> > > >> >> > > > client.
> > > >> >> > > > > > We
> > > >> >> > > > > > > > will
> > > >> >> > > > > > > > > likely need to change how those services use
> Kafka
> > > >> clients
> > > >> >> > > > > > > > > (producer/consumer). Instead of a shared client
> per
> > > >> worker,
> > > >> >> > we
> > > >> >> > > > will
> > > >> >> > > > > > > need
> > > >> >> > > > > > > > a
> > > >> >> > > > > > > > > client per user task since the authentication
> > happens
> > > >> at the
> > > >> >> > > > > > connection
> > > >> >> > > > > > > > > level. For Kafka Connect, the renewer will be the
> > > >> workers.
> > > >> >> > So,
> > > >> >> > > we
> > > >> >> > > > > > > > probably
> > > >> >> > > > > > > > > need to allow multiple renewers. For Kafka rest
> > > proxy,
> > > >> the
> > > >> >> > > > renewer
> > > >> >> > > > > > can
> > > >> >> > > > > > > > > probably just be the creator of the token.
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > 101. Do we need new acl on who can request
> > delegation
> > > >> tokens?
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > 102. Do we recommend people to send delegation
> > tokens
> > > >> in an
> > > >> >> > > > > encrypted
> > > >> >> > > > > > > > > channel?
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > 103. Who is responsible for expiring tokens,
> every
> > > >> broker?
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > 104. For invalidating tokens, would it be better
> to
> > > do
> > > >> it in
> > > >> >> > a
> > > >> >> > > > > > request
> > > >> >> > > > > > > > > instead of going to ZK directly?
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > 105. The terminology of client in the wiki
> > sometimes
> > > >> refers
> > > >> >> > to
> > > >> >> > > > the
> > > >> >> > > > > > end
> > > >> >> > > > > > > > > client and some other times refers to the client
> > > using
> > > >> the
> > > >> >> > > > > delegation
> > > >> >> > > > > > > > > tokens. It would be useful to distinguish between
> > the
> > > >> two.
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > 106. Could you explain the sentence "Broker also
> > > >> stores the
> > > >> >> > > > > > > > DelegationToken
> > > >> >> > > > > > > > > without the hmac in the zookeeper." a bit more? I
> > > >> thought the
> > > >> >> > > > > > > delegation
> > > >> >> > > > > > > > > token is the hmac.
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > Thanks,
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > Jun
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <
> > > >> jun@confluent.io>
> > > >> >> > > > wrote:
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > > Hi, Harsha,
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > > Just sent out a KIP meeting invite. We can
> > discuss
> > > >> this in
> > > >> >> > > the
> > > >> >> > > > > > > meeting
> > > >> >> > > > > > > > > > tomorrow.
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > > Thanks,
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > > Jun
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <
> > > >> kafka@harsha.io>
> > > >> >> > > > wrote:
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > >> Hi All,
> > > >> >> > > > > > > > > >>            Can we have a KIP meeting around
> > this.
> > > >> The KIP
> > > >> >> > is
> > > >> >> > > > up
> > > >> >> > > > > > for
> > > >> >> > > > > > > > > >>            sometime and if there are any
> > questions
> > > >> lets
> > > >> >> > > > quickly
> > > >> >> > > > > > hash
> > > >> >> > > > > > > > out
> > > >> >> > > > > > > > > >>            details.
> > > >> >> > > > > > > > > >>
> > > >> >> > > > > > > > > >> Thanks,
> > > >> >> > > > > > > > > >> Harsha
> > > >> >> > > > > > > > > >>
> > > >> >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40 AM, parth
> > > brahmbhatt
> > > >> wrote:
> > > >> >> > > > > > > > > >> > That is what the hadoop echo system uses so
> no
> > > >> good
> > > >> >> > reason
> > > >> >> > > > > > really.
> > > >> >> > > > > > > > We
> > > >> >> > > > > > > > > >> > could
> > > >> >> > > > > > > > > >> > change it to whatever is the newest
> > recommended
> > > >> standard
> > > >> >> > > is.
> > > >> >> > > > > > > > > >> >
> > > >> >> > > > > > > > > >> > Thanks
> > > >> >> > > > > > > > > >> > Parth
> > > >> >> > > > > > > > > >> >
> > > >> >> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael
> Juma <
> > > >> >> > > > > ismael@juma.me.uk
> > > >> >> > > > > > >
> > > >> >> > > > > > > > > wrote:
> > > >> >> > > > > > > > > >> >
> > > >> >> > > > > > > > > >> > > Hi Parth,
> > > >> >> > > > > > > > > >> > >
> > > >> >> > > > > > > > > >> > > Thanks for the KIP. I only started
> reviewing
> > > >> this and
> > > >> >> > > may
> > > >> >> > > > > have
> > > >> >> > > > > > > > > >> additional
> > > >> >> > > > > > > > > >> > > questions later. The immediate question
> that
> > > >> came to
> > > >> >> > > mind
> > > >> >> > > > is
> > > >> >> > > > > > our
> > > >> >> > > > > > > > > >> choice of
> > > >> >> > > > > > > > > >> > > "DIGEST-MD5" even though it's marked as
> > > >> OBSOLETE in
> > > >> >> > the
> > > >> >> > > > IANA
> > > >> >> > > > > > > > > Registry
> > > >> >> > > > > > > > > >> of
> > > >> >> > > > > > > > > >> > > SASL mechanisms and the original RFC
> (2831)
> > > has
> > > >> been
> > > >> >> > > moved
> > > >> >> > > > > to
> > > >> >> > > > > > > > > Historic
> > > >> >> > > > > > > > > >> > > status:
> > > >> >> > > > > > > > > >> > >
> > > >> >> > > > > > > > > >> > > https://tools.ietf.org/html/rfc6331
> > > >> >> > > > > > > > > >> > >
> > > >> >> > > > > > > > >
> > > >> >> > > > > >
> > > >> >> > >
> > > >>
> http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> > > >> >> > > > > > > > > >> > >
> > > >> >> > > > > > > > > >> > > What is the reasoning behind that choice?
> > > >> >> > > > > > > > > >> > >
> > > >> >> > > > > > > > > >> > > Thanks,
> > > >> >> > > > > > > > > >> > > Ismael
> > > >> >> > > > > > > > > >> > >
> > > >> >> > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen
> > > Shapira <
> > > >> >> > > > > > > gwen@confluent.io
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > >> wrote:
> > > >> >> > > > > > > > > >> > >
> > > >> >> > > > > > > > > >> > > > Also comments inline :)
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > > > > * I want to emphasize that even though
> > > >> delegation
> > > >> >> > > > tokens
> > > >> >> > > > > > > are a
> > > >> >> > > > > > > > > >> Hadoop
> > > >> >> > > > > > > > > >> > > > > innovation, I feel very strongly about
> > not
> > > >> adding
> > > >> >> > > > > > dependency
> > > >> >> > > > > > > > on
> > > >> >> > > > > > > > > >> Hadoop
> > > >> >> > > > > > > > > >> > > > > when implementing delegation tokens
> for
> > > >> Kafka. The
> > > >> >> > > KIP
> > > >> >> > > > > > > doesn't
> > > >> >> > > > > > > > > >> imply
> > > >> >> > > > > > > > > >> > > > > such dependency, but if you can
> > clarify...
> > > >> >> > > > > > > > > >> > > > >
> > > >> >> > > > > > > > > >> > > > >
> > > >> >> > > > > > > > > >> > > > > *No hadoop dependency.*
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > > > Yay! Just add this to the KIP so no one
> > will
> > > >> read
> > > >> >> > the
> > > >> >> > > > KIP
> > > >> >> > > > > > and
> > > >> >> > > > > > > > > panic
> > > >> >> > > > > > > > > >> > > > three weeks before the next release...
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > > > > * Can we get delegation token at any
> > time
> > > >> after
> > > >> >> > > > > > > > authenticating?
> > > >> >> > > > > > > > > >> only
> > > >> >> > > > > > > > > >> > > > > immediately after?
> > > >> >> > > > > > > > > >> > > > >
> > > >> >> > > > > > > > > >> > > > >
> > > >> >> > > > > > > > > >> > > > > *As long as you are authenticated you
> > can
> > > >> get
> > > >> >> > > > delegation
> > > >> >> > > > > > > > tokens.
> > > >> >> > > > > > > > > >> We
> > > >> >> > > > > > > > > >> > > need
> > > >> >> > > > > > > > > >> > > > to
> > > >> >> > > > > > > > > >> > > > > discuss if a client authenticated
> using
> > > >> delegation
> > > >> >> > > > > token,
> > > >> >> > > > > > > can
> > > >> >> > > > > > > > > also
> > > >> >> > > > > > > > > >> > > > acquire
> > > >> >> > > > > > > > > >> > > > > delegation token again or not. Also
> > there
> > > >> is the
> > > >> >> > > > > question
> > > >> >> > > > > > of
> > > >> >> > > > > > > > do
> > > >> >> > > > > > > > > we
> > > >> >> > > > > > > > > >> > > allow
> > > >> >> > > > > > > > > >> > > > > anyone to acquire delegation token or
> we
> > > >> want
> > > >> >> > > specific
> > > >> >> > > > > > ACLs
> > > >> >> > > > > > > (I
> > > >> >> > > > > > > > > >> think
> > > >> >> > > > > > > > > >> > > its
> > > >> >> > > > > > > > > >> > > > an
> > > >> >> > > > > > > > > >> > > > > overkill.)*
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > > > I agree that ACLs is an overkill.
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > > > I think we are debating two options:
> > Either
> > > >> require
> > > >> >> > > > > Kerberos
> > > >> >> > > > > > > > auth
> > > >> >> > > > > > > > > >> for
> > > >> >> > > > > > > > > >> > > > renewal or require non-owners to renew.
> > > >> >> > > > > > > > > >> > > > I *think* the latter is simpler (it
> > > basically
> > > >> >> > require
> > > >> >> > > a
> > > >> >> > > > > "job
> > > >> >> > > > > > > > > master"
> > > >> >> > > > > > > > > >> > > > to take responsibility for the renewal,
> it
> > > >> will have
> > > >> >> > > its
> > > >> >> > > > > own
> > > >> >> > > > > > > > > >> identity
> > > >> >> > > > > > > > > >> > > > anyway and I think this is the correct
> > > design
> > > >> >> > pattern
> > > >> >> > > > > > anyway.
> > > >> >> > > > > > > > For
> > > >> >> > > > > > > > > >> > > > storm, I'd expect Nimbus to coordinate
> > > >> renewals?),
> > > >> >> > but
> > > >> >> > > > it
> > > >> >> > > > > is
> > > >> >> > > > > > > > hard
> > > >> >> > > > > > > > > to
> > > >> >> > > > > > > > > >> > > > debate simplicity without looking at the
> > > code
> > > >> >> > changes
> > > >> >> > > > > > > required.
> > > >> >> > > > > > > > If
> > > >> >> > > > > > > > > >> you
> > > >> >> > > > > > > > > >> > > > have a draft of how the "require
> Kerberos"
> > > >> will look
> > > >> >> > > in
> > > >> >> > > > > > Kafka
> > > >> >> > > > > > > > > code,
> > > >> >> > > > > > > > > >> > > > I'll be happy to take a look.
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > > > > * My understanding is that tokens will
> > > >> propagate
> > > >> >> > via
> > > >> >> > > > ZK
> > > >> >> > > > > > but
> > > >> >> > > > > > > > > >> without
> > > >> >> > > > > > > > > >> > > > > additional changes to UpdateMetadata
> > > >> protocol,
> > > >> >> > > > correct?
> > > >> >> > > > > > > > Clients
> > > >> >> > > > > > > > > >> > > > > currently don't retry on SASL auth
> > failure
> > > >> (IIRC),
> > > >> >> > > but
> > > >> >> > > > > > since
> > > >> >> > > > > > > > the
> > > >> >> > > > > > > > > >> > > > > tokens propagate between brokers
> asynch,
> > > we
> > > >> will
> > > >> >> > > need
> > > >> >> > > > to
> > > >> >> > > > > > > > retry a
> > > >> >> > > > > > > > > >> bit
> > > >> >> > > > > > > > > >> > > > > to avoid clients failing auth due to
> > > timing
> > > >> >> > issues.
> > > >> >> > > > > > > > > >> > > > >
> > > >> >> > > > > > > > > >> > > > > *I am considering 2 alternatives right
> > > now.
> > > >> The
> > > >> >> > > > current
> > > >> >> > > > > > > > > documented
> > > >> >> > > > > > > > > >> > > > approach
> > > >> >> > > > > > > > > >> > > > > is zookeeper based and it does not
> > require
> > > >> any
> > > >> >> > > changes
> > > >> >> > > > > to
> > > >> >> > > > > > > > > >> > > UpdateMetadata
> > > >> >> > > > > > > > > >> > > > > protocol. An alternative approach can
> > > remove
> > > >> >> > > zookeeper
> > > >> >> > > > > > > > > dependency
> > > >> >> > > > > > > > > >> as
> > > >> >> > > > > > > > > >> > > well
> > > >> >> > > > > > > > > >> > > > > but we can discuss that in KIP
> > discussion
> > > >> call.*
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > > > Oooh! Sounds interesting. Do you want to
> > > ping
> > > >> Jun to
> > > >> >> > > > > > arrange a
> > > >> >> > > > > > > > > call?
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > > > > * I liked Ashish's suggestion of
> having
> > > >> just the
> > > >> >> > > > > > controller
> > > >> >> > > > > > > > > issue
> > > >> >> > > > > > > > > >> the
> > > >> >> > > > > > > > > >> > > > > delegation tokens, to avoid syncing a
> > > shared
> > > >> >> > secret.
> > > >> >> > > > Not
> > > >> >> > > > > > > sure
> > > >> >> > > > > > > > if
> > > >> >> > > > > > > > > >> we
> > > >> >> > > > > > > > > >> > > > > want to continue the discussion here
> or
> > on
> > > >> the
> > > >> >> > > wiki. I
> > > >> >> > > > > > think
> > > >> >> > > > > > > > > that
> > > >> >> > > > > > > > > >> we
> > > >> >> > > > > > > > > >> > > > > can decouple the problem of "token
> > > >> distribution"
> > > >> >> > > from
> > > >> >> > > > > > > "shared
> > > >> >> > > > > > > > > >> secret
> > > >> >> > > > > > > > > >> > > > > distribution" and use the controller
> as
> > > the
> > > >> only
> > > >> >> > > token
> > > >> >> > > > > > > > generator
> > > >> >> > > > > > > > > >> to
> > > >> >> > > > > > > > > >> > > > > solve the second issue, while still
> > using
> > > >> ZK async
> > > >> >> > > to
> > > >> >> > > > > > > > distribute
> > > >> >> > > > > > > > > >> > > > > tokens.
> > > >> >> > > > > > > > > >> > > > >
> > > >> >> > > > > > > > > >> > > > >
> > > >> >> > > > > > > > > >> > > > > *As mentioned in the previous Email I
> am
> > > >> fine with
> > > >> >> > > > that
> > > >> >> > > > > > > > approach
> > > >> >> > > > > > > > > >> as
> > > >> >> > > > > > > > > >> > > long
> > > >> >> > > > > > > > > >> > > > as
> > > >> >> > > > > > > > > >> > > > > we agree that the extra complexity of
> > > >> >> > > adding/updating
> > > >> >> > > > > APIS
> > > >> >> > > > > > > > adds
> > > >> >> > > > > > > > > >> enough
> > > >> >> > > > > > > > > >> > > > > value. The advantage with the
> controller
> > > >> approach
> > > >> >> > is
> > > >> >> > > > > > secret
> > > >> >> > > > > > > > > >> rotation
> > > >> >> > > > > > > > > >> > > can
> > > >> >> > > > > > > > > >> > > > be
> > > >> >> > > > > > > > > >> > > > > automated,frequent and would not
> require
> > > >> >> > > deployment. *
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > > > Can you detail the extra complexity (or
> > > point
> > > >> me to
> > > >> >> > > the
> > > >> >> > > > > > email
> > > >> >> > > > > > > I
> > > >> >> > > > > > > > > >> > > > missed?) - which Apis are required?
> > > >> >> > > > > > > > > >> > > > As far as I can tell, clients can
> already
> > > >> find the
> > > >> >> > > > > > controller
> > > >> >> > > > > > > > from
> > > >> >> > > > > > > > > >> > > > metadata. I'm a bit more concerned about
> > > >> controller
> > > >> >> > > > load.
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > > > >
> > > >> >> > > > > > > > > >> > > > > * While I like the idea of forcing
> > > kerberos
> > > >> auth
> > > >> >> > for
> > > >> >> > > > > > > renwal, I
> > > >> >> > > > > > > > > >> think
> > > >> >> > > > > > > > > >> > > > > it mixes the transport layer the the
> > > request
> > > >> >> > content
> > > >> >> > > > in
> > > >> >> > > > > a
> > > >> >> > > > > > > > pretty
> > > >> >> > > > > > > > > >> ugly
> > > >> >> > > > > > > > > >> > > > > way. Perhaps limiting renewer to
> > non-owner
> > > >> is
> > > >> >> > > better.
> > > >> >> > > > > > > > > >> > > > >
> > > >> >> > > > > > > > > >> > > > > *I feel this is a necessary evil.
> While
> > > >> this will
> > > >> >> > > make
> > > >> >> > > > > the
> > > >> >> > > > > > > > kafka
> > > >> >> > > > > > > > > >> code
> > > >> >> > > > > > > > > >> > > > > pretty straight forward , forcing
> > renewer
> > > >> to
> > > >> >> > > > non-owner
> > > >> >> > > > > > > pushes
> > > >> >> > > > > > > > > >> the code
> > > >> >> > > > > > > > > >> > > > > ugliness to client and makes it even
> > > harder
> > > >> to
> > > >> >> > > > > > integrate.  *
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > > > As mentioned before, I don't think the
> > > >> "renewal by
> > > >> >> > > > other"
> > > >> >> > > > > > > > approach
> > > >> >> > > > > > > > > >> is
> > > >> >> > > > > > > > > >> > > > that ugly for the clients we expect to
> use
> > > >> >> > delegation
> > > >> >> > > > > tokens
> > > >> >> > > > > > > > since
> > > >> >> > > > > > > > > >> > > > they will have an app-master of some
> sort
> > > who
> > > >> >> > > requested
> > > >> >> > > > > the
> > > >> >> > > > > > > > token
> > > >> >> > > > > > > > > to
> > > >> >> > > > > > > > > >> > > > begin with.
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > > > >
> > > >> >> > > > > > > > > >> > > > > The response for my question on how
> > > multiple
> > > >> >> > > > identities
> > > >> >> > > > > > will
> > > >> >> > > > > > > > be
> > > >> >> > > > > > > > > >> > > > > handled wasn't super clear to me -
> > AFAIK,
> > > >> we don't
> > > >> >> > > > > > > > authenticate
> > > >> >> > > > > > > > > >> each
> > > >> >> > > > > > > > > >> > > > > request, we authenticate connections.
> > > >> >> > > > > > > > > >> > > > >
> > > >> >> > > > > > > > > >> > > > > *We authenticate connections, and only
> > > when
> > > >> they
> > > >> >> > are
> > > >> >> > > > > being
> > > >> >> > > > > > > > > >> established.
> > > >> >> > > > > > > > > >> > > > Let
> > > >> >> > > > > > > > > >> > > > > me try to phrase this as a question,
> in
> > > >> absence of
> > > >> >> > > > > > > delegation
> > > >> >> > > > > > > > > >> tokens if
> > > >> >> > > > > > > > > >> > > > we
> > > >> >> > > > > > > > > >> > > > > had to support the use case using user
> > > >> TGT's how
> > > >> >> > > would
> > > >> >> > > > > we
> > > >> >> > > > > > do
> > > >> >> > > > > > > > it?
> > > >> >> > > > > > > > > >> My
> > > >> >> > > > > > > > > >> > > point
> > > >> >> > > > > > > > > >> > > > > was it would be no different with
> > > delegation
> > > >> >> > tokens.
> > > >> >> > > > The
> > > >> >> > > > > > use
> > > >> >> > > > > > > > > case
> > > >> >> > > > > > > > > >> you
> > > >> >> > > > > > > > > >> > > are
> > > >> >> > > > > > > > > >> > > > > describing seems more like
> > impersonation.*
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > > > Yeah, I thought that one of the things
> > that
> > > >> >> > delegation
> > > >> >> > > > > > tokens
> > > >> >> > > > > > > > > >> handled.
> > > >> >> > > > > > > > > >> > > > Maybe I got it wrong :)
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > > > Thanks for the detailed answers.
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > > > Gwen
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > > > > Thanks
> > > >> >> > > > > > > > > >> > > > > Parth
> > > >> >> > > > > > > > > >> > > > >
> > > >> >> > > > > > > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen
> > > >> Shapira <
> > > >> >> > > > > > > > > gwen@confluent.io
> > > >> >> > > > > > > > > >> >
> > > >> >> > > > > > > > > >> > > > wrote:
> > > >> >> > > > > > > > > >> > > > >
> > > >> >> > > > > > > > > >> > > > >> Hi Parth and Harsha,
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > > >> Few more comments:
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > > >> * The API / RequestResponse section
> > > >> doesn't seem
> > > >> >> > to
> > > >> >> > > > > have
> > > >> >> > > > > > > good
> > > >> >> > > > > > > > > >> > > > >> description of the changes to the
> Kafka
> > > >> Protocol.
> > > >> >> > > > > Sounds
> > > >> >> > > > > > > like
> > > >> >> > > > > > > > > >> you are
> > > >> >> > > > > > > > > >> > > > >> proposing new DelegationTokenRequest
> > and
> > > >> >> > > > > > RenewTokenRequest
> > > >> >> > > > > > > > (and
> > > >> >> > > > > > > > > >> > > > >> matching responses), without
> detailing
> > > the
> > > >> >> > contents
> > > >> >> > > > of
> > > >> >> > > > > > the
> > > >> >> > > > > > > > > >> requests
> > > >> >> > > > > > > > > >> > > > >> and responses? Or rather, you show
> the
> > > >> class
> > > >> >> > > > interface,
> > > >> >> > > > > > but
> > > >> >> > > > > > > > not
> > > >> >> > > > > > > > > >> the
> > > >> >> > > > > > > > > >> > > > >> underlying protocol. This could be
> seen
> > > as
> > > >> an
> > > >> >> > > > > > > implementation
> > > >> >> > > > > > > > > >> detail,
> > > >> >> > > > > > > > > >> > > > >> but since the binary protocol is what
> > we
> > > >> provide
> > > >> >> > to
> > > >> >> > > > > > > non-Java
> > > >> >> > > > > > > > > >> clients,
> > > >> >> > > > > > > > > >> > > > >> we need to show the changes there.
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > > >> * getDelegationToken sounds like
> > > >> >> > > > > > > > delegationTokenRequestHandler?
> > > >> >> > > > > > > > > >> Is it
> > > >> >> > > > > > > > > >> > > > >> planned to be part of KafkaApi? or
> > > Client?
> > > >> Its
> > > >> >> > > > > unclear...
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > > >> * I want to emphasize that even
> though
> > > >> delegation
> > > >> >> > > > > tokens
> > > >> >> > > > > > > are
> > > >> >> > > > > > > > a
> > > >> >> > > > > > > > > >> Hadoop
> > > >> >> > > > > > > > > >> > > > >> innovation, I feel very strongly
> about
> > > not
> > > >> adding
> > > >> >> > > > > > > dependency
> > > >> >> > > > > > > > on
> > > >> >> > > > > > > > > >> Hadoop
> > > >> >> > > > > > > > > >> > > > >> when implementing delegation tokens
> for
> > > >> Kafka.
> > > >> >> > The
> > > >> >> > > > KIP
> > > >> >> > > > > > > > doesn't
> > > >> >> > > > > > > > > >> imply
> > > >> >> > > > > > > > > >> > > > >> such dependency, but if you can
> > > clarify...
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > > >> * Can we get delegation token at any
> > time
> > > >> after
> > > >> >> > > > > > > > authenticating?
> > > >> >> > > > > > > > > >> only
> > > >> >> > > > > > > > > >> > > > >> immediately after?
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > > >> * My understanding is that tokens
> will
> > > >> propagate
> > > >> >> > > via
> > > >> >> > > > ZK
> > > >> >> > > > > > but
> > > >> >> > > > > > > > > >> without
> > > >> >> > > > > > > > > >> > > > >> additional changes to UpdateMetadata
> > > >> protocol,
> > > >> >> > > > correct?
> > > >> >> > > > > > > > Clients
> > > >> >> > > > > > > > > >> > > > >> currently don't retry on SASL auth
> > > failure
> > > >> >> > (IIRC),
> > > >> >> > > > but
> > > >> >> > > > > > > since
> > > >> >> > > > > > > > > the
> > > >> >> > > > > > > > > >> > > > >> tokens propagate between brokers
> > asynch,
> > > >> we will
> > > >> >> > > need
> > > >> >> > > > > to
> > > >> >> > > > > > > > retry
> > > >> >> > > > > > > > > a
> > > >> >> > > > > > > > > >> bit
> > > >> >> > > > > > > > > >> > > > >> to avoid clients failing auth due to
> > > timing
> > > >> >> > issues.
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > > >> * Strongly agreeing on clients not
> > > >> touching ZK
> > > >> >> > > > directly
> > > >> >> > > > > > :)
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > > >> * I liked Ashish's suggestion of
> having
> > > >> just the
> > > >> >> > > > > > controller
> > > >> >> > > > > > > > > >> issue the
> > > >> >> > > > > > > > > >> > > > >> delegation tokens, to avoid syncing a
> > > >> shared
> > > >> >> > > secret.
> > > >> >> > > > > Not
> > > >> >> > > > > > > sure
> > > >> >> > > > > > > > > if
> > > >> >> > > > > > > > > >> we
> > > >> >> > > > > > > > > >> > > > >> want to continue the discussion here
> or
> > > on
> > > >> the
> > > >> >> > > wiki.
> > > >> >> > > > I
> > > >> >> > > > > > > think
> > > >> >> > > > > > > > > >> that we
> > > >> >> > > > > > > > > >> > > > >> can decouple the problem of "token
> > > >> distribution"
> > > >> >> > > from
> > > >> >> > > > > > > "shared
> > > >> >> > > > > > > > > >> secret
> > > >> >> > > > > > > > > >> > > > >> distribution" and use the controller
> as
> > > >> the only
> > > >> >> > > > token
> > > >> >> > > > > > > > > generator
> > > >> >> > > > > > > > > >> to
> > > >> >> > > > > > > > > >> > > > >> solve the second issue, while still
> > using
> > > >> ZK
> > > >> >> > async
> > > >> >> > > to
> > > >> >> > > > > > > > > distribute
> > > >> >> > > > > > > > > >> > > > >> tokens.
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > > >> * I am also uncomfortable with
> infinite
> > > >> lifetime
> > > >> >> > of
> > > >> >> > > > > > tokens
> > > >> >> > > > > > > > (and
> > > >> >> > > > > > > > > >> hoped
> > > >> >> > > > > > > > > >> > > > >> to hear from others in the
> community) -
> > > but
> > > >> >> > having
> > > >> >> > > > the
> > > >> >> > > > > > > option
> > > >> >> > > > > > > > > and
> > > >> >> > > > > > > > > >> > > > >> documenting it as less secure, allows
> > > >> users to
> > > >> >> > > > > configure
> > > >> >> > > > > > > > their
> > > >> >> > > > > > > > > >> system
> > > >> >> > > > > > > > > >> > > > >> as the wish.
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > > >> * While I like the idea of forcing
> > > >> kerberos auth
> > > >> >> > > for
> > > >> >> > > > > > > renwal,
> > > >> >> > > > > > > > I
> > > >> >> > > > > > > > > >> think
> > > >> >> > > > > > > > > >> > > > >> it mixes the transport layer the the
> > > >> request
> > > >> >> > > content
> > > >> >> > > > > in a
> > > >> >> > > > > > > > > pretty
> > > >> >> > > > > > > > > >> ugly
> > > >> >> > > > > > > > > >> > > > >> way. Perhaps limiting renewer to
> > > non-owner
> > > >> is
> > > >> >> > > better.
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > > >> Things I'd still like to see:
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > > >> * More detailed explanation on what
> we
> > > >> plan to do
> > > >> >> > > for
> > > >> >> > > > > the
> > > >> >> > > > > > > > java
> > > >> >> > > > > > > > > >> clients
> > > >> >> > > > > > > > > >> > > > >> specifically - new configuration? new
> > > APIs?
> > > >> >> > > > > > > > > >> > > > >> The response for my question on how
> > > >> multiple
> > > >> >> > > > identities
> > > >> >> > > > > > > will
> > > >> >> > > > > > > > be
> > > >> >> > > > > > > > > >> > > > >> handled wasn't super clear to me -
> > AFAIK,
> > > >> we
> > > >> >> > don't
> > > >> >> > > > > > > > authenticate
> > > >> >> > > > > > > > > >> each
> > > >> >> > > > > > > > > >> > > > >> request, we authenticate connections.
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > > >> * Alternatives: Delegation tokens are
> > > only
> > > >> used
> > > >> >> > in
> > > >> >> > > > the
> > > >> >> > > > > > > Hadoop
> > > >> >> > > > > > > > > >> > > > >> ecosystem. I'm wondering if there are
> > > >> >> > alternatives
> > > >> >> > > in
> > > >> >> > > > > > other
> > > >> >> > > > > > > > > >> ecosystems
> > > >> >> > > > > > > > > >> > > > >> (Mesos? Tachyon? Cassandra?) and
> > whether
> > > >> there
> > > >> >> > are
> > > >> >> > > > some
> > > >> >> > > > > > > > > >> advantages
> > > >> >> > > > > > > > > >> > > > >> there.
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > > >> Gwen
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM,
> > Harsha <
> > > >> >> > > > > kafka@harsha.io
> > > >> >> > > > > > >
> > > >> >> > > > > > > > > wrote:
> > > >> >> > > > > > > > > >> > > > >> > Hi Gwen,
> > > >> >> > > > > > > > > >> > > > >> >            Can you look at Parth's
> > last
> > > >> reply.
> > > >> >> > > Does
> > > >> >> > > > > it
> > > >> >> > > > > > > > answer
> > > >> >> > > > > > > > > >> your
> > > >> >> > > > > > > > > >> > > > >> >            concerns.
> > > >> >> > > > > > > > > >> > > > >> >
> > > >> >> > > > > > > > > >> > > > >> > Thanks,
> > > >> >> > > > > > > > > >> > > > >> > Harsha
> > > >> >> > > > > > > > > >> > > > >> >
> > > >> >> > > > > > > > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM,
> > parth
> > > >> >> > > brahmbhatt
> > > >> >> > > > > > wrote:
> > > >> >> > > > > > > > > >> > > > >> >> Thanks for reviewing Gwen. The
> wiki
> > > >> already
> > > >> >> > has
> > > >> >> > > > > > details
> > > >> >> > > > > > > on
> > > >> >> > > > > > > > > >> token
> > > >> >> > > > > > > > > >> > > > >> >> expiration
> > > >> >> > > > > > > > > >> > > > >> >> under token acquisition process
> > > >> >> > > > > > > > > >> > > > >> >> <
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > >
> > > >> >> > > > > > > > > >>
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> > > >> >> > > > > > > > > >> > > > >> >.
> > > >> >> > > > > > > > > >> > > > >> >> Current proposal is that tokens
> will
> > > >> expire
> > > >> >> > > based
> > > >> >> > > > > on a
> > > >> >> > > > > > > > > server
> > > >> >> > > > > > > > > >> side
> > > >> >> > > > > > > > > >> > > > >> >> configuration (default 24 hours)
> > > unless
> > > >> >> > renewed.
> > > >> >> > > > > > Renewal
> > > >> >> > > > > > > > is
> > > >> >> > > > > > > > > >> only
> > > >> >> > > > > > > > > >> > > > allowed
> > > >> >> > > > > > > > > >> > > > >> >> until the max life time of token.
> > > >> >> > Alternatively
> > > >> >> > > we
> > > >> >> > > > > > could
> > > >> >> > > > > > > > > also
> > > >> >> > > > > > > > > >> make
> > > >> >> > > > > > > > > >> > > > that
> > > >> >> > > > > > > > > >> > > > >> >> an
> > > >> >> > > > > > > > > >> > > > >> >> optional param and the server side
> > > >> default can
> > > >> >> > > > serve
> > > >> >> > > > > > as
> > > >> >> > > > > > > > the
> > > >> >> > > > > > > > > >> upper
> > > >> >> > > > > > > > > >> > > > bound.
> > > >> >> > > > > > > > > >> > > > >> >>
> > > >> >> > > > > > > > > >> > > > >> >> To your second point it will be
> done
> > > >> exactly
> > > >> >> > the
> > > >> >> > > > > same
> > > >> >> > > > > > > way
> > > >> >> > > > > > > > we
> > > >> >> > > > > > > > > >> would
> > > >> >> > > > > > > > > >> > > > >> >> support
> > > >> >> > > > > > > > > >> > > > >> >> multiple keytabs. The calling
> client
> > > >> will have
> > > >> >> > > to
> > > >> >> > > > > put
> > > >> >> > > > > > > the
> > > >> >> > > > > > > > > >> tokens it
> > > >> >> > > > > > > > > >> > > > >> wants
> > > >> >> > > > > > > > > >> > > > >> >> to use in the subject instance and
> > > call
> > > >> >> > > > > > produce/consume
> > > >> >> > > > > > > > > inside
> > > >> >> > > > > > > > > >> > > > >> >> subject.doas. Each caller will
> have
> > to
> > > >> keep
> > > >> >> > > track
> > > >> >> > > > of
> > > >> >> > > > > > its
> > > >> >> > > > > > > > own
> > > >> >> > > > > > > > > >> > > > subject. I
> > > >> >> > > > > > > > > >> > > > >> >> will have to look at the code to
> see
> > > if
> > > >> we
> > > >> >> > > support
> > > >> >> > > > > > this
> > > >> >> > > > > > > > > >> feature
> > > >> >> > > > > > > > > >> > > right
> > > >> >> > > > > > > > > >> > > > >> now
> > > >> >> > > > > > > > > >> > > > >> >> but my understanding is delegation
> > > token
> > > >> >> > > shouldn't
> > > >> >> > > > > > need
> > > >> >> > > > > > > > any
> > > >> >> > > > > > > > > >> special
> > > >> >> > > > > > > > > >> > > > >> >> treatment as its just another type
> > of
> > > >> >> > Credential
> > > >> >> > > > in
> > > >> >> > > > > > the
> > > >> >> > > > > > > > > >> subject.
> > > >> >> > > > > > > > > >> > > > >> >>
> > > >> >> > > > > > > > > >> > > > >> >> I would also like to know what is
> > your
> > > >> opinion
> > > >> >> > > > about
> > > >> >> > > > > > > > > infinite
> > > >> >> > > > > > > > > >> > > renewal
> > > >> >> > > > > > > > > >> > > > >> (my
> > > >> >> > > > > > > > > >> > > > >> >> recommendation is to not support
> > > this),
> > > >> tokens
> > > >> >> > > > > > renewing
> > > >> >> > > > > > > > them
> > > >> >> > > > > > > > > >> > > self(my
> > > >> >> > > > > > > > > >> > > > >> >> recommendation is to not support
> > this)
> > > >> and
> > > >> >> > most
> > > >> >> > > > > > > > importantly
> > > >> >> > > > > > > > > >> your
> > > >> >> > > > > > > > > >> > > > choice
> > > >> >> > > > > > > > > >> > > > >> >> between the alternatives listed on
> > > this
> > > >> thread
> > > >> >> > > > > > > > > >> > > > >> >> <
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > >
> > > >> >> > > > > > > > > >>
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >>
> > >
> >
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> > > >> >> > > > > > > > > >> > > > >> >
> > > >> >> > > > > > > > > >> > > > >> >> ( I am leaning towards the
> > > >> alternative-2 minus
> > > >> >> > > > > > > controller
> > > >> >> > > > > > > > > >> > > > distributing
> > > >> >> > > > > > > > > >> > > > >> >> secret). Thanks again for
> reviewing.
> > > >> >> > > > > > > > > >> > > > >> >>
> > > >> >> > > > > > > > > >> > > > >> >> Thanks
> > > >> >> > > > > > > > > >> > > > >> >> Parth
> > > >> >> > > > > > > > > >> > > > >> >>
> > > >> >> > > > > > > > > >> > > > >> >>
> > > >> >> > > > > > > > > >> > > > >> >>
> > > >> >> > > > > > > > > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM,
> Gwen
> > > >> Shapira <
> > > >> >> > > > > > > > > >> gwen@confluent.io>
> > > >> >> > > > > > > > > >> > > > wrote:
> > > >> >> > > > > > > > > >> > > > >> >>
> > > >> >> > > > > > > > > >> > > > >> >> > Harsha,
> > > >> >> > > > > > > > > >> > > > >> >> >
> > > >> >> > > > > > > > > >> > > > >> >> > I was thinking of the Rest
> Proxy.
> > I
> > > >> didn't
> > > >> >> > see
> > > >> >> > > > > your
> > > >> >> > > > > > > > design
> > > >> >> > > > > > > > > >> yet,
> > > >> >> > > > > > > > > >> > > > but in
> > > >> >> > > > > > > > > >> > > > >> >> > our proxy, we have a set of
> > > >> producers, which
> > > >> >> > > > will
> > > >> >> > > > > > > serve
> > > >> >> > > > > > > > > >> multiple
> > > >> >> > > > > > > > > >> > > > users
> > > >> >> > > > > > > > > >> > > > >> >> > going through the proxy. Since
> > these
> > > >> users
> > > >> >> > > will
> > > >> >> > > > > have
> > > >> >> > > > > > > > > >> different
> > > >> >> > > > > > > > > >> > > > >> >> > privileges, they'll need to
> > > >> authenticate
> > > >> >> > > > > separately,
> > > >> >> > > > > > > and
> > > >> >> > > > > > > > > >> can't
> > > >> >> > > > > > > > > >> > > > share a
> > > >> >> > > > > > > > > >> > > > >> >> > token.
> > > >> >> > > > > > > > > >> > > > >> >> >
> > > >> >> > > > > > > > > >> > > > >> >> > Am I missing anything?
> > > >> >> > > > > > > > > >> > > > >> >> >
> > > >> >> > > > > > > > > >> > > > >> >> > Gwen
> > > >> >> > > > > > > > > >> > > > >> >> >
> > > >> >> > > > > > > > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM,
> > > >> Harsha <
> > > >> >> > > > > > > kafka@harsha.io
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > >> wrote:
> > > >> >> > > > > > > > > >> > > > >> >> > > Gwen,
> > > >> >> > > > > > > > > >> > > > >> >> > >            On your second
> point.
> > > >> Can you
> > > >> >> > > > > describe
> > > >> >> > > > > > a
> > > >> >> > > > > > > > > >> usecase
> > > >> >> > > > > > > > > >> > > where
> > > >> >> > > > > > > > > >> > > > >> >> > >            mutliple clients
> > ended
> > > up
> > > >> >> > > sharing a
> > > >> >> > > > > > > > producer
> > > >> >> > > > > > > > > >> and
> > > >> >> > > > > > > > > >> > > even
> > > >> >> > > > > > > > > >> > > > if
> > > >> >> > > > > > > > > >> > > > >> they
> > > >> >> > > > > > > > > >> > > > >> >> > >            do why can't they
> not
> > > use
> > > >> >> > single
> > > >> >> > > > > token
> > > >> >> > > > > > > that
> > > >> >> > > > > > > > > >> producer
> > > >> >> > > > > > > > > >> > > > >> >> > >            captures. Why would
> > we
> > > >> need
> > > >> >> > > > multiple
> > > >> >> > > > > > > > clients
> > > >> >> > > > > > > > > >> with
> > > >> >> > > > > > > > > >> > > > >> different
> > > >> >> > > > > > > > > >> > > > >> >> > >            tokens sharing a
> > single
> > > >> >> > instance
> > > >> >> > > of
> > > >> >> > > > > > > > producer.
> > > >> >> > > > > > > > > >> Also
> > > >> >> > > > > > > > > >> > > in
> > > >> >> > > > > > > > > >> > > > >> this
> > > >> >> > > > > > > > > >> > > > >> >> > >            case other clients
> > have
> > > >> access
> > > >> >> > > all
> > > >> >> > > > > the
> > > >> >> > > > > > > > tokens
> > > >> >> > > > > > > > > >> no?
> > > >> >> > > > > > > > > >> > > > >> >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > > Thanks,
> > > >> >> > > > > > > > > >> > > > >> >> > > Harsha
> > > >> >> > > > > > > > > >> > > > >> >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > > On Tue, May 3, 2016, at 11:49
> > AM,
> > > >> Gwen
> > > >> >> > > Shapira
> > > >> >> > > > > > > wrote:
> > > >> >> > > > > > > > > >> > > > >> >> > >> Sorry for the delay:
> > > >> >> > > > > > > > > >> > > > >> >> > >>
> > > >> >> > > > > > > > > >> > > > >> >> > >> Two questions that we didn't
> > see
> > > >> in the
> > > >> >> > > wiki:
> > > >> >> > > > > > > > > >> > > > >> >> > >> 1. Is there an expiration for
> > > >> delegation
> > > >> >> > > > > tokens?
> > > >> >> > > > > > > > > >> Renewal? How
> > > >> >> > > > > > > > > >> > > > do we
> > > >> >> > > > > > > > > >> > > > >> >> > >> revoke them?
> > > >> >> > > > > > > > > >> > > > >> >> > >> 2. If we want to use
> delegation
> > > >> tokens
> > > >> >> > for
> > > >> >> > > > > > "do-as"
> > > >> >> > > > > > > > > (say,
> > > >> >> > > > > > > > > >> > > submit
> > > >> >> > > > > > > > > >> > > > >> Storm
> > > >> >> > > > > > > > > >> > > > >> >> > >> job as my user), we will
> need a
> > > >> producer
> > > >> >> > > for
> > > >> >> > > > > > every
> > > >> >> > > > > > > > job
> > > >> >> > > > > > > > > >> (we
> > > >> >> > > > > > > > > >> > > can't
> > > >> >> > > > > > > > > >> > > > >> share
> > > >> >> > > > > > > > > >> > > > >> >> > >> them between multiple jobs
> > > running
> > > >> on
> > > >> >> > same
> > > >> >> > > > > node),
> > > >> >> > > > > > > > since
> > > >> >> > > > > > > > > >> we
> > > >> >> > > > > > > > > >> > > only
> > > >> >> > > > > > > > > >> > > > >> >> > >> authenticate when connecting.
> > Is
> > > >> there a
> > > >> >> > > plan
> > > >> >> > > > > to
> > > >> >> > > > > > > > change
> > > >> >> > > > > > > > > >> this
> > > >> >> > > > > > > > > >> > > for
> > > >> >> > > > > > > > > >> > > > >> >> > >> delegation tokens, in order
> to
> > > >> allow
> > > >> >> > > multiple
> > > >> >> > > > > > users
> > > >> >> > > > > > > > > with
> > > >> >> > > > > > > > > >> > > > different
> > > >> >> > > > > > > > > >> > > > >> >> > >> tokens to share a client?
> > > >> >> > > > > > > > > >> > > > >> >> > >>
> > > >> >> > > > > > > > > >> > > > >> >> > >> Gwen
> > > >> >> > > > > > > > > >> > > > >> >> > >>
> > > >> >> > > > > > > > > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12
> AM,
> > > >> parth
> > > >> >> > > > > brahmbhatt
> > > >> >> > > > > > > > > >> > > > >> >> > >> <br...@gmail.com>
> > > >> wrote:
> > > >> >> > > > > > > > > >> > > > >> >> > >> > Bumping this up one more
> > time,
> > > >> can
> > > >> >> > other
> > > >> >> > > > > > > committers
> > > >> >> > > > > > > > > >> review?
> > > >> >> > > > > > > > > >> > > > >> >> > >> >
> > > >> >> > > > > > > > > >> > > > >> >> > >> > Thanks
> > > >> >> > > > > > > > > >> > > > >> >> > >> > Parth
> > > >> >> > > > > > > > > >> > > > >> >> > >> >
> > > >> >> > > > > > > > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at
> 9:07
> > > AM,
> > > >> >> > Harsha <
> > > >> >> > > > > > > > > >> kafka@harsha.io>
> > > >> >> > > > > > > > > >> > > > wrote:
> > > >> >> > > > > > > > > >> > > > >> >> > >> >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> Parth,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >>           Overall current
> > > >> design looks
> > > >> >> > > > good
> > > >> >> > > > > to
> > > >> >> > > > > > > > me. I
> > > >> >> > > > > > > > > >> am +1
> > > >> >> > > > > > > > > >> > > on
> > > >> >> > > > > > > > > >> > > > >> the
> > > >> >> > > > > > > > > >> > > > >> >> > KIP.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> Gwen , Jun can you review
> > this
> > > >> as
> > > >> >> > well.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> -Harsha
> > > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at
> > 09:57
> > > >> AM,
> > > >> >> > parth
> > > >> >> > > > > > > > brahmbhatt
> > > >> >> > > > > > > > > >> wrote:
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks for review
> > Jitendra.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > I don't like the idea of
> > > >> infinite
> > > >> >> > > > lifetime
> > > >> >> > > > > > > but I
> > > >> >> > > > > > > > > >> see the
> > > >> >> > > > > > > > > >> > > > >> Streaming
> > > >> >> > > > > > > > > >> > > > >> >> > use
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > case. Even for Streaming
> > use
> > > >> case I
> > > >> >> > > was
> > > >> >> > > > > > hoping
> > > >> >> > > > > > > > > >> there will
> > > >> >> > > > > > > > > >> > > > be
> > > >> >> > > > > > > > > >> > > > >> some
> > > >> >> > > > > > > > > >> > > > >> >> > notion
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > of
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > master/driver that can
> get
> > > new
> > > >> >> > > > delegation
> > > >> >> > > > > > > tokens
> > > >> >> > > > > > > > > at
> > > >> >> > > > > > > > > >> fixed
> > > >> >> > > > > > > > > >> > > > >> interval
> > > >> >> > > > > > > > > >> > > > >> >> > and
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > distribute to workers.
> If
> > > >> that is
> > > >> >> > not
> > > >> >> > > > the
> > > >> >> > > > > > case
> > > >> >> > > > > > > > for
> > > >> >> > > > > > > > > >> we can
> > > >> >> > > > > > > > > >> > > > >> discuss
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > delegation tokens
> renewing
> > > >> them self
> > > >> >> > > and
> > > >> >> > > > > the
> > > >> >> > > > > > > > > >> security
> > > >> >> > > > > > > > > >> > > > >> implications
> > > >> >> > > > > > > > > >> > > > >> >> > of the
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > same.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > I did not want clients
> to
> > > >> fetch
> > > >> >> > tokens
> > > >> >> > > > > from
> > > >> >> > > > > > > > > >> zookeeper,
> > > >> >> > > > > > > > > >> > > > >> overall I
> > > >> >> > > > > > > > > >> > > > >> >> > think
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > its
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > better if clients don't
> > rely
> > > >> on our
> > > >> >> > > > > metadata
> > > >> >> > > > > > > > store
> > > >> >> > > > > > > > > >> and I
> > > >> >> > > > > > > > > >> > > > >> think we
> > > >> >> > > > > > > > > >> > > > >> >> > are
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > moving in that direction
> > > with
> > > >> all
> > > >> >> > the
> > > >> >> > > > > KIP-4
> > > >> >> > > > > > > > > >> improvements.
> > > >> >> > > > > > > > > >> > > > I
> > > >> >> > > > > > > > > >> > > > >> chose
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as in this
> case
> > > the
> > > >> client
> > > >> >> > > > will
> > > >> >> > > > > > > still
> > > >> >> > > > > > > > > >> just talk
> > > >> >> > > > > > > > > >> > > > to
> > > >> >> > > > > > > > > >> > > > >> >> > broker , its
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > the brokers that will
> use
> > > >> zookeeper
> > > >> >> > > > which
> > > >> >> > > > > we
> > > >> >> > > > > > > > > >> already do
> > > >> >> > > > > > > > > >> > > > for a
> > > >> >> > > > > > > > > >> > > > >> lot
> > > >> >> > > > > > > > > >> > > > >> >> > of
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > other
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > usecases + ease of
> > > >> development + and
> > > >> >> > > the
> > > >> >> > > > > > > ability
> > > >> >> > > > > > > > > so
> > > >> >> > > > > > > > > >> > > tokens
> > > >> >> > > > > > > > > >> > > > >> will
> > > >> >> > > > > > > > > >> > > > >> >> > survive
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > even a rolling
> > > restart/cluster
> > > >> >> > > failure.
> > > >> >> > > > > if a
> > > >> >> > > > > > > > > >> majority
> > > >> >> > > > > > > > > >> > > > agrees
> > > >> >> > > > > > > > > >> > > > >> the
> > > >> >> > > > > > > > > >> > > > >> >> > added
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > complexity to have
> > > controller
> > > >> >> > > forwarding
> > > >> >> > > > > > keys
> > > >> >> > > > > > > to
> > > >> >> > > > > > > > > all
> > > >> >> > > > > > > > > >> > > > broker is
> > > >> >> > > > > > > > > >> > > > >> >> > justified
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > as
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > it provides tighter
> > security
> > > >> , I am
> > > >> >> > > fine
> > > >> >> > > > > > with
> > > >> >> > > > > > > > that
> > > >> >> > > > > > > > > >> option
> > > >> >> > > > > > > > > >> > > > too.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > Given zookeeper does not
> > > >> support SSL
> > > >> >> > > we
> > > >> >> > > > > can
> > > >> >> > > > > > > not
> > > >> >> > > > > > > > > >> store
> > > >> >> > > > > > > > > >> > > > master
> > > >> >> > > > > > > > > >> > > > >> keys
> > > >> >> > > > > > > > > >> > > > >> >> > in
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as master keys
> > > will
> > > >> be
> > > >> >> > > exposed
> > > >> >> > > > > on
> > > >> >> > > > > > > > wire.
> > > >> >> > > > > > > > > To
> > > >> >> > > > > > > > > >> > > > support
> > > >> >> > > > > > > > > >> > > > >> >> > rotation
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > without affecting
> current
> > > >> clients is
> > > >> >> > > > > > > something I
> > > >> >> > > > > > > > > >> need to
> > > >> >> > > > > > > > > >> > > > put
> > > >> >> > > > > > > > > >> > > > >> more
> > > >> >> > > > > > > > > >> > > > >> >> > thought
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > in. My current proposal
> > > >> assumes the
> > > >> >> > > > > rotation
> > > >> >> > > > > > > > will
> > > >> >> > > > > > > > > >> > > > invalidate
> > > >> >> > > > > > > > > >> > > > >> all
> > > >> >> > > > > > > > > >> > > > >> >> > current
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > tokens.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > I request committers to
> > also
> > > >> review
> > > >> >> > > and
> > > >> >> > > > > post
> > > >> >> > > > > > > > their
> > > >> >> > > > > > > > > >> > > comments
> > > >> >> > > > > > > > > >> > > > >> so we
> > > >> >> > > > > > > > > >> > > > >> >> > can
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > make
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > progress on this KIP.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > Parth
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at
> > 8:39
> > > >> AM,
> > > >> >> > > Ashish
> > > >> >> > > > > > Singh
> > > >> >> > > > > > > <
> > > >> >> > > > > > > > > >> > > > >> asingh@cloudera.com
> > > >> >> > > > > > > > > >> > > > >> >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > wrote:
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016
> at
> > > >> 11:26 AM,
> > > >> >> > > > > Harsha <
> > > >> >> > > > > > > > > >> > > > kafka@harsha.io>
> > > >> >> > > > > > > > > >> > > > >> >> > wrote:
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Unifying the two
> > > >> discussion
> > > >> >> > > threads
> > > >> >> > > > on
> > > >> >> > > > > > > this
> > > >> >> > > > > > > > > KIP.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Here is the response
> > > from
> > > >> >> > Jitendra
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > "The need for a
> large
> > > >> number of
> > > >> >> > > > > clients
> > > >> >> > > > > > > that
> > > >> >> > > > > > > > > are
> > > >> >> > > > > > > > > >> > > > running
> > > >> >> > > > > > > > > >> > > > >> all
> > > >> >> > > > > > > > > >> > > > >> >> > over the
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > cluster that
> > > authenticate
> > > >> with
> > > >> >> > > Kafka
> > > >> >> > > > > > > > brokers,
> > > >> >> > > > > > > > > >> is very
> > > >> >> > > > > > > > > >> > > > >> similar
> > > >> >> > > > > > > > > >> > > > >> >> > to the
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Hadoop use case of
> > large
> > > >> number
> > > >> >> > of
> > > >> >> > > > > tasks
> > > >> >> > > > > > > > > running
> > > >> >> > > > > > > > > >> > > across
> > > >> >> > > > > > > > > >> > > > >> the
> > > >> >> > > > > > > > > >> > > > >> >> > cluster
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> that
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need authentication
> to
> > > >> Hdfs
> > > >> >> > > > Namenode.
> > > >> >> > > > > > > > > >> Therefore, the
> > > >> >> > > > > > > > > >> > > > >> >> > delegation token
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > approach does seem
> > like
> > > a
> > > >> good
> > > >> >> > fit
> > > >> >> > > > for
> > > >> >> > > > > > > this
> > > >> >> > > > > > > > > use
> > > >> >> > > > > > > > > >> case
> > > >> >> > > > > > > > > >> > > > as we
> > > >> >> > > > > > > > > >> > > > >> >> > have seen
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> it
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > working at large
> scale
> > > in
> > > >> HDFS
> > > >> >> > and
> > > >> >> > > > > YARN.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >   The proposed
> design
> > is
> > > >> very
> > > >> >> > much
> > > >> >> > > > > > inline
> > > >> >> > > > > > > > with
> > > >> >> > > > > > > > > >> Hadoop
> > > >> >> > > > > > > > > >> > > > >> >> > approach. A few
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >   comments:
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 1) Why do you guys
> > want
> > > >> to allow
> > > >> >> > > > > > infinite
> > > >> >> > > > > > > > > >> renewable
> > > >> >> > > > > > > > > >> > > > >> lifetime
> > > >> >> > > > > > > > > >> > > > >> >> > for a
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > token? HDFS
> restricts
> > a
> > > >> token
> > > >> >> > to a
> > > >> >> > > > max
> > > >> >> > > > > > > life
> > > >> >> > > > > > > > > time
> > > >> >> > > > > > > > > >> > > > (default
> > > >> >> > > > > > > > > >> > > > >> 7
> > > >> >> > > > > > > > > >> > > > >> >> > days).  A
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > token's
> vulnerability
> > is
> > > >> >> > believed
> > > >> >> > > to
> > > >> >> > > > > > > > increase
> > > >> >> > > > > > > > > >> with
> > > >> >> > > > > > > > > >> > > > time.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > I agree that having
> > > infinite
> > > >> >> > > lifetime
> > > >> >> > > > > > might
> > > >> >> > > > > > > > not
> > > >> >> > > > > > > > > >> be the
> > > >> >> > > > > > > > > >> > > > best
> > > >> >> > > > > > > > > >> > > > >> idea.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 2) As I understand
> the
> > > >> tokens
> > > >> >> > are
> > > >> >> > > > > stored
> > > >> >> > > > > > > in
> > > >> >> > > > > > > > > >> zookeeper
> > > >> >> > > > > > > > > >> > > > as
> > > >> >> > > > > > > > > >> > > > >> well,
> > > >> >> > > > > > > > > >> > > > >> >> > and
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> can
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > be updated there.
> This
> > > is
> > > >> clever
> > > >> >> > > as
> > > >> >> > > > it
> > > >> >> > > > > > can
> > > >> >> > > > > > > > > allow
> > > >> >> > > > > > > > > >> > > > >> replacing the
> > > >> >> > > > > > > > > >> > > > >> >> > tokens
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > once they run out of
> > max
> > > >> life
> > > >> >> > > time,
> > > >> >> > > > > and
> > > >> >> > > > > > > > > clients
> > > >> >> > > > > > > > > >> can
> > > >> >> > > > > > > > > >> > > > >> download
> > > >> >> > > > > > > > > >> > > > >> >> > new
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> tokens
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from zookeeper. It
> > > >> shouldn't be
> > > >> >> > a
> > > >> >> > > > big
> > > >> >> > > > > > load
> > > >> >> > > > > > > > on
> > > >> >> > > > > > > > > >> > > zookeeper
> > > >> >> > > > > > > > > >> > > > >> as a
> > > >> >> > > > > > > > > >> > > > >> >> > client
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> will
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need to get a new
> > token
> > > >> once in
> > > >> >> > > > > several
> > > >> >> > > > > > > > days.
> > > >> >> > > > > > > > > >> In this
> > > >> >> > > > > > > > > >> > > > >> approach
> > > >> >> > > > > > > > > >> > > > >> >> > you
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> don't
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need infinite
> lifetime
> > > on
> > > >> the
> > > >> >> > > token
> > > >> >> > > > > even
> > > >> >> > > > > > > for
> > > >> >> > > > > > > > > >> long
> > > >> >> > > > > > > > > >> > > > running
> > > >> >> > > > > > > > > >> > > > >> >> > clients.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 3) The token
> password
> > > are
> > > >> >> > > generated
> > > >> >> > > > > > using
> > > >> >> > > > > > > a
> > > >> >> > > > > > > > > >> master
> > > >> >> > > > > > > > > >> > > key.
> > > >> >> > > > > > > > > >> > > > >> The
> > > >> >> > > > > > > > > >> > > > >> >> > master
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> key
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > should also be
> > > >> periodically
> > > >> >> > > changed.
> > > >> >> > > > > In
> > > >> >> > > > > > > > > Hadoop,
> > > >> >> > > > > > > > > >> the
> > > >> >> > > > > > > > > >> > > > >> default
> > > >> >> > > > > > > > > >> > > > >> >> > renewal
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > period is 1 day.?
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > IIUC, this will
> require
> > > >> brokers
> > > >> >> > > > > > maintaining
> > > >> >> > > > > > > a
> > > >> >> > > > > > > > > >> list of X
> > > >> >> > > > > > > > > >> > > > most
> > > >> >> > > > > > > > > >> > > > >> >> > recent
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> master
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > keys. This list will
> > have
> > > >> to be
> > > >> >> > > > > persisted
> > > >> >> > > > > > > > > >> somewhere, as
> > > >> >> > > > > > > > > >> > > > if a
> > > >> >> > > > > > > > > >> > > > >> >> > broker
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> goes
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > down it will have to
> get
> > > >> that list
> > > >> >> > > > again
> > > >> >> > > > > > and
> > > >> >> > > > > > > > > >> storing
> > > >> >> > > > > > > > > >> > > > master
> > > >> >> > > > > > > > > >> > > > >> keys
> > > >> >> > > > > > > > > >> > > > >> >> > on ZK
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> is
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > not the best idea.
> > > However,
> > > >> if a
> > > >> >> > > > broker
> > > >> >> > > > > > goes
> > > >> >> > > > > > > > > down
> > > >> >> > > > > > > > > >> then
> > > >> >> > > > > > > > > >> > > we
> > > >> >> > > > > > > > > >> > > > >> have
> > > >> >> > > > > > > > > >> > > > >> >> > much
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> bigger
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > issue to deal with and
> > > >> client can
> > > >> >> > > > always
> > > >> >> > > > > > > > > >> > > re-authenticate
> > > >> >> > > > > > > > > >> > > > is
> > > >> >> > > > > > > > > >> > > > >> such
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> events.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > Did you happen to
> take a
> > > >> look at
> > > >> >> > > other
> > > >> >> > > > > > > > > >> alternatives
> > > >> >> > > > > > > > > >> > > this
> > > >> >> > > > > > > > > >> > > > >> list has
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > suggested?
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Thanks for a
> thorough
> > > >> proposal,
> > > >> >> > > > great
> > > >> >> > > > > > > work!"
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016,
> > at
> > > >> 10:28
> > > >> >> > PM,
> > > >> >> > > > Gwen
> > > >> >> > > > > > > > Shapira
> > > >> >> > > > > > > > > >> wrote:
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > Makes sense to me.
> > > >> Thanks!
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7,
> 2016
> > at
> > > >> 9:25
> > > >> >> > PM,
> > > >> >> > > > > > Harsha <
> > > >> >> > > > > > > > > >> > > > kafka@harsha.io
> > > >> >> > > > > > > > > >> > > > >> >
> > > >> >> > > > > > > > > >> > > > >> >> > wrote:
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > It doesn't need
> > any
> > > >> release
> > > >> >> > > > > vehicle
> > > >> >> > > > > > > but
> > > >> >> > > > > > > > > >> still the
> > > >> >> > > > > > > > > >> > > > >> work can
> > > >> >> > > > > > > > > >> > > > >> >> > move
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > forward.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > If anyone is
> > > >> interested in
> > > >> >> > the
> > > >> >> > > > KIP
> > > >> >> > > > > > > > please
> > > >> >> > > > > > > > > >> do the
> > > >> >> > > > > > > > > >> > > > >> review and
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> provide
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > the
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > comments.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > -Harsha
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7,
> > 2016,
> > > >> at
> > > >> >> > 04:59
> > > >> >> > > > PM,
> > > >> >> > > > > > > Ismael
> > > >> >> > > > > > > > > >> Juma
> > > >> >> > > > > > > > > >> > > > wrote:
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> I agree that it
> > > >> would be
> > > >> >> > good
> > > >> >> > > > to
> > > >> >> > > > > > have
> > > >> >> > > > > > > > > more
> > > >> >> > > > > > > > > >> time
> > > >> >> > > > > > > > > >> > > to
> > > >> >> > > > > > > > > >> > > > >> review
> > > >> >> > > > > > > > > >> > > > >> >> > and
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > discuss
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> KIP-48.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> Ismael
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8,
> > 2016
> > > >> at
> > > >> >> > 12:55
> > > >> >> > > > AM,
> > > >> >> > > > > > Gwen
> > > >> >> > > > > > > > > >> Shapira <
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> gwen@confluent.io>
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Hi Team,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Since KIP-48
> > > >> depends on
> > > >> >> > > > KIP-43,
> > > >> >> > > > > > > which
> > > >> >> > > > > > > > > is
> > > >> >> > > > > > > > > >> > > > already a
> > > >> >> > > > > > > > > >> > > > >> bit
> > > >> >> > > > > > > > > >> > > > >> >> > of a
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> risk
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > for
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > the next
> > release
> > > -
> > > >> any
> > > >> >> > > chance
> > > >> >> > > > > we
> > > >> >> > > > > > > can
> > > >> >> > > > > > > > > >> delay
> > > >> >> > > > > > > > > >> > > > >> delegation
> > > >> >> > > > > > > > > >> > > > >> >> > tokens
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> to
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Kafka
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > 0.10.1?
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > With the
> > > community
> > > >> >> > working
> > > >> >> > > > on a
> > > >> >> > > > > > > > release
> > > >> >> > > > > > > > > >> every
> > > >> >> > > > > > > > > >> > > 3
> > > >> >> > > > > > > > > >> > > > >> month,
> > > >> >> > > > > > > > > >> > > > >> >> > this
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> is not
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a huge
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > delay.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Gwen
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb
> 26,
> > > >> 2016 at
> > > >> >> > > 5:11
> > > >> >> > > > > PM,
> > > >> >> > > > > > > > Ashish
> > > >> >> > > > > > > > > >> Singh
> > > >> >> > > > > > > > > >> > > <
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > asingh@cloudera.com>
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Parth,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Thanks
> again
> > > for
> > > >> the
> > > >> >> > > > awesome
> > > >> >> > > > > > > write
> > > >> >> > > > > > > > > up.
> > > >> >> > > > > > > > > >> > > > Following
> > > >> >> > > > > > > > > >> > > > >> our
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> discussion
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from the
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I
> think
> > > it
> > > >> will
> > > >> >> > be
> > > >> >> > > > > easier
> > > >> >> > > > > > > to
> > > >> >> > > > > > > > > >> compare
> > > >> >> > > > > > > > > >> > > > >> various
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> alternatives
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > if they
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > are
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > listed
> > > together.
> > > >> I am
> > > >> >> > > > stating
> > > >> >> > > > > > > > below a
> > > >> >> > > > > > > > > >> few
> > > >> >> > > > > > > > > >> > > > >> >> > alternatives along
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > with
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a the
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > current
> > > proposal.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Current
> > > >> proposal)
> > > >> >> > Store
> > > >> >> > > > > > > Delegation
> > > >> >> > > > > > > > > >> Token,
> > > >> >> > > > > > > > > >> > > DT,
> > > >> >> > > > > > > > > >> > > > >> on ZK.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1.
> Client
> > > >> >> > > authenticates
> > > >> >> > > > > > with a
> > > >> >> > > > > > > > > >> broker.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once
> a
> > > >> client is
> > > >> >> > > > > > > > authenticated,
> > > >> >> > > > > > > > > >> it
> > > >> >> > > > > > > > > >> > > will
> > > >> >> > > > > > > > > >> > > > >> make a
> > > >> >> > > > > > > > > >> > > > >> >> > broker
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> side
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a
> > > >> delegation
> > > >> >> > > > token.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The
> > > broker
> > > >> >> > > generates
> > > >> >> > > > a
> > > >> >> > > > > > > shared
> > > >> >> > > > > > > > > >> secret
> > > >> >> > > > > > > > > >> > > > based
> > > >> >> > > > > > > > > >> > > > >> on
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > HMAC-SHA256(a
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> Password/Secret
> > > >> >> > shared
> > > >> >> > > > > > between
> > > >> >> > > > > > > > all
> > > >> >> > > > > > > > > >> > > brokers,
> > > >> >> > > > > > > > > >> > > > >> >> > randomly
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > generated
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > tokenId).
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4.
> Broker
> > > >> stores
> > > >> >> > this
> > > >> >> > > > > token
> > > >> >> > > > > > in
> > > >> >> > > > > > > > its
> > > >> >> > > > > > > > > >> in
> > > >> >> > > > > > > > > >> > > > memory
> > > >> >> > > > > > > > > >> > > > >> cache.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> Broker
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > also stores
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the
> > > >> DelegationToken
> > > >> >> > > > > without
> > > >> >> > > > > > > the
> > > >> >> > > > > > > > > >> hmac in
> > > >> >> > > > > > > > > >> > > the
> > > >> >> > > > > > > > > >> > > > >> >> > zookeeper.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. All
> > > >> brokers will
> > > >> >> > > > have a
> > > >> >> > > > > > > cache
> > > >> >> > > > > > > > > >> backed
> > > >> >> > > > > > > > > >> > > by
> > > >> >> > > > > > > > > >> > > > >> >> > zookeeper so
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> they
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > will all
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    get
> > notified
> > > >> >> > whenever
> > > >> >> > > a
> > > >> >> > > > > new
> > > >> >> > > > > > > > token
> > > >> >> > > > > > > > > is
> > > >> >> > > > > > > > > >> > > > >> generated and
> > > >> >> > > > > > > > > >> > > > >> >> > they
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> will
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > update
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    local
> > cache
> > > >> whenever
> > > >> >> > > > token
> > > >> >> > > > > > > state
> > > >> >> > > > > > > > > >> changes.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6.
> Broker
> > > >> returns
> > > >> >> > the
> > > >> >> > > > > token
> > > >> >> > > > > > to
> > > >> >> > > > > > > > > >> Client.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable
> > issues
> > > >> and
> > > >> >> > fixes
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1.
> > Probable
> > > >> race
> > > >> >> > > > > condition,
> > > >> >> > > > > > > > client
> > > >> >> > > > > > > > > >> tries
> > > >> >> > > > > > > > > >> > > to
> > > >> >> > > > > > > > > >> > > > >> >> > authenticate
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> with
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a broker
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    that is
> > yet
> > > >> to be
> > > >> >> > > > updated
> > > >> >> > > > > > with
> > > >> >> > > > > > > > the
> > > >> >> > > > > > > > > >> newly
> > > >> >> > > > > > > > > >> > > > >> generated
> > > >> >> > > > > > > > > >> > > > >> >> > DT.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> This
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > can
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > probably be
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    dealt
> with
> > > >> making
> > > >> >> > > > > dtRequest
> > > >> >> > > > > > > > block
> > > >> >> > > > > > > > > >> until
> > > >> >> > > > > > > > > >> > > all
> > > >> >> > > > > > > > > >> > > > >> >> > brokers have
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > updated
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their DT
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    cache.
> Zk
> > > >> barrier or
> > > >> >> > > > > similar
> > > >> >> > > > > > > > > >> mechanism
> > > >> >> > > > > > > > > >> > > can
> > > >> >> > > > > > > > > >> > > > be
> > > >> >> > > > > > > > > >> > > > >> used.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> However,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > all such
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> mechanisms
> > > >> will
> > > >> >> > > increase
> > > >> >> > > > > > > > > complexity.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2.
> Using a
> > > >> static
> > > >> >> > > secret
> > > >> >> > > > > key
> > > >> >> > > > > > > > from
> > > >> >> > > > > > > > > >> config
> > > >> >> > > > > > > > > >> > > > >> file. Will
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> require
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > yet
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > another
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    config
> and
> > > >> uses a
> > > >> >> > > static
> > > >> >> > > > > > > secret
> > > >> >> > > > > > > > > >> key. It
> > > >> >> > > > > > > > > >> > > is
> > > >> >> > > > > > > > > >> > > > >> advised
> > > >> >> > > > > > > > > >> > > > >> >> > to
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> rotate
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > secret
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > keys
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > periodically.
> > > >> This
> > > >> >> > can
> > > >> >> > > > be
> > > >> >> > > > > > > > avoided
> > > >> >> > > > > > > > > >> with
> > > >> >> > > > > > > > > >> > > > >> controller
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> generating
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > secretKey and
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    passing
> to
> > > >> brokers
> > > >> >> > > > > > > periodically.
> > > >> >> > > > > > > > > >> However,
> > > >> >> > > > > > > > > >> > > > >> this will
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> require
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > brokers to
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maintain
> > > >> certain
> > > >> >> > > counts
> > > >> >> > > > of
> > > >> >> > > > > > > > > >> secretKeys.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> (Alternative
> > 1)
> > > >> Have
> > > >> >> > > > > controller
> > > >> >> > > > > > > > > >> generate
> > > >> >> > > > > > > > > >> > > > >> delegation
> > > >> >> > > > > > > > > >> > > > >> >> > token.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1.
> Client
> > > >> >> > > authenticates
> > > >> >> > > > > > with a
> > > >> >> > > > > > > > > >> broker.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once
> a
> > > >> client is
> > > >> >> > > > > > > > authenticated,
> > > >> >> > > > > > > > > >> it
> > > >> >> > > > > > > > > >> > > will
> > > >> >> > > > > > > > > >> > > > >> make a
> > > >> >> > > > > > > > > >> > > > >> >> > broker
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> side
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a
> > > >> delegation
> > > >> >> > > > token.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3.
> Broker
> > > >> forwards
> > > >> >> > the
> > > >> >> > > > > > request
> > > >> >> > > > > > > > to
> > > >> >> > > > > > > > > >> > > > controller.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4.
> > > Controller
> > > >> >> > > generates
> > > >> >> > > > a
> > > >> >> > > > > DT
> > > >> >> > > > > > > and
> > > >> >> > > > > > > > > >> > > broadcasts
> > > >> >> > > > > > > > > >> > > > >> to all
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> brokers.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5.
> Broker
> > > >> stores
> > > >> >> > this
> > > >> >> > > > > token
> > > >> >> > > > > > in
> > > >> >> > > > > > > > its
> > > >> >> > > > > > > > > >> memory
> > > >> >> > > > > > > > > >> > > > >> cache.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6.
> > > Controller
> > > >> >> > responds
> > > >> >> > > > to
> > > >> >> > > > > > > > broker’s
> > > >> >> > > > > > > > > >> DT
> > > >> >> > > > > > > > > >> > > req.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    7.
> Broker
> > > >> returns
> > > >> >> > the
> > > >> >> > > > > token
> > > >> >> > > > > > to
> > > >> >> > > > > > > > > >> Client.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable
> > issues
> > > >> and
> > > >> >> > fixes
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. We
> will
> > > >> have to
> > > >> >> > add
> > > >> >> > > > new
> > > >> >> > > > > > > APIs
> > > >> >> > > > > > > > to
> > > >> >> > > > > > > > > >> > > support
> > > >> >> > > > > > > > > >> > > > >> >> > controller
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> pushing
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > to
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    brokers
> on
> > > >> top of
> > > >> >> > the
> > > >> >> > > > > > minimal
> > > >> >> > > > > > > > APIs
> > > >> >> > > > > > > > > >> that
> > > >> >> > > > > > > > > >> > > are
> > > >> >> > > > > > > > > >> > > > >> >> > currently
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > proposed.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. We
> will
> > > >> also have
> > > >> >> > > to
> > > >> >> > > > > add
> > > >> >> > > > > > > APIs
> > > >> >> > > > > > > > > to
> > > >> >> > > > > > > > > >> > > support
> > > >> >> > > > > > > > > >> > > > >> the
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> bootstrapping
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > case,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > i.e,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    when a
> new
> > > >> broker
> > > >> >> > > comes
> > > >> >> > > > up
> > > >> >> > > > > > it
> > > >> >> > > > > > > > will
> > > >> >> > > > > > > > > >> have
> > > >> >> > > > > > > > > >> > > to
> > > >> >> > > > > > > > > >> > > > >> get all
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> delegation
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > from
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the
> > > >> controller.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. In
> > > >> catastrophic
> > > >> >> > > > > failures
> > > >> >> > > > > > > > where
> > > >> >> > > > > > > > > >> all
> > > >> >> > > > > > > > > >> > > > brokers
> > > >> >> > > > > > > > > >> > > > >> go
> > > >> >> > > > > > > > > >> > > > >> >> > down,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> the
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost
> > even
> > > >> if
> > > >> >> > > servers
> > > >> >> > > > > are
> > > >> >> > > > > > > > > >> restarted as
> > > >> >> > > > > > > > > >> > > > >> tokens
> > > >> >> > > > > > > > > >> > > > >> >> > are not
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this
> > > >> happens,
> > > >> >> > then
> > > >> >> > > > > there
> > > >> >> > > > > > > are
> > > >> >> > > > > > > > > more
> > > >> >> > > > > > > > > >> > > > important
> > > >> >> > > > > > > > > >> > > > >> >> > things to
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > worry
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > about
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it
> > is
> > > >> better
> > > >> >> > to
> > > >> >> > > > > > > > > >> re-authenticate.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> (Alternative
> > 2)
> > > >> Do not
> > > >> >> > > > > > distribute
> > > >> >> > > > > > > > DT
> > > >> >> > > > > > > > > to
> > > >> >> > > > > > > > > >> > > other
> > > >> >> > > > > > > > > >> > > > >> brokers
> > > >> >> > > > > > > > > >> > > > >> >> > at
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> all.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1.
> Client
> > > >> >> > > authenticates
> > > >> >> > > > > > with a
> > > >> >> > > > > > > > > >> broker.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once
> a
> > > >> client is
> > > >> >> > > > > > > > authenticated,
> > > >> >> > > > > > > > > >> it
> > > >> >> > > > > > > > > >> > > will
> > > >> >> > > > > > > > > >> > > > >> make a
> > > >> >> > > > > > > > > >> > > > >> >> > broker
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> side
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a
> > > >> delegation
> > > >> >> > > > token.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The
> > > broker
> > > >> >> > > generates
> > > >> >> > > > DT
> > > >> >> > > > > > of
> > > >> >> > > > > > > > > form,
> > > >> >> > > > > > > > > >> > > [hmac +
> > > >> >> > > > > > > > > >> > > > >> (owner,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> renewer,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > maxLifeTime,
> > > >> id,
> > > >> >> > hmac,
> > > >> >> > > > > > > > > >> expirationTime)]
> > > >> >> > > > > > > > > >> > > and
> > > >> >> > > > > > > > > >> > > > >> passes
> > > >> >> > > > > > > > > >> > > > >> >> > back
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> this
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > DT to
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > client.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    hmac is
> > > >> generated
> > > >> >> > via
> > > >> >> > > > > > > > > >> {HMAC-SHA256(owner,
> > > >> >> > > > > > > > > >> > > > >> renewer,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > maxLifeTime, id,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> expirationTime)
> > > >> >> > using
> > > >> >> > > > > > > > SecretKey}.
> > > >> >> > > > > > > > > >> Note
> > > >> >> > > > > > > > > >> > > that
> > > >> >> > > > > > > > > >> > > > >> all
> > > >> >> > > > > > > > > >> > > > >> >> > brokers
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> have
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > this
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > SecretKey.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4.
> Client
> > > >> then goes
> > > >> >> > to
> > > >> >> > > > any
> > > >> >> > > > > > > > broker
> > > >> >> > > > > > > > > >> and to
> > > >> >> > > > > > > > > >> > > > >> >> > authenticate
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> sends
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > the DT.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    Broker
> > > >> recalculates
> > > >> >> > > hmac
> > > >> >> > > > > > using
> > > >> >> > > > > > > > > >> (owner,
> > > >> >> > > > > > > > > >> > > > >> renewer,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> maxLifeTime,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > id, hmac,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> expirationTime) info
> > > >> >> > > > from
> > > >> >> > > > > DT
> > > >> >> > > > > > > and
> > > >> >> > > > > > > > > its
> > > >> >> > > > > > > > > >> > > > >> SecretKey. If
> > > >> >> > > > > > > > > >> > > > >> >> > it
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> matches
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > with
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac of
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    DT,
> client
> > > is
> > > >> >> > > > > authenticated.
> > > >> >> > > > > > > > Yes,
> > > >> >> > > > > > > > > >> it will
> > > >> >> > > > > > > > > >> > > > do
> > > >> >> > > > > > > > > >> > > > >> other
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> obvious
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > checks of
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> timestamp
> > > >> expiry and
> > > >> >> > > > such.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Note that
> > > secret
> > > >> key
> > > >> >> > will
> > > >> >> > > > be
> > > >> >> > > > > > > > > generated
> > > >> >> > > > > > > > > >> by
> > > >> >> > > > > > > > > >> > > > >> controller
> > > >> >> > > > > > > > > >> > > > >> >> > and
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> passed
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > to
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > brokers
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> periodically.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable
> > issues
> > > >> and
> > > >> >> > fixes
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. How
> to
> > > >> delete a
> > > >> >> > DT?
> > > >> >> > > > > Yes,
> > > >> >> > > > > > > that
> > > >> >> > > > > > > > > is
> > > >> >> > > > > > > > > >> a
> > > >> >> > > > > > > > > >> > > > downside
> > > >> >> > > > > > > > > >> > > > >> >> > here.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> However,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > this can
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be
> handled
> > > >> with
> > > >> >> > > brokers
> > > >> >> > > > > > > > > maintaining
> > > >> >> > > > > > > > > >> a
> > > >> >> > > > > > > > > >> > > > >> blacklist of
> > > >> >> > > > > > > > > >> > > > >> >> > DTs,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> DTs
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from this
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > list
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    can be
> > > >> removed after
> > > >> >> > > > > expiry.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. In
> > > >> catastrophic
> > > >> >> > > > > failures
> > > >> >> > > > > > > > where
> > > >> >> > > > > > > > > >> all
> > > >> >> > > > > > > > > >> > > > brokers
> > > >> >> > > > > > > > > >> > > > >> go
> > > >> >> > > > > > > > > >> > > > >> >> > down,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> the
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost
> > even
> > > >> if
> > > >> >> > > servers
> > > >> >> > > > > are
> > > >> >> > > > > > > > > >> restarted as
> > > >> >> > > > > > > > > >> > > > >> tokens
> > > >> >> > > > > > > > > >> > > > >> >> > are not
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this
> > > >> happens,
> > > >> >> > then
> > > >> >> > > > > there
> > > >> >> > > > > > > are
> > > >> >> > > > > > > > > more
> > > >> >> > > > > > > > > >> > > > important
> > > >> >> > > > > > > > > >> > > > >> >> > things to
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > worry
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > about
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it
> > is
> > > >> better
> > > >> >> > to
> > > >> >> > > > > > > > > >> re-authenticate.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb
> > 26,
> > > >> 2016 at
> > > >> >> > > > 1:58
> > > >> >> > > > > > PM,
> > > >> >> > > > > > > > > Parth
> > > >> >> > > > > > > > > >> > > > >> Brahmbhatt <
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > pbrahmbhatt@hortonworks.com>
> > > >> >> > > > > > > > wrote:
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Hi,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> I have
> filed
> > > >> KIP-48 so
> > > >> >> > > we
> > > >> >> > > > > can
> > > >> >> > > > > > > > offer
> > > >> >> > > > > > > > > >> hadoop
> > > >> >> > > > > > > > > >> > > > like
> > > >> >> > > > > > > > > >> > > > >> >> > delegation
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens in
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> kafka. You
> > can
> > > >> review
> > > >> >> > > the
> > > >> >> > > > > > design
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > > >> >> > > > > > > > > >> > > > >> >> >
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > >
> > > >> >> > > > > > > > > >>
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > .
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> This KIP
> > > >> depends on
> > > >> >> > > KIP-43
> > > >> >> > > > > and
> > > >> >> > > > > > > we
> > > >> >> > > > > > > > > >> have also
> > > >> >> > > > > > > > > >> > > > >> >> > discussed an
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > alternative to
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> proposed
> > > design
> > > >> here<
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > > >> >> > > > > > > > > >> > > > >> >> >
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > >
> > > >> >> > > > > > > > > >>
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >>
> > >
> >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >.
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Thanks
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Parth
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > --
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Regards,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Ashish
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > --
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > Regards,
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > > Ashish
> > > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > > >> >> > > > > > > > > >> > > > >> >> >
> > > >> >> > > > > > > > > >> > > > >>
> > > >> >> > > > > > > > > >> > > >
> > > >> >> > > > > > > > > >> > >
> > > >> >> > > > > > > > > >>
> > > >> >> > > > > > > > > >>
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > >
> > > >> >> > > > > >
> > > >> >> > > > > >
> > > >> >> > > > > > --
> > > >> >> > > > > > Regards,
> > > >> >> > > > > >
> > > >> >> > > > > > Rajini
> > > >> >> > > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > > >
> > > >> >> > > >
> > > >> >> > > > --
> > > >> >> > > > Regards,
> > > >> >> > > >
> > > >> >> > > > Rajini
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> --
> > > >> >> Liquan Pei
> > > >> >> Software Engineer, Confluent Inc
> > > >>
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Jun Rao <ju...@confluent.io>]

Parth,

Thanks for the reply. A couple of comments inline below.

On Tue, Jun 21, 2016 at 10:36 AM, parth brahmbhatt <
brahmbhatt.parth@gmail.com> wrote:

> 1. Who / how are tokens renewed? By original requester only? or using
> Kerberos
> auth only?
> My recommendation is to do this only using Kerberos auth and only threw the
> renewer specified during the acquisition request.
>
>
Hmm, not sure that I follow this. Are you saying that any client
authenticated with the delegation token can renew, i.e. there is no renewer
needed?

Also, just to be clear, any authenticated client (either through SASL or
SSL) can request a delegation token for the authenticated user, right?


> 2. Are tokens stored on each broker or in ZK?
> My recommendation is still to store in ZK or not store them at all. The
> whole controller based distribution is too much overhead with not much to
> achieve.
>
> 3. How are tokens invalidated / expired?
> Either by expiration time out or through an explicit request to invalidate.
>
> 4. Which encryption algorithm is used?
> SCRAM
>
> 5. What is the impersonation proposal (it wasn't in the KIP but was
> discussed
> in this thread)?
> There is no imperonation proposal. I tried and explained how its a
> different problem and why its not really necessary to discuss that as part
> of this KIP.  This KIP will not support any impersonation, it will just be
> another way to authenticate.
>
> 6. Do we need new ACLs, if so - for what actions?
> We do not need new ACLs.
>
>
Could we document the format of the new request/response and their
associated Resource and Operation for ACL?


> 7. How would the delegation token be configured in the client?
> Should be through config. I wasn't planning on supporting JAAS for tokens.
> I don't believe hadoop does this either.
>
> Thanks
> Parth
>
>
>
> On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <ju...@confluent.io> wrote:
>
> > Harsha,
> >
> > Another question.
> >
> > 9. How would the delegation token be configured in the client? The
> standard
> > way is to do this through JAAS. However, we will need to think through if
> > this is convenient in a shared environment. For example, when a new task
> is
> > added to a Storm worker node, do we need to dynamically add a new section
> > in the JAAS file? It may be more convenient if we can pass in the token
> > through the config directly w/o going through JAAS.
> >
> > Are you or Parth still actively working on this KIP?
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Just to add on that list.
> > >
> > > 2. It would be good to document the format of the data stored in ZK.
> > > 7. Earlier, there was a discussion on whether the tokens should be
> > > propagated through ZK like config/acl/quota, or through the controller.
> > > Currently, the controller is only designed for propagating topic
> > metadata,
> > > but not other data.
> > > 8. Should we use SCRAM to send the token instead of DIGEST-MD5 since
> it's
> > > deprecated?
> > >
> > > Also, the images in the wiki seem broken.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <gw...@confluent.io>
> > wrote:
> > >
> > >> From what I can see, remaining questions are:
> > >>
> > >> 1. Who / how are tokens renewed? By original requester only? or using
> > >> Kerberos auth only?
> > >> 2. Are tokens stored on each broker or in ZK?
> > >> 3. How are tokens invalidated / expired?
> > >> 4. Which encryption algorithm is used?
> > >> 5. What is the impersonation proposal (it wasn't in the KIP but was
> > >> discussed in this thread)?
> > >> 6. Do we need new ACLs, if so - for what actions?
> > >>
> > >> Gwen
> > >>
> > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <ka...@harsha.io> wrote:
> > >> > Jun & Ismael,
> > >> >                          Unfortunately I couldn't attend the KIP
> > meeting
> > >> >                          when delegation tokens discussed.
> Appreciate
> > if
> > >> >                          you can update the thread if you have any
> > >> >                          further questions.
> > >> > Thanks,
> > >> > Harsha
> > >> >
> > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
> > >> >> It seems that the links to images in the KIP are broken.
> > >> >>
> > >> >> Liquan
> > >> >>
> > >> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
> > >> >> brahmbhatt.parth@gmail.com> wrote:
> > >> >>
> > >> >> > 110. What does getDelegationTokenAs mean?
> > >> >> > In the current proposal we only allow a user to get delegation
> > token
> > >> for
> > >> >> > the identity that it authenticated as using another mechanism,
> i.e.
> > >> A user
> > >> >> > that authenticate using a keytab for principal user1@EXAMPLE.COM
> > >> will get
> > >> >> > delegation tokens for that user only. In future I think we will
> > have
> > >> to
> > >> >> > extend support such that we allow some set of users (
> > >> >> > kafka-rest-user@EXAMPLE.COM, storm-nimbus@EXAMPLE.COM) to
> acquire
> > >> >> > delegation tokens on behalf of other users whose identity they
> have
> > >> >> > verified independently.  Kafka brokers will have ACLs to control
> > >> which
> > >> >> > users are allowed to impersonate other users and get tokens on
> > >> behalf of
> > >> >> > them. Overall Impersonation is a whole different problem in my
> > >> opinion and
> > >> >> > I think we can tackle it in separate KIP.
> > >> >> >
> > >> >> > 111. What's the typical rate of getting and renewing delegation
> > >> tokens?
> > >> >> > Typically this should be very very low, 1 request per minute is a
> > >> >> > relatively high estimate. However it depends on the token
> > >> expiration. I am
> > >> >> > less worried about the extra load it puts on controller vs the
> > added
> > >> >> > complexity and the value it offers.
> > >> >> >
> > >> >> > Thanks
> > >> >> > Parth
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <is...@juma.me.uk>
> > >> wrote:
> > >> >> >
> > >> >> > > Thanks Rajini. It would probably require a separate KIP as it
> > will
> > >> >> > > introduce user visible changes. We could also update KIP-48 to
> > >> have this
> > >> >> > > information, but it seems cleaner to do it separately. We can
> > >> discuss
> > >> >> > that
> > >> >> > > in the KIP call today.
> > >> >> > >
> > >> >> > > Ismael
> > >> >> > >
> > >> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <
> > >> >> > > rajinisivaram@googlemail.com> wrote:
> > >> >> > >
> > >> >> > > > Ismael,
> > >> >> > > >
> > >> >> > > > I have created a JIRA (
> > >> >> > https://issues.apache.org/jira/browse/KAFKA-3751)
> > >> >> > > > for adding SCRAM as a SASL mechanism. Would that need another
> > >> KIP? If
> > >> >> > > > KIP-48 will use this mechanism, can this just be a JIRA that
> > gets
> > >> >> > > reviewed
> > >> >> > > > when the PR is ready?
> > >> >> > > >
> > >> >> > > > Thank you,
> > >> >> > > >
> > >> >> > > > Rajini
> > >> >> > > >
> > >> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <
> > ismael@juma.me.uk>
> > >> >> > wrote:
> > >> >> > > >
> > >> >> > > > > Thanks Rajini, SCRAM seems like a good candidate.
> > >> >> > > > >
> > >> >> > > > > Gwen had independently mentioned this as a SASL mechanism
> > that
> > >> might
> > >> >> > be
> > >> >> > > > > useful for Kafka and I have been meaning to check it in
> more
> > >> detail.
> > >> >> > > Good
> > >> >> > > > > to know that you are willing to contribute an
> implementation.
> > >> Maybe
> > >> >> > we
> > >> >> > > > > should file a separate JIRA for this?
> > >> >> > > > >
> > >> >> > > > > Ismael
> > >> >> > > > >
> > >> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram <
> > >> >> > > > > rajinisivaram@googlemail.com> wrote:
> > >> >> > > > >
> > >> >> > > > > > SCRAM (Salted Challenge Response Authentication
> Mechanism)
> > >> is a
> > >> >> > > better
> > >> >> > > > > > mechanism than Digest-MD5. Java doesn't come with a
> > built-in
> > >> SCRAM
> > >> >> > > > > > SaslServer or SaslClient, but I will be happy to add
> > support
> > >> in
> > >> >> > Kafka
> > >> >> > > > > since
> > >> >> > > > > > it would be a useful mechanism to support anyway.
> > >> >> > > > > > https://tools.ietf.org/html/rfc7677 describes the
> protocol
> > >> for
> > >> >> > > > > > SCRAM-SHA-256.
> > >> >> > > > > >
> > >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <
> jun@confluent.io
> > >
> > >> wrote:
> > >> >> > > > > >
> > >> >> > > > > > > Parth,
> > >> >> > > > > > >
> > >> >> > > > > > > Thanks for the explanation. A couple of more questions.
> > >> >> > > > > > >
> > >> >> > > > > > > 110. What does getDelegationTokenAs mean?
> > >> >> > > > > > >
> > >> >> > > > > > > 111. What's the typical rate of getting and renewing
> > >> delegation
> > >> >> > > > tokens?
> > >> >> > > > > > > That may have an impact on whether they should be
> > directed
> > >> to the
> > >> >> > > > > > > controller.
> > >> >> > > > > > >
> > >> >> > > > > > > Jun
> > >> >> > > > > > >
> > >> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt <
> > >> >> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > >> >> > > > > > >
> > >> >> > > > > > > > Hi Jun,
> > >> >> > > > > > > >
> > >> >> > > > > > > > Thanks for reviewing.
> > >> >> > > > > > > >
> > >> >> > > > > > > > * We could add a Cluster action to add acls on who
> can
> > >> request
> > >> >> > > > > > delegation
> > >> >> > > > > > > > tokens. I don't see the use case for that yet but
> down
> > >> the line
> > >> >> > > > when
> > >> >> > > > > we
> > >> >> > > > > > > > start supporting getDelegationTokenAs it will be
> > >> necessary.
> > >> >> > > > > > > > * Yes we recommend tokens to be only used/distributed
> > >> over
> > >> >> > secure
> > >> >> > > > > > > channels.
> > >> >> > > > > > > > * Depending on what design we end up choosing
> > >> Invalidation will
> > >> >> > > be
> > >> >> > > > > > > > responsibility of every broker or controller.
> > >> >> > > > > > > > * I am not sure if I documented somewhere that
> > >> invalidation
> > >> >> > will
> > >> >> > > > > > directly
> > >> >> > > > > > > > go through zookeeper but that is not the intent.
> > >> Invalidation
> > >> >> > > will
> > >> >> > > > > > either
> > >> >> > > > > > > > be request based or due to expiration. No direct
> > >> zookeeper
> > >> >> > > > > interaction
> > >> >> > > > > > > from
> > >> >> > > > > > > > any client.
> > >> >> > > > > > > > * "Broker also stores the DelegationToken without the
> > >> hmac in
> > >> >> > the
> > >> >> > > > > > > > zookeeper." : Sorry about the confusion. The sole
> > >> purpose of
> > >> >> > > > > zookeeper
> > >> >> > > > > > in
> > >> >> > > > > > > > this design is as distribution channel for tokens
> > >> between all
> > >> >> > > > brokers
> > >> >> > > > > > > and a
> > >> >> > > > > > > > layer that ensures only tokens that were generated by
> > >> making a
> > >> >> > > > > request
> > >> >> > > > > > > to a
> > >> >> > > > > > > > broker will be accepted (more on this in second
> > >> paragraph). The
> > >> >> > > > token
> > >> >> > > > > > > > consists of few elements (owner, renewer, uuid ,
> > >> expiration,
> > >> >> > > hmac)
> > >> >> > > > ,
> > >> >> > > > > > one
> > >> >> > > > > > > of
> > >> >> > > > > > > > which is the finally generated hmac but hmac it self
> is
> > >> >> > derivable
> > >> >> > > > if
> > >> >> > > > > > you
> > >> >> > > > > > > > have all the other elements of the token + secret key
> > to
> > >> >> > generate
> > >> >> > > > > hmac.
> > >> >> > > > > > > > Given zookeeper does not provide SSL support we do
> not
> > >> want the
> > >> >> > > > > entire
> > >> >> > > > > > > > token to be wire transferred to zookeeper as that
> will
> > >> be an
> > >> >> > > > insecure
> > >> >> > > > > > > wire
> > >> >> > > > > > > > transfer. Instead we only store all the other
> elements
> > >> of a
> > >> >> > > > > delegation
> > >> >> > > > > > > > tokens. Brokers can read these elements and because
> > they
> > >> also
> > >> >> > > have
> > >> >> > > > > > access
> > >> >> > > > > > > > to secret key they will be able to generate hmac on
> > >> their end.
> > >> >> > > > > > > >
> > >> >> > > > > > > > One of the alternative proposed is to avoid zookeeper
> > >> >> > > altogether. A
> > >> >> > > > > > > Client
> > >> >> > > > > > > > will call broker with required information (owner,
> > >> renwer,
> > >> >> > > > > expiration)
> > >> >> > > > > > > and
> > >> >> > > > > > > > get back (signed hmac, uuid). Broker won't store this
> > in
> > >> >> > > zookeeper.
> > >> >> > > > > > From
> > >> >> > > > > > > > this point a client can contact any broker with all
> the
> > >> >> > > delegation
> > >> >> > > > > > token
> > >> >> > > > > > > > info (owner, rewner, expiration, hmac, uuid) the
> borker
> > >> will
> > >> >> > > > > regenerate
> > >> >> > > > > > > the
> > >> >> > > > > > > > hmac and as long as it matches with hmac presented by
> > >> client ,
> > >> >> > > > broker
> > >> >> > > > > > > will
> > >> >> > > > > > > > allow the request to authenticate.  Only problem with
> > >> this
> > >> >> > > approach
> > >> >> > > > > is
> > >> >> > > > > > if
> > >> >> > > > > > > > the secret key is compromised any client can now
> > generate
> > >> >> > random
> > >> >> > > > > tokens
> > >> >> > > > > > > and
> > >> >> > > > > > > > they will still be able to authenticate as any user
> > they
> > >> like.
> > >> >> > > with
> > >> >> > > > > > > > zookeeper we guarantee that only tokens acquired via
> a
> > >> broker
> > >> >> > > > (using
> > >> >> > > > > > some
> > >> >> > > > > > > > auth scheme other than delegation token) will be
> > >> accepted. We
> > >> >> > > need
> > >> >> > > > to
> > >> >> > > > > > > > discuss which proposal makes more sense and we can go
> > >> over it
> > >> >> > in
> > >> >> > > > > > > tomorrow's
> > >> >> > > > > > > > meeting.
> > >> >> > > > > > > >
> > >> >> > > > > > > > Also, can you forward the invite to me?
> > >> >> > > > > > > >
> > >> >> > > > > > > > Thanks
> > >> >> > > > > > > > Parth
> > >> >> > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <
> > >> jun@confluent.io>
> > >> >> > > > wrote:
> > >> >> > > > > > > >
> > >> >> > > > > > > > > Thanks for the KIP. A few comments.
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > 100. This potentially can be useful for Kafka
> Connect
> > >> and
> > >> >> > Kafka
> > >> >> > > > > rest
> > >> >> > > > > > > > proxy
> > >> >> > > > > > > > > where a worker agent will need to run a task on
> > behalf
> > >> of a
> > >> >> > > > client.
> > >> >> > > > > > We
> > >> >> > > > > > > > will
> > >> >> > > > > > > > > likely need to change how those services use Kafka
> > >> clients
> > >> >> > > > > > > > > (producer/consumer). Instead of a shared client per
> > >> worker,
> > >> >> > we
> > >> >> > > > will
> > >> >> > > > > > > need
> > >> >> > > > > > > > a
> > >> >> > > > > > > > > client per user task since the authentication
> happens
> > >> at the
> > >> >> > > > > > connection
> > >> >> > > > > > > > > level. For Kafka Connect, the renewer will be the
> > >> workers.
> > >> >> > So,
> > >> >> > > we
> > >> >> > > > > > > > probably
> > >> >> > > > > > > > > need to allow multiple renewers. For Kafka rest
> > proxy,
> > >> the
> > >> >> > > > renewer
> > >> >> > > > > > can
> > >> >> > > > > > > > > probably just be the creator of the token.
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > 101. Do we need new acl on who can request
> delegation
> > >> tokens?
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > 102. Do we recommend people to send delegation
> tokens
> > >> in an
> > >> >> > > > > encrypted
> > >> >> > > > > > > > > channel?
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > 103. Who is responsible for expiring tokens, every
> > >> broker?
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > 104. For invalidating tokens, would it be better to
> > do
> > >> it in
> > >> >> > a
> > >> >> > > > > > request
> > >> >> > > > > > > > > instead of going to ZK directly?
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > 105. The terminology of client in the wiki
> sometimes
> > >> refers
> > >> >> > to
> > >> >> > > > the
> > >> >> > > > > > end
> > >> >> > > > > > > > > client and some other times refers to the client
> > using
> > >> the
> > >> >> > > > > delegation
> > >> >> > > > > > > > > tokens. It would be useful to distinguish between
> the
> > >> two.
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > 106. Could you explain the sentence "Broker also
> > >> stores the
> > >> >> > > > > > > > DelegationToken
> > >> >> > > > > > > > > without the hmac in the zookeeper." a bit more? I
> > >> thought the
> > >> >> > > > > > > delegation
> > >> >> > > > > > > > > token is the hmac.
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > Thanks,
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > Jun
> > >> >> > > > > > > > >
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <
> > >> jun@confluent.io>
> > >> >> > > > wrote:
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > > Hi, Harsha,
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Just sent out a KIP meeting invite. We can
> discuss
> > >> this in
> > >> >> > > the
> > >> >> > > > > > > meeting
> > >> >> > > > > > > > > > tomorrow.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Thanks,
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Jun
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <
> > >> kafka@harsha.io>
> > >> >> > > > wrote:
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > >> Hi All,
> > >> >> > > > > > > > > >>            Can we have a KIP meeting around
> this.
> > >> The KIP
> > >> >> > is
> > >> >> > > > up
> > >> >> > > > > > for
> > >> >> > > > > > > > > >>            sometime and if there are any
> questions
> > >> lets
> > >> >> > > > quickly
> > >> >> > > > > > hash
> > >> >> > > > > > > > out
> > >> >> > > > > > > > > >>            details.
> > >> >> > > > > > > > > >>
> > >> >> > > > > > > > > >> Thanks,
> > >> >> > > > > > > > > >> Harsha
> > >> >> > > > > > > > > >>
> > >> >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40 AM, parth
> > brahmbhatt
> > >> wrote:
> > >> >> > > > > > > > > >> > That is what the hadoop echo system uses so no
> > >> good
> > >> >> > reason
> > >> >> > > > > > really.
> > >> >> > > > > > > > We
> > >> >> > > > > > > > > >> > could
> > >> >> > > > > > > > > >> > change it to whatever is the newest
> recommended
> > >> standard
> > >> >> > > is.
> > >> >> > > > > > > > > >> >
> > >> >> > > > > > > > > >> > Thanks
> > >> >> > > > > > > > > >> > Parth
> > >> >> > > > > > > > > >> >
> > >> >> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <
> > >> >> > > > > ismael@juma.me.uk
> > >> >> > > > > > >
> > >> >> > > > > > > > > wrote:
> > >> >> > > > > > > > > >> >
> > >> >> > > > > > > > > >> > > Hi Parth,
> > >> >> > > > > > > > > >> > >
> > >> >> > > > > > > > > >> > > Thanks for the KIP. I only started reviewing
> > >> this and
> > >> >> > > may
> > >> >> > > > > have
> > >> >> > > > > > > > > >> additional
> > >> >> > > > > > > > > >> > > questions later. The immediate question that
> > >> came to
> > >> >> > > mind
> > >> >> > > > is
> > >> >> > > > > > our
> > >> >> > > > > > > > > >> choice of
> > >> >> > > > > > > > > >> > > "DIGEST-MD5" even though it's marked as
> > >> OBSOLETE in
> > >> >> > the
> > >> >> > > > IANA
> > >> >> > > > > > > > > Registry
> > >> >> > > > > > > > > >> of
> > >> >> > > > > > > > > >> > > SASL mechanisms and the original RFC (2831)
> > has
> > >> been
> > >> >> > > moved
> > >> >> > > > > to
> > >> >> > > > > > > > > Historic
> > >> >> > > > > > > > > >> > > status:
> > >> >> > > > > > > > > >> > >
> > >> >> > > > > > > > > >> > > https://tools.ietf.org/html/rfc6331
> > >> >> > > > > > > > > >> > >
> > >> >> > > > > > > > >
> > >> >> > > > > >
> > >> >> > >
> > >> http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> > >> >> > > > > > > > > >> > >
> > >> >> > > > > > > > > >> > > What is the reasoning behind that choice?
> > >> >> > > > > > > > > >> > >
> > >> >> > > > > > > > > >> > > Thanks,
> > >> >> > > > > > > > > >> > > Ismael
> > >> >> > > > > > > > > >> > >
> > >> >> > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen
> > Shapira <
> > >> >> > > > > > > gwen@confluent.io
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > >> wrote:
> > >> >> > > > > > > > > >> > >
> > >> >> > > > > > > > > >> > > > Also comments inline :)
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > > > > * I want to emphasize that even though
> > >> delegation
> > >> >> > > > tokens
> > >> >> > > > > > > are a
> > >> >> > > > > > > > > >> Hadoop
> > >> >> > > > > > > > > >> > > > > innovation, I feel very strongly about
> not
> > >> adding
> > >> >> > > > > > dependency
> > >> >> > > > > > > > on
> > >> >> > > > > > > > > >> Hadoop
> > >> >> > > > > > > > > >> > > > > when implementing delegation tokens for
> > >> Kafka. The
> > >> >> > > KIP
> > >> >> > > > > > > doesn't
> > >> >> > > > > > > > > >> imply
> > >> >> > > > > > > > > >> > > > > such dependency, but if you can
> clarify...
> > >> >> > > > > > > > > >> > > > >
> > >> >> > > > > > > > > >> > > > >
> > >> >> > > > > > > > > >> > > > > *No hadoop dependency.*
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > > > Yay! Just add this to the KIP so no one
> will
> > >> read
> > >> >> > the
> > >> >> > > > KIP
> > >> >> > > > > > and
> > >> >> > > > > > > > > panic
> > >> >> > > > > > > > > >> > > > three weeks before the next release...
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > > > > * Can we get delegation token at any
> time
> > >> after
> > >> >> > > > > > > > authenticating?
> > >> >> > > > > > > > > >> only
> > >> >> > > > > > > > > >> > > > > immediately after?
> > >> >> > > > > > > > > >> > > > >
> > >> >> > > > > > > > > >> > > > >
> > >> >> > > > > > > > > >> > > > > *As long as you are authenticated you
> can
> > >> get
> > >> >> > > > delegation
> > >> >> > > > > > > > tokens.
> > >> >> > > > > > > > > >> We
> > >> >> > > > > > > > > >> > > need
> > >> >> > > > > > > > > >> > > > to
> > >> >> > > > > > > > > >> > > > > discuss if a client authenticated using
> > >> delegation
> > >> >> > > > > token,
> > >> >> > > > > > > can
> > >> >> > > > > > > > > also
> > >> >> > > > > > > > > >> > > > acquire
> > >> >> > > > > > > > > >> > > > > delegation token again or not. Also
> there
> > >> is the
> > >> >> > > > > question
> > >> >> > > > > > of
> > >> >> > > > > > > > do
> > >> >> > > > > > > > > we
> > >> >> > > > > > > > > >> > > allow
> > >> >> > > > > > > > > >> > > > > anyone to acquire delegation token or we
> > >> want
> > >> >> > > specific
> > >> >> > > > > > ACLs
> > >> >> > > > > > > (I
> > >> >> > > > > > > > > >> think
> > >> >> > > > > > > > > >> > > its
> > >> >> > > > > > > > > >> > > > an
> > >> >> > > > > > > > > >> > > > > overkill.)*
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > > > I agree that ACLs is an overkill.
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > > > I think we are debating two options:
> Either
> > >> require
> > >> >> > > > > Kerberos
> > >> >> > > > > > > > auth
> > >> >> > > > > > > > > >> for
> > >> >> > > > > > > > > >> > > > renewal or require non-owners to renew.
> > >> >> > > > > > > > > >> > > > I *think* the latter is simpler (it
> > basically
> > >> >> > require
> > >> >> > > a
> > >> >> > > > > "job
> > >> >> > > > > > > > > master"
> > >> >> > > > > > > > > >> > > > to take responsibility for the renewal, it
> > >> will have
> > >> >> > > its
> > >> >> > > > > own
> > >> >> > > > > > > > > >> identity
> > >> >> > > > > > > > > >> > > > anyway and I think this is the correct
> > design
> > >> >> > pattern
> > >> >> > > > > > anyway.
> > >> >> > > > > > > > For
> > >> >> > > > > > > > > >> > > > storm, I'd expect Nimbus to coordinate
> > >> renewals?),
> > >> >> > but
> > >> >> > > > it
> > >> >> > > > > is
> > >> >> > > > > > > > hard
> > >> >> > > > > > > > > to
> > >> >> > > > > > > > > >> > > > debate simplicity without looking at the
> > code
> > >> >> > changes
> > >> >> > > > > > > required.
> > >> >> > > > > > > > If
> > >> >> > > > > > > > > >> you
> > >> >> > > > > > > > > >> > > > have a draft of how the "require Kerberos"
> > >> will look
> > >> >> > > in
> > >> >> > > > > > Kafka
> > >> >> > > > > > > > > code,
> > >> >> > > > > > > > > >> > > > I'll be happy to take a look.
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > > > > * My understanding is that tokens will
> > >> propagate
> > >> >> > via
> > >> >> > > > ZK
> > >> >> > > > > > but
> > >> >> > > > > > > > > >> without
> > >> >> > > > > > > > > >> > > > > additional changes to UpdateMetadata
> > >> protocol,
> > >> >> > > > correct?
> > >> >> > > > > > > > Clients
> > >> >> > > > > > > > > >> > > > > currently don't retry on SASL auth
> failure
> > >> (IIRC),
> > >> >> > > but
> > >> >> > > > > > since
> > >> >> > > > > > > > the
> > >> >> > > > > > > > > >> > > > > tokens propagate between brokers asynch,
> > we
> > >> will
> > >> >> > > need
> > >> >> > > > to
> > >> >> > > > > > > > retry a
> > >> >> > > > > > > > > >> bit
> > >> >> > > > > > > > > >> > > > > to avoid clients failing auth due to
> > timing
> > >> >> > issues.
> > >> >> > > > > > > > > >> > > > >
> > >> >> > > > > > > > > >> > > > > *I am considering 2 alternatives right
> > now.
> > >> The
> > >> >> > > > current
> > >> >> > > > > > > > > documented
> > >> >> > > > > > > > > >> > > > approach
> > >> >> > > > > > > > > >> > > > > is zookeeper based and it does not
> require
> > >> any
> > >> >> > > changes
> > >> >> > > > > to
> > >> >> > > > > > > > > >> > > UpdateMetadata
> > >> >> > > > > > > > > >> > > > > protocol. An alternative approach can
> > remove
> > >> >> > > zookeeper
> > >> >> > > > > > > > > dependency
> > >> >> > > > > > > > > >> as
> > >> >> > > > > > > > > >> > > well
> > >> >> > > > > > > > > >> > > > > but we can discuss that in KIP
> discussion
> > >> call.*
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > > > Oooh! Sounds interesting. Do you want to
> > ping
> > >> Jun to
> > >> >> > > > > > arrange a
> > >> >> > > > > > > > > call?
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > > > > * I liked Ashish's suggestion of having
> > >> just the
> > >> >> > > > > > controller
> > >> >> > > > > > > > > issue
> > >> >> > > > > > > > > >> the
> > >> >> > > > > > > > > >> > > > > delegation tokens, to avoid syncing a
> > shared
> > >> >> > secret.
> > >> >> > > > Not
> > >> >> > > > > > > sure
> > >> >> > > > > > > > if
> > >> >> > > > > > > > > >> we
> > >> >> > > > > > > > > >> > > > > want to continue the discussion here or
> on
> > >> the
> > >> >> > > wiki. I
> > >> >> > > > > > think
> > >> >> > > > > > > > > that
> > >> >> > > > > > > > > >> we
> > >> >> > > > > > > > > >> > > > > can decouple the problem of "token
> > >> distribution"
> > >> >> > > from
> > >> >> > > > > > > "shared
> > >> >> > > > > > > > > >> secret
> > >> >> > > > > > > > > >> > > > > distribution" and use the controller as
> > the
> > >> only
> > >> >> > > token
> > >> >> > > > > > > > generator
> > >> >> > > > > > > > > >> to
> > >> >> > > > > > > > > >> > > > > solve the second issue, while still
> using
> > >> ZK async
> > >> >> > > to
> > >> >> > > > > > > > distribute
> > >> >> > > > > > > > > >> > > > > tokens.
> > >> >> > > > > > > > > >> > > > >
> > >> >> > > > > > > > > >> > > > >
> > >> >> > > > > > > > > >> > > > > *As mentioned in the previous Email I am
> > >> fine with
> > >> >> > > > that
> > >> >> > > > > > > > approach
> > >> >> > > > > > > > > >> as
> > >> >> > > > > > > > > >> > > long
> > >> >> > > > > > > > > >> > > > as
> > >> >> > > > > > > > > >> > > > > we agree that the extra complexity of
> > >> >> > > adding/updating
> > >> >> > > > > APIS
> > >> >> > > > > > > > adds
> > >> >> > > > > > > > > >> enough
> > >> >> > > > > > > > > >> > > > > value. The advantage with the controller
> > >> approach
> > >> >> > is
> > >> >> > > > > > secret
> > >> >> > > > > > > > > >> rotation
> > >> >> > > > > > > > > >> > > can
> > >> >> > > > > > > > > >> > > > be
> > >> >> > > > > > > > > >> > > > > automated,frequent and would not require
> > >> >> > > deployment. *
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > > > Can you detail the extra complexity (or
> > point
> > >> me to
> > >> >> > > the
> > >> >> > > > > > email
> > >> >> > > > > > > I
> > >> >> > > > > > > > > >> > > > missed?) - which Apis are required?
> > >> >> > > > > > > > > >> > > > As far as I can tell, clients can already
> > >> find the
> > >> >> > > > > > controller
> > >> >> > > > > > > > from
> > >> >> > > > > > > > > >> > > > metadata. I'm a bit more concerned about
> > >> controller
> > >> >> > > > load.
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > > > >
> > >> >> > > > > > > > > >> > > > > * While I like the idea of forcing
> > kerberos
> > >> auth
> > >> >> > for
> > >> >> > > > > > > renwal, I
> > >> >> > > > > > > > > >> think
> > >> >> > > > > > > > > >> > > > > it mixes the transport layer the the
> > request
> > >> >> > content
> > >> >> > > > in
> > >> >> > > > > a
> > >> >> > > > > > > > pretty
> > >> >> > > > > > > > > >> ugly
> > >> >> > > > > > > > > >> > > > > way. Perhaps limiting renewer to
> non-owner
> > >> is
> > >> >> > > better.
> > >> >> > > > > > > > > >> > > > >
> > >> >> > > > > > > > > >> > > > > *I feel this is a necessary evil. While
> > >> this will
> > >> >> > > make
> > >> >> > > > > the
> > >> >> > > > > > > > kafka
> > >> >> > > > > > > > > >> code
> > >> >> > > > > > > > > >> > > > > pretty straight forward , forcing
> renewer
> > >> to
> > >> >> > > > non-owner
> > >> >> > > > > > > pushes
> > >> >> > > > > > > > > >> the code
> > >> >> > > > > > > > > >> > > > > ugliness to client and makes it even
> > harder
> > >> to
> > >> >> > > > > > integrate.  *
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > > > As mentioned before, I don't think the
> > >> "renewal by
> > >> >> > > > other"
> > >> >> > > > > > > > approach
> > >> >> > > > > > > > > >> is
> > >> >> > > > > > > > > >> > > > that ugly for the clients we expect to use
> > >> >> > delegation
> > >> >> > > > > tokens
> > >> >> > > > > > > > since
> > >> >> > > > > > > > > >> > > > they will have an app-master of some sort
> > who
> > >> >> > > requested
> > >> >> > > > > the
> > >> >> > > > > > > > token
> > >> >> > > > > > > > > to
> > >> >> > > > > > > > > >> > > > begin with.
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > > > >
> > >> >> > > > > > > > > >> > > > > The response for my question on how
> > multiple
> > >> >> > > > identities
> > >> >> > > > > > will
> > >> >> > > > > > > > be
> > >> >> > > > > > > > > >> > > > > handled wasn't super clear to me -
> AFAIK,
> > >> we don't
> > >> >> > > > > > > > authenticate
> > >> >> > > > > > > > > >> each
> > >> >> > > > > > > > > >> > > > > request, we authenticate connections.
> > >> >> > > > > > > > > >> > > > >
> > >> >> > > > > > > > > >> > > > > *We authenticate connections, and only
> > when
> > >> they
> > >> >> > are
> > >> >> > > > > being
> > >> >> > > > > > > > > >> established.
> > >> >> > > > > > > > > >> > > > Let
> > >> >> > > > > > > > > >> > > > > me try to phrase this as a question, in
> > >> absence of
> > >> >> > > > > > > delegation
> > >> >> > > > > > > > > >> tokens if
> > >> >> > > > > > > > > >> > > > we
> > >> >> > > > > > > > > >> > > > > had to support the use case using user
> > >> TGT's how
> > >> >> > > would
> > >> >> > > > > we
> > >> >> > > > > > do
> > >> >> > > > > > > > it?
> > >> >> > > > > > > > > >> My
> > >> >> > > > > > > > > >> > > point
> > >> >> > > > > > > > > >> > > > > was it would be no different with
> > delegation
> > >> >> > tokens.
> > >> >> > > > The
> > >> >> > > > > > use
> > >> >> > > > > > > > > case
> > >> >> > > > > > > > > >> you
> > >> >> > > > > > > > > >> > > are
> > >> >> > > > > > > > > >> > > > > describing seems more like
> impersonation.*
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > > > Yeah, I thought that one of the things
> that
> > >> >> > delegation
> > >> >> > > > > > tokens
> > >> >> > > > > > > > > >> handled.
> > >> >> > > > > > > > > >> > > > Maybe I got it wrong :)
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > > > Thanks for the detailed answers.
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > > > Gwen
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > > > > Thanks
> > >> >> > > > > > > > > >> > > > > Parth
> > >> >> > > > > > > > > >> > > > >
> > >> >> > > > > > > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen
> > >> Shapira <
> > >> >> > > > > > > > > gwen@confluent.io
> > >> >> > > > > > > > > >> >
> > >> >> > > > > > > > > >> > > > wrote:
> > >> >> > > > > > > > > >> > > > >
> > >> >> > > > > > > > > >> > > > >> Hi Parth and Harsha,
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > > >> Few more comments:
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > > >> * The API / RequestResponse section
> > >> doesn't seem
> > >> >> > to
> > >> >> > > > > have
> > >> >> > > > > > > good
> > >> >> > > > > > > > > >> > > > >> description of the changes to the Kafka
> > >> Protocol.
> > >> >> > > > > Sounds
> > >> >> > > > > > > like
> > >> >> > > > > > > > > >> you are
> > >> >> > > > > > > > > >> > > > >> proposing new DelegationTokenRequest
> and
> > >> >> > > > > > RenewTokenRequest
> > >> >> > > > > > > > (and
> > >> >> > > > > > > > > >> > > > >> matching responses), without detailing
> > the
> > >> >> > contents
> > >> >> > > > of
> > >> >> > > > > > the
> > >> >> > > > > > > > > >> requests
> > >> >> > > > > > > > > >> > > > >> and responses? Or rather, you show the
> > >> class
> > >> >> > > > interface,
> > >> >> > > > > > but
> > >> >> > > > > > > > not
> > >> >> > > > > > > > > >> the
> > >> >> > > > > > > > > >> > > > >> underlying protocol. This could be seen
> > as
> > >> an
> > >> >> > > > > > > implementation
> > >> >> > > > > > > > > >> detail,
> > >> >> > > > > > > > > >> > > > >> but since the binary protocol is what
> we
> > >> provide
> > >> >> > to
> > >> >> > > > > > > non-Java
> > >> >> > > > > > > > > >> clients,
> > >> >> > > > > > > > > >> > > > >> we need to show the changes there.
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > > >> * getDelegationToken sounds like
> > >> >> > > > > > > > delegationTokenRequestHandler?
> > >> >> > > > > > > > > >> Is it
> > >> >> > > > > > > > > >> > > > >> planned to be part of KafkaApi? or
> > Client?
> > >> Its
> > >> >> > > > > unclear...
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > > >> * I want to emphasize that even though
> > >> delegation
> > >> >> > > > > tokens
> > >> >> > > > > > > are
> > >> >> > > > > > > > a
> > >> >> > > > > > > > > >> Hadoop
> > >> >> > > > > > > > > >> > > > >> innovation, I feel very strongly about
> > not
> > >> adding
> > >> >> > > > > > > dependency
> > >> >> > > > > > > > on
> > >> >> > > > > > > > > >> Hadoop
> > >> >> > > > > > > > > >> > > > >> when implementing delegation tokens for
> > >> Kafka.
> > >> >> > The
> > >> >> > > > KIP
> > >> >> > > > > > > > doesn't
> > >> >> > > > > > > > > >> imply
> > >> >> > > > > > > > > >> > > > >> such dependency, but if you can
> > clarify...
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > > >> * Can we get delegation token at any
> time
> > >> after
> > >> >> > > > > > > > authenticating?
> > >> >> > > > > > > > > >> only
> > >> >> > > > > > > > > >> > > > >> immediately after?
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > > >> * My understanding is that tokens will
> > >> propagate
> > >> >> > > via
> > >> >> > > > ZK
> > >> >> > > > > > but
> > >> >> > > > > > > > > >> without
> > >> >> > > > > > > > > >> > > > >> additional changes to UpdateMetadata
> > >> protocol,
> > >> >> > > > correct?
> > >> >> > > > > > > > Clients
> > >> >> > > > > > > > > >> > > > >> currently don't retry on SASL auth
> > failure
> > >> >> > (IIRC),
> > >> >> > > > but
> > >> >> > > > > > > since
> > >> >> > > > > > > > > the
> > >> >> > > > > > > > > >> > > > >> tokens propagate between brokers
> asynch,
> > >> we will
> > >> >> > > need
> > >> >> > > > > to
> > >> >> > > > > > > > retry
> > >> >> > > > > > > > > a
> > >> >> > > > > > > > > >> bit
> > >> >> > > > > > > > > >> > > > >> to avoid clients failing auth due to
> > timing
> > >> >> > issues.
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > > >> * Strongly agreeing on clients not
> > >> touching ZK
> > >> >> > > > directly
> > >> >> > > > > > :)
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > > >> * I liked Ashish's suggestion of having
> > >> just the
> > >> >> > > > > > controller
> > >> >> > > > > > > > > >> issue the
> > >> >> > > > > > > > > >> > > > >> delegation tokens, to avoid syncing a
> > >> shared
> > >> >> > > secret.
> > >> >> > > > > Not
> > >> >> > > > > > > sure
> > >> >> > > > > > > > > if
> > >> >> > > > > > > > > >> we
> > >> >> > > > > > > > > >> > > > >> want to continue the discussion here or
> > on
> > >> the
> > >> >> > > wiki.
> > >> >> > > > I
> > >> >> > > > > > > think
> > >> >> > > > > > > > > >> that we
> > >> >> > > > > > > > > >> > > > >> can decouple the problem of "token
> > >> distribution"
> > >> >> > > from
> > >> >> > > > > > > "shared
> > >> >> > > > > > > > > >> secret
> > >> >> > > > > > > > > >> > > > >> distribution" and use the controller as
> > >> the only
> > >> >> > > > token
> > >> >> > > > > > > > > generator
> > >> >> > > > > > > > > >> to
> > >> >> > > > > > > > > >> > > > >> solve the second issue, while still
> using
> > >> ZK
> > >> >> > async
> > >> >> > > to
> > >> >> > > > > > > > > distribute
> > >> >> > > > > > > > > >> > > > >> tokens.
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > > >> * I am also uncomfortable with infinite
> > >> lifetime
> > >> >> > of
> > >> >> > > > > > tokens
> > >> >> > > > > > > > (and
> > >> >> > > > > > > > > >> hoped
> > >> >> > > > > > > > > >> > > > >> to hear from others in the community) -
> > but
> > >> >> > having
> > >> >> > > > the
> > >> >> > > > > > > option
> > >> >> > > > > > > > > and
> > >> >> > > > > > > > > >> > > > >> documenting it as less secure, allows
> > >> users to
> > >> >> > > > > configure
> > >> >> > > > > > > > their
> > >> >> > > > > > > > > >> system
> > >> >> > > > > > > > > >> > > > >> as the wish.
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > > >> * While I like the idea of forcing
> > >> kerberos auth
> > >> >> > > for
> > >> >> > > > > > > renwal,
> > >> >> > > > > > > > I
> > >> >> > > > > > > > > >> think
> > >> >> > > > > > > > > >> > > > >> it mixes the transport layer the the
> > >> request
> > >> >> > > content
> > >> >> > > > > in a
> > >> >> > > > > > > > > pretty
> > >> >> > > > > > > > > >> ugly
> > >> >> > > > > > > > > >> > > > >> way. Perhaps limiting renewer to
> > non-owner
> > >> is
> > >> >> > > better.
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > > >> Things I'd still like to see:
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > > >> * More detailed explanation on what we
> > >> plan to do
> > >> >> > > for
> > >> >> > > > > the
> > >> >> > > > > > > > java
> > >> >> > > > > > > > > >> clients
> > >> >> > > > > > > > > >> > > > >> specifically - new configuration? new
> > APIs?
> > >> >> > > > > > > > > >> > > > >> The response for my question on how
> > >> multiple
> > >> >> > > > identities
> > >> >> > > > > > > will
> > >> >> > > > > > > > be
> > >> >> > > > > > > > > >> > > > >> handled wasn't super clear to me -
> AFAIK,
> > >> we
> > >> >> > don't
> > >> >> > > > > > > > authenticate
> > >> >> > > > > > > > > >> each
> > >> >> > > > > > > > > >> > > > >> request, we authenticate connections.
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > > >> * Alternatives: Delegation tokens are
> > only
> > >> used
> > >> >> > in
> > >> >> > > > the
> > >> >> > > > > > > Hadoop
> > >> >> > > > > > > > > >> > > > >> ecosystem. I'm wondering if there are
> > >> >> > alternatives
> > >> >> > > in
> > >> >> > > > > > other
> > >> >> > > > > > > > > >> ecosystems
> > >> >> > > > > > > > > >> > > > >> (Mesos? Tachyon? Cassandra?) and
> whether
> > >> there
> > >> >> > are
> > >> >> > > > some
> > >> >> > > > > > > > > >> advantages
> > >> >> > > > > > > > > >> > > > >> there.
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > > >> Gwen
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM,
> Harsha <
> > >> >> > > > > kafka@harsha.io
> > >> >> > > > > > >
> > >> >> > > > > > > > > wrote:
> > >> >> > > > > > > > > >> > > > >> > Hi Gwen,
> > >> >> > > > > > > > > >> > > > >> >            Can you look at Parth's
> last
> > >> reply.
> > >> >> > > Does
> > >> >> > > > > it
> > >> >> > > > > > > > answer
> > >> >> > > > > > > > > >> your
> > >> >> > > > > > > > > >> > > > >> >            concerns.
> > >> >> > > > > > > > > >> > > > >> >
> > >> >> > > > > > > > > >> > > > >> > Thanks,
> > >> >> > > > > > > > > >> > > > >> > Harsha
> > >> >> > > > > > > > > >> > > > >> >
> > >> >> > > > > > > > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM,
> parth
> > >> >> > > brahmbhatt
> > >> >> > > > > > wrote:
> > >> >> > > > > > > > > >> > > > >> >> Thanks for reviewing Gwen. The wiki
> > >> already
> > >> >> > has
> > >> >> > > > > > details
> > >> >> > > > > > > on
> > >> >> > > > > > > > > >> token
> > >> >> > > > > > > > > >> > > > >> >> expiration
> > >> >> > > > > > > > > >> > > > >> >> under token acquisition process
> > >> >> > > > > > > > > >> > > > >> >> <
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > >
> > >> >> > > > > > > > > >>
> > >> >> > > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> > >> >> > > > > > > > > >> > > > >> >.
> > >> >> > > > > > > > > >> > > > >> >> Current proposal is that tokens will
> > >> expire
> > >> >> > > based
> > >> >> > > > > on a
> > >> >> > > > > > > > > server
> > >> >> > > > > > > > > >> side
> > >> >> > > > > > > > > >> > > > >> >> configuration (default 24 hours)
> > unless
> > >> >> > renewed.
> > >> >> > > > > > Renewal
> > >> >> > > > > > > > is
> > >> >> > > > > > > > > >> only
> > >> >> > > > > > > > > >> > > > allowed
> > >> >> > > > > > > > > >> > > > >> >> until the max life time of token.
> > >> >> > Alternatively
> > >> >> > > we
> > >> >> > > > > > could
> > >> >> > > > > > > > > also
> > >> >> > > > > > > > > >> make
> > >> >> > > > > > > > > >> > > > that
> > >> >> > > > > > > > > >> > > > >> >> an
> > >> >> > > > > > > > > >> > > > >> >> optional param and the server side
> > >> default can
> > >> >> > > > serve
> > >> >> > > > > > as
> > >> >> > > > > > > > the
> > >> >> > > > > > > > > >> upper
> > >> >> > > > > > > > > >> > > > bound.
> > >> >> > > > > > > > > >> > > > >> >>
> > >> >> > > > > > > > > >> > > > >> >> To your second point it will be done
> > >> exactly
> > >> >> > the
> > >> >> > > > > same
> > >> >> > > > > > > way
> > >> >> > > > > > > > we
> > >> >> > > > > > > > > >> would
> > >> >> > > > > > > > > >> > > > >> >> support
> > >> >> > > > > > > > > >> > > > >> >> multiple keytabs. The calling client
> > >> will have
> > >> >> > > to
> > >> >> > > > > put
> > >> >> > > > > > > the
> > >> >> > > > > > > > > >> tokens it
> > >> >> > > > > > > > > >> > > > >> wants
> > >> >> > > > > > > > > >> > > > >> >> to use in the subject instance and
> > call
> > >> >> > > > > > produce/consume
> > >> >> > > > > > > > > inside
> > >> >> > > > > > > > > >> > > > >> >> subject.doas. Each caller will have
> to
> > >> keep
> > >> >> > > track
> > >> >> > > > of
> > >> >> > > > > > its
> > >> >> > > > > > > > own
> > >> >> > > > > > > > > >> > > > subject. I
> > >> >> > > > > > > > > >> > > > >> >> will have to look at the code to see
> > if
> > >> we
> > >> >> > > support
> > >> >> > > > > > this
> > >> >> > > > > > > > > >> feature
> > >> >> > > > > > > > > >> > > right
> > >> >> > > > > > > > > >> > > > >> now
> > >> >> > > > > > > > > >> > > > >> >> but my understanding is delegation
> > token
> > >> >> > > shouldn't
> > >> >> > > > > > need
> > >> >> > > > > > > > any
> > >> >> > > > > > > > > >> special
> > >> >> > > > > > > > > >> > > > >> >> treatment as its just another type
> of
> > >> >> > Credential
> > >> >> > > > in
> > >> >> > > > > > the
> > >> >> > > > > > > > > >> subject.
> > >> >> > > > > > > > > >> > > > >> >>
> > >> >> > > > > > > > > >> > > > >> >> I would also like to know what is
> your
> > >> opinion
> > >> >> > > > about
> > >> >> > > > > > > > > infinite
> > >> >> > > > > > > > > >> > > renewal
> > >> >> > > > > > > > > >> > > > >> (my
> > >> >> > > > > > > > > >> > > > >> >> recommendation is to not support
> > this),
> > >> tokens
> > >> >> > > > > > renewing
> > >> >> > > > > > > > them
> > >> >> > > > > > > > > >> > > self(my
> > >> >> > > > > > > > > >> > > > >> >> recommendation is to not support
> this)
> > >> and
> > >> >> > most
> > >> >> > > > > > > > importantly
> > >> >> > > > > > > > > >> your
> > >> >> > > > > > > > > >> > > > choice
> > >> >> > > > > > > > > >> > > > >> >> between the alternatives listed on
> > this
> > >> thread
> > >> >> > > > > > > > > >> > > > >> >> <
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > >
> > >> >> > > > > > > > > >>
> > >> >> > > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >>
> >
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> > >> >> > > > > > > > > >> > > > >> >
> > >> >> > > > > > > > > >> > > > >> >> ( I am leaning towards the
> > >> alternative-2 minus
> > >> >> > > > > > > controller
> > >> >> > > > > > > > > >> > > > distributing
> > >> >> > > > > > > > > >> > > > >> >> secret). Thanks again for reviewing.
> > >> >> > > > > > > > > >> > > > >> >>
> > >> >> > > > > > > > > >> > > > >> >> Thanks
> > >> >> > > > > > > > > >> > > > >> >> Parth
> > >> >> > > > > > > > > >> > > > >> >>
> > >> >> > > > > > > > > >> > > > >> >>
> > >> >> > > > > > > > > >> > > > >> >>
> > >> >> > > > > > > > > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen
> > >> Shapira <
> > >> >> > > > > > > > > >> gwen@confluent.io>
> > >> >> > > > > > > > > >> > > > wrote:
> > >> >> > > > > > > > > >> > > > >> >>
> > >> >> > > > > > > > > >> > > > >> >> > Harsha,
> > >> >> > > > > > > > > >> > > > >> >> >
> > >> >> > > > > > > > > >> > > > >> >> > I was thinking of the Rest Proxy.
> I
> > >> didn't
> > >> >> > see
> > >> >> > > > > your
> > >> >> > > > > > > > design
> > >> >> > > > > > > > > >> yet,
> > >> >> > > > > > > > > >> > > > but in
> > >> >> > > > > > > > > >> > > > >> >> > our proxy, we have a set of
> > >> producers, which
> > >> >> > > > will
> > >> >> > > > > > > serve
> > >> >> > > > > > > > > >> multiple
> > >> >> > > > > > > > > >> > > > users
> > >> >> > > > > > > > > >> > > > >> >> > going through the proxy. Since
> these
> > >> users
> > >> >> > > will
> > >> >> > > > > have
> > >> >> > > > > > > > > >> different
> > >> >> > > > > > > > > >> > > > >> >> > privileges, they'll need to
> > >> authenticate
> > >> >> > > > > separately,
> > >> >> > > > > > > and
> > >> >> > > > > > > > > >> can't
> > >> >> > > > > > > > > >> > > > share a
> > >> >> > > > > > > > > >> > > > >> >> > token.
> > >> >> > > > > > > > > >> > > > >> >> >
> > >> >> > > > > > > > > >> > > > >> >> > Am I missing anything?
> > >> >> > > > > > > > > >> > > > >> >> >
> > >> >> > > > > > > > > >> > > > >> >> > Gwen
> > >> >> > > > > > > > > >> > > > >> >> >
> > >> >> > > > > > > > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM,
> > >> Harsha <
> > >> >> > > > > > > kafka@harsha.io
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > >> wrote:
> > >> >> > > > > > > > > >> > > > >> >> > > Gwen,
> > >> >> > > > > > > > > >> > > > >> >> > >            On your second point.
> > >> Can you
> > >> >> > > > > describe
> > >> >> > > > > > a
> > >> >> > > > > > > > > >> usecase
> > >> >> > > > > > > > > >> > > where
> > >> >> > > > > > > > > >> > > > >> >> > >            mutliple clients
> ended
> > up
> > >> >> > > sharing a
> > >> >> > > > > > > > producer
> > >> >> > > > > > > > > >> and
> > >> >> > > > > > > > > >> > > even
> > >> >> > > > > > > > > >> > > > if
> > >> >> > > > > > > > > >> > > > >> they
> > >> >> > > > > > > > > >> > > > >> >> > >            do why can't they not
> > use
> > >> >> > single
> > >> >> > > > > token
> > >> >> > > > > > > that
> > >> >> > > > > > > > > >> producer
> > >> >> > > > > > > > > >> > > > >> >> > >            captures. Why would
> we
> > >> need
> > >> >> > > > multiple
> > >> >> > > > > > > > clients
> > >> >> > > > > > > > > >> with
> > >> >> > > > > > > > > >> > > > >> different
> > >> >> > > > > > > > > >> > > > >> >> > >            tokens sharing a
> single
> > >> >> > instance
> > >> >> > > of
> > >> >> > > > > > > > producer.
> > >> >> > > > > > > > > >> Also
> > >> >> > > > > > > > > >> > > in
> > >> >> > > > > > > > > >> > > > >> this
> > >> >> > > > > > > > > >> > > > >> >> > >            case other clients
> have
> > >> access
> > >> >> > > all
> > >> >> > > > > the
> > >> >> > > > > > > > tokens
> > >> >> > > > > > > > > >> no?
> > >> >> > > > > > > > > >> > > > >> >> > >
> > >> >> > > > > > > > > >> > > > >> >> > > Thanks,
> > >> >> > > > > > > > > >> > > > >> >> > > Harsha
> > >> >> > > > > > > > > >> > > > >> >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >
> > >> >> > > > > > > > > >> > > > >> >> > > On Tue, May 3, 2016, at 11:49
> AM,
> > >> Gwen
> > >> >> > > Shapira
> > >> >> > > > > > > wrote:
> > >> >> > > > > > > > > >> > > > >> >> > >> Sorry for the delay:
> > >> >> > > > > > > > > >> > > > >> >> > >>
> > >> >> > > > > > > > > >> > > > >> >> > >> Two questions that we didn't
> see
> > >> in the
> > >> >> > > wiki:
> > >> >> > > > > > > > > >> > > > >> >> > >> 1. Is there an expiration for
> > >> delegation
> > >> >> > > > > tokens?
> > >> >> > > > > > > > > >> Renewal? How
> > >> >> > > > > > > > > >> > > > do we
> > >> >> > > > > > > > > >> > > > >> >> > >> revoke them?
> > >> >> > > > > > > > > >> > > > >> >> > >> 2. If we want to use delegation
> > >> tokens
> > >> >> > for
> > >> >> > > > > > "do-as"
> > >> >> > > > > > > > > (say,
> > >> >> > > > > > > > > >> > > submit
> > >> >> > > > > > > > > >> > > > >> Storm
> > >> >> > > > > > > > > >> > > > >> >> > >> job as my user), we will need a
> > >> producer
> > >> >> > > for
> > >> >> > > > > > every
> > >> >> > > > > > > > job
> > >> >> > > > > > > > > >> (we
> > >> >> > > > > > > > > >> > > can't
> > >> >> > > > > > > > > >> > > > >> share
> > >> >> > > > > > > > > >> > > > >> >> > >> them between multiple jobs
> > running
> > >> on
> > >> >> > same
> > >> >> > > > > node),
> > >> >> > > > > > > > since
> > >> >> > > > > > > > > >> we
> > >> >> > > > > > > > > >> > > only
> > >> >> > > > > > > > > >> > > > >> >> > >> authenticate when connecting.
> Is
> > >> there a
> > >> >> > > plan
> > >> >> > > > > to
> > >> >> > > > > > > > change
> > >> >> > > > > > > > > >> this
> > >> >> > > > > > > > > >> > > for
> > >> >> > > > > > > > > >> > > > >> >> > >> delegation tokens, in order to
> > >> allow
> > >> >> > > multiple
> > >> >> > > > > > users
> > >> >> > > > > > > > > with
> > >> >> > > > > > > > > >> > > > different
> > >> >> > > > > > > > > >> > > > >> >> > >> tokens to share a client?
> > >> >> > > > > > > > > >> > > > >> >> > >>
> > >> >> > > > > > > > > >> > > > >> >> > >> Gwen
> > >> >> > > > > > > > > >> > > > >> >> > >>
> > >> >> > > > > > > > > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM,
> > >> parth
> > >> >> > > > > brahmbhatt
> > >> >> > > > > > > > > >> > > > >> >> > >> <br...@gmail.com>
> > >> wrote:
> > >> >> > > > > > > > > >> > > > >> >> > >> > Bumping this up one more
> time,
> > >> can
> > >> >> > other
> > >> >> > > > > > > committers
> > >> >> > > > > > > > > >> review?
> > >> >> > > > > > > > > >> > > > >> >> > >> >
> > >> >> > > > > > > > > >> > > > >> >> > >> > Thanks
> > >> >> > > > > > > > > >> > > > >> >> > >> > Parth
> > >> >> > > > > > > > > >> > > > >> >> > >> >
> > >> >> > > > > > > > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07
> > AM,
> > >> >> > Harsha <
> > >> >> > > > > > > > > >> kafka@harsha.io>
> > >> >> > > > > > > > > >> > > > wrote:
> > >> >> > > > > > > > > >> > > > >> >> > >> >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> Parth,
> > >> >> > > > > > > > > >> > > > >> >> > >> >>           Overall current
> > >> design looks
> > >> >> > > > good
> > >> >> > > > > to
> > >> >> > > > > > > > me. I
> > >> >> > > > > > > > > >> am +1
> > >> >> > > > > > > > > >> > > on
> > >> >> > > > > > > > > >> > > > >> the
> > >> >> > > > > > > > > >> > > > >> >> > KIP.
> > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > >> >> > > > > > > > > >> > > > >> >> > >> >> Gwen , Jun can you review
> this
> > >> as
> > >> >> > well.
> > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > >> >> > > > > > > > > >> > > > >> >> > >> >> -Harsha
> > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > >> >> > > > > > > > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at
> 09:57
> > >> AM,
> > >> >> > parth
> > >> >> > > > > > > > brahmbhatt
> > >> >> > > > > > > > > >> wrote:
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks for review
> Jitendra.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > I don't like the idea of
> > >> infinite
> > >> >> > > > lifetime
> > >> >> > > > > > > but I
> > >> >> > > > > > > > > >> see the
> > >> >> > > > > > > > > >> > > > >> Streaming
> > >> >> > > > > > > > > >> > > > >> >> > use
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > case. Even for Streaming
> use
> > >> case I
> > >> >> > > was
> > >> >> > > > > > hoping
> > >> >> > > > > > > > > >> there will
> > >> >> > > > > > > > > >> > > > be
> > >> >> > > > > > > > > >> > > > >> some
> > >> >> > > > > > > > > >> > > > >> >> > notion
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > of
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > master/driver that can get
> > new
> > >> >> > > > delegation
> > >> >> > > > > > > tokens
> > >> >> > > > > > > > > at
> > >> >> > > > > > > > > >> fixed
> > >> >> > > > > > > > > >> > > > >> interval
> > >> >> > > > > > > > > >> > > > >> >> > and
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > distribute to workers. If
> > >> that is
> > >> >> > not
> > >> >> > > > the
> > >> >> > > > > > case
> > >> >> > > > > > > > for
> > >> >> > > > > > > > > >> we can
> > >> >> > > > > > > > > >> > > > >> discuss
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > delegation tokens renewing
> > >> them self
> > >> >> > > and
> > >> >> > > > > the
> > >> >> > > > > > > > > >> security
> > >> >> > > > > > > > > >> > > > >> implications
> > >> >> > > > > > > > > >> > > > >> >> > of the
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > same.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > I did not want clients to
> > >> fetch
> > >> >> > tokens
> > >> >> > > > > from
> > >> >> > > > > > > > > >> zookeeper,
> > >> >> > > > > > > > > >> > > > >> overall I
> > >> >> > > > > > > > > >> > > > >> >> > think
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > its
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > better if clients don't
> rely
> > >> on our
> > >> >> > > > > metadata
> > >> >> > > > > > > > store
> > >> >> > > > > > > > > >> and I
> > >> >> > > > > > > > > >> > > > >> think we
> > >> >> > > > > > > > > >> > > > >> >> > are
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > moving in that direction
> > with
> > >> all
> > >> >> > the
> > >> >> > > > > KIP-4
> > >> >> > > > > > > > > >> improvements.
> > >> >> > > > > > > > > >> > > > I
> > >> >> > > > > > > > > >> > > > >> chose
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as in this case
> > the
> > >> client
> > >> >> > > > will
> > >> >> > > > > > > still
> > >> >> > > > > > > > > >> just talk
> > >> >> > > > > > > > > >> > > > to
> > >> >> > > > > > > > > >> > > > >> >> > broker , its
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > the brokers that will use
> > >> zookeeper
> > >> >> > > > which
> > >> >> > > > > we
> > >> >> > > > > > > > > >> already do
> > >> >> > > > > > > > > >> > > > for a
> > >> >> > > > > > > > > >> > > > >> lot
> > >> >> > > > > > > > > >> > > > >> >> > of
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > other
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > usecases + ease of
> > >> development + and
> > >> >> > > the
> > >> >> > > > > > > ability
> > >> >> > > > > > > > > so
> > >> >> > > > > > > > > >> > > tokens
> > >> >> > > > > > > > > >> > > > >> will
> > >> >> > > > > > > > > >> > > > >> >> > survive
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > even a rolling
> > restart/cluster
> > >> >> > > failure.
> > >> >> > > > > if a
> > >> >> > > > > > > > > >> majority
> > >> >> > > > > > > > > >> > > > agrees
> > >> >> > > > > > > > > >> > > > >> the
> > >> >> > > > > > > > > >> > > > >> >> > added
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > complexity to have
> > controller
> > >> >> > > forwarding
> > >> >> > > > > > keys
> > >> >> > > > > > > to
> > >> >> > > > > > > > > all
> > >> >> > > > > > > > > >> > > > broker is
> > >> >> > > > > > > > > >> > > > >> >> > justified
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > as
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > it provides tighter
> security
> > >> , I am
> > >> >> > > fine
> > >> >> > > > > > with
> > >> >> > > > > > > > that
> > >> >> > > > > > > > > >> option
> > >> >> > > > > > > > > >> > > > too.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > Given zookeeper does not
> > >> support SSL
> > >> >> > > we
> > >> >> > > > > can
> > >> >> > > > > > > not
> > >> >> > > > > > > > > >> store
> > >> >> > > > > > > > > >> > > > master
> > >> >> > > > > > > > > >> > > > >> keys
> > >> >> > > > > > > > > >> > > > >> >> > in
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as master keys
> > will
> > >> be
> > >> >> > > exposed
> > >> >> > > > > on
> > >> >> > > > > > > > wire.
> > >> >> > > > > > > > > To
> > >> >> > > > > > > > > >> > > > support
> > >> >> > > > > > > > > >> > > > >> >> > rotation
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > without affecting current
> > >> clients is
> > >> >> > > > > > > something I
> > >> >> > > > > > > > > >> need to
> > >> >> > > > > > > > > >> > > > put
> > >> >> > > > > > > > > >> > > > >> more
> > >> >> > > > > > > > > >> > > > >> >> > thought
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > in. My current proposal
> > >> assumes the
> > >> >> > > > > rotation
> > >> >> > > > > > > > will
> > >> >> > > > > > > > > >> > > > invalidate
> > >> >> > > > > > > > > >> > > > >> all
> > >> >> > > > > > > > > >> > > > >> >> > current
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > tokens.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > I request committers to
> also
> > >> review
> > >> >> > > and
> > >> >> > > > > post
> > >> >> > > > > > > > their
> > >> >> > > > > > > > > >> > > comments
> > >> >> > > > > > > > > >> > > > >> so we
> > >> >> > > > > > > > > >> > > > >> >> > can
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > make
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > progress on this KIP.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > Parth
> > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at
> 8:39
> > >> AM,
> > >> >> > > Ashish
> > >> >> > > > > > Singh
> > >> >> > > > > > > <
> > >> >> > > > > > > > > >> > > > >> asingh@cloudera.com
> > >> >> > > > > > > > > >> > > > >> >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > wrote:
> > >> >> > > > > > > > > >> > > > >> >> > >> >> >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at
> > >> 11:26 AM,
> > >> >> > > > > Harsha <
> > >> >> > > > > > > > > >> > > > kafka@harsha.io>
> > >> >> > > > > > > > > >> > > > >> >> > wrote:
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Unifying the two
> > >> discussion
> > >> >> > > threads
> > >> >> > > > on
> > >> >> > > > > > > this
> > >> >> > > > > > > > > KIP.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Here is the response
> > from
> > >> >> > Jitendra
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > "The need for a large
> > >> number of
> > >> >> > > > > clients
> > >> >> > > > > > > that
> > >> >> > > > > > > > > are
> > >> >> > > > > > > > > >> > > > running
> > >> >> > > > > > > > > >> > > > >> all
> > >> >> > > > > > > > > >> > > > >> >> > over the
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > cluster that
> > authenticate
> > >> with
> > >> >> > > Kafka
> > >> >> > > > > > > > brokers,
> > >> >> > > > > > > > > >> is very
> > >> >> > > > > > > > > >> > > > >> similar
> > >> >> > > > > > > > > >> > > > >> >> > to the
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Hadoop use case of
> large
> > >> number
> > >> >> > of
> > >> >> > > > > tasks
> > >> >> > > > > > > > > running
> > >> >> > > > > > > > > >> > > across
> > >> >> > > > > > > > > >> > > > >> the
> > >> >> > > > > > > > > >> > > > >> >> > cluster
> > >> >> > > > > > > > > >> > > > >> >> > >> >> that
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need authentication to
> > >> Hdfs
> > >> >> > > > Namenode.
> > >> >> > > > > > > > > >> Therefore, the
> > >> >> > > > > > > > > >> > > > >> >> > delegation token
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > approach does seem
> like
> > a
> > >> good
> > >> >> > fit
> > >> >> > > > for
> > >> >> > > > > > > this
> > >> >> > > > > > > > > use
> > >> >> > > > > > > > > >> case
> > >> >> > > > > > > > > >> > > > as we
> > >> >> > > > > > > > > >> > > > >> >> > have seen
> > >> >> > > > > > > > > >> > > > >> >> > >> >> it
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > working at large scale
> > in
> > >> HDFS
> > >> >> > and
> > >> >> > > > > YARN.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >   The proposed design
> is
> > >> very
> > >> >> > much
> > >> >> > > > > > inline
> > >> >> > > > > > > > with
> > >> >> > > > > > > > > >> Hadoop
> > >> >> > > > > > > > > >> > > > >> >> > approach. A few
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >   comments:
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 1) Why do you guys
> want
> > >> to allow
> > >> >> > > > > > infinite
> > >> >> > > > > > > > > >> renewable
> > >> >> > > > > > > > > >> > > > >> lifetime
> > >> >> > > > > > > > > >> > > > >> >> > for a
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > token? HDFS restricts
> a
> > >> token
> > >> >> > to a
> > >> >> > > > max
> > >> >> > > > > > > life
> > >> >> > > > > > > > > time
> > >> >> > > > > > > > > >> > > > (default
> > >> >> > > > > > > > > >> > > > >> 7
> > >> >> > > > > > > > > >> > > > >> >> > days).  A
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > token's vulnerability
> is
> > >> >> > believed
> > >> >> > > to
> > >> >> > > > > > > > increase
> > >> >> > > > > > > > > >> with
> > >> >> > > > > > > > > >> > > > time.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > I agree that having
> > infinite
> > >> >> > > lifetime
> > >> >> > > > > > might
> > >> >> > > > > > > > not
> > >> >> > > > > > > > > >> be the
> > >> >> > > > > > > > > >> > > > best
> > >> >> > > > > > > > > >> > > > >> idea.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 2) As I understand the
> > >> tokens
> > >> >> > are
> > >> >> > > > > stored
> > >> >> > > > > > > in
> > >> >> > > > > > > > > >> zookeeper
> > >> >> > > > > > > > > >> > > > as
> > >> >> > > > > > > > > >> > > > >> well,
> > >> >> > > > > > > > > >> > > > >> >> > and
> > >> >> > > > > > > > > >> > > > >> >> > >> >> can
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > be updated there. This
> > is
> > >> clever
> > >> >> > > as
> > >> >> > > > it
> > >> >> > > > > > can
> > >> >> > > > > > > > > allow
> > >> >> > > > > > > > > >> > > > >> replacing the
> > >> >> > > > > > > > > >> > > > >> >> > tokens
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > once they run out of
> max
> > >> life
> > >> >> > > time,
> > >> >> > > > > and
> > >> >> > > > > > > > > clients
> > >> >> > > > > > > > > >> can
> > >> >> > > > > > > > > >> > > > >> download
> > >> >> > > > > > > > > >> > > > >> >> > new
> > >> >> > > > > > > > > >> > > > >> >> > >> >> tokens
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from zookeeper. It
> > >> shouldn't be
> > >> >> > a
> > >> >> > > > big
> > >> >> > > > > > load
> > >> >> > > > > > > > on
> > >> >> > > > > > > > > >> > > zookeeper
> > >> >> > > > > > > > > >> > > > >> as a
> > >> >> > > > > > > > > >> > > > >> >> > client
> > >> >> > > > > > > > > >> > > > >> >> > >> >> will
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need to get a new
> token
> > >> once in
> > >> >> > > > > several
> > >> >> > > > > > > > days.
> > >> >> > > > > > > > > >> In this
> > >> >> > > > > > > > > >> > > > >> approach
> > >> >> > > > > > > > > >> > > > >> >> > you
> > >> >> > > > > > > > > >> > > > >> >> > >> >> don't
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need infinite lifetime
> > on
> > >> the
> > >> >> > > token
> > >> >> > > > > even
> > >> >> > > > > > > for
> > >> >> > > > > > > > > >> long
> > >> >> > > > > > > > > >> > > > running
> > >> >> > > > > > > > > >> > > > >> >> > clients.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 3) The token password
> > are
> > >> >> > > generated
> > >> >> > > > > > using
> > >> >> > > > > > > a
> > >> >> > > > > > > > > >> master
> > >> >> > > > > > > > > >> > > key.
> > >> >> > > > > > > > > >> > > > >> The
> > >> >> > > > > > > > > >> > > > >> >> > master
> > >> >> > > > > > > > > >> > > > >> >> > >> >> key
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > should also be
> > >> periodically
> > >> >> > > changed.
> > >> >> > > > > In
> > >> >> > > > > > > > > Hadoop,
> > >> >> > > > > > > > > >> the
> > >> >> > > > > > > > > >> > > > >> default
> > >> >> > > > > > > > > >> > > > >> >> > renewal
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > period is 1 day.?
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > IIUC, this will require
> > >> brokers
> > >> >> > > > > > maintaining
> > >> >> > > > > > > a
> > >> >> > > > > > > > > >> list of X
> > >> >> > > > > > > > > >> > > > most
> > >> >> > > > > > > > > >> > > > >> >> > recent
> > >> >> > > > > > > > > >> > > > >> >> > >> >> master
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > keys. This list will
> have
> > >> to be
> > >> >> > > > > persisted
> > >> >> > > > > > > > > >> somewhere, as
> > >> >> > > > > > > > > >> > > > if a
> > >> >> > > > > > > > > >> > > > >> >> > broker
> > >> >> > > > > > > > > >> > > > >> >> > >> >> goes
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > down it will have to get
> > >> that list
> > >> >> > > > again
> > >> >> > > > > > and
> > >> >> > > > > > > > > >> storing
> > >> >> > > > > > > > > >> > > > master
> > >> >> > > > > > > > > >> > > > >> keys
> > >> >> > > > > > > > > >> > > > >> >> > on ZK
> > >> >> > > > > > > > > >> > > > >> >> > >> >> is
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > not the best idea.
> > However,
> > >> if a
> > >> >> > > > broker
> > >> >> > > > > > goes
> > >> >> > > > > > > > > down
> > >> >> > > > > > > > > >> then
> > >> >> > > > > > > > > >> > > we
> > >> >> > > > > > > > > >> > > > >> have
> > >> >> > > > > > > > > >> > > > >> >> > much
> > >> >> > > > > > > > > >> > > > >> >> > >> >> bigger
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > issue to deal with and
> > >> client can
> > >> >> > > > always
> > >> >> > > > > > > > > >> > > re-authenticate
> > >> >> > > > > > > > > >> > > > is
> > >> >> > > > > > > > > >> > > > >> such
> > >> >> > > > > > > > > >> > > > >> >> > >> >> events.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > Did you happen to take a
> > >> look at
> > >> >> > > other
> > >> >> > > > > > > > > >> alternatives
> > >> >> > > > > > > > > >> > > this
> > >> >> > > > > > > > > >> > > > >> list has
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > suggested?
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Thanks for a thorough
> > >> proposal,
> > >> >> > > > great
> > >> >> > > > > > > work!"
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016,
> at
> > >> 10:28
> > >> >> > PM,
> > >> >> > > > Gwen
> > >> >> > > > > > > > Shapira
> > >> >> > > > > > > > > >> wrote:
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > Makes sense to me.
> > >> Thanks!
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016
> at
> > >> 9:25
> > >> >> > PM,
> > >> >> > > > > > Harsha <
> > >> >> > > > > > > > > >> > > > kafka@harsha.io
> > >> >> > > > > > > > > >> > > > >> >
> > >> >> > > > > > > > > >> > > > >> >> > wrote:
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > It doesn't need
> any
> > >> release
> > >> >> > > > > vehicle
> > >> >> > > > > > > but
> > >> >> > > > > > > > > >> still the
> > >> >> > > > > > > > > >> > > > >> work can
> > >> >> > > > > > > > > >> > > > >> >> > move
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > forward.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > If anyone is
> > >> interested in
> > >> >> > the
> > >> >> > > > KIP
> > >> >> > > > > > > > please
> > >> >> > > > > > > > > >> do the
> > >> >> > > > > > > > > >> > > > >> review and
> > >> >> > > > > > > > > >> > > > >> >> > >> >> provide
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > the
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > comments.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > -Harsha
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7,
> 2016,
> > >> at
> > >> >> > 04:59
> > >> >> > > > PM,
> > >> >> > > > > > > Ismael
> > >> >> > > > > > > > > >> Juma
> > >> >> > > > > > > > > >> > > > wrote:
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> I agree that it
> > >> would be
> > >> >> > good
> > >> >> > > > to
> > >> >> > > > > > have
> > >> >> > > > > > > > > more
> > >> >> > > > > > > > > >> time
> > >> >> > > > > > > > > >> > > to
> > >> >> > > > > > > > > >> > > > >> review
> > >> >> > > > > > > > > >> > > > >> >> > and
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > discuss
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> KIP-48.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> Ismael
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8,
> 2016
> > >> at
> > >> >> > 12:55
> > >> >> > > > AM,
> > >> >> > > > > > Gwen
> > >> >> > > > > > > > > >> Shapira <
> > >> >> > > > > > > > > >> > > > >> >> > >> >> gwen@confluent.io>
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Hi Team,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Since KIP-48
> > >> depends on
> > >> >> > > > KIP-43,
> > >> >> > > > > > > which
> > >> >> > > > > > > > > is
> > >> >> > > > > > > > > >> > > > already a
> > >> >> > > > > > > > > >> > > > >> bit
> > >> >> > > > > > > > > >> > > > >> >> > of a
> > >> >> > > > > > > > > >> > > > >> >> > >> >> risk
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > for
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > the next
> release
> > -
> > >> any
> > >> >> > > chance
> > >> >> > > > > we
> > >> >> > > > > > > can
> > >> >> > > > > > > > > >> delay
> > >> >> > > > > > > > > >> > > > >> delegation
> > >> >> > > > > > > > > >> > > > >> >> > tokens
> > >> >> > > > > > > > > >> > > > >> >> > >> >> to
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Kafka
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > 0.10.1?
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > With the
> > community
> > >> >> > working
> > >> >> > > > on a
> > >> >> > > > > > > > release
> > >> >> > > > > > > > > >> every
> > >> >> > > > > > > > > >> > > 3
> > >> >> > > > > > > > > >> > > > >> month,
> > >> >> > > > > > > > > >> > > > >> >> > this
> > >> >> > > > > > > > > >> > > > >> >> > >> >> is not
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a huge
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > delay.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Gwen
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26,
> > >> 2016 at
> > >> >> > > 5:11
> > >> >> > > > > PM,
> > >> >> > > > > > > > Ashish
> > >> >> > > > > > > > > >> Singh
> > >> >> > > > > > > > > >> > > <
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > asingh@cloudera.com>
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Parth,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Thanks again
> > for
> > >> the
> > >> >> > > > awesome
> > >> >> > > > > > > write
> > >> >> > > > > > > > > up.
> > >> >> > > > > > > > > >> > > > Following
> > >> >> > > > > > > > > >> > > > >> our
> > >> >> > > > > > > > > >> > > > >> >> > >> >> discussion
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from the
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I think
> > it
> > >> will
> > >> >> > be
> > >> >> > > > > easier
> > >> >> > > > > > > to
> > >> >> > > > > > > > > >> compare
> > >> >> > > > > > > > > >> > > > >> various
> > >> >> > > > > > > > > >> > > > >> >> > >> >> alternatives
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > if they
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > are
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > listed
> > together.
> > >> I am
> > >> >> > > > stating
> > >> >> > > > > > > > below a
> > >> >> > > > > > > > > >> few
> > >> >> > > > > > > > > >> > > > >> >> > alternatives along
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > with
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a the
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > current
> > proposal.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Current
> > >> proposal)
> > >> >> > Store
> > >> >> > > > > > > Delegation
> > >> >> > > > > > > > > >> Token,
> > >> >> > > > > > > > > >> > > DT,
> > >> >> > > > > > > > > >> > > > >> on ZK.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> > >> >> > > authenticates
> > >> >> > > > > > with a
> > >> >> > > > > > > > > >> broker.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a
> > >> client is
> > >> >> > > > > > > > authenticated,
> > >> >> > > > > > > > > >> it
> > >> >> > > > > > > > > >> > > will
> > >> >> > > > > > > > > >> > > > >> make a
> > >> >> > > > > > > > > >> > > > >> >> > broker
> > >> >> > > > > > > > > >> > > > >> >> > >> >> side
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a
> > >> delegation
> > >> >> > > > token.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The
> > broker
> > >> >> > > generates
> > >> >> > > > a
> > >> >> > > > > > > shared
> > >> >> > > > > > > > > >> secret
> > >> >> > > > > > > > > >> > > > based
> > >> >> > > > > > > > > >> > > > >> on
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > HMAC-SHA256(a
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> Password/Secret
> > >> >> > shared
> > >> >> > > > > > between
> > >> >> > > > > > > > all
> > >> >> > > > > > > > > >> > > brokers,
> > >> >> > > > > > > > > >> > > > >> >> > randomly
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > generated
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > tokenId).
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Broker
> > >> stores
> > >> >> > this
> > >> >> > > > > token
> > >> >> > > > > > in
> > >> >> > > > > > > > its
> > >> >> > > > > > > > > >> in
> > >> >> > > > > > > > > >> > > > memory
> > >> >> > > > > > > > > >> > > > >> cache.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> Broker
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > also stores
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the
> > >> DelegationToken
> > >> >> > > > > without
> > >> >> > > > > > > the
> > >> >> > > > > > > > > >> hmac in
> > >> >> > > > > > > > > >> > > the
> > >> >> > > > > > > > > >> > > > >> >> > zookeeper.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. All
> > >> brokers will
> > >> >> > > > have a
> > >> >> > > > > > > cache
> > >> >> > > > > > > > > >> backed
> > >> >> > > > > > > > > >> > > by
> > >> >> > > > > > > > > >> > > > >> >> > zookeeper so
> > >> >> > > > > > > > > >> > > > >> >> > >> >> they
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > will all
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    get
> notified
> > >> >> > whenever
> > >> >> > > a
> > >> >> > > > > new
> > >> >> > > > > > > > token
> > >> >> > > > > > > > > is
> > >> >> > > > > > > > > >> > > > >> generated and
> > >> >> > > > > > > > > >> > > > >> >> > they
> > >> >> > > > > > > > > >> > > > >> >> > >> >> will
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > update
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    local
> cache
> > >> whenever
> > >> >> > > > token
> > >> >> > > > > > > state
> > >> >> > > > > > > > > >> changes.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Broker
> > >> returns
> > >> >> > the
> > >> >> > > > > token
> > >> >> > > > > > to
> > >> >> > > > > > > > > >> Client.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable
> issues
> > >> and
> > >> >> > fixes
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1.
> Probable
> > >> race
> > >> >> > > > > condition,
> > >> >> > > > > > > > client
> > >> >> > > > > > > > > >> tries
> > >> >> > > > > > > > > >> > > to
> > >> >> > > > > > > > > >> > > > >> >> > authenticate
> > >> >> > > > > > > > > >> > > > >> >> > >> >> with
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a broker
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    that is
> yet
> > >> to be
> > >> >> > > > updated
> > >> >> > > > > > with
> > >> >> > > > > > > > the
> > >> >> > > > > > > > > >> newly
> > >> >> > > > > > > > > >> > > > >> generated
> > >> >> > > > > > > > > >> > > > >> >> > DT.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> This
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > can
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > probably be
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    dealt with
> > >> making
> > >> >> > > > > dtRequest
> > >> >> > > > > > > > block
> > >> >> > > > > > > > > >> until
> > >> >> > > > > > > > > >> > > all
> > >> >> > > > > > > > > >> > > > >> >> > brokers have
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > updated
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their DT
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    cache. Zk
> > >> barrier or
> > >> >> > > > > similar
> > >> >> > > > > > > > > >> mechanism
> > >> >> > > > > > > > > >> > > can
> > >> >> > > > > > > > > >> > > > be
> > >> >> > > > > > > > > >> > > > >> used.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> However,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > all such
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    mechanisms
> > >> will
> > >> >> > > increase
> > >> >> > > > > > > > > complexity.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Using a
> > >> static
> > >> >> > > secret
> > >> >> > > > > key
> > >> >> > > > > > > > from
> > >> >> > > > > > > > > >> config
> > >> >> > > > > > > > > >> > > > >> file. Will
> > >> >> > > > > > > > > >> > > > >> >> > >> >> require
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > yet
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > another
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    config and
> > >> uses a
> > >> >> > > static
> > >> >> > > > > > > secret
> > >> >> > > > > > > > > >> key. It
> > >> >> > > > > > > > > >> > > is
> > >> >> > > > > > > > > >> > > > >> advised
> > >> >> > > > > > > > > >> > > > >> >> > to
> > >> >> > > > > > > > > >> > > > >> >> > >> >> rotate
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > secret
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > keys
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > periodically.
> > >> This
> > >> >> > can
> > >> >> > > > be
> > >> >> > > > > > > > avoided
> > >> >> > > > > > > > > >> with
> > >> >> > > > > > > > > >> > > > >> controller
> > >> >> > > > > > > > > >> > > > >> >> > >> >> generating
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > secretKey and
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    passing to
> > >> brokers
> > >> >> > > > > > > periodically.
> > >> >> > > > > > > > > >> However,
> > >> >> > > > > > > > > >> > > > >> this will
> > >> >> > > > > > > > > >> > > > >> >> > >> >> require
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > brokers to
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maintain
> > >> certain
> > >> >> > > counts
> > >> >> > > > of
> > >> >> > > > > > > > > >> secretKeys.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative
> 1)
> > >> Have
> > >> >> > > > > controller
> > >> >> > > > > > > > > >> generate
> > >> >> > > > > > > > > >> > > > >> delegation
> > >> >> > > > > > > > > >> > > > >> >> > token.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> > >> >> > > authenticates
> > >> >> > > > > > with a
> > >> >> > > > > > > > > >> broker.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a
> > >> client is
> > >> >> > > > > > > > authenticated,
> > >> >> > > > > > > > > >> it
> > >> >> > > > > > > > > >> > > will
> > >> >> > > > > > > > > >> > > > >> make a
> > >> >> > > > > > > > > >> > > > >> >> > broker
> > >> >> > > > > > > > > >> > > > >> >> > >> >> side
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a
> > >> delegation
> > >> >> > > > token.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. Broker
> > >> forwards
> > >> >> > the
> > >> >> > > > > > request
> > >> >> > > > > > > > to
> > >> >> > > > > > > > > >> > > > controller.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4.
> > Controller
> > >> >> > > generates
> > >> >> > > > a
> > >> >> > > > > DT
> > >> >> > > > > > > and
> > >> >> > > > > > > > > >> > > broadcasts
> > >> >> > > > > > > > > >> > > > >> to all
> > >> >> > > > > > > > > >> > > > >> >> > >> >> brokers.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. Broker
> > >> stores
> > >> >> > this
> > >> >> > > > > token
> > >> >> > > > > > in
> > >> >> > > > > > > > its
> > >> >> > > > > > > > > >> memory
> > >> >> > > > > > > > > >> > > > >> cache.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6.
> > Controller
> > >> >> > responds
> > >> >> > > > to
> > >> >> > > > > > > > broker’s
> > >> >> > > > > > > > > >> DT
> > >> >> > > > > > > > > >> > > req.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    7. Broker
> > >> returns
> > >> >> > the
> > >> >> > > > > token
> > >> >> > > > > > to
> > >> >> > > > > > > > > >> Client.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable
> issues
> > >> and
> > >> >> > fixes
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. We will
> > >> have to
> > >> >> > add
> > >> >> > > > new
> > >> >> > > > > > > APIs
> > >> >> > > > > > > > to
> > >> >> > > > > > > > > >> > > support
> > >> >> > > > > > > > > >> > > > >> >> > controller
> > >> >> > > > > > > > > >> > > > >> >> > >> >> pushing
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > to
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    brokers on
> > >> top of
> > >> >> > the
> > >> >> > > > > > minimal
> > >> >> > > > > > > > APIs
> > >> >> > > > > > > > > >> that
> > >> >> > > > > > > > > >> > > are
> > >> >> > > > > > > > > >> > > > >> >> > currently
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > proposed.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. We will
> > >> also have
> > >> >> > > to
> > >> >> > > > > add
> > >> >> > > > > > > APIs
> > >> >> > > > > > > > > to
> > >> >> > > > > > > > > >> > > support
> > >> >> > > > > > > > > >> > > > >> the
> > >> >> > > > > > > > > >> > > > >> >> > >> >> bootstrapping
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > case,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > i.e,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    when a new
> > >> broker
> > >> >> > > comes
> > >> >> > > > up
> > >> >> > > > > > it
> > >> >> > > > > > > > will
> > >> >> > > > > > > > > >> have
> > >> >> > > > > > > > > >> > > to
> > >> >> > > > > > > > > >> > > > >> get all
> > >> >> > > > > > > > > >> > > > >> >> > >> >> delegation
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > from
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the
> > >> controller.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. In
> > >> catastrophic
> > >> >> > > > > failures
> > >> >> > > > > > > > where
> > >> >> > > > > > > > > >> all
> > >> >> > > > > > > > > >> > > > brokers
> > >> >> > > > > > > > > >> > > > >> go
> > >> >> > > > > > > > > >> > > > >> >> > down,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> the
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost
> even
> > >> if
> > >> >> > > servers
> > >> >> > > > > are
> > >> >> > > > > > > > > >> restarted as
> > >> >> > > > > > > > > >> > > > >> tokens
> > >> >> > > > > > > > > >> > > > >> >> > are not
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this
> > >> happens,
> > >> >> > then
> > >> >> > > > > there
> > >> >> > > > > > > are
> > >> >> > > > > > > > > more
> > >> >> > > > > > > > > >> > > > important
> > >> >> > > > > > > > > >> > > > >> >> > things to
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > worry
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > about
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it
> is
> > >> better
> > >> >> > to
> > >> >> > > > > > > > > >> re-authenticate.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative
> 2)
> > >> Do not
> > >> >> > > > > > distribute
> > >> >> > > > > > > > DT
> > >> >> > > > > > > > > to
> > >> >> > > > > > > > > >> > > other
> > >> >> > > > > > > > > >> > > > >> brokers
> > >> >> > > > > > > > > >> > > > >> >> > at
> > >> >> > > > > > > > > >> > > > >> >> > >> >> all.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> > >> >> > > authenticates
> > >> >> > > > > > with a
> > >> >> > > > > > > > > >> broker.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a
> > >> client is
> > >> >> > > > > > > > authenticated,
> > >> >> > > > > > > > > >> it
> > >> >> > > > > > > > > >> > > will
> > >> >> > > > > > > > > >> > > > >> make a
> > >> >> > > > > > > > > >> > > > >> >> > broker
> > >> >> > > > > > > > > >> > > > >> >> > >> >> side
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a
> > >> delegation
> > >> >> > > > token.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The
> > broker
> > >> >> > > generates
> > >> >> > > > DT
> > >> >> > > > > > of
> > >> >> > > > > > > > > form,
> > >> >> > > > > > > > > >> > > [hmac +
> > >> >> > > > > > > > > >> > > > >> (owner,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> renewer,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> maxLifeTime,
> > >> id,
> > >> >> > hmac,
> > >> >> > > > > > > > > >> expirationTime)]
> > >> >> > > > > > > > > >> > > and
> > >> >> > > > > > > > > >> > > > >> passes
> > >> >> > > > > > > > > >> > > > >> >> > back
> > >> >> > > > > > > > > >> > > > >> >> > >> >> this
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > DT to
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > client.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    hmac is
> > >> generated
> > >> >> > via
> > >> >> > > > > > > > > >> {HMAC-SHA256(owner,
> > >> >> > > > > > > > > >> > > > >> renewer,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > maxLifeTime, id,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> expirationTime)
> > >> >> > using
> > >> >> > > > > > > > SecretKey}.
> > >> >> > > > > > > > > >> Note
> > >> >> > > > > > > > > >> > > that
> > >> >> > > > > > > > > >> > > > >> all
> > >> >> > > > > > > > > >> > > > >> >> > brokers
> > >> >> > > > > > > > > >> > > > >> >> > >> >> have
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > this
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > SecretKey.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Client
> > >> then goes
> > >> >> > to
> > >> >> > > > any
> > >> >> > > > > > > > broker
> > >> >> > > > > > > > > >> and to
> > >> >> > > > > > > > > >> > > > >> >> > authenticate
> > >> >> > > > > > > > > >> > > > >> >> > >> >> sends
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > the DT.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    Broker
> > >> recalculates
> > >> >> > > hmac
> > >> >> > > > > > using
> > >> >> > > > > > > > > >> (owner,
> > >> >> > > > > > > > > >> > > > >> renewer,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> maxLifeTime,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > id, hmac,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> expirationTime) info
> > >> >> > > > from
> > >> >> > > > > DT
> > >> >> > > > > > > and
> > >> >> > > > > > > > > its
> > >> >> > > > > > > > > >> > > > >> SecretKey. If
> > >> >> > > > > > > > > >> > > > >> >> > it
> > >> >> > > > > > > > > >> > > > >> >> > >> >> matches
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > with
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac of
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    DT, client
> > is
> > >> >> > > > > authenticated.
> > >> >> > > > > > > > Yes,
> > >> >> > > > > > > > > >> it will
> > >> >> > > > > > > > > >> > > > do
> > >> >> > > > > > > > > >> > > > >> other
> > >> >> > > > > > > > > >> > > > >> >> > >> >> obvious
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > checks of
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    timestamp
> > >> expiry and
> > >> >> > > > such.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Note that
> > secret
> > >> key
> > >> >> > will
> > >> >> > > > be
> > >> >> > > > > > > > > generated
> > >> >> > > > > > > > > >> by
> > >> >> > > > > > > > > >> > > > >> controller
> > >> >> > > > > > > > > >> > > > >> >> > and
> > >> >> > > > > > > > > >> > > > >> >> > >> >> passed
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > to
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > brokers
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > periodically.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable
> issues
> > >> and
> > >> >> > fixes
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. How to
> > >> delete a
> > >> >> > DT?
> > >> >> > > > > Yes,
> > >> >> > > > > > > that
> > >> >> > > > > > > > > is
> > >> >> > > > > > > > > >> a
> > >> >> > > > > > > > > >> > > > downside
> > >> >> > > > > > > > > >> > > > >> >> > here.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> However,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > this can
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be handled
> > >> with
> > >> >> > > brokers
> > >> >> > > > > > > > > maintaining
> > >> >> > > > > > > > > >> a
> > >> >> > > > > > > > > >> > > > >> blacklist of
> > >> >> > > > > > > > > >> > > > >> >> > DTs,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> DTs
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from this
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > list
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    can be
> > >> removed after
> > >> >> > > > > expiry.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. In
> > >> catastrophic
> > >> >> > > > > failures
> > >> >> > > > > > > > where
> > >> >> > > > > > > > > >> all
> > >> >> > > > > > > > > >> > > > brokers
> > >> >> > > > > > > > > >> > > > >> go
> > >> >> > > > > > > > > >> > > > >> >> > down,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> the
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost
> even
> > >> if
> > >> >> > > servers
> > >> >> > > > > are
> > >> >> > > > > > > > > >> restarted as
> > >> >> > > > > > > > > >> > > > >> tokens
> > >> >> > > > > > > > > >> > > > >> >> > are not
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this
> > >> happens,
> > >> >> > then
> > >> >> > > > > there
> > >> >> > > > > > > are
> > >> >> > > > > > > > > more
> > >> >> > > > > > > > > >> > > > important
> > >> >> > > > > > > > > >> > > > >> >> > things to
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > worry
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > about
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it
> is
> > >> better
> > >> >> > to
> > >> >> > > > > > > > > >> re-authenticate.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb
> 26,
> > >> 2016 at
> > >> >> > > > 1:58
> > >> >> > > > > > PM,
> > >> >> > > > > > > > > Parth
> > >> >> > > > > > > > > >> > > > >> Brahmbhatt <
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > pbrahmbhatt@hortonworks.com>
> > >> >> > > > > > > > wrote:
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Hi,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> I have filed
> > >> KIP-48 so
> > >> >> > > we
> > >> >> > > > > can
> > >> >> > > > > > > > offer
> > >> >> > > > > > > > > >> hadoop
> > >> >> > > > > > > > > >> > > > like
> > >> >> > > > > > > > > >> > > > >> >> > delegation
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens in
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> kafka. You
> can
> > >> review
> > >> >> > > the
> > >> >> > > > > > design
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > >> >> > > > > > > > > >> > > > >> >> >
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > >
> > >> >> > > > > > > > > >>
> > >> >> > > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > .
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> This KIP
> > >> depends on
> > >> >> > > KIP-43
> > >> >> > > > > and
> > >> >> > > > > > > we
> > >> >> > > > > > > > > >> have also
> > >> >> > > > > > > > > >> > > > >> >> > discussed an
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > alternative to
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> proposed
> > design
> > >> here<
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > >> >> > > > > > > > > >> > > > >> >> >
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > >
> > >> >> > > > > > > > > >>
> > >> >> > > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >>
> >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >.
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Thanks
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Parth
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > --
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Regards,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Ashish
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > --
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > Regards,
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > > Ashish
> > >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> > >> >> > > > > > > > > >> > > > >> >> > >> >>
> > >> >> > > > > > > > > >> > > > >> >> >
> > >> >> > > > > > > > > >> > > > >>
> > >> >> > > > > > > > > >> > > >
> > >> >> > > > > > > > > >> > >
> > >> >> > > > > > > > > >>
> > >> >> > > > > > > > > >>
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > >
> > >> >> > > > > >
> > >> >> > > > > >
> > >> >> > > > > > --
> > >> >> > > > > > Regards,
> > >> >> > > > > >
> > >> >> > > > > > Rajini
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > > >
> > >> >> > > >
> > >> >> > > > --
> > >> >> > > > Regards,
> > >> >> > > >
> > >> >> > > > Rajini
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >>
> > >> >>
> > >> >> --
> > >> >> Liquan Pei
> > >> >> Software Engineer, Confluent Inc
> > >>
> > >
> > >
> >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[parth brahmbhatt <br...@gmail.com>]

1. Who / how are tokens renewed? By original requester only? or using Kerberos
auth only?
My recommendation is to do this only using Kerberos auth and only threw the
renewer specified during the acquisition request.

2. Are tokens stored on each broker or in ZK?
My recommendation is still to store in ZK or not store them at all. The
whole controller based distribution is too much overhead with not much to
achieve.

3. How are tokens invalidated / expired?
Either by expiration time out or through an explicit request to invalidate.

4. Which encryption algorithm is used?
SCRAM

5. What is the impersonation proposal (it wasn't in the KIP but was discussed
in this thread)?
There is no imperonation proposal. I tried and explained how its a
different problem and why its not really necessary to discuss that as part
of this KIP.  This KIP will not support any impersonation, it will just be
another way to authenticate.

6. Do we need new ACLs, if so - for what actions?
We do not need new ACLs.

7. How would the delegation token be configured in the client?
Should be through config. I wasn't planning on supporting JAAS for tokens.
I don't believe hadoop does this either.

Thanks
Parth



On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao <ju...@confluent.io> wrote:

> Harsha,
>
> Another question.
>
> 9. How would the delegation token be configured in the client? The standard
> way is to do this through JAAS. However, we will need to think through if
> this is convenient in a shared environment. For example, when a new task is
> added to a Storm worker node, do we need to dynamically add a new section
> in the JAAS file? It may be more convenient if we can pass in the token
> through the config directly w/o going through JAAS.
>
> Are you or Parth still actively working on this KIP?
>
> Thanks,
>
> Jun
>
>
>
> On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <ju...@confluent.io> wrote:
>
> > Just to add on that list.
> >
> > 2. It would be good to document the format of the data stored in ZK.
> > 7. Earlier, there was a discussion on whether the tokens should be
> > propagated through ZK like config/acl/quota, or through the controller.
> > Currently, the controller is only designed for propagating topic
> metadata,
> > but not other data.
> > 8. Should we use SCRAM to send the token instead of DIGEST-MD5 since it's
> > deprecated?
> >
> > Also, the images in the wiki seem broken.
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <gw...@confluent.io>
> wrote:
> >
> >> From what I can see, remaining questions are:
> >>
> >> 1. Who / how are tokens renewed? By original requester only? or using
> >> Kerberos auth only?
> >> 2. Are tokens stored on each broker or in ZK?
> >> 3. How are tokens invalidated / expired?
> >> 4. Which encryption algorithm is used?
> >> 5. What is the impersonation proposal (it wasn't in the KIP but was
> >> discussed in this thread)?
> >> 6. Do we need new ACLs, if so - for what actions?
> >>
> >> Gwen
> >>
> >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <ka...@harsha.io> wrote:
> >> > Jun & Ismael,
> >> >                          Unfortunately I couldn't attend the KIP
> meeting
> >> >                          when delegation tokens discussed. Appreciate
> if
> >> >                          you can update the thread if you have any
> >> >                          further questions.
> >> > Thanks,
> >> > Harsha
> >> >
> >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
> >> >> It seems that the links to images in the KIP are broken.
> >> >>
> >> >> Liquan
> >> >>
> >> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
> >> >> brahmbhatt.parth@gmail.com> wrote:
> >> >>
> >> >> > 110. What does getDelegationTokenAs mean?
> >> >> > In the current proposal we only allow a user to get delegation
> token
> >> for
> >> >> > the identity that it authenticated as using another mechanism, i.e.
> >> A user
> >> >> > that authenticate using a keytab for principal user1@EXAMPLE.COM
> >> will get
> >> >> > delegation tokens for that user only. In future I think we will
> have
> >> to
> >> >> > extend support such that we allow some set of users (
> >> >> > kafka-rest-user@EXAMPLE.COM, storm-nimbus@EXAMPLE.COM) to acquire
> >> >> > delegation tokens on behalf of other users whose identity they have
> >> >> > verified independently.  Kafka brokers will have ACLs to control
> >> which
> >> >> > users are allowed to impersonate other users and get tokens on
> >> behalf of
> >> >> > them. Overall Impersonation is a whole different problem in my
> >> opinion and
> >> >> > I think we can tackle it in separate KIP.
> >> >> >
> >> >> > 111. What's the typical rate of getting and renewing delegation
> >> tokens?
> >> >> > Typically this should be very very low, 1 request per minute is a
> >> >> > relatively high estimate. However it depends on the token
> >> expiration. I am
> >> >> > less worried about the extra load it puts on controller vs the
> added
> >> >> > complexity and the value it offers.
> >> >> >
> >> >> > Thanks
> >> >> > Parth
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <is...@juma.me.uk>
> >> wrote:
> >> >> >
> >> >> > > Thanks Rajini. It would probably require a separate KIP as it
> will
> >> >> > > introduce user visible changes. We could also update KIP-48 to
> >> have this
> >> >> > > information, but it seems cleaner to do it separately. We can
> >> discuss
> >> >> > that
> >> >> > > in the KIP call today.
> >> >> > >
> >> >> > > Ismael
> >> >> > >
> >> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <
> >> >> > > rajinisivaram@googlemail.com> wrote:
> >> >> > >
> >> >> > > > Ismael,
> >> >> > > >
> >> >> > > > I have created a JIRA (
> >> >> > https://issues.apache.org/jira/browse/KAFKA-3751)
> >> >> > > > for adding SCRAM as a SASL mechanism. Would that need another
> >> KIP? If
> >> >> > > > KIP-48 will use this mechanism, can this just be a JIRA that
> gets
> >> >> > > reviewed
> >> >> > > > when the PR is ready?
> >> >> > > >
> >> >> > > > Thank you,
> >> >> > > >
> >> >> > > > Rajini
> >> >> > > >
> >> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <
> ismael@juma.me.uk>
> >> >> > wrote:
> >> >> > > >
> >> >> > > > > Thanks Rajini, SCRAM seems like a good candidate.
> >> >> > > > >
> >> >> > > > > Gwen had independently mentioned this as a SASL mechanism
> that
> >> might
> >> >> > be
> >> >> > > > > useful for Kafka and I have been meaning to check it in more
> >> detail.
> >> >> > > Good
> >> >> > > > > to know that you are willing to contribute an implementation.
> >> Maybe
> >> >> > we
> >> >> > > > > should file a separate JIRA for this?
> >> >> > > > >
> >> >> > > > > Ismael
> >> >> > > > >
> >> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram <
> >> >> > > > > rajinisivaram@googlemail.com> wrote:
> >> >> > > > >
> >> >> > > > > > SCRAM (Salted Challenge Response Authentication Mechanism)
> >> is a
> >> >> > > better
> >> >> > > > > > mechanism than Digest-MD5. Java doesn't come with a
> built-in
> >> SCRAM
> >> >> > > > > > SaslServer or SaslClient, but I will be happy to add
> support
> >> in
> >> >> > Kafka
> >> >> > > > > since
> >> >> > > > > > it would be a useful mechanism to support anyway.
> >> >> > > > > > https://tools.ietf.org/html/rfc7677 describes the protocol
> >> for
> >> >> > > > > > SCRAM-SHA-256.
> >> >> > > > > >
> >> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <jun@confluent.io
> >
> >> wrote:
> >> >> > > > > >
> >> >> > > > > > > Parth,
> >> >> > > > > > >
> >> >> > > > > > > Thanks for the explanation. A couple of more questions.
> >> >> > > > > > >
> >> >> > > > > > > 110. What does getDelegationTokenAs mean?
> >> >> > > > > > >
> >> >> > > > > > > 111. What's the typical rate of getting and renewing
> >> delegation
> >> >> > > > tokens?
> >> >> > > > > > > That may have an impact on whether they should be
> directed
> >> to the
> >> >> > > > > > > controller.
> >> >> > > > > > >
> >> >> > > > > > > Jun
> >> >> > > > > > >
> >> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt <
> >> >> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> >> >> > > > > > >
> >> >> > > > > > > > Hi Jun,
> >> >> > > > > > > >
> >> >> > > > > > > > Thanks for reviewing.
> >> >> > > > > > > >
> >> >> > > > > > > > * We could add a Cluster action to add acls on who can
> >> request
> >> >> > > > > > delegation
> >> >> > > > > > > > tokens. I don't see the use case for that yet but down
> >> the line
> >> >> > > > when
> >> >> > > > > we
> >> >> > > > > > > > start supporting getDelegationTokenAs it will be
> >> necessary.
> >> >> > > > > > > > * Yes we recommend tokens to be only used/distributed
> >> over
> >> >> > secure
> >> >> > > > > > > channels.
> >> >> > > > > > > > * Depending on what design we end up choosing
> >> Invalidation will
> >> >> > > be
> >> >> > > > > > > > responsibility of every broker or controller.
> >> >> > > > > > > > * I am not sure if I documented somewhere that
> >> invalidation
> >> >> > will
> >> >> > > > > > directly
> >> >> > > > > > > > go through zookeeper but that is not the intent.
> >> Invalidation
> >> >> > > will
> >> >> > > > > > either
> >> >> > > > > > > > be request based or due to expiration. No direct
> >> zookeeper
> >> >> > > > > interaction
> >> >> > > > > > > from
> >> >> > > > > > > > any client.
> >> >> > > > > > > > * "Broker also stores the DelegationToken without the
> >> hmac in
> >> >> > the
> >> >> > > > > > > > zookeeper." : Sorry about the confusion. The sole
> >> purpose of
> >> >> > > > > zookeeper
> >> >> > > > > > in
> >> >> > > > > > > > this design is as distribution channel for tokens
> >> between all
> >> >> > > > brokers
> >> >> > > > > > > and a
> >> >> > > > > > > > layer that ensures only tokens that were generated by
> >> making a
> >> >> > > > > request
> >> >> > > > > > > to a
> >> >> > > > > > > > broker will be accepted (more on this in second
> >> paragraph). The
> >> >> > > > token
> >> >> > > > > > > > consists of few elements (owner, renewer, uuid ,
> >> expiration,
> >> >> > > hmac)
> >> >> > > > ,
> >> >> > > > > > one
> >> >> > > > > > > of
> >> >> > > > > > > > which is the finally generated hmac but hmac it self is
> >> >> > derivable
> >> >> > > > if
> >> >> > > > > > you
> >> >> > > > > > > > have all the other elements of the token + secret key
> to
> >> >> > generate
> >> >> > > > > hmac.
> >> >> > > > > > > > Given zookeeper does not provide SSL support we do not
> >> want the
> >> >> > > > > entire
> >> >> > > > > > > > token to be wire transferred to zookeeper as that will
> >> be an
> >> >> > > > insecure
> >> >> > > > > > > wire
> >> >> > > > > > > > transfer. Instead we only store all the other elements
> >> of a
> >> >> > > > > delegation
> >> >> > > > > > > > tokens. Brokers can read these elements and because
> they
> >> also
> >> >> > > have
> >> >> > > > > > access
> >> >> > > > > > > > to secret key they will be able to generate hmac on
> >> their end.
> >> >> > > > > > > >
> >> >> > > > > > > > One of the alternative proposed is to avoid zookeeper
> >> >> > > altogether. A
> >> >> > > > > > > Client
> >> >> > > > > > > > will call broker with required information (owner,
> >> renwer,
> >> >> > > > > expiration)
> >> >> > > > > > > and
> >> >> > > > > > > > get back (signed hmac, uuid). Broker won't store this
> in
> >> >> > > zookeeper.
> >> >> > > > > > From
> >> >> > > > > > > > this point a client can contact any broker with all the
> >> >> > > delegation
> >> >> > > > > > token
> >> >> > > > > > > > info (owner, rewner, expiration, hmac, uuid) the borker
> >> will
> >> >> > > > > regenerate
> >> >> > > > > > > the
> >> >> > > > > > > > hmac and as long as it matches with hmac presented by
> >> client ,
> >> >> > > > broker
> >> >> > > > > > > will
> >> >> > > > > > > > allow the request to authenticate.  Only problem with
> >> this
> >> >> > > approach
> >> >> > > > > is
> >> >> > > > > > if
> >> >> > > > > > > > the secret key is compromised any client can now
> generate
> >> >> > random
> >> >> > > > > tokens
> >> >> > > > > > > and
> >> >> > > > > > > > they will still be able to authenticate as any user
> they
> >> like.
> >> >> > > with
> >> >> > > > > > > > zookeeper we guarantee that only tokens acquired via a
> >> broker
> >> >> > > > (using
> >> >> > > > > > some
> >> >> > > > > > > > auth scheme other than delegation token) will be
> >> accepted. We
> >> >> > > need
> >> >> > > > to
> >> >> > > > > > > > discuss which proposal makes more sense and we can go
> >> over it
> >> >> > in
> >> >> > > > > > > tomorrow's
> >> >> > > > > > > > meeting.
> >> >> > > > > > > >
> >> >> > > > > > > > Also, can you forward the invite to me?
> >> >> > > > > > > >
> >> >> > > > > > > > Thanks
> >> >> > > > > > > > Parth
> >> >> > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <
> >> jun@confluent.io>
> >> >> > > > wrote:
> >> >> > > > > > > >
> >> >> > > > > > > > > Thanks for the KIP. A few comments.
> >> >> > > > > > > > >
> >> >> > > > > > > > > 100. This potentially can be useful for Kafka Connect
> >> and
> >> >> > Kafka
> >> >> > > > > rest
> >> >> > > > > > > > proxy
> >> >> > > > > > > > > where a worker agent will need to run a task on
> behalf
> >> of a
> >> >> > > > client.
> >> >> > > > > > We
> >> >> > > > > > > > will
> >> >> > > > > > > > > likely need to change how those services use Kafka
> >> clients
> >> >> > > > > > > > > (producer/consumer). Instead of a shared client per
> >> worker,
> >> >> > we
> >> >> > > > will
> >> >> > > > > > > need
> >> >> > > > > > > > a
> >> >> > > > > > > > > client per user task since the authentication happens
> >> at the
> >> >> > > > > > connection
> >> >> > > > > > > > > level. For Kafka Connect, the renewer will be the
> >> workers.
> >> >> > So,
> >> >> > > we
> >> >> > > > > > > > probably
> >> >> > > > > > > > > need to allow multiple renewers. For Kafka rest
> proxy,
> >> the
> >> >> > > > renewer
> >> >> > > > > > can
> >> >> > > > > > > > > probably just be the creator of the token.
> >> >> > > > > > > > >
> >> >> > > > > > > > > 101. Do we need new acl on who can request delegation
> >> tokens?
> >> >> > > > > > > > >
> >> >> > > > > > > > > 102. Do we recommend people to send delegation tokens
> >> in an
> >> >> > > > > encrypted
> >> >> > > > > > > > > channel?
> >> >> > > > > > > > >
> >> >> > > > > > > > > 103. Who is responsible for expiring tokens, every
> >> broker?
> >> >> > > > > > > > >
> >> >> > > > > > > > > 104. For invalidating tokens, would it be better to
> do
> >> it in
> >> >> > a
> >> >> > > > > > request
> >> >> > > > > > > > > instead of going to ZK directly?
> >> >> > > > > > > > >
> >> >> > > > > > > > > 105. The terminology of client in the wiki sometimes
> >> refers
> >> >> > to
> >> >> > > > the
> >> >> > > > > > end
> >> >> > > > > > > > > client and some other times refers to the client
> using
> >> the
> >> >> > > > > delegation
> >> >> > > > > > > > > tokens. It would be useful to distinguish between the
> >> two.
> >> >> > > > > > > > >
> >> >> > > > > > > > > 106. Could you explain the sentence "Broker also
> >> stores the
> >> >> > > > > > > > DelegationToken
> >> >> > > > > > > > > without the hmac in the zookeeper." a bit more? I
> >> thought the
> >> >> > > > > > > delegation
> >> >> > > > > > > > > token is the hmac.
> >> >> > > > > > > > >
> >> >> > > > > > > > > Thanks,
> >> >> > > > > > > > >
> >> >> > > > > > > > > Jun
> >> >> > > > > > > > >
> >> >> > > > > > > > >
> >> >> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <
> >> jun@confluent.io>
> >> >> > > > wrote:
> >> >> > > > > > > > >
> >> >> > > > > > > > > > Hi, Harsha,
> >> >> > > > > > > > > >
> >> >> > > > > > > > > > Just sent out a KIP meeting invite. We can discuss
> >> this in
> >> >> > > the
> >> >> > > > > > > meeting
> >> >> > > > > > > > > > tomorrow.
> >> >> > > > > > > > > >
> >> >> > > > > > > > > > Thanks,
> >> >> > > > > > > > > >
> >> >> > > > > > > > > > Jun
> >> >> > > > > > > > > >
> >> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <
> >> kafka@harsha.io>
> >> >> > > > wrote:
> >> >> > > > > > > > > >
> >> >> > > > > > > > > >> Hi All,
> >> >> > > > > > > > > >>            Can we have a KIP meeting around this.
> >> The KIP
> >> >> > is
> >> >> > > > up
> >> >> > > > > > for
> >> >> > > > > > > > > >>            sometime and if there are any questions
> >> lets
> >> >> > > > quickly
> >> >> > > > > > hash
> >> >> > > > > > > > out
> >> >> > > > > > > > > >>            details.
> >> >> > > > > > > > > >>
> >> >> > > > > > > > > >> Thanks,
> >> >> > > > > > > > > >> Harsha
> >> >> > > > > > > > > >>
> >> >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40 AM, parth
> brahmbhatt
> >> wrote:
> >> >> > > > > > > > > >> > That is what the hadoop echo system uses so no
> >> good
> >> >> > reason
> >> >> > > > > > really.
> >> >> > > > > > > > We
> >> >> > > > > > > > > >> > could
> >> >> > > > > > > > > >> > change it to whatever is the newest recommended
> >> standard
> >> >> > > is.
> >> >> > > > > > > > > >> >
> >> >> > > > > > > > > >> > Thanks
> >> >> > > > > > > > > >> > Parth
> >> >> > > > > > > > > >> >
> >> >> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <
> >> >> > > > > ismael@juma.me.uk
> >> >> > > > > > >
> >> >> > > > > > > > > wrote:
> >> >> > > > > > > > > >> >
> >> >> > > > > > > > > >> > > Hi Parth,
> >> >> > > > > > > > > >> > >
> >> >> > > > > > > > > >> > > Thanks for the KIP. I only started reviewing
> >> this and
> >> >> > > may
> >> >> > > > > have
> >> >> > > > > > > > > >> additional
> >> >> > > > > > > > > >> > > questions later. The immediate question that
> >> came to
> >> >> > > mind
> >> >> > > > is
> >> >> > > > > > our
> >> >> > > > > > > > > >> choice of
> >> >> > > > > > > > > >> > > "DIGEST-MD5" even though it's marked as
> >> OBSOLETE in
> >> >> > the
> >> >> > > > IANA
> >> >> > > > > > > > > Registry
> >> >> > > > > > > > > >> of
> >> >> > > > > > > > > >> > > SASL mechanisms and the original RFC (2831)
> has
> >> been
> >> >> > > moved
> >> >> > > > > to
> >> >> > > > > > > > > Historic
> >> >> > > > > > > > > >> > > status:
> >> >> > > > > > > > > >> > >
> >> >> > > > > > > > > >> > > https://tools.ietf.org/html/rfc6331
> >> >> > > > > > > > > >> > >
> >> >> > > > > > > > >
> >> >> > > > > >
> >> >> > >
> >> http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> >> >> > > > > > > > > >> > >
> >> >> > > > > > > > > >> > > What is the reasoning behind that choice?
> >> >> > > > > > > > > >> > >
> >> >> > > > > > > > > >> > > Thanks,
> >> >> > > > > > > > > >> > > Ismael
> >> >> > > > > > > > > >> > >
> >> >> > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen
> Shapira <
> >> >> > > > > > > gwen@confluent.io
> >> >> > > > > > > > >
> >> >> > > > > > > > > >> wrote:
> >> >> > > > > > > > > >> > >
> >> >> > > > > > > > > >> > > > Also comments inline :)
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > > > > * I want to emphasize that even though
> >> delegation
> >> >> > > > tokens
> >> >> > > > > > > are a
> >> >> > > > > > > > > >> Hadoop
> >> >> > > > > > > > > >> > > > > innovation, I feel very strongly about not
> >> adding
> >> >> > > > > > dependency
> >> >> > > > > > > > on
> >> >> > > > > > > > > >> Hadoop
> >> >> > > > > > > > > >> > > > > when implementing delegation tokens for
> >> Kafka. The
> >> >> > > KIP
> >> >> > > > > > > doesn't
> >> >> > > > > > > > > >> imply
> >> >> > > > > > > > > >> > > > > such dependency, but if you can clarify...
> >> >> > > > > > > > > >> > > > >
> >> >> > > > > > > > > >> > > > >
> >> >> > > > > > > > > >> > > > > *No hadoop dependency.*
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > > > Yay! Just add this to the KIP so no one will
> >> read
> >> >> > the
> >> >> > > > KIP
> >> >> > > > > > and
> >> >> > > > > > > > > panic
> >> >> > > > > > > > > >> > > > three weeks before the next release...
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > > > > * Can we get delegation token at any time
> >> after
> >> >> > > > > > > > authenticating?
> >> >> > > > > > > > > >> only
> >> >> > > > > > > > > >> > > > > immediately after?
> >> >> > > > > > > > > >> > > > >
> >> >> > > > > > > > > >> > > > >
> >> >> > > > > > > > > >> > > > > *As long as you are authenticated you can
> >> get
> >> >> > > > delegation
> >> >> > > > > > > > tokens.
> >> >> > > > > > > > > >> We
> >> >> > > > > > > > > >> > > need
> >> >> > > > > > > > > >> > > > to
> >> >> > > > > > > > > >> > > > > discuss if a client authenticated using
> >> delegation
> >> >> > > > > token,
> >> >> > > > > > > can
> >> >> > > > > > > > > also
> >> >> > > > > > > > > >> > > > acquire
> >> >> > > > > > > > > >> > > > > delegation token again or not. Also there
> >> is the
> >> >> > > > > question
> >> >> > > > > > of
> >> >> > > > > > > > do
> >> >> > > > > > > > > we
> >> >> > > > > > > > > >> > > allow
> >> >> > > > > > > > > >> > > > > anyone to acquire delegation token or we
> >> want
> >> >> > > specific
> >> >> > > > > > ACLs
> >> >> > > > > > > (I
> >> >> > > > > > > > > >> think
> >> >> > > > > > > > > >> > > its
> >> >> > > > > > > > > >> > > > an
> >> >> > > > > > > > > >> > > > > overkill.)*
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > > > I agree that ACLs is an overkill.
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > > > I think we are debating two options: Either
> >> require
> >> >> > > > > Kerberos
> >> >> > > > > > > > auth
> >> >> > > > > > > > > >> for
> >> >> > > > > > > > > >> > > > renewal or require non-owners to renew.
> >> >> > > > > > > > > >> > > > I *think* the latter is simpler (it
> basically
> >> >> > require
> >> >> > > a
> >> >> > > > > "job
> >> >> > > > > > > > > master"
> >> >> > > > > > > > > >> > > > to take responsibility for the renewal, it
> >> will have
> >> >> > > its
> >> >> > > > > own
> >> >> > > > > > > > > >> identity
> >> >> > > > > > > > > >> > > > anyway and I think this is the correct
> design
> >> >> > pattern
> >> >> > > > > > anyway.
> >> >> > > > > > > > For
> >> >> > > > > > > > > >> > > > storm, I'd expect Nimbus to coordinate
> >> renewals?),
> >> >> > but
> >> >> > > > it
> >> >> > > > > is
> >> >> > > > > > > > hard
> >> >> > > > > > > > > to
> >> >> > > > > > > > > >> > > > debate simplicity without looking at the
> code
> >> >> > changes
> >> >> > > > > > > required.
> >> >> > > > > > > > If
> >> >> > > > > > > > > >> you
> >> >> > > > > > > > > >> > > > have a draft of how the "require Kerberos"
> >> will look
> >> >> > > in
> >> >> > > > > > Kafka
> >> >> > > > > > > > > code,
> >> >> > > > > > > > > >> > > > I'll be happy to take a look.
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > > > > * My understanding is that tokens will
> >> propagate
> >> >> > via
> >> >> > > > ZK
> >> >> > > > > > but
> >> >> > > > > > > > > >> without
> >> >> > > > > > > > > >> > > > > additional changes to UpdateMetadata
> >> protocol,
> >> >> > > > correct?
> >> >> > > > > > > > Clients
> >> >> > > > > > > > > >> > > > > currently don't retry on SASL auth failure
> >> (IIRC),
> >> >> > > but
> >> >> > > > > > since
> >> >> > > > > > > > the
> >> >> > > > > > > > > >> > > > > tokens propagate between brokers asynch,
> we
> >> will
> >> >> > > need
> >> >> > > > to
> >> >> > > > > > > > retry a
> >> >> > > > > > > > > >> bit
> >> >> > > > > > > > > >> > > > > to avoid clients failing auth due to
> timing
> >> >> > issues.
> >> >> > > > > > > > > >> > > > >
> >> >> > > > > > > > > >> > > > > *I am considering 2 alternatives right
> now.
> >> The
> >> >> > > > current
> >> >> > > > > > > > > documented
> >> >> > > > > > > > > >> > > > approach
> >> >> > > > > > > > > >> > > > > is zookeeper based and it does not require
> >> any
> >> >> > > changes
> >> >> > > > > to
> >> >> > > > > > > > > >> > > UpdateMetadata
> >> >> > > > > > > > > >> > > > > protocol. An alternative approach can
> remove
> >> >> > > zookeeper
> >> >> > > > > > > > > dependency
> >> >> > > > > > > > > >> as
> >> >> > > > > > > > > >> > > well
> >> >> > > > > > > > > >> > > > > but we can discuss that in KIP discussion
> >> call.*
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > > > Oooh! Sounds interesting. Do you want to
> ping
> >> Jun to
> >> >> > > > > > arrange a
> >> >> > > > > > > > > call?
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > > > > * I liked Ashish's suggestion of having
> >> just the
> >> >> > > > > > controller
> >> >> > > > > > > > > issue
> >> >> > > > > > > > > >> the
> >> >> > > > > > > > > >> > > > > delegation tokens, to avoid syncing a
> shared
> >> >> > secret.
> >> >> > > > Not
> >> >> > > > > > > sure
> >> >> > > > > > > > if
> >> >> > > > > > > > > >> we
> >> >> > > > > > > > > >> > > > > want to continue the discussion here or on
> >> the
> >> >> > > wiki. I
> >> >> > > > > > think
> >> >> > > > > > > > > that
> >> >> > > > > > > > > >> we
> >> >> > > > > > > > > >> > > > > can decouple the problem of "token
> >> distribution"
> >> >> > > from
> >> >> > > > > > > "shared
> >> >> > > > > > > > > >> secret
> >> >> > > > > > > > > >> > > > > distribution" and use the controller as
> the
> >> only
> >> >> > > token
> >> >> > > > > > > > generator
> >> >> > > > > > > > > >> to
> >> >> > > > > > > > > >> > > > > solve the second issue, while still using
> >> ZK async
> >> >> > > to
> >> >> > > > > > > > distribute
> >> >> > > > > > > > > >> > > > > tokens.
> >> >> > > > > > > > > >> > > > >
> >> >> > > > > > > > > >> > > > >
> >> >> > > > > > > > > >> > > > > *As mentioned in the previous Email I am
> >> fine with
> >> >> > > > that
> >> >> > > > > > > > approach
> >> >> > > > > > > > > >> as
> >> >> > > > > > > > > >> > > long
> >> >> > > > > > > > > >> > > > as
> >> >> > > > > > > > > >> > > > > we agree that the extra complexity of
> >> >> > > adding/updating
> >> >> > > > > APIS
> >> >> > > > > > > > adds
> >> >> > > > > > > > > >> enough
> >> >> > > > > > > > > >> > > > > value. The advantage with the controller
> >> approach
> >> >> > is
> >> >> > > > > > secret
> >> >> > > > > > > > > >> rotation
> >> >> > > > > > > > > >> > > can
> >> >> > > > > > > > > >> > > > be
> >> >> > > > > > > > > >> > > > > automated,frequent and would not require
> >> >> > > deployment. *
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > > > Can you detail the extra complexity (or
> point
> >> me to
> >> >> > > the
> >> >> > > > > > email
> >> >> > > > > > > I
> >> >> > > > > > > > > >> > > > missed?) - which Apis are required?
> >> >> > > > > > > > > >> > > > As far as I can tell, clients can already
> >> find the
> >> >> > > > > > controller
> >> >> > > > > > > > from
> >> >> > > > > > > > > >> > > > metadata. I'm a bit more concerned about
> >> controller
> >> >> > > > load.
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > > > >
> >> >> > > > > > > > > >> > > > > * While I like the idea of forcing
> kerberos
> >> auth
> >> >> > for
> >> >> > > > > > > renwal, I
> >> >> > > > > > > > > >> think
> >> >> > > > > > > > > >> > > > > it mixes the transport layer the the
> request
> >> >> > content
> >> >> > > > in
> >> >> > > > > a
> >> >> > > > > > > > pretty
> >> >> > > > > > > > > >> ugly
> >> >> > > > > > > > > >> > > > > way. Perhaps limiting renewer to non-owner
> >> is
> >> >> > > better.
> >> >> > > > > > > > > >> > > > >
> >> >> > > > > > > > > >> > > > > *I feel this is a necessary evil. While
> >> this will
> >> >> > > make
> >> >> > > > > the
> >> >> > > > > > > > kafka
> >> >> > > > > > > > > >> code
> >> >> > > > > > > > > >> > > > > pretty straight forward , forcing  renewer
> >> to
> >> >> > > > non-owner
> >> >> > > > > > > pushes
> >> >> > > > > > > > > >> the code
> >> >> > > > > > > > > >> > > > > ugliness to client and makes it even
> harder
> >> to
> >> >> > > > > > integrate.  *
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > > > As mentioned before, I don't think the
> >> "renewal by
> >> >> > > > other"
> >> >> > > > > > > > approach
> >> >> > > > > > > > > >> is
> >> >> > > > > > > > > >> > > > that ugly for the clients we expect to use
> >> >> > delegation
> >> >> > > > > tokens
> >> >> > > > > > > > since
> >> >> > > > > > > > > >> > > > they will have an app-master of some sort
> who
> >> >> > > requested
> >> >> > > > > the
> >> >> > > > > > > > token
> >> >> > > > > > > > > to
> >> >> > > > > > > > > >> > > > begin with.
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > > > >
> >> >> > > > > > > > > >> > > > > The response for my question on how
> multiple
> >> >> > > > identities
> >> >> > > > > > will
> >> >> > > > > > > > be
> >> >> > > > > > > > > >> > > > > handled wasn't super clear to me - AFAIK,
> >> we don't
> >> >> > > > > > > > authenticate
> >> >> > > > > > > > > >> each
> >> >> > > > > > > > > >> > > > > request, we authenticate connections.
> >> >> > > > > > > > > >> > > > >
> >> >> > > > > > > > > >> > > > > *We authenticate connections, and only
> when
> >> they
> >> >> > are
> >> >> > > > > being
> >> >> > > > > > > > > >> established.
> >> >> > > > > > > > > >> > > > Let
> >> >> > > > > > > > > >> > > > > me try to phrase this as a question, in
> >> absence of
> >> >> > > > > > > delegation
> >> >> > > > > > > > > >> tokens if
> >> >> > > > > > > > > >> > > > we
> >> >> > > > > > > > > >> > > > > had to support the use case using user
> >> TGT's how
> >> >> > > would
> >> >> > > > > we
> >> >> > > > > > do
> >> >> > > > > > > > it?
> >> >> > > > > > > > > >> My
> >> >> > > > > > > > > >> > > point
> >> >> > > > > > > > > >> > > > > was it would be no different with
> delegation
> >> >> > tokens.
> >> >> > > > The
> >> >> > > > > > use
> >> >> > > > > > > > > case
> >> >> > > > > > > > > >> you
> >> >> > > > > > > > > >> > > are
> >> >> > > > > > > > > >> > > > > describing seems more like impersonation.*
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > > > Yeah, I thought that one of the things that
> >> >> > delegation
> >> >> > > > > > tokens
> >> >> > > > > > > > > >> handled.
> >> >> > > > > > > > > >> > > > Maybe I got it wrong :)
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > > > Thanks for the detailed answers.
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > > > Gwen
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > > > > Thanks
> >> >> > > > > > > > > >> > > > > Parth
> >> >> > > > > > > > > >> > > > >
> >> >> > > > > > > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen
> >> Shapira <
> >> >> > > > > > > > > gwen@confluent.io
> >> >> > > > > > > > > >> >
> >> >> > > > > > > > > >> > > > wrote:
> >> >> > > > > > > > > >> > > > >
> >> >> > > > > > > > > >> > > > >> Hi Parth and Harsha,
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > > >> Few more comments:
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > > >> * The API / RequestResponse section
> >> doesn't seem
> >> >> > to
> >> >> > > > > have
> >> >> > > > > > > good
> >> >> > > > > > > > > >> > > > >> description of the changes to the Kafka
> >> Protocol.
> >> >> > > > > Sounds
> >> >> > > > > > > like
> >> >> > > > > > > > > >> you are
> >> >> > > > > > > > > >> > > > >> proposing new DelegationTokenRequest and
> >> >> > > > > > RenewTokenRequest
> >> >> > > > > > > > (and
> >> >> > > > > > > > > >> > > > >> matching responses), without detailing
> the
> >> >> > contents
> >> >> > > > of
> >> >> > > > > > the
> >> >> > > > > > > > > >> requests
> >> >> > > > > > > > > >> > > > >> and responses? Or rather, you show the
> >> class
> >> >> > > > interface,
> >> >> > > > > > but
> >> >> > > > > > > > not
> >> >> > > > > > > > > >> the
> >> >> > > > > > > > > >> > > > >> underlying protocol. This could be seen
> as
> >> an
> >> >> > > > > > > implementation
> >> >> > > > > > > > > >> detail,
> >> >> > > > > > > > > >> > > > >> but since the binary protocol is what we
> >> provide
> >> >> > to
> >> >> > > > > > > non-Java
> >> >> > > > > > > > > >> clients,
> >> >> > > > > > > > > >> > > > >> we need to show the changes there.
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > > >> * getDelegationToken sounds like
> >> >> > > > > > > > delegationTokenRequestHandler?
> >> >> > > > > > > > > >> Is it
> >> >> > > > > > > > > >> > > > >> planned to be part of KafkaApi? or
> Client?
> >> Its
> >> >> > > > > unclear...
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > > >> * I want to emphasize that even though
> >> delegation
> >> >> > > > > tokens
> >> >> > > > > > > are
> >> >> > > > > > > > a
> >> >> > > > > > > > > >> Hadoop
> >> >> > > > > > > > > >> > > > >> innovation, I feel very strongly about
> not
> >> adding
> >> >> > > > > > > dependency
> >> >> > > > > > > > on
> >> >> > > > > > > > > >> Hadoop
> >> >> > > > > > > > > >> > > > >> when implementing delegation tokens for
> >> Kafka.
> >> >> > The
> >> >> > > > KIP
> >> >> > > > > > > > doesn't
> >> >> > > > > > > > > >> imply
> >> >> > > > > > > > > >> > > > >> such dependency, but if you can
> clarify...
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > > >> * Can we get delegation token at any time
> >> after
> >> >> > > > > > > > authenticating?
> >> >> > > > > > > > > >> only
> >> >> > > > > > > > > >> > > > >> immediately after?
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > > >> * My understanding is that tokens will
> >> propagate
> >> >> > > via
> >> >> > > > ZK
> >> >> > > > > > but
> >> >> > > > > > > > > >> without
> >> >> > > > > > > > > >> > > > >> additional changes to UpdateMetadata
> >> protocol,
> >> >> > > > correct?
> >> >> > > > > > > > Clients
> >> >> > > > > > > > > >> > > > >> currently don't retry on SASL auth
> failure
> >> >> > (IIRC),
> >> >> > > > but
> >> >> > > > > > > since
> >> >> > > > > > > > > the
> >> >> > > > > > > > > >> > > > >> tokens propagate between brokers asynch,
> >> we will
> >> >> > > need
> >> >> > > > > to
> >> >> > > > > > > > retry
> >> >> > > > > > > > > a
> >> >> > > > > > > > > >> bit
> >> >> > > > > > > > > >> > > > >> to avoid clients failing auth due to
> timing
> >> >> > issues.
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > > >> * Strongly agreeing on clients not
> >> touching ZK
> >> >> > > > directly
> >> >> > > > > > :)
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > > >> * I liked Ashish's suggestion of having
> >> just the
> >> >> > > > > > controller
> >> >> > > > > > > > > >> issue the
> >> >> > > > > > > > > >> > > > >> delegation tokens, to avoid syncing a
> >> shared
> >> >> > > secret.
> >> >> > > > > Not
> >> >> > > > > > > sure
> >> >> > > > > > > > > if
> >> >> > > > > > > > > >> we
> >> >> > > > > > > > > >> > > > >> want to continue the discussion here or
> on
> >> the
> >> >> > > wiki.
> >> >> > > > I
> >> >> > > > > > > think
> >> >> > > > > > > > > >> that we
> >> >> > > > > > > > > >> > > > >> can decouple the problem of "token
> >> distribution"
> >> >> > > from
> >> >> > > > > > > "shared
> >> >> > > > > > > > > >> secret
> >> >> > > > > > > > > >> > > > >> distribution" and use the controller as
> >> the only
> >> >> > > > token
> >> >> > > > > > > > > generator
> >> >> > > > > > > > > >> to
> >> >> > > > > > > > > >> > > > >> solve the second issue, while still using
> >> ZK
> >> >> > async
> >> >> > > to
> >> >> > > > > > > > > distribute
> >> >> > > > > > > > > >> > > > >> tokens.
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > > >> * I am also uncomfortable with infinite
> >> lifetime
> >> >> > of
> >> >> > > > > > tokens
> >> >> > > > > > > > (and
> >> >> > > > > > > > > >> hoped
> >> >> > > > > > > > > >> > > > >> to hear from others in the community) -
> but
> >> >> > having
> >> >> > > > the
> >> >> > > > > > > option
> >> >> > > > > > > > > and
> >> >> > > > > > > > > >> > > > >> documenting it as less secure, allows
> >> users to
> >> >> > > > > configure
> >> >> > > > > > > > their
> >> >> > > > > > > > > >> system
> >> >> > > > > > > > > >> > > > >> as the wish.
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > > >> * While I like the idea of forcing
> >> kerberos auth
> >> >> > > for
> >> >> > > > > > > renwal,
> >> >> > > > > > > > I
> >> >> > > > > > > > > >> think
> >> >> > > > > > > > > >> > > > >> it mixes the transport layer the the
> >> request
> >> >> > > content
> >> >> > > > > in a
> >> >> > > > > > > > > pretty
> >> >> > > > > > > > > >> ugly
> >> >> > > > > > > > > >> > > > >> way. Perhaps limiting renewer to
> non-owner
> >> is
> >> >> > > better.
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > > >> Things I'd still like to see:
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > > >> * More detailed explanation on what we
> >> plan to do
> >> >> > > for
> >> >> > > > > the
> >> >> > > > > > > > java
> >> >> > > > > > > > > >> clients
> >> >> > > > > > > > > >> > > > >> specifically - new configuration? new
> APIs?
> >> >> > > > > > > > > >> > > > >> The response for my question on how
> >> multiple
> >> >> > > > identities
> >> >> > > > > > > will
> >> >> > > > > > > > be
> >> >> > > > > > > > > >> > > > >> handled wasn't super clear to me - AFAIK,
> >> we
> >> >> > don't
> >> >> > > > > > > > authenticate
> >> >> > > > > > > > > >> each
> >> >> > > > > > > > > >> > > > >> request, we authenticate connections.
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > > >> * Alternatives: Delegation tokens are
> only
> >> used
> >> >> > in
> >> >> > > > the
> >> >> > > > > > > Hadoop
> >> >> > > > > > > > > >> > > > >> ecosystem. I'm wondering if there are
> >> >> > alternatives
> >> >> > > in
> >> >> > > > > > other
> >> >> > > > > > > > > >> ecosystems
> >> >> > > > > > > > > >> > > > >> (Mesos? Tachyon? Cassandra?) and whether
> >> there
> >> >> > are
> >> >> > > > some
> >> >> > > > > > > > > >> advantages
> >> >> > > > > > > > > >> > > > >> there.
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > > >> Gwen
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <
> >> >> > > > > kafka@harsha.io
> >> >> > > > > > >
> >> >> > > > > > > > > wrote:
> >> >> > > > > > > > > >> > > > >> > Hi Gwen,
> >> >> > > > > > > > > >> > > > >> >            Can you look at Parth's last
> >> reply.
> >> >> > > Does
> >> >> > > > > it
> >> >> > > > > > > > answer
> >> >> > > > > > > > > >> your
> >> >> > > > > > > > > >> > > > >> >            concerns.
> >> >> > > > > > > > > >> > > > >> >
> >> >> > > > > > > > > >> > > > >> > Thanks,
> >> >> > > > > > > > > >> > > > >> > Harsha
> >> >> > > > > > > > > >> > > > >> >
> >> >> > > > > > > > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth
> >> >> > > brahmbhatt
> >> >> > > > > > wrote:
> >> >> > > > > > > > > >> > > > >> >> Thanks for reviewing Gwen. The wiki
> >> already
> >> >> > has
> >> >> > > > > > details
> >> >> > > > > > > on
> >> >> > > > > > > > > >> token
> >> >> > > > > > > > > >> > > > >> >> expiration
> >> >> > > > > > > > > >> > > > >> >> under token acquisition process
> >> >> > > > > > > > > >> > > > >> >> <
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > >
> >> >> > > > > > > > > >>
> >> >> > > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > >
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> >> >> > > > > > > > > >> > > > >> >.
> >> >> > > > > > > > > >> > > > >> >> Current proposal is that tokens will
> >> expire
> >> >> > > based
> >> >> > > > > on a
> >> >> > > > > > > > > server
> >> >> > > > > > > > > >> side
> >> >> > > > > > > > > >> > > > >> >> configuration (default 24 hours)
> unless
> >> >> > renewed.
> >> >> > > > > > Renewal
> >> >> > > > > > > > is
> >> >> > > > > > > > > >> only
> >> >> > > > > > > > > >> > > > allowed
> >> >> > > > > > > > > >> > > > >> >> until the max life time of token.
> >> >> > Alternatively
> >> >> > > we
> >> >> > > > > > could
> >> >> > > > > > > > > also
> >> >> > > > > > > > > >> make
> >> >> > > > > > > > > >> > > > that
> >> >> > > > > > > > > >> > > > >> >> an
> >> >> > > > > > > > > >> > > > >> >> optional param and the server side
> >> default can
> >> >> > > > serve
> >> >> > > > > > as
> >> >> > > > > > > > the
> >> >> > > > > > > > > >> upper
> >> >> > > > > > > > > >> > > > bound.
> >> >> > > > > > > > > >> > > > >> >>
> >> >> > > > > > > > > >> > > > >> >> To your second point it will be done
> >> exactly
> >> >> > the
> >> >> > > > > same
> >> >> > > > > > > way
> >> >> > > > > > > > we
> >> >> > > > > > > > > >> would
> >> >> > > > > > > > > >> > > > >> >> support
> >> >> > > > > > > > > >> > > > >> >> multiple keytabs. The calling client
> >> will have
> >> >> > > to
> >> >> > > > > put
> >> >> > > > > > > the
> >> >> > > > > > > > > >> tokens it
> >> >> > > > > > > > > >> > > > >> wants
> >> >> > > > > > > > > >> > > > >> >> to use in the subject instance and
> call
> >> >> > > > > > produce/consume
> >> >> > > > > > > > > inside
> >> >> > > > > > > > > >> > > > >> >> subject.doas. Each caller will have to
> >> keep
> >> >> > > track
> >> >> > > > of
> >> >> > > > > > its
> >> >> > > > > > > > own
> >> >> > > > > > > > > >> > > > subject. I
> >> >> > > > > > > > > >> > > > >> >> will have to look at the code to see
> if
> >> we
> >> >> > > support
> >> >> > > > > > this
> >> >> > > > > > > > > >> feature
> >> >> > > > > > > > > >> > > right
> >> >> > > > > > > > > >> > > > >> now
> >> >> > > > > > > > > >> > > > >> >> but my understanding is delegation
> token
> >> >> > > shouldn't
> >> >> > > > > > need
> >> >> > > > > > > > any
> >> >> > > > > > > > > >> special
> >> >> > > > > > > > > >> > > > >> >> treatment as its just another type of
> >> >> > Credential
> >> >> > > > in
> >> >> > > > > > the
> >> >> > > > > > > > > >> subject.
> >> >> > > > > > > > > >> > > > >> >>
> >> >> > > > > > > > > >> > > > >> >> I would also like to know what is your
> >> opinion
> >> >> > > > about
> >> >> > > > > > > > > infinite
> >> >> > > > > > > > > >> > > renewal
> >> >> > > > > > > > > >> > > > >> (my
> >> >> > > > > > > > > >> > > > >> >> recommendation is to not support
> this),
> >> tokens
> >> >> > > > > > renewing
> >> >> > > > > > > > them
> >> >> > > > > > > > > >> > > self(my
> >> >> > > > > > > > > >> > > > >> >> recommendation is to not support this)
> >> and
> >> >> > most
> >> >> > > > > > > > importantly
> >> >> > > > > > > > > >> your
> >> >> > > > > > > > > >> > > > choice
> >> >> > > > > > > > > >> > > > >> >> between the alternatives listed on
> this
> >> thread
> >> >> > > > > > > > > >> > > > >> >> <
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > >
> >> >> > > > > > > > > >>
> >> >> > > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > >
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >>
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> >> >> > > > > > > > > >> > > > >> >
> >> >> > > > > > > > > >> > > > >> >> ( I am leaning towards the
> >> alternative-2 minus
> >> >> > > > > > > controller
> >> >> > > > > > > > > >> > > > distributing
> >> >> > > > > > > > > >> > > > >> >> secret). Thanks again for reviewing.
> >> >> > > > > > > > > >> > > > >> >>
> >> >> > > > > > > > > >> > > > >> >> Thanks
> >> >> > > > > > > > > >> > > > >> >> Parth
> >> >> > > > > > > > > >> > > > >> >>
> >> >> > > > > > > > > >> > > > >> >>
> >> >> > > > > > > > > >> > > > >> >>
> >> >> > > > > > > > > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen
> >> Shapira <
> >> >> > > > > > > > > >> gwen@confluent.io>
> >> >> > > > > > > > > >> > > > wrote:
> >> >> > > > > > > > > >> > > > >> >>
> >> >> > > > > > > > > >> > > > >> >> > Harsha,
> >> >> > > > > > > > > >> > > > >> >> >
> >> >> > > > > > > > > >> > > > >> >> > I was thinking of the Rest Proxy. I
> >> didn't
> >> >> > see
> >> >> > > > > your
> >> >> > > > > > > > design
> >> >> > > > > > > > > >> yet,
> >> >> > > > > > > > > >> > > > but in
> >> >> > > > > > > > > >> > > > >> >> > our proxy, we have a set of
> >> producers, which
> >> >> > > > will
> >> >> > > > > > > serve
> >> >> > > > > > > > > >> multiple
> >> >> > > > > > > > > >> > > > users
> >> >> > > > > > > > > >> > > > >> >> > going through the proxy. Since these
> >> users
> >> >> > > will
> >> >> > > > > have
> >> >> > > > > > > > > >> different
> >> >> > > > > > > > > >> > > > >> >> > privileges, they'll need to
> >> authenticate
> >> >> > > > > separately,
> >> >> > > > > > > and
> >> >> > > > > > > > > >> can't
> >> >> > > > > > > > > >> > > > share a
> >> >> > > > > > > > > >> > > > >> >> > token.
> >> >> > > > > > > > > >> > > > >> >> >
> >> >> > > > > > > > > >> > > > >> >> > Am I missing anything?
> >> >> > > > > > > > > >> > > > >> >> >
> >> >> > > > > > > > > >> > > > >> >> > Gwen
> >> >> > > > > > > > > >> > > > >> >> >
> >> >> > > > > > > > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM,
> >> Harsha <
> >> >> > > > > > > kafka@harsha.io
> >> >> > > > > > > > >
> >> >> > > > > > > > > >> wrote:
> >> >> > > > > > > > > >> > > > >> >> > > Gwen,
> >> >> > > > > > > > > >> > > > >> >> > >            On your second point.
> >> Can you
> >> >> > > > > describe
> >> >> > > > > > a
> >> >> > > > > > > > > >> usecase
> >> >> > > > > > > > > >> > > where
> >> >> > > > > > > > > >> > > > >> >> > >            mutliple clients ended
> up
> >> >> > > sharing a
> >> >> > > > > > > > producer
> >> >> > > > > > > > > >> and
> >> >> > > > > > > > > >> > > even
> >> >> > > > > > > > > >> > > > if
> >> >> > > > > > > > > >> > > > >> they
> >> >> > > > > > > > > >> > > > >> >> > >            do why can't they not
> use
> >> >> > single
> >> >> > > > > token
> >> >> > > > > > > that
> >> >> > > > > > > > > >> producer
> >> >> > > > > > > > > >> > > > >> >> > >            captures. Why would we
> >> need
> >> >> > > > multiple
> >> >> > > > > > > > clients
> >> >> > > > > > > > > >> with
> >> >> > > > > > > > > >> > > > >> different
> >> >> > > > > > > > > >> > > > >> >> > >            tokens sharing a single
> >> >> > instance
> >> >> > > of
> >> >> > > > > > > > producer.
> >> >> > > > > > > > > >> Also
> >> >> > > > > > > > > >> > > in
> >> >> > > > > > > > > >> > > > >> this
> >> >> > > > > > > > > >> > > > >> >> > >            case other clients have
> >> access
> >> >> > > all
> >> >> > > > > the
> >> >> > > > > > > > tokens
> >> >> > > > > > > > > >> no?
> >> >> > > > > > > > > >> > > > >> >> > >
> >> >> > > > > > > > > >> > > > >> >> > > Thanks,
> >> >> > > > > > > > > >> > > > >> >> > > Harsha
> >> >> > > > > > > > > >> > > > >> >> > >
> >> >> > > > > > > > > >> > > > >> >> > >
> >> >> > > > > > > > > >> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM,
> >> Gwen
> >> >> > > Shapira
> >> >> > > > > > > wrote:
> >> >> > > > > > > > > >> > > > >> >> > >> Sorry for the delay:
> >> >> > > > > > > > > >> > > > >> >> > >>
> >> >> > > > > > > > > >> > > > >> >> > >> Two questions that we didn't see
> >> in the
> >> >> > > wiki:
> >> >> > > > > > > > > >> > > > >> >> > >> 1. Is there an expiration for
> >> delegation
> >> >> > > > > tokens?
> >> >> > > > > > > > > >> Renewal? How
> >> >> > > > > > > > > >> > > > do we
> >> >> > > > > > > > > >> > > > >> >> > >> revoke them?
> >> >> > > > > > > > > >> > > > >> >> > >> 2. If we want to use delegation
> >> tokens
> >> >> > for
> >> >> > > > > > "do-as"
> >> >> > > > > > > > > (say,
> >> >> > > > > > > > > >> > > submit
> >> >> > > > > > > > > >> > > > >> Storm
> >> >> > > > > > > > > >> > > > >> >> > >> job as my user), we will need a
> >> producer
> >> >> > > for
> >> >> > > > > > every
> >> >> > > > > > > > job
> >> >> > > > > > > > > >> (we
> >> >> > > > > > > > > >> > > can't
> >> >> > > > > > > > > >> > > > >> share
> >> >> > > > > > > > > >> > > > >> >> > >> them between multiple jobs
> running
> >> on
> >> >> > same
> >> >> > > > > node),
> >> >> > > > > > > > since
> >> >> > > > > > > > > >> we
> >> >> > > > > > > > > >> > > only
> >> >> > > > > > > > > >> > > > >> >> > >> authenticate when connecting. Is
> >> there a
> >> >> > > plan
> >> >> > > > > to
> >> >> > > > > > > > change
> >> >> > > > > > > > > >> this
> >> >> > > > > > > > > >> > > for
> >> >> > > > > > > > > >> > > > >> >> > >> delegation tokens, in order to
> >> allow
> >> >> > > multiple
> >> >> > > > > > users
> >> >> > > > > > > > > with
> >> >> > > > > > > > > >> > > > different
> >> >> > > > > > > > > >> > > > >> >> > >> tokens to share a client?
> >> >> > > > > > > > > >> > > > >> >> > >>
> >> >> > > > > > > > > >> > > > >> >> > >> Gwen
> >> >> > > > > > > > > >> > > > >> >> > >>
> >> >> > > > > > > > > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM,
> >> parth
> >> >> > > > > brahmbhatt
> >> >> > > > > > > > > >> > > > >> >> > >> <br...@gmail.com>
> >> wrote:
> >> >> > > > > > > > > >> > > > >> >> > >> > Bumping this up one more time,
> >> can
> >> >> > other
> >> >> > > > > > > committers
> >> >> > > > > > > > > >> review?
> >> >> > > > > > > > > >> > > > >> >> > >> >
> >> >> > > > > > > > > >> > > > >> >> > >> > Thanks
> >> >> > > > > > > > > >> > > > >> >> > >> > Parth
> >> >> > > > > > > > > >> > > > >> >> > >> >
> >> >> > > > > > > > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07
> AM,
> >> >> > Harsha <
> >> >> > > > > > > > > >> kafka@harsha.io>
> >> >> > > > > > > > > >> > > > wrote:
> >> >> > > > > > > > > >> > > > >> >> > >> >
> >> >> > > > > > > > > >> > > > >> >> > >> >> Parth,
> >> >> > > > > > > > > >> > > > >> >> > >> >>           Overall current
> >> design looks
> >> >> > > > good
> >> >> > > > > to
> >> >> > > > > > > > me. I
> >> >> > > > > > > > > >> am +1
> >> >> > > > > > > > > >> > > on
> >> >> > > > > > > > > >> > > > >> the
> >> >> > > > > > > > > >> > > > >> >> > KIP.
> >> >> > > > > > > > > >> > > > >> >> > >> >>
> >> >> > > > > > > > > >> > > > >> >> > >> >> Gwen , Jun can you review this
> >> as
> >> >> > well.
> >> >> > > > > > > > > >> > > > >> >> > >> >>
> >> >> > > > > > > > > >> > > > >> >> > >> >> -Harsha
> >> >> > > > > > > > > >> > > > >> >> > >> >>
> >> >> > > > > > > > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57
> >> AM,
> >> >> > parth
> >> >> > > > > > > > brahmbhatt
> >> >> > > > > > > > > >> wrote:
> >> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks for review Jitendra.
> >> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > I don't like the idea of
> >> infinite
> >> >> > > > lifetime
> >> >> > > > > > > but I
> >> >> > > > > > > > > >> see the
> >> >> > > > > > > > > >> > > > >> Streaming
> >> >> > > > > > > > > >> > > > >> >> > use
> >> >> > > > > > > > > >> > > > >> >> > >> >> > case. Even for Streaming use
> >> case I
> >> >> > > was
> >> >> > > > > > hoping
> >> >> > > > > > > > > >> there will
> >> >> > > > > > > > > >> > > > be
> >> >> > > > > > > > > >> > > > >> some
> >> >> > > > > > > > > >> > > > >> >> > notion
> >> >> > > > > > > > > >> > > > >> >> > >> >> > of
> >> >> > > > > > > > > >> > > > >> >> > >> >> > master/driver that can get
> new
> >> >> > > > delegation
> >> >> > > > > > > tokens
> >> >> > > > > > > > > at
> >> >> > > > > > > > > >> fixed
> >> >> > > > > > > > > >> > > > >> interval
> >> >> > > > > > > > > >> > > > >> >> > and
> >> >> > > > > > > > > >> > > > >> >> > >> >> > distribute to workers. If
> >> that is
> >> >> > not
> >> >> > > > the
> >> >> > > > > > case
> >> >> > > > > > > > for
> >> >> > > > > > > > > >> we can
> >> >> > > > > > > > > >> > > > >> discuss
> >> >> > > > > > > > > >> > > > >> >> > >> >> > delegation tokens renewing
> >> them self
> >> >> > > and
> >> >> > > > > the
> >> >> > > > > > > > > >> security
> >> >> > > > > > > > > >> > > > >> implications
> >> >> > > > > > > > > >> > > > >> >> > of the
> >> >> > > > > > > > > >> > > > >> >> > >> >> > same.
> >> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > I did not want clients to
> >> fetch
> >> >> > tokens
> >> >> > > > > from
> >> >> > > > > > > > > >> zookeeper,
> >> >> > > > > > > > > >> > > > >> overall I
> >> >> > > > > > > > > >> > > > >> >> > think
> >> >> > > > > > > > > >> > > > >> >> > >> >> > its
> >> >> > > > > > > > > >> > > > >> >> > >> >> > better if clients don't rely
> >> on our
> >> >> > > > > metadata
> >> >> > > > > > > > store
> >> >> > > > > > > > > >> and I
> >> >> > > > > > > > > >> > > > >> think we
> >> >> > > > > > > > > >> > > > >> >> > are
> >> >> > > > > > > > > >> > > > >> >> > >> >> > moving in that direction
> with
> >> all
> >> >> > the
> >> >> > > > > KIP-4
> >> >> > > > > > > > > >> improvements.
> >> >> > > > > > > > > >> > > > I
> >> >> > > > > > > > > >> > > > >> chose
> >> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as in this case
> the
> >> client
> >> >> > > > will
> >> >> > > > > > > still
> >> >> > > > > > > > > >> just talk
> >> >> > > > > > > > > >> > > > to
> >> >> > > > > > > > > >> > > > >> >> > broker , its
> >> >> > > > > > > > > >> > > > >> >> > >> >> > the brokers that will use
> >> zookeeper
> >> >> > > > which
> >> >> > > > > we
> >> >> > > > > > > > > >> already do
> >> >> > > > > > > > > >> > > > for a
> >> >> > > > > > > > > >> > > > >> lot
> >> >> > > > > > > > > >> > > > >> >> > of
> >> >> > > > > > > > > >> > > > >> >> > >> >> > other
> >> >> > > > > > > > > >> > > > >> >> > >> >> > usecases + ease of
> >> development + and
> >> >> > > the
> >> >> > > > > > > ability
> >> >> > > > > > > > > so
> >> >> > > > > > > > > >> > > tokens
> >> >> > > > > > > > > >> > > > >> will
> >> >> > > > > > > > > >> > > > >> >> > survive
> >> >> > > > > > > > > >> > > > >> >> > >> >> > even a rolling
> restart/cluster
> >> >> > > failure.
> >> >> > > > > if a
> >> >> > > > > > > > > >> majority
> >> >> > > > > > > > > >> > > > agrees
> >> >> > > > > > > > > >> > > > >> the
> >> >> > > > > > > > > >> > > > >> >> > added
> >> >> > > > > > > > > >> > > > >> >> > >> >> > complexity to have
> controller
> >> >> > > forwarding
> >> >> > > > > > keys
> >> >> > > > > > > to
> >> >> > > > > > > > > all
> >> >> > > > > > > > > >> > > > broker is
> >> >> > > > > > > > > >> > > > >> >> > justified
> >> >> > > > > > > > > >> > > > >> >> > >> >> > as
> >> >> > > > > > > > > >> > > > >> >> > >> >> > it provides tighter security
> >> , I am
> >> >> > > fine
> >> >> > > > > > with
> >> >> > > > > > > > that
> >> >> > > > > > > > > >> option
> >> >> > > > > > > > > >> > > > too.
> >> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > Given zookeeper does not
> >> support SSL
> >> >> > > we
> >> >> > > > > can
> >> >> > > > > > > not
> >> >> > > > > > > > > >> store
> >> >> > > > > > > > > >> > > > master
> >> >> > > > > > > > > >> > > > >> keys
> >> >> > > > > > > > > >> > > > >> >> > in
> >> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as master keys
> will
> >> be
> >> >> > > exposed
> >> >> > > > > on
> >> >> > > > > > > > wire.
> >> >> > > > > > > > > To
> >> >> > > > > > > > > >> > > > support
> >> >> > > > > > > > > >> > > > >> >> > rotation
> >> >> > > > > > > > > >> > > > >> >> > >> >> > without affecting current
> >> clients is
> >> >> > > > > > > something I
> >> >> > > > > > > > > >> need to
> >> >> > > > > > > > > >> > > > put
> >> >> > > > > > > > > >> > > > >> more
> >> >> > > > > > > > > >> > > > >> >> > thought
> >> >> > > > > > > > > >> > > > >> >> > >> >> > in. My current proposal
> >> assumes the
> >> >> > > > > rotation
> >> >> > > > > > > > will
> >> >> > > > > > > > > >> > > > invalidate
> >> >> > > > > > > > > >> > > > >> all
> >> >> > > > > > > > > >> > > > >> >> > current
> >> >> > > > > > > > > >> > > > >> >> > >> >> > tokens.
> >> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > I request committers to also
> >> review
> >> >> > > and
> >> >> > > > > post
> >> >> > > > > > > > their
> >> >> > > > > > > > > >> > > comments
> >> >> > > > > > > > > >> > > > >> so we
> >> >> > > > > > > > > >> > > > >> >> > can
> >> >> > > > > > > > > >> > > > >> >> > >> >> > make
> >> >> > > > > > > > > >> > > > >> >> > >> >> > progress on this KIP.
> >> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks
> >> >> > > > > > > > > >> > > > >> >> > >> >> > Parth
> >> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39
> >> AM,
> >> >> > > Ashish
> >> >> > > > > > Singh
> >> >> > > > > > > <
> >> >> > > > > > > > > >> > > > >> asingh@cloudera.com
> >> >> > > > > > > > > >> > > > >> >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > wrote:
> >> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at
> >> 11:26 AM,
> >> >> > > > > Harsha <
> >> >> > > > > > > > > >> > > > kafka@harsha.io>
> >> >> > > > > > > > > >> > > > >> >> > wrote:
> >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Unifying the two
> >> discussion
> >> >> > > threads
> >> >> > > > on
> >> >> > > > > > > this
> >> >> > > > > > > > > KIP.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Here is the response
> from
> >> >> > Jitendra
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > "The need for a large
> >> number of
> >> >> > > > > clients
> >> >> > > > > > > that
> >> >> > > > > > > > > are
> >> >> > > > > > > > > >> > > > running
> >> >> > > > > > > > > >> > > > >> all
> >> >> > > > > > > > > >> > > > >> >> > over the
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > cluster that
> authenticate
> >> with
> >> >> > > Kafka
> >> >> > > > > > > > brokers,
> >> >> > > > > > > > > >> is very
> >> >> > > > > > > > > >> > > > >> similar
> >> >> > > > > > > > > >> > > > >> >> > to the
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Hadoop use case of large
> >> number
> >> >> > of
> >> >> > > > > tasks
> >> >> > > > > > > > > running
> >> >> > > > > > > > > >> > > across
> >> >> > > > > > > > > >> > > > >> the
> >> >> > > > > > > > > >> > > > >> >> > cluster
> >> >> > > > > > > > > >> > > > >> >> > >> >> that
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need authentication to
> >> Hdfs
> >> >> > > > Namenode.
> >> >> > > > > > > > > >> Therefore, the
> >> >> > > > > > > > > >> > > > >> >> > delegation token
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > approach does seem like
> a
> >> good
> >> >> > fit
> >> >> > > > for
> >> >> > > > > > > this
> >> >> > > > > > > > > use
> >> >> > > > > > > > > >> case
> >> >> > > > > > > > > >> > > > as we
> >> >> > > > > > > > > >> > > > >> >> > have seen
> >> >> > > > > > > > > >> > > > >> >> > >> >> it
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > working at large scale
> in
> >> HDFS
> >> >> > and
> >> >> > > > > YARN.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > >   The proposed design is
> >> very
> >> >> > much
> >> >> > > > > > inline
> >> >> > > > > > > > with
> >> >> > > > > > > > > >> Hadoop
> >> >> > > > > > > > > >> > > > >> >> > approach. A few
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > >   comments:
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 1) Why do you guys want
> >> to allow
> >> >> > > > > > infinite
> >> >> > > > > > > > > >> renewable
> >> >> > > > > > > > > >> > > > >> lifetime
> >> >> > > > > > > > > >> > > > >> >> > for a
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > token? HDFS restricts a
> >> token
> >> >> > to a
> >> >> > > > max
> >> >> > > > > > > life
> >> >> > > > > > > > > time
> >> >> > > > > > > > > >> > > > (default
> >> >> > > > > > > > > >> > > > >> 7
> >> >> > > > > > > > > >> > > > >> >> > days).  A
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > token's vulnerability is
> >> >> > believed
> >> >> > > to
> >> >> > > > > > > > increase
> >> >> > > > > > > > > >> with
> >> >> > > > > > > > > >> > > > time.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > I agree that having
> infinite
> >> >> > > lifetime
> >> >> > > > > > might
> >> >> > > > > > > > not
> >> >> > > > > > > > > >> be the
> >> >> > > > > > > > > >> > > > best
> >> >> > > > > > > > > >> > > > >> idea.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 2) As I understand the
> >> tokens
> >> >> > are
> >> >> > > > > stored
> >> >> > > > > > > in
> >> >> > > > > > > > > >> zookeeper
> >> >> > > > > > > > > >> > > > as
> >> >> > > > > > > > > >> > > > >> well,
> >> >> > > > > > > > > >> > > > >> >> > and
> >> >> > > > > > > > > >> > > > >> >> > >> >> can
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > be updated there. This
> is
> >> clever
> >> >> > > as
> >> >> > > > it
> >> >> > > > > > can
> >> >> > > > > > > > > allow
> >> >> > > > > > > > > >> > > > >> replacing the
> >> >> > > > > > > > > >> > > > >> >> > tokens
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > once they run out of max
> >> life
> >> >> > > time,
> >> >> > > > > and
> >> >> > > > > > > > > clients
> >> >> > > > > > > > > >> can
> >> >> > > > > > > > > >> > > > >> download
> >> >> > > > > > > > > >> > > > >> >> > new
> >> >> > > > > > > > > >> > > > >> >> > >> >> tokens
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from zookeeper. It
> >> shouldn't be
> >> >> > a
> >> >> > > > big
> >> >> > > > > > load
> >> >> > > > > > > > on
> >> >> > > > > > > > > >> > > zookeeper
> >> >> > > > > > > > > >> > > > >> as a
> >> >> > > > > > > > > >> > > > >> >> > client
> >> >> > > > > > > > > >> > > > >> >> > >> >> will
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need to get a new token
> >> once in
> >> >> > > > > several
> >> >> > > > > > > > days.
> >> >> > > > > > > > > >> In this
> >> >> > > > > > > > > >> > > > >> approach
> >> >> > > > > > > > > >> > > > >> >> > you
> >> >> > > > > > > > > >> > > > >> >> > >> >> don't
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > need infinite lifetime
> on
> >> the
> >> >> > > token
> >> >> > > > > even
> >> >> > > > > > > for
> >> >> > > > > > > > > >> long
> >> >> > > > > > > > > >> > > > running
> >> >> > > > > > > > > >> > > > >> >> > clients.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > 3) The token password
> are
> >> >> > > generated
> >> >> > > > > > using
> >> >> > > > > > > a
> >> >> > > > > > > > > >> master
> >> >> > > > > > > > > >> > > key.
> >> >> > > > > > > > > >> > > > >> The
> >> >> > > > > > > > > >> > > > >> >> > master
> >> >> > > > > > > > > >> > > > >> >> > >> >> key
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > should also be
> >> periodically
> >> >> > > changed.
> >> >> > > > > In
> >> >> > > > > > > > > Hadoop,
> >> >> > > > > > > > > >> the
> >> >> > > > > > > > > >> > > > >> default
> >> >> > > > > > > > > >> > > > >> >> > renewal
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > period is 1 day.?
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > IIUC, this will require
> >> brokers
> >> >> > > > > > maintaining
> >> >> > > > > > > a
> >> >> > > > > > > > > >> list of X
> >> >> > > > > > > > > >> > > > most
> >> >> > > > > > > > > >> > > > >> >> > recent
> >> >> > > > > > > > > >> > > > >> >> > >> >> master
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > keys. This list will have
> >> to be
> >> >> > > > > persisted
> >> >> > > > > > > > > >> somewhere, as
> >> >> > > > > > > > > >> > > > if a
> >> >> > > > > > > > > >> > > > >> >> > broker
> >> >> > > > > > > > > >> > > > >> >> > >> >> goes
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > down it will have to get
> >> that list
> >> >> > > > again
> >> >> > > > > > and
> >> >> > > > > > > > > >> storing
> >> >> > > > > > > > > >> > > > master
> >> >> > > > > > > > > >> > > > >> keys
> >> >> > > > > > > > > >> > > > >> >> > on ZK
> >> >> > > > > > > > > >> > > > >> >> > >> >> is
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > not the best idea.
> However,
> >> if a
> >> >> > > > broker
> >> >> > > > > > goes
> >> >> > > > > > > > > down
> >> >> > > > > > > > > >> then
> >> >> > > > > > > > > >> > > we
> >> >> > > > > > > > > >> > > > >> have
> >> >> > > > > > > > > >> > > > >> >> > much
> >> >> > > > > > > > > >> > > > >> >> > >> >> bigger
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > issue to deal with and
> >> client can
> >> >> > > > always
> >> >> > > > > > > > > >> > > re-authenticate
> >> >> > > > > > > > > >> > > > is
> >> >> > > > > > > > > >> > > > >> such
> >> >> > > > > > > > > >> > > > >> >> > >> >> events.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > Did you happen to take a
> >> look at
> >> >> > > other
> >> >> > > > > > > > > >> alternatives
> >> >> > > > > > > > > >> > > this
> >> >> > > > > > > > > >> > > > >> list has
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > suggested?
> >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Thanks for a thorough
> >> proposal,
> >> >> > > > great
> >> >> > > > > > > work!"
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at
> >> 10:28
> >> >> > PM,
> >> >> > > > Gwen
> >> >> > > > > > > > Shapira
> >> >> > > > > > > > > >> wrote:
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > Makes sense to me.
> >> Thanks!
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at
> >> 9:25
> >> >> > PM,
> >> >> > > > > > Harsha <
> >> >> > > > > > > > > >> > > > kafka@harsha.io
> >> >> > > > > > > > > >> > > > >> >
> >> >> > > > > > > > > >> > > > >> >> > wrote:
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > It doesn't need any
> >> release
> >> >> > > > > vehicle
> >> >> > > > > > > but
> >> >> > > > > > > > > >> still the
> >> >> > > > > > > > > >> > > > >> work can
> >> >> > > > > > > > > >> > > > >> >> > move
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > forward.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > If anyone is
> >> interested in
> >> >> > the
> >> >> > > > KIP
> >> >> > > > > > > > please
> >> >> > > > > > > > > >> do the
> >> >> > > > > > > > > >> > > > >> review and
> >> >> > > > > > > > > >> > > > >> >> > >> >> provide
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > the
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > comments.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > -Harsha
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016,
> >> at
> >> >> > 04:59
> >> >> > > > PM,
> >> >> > > > > > > Ismael
> >> >> > > > > > > > > >> Juma
> >> >> > > > > > > > > >> > > > wrote:
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> I agree that it
> >> would be
> >> >> > good
> >> >> > > > to
> >> >> > > > > > have
> >> >> > > > > > > > > more
> >> >> > > > > > > > > >> time
> >> >> > > > > > > > > >> > > to
> >> >> > > > > > > > > >> > > > >> review
> >> >> > > > > > > > > >> > > > >> >> > and
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > discuss
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> KIP-48.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> Ismael
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016
> >> at
> >> >> > 12:55
> >> >> > > > AM,
> >> >> > > > > > Gwen
> >> >> > > > > > > > > >> Shapira <
> >> >> > > > > > > > > >> > > > >> >> > >> >> gwen@confluent.io>
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Hi Team,
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Since KIP-48
> >> depends on
> >> >> > > > KIP-43,
> >> >> > > > > > > which
> >> >> > > > > > > > > is
> >> >> > > > > > > > > >> > > > already a
> >> >> > > > > > > > > >> > > > >> bit
> >> >> > > > > > > > > >> > > > >> >> > of a
> >> >> > > > > > > > > >> > > > >> >> > >> >> risk
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > for
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > the next release
> -
> >> any
> >> >> > > chance
> >> >> > > > > we
> >> >> > > > > > > can
> >> >> > > > > > > > > >> delay
> >> >> > > > > > > > > >> > > > >> delegation
> >> >> > > > > > > > > >> > > > >> >> > tokens
> >> >> > > > > > > > > >> > > > >> >> > >> >> to
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > Kafka
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > 0.10.1?
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > With the
> community
> >> >> > working
> >> >> > > > on a
> >> >> > > > > > > > release
> >> >> > > > > > > > > >> every
> >> >> > > > > > > > > >> > > 3
> >> >> > > > > > > > > >> > > > >> month,
> >> >> > > > > > > > > >> > > > >> >> > this
> >> >> > > > > > > > > >> > > > >> >> > >> >> is not
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a huge
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > delay.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Gwen
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26,
> >> 2016 at
> >> >> > > 5:11
> >> >> > > > > PM,
> >> >> > > > > > > > Ashish
> >> >> > > > > > > > > >> Singh
> >> >> > > > > > > > > >> > > <
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > asingh@cloudera.com>
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Parth,
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Thanks again
> for
> >> the
> >> >> > > > awesome
> >> >> > > > > > > write
> >> >> > > > > > > > > up.
> >> >> > > > > > > > > >> > > > Following
> >> >> > > > > > > > > >> > > > >> our
> >> >> > > > > > > > > >> > > > >> >> > >> >> discussion
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from the
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I think
> it
> >> will
> >> >> > be
> >> >> > > > > easier
> >> >> > > > > > > to
> >> >> > > > > > > > > >> compare
> >> >> > > > > > > > > >> > > > >> various
> >> >> > > > > > > > > >> > > > >> >> > >> >> alternatives
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > if they
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > are
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > listed
> together.
> >> I am
> >> >> > > > stating
> >> >> > > > > > > > below a
> >> >> > > > > > > > > >> few
> >> >> > > > > > > > > >> > > > >> >> > alternatives along
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > with
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a the
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > current
> proposal.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Current
> >> proposal)
> >> >> > Store
> >> >> > > > > > > Delegation
> >> >> > > > > > > > > >> Token,
> >> >> > > > > > > > > >> > > DT,
> >> >> > > > > > > > > >> > > > >> on ZK.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> >> >> > > authenticates
> >> >> > > > > > with a
> >> >> > > > > > > > > >> broker.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a
> >> client is
> >> >> > > > > > > > authenticated,
> >> >> > > > > > > > > >> it
> >> >> > > > > > > > > >> > > will
> >> >> > > > > > > > > >> > > > >> make a
> >> >> > > > > > > > > >> > > > >> >> > broker
> >> >> > > > > > > > > >> > > > >> >> > >> >> side
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a
> >> delegation
> >> >> > > > token.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The
> broker
> >> >> > > generates
> >> >> > > > a
> >> >> > > > > > > shared
> >> >> > > > > > > > > >> secret
> >> >> > > > > > > > > >> > > > based
> >> >> > > > > > > > > >> > > > >> on
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > HMAC-SHA256(a
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> Password/Secret
> >> >> > shared
> >> >> > > > > > between
> >> >> > > > > > > > all
> >> >> > > > > > > > > >> > > brokers,
> >> >> > > > > > > > > >> > > > >> >> > randomly
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > generated
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > tokenId).
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Broker
> >> stores
> >> >> > this
> >> >> > > > > token
> >> >> > > > > > in
> >> >> > > > > > > > its
> >> >> > > > > > > > > >> in
> >> >> > > > > > > > > >> > > > memory
> >> >> > > > > > > > > >> > > > >> cache.
> >> >> > > > > > > > > >> > > > >> >> > >> >> Broker
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > also stores
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the
> >> DelegationToken
> >> >> > > > > without
> >> >> > > > > > > the
> >> >> > > > > > > > > >> hmac in
> >> >> > > > > > > > > >> > > the
> >> >> > > > > > > > > >> > > > >> >> > zookeeper.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. All
> >> brokers will
> >> >> > > > have a
> >> >> > > > > > > cache
> >> >> > > > > > > > > >> backed
> >> >> > > > > > > > > >> > > by
> >> >> > > > > > > > > >> > > > >> >> > zookeeper so
> >> >> > > > > > > > > >> > > > >> >> > >> >> they
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > will all
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    get notified
> >> >> > whenever
> >> >> > > a
> >> >> > > > > new
> >> >> > > > > > > > token
> >> >> > > > > > > > > is
> >> >> > > > > > > > > >> > > > >> generated and
> >> >> > > > > > > > > >> > > > >> >> > they
> >> >> > > > > > > > > >> > > > >> >> > >> >> will
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > update
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    local cache
> >> whenever
> >> >> > > > token
> >> >> > > > > > > state
> >> >> > > > > > > > > >> changes.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Broker
> >> returns
> >> >> > the
> >> >> > > > > token
> >> >> > > > > > to
> >> >> > > > > > > > > >> Client.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues
> >> and
> >> >> > fixes
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Probable
> >> race
> >> >> > > > > condition,
> >> >> > > > > > > > client
> >> >> > > > > > > > > >> tries
> >> >> > > > > > > > > >> > > to
> >> >> > > > > > > > > >> > > > >> >> > authenticate
> >> >> > > > > > > > > >> > > > >> >> > >> >> with
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > a broker
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    that is yet
> >> to be
> >> >> > > > updated
> >> >> > > > > > with
> >> >> > > > > > > > the
> >> >> > > > > > > > > >> newly
> >> >> > > > > > > > > >> > > > >> generated
> >> >> > > > > > > > > >> > > > >> >> > DT.
> >> >> > > > > > > > > >> > > > >> >> > >> >> This
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > can
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > probably be
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    dealt with
> >> making
> >> >> > > > > dtRequest
> >> >> > > > > > > > block
> >> >> > > > > > > > > >> until
> >> >> > > > > > > > > >> > > all
> >> >> > > > > > > > > >> > > > >> >> > brokers have
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > updated
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their DT
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    cache. Zk
> >> barrier or
> >> >> > > > > similar
> >> >> > > > > > > > > >> mechanism
> >> >> > > > > > > > > >> > > can
> >> >> > > > > > > > > >> > > > be
> >> >> > > > > > > > > >> > > > >> used.
> >> >> > > > > > > > > >> > > > >> >> > >> >> However,
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > all such
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    mechanisms
> >> will
> >> >> > > increase
> >> >> > > > > > > > > complexity.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Using a
> >> static
> >> >> > > secret
> >> >> > > > > key
> >> >> > > > > > > > from
> >> >> > > > > > > > > >> config
> >> >> > > > > > > > > >> > > > >> file. Will
> >> >> > > > > > > > > >> > > > >> >> > >> >> require
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > yet
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > another
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    config and
> >> uses a
> >> >> > > static
> >> >> > > > > > > secret
> >> >> > > > > > > > > >> key. It
> >> >> > > > > > > > > >> > > is
> >> >> > > > > > > > > >> > > > >> advised
> >> >> > > > > > > > > >> > > > >> >> > to
> >> >> > > > > > > > > >> > > > >> >> > >> >> rotate
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > secret
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > keys
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> periodically.
> >> This
> >> >> > can
> >> >> > > > be
> >> >> > > > > > > > avoided
> >> >> > > > > > > > > >> with
> >> >> > > > > > > > > >> > > > >> controller
> >> >> > > > > > > > > >> > > > >> >> > >> >> generating
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > secretKey and
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    passing to
> >> brokers
> >> >> > > > > > > periodically.
> >> >> > > > > > > > > >> However,
> >> >> > > > > > > > > >> > > > >> this will
> >> >> > > > > > > > > >> > > > >> >> > >> >> require
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > brokers to
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maintain
> >> certain
> >> >> > > counts
> >> >> > > > of
> >> >> > > > > > > > > >> secretKeys.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 1)
> >> Have
> >> >> > > > > controller
> >> >> > > > > > > > > >> generate
> >> >> > > > > > > > > >> > > > >> delegation
> >> >> > > > > > > > > >> > > > >> >> > token.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> >> >> > > authenticates
> >> >> > > > > > with a
> >> >> > > > > > > > > >> broker.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a
> >> client is
> >> >> > > > > > > > authenticated,
> >> >> > > > > > > > > >> it
> >> >> > > > > > > > > >> > > will
> >> >> > > > > > > > > >> > > > >> make a
> >> >> > > > > > > > > >> > > > >> >> > broker
> >> >> > > > > > > > > >> > > > >> >> > >> >> side
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a
> >> delegation
> >> >> > > > token.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. Broker
> >> forwards
> >> >> > the
> >> >> > > > > > request
> >> >> > > > > > > > to
> >> >> > > > > > > > > >> > > > controller.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4.
> Controller
> >> >> > > generates
> >> >> > > > a
> >> >> > > > > DT
> >> >> > > > > > > and
> >> >> > > > > > > > > >> > > broadcasts
> >> >> > > > > > > > > >> > > > >> to all
> >> >> > > > > > > > > >> > > > >> >> > >> >> brokers.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. Broker
> >> stores
> >> >> > this
> >> >> > > > > token
> >> >> > > > > > in
> >> >> > > > > > > > its
> >> >> > > > > > > > > >> memory
> >> >> > > > > > > > > >> > > > >> cache.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6.
> Controller
> >> >> > responds
> >> >> > > > to
> >> >> > > > > > > > broker’s
> >> >> > > > > > > > > >> DT
> >> >> > > > > > > > > >> > > req.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    7. Broker
> >> returns
> >> >> > the
> >> >> > > > > token
> >> >> > > > > > to
> >> >> > > > > > > > > >> Client.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues
> >> and
> >> >> > fixes
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. We will
> >> have to
> >> >> > add
> >> >> > > > new
> >> >> > > > > > > APIs
> >> >> > > > > > > > to
> >> >> > > > > > > > > >> > > support
> >> >> > > > > > > > > >> > > > >> >> > controller
> >> >> > > > > > > > > >> > > > >> >> > >> >> pushing
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > to
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    brokers on
> >> top of
> >> >> > the
> >> >> > > > > > minimal
> >> >> > > > > > > > APIs
> >> >> > > > > > > > > >> that
> >> >> > > > > > > > > >> > > are
> >> >> > > > > > > > > >> > > > >> >> > currently
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > proposed.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. We will
> >> also have
> >> >> > > to
> >> >> > > > > add
> >> >> > > > > > > APIs
> >> >> > > > > > > > > to
> >> >> > > > > > > > > >> > > support
> >> >> > > > > > > > > >> > > > >> the
> >> >> > > > > > > > > >> > > > >> >> > >> >> bootstrapping
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > case,
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > i.e,
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    when a new
> >> broker
> >> >> > > comes
> >> >> > > > up
> >> >> > > > > > it
> >> >> > > > > > > > will
> >> >> > > > > > > > > >> have
> >> >> > > > > > > > > >> > > to
> >> >> > > > > > > > > >> > > > >> get all
> >> >> > > > > > > > > >> > > > >> >> > >> >> delegation
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > from
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the
> >> controller.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. In
> >> catastrophic
> >> >> > > > > failures
> >> >> > > > > > > > where
> >> >> > > > > > > > > >> all
> >> >> > > > > > > > > >> > > > brokers
> >> >> > > > > > > > > >> > > > >> go
> >> >> > > > > > > > > >> > > > >> >> > down,
> >> >> > > > > > > > > >> > > > >> >> > >> >> the
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even
> >> if
> >> >> > > servers
> >> >> > > > > are
> >> >> > > > > > > > > >> restarted as
> >> >> > > > > > > > > >> > > > >> tokens
> >> >> > > > > > > > > >> > > > >> >> > are not
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this
> >> happens,
> >> >> > then
> >> >> > > > > there
> >> >> > > > > > > are
> >> >> > > > > > > > > more
> >> >> > > > > > > > > >> > > > important
> >> >> > > > > > > > > >> > > > >> >> > things to
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > worry
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > about
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is
> >> better
> >> >> > to
> >> >> > > > > > > > > >> re-authenticate.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 2)
> >> Do not
> >> >> > > > > > distribute
> >> >> > > > > > > > DT
> >> >> > > > > > > > > to
> >> >> > > > > > > > > >> > > other
> >> >> > > > > > > > > >> > > > >> brokers
> >> >> > > > > > > > > >> > > > >> >> > at
> >> >> > > > > > > > > >> > > > >> >> > >> >> all.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> >> >> > > authenticates
> >> >> > > > > > with a
> >> >> > > > > > > > > >> broker.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a
> >> client is
> >> >> > > > > > > > authenticated,
> >> >> > > > > > > > > >> it
> >> >> > > > > > > > > >> > > will
> >> >> > > > > > > > > >> > > > >> make a
> >> >> > > > > > > > > >> > > > >> >> > broker
> >> >> > > > > > > > > >> > > > >> >> > >> >> side
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a
> >> delegation
> >> >> > > > token.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The
> broker
> >> >> > > generates
> >> >> > > > DT
> >> >> > > > > > of
> >> >> > > > > > > > > form,
> >> >> > > > > > > > > >> > > [hmac +
> >> >> > > > > > > > > >> > > > >> (owner,
> >> >> > > > > > > > > >> > > > >> >> > >> >> renewer,
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maxLifeTime,
> >> id,
> >> >> > hmac,
> >> >> > > > > > > > > >> expirationTime)]
> >> >> > > > > > > > > >> > > and
> >> >> > > > > > > > > >> > > > >> passes
> >> >> > > > > > > > > >> > > > >> >> > back
> >> >> > > > > > > > > >> > > > >> >> > >> >> this
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > DT to
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > client.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    hmac is
> >> generated
> >> >> > via
> >> >> > > > > > > > > >> {HMAC-SHA256(owner,
> >> >> > > > > > > > > >> > > > >> renewer,
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > maxLifeTime, id,
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac,
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> expirationTime)
> >> >> > using
> >> >> > > > > > > > SecretKey}.
> >> >> > > > > > > > > >> Note
> >> >> > > > > > > > > >> > > that
> >> >> > > > > > > > > >> > > > >> all
> >> >> > > > > > > > > >> > > > >> >> > brokers
> >> >> > > > > > > > > >> > > > >> >> > >> >> have
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > this
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > SecretKey.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Client
> >> then goes
> >> >> > to
> >> >> > > > any
> >> >> > > > > > > > broker
> >> >> > > > > > > > > >> and to
> >> >> > > > > > > > > >> > > > >> >> > authenticate
> >> >> > > > > > > > > >> > > > >> >> > >> >> sends
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > the DT.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    Broker
> >> recalculates
> >> >> > > hmac
> >> >> > > > > > using
> >> >> > > > > > > > > >> (owner,
> >> >> > > > > > > > > >> > > > >> renewer,
> >> >> > > > > > > > > >> > > > >> >> > >> >> maxLifeTime,
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > id, hmac,
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> expirationTime) info
> >> >> > > > from
> >> >> > > > > DT
> >> >> > > > > > > and
> >> >> > > > > > > > > its
> >> >> > > > > > > > > >> > > > >> SecretKey. If
> >> >> > > > > > > > > >> > > > >> >> > it
> >> >> > > > > > > > > >> > > > >> >> > >> >> matches
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > with
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac of
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    DT, client
> is
> >> >> > > > > authenticated.
> >> >> > > > > > > > Yes,
> >> >> > > > > > > > > >> it will
> >> >> > > > > > > > > >> > > > do
> >> >> > > > > > > > > >> > > > >> other
> >> >> > > > > > > > > >> > > > >> >> > >> >> obvious
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > checks of
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    timestamp
> >> expiry and
> >> >> > > > such.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Note that
> secret
> >> key
> >> >> > will
> >> >> > > > be
> >> >> > > > > > > > > generated
> >> >> > > > > > > > > >> by
> >> >> > > > > > > > > >> > > > >> controller
> >> >> > > > > > > > > >> > > > >> >> > and
> >> >> > > > > > > > > >> > > > >> >> > >> >> passed
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > to
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > brokers
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > periodically.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues
> >> and
> >> >> > fixes
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. How to
> >> delete a
> >> >> > DT?
> >> >> > > > > Yes,
> >> >> > > > > > > that
> >> >> > > > > > > > > is
> >> >> > > > > > > > > >> a
> >> >> > > > > > > > > >> > > > downside
> >> >> > > > > > > > > >> > > > >> >> > here.
> >> >> > > > > > > > > >> > > > >> >> > >> >> However,
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > this can
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be handled
> >> with
> >> >> > > brokers
> >> >> > > > > > > > > maintaining
> >> >> > > > > > > > > >> a
> >> >> > > > > > > > > >> > > > >> blacklist of
> >> >> > > > > > > > > >> > > > >> >> > DTs,
> >> >> > > > > > > > > >> > > > >> >> > >> >> DTs
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > from this
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > list
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    can be
> >> removed after
> >> >> > > > > expiry.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. In
> >> catastrophic
> >> >> > > > > failures
> >> >> > > > > > > > where
> >> >> > > > > > > > > >> all
> >> >> > > > > > > > > >> > > > brokers
> >> >> > > > > > > > > >> > > > >> go
> >> >> > > > > > > > > >> > > > >> >> > down,
> >> >> > > > > > > > > >> > > > >> >> > >> >> the
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even
> >> if
> >> >> > > servers
> >> >> > > > > are
> >> >> > > > > > > > > >> restarted as
> >> >> > > > > > > > > >> > > > >> tokens
> >> >> > > > > > > > > >> > > > >> >> > are not
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this
> >> happens,
> >> >> > then
> >> >> > > > > there
> >> >> > > > > > > are
> >> >> > > > > > > > > more
> >> >> > > > > > > > > >> > > > important
> >> >> > > > > > > > > >> > > > >> >> > things to
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > worry
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > about
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is
> >> better
> >> >> > to
> >> >> > > > > > > > > >> re-authenticate.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26,
> >> 2016 at
> >> >> > > > 1:58
> >> >> > > > > > PM,
> >> >> > > > > > > > > Parth
> >> >> > > > > > > > > >> > > > >> Brahmbhatt <
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > pbrahmbhatt@hortonworks.com>
> >> >> > > > > > > > wrote:
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Hi,
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> I have filed
> >> KIP-48 so
> >> >> > > we
> >> >> > > > > can
> >> >> > > > > > > > offer
> >> >> > > > > > > > > >> hadoop
> >> >> > > > > > > > > >> > > > like
> >> >> > > > > > > > > >> > > > >> >> > delegation
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens in
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> kafka. You can
> >> review
> >> >> > > the
> >> >> > > > > > design
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >>
> >> >> > > > > > > > > >> > > > >> >> >
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > >
> >> >> > > > > > > > > >>
> >> >> > > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > >
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > .
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> This KIP
> >> depends on
> >> >> > > KIP-43
> >> >> > > > > and
> >> >> > > > > > > we
> >> >> > > > > > > > > >> have also
> >> >> > > > > > > > > >> > > > >> >> > discussed an
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > alternative to
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> proposed
> design
> >> here<
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >>
> >> >> > > > > > > > > >> > > > >> >> >
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > >
> >> >> > > > > > > > > >>
> >> >> > > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > >
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >>
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >.
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Thanks
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Parth
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > --
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Regards,
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Ashish
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > --
> >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > Regards,
> >> >> > > > > > > > > >> > > > >> >> > >> >> > > Ashish
> >> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> >> > > > > > > > > >> > > > >> >> > >> >>
> >> >> > > > > > > > > >> > > > >> >> >
> >> >> > > > > > > > > >> > > > >>
> >> >> > > > > > > > > >> > > >
> >> >> > > > > > > > > >> > >
> >> >> > > > > > > > > >>
> >> >> > > > > > > > > >>
> >> >> > > > > > > > > >
> >> >> > > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > >
> >> >> > > > > >
> >> >> > > > > >
> >> >> > > > > >
> >> >> > > > > > --
> >> >> > > > > > Regards,
> >> >> > > > > >
> >> >> > > > > > Rajini
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > > >
> >> >> > > >
> >> >> > > > --
> >> >> > > > Regards,
> >> >> > > >
> >> >> > > > Rajini
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Liquan Pei
> >> >> Software Engineer, Confluent Inc
> >>
> >
> >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Jun Rao <ju...@confluent.io>]

Harsha,

Another question.

9. How would the delegation token be configured in the client? The standard
way is to do this through JAAS. However, we will need to think through if
this is convenient in a shared environment. For example, when a new task is
added to a Storm worker node, do we need to dynamically add a new section
in the JAAS file? It may be more convenient if we can pass in the token
through the config directly w/o going through JAAS.

Are you or Parth still actively working on this KIP?

Thanks,

Jun



On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao <ju...@confluent.io> wrote:

> Just to add on that list.
>
> 2. It would be good to document the format of the data stored in ZK.
> 7. Earlier, there was a discussion on whether the tokens should be
> propagated through ZK like config/acl/quota, or through the controller.
> Currently, the controller is only designed for propagating topic metadata,
> but not other data.
> 8. Should we use SCRAM to send the token instead of DIGEST-MD5 since it's
> deprecated?
>
> Also, the images in the wiki seem broken.
>
> Thanks,
>
> Jun
>
> On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <gw...@confluent.io> wrote:
>
>> From what I can see, remaining questions are:
>>
>> 1. Who / how are tokens renewed? By original requester only? or using
>> Kerberos auth only?
>> 2. Are tokens stored on each broker or in ZK?
>> 3. How are tokens invalidated / expired?
>> 4. Which encryption algorithm is used?
>> 5. What is the impersonation proposal (it wasn't in the KIP but was
>> discussed in this thread)?
>> 6. Do we need new ACLs, if so - for what actions?
>>
>> Gwen
>>
>> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <ka...@harsha.io> wrote:
>> > Jun & Ismael,
>> >                          Unfortunately I couldn't attend the KIP meeting
>> >                          when delegation tokens discussed. Appreciate if
>> >                          you can update the thread if you have any
>> >                          further questions.
>> > Thanks,
>> > Harsha
>> >
>> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
>> >> It seems that the links to images in the KIP are broken.
>> >>
>> >> Liquan
>> >>
>> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
>> >> brahmbhatt.parth@gmail.com> wrote:
>> >>
>> >> > 110. What does getDelegationTokenAs mean?
>> >> > In the current proposal we only allow a user to get delegation token
>> for
>> >> > the identity that it authenticated as using another mechanism, i.e.
>> A user
>> >> > that authenticate using a keytab for principal user1@EXAMPLE.COM
>> will get
>> >> > delegation tokens for that user only. In future I think we will have
>> to
>> >> > extend support such that we allow some set of users (
>> >> > kafka-rest-user@EXAMPLE.COM, storm-nimbus@EXAMPLE.COM) to acquire
>> >> > delegation tokens on behalf of other users whose identity they have
>> >> > verified independently.  Kafka brokers will have ACLs to control
>> which
>> >> > users are allowed to impersonate other users and get tokens on
>> behalf of
>> >> > them. Overall Impersonation is a whole different problem in my
>> opinion and
>> >> > I think we can tackle it in separate KIP.
>> >> >
>> >> > 111. What's the typical rate of getting and renewing delegation
>> tokens?
>> >> > Typically this should be very very low, 1 request per minute is a
>> >> > relatively high estimate. However it depends on the token
>> expiration. I am
>> >> > less worried about the extra load it puts on controller vs the added
>> >> > complexity and the value it offers.
>> >> >
>> >> > Thanks
>> >> > Parth
>> >> >
>> >> >
>> >> >
>> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <is...@juma.me.uk>
>> wrote:
>> >> >
>> >> > > Thanks Rajini. It would probably require a separate KIP as it will
>> >> > > introduce user visible changes. We could also update KIP-48 to
>> have this
>> >> > > information, but it seems cleaner to do it separately. We can
>> discuss
>> >> > that
>> >> > > in the KIP call today.
>> >> > >
>> >> > > Ismael
>> >> > >
>> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <
>> >> > > rajinisivaram@googlemail.com> wrote:
>> >> > >
>> >> > > > Ismael,
>> >> > > >
>> >> > > > I have created a JIRA (
>> >> > https://issues.apache.org/jira/browse/KAFKA-3751)
>> >> > > > for adding SCRAM as a SASL mechanism. Would that need another
>> KIP? If
>> >> > > > KIP-48 will use this mechanism, can this just be a JIRA that gets
>> >> > > reviewed
>> >> > > > when the PR is ready?
>> >> > > >
>> >> > > > Thank you,
>> >> > > >
>> >> > > > Rajini
>> >> > > >
>> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <is...@juma.me.uk>
>> >> > wrote:
>> >> > > >
>> >> > > > > Thanks Rajini, SCRAM seems like a good candidate.
>> >> > > > >
>> >> > > > > Gwen had independently mentioned this as a SASL mechanism that
>> might
>> >> > be
>> >> > > > > useful for Kafka and I have been meaning to check it in more
>> detail.
>> >> > > Good
>> >> > > > > to know that you are willing to contribute an implementation.
>> Maybe
>> >> > we
>> >> > > > > should file a separate JIRA for this?
>> >> > > > >
>> >> > > > > Ismael
>> >> > > > >
>> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram <
>> >> > > > > rajinisivaram@googlemail.com> wrote:
>> >> > > > >
>> >> > > > > > SCRAM (Salted Challenge Response Authentication Mechanism)
>> is a
>> >> > > better
>> >> > > > > > mechanism than Digest-MD5. Java doesn't come with a built-in
>> SCRAM
>> >> > > > > > SaslServer or SaslClient, but I will be happy to add support
>> in
>> >> > Kafka
>> >> > > > > since
>> >> > > > > > it would be a useful mechanism to support anyway.
>> >> > > > > > https://tools.ietf.org/html/rfc7677 describes the protocol
>> for
>> >> > > > > > SCRAM-SHA-256.
>> >> > > > > >
>> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <ju...@confluent.io>
>> wrote:
>> >> > > > > >
>> >> > > > > > > Parth,
>> >> > > > > > >
>> >> > > > > > > Thanks for the explanation. A couple of more questions.
>> >> > > > > > >
>> >> > > > > > > 110. What does getDelegationTokenAs mean?
>> >> > > > > > >
>> >> > > > > > > 111. What's the typical rate of getting and renewing
>> delegation
>> >> > > > tokens?
>> >> > > > > > > That may have an impact on whether they should be directed
>> to the
>> >> > > > > > > controller.
>> >> > > > > > >
>> >> > > > > > > Jun
>> >> > > > > > >
>> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt <
>> >> > > > > > > brahmbhatt.parth@gmail.com> wrote:
>> >> > > > > > >
>> >> > > > > > > > Hi Jun,
>> >> > > > > > > >
>> >> > > > > > > > Thanks for reviewing.
>> >> > > > > > > >
>> >> > > > > > > > * We could add a Cluster action to add acls on who can
>> request
>> >> > > > > > delegation
>> >> > > > > > > > tokens. I don't see the use case for that yet but down
>> the line
>> >> > > > when
>> >> > > > > we
>> >> > > > > > > > start supporting getDelegationTokenAs it will be
>> necessary.
>> >> > > > > > > > * Yes we recommend tokens to be only used/distributed
>> over
>> >> > secure
>> >> > > > > > > channels.
>> >> > > > > > > > * Depending on what design we end up choosing
>> Invalidation will
>> >> > > be
>> >> > > > > > > > responsibility of every broker or controller.
>> >> > > > > > > > * I am not sure if I documented somewhere that
>> invalidation
>> >> > will
>> >> > > > > > directly
>> >> > > > > > > > go through zookeeper but that is not the intent.
>> Invalidation
>> >> > > will
>> >> > > > > > either
>> >> > > > > > > > be request based or due to expiration. No direct
>> zookeeper
>> >> > > > > interaction
>> >> > > > > > > from
>> >> > > > > > > > any client.
>> >> > > > > > > > * "Broker also stores the DelegationToken without the
>> hmac in
>> >> > the
>> >> > > > > > > > zookeeper." : Sorry about the confusion. The sole
>> purpose of
>> >> > > > > zookeeper
>> >> > > > > > in
>> >> > > > > > > > this design is as distribution channel for tokens
>> between all
>> >> > > > brokers
>> >> > > > > > > and a
>> >> > > > > > > > layer that ensures only tokens that were generated by
>> making a
>> >> > > > > request
>> >> > > > > > > to a
>> >> > > > > > > > broker will be accepted (more on this in second
>> paragraph). The
>> >> > > > token
>> >> > > > > > > > consists of few elements (owner, renewer, uuid ,
>> expiration,
>> >> > > hmac)
>> >> > > > ,
>> >> > > > > > one
>> >> > > > > > > of
>> >> > > > > > > > which is the finally generated hmac but hmac it self is
>> >> > derivable
>> >> > > > if
>> >> > > > > > you
>> >> > > > > > > > have all the other elements of the token + secret key to
>> >> > generate
>> >> > > > > hmac.
>> >> > > > > > > > Given zookeeper does not provide SSL support we do not
>> want the
>> >> > > > > entire
>> >> > > > > > > > token to be wire transferred to zookeeper as that will
>> be an
>> >> > > > insecure
>> >> > > > > > > wire
>> >> > > > > > > > transfer. Instead we only store all the other elements
>> of a
>> >> > > > > delegation
>> >> > > > > > > > tokens. Brokers can read these elements and because they
>> also
>> >> > > have
>> >> > > > > > access
>> >> > > > > > > > to secret key they will be able to generate hmac on
>> their end.
>> >> > > > > > > >
>> >> > > > > > > > One of the alternative proposed is to avoid zookeeper
>> >> > > altogether. A
>> >> > > > > > > Client
>> >> > > > > > > > will call broker with required information (owner,
>> renwer,
>> >> > > > > expiration)
>> >> > > > > > > and
>> >> > > > > > > > get back (signed hmac, uuid). Broker won't store this in
>> >> > > zookeeper.
>> >> > > > > > From
>> >> > > > > > > > this point a client can contact any broker with all the
>> >> > > delegation
>> >> > > > > > token
>> >> > > > > > > > info (owner, rewner, expiration, hmac, uuid) the borker
>> will
>> >> > > > > regenerate
>> >> > > > > > > the
>> >> > > > > > > > hmac and as long as it matches with hmac presented by
>> client ,
>> >> > > > broker
>> >> > > > > > > will
>> >> > > > > > > > allow the request to authenticate.  Only problem with
>> this
>> >> > > approach
>> >> > > > > is
>> >> > > > > > if
>> >> > > > > > > > the secret key is compromised any client can now generate
>> >> > random
>> >> > > > > tokens
>> >> > > > > > > and
>> >> > > > > > > > they will still be able to authenticate as any user they
>> like.
>> >> > > with
>> >> > > > > > > > zookeeper we guarantee that only tokens acquired via a
>> broker
>> >> > > > (using
>> >> > > > > > some
>> >> > > > > > > > auth scheme other than delegation token) will be
>> accepted. We
>> >> > > need
>> >> > > > to
>> >> > > > > > > > discuss which proposal makes more sense and we can go
>> over it
>> >> > in
>> >> > > > > > > tomorrow's
>> >> > > > > > > > meeting.
>> >> > > > > > > >
>> >> > > > > > > > Also, can you forward the invite to me?
>> >> > > > > > > >
>> >> > > > > > > > Thanks
>> >> > > > > > > > Parth
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <
>> jun@confluent.io>
>> >> > > > wrote:
>> >> > > > > > > >
>> >> > > > > > > > > Thanks for the KIP. A few comments.
>> >> > > > > > > > >
>> >> > > > > > > > > 100. This potentially can be useful for Kafka Connect
>> and
>> >> > Kafka
>> >> > > > > rest
>> >> > > > > > > > proxy
>> >> > > > > > > > > where a worker agent will need to run a task on behalf
>> of a
>> >> > > > client.
>> >> > > > > > We
>> >> > > > > > > > will
>> >> > > > > > > > > likely need to change how those services use Kafka
>> clients
>> >> > > > > > > > > (producer/consumer). Instead of a shared client per
>> worker,
>> >> > we
>> >> > > > will
>> >> > > > > > > need
>> >> > > > > > > > a
>> >> > > > > > > > > client per user task since the authentication happens
>> at the
>> >> > > > > > connection
>> >> > > > > > > > > level. For Kafka Connect, the renewer will be the
>> workers.
>> >> > So,
>> >> > > we
>> >> > > > > > > > probably
>> >> > > > > > > > > need to allow multiple renewers. For Kafka rest proxy,
>> the
>> >> > > > renewer
>> >> > > > > > can
>> >> > > > > > > > > probably just be the creator of the token.
>> >> > > > > > > > >
>> >> > > > > > > > > 101. Do we need new acl on who can request delegation
>> tokens?
>> >> > > > > > > > >
>> >> > > > > > > > > 102. Do we recommend people to send delegation tokens
>> in an
>> >> > > > > encrypted
>> >> > > > > > > > > channel?
>> >> > > > > > > > >
>> >> > > > > > > > > 103. Who is responsible for expiring tokens, every
>> broker?
>> >> > > > > > > > >
>> >> > > > > > > > > 104. For invalidating tokens, would it be better to do
>> it in
>> >> > a
>> >> > > > > > request
>> >> > > > > > > > > instead of going to ZK directly?
>> >> > > > > > > > >
>> >> > > > > > > > > 105. The terminology of client in the wiki sometimes
>> refers
>> >> > to
>> >> > > > the
>> >> > > > > > end
>> >> > > > > > > > > client and some other times refers to the client using
>> the
>> >> > > > > delegation
>> >> > > > > > > > > tokens. It would be useful to distinguish between the
>> two.
>> >> > > > > > > > >
>> >> > > > > > > > > 106. Could you explain the sentence "Broker also
>> stores the
>> >> > > > > > > > DelegationToken
>> >> > > > > > > > > without the hmac in the zookeeper." a bit more? I
>> thought the
>> >> > > > > > > delegation
>> >> > > > > > > > > token is the hmac.
>> >> > > > > > > > >
>> >> > > > > > > > > Thanks,
>> >> > > > > > > > >
>> >> > > > > > > > > Jun
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <
>> jun@confluent.io>
>> >> > > > wrote:
>> >> > > > > > > > >
>> >> > > > > > > > > > Hi, Harsha,
>> >> > > > > > > > > >
>> >> > > > > > > > > > Just sent out a KIP meeting invite. We can discuss
>> this in
>> >> > > the
>> >> > > > > > > meeting
>> >> > > > > > > > > > tomorrow.
>> >> > > > > > > > > >
>> >> > > > > > > > > > Thanks,
>> >> > > > > > > > > >
>> >> > > > > > > > > > Jun
>> >> > > > > > > > > >
>> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <
>> kafka@harsha.io>
>> >> > > > wrote:
>> >> > > > > > > > > >
>> >> > > > > > > > > >> Hi All,
>> >> > > > > > > > > >>            Can we have a KIP meeting around this.
>> The KIP
>> >> > is
>> >> > > > up
>> >> > > > > > for
>> >> > > > > > > > > >>            sometime and if there are any questions
>> lets
>> >> > > > quickly
>> >> > > > > > hash
>> >> > > > > > > > out
>> >> > > > > > > > > >>            details.
>> >> > > > > > > > > >>
>> >> > > > > > > > > >> Thanks,
>> >> > > > > > > > > >> Harsha
>> >> > > > > > > > > >>
>> >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt
>> wrote:
>> >> > > > > > > > > >> > That is what the hadoop echo system uses so no
>> good
>> >> > reason
>> >> > > > > > really.
>> >> > > > > > > > We
>> >> > > > > > > > > >> > could
>> >> > > > > > > > > >> > change it to whatever is the newest recommended
>> standard
>> >> > > is.
>> >> > > > > > > > > >> >
>> >> > > > > > > > > >> > Thanks
>> >> > > > > > > > > >> > Parth
>> >> > > > > > > > > >> >
>> >> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <
>> >> > > > > ismael@juma.me.uk
>> >> > > > > > >
>> >> > > > > > > > > wrote:
>> >> > > > > > > > > >> >
>> >> > > > > > > > > >> > > Hi Parth,
>> >> > > > > > > > > >> > >
>> >> > > > > > > > > >> > > Thanks for the KIP. I only started reviewing
>> this and
>> >> > > may
>> >> > > > > have
>> >> > > > > > > > > >> additional
>> >> > > > > > > > > >> > > questions later. The immediate question that
>> came to
>> >> > > mind
>> >> > > > is
>> >> > > > > > our
>> >> > > > > > > > > >> choice of
>> >> > > > > > > > > >> > > "DIGEST-MD5" even though it's marked as
>> OBSOLETE in
>> >> > the
>> >> > > > IANA
>> >> > > > > > > > > Registry
>> >> > > > > > > > > >> of
>> >> > > > > > > > > >> > > SASL mechanisms and the original RFC (2831) has
>> been
>> >> > > moved
>> >> > > > > to
>> >> > > > > > > > > Historic
>> >> > > > > > > > > >> > > status:
>> >> > > > > > > > > >> > >
>> >> > > > > > > > > >> > > https://tools.ietf.org/html/rfc6331
>> >> > > > > > > > > >> > >
>> >> > > > > > > > >
>> >> > > > > >
>> >> > >
>> http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
>> >> > > > > > > > > >> > >
>> >> > > > > > > > > >> > > What is the reasoning behind that choice?
>> >> > > > > > > > > >> > >
>> >> > > > > > > > > >> > > Thanks,
>> >> > > > > > > > > >> > > Ismael
>> >> > > > > > > > > >> > >
>> >> > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <
>> >> > > > > > > gwen@confluent.io
>> >> > > > > > > > >
>> >> > > > > > > > > >> wrote:
>> >> > > > > > > > > >> > >
>> >> > > > > > > > > >> > > > Also comments inline :)
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > > > > * I want to emphasize that even though
>> delegation
>> >> > > > tokens
>> >> > > > > > > are a
>> >> > > > > > > > > >> Hadoop
>> >> > > > > > > > > >> > > > > innovation, I feel very strongly about not
>> adding
>> >> > > > > > dependency
>> >> > > > > > > > on
>> >> > > > > > > > > >> Hadoop
>> >> > > > > > > > > >> > > > > when implementing delegation tokens for
>> Kafka. The
>> >> > > KIP
>> >> > > > > > > doesn't
>> >> > > > > > > > > >> imply
>> >> > > > > > > > > >> > > > > such dependency, but if you can clarify...
>> >> > > > > > > > > >> > > > >
>> >> > > > > > > > > >> > > > >
>> >> > > > > > > > > >> > > > > *No hadoop dependency.*
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > > > Yay! Just add this to the KIP so no one will
>> read
>> >> > the
>> >> > > > KIP
>> >> > > > > > and
>> >> > > > > > > > > panic
>> >> > > > > > > > > >> > > > three weeks before the next release...
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > > > > * Can we get delegation token at any time
>> after
>> >> > > > > > > > authenticating?
>> >> > > > > > > > > >> only
>> >> > > > > > > > > >> > > > > immediately after?
>> >> > > > > > > > > >> > > > >
>> >> > > > > > > > > >> > > > >
>> >> > > > > > > > > >> > > > > *As long as you are authenticated you can
>> get
>> >> > > > delegation
>> >> > > > > > > > tokens.
>> >> > > > > > > > > >> We
>> >> > > > > > > > > >> > > need
>> >> > > > > > > > > >> > > > to
>> >> > > > > > > > > >> > > > > discuss if a client authenticated using
>> delegation
>> >> > > > > token,
>> >> > > > > > > can
>> >> > > > > > > > > also
>> >> > > > > > > > > >> > > > acquire
>> >> > > > > > > > > >> > > > > delegation token again or not. Also there
>> is the
>> >> > > > > question
>> >> > > > > > of
>> >> > > > > > > > do
>> >> > > > > > > > > we
>> >> > > > > > > > > >> > > allow
>> >> > > > > > > > > >> > > > > anyone to acquire delegation token or we
>> want
>> >> > > specific
>> >> > > > > > ACLs
>> >> > > > > > > (I
>> >> > > > > > > > > >> think
>> >> > > > > > > > > >> > > its
>> >> > > > > > > > > >> > > > an
>> >> > > > > > > > > >> > > > > overkill.)*
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > > > I agree that ACLs is an overkill.
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > > > I think we are debating two options: Either
>> require
>> >> > > > > Kerberos
>> >> > > > > > > > auth
>> >> > > > > > > > > >> for
>> >> > > > > > > > > >> > > > renewal or require non-owners to renew.
>> >> > > > > > > > > >> > > > I *think* the latter is simpler (it basically
>> >> > require
>> >> > > a
>> >> > > > > "job
>> >> > > > > > > > > master"
>> >> > > > > > > > > >> > > > to take responsibility for the renewal, it
>> will have
>> >> > > its
>> >> > > > > own
>> >> > > > > > > > > >> identity
>> >> > > > > > > > > >> > > > anyway and I think this is the correct design
>> >> > pattern
>> >> > > > > > anyway.
>> >> > > > > > > > For
>> >> > > > > > > > > >> > > > storm, I'd expect Nimbus to coordinate
>> renewals?),
>> >> > but
>> >> > > > it
>> >> > > > > is
>> >> > > > > > > > hard
>> >> > > > > > > > > to
>> >> > > > > > > > > >> > > > debate simplicity without looking at the code
>> >> > changes
>> >> > > > > > > required.
>> >> > > > > > > > If
>> >> > > > > > > > > >> you
>> >> > > > > > > > > >> > > > have a draft of how the "require Kerberos"
>> will look
>> >> > > in
>> >> > > > > > Kafka
>> >> > > > > > > > > code,
>> >> > > > > > > > > >> > > > I'll be happy to take a look.
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > > > > * My understanding is that tokens will
>> propagate
>> >> > via
>> >> > > > ZK
>> >> > > > > > but
>> >> > > > > > > > > >> without
>> >> > > > > > > > > >> > > > > additional changes to UpdateMetadata
>> protocol,
>> >> > > > correct?
>> >> > > > > > > > Clients
>> >> > > > > > > > > >> > > > > currently don't retry on SASL auth failure
>> (IIRC),
>> >> > > but
>> >> > > > > > since
>> >> > > > > > > > the
>> >> > > > > > > > > >> > > > > tokens propagate between brokers asynch, we
>> will
>> >> > > need
>> >> > > > to
>> >> > > > > > > > retry a
>> >> > > > > > > > > >> bit
>> >> > > > > > > > > >> > > > > to avoid clients failing auth due to timing
>> >> > issues.
>> >> > > > > > > > > >> > > > >
>> >> > > > > > > > > >> > > > > *I am considering 2 alternatives right now.
>> The
>> >> > > > current
>> >> > > > > > > > > documented
>> >> > > > > > > > > >> > > > approach
>> >> > > > > > > > > >> > > > > is zookeeper based and it does not require
>> any
>> >> > > changes
>> >> > > > > to
>> >> > > > > > > > > >> > > UpdateMetadata
>> >> > > > > > > > > >> > > > > protocol. An alternative approach can remove
>> >> > > zookeeper
>> >> > > > > > > > > dependency
>> >> > > > > > > > > >> as
>> >> > > > > > > > > >> > > well
>> >> > > > > > > > > >> > > > > but we can discuss that in KIP discussion
>> call.*
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > > > Oooh! Sounds interesting. Do you want to ping
>> Jun to
>> >> > > > > > arrange a
>> >> > > > > > > > > call?
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > > > > * I liked Ashish's suggestion of having
>> just the
>> >> > > > > > controller
>> >> > > > > > > > > issue
>> >> > > > > > > > > >> the
>> >> > > > > > > > > >> > > > > delegation tokens, to avoid syncing a shared
>> >> > secret.
>> >> > > > Not
>> >> > > > > > > sure
>> >> > > > > > > > if
>> >> > > > > > > > > >> we
>> >> > > > > > > > > >> > > > > want to continue the discussion here or on
>> the
>> >> > > wiki. I
>> >> > > > > > think
>> >> > > > > > > > > that
>> >> > > > > > > > > >> we
>> >> > > > > > > > > >> > > > > can decouple the problem of "token
>> distribution"
>> >> > > from
>> >> > > > > > > "shared
>> >> > > > > > > > > >> secret
>> >> > > > > > > > > >> > > > > distribution" and use the controller as the
>> only
>> >> > > token
>> >> > > > > > > > generator
>> >> > > > > > > > > >> to
>> >> > > > > > > > > >> > > > > solve the second issue, while still using
>> ZK async
>> >> > > to
>> >> > > > > > > > distribute
>> >> > > > > > > > > >> > > > > tokens.
>> >> > > > > > > > > >> > > > >
>> >> > > > > > > > > >> > > > >
>> >> > > > > > > > > >> > > > > *As mentioned in the previous Email I am
>> fine with
>> >> > > > that
>> >> > > > > > > > approach
>> >> > > > > > > > > >> as
>> >> > > > > > > > > >> > > long
>> >> > > > > > > > > >> > > > as
>> >> > > > > > > > > >> > > > > we agree that the extra complexity of
>> >> > > adding/updating
>> >> > > > > APIS
>> >> > > > > > > > adds
>> >> > > > > > > > > >> enough
>> >> > > > > > > > > >> > > > > value. The advantage with the controller
>> approach
>> >> > is
>> >> > > > > > secret
>> >> > > > > > > > > >> rotation
>> >> > > > > > > > > >> > > can
>> >> > > > > > > > > >> > > > be
>> >> > > > > > > > > >> > > > > automated,frequent and would not require
>> >> > > deployment. *
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > > > Can you detail the extra complexity (or point
>> me to
>> >> > > the
>> >> > > > > > email
>> >> > > > > > > I
>> >> > > > > > > > > >> > > > missed?) - which Apis are required?
>> >> > > > > > > > > >> > > > As far as I can tell, clients can already
>> find the
>> >> > > > > > controller
>> >> > > > > > > > from
>> >> > > > > > > > > >> > > > metadata. I'm a bit more concerned about
>> controller
>> >> > > > load.
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > > > >
>> >> > > > > > > > > >> > > > > * While I like the idea of forcing kerberos
>> auth
>> >> > for
>> >> > > > > > > renwal, I
>> >> > > > > > > > > >> think
>> >> > > > > > > > > >> > > > > it mixes the transport layer the the request
>> >> > content
>> >> > > > in
>> >> > > > > a
>> >> > > > > > > > pretty
>> >> > > > > > > > > >> ugly
>> >> > > > > > > > > >> > > > > way. Perhaps limiting renewer to non-owner
>> is
>> >> > > better.
>> >> > > > > > > > > >> > > > >
>> >> > > > > > > > > >> > > > > *I feel this is a necessary evil. While
>> this will
>> >> > > make
>> >> > > > > the
>> >> > > > > > > > kafka
>> >> > > > > > > > > >> code
>> >> > > > > > > > > >> > > > > pretty straight forward , forcing  renewer
>> to
>> >> > > > non-owner
>> >> > > > > > > pushes
>> >> > > > > > > > > >> the code
>> >> > > > > > > > > >> > > > > ugliness to client and makes it even harder
>> to
>> >> > > > > > integrate.  *
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > > > As mentioned before, I don't think the
>> "renewal by
>> >> > > > other"
>> >> > > > > > > > approach
>> >> > > > > > > > > >> is
>> >> > > > > > > > > >> > > > that ugly for the clients we expect to use
>> >> > delegation
>> >> > > > > tokens
>> >> > > > > > > > since
>> >> > > > > > > > > >> > > > they will have an app-master of some sort who
>> >> > > requested
>> >> > > > > the
>> >> > > > > > > > token
>> >> > > > > > > > > to
>> >> > > > > > > > > >> > > > begin with.
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > > > >
>> >> > > > > > > > > >> > > > > The response for my question on how multiple
>> >> > > > identities
>> >> > > > > > will
>> >> > > > > > > > be
>> >> > > > > > > > > >> > > > > handled wasn't super clear to me - AFAIK,
>> we don't
>> >> > > > > > > > authenticate
>> >> > > > > > > > > >> each
>> >> > > > > > > > > >> > > > > request, we authenticate connections.
>> >> > > > > > > > > >> > > > >
>> >> > > > > > > > > >> > > > > *We authenticate connections, and only when
>> they
>> >> > are
>> >> > > > > being
>> >> > > > > > > > > >> established.
>> >> > > > > > > > > >> > > > Let
>> >> > > > > > > > > >> > > > > me try to phrase this as a question, in
>> absence of
>> >> > > > > > > delegation
>> >> > > > > > > > > >> tokens if
>> >> > > > > > > > > >> > > > we
>> >> > > > > > > > > >> > > > > had to support the use case using user
>> TGT's how
>> >> > > would
>> >> > > > > we
>> >> > > > > > do
>> >> > > > > > > > it?
>> >> > > > > > > > > >> My
>> >> > > > > > > > > >> > > point
>> >> > > > > > > > > >> > > > > was it would be no different with delegation
>> >> > tokens.
>> >> > > > The
>> >> > > > > > use
>> >> > > > > > > > > case
>> >> > > > > > > > > >> you
>> >> > > > > > > > > >> > > are
>> >> > > > > > > > > >> > > > > describing seems more like impersonation.*
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > > > Yeah, I thought that one of the things that
>> >> > delegation
>> >> > > > > > tokens
>> >> > > > > > > > > >> handled.
>> >> > > > > > > > > >> > > > Maybe I got it wrong :)
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > > > Thanks for the detailed answers.
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > > > Gwen
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > > > > Thanks
>> >> > > > > > > > > >> > > > > Parth
>> >> > > > > > > > > >> > > > >
>> >> > > > > > > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen
>> Shapira <
>> >> > > > > > > > > gwen@confluent.io
>> >> > > > > > > > > >> >
>> >> > > > > > > > > >> > > > wrote:
>> >> > > > > > > > > >> > > > >
>> >> > > > > > > > > >> > > > >> Hi Parth and Harsha,
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > > >> Few more comments:
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > > >> * The API / RequestResponse section
>> doesn't seem
>> >> > to
>> >> > > > > have
>> >> > > > > > > good
>> >> > > > > > > > > >> > > > >> description of the changes to the Kafka
>> Protocol.
>> >> > > > > Sounds
>> >> > > > > > > like
>> >> > > > > > > > > >> you are
>> >> > > > > > > > > >> > > > >> proposing new DelegationTokenRequest and
>> >> > > > > > RenewTokenRequest
>> >> > > > > > > > (and
>> >> > > > > > > > > >> > > > >> matching responses), without detailing the
>> >> > contents
>> >> > > > of
>> >> > > > > > the
>> >> > > > > > > > > >> requests
>> >> > > > > > > > > >> > > > >> and responses? Or rather, you show the
>> class
>> >> > > > interface,
>> >> > > > > > but
>> >> > > > > > > > not
>> >> > > > > > > > > >> the
>> >> > > > > > > > > >> > > > >> underlying protocol. This could be seen as
>> an
>> >> > > > > > > implementation
>> >> > > > > > > > > >> detail,
>> >> > > > > > > > > >> > > > >> but since the binary protocol is what we
>> provide
>> >> > to
>> >> > > > > > > non-Java
>> >> > > > > > > > > >> clients,
>> >> > > > > > > > > >> > > > >> we need to show the changes there.
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > > >> * getDelegationToken sounds like
>> >> > > > > > > > delegationTokenRequestHandler?
>> >> > > > > > > > > >> Is it
>> >> > > > > > > > > >> > > > >> planned to be part of KafkaApi? or Client?
>> Its
>> >> > > > > unclear...
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > > >> * I want to emphasize that even though
>> delegation
>> >> > > > > tokens
>> >> > > > > > > are
>> >> > > > > > > > a
>> >> > > > > > > > > >> Hadoop
>> >> > > > > > > > > >> > > > >> innovation, I feel very strongly about not
>> adding
>> >> > > > > > > dependency
>> >> > > > > > > > on
>> >> > > > > > > > > >> Hadoop
>> >> > > > > > > > > >> > > > >> when implementing delegation tokens for
>> Kafka.
>> >> > The
>> >> > > > KIP
>> >> > > > > > > > doesn't
>> >> > > > > > > > > >> imply
>> >> > > > > > > > > >> > > > >> such dependency, but if you can clarify...
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > > >> * Can we get delegation token at any time
>> after
>> >> > > > > > > > authenticating?
>> >> > > > > > > > > >> only
>> >> > > > > > > > > >> > > > >> immediately after?
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > > >> * My understanding is that tokens will
>> propagate
>> >> > > via
>> >> > > > ZK
>> >> > > > > > but
>> >> > > > > > > > > >> without
>> >> > > > > > > > > >> > > > >> additional changes to UpdateMetadata
>> protocol,
>> >> > > > correct?
>> >> > > > > > > > Clients
>> >> > > > > > > > > >> > > > >> currently don't retry on SASL auth failure
>> >> > (IIRC),
>> >> > > > but
>> >> > > > > > > since
>> >> > > > > > > > > the
>> >> > > > > > > > > >> > > > >> tokens propagate between brokers asynch,
>> we will
>> >> > > need
>> >> > > > > to
>> >> > > > > > > > retry
>> >> > > > > > > > > a
>> >> > > > > > > > > >> bit
>> >> > > > > > > > > >> > > > >> to avoid clients failing auth due to timing
>> >> > issues.
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > > >> * Strongly agreeing on clients not
>> touching ZK
>> >> > > > directly
>> >> > > > > > :)
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > > >> * I liked Ashish's suggestion of having
>> just the
>> >> > > > > > controller
>> >> > > > > > > > > >> issue the
>> >> > > > > > > > > >> > > > >> delegation tokens, to avoid syncing a
>> shared
>> >> > > secret.
>> >> > > > > Not
>> >> > > > > > > sure
>> >> > > > > > > > > if
>> >> > > > > > > > > >> we
>> >> > > > > > > > > >> > > > >> want to continue the discussion here or on
>> the
>> >> > > wiki.
>> >> > > > I
>> >> > > > > > > think
>> >> > > > > > > > > >> that we
>> >> > > > > > > > > >> > > > >> can decouple the problem of "token
>> distribution"
>> >> > > from
>> >> > > > > > > "shared
>> >> > > > > > > > > >> secret
>> >> > > > > > > > > >> > > > >> distribution" and use the controller as
>> the only
>> >> > > > token
>> >> > > > > > > > > generator
>> >> > > > > > > > > >> to
>> >> > > > > > > > > >> > > > >> solve the second issue, while still using
>> ZK
>> >> > async
>> >> > > to
>> >> > > > > > > > > distribute
>> >> > > > > > > > > >> > > > >> tokens.
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > > >> * I am also uncomfortable with infinite
>> lifetime
>> >> > of
>> >> > > > > > tokens
>> >> > > > > > > > (and
>> >> > > > > > > > > >> hoped
>> >> > > > > > > > > >> > > > >> to hear from others in the community) - but
>> >> > having
>> >> > > > the
>> >> > > > > > > option
>> >> > > > > > > > > and
>> >> > > > > > > > > >> > > > >> documenting it as less secure, allows
>> users to
>> >> > > > > configure
>> >> > > > > > > > their
>> >> > > > > > > > > >> system
>> >> > > > > > > > > >> > > > >> as the wish.
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > > >> * While I like the idea of forcing
>> kerberos auth
>> >> > > for
>> >> > > > > > > renwal,
>> >> > > > > > > > I
>> >> > > > > > > > > >> think
>> >> > > > > > > > > >> > > > >> it mixes the transport layer the the
>> request
>> >> > > content
>> >> > > > > in a
>> >> > > > > > > > > pretty
>> >> > > > > > > > > >> ugly
>> >> > > > > > > > > >> > > > >> way. Perhaps limiting renewer to non-owner
>> is
>> >> > > better.
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > > >> Things I'd still like to see:
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > > >> * More detailed explanation on what we
>> plan to do
>> >> > > for
>> >> > > > > the
>> >> > > > > > > > java
>> >> > > > > > > > > >> clients
>> >> > > > > > > > > >> > > > >> specifically - new configuration? new APIs?
>> >> > > > > > > > > >> > > > >> The response for my question on how
>> multiple
>> >> > > > identities
>> >> > > > > > > will
>> >> > > > > > > > be
>> >> > > > > > > > > >> > > > >> handled wasn't super clear to me - AFAIK,
>> we
>> >> > don't
>> >> > > > > > > > authenticate
>> >> > > > > > > > > >> each
>> >> > > > > > > > > >> > > > >> request, we authenticate connections.
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > > >> * Alternatives: Delegation tokens are only
>> used
>> >> > in
>> >> > > > the
>> >> > > > > > > Hadoop
>> >> > > > > > > > > >> > > > >> ecosystem. I'm wondering if there are
>> >> > alternatives
>> >> > > in
>> >> > > > > > other
>> >> > > > > > > > > >> ecosystems
>> >> > > > > > > > > >> > > > >> (Mesos? Tachyon? Cassandra?) and whether
>> there
>> >> > are
>> >> > > > some
>> >> > > > > > > > > >> advantages
>> >> > > > > > > > > >> > > > >> there.
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > > >> Gwen
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <
>> >> > > > > kafka@harsha.io
>> >> > > > > > >
>> >> > > > > > > > > wrote:
>> >> > > > > > > > > >> > > > >> > Hi Gwen,
>> >> > > > > > > > > >> > > > >> >            Can you look at Parth's last
>> reply.
>> >> > > Does
>> >> > > > > it
>> >> > > > > > > > answer
>> >> > > > > > > > > >> your
>> >> > > > > > > > > >> > > > >> >            concerns.
>> >> > > > > > > > > >> > > > >> >
>> >> > > > > > > > > >> > > > >> > Thanks,
>> >> > > > > > > > > >> > > > >> > Harsha
>> >> > > > > > > > > >> > > > >> >
>> >> > > > > > > > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth
>> >> > > brahmbhatt
>> >> > > > > > wrote:
>> >> > > > > > > > > >> > > > >> >> Thanks for reviewing Gwen. The wiki
>> already
>> >> > has
>> >> > > > > > details
>> >> > > > > > > on
>> >> > > > > > > > > >> token
>> >> > > > > > > > > >> > > > >> >> expiration
>> >> > > > > > > > > >> > > > >> >> under token acquisition process
>> >> > > > > > > > > >> > > > >> >> <
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > >
>> >> > > > > > > > > >>
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
>> >> > > > > > > > > >> > > > >> >.
>> >> > > > > > > > > >> > > > >> >> Current proposal is that tokens will
>> expire
>> >> > > based
>> >> > > > > on a
>> >> > > > > > > > > server
>> >> > > > > > > > > >> side
>> >> > > > > > > > > >> > > > >> >> configuration (default 24 hours) unless
>> >> > renewed.
>> >> > > > > > Renewal
>> >> > > > > > > > is
>> >> > > > > > > > > >> only
>> >> > > > > > > > > >> > > > allowed
>> >> > > > > > > > > >> > > > >> >> until the max life time of token.
>> >> > Alternatively
>> >> > > we
>> >> > > > > > could
>> >> > > > > > > > > also
>> >> > > > > > > > > >> make
>> >> > > > > > > > > >> > > > that
>> >> > > > > > > > > >> > > > >> >> an
>> >> > > > > > > > > >> > > > >> >> optional param and the server side
>> default can
>> >> > > > serve
>> >> > > > > > as
>> >> > > > > > > > the
>> >> > > > > > > > > >> upper
>> >> > > > > > > > > >> > > > bound.
>> >> > > > > > > > > >> > > > >> >>
>> >> > > > > > > > > >> > > > >> >> To your second point it will be done
>> exactly
>> >> > the
>> >> > > > > same
>> >> > > > > > > way
>> >> > > > > > > > we
>> >> > > > > > > > > >> would
>> >> > > > > > > > > >> > > > >> >> support
>> >> > > > > > > > > >> > > > >> >> multiple keytabs. The calling client
>> will have
>> >> > > to
>> >> > > > > put
>> >> > > > > > > the
>> >> > > > > > > > > >> tokens it
>> >> > > > > > > > > >> > > > >> wants
>> >> > > > > > > > > >> > > > >> >> to use in the subject instance and call
>> >> > > > > > produce/consume
>> >> > > > > > > > > inside
>> >> > > > > > > > > >> > > > >> >> subject.doas. Each caller will have to
>> keep
>> >> > > track
>> >> > > > of
>> >> > > > > > its
>> >> > > > > > > > own
>> >> > > > > > > > > >> > > > subject. I
>> >> > > > > > > > > >> > > > >> >> will have to look at the code to see if
>> we
>> >> > > support
>> >> > > > > > this
>> >> > > > > > > > > >> feature
>> >> > > > > > > > > >> > > right
>> >> > > > > > > > > >> > > > >> now
>> >> > > > > > > > > >> > > > >> >> but my understanding is delegation token
>> >> > > shouldn't
>> >> > > > > > need
>> >> > > > > > > > any
>> >> > > > > > > > > >> special
>> >> > > > > > > > > >> > > > >> >> treatment as its just another type of
>> >> > Credential
>> >> > > > in
>> >> > > > > > the
>> >> > > > > > > > > >> subject.
>> >> > > > > > > > > >> > > > >> >>
>> >> > > > > > > > > >> > > > >> >> I would also like to know what is your
>> opinion
>> >> > > > about
>> >> > > > > > > > > infinite
>> >> > > > > > > > > >> > > renewal
>> >> > > > > > > > > >> > > > >> (my
>> >> > > > > > > > > >> > > > >> >> recommendation is to not support this),
>> tokens
>> >> > > > > > renewing
>> >> > > > > > > > them
>> >> > > > > > > > > >> > > self(my
>> >> > > > > > > > > >> > > > >> >> recommendation is to not support this)
>> and
>> >> > most
>> >> > > > > > > > importantly
>> >> > > > > > > > > >> your
>> >> > > > > > > > > >> > > > choice
>> >> > > > > > > > > >> > > > >> >> between the alternatives listed on this
>> thread
>> >> > > > > > > > > >> > > > >> >> <
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > >
>> >> > > > > > > > > >>
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
>> >> > > > > > > > > >> > > > >> >
>> >> > > > > > > > > >> > > > >> >> ( I am leaning towards the
>> alternative-2 minus
>> >> > > > > > > controller
>> >> > > > > > > > > >> > > > distributing
>> >> > > > > > > > > >> > > > >> >> secret). Thanks again for reviewing.
>> >> > > > > > > > > >> > > > >> >>
>> >> > > > > > > > > >> > > > >> >> Thanks
>> >> > > > > > > > > >> > > > >> >> Parth
>> >> > > > > > > > > >> > > > >> >>
>> >> > > > > > > > > >> > > > >> >>
>> >> > > > > > > > > >> > > > >> >>
>> >> > > > > > > > > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen
>> Shapira <
>> >> > > > > > > > > >> gwen@confluent.io>
>> >> > > > > > > > > >> > > > wrote:
>> >> > > > > > > > > >> > > > >> >>
>> >> > > > > > > > > >> > > > >> >> > Harsha,
>> >> > > > > > > > > >> > > > >> >> >
>> >> > > > > > > > > >> > > > >> >> > I was thinking of the Rest Proxy. I
>> didn't
>> >> > see
>> >> > > > > your
>> >> > > > > > > > design
>> >> > > > > > > > > >> yet,
>> >> > > > > > > > > >> > > > but in
>> >> > > > > > > > > >> > > > >> >> > our proxy, we have a set of
>> producers, which
>> >> > > > will
>> >> > > > > > > serve
>> >> > > > > > > > > >> multiple
>> >> > > > > > > > > >> > > > users
>> >> > > > > > > > > >> > > > >> >> > going through the proxy. Since these
>> users
>> >> > > will
>> >> > > > > have
>> >> > > > > > > > > >> different
>> >> > > > > > > > > >> > > > >> >> > privileges, they'll need to
>> authenticate
>> >> > > > > separately,
>> >> > > > > > > and
>> >> > > > > > > > > >> can't
>> >> > > > > > > > > >> > > > share a
>> >> > > > > > > > > >> > > > >> >> > token.
>> >> > > > > > > > > >> > > > >> >> >
>> >> > > > > > > > > >> > > > >> >> > Am I missing anything?
>> >> > > > > > > > > >> > > > >> >> >
>> >> > > > > > > > > >> > > > >> >> > Gwen
>> >> > > > > > > > > >> > > > >> >> >
>> >> > > > > > > > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM,
>> Harsha <
>> >> > > > > > > kafka@harsha.io
>> >> > > > > > > > >
>> >> > > > > > > > > >> wrote:
>> >> > > > > > > > > >> > > > >> >> > > Gwen,
>> >> > > > > > > > > >> > > > >> >> > >            On your second point.
>> Can you
>> >> > > > > describe
>> >> > > > > > a
>> >> > > > > > > > > >> usecase
>> >> > > > > > > > > >> > > where
>> >> > > > > > > > > >> > > > >> >> > >            mutliple clients ended up
>> >> > > sharing a
>> >> > > > > > > > producer
>> >> > > > > > > > > >> and
>> >> > > > > > > > > >> > > even
>> >> > > > > > > > > >> > > > if
>> >> > > > > > > > > >> > > > >> they
>> >> > > > > > > > > >> > > > >> >> > >            do why can't they not use
>> >> > single
>> >> > > > > token
>> >> > > > > > > that
>> >> > > > > > > > > >> producer
>> >> > > > > > > > > >> > > > >> >> > >            captures. Why would we
>> need
>> >> > > > multiple
>> >> > > > > > > > clients
>> >> > > > > > > > > >> with
>> >> > > > > > > > > >> > > > >> different
>> >> > > > > > > > > >> > > > >> >> > >            tokens sharing a single
>> >> > instance
>> >> > > of
>> >> > > > > > > > producer.
>> >> > > > > > > > > >> Also
>> >> > > > > > > > > >> > > in
>> >> > > > > > > > > >> > > > >> this
>> >> > > > > > > > > >> > > > >> >> > >            case other clients have
>> access
>> >> > > all
>> >> > > > > the
>> >> > > > > > > > tokens
>> >> > > > > > > > > >> no?
>> >> > > > > > > > > >> > > > >> >> > >
>> >> > > > > > > > > >> > > > >> >> > > Thanks,
>> >> > > > > > > > > >> > > > >> >> > > Harsha
>> >> > > > > > > > > >> > > > >> >> > >
>> >> > > > > > > > > >> > > > >> >> > >
>> >> > > > > > > > > >> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM,
>> Gwen
>> >> > > Shapira
>> >> > > > > > > wrote:
>> >> > > > > > > > > >> > > > >> >> > >> Sorry for the delay:
>> >> > > > > > > > > >> > > > >> >> > >>
>> >> > > > > > > > > >> > > > >> >> > >> Two questions that we didn't see
>> in the
>> >> > > wiki:
>> >> > > > > > > > > >> > > > >> >> > >> 1. Is there an expiration for
>> delegation
>> >> > > > > tokens?
>> >> > > > > > > > > >> Renewal? How
>> >> > > > > > > > > >> > > > do we
>> >> > > > > > > > > >> > > > >> >> > >> revoke them?
>> >> > > > > > > > > >> > > > >> >> > >> 2. If we want to use delegation
>> tokens
>> >> > for
>> >> > > > > > "do-as"
>> >> > > > > > > > > (say,
>> >> > > > > > > > > >> > > submit
>> >> > > > > > > > > >> > > > >> Storm
>> >> > > > > > > > > >> > > > >> >> > >> job as my user), we will need a
>> producer
>> >> > > for
>> >> > > > > > every
>> >> > > > > > > > job
>> >> > > > > > > > > >> (we
>> >> > > > > > > > > >> > > can't
>> >> > > > > > > > > >> > > > >> share
>> >> > > > > > > > > >> > > > >> >> > >> them between multiple jobs running
>> on
>> >> > same
>> >> > > > > node),
>> >> > > > > > > > since
>> >> > > > > > > > > >> we
>> >> > > > > > > > > >> > > only
>> >> > > > > > > > > >> > > > >> >> > >> authenticate when connecting. Is
>> there a
>> >> > > plan
>> >> > > > > to
>> >> > > > > > > > change
>> >> > > > > > > > > >> this
>> >> > > > > > > > > >> > > for
>> >> > > > > > > > > >> > > > >> >> > >> delegation tokens, in order to
>> allow
>> >> > > multiple
>> >> > > > > > users
>> >> > > > > > > > > with
>> >> > > > > > > > > >> > > > different
>> >> > > > > > > > > >> > > > >> >> > >> tokens to share a client?
>> >> > > > > > > > > >> > > > >> >> > >>
>> >> > > > > > > > > >> > > > >> >> > >> Gwen
>> >> > > > > > > > > >> > > > >> >> > >>
>> >> > > > > > > > > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM,
>> parth
>> >> > > > > brahmbhatt
>> >> > > > > > > > > >> > > > >> >> > >> <br...@gmail.com>
>> wrote:
>> >> > > > > > > > > >> > > > >> >> > >> > Bumping this up one more time,
>> can
>> >> > other
>> >> > > > > > > committers
>> >> > > > > > > > > >> review?
>> >> > > > > > > > > >> > > > >> >> > >> >
>> >> > > > > > > > > >> > > > >> >> > >> > Thanks
>> >> > > > > > > > > >> > > > >> >> > >> > Parth
>> >> > > > > > > > > >> > > > >> >> > >> >
>> >> > > > > > > > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM,
>> >> > Harsha <
>> >> > > > > > > > > >> kafka@harsha.io>
>> >> > > > > > > > > >> > > > wrote:
>> >> > > > > > > > > >> > > > >> >> > >> >
>> >> > > > > > > > > >> > > > >> >> > >> >> Parth,
>> >> > > > > > > > > >> > > > >> >> > >> >>           Overall current
>> design looks
>> >> > > > good
>> >> > > > > to
>> >> > > > > > > > me. I
>> >> > > > > > > > > >> am +1
>> >> > > > > > > > > >> > > on
>> >> > > > > > > > > >> > > > >> the
>> >> > > > > > > > > >> > > > >> >> > KIP.
>> >> > > > > > > > > >> > > > >> >> > >> >>
>> >> > > > > > > > > >> > > > >> >> > >> >> Gwen , Jun can you review this
>> as
>> >> > well.
>> >> > > > > > > > > >> > > > >> >> > >> >>
>> >> > > > > > > > > >> > > > >> >> > >> >> -Harsha
>> >> > > > > > > > > >> > > > >> >> > >> >>
>> >> > > > > > > > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57
>> AM,
>> >> > parth
>> >> > > > > > > > brahmbhatt
>> >> > > > > > > > > >> wrote:
>> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks for review Jitendra.
>> >> > > > > > > > > >> > > > >> >> > >> >> >
>> >> > > > > > > > > >> > > > >> >> > >> >> > I don't like the idea of
>> infinite
>> >> > > > lifetime
>> >> > > > > > > but I
>> >> > > > > > > > > >> see the
>> >> > > > > > > > > >> > > > >> Streaming
>> >> > > > > > > > > >> > > > >> >> > use
>> >> > > > > > > > > >> > > > >> >> > >> >> > case. Even for Streaming use
>> case I
>> >> > > was
>> >> > > > > > hoping
>> >> > > > > > > > > >> there will
>> >> > > > > > > > > >> > > > be
>> >> > > > > > > > > >> > > > >> some
>> >> > > > > > > > > >> > > > >> >> > notion
>> >> > > > > > > > > >> > > > >> >> > >> >> > of
>> >> > > > > > > > > >> > > > >> >> > >> >> > master/driver that can get new
>> >> > > > delegation
>> >> > > > > > > tokens
>> >> > > > > > > > > at
>> >> > > > > > > > > >> fixed
>> >> > > > > > > > > >> > > > >> interval
>> >> > > > > > > > > >> > > > >> >> > and
>> >> > > > > > > > > >> > > > >> >> > >> >> > distribute to workers. If
>> that is
>> >> > not
>> >> > > > the
>> >> > > > > > case
>> >> > > > > > > > for
>> >> > > > > > > > > >> we can
>> >> > > > > > > > > >> > > > >> discuss
>> >> > > > > > > > > >> > > > >> >> > >> >> > delegation tokens renewing
>> them self
>> >> > > and
>> >> > > > > the
>> >> > > > > > > > > >> security
>> >> > > > > > > > > >> > > > >> implications
>> >> > > > > > > > > >> > > > >> >> > of the
>> >> > > > > > > > > >> > > > >> >> > >> >> > same.
>> >> > > > > > > > > >> > > > >> >> > >> >> >
>> >> > > > > > > > > >> > > > >> >> > >> >> > I did not want clients to
>> fetch
>> >> > tokens
>> >> > > > > from
>> >> > > > > > > > > >> zookeeper,
>> >> > > > > > > > > >> > > > >> overall I
>> >> > > > > > > > > >> > > > >> >> > think
>> >> > > > > > > > > >> > > > >> >> > >> >> > its
>> >> > > > > > > > > >> > > > >> >> > >> >> > better if clients don't rely
>> on our
>> >> > > > > metadata
>> >> > > > > > > > store
>> >> > > > > > > > > >> and I
>> >> > > > > > > > > >> > > > >> think we
>> >> > > > > > > > > >> > > > >> >> > are
>> >> > > > > > > > > >> > > > >> >> > >> >> > moving in that direction with
>> all
>> >> > the
>> >> > > > > KIP-4
>> >> > > > > > > > > >> improvements.
>> >> > > > > > > > > >> > > > I
>> >> > > > > > > > > >> > > > >> chose
>> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as in this case the
>> client
>> >> > > > will
>> >> > > > > > > still
>> >> > > > > > > > > >> just talk
>> >> > > > > > > > > >> > > > to
>> >> > > > > > > > > >> > > > >> >> > broker , its
>> >> > > > > > > > > >> > > > >> >> > >> >> > the brokers that will use
>> zookeeper
>> >> > > > which
>> >> > > > > we
>> >> > > > > > > > > >> already do
>> >> > > > > > > > > >> > > > for a
>> >> > > > > > > > > >> > > > >> lot
>> >> > > > > > > > > >> > > > >> >> > of
>> >> > > > > > > > > >> > > > >> >> > >> >> > other
>> >> > > > > > > > > >> > > > >> >> > >> >> > usecases + ease of
>> development + and
>> >> > > the
>> >> > > > > > > ability
>> >> > > > > > > > > so
>> >> > > > > > > > > >> > > tokens
>> >> > > > > > > > > >> > > > >> will
>> >> > > > > > > > > >> > > > >> >> > survive
>> >> > > > > > > > > >> > > > >> >> > >> >> > even a rolling restart/cluster
>> >> > > failure.
>> >> > > > > if a
>> >> > > > > > > > > >> majority
>> >> > > > > > > > > >> > > > agrees
>> >> > > > > > > > > >> > > > >> the
>> >> > > > > > > > > >> > > > >> >> > added
>> >> > > > > > > > > >> > > > >> >> > >> >> > complexity to have controller
>> >> > > forwarding
>> >> > > > > > keys
>> >> > > > > > > to
>> >> > > > > > > > > all
>> >> > > > > > > > > >> > > > broker is
>> >> > > > > > > > > >> > > > >> >> > justified
>> >> > > > > > > > > >> > > > >> >> > >> >> > as
>> >> > > > > > > > > >> > > > >> >> > >> >> > it provides tighter security
>> , I am
>> >> > > fine
>> >> > > > > > with
>> >> > > > > > > > that
>> >> > > > > > > > > >> option
>> >> > > > > > > > > >> > > > too.
>> >> > > > > > > > > >> > > > >> >> > >> >> >
>> >> > > > > > > > > >> > > > >> >> > >> >> > Given zookeeper does not
>> support SSL
>> >> > > we
>> >> > > > > can
>> >> > > > > > > not
>> >> > > > > > > > > >> store
>> >> > > > > > > > > >> > > > master
>> >> > > > > > > > > >> > > > >> keys
>> >> > > > > > > > > >> > > > >> >> > in
>> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as master keys will
>> be
>> >> > > exposed
>> >> > > > > on
>> >> > > > > > > > wire.
>> >> > > > > > > > > To
>> >> > > > > > > > > >> > > > support
>> >> > > > > > > > > >> > > > >> >> > rotation
>> >> > > > > > > > > >> > > > >> >> > >> >> > without affecting current
>> clients is
>> >> > > > > > > something I
>> >> > > > > > > > > >> need to
>> >> > > > > > > > > >> > > > put
>> >> > > > > > > > > >> > > > >> more
>> >> > > > > > > > > >> > > > >> >> > thought
>> >> > > > > > > > > >> > > > >> >> > >> >> > in. My current proposal
>> assumes the
>> >> > > > > rotation
>> >> > > > > > > > will
>> >> > > > > > > > > >> > > > invalidate
>> >> > > > > > > > > >> > > > >> all
>> >> > > > > > > > > >> > > > >> >> > current
>> >> > > > > > > > > >> > > > >> >> > >> >> > tokens.
>> >> > > > > > > > > >> > > > >> >> > >> >> >
>> >> > > > > > > > > >> > > > >> >> > >> >> > I request committers to also
>> review
>> >> > > and
>> >> > > > > post
>> >> > > > > > > > their
>> >> > > > > > > > > >> > > comments
>> >> > > > > > > > > >> > > > >> so we
>> >> > > > > > > > > >> > > > >> >> > can
>> >> > > > > > > > > >> > > > >> >> > >> >> > make
>> >> > > > > > > > > >> > > > >> >> > >> >> > progress on this KIP.
>> >> > > > > > > > > >> > > > >> >> > >> >> >
>> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks
>> >> > > > > > > > > >> > > > >> >> > >> >> > Parth
>> >> > > > > > > > > >> > > > >> >> > >> >> >
>> >> > > > > > > > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39
>> AM,
>> >> > > Ashish
>> >> > > > > > Singh
>> >> > > > > > > <
>> >> > > > > > > > > >> > > > >> asingh@cloudera.com
>> >> > > > > > > > > >> > > > >> >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > wrote:
>> >> > > > > > > > > >> > > > >> >> > >> >> >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at
>> 11:26 AM,
>> >> > > > > Harsha <
>> >> > > > > > > > > >> > > > kafka@harsha.io>
>> >> > > > > > > > > >> > > > >> >> > wrote:
>> >> > > > > > > > > >> > > > >> >> > >> >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > Unifying the two
>> discussion
>> >> > > threads
>> >> > > > on
>> >> > > > > > > this
>> >> > > > > > > > > KIP.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > Here is the response from
>> >> > Jitendra
>> >> > > > > > > > > >> > > > >> >> > >> >> > > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > "The need for a large
>> number of
>> >> > > > > clients
>> >> > > > > > > that
>> >> > > > > > > > > are
>> >> > > > > > > > > >> > > > running
>> >> > > > > > > > > >> > > > >> all
>> >> > > > > > > > > >> > > > >> >> > over the
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > cluster that authenticate
>> with
>> >> > > Kafka
>> >> > > > > > > > brokers,
>> >> > > > > > > > > >> is very
>> >> > > > > > > > > >> > > > >> similar
>> >> > > > > > > > > >> > > > >> >> > to the
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > Hadoop use case of large
>> number
>> >> > of
>> >> > > > > tasks
>> >> > > > > > > > > running
>> >> > > > > > > > > >> > > across
>> >> > > > > > > > > >> > > > >> the
>> >> > > > > > > > > >> > > > >> >> > cluster
>> >> > > > > > > > > >> > > > >> >> > >> >> that
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > need authentication to
>> Hdfs
>> >> > > > Namenode.
>> >> > > > > > > > > >> Therefore, the
>> >> > > > > > > > > >> > > > >> >> > delegation token
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > approach does seem like a
>> good
>> >> > fit
>> >> > > > for
>> >> > > > > > > this
>> >> > > > > > > > > use
>> >> > > > > > > > > >> case
>> >> > > > > > > > > >> > > > as we
>> >> > > > > > > > > >> > > > >> >> > have seen
>> >> > > > > > > > > >> > > > >> >> > >> >> it
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > working at large scale in
>> HDFS
>> >> > and
>> >> > > > > YARN.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > >   The proposed design is
>> very
>> >> > much
>> >> > > > > > inline
>> >> > > > > > > > with
>> >> > > > > > > > > >> Hadoop
>> >> > > > > > > > > >> > > > >> >> > approach. A few
>> >> > > > > > > > > >> > > > >> >> > >> >> > > >   comments:
>> >> > > > > > > > > >> > > > >> >> > >> >> > > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > 1) Why do you guys want
>> to allow
>> >> > > > > > infinite
>> >> > > > > > > > > >> renewable
>> >> > > > > > > > > >> > > > >> lifetime
>> >> > > > > > > > > >> > > > >> >> > for a
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > token? HDFS restricts a
>> token
>> >> > to a
>> >> > > > max
>> >> > > > > > > life
>> >> > > > > > > > > time
>> >> > > > > > > > > >> > > > (default
>> >> > > > > > > > > >> > > > >> 7
>> >> > > > > > > > > >> > > > >> >> > days).  A
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > token's vulnerability is
>> >> > believed
>> >> > > to
>> >> > > > > > > > increase
>> >> > > > > > > > > >> with
>> >> > > > > > > > > >> > > > time.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > I agree that having infinite
>> >> > > lifetime
>> >> > > > > > might
>> >> > > > > > > > not
>> >> > > > > > > > > >> be the
>> >> > > > > > > > > >> > > > best
>> >> > > > > > > > > >> > > > >> idea.
>> >> > > > > > > > > >> > > > >> >> > >> >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > 2) As I understand the
>> tokens
>> >> > are
>> >> > > > > stored
>> >> > > > > > > in
>> >> > > > > > > > > >> zookeeper
>> >> > > > > > > > > >> > > > as
>> >> > > > > > > > > >> > > > >> well,
>> >> > > > > > > > > >> > > > >> >> > and
>> >> > > > > > > > > >> > > > >> >> > >> >> can
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > be updated there. This is
>> clever
>> >> > > as
>> >> > > > it
>> >> > > > > > can
>> >> > > > > > > > > allow
>> >> > > > > > > > > >> > > > >> replacing the
>> >> > > > > > > > > >> > > > >> >> > tokens
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > once they run out of max
>> life
>> >> > > time,
>> >> > > > > and
>> >> > > > > > > > > clients
>> >> > > > > > > > > >> can
>> >> > > > > > > > > >> > > > >> download
>> >> > > > > > > > > >> > > > >> >> > new
>> >> > > > > > > > > >> > > > >> >> > >> >> tokens
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > from zookeeper. It
>> shouldn't be
>> >> > a
>> >> > > > big
>> >> > > > > > load
>> >> > > > > > > > on
>> >> > > > > > > > > >> > > zookeeper
>> >> > > > > > > > > >> > > > >> as a
>> >> > > > > > > > > >> > > > >> >> > client
>> >> > > > > > > > > >> > > > >> >> > >> >> will
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > need to get a new token
>> once in
>> >> > > > > several
>> >> > > > > > > > days.
>> >> > > > > > > > > >> In this
>> >> > > > > > > > > >> > > > >> approach
>> >> > > > > > > > > >> > > > >> >> > you
>> >> > > > > > > > > >> > > > >> >> > >> >> don't
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > need infinite lifetime on
>> the
>> >> > > token
>> >> > > > > even
>> >> > > > > > > for
>> >> > > > > > > > > >> long
>> >> > > > > > > > > >> > > > running
>> >> > > > > > > > > >> > > > >> >> > clients.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > 3) The token password are
>> >> > > generated
>> >> > > > > > using
>> >> > > > > > > a
>> >> > > > > > > > > >> master
>> >> > > > > > > > > >> > > key.
>> >> > > > > > > > > >> > > > >> The
>> >> > > > > > > > > >> > > > >> >> > master
>> >> > > > > > > > > >> > > > >> >> > >> >> key
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > should also be
>> periodically
>> >> > > changed.
>> >> > > > > In
>> >> > > > > > > > > Hadoop,
>> >> > > > > > > > > >> the
>> >> > > > > > > > > >> > > > >> default
>> >> > > > > > > > > >> > > > >> >> > renewal
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > period is 1 day.?
>> >> > > > > > > > > >> > > > >> >> > >> >> > > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > IIUC, this will require
>> brokers
>> >> > > > > > maintaining
>> >> > > > > > > a
>> >> > > > > > > > > >> list of X
>> >> > > > > > > > > >> > > > most
>> >> > > > > > > > > >> > > > >> >> > recent
>> >> > > > > > > > > >> > > > >> >> > >> >> master
>> >> > > > > > > > > >> > > > >> >> > >> >> > > keys. This list will have
>> to be
>> >> > > > > persisted
>> >> > > > > > > > > >> somewhere, as
>> >> > > > > > > > > >> > > > if a
>> >> > > > > > > > > >> > > > >> >> > broker
>> >> > > > > > > > > >> > > > >> >> > >> >> goes
>> >> > > > > > > > > >> > > > >> >> > >> >> > > down it will have to get
>> that list
>> >> > > > again
>> >> > > > > > and
>> >> > > > > > > > > >> storing
>> >> > > > > > > > > >> > > > master
>> >> > > > > > > > > >> > > > >> keys
>> >> > > > > > > > > >> > > > >> >> > on ZK
>> >> > > > > > > > > >> > > > >> >> > >> >> is
>> >> > > > > > > > > >> > > > >> >> > >> >> > > not the best idea. However,
>> if a
>> >> > > > broker
>> >> > > > > > goes
>> >> > > > > > > > > down
>> >> > > > > > > > > >> then
>> >> > > > > > > > > >> > > we
>> >> > > > > > > > > >> > > > >> have
>> >> > > > > > > > > >> > > > >> >> > much
>> >> > > > > > > > > >> > > > >> >> > >> >> bigger
>> >> > > > > > > > > >> > > > >> >> > >> >> > > issue to deal with and
>> client can
>> >> > > > always
>> >> > > > > > > > > >> > > re-authenticate
>> >> > > > > > > > > >> > > > is
>> >> > > > > > > > > >> > > > >> such
>> >> > > > > > > > > >> > > > >> >> > >> >> events.
>> >> > > > > > > > > >> > > > >> >> > >> >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > Did you happen to take a
>> look at
>> >> > > other
>> >> > > > > > > > > >> alternatives
>> >> > > > > > > > > >> > > this
>> >> > > > > > > > > >> > > > >> list has
>> >> > > > > > > > > >> > > > >> >> > >> >> > > suggested?
>> >> > > > > > > > > >> > > > >> >> > >> >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > Thanks for a thorough
>> proposal,
>> >> > > > great
>> >> > > > > > > work!"
>> >> > > > > > > > > >> > > > >> >> > >> >> > > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at
>> 10:28
>> >> > PM,
>> >> > > > Gwen
>> >> > > > > > > > Shapira
>> >> > > > > > > > > >> wrote:
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > Makes sense to me.
>> Thanks!
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at
>> 9:25
>> >> > PM,
>> >> > > > > > Harsha <
>> >> > > > > > > > > >> > > > kafka@harsha.io
>> >> > > > > > > > > >> > > > >> >
>> >> > > > > > > > > >> > > > >> >> > wrote:
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > It doesn't need any
>> release
>> >> > > > > vehicle
>> >> > > > > > > but
>> >> > > > > > > > > >> still the
>> >> > > > > > > > > >> > > > >> work can
>> >> > > > > > > > > >> > > > >> >> > move
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > forward.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > If anyone is
>> interested in
>> >> > the
>> >> > > > KIP
>> >> > > > > > > > please
>> >> > > > > > > > > >> do the
>> >> > > > > > > > > >> > > > >> review and
>> >> > > > > > > > > >> > > > >> >> > >> >> provide
>> >> > > > > > > > > >> > > > >> >> > >> >> > > the
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > comments.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > -Harsha
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016,
>> at
>> >> > 04:59
>> >> > > > PM,
>> >> > > > > > > Ismael
>> >> > > > > > > > > >> Juma
>> >> > > > > > > > > >> > > > wrote:
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> I agree that it
>> would be
>> >> > good
>> >> > > > to
>> >> > > > > > have
>> >> > > > > > > > > more
>> >> > > > > > > > > >> time
>> >> > > > > > > > > >> > > to
>> >> > > > > > > > > >> > > > >> review
>> >> > > > > > > > > >> > > > >> >> > and
>> >> > > > > > > > > >> > > > >> >> > >> >> > > discuss
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> KIP-48.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> Ismael
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016
>> at
>> >> > 12:55
>> >> > > > AM,
>> >> > > > > > Gwen
>> >> > > > > > > > > >> Shapira <
>> >> > > > > > > > > >> > > > >> >> > >> >> gwen@confluent.io>
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Hi Team,
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Since KIP-48
>> depends on
>> >> > > > KIP-43,
>> >> > > > > > > which
>> >> > > > > > > > > is
>> >> > > > > > > > > >> > > > already a
>> >> > > > > > > > > >> > > > >> bit
>> >> > > > > > > > > >> > > > >> >> > of a
>> >> > > > > > > > > >> > > > >> >> > >> >> risk
>> >> > > > > > > > > >> > > > >> >> > >> >> > > for
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > the next release -
>> any
>> >> > > chance
>> >> > > > > we
>> >> > > > > > > can
>> >> > > > > > > > > >> delay
>> >> > > > > > > > > >> > > > >> delegation
>> >> > > > > > > > > >> > > > >> >> > tokens
>> >> > > > > > > > > >> > > > >> >> > >> >> to
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > Kafka
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > 0.10.1?
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > With the community
>> >> > working
>> >> > > > on a
>> >> > > > > > > > release
>> >> > > > > > > > > >> every
>> >> > > > > > > > > >> > > 3
>> >> > > > > > > > > >> > > > >> month,
>> >> > > > > > > > > >> > > > >> >> > this
>> >> > > > > > > > > >> > > > >> >> > >> >> is not
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > a huge
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > delay.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Gwen
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26,
>> 2016 at
>> >> > > 5:11
>> >> > > > > PM,
>> >> > > > > > > > Ashish
>> >> > > > > > > > > >> Singh
>> >> > > > > > > > > >> > > <
>> >> > > > > > > > > >> > > > >> >> > >> >> > > asingh@cloudera.com>
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Parth,
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Thanks again for
>> the
>> >> > > > awesome
>> >> > > > > > > write
>> >> > > > > > > > > up.
>> >> > > > > > > > > >> > > > Following
>> >> > > > > > > > > >> > > > >> our
>> >> > > > > > > > > >> > > > >> >> > >> >> discussion
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > from the
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I think it
>> will
>> >> > be
>> >> > > > > easier
>> >> > > > > > > to
>> >> > > > > > > > > >> compare
>> >> > > > > > > > > >> > > > >> various
>> >> > > > > > > > > >> > > > >> >> > >> >> alternatives
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > if they
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > are
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > listed together.
>> I am
>> >> > > > stating
>> >> > > > > > > > below a
>> >> > > > > > > > > >> few
>> >> > > > > > > > > >> > > > >> >> > alternatives along
>> >> > > > > > > > > >> > > > >> >> > >> >> > > with
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > a the
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > current proposal.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Current
>> proposal)
>> >> > Store
>> >> > > > > > > Delegation
>> >> > > > > > > > > >> Token,
>> >> > > > > > > > > >> > > DT,
>> >> > > > > > > > > >> > > > >> on ZK.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
>> >> > > authenticates
>> >> > > > > > with a
>> >> > > > > > > > > >> broker.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a
>> client is
>> >> > > > > > > > authenticated,
>> >> > > > > > > > > >> it
>> >> > > > > > > > > >> > > will
>> >> > > > > > > > > >> > > > >> make a
>> >> > > > > > > > > >> > > > >> >> > broker
>> >> > > > > > > > > >> > > > >> >> > >> >> side
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a
>> delegation
>> >> > > > token.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker
>> >> > > generates
>> >> > > > a
>> >> > > > > > > shared
>> >> > > > > > > > > >> secret
>> >> > > > > > > > > >> > > > based
>> >> > > > > > > > > >> > > > >> on
>> >> > > > > > > > > >> > > > >> >> > >> >> > > HMAC-SHA256(a
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> Password/Secret
>> >> > shared
>> >> > > > > > between
>> >> > > > > > > > all
>> >> > > > > > > > > >> > > brokers,
>> >> > > > > > > > > >> > > > >> >> > randomly
>> >> > > > > > > > > >> > > > >> >> > >> >> > > generated
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > tokenId).
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Broker
>> stores
>> >> > this
>> >> > > > > token
>> >> > > > > > in
>> >> > > > > > > > its
>> >> > > > > > > > > >> in
>> >> > > > > > > > > >> > > > memory
>> >> > > > > > > > > >> > > > >> cache.
>> >> > > > > > > > > >> > > > >> >> > >> >> Broker
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > also stores
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the
>> DelegationToken
>> >> > > > > without
>> >> > > > > > > the
>> >> > > > > > > > > >> hmac in
>> >> > > > > > > > > >> > > the
>> >> > > > > > > > > >> > > > >> >> > zookeeper.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. All
>> brokers will
>> >> > > > have a
>> >> > > > > > > cache
>> >> > > > > > > > > >> backed
>> >> > > > > > > > > >> > > by
>> >> > > > > > > > > >> > > > >> >> > zookeeper so
>> >> > > > > > > > > >> > > > >> >> > >> >> they
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > will all
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    get notified
>> >> > whenever
>> >> > > a
>> >> > > > > new
>> >> > > > > > > > token
>> >> > > > > > > > > is
>> >> > > > > > > > > >> > > > >> generated and
>> >> > > > > > > > > >> > > > >> >> > they
>> >> > > > > > > > > >> > > > >> >> > >> >> will
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > update
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    local cache
>> whenever
>> >> > > > token
>> >> > > > > > > state
>> >> > > > > > > > > >> changes.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Broker
>> returns
>> >> > the
>> >> > > > > token
>> >> > > > > > to
>> >> > > > > > > > > >> Client.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues
>> and
>> >> > fixes
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Probable
>> race
>> >> > > > > condition,
>> >> > > > > > > > client
>> >> > > > > > > > > >> tries
>> >> > > > > > > > > >> > > to
>> >> > > > > > > > > >> > > > >> >> > authenticate
>> >> > > > > > > > > >> > > > >> >> > >> >> with
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > a broker
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    that is yet
>> to be
>> >> > > > updated
>> >> > > > > > with
>> >> > > > > > > > the
>> >> > > > > > > > > >> newly
>> >> > > > > > > > > >> > > > >> generated
>> >> > > > > > > > > >> > > > >> >> > DT.
>> >> > > > > > > > > >> > > > >> >> > >> >> This
>> >> > > > > > > > > >> > > > >> >> > >> >> > > can
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > probably be
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    dealt with
>> making
>> >> > > > > dtRequest
>> >> > > > > > > > block
>> >> > > > > > > > > >> until
>> >> > > > > > > > > >> > > all
>> >> > > > > > > > > >> > > > >> >> > brokers have
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > updated
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their DT
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    cache. Zk
>> barrier or
>> >> > > > > similar
>> >> > > > > > > > > >> mechanism
>> >> > > > > > > > > >> > > can
>> >> > > > > > > > > >> > > > be
>> >> > > > > > > > > >> > > > >> used.
>> >> > > > > > > > > >> > > > >> >> > >> >> However,
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > all such
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    mechanisms
>> will
>> >> > > increase
>> >> > > > > > > > > complexity.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Using a
>> static
>> >> > > secret
>> >> > > > > key
>> >> > > > > > > > from
>> >> > > > > > > > > >> config
>> >> > > > > > > > > >> > > > >> file. Will
>> >> > > > > > > > > >> > > > >> >> > >> >> require
>> >> > > > > > > > > >> > > > >> >> > >> >> > > yet
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > another
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    config and
>> uses a
>> >> > > static
>> >> > > > > > > secret
>> >> > > > > > > > > >> key. It
>> >> > > > > > > > > >> > > is
>> >> > > > > > > > > >> > > > >> advised
>> >> > > > > > > > > >> > > > >> >> > to
>> >> > > > > > > > > >> > > > >> >> > >> >> rotate
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > secret
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > keys
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    periodically.
>> This
>> >> > can
>> >> > > > be
>> >> > > > > > > > avoided
>> >> > > > > > > > > >> with
>> >> > > > > > > > > >> > > > >> controller
>> >> > > > > > > > > >> > > > >> >> > >> >> generating
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > secretKey and
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    passing to
>> brokers
>> >> > > > > > > periodically.
>> >> > > > > > > > > >> However,
>> >> > > > > > > > > >> > > > >> this will
>> >> > > > > > > > > >> > > > >> >> > >> >> require
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > brokers to
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maintain
>> certain
>> >> > > counts
>> >> > > > of
>> >> > > > > > > > > >> secretKeys.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 1)
>> Have
>> >> > > > > controller
>> >> > > > > > > > > >> generate
>> >> > > > > > > > > >> > > > >> delegation
>> >> > > > > > > > > >> > > > >> >> > token.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
>> >> > > authenticates
>> >> > > > > > with a
>> >> > > > > > > > > >> broker.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a
>> client is
>> >> > > > > > > > authenticated,
>> >> > > > > > > > > >> it
>> >> > > > > > > > > >> > > will
>> >> > > > > > > > > >> > > > >> make a
>> >> > > > > > > > > >> > > > >> >> > broker
>> >> > > > > > > > > >> > > > >> >> > >> >> side
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a
>> delegation
>> >> > > > token.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. Broker
>> forwards
>> >> > the
>> >> > > > > > request
>> >> > > > > > > > to
>> >> > > > > > > > > >> > > > controller.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Controller
>> >> > > generates
>> >> > > > a
>> >> > > > > DT
>> >> > > > > > > and
>> >> > > > > > > > > >> > > broadcasts
>> >> > > > > > > > > >> > > > >> to all
>> >> > > > > > > > > >> > > > >> >> > >> >> brokers.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. Broker
>> stores
>> >> > this
>> >> > > > > token
>> >> > > > > > in
>> >> > > > > > > > its
>> >> > > > > > > > > >> memory
>> >> > > > > > > > > >> > > > >> cache.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Controller
>> >> > responds
>> >> > > > to
>> >> > > > > > > > broker’s
>> >> > > > > > > > > >> DT
>> >> > > > > > > > > >> > > req.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    7. Broker
>> returns
>> >> > the
>> >> > > > > token
>> >> > > > > > to
>> >> > > > > > > > > >> Client.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues
>> and
>> >> > fixes
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. We will
>> have to
>> >> > add
>> >> > > > new
>> >> > > > > > > APIs
>> >> > > > > > > > to
>> >> > > > > > > > > >> > > support
>> >> > > > > > > > > >> > > > >> >> > controller
>> >> > > > > > > > > >> > > > >> >> > >> >> pushing
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > to
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    brokers on
>> top of
>> >> > the
>> >> > > > > > minimal
>> >> > > > > > > > APIs
>> >> > > > > > > > > >> that
>> >> > > > > > > > > >> > > are
>> >> > > > > > > > > >> > > > >> >> > currently
>> >> > > > > > > > > >> > > > >> >> > >> >> > > proposed.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. We will
>> also have
>> >> > > to
>> >> > > > > add
>> >> > > > > > > APIs
>> >> > > > > > > > > to
>> >> > > > > > > > > >> > > support
>> >> > > > > > > > > >> > > > >> the
>> >> > > > > > > > > >> > > > >> >> > >> >> bootstrapping
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > case,
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > i.e,
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    when a new
>> broker
>> >> > > comes
>> >> > > > up
>> >> > > > > > it
>> >> > > > > > > > will
>> >> > > > > > > > > >> have
>> >> > > > > > > > > >> > > to
>> >> > > > > > > > > >> > > > >> get all
>> >> > > > > > > > > >> > > > >> >> > >> >> delegation
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > from
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the
>> controller.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. In
>> catastrophic
>> >> > > > > failures
>> >> > > > > > > > where
>> >> > > > > > > > > >> all
>> >> > > > > > > > > >> > > > brokers
>> >> > > > > > > > > >> > > > >> go
>> >> > > > > > > > > >> > > > >> >> > down,
>> >> > > > > > > > > >> > > > >> >> > >> >> the
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even
>> if
>> >> > > servers
>> >> > > > > are
>> >> > > > > > > > > >> restarted as
>> >> > > > > > > > > >> > > > >> tokens
>> >> > > > > > > > > >> > > > >> >> > are not
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this
>> happens,
>> >> > then
>> >> > > > > there
>> >> > > > > > > are
>> >> > > > > > > > > more
>> >> > > > > > > > > >> > > > important
>> >> > > > > > > > > >> > > > >> >> > things to
>> >> > > > > > > > > >> > > > >> >> > >> >> > > worry
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > about
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is
>> better
>> >> > to
>> >> > > > > > > > > >> re-authenticate.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 2)
>> Do not
>> >> > > > > > distribute
>> >> > > > > > > > DT
>> >> > > > > > > > > to
>> >> > > > > > > > > >> > > other
>> >> > > > > > > > > >> > > > >> brokers
>> >> > > > > > > > > >> > > > >> >> > at
>> >> > > > > > > > > >> > > > >> >> > >> >> all.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
>> >> > > authenticates
>> >> > > > > > with a
>> >> > > > > > > > > >> broker.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a
>> client is
>> >> > > > > > > > authenticated,
>> >> > > > > > > > > >> it
>> >> > > > > > > > > >> > > will
>> >> > > > > > > > > >> > > > >> make a
>> >> > > > > > > > > >> > > > >> >> > broker
>> >> > > > > > > > > >> > > > >> >> > >> >> side
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a
>> delegation
>> >> > > > token.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker
>> >> > > generates
>> >> > > > DT
>> >> > > > > > of
>> >> > > > > > > > > form,
>> >> > > > > > > > > >> > > [hmac +
>> >> > > > > > > > > >> > > > >> (owner,
>> >> > > > > > > > > >> > > > >> >> > >> >> renewer,
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maxLifeTime,
>> id,
>> >> > hmac,
>> >> > > > > > > > > >> expirationTime)]
>> >> > > > > > > > > >> > > and
>> >> > > > > > > > > >> > > > >> passes
>> >> > > > > > > > > >> > > > >> >> > back
>> >> > > > > > > > > >> > > > >> >> > >> >> this
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > DT to
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > client.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    hmac is
>> generated
>> >> > via
>> >> > > > > > > > > >> {HMAC-SHA256(owner,
>> >> > > > > > > > > >> > > > >> renewer,
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > maxLifeTime, id,
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac,
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> expirationTime)
>> >> > using
>> >> > > > > > > > SecretKey}.
>> >> > > > > > > > > >> Note
>> >> > > > > > > > > >> > > that
>> >> > > > > > > > > >> > > > >> all
>> >> > > > > > > > > >> > > > >> >> > brokers
>> >> > > > > > > > > >> > > > >> >> > >> >> have
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > this
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > SecretKey.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Client
>> then goes
>> >> > to
>> >> > > > any
>> >> > > > > > > > broker
>> >> > > > > > > > > >> and to
>> >> > > > > > > > > >> > > > >> >> > authenticate
>> >> > > > > > > > > >> > > > >> >> > >> >> sends
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > the DT.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    Broker
>> recalculates
>> >> > > hmac
>> >> > > > > > using
>> >> > > > > > > > > >> (owner,
>> >> > > > > > > > > >> > > > >> renewer,
>> >> > > > > > > > > >> > > > >> >> > >> >> maxLifeTime,
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > id, hmac,
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> expirationTime) info
>> >> > > > from
>> >> > > > > DT
>> >> > > > > > > and
>> >> > > > > > > > > its
>> >> > > > > > > > > >> > > > >> SecretKey. If
>> >> > > > > > > > > >> > > > >> >> > it
>> >> > > > > > > > > >> > > > >> >> > >> >> matches
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > with
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac of
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    DT, client is
>> >> > > > > authenticated.
>> >> > > > > > > > Yes,
>> >> > > > > > > > > >> it will
>> >> > > > > > > > > >> > > > do
>> >> > > > > > > > > >> > > > >> other
>> >> > > > > > > > > >> > > > >> >> > >> >> obvious
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > checks of
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    timestamp
>> expiry and
>> >> > > > such.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Note that secret
>> key
>> >> > will
>> >> > > > be
>> >> > > > > > > > > generated
>> >> > > > > > > > > >> by
>> >> > > > > > > > > >> > > > >> controller
>> >> > > > > > > > > >> > > > >> >> > and
>> >> > > > > > > > > >> > > > >> >> > >> >> passed
>> >> > > > > > > > > >> > > > >> >> > >> >> > > to
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > brokers
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > periodically.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues
>> and
>> >> > fixes
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. How to
>> delete a
>> >> > DT?
>> >> > > > > Yes,
>> >> > > > > > > that
>> >> > > > > > > > > is
>> >> > > > > > > > > >> a
>> >> > > > > > > > > >> > > > downside
>> >> > > > > > > > > >> > > > >> >> > here.
>> >> > > > > > > > > >> > > > >> >> > >> >> However,
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > this can
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be handled
>> with
>> >> > > brokers
>> >> > > > > > > > > maintaining
>> >> > > > > > > > > >> a
>> >> > > > > > > > > >> > > > >> blacklist of
>> >> > > > > > > > > >> > > > >> >> > DTs,
>> >> > > > > > > > > >> > > > >> >> > >> >> DTs
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > from this
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > list
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    can be
>> removed after
>> >> > > > > expiry.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. In
>> catastrophic
>> >> > > > > failures
>> >> > > > > > > > where
>> >> > > > > > > > > >> all
>> >> > > > > > > > > >> > > > brokers
>> >> > > > > > > > > >> > > > >> go
>> >> > > > > > > > > >> > > > >> >> > down,
>> >> > > > > > > > > >> > > > >> >> > >> >> the
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even
>> if
>> >> > > servers
>> >> > > > > are
>> >> > > > > > > > > >> restarted as
>> >> > > > > > > > > >> > > > >> tokens
>> >> > > > > > > > > >> > > > >> >> > are not
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this
>> happens,
>> >> > then
>> >> > > > > there
>> >> > > > > > > are
>> >> > > > > > > > > more
>> >> > > > > > > > > >> > > > important
>> >> > > > > > > > > >> > > > >> >> > things to
>> >> > > > > > > > > >> > > > >> >> > >> >> > > worry
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > about
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is
>> better
>> >> > to
>> >> > > > > > > > > >> re-authenticate.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26,
>> 2016 at
>> >> > > > 1:58
>> >> > > > > > PM,
>> >> > > > > > > > > Parth
>> >> > > > > > > > > >> > > > >> Brahmbhatt <
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > pbrahmbhatt@hortonworks.com>
>> >> > > > > > > > wrote:
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Hi,
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> I have filed
>> KIP-48 so
>> >> > > we
>> >> > > > > can
>> >> > > > > > > > offer
>> >> > > > > > > > > >> hadoop
>> >> > > > > > > > > >> > > > like
>> >> > > > > > > > > >> > > > >> >> > delegation
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens in
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> kafka. You can
>> review
>> >> > > the
>> >> > > > > > design
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >>
>> >> > > > > > > > > >> > > > >> >> >
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > >
>> >> > > > > > > > > >>
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > .
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> This KIP
>> depends on
>> >> > > KIP-43
>> >> > > > > and
>> >> > > > > > > we
>> >> > > > > > > > > >> have also
>> >> > > > > > > > > >> > > > >> >> > discussed an
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > alternative to
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> proposed design
>> here<
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >>
>> >> > > > > > > > > >> > > > >> >> >
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > >
>> >> > > > > > > > > >>
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >.
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Thanks
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Parth
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > --
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Regards,
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Ashish
>> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > --
>> >> > > > > > > > > >> > > > >> >> > >> >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >> > > Regards,
>> >> > > > > > > > > >> > > > >> >> > >> >> > > Ashish
>> >> > > > > > > > > >> > > > >> >> > >> >> > >
>> >> > > > > > > > > >> > > > >> >> > >> >>
>> >> > > > > > > > > >> > > > >> >> >
>> >> > > > > > > > > >> > > > >>
>> >> > > > > > > > > >> > > >
>> >> > > > > > > > > >> > >
>> >> > > > > > > > > >>
>> >> > > > > > > > > >>
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > --
>> >> > > > > > Regards,
>> >> > > > > >
>> >> > > > > > Rajini
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > --
>> >> > > > Regards,
>> >> > > >
>> >> > > > Rajini
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Liquan Pei
>> >> Software Engineer, Confluent Inc
>>
>
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Jun Rao <ju...@confluent.io>]

Just to add on that list.

2. It would be good to document the format of the data stored in ZK.
7. Earlier, there was a discussion on whether the tokens should be
propagated through ZK like config/acl/quota, or through the controller.
Currently, the controller is only designed for propagating topic metadata,
but not other data.
8. Should we use SCRAM to send the token instead of DIGEST-MD5 since it's
deprecated?

Also, the images in the wiki seem broken.

Thanks,

Jun

On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira <gw...@confluent.io> wrote:

> From what I can see, remaining questions are:
>
> 1. Who / how are tokens renewed? By original requester only? or using
> Kerberos auth only?
> 2. Are tokens stored on each broker or in ZK?
> 3. How are tokens invalidated / expired?
> 4. Which encryption algorithm is used?
> 5. What is the impersonation proposal (it wasn't in the KIP but was
> discussed in this thread)?
> 6. Do we need new ACLs, if so - for what actions?
>
> Gwen
>
> On Thu, Jun 9, 2016 at 7:48 PM, Harsha <ka...@harsha.io> wrote:
> > Jun & Ismael,
> >                          Unfortunately I couldn't attend the KIP meeting
> >                          when delegation tokens discussed. Appreciate if
> >                          you can update the thread if you have any
> >                          further questions.
> > Thanks,
> > Harsha
> >
> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
> >> It seems that the links to images in the KIP are broken.
> >>
> >> Liquan
> >>
> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
> >> brahmbhatt.parth@gmail.com> wrote:
> >>
> >> > 110. What does getDelegationTokenAs mean?
> >> > In the current proposal we only allow a user to get delegation token
> for
> >> > the identity that it authenticated as using another mechanism, i.e. A
> user
> >> > that authenticate using a keytab for principal user1@EXAMPLE.COM
> will get
> >> > delegation tokens for that user only. In future I think we will have
> to
> >> > extend support such that we allow some set of users (
> >> > kafka-rest-user@EXAMPLE.COM, storm-nimbus@EXAMPLE.COM) to acquire
> >> > delegation tokens on behalf of other users whose identity they have
> >> > verified independently.  Kafka brokers will have ACLs to control which
> >> > users are allowed to impersonate other users and get tokens on behalf
> of
> >> > them. Overall Impersonation is a whole different problem in my
> opinion and
> >> > I think we can tackle it in separate KIP.
> >> >
> >> > 111. What's the typical rate of getting and renewing delegation
> tokens?
> >> > Typically this should be very very low, 1 request per minute is a
> >> > relatively high estimate. However it depends on the token expiration.
> I am
> >> > less worried about the extra load it puts on controller vs the added
> >> > complexity and the value it offers.
> >> >
> >> > Thanks
> >> > Parth
> >> >
> >> >
> >> >
> >> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <is...@juma.me.uk>
> wrote:
> >> >
> >> > > Thanks Rajini. It would probably require a separate KIP as it will
> >> > > introduce user visible changes. We could also update KIP-48 to have
> this
> >> > > information, but it seems cleaner to do it separately. We can
> discuss
> >> > that
> >> > > in the KIP call today.
> >> > >
> >> > > Ismael
> >> > >
> >> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <
> >> > > rajinisivaram@googlemail.com> wrote:
> >> > >
> >> > > > Ismael,
> >> > > >
> >> > > > I have created a JIRA (
> >> > https://issues.apache.org/jira/browse/KAFKA-3751)
> >> > > > for adding SCRAM as a SASL mechanism. Would that need another
> KIP? If
> >> > > > KIP-48 will use this mechanism, can this just be a JIRA that gets
> >> > > reviewed
> >> > > > when the PR is ready?
> >> > > >
> >> > > > Thank you,
> >> > > >
> >> > > > Rajini
> >> > > >
> >> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <is...@juma.me.uk>
> >> > wrote:
> >> > > >
> >> > > > > Thanks Rajini, SCRAM seems like a good candidate.
> >> > > > >
> >> > > > > Gwen had independently mentioned this as a SASL mechanism that
> might
> >> > be
> >> > > > > useful for Kafka and I have been meaning to check it in more
> detail.
> >> > > Good
> >> > > > > to know that you are willing to contribute an implementation.
> Maybe
> >> > we
> >> > > > > should file a separate JIRA for this?
> >> > > > >
> >> > > > > Ismael
> >> > > > >
> >> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram <
> >> > > > > rajinisivaram@googlemail.com> wrote:
> >> > > > >
> >> > > > > > SCRAM (Salted Challenge Response Authentication Mechanism) is
> a
> >> > > better
> >> > > > > > mechanism than Digest-MD5. Java doesn't come with a built-in
> SCRAM
> >> > > > > > SaslServer or SaslClient, but I will be happy to add support
> in
> >> > Kafka
> >> > > > > since
> >> > > > > > it would be a useful mechanism to support anyway.
> >> > > > > > https://tools.ietf.org/html/rfc7677 describes the protocol
> for
> >> > > > > > SCRAM-SHA-256.
> >> > > > > >
> >> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <ju...@confluent.io>
> wrote:
> >> > > > > >
> >> > > > > > > Parth,
> >> > > > > > >
> >> > > > > > > Thanks for the explanation. A couple of more questions.
> >> > > > > > >
> >> > > > > > > 110. What does getDelegationTokenAs mean?
> >> > > > > > >
> >> > > > > > > 111. What's the typical rate of getting and renewing
> delegation
> >> > > > tokens?
> >> > > > > > > That may have an impact on whether they should be directed
> to the
> >> > > > > > > controller.
> >> > > > > > >
> >> > > > > > > Jun
> >> > > > > > >
> >> > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt <
> >> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> >> > > > > > >
> >> > > > > > > > Hi Jun,
> >> > > > > > > >
> >> > > > > > > > Thanks for reviewing.
> >> > > > > > > >
> >> > > > > > > > * We could add a Cluster action to add acls on who can
> request
> >> > > > > > delegation
> >> > > > > > > > tokens. I don't see the use case for that yet but down
> the line
> >> > > > when
> >> > > > > we
> >> > > > > > > > start supporting getDelegationTokenAs it will be
> necessary.
> >> > > > > > > > * Yes we recommend tokens to be only used/distributed over
> >> > secure
> >> > > > > > > channels.
> >> > > > > > > > * Depending on what design we end up choosing
> Invalidation will
> >> > > be
> >> > > > > > > > responsibility of every broker or controller.
> >> > > > > > > > * I am not sure if I documented somewhere that
> invalidation
> >> > will
> >> > > > > > directly
> >> > > > > > > > go through zookeeper but that is not the intent.
> Invalidation
> >> > > will
> >> > > > > > either
> >> > > > > > > > be request based or due to expiration. No direct zookeeper
> >> > > > > interaction
> >> > > > > > > from
> >> > > > > > > > any client.
> >> > > > > > > > * "Broker also stores the DelegationToken without the
> hmac in
> >> > the
> >> > > > > > > > zookeeper." : Sorry about the confusion. The sole purpose
> of
> >> > > > > zookeeper
> >> > > > > > in
> >> > > > > > > > this design is as distribution channel for tokens between
> all
> >> > > > brokers
> >> > > > > > > and a
> >> > > > > > > > layer that ensures only tokens that were generated by
> making a
> >> > > > > request
> >> > > > > > > to a
> >> > > > > > > > broker will be accepted (more on this in second
> paragraph). The
> >> > > > token
> >> > > > > > > > consists of few elements (owner, renewer, uuid ,
> expiration,
> >> > > hmac)
> >> > > > ,
> >> > > > > > one
> >> > > > > > > of
> >> > > > > > > > which is the finally generated hmac but hmac it self is
> >> > derivable
> >> > > > if
> >> > > > > > you
> >> > > > > > > > have all the other elements of the token + secret key to
> >> > generate
> >> > > > > hmac.
> >> > > > > > > > Given zookeeper does not provide SSL support we do not
> want the
> >> > > > > entire
> >> > > > > > > > token to be wire transferred to zookeeper as that will be
> an
> >> > > > insecure
> >> > > > > > > wire
> >> > > > > > > > transfer. Instead we only store all the other elements of
> a
> >> > > > > delegation
> >> > > > > > > > tokens. Brokers can read these elements and because they
> also
> >> > > have
> >> > > > > > access
> >> > > > > > > > to secret key they will be able to generate hmac on their
> end.
> >> > > > > > > >
> >> > > > > > > > One of the alternative proposed is to avoid zookeeper
> >> > > altogether. A
> >> > > > > > > Client
> >> > > > > > > > will call broker with required information (owner, renwer,
> >> > > > > expiration)
> >> > > > > > > and
> >> > > > > > > > get back (signed hmac, uuid). Broker won't store this in
> >> > > zookeeper.
> >> > > > > > From
> >> > > > > > > > this point a client can contact any broker with all the
> >> > > delegation
> >> > > > > > token
> >> > > > > > > > info (owner, rewner, expiration, hmac, uuid) the borker
> will
> >> > > > > regenerate
> >> > > > > > > the
> >> > > > > > > > hmac and as long as it matches with hmac presented by
> client ,
> >> > > > broker
> >> > > > > > > will
> >> > > > > > > > allow the request to authenticate.  Only problem with this
> >> > > approach
> >> > > > > is
> >> > > > > > if
> >> > > > > > > > the secret key is compromised any client can now generate
> >> > random
> >> > > > > tokens
> >> > > > > > > and
> >> > > > > > > > they will still be able to authenticate as any user they
> like.
> >> > > with
> >> > > > > > > > zookeeper we guarantee that only tokens acquired via a
> broker
> >> > > > (using
> >> > > > > > some
> >> > > > > > > > auth scheme other than delegation token) will be
> accepted. We
> >> > > need
> >> > > > to
> >> > > > > > > > discuss which proposal makes more sense and we can go
> over it
> >> > in
> >> > > > > > > tomorrow's
> >> > > > > > > > meeting.
> >> > > > > > > >
> >> > > > > > > > Also, can you forward the invite to me?
> >> > > > > > > >
> >> > > > > > > > Thanks
> >> > > > > > > > Parth
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <
> jun@confluent.io>
> >> > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Thanks for the KIP. A few comments.
> >> > > > > > > > >
> >> > > > > > > > > 100. This potentially can be useful for Kafka Connect
> and
> >> > Kafka
> >> > > > > rest
> >> > > > > > > > proxy
> >> > > > > > > > > where a worker agent will need to run a task on behalf
> of a
> >> > > > client.
> >> > > > > > We
> >> > > > > > > > will
> >> > > > > > > > > likely need to change how those services use Kafka
> clients
> >> > > > > > > > > (producer/consumer). Instead of a shared client per
> worker,
> >> > we
> >> > > > will
> >> > > > > > > need
> >> > > > > > > > a
> >> > > > > > > > > client per user task since the authentication happens
> at the
> >> > > > > > connection
> >> > > > > > > > > level. For Kafka Connect, the renewer will be the
> workers.
> >> > So,
> >> > > we
> >> > > > > > > > probably
> >> > > > > > > > > need to allow multiple renewers. For Kafka rest proxy,
> the
> >> > > > renewer
> >> > > > > > can
> >> > > > > > > > > probably just be the creator of the token.
> >> > > > > > > > >
> >> > > > > > > > > 101. Do we need new acl on who can request delegation
> tokens?
> >> > > > > > > > >
> >> > > > > > > > > 102. Do we recommend people to send delegation tokens
> in an
> >> > > > > encrypted
> >> > > > > > > > > channel?
> >> > > > > > > > >
> >> > > > > > > > > 103. Who is responsible for expiring tokens, every
> broker?
> >> > > > > > > > >
> >> > > > > > > > > 104. For invalidating tokens, would it be better to do
> it in
> >> > a
> >> > > > > > request
> >> > > > > > > > > instead of going to ZK directly?
> >> > > > > > > > >
> >> > > > > > > > > 105. The terminology of client in the wiki sometimes
> refers
> >> > to
> >> > > > the
> >> > > > > > end
> >> > > > > > > > > client and some other times refers to the client using
> the
> >> > > > > delegation
> >> > > > > > > > > tokens. It would be useful to distinguish between the
> two.
> >> > > > > > > > >
> >> > > > > > > > > 106. Could you explain the sentence "Broker also stores
> the
> >> > > > > > > > DelegationToken
> >> > > > > > > > > without the hmac in the zookeeper." a bit more? I
> thought the
> >> > > > > > > delegation
> >> > > > > > > > > token is the hmac.
> >> > > > > > > > >
> >> > > > > > > > > Thanks,
> >> > > > > > > > >
> >> > > > > > > > > Jun
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <
> jun@confluent.io>
> >> > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi, Harsha,
> >> > > > > > > > > >
> >> > > > > > > > > > Just sent out a KIP meeting invite. We can discuss
> this in
> >> > > the
> >> > > > > > > meeting
> >> > > > > > > > > > tomorrow.
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > >
> >> > > > > > > > > > Jun
> >> > > > > > > > > >
> >> > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <
> kafka@harsha.io>
> >> > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > >> Hi All,
> >> > > > > > > > > >>            Can we have a KIP meeting around this.
> The KIP
> >> > is
> >> > > > up
> >> > > > > > for
> >> > > > > > > > > >>            sometime and if there are any questions
> lets
> >> > > > quickly
> >> > > > > > hash
> >> > > > > > > > out
> >> > > > > > > > > >>            details.
> >> > > > > > > > > >>
> >> > > > > > > > > >> Thanks,
> >> > > > > > > > > >> Harsha
> >> > > > > > > > > >>
> >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt
> wrote:
> >> > > > > > > > > >> > That is what the hadoop echo system uses so no good
> >> > reason
> >> > > > > > really.
> >> > > > > > > > We
> >> > > > > > > > > >> > could
> >> > > > > > > > > >> > change it to whatever is the newest recommended
> standard
> >> > > is.
> >> > > > > > > > > >> >
> >> > > > > > > > > >> > Thanks
> >> > > > > > > > > >> > Parth
> >> > > > > > > > > >> >
> >> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <
> >> > > > > ismael@juma.me.uk
> >> > > > > > >
> >> > > > > > > > > wrote:
> >> > > > > > > > > >> >
> >> > > > > > > > > >> > > Hi Parth,
> >> > > > > > > > > >> > >
> >> > > > > > > > > >> > > Thanks for the KIP. I only started reviewing
> this and
> >> > > may
> >> > > > > have
> >> > > > > > > > > >> additional
> >> > > > > > > > > >> > > questions later. The immediate question that
> came to
> >> > > mind
> >> > > > is
> >> > > > > > our
> >> > > > > > > > > >> choice of
> >> > > > > > > > > >> > > "DIGEST-MD5" even though it's marked as OBSOLETE
> in
> >> > the
> >> > > > IANA
> >> > > > > > > > > Registry
> >> > > > > > > > > >> of
> >> > > > > > > > > >> > > SASL mechanisms and the original RFC (2831) has
> been
> >> > > moved
> >> > > > > to
> >> > > > > > > > > Historic
> >> > > > > > > > > >> > > status:
> >> > > > > > > > > >> > >
> >> > > > > > > > > >> > > https://tools.ietf.org/html/rfc6331
> >> > > > > > > > > >> > >
> >> > > > > > > > >
> >> > > > > >
> >> > >
> http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> >> > > > > > > > > >> > >
> >> > > > > > > > > >> > > What is the reasoning behind that choice?
> >> > > > > > > > > >> > >
> >> > > > > > > > > >> > > Thanks,
> >> > > > > > > > > >> > > Ismael
> >> > > > > > > > > >> > >
> >> > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <
> >> > > > > > > gwen@confluent.io
> >> > > > > > > > >
> >> > > > > > > > > >> wrote:
> >> > > > > > > > > >> > >
> >> > > > > > > > > >> > > > Also comments inline :)
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > > * I want to emphasize that even though
> delegation
> >> > > > tokens
> >> > > > > > > are a
> >> > > > > > > > > >> Hadoop
> >> > > > > > > > > >> > > > > innovation, I feel very strongly about not
> adding
> >> > > > > > dependency
> >> > > > > > > > on
> >> > > > > > > > > >> Hadoop
> >> > > > > > > > > >> > > > > when implementing delegation tokens for
> Kafka. The
> >> > > KIP
> >> > > > > > > doesn't
> >> > > > > > > > > >> imply
> >> > > > > > > > > >> > > > > such dependency, but if you can clarify...
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > > *No hadoop dependency.*
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > Yay! Just add this to the KIP so no one will
> read
> >> > the
> >> > > > KIP
> >> > > > > > and
> >> > > > > > > > > panic
> >> > > > > > > > > >> > > > three weeks before the next release...
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > > * Can we get delegation token at any time
> after
> >> > > > > > > > authenticating?
> >> > > > > > > > > >> only
> >> > > > > > > > > >> > > > > immediately after?
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > > *As long as you are authenticated you can get
> >> > > > delegation
> >> > > > > > > > tokens.
> >> > > > > > > > > >> We
> >> > > > > > > > > >> > > need
> >> > > > > > > > > >> > > > to
> >> > > > > > > > > >> > > > > discuss if a client authenticated using
> delegation
> >> > > > > token,
> >> > > > > > > can
> >> > > > > > > > > also
> >> > > > > > > > > >> > > > acquire
> >> > > > > > > > > >> > > > > delegation token again or not. Also there is
> the
> >> > > > > question
> >> > > > > > of
> >> > > > > > > > do
> >> > > > > > > > > we
> >> > > > > > > > > >> > > allow
> >> > > > > > > > > >> > > > > anyone to acquire delegation token or we want
> >> > > specific
> >> > > > > > ACLs
> >> > > > > > > (I
> >> > > > > > > > > >> think
> >> > > > > > > > > >> > > its
> >> > > > > > > > > >> > > > an
> >> > > > > > > > > >> > > > > overkill.)*
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > I agree that ACLs is an overkill.
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > I think we are debating two options: Either
> require
> >> > > > > Kerberos
> >> > > > > > > > auth
> >> > > > > > > > > >> for
> >> > > > > > > > > >> > > > renewal or require non-owners to renew.
> >> > > > > > > > > >> > > > I *think* the latter is simpler (it basically
> >> > require
> >> > > a
> >> > > > > "job
> >> > > > > > > > > master"
> >> > > > > > > > > >> > > > to take responsibility for the renewal, it
> will have
> >> > > its
> >> > > > > own
> >> > > > > > > > > >> identity
> >> > > > > > > > > >> > > > anyway and I think this is the correct design
> >> > pattern
> >> > > > > > anyway.
> >> > > > > > > > For
> >> > > > > > > > > >> > > > storm, I'd expect Nimbus to coordinate
> renewals?),
> >> > but
> >> > > > it
> >> > > > > is
> >> > > > > > > > hard
> >> > > > > > > > > to
> >> > > > > > > > > >> > > > debate simplicity without looking at the code
> >> > changes
> >> > > > > > > required.
> >> > > > > > > > If
> >> > > > > > > > > >> you
> >> > > > > > > > > >> > > > have a draft of how the "require Kerberos"
> will look
> >> > > in
> >> > > > > > Kafka
> >> > > > > > > > > code,
> >> > > > > > > > > >> > > > I'll be happy to take a look.
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > > * My understanding is that tokens will
> propagate
> >> > via
> >> > > > ZK
> >> > > > > > but
> >> > > > > > > > > >> without
> >> > > > > > > > > >> > > > > additional changes to UpdateMetadata
> protocol,
> >> > > > correct?
> >> > > > > > > > Clients
> >> > > > > > > > > >> > > > > currently don't retry on SASL auth failure
> (IIRC),
> >> > > but
> >> > > > > > since
> >> > > > > > > > the
> >> > > > > > > > > >> > > > > tokens propagate between brokers asynch, we
> will
> >> > > need
> >> > > > to
> >> > > > > > > > retry a
> >> > > > > > > > > >> bit
> >> > > > > > > > > >> > > > > to avoid clients failing auth due to timing
> >> > issues.
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > > *I am considering 2 alternatives right now.
> The
> >> > > > current
> >> > > > > > > > > documented
> >> > > > > > > > > >> > > > approach
> >> > > > > > > > > >> > > > > is zookeeper based and it does not require
> any
> >> > > changes
> >> > > > > to
> >> > > > > > > > > >> > > UpdateMetadata
> >> > > > > > > > > >> > > > > protocol. An alternative approach can remove
> >> > > zookeeper
> >> > > > > > > > > dependency
> >> > > > > > > > > >> as
> >> > > > > > > > > >> > > well
> >> > > > > > > > > >> > > > > but we can discuss that in KIP discussion
> call.*
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > Oooh! Sounds interesting. Do you want to ping
> Jun to
> >> > > > > > arrange a
> >> > > > > > > > > call?
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > > * I liked Ashish's suggestion of having just
> the
> >> > > > > > controller
> >> > > > > > > > > issue
> >> > > > > > > > > >> the
> >> > > > > > > > > >> > > > > delegation tokens, to avoid syncing a shared
> >> > secret.
> >> > > > Not
> >> > > > > > > sure
> >> > > > > > > > if
> >> > > > > > > > > >> we
> >> > > > > > > > > >> > > > > want to continue the discussion here or on
> the
> >> > > wiki. I
> >> > > > > > think
> >> > > > > > > > > that
> >> > > > > > > > > >> we
> >> > > > > > > > > >> > > > > can decouple the problem of "token
> distribution"
> >> > > from
> >> > > > > > > "shared
> >> > > > > > > > > >> secret
> >> > > > > > > > > >> > > > > distribution" and use the controller as the
> only
> >> > > token
> >> > > > > > > > generator
> >> > > > > > > > > >> to
> >> > > > > > > > > >> > > > > solve the second issue, while still using ZK
> async
> >> > > to
> >> > > > > > > > distribute
> >> > > > > > > > > >> > > > > tokens.
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > > *As mentioned in the previous Email I am
> fine with
> >> > > > that
> >> > > > > > > > approach
> >> > > > > > > > > >> as
> >> > > > > > > > > >> > > long
> >> > > > > > > > > >> > > > as
> >> > > > > > > > > >> > > > > we agree that the extra complexity of
> >> > > adding/updating
> >> > > > > APIS
> >> > > > > > > > adds
> >> > > > > > > > > >> enough
> >> > > > > > > > > >> > > > > value. The advantage with the controller
> approach
> >> > is
> >> > > > > > secret
> >> > > > > > > > > >> rotation
> >> > > > > > > > > >> > > can
> >> > > > > > > > > >> > > > be
> >> > > > > > > > > >> > > > > automated,frequent and would not require
> >> > > deployment. *
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > Can you detail the extra complexity (or point
> me to
> >> > > the
> >> > > > > > email
> >> > > > > > > I
> >> > > > > > > > > >> > > > missed?) - which Apis are required?
> >> > > > > > > > > >> > > > As far as I can tell, clients can already find
> the
> >> > > > > > controller
> >> > > > > > > > from
> >> > > > > > > > > >> > > > metadata. I'm a bit more concerned about
> controller
> >> > > > load.
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > > * While I like the idea of forcing kerberos
> auth
> >> > for
> >> > > > > > > renwal, I
> >> > > > > > > > > >> think
> >> > > > > > > > > >> > > > > it mixes the transport layer the the request
> >> > content
> >> > > > in
> >> > > > > a
> >> > > > > > > > pretty
> >> > > > > > > > > >> ugly
> >> > > > > > > > > >> > > > > way. Perhaps limiting renewer to non-owner is
> >> > > better.
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > > *I feel this is a necessary evil. While this
> will
> >> > > make
> >> > > > > the
> >> > > > > > > > kafka
> >> > > > > > > > > >> code
> >> > > > > > > > > >> > > > > pretty straight forward , forcing  renewer to
> >> > > > non-owner
> >> > > > > > > pushes
> >> > > > > > > > > >> the code
> >> > > > > > > > > >> > > > > ugliness to client and makes it even harder
> to
> >> > > > > > integrate.  *
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > As mentioned before, I don't think the
> "renewal by
> >> > > > other"
> >> > > > > > > > approach
> >> > > > > > > > > >> is
> >> > > > > > > > > >> > > > that ugly for the clients we expect to use
> >> > delegation
> >> > > > > tokens
> >> > > > > > > > since
> >> > > > > > > > > >> > > > they will have an app-master of some sort who
> >> > > requested
> >> > > > > the
> >> > > > > > > > token
> >> > > > > > > > > to
> >> > > > > > > > > >> > > > begin with.
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > > The response for my question on how multiple
> >> > > > identities
> >> > > > > > will
> >> > > > > > > > be
> >> > > > > > > > > >> > > > > handled wasn't super clear to me - AFAIK, we
> don't
> >> > > > > > > > authenticate
> >> > > > > > > > > >> each
> >> > > > > > > > > >> > > > > request, we authenticate connections.
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > > *We authenticate connections, and only when
> they
> >> > are
> >> > > > > being
> >> > > > > > > > > >> established.
> >> > > > > > > > > >> > > > Let
> >> > > > > > > > > >> > > > > me try to phrase this as a question, in
> absence of
> >> > > > > > > delegation
> >> > > > > > > > > >> tokens if
> >> > > > > > > > > >> > > > we
> >> > > > > > > > > >> > > > > had to support the use case using user TGT's
> how
> >> > > would
> >> > > > > we
> >> > > > > > do
> >> > > > > > > > it?
> >> > > > > > > > > >> My
> >> > > > > > > > > >> > > point
> >> > > > > > > > > >> > > > > was it would be no different with delegation
> >> > tokens.
> >> > > > The
> >> > > > > > use
> >> > > > > > > > > case
> >> > > > > > > > > >> you
> >> > > > > > > > > >> > > are
> >> > > > > > > > > >> > > > > describing seems more like impersonation.*
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > Yeah, I thought that one of the things that
> >> > delegation
> >> > > > > > tokens
> >> > > > > > > > > >> handled.
> >> > > > > > > > > >> > > > Maybe I got it wrong :)
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > Thanks for the detailed answers.
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > Gwen
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > > > > Thanks
> >> > > > > > > > > >> > > > > Parth
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen
> Shapira <
> >> > > > > > > > > gwen@confluent.io
> >> > > > > > > > > >> >
> >> > > > > > > > > >> > > > wrote:
> >> > > > > > > > > >> > > > >
> >> > > > > > > > > >> > > > >> Hi Parth and Harsha,
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> Few more comments:
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * The API / RequestResponse section doesn't
> seem
> >> > to
> >> > > > > have
> >> > > > > > > good
> >> > > > > > > > > >> > > > >> description of the changes to the Kafka
> Protocol.
> >> > > > > Sounds
> >> > > > > > > like
> >> > > > > > > > > >> you are
> >> > > > > > > > > >> > > > >> proposing new DelegationTokenRequest and
> >> > > > > > RenewTokenRequest
> >> > > > > > > > (and
> >> > > > > > > > > >> > > > >> matching responses), without detailing the
> >> > contents
> >> > > > of
> >> > > > > > the
> >> > > > > > > > > >> requests
> >> > > > > > > > > >> > > > >> and responses? Or rather, you show the class
> >> > > > interface,
> >> > > > > > but
> >> > > > > > > > not
> >> > > > > > > > > >> the
> >> > > > > > > > > >> > > > >> underlying protocol. This could be seen as
> an
> >> > > > > > > implementation
> >> > > > > > > > > >> detail,
> >> > > > > > > > > >> > > > >> but since the binary protocol is what we
> provide
> >> > to
> >> > > > > > > non-Java
> >> > > > > > > > > >> clients,
> >> > > > > > > > > >> > > > >> we need to show the changes there.
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * getDelegationToken sounds like
> >> > > > > > > > delegationTokenRequestHandler?
> >> > > > > > > > > >> Is it
> >> > > > > > > > > >> > > > >> planned to be part of KafkaApi? or Client?
> Its
> >> > > > > unclear...
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * I want to emphasize that even though
> delegation
> >> > > > > tokens
> >> > > > > > > are
> >> > > > > > > > a
> >> > > > > > > > > >> Hadoop
> >> > > > > > > > > >> > > > >> innovation, I feel very strongly about not
> adding
> >> > > > > > > dependency
> >> > > > > > > > on
> >> > > > > > > > > >> Hadoop
> >> > > > > > > > > >> > > > >> when implementing delegation tokens for
> Kafka.
> >> > The
> >> > > > KIP
> >> > > > > > > > doesn't
> >> > > > > > > > > >> imply
> >> > > > > > > > > >> > > > >> such dependency, but if you can clarify...
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * Can we get delegation token at any time
> after
> >> > > > > > > > authenticating?
> >> > > > > > > > > >> only
> >> > > > > > > > > >> > > > >> immediately after?
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * My understanding is that tokens will
> propagate
> >> > > via
> >> > > > ZK
> >> > > > > > but
> >> > > > > > > > > >> without
> >> > > > > > > > > >> > > > >> additional changes to UpdateMetadata
> protocol,
> >> > > > correct?
> >> > > > > > > > Clients
> >> > > > > > > > > >> > > > >> currently don't retry on SASL auth failure
> >> > (IIRC),
> >> > > > but
> >> > > > > > > since
> >> > > > > > > > > the
> >> > > > > > > > > >> > > > >> tokens propagate between brokers asynch, we
> will
> >> > > need
> >> > > > > to
> >> > > > > > > > retry
> >> > > > > > > > > a
> >> > > > > > > > > >> bit
> >> > > > > > > > > >> > > > >> to avoid clients failing auth due to timing
> >> > issues.
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * Strongly agreeing on clients not touching
> ZK
> >> > > > directly
> >> > > > > > :)
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * I liked Ashish's suggestion of having
> just the
> >> > > > > > controller
> >> > > > > > > > > >> issue the
> >> > > > > > > > > >> > > > >> delegation tokens, to avoid syncing a shared
> >> > > secret.
> >> > > > > Not
> >> > > > > > > sure
> >> > > > > > > > > if
> >> > > > > > > > > >> we
> >> > > > > > > > > >> > > > >> want to continue the discussion here or on
> the
> >> > > wiki.
> >> > > > I
> >> > > > > > > think
> >> > > > > > > > > >> that we
> >> > > > > > > > > >> > > > >> can decouple the problem of "token
> distribution"
> >> > > from
> >> > > > > > > "shared
> >> > > > > > > > > >> secret
> >> > > > > > > > > >> > > > >> distribution" and use the controller as the
> only
> >> > > > token
> >> > > > > > > > > generator
> >> > > > > > > > > >> to
> >> > > > > > > > > >> > > > >> solve the second issue, while still using ZK
> >> > async
> >> > > to
> >> > > > > > > > > distribute
> >> > > > > > > > > >> > > > >> tokens.
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * I am also uncomfortable with infinite
> lifetime
> >> > of
> >> > > > > > tokens
> >> > > > > > > > (and
> >> > > > > > > > > >> hoped
> >> > > > > > > > > >> > > > >> to hear from others in the community) - but
> >> > having
> >> > > > the
> >> > > > > > > option
> >> > > > > > > > > and
> >> > > > > > > > > >> > > > >> documenting it as less secure, allows users
> to
> >> > > > > configure
> >> > > > > > > > their
> >> > > > > > > > > >> system
> >> > > > > > > > > >> > > > >> as the wish.
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * While I like the idea of forcing kerberos
> auth
> >> > > for
> >> > > > > > > renwal,
> >> > > > > > > > I
> >> > > > > > > > > >> think
> >> > > > > > > > > >> > > > >> it mixes the transport layer the the request
> >> > > content
> >> > > > > in a
> >> > > > > > > > > pretty
> >> > > > > > > > > >> ugly
> >> > > > > > > > > >> > > > >> way. Perhaps limiting renewer to non-owner
> is
> >> > > better.
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> Things I'd still like to see:
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * More detailed explanation on what we plan
> to do
> >> > > for
> >> > > > > the
> >> > > > > > > > java
> >> > > > > > > > > >> clients
> >> > > > > > > > > >> > > > >> specifically - new configuration? new APIs?
> >> > > > > > > > > >> > > > >> The response for my question on how multiple
> >> > > > identities
> >> > > > > > > will
> >> > > > > > > > be
> >> > > > > > > > > >> > > > >> handled wasn't super clear to me - AFAIK, we
> >> > don't
> >> > > > > > > > authenticate
> >> > > > > > > > > >> each
> >> > > > > > > > > >> > > > >> request, we authenticate connections.
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> * Alternatives: Delegation tokens are only
> used
> >> > in
> >> > > > the
> >> > > > > > > Hadoop
> >> > > > > > > > > >> > > > >> ecosystem. I'm wondering if there are
> >> > alternatives
> >> > > in
> >> > > > > > other
> >> > > > > > > > > >> ecosystems
> >> > > > > > > > > >> > > > >> (Mesos? Tachyon? Cassandra?) and whether
> there
> >> > are
> >> > > > some
> >> > > > > > > > > >> advantages
> >> > > > > > > > > >> > > > >> there.
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> Gwen
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <
> >> > > > > kafka@harsha.io
> >> > > > > > >
> >> > > > > > > > > wrote:
> >> > > > > > > > > >> > > > >> > Hi Gwen,
> >> > > > > > > > > >> > > > >> >            Can you look at Parth's last
> reply.
> >> > > Does
> >> > > > > it
> >> > > > > > > > answer
> >> > > > > > > > > >> your
> >> > > > > > > > > >> > > > >> >            concerns.
> >> > > > > > > > > >> > > > >> >
> >> > > > > > > > > >> > > > >> > Thanks,
> >> > > > > > > > > >> > > > >> > Harsha
> >> > > > > > > > > >> > > > >> >
> >> > > > > > > > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth
> >> > > brahmbhatt
> >> > > > > > wrote:
> >> > > > > > > > > >> > > > >> >> Thanks for reviewing Gwen. The wiki
> already
> >> > has
> >> > > > > > details
> >> > > > > > > on
> >> > > > > > > > > >> token
> >> > > > > > > > > >> > > > >> >> expiration
> >> > > > > > > > > >> > > > >> >> under token acquisition process
> >> > > > > > > > > >> > > > >> >> <
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > >
> >> > > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> >> > > > > > > > > >> > > > >> >.
> >> > > > > > > > > >> > > > >> >> Current proposal is that tokens will
> expire
> >> > > based
> >> > > > > on a
> >> > > > > > > > > server
> >> > > > > > > > > >> side
> >> > > > > > > > > >> > > > >> >> configuration (default 24 hours) unless
> >> > renewed.
> >> > > > > > Renewal
> >> > > > > > > > is
> >> > > > > > > > > >> only
> >> > > > > > > > > >> > > > allowed
> >> > > > > > > > > >> > > > >> >> until the max life time of token.
> >> > Alternatively
> >> > > we
> >> > > > > > could
> >> > > > > > > > > also
> >> > > > > > > > > >> make
> >> > > > > > > > > >> > > > that
> >> > > > > > > > > >> > > > >> >> an
> >> > > > > > > > > >> > > > >> >> optional param and the server side
> default can
> >> > > > serve
> >> > > > > > as
> >> > > > > > > > the
> >> > > > > > > > > >> upper
> >> > > > > > > > > >> > > > bound.
> >> > > > > > > > > >> > > > >> >>
> >> > > > > > > > > >> > > > >> >> To your second point it will be done
> exactly
> >> > the
> >> > > > > same
> >> > > > > > > way
> >> > > > > > > > we
> >> > > > > > > > > >> would
> >> > > > > > > > > >> > > > >> >> support
> >> > > > > > > > > >> > > > >> >> multiple keytabs. The calling client
> will have
> >> > > to
> >> > > > > put
> >> > > > > > > the
> >> > > > > > > > > >> tokens it
> >> > > > > > > > > >> > > > >> wants
> >> > > > > > > > > >> > > > >> >> to use in the subject instance and call
> >> > > > > > produce/consume
> >> > > > > > > > > inside
> >> > > > > > > > > >> > > > >> >> subject.doas. Each caller will have to
> keep
> >> > > track
> >> > > > of
> >> > > > > > its
> >> > > > > > > > own
> >> > > > > > > > > >> > > > subject. I
> >> > > > > > > > > >> > > > >> >> will have to look at the code to see if
> we
> >> > > support
> >> > > > > > this
> >> > > > > > > > > >> feature
> >> > > > > > > > > >> > > right
> >> > > > > > > > > >> > > > >> now
> >> > > > > > > > > >> > > > >> >> but my understanding is delegation token
> >> > > shouldn't
> >> > > > > > need
> >> > > > > > > > any
> >> > > > > > > > > >> special
> >> > > > > > > > > >> > > > >> >> treatment as its just another type of
> >> > Credential
> >> > > > in
> >> > > > > > the
> >> > > > > > > > > >> subject.
> >> > > > > > > > > >> > > > >> >>
> >> > > > > > > > > >> > > > >> >> I would also like to know what is your
> opinion
> >> > > > about
> >> > > > > > > > > infinite
> >> > > > > > > > > >> > > renewal
> >> > > > > > > > > >> > > > >> (my
> >> > > > > > > > > >> > > > >> >> recommendation is to not support this),
> tokens
> >> > > > > > renewing
> >> > > > > > > > them
> >> > > > > > > > > >> > > self(my
> >> > > > > > > > > >> > > > >> >> recommendation is to not support this)
> and
> >> > most
> >> > > > > > > > importantly
> >> > > > > > > > > >> your
> >> > > > > > > > > >> > > > choice
> >> > > > > > > > > >> > > > >> >> between the alternatives listed on this
> thread
> >> > > > > > > > > >> > > > >> >> <
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > >
> >> > > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> >> > > > > > > > > >> > > > >> >
> >> > > > > > > > > >> > > > >> >> ( I am leaning towards the alternative-2
> minus
> >> > > > > > > controller
> >> > > > > > > > > >> > > > distributing
> >> > > > > > > > > >> > > > >> >> secret). Thanks again for reviewing.
> >> > > > > > > > > >> > > > >> >>
> >> > > > > > > > > >> > > > >> >> Thanks
> >> > > > > > > > > >> > > > >> >> Parth
> >> > > > > > > > > >> > > > >> >>
> >> > > > > > > > > >> > > > >> >>
> >> > > > > > > > > >> > > > >> >>
> >> > > > > > > > > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen
> Shapira <
> >> > > > > > > > > >> gwen@confluent.io>
> >> > > > > > > > > >> > > > wrote:
> >> > > > > > > > > >> > > > >> >>
> >> > > > > > > > > >> > > > >> >> > Harsha,
> >> > > > > > > > > >> > > > >> >> >
> >> > > > > > > > > >> > > > >> >> > I was thinking of the Rest Proxy. I
> didn't
> >> > see
> >> > > > > your
> >> > > > > > > > design
> >> > > > > > > > > >> yet,
> >> > > > > > > > > >> > > > but in
> >> > > > > > > > > >> > > > >> >> > our proxy, we have a set of producers,
> which
> >> > > > will
> >> > > > > > > serve
> >> > > > > > > > > >> multiple
> >> > > > > > > > > >> > > > users
> >> > > > > > > > > >> > > > >> >> > going through the proxy. Since these
> users
> >> > > will
> >> > > > > have
> >> > > > > > > > > >> different
> >> > > > > > > > > >> > > > >> >> > privileges, they'll need to
> authenticate
> >> > > > > separately,
> >> > > > > > > and
> >> > > > > > > > > >> can't
> >> > > > > > > > > >> > > > share a
> >> > > > > > > > > >> > > > >> >> > token.
> >> > > > > > > > > >> > > > >> >> >
> >> > > > > > > > > >> > > > >> >> > Am I missing anything?
> >> > > > > > > > > >> > > > >> >> >
> >> > > > > > > > > >> > > > >> >> > Gwen
> >> > > > > > > > > >> > > > >> >> >
> >> > > > > > > > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha
> <
> >> > > > > > > kafka@harsha.io
> >> > > > > > > > >
> >> > > > > > > > > >> wrote:
> >> > > > > > > > > >> > > > >> >> > > Gwen,
> >> > > > > > > > > >> > > > >> >> > >            On your second point. Can
> you
> >> > > > > describe
> >> > > > > > a
> >> > > > > > > > > >> usecase
> >> > > > > > > > > >> > > where
> >> > > > > > > > > >> > > > >> >> > >            mutliple clients ended up
> >> > > sharing a
> >> > > > > > > > producer
> >> > > > > > > > > >> and
> >> > > > > > > > > >> > > even
> >> > > > > > > > > >> > > > if
> >> > > > > > > > > >> > > > >> they
> >> > > > > > > > > >> > > > >> >> > >            do why can't they not use
> >> > single
> >> > > > > token
> >> > > > > > > that
> >> > > > > > > > > >> producer
> >> > > > > > > > > >> > > > >> >> > >            captures. Why would we
> need
> >> > > > multiple
> >> > > > > > > > clients
> >> > > > > > > > > >> with
> >> > > > > > > > > >> > > > >> different
> >> > > > > > > > > >> > > > >> >> > >            tokens sharing a single
> >> > instance
> >> > > of
> >> > > > > > > > producer.
> >> > > > > > > > > >> Also
> >> > > > > > > > > >> > > in
> >> > > > > > > > > >> > > > >> this
> >> > > > > > > > > >> > > > >> >> > >            case other clients have
> access
> >> > > all
> >> > > > > the
> >> > > > > > > > tokens
> >> > > > > > > > > >> no?
> >> > > > > > > > > >> > > > >> >> > >
> >> > > > > > > > > >> > > > >> >> > > Thanks,
> >> > > > > > > > > >> > > > >> >> > > Harsha
> >> > > > > > > > > >> > > > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >
> >> > > > > > > > > >> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM,
> Gwen
> >> > > Shapira
> >> > > > > > > wrote:
> >> > > > > > > > > >> > > > >> >> > >> Sorry for the delay:
> >> > > > > > > > > >> > > > >> >> > >>
> >> > > > > > > > > >> > > > >> >> > >> Two questions that we didn't see in
> the
> >> > > wiki:
> >> > > > > > > > > >> > > > >> >> > >> 1. Is there an expiration for
> delegation
> >> > > > > tokens?
> >> > > > > > > > > >> Renewal? How
> >> > > > > > > > > >> > > > do we
> >> > > > > > > > > >> > > > >> >> > >> revoke them?
> >> > > > > > > > > >> > > > >> >> > >> 2. If we want to use delegation
> tokens
> >> > for
> >> > > > > > "do-as"
> >> > > > > > > > > (say,
> >> > > > > > > > > >> > > submit
> >> > > > > > > > > >> > > > >> Storm
> >> > > > > > > > > >> > > > >> >> > >> job as my user), we will need a
> producer
> >> > > for
> >> > > > > > every
> >> > > > > > > > job
> >> > > > > > > > > >> (we
> >> > > > > > > > > >> > > can't
> >> > > > > > > > > >> > > > >> share
> >> > > > > > > > > >> > > > >> >> > >> them between multiple jobs running
> on
> >> > same
> >> > > > > node),
> >> > > > > > > > since
> >> > > > > > > > > >> we
> >> > > > > > > > > >> > > only
> >> > > > > > > > > >> > > > >> >> > >> authenticate when connecting. Is
> there a
> >> > > plan
> >> > > > > to
> >> > > > > > > > change
> >> > > > > > > > > >> this
> >> > > > > > > > > >> > > for
> >> > > > > > > > > >> > > > >> >> > >> delegation tokens, in order to allow
> >> > > multiple
> >> > > > > > users
> >> > > > > > > > > with
> >> > > > > > > > > >> > > > different
> >> > > > > > > > > >> > > > >> >> > >> tokens to share a client?
> >> > > > > > > > > >> > > > >> >> > >>
> >> > > > > > > > > >> > > > >> >> > >> Gwen
> >> > > > > > > > > >> > > > >> >> > >>
> >> > > > > > > > > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM,
> parth
> >> > > > > brahmbhatt
> >> > > > > > > > > >> > > > >> >> > >> <br...@gmail.com> wrote:
> >> > > > > > > > > >> > > > >> >> > >> > Bumping this up one more time, can
> >> > other
> >> > > > > > > committers
> >> > > > > > > > > >> review?
> >> > > > > > > > > >> > > > >> >> > >> >
> >> > > > > > > > > >> > > > >> >> > >> > Thanks
> >> > > > > > > > > >> > > > >> >> > >> > Parth
> >> > > > > > > > > >> > > > >> >> > >> >
> >> > > > > > > > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM,
> >> > Harsha <
> >> > > > > > > > > >> kafka@harsha.io>
> >> > > > > > > > > >> > > > wrote:
> >> > > > > > > > > >> > > > >> >> > >> >
> >> > > > > > > > > >> > > > >> >> > >> >> Parth,
> >> > > > > > > > > >> > > > >> >> > >> >>           Overall current design
> looks
> >> > > > good
> >> > > > > to
> >> > > > > > > > me. I
> >> > > > > > > > > >> am +1
> >> > > > > > > > > >> > > on
> >> > > > > > > > > >> > > > >> the
> >> > > > > > > > > >> > > > >> >> > KIP.
> >> > > > > > > > > >> > > > >> >> > >> >>
> >> > > > > > > > > >> > > > >> >> > >> >> Gwen , Jun can you review this as
> >> > well.
> >> > > > > > > > > >> > > > >> >> > >> >>
> >> > > > > > > > > >> > > > >> >> > >> >> -Harsha
> >> > > > > > > > > >> > > > >> >> > >> >>
> >> > > > > > > > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57
> AM,
> >> > parth
> >> > > > > > > > brahmbhatt
> >> > > > > > > > > >> wrote:
> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks for review Jitendra.
> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > I don't like the idea of
> infinite
> >> > > > lifetime
> >> > > > > > > but I
> >> > > > > > > > > >> see the
> >> > > > > > > > > >> > > > >> Streaming
> >> > > > > > > > > >> > > > >> >> > use
> >> > > > > > > > > >> > > > >> >> > >> >> > case. Even for Streaming use
> case I
> >> > > was
> >> > > > > > hoping
> >> > > > > > > > > >> there will
> >> > > > > > > > > >> > > > be
> >> > > > > > > > > >> > > > >> some
> >> > > > > > > > > >> > > > >> >> > notion
> >> > > > > > > > > >> > > > >> >> > >> >> > of
> >> > > > > > > > > >> > > > >> >> > >> >> > master/driver that can get new
> >> > > > delegation
> >> > > > > > > tokens
> >> > > > > > > > > at
> >> > > > > > > > > >> fixed
> >> > > > > > > > > >> > > > >> interval
> >> > > > > > > > > >> > > > >> >> > and
> >> > > > > > > > > >> > > > >> >> > >> >> > distribute to workers. If that
> is
> >> > not
> >> > > > the
> >> > > > > > case
> >> > > > > > > > for
> >> > > > > > > > > >> we can
> >> > > > > > > > > >> > > > >> discuss
> >> > > > > > > > > >> > > > >> >> > >> >> > delegation tokens renewing
> them self
> >> > > and
> >> > > > > the
> >> > > > > > > > > >> security
> >> > > > > > > > > >> > > > >> implications
> >> > > > > > > > > >> > > > >> >> > of the
> >> > > > > > > > > >> > > > >> >> > >> >> > same.
> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > I did not want clients to fetch
> >> > tokens
> >> > > > > from
> >> > > > > > > > > >> zookeeper,
> >> > > > > > > > > >> > > > >> overall I
> >> > > > > > > > > >> > > > >> >> > think
> >> > > > > > > > > >> > > > >> >> > >> >> > its
> >> > > > > > > > > >> > > > >> >> > >> >> > better if clients don't rely
> on our
> >> > > > > metadata
> >> > > > > > > > store
> >> > > > > > > > > >> and I
> >> > > > > > > > > >> > > > >> think we
> >> > > > > > > > > >> > > > >> >> > are
> >> > > > > > > > > >> > > > >> >> > >> >> > moving in that direction with
> all
> >> > the
> >> > > > > KIP-4
> >> > > > > > > > > >> improvements.
> >> > > > > > > > > >> > > > I
> >> > > > > > > > > >> > > > >> chose
> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as in this case the
> client
> >> > > > will
> >> > > > > > > still
> >> > > > > > > > > >> just talk
> >> > > > > > > > > >> > > > to
> >> > > > > > > > > >> > > > >> >> > broker , its
> >> > > > > > > > > >> > > > >> >> > >> >> > the brokers that will use
> zookeeper
> >> > > > which
> >> > > > > we
> >> > > > > > > > > >> already do
> >> > > > > > > > > >> > > > for a
> >> > > > > > > > > >> > > > >> lot
> >> > > > > > > > > >> > > > >> >> > of
> >> > > > > > > > > >> > > > >> >> > >> >> > other
> >> > > > > > > > > >> > > > >> >> > >> >> > usecases + ease of development
> + and
> >> > > the
> >> > > > > > > ability
> >> > > > > > > > > so
> >> > > > > > > > > >> > > tokens
> >> > > > > > > > > >> > > > >> will
> >> > > > > > > > > >> > > > >> >> > survive
> >> > > > > > > > > >> > > > >> >> > >> >> > even a rolling restart/cluster
> >> > > failure.
> >> > > > > if a
> >> > > > > > > > > >> majority
> >> > > > > > > > > >> > > > agrees
> >> > > > > > > > > >> > > > >> the
> >> > > > > > > > > >> > > > >> >> > added
> >> > > > > > > > > >> > > > >> >> > >> >> > complexity to have controller
> >> > > forwarding
> >> > > > > > keys
> >> > > > > > > to
> >> > > > > > > > > all
> >> > > > > > > > > >> > > > broker is
> >> > > > > > > > > >> > > > >> >> > justified
> >> > > > > > > > > >> > > > >> >> > >> >> > as
> >> > > > > > > > > >> > > > >> >> > >> >> > it provides tighter security ,
> I am
> >> > > fine
> >> > > > > > with
> >> > > > > > > > that
> >> > > > > > > > > >> option
> >> > > > > > > > > >> > > > too.
> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > Given zookeeper does not
> support SSL
> >> > > we
> >> > > > > can
> >> > > > > > > not
> >> > > > > > > > > >> store
> >> > > > > > > > > >> > > > master
> >> > > > > > > > > >> > > > >> keys
> >> > > > > > > > > >> > > > >> >> > in
> >> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as master keys will
> be
> >> > > exposed
> >> > > > > on
> >> > > > > > > > wire.
> >> > > > > > > > > To
> >> > > > > > > > > >> > > > support
> >> > > > > > > > > >> > > > >> >> > rotation
> >> > > > > > > > > >> > > > >> >> > >> >> > without affecting current
> clients is
> >> > > > > > > something I
> >> > > > > > > > > >> need to
> >> > > > > > > > > >> > > > put
> >> > > > > > > > > >> > > > >> more
> >> > > > > > > > > >> > > > >> >> > thought
> >> > > > > > > > > >> > > > >> >> > >> >> > in. My current proposal
> assumes the
> >> > > > > rotation
> >> > > > > > > > will
> >> > > > > > > > > >> > > > invalidate
> >> > > > > > > > > >> > > > >> all
> >> > > > > > > > > >> > > > >> >> > current
> >> > > > > > > > > >> > > > >> >> > >> >> > tokens.
> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > I request committers to also
> review
> >> > > and
> >> > > > > post
> >> > > > > > > > their
> >> > > > > > > > > >> > > comments
> >> > > > > > > > > >> > > > >> so we
> >> > > > > > > > > >> > > > >> >> > can
> >> > > > > > > > > >> > > > >> >> > >> >> > make
> >> > > > > > > > > >> > > > >> >> > >> >> > progress on this KIP.
> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > Thanks
> >> > > > > > > > > >> > > > >> >> > >> >> > Parth
> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39
> AM,
> >> > > Ashish
> >> > > > > > Singh
> >> > > > > > > <
> >> > > > > > > > > >> > > > >> asingh@cloudera.com
> >> > > > > > > > > >> > > > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > wrote:
> >> > > > > > > > > >> > > > >> >> > >> >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at
> 11:26 AM,
> >> > > > > Harsha <
> >> > > > > > > > > >> > > > kafka@harsha.io>
> >> > > > > > > > > >> > > > >> >> > wrote:
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > Unifying the two discussion
> >> > > threads
> >> > > > on
> >> > > > > > > this
> >> > > > > > > > > KIP.
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > Here is the response from
> >> > Jitendra
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > "The need for a large
> number of
> >> > > > > clients
> >> > > > > > > that
> >> > > > > > > > > are
> >> > > > > > > > > >> > > > running
> >> > > > > > > > > >> > > > >> all
> >> > > > > > > > > >> > > > >> >> > over the
> >> > > > > > > > > >> > > > >> >> > >> >> > > > cluster that authenticate
> with
> >> > > Kafka
> >> > > > > > > > brokers,
> >> > > > > > > > > >> is very
> >> > > > > > > > > >> > > > >> similar
> >> > > > > > > > > >> > > > >> >> > to the
> >> > > > > > > > > >> > > > >> >> > >> >> > > > Hadoop use case of large
> number
> >> > of
> >> > > > > tasks
> >> > > > > > > > > running
> >> > > > > > > > > >> > > across
> >> > > > > > > > > >> > > > >> the
> >> > > > > > > > > >> > > > >> >> > cluster
> >> > > > > > > > > >> > > > >> >> > >> >> that
> >> > > > > > > > > >> > > > >> >> > >> >> > > > need authentication to Hdfs
> >> > > > Namenode.
> >> > > > > > > > > >> Therefore, the
> >> > > > > > > > > >> > > > >> >> > delegation token
> >> > > > > > > > > >> > > > >> >> > >> >> > > > approach does seem like a
> good
> >> > fit
> >> > > > for
> >> > > > > > > this
> >> > > > > > > > > use
> >> > > > > > > > > >> case
> >> > > > > > > > > >> > > > as we
> >> > > > > > > > > >> > > > >> >> > have seen
> >> > > > > > > > > >> > > > >> >> > >> >> it
> >> > > > > > > > > >> > > > >> >> > >> >> > > > working at large scale in
> HDFS
> >> > and
> >> > > > > YARN.
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > >   The proposed design is
> very
> >> > much
> >> > > > > > inline
> >> > > > > > > > with
> >> > > > > > > > > >> Hadoop
> >> > > > > > > > > >> > > > >> >> > approach. A few
> >> > > > > > > > > >> > > > >> >> > >> >> > > >   comments:
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > 1) Why do you guys want to
> allow
> >> > > > > > infinite
> >> > > > > > > > > >> renewable
> >> > > > > > > > > >> > > > >> lifetime
> >> > > > > > > > > >> > > > >> >> > for a
> >> > > > > > > > > >> > > > >> >> > >> >> > > > token? HDFS restricts a
> token
> >> > to a
> >> > > > max
> >> > > > > > > life
> >> > > > > > > > > time
> >> > > > > > > > > >> > > > (default
> >> > > > > > > > > >> > > > >> 7
> >> > > > > > > > > >> > > > >> >> > days).  A
> >> > > > > > > > > >> > > > >> >> > >> >> > > > token's vulnerability is
> >> > believed
> >> > > to
> >> > > > > > > > increase
> >> > > > > > > > > >> with
> >> > > > > > > > > >> > > > time.
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > I agree that having infinite
> >> > > lifetime
> >> > > > > > might
> >> > > > > > > > not
> >> > > > > > > > > >> be the
> >> > > > > > > > > >> > > > best
> >> > > > > > > > > >> > > > >> idea.
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > 2) As I understand the
> tokens
> >> > are
> >> > > > > stored
> >> > > > > > > in
> >> > > > > > > > > >> zookeeper
> >> > > > > > > > > >> > > > as
> >> > > > > > > > > >> > > > >> well,
> >> > > > > > > > > >> > > > >> >> > and
> >> > > > > > > > > >> > > > >> >> > >> >> can
> >> > > > > > > > > >> > > > >> >> > >> >> > > > be updated there. This is
> clever
> >> > > as
> >> > > > it
> >> > > > > > can
> >> > > > > > > > > allow
> >> > > > > > > > > >> > > > >> replacing the
> >> > > > > > > > > >> > > > >> >> > tokens
> >> > > > > > > > > >> > > > >> >> > >> >> > > > once they run out of max
> life
> >> > > time,
> >> > > > > and
> >> > > > > > > > > clients
> >> > > > > > > > > >> can
> >> > > > > > > > > >> > > > >> download
> >> > > > > > > > > >> > > > >> >> > new
> >> > > > > > > > > >> > > > >> >> > >> >> tokens
> >> > > > > > > > > >> > > > >> >> > >> >> > > > from zookeeper. It
> shouldn't be
> >> > a
> >> > > > big
> >> > > > > > load
> >> > > > > > > > on
> >> > > > > > > > > >> > > zookeeper
> >> > > > > > > > > >> > > > >> as a
> >> > > > > > > > > >> > > > >> >> > client
> >> > > > > > > > > >> > > > >> >> > >> >> will
> >> > > > > > > > > >> > > > >> >> > >> >> > > > need to get a new token
> once in
> >> > > > > several
> >> > > > > > > > days.
> >> > > > > > > > > >> In this
> >> > > > > > > > > >> > > > >> approach
> >> > > > > > > > > >> > > > >> >> > you
> >> > > > > > > > > >> > > > >> >> > >> >> don't
> >> > > > > > > > > >> > > > >> >> > >> >> > > > need infinite lifetime on
> the
> >> > > token
> >> > > > > even
> >> > > > > > > for
> >> > > > > > > > > >> long
> >> > > > > > > > > >> > > > running
> >> > > > > > > > > >> > > > >> >> > clients.
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > 3) The token password are
> >> > > generated
> >> > > > > > using
> >> > > > > > > a
> >> > > > > > > > > >> master
> >> > > > > > > > > >> > > key.
> >> > > > > > > > > >> > > > >> The
> >> > > > > > > > > >> > > > >> >> > master
> >> > > > > > > > > >> > > > >> >> > >> >> key
> >> > > > > > > > > >> > > > >> >> > >> >> > > > should also be periodically
> >> > > changed.
> >> > > > > In
> >> > > > > > > > > Hadoop,
> >> > > > > > > > > >> the
> >> > > > > > > > > >> > > > >> default
> >> > > > > > > > > >> > > > >> >> > renewal
> >> > > > > > > > > >> > > > >> >> > >> >> > > > period is 1 day.?
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > IIUC, this will require
> brokers
> >> > > > > > maintaining
> >> > > > > > > a
> >> > > > > > > > > >> list of X
> >> > > > > > > > > >> > > > most
> >> > > > > > > > > >> > > > >> >> > recent
> >> > > > > > > > > >> > > > >> >> > >> >> master
> >> > > > > > > > > >> > > > >> >> > >> >> > > keys. This list will have to
> be
> >> > > > > persisted
> >> > > > > > > > > >> somewhere, as
> >> > > > > > > > > >> > > > if a
> >> > > > > > > > > >> > > > >> >> > broker
> >> > > > > > > > > >> > > > >> >> > >> >> goes
> >> > > > > > > > > >> > > > >> >> > >> >> > > down it will have to get
> that list
> >> > > > again
> >> > > > > > and
> >> > > > > > > > > >> storing
> >> > > > > > > > > >> > > > master
> >> > > > > > > > > >> > > > >> keys
> >> > > > > > > > > >> > > > >> >> > on ZK
> >> > > > > > > > > >> > > > >> >> > >> >> is
> >> > > > > > > > > >> > > > >> >> > >> >> > > not the best idea. However,
> if a
> >> > > > broker
> >> > > > > > goes
> >> > > > > > > > > down
> >> > > > > > > > > >> then
> >> > > > > > > > > >> > > we
> >> > > > > > > > > >> > > > >> have
> >> > > > > > > > > >> > > > >> >> > much
> >> > > > > > > > > >> > > > >> >> > >> >> bigger
> >> > > > > > > > > >> > > > >> >> > >> >> > > issue to deal with and
> client can
> >> > > > always
> >> > > > > > > > > >> > > re-authenticate
> >> > > > > > > > > >> > > > is
> >> > > > > > > > > >> > > > >> such
> >> > > > > > > > > >> > > > >> >> > >> >> events.
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > Did you happen to take a
> look at
> >> > > other
> >> > > > > > > > > >> alternatives
> >> > > > > > > > > >> > > this
> >> > > > > > > > > >> > > > >> list has
> >> > > > > > > > > >> > > > >> >> > >> >> > > suggested?
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > Thanks for a thorough
> proposal,
> >> > > > great
> >> > > > > > > work!"
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at
> 10:28
> >> > PM,
> >> > > > Gwen
> >> > > > > > > > Shapira
> >> > > > > > > > > >> wrote:
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > Makes sense to me.
> Thanks!
> >> > > > > > > > > >> > > > >> >> > >> >> > > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at
> 9:25
> >> > PM,
> >> > > > > > Harsha <
> >> > > > > > > > > >> > > > kafka@harsha.io
> >> > > > > > > > > >> > > > >> >
> >> > > > > > > > > >> > > > >> >> > wrote:
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > It doesn't need any
> release
> >> > > > > vehicle
> >> > > > > > > but
> >> > > > > > > > > >> still the
> >> > > > > > > > > >> > > > >> work can
> >> > > > > > > > > >> > > > >> >> > move
> >> > > > > > > > > >> > > > >> >> > >> >> > > > forward.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > If anyone is
> interested in
> >> > the
> >> > > > KIP
> >> > > > > > > > please
> >> > > > > > > > > >> do the
> >> > > > > > > > > >> > > > >> review and
> >> > > > > > > > > >> > > > >> >> > >> >> provide
> >> > > > > > > > > >> > > > >> >> > >> >> > > the
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > comments.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > -Harsha
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at
> >> > 04:59
> >> > > > PM,
> >> > > > > > > Ismael
> >> > > > > > > > > >> Juma
> >> > > > > > > > > >> > > > wrote:
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> I agree that it would
> be
> >> > good
> >> > > > to
> >> > > > > > have
> >> > > > > > > > > more
> >> > > > > > > > > >> time
> >> > > > > > > > > >> > > to
> >> > > > > > > > > >> > > > >> review
> >> > > > > > > > > >> > > > >> >> > and
> >> > > > > > > > > >> > > > >> >> > >> >> > > discuss
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> KIP-48.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> Ismael
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at
> >> > 12:55
> >> > > > AM,
> >> > > > > > Gwen
> >> > > > > > > > > >> Shapira <
> >> > > > > > > > > >> > > > >> >> > >> >> gwen@confluent.io>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Hi Team,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Since KIP-48
> depends on
> >> > > > KIP-43,
> >> > > > > > > which
> >> > > > > > > > > is
> >> > > > > > > > > >> > > > already a
> >> > > > > > > > > >> > > > >> bit
> >> > > > > > > > > >> > > > >> >> > of a
> >> > > > > > > > > >> > > > >> >> > >> >> risk
> >> > > > > > > > > >> > > > >> >> > >> >> > > for
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > the next release -
> any
> >> > > chance
> >> > > > > we
> >> > > > > > > can
> >> > > > > > > > > >> delay
> >> > > > > > > > > >> > > > >> delegation
> >> > > > > > > > > >> > > > >> >> > tokens
> >> > > > > > > > > >> > > > >> >> > >> >> to
> >> > > > > > > > > >> > > > >> >> > >> >> > > > Kafka
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > 0.10.1?
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > With the community
> >> > working
> >> > > > on a
> >> > > > > > > > release
> >> > > > > > > > > >> every
> >> > > > > > > > > >> > > 3
> >> > > > > > > > > >> > > > >> month,
> >> > > > > > > > > >> > > > >> >> > this
> >> > > > > > > > > >> > > > >> >> > >> >> is not
> >> > > > > > > > > >> > > > >> >> > >> >> > > > a huge
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > delay.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Gwen
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26,
> 2016 at
> >> > > 5:11
> >> > > > > PM,
> >> > > > > > > > Ashish
> >> > > > > > > > > >> Singh
> >> > > > > > > > > >> > > <
> >> > > > > > > > > >> > > > >> >> > >> >> > > asingh@cloudera.com>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Parth,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Thanks again for
> the
> >> > > > awesome
> >> > > > > > > write
> >> > > > > > > > > up.
> >> > > > > > > > > >> > > > Following
> >> > > > > > > > > >> > > > >> our
> >> > > > > > > > > >> > > > >> >> > >> >> discussion
> >> > > > > > > > > >> > > > >> >> > >> >> > > > from the
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I think it
> will
> >> > be
> >> > > > > easier
> >> > > > > > > to
> >> > > > > > > > > >> compare
> >> > > > > > > > > >> > > > >> various
> >> > > > > > > > > >> > > > >> >> > >> >> alternatives
> >> > > > > > > > > >> > > > >> >> > >> >> > > > if they
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > are
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > listed together.
> I am
> >> > > > stating
> >> > > > > > > > below a
> >> > > > > > > > > >> few
> >> > > > > > > > > >> > > > >> >> > alternatives along
> >> > > > > > > > > >> > > > >> >> > >> >> > > with
> >> > > > > > > > > >> > > > >> >> > >> >> > > > a the
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > current proposal.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Current proposal)
> >> > Store
> >> > > > > > > Delegation
> >> > > > > > > > > >> Token,
> >> > > > > > > > > >> > > DT,
> >> > > > > > > > > >> > > > >> on ZK.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> >> > > authenticates
> >> > > > > > with a
> >> > > > > > > > > >> broker.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a
> client is
> >> > > > > > > > authenticated,
> >> > > > > > > > > >> it
> >> > > > > > > > > >> > > will
> >> > > > > > > > > >> > > > >> make a
> >> > > > > > > > > >> > > > >> >> > broker
> >> > > > > > > > > >> > > > >> >> > >> >> side
> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a
> delegation
> >> > > > token.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker
> >> > > generates
> >> > > > a
> >> > > > > > > shared
> >> > > > > > > > > >> secret
> >> > > > > > > > > >> > > > based
> >> > > > > > > > > >> > > > >> on
> >> > > > > > > > > >> > > > >> >> > >> >> > > HMAC-SHA256(a
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    Password/Secret
> >> > shared
> >> > > > > > between
> >> > > > > > > > all
> >> > > > > > > > > >> > > brokers,
> >> > > > > > > > > >> > > > >> >> > randomly
> >> > > > > > > > > >> > > > >> >> > >> >> > > generated
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > tokenId).
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Broker
> stores
> >> > this
> >> > > > > token
> >> > > > > > in
> >> > > > > > > > its
> >> > > > > > > > > >> in
> >> > > > > > > > > >> > > > memory
> >> > > > > > > > > >> > > > >> cache.
> >> > > > > > > > > >> > > > >> >> > >> >> Broker
> >> > > > > > > > > >> > > > >> >> > >> >> > > > also stores
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the
> DelegationToken
> >> > > > > without
> >> > > > > > > the
> >> > > > > > > > > >> hmac in
> >> > > > > > > > > >> > > the
> >> > > > > > > > > >> > > > >> >> > zookeeper.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. All brokers
> will
> >> > > > have a
> >> > > > > > > cache
> >> > > > > > > > > >> backed
> >> > > > > > > > > >> > > by
> >> > > > > > > > > >> > > > >> >> > zookeeper so
> >> > > > > > > > > >> > > > >> >> > >> >> they
> >> > > > > > > > > >> > > > >> >> > >> >> > > > will all
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    get notified
> >> > whenever
> >> > > a
> >> > > > > new
> >> > > > > > > > token
> >> > > > > > > > > is
> >> > > > > > > > > >> > > > >> generated and
> >> > > > > > > > > >> > > > >> >> > they
> >> > > > > > > > > >> > > > >> >> > >> >> will
> >> > > > > > > > > >> > > > >> >> > >> >> > > > update
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    local cache
> whenever
> >> > > > token
> >> > > > > > > state
> >> > > > > > > > > >> changes.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Broker
> returns
> >> > the
> >> > > > > token
> >> > > > > > to
> >> > > > > > > > > >> Client.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues
> and
> >> > fixes
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Probable
> race
> >> > > > > condition,
> >> > > > > > > > client
> >> > > > > > > > > >> tries
> >> > > > > > > > > >> > > to
> >> > > > > > > > > >> > > > >> >> > authenticate
> >> > > > > > > > > >> > > > >> >> > >> >> with
> >> > > > > > > > > >> > > > >> >> > >> >> > > > a broker
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    that is yet to
> be
> >> > > > updated
> >> > > > > > with
> >> > > > > > > > the
> >> > > > > > > > > >> newly
> >> > > > > > > > > >> > > > >> generated
> >> > > > > > > > > >> > > > >> >> > DT.
> >> > > > > > > > > >> > > > >> >> > >> >> This
> >> > > > > > > > > >> > > > >> >> > >> >> > > can
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > probably be
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    dealt with
> making
> >> > > > > dtRequest
> >> > > > > > > > block
> >> > > > > > > > > >> until
> >> > > > > > > > > >> > > all
> >> > > > > > > > > >> > > > >> >> > brokers have
> >> > > > > > > > > >> > > > >> >> > >> >> > > > updated
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their DT
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    cache. Zk
> barrier or
> >> > > > > similar
> >> > > > > > > > > >> mechanism
> >> > > > > > > > > >> > > can
> >> > > > > > > > > >> > > > be
> >> > > > > > > > > >> > > > >> used.
> >> > > > > > > > > >> > > > >> >> > >> >> However,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > all such
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    mechanisms will
> >> > > increase
> >> > > > > > > > > complexity.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Using a
> static
> >> > > secret
> >> > > > > key
> >> > > > > > > > from
> >> > > > > > > > > >> config
> >> > > > > > > > > >> > > > >> file. Will
> >> > > > > > > > > >> > > > >> >> > >> >> require
> >> > > > > > > > > >> > > > >> >> > >> >> > > yet
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > another
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    config and
> uses a
> >> > > static
> >> > > > > > > secret
> >> > > > > > > > > >> key. It
> >> > > > > > > > > >> > > is
> >> > > > > > > > > >> > > > >> advised
> >> > > > > > > > > >> > > > >> >> > to
> >> > > > > > > > > >> > > > >> >> > >> >> rotate
> >> > > > > > > > > >> > > > >> >> > >> >> > > > secret
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > keys
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    periodically.
> This
> >> > can
> >> > > > be
> >> > > > > > > > avoided
> >> > > > > > > > > >> with
> >> > > > > > > > > >> > > > >> controller
> >> > > > > > > > > >> > > > >> >> > >> >> generating
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > secretKey and
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    passing to
> brokers
> >> > > > > > > periodically.
> >> > > > > > > > > >> However,
> >> > > > > > > > > >> > > > >> this will
> >> > > > > > > > > >> > > > >> >> > >> >> require
> >> > > > > > > > > >> > > > >> >> > >> >> > > > brokers to
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maintain
> certain
> >> > > counts
> >> > > > of
> >> > > > > > > > > >> secretKeys.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 1)
> Have
> >> > > > > controller
> >> > > > > > > > > >> generate
> >> > > > > > > > > >> > > > >> delegation
> >> > > > > > > > > >> > > > >> >> > token.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> >> > > authenticates
> >> > > > > > with a
> >> > > > > > > > > >> broker.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a
> client is
> >> > > > > > > > authenticated,
> >> > > > > > > > > >> it
> >> > > > > > > > > >> > > will
> >> > > > > > > > > >> > > > >> make a
> >> > > > > > > > > >> > > > >> >> > broker
> >> > > > > > > > > >> > > > >> >> > >> >> side
> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a
> delegation
> >> > > > token.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. Broker
> forwards
> >> > the
> >> > > > > > request
> >> > > > > > > > to
> >> > > > > > > > > >> > > > controller.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Controller
> >> > > generates
> >> > > > a
> >> > > > > DT
> >> > > > > > > and
> >> > > > > > > > > >> > > broadcasts
> >> > > > > > > > > >> > > > >> to all
> >> > > > > > > > > >> > > > >> >> > >> >> brokers.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. Broker
> stores
> >> > this
> >> > > > > token
> >> > > > > > in
> >> > > > > > > > its
> >> > > > > > > > > >> memory
> >> > > > > > > > > >> > > > >> cache.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Controller
> >> > responds
> >> > > > to
> >> > > > > > > > broker’s
> >> > > > > > > > > >> DT
> >> > > > > > > > > >> > > req.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    7. Broker
> returns
> >> > the
> >> > > > > token
> >> > > > > > to
> >> > > > > > > > > >> Client.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues
> and
> >> > fixes
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. We will
> have to
> >> > add
> >> > > > new
> >> > > > > > > APIs
> >> > > > > > > > to
> >> > > > > > > > > >> > > support
> >> > > > > > > > > >> > > > >> >> > controller
> >> > > > > > > > > >> > > > >> >> > >> >> pushing
> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > to
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    brokers on top
> of
> >> > the
> >> > > > > > minimal
> >> > > > > > > > APIs
> >> > > > > > > > > >> that
> >> > > > > > > > > >> > > are
> >> > > > > > > > > >> > > > >> >> > currently
> >> > > > > > > > > >> > > > >> >> > >> >> > > proposed.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. We will
> also have
> >> > > to
> >> > > > > add
> >> > > > > > > APIs
> >> > > > > > > > > to
> >> > > > > > > > > >> > > support
> >> > > > > > > > > >> > > > >> the
> >> > > > > > > > > >> > > > >> >> > >> >> bootstrapping
> >> > > > > > > > > >> > > > >> >> > >> >> > > > case,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > i.e,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    when a new
> broker
> >> > > comes
> >> > > > up
> >> > > > > > it
> >> > > > > > > > will
> >> > > > > > > > > >> have
> >> > > > > > > > > >> > > to
> >> > > > > > > > > >> > > > >> get all
> >> > > > > > > > > >> > > > >> >> > >> >> delegation
> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > from
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the controller.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. In
> catastrophic
> >> > > > > failures
> >> > > > > > > > where
> >> > > > > > > > > >> all
> >> > > > > > > > > >> > > > brokers
> >> > > > > > > > > >> > > > >> go
> >> > > > > > > > > >> > > > >> >> > down,
> >> > > > > > > > > >> > > > >> >> > >> >> the
> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if
> >> > > servers
> >> > > > > are
> >> > > > > > > > > >> restarted as
> >> > > > > > > > > >> > > > >> tokens
> >> > > > > > > > > >> > > > >> >> > are not
> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this
> happens,
> >> > then
> >> > > > > there
> >> > > > > > > are
> >> > > > > > > > > more
> >> > > > > > > > > >> > > > important
> >> > > > > > > > > >> > > > >> >> > things to
> >> > > > > > > > > >> > > > >> >> > >> >> > > worry
> >> > > > > > > > > >> > > > >> >> > >> >> > > > about
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is
> better
> >> > to
> >> > > > > > > > > >> re-authenticate.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 2)
> Do not
> >> > > > > > distribute
> >> > > > > > > > DT
> >> > > > > > > > > to
> >> > > > > > > > > >> > > other
> >> > > > > > > > > >> > > > >> brokers
> >> > > > > > > > > >> > > > >> >> > at
> >> > > > > > > > > >> > > > >> >> > >> >> all.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> >> > > authenticates
> >> > > > > > with a
> >> > > > > > > > > >> broker.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a
> client is
> >> > > > > > > > authenticated,
> >> > > > > > > > > >> it
> >> > > > > > > > > >> > > will
> >> > > > > > > > > >> > > > >> make a
> >> > > > > > > > > >> > > > >> >> > broker
> >> > > > > > > > > >> > > > >> >> > >> >> side
> >> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a
> delegation
> >> > > > token.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker
> >> > > generates
> >> > > > DT
> >> > > > > > of
> >> > > > > > > > > form,
> >> > > > > > > > > >> > > [hmac +
> >> > > > > > > > > >> > > > >> (owner,
> >> > > > > > > > > >> > > > >> >> > >> >> renewer,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maxLifeTime,
> id,
> >> > hmac,
> >> > > > > > > > > >> expirationTime)]
> >> > > > > > > > > >> > > and
> >> > > > > > > > > >> > > > >> passes
> >> > > > > > > > > >> > > > >> >> > back
> >> > > > > > > > > >> > > > >> >> > >> >> this
> >> > > > > > > > > >> > > > >> >> > >> >> > > > DT to
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > client.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    hmac is
> generated
> >> > via
> >> > > > > > > > > >> {HMAC-SHA256(owner,
> >> > > > > > > > > >> > > > >> renewer,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > maxLifeTime, id,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    expirationTime)
> >> > using
> >> > > > > > > > SecretKey}.
> >> > > > > > > > > >> Note
> >> > > > > > > > > >> > > that
> >> > > > > > > > > >> > > > >> all
> >> > > > > > > > > >> > > > >> >> > brokers
> >> > > > > > > > > >> > > > >> >> > >> >> have
> >> > > > > > > > > >> > > > >> >> > >> >> > > > this
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > SecretKey.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Client then
> goes
> >> > to
> >> > > > any
> >> > > > > > > > broker
> >> > > > > > > > > >> and to
> >> > > > > > > > > >> > > > >> >> > authenticate
> >> > > > > > > > > >> > > > >> >> > >> >> sends
> >> > > > > > > > > >> > > > >> >> > >> >> > > > the DT.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    Broker
> recalculates
> >> > > hmac
> >> > > > > > using
> >> > > > > > > > > >> (owner,
> >> > > > > > > > > >> > > > >> renewer,
> >> > > > > > > > > >> > > > >> >> > >> >> maxLifeTime,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > id, hmac,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> expirationTime) info
> >> > > > from
> >> > > > > DT
> >> > > > > > > and
> >> > > > > > > > > its
> >> > > > > > > > > >> > > > >> SecretKey. If
> >> > > > > > > > > >> > > > >> >> > it
> >> > > > > > > > > >> > > > >> >> > >> >> matches
> >> > > > > > > > > >> > > > >> >> > >> >> > > > with
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac of
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    DT, client is
> >> > > > > authenticated.
> >> > > > > > > > Yes,
> >> > > > > > > > > >> it will
> >> > > > > > > > > >> > > > do
> >> > > > > > > > > >> > > > >> other
> >> > > > > > > > > >> > > > >> >> > >> >> obvious
> >> > > > > > > > > >> > > > >> >> > >> >> > > > checks of
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    timestamp
> expiry and
> >> > > > such.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Note that secret
> key
> >> > will
> >> > > > be
> >> > > > > > > > > generated
> >> > > > > > > > > >> by
> >> > > > > > > > > >> > > > >> controller
> >> > > > > > > > > >> > > > >> >> > and
> >> > > > > > > > > >> > > > >> >> > >> >> passed
> >> > > > > > > > > >> > > > >> >> > >> >> > > to
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > brokers
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > periodically.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues
> and
> >> > fixes
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. How to
> delete a
> >> > DT?
> >> > > > > Yes,
> >> > > > > > > that
> >> > > > > > > > > is
> >> > > > > > > > > >> a
> >> > > > > > > > > >> > > > downside
> >> > > > > > > > > >> > > > >> >> > here.
> >> > > > > > > > > >> > > > >> >> > >> >> However,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > this can
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be handled with
> >> > > brokers
> >> > > > > > > > > maintaining
> >> > > > > > > > > >> a
> >> > > > > > > > > >> > > > >> blacklist of
> >> > > > > > > > > >> > > > >> >> > DTs,
> >> > > > > > > > > >> > > > >> >> > >> >> DTs
> >> > > > > > > > > >> > > > >> >> > >> >> > > > from this
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > list
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    can be removed
> after
> >> > > > > expiry.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. In
> catastrophic
> >> > > > > failures
> >> > > > > > > > where
> >> > > > > > > > > >> all
> >> > > > > > > > > >> > > > brokers
> >> > > > > > > > > >> > > > >> go
> >> > > > > > > > > >> > > > >> >> > down,
> >> > > > > > > > > >> > > > >> >> > >> >> the
> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if
> >> > > servers
> >> > > > > are
> >> > > > > > > > > >> restarted as
> >> > > > > > > > > >> > > > >> tokens
> >> > > > > > > > > >> > > > >> >> > are not
> >> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this
> happens,
> >> > then
> >> > > > > there
> >> > > > > > > are
> >> > > > > > > > > more
> >> > > > > > > > > >> > > > important
> >> > > > > > > > > >> > > > >> >> > things to
> >> > > > > > > > > >> > > > >> >> > >> >> > > worry
> >> > > > > > > > > >> > > > >> >> > >> >> > > > about
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is
> better
> >> > to
> >> > > > > > > > > >> re-authenticate.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26,
> 2016 at
> >> > > > 1:58
> >> > > > > > PM,
> >> > > > > > > > > Parth
> >> > > > > > > > > >> > > > >> Brahmbhatt <
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > pbrahmbhatt@hortonworks.com>
> >> > > > > > > > wrote:
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Hi,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> I have filed
> KIP-48 so
> >> > > we
> >> > > > > can
> >> > > > > > > > offer
> >> > > > > > > > > >> hadoop
> >> > > > > > > > > >> > > > like
> >> > > > > > > > > >> > > > >> >> > delegation
> >> > > > > > > > > >> > > > >> >> > >> >> > > > tokens in
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> kafka. You can
> review
> >> > > the
> >> > > > > > design
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >>
> >> > > > > > > > > >> > > > >> >> >
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > >
> >> > > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > .
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> This KIP depends
> on
> >> > > KIP-43
> >> > > > > and
> >> > > > > > > we
> >> > > > > > > > > >> have also
> >> > > > > > > > > >> > > > >> >> > discussed an
> >> > > > > > > > > >> > > > >> >> > >> >> > > > alternative to
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> proposed design
> here<
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >>
> >> > > > > > > > > >> > > > >> >> >
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > >
> >> > > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >.
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Thanks
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Parth
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > --
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Regards,
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Ashish
> >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> >> > > > > > > > > >> > > > >> >> > >> >> > > >
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > --
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >> > > Regards,
> >> > > > > > > > > >> > > > >> >> > >> >> > > Ashish
> >> > > > > > > > > >> > > > >> >> > >> >> > >
> >> > > > > > > > > >> > > > >> >> > >> >>
> >> > > > > > > > > >> > > > >> >> >
> >> > > > > > > > > >> > > > >>
> >> > > > > > > > > >> > > >
> >> > > > > > > > > >> > >
> >> > > > > > > > > >>
> >> > > > > > > > > >>
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > --
> >> > > > > > Regards,
> >> > > > > >
> >> > > > > > Rajini
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Regards,
> >> > > >
> >> > > > Rajini
> >> > > >
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Liquan Pei
> >> Software Engineer, Confluent Inc
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Gwen Shapira <gw...@confluent.io>]

From what I can see, remaining questions are:

1. Who / how are tokens renewed? By original requester only? or using
Kerberos auth only?
2. Are tokens stored on each broker or in ZK?
3. How are tokens invalidated / expired?
4. Which encryption algorithm is used?
5. What is the impersonation proposal (it wasn't in the KIP but was
discussed in this thread)?
6. Do we need new ACLs, if so - for what actions?

Gwen

On Thu, Jun 9, 2016 at 7:48 PM, Harsha <ka...@harsha.io> wrote:
> Jun & Ismael,
>                          Unfortunately I couldn't attend the KIP meeting
>                          when delegation tokens discussed. Appreciate if
>                          you can update the thread if you have any
>                          further questions.
> Thanks,
> Harsha
>
> On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
>> It seems that the links to images in the KIP are broken.
>>
>> Liquan
>>
>> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
>> brahmbhatt.parth@gmail.com> wrote:
>>
>> > 110. What does getDelegationTokenAs mean?
>> > In the current proposal we only allow a user to get delegation token for
>> > the identity that it authenticated as using another mechanism, i.e. A user
>> > that authenticate using a keytab for principal user1@EXAMPLE.COM will get
>> > delegation tokens for that user only. In future I think we will have to
>> > extend support such that we allow some set of users (
>> > kafka-rest-user@EXAMPLE.COM, storm-nimbus@EXAMPLE.COM) to acquire
>> > delegation tokens on behalf of other users whose identity they have
>> > verified independently.  Kafka brokers will have ACLs to control which
>> > users are allowed to impersonate other users and get tokens on behalf of
>> > them. Overall Impersonation is a whole different problem in my opinion and
>> > I think we can tackle it in separate KIP.
>> >
>> > 111. What's the typical rate of getting and renewing delegation tokens?
>> > Typically this should be very very low, 1 request per minute is a
>> > relatively high estimate. However it depends on the token expiration. I am
>> > less worried about the extra load it puts on controller vs the added
>> > complexity and the value it offers.
>> >
>> > Thanks
>> > Parth
>> >
>> >
>> >
>> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <is...@juma.me.uk> wrote:
>> >
>> > > Thanks Rajini. It would probably require a separate KIP as it will
>> > > introduce user visible changes. We could also update KIP-48 to have this
>> > > information, but it seems cleaner to do it separately. We can discuss
>> > that
>> > > in the KIP call today.
>> > >
>> > > Ismael
>> > >
>> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <
>> > > rajinisivaram@googlemail.com> wrote:
>> > >
>> > > > Ismael,
>> > > >
>> > > > I have created a JIRA (
>> > https://issues.apache.org/jira/browse/KAFKA-3751)
>> > > > for adding SCRAM as a SASL mechanism. Would that need another KIP? If
>> > > > KIP-48 will use this mechanism, can this just be a JIRA that gets
>> > > reviewed
>> > > > when the PR is ready?
>> > > >
>> > > > Thank you,
>> > > >
>> > > > Rajini
>> > > >
>> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <is...@juma.me.uk>
>> > wrote:
>> > > >
>> > > > > Thanks Rajini, SCRAM seems like a good candidate.
>> > > > >
>> > > > > Gwen had independently mentioned this as a SASL mechanism that might
>> > be
>> > > > > useful for Kafka and I have been meaning to check it in more detail.
>> > > Good
>> > > > > to know that you are willing to contribute an implementation. Maybe
>> > we
>> > > > > should file a separate JIRA for this?
>> > > > >
>> > > > > Ismael
>> > > > >
>> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram <
>> > > > > rajinisivaram@googlemail.com> wrote:
>> > > > >
>> > > > > > SCRAM (Salted Challenge Response Authentication Mechanism) is a
>> > > better
>> > > > > > mechanism than Digest-MD5. Java doesn't come with a built-in SCRAM
>> > > > > > SaslServer or SaslClient, but I will be happy to add support in
>> > Kafka
>> > > > > since
>> > > > > > it would be a useful mechanism to support anyway.
>> > > > > > https://tools.ietf.org/html/rfc7677 describes the protocol for
>> > > > > > SCRAM-SHA-256.
>> > > > > >
>> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <ju...@confluent.io> wrote:
>> > > > > >
>> > > > > > > Parth,
>> > > > > > >
>> > > > > > > Thanks for the explanation. A couple of more questions.
>> > > > > > >
>> > > > > > > 110. What does getDelegationTokenAs mean?
>> > > > > > >
>> > > > > > > 111. What's the typical rate of getting and renewing delegation
>> > > > tokens?
>> > > > > > > That may have an impact on whether they should be directed to the
>> > > > > > > controller.
>> > > > > > >
>> > > > > > > Jun
>> > > > > > >
>> > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt <
>> > > > > > > brahmbhatt.parth@gmail.com> wrote:
>> > > > > > >
>> > > > > > > > Hi Jun,
>> > > > > > > >
>> > > > > > > > Thanks for reviewing.
>> > > > > > > >
>> > > > > > > > * We could add a Cluster action to add acls on who can request
>> > > > > > delegation
>> > > > > > > > tokens. I don't see the use case for that yet but down the line
>> > > > when
>> > > > > we
>> > > > > > > > start supporting getDelegationTokenAs it will be necessary.
>> > > > > > > > * Yes we recommend tokens to be only used/distributed over
>> > secure
>> > > > > > > channels.
>> > > > > > > > * Depending on what design we end up choosing Invalidation will
>> > > be
>> > > > > > > > responsibility of every broker or controller.
>> > > > > > > > * I am not sure if I documented somewhere that invalidation
>> > will
>> > > > > > directly
>> > > > > > > > go through zookeeper but that is not the intent. Invalidation
>> > > will
>> > > > > > either
>> > > > > > > > be request based or due to expiration. No direct zookeeper
>> > > > > interaction
>> > > > > > > from
>> > > > > > > > any client.
>> > > > > > > > * "Broker also stores the DelegationToken without the hmac in
>> > the
>> > > > > > > > zookeeper." : Sorry about the confusion. The sole purpose of
>> > > > > zookeeper
>> > > > > > in
>> > > > > > > > this design is as distribution channel for tokens between all
>> > > > brokers
>> > > > > > > and a
>> > > > > > > > layer that ensures only tokens that were generated by making a
>> > > > > request
>> > > > > > > to a
>> > > > > > > > broker will be accepted (more on this in second paragraph). The
>> > > > token
>> > > > > > > > consists of few elements (owner, renewer, uuid , expiration,
>> > > hmac)
>> > > > ,
>> > > > > > one
>> > > > > > > of
>> > > > > > > > which is the finally generated hmac but hmac it self is
>> > derivable
>> > > > if
>> > > > > > you
>> > > > > > > > have all the other elements of the token + secret key to
>> > generate
>> > > > > hmac.
>> > > > > > > > Given zookeeper does not provide SSL support we do not want the
>> > > > > entire
>> > > > > > > > token to be wire transferred to zookeeper as that will be an
>> > > > insecure
>> > > > > > > wire
>> > > > > > > > transfer. Instead we only store all the other elements of a
>> > > > > delegation
>> > > > > > > > tokens. Brokers can read these elements and because they also
>> > > have
>> > > > > > access
>> > > > > > > > to secret key they will be able to generate hmac on their end.
>> > > > > > > >
>> > > > > > > > One of the alternative proposed is to avoid zookeeper
>> > > altogether. A
>> > > > > > > Client
>> > > > > > > > will call broker with required information (owner, renwer,
>> > > > > expiration)
>> > > > > > > and
>> > > > > > > > get back (signed hmac, uuid). Broker won't store this in
>> > > zookeeper.
>> > > > > > From
>> > > > > > > > this point a client can contact any broker with all the
>> > > delegation
>> > > > > > token
>> > > > > > > > info (owner, rewner, expiration, hmac, uuid) the borker will
>> > > > > regenerate
>> > > > > > > the
>> > > > > > > > hmac and as long as it matches with hmac presented by client ,
>> > > > broker
>> > > > > > > will
>> > > > > > > > allow the request to authenticate.  Only problem with this
>> > > approach
>> > > > > is
>> > > > > > if
>> > > > > > > > the secret key is compromised any client can now generate
>> > random
>> > > > > tokens
>> > > > > > > and
>> > > > > > > > they will still be able to authenticate as any user they like.
>> > > with
>> > > > > > > > zookeeper we guarantee that only tokens acquired via a broker
>> > > > (using
>> > > > > > some
>> > > > > > > > auth scheme other than delegation token) will be accepted. We
>> > > need
>> > > > to
>> > > > > > > > discuss which proposal makes more sense and we can go over it
>> > in
>> > > > > > > tomorrow's
>> > > > > > > > meeting.
>> > > > > > > >
>> > > > > > > > Also, can you forward the invite to me?
>> > > > > > > >
>> > > > > > > > Thanks
>> > > > > > > > Parth
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <ju...@confluent.io>
>> > > > wrote:
>> > > > > > > >
>> > > > > > > > > Thanks for the KIP. A few comments.
>> > > > > > > > >
>> > > > > > > > > 100. This potentially can be useful for Kafka Connect and
>> > Kafka
>> > > > > rest
>> > > > > > > > proxy
>> > > > > > > > > where a worker agent will need to run a task on behalf of a
>> > > > client.
>> > > > > > We
>> > > > > > > > will
>> > > > > > > > > likely need to change how those services use Kafka clients
>> > > > > > > > > (producer/consumer). Instead of a shared client per worker,
>> > we
>> > > > will
>> > > > > > > need
>> > > > > > > > a
>> > > > > > > > > client per user task since the authentication happens at the
>> > > > > > connection
>> > > > > > > > > level. For Kafka Connect, the renewer will be the workers.
>> > So,
>> > > we
>> > > > > > > > probably
>> > > > > > > > > need to allow multiple renewers. For Kafka rest proxy, the
>> > > > renewer
>> > > > > > can
>> > > > > > > > > probably just be the creator of the token.
>> > > > > > > > >
>> > > > > > > > > 101. Do we need new acl on who can request delegation tokens?
>> > > > > > > > >
>> > > > > > > > > 102. Do we recommend people to send delegation tokens in an
>> > > > > encrypted
>> > > > > > > > > channel?
>> > > > > > > > >
>> > > > > > > > > 103. Who is responsible for expiring tokens, every broker?
>> > > > > > > > >
>> > > > > > > > > 104. For invalidating tokens, would it be better to do it in
>> > a
>> > > > > > request
>> > > > > > > > > instead of going to ZK directly?
>> > > > > > > > >
>> > > > > > > > > 105. The terminology of client in the wiki sometimes refers
>> > to
>> > > > the
>> > > > > > end
>> > > > > > > > > client and some other times refers to the client using the
>> > > > > delegation
>> > > > > > > > > tokens. It would be useful to distinguish between the two.
>> > > > > > > > >
>> > > > > > > > > 106. Could you explain the sentence "Broker also stores the
>> > > > > > > > DelegationToken
>> > > > > > > > > without the hmac in the zookeeper." a bit more? I thought the
>> > > > > > > delegation
>> > > > > > > > > token is the hmac.
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > >
>> > > > > > > > > Jun
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <ju...@confluent.io>
>> > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi, Harsha,
>> > > > > > > > > >
>> > > > > > > > > > Just sent out a KIP meeting invite. We can discuss this in
>> > > the
>> > > > > > > meeting
>> > > > > > > > > > tomorrow.
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > >
>> > > > > > > > > > Jun
>> > > > > > > > > >
>> > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <ka...@harsha.io>
>> > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > >> Hi All,
>> > > > > > > > > >>            Can we have a KIP meeting around this. The KIP
>> > is
>> > > > up
>> > > > > > for
>> > > > > > > > > >>            sometime and if there are any questions lets
>> > > > quickly
>> > > > > > hash
>> > > > > > > > out
>> > > > > > > > > >>            details.
>> > > > > > > > > >>
>> > > > > > > > > >> Thanks,
>> > > > > > > > > >> Harsha
>> > > > > > > > > >>
>> > > > > > > > > >> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt wrote:
>> > > > > > > > > >> > That is what the hadoop echo system uses so no good
>> > reason
>> > > > > > really.
>> > > > > > > > We
>> > > > > > > > > >> > could
>> > > > > > > > > >> > change it to whatever is the newest recommended standard
>> > > is.
>> > > > > > > > > >> >
>> > > > > > > > > >> > Thanks
>> > > > > > > > > >> > Parth
>> > > > > > > > > >> >
>> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <
>> > > > > ismael@juma.me.uk
>> > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > > >> >
>> > > > > > > > > >> > > Hi Parth,
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > Thanks for the KIP. I only started reviewing this and
>> > > may
>> > > > > have
>> > > > > > > > > >> additional
>> > > > > > > > > >> > > questions later. The immediate question that came to
>> > > mind
>> > > > is
>> > > > > > our
>> > > > > > > > > >> choice of
>> > > > > > > > > >> > > "DIGEST-MD5" even though it's marked as OBSOLETE in
>> > the
>> > > > IANA
>> > > > > > > > > Registry
>> > > > > > > > > >> of
>> > > > > > > > > >> > > SASL mechanisms and the original RFC (2831) has been
>> > > moved
>> > > > > to
>> > > > > > > > > Historic
>> > > > > > > > > >> > > status:
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > https://tools.ietf.org/html/rfc6331
>> > > > > > > > > >> > >
>> > > > > > > > >
>> > > > > >
>> > > http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > What is the reasoning behind that choice?
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > Thanks,
>> > > > > > > > > >> > > Ismael
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <
>> > > > > > > gwen@confluent.io
>> > > > > > > > >
>> > > > > > > > > >> wrote:
>> > > > > > > > > >> > >
>> > > > > > > > > >> > > > Also comments inline :)
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > > * I want to emphasize that even though delegation
>> > > > tokens
>> > > > > > > are a
>> > > > > > > > > >> Hadoop
>> > > > > > > > > >> > > > > innovation, I feel very strongly about not adding
>> > > > > > dependency
>> > > > > > > > on
>> > > > > > > > > >> Hadoop
>> > > > > > > > > >> > > > > when implementing delegation tokens for Kafka. The
>> > > KIP
>> > > > > > > doesn't
>> > > > > > > > > >> imply
>> > > > > > > > > >> > > > > such dependency, but if you can clarify...
>> > > > > > > > > >> > > > >
>> > > > > > > > > >> > > > >
>> > > > > > > > > >> > > > > *No hadoop dependency.*
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > Yay! Just add this to the KIP so no one will read
>> > the
>> > > > KIP
>> > > > > > and
>> > > > > > > > > panic
>> > > > > > > > > >> > > > three weeks before the next release...
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > > * Can we get delegation token at any time after
>> > > > > > > > authenticating?
>> > > > > > > > > >> only
>> > > > > > > > > >> > > > > immediately after?
>> > > > > > > > > >> > > > >
>> > > > > > > > > >> > > > >
>> > > > > > > > > >> > > > > *As long as you are authenticated you can get
>> > > > delegation
>> > > > > > > > tokens.
>> > > > > > > > > >> We
>> > > > > > > > > >> > > need
>> > > > > > > > > >> > > > to
>> > > > > > > > > >> > > > > discuss if a client authenticated using delegation
>> > > > > token,
>> > > > > > > can
>> > > > > > > > > also
>> > > > > > > > > >> > > > acquire
>> > > > > > > > > >> > > > > delegation token again or not. Also there is the
>> > > > > question
>> > > > > > of
>> > > > > > > > do
>> > > > > > > > > we
>> > > > > > > > > >> > > allow
>> > > > > > > > > >> > > > > anyone to acquire delegation token or we want
>> > > specific
>> > > > > > ACLs
>> > > > > > > (I
>> > > > > > > > > >> think
>> > > > > > > > > >> > > its
>> > > > > > > > > >> > > > an
>> > > > > > > > > >> > > > > overkill.)*
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > I agree that ACLs is an overkill.
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > I think we are debating two options: Either require
>> > > > > Kerberos
>> > > > > > > > auth
>> > > > > > > > > >> for
>> > > > > > > > > >> > > > renewal or require non-owners to renew.
>> > > > > > > > > >> > > > I *think* the latter is simpler (it basically
>> > require
>> > > a
>> > > > > "job
>> > > > > > > > > master"
>> > > > > > > > > >> > > > to take responsibility for the renewal, it will have
>> > > its
>> > > > > own
>> > > > > > > > > >> identity
>> > > > > > > > > >> > > > anyway and I think this is the correct design
>> > pattern
>> > > > > > anyway.
>> > > > > > > > For
>> > > > > > > > > >> > > > storm, I'd expect Nimbus to coordinate renewals?),
>> > but
>> > > > it
>> > > > > is
>> > > > > > > > hard
>> > > > > > > > > to
>> > > > > > > > > >> > > > debate simplicity without looking at the code
>> > changes
>> > > > > > > required.
>> > > > > > > > If
>> > > > > > > > > >> you
>> > > > > > > > > >> > > > have a draft of how the "require Kerberos" will look
>> > > in
>> > > > > > Kafka
>> > > > > > > > > code,
>> > > > > > > > > >> > > > I'll be happy to take a look.
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > > * My understanding is that tokens will propagate
>> > via
>> > > > ZK
>> > > > > > but
>> > > > > > > > > >> without
>> > > > > > > > > >> > > > > additional changes to UpdateMetadata protocol,
>> > > > correct?
>> > > > > > > > Clients
>> > > > > > > > > >> > > > > currently don't retry on SASL auth failure (IIRC),
>> > > but
>> > > > > > since
>> > > > > > > > the
>> > > > > > > > > >> > > > > tokens propagate between brokers asynch, we will
>> > > need
>> > > > to
>> > > > > > > > retry a
>> > > > > > > > > >> bit
>> > > > > > > > > >> > > > > to avoid clients failing auth due to timing
>> > issues.
>> > > > > > > > > >> > > > >
>> > > > > > > > > >> > > > > *I am considering 2 alternatives right now. The
>> > > > current
>> > > > > > > > > documented
>> > > > > > > > > >> > > > approach
>> > > > > > > > > >> > > > > is zookeeper based and it does not require any
>> > > changes
>> > > > > to
>> > > > > > > > > >> > > UpdateMetadata
>> > > > > > > > > >> > > > > protocol. An alternative approach can remove
>> > > zookeeper
>> > > > > > > > > dependency
>> > > > > > > > > >> as
>> > > > > > > > > >> > > well
>> > > > > > > > > >> > > > > but we can discuss that in KIP discussion call.*
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > Oooh! Sounds interesting. Do you want to ping Jun to
>> > > > > > arrange a
>> > > > > > > > > call?
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > > * I liked Ashish's suggestion of having just the
>> > > > > > controller
>> > > > > > > > > issue
>> > > > > > > > > >> the
>> > > > > > > > > >> > > > > delegation tokens, to avoid syncing a shared
>> > secret.
>> > > > Not
>> > > > > > > sure
>> > > > > > > > if
>> > > > > > > > > >> we
>> > > > > > > > > >> > > > > want to continue the discussion here or on the
>> > > wiki. I
>> > > > > > think
>> > > > > > > > > that
>> > > > > > > > > >> we
>> > > > > > > > > >> > > > > can decouple the problem of "token distribution"
>> > > from
>> > > > > > > "shared
>> > > > > > > > > >> secret
>> > > > > > > > > >> > > > > distribution" and use the controller as the only
>> > > token
>> > > > > > > > generator
>> > > > > > > > > >> to
>> > > > > > > > > >> > > > > solve the second issue, while still using ZK async
>> > > to
>> > > > > > > > distribute
>> > > > > > > > > >> > > > > tokens.
>> > > > > > > > > >> > > > >
>> > > > > > > > > >> > > > >
>> > > > > > > > > >> > > > > *As mentioned in the previous Email I am fine with
>> > > > that
>> > > > > > > > approach
>> > > > > > > > > >> as
>> > > > > > > > > >> > > long
>> > > > > > > > > >> > > > as
>> > > > > > > > > >> > > > > we agree that the extra complexity of
>> > > adding/updating
>> > > > > APIS
>> > > > > > > > adds
>> > > > > > > > > >> enough
>> > > > > > > > > >> > > > > value. The advantage with the controller approach
>> > is
>> > > > > > secret
>> > > > > > > > > >> rotation
>> > > > > > > > > >> > > can
>> > > > > > > > > >> > > > be
>> > > > > > > > > >> > > > > automated,frequent and would not require
>> > > deployment. *
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > Can you detail the extra complexity (or point me to
>> > > the
>> > > > > > email
>> > > > > > > I
>> > > > > > > > > >> > > > missed?) - which Apis are required?
>> > > > > > > > > >> > > > As far as I can tell, clients can already find the
>> > > > > > controller
>> > > > > > > > from
>> > > > > > > > > >> > > > metadata. I'm a bit more concerned about controller
>> > > > load.
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > >
>> > > > > > > > > >> > > > > * While I like the idea of forcing kerberos auth
>> > for
>> > > > > > > renwal, I
>> > > > > > > > > >> think
>> > > > > > > > > >> > > > > it mixes the transport layer the the request
>> > content
>> > > > in
>> > > > > a
>> > > > > > > > pretty
>> > > > > > > > > >> ugly
>> > > > > > > > > >> > > > > way. Perhaps limiting renewer to non-owner is
>> > > better.
>> > > > > > > > > >> > > > >
>> > > > > > > > > >> > > > > *I feel this is a necessary evil. While this will
>> > > make
>> > > > > the
>> > > > > > > > kafka
>> > > > > > > > > >> code
>> > > > > > > > > >> > > > > pretty straight forward , forcing  renewer to
>> > > > non-owner
>> > > > > > > pushes
>> > > > > > > > > >> the code
>> > > > > > > > > >> > > > > ugliness to client and makes it even harder to
>> > > > > > integrate.  *
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > As mentioned before, I don't think the "renewal by
>> > > > other"
>> > > > > > > > approach
>> > > > > > > > > >> is
>> > > > > > > > > >> > > > that ugly for the clients we expect to use
>> > delegation
>> > > > > tokens
>> > > > > > > > since
>> > > > > > > > > >> > > > they will have an app-master of some sort who
>> > > requested
>> > > > > the
>> > > > > > > > token
>> > > > > > > > > to
>> > > > > > > > > >> > > > begin with.
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > >
>> > > > > > > > > >> > > > > The response for my question on how multiple
>> > > > identities
>> > > > > > will
>> > > > > > > > be
>> > > > > > > > > >> > > > > handled wasn't super clear to me - AFAIK, we don't
>> > > > > > > > authenticate
>> > > > > > > > > >> each
>> > > > > > > > > >> > > > > request, we authenticate connections.
>> > > > > > > > > >> > > > >
>> > > > > > > > > >> > > > > *We authenticate connections, and only when they
>> > are
>> > > > > being
>> > > > > > > > > >> established.
>> > > > > > > > > >> > > > Let
>> > > > > > > > > >> > > > > me try to phrase this as a question, in absence of
>> > > > > > > delegation
>> > > > > > > > > >> tokens if
>> > > > > > > > > >> > > > we
>> > > > > > > > > >> > > > > had to support the use case using user TGT's how
>> > > would
>> > > > > we
>> > > > > > do
>> > > > > > > > it?
>> > > > > > > > > >> My
>> > > > > > > > > >> > > point
>> > > > > > > > > >> > > > > was it would be no different with delegation
>> > tokens.
>> > > > The
>> > > > > > use
>> > > > > > > > > case
>> > > > > > > > > >> you
>> > > > > > > > > >> > > are
>> > > > > > > > > >> > > > > describing seems more like impersonation.*
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > Yeah, I thought that one of the things that
>> > delegation
>> > > > > > tokens
>> > > > > > > > > >> handled.
>> > > > > > > > > >> > > > Maybe I got it wrong :)
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > Thanks for the detailed answers.
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > Gwen
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > > > > Thanks
>> > > > > > > > > >> > > > > Parth
>> > > > > > > > > >> > > > >
>> > > > > > > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <
>> > > > > > > > > gwen@confluent.io
>> > > > > > > > > >> >
>> > > > > > > > > >> > > > wrote:
>> > > > > > > > > >> > > > >
>> > > > > > > > > >> > > > >> Hi Parth and Harsha,
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >> Few more comments:
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >> * The API / RequestResponse section doesn't seem
>> > to
>> > > > > have
>> > > > > > > good
>> > > > > > > > > >> > > > >> description of the changes to the Kafka Protocol.
>> > > > > Sounds
>> > > > > > > like
>> > > > > > > > > >> you are
>> > > > > > > > > >> > > > >> proposing new DelegationTokenRequest and
>> > > > > > RenewTokenRequest
>> > > > > > > > (and
>> > > > > > > > > >> > > > >> matching responses), without detailing the
>> > contents
>> > > > of
>> > > > > > the
>> > > > > > > > > >> requests
>> > > > > > > > > >> > > > >> and responses? Or rather, you show the class
>> > > > interface,
>> > > > > > but
>> > > > > > > > not
>> > > > > > > > > >> the
>> > > > > > > > > >> > > > >> underlying protocol. This could be seen as an
>> > > > > > > implementation
>> > > > > > > > > >> detail,
>> > > > > > > > > >> > > > >> but since the binary protocol is what we provide
>> > to
>> > > > > > > non-Java
>> > > > > > > > > >> clients,
>> > > > > > > > > >> > > > >> we need to show the changes there.
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >> * getDelegationToken sounds like
>> > > > > > > > delegationTokenRequestHandler?
>> > > > > > > > > >> Is it
>> > > > > > > > > >> > > > >> planned to be part of KafkaApi? or Client? Its
>> > > > > unclear...
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >> * I want to emphasize that even though delegation
>> > > > > tokens
>> > > > > > > are
>> > > > > > > > a
>> > > > > > > > > >> Hadoop
>> > > > > > > > > >> > > > >> innovation, I feel very strongly about not adding
>> > > > > > > dependency
>> > > > > > > > on
>> > > > > > > > > >> Hadoop
>> > > > > > > > > >> > > > >> when implementing delegation tokens for Kafka.
>> > The
>> > > > KIP
>> > > > > > > > doesn't
>> > > > > > > > > >> imply
>> > > > > > > > > >> > > > >> such dependency, but if you can clarify...
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >> * Can we get delegation token at any time after
>> > > > > > > > authenticating?
>> > > > > > > > > >> only
>> > > > > > > > > >> > > > >> immediately after?
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >> * My understanding is that tokens will propagate
>> > > via
>> > > > ZK
>> > > > > > but
>> > > > > > > > > >> without
>> > > > > > > > > >> > > > >> additional changes to UpdateMetadata protocol,
>> > > > correct?
>> > > > > > > > Clients
>> > > > > > > > > >> > > > >> currently don't retry on SASL auth failure
>> > (IIRC),
>> > > > but
>> > > > > > > since
>> > > > > > > > > the
>> > > > > > > > > >> > > > >> tokens propagate between brokers asynch, we will
>> > > need
>> > > > > to
>> > > > > > > > retry
>> > > > > > > > > a
>> > > > > > > > > >> bit
>> > > > > > > > > >> > > > >> to avoid clients failing auth due to timing
>> > issues.
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >> * Strongly agreeing on clients not touching ZK
>> > > > directly
>> > > > > > :)
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >> * I liked Ashish's suggestion of having just the
>> > > > > > controller
>> > > > > > > > > >> issue the
>> > > > > > > > > >> > > > >> delegation tokens, to avoid syncing a shared
>> > > secret.
>> > > > > Not
>> > > > > > > sure
>> > > > > > > > > if
>> > > > > > > > > >> we
>> > > > > > > > > >> > > > >> want to continue the discussion here or on the
>> > > wiki.
>> > > > I
>> > > > > > > think
>> > > > > > > > > >> that we
>> > > > > > > > > >> > > > >> can decouple the problem of "token distribution"
>> > > from
>> > > > > > > "shared
>> > > > > > > > > >> secret
>> > > > > > > > > >> > > > >> distribution" and use the controller as the only
>> > > > token
>> > > > > > > > > generator
>> > > > > > > > > >> to
>> > > > > > > > > >> > > > >> solve the second issue, while still using ZK
>> > async
>> > > to
>> > > > > > > > > distribute
>> > > > > > > > > >> > > > >> tokens.
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >> * I am also uncomfortable with infinite lifetime
>> > of
>> > > > > > tokens
>> > > > > > > > (and
>> > > > > > > > > >> hoped
>> > > > > > > > > >> > > > >> to hear from others in the community) - but
>> > having
>> > > > the
>> > > > > > > option
>> > > > > > > > > and
>> > > > > > > > > >> > > > >> documenting it as less secure, allows users to
>> > > > > configure
>> > > > > > > > their
>> > > > > > > > > >> system
>> > > > > > > > > >> > > > >> as the wish.
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >> * While I like the idea of forcing kerberos auth
>> > > for
>> > > > > > > renwal,
>> > > > > > > > I
>> > > > > > > > > >> think
>> > > > > > > > > >> > > > >> it mixes the transport layer the the request
>> > > content
>> > > > > in a
>> > > > > > > > > pretty
>> > > > > > > > > >> ugly
>> > > > > > > > > >> > > > >> way. Perhaps limiting renewer to non-owner is
>> > > better.
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >> Things I'd still like to see:
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >> * More detailed explanation on what we plan to do
>> > > for
>> > > > > the
>> > > > > > > > java
>> > > > > > > > > >> clients
>> > > > > > > > > >> > > > >> specifically - new configuration? new APIs?
>> > > > > > > > > >> > > > >> The response for my question on how multiple
>> > > > identities
>> > > > > > > will
>> > > > > > > > be
>> > > > > > > > > >> > > > >> handled wasn't super clear to me - AFAIK, we
>> > don't
>> > > > > > > > authenticate
>> > > > > > > > > >> each
>> > > > > > > > > >> > > > >> request, we authenticate connections.
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >> * Alternatives: Delegation tokens are only used
>> > in
>> > > > the
>> > > > > > > Hadoop
>> > > > > > > > > >> > > > >> ecosystem. I'm wondering if there are
>> > alternatives
>> > > in
>> > > > > > other
>> > > > > > > > > >> ecosystems
>> > > > > > > > > >> > > > >> (Mesos? Tachyon? Cassandra?) and whether there
>> > are
>> > > > some
>> > > > > > > > > >> advantages
>> > > > > > > > > >> > > > >> there.
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >> Gwen
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <
>> > > > > kafka@harsha.io
>> > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > > >> > > > >> > Hi Gwen,
>> > > > > > > > > >> > > > >> >            Can you look at Parth's last reply.
>> > > Does
>> > > > > it
>> > > > > > > > answer
>> > > > > > > > > >> your
>> > > > > > > > > >> > > > >> >            concerns.
>> > > > > > > > > >> > > > >> >
>> > > > > > > > > >> > > > >> > Thanks,
>> > > > > > > > > >> > > > >> > Harsha
>> > > > > > > > > >> > > > >> >
>> > > > > > > > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth
>> > > brahmbhatt
>> > > > > > wrote:
>> > > > > > > > > >> > > > >> >> Thanks for reviewing Gwen. The wiki already
>> > has
>> > > > > > details
>> > > > > > > on
>> > > > > > > > > >> token
>> > > > > > > > > >> > > > >> >> expiration
>> > > > > > > > > >> > > > >> >> under token acquisition process
>> > > > > > > > > >> > > > >> >> <
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > >
>> > > > > > > > > >>
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
>> > > > > > > > > >> > > > >> >.
>> > > > > > > > > >> > > > >> >> Current proposal is that tokens will expire
>> > > based
>> > > > > on a
>> > > > > > > > > server
>> > > > > > > > > >> side
>> > > > > > > > > >> > > > >> >> configuration (default 24 hours) unless
>> > renewed.
>> > > > > > Renewal
>> > > > > > > > is
>> > > > > > > > > >> only
>> > > > > > > > > >> > > > allowed
>> > > > > > > > > >> > > > >> >> until the max life time of token.
>> > Alternatively
>> > > we
>> > > > > > could
>> > > > > > > > > also
>> > > > > > > > > >> make
>> > > > > > > > > >> > > > that
>> > > > > > > > > >> > > > >> >> an
>> > > > > > > > > >> > > > >> >> optional param and the server side default can
>> > > > serve
>> > > > > > as
>> > > > > > > > the
>> > > > > > > > > >> upper
>> > > > > > > > > >> > > > bound.
>> > > > > > > > > >> > > > >> >>
>> > > > > > > > > >> > > > >> >> To your second point it will be done exactly
>> > the
>> > > > > same
>> > > > > > > way
>> > > > > > > > we
>> > > > > > > > > >> would
>> > > > > > > > > >> > > > >> >> support
>> > > > > > > > > >> > > > >> >> multiple keytabs. The calling client will have
>> > > to
>> > > > > put
>> > > > > > > the
>> > > > > > > > > >> tokens it
>> > > > > > > > > >> > > > >> wants
>> > > > > > > > > >> > > > >> >> to use in the subject instance and call
>> > > > > > produce/consume
>> > > > > > > > > inside
>> > > > > > > > > >> > > > >> >> subject.doas. Each caller will have to keep
>> > > track
>> > > > of
>> > > > > > its
>> > > > > > > > own
>> > > > > > > > > >> > > > subject. I
>> > > > > > > > > >> > > > >> >> will have to look at the code to see if we
>> > > support
>> > > > > > this
>> > > > > > > > > >> feature
>> > > > > > > > > >> > > right
>> > > > > > > > > >> > > > >> now
>> > > > > > > > > >> > > > >> >> but my understanding is delegation token
>> > > shouldn't
>> > > > > > need
>> > > > > > > > any
>> > > > > > > > > >> special
>> > > > > > > > > >> > > > >> >> treatment as its just another type of
>> > Credential
>> > > > in
>> > > > > > the
>> > > > > > > > > >> subject.
>> > > > > > > > > >> > > > >> >>
>> > > > > > > > > >> > > > >> >> I would also like to know what is your opinion
>> > > > about
>> > > > > > > > > infinite
>> > > > > > > > > >> > > renewal
>> > > > > > > > > >> > > > >> (my
>> > > > > > > > > >> > > > >> >> recommendation is to not support this), tokens
>> > > > > > renewing
>> > > > > > > > them
>> > > > > > > > > >> > > self(my
>> > > > > > > > > >> > > > >> >> recommendation is to not support this) and
>> > most
>> > > > > > > > importantly
>> > > > > > > > > >> your
>> > > > > > > > > >> > > > choice
>> > > > > > > > > >> > > > >> >> between the alternatives listed on this thread
>> > > > > > > > > >> > > > >> >> <
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > >
>> > > > > > > > > >>
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
>> > > > > > > > > >> > > > >> >
>> > > > > > > > > >> > > > >> >> ( I am leaning towards the alternative-2 minus
>> > > > > > > controller
>> > > > > > > > > >> > > > distributing
>> > > > > > > > > >> > > > >> >> secret). Thanks again for reviewing.
>> > > > > > > > > >> > > > >> >>
>> > > > > > > > > >> > > > >> >> Thanks
>> > > > > > > > > >> > > > >> >> Parth
>> > > > > > > > > >> > > > >> >>
>> > > > > > > > > >> > > > >> >>
>> > > > > > > > > >> > > > >> >>
>> > > > > > > > > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <
>> > > > > > > > > >> gwen@confluent.io>
>> > > > > > > > > >> > > > wrote:
>> > > > > > > > > >> > > > >> >>
>> > > > > > > > > >> > > > >> >> > Harsha,
>> > > > > > > > > >> > > > >> >> >
>> > > > > > > > > >> > > > >> >> > I was thinking of the Rest Proxy. I didn't
>> > see
>> > > > > your
>> > > > > > > > design
>> > > > > > > > > >> yet,
>> > > > > > > > > >> > > > but in
>> > > > > > > > > >> > > > >> >> > our proxy, we have a set of producers, which
>> > > > will
>> > > > > > > serve
>> > > > > > > > > >> multiple
>> > > > > > > > > >> > > > users
>> > > > > > > > > >> > > > >> >> > going through the proxy. Since these users
>> > > will
>> > > > > have
>> > > > > > > > > >> different
>> > > > > > > > > >> > > > >> >> > privileges, they'll need to authenticate
>> > > > > separately,
>> > > > > > > and
>> > > > > > > > > >> can't
>> > > > > > > > > >> > > > share a
>> > > > > > > > > >> > > > >> >> > token.
>> > > > > > > > > >> > > > >> >> >
>> > > > > > > > > >> > > > >> >> > Am I missing anything?
>> > > > > > > > > >> > > > >> >> >
>> > > > > > > > > >> > > > >> >> > Gwen
>> > > > > > > > > >> > > > >> >> >
>> > > > > > > > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <
>> > > > > > > kafka@harsha.io
>> > > > > > > > >
>> > > > > > > > > >> wrote:
>> > > > > > > > > >> > > > >> >> > > Gwen,
>> > > > > > > > > >> > > > >> >> > >            On your second point. Can you
>> > > > > describe
>> > > > > > a
>> > > > > > > > > >> usecase
>> > > > > > > > > >> > > where
>> > > > > > > > > >> > > > >> >> > >            mutliple clients ended up
>> > > sharing a
>> > > > > > > > producer
>> > > > > > > > > >> and
>> > > > > > > > > >> > > even
>> > > > > > > > > >> > > > if
>> > > > > > > > > >> > > > >> they
>> > > > > > > > > >> > > > >> >> > >            do why can't they not use
>> > single
>> > > > > token
>> > > > > > > that
>> > > > > > > > > >> producer
>> > > > > > > > > >> > > > >> >> > >            captures. Why would we need
>> > > > multiple
>> > > > > > > > clients
>> > > > > > > > > >> with
>> > > > > > > > > >> > > > >> different
>> > > > > > > > > >> > > > >> >> > >            tokens sharing a single
>> > instance
>> > > of
>> > > > > > > > producer.
>> > > > > > > > > >> Also
>> > > > > > > > > >> > > in
>> > > > > > > > > >> > > > >> this
>> > > > > > > > > >> > > > >> >> > >            case other clients have access
>> > > all
>> > > > > the
>> > > > > > > > tokens
>> > > > > > > > > >> no?
>> > > > > > > > > >> > > > >> >> > >
>> > > > > > > > > >> > > > >> >> > > Thanks,
>> > > > > > > > > >> > > > >> >> > > Harsha
>> > > > > > > > > >> > > > >> >> > >
>> > > > > > > > > >> > > > >> >> > >
>> > > > > > > > > >> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen
>> > > Shapira
>> > > > > > > wrote:
>> > > > > > > > > >> > > > >> >> > >> Sorry for the delay:
>> > > > > > > > > >> > > > >> >> > >>
>> > > > > > > > > >> > > > >> >> > >> Two questions that we didn't see in the
>> > > wiki:
>> > > > > > > > > >> > > > >> >> > >> 1. Is there an expiration for delegation
>> > > > > tokens?
>> > > > > > > > > >> Renewal? How
>> > > > > > > > > >> > > > do we
>> > > > > > > > > >> > > > >> >> > >> revoke them?
>> > > > > > > > > >> > > > >> >> > >> 2. If we want to use delegation tokens
>> > for
>> > > > > > "do-as"
>> > > > > > > > > (say,
>> > > > > > > > > >> > > submit
>> > > > > > > > > >> > > > >> Storm
>> > > > > > > > > >> > > > >> >> > >> job as my user), we will need a producer
>> > > for
>> > > > > > every
>> > > > > > > > job
>> > > > > > > > > >> (we
>> > > > > > > > > >> > > can't
>> > > > > > > > > >> > > > >> share
>> > > > > > > > > >> > > > >> >> > >> them between multiple jobs running on
>> > same
>> > > > > node),
>> > > > > > > > since
>> > > > > > > > > >> we
>> > > > > > > > > >> > > only
>> > > > > > > > > >> > > > >> >> > >> authenticate when connecting. Is there a
>> > > plan
>> > > > > to
>> > > > > > > > change
>> > > > > > > > > >> this
>> > > > > > > > > >> > > for
>> > > > > > > > > >> > > > >> >> > >> delegation tokens, in order to allow
>> > > multiple
>> > > > > > users
>> > > > > > > > > with
>> > > > > > > > > >> > > > different
>> > > > > > > > > >> > > > >> >> > >> tokens to share a client?
>> > > > > > > > > >> > > > >> >> > >>
>> > > > > > > > > >> > > > >> >> > >> Gwen
>> > > > > > > > > >> > > > >> >> > >>
>> > > > > > > > > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth
>> > > > > brahmbhatt
>> > > > > > > > > >> > > > >> >> > >> <br...@gmail.com> wrote:
>> > > > > > > > > >> > > > >> >> > >> > Bumping this up one more time, can
>> > other
>> > > > > > > committers
>> > > > > > > > > >> review?
>> > > > > > > > > >> > > > >> >> > >> >
>> > > > > > > > > >> > > > >> >> > >> > Thanks
>> > > > > > > > > >> > > > >> >> > >> > Parth
>> > > > > > > > > >> > > > >> >> > >> >
>> > > > > > > > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM,
>> > Harsha <
>> > > > > > > > > >> kafka@harsha.io>
>> > > > > > > > > >> > > > wrote:
>> > > > > > > > > >> > > > >> >> > >> >
>> > > > > > > > > >> > > > >> >> > >> >> Parth,
>> > > > > > > > > >> > > > >> >> > >> >>           Overall current design looks
>> > > > good
>> > > > > to
>> > > > > > > > me. I
>> > > > > > > > > >> am +1
>> > > > > > > > > >> > > on
>> > > > > > > > > >> > > > >> the
>> > > > > > > > > >> > > > >> >> > KIP.
>> > > > > > > > > >> > > > >> >> > >> >>
>> > > > > > > > > >> > > > >> >> > >> >> Gwen , Jun can you review this as
>> > well.
>> > > > > > > > > >> > > > >> >> > >> >>
>> > > > > > > > > >> > > > >> >> > >> >> -Harsha
>> > > > > > > > > >> > > > >> >> > >> >>
>> > > > > > > > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM,
>> > parth
>> > > > > > > > brahmbhatt
>> > > > > > > > > >> wrote:
>> > > > > > > > > >> > > > >> >> > >> >> > Thanks for review Jitendra.
>> > > > > > > > > >> > > > >> >> > >> >> >
>> > > > > > > > > >> > > > >> >> > >> >> > I don't like the idea of infinite
>> > > > lifetime
>> > > > > > > but I
>> > > > > > > > > >> see the
>> > > > > > > > > >> > > > >> Streaming
>> > > > > > > > > >> > > > >> >> > use
>> > > > > > > > > >> > > > >> >> > >> >> > case. Even for Streaming use case I
>> > > was
>> > > > > > hoping
>> > > > > > > > > >> there will
>> > > > > > > > > >> > > > be
>> > > > > > > > > >> > > > >> some
>> > > > > > > > > >> > > > >> >> > notion
>> > > > > > > > > >> > > > >> >> > >> >> > of
>> > > > > > > > > >> > > > >> >> > >> >> > master/driver that can get new
>> > > > delegation
>> > > > > > > tokens
>> > > > > > > > > at
>> > > > > > > > > >> fixed
>> > > > > > > > > >> > > > >> interval
>> > > > > > > > > >> > > > >> >> > and
>> > > > > > > > > >> > > > >> >> > >> >> > distribute to workers. If that is
>> > not
>> > > > the
>> > > > > > case
>> > > > > > > > for
>> > > > > > > > > >> we can
>> > > > > > > > > >> > > > >> discuss
>> > > > > > > > > >> > > > >> >> > >> >> > delegation tokens renewing them self
>> > > and
>> > > > > the
>> > > > > > > > > >> security
>> > > > > > > > > >> > > > >> implications
>> > > > > > > > > >> > > > >> >> > of the
>> > > > > > > > > >> > > > >> >> > >> >> > same.
>> > > > > > > > > >> > > > >> >> > >> >> >
>> > > > > > > > > >> > > > >> >> > >> >> > I did not want clients to fetch
>> > tokens
>> > > > > from
>> > > > > > > > > >> zookeeper,
>> > > > > > > > > >> > > > >> overall I
>> > > > > > > > > >> > > > >> >> > think
>> > > > > > > > > >> > > > >> >> > >> >> > its
>> > > > > > > > > >> > > > >> >> > >> >> > better if clients don't rely on our
>> > > > > metadata
>> > > > > > > > store
>> > > > > > > > > >> and I
>> > > > > > > > > >> > > > >> think we
>> > > > > > > > > >> > > > >> >> > are
>> > > > > > > > > >> > > > >> >> > >> >> > moving in that direction with all
>> > the
>> > > > > KIP-4
>> > > > > > > > > >> improvements.
>> > > > > > > > > >> > > > I
>> > > > > > > > > >> > > > >> chose
>> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as in this case the client
>> > > > will
>> > > > > > > still
>> > > > > > > > > >> just talk
>> > > > > > > > > >> > > > to
>> > > > > > > > > >> > > > >> >> > broker , its
>> > > > > > > > > >> > > > >> >> > >> >> > the brokers that will use zookeeper
>> > > > which
>> > > > > we
>> > > > > > > > > >> already do
>> > > > > > > > > >> > > > for a
>> > > > > > > > > >> > > > >> lot
>> > > > > > > > > >> > > > >> >> > of
>> > > > > > > > > >> > > > >> >> > >> >> > other
>> > > > > > > > > >> > > > >> >> > >> >> > usecases + ease of development + and
>> > > the
>> > > > > > > ability
>> > > > > > > > > so
>> > > > > > > > > >> > > tokens
>> > > > > > > > > >> > > > >> will
>> > > > > > > > > >> > > > >> >> > survive
>> > > > > > > > > >> > > > >> >> > >> >> > even a rolling restart/cluster
>> > > failure.
>> > > > > if a
>> > > > > > > > > >> majority
>> > > > > > > > > >> > > > agrees
>> > > > > > > > > >> > > > >> the
>> > > > > > > > > >> > > > >> >> > added
>> > > > > > > > > >> > > > >> >> > >> >> > complexity to have controller
>> > > forwarding
>> > > > > > keys
>> > > > > > > to
>> > > > > > > > > all
>> > > > > > > > > >> > > > broker is
>> > > > > > > > > >> > > > >> >> > justified
>> > > > > > > > > >> > > > >> >> > >> >> > as
>> > > > > > > > > >> > > > >> >> > >> >> > it provides tighter security , I am
>> > > fine
>> > > > > > with
>> > > > > > > > that
>> > > > > > > > > >> option
>> > > > > > > > > >> > > > too.
>> > > > > > > > > >> > > > >> >> > >> >> >
>> > > > > > > > > >> > > > >> >> > >> >> > Given zookeeper does not support SSL
>> > > we
>> > > > > can
>> > > > > > > not
>> > > > > > > > > >> store
>> > > > > > > > > >> > > > master
>> > > > > > > > > >> > > > >> keys
>> > > > > > > > > >> > > > >> >> > in
>> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as master keys will be
>> > > exposed
>> > > > > on
>> > > > > > > > wire.
>> > > > > > > > > To
>> > > > > > > > > >> > > > support
>> > > > > > > > > >> > > > >> >> > rotation
>> > > > > > > > > >> > > > >> >> > >> >> > without affecting current clients is
>> > > > > > > something I
>> > > > > > > > > >> need to
>> > > > > > > > > >> > > > put
>> > > > > > > > > >> > > > >> more
>> > > > > > > > > >> > > > >> >> > thought
>> > > > > > > > > >> > > > >> >> > >> >> > in. My current proposal assumes the
>> > > > > rotation
>> > > > > > > > will
>> > > > > > > > > >> > > > invalidate
>> > > > > > > > > >> > > > >> all
>> > > > > > > > > >> > > > >> >> > current
>> > > > > > > > > >> > > > >> >> > >> >> > tokens.
>> > > > > > > > > >> > > > >> >> > >> >> >
>> > > > > > > > > >> > > > >> >> > >> >> > I request committers to also review
>> > > and
>> > > > > post
>> > > > > > > > their
>> > > > > > > > > >> > > comments
>> > > > > > > > > >> > > > >> so we
>> > > > > > > > > >> > > > >> >> > can
>> > > > > > > > > >> > > > >> >> > >> >> > make
>> > > > > > > > > >> > > > >> >> > >> >> > progress on this KIP.
>> > > > > > > > > >> > > > >> >> > >> >> >
>> > > > > > > > > >> > > > >> >> > >> >> > Thanks
>> > > > > > > > > >> > > > >> >> > >> >> > Parth
>> > > > > > > > > >> > > > >> >> > >> >> >
>> > > > > > > > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM,
>> > > Ashish
>> > > > > > Singh
>> > > > > > > <
>> > > > > > > > > >> > > > >> asingh@cloudera.com
>> > > > > > > > > >> > > > >> >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > wrote:
>> > > > > > > > > >> > > > >> >> > >> >> >
>> > > > > > > > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM,
>> > > > > Harsha <
>> > > > > > > > > >> > > > kafka@harsha.io>
>> > > > > > > > > >> > > > >> >> > wrote:
>> > > > > > > > > >> > > > >> >> > >> >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > Unifying the two discussion
>> > > threads
>> > > > on
>> > > > > > > this
>> > > > > > > > > KIP.
>> > > > > > > > > >> > > > >> >> > >> >> > > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > Here is the response from
>> > Jitendra
>> > > > > > > > > >> > > > >> >> > >> >> > > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > "The need for a large number of
>> > > > > clients
>> > > > > > > that
>> > > > > > > > > are
>> > > > > > > > > >> > > > running
>> > > > > > > > > >> > > > >> all
>> > > > > > > > > >> > > > >> >> > over the
>> > > > > > > > > >> > > > >> >> > >> >> > > > cluster that authenticate with
>> > > Kafka
>> > > > > > > > brokers,
>> > > > > > > > > >> is very
>> > > > > > > > > >> > > > >> similar
>> > > > > > > > > >> > > > >> >> > to the
>> > > > > > > > > >> > > > >> >> > >> >> > > > Hadoop use case of large number
>> > of
>> > > > > tasks
>> > > > > > > > > running
>> > > > > > > > > >> > > across
>> > > > > > > > > >> > > > >> the
>> > > > > > > > > >> > > > >> >> > cluster
>> > > > > > > > > >> > > > >> >> > >> >> that
>> > > > > > > > > >> > > > >> >> > >> >> > > > need authentication to Hdfs
>> > > > Namenode.
>> > > > > > > > > >> Therefore, the
>> > > > > > > > > >> > > > >> >> > delegation token
>> > > > > > > > > >> > > > >> >> > >> >> > > > approach does seem like a good
>> > fit
>> > > > for
>> > > > > > > this
>> > > > > > > > > use
>> > > > > > > > > >> case
>> > > > > > > > > >> > > > as we
>> > > > > > > > > >> > > > >> >> > have seen
>> > > > > > > > > >> > > > >> >> > >> >> it
>> > > > > > > > > >> > > > >> >> > >> >> > > > working at large scale in HDFS
>> > and
>> > > > > YARN.
>> > > > > > > > > >> > > > >> >> > >> >> > > >
>> > > > > > > > > >> > > > >> >> > >> >> > > >   The proposed design is very
>> > much
>> > > > > > inline
>> > > > > > > > with
>> > > > > > > > > >> Hadoop
>> > > > > > > > > >> > > > >> >> > approach. A few
>> > > > > > > > > >> > > > >> >> > >> >> > > >   comments:
>> > > > > > > > > >> > > > >> >> > >> >> > > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > 1) Why do you guys want to allow
>> > > > > > infinite
>> > > > > > > > > >> renewable
>> > > > > > > > > >> > > > >> lifetime
>> > > > > > > > > >> > > > >> >> > for a
>> > > > > > > > > >> > > > >> >> > >> >> > > > token? HDFS restricts a token
>> > to a
>> > > > max
>> > > > > > > life
>> > > > > > > > > time
>> > > > > > > > > >> > > > (default
>> > > > > > > > > >> > > > >> 7
>> > > > > > > > > >> > > > >> >> > days).  A
>> > > > > > > > > >> > > > >> >> > >> >> > > > token's vulnerability is
>> > believed
>> > > to
>> > > > > > > > increase
>> > > > > > > > > >> with
>> > > > > > > > > >> > > > time.
>> > > > > > > > > >> > > > >> >> > >> >> > > >
>> > > > > > > > > >> > > > >> >> > >> >> > > I agree that having infinite
>> > > lifetime
>> > > > > > might
>> > > > > > > > not
>> > > > > > > > > >> be the
>> > > > > > > > > >> > > > best
>> > > > > > > > > >> > > > >> idea.
>> > > > > > > > > >> > > > >> >> > >> >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > 2) As I understand the tokens
>> > are
>> > > > > stored
>> > > > > > > in
>> > > > > > > > > >> zookeeper
>> > > > > > > > > >> > > > as
>> > > > > > > > > >> > > > >> well,
>> > > > > > > > > >> > > > >> >> > and
>> > > > > > > > > >> > > > >> >> > >> >> can
>> > > > > > > > > >> > > > >> >> > >> >> > > > be updated there. This is clever
>> > > as
>> > > > it
>> > > > > > can
>> > > > > > > > > allow
>> > > > > > > > > >> > > > >> replacing the
>> > > > > > > > > >> > > > >> >> > tokens
>> > > > > > > > > >> > > > >> >> > >> >> > > > once they run out of max life
>> > > time,
>> > > > > and
>> > > > > > > > > clients
>> > > > > > > > > >> can
>> > > > > > > > > >> > > > >> download
>> > > > > > > > > >> > > > >> >> > new
>> > > > > > > > > >> > > > >> >> > >> >> tokens
>> > > > > > > > > >> > > > >> >> > >> >> > > > from zookeeper. It shouldn't be
>> > a
>> > > > big
>> > > > > > load
>> > > > > > > > on
>> > > > > > > > > >> > > zookeeper
>> > > > > > > > > >> > > > >> as a
>> > > > > > > > > >> > > > >> >> > client
>> > > > > > > > > >> > > > >> >> > >> >> will
>> > > > > > > > > >> > > > >> >> > >> >> > > > need to get a new token once in
>> > > > > several
>> > > > > > > > days.
>> > > > > > > > > >> In this
>> > > > > > > > > >> > > > >> approach
>> > > > > > > > > >> > > > >> >> > you
>> > > > > > > > > >> > > > >> >> > >> >> don't
>> > > > > > > > > >> > > > >> >> > >> >> > > > need infinite lifetime on the
>> > > token
>> > > > > even
>> > > > > > > for
>> > > > > > > > > >> long
>> > > > > > > > > >> > > > running
>> > > > > > > > > >> > > > >> >> > clients.
>> > > > > > > > > >> > > > >> >> > >> >> > > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > 3) The token password are
>> > > generated
>> > > > > > using
>> > > > > > > a
>> > > > > > > > > >> master
>> > > > > > > > > >> > > key.
>> > > > > > > > > >> > > > >> The
>> > > > > > > > > >> > > > >> >> > master
>> > > > > > > > > >> > > > >> >> > >> >> key
>> > > > > > > > > >> > > > >> >> > >> >> > > > should also be periodically
>> > > changed.
>> > > > > In
>> > > > > > > > > Hadoop,
>> > > > > > > > > >> the
>> > > > > > > > > >> > > > >> default
>> > > > > > > > > >> > > > >> >> > renewal
>> > > > > > > > > >> > > > >> >> > >> >> > > > period is 1 day.?
>> > > > > > > > > >> > > > >> >> > >> >> > > >
>> > > > > > > > > >> > > > >> >> > >> >> > > IIUC, this will require brokers
>> > > > > > maintaining
>> > > > > > > a
>> > > > > > > > > >> list of X
>> > > > > > > > > >> > > > most
>> > > > > > > > > >> > > > >> >> > recent
>> > > > > > > > > >> > > > >> >> > >> >> master
>> > > > > > > > > >> > > > >> >> > >> >> > > keys. This list will have to be
>> > > > > persisted
>> > > > > > > > > >> somewhere, as
>> > > > > > > > > >> > > > if a
>> > > > > > > > > >> > > > >> >> > broker
>> > > > > > > > > >> > > > >> >> > >> >> goes
>> > > > > > > > > >> > > > >> >> > >> >> > > down it will have to get that list
>> > > > again
>> > > > > > and
>> > > > > > > > > >> storing
>> > > > > > > > > >> > > > master
>> > > > > > > > > >> > > > >> keys
>> > > > > > > > > >> > > > >> >> > on ZK
>> > > > > > > > > >> > > > >> >> > >> >> is
>> > > > > > > > > >> > > > >> >> > >> >> > > not the best idea. However, if a
>> > > > broker
>> > > > > > goes
>> > > > > > > > > down
>> > > > > > > > > >> then
>> > > > > > > > > >> > > we
>> > > > > > > > > >> > > > >> have
>> > > > > > > > > >> > > > >> >> > much
>> > > > > > > > > >> > > > >> >> > >> >> bigger
>> > > > > > > > > >> > > > >> >> > >> >> > > issue to deal with and client can
>> > > > always
>> > > > > > > > > >> > > re-authenticate
>> > > > > > > > > >> > > > is
>> > > > > > > > > >> > > > >> such
>> > > > > > > > > >> > > > >> >> > >> >> events.
>> > > > > > > > > >> > > > >> >> > >> >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > Did you happen to take a look at
>> > > other
>> > > > > > > > > >> alternatives
>> > > > > > > > > >> > > this
>> > > > > > > > > >> > > > >> list has
>> > > > > > > > > >> > > > >> >> > >> >> > > suggested?
>> > > > > > > > > >> > > > >> >> > >> >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > Thanks for a thorough proposal,
>> > > > great
>> > > > > > > work!"
>> > > > > > > > > >> > > > >> >> > >> >> > > >
>> > > > > > > > > >> > > > >> >> > >> >> > > >
>> > > > > > > > > >> > > > >> >> > >> >> > > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28
>> > PM,
>> > > > Gwen
>> > > > > > > > Shapira
>> > > > > > > > > >> wrote:
>> > > > > > > > > >> > > > >> >> > >> >> > > > > Makes sense to me. Thanks!
>> > > > > > > > > >> > > > >> >> > >> >> > > > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25
>> > PM,
>> > > > > > Harsha <
>> > > > > > > > > >> > > > kafka@harsha.io
>> > > > > > > > > >> > > > >> >
>> > > > > > > > > >> > > > >> >> > wrote:
>> > > > > > > > > >> > > > >> >> > >> >> > > > > > It doesn't need any release
>> > > > > vehicle
>> > > > > > > but
>> > > > > > > > > >> still the
>> > > > > > > > > >> > > > >> work can
>> > > > > > > > > >> > > > >> >> > move
>> > > > > > > > > >> > > > >> >> > >> >> > > > forward.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > > If anyone is interested in
>> > the
>> > > > KIP
>> > > > > > > > please
>> > > > > > > > > >> do the
>> > > > > > > > > >> > > > >> review and
>> > > > > > > > > >> > > > >> >> > >> >> provide
>> > > > > > > > > >> > > > >> >> > >> >> > > the
>> > > > > > > > > >> > > > >> >> > >> >> > > > > > comments.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > > -Harsha
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at
>> > 04:59
>> > > > PM,
>> > > > > > > Ismael
>> > > > > > > > > >> Juma
>> > > > > > > > > >> > > > wrote:
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> I agree that it would be
>> > good
>> > > > to
>> > > > > > have
>> > > > > > > > > more
>> > > > > > > > > >> time
>> > > > > > > > > >> > > to
>> > > > > > > > > >> > > > >> review
>> > > > > > > > > >> > > > >> >> > and
>> > > > > > > > > >> > > > >> >> > >> >> > > discuss
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> KIP-48.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> Ismael
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at
>> > 12:55
>> > > > AM,
>> > > > > > Gwen
>> > > > > > > > > >> Shapira <
>> > > > > > > > > >> > > > >> >> > >> >> gwen@confluent.io>
>> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Hi Team,
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Since KIP-48 depends on
>> > > > KIP-43,
>> > > > > > > which
>> > > > > > > > > is
>> > > > > > > > > >> > > > already a
>> > > > > > > > > >> > > > >> bit
>> > > > > > > > > >> > > > >> >> > of a
>> > > > > > > > > >> > > > >> >> > >> >> risk
>> > > > > > > > > >> > > > >> >> > >> >> > > for
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > the next release - any
>> > > chance
>> > > > > we
>> > > > > > > can
>> > > > > > > > > >> delay
>> > > > > > > > > >> > > > >> delegation
>> > > > > > > > > >> > > > >> >> > tokens
>> > > > > > > > > >> > > > >> >> > >> >> to
>> > > > > > > > > >> > > > >> >> > >> >> > > > Kafka
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > 0.10.1?
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > With the community
>> > working
>> > > > on a
>> > > > > > > > release
>> > > > > > > > > >> every
>> > > > > > > > > >> > > 3
>> > > > > > > > > >> > > > >> month,
>> > > > > > > > > >> > > > >> >> > this
>> > > > > > > > > >> > > > >> >> > >> >> is not
>> > > > > > > > > >> > > > >> >> > >> >> > > > a huge
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > delay.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Gwen
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at
>> > > 5:11
>> > > > > PM,
>> > > > > > > > Ashish
>> > > > > > > > > >> Singh
>> > > > > > > > > >> > > <
>> > > > > > > > > >> > > > >> >> > >> >> > > asingh@cloudera.com>
>> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Parth,
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Thanks again for the
>> > > > awesome
>> > > > > > > write
>> > > > > > > > > up.
>> > > > > > > > > >> > > > Following
>> > > > > > > > > >> > > > >> our
>> > > > > > > > > >> > > > >> >> > >> >> discussion
>> > > > > > > > > >> > > > >> >> > >> >> > > > from the
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I think it will
>> > be
>> > > > > easier
>> > > > > > > to
>> > > > > > > > > >> compare
>> > > > > > > > > >> > > > >> various
>> > > > > > > > > >> > > > >> >> > >> >> alternatives
>> > > > > > > > > >> > > > >> >> > >> >> > > > if they
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > are
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > listed together. I am
>> > > > stating
>> > > > > > > > below a
>> > > > > > > > > >> few
>> > > > > > > > > >> > > > >> >> > alternatives along
>> > > > > > > > > >> > > > >> >> > >> >> > > with
>> > > > > > > > > >> > > > >> >> > >> >> > > > a the
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > current proposal.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Current proposal)
>> > Store
>> > > > > > > Delegation
>> > > > > > > > > >> Token,
>> > > > > > > > > >> > > DT,
>> > > > > > > > > >> > > > >> on ZK.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
>> > > authenticates
>> > > > > > with a
>> > > > > > > > > >> broker.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
>> > > > > > > > authenticated,
>> > > > > > > > > >> it
>> > > > > > > > > >> > > will
>> > > > > > > > > >> > > > >> make a
>> > > > > > > > > >> > > > >> >> > broker
>> > > > > > > > > >> > > > >> >> > >> >> side
>> > > > > > > > > >> > > > >> >> > >> >> > > > call to
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation
>> > > > token.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker
>> > > generates
>> > > > a
>> > > > > > > shared
>> > > > > > > > > >> secret
>> > > > > > > > > >> > > > based
>> > > > > > > > > >> > > > >> on
>> > > > > > > > > >> > > > >> >> > >> >> > > HMAC-SHA256(a
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    Password/Secret
>> > shared
>> > > > > > between
>> > > > > > > > all
>> > > > > > > > > >> > > brokers,
>> > > > > > > > > >> > > > >> >> > randomly
>> > > > > > > > > >> > > > >> >> > >> >> > > generated
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > tokenId).
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Broker stores
>> > this
>> > > > > token
>> > > > > > in
>> > > > > > > > its
>> > > > > > > > > >> in
>> > > > > > > > > >> > > > memory
>> > > > > > > > > >> > > > >> cache.
>> > > > > > > > > >> > > > >> >> > >> >> Broker
>> > > > > > > > > >> > > > >> >> > >> >> > > > also stores
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the DelegationToken
>> > > > > without
>> > > > > > > the
>> > > > > > > > > >> hmac in
>> > > > > > > > > >> > > the
>> > > > > > > > > >> > > > >> >> > zookeeper.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. All brokers will
>> > > > have a
>> > > > > > > cache
>> > > > > > > > > >> backed
>> > > > > > > > > >> > > by
>> > > > > > > > > >> > > > >> >> > zookeeper so
>> > > > > > > > > >> > > > >> >> > >> >> they
>> > > > > > > > > >> > > > >> >> > >> >> > > > will all
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    get notified
>> > whenever
>> > > a
>> > > > > new
>> > > > > > > > token
>> > > > > > > > > is
>> > > > > > > > > >> > > > >> generated and
>> > > > > > > > > >> > > > >> >> > they
>> > > > > > > > > >> > > > >> >> > >> >> will
>> > > > > > > > > >> > > > >> >> > >> >> > > > update
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    local cache whenever
>> > > > token
>> > > > > > > state
>> > > > > > > > > >> changes.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Broker returns
>> > the
>> > > > > token
>> > > > > > to
>> > > > > > > > > >> Client.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and
>> > fixes
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Probable race
>> > > > > condition,
>> > > > > > > > client
>> > > > > > > > > >> tries
>> > > > > > > > > >> > > to
>> > > > > > > > > >> > > > >> >> > authenticate
>> > > > > > > > > >> > > > >> >> > >> >> with
>> > > > > > > > > >> > > > >> >> > >> >> > > > a broker
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    that is yet to be
>> > > > updated
>> > > > > > with
>> > > > > > > > the
>> > > > > > > > > >> newly
>> > > > > > > > > >> > > > >> generated
>> > > > > > > > > >> > > > >> >> > DT.
>> > > > > > > > > >> > > > >> >> > >> >> This
>> > > > > > > > > >> > > > >> >> > >> >> > > can
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > probably be
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    dealt with making
>> > > > > dtRequest
>> > > > > > > > block
>> > > > > > > > > >> until
>> > > > > > > > > >> > > all
>> > > > > > > > > >> > > > >> >> > brokers have
>> > > > > > > > > >> > > > >> >> > >> >> > > > updated
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their DT
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    cache. Zk barrier or
>> > > > > similar
>> > > > > > > > > >> mechanism
>> > > > > > > > > >> > > can
>> > > > > > > > > >> > > > be
>> > > > > > > > > >> > > > >> used.
>> > > > > > > > > >> > > > >> >> > >> >> However,
>> > > > > > > > > >> > > > >> >> > >> >> > > > all such
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    mechanisms will
>> > > increase
>> > > > > > > > > complexity.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Using a static
>> > > secret
>> > > > > key
>> > > > > > > > from
>> > > > > > > > > >> config
>> > > > > > > > > >> > > > >> file. Will
>> > > > > > > > > >> > > > >> >> > >> >> require
>> > > > > > > > > >> > > > >> >> > >> >> > > yet
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > another
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    config and uses a
>> > > static
>> > > > > > > secret
>> > > > > > > > > >> key. It
>> > > > > > > > > >> > > is
>> > > > > > > > > >> > > > >> advised
>> > > > > > > > > >> > > > >> >> > to
>> > > > > > > > > >> > > > >> >> > >> >> rotate
>> > > > > > > > > >> > > > >> >> > >> >> > > > secret
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > keys
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    periodically. This
>> > can
>> > > > be
>> > > > > > > > avoided
>> > > > > > > > > >> with
>> > > > > > > > > >> > > > >> controller
>> > > > > > > > > >> > > > >> >> > >> >> generating
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > secretKey and
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    passing to brokers
>> > > > > > > periodically.
>> > > > > > > > > >> However,
>> > > > > > > > > >> > > > >> this will
>> > > > > > > > > >> > > > >> >> > >> >> require
>> > > > > > > > > >> > > > >> >> > >> >> > > > brokers to
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maintain certain
>> > > counts
>> > > > of
>> > > > > > > > > >> secretKeys.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 1) Have
>> > > > > controller
>> > > > > > > > > >> generate
>> > > > > > > > > >> > > > >> delegation
>> > > > > > > > > >> > > > >> >> > token.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
>> > > authenticates
>> > > > > > with a
>> > > > > > > > > >> broker.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
>> > > > > > > > authenticated,
>> > > > > > > > > >> it
>> > > > > > > > > >> > > will
>> > > > > > > > > >> > > > >> make a
>> > > > > > > > > >> > > > >> >> > broker
>> > > > > > > > > >> > > > >> >> > >> >> side
>> > > > > > > > > >> > > > >> >> > >> >> > > > call to
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation
>> > > > token.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. Broker forwards
>> > the
>> > > > > > request
>> > > > > > > > to
>> > > > > > > > > >> > > > controller.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Controller
>> > > generates
>> > > > a
>> > > > > DT
>> > > > > > > and
>> > > > > > > > > >> > > broadcasts
>> > > > > > > > > >> > > > >> to all
>> > > > > > > > > >> > > > >> >> > >> >> brokers.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. Broker stores
>> > this
>> > > > > token
>> > > > > > in
>> > > > > > > > its
>> > > > > > > > > >> memory
>> > > > > > > > > >> > > > >> cache.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Controller
>> > responds
>> > > > to
>> > > > > > > > broker’s
>> > > > > > > > > >> DT
>> > > > > > > > > >> > > req.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    7. Broker returns
>> > the
>> > > > > token
>> > > > > > to
>> > > > > > > > > >> Client.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and
>> > fixes
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. We will have to
>> > add
>> > > > new
>> > > > > > > APIs
>> > > > > > > > to
>> > > > > > > > > >> > > support
>> > > > > > > > > >> > > > >> >> > controller
>> > > > > > > > > >> > > > >> >> > >> >> pushing
>> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > to
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    brokers on top of
>> > the
>> > > > > > minimal
>> > > > > > > > APIs
>> > > > > > > > > >> that
>> > > > > > > > > >> > > are
>> > > > > > > > > >> > > > >> >> > currently
>> > > > > > > > > >> > > > >> >> > >> >> > > proposed.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. We will also have
>> > > to
>> > > > > add
>> > > > > > > APIs
>> > > > > > > > > to
>> > > > > > > > > >> > > support
>> > > > > > > > > >> > > > >> the
>> > > > > > > > > >> > > > >> >> > >> >> bootstrapping
>> > > > > > > > > >> > > > >> >> > >> >> > > > case,
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > i.e,
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    when a new broker
>> > > comes
>> > > > up
>> > > > > > it
>> > > > > > > > will
>> > > > > > > > > >> have
>> > > > > > > > > >> > > to
>> > > > > > > > > >> > > > >> get all
>> > > > > > > > > >> > > > >> >> > >> >> delegation
>> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > from
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the controller.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. In catastrophic
>> > > > > failures
>> > > > > > > > where
>> > > > > > > > > >> all
>> > > > > > > > > >> > > > brokers
>> > > > > > > > > >> > > > >> go
>> > > > > > > > > >> > > > >> >> > down,
>> > > > > > > > > >> > > > >> >> > >> >> the
>> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if
>> > > servers
>> > > > > are
>> > > > > > > > > >> restarted as
>> > > > > > > > > >> > > > >> tokens
>> > > > > > > > > >> > > > >> >> > are not
>> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this happens,
>> > then
>> > > > > there
>> > > > > > > are
>> > > > > > > > > more
>> > > > > > > > > >> > > > important
>> > > > > > > > > >> > > > >> >> > things to
>> > > > > > > > > >> > > > >> >> > >> >> > > worry
>> > > > > > > > > >> > > > >> >> > >> >> > > > about
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is better
>> > to
>> > > > > > > > > >> re-authenticate.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 2) Do not
>> > > > > > distribute
>> > > > > > > > DT
>> > > > > > > > > to
>> > > > > > > > > >> > > other
>> > > > > > > > > >> > > > >> brokers
>> > > > > > > > > >> > > > >> >> > at
>> > > > > > > > > >> > > > >> >> > >> >> all.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
>> > > authenticates
>> > > > > > with a
>> > > > > > > > > >> broker.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
>> > > > > > > > authenticated,
>> > > > > > > > > >> it
>> > > > > > > > > >> > > will
>> > > > > > > > > >> > > > >> make a
>> > > > > > > > > >> > > > >> >> > broker
>> > > > > > > > > >> > > > >> >> > >> >> side
>> > > > > > > > > >> > > > >> >> > >> >> > > > call to
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation
>> > > > token.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker
>> > > generates
>> > > > DT
>> > > > > > of
>> > > > > > > > > form,
>> > > > > > > > > >> > > [hmac +
>> > > > > > > > > >> > > > >> (owner,
>> > > > > > > > > >> > > > >> >> > >> >> renewer,
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maxLifeTime, id,
>> > hmac,
>> > > > > > > > > >> expirationTime)]
>> > > > > > > > > >> > > and
>> > > > > > > > > >> > > > >> passes
>> > > > > > > > > >> > > > >> >> > back
>> > > > > > > > > >> > > > >> >> > >> >> this
>> > > > > > > > > >> > > > >> >> > >> >> > > > DT to
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > client.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    hmac is generated
>> > via
>> > > > > > > > > >> {HMAC-SHA256(owner,
>> > > > > > > > > >> > > > >> renewer,
>> > > > > > > > > >> > > > >> >> > >> >> > > > maxLifeTime, id,
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac,
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    expirationTime)
>> > using
>> > > > > > > > SecretKey}.
>> > > > > > > > > >> Note
>> > > > > > > > > >> > > that
>> > > > > > > > > >> > > > >> all
>> > > > > > > > > >> > > > >> >> > brokers
>> > > > > > > > > >> > > > >> >> > >> >> have
>> > > > > > > > > >> > > > >> >> > >> >> > > > this
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > SecretKey.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Client then goes
>> > to
>> > > > any
>> > > > > > > > broker
>> > > > > > > > > >> and to
>> > > > > > > > > >> > > > >> >> > authenticate
>> > > > > > > > > >> > > > >> >> > >> >> sends
>> > > > > > > > > >> > > > >> >> > >> >> > > > the DT.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    Broker recalculates
>> > > hmac
>> > > > > > using
>> > > > > > > > > >> (owner,
>> > > > > > > > > >> > > > >> renewer,
>> > > > > > > > > >> > > > >> >> > >> >> maxLifeTime,
>> > > > > > > > > >> > > > >> >> > >> >> > > > id, hmac,
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    expirationTime) info
>> > > > from
>> > > > > DT
>> > > > > > > and
>> > > > > > > > > its
>> > > > > > > > > >> > > > >> SecretKey. If
>> > > > > > > > > >> > > > >> >> > it
>> > > > > > > > > >> > > > >> >> > >> >> matches
>> > > > > > > > > >> > > > >> >> > >> >> > > > with
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac of
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    DT, client is
>> > > > > authenticated.
>> > > > > > > > Yes,
>> > > > > > > > > >> it will
>> > > > > > > > > >> > > > do
>> > > > > > > > > >> > > > >> other
>> > > > > > > > > >> > > > >> >> > >> >> obvious
>> > > > > > > > > >> > > > >> >> > >> >> > > > checks of
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    timestamp expiry and
>> > > > such.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Note that secret key
>> > will
>> > > > be
>> > > > > > > > > generated
>> > > > > > > > > >> by
>> > > > > > > > > >> > > > >> controller
>> > > > > > > > > >> > > > >> >> > and
>> > > > > > > > > >> > > > >> >> > >> >> passed
>> > > > > > > > > >> > > > >> >> > >> >> > > to
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > brokers
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > periodically.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and
>> > fixes
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. How to delete a
>> > DT?
>> > > > > Yes,
>> > > > > > > that
>> > > > > > > > > is
>> > > > > > > > > >> a
>> > > > > > > > > >> > > > downside
>> > > > > > > > > >> > > > >> >> > here.
>> > > > > > > > > >> > > > >> >> > >> >> However,
>> > > > > > > > > >> > > > >> >> > >> >> > > > this can
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be handled with
>> > > brokers
>> > > > > > > > > maintaining
>> > > > > > > > > >> a
>> > > > > > > > > >> > > > >> blacklist of
>> > > > > > > > > >> > > > >> >> > DTs,
>> > > > > > > > > >> > > > >> >> > >> >> DTs
>> > > > > > > > > >> > > > >> >> > >> >> > > > from this
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > list
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    can be removed after
>> > > > > expiry.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. In catastrophic
>> > > > > failures
>> > > > > > > > where
>> > > > > > > > > >> all
>> > > > > > > > > >> > > > brokers
>> > > > > > > > > >> > > > >> go
>> > > > > > > > > >> > > > >> >> > down,
>> > > > > > > > > >> > > > >> >> > >> >> the
>> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if
>> > > servers
>> > > > > are
>> > > > > > > > > >> restarted as
>> > > > > > > > > >> > > > >> tokens
>> > > > > > > > > >> > > > >> >> > are not
>> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this happens,
>> > then
>> > > > > there
>> > > > > > > are
>> > > > > > > > > more
>> > > > > > > > > >> > > > important
>> > > > > > > > > >> > > > >> >> > things to
>> > > > > > > > > >> > > > >> >> > >> >> > > worry
>> > > > > > > > > >> > > > >> >> > >> >> > > > about
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is better
>> > to
>> > > > > > > > > >> re-authenticate.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at
>> > > > 1:58
>> > > > > > PM,
>> > > > > > > > > Parth
>> > > > > > > > > >> > > > >> Brahmbhatt <
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > pbrahmbhatt@hortonworks.com>
>> > > > > > > > wrote:
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Hi,
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> I have filed KIP-48 so
>> > > we
>> > > > > can
>> > > > > > > > offer
>> > > > > > > > > >> hadoop
>> > > > > > > > > >> > > > like
>> > > > > > > > > >> > > > >> >> > delegation
>> > > > > > > > > >> > > > >> >> > >> >> > > > tokens in
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> kafka. You can review
>> > > the
>> > > > > > design
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
>> > > > > > > > > >> > > > >> >> > >> >> > > >
>> > > > > > > > > >> > > > >> >> > >> >> > >
>> > > > > > > > > >> > > > >> >> > >> >>
>> > > > > > > > > >> > > > >> >> >
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > >
>> > > > > > > > > >>
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > .
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> This KIP depends on
>> > > KIP-43
>> > > > > and
>> > > > > > > we
>> > > > > > > > > >> have also
>> > > > > > > > > >> > > > >> >> > discussed an
>> > > > > > > > > >> > > > >> >> > >> >> > > > alternative to
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> proposed design here<
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
>> > > > > > > > > >> > > > >> >> > >> >> > > >
>> > > > > > > > > >> > > > >> >> > >> >> > >
>> > > > > > > > > >> > > > >> >> > >> >>
>> > > > > > > > > >> > > > >> >> >
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > >
>> > > > > > > > > >>
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >.
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Thanks
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Parth
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > --
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Regards,
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Ashish
>> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
>> > > > > > > > > >> > > > >> >> > >> >> > > >
>> > > > > > > > > >> > > > >> >> > >> >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > --
>> > > > > > > > > >> > > > >> >> > >> >> > >
>> > > > > > > > > >> > > > >> >> > >> >> > > Regards,
>> > > > > > > > > >> > > > >> >> > >> >> > > Ashish
>> > > > > > > > > >> > > > >> >> > >> >> > >
>> > > > > > > > > >> > > > >> >> > >> >>
>> > > > > > > > > >> > > > >> >> >
>> > > > > > > > > >> > > > >>
>> > > > > > > > > >> > > >
>> > > > > > > > > >> > >
>> > > > > > > > > >>
>> > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Regards,
>> > > > > >
>> > > > > > Rajini
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Regards,
>> > > >
>> > > > Rajini
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> Liquan Pei
>> Software Engineer, Confluent Inc

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Harsha <ka...@harsha.io>]

Jun & Ismael,
                         Unfortunately I couldn't attend the KIP meeting
                         when delegation tokens discussed. Appreciate if
                         you can update the thread if you have any
                         further questions.
Thanks,
Harsha

On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
> It seems that the links to images in the KIP are broken.
> 
> Liquan
> 
> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
> brahmbhatt.parth@gmail.com> wrote:
> 
> > 110. What does getDelegationTokenAs mean?
> > In the current proposal we only allow a user to get delegation token for
> > the identity that it authenticated as using another mechanism, i.e. A user
> > that authenticate using a keytab for principal user1@EXAMPLE.COM will get
> > delegation tokens for that user only. In future I think we will have to
> > extend support such that we allow some set of users (
> > kafka-rest-user@EXAMPLE.COM, storm-nimbus@EXAMPLE.COM) to acquire
> > delegation tokens on behalf of other users whose identity they have
> > verified independently.  Kafka brokers will have ACLs to control which
> > users are allowed to impersonate other users and get tokens on behalf of
> > them. Overall Impersonation is a whole different problem in my opinion and
> > I think we can tackle it in separate KIP.
> >
> > 111. What's the typical rate of getting and renewing delegation tokens?
> > Typically this should be very very low, 1 request per minute is a
> > relatively high estimate. However it depends on the token expiration. I am
> > less worried about the extra load it puts on controller vs the added
> > complexity and the value it offers.
> >
> > Thanks
> > Parth
> >
> >
> >
> > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Thanks Rajini. It would probably require a separate KIP as it will
> > > introduce user visible changes. We could also update KIP-48 to have this
> > > information, but it seems cleaner to do it separately. We can discuss
> > that
> > > in the KIP call today.
> > >
> > > Ismael
> > >
> > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <
> > > rajinisivaram@googlemail.com> wrote:
> > >
> > > > Ismael,
> > > >
> > > > I have created a JIRA (
> > https://issues.apache.org/jira/browse/KAFKA-3751)
> > > > for adding SCRAM as a SASL mechanism. Would that need another KIP? If
> > > > KIP-48 will use this mechanism, can this just be a JIRA that gets
> > > reviewed
> > > > when the PR is ready?
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > >
> > > > > Thanks Rajini, SCRAM seems like a good candidate.
> > > > >
> > > > > Gwen had independently mentioned this as a SASL mechanism that might
> > be
> > > > > useful for Kafka and I have been meaning to check it in more detail.
> > > Good
> > > > > to know that you are willing to contribute an implementation. Maybe
> > we
> > > > > should file a separate JIRA for this?
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram <
> > > > > rajinisivaram@googlemail.com> wrote:
> > > > >
> > > > > > SCRAM (Salted Challenge Response Authentication Mechanism) is a
> > > better
> > > > > > mechanism than Digest-MD5. Java doesn't come with a built-in SCRAM
> > > > > > SaslServer or SaslClient, but I will be happy to add support in
> > Kafka
> > > > > since
> > > > > > it would be a useful mechanism to support anyway.
> > > > > > https://tools.ietf.org/html/rfc7677 describes the protocol for
> > > > > > SCRAM-SHA-256.
> > > > > >
> > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <ju...@confluent.io> wrote:
> > > > > >
> > > > > > > Parth,
> > > > > > >
> > > > > > > Thanks for the explanation. A couple of more questions.
> > > > > > >
> > > > > > > 110. What does getDelegationTokenAs mean?
> > > > > > >
> > > > > > > 111. What's the typical rate of getting and renewing delegation
> > > > tokens?
> > > > > > > That may have an impact on whether they should be directed to the
> > > > > > > controller.
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt <
> > > > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > > > >
> > > > > > > > Hi Jun,
> > > > > > > >
> > > > > > > > Thanks for reviewing.
> > > > > > > >
> > > > > > > > * We could add a Cluster action to add acls on who can request
> > > > > > delegation
> > > > > > > > tokens. I don't see the use case for that yet but down the line
> > > > when
> > > > > we
> > > > > > > > start supporting getDelegationTokenAs it will be necessary.
> > > > > > > > * Yes we recommend tokens to be only used/distributed over
> > secure
> > > > > > > channels.
> > > > > > > > * Depending on what design we end up choosing Invalidation will
> > > be
> > > > > > > > responsibility of every broker or controller.
> > > > > > > > * I am not sure if I documented somewhere that invalidation
> > will
> > > > > > directly
> > > > > > > > go through zookeeper but that is not the intent. Invalidation
> > > will
> > > > > > either
> > > > > > > > be request based or due to expiration. No direct zookeeper
> > > > > interaction
> > > > > > > from
> > > > > > > > any client.
> > > > > > > > * "Broker also stores the DelegationToken without the hmac in
> > the
> > > > > > > > zookeeper." : Sorry about the confusion. The sole purpose of
> > > > > zookeeper
> > > > > > in
> > > > > > > > this design is as distribution channel for tokens between all
> > > > brokers
> > > > > > > and a
> > > > > > > > layer that ensures only tokens that were generated by making a
> > > > > request
> > > > > > > to a
> > > > > > > > broker will be accepted (more on this in second paragraph). The
> > > > token
> > > > > > > > consists of few elements (owner, renewer, uuid , expiration,
> > > hmac)
> > > > ,
> > > > > > one
> > > > > > > of
> > > > > > > > which is the finally generated hmac but hmac it self is
> > derivable
> > > > if
> > > > > > you
> > > > > > > > have all the other elements of the token + secret key to
> > generate
> > > > > hmac.
> > > > > > > > Given zookeeper does not provide SSL support we do not want the
> > > > > entire
> > > > > > > > token to be wire transferred to zookeeper as that will be an
> > > > insecure
> > > > > > > wire
> > > > > > > > transfer. Instead we only store all the other elements of a
> > > > > delegation
> > > > > > > > tokens. Brokers can read these elements and because they also
> > > have
> > > > > > access
> > > > > > > > to secret key they will be able to generate hmac on their end.
> > > > > > > >
> > > > > > > > One of the alternative proposed is to avoid zookeeper
> > > altogether. A
> > > > > > > Client
> > > > > > > > will call broker with required information (owner, renwer,
> > > > > expiration)
> > > > > > > and
> > > > > > > > get back (signed hmac, uuid). Broker won't store this in
> > > zookeeper.
> > > > > > From
> > > > > > > > this point a client can contact any broker with all the
> > > delegation
> > > > > > token
> > > > > > > > info (owner, rewner, expiration, hmac, uuid) the borker will
> > > > > regenerate
> > > > > > > the
> > > > > > > > hmac and as long as it matches with hmac presented by client ,
> > > > broker
> > > > > > > will
> > > > > > > > allow the request to authenticate.  Only problem with this
> > > approach
> > > > > is
> > > > > > if
> > > > > > > > the secret key is compromised any client can now generate
> > random
> > > > > tokens
> > > > > > > and
> > > > > > > > they will still be able to authenticate as any user they like.
> > > with
> > > > > > > > zookeeper we guarantee that only tokens acquired via a broker
> > > > (using
> > > > > > some
> > > > > > > > auth scheme other than delegation token) will be accepted. We
> > > need
> > > > to
> > > > > > > > discuss which proposal makes more sense and we can go over it
> > in
> > > > > > > tomorrow's
> > > > > > > > meeting.
> > > > > > > >
> > > > > > > > Also, can you forward the invite to me?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Parth
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <ju...@confluent.io>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks for the KIP. A few comments.
> > > > > > > > >
> > > > > > > > > 100. This potentially can be useful for Kafka Connect and
> > Kafka
> > > > > rest
> > > > > > > > proxy
> > > > > > > > > where a worker agent will need to run a task on behalf of a
> > > > client.
> > > > > > We
> > > > > > > > will
> > > > > > > > > likely need to change how those services use Kafka clients
> > > > > > > > > (producer/consumer). Instead of a shared client per worker,
> > we
> > > > will
> > > > > > > need
> > > > > > > > a
> > > > > > > > > client per user task since the authentication happens at the
> > > > > > connection
> > > > > > > > > level. For Kafka Connect, the renewer will be the workers.
> > So,
> > > we
> > > > > > > > probably
> > > > > > > > > need to allow multiple renewers. For Kafka rest proxy, the
> > > > renewer
> > > > > > can
> > > > > > > > > probably just be the creator of the token.
> > > > > > > > >
> > > > > > > > > 101. Do we need new acl on who can request delegation tokens?
> > > > > > > > >
> > > > > > > > > 102. Do we recommend people to send delegation tokens in an
> > > > > encrypted
> > > > > > > > > channel?
> > > > > > > > >
> > > > > > > > > 103. Who is responsible for expiring tokens, every broker?
> > > > > > > > >
> > > > > > > > > 104. For invalidating tokens, would it be better to do it in
> > a
> > > > > > request
> > > > > > > > > instead of going to ZK directly?
> > > > > > > > >
> > > > > > > > > 105. The terminology of client in the wiki sometimes refers
> > to
> > > > the
> > > > > > end
> > > > > > > > > client and some other times refers to the client using the
> > > > > delegation
> > > > > > > > > tokens. It would be useful to distinguish between the two.
> > > > > > > > >
> > > > > > > > > 106. Could you explain the sentence "Broker also stores the
> > > > > > > > DelegationToken
> > > > > > > > > without the hmac in the zookeeper." a bit more? I thought the
> > > > > > > delegation
> > > > > > > > > token is the hmac.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <ju...@confluent.io>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi, Harsha,
> > > > > > > > > >
> > > > > > > > > > Just sent out a KIP meeting invite. We can discuss this in
> > > the
> > > > > > > meeting
> > > > > > > > > > tomorrow.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <ka...@harsha.io>
> > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi All,
> > > > > > > > > >>            Can we have a KIP meeting around this. The KIP
> > is
> > > > up
> > > > > > for
> > > > > > > > > >>            sometime and if there are any questions lets
> > > > quickly
> > > > > > hash
> > > > > > > > out
> > > > > > > > > >>            details.
> > > > > > > > > >>
> > > > > > > > > >> Thanks,
> > > > > > > > > >> Harsha
> > > > > > > > > >>
> > > > > > > > > >> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt wrote:
> > > > > > > > > >> > That is what the hadoop echo system uses so no good
> > reason
> > > > > > really.
> > > > > > > > We
> > > > > > > > > >> > could
> > > > > > > > > >> > change it to whatever is the newest recommended standard
> > > is.
> > > > > > > > > >> >
> > > > > > > > > >> > Thanks
> > > > > > > > > >> > Parth
> > > > > > > > > >> >
> > > > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <
> > > > > ismael@juma.me.uk
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > > Hi Parth,
> > > > > > > > > >> > >
> > > > > > > > > >> > > Thanks for the KIP. I only started reviewing this and
> > > may
> > > > > have
> > > > > > > > > >> additional
> > > > > > > > > >> > > questions later. The immediate question that came to
> > > mind
> > > > is
> > > > > > our
> > > > > > > > > >> choice of
> > > > > > > > > >> > > "DIGEST-MD5" even though it's marked as OBSOLETE in
> > the
> > > > IANA
> > > > > > > > > Registry
> > > > > > > > > >> of
> > > > > > > > > >> > > SASL mechanisms and the original RFC (2831) has been
> > > moved
> > > > > to
> > > > > > > > > Historic
> > > > > > > > > >> > > status:
> > > > > > > > > >> > >
> > > > > > > > > >> > > https://tools.ietf.org/html/rfc6331
> > > > > > > > > >> > >
> > > > > > > > >
> > > > > >
> > > http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> > > > > > > > > >> > >
> > > > > > > > > >> > > What is the reasoning behind that choice?
> > > > > > > > > >> > >
> > > > > > > > > >> > > Thanks,
> > > > > > > > > >> > > Ismael
> > > > > > > > > >> > >
> > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <
> > > > > > > gwen@confluent.io
> > > > > > > > >
> > > > > > > > > >> wrote:
> > > > > > > > > >> > >
> > > > > > > > > >> > > > Also comments inline :)
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > > * I want to emphasize that even though delegation
> > > > tokens
> > > > > > > are a
> > > > > > > > > >> Hadoop
> > > > > > > > > >> > > > > innovation, I feel very strongly about not adding
> > > > > > dependency
> > > > > > > > on
> > > > > > > > > >> Hadoop
> > > > > > > > > >> > > > > when implementing delegation tokens for Kafka. The
> > > KIP
> > > > > > > doesn't
> > > > > > > > > >> imply
> > > > > > > > > >> > > > > such dependency, but if you can clarify...
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > *No hadoop dependency.*
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Yay! Just add this to the KIP so no one will read
> > the
> > > > KIP
> > > > > > and
> > > > > > > > > panic
> > > > > > > > > >> > > > three weeks before the next release...
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > > * Can we get delegation token at any time after
> > > > > > > > authenticating?
> > > > > > > > > >> only
> > > > > > > > > >> > > > > immediately after?
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > *As long as you are authenticated you can get
> > > > delegation
> > > > > > > > tokens.
> > > > > > > > > >> We
> > > > > > > > > >> > > need
> > > > > > > > > >> > > > to
> > > > > > > > > >> > > > > discuss if a client authenticated using delegation
> > > > > token,
> > > > > > > can
> > > > > > > > > also
> > > > > > > > > >> > > > acquire
> > > > > > > > > >> > > > > delegation token again or not. Also there is the
> > > > > question
> > > > > > of
> > > > > > > > do
> > > > > > > > > we
> > > > > > > > > >> > > allow
> > > > > > > > > >> > > > > anyone to acquire delegation token or we want
> > > specific
> > > > > > ACLs
> > > > > > > (I
> > > > > > > > > >> think
> > > > > > > > > >> > > its
> > > > > > > > > >> > > > an
> > > > > > > > > >> > > > > overkill.)*
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > I agree that ACLs is an overkill.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > I think we are debating two options: Either require
> > > > > Kerberos
> > > > > > > > auth
> > > > > > > > > >> for
> > > > > > > > > >> > > > renewal or require non-owners to renew.
> > > > > > > > > >> > > > I *think* the latter is simpler (it basically
> > require
> > > a
> > > > > "job
> > > > > > > > > master"
> > > > > > > > > >> > > > to take responsibility for the renewal, it will have
> > > its
> > > > > own
> > > > > > > > > >> identity
> > > > > > > > > >> > > > anyway and I think this is the correct design
> > pattern
> > > > > > anyway.
> > > > > > > > For
> > > > > > > > > >> > > > storm, I'd expect Nimbus to coordinate renewals?),
> > but
> > > > it
> > > > > is
> > > > > > > > hard
> > > > > > > > > to
> > > > > > > > > >> > > > debate simplicity without looking at the code
> > changes
> > > > > > > required.
> > > > > > > > If
> > > > > > > > > >> you
> > > > > > > > > >> > > > have a draft of how the "require Kerberos" will look
> > > in
> > > > > > Kafka
> > > > > > > > > code,
> > > > > > > > > >> > > > I'll be happy to take a look.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > > * My understanding is that tokens will propagate
> > via
> > > > ZK
> > > > > > but
> > > > > > > > > >> without
> > > > > > > > > >> > > > > additional changes to UpdateMetadata protocol,
> > > > correct?
> > > > > > > > Clients
> > > > > > > > > >> > > > > currently don't retry on SASL auth failure (IIRC),
> > > but
> > > > > > since
> > > > > > > > the
> > > > > > > > > >> > > > > tokens propagate between brokers asynch, we will
> > > need
> > > > to
> > > > > > > > retry a
> > > > > > > > > >> bit
> > > > > > > > > >> > > > > to avoid clients failing auth due to timing
> > issues.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > *I am considering 2 alternatives right now. The
> > > > current
> > > > > > > > > documented
> > > > > > > > > >> > > > approach
> > > > > > > > > >> > > > > is zookeeper based and it does not require any
> > > changes
> > > > > to
> > > > > > > > > >> > > UpdateMetadata
> > > > > > > > > >> > > > > protocol. An alternative approach can remove
> > > zookeeper
> > > > > > > > > dependency
> > > > > > > > > >> as
> > > > > > > > > >> > > well
> > > > > > > > > >> > > > > but we can discuss that in KIP discussion call.*
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Oooh! Sounds interesting. Do you want to ping Jun to
> > > > > > arrange a
> > > > > > > > > call?
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > > * I liked Ashish's suggestion of having just the
> > > > > > controller
> > > > > > > > > issue
> > > > > > > > > >> the
> > > > > > > > > >> > > > > delegation tokens, to avoid syncing a shared
> > secret.
> > > > Not
> > > > > > > sure
> > > > > > > > if
> > > > > > > > > >> we
> > > > > > > > > >> > > > > want to continue the discussion here or on the
> > > wiki. I
> > > > > > think
> > > > > > > > > that
> > > > > > > > > >> we
> > > > > > > > > >> > > > > can decouple the problem of "token distribution"
> > > from
> > > > > > > "shared
> > > > > > > > > >> secret
> > > > > > > > > >> > > > > distribution" and use the controller as the only
> > > token
> > > > > > > > generator
> > > > > > > > > >> to
> > > > > > > > > >> > > > > solve the second issue, while still using ZK async
> > > to
> > > > > > > > distribute
> > > > > > > > > >> > > > > tokens.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > *As mentioned in the previous Email I am fine with
> > > > that
> > > > > > > > approach
> > > > > > > > > >> as
> > > > > > > > > >> > > long
> > > > > > > > > >> > > > as
> > > > > > > > > >> > > > > we agree that the extra complexity of
> > > adding/updating
> > > > > APIS
> > > > > > > > adds
> > > > > > > > > >> enough
> > > > > > > > > >> > > > > value. The advantage with the controller approach
> > is
> > > > > > secret
> > > > > > > > > >> rotation
> > > > > > > > > >> > > can
> > > > > > > > > >> > > > be
> > > > > > > > > >> > > > > automated,frequent and would not require
> > > deployment. *
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Can you detail the extra complexity (or point me to
> > > the
> > > > > > email
> > > > > > > I
> > > > > > > > > >> > > > missed?) - which Apis are required?
> > > > > > > > > >> > > > As far as I can tell, clients can already find the
> > > > > > controller
> > > > > > > > from
> > > > > > > > > >> > > > metadata. I'm a bit more concerned about controller
> > > > load.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > * While I like the idea of forcing kerberos auth
> > for
> > > > > > > renwal, I
> > > > > > > > > >> think
> > > > > > > > > >> > > > > it mixes the transport layer the the request
> > content
> > > > in
> > > > > a
> > > > > > > > pretty
> > > > > > > > > >> ugly
> > > > > > > > > >> > > > > way. Perhaps limiting renewer to non-owner is
> > > better.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > *I feel this is a necessary evil. While this will
> > > make
> > > > > the
> > > > > > > > kafka
> > > > > > > > > >> code
> > > > > > > > > >> > > > > pretty straight forward , forcing  renewer to
> > > > non-owner
> > > > > > > pushes
> > > > > > > > > >> the code
> > > > > > > > > >> > > > > ugliness to client and makes it even harder to
> > > > > > integrate.  *
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > As mentioned before, I don't think the "renewal by
> > > > other"
> > > > > > > > approach
> > > > > > > > > >> is
> > > > > > > > > >> > > > that ugly for the clients we expect to use
> > delegation
> > > > > tokens
> > > > > > > > since
> > > > > > > > > >> > > > they will have an app-master of some sort who
> > > requested
> > > > > the
> > > > > > > > token
> > > > > > > > > to
> > > > > > > > > >> > > > begin with.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > The response for my question on how multiple
> > > > identities
> > > > > > will
> > > > > > > > be
> > > > > > > > > >> > > > > handled wasn't super clear to me - AFAIK, we don't
> > > > > > > > authenticate
> > > > > > > > > >> each
> > > > > > > > > >> > > > > request, we authenticate connections.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > *We authenticate connections, and only when they
> > are
> > > > > being
> > > > > > > > > >> established.
> > > > > > > > > >> > > > Let
> > > > > > > > > >> > > > > me try to phrase this as a question, in absence of
> > > > > > > delegation
> > > > > > > > > >> tokens if
> > > > > > > > > >> > > > we
> > > > > > > > > >> > > > > had to support the use case using user TGT's how
> > > would
> > > > > we
> > > > > > do
> > > > > > > > it?
> > > > > > > > > >> My
> > > > > > > > > >> > > point
> > > > > > > > > >> > > > > was it would be no different with delegation
> > tokens.
> > > > The
> > > > > > use
> > > > > > > > > case
> > > > > > > > > >> you
> > > > > > > > > >> > > are
> > > > > > > > > >> > > > > describing seems more like impersonation.*
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Yeah, I thought that one of the things that
> > delegation
> > > > > > tokens
> > > > > > > > > >> handled.
> > > > > > > > > >> > > > Maybe I got it wrong :)
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Thanks for the detailed answers.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Gwen
> > > > > > > > > >> > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > > Thanks
> > > > > > > > > >> > > > > Parth
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <
> > > > > > > > > gwen@confluent.io
> > > > > > > > > >> >
> > > > > > > > > >> > > > wrote:
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >> Hi Parth and Harsha,
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> Few more comments:
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> * The API / RequestResponse section doesn't seem
> > to
> > > > > have
> > > > > > > good
> > > > > > > > > >> > > > >> description of the changes to the Kafka Protocol.
> > > > > Sounds
> > > > > > > like
> > > > > > > > > >> you are
> > > > > > > > > >> > > > >> proposing new DelegationTokenRequest and
> > > > > > RenewTokenRequest
> > > > > > > > (and
> > > > > > > > > >> > > > >> matching responses), without detailing the
> > contents
> > > > of
> > > > > > the
> > > > > > > > > >> requests
> > > > > > > > > >> > > > >> and responses? Or rather, you show the class
> > > > interface,
> > > > > > but
> > > > > > > > not
> > > > > > > > > >> the
> > > > > > > > > >> > > > >> underlying protocol. This could be seen as an
> > > > > > > implementation
> > > > > > > > > >> detail,
> > > > > > > > > >> > > > >> but since the binary protocol is what we provide
> > to
> > > > > > > non-Java
> > > > > > > > > >> clients,
> > > > > > > > > >> > > > >> we need to show the changes there.
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> * getDelegationToken sounds like
> > > > > > > > delegationTokenRequestHandler?
> > > > > > > > > >> Is it
> > > > > > > > > >> > > > >> planned to be part of KafkaApi? or Client? Its
> > > > > unclear...
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> * I want to emphasize that even though delegation
> > > > > tokens
> > > > > > > are
> > > > > > > > a
> > > > > > > > > >> Hadoop
> > > > > > > > > >> > > > >> innovation, I feel very strongly about not adding
> > > > > > > dependency
> > > > > > > > on
> > > > > > > > > >> Hadoop
> > > > > > > > > >> > > > >> when implementing delegation tokens for Kafka.
> > The
> > > > KIP
> > > > > > > > doesn't
> > > > > > > > > >> imply
> > > > > > > > > >> > > > >> such dependency, but if you can clarify...
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> * Can we get delegation token at any time after
> > > > > > > > authenticating?
> > > > > > > > > >> only
> > > > > > > > > >> > > > >> immediately after?
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> * My understanding is that tokens will propagate
> > > via
> > > > ZK
> > > > > > but
> > > > > > > > > >> without
> > > > > > > > > >> > > > >> additional changes to UpdateMetadata protocol,
> > > > correct?
> > > > > > > > Clients
> > > > > > > > > >> > > > >> currently don't retry on SASL auth failure
> > (IIRC),
> > > > but
> > > > > > > since
> > > > > > > > > the
> > > > > > > > > >> > > > >> tokens propagate between brokers asynch, we will
> > > need
> > > > > to
> > > > > > > > retry
> > > > > > > > > a
> > > > > > > > > >> bit
> > > > > > > > > >> > > > >> to avoid clients failing auth due to timing
> > issues.
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> * Strongly agreeing on clients not touching ZK
> > > > directly
> > > > > > :)
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> * I liked Ashish's suggestion of having just the
> > > > > > controller
> > > > > > > > > >> issue the
> > > > > > > > > >> > > > >> delegation tokens, to avoid syncing a shared
> > > secret.
> > > > > Not
> > > > > > > sure
> > > > > > > > > if
> > > > > > > > > >> we
> > > > > > > > > >> > > > >> want to continue the discussion here or on the
> > > wiki.
> > > > I
> > > > > > > think
> > > > > > > > > >> that we
> > > > > > > > > >> > > > >> can decouple the problem of "token distribution"
> > > from
> > > > > > > "shared
> > > > > > > > > >> secret
> > > > > > > > > >> > > > >> distribution" and use the controller as the only
> > > > token
> > > > > > > > > generator
> > > > > > > > > >> to
> > > > > > > > > >> > > > >> solve the second issue, while still using ZK
> > async
> > > to
> > > > > > > > > distribute
> > > > > > > > > >> > > > >> tokens.
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> * I am also uncomfortable with infinite lifetime
> > of
> > > > > > tokens
> > > > > > > > (and
> > > > > > > > > >> hoped
> > > > > > > > > >> > > > >> to hear from others in the community) - but
> > having
> > > > the
> > > > > > > option
> > > > > > > > > and
> > > > > > > > > >> > > > >> documenting it as less secure, allows users to
> > > > > configure
> > > > > > > > their
> > > > > > > > > >> system
> > > > > > > > > >> > > > >> as the wish.
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> * While I like the idea of forcing kerberos auth
> > > for
> > > > > > > renwal,
> > > > > > > > I
> > > > > > > > > >> think
> > > > > > > > > >> > > > >> it mixes the transport layer the the request
> > > content
> > > > > in a
> > > > > > > > > pretty
> > > > > > > > > >> ugly
> > > > > > > > > >> > > > >> way. Perhaps limiting renewer to non-owner is
> > > better.
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> Things I'd still like to see:
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> * More detailed explanation on what we plan to do
> > > for
> > > > > the
> > > > > > > > java
> > > > > > > > > >> clients
> > > > > > > > > >> > > > >> specifically - new configuration? new APIs?
> > > > > > > > > >> > > > >> The response for my question on how multiple
> > > > identities
> > > > > > > will
> > > > > > > > be
> > > > > > > > > >> > > > >> handled wasn't super clear to me - AFAIK, we
> > don't
> > > > > > > > authenticate
> > > > > > > > > >> each
> > > > > > > > > >> > > > >> request, we authenticate connections.
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> * Alternatives: Delegation tokens are only used
> > in
> > > > the
> > > > > > > Hadoop
> > > > > > > > > >> > > > >> ecosystem. I'm wondering if there are
> > alternatives
> > > in
> > > > > > other
> > > > > > > > > >> ecosystems
> > > > > > > > > >> > > > >> (Mesos? Tachyon? Cassandra?) and whether there
> > are
> > > > some
> > > > > > > > > >> advantages
> > > > > > > > > >> > > > >> there.
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> Gwen
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <
> > > > > kafka@harsha.io
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >> > > > >> > Hi Gwen,
> > > > > > > > > >> > > > >> >            Can you look at Parth's last reply.
> > > Does
> > > > > it
> > > > > > > > answer
> > > > > > > > > >> your
> > > > > > > > > >> > > > >> >            concerns.
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >> > Thanks,
> > > > > > > > > >> > > > >> > Harsha
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth
> > > brahmbhatt
> > > > > > wrote:
> > > > > > > > > >> > > > >> >> Thanks for reviewing Gwen. The wiki already
> > has
> > > > > > details
> > > > > > > on
> > > > > > > > > >> token
> > > > > > > > > >> > > > >> >> expiration
> > > > > > > > > >> > > > >> >> under token acquisition process
> > > > > > > > > >> > > > >> >> <
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> > > > > > > > > >> > > > >> >.
> > > > > > > > > >> > > > >> >> Current proposal is that tokens will expire
> > > based
> > > > > on a
> > > > > > > > > server
> > > > > > > > > >> side
> > > > > > > > > >> > > > >> >> configuration (default 24 hours) unless
> > renewed.
> > > > > > Renewal
> > > > > > > > is
> > > > > > > > > >> only
> > > > > > > > > >> > > > allowed
> > > > > > > > > >> > > > >> >> until the max life time of token.
> > Alternatively
> > > we
> > > > > > could
> > > > > > > > > also
> > > > > > > > > >> make
> > > > > > > > > >> > > > that
> > > > > > > > > >> > > > >> >> an
> > > > > > > > > >> > > > >> >> optional param and the server side default can
> > > > serve
> > > > > > as
> > > > > > > > the
> > > > > > > > > >> upper
> > > > > > > > > >> > > > bound.
> > > > > > > > > >> > > > >> >>
> > > > > > > > > >> > > > >> >> To your second point it will be done exactly
> > the
> > > > > same
> > > > > > > way
> > > > > > > > we
> > > > > > > > > >> would
> > > > > > > > > >> > > > >> >> support
> > > > > > > > > >> > > > >> >> multiple keytabs. The calling client will have
> > > to
> > > > > put
> > > > > > > the
> > > > > > > > > >> tokens it
> > > > > > > > > >> > > > >> wants
> > > > > > > > > >> > > > >> >> to use in the subject instance and call
> > > > > > produce/consume
> > > > > > > > > inside
> > > > > > > > > >> > > > >> >> subject.doas. Each caller will have to keep
> > > track
> > > > of
> > > > > > its
> > > > > > > > own
> > > > > > > > > >> > > > subject. I
> > > > > > > > > >> > > > >> >> will have to look at the code to see if we
> > > support
> > > > > > this
> > > > > > > > > >> feature
> > > > > > > > > >> > > right
> > > > > > > > > >> > > > >> now
> > > > > > > > > >> > > > >> >> but my understanding is delegation token
> > > shouldn't
> > > > > > need
> > > > > > > > any
> > > > > > > > > >> special
> > > > > > > > > >> > > > >> >> treatment as its just another type of
> > Credential
> > > > in
> > > > > > the
> > > > > > > > > >> subject.
> > > > > > > > > >> > > > >> >>
> > > > > > > > > >> > > > >> >> I would also like to know what is your opinion
> > > > about
> > > > > > > > > infinite
> > > > > > > > > >> > > renewal
> > > > > > > > > >> > > > >> (my
> > > > > > > > > >> > > > >> >> recommendation is to not support this), tokens
> > > > > > renewing
> > > > > > > > them
> > > > > > > > > >> > > self(my
> > > > > > > > > >> > > > >> >> recommendation is to not support this) and
> > most
> > > > > > > > importantly
> > > > > > > > > >> your
> > > > > > > > > >> > > > choice
> > > > > > > > > >> > > > >> >> between the alternatives listed on this thread
> > > > > > > > > >> > > > >> >> <
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >> >> ( I am leaning towards the alternative-2 minus
> > > > > > > controller
> > > > > > > > > >> > > > distributing
> > > > > > > > > >> > > > >> >> secret). Thanks again for reviewing.
> > > > > > > > > >> > > > >> >>
> > > > > > > > > >> > > > >> >> Thanks
> > > > > > > > > >> > > > >> >> Parth
> > > > > > > > > >> > > > >> >>
> > > > > > > > > >> > > > >> >>
> > > > > > > > > >> > > > >> >>
> > > > > > > > > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <
> > > > > > > > > >> gwen@confluent.io>
> > > > > > > > > >> > > > wrote:
> > > > > > > > > >> > > > >> >>
> > > > > > > > > >> > > > >> >> > Harsha,
> > > > > > > > > >> > > > >> >> >
> > > > > > > > > >> > > > >> >> > I was thinking of the Rest Proxy. I didn't
> > see
> > > > > your
> > > > > > > > design
> > > > > > > > > >> yet,
> > > > > > > > > >> > > > but in
> > > > > > > > > >> > > > >> >> > our proxy, we have a set of producers, which
> > > > will
> > > > > > > serve
> > > > > > > > > >> multiple
> > > > > > > > > >> > > > users
> > > > > > > > > >> > > > >> >> > going through the proxy. Since these users
> > > will
> > > > > have
> > > > > > > > > >> different
> > > > > > > > > >> > > > >> >> > privileges, they'll need to authenticate
> > > > > separately,
> > > > > > > and
> > > > > > > > > >> can't
> > > > > > > > > >> > > > share a
> > > > > > > > > >> > > > >> >> > token.
> > > > > > > > > >> > > > >> >> >
> > > > > > > > > >> > > > >> >> > Am I missing anything?
> > > > > > > > > >> > > > >> >> >
> > > > > > > > > >> > > > >> >> > Gwen
> > > > > > > > > >> > > > >> >> >
> > > > > > > > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <
> > > > > > > kafka@harsha.io
> > > > > > > > >
> > > > > > > > > >> wrote:
> > > > > > > > > >> > > > >> >> > > Gwen,
> > > > > > > > > >> > > > >> >> > >            On your second point. Can you
> > > > > describe
> > > > > > a
> > > > > > > > > >> usecase
> > > > > > > > > >> > > where
> > > > > > > > > >> > > > >> >> > >            mutliple clients ended up
> > > sharing a
> > > > > > > > producer
> > > > > > > > > >> and
> > > > > > > > > >> > > even
> > > > > > > > > >> > > > if
> > > > > > > > > >> > > > >> they
> > > > > > > > > >> > > > >> >> > >            do why can't they not use
> > single
> > > > > token
> > > > > > > that
> > > > > > > > > >> producer
> > > > > > > > > >> > > > >> >> > >            captures. Why would we need
> > > > multiple
> > > > > > > > clients
> > > > > > > > > >> with
> > > > > > > > > >> > > > >> different
> > > > > > > > > >> > > > >> >> > >            tokens sharing a single
> > instance
> > > of
> > > > > > > > producer.
> > > > > > > > > >> Also
> > > > > > > > > >> > > in
> > > > > > > > > >> > > > >> this
> > > > > > > > > >> > > > >> >> > >            case other clients have access
> > > all
> > > > > the
> > > > > > > > tokens
> > > > > > > > > >> no?
> > > > > > > > > >> > > > >> >> > >
> > > > > > > > > >> > > > >> >> > > Thanks,
> > > > > > > > > >> > > > >> >> > > Harsha
> > > > > > > > > >> > > > >> >> > >
> > > > > > > > > >> > > > >> >> > >
> > > > > > > > > >> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen
> > > Shapira
> > > > > > > wrote:
> > > > > > > > > >> > > > >> >> > >> Sorry for the delay:
> > > > > > > > > >> > > > >> >> > >>
> > > > > > > > > >> > > > >> >> > >> Two questions that we didn't see in the
> > > wiki:
> > > > > > > > > >> > > > >> >> > >> 1. Is there an expiration for delegation
> > > > > tokens?
> > > > > > > > > >> Renewal? How
> > > > > > > > > >> > > > do we
> > > > > > > > > >> > > > >> >> > >> revoke them?
> > > > > > > > > >> > > > >> >> > >> 2. If we want to use delegation tokens
> > for
> > > > > > "do-as"
> > > > > > > > > (say,
> > > > > > > > > >> > > submit
> > > > > > > > > >> > > > >> Storm
> > > > > > > > > >> > > > >> >> > >> job as my user), we will need a producer
> > > for
> > > > > > every
> > > > > > > > job
> > > > > > > > > >> (we
> > > > > > > > > >> > > can't
> > > > > > > > > >> > > > >> share
> > > > > > > > > >> > > > >> >> > >> them between multiple jobs running on
> > same
> > > > > node),
> > > > > > > > since
> > > > > > > > > >> we
> > > > > > > > > >> > > only
> > > > > > > > > >> > > > >> >> > >> authenticate when connecting. Is there a
> > > plan
> > > > > to
> > > > > > > > change
> > > > > > > > > >> this
> > > > > > > > > >> > > for
> > > > > > > > > >> > > > >> >> > >> delegation tokens, in order to allow
> > > multiple
> > > > > > users
> > > > > > > > > with
> > > > > > > > > >> > > > different
> > > > > > > > > >> > > > >> >> > >> tokens to share a client?
> > > > > > > > > >> > > > >> >> > >>
> > > > > > > > > >> > > > >> >> > >> Gwen
> > > > > > > > > >> > > > >> >> > >>
> > > > > > > > > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth
> > > > > brahmbhatt
> > > > > > > > > >> > > > >> >> > >> <br...@gmail.com> wrote:
> > > > > > > > > >> > > > >> >> > >> > Bumping this up one more time, can
> > other
> > > > > > > committers
> > > > > > > > > >> review?
> > > > > > > > > >> > > > >> >> > >> >
> > > > > > > > > >> > > > >> >> > >> > Thanks
> > > > > > > > > >> > > > >> >> > >> > Parth
> > > > > > > > > >> > > > >> >> > >> >
> > > > > > > > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM,
> > Harsha <
> > > > > > > > > >> kafka@harsha.io>
> > > > > > > > > >> > > > wrote:
> > > > > > > > > >> > > > >> >> > >> >
> > > > > > > > > >> > > > >> >> > >> >> Parth,
> > > > > > > > > >> > > > >> >> > >> >>           Overall current design looks
> > > > good
> > > > > to
> > > > > > > > me. I
> > > > > > > > > >> am +1
> > > > > > > > > >> > > on
> > > > > > > > > >> > > > >> the
> > > > > > > > > >> > > > >> >> > KIP.
> > > > > > > > > >> > > > >> >> > >> >>
> > > > > > > > > >> > > > >> >> > >> >> Gwen , Jun can you review this as
> > well.
> > > > > > > > > >> > > > >> >> > >> >>
> > > > > > > > > >> > > > >> >> > >> >> -Harsha
> > > > > > > > > >> > > > >> >> > >> >>
> > > > > > > > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM,
> > parth
> > > > > > > > brahmbhatt
> > > > > > > > > >> wrote:
> > > > > > > > > >> > > > >> >> > >> >> > Thanks for review Jitendra.
> > > > > > > > > >> > > > >> >> > >> >> >
> > > > > > > > > >> > > > >> >> > >> >> > I don't like the idea of infinite
> > > > lifetime
> > > > > > > but I
> > > > > > > > > >> see the
> > > > > > > > > >> > > > >> Streaming
> > > > > > > > > >> > > > >> >> > use
> > > > > > > > > >> > > > >> >> > >> >> > case. Even for Streaming use case I
> > > was
> > > > > > hoping
> > > > > > > > > >> there will
> > > > > > > > > >> > > > be
> > > > > > > > > >> > > > >> some
> > > > > > > > > >> > > > >> >> > notion
> > > > > > > > > >> > > > >> >> > >> >> > of
> > > > > > > > > >> > > > >> >> > >> >> > master/driver that can get new
> > > > delegation
> > > > > > > tokens
> > > > > > > > > at
> > > > > > > > > >> fixed
> > > > > > > > > >> > > > >> interval
> > > > > > > > > >> > > > >> >> > and
> > > > > > > > > >> > > > >> >> > >> >> > distribute to workers. If that is
> > not
> > > > the
> > > > > > case
> > > > > > > > for
> > > > > > > > > >> we can
> > > > > > > > > >> > > > >> discuss
> > > > > > > > > >> > > > >> >> > >> >> > delegation tokens renewing them self
> > > and
> > > > > the
> > > > > > > > > >> security
> > > > > > > > > >> > > > >> implications
> > > > > > > > > >> > > > >> >> > of the
> > > > > > > > > >> > > > >> >> > >> >> > same.
> > > > > > > > > >> > > > >> >> > >> >> >
> > > > > > > > > >> > > > >> >> > >> >> > I did not want clients to fetch
> > tokens
> > > > > from
> > > > > > > > > >> zookeeper,
> > > > > > > > > >> > > > >> overall I
> > > > > > > > > >> > > > >> >> > think
> > > > > > > > > >> > > > >> >> > >> >> > its
> > > > > > > > > >> > > > >> >> > >> >> > better if clients don't rely on our
> > > > > metadata
> > > > > > > > store
> > > > > > > > > >> and I
> > > > > > > > > >> > > > >> think we
> > > > > > > > > >> > > > >> >> > are
> > > > > > > > > >> > > > >> >> > >> >> > moving in that direction with all
> > the
> > > > > KIP-4
> > > > > > > > > >> improvements.
> > > > > > > > > >> > > > I
> > > > > > > > > >> > > > >> chose
> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as in this case the client
> > > > will
> > > > > > > still
> > > > > > > > > >> just talk
> > > > > > > > > >> > > > to
> > > > > > > > > >> > > > >> >> > broker , its
> > > > > > > > > >> > > > >> >> > >> >> > the brokers that will use zookeeper
> > > > which
> > > > > we
> > > > > > > > > >> already do
> > > > > > > > > >> > > > for a
> > > > > > > > > >> > > > >> lot
> > > > > > > > > >> > > > >> >> > of
> > > > > > > > > >> > > > >> >> > >> >> > other
> > > > > > > > > >> > > > >> >> > >> >> > usecases + ease of development + and
> > > the
> > > > > > > ability
> > > > > > > > > so
> > > > > > > > > >> > > tokens
> > > > > > > > > >> > > > >> will
> > > > > > > > > >> > > > >> >> > survive
> > > > > > > > > >> > > > >> >> > >> >> > even a rolling restart/cluster
> > > failure.
> > > > > if a
> > > > > > > > > >> majority
> > > > > > > > > >> > > > agrees
> > > > > > > > > >> > > > >> the
> > > > > > > > > >> > > > >> >> > added
> > > > > > > > > >> > > > >> >> > >> >> > complexity to have controller
> > > forwarding
> > > > > > keys
> > > > > > > to
> > > > > > > > > all
> > > > > > > > > >> > > > broker is
> > > > > > > > > >> > > > >> >> > justified
> > > > > > > > > >> > > > >> >> > >> >> > as
> > > > > > > > > >> > > > >> >> > >> >> > it provides tighter security , I am
> > > fine
> > > > > > with
> > > > > > > > that
> > > > > > > > > >> option
> > > > > > > > > >> > > > too.
> > > > > > > > > >> > > > >> >> > >> >> >
> > > > > > > > > >> > > > >> >> > >> >> > Given zookeeper does not support SSL
> > > we
> > > > > can
> > > > > > > not
> > > > > > > > > >> store
> > > > > > > > > >> > > > master
> > > > > > > > > >> > > > >> keys
> > > > > > > > > >> > > > >> >> > in
> > > > > > > > > >> > > > >> >> > >> >> > zookeeper as master keys will be
> > > exposed
> > > > > on
> > > > > > > > wire.
> > > > > > > > > To
> > > > > > > > > >> > > > support
> > > > > > > > > >> > > > >> >> > rotation
> > > > > > > > > >> > > > >> >> > >> >> > without affecting current clients is
> > > > > > > something I
> > > > > > > > > >> need to
> > > > > > > > > >> > > > put
> > > > > > > > > >> > > > >> more
> > > > > > > > > >> > > > >> >> > thought
> > > > > > > > > >> > > > >> >> > >> >> > in. My current proposal assumes the
> > > > > rotation
> > > > > > > > will
> > > > > > > > > >> > > > invalidate
> > > > > > > > > >> > > > >> all
> > > > > > > > > >> > > > >> >> > current
> > > > > > > > > >> > > > >> >> > >> >> > tokens.
> > > > > > > > > >> > > > >> >> > >> >> >
> > > > > > > > > >> > > > >> >> > >> >> > I request committers to also review
> > > and
> > > > > post
> > > > > > > > their
> > > > > > > > > >> > > comments
> > > > > > > > > >> > > > >> so we
> > > > > > > > > >> > > > >> >> > can
> > > > > > > > > >> > > > >> >> > >> >> > make
> > > > > > > > > >> > > > >> >> > >> >> > progress on this KIP.
> > > > > > > > > >> > > > >> >> > >> >> >
> > > > > > > > > >> > > > >> >> > >> >> > Thanks
> > > > > > > > > >> > > > >> >> > >> >> > Parth
> > > > > > > > > >> > > > >> >> > >> >> >
> > > > > > > > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM,
> > > Ashish
> > > > > > Singh
> > > > > > > <
> > > > > > > > > >> > > > >> asingh@cloudera.com
> > > > > > > > > >> > > > >> >> > >
> > > > > > > > > >> > > > >> >> > >> >> > wrote:
> > > > > > > > > >> > > > >> >> > >> >> >
> > > > > > > > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM,
> > > > > Harsha <
> > > > > > > > > >> > > > kafka@harsha.io>
> > > > > > > > > >> > > > >> >> > wrote:
> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > Unifying the two discussion
> > > threads
> > > > on
> > > > > > > this
> > > > > > > > > KIP.
> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > > >> > > > >> >> > >> >> > > > Here is the response from
> > Jitendra
> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > > >> > > > >> >> > >> >> > > > "The need for a large number of
> > > > > clients
> > > > > > > that
> > > > > > > > > are
> > > > > > > > > >> > > > running
> > > > > > > > > >> > > > >> all
> > > > > > > > > >> > > > >> >> > over the
> > > > > > > > > >> > > > >> >> > >> >> > > > cluster that authenticate with
> > > Kafka
> > > > > > > > brokers,
> > > > > > > > > >> is very
> > > > > > > > > >> > > > >> similar
> > > > > > > > > >> > > > >> >> > to the
> > > > > > > > > >> > > > >> >> > >> >> > > > Hadoop use case of large number
> > of
> > > > > tasks
> > > > > > > > > running
> > > > > > > > > >> > > across
> > > > > > > > > >> > > > >> the
> > > > > > > > > >> > > > >> >> > cluster
> > > > > > > > > >> > > > >> >> > >> >> that
> > > > > > > > > >> > > > >> >> > >> >> > > > need authentication to Hdfs
> > > > Namenode.
> > > > > > > > > >> Therefore, the
> > > > > > > > > >> > > > >> >> > delegation token
> > > > > > > > > >> > > > >> >> > >> >> > > > approach does seem like a good
> > fit
> > > > for
> > > > > > > this
> > > > > > > > > use
> > > > > > > > > >> case
> > > > > > > > > >> > > > as we
> > > > > > > > > >> > > > >> >> > have seen
> > > > > > > > > >> > > > >> >> > >> >> it
> > > > > > > > > >> > > > >> >> > >> >> > > > working at large scale in HDFS
> > and
> > > > > YARN.
> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > > >> > > > >> >> > >> >> > > >   The proposed design is very
> > much
> > > > > > inline
> > > > > > > > with
> > > > > > > > > >> Hadoop
> > > > > > > > > >> > > > >> >> > approach. A few
> > > > > > > > > >> > > > >> >> > >> >> > > >   comments:
> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > > >> > > > >> >> > >> >> > > > 1) Why do you guys want to allow
> > > > > > infinite
> > > > > > > > > >> renewable
> > > > > > > > > >> > > > >> lifetime
> > > > > > > > > >> > > > >> >> > for a
> > > > > > > > > >> > > > >> >> > >> >> > > > token? HDFS restricts a token
> > to a
> > > > max
> > > > > > > life
> > > > > > > > > time
> > > > > > > > > >> > > > (default
> > > > > > > > > >> > > > >> 7
> > > > > > > > > >> > > > >> >> > days).  A
> > > > > > > > > >> > > > >> >> > >> >> > > > token's vulnerability is
> > believed
> > > to
> > > > > > > > increase
> > > > > > > > > >> with
> > > > > > > > > >> > > > time.
> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > > >> > > > >> >> > >> >> > > I agree that having infinite
> > > lifetime
> > > > > > might
> > > > > > > > not
> > > > > > > > > >> be the
> > > > > > > > > >> > > > best
> > > > > > > > > >> > > > >> idea.
> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > > >> > > > >> >> > >> >> > > > 2) As I understand the tokens
> > are
> > > > > stored
> > > > > > > in
> > > > > > > > > >> zookeeper
> > > > > > > > > >> > > > as
> > > > > > > > > >> > > > >> well,
> > > > > > > > > >> > > > >> >> > and
> > > > > > > > > >> > > > >> >> > >> >> can
> > > > > > > > > >> > > > >> >> > >> >> > > > be updated there. This is clever
> > > as
> > > > it
> > > > > > can
> > > > > > > > > allow
> > > > > > > > > >> > > > >> replacing the
> > > > > > > > > >> > > > >> >> > tokens
> > > > > > > > > >> > > > >> >> > >> >> > > > once they run out of max life
> > > time,
> > > > > and
> > > > > > > > > clients
> > > > > > > > > >> can
> > > > > > > > > >> > > > >> download
> > > > > > > > > >> > > > >> >> > new
> > > > > > > > > >> > > > >> >> > >> >> tokens
> > > > > > > > > >> > > > >> >> > >> >> > > > from zookeeper. It shouldn't be
> > a
> > > > big
> > > > > > load
> > > > > > > > on
> > > > > > > > > >> > > zookeeper
> > > > > > > > > >> > > > >> as a
> > > > > > > > > >> > > > >> >> > client
> > > > > > > > > >> > > > >> >> > >> >> will
> > > > > > > > > >> > > > >> >> > >> >> > > > need to get a new token once in
> > > > > several
> > > > > > > > days.
> > > > > > > > > >> In this
> > > > > > > > > >> > > > >> approach
> > > > > > > > > >> > > > >> >> > you
> > > > > > > > > >> > > > >> >> > >> >> don't
> > > > > > > > > >> > > > >> >> > >> >> > > > need infinite lifetime on the
> > > token
> > > > > even
> > > > > > > for
> > > > > > > > > >> long
> > > > > > > > > >> > > > running
> > > > > > > > > >> > > > >> >> > clients.
> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > > >> > > > >> >> > >> >> > > > 3) The token password are
> > > generated
> > > > > > using
> > > > > > > a
> > > > > > > > > >> master
> > > > > > > > > >> > > key.
> > > > > > > > > >> > > > >> The
> > > > > > > > > >> > > > >> >> > master
> > > > > > > > > >> > > > >> >> > >> >> key
> > > > > > > > > >> > > > >> >> > >> >> > > > should also be periodically
> > > changed.
> > > > > In
> > > > > > > > > Hadoop,
> > > > > > > > > >> the
> > > > > > > > > >> > > > >> default
> > > > > > > > > >> > > > >> >> > renewal
> > > > > > > > > >> > > > >> >> > >> >> > > > period is 1 day.?
> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > > >> > > > >> >> > >> >> > > IIUC, this will require brokers
> > > > > > maintaining
> > > > > > > a
> > > > > > > > > >> list of X
> > > > > > > > > >> > > > most
> > > > > > > > > >> > > > >> >> > recent
> > > > > > > > > >> > > > >> >> > >> >> master
> > > > > > > > > >> > > > >> >> > >> >> > > keys. This list will have to be
> > > > > persisted
> > > > > > > > > >> somewhere, as
> > > > > > > > > >> > > > if a
> > > > > > > > > >> > > > >> >> > broker
> > > > > > > > > >> > > > >> >> > >> >> goes
> > > > > > > > > >> > > > >> >> > >> >> > > down it will have to get that list
> > > > again
> > > > > > and
> > > > > > > > > >> storing
> > > > > > > > > >> > > > master
> > > > > > > > > >> > > > >> keys
> > > > > > > > > >> > > > >> >> > on ZK
> > > > > > > > > >> > > > >> >> > >> >> is
> > > > > > > > > >> > > > >> >> > >> >> > > not the best idea. However, if a
> > > > broker
> > > > > > goes
> > > > > > > > > down
> > > > > > > > > >> then
> > > > > > > > > >> > > we
> > > > > > > > > >> > > > >> have
> > > > > > > > > >> > > > >> >> > much
> > > > > > > > > >> > > > >> >> > >> >> bigger
> > > > > > > > > >> > > > >> >> > >> >> > > issue to deal with and client can
> > > > always
> > > > > > > > > >> > > re-authenticate
> > > > > > > > > >> > > > is
> > > > > > > > > >> > > > >> such
> > > > > > > > > >> > > > >> >> > >> >> events.
> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > Did you happen to take a look at
> > > other
> > > > > > > > > >> alternatives
> > > > > > > > > >> > > this
> > > > > > > > > >> > > > >> list has
> > > > > > > > > >> > > > >> >> > >> >> > > suggested?
> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > > >> > > > >> >> > >> >> > > > Thanks for a thorough proposal,
> > > > great
> > > > > > > work!"
> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28
> > PM,
> > > > Gwen
> > > > > > > > Shapira
> > > > > > > > > >> wrote:
> > > > > > > > > >> > > > >> >> > >> >> > > > > Makes sense to me. Thanks!
> > > > > > > > > >> > > > >> >> > >> >> > > > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25
> > PM,
> > > > > > Harsha <
> > > > > > > > > >> > > > kafka@harsha.io
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >> >> > wrote:
> > > > > > > > > >> > > > >> >> > >> >> > > > > > It doesn't need any release
> > > > > vehicle
> > > > > > > but
> > > > > > > > > >> still the
> > > > > > > > > >> > > > >> work can
> > > > > > > > > >> > > > >> >> > move
> > > > > > > > > >> > > > >> >> > >> >> > > > forward.
> > > > > > > > > >> > > > >> >> > >> >> > > > > > If anyone is interested in
> > the
> > > > KIP
> > > > > > > > please
> > > > > > > > > >> do the
> > > > > > > > > >> > > > >> review and
> > > > > > > > > >> > > > >> >> > >> >> provide
> > > > > > > > > >> > > > >> >> > >> >> > > the
> > > > > > > > > >> > > > >> >> > >> >> > > > > > comments.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > > -Harsha
> > > > > > > > > >> > > > >> >> > >> >> > > > > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at
> > 04:59
> > > > PM,
> > > > > > > Ismael
> > > > > > > > > >> Juma
> > > > > > > > > >> > > > wrote:
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> I agree that it would be
> > good
> > > > to
> > > > > > have
> > > > > > > > > more
> > > > > > > > > >> time
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > >> review
> > > > > > > > > >> > > > >> >> > and
> > > > > > > > > >> > > > >> >> > >> >> > > discuss
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> KIP-48.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> Ismael
> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at
> > 12:55
> > > > AM,
> > > > > > Gwen
> > > > > > > > > >> Shapira <
> > > > > > > > > >> > > > >> >> > >> >> gwen@confluent.io>
> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
> > > > > > > > > >> > > > >> >> > >> >> > > > > >>
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Hi Team,
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Since KIP-48 depends on
> > > > KIP-43,
> > > > > > > which
> > > > > > > > > is
> > > > > > > > > >> > > > already a
> > > > > > > > > >> > > > >> bit
> > > > > > > > > >> > > > >> >> > of a
> > > > > > > > > >> > > > >> >> > >> >> risk
> > > > > > > > > >> > > > >> >> > >> >> > > for
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > the next release - any
> > > chance
> > > > > we
> > > > > > > can
> > > > > > > > > >> delay
> > > > > > > > > >> > > > >> delegation
> > > > > > > > > >> > > > >> >> > tokens
> > > > > > > > > >> > > > >> >> > >> >> to
> > > > > > > > > >> > > > >> >> > >> >> > > > Kafka
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > 0.10.1?
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > With the community
> > working
> > > > on a
> > > > > > > > release
> > > > > > > > > >> every
> > > > > > > > > >> > > 3
> > > > > > > > > >> > > > >> month,
> > > > > > > > > >> > > > >> >> > this
> > > > > > > > > >> > > > >> >> > >> >> is not
> > > > > > > > > >> > > > >> >> > >> >> > > > a huge
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > delay.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Gwen
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at
> > > 5:11
> > > > > PM,
> > > > > > > > Ashish
> > > > > > > > > >> Singh
> > > > > > > > > >> > > <
> > > > > > > > > >> > > > >> >> > >> >> > > asingh@cloudera.com>
> > > > > > > > > >> > > > >> >> > >> >> > > > wrote:
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Parth,
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Thanks again for the
> > > > awesome
> > > > > > > write
> > > > > > > > > up.
> > > > > > > > > >> > > > Following
> > > > > > > > > >> > > > >> our
> > > > > > > > > >> > > > >> >> > >> >> discussion
> > > > > > > > > >> > > > >> >> > >> >> > > > from the
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I think it will
> > be
> > > > > easier
> > > > > > > to
> > > > > > > > > >> compare
> > > > > > > > > >> > > > >> various
> > > > > > > > > >> > > > >> >> > >> >> alternatives
> > > > > > > > > >> > > > >> >> > >> >> > > > if they
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > are
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > listed together. I am
> > > > stating
> > > > > > > > below a
> > > > > > > > > >> few
> > > > > > > > > >> > > > >> >> > alternatives along
> > > > > > > > > >> > > > >> >> > >> >> > > with
> > > > > > > > > >> > > > >> >> > >> >> > > > a the
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > current proposal.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Current proposal)
> > Store
> > > > > > > Delegation
> > > > > > > > > >> Token,
> > > > > > > > > >> > > DT,
> > > > > > > > > >> > > > >> on ZK.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> > > authenticates
> > > > > > with a
> > > > > > > > > >> broker.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > > > > > > > authenticated,
> > > > > > > > > >> it
> > > > > > > > > >> > > will
> > > > > > > > > >> > > > >> make a
> > > > > > > > > >> > > > >> >> > broker
> > > > > > > > > >> > > > >> >> > >> >> side
> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation
> > > > token.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker
> > > generates
> > > > a
> > > > > > > shared
> > > > > > > > > >> secret
> > > > > > > > > >> > > > based
> > > > > > > > > >> > > > >> on
> > > > > > > > > >> > > > >> >> > >> >> > > HMAC-SHA256(a
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    Password/Secret
> > shared
> > > > > > between
> > > > > > > > all
> > > > > > > > > >> > > brokers,
> > > > > > > > > >> > > > >> >> > randomly
> > > > > > > > > >> > > > >> >> > >> >> > > generated
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > tokenId).
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Broker stores
> > this
> > > > > token
> > > > > > in
> > > > > > > > its
> > > > > > > > > >> in
> > > > > > > > > >> > > > memory
> > > > > > > > > >> > > > >> cache.
> > > > > > > > > >> > > > >> >> > >> >> Broker
> > > > > > > > > >> > > > >> >> > >> >> > > > also stores
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the DelegationToken
> > > > > without
> > > > > > > the
> > > > > > > > > >> hmac in
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > >> >> > zookeeper.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. All brokers will
> > > > have a
> > > > > > > cache
> > > > > > > > > >> backed
> > > > > > > > > >> > > by
> > > > > > > > > >> > > > >> >> > zookeeper so
> > > > > > > > > >> > > > >> >> > >> >> they
> > > > > > > > > >> > > > >> >> > >> >> > > > will all
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    get notified
> > whenever
> > > a
> > > > > new
> > > > > > > > token
> > > > > > > > > is
> > > > > > > > > >> > > > >> generated and
> > > > > > > > > >> > > > >> >> > they
> > > > > > > > > >> > > > >> >> > >> >> will
> > > > > > > > > >> > > > >> >> > >> >> > > > update
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    local cache whenever
> > > > token
> > > > > > > state
> > > > > > > > > >> changes.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Broker returns
> > the
> > > > > token
> > > > > > to
> > > > > > > > > >> Client.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and
> > fixes
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Probable race
> > > > > condition,
> > > > > > > > client
> > > > > > > > > >> tries
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > >> >> > authenticate
> > > > > > > > > >> > > > >> >> > >> >> with
> > > > > > > > > >> > > > >> >> > >> >> > > > a broker
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    that is yet to be
> > > > updated
> > > > > > with
> > > > > > > > the
> > > > > > > > > >> newly
> > > > > > > > > >> > > > >> generated
> > > > > > > > > >> > > > >> >> > DT.
> > > > > > > > > >> > > > >> >> > >> >> This
> > > > > > > > > >> > > > >> >> > >> >> > > can
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > probably be
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    dealt with making
> > > > > dtRequest
> > > > > > > > block
> > > > > > > > > >> until
> > > > > > > > > >> > > all
> > > > > > > > > >> > > > >> >> > brokers have
> > > > > > > > > >> > > > >> >> > >> >> > > > updated
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their DT
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    cache. Zk barrier or
> > > > > similar
> > > > > > > > > >> mechanism
> > > > > > > > > >> > > can
> > > > > > > > > >> > > > be
> > > > > > > > > >> > > > >> used.
> > > > > > > > > >> > > > >> >> > >> >> However,
> > > > > > > > > >> > > > >> >> > >> >> > > > all such
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    mechanisms will
> > > increase
> > > > > > > > > complexity.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Using a static
> > > secret
> > > > > key
> > > > > > > > from
> > > > > > > > > >> config
> > > > > > > > > >> > > > >> file. Will
> > > > > > > > > >> > > > >> >> > >> >> require
> > > > > > > > > >> > > > >> >> > >> >> > > yet
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > another
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    config and uses a
> > > static
> > > > > > > secret
> > > > > > > > > >> key. It
> > > > > > > > > >> > > is
> > > > > > > > > >> > > > >> advised
> > > > > > > > > >> > > > >> >> > to
> > > > > > > > > >> > > > >> >> > >> >> rotate
> > > > > > > > > >> > > > >> >> > >> >> > > > secret
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > keys
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    periodically. This
> > can
> > > > be
> > > > > > > > avoided
> > > > > > > > > >> with
> > > > > > > > > >> > > > >> controller
> > > > > > > > > >> > > > >> >> > >> >> generating
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > secretKey and
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    passing to brokers
> > > > > > > periodically.
> > > > > > > > > >> However,
> > > > > > > > > >> > > > >> this will
> > > > > > > > > >> > > > >> >> > >> >> require
> > > > > > > > > >> > > > >> >> > >> >> > > > brokers to
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maintain certain
> > > counts
> > > > of
> > > > > > > > > >> secretKeys.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 1) Have
> > > > > controller
> > > > > > > > > >> generate
> > > > > > > > > >> > > > >> delegation
> > > > > > > > > >> > > > >> >> > token.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> > > authenticates
> > > > > > with a
> > > > > > > > > >> broker.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > > > > > > > authenticated,
> > > > > > > > > >> it
> > > > > > > > > >> > > will
> > > > > > > > > >> > > > >> make a
> > > > > > > > > >> > > > >> >> > broker
> > > > > > > > > >> > > > >> >> > >> >> side
> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation
> > > > token.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. Broker forwards
> > the
> > > > > > request
> > > > > > > > to
> > > > > > > > > >> > > > controller.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Controller
> > > generates
> > > > a
> > > > > DT
> > > > > > > and
> > > > > > > > > >> > > broadcasts
> > > > > > > > > >> > > > >> to all
> > > > > > > > > >> > > > >> >> > >> >> brokers.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. Broker stores
> > this
> > > > > token
> > > > > > in
> > > > > > > > its
> > > > > > > > > >> memory
> > > > > > > > > >> > > > >> cache.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Controller
> > responds
> > > > to
> > > > > > > > broker’s
> > > > > > > > > >> DT
> > > > > > > > > >> > > req.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    7. Broker returns
> > the
> > > > > token
> > > > > > to
> > > > > > > > > >> Client.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and
> > fixes
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. We will have to
> > add
> > > > new
> > > > > > > APIs
> > > > > > > > to
> > > > > > > > > >> > > support
> > > > > > > > > >> > > > >> >> > controller
> > > > > > > > > >> > > > >> >> > >> >> pushing
> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > to
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    brokers on top of
> > the
> > > > > > minimal
> > > > > > > > APIs
> > > > > > > > > >> that
> > > > > > > > > >> > > are
> > > > > > > > > >> > > > >> >> > currently
> > > > > > > > > >> > > > >> >> > >> >> > > proposed.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. We will also have
> > > to
> > > > > add
> > > > > > > APIs
> > > > > > > > > to
> > > > > > > > > >> > > support
> > > > > > > > > >> > > > >> the
> > > > > > > > > >> > > > >> >> > >> >> bootstrapping
> > > > > > > > > >> > > > >> >> > >> >> > > > case,
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > i.e,
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    when a new broker
> > > comes
> > > > up
> > > > > > it
> > > > > > > > will
> > > > > > > > > >> have
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > >> get all
> > > > > > > > > >> > > > >> >> > >> >> delegation
> > > > > > > > > >> > > > >> >> > >> >> > > > tokens
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > from
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the controller.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. In catastrophic
> > > > > failures
> > > > > > > > where
> > > > > > > > > >> all
> > > > > > > > > >> > > > brokers
> > > > > > > > > >> > > > >> go
> > > > > > > > > >> > > > >> >> > down,
> > > > > > > > > >> > > > >> >> > >> >> the
> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if
> > > servers
> > > > > are
> > > > > > > > > >> restarted as
> > > > > > > > > >> > > > >> tokens
> > > > > > > > > >> > > > >> >> > are not
> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this happens,
> > then
> > > > > there
> > > > > > > are
> > > > > > > > > more
> > > > > > > > > >> > > > important
> > > > > > > > > >> > > > >> >> > things to
> > > > > > > > > >> > > > >> >> > >> >> > > worry
> > > > > > > > > >> > > > >> >> > >> >> > > > about
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is better
> > to
> > > > > > > > > >> re-authenticate.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 2) Do not
> > > > > > distribute
> > > > > > > > DT
> > > > > > > > > to
> > > > > > > > > >> > > other
> > > > > > > > > >> > > > >> brokers
> > > > > > > > > >> > > > >> >> > at
> > > > > > > > > >> > > > >> >> > >> >> all.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> > > authenticates
> > > > > > with a
> > > > > > > > > >> broker.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > > > > > > > authenticated,
> > > > > > > > > >> it
> > > > > > > > > >> > > will
> > > > > > > > > >> > > > >> make a
> > > > > > > > > >> > > > >> >> > broker
> > > > > > > > > >> > > > >> >> > >> >> side
> > > > > > > > > >> > > > >> >> > >> >> > > > call to
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation
> > > > token.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker
> > > generates
> > > > DT
> > > > > > of
> > > > > > > > > form,
> > > > > > > > > >> > > [hmac +
> > > > > > > > > >> > > > >> (owner,
> > > > > > > > > >> > > > >> >> > >> >> renewer,
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maxLifeTime, id,
> > hmac,
> > > > > > > > > >> expirationTime)]
> > > > > > > > > >> > > and
> > > > > > > > > >> > > > >> passes
> > > > > > > > > >> > > > >> >> > back
> > > > > > > > > >> > > > >> >> > >> >> this
> > > > > > > > > >> > > > >> >> > >> >> > > > DT to
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > client.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    hmac is generated
> > via
> > > > > > > > > >> {HMAC-SHA256(owner,
> > > > > > > > > >> > > > >> renewer,
> > > > > > > > > >> > > > >> >> > >> >> > > > maxLifeTime, id,
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac,
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    expirationTime)
> > using
> > > > > > > > SecretKey}.
> > > > > > > > > >> Note
> > > > > > > > > >> > > that
> > > > > > > > > >> > > > >> all
> > > > > > > > > >> > > > >> >> > brokers
> > > > > > > > > >> > > > >> >> > >> >> have
> > > > > > > > > >> > > > >> >> > >> >> > > > this
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > SecretKey.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Client then goes
> > to
> > > > any
> > > > > > > > broker
> > > > > > > > > >> and to
> > > > > > > > > >> > > > >> >> > authenticate
> > > > > > > > > >> > > > >> >> > >> >> sends
> > > > > > > > > >> > > > >> >> > >> >> > > > the DT.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    Broker recalculates
> > > hmac
> > > > > > using
> > > > > > > > > >> (owner,
> > > > > > > > > >> > > > >> renewer,
> > > > > > > > > >> > > > >> >> > >> >> maxLifeTime,
> > > > > > > > > >> > > > >> >> > >> >> > > > id, hmac,
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    expirationTime) info
> > > > from
> > > > > DT
> > > > > > > and
> > > > > > > > > its
> > > > > > > > > >> > > > >> SecretKey. If
> > > > > > > > > >> > > > >> >> > it
> > > > > > > > > >> > > > >> >> > >> >> matches
> > > > > > > > > >> > > > >> >> > >> >> > > > with
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac of
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    DT, client is
> > > > > authenticated.
> > > > > > > > Yes,
> > > > > > > > > >> it will
> > > > > > > > > >> > > > do
> > > > > > > > > >> > > > >> other
> > > > > > > > > >> > > > >> >> > >> >> obvious
> > > > > > > > > >> > > > >> >> > >> >> > > > checks of
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    timestamp expiry and
> > > > such.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Note that secret key
> > will
> > > > be
> > > > > > > > > generated
> > > > > > > > > >> by
> > > > > > > > > >> > > > >> controller
> > > > > > > > > >> > > > >> >> > and
> > > > > > > > > >> > > > >> >> > >> >> passed
> > > > > > > > > >> > > > >> >> > >> >> > > to
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > brokers
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > periodically.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and
> > fixes
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. How to delete a
> > DT?
> > > > > Yes,
> > > > > > > that
> > > > > > > > > is
> > > > > > > > > >> a
> > > > > > > > > >> > > > downside
> > > > > > > > > >> > > > >> >> > here.
> > > > > > > > > >> > > > >> >> > >> >> However,
> > > > > > > > > >> > > > >> >> > >> >> > > > this can
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be handled with
> > > brokers
> > > > > > > > > maintaining
> > > > > > > > > >> a
> > > > > > > > > >> > > > >> blacklist of
> > > > > > > > > >> > > > >> >> > DTs,
> > > > > > > > > >> > > > >> >> > >> >> DTs
> > > > > > > > > >> > > > >> >> > >> >> > > > from this
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > list
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    can be removed after
> > > > > expiry.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. In catastrophic
> > > > > failures
> > > > > > > > where
> > > > > > > > > >> all
> > > > > > > > > >> > > > brokers
> > > > > > > > > >> > > > >> go
> > > > > > > > > >> > > > >> >> > down,
> > > > > > > > > >> > > > >> >> > >> >> the
> > > > > > > > > >> > > > >> >> > >> >> > > > tokens will
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if
> > > servers
> > > > > are
> > > > > > > > > >> restarted as
> > > > > > > > > >> > > > >> tokens
> > > > > > > > > >> > > > >> >> > are not
> > > > > > > > > >> > > > >> >> > >> >> > > > persisted
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this happens,
> > then
> > > > > there
> > > > > > > are
> > > > > > > > > more
> > > > > > > > > >> > > > important
> > > > > > > > > >> > > > >> >> > things to
> > > > > > > > > >> > > > >> >> > >> >> > > worry
> > > > > > > > > >> > > > >> >> > >> >> > > > about
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is better
> > to
> > > > > > > > > >> re-authenticate.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at
> > > > 1:58
> > > > > > PM,
> > > > > > > > > Parth
> > > > > > > > > >> > > > >> Brahmbhatt <
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > pbrahmbhatt@hortonworks.com>
> > > > > > > > wrote:
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Hi,
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> I have filed KIP-48 so
> > > we
> > > > > can
> > > > > > > > offer
> > > > > > > > > >> hadoop
> > > > > > > > > >> > > > like
> > > > > > > > > >> > > > >> >> > delegation
> > > > > > > > > >> > > > >> >> > >> >> > > > tokens in
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> kafka. You can review
> > > the
> > > > > > design
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > > >> > > > >> >> > >> >>
> > > > > > > > > >> > > > >> >> >
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > .
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> This KIP depends on
> > > KIP-43
> > > > > and
> > > > > > > we
> > > > > > > > > >> have also
> > > > > > > > > >> > > > >> >> > discussed an
> > > > > > > > > >> > > > >> >> > >> >> > > > alternative to
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> proposed design here<
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > > >> > > > >> >> > >> >>
> > > > > > > > > >> > > > >> >> >
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >.
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Thanks
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Parth
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > --
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Regards,
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Ashish
> > > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > --
> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > > >> > > > >> >> > >> >> > > Regards,
> > > > > > > > > >> > > > >> >> > >> >> > > Ashish
> > > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > > >> > > > >> >> > >> >>
> > > > > > > > > >> > > > >> >> >
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> >
> 
> 
> 
> -- 
> Liquan Pei
> Software Engineer, Confluent Inc

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Liquan Pei <li...@gmail.com>]

It seems that the links to images in the KIP are broken.

Liquan

On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
brahmbhatt.parth@gmail.com> wrote:

> 110. What does getDelegationTokenAs mean?
> In the current proposal we only allow a user to get delegation token for
> the identity that it authenticated as using another mechanism, i.e. A user
> that authenticate using a keytab for principal user1@EXAMPLE.COM will get
> delegation tokens for that user only. In future I think we will have to
> extend support such that we allow some set of users (
> kafka-rest-user@EXAMPLE.COM, storm-nimbus@EXAMPLE.COM) to acquire
> delegation tokens on behalf of other users whose identity they have
> verified independently.  Kafka brokers will have ACLs to control which
> users are allowed to impersonate other users and get tokens on behalf of
> them. Overall Impersonation is a whole different problem in my opinion and
> I think we can tackle it in separate KIP.
>
> 111. What's the typical rate of getting and renewing delegation tokens?
> Typically this should be very very low, 1 request per minute is a
> relatively high estimate. However it depends on the token expiration. I am
> less worried about the extra load it puts on controller vs the added
> complexity and the value it offers.
>
> Thanks
> Parth
>
>
>
> On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Thanks Rajini. It would probably require a separate KIP as it will
> > introduce user visible changes. We could also update KIP-48 to have this
> > information, but it seems cleaner to do it separately. We can discuss
> that
> > in the KIP call today.
> >
> > Ismael
> >
> > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <
> > rajinisivaram@googlemail.com> wrote:
> >
> > > Ismael,
> > >
> > > I have created a JIRA (
> https://issues.apache.org/jira/browse/KAFKA-3751)
> > > for adding SCRAM as a SASL mechanism. Would that need another KIP? If
> > > KIP-48 will use this mechanism, can this just be a JIRA that gets
> > reviewed
> > > when the PR is ready?
> > >
> > > Thank you,
> > >
> > > Rajini
> > >
> > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <is...@juma.me.uk>
> wrote:
> > >
> > > > Thanks Rajini, SCRAM seems like a good candidate.
> > > >
> > > > Gwen had independently mentioned this as a SASL mechanism that might
> be
> > > > useful for Kafka and I have been meaning to check it in more detail.
> > Good
> > > > to know that you are willing to contribute an implementation. Maybe
> we
> > > > should file a separate JIRA for this?
> > > >
> > > > Ismael
> > > >
> > > > On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram <
> > > > rajinisivaram@googlemail.com> wrote:
> > > >
> > > > > SCRAM (Salted Challenge Response Authentication Mechanism) is a
> > better
> > > > > mechanism than Digest-MD5. Java doesn't come with a built-in SCRAM
> > > > > SaslServer or SaslClient, but I will be happy to add support in
> Kafka
> > > > since
> > > > > it would be a useful mechanism to support anyway.
> > > > > https://tools.ietf.org/html/rfc7677 describes the protocol for
> > > > > SCRAM-SHA-256.
> > > > >
> > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <ju...@confluent.io> wrote:
> > > > >
> > > > > > Parth,
> > > > > >
> > > > > > Thanks for the explanation. A couple of more questions.
> > > > > >
> > > > > > 110. What does getDelegationTokenAs mean?
> > > > > >
> > > > > > 111. What's the typical rate of getting and renewing delegation
> > > tokens?
> > > > > > That may have an impact on whether they should be directed to the
> > > > > > controller.
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt <
> > > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > Thanks for reviewing.
> > > > > > >
> > > > > > > * We could add a Cluster action to add acls on who can request
> > > > > delegation
> > > > > > > tokens. I don't see the use case for that yet but down the line
> > > when
> > > > we
> > > > > > > start supporting getDelegationTokenAs it will be necessary.
> > > > > > > * Yes we recommend tokens to be only used/distributed over
> secure
> > > > > > channels.
> > > > > > > * Depending on what design we end up choosing Invalidation will
> > be
> > > > > > > responsibility of every broker or controller.
> > > > > > > * I am not sure if I documented somewhere that invalidation
> will
> > > > > directly
> > > > > > > go through zookeeper but that is not the intent. Invalidation
> > will
> > > > > either
> > > > > > > be request based or due to expiration. No direct zookeeper
> > > > interaction
> > > > > > from
> > > > > > > any client.
> > > > > > > * "Broker also stores the DelegationToken without the hmac in
> the
> > > > > > > zookeeper." : Sorry about the confusion. The sole purpose of
> > > > zookeeper
> > > > > in
> > > > > > > this design is as distribution channel for tokens between all
> > > brokers
> > > > > > and a
> > > > > > > layer that ensures only tokens that were generated by making a
> > > > request
> > > > > > to a
> > > > > > > broker will be accepted (more on this in second paragraph). The
> > > token
> > > > > > > consists of few elements (owner, renewer, uuid , expiration,
> > hmac)
> > > ,
> > > > > one
> > > > > > of
> > > > > > > which is the finally generated hmac but hmac it self is
> derivable
> > > if
> > > > > you
> > > > > > > have all the other elements of the token + secret key to
> generate
> > > > hmac.
> > > > > > > Given zookeeper does not provide SSL support we do not want the
> > > > entire
> > > > > > > token to be wire transferred to zookeeper as that will be an
> > > insecure
> > > > > > wire
> > > > > > > transfer. Instead we only store all the other elements of a
> > > > delegation
> > > > > > > tokens. Brokers can read these elements and because they also
> > have
> > > > > access
> > > > > > > to secret key they will be able to generate hmac on their end.
> > > > > > >
> > > > > > > One of the alternative proposed is to avoid zookeeper
> > altogether. A
> > > > > > Client
> > > > > > > will call broker with required information (owner, renwer,
> > > > expiration)
> > > > > > and
> > > > > > > get back (signed hmac, uuid). Broker won't store this in
> > zookeeper.
> > > > > From
> > > > > > > this point a client can contact any broker with all the
> > delegation
> > > > > token
> > > > > > > info (owner, rewner, expiration, hmac, uuid) the borker will
> > > > regenerate
> > > > > > the
> > > > > > > hmac and as long as it matches with hmac presented by client ,
> > > broker
> > > > > > will
> > > > > > > allow the request to authenticate.  Only problem with this
> > approach
> > > > is
> > > > > if
> > > > > > > the secret key is compromised any client can now generate
> random
> > > > tokens
> > > > > > and
> > > > > > > they will still be able to authenticate as any user they like.
> > with
> > > > > > > zookeeper we guarantee that only tokens acquired via a broker
> > > (using
> > > > > some
> > > > > > > auth scheme other than delegation token) will be accepted. We
> > need
> > > to
> > > > > > > discuss which proposal makes more sense and we can go over it
> in
> > > > > > tomorrow's
> > > > > > > meeting.
> > > > > > >
> > > > > > > Also, can you forward the invite to me?
> > > > > > >
> > > > > > > Thanks
> > > > > > > Parth
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <ju...@confluent.io>
> > > wrote:
> > > > > > >
> > > > > > > > Thanks for the KIP. A few comments.
> > > > > > > >
> > > > > > > > 100. This potentially can be useful for Kafka Connect and
> Kafka
> > > > rest
> > > > > > > proxy
> > > > > > > > where a worker agent will need to run a task on behalf of a
> > > client.
> > > > > We
> > > > > > > will
> > > > > > > > likely need to change how those services use Kafka clients
> > > > > > > > (producer/consumer). Instead of a shared client per worker,
> we
> > > will
> > > > > > need
> > > > > > > a
> > > > > > > > client per user task since the authentication happens at the
> > > > > connection
> > > > > > > > level. For Kafka Connect, the renewer will be the workers.
> So,
> > we
> > > > > > > probably
> > > > > > > > need to allow multiple renewers. For Kafka rest proxy, the
> > > renewer
> > > > > can
> > > > > > > > probably just be the creator of the token.
> > > > > > > >
> > > > > > > > 101. Do we need new acl on who can request delegation tokens?
> > > > > > > >
> > > > > > > > 102. Do we recommend people to send delegation tokens in an
> > > > encrypted
> > > > > > > > channel?
> > > > > > > >
> > > > > > > > 103. Who is responsible for expiring tokens, every broker?
> > > > > > > >
> > > > > > > > 104. For invalidating tokens, would it be better to do it in
> a
> > > > > request
> > > > > > > > instead of going to ZK directly?
> > > > > > > >
> > > > > > > > 105. The terminology of client in the wiki sometimes refers
> to
> > > the
> > > > > end
> > > > > > > > client and some other times refers to the client using the
> > > > delegation
> > > > > > > > tokens. It would be useful to distinguish between the two.
> > > > > > > >
> > > > > > > > 106. Could you explain the sentence "Broker also stores the
> > > > > > > DelegationToken
> > > > > > > > without the hmac in the zookeeper." a bit more? I thought the
> > > > > > delegation
> > > > > > > > token is the hmac.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <ju...@confluent.io>
> > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Harsha,
> > > > > > > > >
> > > > > > > > > Just sent out a KIP meeting invite. We can discuss this in
> > the
> > > > > > meeting
> > > > > > > > > tomorrow.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <ka...@harsha.io>
> > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi All,
> > > > > > > > >>            Can we have a KIP meeting around this. The KIP
> is
> > > up
> > > > > for
> > > > > > > > >>            sometime and if there are any questions lets
> > > quickly
> > > > > hash
> > > > > > > out
> > > > > > > > >>            details.
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> Harsha
> > > > > > > > >>
> > > > > > > > >> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt wrote:
> > > > > > > > >> > That is what the hadoop echo system uses so no good
> reason
> > > > > really.
> > > > > > > We
> > > > > > > > >> > could
> > > > > > > > >> > change it to whatever is the newest recommended standard
> > is.
> > > > > > > > >> >
> > > > > > > > >> > Thanks
> > > > > > > > >> > Parth
> > > > > > > > >> >
> > > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <
> > > > ismael@juma.me.uk
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Hi Parth,
> > > > > > > > >> > >
> > > > > > > > >> > > Thanks for the KIP. I only started reviewing this and
> > may
> > > > have
> > > > > > > > >> additional
> > > > > > > > >> > > questions later. The immediate question that came to
> > mind
> > > is
> > > > > our
> > > > > > > > >> choice of
> > > > > > > > >> > > "DIGEST-MD5" even though it's marked as OBSOLETE in
> the
> > > IANA
> > > > > > > > Registry
> > > > > > > > >> of
> > > > > > > > >> > > SASL mechanisms and the original RFC (2831) has been
> > moved
> > > > to
> > > > > > > > Historic
> > > > > > > > >> > > status:
> > > > > > > > >> > >
> > > > > > > > >> > > https://tools.ietf.org/html/rfc6331
> > > > > > > > >> > >
> > > > > > > >
> > > > >
> > http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> > > > > > > > >> > >
> > > > > > > > >> > > What is the reasoning behind that choice?
> > > > > > > > >> > >
> > > > > > > > >> > > Thanks,
> > > > > > > > >> > > Ismael
> > > > > > > > >> > >
> > > > > > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <
> > > > > > gwen@confluent.io
> > > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > > Also comments inline :)
> > > > > > > > >> > > >
> > > > > > > > >> > > > > * I want to emphasize that even though delegation
> > > tokens
> > > > > > are a
> > > > > > > > >> Hadoop
> > > > > > > > >> > > > > innovation, I feel very strongly about not adding
> > > > > dependency
> > > > > > > on
> > > > > > > > >> Hadoop
> > > > > > > > >> > > > > when implementing delegation tokens for Kafka. The
> > KIP
> > > > > > doesn't
> > > > > > > > >> imply
> > > > > > > > >> > > > > such dependency, but if you can clarify...
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > *No hadoop dependency.*
> > > > > > > > >> > > >
> > > > > > > > >> > > > Yay! Just add this to the KIP so no one will read
> the
> > > KIP
> > > > > and
> > > > > > > > panic
> > > > > > > > >> > > > three weeks before the next release...
> > > > > > > > >> > > >
> > > > > > > > >> > > > > * Can we get delegation token at any time after
> > > > > > > authenticating?
> > > > > > > > >> only
> > > > > > > > >> > > > > immediately after?
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > *As long as you are authenticated you can get
> > > delegation
> > > > > > > tokens.
> > > > > > > > >> We
> > > > > > > > >> > > need
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > discuss if a client authenticated using delegation
> > > > token,
> > > > > > can
> > > > > > > > also
> > > > > > > > >> > > > acquire
> > > > > > > > >> > > > > delegation token again or not. Also there is the
> > > > question
> > > > > of
> > > > > > > do
> > > > > > > > we
> > > > > > > > >> > > allow
> > > > > > > > >> > > > > anyone to acquire delegation token or we want
> > specific
> > > > > ACLs
> > > > > > (I
> > > > > > > > >> think
> > > > > > > > >> > > its
> > > > > > > > >> > > > an
> > > > > > > > >> > > > > overkill.)*
> > > > > > > > >> > > >
> > > > > > > > >> > > > I agree that ACLs is an overkill.
> > > > > > > > >> > > >
> > > > > > > > >> > > > I think we are debating two options: Either require
> > > > Kerberos
> > > > > > > auth
> > > > > > > > >> for
> > > > > > > > >> > > > renewal or require non-owners to renew.
> > > > > > > > >> > > > I *think* the latter is simpler (it basically
> require
> > a
> > > > "job
> > > > > > > > master"
> > > > > > > > >> > > > to take responsibility for the renewal, it will have
> > its
> > > > own
> > > > > > > > >> identity
> > > > > > > > >> > > > anyway and I think this is the correct design
> pattern
> > > > > anyway.
> > > > > > > For
> > > > > > > > >> > > > storm, I'd expect Nimbus to coordinate renewals?),
> but
> > > it
> > > > is
> > > > > > > hard
> > > > > > > > to
> > > > > > > > >> > > > debate simplicity without looking at the code
> changes
> > > > > > required.
> > > > > > > If
> > > > > > > > >> you
> > > > > > > > >> > > > have a draft of how the "require Kerberos" will look
> > in
> > > > > Kafka
> > > > > > > > code,
> > > > > > > > >> > > > I'll be happy to take a look.
> > > > > > > > >> > > >
> > > > > > > > >> > > > > * My understanding is that tokens will propagate
> via
> > > ZK
> > > > > but
> > > > > > > > >> without
> > > > > > > > >> > > > > additional changes to UpdateMetadata protocol,
> > > correct?
> > > > > > > Clients
> > > > > > > > >> > > > > currently don't retry on SASL auth failure (IIRC),
> > but
> > > > > since
> > > > > > > the
> > > > > > > > >> > > > > tokens propagate between brokers asynch, we will
> > need
> > > to
> > > > > > > retry a
> > > > > > > > >> bit
> > > > > > > > >> > > > > to avoid clients failing auth due to timing
> issues.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > *I am considering 2 alternatives right now. The
> > > current
> > > > > > > > documented
> > > > > > > > >> > > > approach
> > > > > > > > >> > > > > is zookeeper based and it does not require any
> > changes
> > > > to
> > > > > > > > >> > > UpdateMetadata
> > > > > > > > >> > > > > protocol. An alternative approach can remove
> > zookeeper
> > > > > > > > dependency
> > > > > > > > >> as
> > > > > > > > >> > > well
> > > > > > > > >> > > > > but we can discuss that in KIP discussion call.*
> > > > > > > > >> > > >
> > > > > > > > >> > > > Oooh! Sounds interesting. Do you want to ping Jun to
> > > > > arrange a
> > > > > > > > call?
> > > > > > > > >> > > >
> > > > > > > > >> > > > > * I liked Ashish's suggestion of having just the
> > > > > controller
> > > > > > > > issue
> > > > > > > > >> the
> > > > > > > > >> > > > > delegation tokens, to avoid syncing a shared
> secret.
> > > Not
> > > > > > sure
> > > > > > > if
> > > > > > > > >> we
> > > > > > > > >> > > > > want to continue the discussion here or on the
> > wiki. I
> > > > > think
> > > > > > > > that
> > > > > > > > >> we
> > > > > > > > >> > > > > can decouple the problem of "token distribution"
> > from
> > > > > > "shared
> > > > > > > > >> secret
> > > > > > > > >> > > > > distribution" and use the controller as the only
> > token
> > > > > > > generator
> > > > > > > > >> to
> > > > > > > > >> > > > > solve the second issue, while still using ZK async
> > to
> > > > > > > distribute
> > > > > > > > >> > > > > tokens.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > *As mentioned in the previous Email I am fine with
> > > that
> > > > > > > approach
> > > > > > > > >> as
> > > > > > > > >> > > long
> > > > > > > > >> > > > as
> > > > > > > > >> > > > > we agree that the extra complexity of
> > adding/updating
> > > > APIS
> > > > > > > adds
> > > > > > > > >> enough
> > > > > > > > >> > > > > value. The advantage with the controller approach
> is
> > > > > secret
> > > > > > > > >> rotation
> > > > > > > > >> > > can
> > > > > > > > >> > > > be
> > > > > > > > >> > > > > automated,frequent and would not require
> > deployment. *
> > > > > > > > >> > > >
> > > > > > > > >> > > > Can you detail the extra complexity (or point me to
> > the
> > > > > email
> > > > > > I
> > > > > > > > >> > > > missed?) - which Apis are required?
> > > > > > > > >> > > > As far as I can tell, clients can already find the
> > > > > controller
> > > > > > > from
> > > > > > > > >> > > > metadata. I'm a bit more concerned about controller
> > > load.
> > > > > > > > >> > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > * While I like the idea of forcing kerberos auth
> for
> > > > > > renwal, I
> > > > > > > > >> think
> > > > > > > > >> > > > > it mixes the transport layer the the request
> content
> > > in
> > > > a
> > > > > > > pretty
> > > > > > > > >> ugly
> > > > > > > > >> > > > > way. Perhaps limiting renewer to non-owner is
> > better.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > *I feel this is a necessary evil. While this will
> > make
> > > > the
> > > > > > > kafka
> > > > > > > > >> code
> > > > > > > > >> > > > > pretty straight forward , forcing  renewer to
> > > non-owner
> > > > > > pushes
> > > > > > > > >> the code
> > > > > > > > >> > > > > ugliness to client and makes it even harder to
> > > > > integrate.  *
> > > > > > > > >> > > >
> > > > > > > > >> > > > As mentioned before, I don't think the "renewal by
> > > other"
> > > > > > > approach
> > > > > > > > >> is
> > > > > > > > >> > > > that ugly for the clients we expect to use
> delegation
> > > > tokens
> > > > > > > since
> > > > > > > > >> > > > they will have an app-master of some sort who
> > requested
> > > > the
> > > > > > > token
> > > > > > > > to
> > > > > > > > >> > > > begin with.
> > > > > > > > >> > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > The response for my question on how multiple
> > > identities
> > > > > will
> > > > > > > be
> > > > > > > > >> > > > > handled wasn't super clear to me - AFAIK, we don't
> > > > > > > authenticate
> > > > > > > > >> each
> > > > > > > > >> > > > > request, we authenticate connections.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > *We authenticate connections, and only when they
> are
> > > > being
> > > > > > > > >> established.
> > > > > > > > >> > > > Let
> > > > > > > > >> > > > > me try to phrase this as a question, in absence of
> > > > > > delegation
> > > > > > > > >> tokens if
> > > > > > > > >> > > > we
> > > > > > > > >> > > > > had to support the use case using user TGT's how
> > would
> > > > we
> > > > > do
> > > > > > > it?
> > > > > > > > >> My
> > > > > > > > >> > > point
> > > > > > > > >> > > > > was it would be no different with delegation
> tokens.
> > > The
> > > > > use
> > > > > > > > case
> > > > > > > > >> you
> > > > > > > > >> > > are
> > > > > > > > >> > > > > describing seems more like impersonation.*
> > > > > > > > >> > > >
> > > > > > > > >> > > > Yeah, I thought that one of the things that
> delegation
> > > > > tokens
> > > > > > > > >> handled.
> > > > > > > > >> > > > Maybe I got it wrong :)
> > > > > > > > >> > > >
> > > > > > > > >> > > > Thanks for the detailed answers.
> > > > > > > > >> > > >
> > > > > > > > >> > > > Gwen
> > > > > > > > >> > > >
> > > > > > > > >> > > >
> > > > > > > > >> > > > > Thanks
> > > > > > > > >> > > > > Parth
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <
> > > > > > > > gwen@confluent.io
> > > > > > > > >> >
> > > > > > > > >> > > > wrote:
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >> Hi Parth and Harsha,
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> Few more comments:
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> * The API / RequestResponse section doesn't seem
> to
> > > > have
> > > > > > good
> > > > > > > > >> > > > >> description of the changes to the Kafka Protocol.
> > > > Sounds
> > > > > > like
> > > > > > > > >> you are
> > > > > > > > >> > > > >> proposing new DelegationTokenRequest and
> > > > > RenewTokenRequest
> > > > > > > (and
> > > > > > > > >> > > > >> matching responses), without detailing the
> contents
> > > of
> > > > > the
> > > > > > > > >> requests
> > > > > > > > >> > > > >> and responses? Or rather, you show the class
> > > interface,
> > > > > but
> > > > > > > not
> > > > > > > > >> the
> > > > > > > > >> > > > >> underlying protocol. This could be seen as an
> > > > > > implementation
> > > > > > > > >> detail,
> > > > > > > > >> > > > >> but since the binary protocol is what we provide
> to
> > > > > > non-Java
> > > > > > > > >> clients,
> > > > > > > > >> > > > >> we need to show the changes there.
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> * getDelegationToken sounds like
> > > > > > > delegationTokenRequestHandler?
> > > > > > > > >> Is it
> > > > > > > > >> > > > >> planned to be part of KafkaApi? or Client? Its
> > > > unclear...
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> * I want to emphasize that even though delegation
> > > > tokens
> > > > > > are
> > > > > > > a
> > > > > > > > >> Hadoop
> > > > > > > > >> > > > >> innovation, I feel very strongly about not adding
> > > > > > dependency
> > > > > > > on
> > > > > > > > >> Hadoop
> > > > > > > > >> > > > >> when implementing delegation tokens for Kafka.
> The
> > > KIP
> > > > > > > doesn't
> > > > > > > > >> imply
> > > > > > > > >> > > > >> such dependency, but if you can clarify...
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> * Can we get delegation token at any time after
> > > > > > > authenticating?
> > > > > > > > >> only
> > > > > > > > >> > > > >> immediately after?
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> * My understanding is that tokens will propagate
> > via
> > > ZK
> > > > > but
> > > > > > > > >> without
> > > > > > > > >> > > > >> additional changes to UpdateMetadata protocol,
> > > correct?
> > > > > > > Clients
> > > > > > > > >> > > > >> currently don't retry on SASL auth failure
> (IIRC),
> > > but
> > > > > > since
> > > > > > > > the
> > > > > > > > >> > > > >> tokens propagate between brokers asynch, we will
> > need
> > > > to
> > > > > > > retry
> > > > > > > > a
> > > > > > > > >> bit
> > > > > > > > >> > > > >> to avoid clients failing auth due to timing
> issues.
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> * Strongly agreeing on clients not touching ZK
> > > directly
> > > > > :)
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> * I liked Ashish's suggestion of having just the
> > > > > controller
> > > > > > > > >> issue the
> > > > > > > > >> > > > >> delegation tokens, to avoid syncing a shared
> > secret.
> > > > Not
> > > > > > sure
> > > > > > > > if
> > > > > > > > >> we
> > > > > > > > >> > > > >> want to continue the discussion here or on the
> > wiki.
> > > I
> > > > > > think
> > > > > > > > >> that we
> > > > > > > > >> > > > >> can decouple the problem of "token distribution"
> > from
> > > > > > "shared
> > > > > > > > >> secret
> > > > > > > > >> > > > >> distribution" and use the controller as the only
> > > token
> > > > > > > > generator
> > > > > > > > >> to
> > > > > > > > >> > > > >> solve the second issue, while still using ZK
> async
> > to
> > > > > > > > distribute
> > > > > > > > >> > > > >> tokens.
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> * I am also uncomfortable with infinite lifetime
> of
> > > > > tokens
> > > > > > > (and
> > > > > > > > >> hoped
> > > > > > > > >> > > > >> to hear from others in the community) - but
> having
> > > the
> > > > > > option
> > > > > > > > and
> > > > > > > > >> > > > >> documenting it as less secure, allows users to
> > > > configure
> > > > > > > their
> > > > > > > > >> system
> > > > > > > > >> > > > >> as the wish.
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> * While I like the idea of forcing kerberos auth
> > for
> > > > > > renwal,
> > > > > > > I
> > > > > > > > >> think
> > > > > > > > >> > > > >> it mixes the transport layer the the request
> > content
> > > > in a
> > > > > > > > pretty
> > > > > > > > >> ugly
> > > > > > > > >> > > > >> way. Perhaps limiting renewer to non-owner is
> > better.
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> Things I'd still like to see:
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> * More detailed explanation on what we plan to do
> > for
> > > > the
> > > > > > > java
> > > > > > > > >> clients
> > > > > > > > >> > > > >> specifically - new configuration? new APIs?
> > > > > > > > >> > > > >> The response for my question on how multiple
> > > identities
> > > > > > will
> > > > > > > be
> > > > > > > > >> > > > >> handled wasn't super clear to me - AFAIK, we
> don't
> > > > > > > authenticate
> > > > > > > > >> each
> > > > > > > > >> > > > >> request, we authenticate connections.
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> * Alternatives: Delegation tokens are only used
> in
> > > the
> > > > > > Hadoop
> > > > > > > > >> > > > >> ecosystem. I'm wondering if there are
> alternatives
> > in
> > > > > other
> > > > > > > > >> ecosystems
> > > > > > > > >> > > > >> (Mesos? Tachyon? Cassandra?) and whether there
> are
> > > some
> > > > > > > > >> advantages
> > > > > > > > >> > > > >> there.
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> Gwen
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <
> > > > kafka@harsha.io
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >> > > > >> > Hi Gwen,
> > > > > > > > >> > > > >> >            Can you look at Parth's last reply.
> > Does
> > > > it
> > > > > > > answer
> > > > > > > > >> your
> > > > > > > > >> > > > >> >            concerns.
> > > > > > > > >> > > > >> >
> > > > > > > > >> > > > >> > Thanks,
> > > > > > > > >> > > > >> > Harsha
> > > > > > > > >> > > > >> >
> > > > > > > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth
> > brahmbhatt
> > > > > wrote:
> > > > > > > > >> > > > >> >> Thanks for reviewing Gwen. The wiki already
> has
> > > > > details
> > > > > > on
> > > > > > > > >> token
> > > > > > > > >> > > > >> >> expiration
> > > > > > > > >> > > > >> >> under token acquisition process
> > > > > > > > >> > > > >> >> <
> > > > > > > > >> > > > >>
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> > > > > > > > >> > > > >> >.
> > > > > > > > >> > > > >> >> Current proposal is that tokens will expire
> > based
> > > > on a
> > > > > > > > server
> > > > > > > > >> side
> > > > > > > > >> > > > >> >> configuration (default 24 hours) unless
> renewed.
> > > > > Renewal
> > > > > > > is
> > > > > > > > >> only
> > > > > > > > >> > > > allowed
> > > > > > > > >> > > > >> >> until the max life time of token.
> Alternatively
> > we
> > > > > could
> > > > > > > > also
> > > > > > > > >> make
> > > > > > > > >> > > > that
> > > > > > > > >> > > > >> >> an
> > > > > > > > >> > > > >> >> optional param and the server side default can
> > > serve
> > > > > as
> > > > > > > the
> > > > > > > > >> upper
> > > > > > > > >> > > > bound.
> > > > > > > > >> > > > >> >>
> > > > > > > > >> > > > >> >> To your second point it will be done exactly
> the
> > > > same
> > > > > > way
> > > > > > > we
> > > > > > > > >> would
> > > > > > > > >> > > > >> >> support
> > > > > > > > >> > > > >> >> multiple keytabs. The calling client will have
> > to
> > > > put
> > > > > > the
> > > > > > > > >> tokens it
> > > > > > > > >> > > > >> wants
> > > > > > > > >> > > > >> >> to use in the subject instance and call
> > > > > produce/consume
> > > > > > > > inside
> > > > > > > > >> > > > >> >> subject.doas. Each caller will have to keep
> > track
> > > of
> > > > > its
> > > > > > > own
> > > > > > > > >> > > > subject. I
> > > > > > > > >> > > > >> >> will have to look at the code to see if we
> > support
> > > > > this
> > > > > > > > >> feature
> > > > > > > > >> > > right
> > > > > > > > >> > > > >> now
> > > > > > > > >> > > > >> >> but my understanding is delegation token
> > shouldn't
> > > > > need
> > > > > > > any
> > > > > > > > >> special
> > > > > > > > >> > > > >> >> treatment as its just another type of
> Credential
> > > in
> > > > > the
> > > > > > > > >> subject.
> > > > > > > > >> > > > >> >>
> > > > > > > > >> > > > >> >> I would also like to know what is your opinion
> > > about
> > > > > > > > infinite
> > > > > > > > >> > > renewal
> > > > > > > > >> > > > >> (my
> > > > > > > > >> > > > >> >> recommendation is to not support this), tokens
> > > > > renewing
> > > > > > > them
> > > > > > > > >> > > self(my
> > > > > > > > >> > > > >> >> recommendation is to not support this) and
> most
> > > > > > > importantly
> > > > > > > > >> your
> > > > > > > > >> > > > choice
> > > > > > > > >> > > > >> >> between the alternatives listed on this thread
> > > > > > > > >> > > > >> >> <
> > > > > > > > >> > > > >>
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> > > > > > > > >> > > > >> >
> > > > > > > > >> > > > >> >> ( I am leaning towards the alternative-2 minus
> > > > > > controller
> > > > > > > > >> > > > distributing
> > > > > > > > >> > > > >> >> secret). Thanks again for reviewing.
> > > > > > > > >> > > > >> >>
> > > > > > > > >> > > > >> >> Thanks
> > > > > > > > >> > > > >> >> Parth
> > > > > > > > >> > > > >> >>
> > > > > > > > >> > > > >> >>
> > > > > > > > >> > > > >> >>
> > > > > > > > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <
> > > > > > > > >> gwen@confluent.io>
> > > > > > > > >> > > > wrote:
> > > > > > > > >> > > > >> >>
> > > > > > > > >> > > > >> >> > Harsha,
> > > > > > > > >> > > > >> >> >
> > > > > > > > >> > > > >> >> > I was thinking of the Rest Proxy. I didn't
> see
> > > > your
> > > > > > > design
> > > > > > > > >> yet,
> > > > > > > > >> > > > but in
> > > > > > > > >> > > > >> >> > our proxy, we have a set of producers, which
> > > will
> > > > > > serve
> > > > > > > > >> multiple
> > > > > > > > >> > > > users
> > > > > > > > >> > > > >> >> > going through the proxy. Since these users
> > will
> > > > have
> > > > > > > > >> different
> > > > > > > > >> > > > >> >> > privileges, they'll need to authenticate
> > > > separately,
> > > > > > and
> > > > > > > > >> can't
> > > > > > > > >> > > > share a
> > > > > > > > >> > > > >> >> > token.
> > > > > > > > >> > > > >> >> >
> > > > > > > > >> > > > >> >> > Am I missing anything?
> > > > > > > > >> > > > >> >> >
> > > > > > > > >> > > > >> >> > Gwen
> > > > > > > > >> > > > >> >> >
> > > > > > > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <
> > > > > > kafka@harsha.io
> > > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >> > > > >> >> > > Gwen,
> > > > > > > > >> > > > >> >> > >            On your second point. Can you
> > > > describe
> > > > > a
> > > > > > > > >> usecase
> > > > > > > > >> > > where
> > > > > > > > >> > > > >> >> > >            mutliple clients ended up
> > sharing a
> > > > > > > producer
> > > > > > > > >> and
> > > > > > > > >> > > even
> > > > > > > > >> > > > if
> > > > > > > > >> > > > >> they
> > > > > > > > >> > > > >> >> > >            do why can't they not use
> single
> > > > token
> > > > > > that
> > > > > > > > >> producer
> > > > > > > > >> > > > >> >> > >            captures. Why would we need
> > > multiple
> > > > > > > clients
> > > > > > > > >> with
> > > > > > > > >> > > > >> different
> > > > > > > > >> > > > >> >> > >            tokens sharing a single
> instance
> > of
> > > > > > > producer.
> > > > > > > > >> Also
> > > > > > > > >> > > in
> > > > > > > > >> > > > >> this
> > > > > > > > >> > > > >> >> > >            case other clients have access
> > all
> > > > the
> > > > > > > tokens
> > > > > > > > >> no?
> > > > > > > > >> > > > >> >> > >
> > > > > > > > >> > > > >> >> > > Thanks,
> > > > > > > > >> > > > >> >> > > Harsha
> > > > > > > > >> > > > >> >> > >
> > > > > > > > >> > > > >> >> > >
> > > > > > > > >> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen
> > Shapira
> > > > > > wrote:
> > > > > > > > >> > > > >> >> > >> Sorry for the delay:
> > > > > > > > >> > > > >> >> > >>
> > > > > > > > >> > > > >> >> > >> Two questions that we didn't see in the
> > wiki:
> > > > > > > > >> > > > >> >> > >> 1. Is there an expiration for delegation
> > > > tokens?
> > > > > > > > >> Renewal? How
> > > > > > > > >> > > > do we
> > > > > > > > >> > > > >> >> > >> revoke them?
> > > > > > > > >> > > > >> >> > >> 2. If we want to use delegation tokens
> for
> > > > > "do-as"
> > > > > > > > (say,
> > > > > > > > >> > > submit
> > > > > > > > >> > > > >> Storm
> > > > > > > > >> > > > >> >> > >> job as my user), we will need a producer
> > for
> > > > > every
> > > > > > > job
> > > > > > > > >> (we
> > > > > > > > >> > > can't
> > > > > > > > >> > > > >> share
> > > > > > > > >> > > > >> >> > >> them between multiple jobs running on
> same
> > > > node),
> > > > > > > since
> > > > > > > > >> we
> > > > > > > > >> > > only
> > > > > > > > >> > > > >> >> > >> authenticate when connecting. Is there a
> > plan
> > > > to
> > > > > > > change
> > > > > > > > >> this
> > > > > > > > >> > > for
> > > > > > > > >> > > > >> >> > >> delegation tokens, in order to allow
> > multiple
> > > > > users
> > > > > > > > with
> > > > > > > > >> > > > different
> > > > > > > > >> > > > >> >> > >> tokens to share a client?
> > > > > > > > >> > > > >> >> > >>
> > > > > > > > >> > > > >> >> > >> Gwen
> > > > > > > > >> > > > >> >> > >>
> > > > > > > > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth
> > > > brahmbhatt
> > > > > > > > >> > > > >> >> > >> <br...@gmail.com> wrote:
> > > > > > > > >> > > > >> >> > >> > Bumping this up one more time, can
> other
> > > > > > committers
> > > > > > > > >> review?
> > > > > > > > >> > > > >> >> > >> >
> > > > > > > > >> > > > >> >> > >> > Thanks
> > > > > > > > >> > > > >> >> > >> > Parth
> > > > > > > > >> > > > >> >> > >> >
> > > > > > > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM,
> Harsha <
> > > > > > > > >> kafka@harsha.io>
> > > > > > > > >> > > > wrote:
> > > > > > > > >> > > > >> >> > >> >
> > > > > > > > >> > > > >> >> > >> >> Parth,
> > > > > > > > >> > > > >> >> > >> >>           Overall current design looks
> > > good
> > > > to
> > > > > > > me. I
> > > > > > > > >> am +1
> > > > > > > > >> > > on
> > > > > > > > >> > > > >> the
> > > > > > > > >> > > > >> >> > KIP.
> > > > > > > > >> > > > >> >> > >> >>
> > > > > > > > >> > > > >> >> > >> >> Gwen , Jun can you review this as
> well.
> > > > > > > > >> > > > >> >> > >> >>
> > > > > > > > >> > > > >> >> > >> >> -Harsha
> > > > > > > > >> > > > >> >> > >> >>
> > > > > > > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM,
> parth
> > > > > > > brahmbhatt
> > > > > > > > >> wrote:
> > > > > > > > >> > > > >> >> > >> >> > Thanks for review Jitendra.
> > > > > > > > >> > > > >> >> > >> >> >
> > > > > > > > >> > > > >> >> > >> >> > I don't like the idea of infinite
> > > lifetime
> > > > > > but I
> > > > > > > > >> see the
> > > > > > > > >> > > > >> Streaming
> > > > > > > > >> > > > >> >> > use
> > > > > > > > >> > > > >> >> > >> >> > case. Even for Streaming use case I
> > was
> > > > > hoping
> > > > > > > > >> there will
> > > > > > > > >> > > > be
> > > > > > > > >> > > > >> some
> > > > > > > > >> > > > >> >> > notion
> > > > > > > > >> > > > >> >> > >> >> > of
> > > > > > > > >> > > > >> >> > >> >> > master/driver that can get new
> > > delegation
> > > > > > tokens
> > > > > > > > at
> > > > > > > > >> fixed
> > > > > > > > >> > > > >> interval
> > > > > > > > >> > > > >> >> > and
> > > > > > > > >> > > > >> >> > >> >> > distribute to workers. If that is
> not
> > > the
> > > > > case
> > > > > > > for
> > > > > > > > >> we can
> > > > > > > > >> > > > >> discuss
> > > > > > > > >> > > > >> >> > >> >> > delegation tokens renewing them self
> > and
> > > > the
> > > > > > > > >> security
> > > > > > > > >> > > > >> implications
> > > > > > > > >> > > > >> >> > of the
> > > > > > > > >> > > > >> >> > >> >> > same.
> > > > > > > > >> > > > >> >> > >> >> >
> > > > > > > > >> > > > >> >> > >> >> > I did not want clients to fetch
> tokens
> > > > from
> > > > > > > > >> zookeeper,
> > > > > > > > >> > > > >> overall I
> > > > > > > > >> > > > >> >> > think
> > > > > > > > >> > > > >> >> > >> >> > its
> > > > > > > > >> > > > >> >> > >> >> > better if clients don't rely on our
> > > > metadata
> > > > > > > store
> > > > > > > > >> and I
> > > > > > > > >> > > > >> think we
> > > > > > > > >> > > > >> >> > are
> > > > > > > > >> > > > >> >> > >> >> > moving in that direction with all
> the
> > > > KIP-4
> > > > > > > > >> improvements.
> > > > > > > > >> > > > I
> > > > > > > > >> > > > >> chose
> > > > > > > > >> > > > >> >> > >> >> > zookeeper as in this case the client
> > > will
> > > > > > still
> > > > > > > > >> just talk
> > > > > > > > >> > > > to
> > > > > > > > >> > > > >> >> > broker , its
> > > > > > > > >> > > > >> >> > >> >> > the brokers that will use zookeeper
> > > which
> > > > we
> > > > > > > > >> already do
> > > > > > > > >> > > > for a
> > > > > > > > >> > > > >> lot
> > > > > > > > >> > > > >> >> > of
> > > > > > > > >> > > > >> >> > >> >> > other
> > > > > > > > >> > > > >> >> > >> >> > usecases + ease of development + and
> > the
> > > > > > ability
> > > > > > > > so
> > > > > > > > >> > > tokens
> > > > > > > > >> > > > >> will
> > > > > > > > >> > > > >> >> > survive
> > > > > > > > >> > > > >> >> > >> >> > even a rolling restart/cluster
> > failure.
> > > > if a
> > > > > > > > >> majority
> > > > > > > > >> > > > agrees
> > > > > > > > >> > > > >> the
> > > > > > > > >> > > > >> >> > added
> > > > > > > > >> > > > >> >> > >> >> > complexity to have controller
> > forwarding
> > > > > keys
> > > > > > to
> > > > > > > > all
> > > > > > > > >> > > > broker is
> > > > > > > > >> > > > >> >> > justified
> > > > > > > > >> > > > >> >> > >> >> > as
> > > > > > > > >> > > > >> >> > >> >> > it provides tighter security , I am
> > fine
> > > > > with
> > > > > > > that
> > > > > > > > >> option
> > > > > > > > >> > > > too.
> > > > > > > > >> > > > >> >> > >> >> >
> > > > > > > > >> > > > >> >> > >> >> > Given zookeeper does not support SSL
> > we
> > > > can
> > > > > > not
> > > > > > > > >> store
> > > > > > > > >> > > > master
> > > > > > > > >> > > > >> keys
> > > > > > > > >> > > > >> >> > in
> > > > > > > > >> > > > >> >> > >> >> > zookeeper as master keys will be
> > exposed
> > > > on
> > > > > > > wire.
> > > > > > > > To
> > > > > > > > >> > > > support
> > > > > > > > >> > > > >> >> > rotation
> > > > > > > > >> > > > >> >> > >> >> > without affecting current clients is
> > > > > > something I
> > > > > > > > >> need to
> > > > > > > > >> > > > put
> > > > > > > > >> > > > >> more
> > > > > > > > >> > > > >> >> > thought
> > > > > > > > >> > > > >> >> > >> >> > in. My current proposal assumes the
> > > > rotation
> > > > > > > will
> > > > > > > > >> > > > invalidate
> > > > > > > > >> > > > >> all
> > > > > > > > >> > > > >> >> > current
> > > > > > > > >> > > > >> >> > >> >> > tokens.
> > > > > > > > >> > > > >> >> > >> >> >
> > > > > > > > >> > > > >> >> > >> >> > I request committers to also review
> > and
> > > > post
> > > > > > > their
> > > > > > > > >> > > comments
> > > > > > > > >> > > > >> so we
> > > > > > > > >> > > > >> >> > can
> > > > > > > > >> > > > >> >> > >> >> > make
> > > > > > > > >> > > > >> >> > >> >> > progress on this KIP.
> > > > > > > > >> > > > >> >> > >> >> >
> > > > > > > > >> > > > >> >> > >> >> > Thanks
> > > > > > > > >> > > > >> >> > >> >> > Parth
> > > > > > > > >> > > > >> >> > >> >> >
> > > > > > > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM,
> > Ashish
> > > > > Singh
> > > > > > <
> > > > > > > > >> > > > >> asingh@cloudera.com
> > > > > > > > >> > > > >> >> > >
> > > > > > > > >> > > > >> >> > >> >> > wrote:
> > > > > > > > >> > > > >> >> > >> >> >
> > > > > > > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM,
> > > > Harsha <
> > > > > > > > >> > > > kafka@harsha.io>
> > > > > > > > >> > > > >> >> > wrote:
> > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > Unifying the two discussion
> > threads
> > > on
> > > > > > this
> > > > > > > > KIP.
> > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > >> > > > >> >> > >> >> > > > Here is the response from
> Jitendra
> > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > >> > > > >> >> > >> >> > > > "The need for a large number of
> > > > clients
> > > > > > that
> > > > > > > > are
> > > > > > > > >> > > > running
> > > > > > > > >> > > > >> all
> > > > > > > > >> > > > >> >> > over the
> > > > > > > > >> > > > >> >> > >> >> > > > cluster that authenticate with
> > Kafka
> > > > > > > brokers,
> > > > > > > > >> is very
> > > > > > > > >> > > > >> similar
> > > > > > > > >> > > > >> >> > to the
> > > > > > > > >> > > > >> >> > >> >> > > > Hadoop use case of large number
> of
> > > > tasks
> > > > > > > > running
> > > > > > > > >> > > across
> > > > > > > > >> > > > >> the
> > > > > > > > >> > > > >> >> > cluster
> > > > > > > > >> > > > >> >> > >> >> that
> > > > > > > > >> > > > >> >> > >> >> > > > need authentication to Hdfs
> > > Namenode.
> > > > > > > > >> Therefore, the
> > > > > > > > >> > > > >> >> > delegation token
> > > > > > > > >> > > > >> >> > >> >> > > > approach does seem like a good
> fit
> > > for
> > > > > > this
> > > > > > > > use
> > > > > > > > >> case
> > > > > > > > >> > > > as we
> > > > > > > > >> > > > >> >> > have seen
> > > > > > > > >> > > > >> >> > >> >> it
> > > > > > > > >> > > > >> >> > >> >> > > > working at large scale in HDFS
> and
> > > > YARN.
> > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > >> > > > >> >> > >> >> > > >   The proposed design is very
> much
> > > > > inline
> > > > > > > with
> > > > > > > > >> Hadoop
> > > > > > > > >> > > > >> >> > approach. A few
> > > > > > > > >> > > > >> >> > >> >> > > >   comments:
> > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > >> > > > >> >> > >> >> > > > 1) Why do you guys want to allow
> > > > > infinite
> > > > > > > > >> renewable
> > > > > > > > >> > > > >> lifetime
> > > > > > > > >> > > > >> >> > for a
> > > > > > > > >> > > > >> >> > >> >> > > > token? HDFS restricts a token
> to a
> > > max
> > > > > > life
> > > > > > > > time
> > > > > > > > >> > > > (default
> > > > > > > > >> > > > >> 7
> > > > > > > > >> > > > >> >> > days).  A
> > > > > > > > >> > > > >> >> > >> >> > > > token's vulnerability is
> believed
> > to
> > > > > > > increase
> > > > > > > > >> with
> > > > > > > > >> > > > time.
> > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > >> > > > >> >> > >> >> > > I agree that having infinite
> > lifetime
> > > > > might
> > > > > > > not
> > > > > > > > >> be the
> > > > > > > > >> > > > best
> > > > > > > > >> > > > >> idea.
> > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > >> > > > >> >> > >> >> > > > 2) As I understand the tokens
> are
> > > > stored
> > > > > > in
> > > > > > > > >> zookeeper
> > > > > > > > >> > > > as
> > > > > > > > >> > > > >> well,
> > > > > > > > >> > > > >> >> > and
> > > > > > > > >> > > > >> >> > >> >> can
> > > > > > > > >> > > > >> >> > >> >> > > > be updated there. This is clever
> > as
> > > it
> > > > > can
> > > > > > > > allow
> > > > > > > > >> > > > >> replacing the
> > > > > > > > >> > > > >> >> > tokens
> > > > > > > > >> > > > >> >> > >> >> > > > once they run out of max life
> > time,
> > > > and
> > > > > > > > clients
> > > > > > > > >> can
> > > > > > > > >> > > > >> download
> > > > > > > > >> > > > >> >> > new
> > > > > > > > >> > > > >> >> > >> >> tokens
> > > > > > > > >> > > > >> >> > >> >> > > > from zookeeper. It shouldn't be
> a
> > > big
> > > > > load
> > > > > > > on
> > > > > > > > >> > > zookeeper
> > > > > > > > >> > > > >> as a
> > > > > > > > >> > > > >> >> > client
> > > > > > > > >> > > > >> >> > >> >> will
> > > > > > > > >> > > > >> >> > >> >> > > > need to get a new token once in
> > > > several
> > > > > > > days.
> > > > > > > > >> In this
> > > > > > > > >> > > > >> approach
> > > > > > > > >> > > > >> >> > you
> > > > > > > > >> > > > >> >> > >> >> don't
> > > > > > > > >> > > > >> >> > >> >> > > > need infinite lifetime on the
> > token
> > > > even
> > > > > > for
> > > > > > > > >> long
> > > > > > > > >> > > > running
> > > > > > > > >> > > > >> >> > clients.
> > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > >> > > > >> >> > >> >> > > > 3) The token password are
> > generated
> > > > > using
> > > > > > a
> > > > > > > > >> master
> > > > > > > > >> > > key.
> > > > > > > > >> > > > >> The
> > > > > > > > >> > > > >> >> > master
> > > > > > > > >> > > > >> >> > >> >> key
> > > > > > > > >> > > > >> >> > >> >> > > > should also be periodically
> > changed.
> > > > In
> > > > > > > > Hadoop,
> > > > > > > > >> the
> > > > > > > > >> > > > >> default
> > > > > > > > >> > > > >> >> > renewal
> > > > > > > > >> > > > >> >> > >> >> > > > period is 1 day.?
> > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > >> > > > >> >> > >> >> > > IIUC, this will require brokers
> > > > > maintaining
> > > > > > a
> > > > > > > > >> list of X
> > > > > > > > >> > > > most
> > > > > > > > >> > > > >> >> > recent
> > > > > > > > >> > > > >> >> > >> >> master
> > > > > > > > >> > > > >> >> > >> >> > > keys. This list will have to be
> > > > persisted
> > > > > > > > >> somewhere, as
> > > > > > > > >> > > > if a
> > > > > > > > >> > > > >> >> > broker
> > > > > > > > >> > > > >> >> > >> >> goes
> > > > > > > > >> > > > >> >> > >> >> > > down it will have to get that list
> > > again
> > > > > and
> > > > > > > > >> storing
> > > > > > > > >> > > > master
> > > > > > > > >> > > > >> keys
> > > > > > > > >> > > > >> >> > on ZK
> > > > > > > > >> > > > >> >> > >> >> is
> > > > > > > > >> > > > >> >> > >> >> > > not the best idea. However, if a
> > > broker
> > > > > goes
> > > > > > > > down
> > > > > > > > >> then
> > > > > > > > >> > > we
> > > > > > > > >> > > > >> have
> > > > > > > > >> > > > >> >> > much
> > > > > > > > >> > > > >> >> > >> >> bigger
> > > > > > > > >> > > > >> >> > >> >> > > issue to deal with and client can
> > > always
> > > > > > > > >> > > re-authenticate
> > > > > > > > >> > > > is
> > > > > > > > >> > > > >> such
> > > > > > > > >> > > > >> >> > >> >> events.
> > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > >> > > > >> >> > >> >> > > Did you happen to take a look at
> > other
> > > > > > > > >> alternatives
> > > > > > > > >> > > this
> > > > > > > > >> > > > >> list has
> > > > > > > > >> > > > >> >> > >> >> > > suggested?
> > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > >> > > > >> >> > >> >> > > > Thanks for a thorough proposal,
> > > great
> > > > > > work!"
> > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28
> PM,
> > > Gwen
> > > > > > > Shapira
> > > > > > > > >> wrote:
> > > > > > > > >> > > > >> >> > >> >> > > > > Makes sense to me. Thanks!
> > > > > > > > >> > > > >> >> > >> >> > > > >
> > > > > > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25
> PM,
> > > > > Harsha <
> > > > > > > > >> > > > kafka@harsha.io
> > > > > > > > >> > > > >> >
> > > > > > > > >> > > > >> >> > wrote:
> > > > > > > > >> > > > >> >> > >> >> > > > > > It doesn't need any release
> > > > vehicle
> > > > > > but
> > > > > > > > >> still the
> > > > > > > > >> > > > >> work can
> > > > > > > > >> > > > >> >> > move
> > > > > > > > >> > > > >> >> > >> >> > > > forward.
> > > > > > > > >> > > > >> >> > >> >> > > > > > If anyone is interested in
> the
> > > KIP
> > > > > > > please
> > > > > > > > >> do the
> > > > > > > > >> > > > >> review and
> > > > > > > > >> > > > >> >> > >> >> provide
> > > > > > > > >> > > > >> >> > >> >> > > the
> > > > > > > > >> > > > >> >> > >> >> > > > > > comments.
> > > > > > > > >> > > > >> >> > >> >> > > > > >
> > > > > > > > >> > > > >> >> > >> >> > > > > > -Harsha
> > > > > > > > >> > > > >> >> > >> >> > > > > >
> > > > > > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at
> 04:59
> > > PM,
> > > > > > Ismael
> > > > > > > > >> Juma
> > > > > > > > >> > > > wrote:
> > > > > > > > >> > > > >> >> > >> >> > > > > >> I agree that it would be
> good
> > > to
> > > > > have
> > > > > > > > more
> > > > > > > > >> time
> > > > > > > > >> > > to
> > > > > > > > >> > > > >> review
> > > > > > > > >> > > > >> >> > and
> > > > > > > > >> > > > >> >> > >> >> > > discuss
> > > > > > > > >> > > > >> >> > >> >> > > > > >> KIP-48.
> > > > > > > > >> > > > >> >> > >> >> > > > > >>
> > > > > > > > >> > > > >> >> > >> >> > > > > >> Ismael
> > > > > > > > >> > > > >> >> > >> >> > > > > >>
> > > > > > > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at
> 12:55
> > > AM,
> > > > > Gwen
> > > > > > > > >> Shapira <
> > > > > > > > >> > > > >> >> > >> >> gwen@confluent.io>
> > > > > > > > >> > > > >> >> > >> >> > > > wrote:
> > > > > > > > >> > > > >> >> > >> >> > > > > >>
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > Hi Team,
> > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > Since KIP-48 depends on
> > > KIP-43,
> > > > > > which
> > > > > > > > is
> > > > > > > > >> > > > already a
> > > > > > > > >> > > > >> bit
> > > > > > > > >> > > > >> >> > of a
> > > > > > > > >> > > > >> >> > >> >> risk
> > > > > > > > >> > > > >> >> > >> >> > > for
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > the next release - any
> > chance
> > > > we
> > > > > > can
> > > > > > > > >> delay
> > > > > > > > >> > > > >> delegation
> > > > > > > > >> > > > >> >> > tokens
> > > > > > > > >> > > > >> >> > >> >> to
> > > > > > > > >> > > > >> >> > >> >> > > > Kafka
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > 0.10.1?
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > With the community
> working
> > > on a
> > > > > > > release
> > > > > > > > >> every
> > > > > > > > >> > > 3
> > > > > > > > >> > > > >> month,
> > > > > > > > >> > > > >> >> > this
> > > > > > > > >> > > > >> >> > >> >> is not
> > > > > > > > >> > > > >> >> > >> >> > > > a huge
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > delay.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > Gwen
> > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at
> > 5:11
> > > > PM,
> > > > > > > Ashish
> > > > > > > > >> Singh
> > > > > > > > >> > > <
> > > > > > > > >> > > > >> >> > >> >> > > asingh@cloudera.com>
> > > > > > > > >> > > > >> >> > >> >> > > > wrote:
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Parth,
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Thanks again for the
> > > awesome
> > > > > > write
> > > > > > > > up.
> > > > > > > > >> > > > Following
> > > > > > > > >> > > > >> our
> > > > > > > > >> > > > >> >> > >> >> discussion
> > > > > > > > >> > > > >> >> > >> >> > > > from the
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I think it will
> be
> > > > easier
> > > > > > to
> > > > > > > > >> compare
> > > > > > > > >> > > > >> various
> > > > > > > > >> > > > >> >> > >> >> alternatives
> > > > > > > > >> > > > >> >> > >> >> > > > if they
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > are
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > > listed together. I am
> > > stating
> > > > > > > below a
> > > > > > > > >> few
> > > > > > > > >> > > > >> >> > alternatives along
> > > > > > > > >> > > > >> >> > >> >> > > with
> > > > > > > > >> > > > >> >> > >> >> > > > a the
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > > current proposal.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Current proposal)
> Store
> > > > > > Delegation
> > > > > > > > >> Token,
> > > > > > > > >> > > DT,
> > > > > > > > >> > > > >> on ZK.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> > authenticates
> > > > > with a
> > > > > > > > >> broker.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > > > > > > authenticated,
> > > > > > > > >> it
> > > > > > > > >> > > will
> > > > > > > > >> > > > >> make a
> > > > > > > > >> > > > >> >> > broker
> > > > > > > > >> > > > >> >> > >> >> side
> > > > > > > > >> > > > >> >> > >> >> > > > call to
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation
> > > token.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker
> > generates
> > > a
> > > > > > shared
> > > > > > > > >> secret
> > > > > > > > >> > > > based
> > > > > > > > >> > > > >> on
> > > > > > > > >> > > > >> >> > >> >> > > HMAC-SHA256(a
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    Password/Secret
> shared
> > > > > between
> > > > > > > all
> > > > > > > > >> > > brokers,
> > > > > > > > >> > > > >> >> > randomly
> > > > > > > > >> > > > >> >> > >> >> > > generated
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > tokenId).
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Broker stores
> this
> > > > token
> > > > > in
> > > > > > > its
> > > > > > > > >> in
> > > > > > > > >> > > > memory
> > > > > > > > >> > > > >> cache.
> > > > > > > > >> > > > >> >> > >> >> Broker
> > > > > > > > >> > > > >> >> > >> >> > > > also stores
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the DelegationToken
> > > > without
> > > > > > the
> > > > > > > > >> hmac in
> > > > > > > > >> > > the
> > > > > > > > >> > > > >> >> > zookeeper.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. All brokers will
> > > have a
> > > > > > cache
> > > > > > > > >> backed
> > > > > > > > >> > > by
> > > > > > > > >> > > > >> >> > zookeeper so
> > > > > > > > >> > > > >> >> > >> >> they
> > > > > > > > >> > > > >> >> > >> >> > > > will all
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    get notified
> whenever
> > a
> > > > new
> > > > > > > token
> > > > > > > > is
> > > > > > > > >> > > > >> generated and
> > > > > > > > >> > > > >> >> > they
> > > > > > > > >> > > > >> >> > >> >> will
> > > > > > > > >> > > > >> >> > >> >> > > > update
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > their
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    local cache whenever
> > > token
> > > > > > state
> > > > > > > > >> changes.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Broker returns
> the
> > > > token
> > > > > to
> > > > > > > > >> Client.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and
> fixes
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Probable race
> > > > condition,
> > > > > > > client
> > > > > > > > >> tries
> > > > > > > > >> > > to
> > > > > > > > >> > > > >> >> > authenticate
> > > > > > > > >> > > > >> >> > >> >> with
> > > > > > > > >> > > > >> >> > >> >> > > > a broker
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    that is yet to be
> > > updated
> > > > > with
> > > > > > > the
> > > > > > > > >> newly
> > > > > > > > >> > > > >> generated
> > > > > > > > >> > > > >> >> > DT.
> > > > > > > > >> > > > >> >> > >> >> This
> > > > > > > > >> > > > >> >> > >> >> > > can
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > probably be
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    dealt with making
> > > > dtRequest
> > > > > > > block
> > > > > > > > >> until
> > > > > > > > >> > > all
> > > > > > > > >> > > > >> >> > brokers have
> > > > > > > > >> > > > >> >> > >> >> > > > updated
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > their DT
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    cache. Zk barrier or
> > > > similar
> > > > > > > > >> mechanism
> > > > > > > > >> > > can
> > > > > > > > >> > > > be
> > > > > > > > >> > > > >> used.
> > > > > > > > >> > > > >> >> > >> >> However,
> > > > > > > > >> > > > >> >> > >> >> > > > all such
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    mechanisms will
> > increase
> > > > > > > > complexity.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Using a static
> > secret
> > > > key
> > > > > > > from
> > > > > > > > >> config
> > > > > > > > >> > > > >> file. Will
> > > > > > > > >> > > > >> >> > >> >> require
> > > > > > > > >> > > > >> >> > >> >> > > yet
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > another
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    config and uses a
> > static
> > > > > > secret
> > > > > > > > >> key. It
> > > > > > > > >> > > is
> > > > > > > > >> > > > >> advised
> > > > > > > > >> > > > >> >> > to
> > > > > > > > >> > > > >> >> > >> >> rotate
> > > > > > > > >> > > > >> >> > >> >> > > > secret
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > keys
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    periodically. This
> can
> > > be
> > > > > > > avoided
> > > > > > > > >> with
> > > > > > > > >> > > > >> controller
> > > > > > > > >> > > > >> >> > >> >> generating
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > secretKey and
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    passing to brokers
> > > > > > periodically.
> > > > > > > > >> However,
> > > > > > > > >> > > > >> this will
> > > > > > > > >> > > > >> >> > >> >> require
> > > > > > > > >> > > > >> >> > >> >> > > > brokers to
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maintain certain
> > counts
> > > of
> > > > > > > > >> secretKeys.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 1) Have
> > > > controller
> > > > > > > > >> generate
> > > > > > > > >> > > > >> delegation
> > > > > > > > >> > > > >> >> > token.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> > authenticates
> > > > > with a
> > > > > > > > >> broker.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > > > > > > authenticated,
> > > > > > > > >> it
> > > > > > > > >> > > will
> > > > > > > > >> > > > >> make a
> > > > > > > > >> > > > >> >> > broker
> > > > > > > > >> > > > >> >> > >> >> side
> > > > > > > > >> > > > >> >> > >> >> > > > call to
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation
> > > token.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. Broker forwards
> the
> > > > > request
> > > > > > > to
> > > > > > > > >> > > > controller.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Controller
> > generates
> > > a
> > > > DT
> > > > > > and
> > > > > > > > >> > > broadcasts
> > > > > > > > >> > > > >> to all
> > > > > > > > >> > > > >> >> > >> >> brokers.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. Broker stores
> this
> > > > token
> > > > > in
> > > > > > > its
> > > > > > > > >> memory
> > > > > > > > >> > > > >> cache.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Controller
> responds
> > > to
> > > > > > > broker’s
> > > > > > > > >> DT
> > > > > > > > >> > > req.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    7. Broker returns
> the
> > > > token
> > > > > to
> > > > > > > > >> Client.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and
> fixes
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. We will have to
> add
> > > new
> > > > > > APIs
> > > > > > > to
> > > > > > > > >> > > support
> > > > > > > > >> > > > >> >> > controller
> > > > > > > > >> > > > >> >> > >> >> pushing
> > > > > > > > >> > > > >> >> > >> >> > > > tokens
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > to
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    brokers on top of
> the
> > > > > minimal
> > > > > > > APIs
> > > > > > > > >> that
> > > > > > > > >> > > are
> > > > > > > > >> > > > >> >> > currently
> > > > > > > > >> > > > >> >> > >> >> > > proposed.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. We will also have
> > to
> > > > add
> > > > > > APIs
> > > > > > > > to
> > > > > > > > >> > > support
> > > > > > > > >> > > > >> the
> > > > > > > > >> > > > >> >> > >> >> bootstrapping
> > > > > > > > >> > > > >> >> > >> >> > > > case,
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > i.e,
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    when a new broker
> > comes
> > > up
> > > > > it
> > > > > > > will
> > > > > > > > >> have
> > > > > > > > >> > > to
> > > > > > > > >> > > > >> get all
> > > > > > > > >> > > > >> >> > >> >> delegation
> > > > > > > > >> > > > >> >> > >> >> > > > tokens
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > from
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the controller.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. In catastrophic
> > > > failures
> > > > > > > where
> > > > > > > > >> all
> > > > > > > > >> > > > brokers
> > > > > > > > >> > > > >> go
> > > > > > > > >> > > > >> >> > down,
> > > > > > > > >> > > > >> >> > >> >> the
> > > > > > > > >> > > > >> >> > >> >> > > > tokens will
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if
> > servers
> > > > are
> > > > > > > > >> restarted as
> > > > > > > > >> > > > >> tokens
> > > > > > > > >> > > > >> >> > are not
> > > > > > > > >> > > > >> >> > >> >> > > > persisted
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this happens,
> then
> > > > there
> > > > > > are
> > > > > > > > more
> > > > > > > > >> > > > important
> > > > > > > > >> > > > >> >> > things to
> > > > > > > > >> > > > >> >> > >> >> > > worry
> > > > > > > > >> > > > >> >> > >> >> > > > about
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is better
> to
> > > > > > > > >> re-authenticate.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 2) Do not
> > > > > distribute
> > > > > > > DT
> > > > > > > > to
> > > > > > > > >> > > other
> > > > > > > > >> > > > >> brokers
> > > > > > > > >> > > > >> >> > at
> > > > > > > > >> > > > >> >> > >> >> all.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> > authenticates
> > > > > with a
> > > > > > > > >> broker.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > > > > > > authenticated,
> > > > > > > > >> it
> > > > > > > > >> > > will
> > > > > > > > >> > > > >> make a
> > > > > > > > >> > > > >> >> > broker
> > > > > > > > >> > > > >> >> > >> >> side
> > > > > > > > >> > > > >> >> > >> >> > > > call to
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation
> > > token.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker
> > generates
> > > DT
> > > > > of
> > > > > > > > form,
> > > > > > > > >> > > [hmac +
> > > > > > > > >> > > > >> (owner,
> > > > > > > > >> > > > >> >> > >> >> renewer,
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maxLifeTime, id,
> hmac,
> > > > > > > > >> expirationTime)]
> > > > > > > > >> > > and
> > > > > > > > >> > > > >> passes
> > > > > > > > >> > > > >> >> > back
> > > > > > > > >> > > > >> >> > >> >> this
> > > > > > > > >> > > > >> >> > >> >> > > > DT to
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > client.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    hmac is generated
> via
> > > > > > > > >> {HMAC-SHA256(owner,
> > > > > > > > >> > > > >> renewer,
> > > > > > > > >> > > > >> >> > >> >> > > > maxLifeTime, id,
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac,
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    expirationTime)
> using
> > > > > > > SecretKey}.
> > > > > > > > >> Note
> > > > > > > > >> > > that
> > > > > > > > >> > > > >> all
> > > > > > > > >> > > > >> >> > brokers
> > > > > > > > >> > > > >> >> > >> >> have
> > > > > > > > >> > > > >> >> > >> >> > > > this
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > SecretKey.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Client then goes
> to
> > > any
> > > > > > > broker
> > > > > > > > >> and to
> > > > > > > > >> > > > >> >> > authenticate
> > > > > > > > >> > > > >> >> > >> >> sends
> > > > > > > > >> > > > >> >> > >> >> > > > the DT.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    Broker recalculates
> > hmac
> > > > > using
> > > > > > > > >> (owner,
> > > > > > > > >> > > > >> renewer,
> > > > > > > > >> > > > >> >> > >> >> maxLifeTime,
> > > > > > > > >> > > > >> >> > >> >> > > > id, hmac,
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    expirationTime) info
> > > from
> > > > DT
> > > > > > and
> > > > > > > > its
> > > > > > > > >> > > > >> SecretKey. If
> > > > > > > > >> > > > >> >> > it
> > > > > > > > >> > > > >> >> > >> >> matches
> > > > > > > > >> > > > >> >> > >> >> > > > with
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac of
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    DT, client is
> > > > authenticated.
> > > > > > > Yes,
> > > > > > > > >> it will
> > > > > > > > >> > > > do
> > > > > > > > >> > > > >> other
> > > > > > > > >> > > > >> >> > >> >> obvious
> > > > > > > > >> > > > >> >> > >> >> > > > checks of
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    timestamp expiry and
> > > such.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Note that secret key
> will
> > > be
> > > > > > > > generated
> > > > > > > > >> by
> > > > > > > > >> > > > >> controller
> > > > > > > > >> > > > >> >> > and
> > > > > > > > >> > > > >> >> > >> >> passed
> > > > > > > > >> > > > >> >> > >> >> > > to
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > brokers
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > > periodically.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and
> fixes
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. How to delete a
> DT?
> > > > Yes,
> > > > > > that
> > > > > > > > is
> > > > > > > > >> a
> > > > > > > > >> > > > downside
> > > > > > > > >> > > > >> >> > here.
> > > > > > > > >> > > > >> >> > >> >> However,
> > > > > > > > >> > > > >> >> > >> >> > > > this can
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be handled with
> > brokers
> > > > > > > > maintaining
> > > > > > > > >> a
> > > > > > > > >> > > > >> blacklist of
> > > > > > > > >> > > > >> >> > DTs,
> > > > > > > > >> > > > >> >> > >> >> DTs
> > > > > > > > >> > > > >> >> > >> >> > > > from this
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > list
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    can be removed after
> > > > expiry.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. In catastrophic
> > > > failures
> > > > > > > where
> > > > > > > > >> all
> > > > > > > > >> > > > brokers
> > > > > > > > >> > > > >> go
> > > > > > > > >> > > > >> >> > down,
> > > > > > > > >> > > > >> >> > >> >> the
> > > > > > > > >> > > > >> >> > >> >> > > > tokens will
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if
> > servers
> > > > are
> > > > > > > > >> restarted as
> > > > > > > > >> > > > >> tokens
> > > > > > > > >> > > > >> >> > are not
> > > > > > > > >> > > > >> >> > >> >> > > > persisted
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this happens,
> then
> > > > there
> > > > > > are
> > > > > > > > more
> > > > > > > > >> > > > important
> > > > > > > > >> > > > >> >> > things to
> > > > > > > > >> > > > >> >> > >> >> > > worry
> > > > > > > > >> > > > >> >> > >> >> > > > about
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is better
> to
> > > > > > > > >> re-authenticate.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at
> > > 1:58
> > > > > PM,
> > > > > > > > Parth
> > > > > > > > >> > > > >> Brahmbhatt <
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > pbrahmbhatt@hortonworks.com>
> > > > > > > wrote:
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Hi,
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> I have filed KIP-48 so
> > we
> > > > can
> > > > > > > offer
> > > > > > > > >> hadoop
> > > > > > > > >> > > > like
> > > > > > > > >> > > > >> >> > delegation
> > > > > > > > >> > > > >> >> > >> >> > > > tokens in
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> kafka. You can review
> > the
> > > > > design
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > >> > > > >> >> > >> >>
> > > > > > > > >> > > > >> >> >
> > > > > > > > >> > > > >>
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > .
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> This KIP depends on
> > KIP-43
> > > > and
> > > > > > we
> > > > > > > > >> have also
> > > > > > > > >> > > > >> >> > discussed an
> > > > > > > > >> > > > >> >> > >> >> > > > alternative to
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> proposed design here<
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > >> > > > >> >> > >> >>
> > > > > > > > >> > > > >> >> >
> > > > > > > > >> > > > >>
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >.
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Thanks
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Parth
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > > --
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Regards,
> > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Ashish
> > > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > >> > > > >> >> > >> >> > > --
> > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > >> > > > >> >> > >> >> > > Regards,
> > > > > > > > >> > > > >> >> > >> >> > > Ashish
> > > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > > >> > > > >> >> > >> >>
> > > > > > > > >> > > > >> >> >
> > > > > > > > >> > > > >>
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Rajini
> > >
> >
>



-- 
Liquan Pei
Software Engineer, Confluent Inc

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[parth brahmbhatt <br...@gmail.com>]

110. What does getDelegationTokenAs mean?
In the current proposal we only allow a user to get delegation token for
the identity that it authenticated as using another mechanism, i.e. A user
that authenticate using a keytab for principal user1@EXAMPLE.COM will get
delegation tokens for that user only. In future I think we will have to
extend support such that we allow some set of users (
kafka-rest-user@EXAMPLE.COM, storm-nimbus@EXAMPLE.COM) to acquire
delegation tokens on behalf of other users whose identity they have
verified independently.  Kafka brokers will have ACLs to control which
users are allowed to impersonate other users and get tokens on behalf of
them. Overall Impersonation is a whole different problem in my opinion and
I think we can tackle it in separate KIP.

111. What's the typical rate of getting and renewing delegation tokens?
Typically this should be very very low, 1 request per minute is a
relatively high estimate. However it depends on the token expiration. I am
less worried about the extra load it puts on controller vs the added
complexity and the value it offers.

Thanks
Parth



On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <is...@juma.me.uk> wrote:

> Thanks Rajini. It would probably require a separate KIP as it will
> introduce user visible changes. We could also update KIP-48 to have this
> information, but it seems cleaner to do it separately. We can discuss that
> in the KIP call today.
>
> Ismael
>
> On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <
> rajinisivaram@googlemail.com> wrote:
>
> > Ismael,
> >
> > I have created a JIRA (https://issues.apache.org/jira/browse/KAFKA-3751)
> > for adding SCRAM as a SASL mechanism. Would that need another KIP? If
> > KIP-48 will use this mechanism, can this just be a JIRA that gets
> reviewed
> > when the PR is ready?
> >
> > Thank you,
> >
> > Rajini
> >
> > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Thanks Rajini, SCRAM seems like a good candidate.
> > >
> > > Gwen had independently mentioned this as a SASL mechanism that might be
> > > useful for Kafka and I have been meaning to check it in more detail.
> Good
> > > to know that you are willing to contribute an implementation. Maybe we
> > > should file a separate JIRA for this?
> > >
> > > Ismael
> > >
> > > On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram <
> > > rajinisivaram@googlemail.com> wrote:
> > >
> > > > SCRAM (Salted Challenge Response Authentication Mechanism) is a
> better
> > > > mechanism than Digest-MD5. Java doesn't come with a built-in SCRAM
> > > > SaslServer or SaslClient, but I will be happy to add support in Kafka
> > > since
> > > > it would be a useful mechanism to support anyway.
> > > > https://tools.ietf.org/html/rfc7677 describes the protocol for
> > > > SCRAM-SHA-256.
> > > >
> > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <ju...@confluent.io> wrote:
> > > >
> > > > > Parth,
> > > > >
> > > > > Thanks for the explanation. A couple of more questions.
> > > > >
> > > > > 110. What does getDelegationTokenAs mean?
> > > > >
> > > > > 111. What's the typical rate of getting and renewing delegation
> > tokens?
> > > > > That may have an impact on whether they should be directed to the
> > > > > controller.
> > > > >
> > > > > Jun
> > > > >
> > > > > On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt <
> > > > > brahmbhatt.parth@gmail.com> wrote:
> > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > Thanks for reviewing.
> > > > > >
> > > > > > * We could add a Cluster action to add acls on who can request
> > > > delegation
> > > > > > tokens. I don't see the use case for that yet but down the line
> > when
> > > we
> > > > > > start supporting getDelegationTokenAs it will be necessary.
> > > > > > * Yes we recommend tokens to be only used/distributed over secure
> > > > > channels.
> > > > > > * Depending on what design we end up choosing Invalidation will
> be
> > > > > > responsibility of every broker or controller.
> > > > > > * I am not sure if I documented somewhere that invalidation will
> > > > directly
> > > > > > go through zookeeper but that is not the intent. Invalidation
> will
> > > > either
> > > > > > be request based or due to expiration. No direct zookeeper
> > > interaction
> > > > > from
> > > > > > any client.
> > > > > > * "Broker also stores the DelegationToken without the hmac in the
> > > > > > zookeeper." : Sorry about the confusion. The sole purpose of
> > > zookeeper
> > > > in
> > > > > > this design is as distribution channel for tokens between all
> > brokers
> > > > > and a
> > > > > > layer that ensures only tokens that were generated by making a
> > > request
> > > > > to a
> > > > > > broker will be accepted (more on this in second paragraph). The
> > token
> > > > > > consists of few elements (owner, renewer, uuid , expiration,
> hmac)
> > ,
> > > > one
> > > > > of
> > > > > > which is the finally generated hmac but hmac it self is derivable
> > if
> > > > you
> > > > > > have all the other elements of the token + secret key to generate
> > > hmac.
> > > > > > Given zookeeper does not provide SSL support we do not want the
> > > entire
> > > > > > token to be wire transferred to zookeeper as that will be an
> > insecure
> > > > > wire
> > > > > > transfer. Instead we only store all the other elements of a
> > > delegation
> > > > > > tokens. Brokers can read these elements and because they also
> have
> > > > access
> > > > > > to secret key they will be able to generate hmac on their end.
> > > > > >
> > > > > > One of the alternative proposed is to avoid zookeeper
> altogether. A
> > > > > Client
> > > > > > will call broker with required information (owner, renwer,
> > > expiration)
> > > > > and
> > > > > > get back (signed hmac, uuid). Broker won't store this in
> zookeeper.
> > > > From
> > > > > > this point a client can contact any broker with all the
> delegation
> > > > token
> > > > > > info (owner, rewner, expiration, hmac, uuid) the borker will
> > > regenerate
> > > > > the
> > > > > > hmac and as long as it matches with hmac presented by client ,
> > broker
> > > > > will
> > > > > > allow the request to authenticate.  Only problem with this
> approach
> > > is
> > > > if
> > > > > > the secret key is compromised any client can now generate random
> > > tokens
> > > > > and
> > > > > > they will still be able to authenticate as any user they like.
> with
> > > > > > zookeeper we guarantee that only tokens acquired via a broker
> > (using
> > > > some
> > > > > > auth scheme other than delegation token) will be accepted. We
> need
> > to
> > > > > > discuss which proposal makes more sense and we can go over it in
> > > > > tomorrow's
> > > > > > meeting.
> > > > > >
> > > > > > Also, can you forward the invite to me?
> > > > > >
> > > > > > Thanks
> > > > > > Parth
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <ju...@confluent.io>
> > wrote:
> > > > > >
> > > > > > > Thanks for the KIP. A few comments.
> > > > > > >
> > > > > > > 100. This potentially can be useful for Kafka Connect and Kafka
> > > rest
> > > > > > proxy
> > > > > > > where a worker agent will need to run a task on behalf of a
> > client.
> > > > We
> > > > > > will
> > > > > > > likely need to change how those services use Kafka clients
> > > > > > > (producer/consumer). Instead of a shared client per worker, we
> > will
> > > > > need
> > > > > > a
> > > > > > > client per user task since the authentication happens at the
> > > > connection
> > > > > > > level. For Kafka Connect, the renewer will be the workers. So,
> we
> > > > > > probably
> > > > > > > need to allow multiple renewers. For Kafka rest proxy, the
> > renewer
> > > > can
> > > > > > > probably just be the creator of the token.
> > > > > > >
> > > > > > > 101. Do we need new acl on who can request delegation tokens?
> > > > > > >
> > > > > > > 102. Do we recommend people to send delegation tokens in an
> > > encrypted
> > > > > > > channel?
> > > > > > >
> > > > > > > 103. Who is responsible for expiring tokens, every broker?
> > > > > > >
> > > > > > > 104. For invalidating tokens, would it be better to do it in a
> > > > request
> > > > > > > instead of going to ZK directly?
> > > > > > >
> > > > > > > 105. The terminology of client in the wiki sometimes refers to
> > the
> > > > end
> > > > > > > client and some other times refers to the client using the
> > > delegation
> > > > > > > tokens. It would be useful to distinguish between the two.
> > > > > > >
> > > > > > > 106. Could you explain the sentence "Broker also stores the
> > > > > > DelegationToken
> > > > > > > without the hmac in the zookeeper." a bit more? I thought the
> > > > > delegation
> > > > > > > token is the hmac.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <ju...@confluent.io>
> > wrote:
> > > > > > >
> > > > > > > > Hi, Harsha,
> > > > > > > >
> > > > > > > > Just sent out a KIP meeting invite. We can discuss this in
> the
> > > > > meeting
> > > > > > > > tomorrow.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <ka...@harsha.io>
> > wrote:
> > > > > > > >
> > > > > > > >> Hi All,
> > > > > > > >>            Can we have a KIP meeting around this. The KIP is
> > up
> > > > for
> > > > > > > >>            sometime and if there are any questions lets
> > quickly
> > > > hash
> > > > > > out
> > > > > > > >>            details.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> Harsha
> > > > > > > >>
> > > > > > > >> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt wrote:
> > > > > > > >> > That is what the hadoop echo system uses so no good reason
> > > > really.
> > > > > > We
> > > > > > > >> > could
> > > > > > > >> > change it to whatever is the newest recommended standard
> is.
> > > > > > > >> >
> > > > > > > >> > Thanks
> > > > > > > >> > Parth
> > > > > > > >> >
> > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <
> > > ismael@juma.me.uk
> > > > >
> > > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > > Hi Parth,
> > > > > > > >> > >
> > > > > > > >> > > Thanks for the KIP. I only started reviewing this and
> may
> > > have
> > > > > > > >> additional
> > > > > > > >> > > questions later. The immediate question that came to
> mind
> > is
> > > > our
> > > > > > > >> choice of
> > > > > > > >> > > "DIGEST-MD5" even though it's marked as OBSOLETE in the
> > IANA
> > > > > > > Registry
> > > > > > > >> of
> > > > > > > >> > > SASL mechanisms and the original RFC (2831) has been
> moved
> > > to
> > > > > > > Historic
> > > > > > > >> > > status:
> > > > > > > >> > >
> > > > > > > >> > > https://tools.ietf.org/html/rfc6331
> > > > > > > >> > >
> > > > > > >
> > > >
> http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> > > > > > > >> > >
> > > > > > > >> > > What is the reasoning behind that choice?
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > > Ismael
> > > > > > > >> > >
> > > > > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <
> > > > > gwen@confluent.io
> > > > > > >
> > > > > > > >> wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Also comments inline :)
> > > > > > > >> > > >
> > > > > > > >> > > > > * I want to emphasize that even though delegation
> > tokens
> > > > > are a
> > > > > > > >> Hadoop
> > > > > > > >> > > > > innovation, I feel very strongly about not adding
> > > > dependency
> > > > > > on
> > > > > > > >> Hadoop
> > > > > > > >> > > > > when implementing delegation tokens for Kafka. The
> KIP
> > > > > doesn't
> > > > > > > >> imply
> > > > > > > >> > > > > such dependency, but if you can clarify...
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > *No hadoop dependency.*
> > > > > > > >> > > >
> > > > > > > >> > > > Yay! Just add this to the KIP so no one will read the
> > KIP
> > > > and
> > > > > > > panic
> > > > > > > >> > > > three weeks before the next release...
> > > > > > > >> > > >
> > > > > > > >> > > > > * Can we get delegation token at any time after
> > > > > > authenticating?
> > > > > > > >> only
> > > > > > > >> > > > > immediately after?
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > *As long as you are authenticated you can get
> > delegation
> > > > > > tokens.
> > > > > > > >> We
> > > > > > > >> > > need
> > > > > > > >> > > > to
> > > > > > > >> > > > > discuss if a client authenticated using delegation
> > > token,
> > > > > can
> > > > > > > also
> > > > > > > >> > > > acquire
> > > > > > > >> > > > > delegation token again or not. Also there is the
> > > question
> > > > of
> > > > > > do
> > > > > > > we
> > > > > > > >> > > allow
> > > > > > > >> > > > > anyone to acquire delegation token or we want
> specific
> > > > ACLs
> > > > > (I
> > > > > > > >> think
> > > > > > > >> > > its
> > > > > > > >> > > > an
> > > > > > > >> > > > > overkill.)*
> > > > > > > >> > > >
> > > > > > > >> > > > I agree that ACLs is an overkill.
> > > > > > > >> > > >
> > > > > > > >> > > > I think we are debating two options: Either require
> > > Kerberos
> > > > > > auth
> > > > > > > >> for
> > > > > > > >> > > > renewal or require non-owners to renew.
> > > > > > > >> > > > I *think* the latter is simpler (it basically require
> a
> > > "job
> > > > > > > master"
> > > > > > > >> > > > to take responsibility for the renewal, it will have
> its
> > > own
> > > > > > > >> identity
> > > > > > > >> > > > anyway and I think this is the correct design pattern
> > > > anyway.
> > > > > > For
> > > > > > > >> > > > storm, I'd expect Nimbus to coordinate renewals?), but
> > it
> > > is
> > > > > > hard
> > > > > > > to
> > > > > > > >> > > > debate simplicity without looking at the code changes
> > > > > required.
> > > > > > If
> > > > > > > >> you
> > > > > > > >> > > > have a draft of how the "require Kerberos" will look
> in
> > > > Kafka
> > > > > > > code,
> > > > > > > >> > > > I'll be happy to take a look.
> > > > > > > >> > > >
> > > > > > > >> > > > > * My understanding is that tokens will propagate via
> > ZK
> > > > but
> > > > > > > >> without
> > > > > > > >> > > > > additional changes to UpdateMetadata protocol,
> > correct?
> > > > > > Clients
> > > > > > > >> > > > > currently don't retry on SASL auth failure (IIRC),
> but
> > > > since
> > > > > > the
> > > > > > > >> > > > > tokens propagate between brokers asynch, we will
> need
> > to
> > > > > > retry a
> > > > > > > >> bit
> > > > > > > >> > > > > to avoid clients failing auth due to timing issues.
> > > > > > > >> > > > >
> > > > > > > >> > > > > *I am considering 2 alternatives right now. The
> > current
> > > > > > > documented
> > > > > > > >> > > > approach
> > > > > > > >> > > > > is zookeeper based and it does not require any
> changes
> > > to
> > > > > > > >> > > UpdateMetadata
> > > > > > > >> > > > > protocol. An alternative approach can remove
> zookeeper
> > > > > > > dependency
> > > > > > > >> as
> > > > > > > >> > > well
> > > > > > > >> > > > > but we can discuss that in KIP discussion call.*
> > > > > > > >> > > >
> > > > > > > >> > > > Oooh! Sounds interesting. Do you want to ping Jun to
> > > > arrange a
> > > > > > > call?
> > > > > > > >> > > >
> > > > > > > >> > > > > * I liked Ashish's suggestion of having just the
> > > > controller
> > > > > > > issue
> > > > > > > >> the
> > > > > > > >> > > > > delegation tokens, to avoid syncing a shared secret.
> > Not
> > > > > sure
> > > > > > if
> > > > > > > >> we
> > > > > > > >> > > > > want to continue the discussion here or on the
> wiki. I
> > > > think
> > > > > > > that
> > > > > > > >> we
> > > > > > > >> > > > > can decouple the problem of "token distribution"
> from
> > > > > "shared
> > > > > > > >> secret
> > > > > > > >> > > > > distribution" and use the controller as the only
> token
> > > > > > generator
> > > > > > > >> to
> > > > > > > >> > > > > solve the second issue, while still using ZK async
> to
> > > > > > distribute
> > > > > > > >> > > > > tokens.
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > *As mentioned in the previous Email I am fine with
> > that
> > > > > > approach
> > > > > > > >> as
> > > > > > > >> > > long
> > > > > > > >> > > > as
> > > > > > > >> > > > > we agree that the extra complexity of
> adding/updating
> > > APIS
> > > > > > adds
> > > > > > > >> enough
> > > > > > > >> > > > > value. The advantage with the controller approach is
> > > > secret
> > > > > > > >> rotation
> > > > > > > >> > > can
> > > > > > > >> > > > be
> > > > > > > >> > > > > automated,frequent and would not require
> deployment. *
> > > > > > > >> > > >
> > > > > > > >> > > > Can you detail the extra complexity (or point me to
> the
> > > > email
> > > > > I
> > > > > > > >> > > > missed?) - which Apis are required?
> > > > > > > >> > > > As far as I can tell, clients can already find the
> > > > controller
> > > > > > from
> > > > > > > >> > > > metadata. I'm a bit more concerned about controller
> > load.
> > > > > > > >> > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > * While I like the idea of forcing kerberos auth for
> > > > > renwal, I
> > > > > > > >> think
> > > > > > > >> > > > > it mixes the transport layer the the request content
> > in
> > > a
> > > > > > pretty
> > > > > > > >> ugly
> > > > > > > >> > > > > way. Perhaps limiting renewer to non-owner is
> better.
> > > > > > > >> > > > >
> > > > > > > >> > > > > *I feel this is a necessary evil. While this will
> make
> > > the
> > > > > > kafka
> > > > > > > >> code
> > > > > > > >> > > > > pretty straight forward , forcing  renewer to
> > non-owner
> > > > > pushes
> > > > > > > >> the code
> > > > > > > >> > > > > ugliness to client and makes it even harder to
> > > > integrate.  *
> > > > > > > >> > > >
> > > > > > > >> > > > As mentioned before, I don't think the "renewal by
> > other"
> > > > > > approach
> > > > > > > >> is
> > > > > > > >> > > > that ugly for the clients we expect to use delegation
> > > tokens
> > > > > > since
> > > > > > > >> > > > they will have an app-master of some sort who
> requested
> > > the
> > > > > > token
> > > > > > > to
> > > > > > > >> > > > begin with.
> > > > > > > >> > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > The response for my question on how multiple
> > identities
> > > > will
> > > > > > be
> > > > > > > >> > > > > handled wasn't super clear to me - AFAIK, we don't
> > > > > > authenticate
> > > > > > > >> each
> > > > > > > >> > > > > request, we authenticate connections.
> > > > > > > >> > > > >
> > > > > > > >> > > > > *We authenticate connections, and only when they are
> > > being
> > > > > > > >> established.
> > > > > > > >> > > > Let
> > > > > > > >> > > > > me try to phrase this as a question, in absence of
> > > > > delegation
> > > > > > > >> tokens if
> > > > > > > >> > > > we
> > > > > > > >> > > > > had to support the use case using user TGT's how
> would
> > > we
> > > > do
> > > > > > it?
> > > > > > > >> My
> > > > > > > >> > > point
> > > > > > > >> > > > > was it would be no different with delegation tokens.
> > The
> > > > use
> > > > > > > case
> > > > > > > >> you
> > > > > > > >> > > are
> > > > > > > >> > > > > describing seems more like impersonation.*
> > > > > > > >> > > >
> > > > > > > >> > > > Yeah, I thought that one of the things that delegation
> > > > tokens
> > > > > > > >> handled.
> > > > > > > >> > > > Maybe I got it wrong :)
> > > > > > > >> > > >
> > > > > > > >> > > > Thanks for the detailed answers.
> > > > > > > >> > > >
> > > > > > > >> > > > Gwen
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > > > Thanks
> > > > > > > >> > > > > Parth
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <
> > > > > > > gwen@confluent.io
> > > > > > > >> >
> > > > > > > >> > > > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > >> Hi Parth and Harsha,
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> Few more comments:
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> * The API / RequestResponse section doesn't seem to
> > > have
> > > > > good
> > > > > > > >> > > > >> description of the changes to the Kafka Protocol.
> > > Sounds
> > > > > like
> > > > > > > >> you are
> > > > > > > >> > > > >> proposing new DelegationTokenRequest and
> > > > RenewTokenRequest
> > > > > > (and
> > > > > > > >> > > > >> matching responses), without detailing the contents
> > of
> > > > the
> > > > > > > >> requests
> > > > > > > >> > > > >> and responses? Or rather, you show the class
> > interface,
> > > > but
> > > > > > not
> > > > > > > >> the
> > > > > > > >> > > > >> underlying protocol. This could be seen as an
> > > > > implementation
> > > > > > > >> detail,
> > > > > > > >> > > > >> but since the binary protocol is what we provide to
> > > > > non-Java
> > > > > > > >> clients,
> > > > > > > >> > > > >> we need to show the changes there.
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> * getDelegationToken sounds like
> > > > > > delegationTokenRequestHandler?
> > > > > > > >> Is it
> > > > > > > >> > > > >> planned to be part of KafkaApi? or Client? Its
> > > unclear...
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> * I want to emphasize that even though delegation
> > > tokens
> > > > > are
> > > > > > a
> > > > > > > >> Hadoop
> > > > > > > >> > > > >> innovation, I feel very strongly about not adding
> > > > > dependency
> > > > > > on
> > > > > > > >> Hadoop
> > > > > > > >> > > > >> when implementing delegation tokens for Kafka. The
> > KIP
> > > > > > doesn't
> > > > > > > >> imply
> > > > > > > >> > > > >> such dependency, but if you can clarify...
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> * Can we get delegation token at any time after
> > > > > > authenticating?
> > > > > > > >> only
> > > > > > > >> > > > >> immediately after?
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> * My understanding is that tokens will propagate
> via
> > ZK
> > > > but
> > > > > > > >> without
> > > > > > > >> > > > >> additional changes to UpdateMetadata protocol,
> > correct?
> > > > > > Clients
> > > > > > > >> > > > >> currently don't retry on SASL auth failure (IIRC),
> > but
> > > > > since
> > > > > > > the
> > > > > > > >> > > > >> tokens propagate between brokers asynch, we will
> need
> > > to
> > > > > > retry
> > > > > > > a
> > > > > > > >> bit
> > > > > > > >> > > > >> to avoid clients failing auth due to timing issues.
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> * Strongly agreeing on clients not touching ZK
> > directly
> > > > :)
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> * I liked Ashish's suggestion of having just the
> > > > controller
> > > > > > > >> issue the
> > > > > > > >> > > > >> delegation tokens, to avoid syncing a shared
> secret.
> > > Not
> > > > > sure
> > > > > > > if
> > > > > > > >> we
> > > > > > > >> > > > >> want to continue the discussion here or on the
> wiki.
> > I
> > > > > think
> > > > > > > >> that we
> > > > > > > >> > > > >> can decouple the problem of "token distribution"
> from
> > > > > "shared
> > > > > > > >> secret
> > > > > > > >> > > > >> distribution" and use the controller as the only
> > token
> > > > > > > generator
> > > > > > > >> to
> > > > > > > >> > > > >> solve the second issue, while still using ZK async
> to
> > > > > > > distribute
> > > > > > > >> > > > >> tokens.
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> * I am also uncomfortable with infinite lifetime of
> > > > tokens
> > > > > > (and
> > > > > > > >> hoped
> > > > > > > >> > > > >> to hear from others in the community) - but having
> > the
> > > > > option
> > > > > > > and
> > > > > > > >> > > > >> documenting it as less secure, allows users to
> > > configure
> > > > > > their
> > > > > > > >> system
> > > > > > > >> > > > >> as the wish.
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> * While I like the idea of forcing kerberos auth
> for
> > > > > renwal,
> > > > > > I
> > > > > > > >> think
> > > > > > > >> > > > >> it mixes the transport layer the the request
> content
> > > in a
> > > > > > > pretty
> > > > > > > >> ugly
> > > > > > > >> > > > >> way. Perhaps limiting renewer to non-owner is
> better.
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> Things I'd still like to see:
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> * More detailed explanation on what we plan to do
> for
> > > the
> > > > > > java
> > > > > > > >> clients
> > > > > > > >> > > > >> specifically - new configuration? new APIs?
> > > > > > > >> > > > >> The response for my question on how multiple
> > identities
> > > > > will
> > > > > > be
> > > > > > > >> > > > >> handled wasn't super clear to me - AFAIK, we don't
> > > > > > authenticate
> > > > > > > >> each
> > > > > > > >> > > > >> request, we authenticate connections.
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> * Alternatives: Delegation tokens are only used in
> > the
> > > > > Hadoop
> > > > > > > >> > > > >> ecosystem. I'm wondering if there are alternatives
> in
> > > > other
> > > > > > > >> ecosystems
> > > > > > > >> > > > >> (Mesos? Tachyon? Cassandra?) and whether there are
> > some
> > > > > > > >> advantages
> > > > > > > >> > > > >> there.
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> Gwen
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <
> > > kafka@harsha.io
> > > > >
> > > > > > > wrote:
> > > > > > > >> > > > >> > Hi Gwen,
> > > > > > > >> > > > >> >            Can you look at Parth's last reply.
> Does
> > > it
> > > > > > answer
> > > > > > > >> your
> > > > > > > >> > > > >> >            concerns.
> > > > > > > >> > > > >> >
> > > > > > > >> > > > >> > Thanks,
> > > > > > > >> > > > >> > Harsha
> > > > > > > >> > > > >> >
> > > > > > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth
> brahmbhatt
> > > > wrote:
> > > > > > > >> > > > >> >> Thanks for reviewing Gwen. The wiki already has
> > > > details
> > > > > on
> > > > > > > >> token
> > > > > > > >> > > > >> >> expiration
> > > > > > > >> > > > >> >> under token acquisition process
> > > > > > > >> > > > >> >> <
> > > > > > > >> > > > >>
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> > > > > > > >> > > > >> >.
> > > > > > > >> > > > >> >> Current proposal is that tokens will expire
> based
> > > on a
> > > > > > > server
> > > > > > > >> side
> > > > > > > >> > > > >> >> configuration (default 24 hours) unless renewed.
> > > > Renewal
> > > > > > is
> > > > > > > >> only
> > > > > > > >> > > > allowed
> > > > > > > >> > > > >> >> until the max life time of token. Alternatively
> we
> > > > could
> > > > > > > also
> > > > > > > >> make
> > > > > > > >> > > > that
> > > > > > > >> > > > >> >> an
> > > > > > > >> > > > >> >> optional param and the server side default can
> > serve
> > > > as
> > > > > > the
> > > > > > > >> upper
> > > > > > > >> > > > bound.
> > > > > > > >> > > > >> >>
> > > > > > > >> > > > >> >> To your second point it will be done exactly the
> > > same
> > > > > way
> > > > > > we
> > > > > > > >> would
> > > > > > > >> > > > >> >> support
> > > > > > > >> > > > >> >> multiple keytabs. The calling client will have
> to
> > > put
> > > > > the
> > > > > > > >> tokens it
> > > > > > > >> > > > >> wants
> > > > > > > >> > > > >> >> to use in the subject instance and call
> > > > produce/consume
> > > > > > > inside
> > > > > > > >> > > > >> >> subject.doas. Each caller will have to keep
> track
> > of
> > > > its
> > > > > > own
> > > > > > > >> > > > subject. I
> > > > > > > >> > > > >> >> will have to look at the code to see if we
> support
> > > > this
> > > > > > > >> feature
> > > > > > > >> > > right
> > > > > > > >> > > > >> now
> > > > > > > >> > > > >> >> but my understanding is delegation token
> shouldn't
> > > > need
> > > > > > any
> > > > > > > >> special
> > > > > > > >> > > > >> >> treatment as its just another type of Credential
> > in
> > > > the
> > > > > > > >> subject.
> > > > > > > >> > > > >> >>
> > > > > > > >> > > > >> >> I would also like to know what is your opinion
> > about
> > > > > > > infinite
> > > > > > > >> > > renewal
> > > > > > > >> > > > >> (my
> > > > > > > >> > > > >> >> recommendation is to not support this), tokens
> > > > renewing
> > > > > > them
> > > > > > > >> > > self(my
> > > > > > > >> > > > >> >> recommendation is to not support this) and most
> > > > > > importantly
> > > > > > > >> your
> > > > > > > >> > > > choice
> > > > > > > >> > > > >> >> between the alternatives listed on this thread
> > > > > > > >> > > > >> >> <
> > > > > > > >> > > > >>
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> > > > > > > >> > > > >> >
> > > > > > > >> > > > >> >> ( I am leaning towards the alternative-2 minus
> > > > > controller
> > > > > > > >> > > > distributing
> > > > > > > >> > > > >> >> secret). Thanks again for reviewing.
> > > > > > > >> > > > >> >>
> > > > > > > >> > > > >> >> Thanks
> > > > > > > >> > > > >> >> Parth
> > > > > > > >> > > > >> >>
> > > > > > > >> > > > >> >>
> > > > > > > >> > > > >> >>
> > > > > > > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <
> > > > > > > >> gwen@confluent.io>
> > > > > > > >> > > > wrote:
> > > > > > > >> > > > >> >>
> > > > > > > >> > > > >> >> > Harsha,
> > > > > > > >> > > > >> >> >
> > > > > > > >> > > > >> >> > I was thinking of the Rest Proxy. I didn't see
> > > your
> > > > > > design
> > > > > > > >> yet,
> > > > > > > >> > > > but in
> > > > > > > >> > > > >> >> > our proxy, we have a set of producers, which
> > will
> > > > > serve
> > > > > > > >> multiple
> > > > > > > >> > > > users
> > > > > > > >> > > > >> >> > going through the proxy. Since these users
> will
> > > have
> > > > > > > >> different
> > > > > > > >> > > > >> >> > privileges, they'll need to authenticate
> > > separately,
> > > > > and
> > > > > > > >> can't
> > > > > > > >> > > > share a
> > > > > > > >> > > > >> >> > token.
> > > > > > > >> > > > >> >> >
> > > > > > > >> > > > >> >> > Am I missing anything?
> > > > > > > >> > > > >> >> >
> > > > > > > >> > > > >> >> > Gwen
> > > > > > > >> > > > >> >> >
> > > > > > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <
> > > > > kafka@harsha.io
> > > > > > >
> > > > > > > >> wrote:
> > > > > > > >> > > > >> >> > > Gwen,
> > > > > > > >> > > > >> >> > >            On your second point. Can you
> > > describe
> > > > a
> > > > > > > >> usecase
> > > > > > > >> > > where
> > > > > > > >> > > > >> >> > >            mutliple clients ended up
> sharing a
> > > > > > producer
> > > > > > > >> and
> > > > > > > >> > > even
> > > > > > > >> > > > if
> > > > > > > >> > > > >> they
> > > > > > > >> > > > >> >> > >            do why can't they not use single
> > > token
> > > > > that
> > > > > > > >> producer
> > > > > > > >> > > > >> >> > >            captures. Why would we need
> > multiple
> > > > > > clients
> > > > > > > >> with
> > > > > > > >> > > > >> different
> > > > > > > >> > > > >> >> > >            tokens sharing a single instance
> of
> > > > > > producer.
> > > > > > > >> Also
> > > > > > > >> > > in
> > > > > > > >> > > > >> this
> > > > > > > >> > > > >> >> > >            case other clients have access
> all
> > > the
> > > > > > tokens
> > > > > > > >> no?
> > > > > > > >> > > > >> >> > >
> > > > > > > >> > > > >> >> > > Thanks,
> > > > > > > >> > > > >> >> > > Harsha
> > > > > > > >> > > > >> >> > >
> > > > > > > >> > > > >> >> > >
> > > > > > > >> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen
> Shapira
> > > > > wrote:
> > > > > > > >> > > > >> >> > >> Sorry for the delay:
> > > > > > > >> > > > >> >> > >>
> > > > > > > >> > > > >> >> > >> Two questions that we didn't see in the
> wiki:
> > > > > > > >> > > > >> >> > >> 1. Is there an expiration for delegation
> > > tokens?
> > > > > > > >> Renewal? How
> > > > > > > >> > > > do we
> > > > > > > >> > > > >> >> > >> revoke them?
> > > > > > > >> > > > >> >> > >> 2. If we want to use delegation tokens for
> > > > "do-as"
> > > > > > > (say,
> > > > > > > >> > > submit
> > > > > > > >> > > > >> Storm
> > > > > > > >> > > > >> >> > >> job as my user), we will need a producer
> for
> > > > every
> > > > > > job
> > > > > > > >> (we
> > > > > > > >> > > can't
> > > > > > > >> > > > >> share
> > > > > > > >> > > > >> >> > >> them between multiple jobs running on same
> > > node),
> > > > > > since
> > > > > > > >> we
> > > > > > > >> > > only
> > > > > > > >> > > > >> >> > >> authenticate when connecting. Is there a
> plan
> > > to
> > > > > > change
> > > > > > > >> this
> > > > > > > >> > > for
> > > > > > > >> > > > >> >> > >> delegation tokens, in order to allow
> multiple
> > > > users
> > > > > > > with
> > > > > > > >> > > > different
> > > > > > > >> > > > >> >> > >> tokens to share a client?
> > > > > > > >> > > > >> >> > >>
> > > > > > > >> > > > >> >> > >> Gwen
> > > > > > > >> > > > >> >> > >>
> > > > > > > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth
> > > brahmbhatt
> > > > > > > >> > > > >> >> > >> <br...@gmail.com> wrote:
> > > > > > > >> > > > >> >> > >> > Bumping this up one more time, can other
> > > > > committers
> > > > > > > >> review?
> > > > > > > >> > > > >> >> > >> >
> > > > > > > >> > > > >> >> > >> > Thanks
> > > > > > > >> > > > >> >> > >> > Parth
> > > > > > > >> > > > >> >> > >> >
> > > > > > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <
> > > > > > > >> kafka@harsha.io>
> > > > > > > >> > > > wrote:
> > > > > > > >> > > > >> >> > >> >
> > > > > > > >> > > > >> >> > >> >> Parth,
> > > > > > > >> > > > >> >> > >> >>           Overall current design looks
> > good
> > > to
> > > > > > me. I
> > > > > > > >> am +1
> > > > > > > >> > > on
> > > > > > > >> > > > >> the
> > > > > > > >> > > > >> >> > KIP.
> > > > > > > >> > > > >> >> > >> >>
> > > > > > > >> > > > >> >> > >> >> Gwen , Jun can you review this as well.
> > > > > > > >> > > > >> >> > >> >>
> > > > > > > >> > > > >> >> > >> >> -Harsha
> > > > > > > >> > > > >> >> > >> >>
> > > > > > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth
> > > > > > brahmbhatt
> > > > > > > >> wrote:
> > > > > > > >> > > > >> >> > >> >> > Thanks for review Jitendra.
> > > > > > > >> > > > >> >> > >> >> >
> > > > > > > >> > > > >> >> > >> >> > I don't like the idea of infinite
> > lifetime
> > > > > but I
> > > > > > > >> see the
> > > > > > > >> > > > >> Streaming
> > > > > > > >> > > > >> >> > use
> > > > > > > >> > > > >> >> > >> >> > case. Even for Streaming use case I
> was
> > > > hoping
> > > > > > > >> there will
> > > > > > > >> > > > be
> > > > > > > >> > > > >> some
> > > > > > > >> > > > >> >> > notion
> > > > > > > >> > > > >> >> > >> >> > of
> > > > > > > >> > > > >> >> > >> >> > master/driver that can get new
> > delegation
> > > > > tokens
> > > > > > > at
> > > > > > > >> fixed
> > > > > > > >> > > > >> interval
> > > > > > > >> > > > >> >> > and
> > > > > > > >> > > > >> >> > >> >> > distribute to workers. If that is not
> > the
> > > > case
> > > > > > for
> > > > > > > >> we can
> > > > > > > >> > > > >> discuss
> > > > > > > >> > > > >> >> > >> >> > delegation tokens renewing them self
> and
> > > the
> > > > > > > >> security
> > > > > > > >> > > > >> implications
> > > > > > > >> > > > >> >> > of the
> > > > > > > >> > > > >> >> > >> >> > same.
> > > > > > > >> > > > >> >> > >> >> >
> > > > > > > >> > > > >> >> > >> >> > I did not want clients to fetch tokens
> > > from
> > > > > > > >> zookeeper,
> > > > > > > >> > > > >> overall I
> > > > > > > >> > > > >> >> > think
> > > > > > > >> > > > >> >> > >> >> > its
> > > > > > > >> > > > >> >> > >> >> > better if clients don't rely on our
> > > metadata
> > > > > > store
> > > > > > > >> and I
> > > > > > > >> > > > >> think we
> > > > > > > >> > > > >> >> > are
> > > > > > > >> > > > >> >> > >> >> > moving in that direction with all the
> > > KIP-4
> > > > > > > >> improvements.
> > > > > > > >> > > > I
> > > > > > > >> > > > >> chose
> > > > > > > >> > > > >> >> > >> >> > zookeeper as in this case the client
> > will
> > > > > still
> > > > > > > >> just talk
> > > > > > > >> > > > to
> > > > > > > >> > > > >> >> > broker , its
> > > > > > > >> > > > >> >> > >> >> > the brokers that will use zookeeper
> > which
> > > we
> > > > > > > >> already do
> > > > > > > >> > > > for a
> > > > > > > >> > > > >> lot
> > > > > > > >> > > > >> >> > of
> > > > > > > >> > > > >> >> > >> >> > other
> > > > > > > >> > > > >> >> > >> >> > usecases + ease of development + and
> the
> > > > > ability
> > > > > > > so
> > > > > > > >> > > tokens
> > > > > > > >> > > > >> will
> > > > > > > >> > > > >> >> > survive
> > > > > > > >> > > > >> >> > >> >> > even a rolling restart/cluster
> failure.
> > > if a
> > > > > > > >> majority
> > > > > > > >> > > > agrees
> > > > > > > >> > > > >> the
> > > > > > > >> > > > >> >> > added
> > > > > > > >> > > > >> >> > >> >> > complexity to have controller
> forwarding
> > > > keys
> > > > > to
> > > > > > > all
> > > > > > > >> > > > broker is
> > > > > > > >> > > > >> >> > justified
> > > > > > > >> > > > >> >> > >> >> > as
> > > > > > > >> > > > >> >> > >> >> > it provides tighter security , I am
> fine
> > > > with
> > > > > > that
> > > > > > > >> option
> > > > > > > >> > > > too.
> > > > > > > >> > > > >> >> > >> >> >
> > > > > > > >> > > > >> >> > >> >> > Given zookeeper does not support SSL
> we
> > > can
> > > > > not
> > > > > > > >> store
> > > > > > > >> > > > master
> > > > > > > >> > > > >> keys
> > > > > > > >> > > > >> >> > in
> > > > > > > >> > > > >> >> > >> >> > zookeeper as master keys will be
> exposed
> > > on
> > > > > > wire.
> > > > > > > To
> > > > > > > >> > > > support
> > > > > > > >> > > > >> >> > rotation
> > > > > > > >> > > > >> >> > >> >> > without affecting current clients is
> > > > > something I
> > > > > > > >> need to
> > > > > > > >> > > > put
> > > > > > > >> > > > >> more
> > > > > > > >> > > > >> >> > thought
> > > > > > > >> > > > >> >> > >> >> > in. My current proposal assumes the
> > > rotation
> > > > > > will
> > > > > > > >> > > > invalidate
> > > > > > > >> > > > >> all
> > > > > > > >> > > > >> >> > current
> > > > > > > >> > > > >> >> > >> >> > tokens.
> > > > > > > >> > > > >> >> > >> >> >
> > > > > > > >> > > > >> >> > >> >> > I request committers to also review
> and
> > > post
> > > > > > their
> > > > > > > >> > > comments
> > > > > > > >> > > > >> so we
> > > > > > > >> > > > >> >> > can
> > > > > > > >> > > > >> >> > >> >> > make
> > > > > > > >> > > > >> >> > >> >> > progress on this KIP.
> > > > > > > >> > > > >> >> > >> >> >
> > > > > > > >> > > > >> >> > >> >> > Thanks
> > > > > > > >> > > > >> >> > >> >> > Parth
> > > > > > > >> > > > >> >> > >> >> >
> > > > > > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM,
> Ashish
> > > > Singh
> > > > > <
> > > > > > > >> > > > >> asingh@cloudera.com
> > > > > > > >> > > > >> >> > >
> > > > > > > >> > > > >> >> > >> >> > wrote:
> > > > > > > >> > > > >> >> > >> >> >
> > > > > > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM,
> > > Harsha <
> > > > > > > >> > > > kafka@harsha.io>
> > > > > > > >> > > > >> >> > wrote:
> > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > >> > > > >> >> > >> >> > > > Unifying the two discussion
> threads
> > on
> > > > > this
> > > > > > > KIP.
> > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > >> > > > >> >> > >> >> > > > Here is the response from Jitendra
> > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > >> > > > >> >> > >> >> > > > "The need for a large number of
> > > clients
> > > > > that
> > > > > > > are
> > > > > > > >> > > > running
> > > > > > > >> > > > >> all
> > > > > > > >> > > > >> >> > over the
> > > > > > > >> > > > >> >> > >> >> > > > cluster that authenticate with
> Kafka
> > > > > > brokers,
> > > > > > > >> is very
> > > > > > > >> > > > >> similar
> > > > > > > >> > > > >> >> > to the
> > > > > > > >> > > > >> >> > >> >> > > > Hadoop use case of large number of
> > > tasks
> > > > > > > running
> > > > > > > >> > > across
> > > > > > > >> > > > >> the
> > > > > > > >> > > > >> >> > cluster
> > > > > > > >> > > > >> >> > >> >> that
> > > > > > > >> > > > >> >> > >> >> > > > need authentication to Hdfs
> > Namenode.
> > > > > > > >> Therefore, the
> > > > > > > >> > > > >> >> > delegation token
> > > > > > > >> > > > >> >> > >> >> > > > approach does seem like a good fit
> > for
> > > > > this
> > > > > > > use
> > > > > > > >> case
> > > > > > > >> > > > as we
> > > > > > > >> > > > >> >> > have seen
> > > > > > > >> > > > >> >> > >> >> it
> > > > > > > >> > > > >> >> > >> >> > > > working at large scale in HDFS and
> > > YARN.
> > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > >> > > > >> >> > >> >> > > >   The proposed design is very much
> > > > inline
> > > > > > with
> > > > > > > >> Hadoop
> > > > > > > >> > > > >> >> > approach. A few
> > > > > > > >> > > > >> >> > >> >> > > >   comments:
> > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > >> > > > >> >> > >> >> > > > 1) Why do you guys want to allow
> > > > infinite
> > > > > > > >> renewable
> > > > > > > >> > > > >> lifetime
> > > > > > > >> > > > >> >> > for a
> > > > > > > >> > > > >> >> > >> >> > > > token? HDFS restricts a token to a
> > max
> > > > > life
> > > > > > > time
> > > > > > > >> > > > (default
> > > > > > > >> > > > >> 7
> > > > > > > >> > > > >> >> > days).  A
> > > > > > > >> > > > >> >> > >> >> > > > token's vulnerability is believed
> to
> > > > > > increase
> > > > > > > >> with
> > > > > > > >> > > > time.
> > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > >> > > > >> >> > >> >> > > I agree that having infinite
> lifetime
> > > > might
> > > > > > not
> > > > > > > >> be the
> > > > > > > >> > > > best
> > > > > > > >> > > > >> idea.
> > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > >> > > > >> >> > >> >> > > > 2) As I understand the tokens are
> > > stored
> > > > > in
> > > > > > > >> zookeeper
> > > > > > > >> > > > as
> > > > > > > >> > > > >> well,
> > > > > > > >> > > > >> >> > and
> > > > > > > >> > > > >> >> > >> >> can
> > > > > > > >> > > > >> >> > >> >> > > > be updated there. This is clever
> as
> > it
> > > > can
> > > > > > > allow
> > > > > > > >> > > > >> replacing the
> > > > > > > >> > > > >> >> > tokens
> > > > > > > >> > > > >> >> > >> >> > > > once they run out of max life
> time,
> > > and
> > > > > > > clients
> > > > > > > >> can
> > > > > > > >> > > > >> download
> > > > > > > >> > > > >> >> > new
> > > > > > > >> > > > >> >> > >> >> tokens
> > > > > > > >> > > > >> >> > >> >> > > > from zookeeper. It shouldn't be a
> > big
> > > > load
> > > > > > on
> > > > > > > >> > > zookeeper
> > > > > > > >> > > > >> as a
> > > > > > > >> > > > >> >> > client
> > > > > > > >> > > > >> >> > >> >> will
> > > > > > > >> > > > >> >> > >> >> > > > need to get a new token once in
> > > several
> > > > > > days.
> > > > > > > >> In this
> > > > > > > >> > > > >> approach
> > > > > > > >> > > > >> >> > you
> > > > > > > >> > > > >> >> > >> >> don't
> > > > > > > >> > > > >> >> > >> >> > > > need infinite lifetime on the
> token
> > > even
> > > > > for
> > > > > > > >> long
> > > > > > > >> > > > running
> > > > > > > >> > > > >> >> > clients.
> > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > >> > > > >> >> > >> >> > > > 3) The token password are
> generated
> > > > using
> > > > > a
> > > > > > > >> master
> > > > > > > >> > > key.
> > > > > > > >> > > > >> The
> > > > > > > >> > > > >> >> > master
> > > > > > > >> > > > >> >> > >> >> key
> > > > > > > >> > > > >> >> > >> >> > > > should also be periodically
> changed.
> > > In
> > > > > > > Hadoop,
> > > > > > > >> the
> > > > > > > >> > > > >> default
> > > > > > > >> > > > >> >> > renewal
> > > > > > > >> > > > >> >> > >> >> > > > period is 1 day.?
> > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > >> > > > >> >> > >> >> > > IIUC, this will require brokers
> > > > maintaining
> > > > > a
> > > > > > > >> list of X
> > > > > > > >> > > > most
> > > > > > > >> > > > >> >> > recent
> > > > > > > >> > > > >> >> > >> >> master
> > > > > > > >> > > > >> >> > >> >> > > keys. This list will have to be
> > > persisted
> > > > > > > >> somewhere, as
> > > > > > > >> > > > if a
> > > > > > > >> > > > >> >> > broker
> > > > > > > >> > > > >> >> > >> >> goes
> > > > > > > >> > > > >> >> > >> >> > > down it will have to get that list
> > again
> > > > and
> > > > > > > >> storing
> > > > > > > >> > > > master
> > > > > > > >> > > > >> keys
> > > > > > > >> > > > >> >> > on ZK
> > > > > > > >> > > > >> >> > >> >> is
> > > > > > > >> > > > >> >> > >> >> > > not the best idea. However, if a
> > broker
> > > > goes
> > > > > > > down
> > > > > > > >> then
> > > > > > > >> > > we
> > > > > > > >> > > > >> have
> > > > > > > >> > > > >> >> > much
> > > > > > > >> > > > >> >> > >> >> bigger
> > > > > > > >> > > > >> >> > >> >> > > issue to deal with and client can
> > always
> > > > > > > >> > > re-authenticate
> > > > > > > >> > > > is
> > > > > > > >> > > > >> such
> > > > > > > >> > > > >> >> > >> >> events.
> > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > >> > > > >> >> > >> >> > > Did you happen to take a look at
> other
> > > > > > > >> alternatives
> > > > > > > >> > > this
> > > > > > > >> > > > >> list has
> > > > > > > >> > > > >> >> > >> >> > > suggested?
> > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > >> > > > >> >> > >> >> > > > Thanks for a thorough proposal,
> > great
> > > > > work!"
> > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM,
> > Gwen
> > > > > > Shapira
> > > > > > > >> wrote:
> > > > > > > >> > > > >> >> > >> >> > > > > Makes sense to me. Thanks!
> > > > > > > >> > > > >> >> > >> >> > > > >
> > > > > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM,
> > > > Harsha <
> > > > > > > >> > > > kafka@harsha.io
> > > > > > > >> > > > >> >
> > > > > > > >> > > > >> >> > wrote:
> > > > > > > >> > > > >> >> > >> >> > > > > > It doesn't need any release
> > > vehicle
> > > > > but
> > > > > > > >> still the
> > > > > > > >> > > > >> work can
> > > > > > > >> > > > >> >> > move
> > > > > > > >> > > > >> >> > >> >> > > > forward.
> > > > > > > >> > > > >> >> > >> >> > > > > > If anyone is interested in the
> > KIP
> > > > > > please
> > > > > > > >> do the
> > > > > > > >> > > > >> review and
> > > > > > > >> > > > >> >> > >> >> provide
> > > > > > > >> > > > >> >> > >> >> > > the
> > > > > > > >> > > > >> >> > >> >> > > > > > comments.
> > > > > > > >> > > > >> >> > >> >> > > > > >
> > > > > > > >> > > > >> >> > >> >> > > > > > -Harsha
> > > > > > > >> > > > >> >> > >> >> > > > > >
> > > > > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59
> > PM,
> > > > > Ismael
> > > > > > > >> Juma
> > > > > > > >> > > > wrote:
> > > > > > > >> > > > >> >> > >> >> > > > > >> I agree that it would be good
> > to
> > > > have
> > > > > > > more
> > > > > > > >> time
> > > > > > > >> > > to
> > > > > > > >> > > > >> review
> > > > > > > >> > > > >> >> > and
> > > > > > > >> > > > >> >> > >> >> > > discuss
> > > > > > > >> > > > >> >> > >> >> > > > > >> KIP-48.
> > > > > > > >> > > > >> >> > >> >> > > > > >>
> > > > > > > >> > > > >> >> > >> >> > > > > >> Ismael
> > > > > > > >> > > > >> >> > >> >> > > > > >>
> > > > > > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55
> > AM,
> > > > Gwen
> > > > > > > >> Shapira <
> > > > > > > >> > > > >> >> > >> >> gwen@confluent.io>
> > > > > > > >> > > > >> >> > >> >> > > > wrote:
> > > > > > > >> > > > >> >> > >> >> > > > > >>
> > > > > > > >> > > > >> >> > >> >> > > > > >> > Hi Team,
> > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > Since KIP-48 depends on
> > KIP-43,
> > > > > which
> > > > > > > is
> > > > > > > >> > > > already a
> > > > > > > >> > > > >> bit
> > > > > > > >> > > > >> >> > of a
> > > > > > > >> > > > >> >> > >> >> risk
> > > > > > > >> > > > >> >> > >> >> > > for
> > > > > > > >> > > > >> >> > >> >> > > > > >> > the next release - any
> chance
> > > we
> > > > > can
> > > > > > > >> delay
> > > > > > > >> > > > >> delegation
> > > > > > > >> > > > >> >> > tokens
> > > > > > > >> > > > >> >> > >> >> to
> > > > > > > >> > > > >> >> > >> >> > > > Kafka
> > > > > > > >> > > > >> >> > >> >> > > > > >> > 0.10.1?
> > > > > > > >> > > > >> >> > >> >> > > > > >> > With the community working
> > on a
> > > > > > release
> > > > > > > >> every
> > > > > > > >> > > 3
> > > > > > > >> > > > >> month,
> > > > > > > >> > > > >> >> > this
> > > > > > > >> > > > >> >> > >> >> is not
> > > > > > > >> > > > >> >> > >> >> > > > a huge
> > > > > > > >> > > > >> >> > >> >> > > > > >> > delay.
> > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > Gwen
> > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at
> 5:11
> > > PM,
> > > > > > Ashish
> > > > > > > >> Singh
> > > > > > > >> > > <
> > > > > > > >> > > > >> >> > >> >> > > asingh@cloudera.com>
> > > > > > > >> > > > >> >> > >> >> > > > wrote:
> > > > > > > >> > > > >> >> > >> >> > > > > >> > > Parth,
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > > Thanks again for the
> > awesome
> > > > > write
> > > > > > > up.
> > > > > > > >> > > > Following
> > > > > > > >> > > > >> our
> > > > > > > >> > > > >> >> > >> >> discussion
> > > > > > > >> > > > >> >> > >> >> > > > from the
> > > > > > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I think it will be
> > > easier
> > > > > to
> > > > > > > >> compare
> > > > > > > >> > > > >> various
> > > > > > > >> > > > >> >> > >> >> alternatives
> > > > > > > >> > > > >> >> > >> >> > > > if they
> > > > > > > >> > > > >> >> > >> >> > > > > >> > are
> > > > > > > >> > > > >> >> > >> >> > > > > >> > > listed together. I am
> > stating
> > > > > > below a
> > > > > > > >> few
> > > > > > > >> > > > >> >> > alternatives along
> > > > > > > >> > > > >> >> > >> >> > > with
> > > > > > > >> > > > >> >> > >> >> > > > a the
> > > > > > > >> > > > >> >> > >> >> > > > > >> > > current proposal.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Current proposal) Store
> > > > > Delegation
> > > > > > > >> Token,
> > > > > > > >> > > DT,
> > > > > > > >> > > > >> on ZK.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> authenticates
> > > > with a
> > > > > > > >> broker.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > > > > > authenticated,
> > > > > > > >> it
> > > > > > > >> > > will
> > > > > > > >> > > > >> make a
> > > > > > > >> > > > >> >> > broker
> > > > > > > >> > > > >> >> > >> >> side
> > > > > > > >> > > > >> >> > >> >> > > > call to
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation
> > token.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker
> generates
> > a
> > > > > shared
> > > > > > > >> secret
> > > > > > > >> > > > based
> > > > > > > >> > > > >> on
> > > > > > > >> > > > >> >> > >> >> > > HMAC-SHA256(a
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    Password/Secret shared
> > > > between
> > > > > > all
> > > > > > > >> > > brokers,
> > > > > > > >> > > > >> >> > randomly
> > > > > > > >> > > > >> >> > >> >> > > generated
> > > > > > > >> > > > >> >> > >> >> > > > > >> > tokenId).
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Broker stores this
> > > token
> > > > in
> > > > > > its
> > > > > > > >> in
> > > > > > > >> > > > memory
> > > > > > > >> > > > >> cache.
> > > > > > > >> > > > >> >> > >> >> Broker
> > > > > > > >> > > > >> >> > >> >> > > > also stores
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the DelegationToken
> > > without
> > > > > the
> > > > > > > >> hmac in
> > > > > > > >> > > the
> > > > > > > >> > > > >> >> > zookeeper.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. All brokers will
> > have a
> > > > > cache
> > > > > > > >> backed
> > > > > > > >> > > by
> > > > > > > >> > > > >> >> > zookeeper so
> > > > > > > >> > > > >> >> > >> >> they
> > > > > > > >> > > > >> >> > >> >> > > > will all
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    get notified whenever
> a
> > > new
> > > > > > token
> > > > > > > is
> > > > > > > >> > > > >> generated and
> > > > > > > >> > > > >> >> > they
> > > > > > > >> > > > >> >> > >> >> will
> > > > > > > >> > > > >> >> > >> >> > > > update
> > > > > > > >> > > > >> >> > >> >> > > > > >> > their
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    local cache whenever
> > token
> > > > > state
> > > > > > > >> changes.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Broker returns the
> > > token
> > > > to
> > > > > > > >> Client.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Probable race
> > > condition,
> > > > > > client
> > > > > > > >> tries
> > > > > > > >> > > to
> > > > > > > >> > > > >> >> > authenticate
> > > > > > > >> > > > >> >> > >> >> with
> > > > > > > >> > > > >> >> > >> >> > > > a broker
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    that is yet to be
> > updated
> > > > with
> > > > > > the
> > > > > > > >> newly
> > > > > > > >> > > > >> generated
> > > > > > > >> > > > >> >> > DT.
> > > > > > > >> > > > >> >> > >> >> This
> > > > > > > >> > > > >> >> > >> >> > > can
> > > > > > > >> > > > >> >> > >> >> > > > > >> > probably be
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    dealt with making
> > > dtRequest
> > > > > > block
> > > > > > > >> until
> > > > > > > >> > > all
> > > > > > > >> > > > >> >> > brokers have
> > > > > > > >> > > > >> >> > >> >> > > > updated
> > > > > > > >> > > > >> >> > >> >> > > > > >> > their DT
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    cache. Zk barrier or
> > > similar
> > > > > > > >> mechanism
> > > > > > > >> > > can
> > > > > > > >> > > > be
> > > > > > > >> > > > >> used.
> > > > > > > >> > > > >> >> > >> >> However,
> > > > > > > >> > > > >> >> > >> >> > > > all such
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    mechanisms will
> increase
> > > > > > > complexity.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Using a static
> secret
> > > key
> > > > > > from
> > > > > > > >> config
> > > > > > > >> > > > >> file. Will
> > > > > > > >> > > > >> >> > >> >> require
> > > > > > > >> > > > >> >> > >> >> > > yet
> > > > > > > >> > > > >> >> > >> >> > > > > >> > another
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    config and uses a
> static
> > > > > secret
> > > > > > > >> key. It
> > > > > > > >> > > is
> > > > > > > >> > > > >> advised
> > > > > > > >> > > > >> >> > to
> > > > > > > >> > > > >> >> > >> >> rotate
> > > > > > > >> > > > >> >> > >> >> > > > secret
> > > > > > > >> > > > >> >> > >> >> > > > > >> > keys
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    periodically. This can
> > be
> > > > > > avoided
> > > > > > > >> with
> > > > > > > >> > > > >> controller
> > > > > > > >> > > > >> >> > >> >> generating
> > > > > > > >> > > > >> >> > >> >> > > > > >> > secretKey and
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    passing to brokers
> > > > > periodically.
> > > > > > > >> However,
> > > > > > > >> > > > >> this will
> > > > > > > >> > > > >> >> > >> >> require
> > > > > > > >> > > > >> >> > >> >> > > > brokers to
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maintain certain
> counts
> > of
> > > > > > > >> secretKeys.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 1) Have
> > > controller
> > > > > > > >> generate
> > > > > > > >> > > > >> delegation
> > > > > > > >> > > > >> >> > token.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> authenticates
> > > > with a
> > > > > > > >> broker.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > > > > > authenticated,
> > > > > > > >> it
> > > > > > > >> > > will
> > > > > > > >> > > > >> make a
> > > > > > > >> > > > >> >> > broker
> > > > > > > >> > > > >> >> > >> >> side
> > > > > > > >> > > > >> >> > >> >> > > > call to
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation
> > token.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. Broker forwards the
> > > > request
> > > > > > to
> > > > > > > >> > > > controller.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Controller
> generates
> > a
> > > DT
> > > > > and
> > > > > > > >> > > broadcasts
> > > > > > > >> > > > >> to all
> > > > > > > >> > > > >> >> > >> >> brokers.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. Broker stores this
> > > token
> > > > in
> > > > > > its
> > > > > > > >> memory
> > > > > > > >> > > > >> cache.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Controller responds
> > to
> > > > > > broker’s
> > > > > > > >> DT
> > > > > > > >> > > req.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    7. Broker returns the
> > > token
> > > > to
> > > > > > > >> Client.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. We will have to add
> > new
> > > > > APIs
> > > > > > to
> > > > > > > >> > > support
> > > > > > > >> > > > >> >> > controller
> > > > > > > >> > > > >> >> > >> >> pushing
> > > > > > > >> > > > >> >> > >> >> > > > tokens
> > > > > > > >> > > > >> >> > >> >> > > > > >> > to
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    brokers on top of the
> > > > minimal
> > > > > > APIs
> > > > > > > >> that
> > > > > > > >> > > are
> > > > > > > >> > > > >> >> > currently
> > > > > > > >> > > > >> >> > >> >> > > proposed.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. We will also have
> to
> > > add
> > > > > APIs
> > > > > > > to
> > > > > > > >> > > support
> > > > > > > >> > > > >> the
> > > > > > > >> > > > >> >> > >> >> bootstrapping
> > > > > > > >> > > > >> >> > >> >> > > > case,
> > > > > > > >> > > > >> >> > >> >> > > > > >> > i.e,
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    when a new broker
> comes
> > up
> > > > it
> > > > > > will
> > > > > > > >> have
> > > > > > > >> > > to
> > > > > > > >> > > > >> get all
> > > > > > > >> > > > >> >> > >> >> delegation
> > > > > > > >> > > > >> >> > >> >> > > > tokens
> > > > > > > >> > > > >> >> > >> >> > > > > >> > from
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    the controller.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. In catastrophic
> > > failures
> > > > > > where
> > > > > > > >> all
> > > > > > > >> > > > brokers
> > > > > > > >> > > > >> go
> > > > > > > >> > > > >> >> > down,
> > > > > > > >> > > > >> >> > >> >> the
> > > > > > > >> > > > >> >> > >> >> > > > tokens will
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if
> servers
> > > are
> > > > > > > >> restarted as
> > > > > > > >> > > > >> tokens
> > > > > > > >> > > > >> >> > are not
> > > > > > > >> > > > >> >> > >> >> > > > persisted
> > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this happens, then
> > > there
> > > > > are
> > > > > > > more
> > > > > > > >> > > > important
> > > > > > > >> > > > >> >> > things to
> > > > > > > >> > > > >> >> > >> >> > > worry
> > > > > > > >> > > > >> >> > >> >> > > > about
> > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
> > > > > > > >> re-authenticate.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 2) Do not
> > > > distribute
> > > > > > DT
> > > > > > > to
> > > > > > > >> > > other
> > > > > > > >> > > > >> brokers
> > > > > > > >> > > > >> >> > at
> > > > > > > >> > > > >> >> > >> >> all.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client
> authenticates
> > > > with a
> > > > > > > >> broker.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > > > > > authenticated,
> > > > > > > >> it
> > > > > > > >> > > will
> > > > > > > >> > > > >> make a
> > > > > > > >> > > > >> >> > broker
> > > > > > > >> > > > >> >> > >> >> side
> > > > > > > >> > > > >> >> > >> >> > > > call to
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation
> > token.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker
> generates
> > DT
> > > > of
> > > > > > > form,
> > > > > > > >> > > [hmac +
> > > > > > > >> > > > >> (owner,
> > > > > > > >> > > > >> >> > >> >> renewer,
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maxLifeTime, id, hmac,
> > > > > > > >> expirationTime)]
> > > > > > > >> > > and
> > > > > > > >> > > > >> passes
> > > > > > > >> > > > >> >> > back
> > > > > > > >> > > > >> >> > >> >> this
> > > > > > > >> > > > >> >> > >> >> > > > DT to
> > > > > > > >> > > > >> >> > >> >> > > > > >> > client.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    hmac is generated via
> > > > > > > >> {HMAC-SHA256(owner,
> > > > > > > >> > > > >> renewer,
> > > > > > > >> > > > >> >> > >> >> > > > maxLifeTime, id,
> > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac,
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    expirationTime) using
> > > > > > SecretKey}.
> > > > > > > >> Note
> > > > > > > >> > > that
> > > > > > > >> > > > >> all
> > > > > > > >> > > > >> >> > brokers
> > > > > > > >> > > > >> >> > >> >> have
> > > > > > > >> > > > >> >> > >> >> > > > this
> > > > > > > >> > > > >> >> > >> >> > > > > >> > SecretKey.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Client then goes to
> > any
> > > > > > broker
> > > > > > > >> and to
> > > > > > > >> > > > >> >> > authenticate
> > > > > > > >> > > > >> >> > >> >> sends
> > > > > > > >> > > > >> >> > >> >> > > > the DT.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    Broker recalculates
> hmac
> > > > using
> > > > > > > >> (owner,
> > > > > > > >> > > > >> renewer,
> > > > > > > >> > > > >> >> > >> >> maxLifeTime,
> > > > > > > >> > > > >> >> > >> >> > > > id, hmac,
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    expirationTime) info
> > from
> > > DT
> > > > > and
> > > > > > > its
> > > > > > > >> > > > >> SecretKey. If
> > > > > > > >> > > > >> >> > it
> > > > > > > >> > > > >> >> > >> >> matches
> > > > > > > >> > > > >> >> > >> >> > > > with
> > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac of
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    DT, client is
> > > authenticated.
> > > > > > Yes,
> > > > > > > >> it will
> > > > > > > >> > > > do
> > > > > > > >> > > > >> other
> > > > > > > >> > > > >> >> > >> >> obvious
> > > > > > > >> > > > >> >> > >> >> > > > checks of
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    timestamp expiry and
> > such.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > > Note that secret key will
> > be
> > > > > > > generated
> > > > > > > >> by
> > > > > > > >> > > > >> controller
> > > > > > > >> > > > >> >> > and
> > > > > > > >> > > > >> >> > >> >> passed
> > > > > > > >> > > > >> >> > >> >> > > to
> > > > > > > >> > > > >> >> > >> >> > > > > >> > brokers
> > > > > > > >> > > > >> >> > >> >> > > > > >> > > periodically.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. How to delete a DT?
> > > Yes,
> > > > > that
> > > > > > > is
> > > > > > > >> a
> > > > > > > >> > > > downside
> > > > > > > >> > > > >> >> > here.
> > > > > > > >> > > > >> >> > >> >> However,
> > > > > > > >> > > > >> >> > >> >> > > > this can
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be handled with
> brokers
> > > > > > > maintaining
> > > > > > > >> a
> > > > > > > >> > > > >> blacklist of
> > > > > > > >> > > > >> >> > DTs,
> > > > > > > >> > > > >> >> > >> >> DTs
> > > > > > > >> > > > >> >> > >> >> > > > from this
> > > > > > > >> > > > >> >> > >> >> > > > > >> > list
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    can be removed after
> > > expiry.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. In catastrophic
> > > failures
> > > > > > where
> > > > > > > >> all
> > > > > > > >> > > > brokers
> > > > > > > >> > > > >> go
> > > > > > > >> > > > >> >> > down,
> > > > > > > >> > > > >> >> > >> >> the
> > > > > > > >> > > > >> >> > >> >> > > > tokens will
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if
> servers
> > > are
> > > > > > > >> restarted as
> > > > > > > >> > > > >> tokens
> > > > > > > >> > > > >> >> > are not
> > > > > > > >> > > > >> >> > >> >> > > > persisted
> > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this happens, then
> > > there
> > > > > are
> > > > > > > more
> > > > > > > >> > > > important
> > > > > > > >> > > > >> >> > things to
> > > > > > > >> > > > >> >> > >> >> > > worry
> > > > > > > >> > > > >> >> > >> >> > > > about
> > > > > > > >> > > > >> >> > >> >> > > > > >> > and
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
> > > > > > > >> re-authenticate.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at
> > 1:58
> > > > PM,
> > > > > > > Parth
> > > > > > > >> > > > >> Brahmbhatt <
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > pbrahmbhatt@hortonworks.com>
> > > > > > wrote:
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Hi,
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >> I have filed KIP-48 so
> we
> > > can
> > > > > > offer
> > > > > > > >> hadoop
> > > > > > > >> > > > like
> > > > > > > >> > > > >> >> > delegation
> > > > > > > >> > > > >> >> > >> >> > > > tokens in
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >> kafka. You can review
> the
> > > > design
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > >> > > > >> >> > >> >>
> > > > > > > >> > > > >> >> >
> > > > > > > >> > > > >>
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > > > > > > >> > > > >> >> > >> >> > > > > >> > .
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >> This KIP depends on
> KIP-43
> > > and
> > > > > we
> > > > > > > >> have also
> > > > > > > >> > > > >> >> > discussed an
> > > > > > > >> > > > >> >> > >> >> > > > alternative to
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >> proposed design here<
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > >> > > > >> >> > >> >>
> > > > > > > >> > > > >> >> >
> > > > > > > >> > > > >>
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >.
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Thanks
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Parth
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > > --
> > > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > > >> > > > >> >> > >> >> > > > > >> > > Regards,
> > > > > > > >> > > > >> >> > >> >> > > > > >> > > Ashish
> > > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > > >> > > > >> >> > >> >> > > >
> > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > >> > > > >> >> > >> >> > > --
> > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > >> > > > >> >> > >> >> > > Regards,
> > > > > > > >> > > > >> >> > >> >> > > Ashish
> > > > > > > >> > > > >> >> > >> >> > >
> > > > > > > >> > > > >> >> > >> >>
> > > > > > > >> > > > >> >> >
> > > > > > > >> > > > >>
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Ismael Juma <is...@juma.me.uk>]

Thanks Rajini. It would probably require a separate KIP as it will
introduce user visible changes. We could also update KIP-48 to have this
information, but it seems cleaner to do it separately. We can discuss that
in the KIP call today.

Ismael

On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <
rajinisivaram@googlemail.com> wrote:

> Ismael,
>
> I have created a JIRA (https://issues.apache.org/jira/browse/KAFKA-3751)
> for adding SCRAM as a SASL mechanism. Would that need another KIP? If
> KIP-48 will use this mechanism, can this just be a JIRA that gets reviewed
> when the PR is ready?
>
> Thank you,
>
> Rajini
>
> On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Thanks Rajini, SCRAM seems like a good candidate.
> >
> > Gwen had independently mentioned this as a SASL mechanism that might be
> > useful for Kafka and I have been meaning to check it in more detail. Good
> > to know that you are willing to contribute an implementation. Maybe we
> > should file a separate JIRA for this?
> >
> > Ismael
> >
> > On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram <
> > rajinisivaram@googlemail.com> wrote:
> >
> > > SCRAM (Salted Challenge Response Authentication Mechanism) is a better
> > > mechanism than Digest-MD5. Java doesn't come with a built-in SCRAM
> > > SaslServer or SaslClient, but I will be happy to add support in Kafka
> > since
> > > it would be a useful mechanism to support anyway.
> > > https://tools.ietf.org/html/rfc7677 describes the protocol for
> > > SCRAM-SHA-256.
> > >
> > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Parth,
> > > >
> > > > Thanks for the explanation. A couple of more questions.
> > > >
> > > > 110. What does getDelegationTokenAs mean?
> > > >
> > > > 111. What's the typical rate of getting and renewing delegation
> tokens?
> > > > That may have an impact on whether they should be directed to the
> > > > controller.
> > > >
> > > > Jun
> > > >
> > > > On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt <
> > > > brahmbhatt.parth@gmail.com> wrote:
> > > >
> > > > > Hi Jun,
> > > > >
> > > > > Thanks for reviewing.
> > > > >
> > > > > * We could add a Cluster action to add acls on who can request
> > > delegation
> > > > > tokens. I don't see the use case for that yet but down the line
> when
> > we
> > > > > start supporting getDelegationTokenAs it will be necessary.
> > > > > * Yes we recommend tokens to be only used/distributed over secure
> > > > channels.
> > > > > * Depending on what design we end up choosing Invalidation will be
> > > > > responsibility of every broker or controller.
> > > > > * I am not sure if I documented somewhere that invalidation will
> > > directly
> > > > > go through zookeeper but that is not the intent. Invalidation will
> > > either
> > > > > be request based or due to expiration. No direct zookeeper
> > interaction
> > > > from
> > > > > any client.
> > > > > * "Broker also stores the DelegationToken without the hmac in the
> > > > > zookeeper." : Sorry about the confusion. The sole purpose of
> > zookeeper
> > > in
> > > > > this design is as distribution channel for tokens between all
> brokers
> > > > and a
> > > > > layer that ensures only tokens that were generated by making a
> > request
> > > > to a
> > > > > broker will be accepted (more on this in second paragraph). The
> token
> > > > > consists of few elements (owner, renewer, uuid , expiration, hmac)
> ,
> > > one
> > > > of
> > > > > which is the finally generated hmac but hmac it self is derivable
> if
> > > you
> > > > > have all the other elements of the token + secret key to generate
> > hmac.
> > > > > Given zookeeper does not provide SSL support we do not want the
> > entire
> > > > > token to be wire transferred to zookeeper as that will be an
> insecure
> > > > wire
> > > > > transfer. Instead we only store all the other elements of a
> > delegation
> > > > > tokens. Brokers can read these elements and because they also have
> > > access
> > > > > to secret key they will be able to generate hmac on their end.
> > > > >
> > > > > One of the alternative proposed is to avoid zookeeper altogether. A
> > > > Client
> > > > > will call broker with required information (owner, renwer,
> > expiration)
> > > > and
> > > > > get back (signed hmac, uuid). Broker won't store this in zookeeper.
> > > From
> > > > > this point a client can contact any broker with all the delegation
> > > token
> > > > > info (owner, rewner, expiration, hmac, uuid) the borker will
> > regenerate
> > > > the
> > > > > hmac and as long as it matches with hmac presented by client ,
> broker
> > > > will
> > > > > allow the request to authenticate.  Only problem with this approach
> > is
> > > if
> > > > > the secret key is compromised any client can now generate random
> > tokens
> > > > and
> > > > > they will still be able to authenticate as any user they like. with
> > > > > zookeeper we guarantee that only tokens acquired via a broker
> (using
> > > some
> > > > > auth scheme other than delegation token) will be accepted. We need
> to
> > > > > discuss which proposal makes more sense and we can go over it in
> > > > tomorrow's
> > > > > meeting.
> > > > >
> > > > > Also, can you forward the invite to me?
> > > > >
> > > > > Thanks
> > > > > Parth
> > > > >
> > > > >
> > > > >
> > > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <ju...@confluent.io>
> wrote:
> > > > >
> > > > > > Thanks for the KIP. A few comments.
> > > > > >
> > > > > > 100. This potentially can be useful for Kafka Connect and Kafka
> > rest
> > > > > proxy
> > > > > > where a worker agent will need to run a task on behalf of a
> client.
> > > We
> > > > > will
> > > > > > likely need to change how those services use Kafka clients
> > > > > > (producer/consumer). Instead of a shared client per worker, we
> will
> > > > need
> > > > > a
> > > > > > client per user task since the authentication happens at the
> > > connection
> > > > > > level. For Kafka Connect, the renewer will be the workers. So, we
> > > > > probably
> > > > > > need to allow multiple renewers. For Kafka rest proxy, the
> renewer
> > > can
> > > > > > probably just be the creator of the token.
> > > > > >
> > > > > > 101. Do we need new acl on who can request delegation tokens?
> > > > > >
> > > > > > 102. Do we recommend people to send delegation tokens in an
> > encrypted
> > > > > > channel?
> > > > > >
> > > > > > 103. Who is responsible for expiring tokens, every broker?
> > > > > >
> > > > > > 104. For invalidating tokens, would it be better to do it in a
> > > request
> > > > > > instead of going to ZK directly?
> > > > > >
> > > > > > 105. The terminology of client in the wiki sometimes refers to
> the
> > > end
> > > > > > client and some other times refers to the client using the
> > delegation
> > > > > > tokens. It would be useful to distinguish between the two.
> > > > > >
> > > > > > 106. Could you explain the sentence "Broker also stores the
> > > > > DelegationToken
> > > > > > without the hmac in the zookeeper." a bit more? I thought the
> > > > delegation
> > > > > > token is the hmac.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <ju...@confluent.io>
> wrote:
> > > > > >
> > > > > > > Hi, Harsha,
> > > > > > >
> > > > > > > Just sent out a KIP meeting invite. We can discuss this in the
> > > > meeting
> > > > > > > tomorrow.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <ka...@harsha.io>
> wrote:
> > > > > > >
> > > > > > >> Hi All,
> > > > > > >>            Can we have a KIP meeting around this. The KIP is
> up
> > > for
> > > > > > >>            sometime and if there are any questions lets
> quickly
> > > hash
> > > > > out
> > > > > > >>            details.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Harsha
> > > > > > >>
> > > > > > >> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt wrote:
> > > > > > >> > That is what the hadoop echo system uses so no good reason
> > > really.
> > > > > We
> > > > > > >> > could
> > > > > > >> > change it to whatever is the newest recommended standard is.
> > > > > > >> >
> > > > > > >> > Thanks
> > > > > > >> > Parth
> > > > > > >> >
> > > > > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <
> > ismael@juma.me.uk
> > > >
> > > > > > wrote:
> > > > > > >> >
> > > > > > >> > > Hi Parth,
> > > > > > >> > >
> > > > > > >> > > Thanks for the KIP. I only started reviewing this and may
> > have
> > > > > > >> additional
> > > > > > >> > > questions later. The immediate question that came to mind
> is
> > > our
> > > > > > >> choice of
> > > > > > >> > > "DIGEST-MD5" even though it's marked as OBSOLETE in the
> IANA
> > > > > > Registry
> > > > > > >> of
> > > > > > >> > > SASL mechanisms and the original RFC (2831) has been moved
> > to
> > > > > > Historic
> > > > > > >> > > status:
> > > > > > >> > >
> > > > > > >> > > https://tools.ietf.org/html/rfc6331
> > > > > > >> > >
> > > > > >
> > > http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> > > > > > >> > >
> > > > > > >> > > What is the reasoning behind that choice?
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Ismael
> > > > > > >> > >
> > > > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <
> > > > gwen@confluent.io
> > > > > >
> > > > > > >> wrote:
> > > > > > >> > >
> > > > > > >> > > > Also comments inline :)
> > > > > > >> > > >
> > > > > > >> > > > > * I want to emphasize that even though delegation
> tokens
> > > > are a
> > > > > > >> Hadoop
> > > > > > >> > > > > innovation, I feel very strongly about not adding
> > > dependency
> > > > > on
> > > > > > >> Hadoop
> > > > > > >> > > > > when implementing delegation tokens for Kafka. The KIP
> > > > doesn't
> > > > > > >> imply
> > > > > > >> > > > > such dependency, but if you can clarify...
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > *No hadoop dependency.*
> > > > > > >> > > >
> > > > > > >> > > > Yay! Just add this to the KIP so no one will read the
> KIP
> > > and
> > > > > > panic
> > > > > > >> > > > three weeks before the next release...
> > > > > > >> > > >
> > > > > > >> > > > > * Can we get delegation token at any time after
> > > > > authenticating?
> > > > > > >> only
> > > > > > >> > > > > immediately after?
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > *As long as you are authenticated you can get
> delegation
> > > > > tokens.
> > > > > > >> We
> > > > > > >> > > need
> > > > > > >> > > > to
> > > > > > >> > > > > discuss if a client authenticated using delegation
> > token,
> > > > can
> > > > > > also
> > > > > > >> > > > acquire
> > > > > > >> > > > > delegation token again or not. Also there is the
> > question
> > > of
> > > > > do
> > > > > > we
> > > > > > >> > > allow
> > > > > > >> > > > > anyone to acquire delegation token or we want specific
> > > ACLs
> > > > (I
> > > > > > >> think
> > > > > > >> > > its
> > > > > > >> > > > an
> > > > > > >> > > > > overkill.)*
> > > > > > >> > > >
> > > > > > >> > > > I agree that ACLs is an overkill.
> > > > > > >> > > >
> > > > > > >> > > > I think we are debating two options: Either require
> > Kerberos
> > > > > auth
> > > > > > >> for
> > > > > > >> > > > renewal or require non-owners to renew.
> > > > > > >> > > > I *think* the latter is simpler (it basically require a
> > "job
> > > > > > master"
> > > > > > >> > > > to take responsibility for the renewal, it will have its
> > own
> > > > > > >> identity
> > > > > > >> > > > anyway and I think this is the correct design pattern
> > > anyway.
> > > > > For
> > > > > > >> > > > storm, I'd expect Nimbus to coordinate renewals?), but
> it
> > is
> > > > > hard
> > > > > > to
> > > > > > >> > > > debate simplicity without looking at the code changes
> > > > required.
> > > > > If
> > > > > > >> you
> > > > > > >> > > > have a draft of how the "require Kerberos" will look in
> > > Kafka
> > > > > > code,
> > > > > > >> > > > I'll be happy to take a look.
> > > > > > >> > > >
> > > > > > >> > > > > * My understanding is that tokens will propagate via
> ZK
> > > but
> > > > > > >> without
> > > > > > >> > > > > additional changes to UpdateMetadata protocol,
> correct?
> > > > > Clients
> > > > > > >> > > > > currently don't retry on SASL auth failure (IIRC), but
> > > since
> > > > > the
> > > > > > >> > > > > tokens propagate between brokers asynch, we will need
> to
> > > > > retry a
> > > > > > >> bit
> > > > > > >> > > > > to avoid clients failing auth due to timing issues.
> > > > > > >> > > > >
> > > > > > >> > > > > *I am considering 2 alternatives right now. The
> current
> > > > > > documented
> > > > > > >> > > > approach
> > > > > > >> > > > > is zookeeper based and it does not require any changes
> > to
> > > > > > >> > > UpdateMetadata
> > > > > > >> > > > > protocol. An alternative approach can remove zookeeper
> > > > > > dependency
> > > > > > >> as
> > > > > > >> > > well
> > > > > > >> > > > > but we can discuss that in KIP discussion call.*
> > > > > > >> > > >
> > > > > > >> > > > Oooh! Sounds interesting. Do you want to ping Jun to
> > > arrange a
> > > > > > call?
> > > > > > >> > > >
> > > > > > >> > > > > * I liked Ashish's suggestion of having just the
> > > controller
> > > > > > issue
> > > > > > >> the
> > > > > > >> > > > > delegation tokens, to avoid syncing a shared secret.
> Not
> > > > sure
> > > > > if
> > > > > > >> we
> > > > > > >> > > > > want to continue the discussion here or on the wiki. I
> > > think
> > > > > > that
> > > > > > >> we
> > > > > > >> > > > > can decouple the problem of "token distribution" from
> > > > "shared
> > > > > > >> secret
> > > > > > >> > > > > distribution" and use the controller as the only token
> > > > > generator
> > > > > > >> to
> > > > > > >> > > > > solve the second issue, while still using ZK async to
> > > > > distribute
> > > > > > >> > > > > tokens.
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > *As mentioned in the previous Email I am fine with
> that
> > > > > approach
> > > > > > >> as
> > > > > > >> > > long
> > > > > > >> > > > as
> > > > > > >> > > > > we agree that the extra complexity of adding/updating
> > APIS
> > > > > adds
> > > > > > >> enough
> > > > > > >> > > > > value. The advantage with the controller approach is
> > > secret
> > > > > > >> rotation
> > > > > > >> > > can
> > > > > > >> > > > be
> > > > > > >> > > > > automated,frequent and would not require deployment. *
> > > > > > >> > > >
> > > > > > >> > > > Can you detail the extra complexity (or point me to the
> > > email
> > > > I
> > > > > > >> > > > missed?) - which Apis are required?
> > > > > > >> > > > As far as I can tell, clients can already find the
> > > controller
> > > > > from
> > > > > > >> > > > metadata. I'm a bit more concerned about controller
> load.
> > > > > > >> > > >
> > > > > > >> > > > >
> > > > > > >> > > > > * While I like the idea of forcing kerberos auth for
> > > > renwal, I
> > > > > > >> think
> > > > > > >> > > > > it mixes the transport layer the the request content
> in
> > a
> > > > > pretty
> > > > > > >> ugly
> > > > > > >> > > > > way. Perhaps limiting renewer to non-owner is better.
> > > > > > >> > > > >
> > > > > > >> > > > > *I feel this is a necessary evil. While this will make
> > the
> > > > > kafka
> > > > > > >> code
> > > > > > >> > > > > pretty straight forward , forcing  renewer to
> non-owner
> > > > pushes
> > > > > > >> the code
> > > > > > >> > > > > ugliness to client and makes it even harder to
> > > integrate.  *
> > > > > > >> > > >
> > > > > > >> > > > As mentioned before, I don't think the "renewal by
> other"
> > > > > approach
> > > > > > >> is
> > > > > > >> > > > that ugly for the clients we expect to use delegation
> > tokens
> > > > > since
> > > > > > >> > > > they will have an app-master of some sort who requested
> > the
> > > > > token
> > > > > > to
> > > > > > >> > > > begin with.
> > > > > > >> > > >
> > > > > > >> > > > >
> > > > > > >> > > > > The response for my question on how multiple
> identities
> > > will
> > > > > be
> > > > > > >> > > > > handled wasn't super clear to me - AFAIK, we don't
> > > > > authenticate
> > > > > > >> each
> > > > > > >> > > > > request, we authenticate connections.
> > > > > > >> > > > >
> > > > > > >> > > > > *We authenticate connections, and only when they are
> > being
> > > > > > >> established.
> > > > > > >> > > > Let
> > > > > > >> > > > > me try to phrase this as a question, in absence of
> > > > delegation
> > > > > > >> tokens if
> > > > > > >> > > > we
> > > > > > >> > > > > had to support the use case using user TGT's how would
> > we
> > > do
> > > > > it?
> > > > > > >> My
> > > > > > >> > > point
> > > > > > >> > > > > was it would be no different with delegation tokens.
> The
> > > use
> > > > > > case
> > > > > > >> you
> > > > > > >> > > are
> > > > > > >> > > > > describing seems more like impersonation.*
> > > > > > >> > > >
> > > > > > >> > > > Yeah, I thought that one of the things that delegation
> > > tokens
> > > > > > >> handled.
> > > > > > >> > > > Maybe I got it wrong :)
> > > > > > >> > > >
> > > > > > >> > > > Thanks for the detailed answers.
> > > > > > >> > > >
> > > > > > >> > > > Gwen
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > > Thanks
> > > > > > >> > > > > Parth
> > > > > > >> > > > >
> > > > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <
> > > > > > gwen@confluent.io
> > > > > > >> >
> > > > > > >> > > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > >> Hi Parth and Harsha,
> > > > > > >> > > > >>
> > > > > > >> > > > >> Few more comments:
> > > > > > >> > > > >>
> > > > > > >> > > > >> * The API / RequestResponse section doesn't seem to
> > have
> > > > good
> > > > > > >> > > > >> description of the changes to the Kafka Protocol.
> > Sounds
> > > > like
> > > > > > >> you are
> > > > > > >> > > > >> proposing new DelegationTokenRequest and
> > > RenewTokenRequest
> > > > > (and
> > > > > > >> > > > >> matching responses), without detailing the contents
> of
> > > the
> > > > > > >> requests
> > > > > > >> > > > >> and responses? Or rather, you show the class
> interface,
> > > but
> > > > > not
> > > > > > >> the
> > > > > > >> > > > >> underlying protocol. This could be seen as an
> > > > implementation
> > > > > > >> detail,
> > > > > > >> > > > >> but since the binary protocol is what we provide to
> > > > non-Java
> > > > > > >> clients,
> > > > > > >> > > > >> we need to show the changes there.
> > > > > > >> > > > >>
> > > > > > >> > > > >> * getDelegationToken sounds like
> > > > > delegationTokenRequestHandler?
> > > > > > >> Is it
> > > > > > >> > > > >> planned to be part of KafkaApi? or Client? Its
> > unclear...
> > > > > > >> > > > >>
> > > > > > >> > > > >> * I want to emphasize that even though delegation
> > tokens
> > > > are
> > > > > a
> > > > > > >> Hadoop
> > > > > > >> > > > >> innovation, I feel very strongly about not adding
> > > > dependency
> > > > > on
> > > > > > >> Hadoop
> > > > > > >> > > > >> when implementing delegation tokens for Kafka. The
> KIP
> > > > > doesn't
> > > > > > >> imply
> > > > > > >> > > > >> such dependency, but if you can clarify...
> > > > > > >> > > > >>
> > > > > > >> > > > >> * Can we get delegation token at any time after
> > > > > authenticating?
> > > > > > >> only
> > > > > > >> > > > >> immediately after?
> > > > > > >> > > > >>
> > > > > > >> > > > >> * My understanding is that tokens will propagate via
> ZK
> > > but
> > > > > > >> without
> > > > > > >> > > > >> additional changes to UpdateMetadata protocol,
> correct?
> > > > > Clients
> > > > > > >> > > > >> currently don't retry on SASL auth failure (IIRC),
> but
> > > > since
> > > > > > the
> > > > > > >> > > > >> tokens propagate between brokers asynch, we will need
> > to
> > > > > retry
> > > > > > a
> > > > > > >> bit
> > > > > > >> > > > >> to avoid clients failing auth due to timing issues.
> > > > > > >> > > > >>
> > > > > > >> > > > >> * Strongly agreeing on clients not touching ZK
> directly
> > > :)
> > > > > > >> > > > >>
> > > > > > >> > > > >> * I liked Ashish's suggestion of having just the
> > > controller
> > > > > > >> issue the
> > > > > > >> > > > >> delegation tokens, to avoid syncing a shared secret.
> > Not
> > > > sure
> > > > > > if
> > > > > > >> we
> > > > > > >> > > > >> want to continue the discussion here or on the wiki.
> I
> > > > think
> > > > > > >> that we
> > > > > > >> > > > >> can decouple the problem of "token distribution" from
> > > > "shared
> > > > > > >> secret
> > > > > > >> > > > >> distribution" and use the controller as the only
> token
> > > > > > generator
> > > > > > >> to
> > > > > > >> > > > >> solve the second issue, while still using ZK async to
> > > > > > distribute
> > > > > > >> > > > >> tokens.
> > > > > > >> > > > >>
> > > > > > >> > > > >> * I am also uncomfortable with infinite lifetime of
> > > tokens
> > > > > (and
> > > > > > >> hoped
> > > > > > >> > > > >> to hear from others in the community) - but having
> the
> > > > option
> > > > > > and
> > > > > > >> > > > >> documenting it as less secure, allows users to
> > configure
> > > > > their
> > > > > > >> system
> > > > > > >> > > > >> as the wish.
> > > > > > >> > > > >>
> > > > > > >> > > > >> * While I like the idea of forcing kerberos auth for
> > > > renwal,
> > > > > I
> > > > > > >> think
> > > > > > >> > > > >> it mixes the transport layer the the request content
> > in a
> > > > > > pretty
> > > > > > >> ugly
> > > > > > >> > > > >> way. Perhaps limiting renewer to non-owner is better.
> > > > > > >> > > > >>
> > > > > > >> > > > >> Things I'd still like to see:
> > > > > > >> > > > >>
> > > > > > >> > > > >> * More detailed explanation on what we plan to do for
> > the
> > > > > java
> > > > > > >> clients
> > > > > > >> > > > >> specifically - new configuration? new APIs?
> > > > > > >> > > > >> The response for my question on how multiple
> identities
> > > > will
> > > > > be
> > > > > > >> > > > >> handled wasn't super clear to me - AFAIK, we don't
> > > > > authenticate
> > > > > > >> each
> > > > > > >> > > > >> request, we authenticate connections.
> > > > > > >> > > > >>
> > > > > > >> > > > >> * Alternatives: Delegation tokens are only used in
> the
> > > > Hadoop
> > > > > > >> > > > >> ecosystem. I'm wondering if there are alternatives in
> > > other
> > > > > > >> ecosystems
> > > > > > >> > > > >> (Mesos? Tachyon? Cassandra?) and whether there are
> some
> > > > > > >> advantages
> > > > > > >> > > > >> there.
> > > > > > >> > > > >>
> > > > > > >> > > > >> Gwen
> > > > > > >> > > > >>
> > > > > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <
> > kafka@harsha.io
> > > >
> > > > > > wrote:
> > > > > > >> > > > >> > Hi Gwen,
> > > > > > >> > > > >> >            Can you look at Parth's last reply. Does
> > it
> > > > > answer
> > > > > > >> your
> > > > > > >> > > > >> >            concerns.
> > > > > > >> > > > >> >
> > > > > > >> > > > >> > Thanks,
> > > > > > >> > > > >> > Harsha
> > > > > > >> > > > >> >
> > > > > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt
> > > wrote:
> > > > > > >> > > > >> >> Thanks for reviewing Gwen. The wiki already has
> > > details
> > > > on
> > > > > > >> token
> > > > > > >> > > > >> >> expiration
> > > > > > >> > > > >> >> under token acquisition process
> > > > > > >> > > > >> >> <
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> > > > > > >> > > > >> >.
> > > > > > >> > > > >> >> Current proposal is that tokens will expire based
> > on a
> > > > > > server
> > > > > > >> side
> > > > > > >> > > > >> >> configuration (default 24 hours) unless renewed.
> > > Renewal
> > > > > is
> > > > > > >> only
> > > > > > >> > > > allowed
> > > > > > >> > > > >> >> until the max life time of token. Alternatively we
> > > could
> > > > > > also
> > > > > > >> make
> > > > > > >> > > > that
> > > > > > >> > > > >> >> an
> > > > > > >> > > > >> >> optional param and the server side default can
> serve
> > > as
> > > > > the
> > > > > > >> upper
> > > > > > >> > > > bound.
> > > > > > >> > > > >> >>
> > > > > > >> > > > >> >> To your second point it will be done exactly the
> > same
> > > > way
> > > > > we
> > > > > > >> would
> > > > > > >> > > > >> >> support
> > > > > > >> > > > >> >> multiple keytabs. The calling client will have to
> > put
> > > > the
> > > > > > >> tokens it
> > > > > > >> > > > >> wants
> > > > > > >> > > > >> >> to use in the subject instance and call
> > > produce/consume
> > > > > > inside
> > > > > > >> > > > >> >> subject.doas. Each caller will have to keep track
> of
> > > its
> > > > > own
> > > > > > >> > > > subject. I
> > > > > > >> > > > >> >> will have to look at the code to see if we support
> > > this
> > > > > > >> feature
> > > > > > >> > > right
> > > > > > >> > > > >> now
> > > > > > >> > > > >> >> but my understanding is delegation token shouldn't
> > > need
> > > > > any
> > > > > > >> special
> > > > > > >> > > > >> >> treatment as its just another type of Credential
> in
> > > the
> > > > > > >> subject.
> > > > > > >> > > > >> >>
> > > > > > >> > > > >> >> I would also like to know what is your opinion
> about
> > > > > > infinite
> > > > > > >> > > renewal
> > > > > > >> > > > >> (my
> > > > > > >> > > > >> >> recommendation is to not support this), tokens
> > > renewing
> > > > > them
> > > > > > >> > > self(my
> > > > > > >> > > > >> >> recommendation is to not support this) and most
> > > > > importantly
> > > > > > >> your
> > > > > > >> > > > choice
> > > > > > >> > > > >> >> between the alternatives listed on this thread
> > > > > > >> > > > >> >> <
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> > > > > > >> > > > >> >
> > > > > > >> > > > >> >> ( I am leaning towards the alternative-2 minus
> > > > controller
> > > > > > >> > > > distributing
> > > > > > >> > > > >> >> secret). Thanks again for reviewing.
> > > > > > >> > > > >> >>
> > > > > > >> > > > >> >> Thanks
> > > > > > >> > > > >> >> Parth
> > > > > > >> > > > >> >>
> > > > > > >> > > > >> >>
> > > > > > >> > > > >> >>
> > > > > > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <
> > > > > > >> gwen@confluent.io>
> > > > > > >> > > > wrote:
> > > > > > >> > > > >> >>
> > > > > > >> > > > >> >> > Harsha,
> > > > > > >> > > > >> >> >
> > > > > > >> > > > >> >> > I was thinking of the Rest Proxy. I didn't see
> > your
> > > > > design
> > > > > > >> yet,
> > > > > > >> > > > but in
> > > > > > >> > > > >> >> > our proxy, we have a set of producers, which
> will
> > > > serve
> > > > > > >> multiple
> > > > > > >> > > > users
> > > > > > >> > > > >> >> > going through the proxy. Since these users will
> > have
> > > > > > >> different
> > > > > > >> > > > >> >> > privileges, they'll need to authenticate
> > separately,
> > > > and
> > > > > > >> can't
> > > > > > >> > > > share a
> > > > > > >> > > > >> >> > token.
> > > > > > >> > > > >> >> >
> > > > > > >> > > > >> >> > Am I missing anything?
> > > > > > >> > > > >> >> >
> > > > > > >> > > > >> >> > Gwen
> > > > > > >> > > > >> >> >
> > > > > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <
> > > > kafka@harsha.io
> > > > > >
> > > > > > >> wrote:
> > > > > > >> > > > >> >> > > Gwen,
> > > > > > >> > > > >> >> > >            On your second point. Can you
> > describe
> > > a
> > > > > > >> usecase
> > > > > > >> > > where
> > > > > > >> > > > >> >> > >            mutliple clients ended up sharing a
> > > > > producer
> > > > > > >> and
> > > > > > >> > > even
> > > > > > >> > > > if
> > > > > > >> > > > >> they
> > > > > > >> > > > >> >> > >            do why can't they not use single
> > token
> > > > that
> > > > > > >> producer
> > > > > > >> > > > >> >> > >            captures. Why would we need
> multiple
> > > > > clients
> > > > > > >> with
> > > > > > >> > > > >> different
> > > > > > >> > > > >> >> > >            tokens sharing a single instance of
> > > > > producer.
> > > > > > >> Also
> > > > > > >> > > in
> > > > > > >> > > > >> this
> > > > > > >> > > > >> >> > >            case other clients have access all
> > the
> > > > > tokens
> > > > > > >> no?
> > > > > > >> > > > >> >> > >
> > > > > > >> > > > >> >> > > Thanks,
> > > > > > >> > > > >> >> > > Harsha
> > > > > > >> > > > >> >> > >
> > > > > > >> > > > >> >> > >
> > > > > > >> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira
> > > > wrote:
> > > > > > >> > > > >> >> > >> Sorry for the delay:
> > > > > > >> > > > >> >> > >>
> > > > > > >> > > > >> >> > >> Two questions that we didn't see in the wiki:
> > > > > > >> > > > >> >> > >> 1. Is there an expiration for delegation
> > tokens?
> > > > > > >> Renewal? How
> > > > > > >> > > > do we
> > > > > > >> > > > >> >> > >> revoke them?
> > > > > > >> > > > >> >> > >> 2. If we want to use delegation tokens for
> > > "do-as"
> > > > > > (say,
> > > > > > >> > > submit
> > > > > > >> > > > >> Storm
> > > > > > >> > > > >> >> > >> job as my user), we will need a producer for
> > > every
> > > > > job
> > > > > > >> (we
> > > > > > >> > > can't
> > > > > > >> > > > >> share
> > > > > > >> > > > >> >> > >> them between multiple jobs running on same
> > node),
> > > > > since
> > > > > > >> we
> > > > > > >> > > only
> > > > > > >> > > > >> >> > >> authenticate when connecting. Is there a plan
> > to
> > > > > change
> > > > > > >> this
> > > > > > >> > > for
> > > > > > >> > > > >> >> > >> delegation tokens, in order to allow multiple
> > > users
> > > > > > with
> > > > > > >> > > > different
> > > > > > >> > > > >> >> > >> tokens to share a client?
> > > > > > >> > > > >> >> > >>
> > > > > > >> > > > >> >> > >> Gwen
> > > > > > >> > > > >> >> > >>
> > > > > > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth
> > brahmbhatt
> > > > > > >> > > > >> >> > >> <br...@gmail.com> wrote:
> > > > > > >> > > > >> >> > >> > Bumping this up one more time, can other
> > > > committers
> > > > > > >> review?
> > > > > > >> > > > >> >> > >> >
> > > > > > >> > > > >> >> > >> > Thanks
> > > > > > >> > > > >> >> > >> > Parth
> > > > > > >> > > > >> >> > >> >
> > > > > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <
> > > > > > >> kafka@harsha.io>
> > > > > > >> > > > wrote:
> > > > > > >> > > > >> >> > >> >
> > > > > > >> > > > >> >> > >> >> Parth,
> > > > > > >> > > > >> >> > >> >>           Overall current design looks
> good
> > to
> > > > > me. I
> > > > > > >> am +1
> > > > > > >> > > on
> > > > > > >> > > > >> the
> > > > > > >> > > > >> >> > KIP.
> > > > > > >> > > > >> >> > >> >>
> > > > > > >> > > > >> >> > >> >> Gwen , Jun can you review this as well.
> > > > > > >> > > > >> >> > >> >>
> > > > > > >> > > > >> >> > >> >> -Harsha
> > > > > > >> > > > >> >> > >> >>
> > > > > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth
> > > > > brahmbhatt
> > > > > > >> wrote:
> > > > > > >> > > > >> >> > >> >> > Thanks for review Jitendra.
> > > > > > >> > > > >> >> > >> >> >
> > > > > > >> > > > >> >> > >> >> > I don't like the idea of infinite
> lifetime
> > > > but I
> > > > > > >> see the
> > > > > > >> > > > >> Streaming
> > > > > > >> > > > >> >> > use
> > > > > > >> > > > >> >> > >> >> > case. Even for Streaming use case I was
> > > hoping
> > > > > > >> there will
> > > > > > >> > > > be
> > > > > > >> > > > >> some
> > > > > > >> > > > >> >> > notion
> > > > > > >> > > > >> >> > >> >> > of
> > > > > > >> > > > >> >> > >> >> > master/driver that can get new
> delegation
> > > > tokens
> > > > > > at
> > > > > > >> fixed
> > > > > > >> > > > >> interval
> > > > > > >> > > > >> >> > and
> > > > > > >> > > > >> >> > >> >> > distribute to workers. If that is not
> the
> > > case
> > > > > for
> > > > > > >> we can
> > > > > > >> > > > >> discuss
> > > > > > >> > > > >> >> > >> >> > delegation tokens renewing them self and
> > the
> > > > > > >> security
> > > > > > >> > > > >> implications
> > > > > > >> > > > >> >> > of the
> > > > > > >> > > > >> >> > >> >> > same.
> > > > > > >> > > > >> >> > >> >> >
> > > > > > >> > > > >> >> > >> >> > I did not want clients to fetch tokens
> > from
> > > > > > >> zookeeper,
> > > > > > >> > > > >> overall I
> > > > > > >> > > > >> >> > think
> > > > > > >> > > > >> >> > >> >> > its
> > > > > > >> > > > >> >> > >> >> > better if clients don't rely on our
> > metadata
> > > > > store
> > > > > > >> and I
> > > > > > >> > > > >> think we
> > > > > > >> > > > >> >> > are
> > > > > > >> > > > >> >> > >> >> > moving in that direction with all the
> > KIP-4
> > > > > > >> improvements.
> > > > > > >> > > > I
> > > > > > >> > > > >> chose
> > > > > > >> > > > >> >> > >> >> > zookeeper as in this case the client
> will
> > > > still
> > > > > > >> just talk
> > > > > > >> > > > to
> > > > > > >> > > > >> >> > broker , its
> > > > > > >> > > > >> >> > >> >> > the brokers that will use zookeeper
> which
> > we
> > > > > > >> already do
> > > > > > >> > > > for a
> > > > > > >> > > > >> lot
> > > > > > >> > > > >> >> > of
> > > > > > >> > > > >> >> > >> >> > other
> > > > > > >> > > > >> >> > >> >> > usecases + ease of development + and the
> > > > ability
> > > > > > so
> > > > > > >> > > tokens
> > > > > > >> > > > >> will
> > > > > > >> > > > >> >> > survive
> > > > > > >> > > > >> >> > >> >> > even a rolling restart/cluster failure.
> > if a
> > > > > > >> majority
> > > > > > >> > > > agrees
> > > > > > >> > > > >> the
> > > > > > >> > > > >> >> > added
> > > > > > >> > > > >> >> > >> >> > complexity to have controller forwarding
> > > keys
> > > > to
> > > > > > all
> > > > > > >> > > > broker is
> > > > > > >> > > > >> >> > justified
> > > > > > >> > > > >> >> > >> >> > as
> > > > > > >> > > > >> >> > >> >> > it provides tighter security , I am fine
> > > with
> > > > > that
> > > > > > >> option
> > > > > > >> > > > too.
> > > > > > >> > > > >> >> > >> >> >
> > > > > > >> > > > >> >> > >> >> > Given zookeeper does not support SSL we
> > can
> > > > not
> > > > > > >> store
> > > > > > >> > > > master
> > > > > > >> > > > >> keys
> > > > > > >> > > > >> >> > in
> > > > > > >> > > > >> >> > >> >> > zookeeper as master keys will be exposed
> > on
> > > > > wire.
> > > > > > To
> > > > > > >> > > > support
> > > > > > >> > > > >> >> > rotation
> > > > > > >> > > > >> >> > >> >> > without affecting current clients is
> > > > something I
> > > > > > >> need to
> > > > > > >> > > > put
> > > > > > >> > > > >> more
> > > > > > >> > > > >> >> > thought
> > > > > > >> > > > >> >> > >> >> > in. My current proposal assumes the
> > rotation
> > > > > will
> > > > > > >> > > > invalidate
> > > > > > >> > > > >> all
> > > > > > >> > > > >> >> > current
> > > > > > >> > > > >> >> > >> >> > tokens.
> > > > > > >> > > > >> >> > >> >> >
> > > > > > >> > > > >> >> > >> >> > I request committers to also review and
> > post
> > > > > their
> > > > > > >> > > comments
> > > > > > >> > > > >> so we
> > > > > > >> > > > >> >> > can
> > > > > > >> > > > >> >> > >> >> > make
> > > > > > >> > > > >> >> > >> >> > progress on this KIP.
> > > > > > >> > > > >> >> > >> >> >
> > > > > > >> > > > >> >> > >> >> > Thanks
> > > > > > >> > > > >> >> > >> >> > Parth
> > > > > > >> > > > >> >> > >> >> >
> > > > > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish
> > > Singh
> > > > <
> > > > > > >> > > > >> asingh@cloudera.com
> > > > > > >> > > > >> >> > >
> > > > > > >> > > > >> >> > >> >> > wrote:
> > > > > > >> > > > >> >> > >> >> >
> > > > > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM,
> > Harsha <
> > > > > > >> > > > kafka@harsha.io>
> > > > > > >> > > > >> >> > wrote:
> > > > > > >> > > > >> >> > >> >> > >
> > > > > > >> > > > >> >> > >> >> > > > Unifying the two discussion threads
> on
> > > > this
> > > > > > KIP.
> > > > > > >> > > > >> >> > >> >> > > >
> > > > > > >> > > > >> >> > >> >> > > > Here is the response from Jitendra
> > > > > > >> > > > >> >> > >> >> > > >
> > > > > > >> > > > >> >> > >> >> > > > "The need for a large number of
> > clients
> > > > that
> > > > > > are
> > > > > > >> > > > running
> > > > > > >> > > > >> all
> > > > > > >> > > > >> >> > over the
> > > > > > >> > > > >> >> > >> >> > > > cluster that authenticate with Kafka
> > > > > brokers,
> > > > > > >> is very
> > > > > > >> > > > >> similar
> > > > > > >> > > > >> >> > to the
> > > > > > >> > > > >> >> > >> >> > > > Hadoop use case of large number of
> > tasks
> > > > > > running
> > > > > > >> > > across
> > > > > > >> > > > >> the
> > > > > > >> > > > >> >> > cluster
> > > > > > >> > > > >> >> > >> >> that
> > > > > > >> > > > >> >> > >> >> > > > need authentication to Hdfs
> Namenode.
> > > > > > >> Therefore, the
> > > > > > >> > > > >> >> > delegation token
> > > > > > >> > > > >> >> > >> >> > > > approach does seem like a good fit
> for
> > > > this
> > > > > > use
> > > > > > >> case
> > > > > > >> > > > as we
> > > > > > >> > > > >> >> > have seen
> > > > > > >> > > > >> >> > >> >> it
> > > > > > >> > > > >> >> > >> >> > > > working at large scale in HDFS and
> > YARN.
> > > > > > >> > > > >> >> > >> >> > > >
> > > > > > >> > > > >> >> > >> >> > > >   The proposed design is very much
> > > inline
> > > > > with
> > > > > > >> Hadoop
> > > > > > >> > > > >> >> > approach. A few
> > > > > > >> > > > >> >> > >> >> > > >   comments:
> > > > > > >> > > > >> >> > >> >> > > >
> > > > > > >> > > > >> >> > >> >> > > > 1) Why do you guys want to allow
> > > infinite
> > > > > > >> renewable
> > > > > > >> > > > >> lifetime
> > > > > > >> > > > >> >> > for a
> > > > > > >> > > > >> >> > >> >> > > > token? HDFS restricts a token to a
> max
> > > > life
> > > > > > time
> > > > > > >> > > > (default
> > > > > > >> > > > >> 7
> > > > > > >> > > > >> >> > days).  A
> > > > > > >> > > > >> >> > >> >> > > > token's vulnerability is believed to
> > > > > increase
> > > > > > >> with
> > > > > > >> > > > time.
> > > > > > >> > > > >> >> > >> >> > > >
> > > > > > >> > > > >> >> > >> >> > > I agree that having infinite lifetime
> > > might
> > > > > not
> > > > > > >> be the
> > > > > > >> > > > best
> > > > > > >> > > > >> idea.
> > > > > > >> > > > >> >> > >> >> > >
> > > > > > >> > > > >> >> > >> >> > > >
> > > > > > >> > > > >> >> > >> >> > > > 2) As I understand the tokens are
> > stored
> > > > in
> > > > > > >> zookeeper
> > > > > > >> > > > as
> > > > > > >> > > > >> well,
> > > > > > >> > > > >> >> > and
> > > > > > >> > > > >> >> > >> >> can
> > > > > > >> > > > >> >> > >> >> > > > be updated there. This is clever as
> it
> > > can
> > > > > > allow
> > > > > > >> > > > >> replacing the
> > > > > > >> > > > >> >> > tokens
> > > > > > >> > > > >> >> > >> >> > > > once they run out of max life time,
> > and
> > > > > > clients
> > > > > > >> can
> > > > > > >> > > > >> download
> > > > > > >> > > > >> >> > new
> > > > > > >> > > > >> >> > >> >> tokens
> > > > > > >> > > > >> >> > >> >> > > > from zookeeper. It shouldn't be a
> big
> > > load
> > > > > on
> > > > > > >> > > zookeeper
> > > > > > >> > > > >> as a
> > > > > > >> > > > >> >> > client
> > > > > > >> > > > >> >> > >> >> will
> > > > > > >> > > > >> >> > >> >> > > > need to get a new token once in
> > several
> > > > > days.
> > > > > > >> In this
> > > > > > >> > > > >> approach
> > > > > > >> > > > >> >> > you
> > > > > > >> > > > >> >> > >> >> don't
> > > > > > >> > > > >> >> > >> >> > > > need infinite lifetime on the token
> > even
> > > > for
> > > > > > >> long
> > > > > > >> > > > running
> > > > > > >> > > > >> >> > clients.
> > > > > > >> > > > >> >> > >> >> > > >
> > > > > > >> > > > >> >> > >> >> > > > 3) The token password are generated
> > > using
> > > > a
> > > > > > >> master
> > > > > > >> > > key.
> > > > > > >> > > > >> The
> > > > > > >> > > > >> >> > master
> > > > > > >> > > > >> >> > >> >> key
> > > > > > >> > > > >> >> > >> >> > > > should also be periodically changed.
> > In
> > > > > > Hadoop,
> > > > > > >> the
> > > > > > >> > > > >> default
> > > > > > >> > > > >> >> > renewal
> > > > > > >> > > > >> >> > >> >> > > > period is 1 day.?
> > > > > > >> > > > >> >> > >> >> > > >
> > > > > > >> > > > >> >> > >> >> > > IIUC, this will require brokers
> > > maintaining
> > > > a
> > > > > > >> list of X
> > > > > > >> > > > most
> > > > > > >> > > > >> >> > recent
> > > > > > >> > > > >> >> > >> >> master
> > > > > > >> > > > >> >> > >> >> > > keys. This list will have to be
> > persisted
> > > > > > >> somewhere, as
> > > > > > >> > > > if a
> > > > > > >> > > > >> >> > broker
> > > > > > >> > > > >> >> > >> >> goes
> > > > > > >> > > > >> >> > >> >> > > down it will have to get that list
> again
> > > and
> > > > > > >> storing
> > > > > > >> > > > master
> > > > > > >> > > > >> keys
> > > > > > >> > > > >> >> > on ZK
> > > > > > >> > > > >> >> > >> >> is
> > > > > > >> > > > >> >> > >> >> > > not the best idea. However, if a
> broker
> > > goes
> > > > > > down
> > > > > > >> then
> > > > > > >> > > we
> > > > > > >> > > > >> have
> > > > > > >> > > > >> >> > much
> > > > > > >> > > > >> >> > >> >> bigger
> > > > > > >> > > > >> >> > >> >> > > issue to deal with and client can
> always
> > > > > > >> > > re-authenticate
> > > > > > >> > > > is
> > > > > > >> > > > >> such
> > > > > > >> > > > >> >> > >> >> events.
> > > > > > >> > > > >> >> > >> >> > >
> > > > > > >> > > > >> >> > >> >> > > Did you happen to take a look at other
> > > > > > >> alternatives
> > > > > > >> > > this
> > > > > > >> > > > >> list has
> > > > > > >> > > > >> >> > >> >> > > suggested?
> > > > > > >> > > > >> >> > >> >> > >
> > > > > > >> > > > >> >> > >> >> > > >
> > > > > > >> > > > >> >> > >> >> > > > Thanks for a thorough proposal,
> great
> > > > work!"
> > > > > > >> > > > >> >> > >> >> > > >
> > > > > > >> > > > >> >> > >> >> > > >
> > > > > > >> > > > >> >> > >> >> > > >
> > > > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM,
> Gwen
> > > > > Shapira
> > > > > > >> wrote:
> > > > > > >> > > > >> >> > >> >> > > > > Makes sense to me. Thanks!
> > > > > > >> > > > >> >> > >> >> > > > >
> > > > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM,
> > > Harsha <
> > > > > > >> > > > kafka@harsha.io
> > > > > > >> > > > >> >
> > > > > > >> > > > >> >> > wrote:
> > > > > > >> > > > >> >> > >> >> > > > > > It doesn't need any release
> > vehicle
> > > > but
> > > > > > >> still the
> > > > > > >> > > > >> work can
> > > > > > >> > > > >> >> > move
> > > > > > >> > > > >> >> > >> >> > > > forward.
> > > > > > >> > > > >> >> > >> >> > > > > > If anyone is interested in the
> KIP
> > > > > please
> > > > > > >> do the
> > > > > > >> > > > >> review and
> > > > > > >> > > > >> >> > >> >> provide
> > > > > > >> > > > >> >> > >> >> > > the
> > > > > > >> > > > >> >> > >> >> > > > > > comments.
> > > > > > >> > > > >> >> > >> >> > > > > >
> > > > > > >> > > > >> >> > >> >> > > > > > -Harsha
> > > > > > >> > > > >> >> > >> >> > > > > >
> > > > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59
> PM,
> > > > Ismael
> > > > > > >> Juma
> > > > > > >> > > > wrote:
> > > > > > >> > > > >> >> > >> >> > > > > >> I agree that it would be good
> to
> > > have
> > > > > > more
> > > > > > >> time
> > > > > > >> > > to
> > > > > > >> > > > >> review
> > > > > > >> > > > >> >> > and
> > > > > > >> > > > >> >> > >> >> > > discuss
> > > > > > >> > > > >> >> > >> >> > > > > >> KIP-48.
> > > > > > >> > > > >> >> > >> >> > > > > >>
> > > > > > >> > > > >> >> > >> >> > > > > >> Ismael
> > > > > > >> > > > >> >> > >> >> > > > > >>
> > > > > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55
> AM,
> > > Gwen
> > > > > > >> Shapira <
> > > > > > >> > > > >> >> > >> >> gwen@confluent.io>
> > > > > > >> > > > >> >> > >> >> > > > wrote:
> > > > > > >> > > > >> >> > >> >> > > > > >>
> > > > > > >> > > > >> >> > >> >> > > > > >> > Hi Team,
> > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > >> > > > >> >> > >> >> > > > > >> > Since KIP-48 depends on
> KIP-43,
> > > > which
> > > > > > is
> > > > > > >> > > > already a
> > > > > > >> > > > >> bit
> > > > > > >> > > > >> >> > of a
> > > > > > >> > > > >> >> > >> >> risk
> > > > > > >> > > > >> >> > >> >> > > for
> > > > > > >> > > > >> >> > >> >> > > > > >> > the next release - any chance
> > we
> > > > can
> > > > > > >> delay
> > > > > > >> > > > >> delegation
> > > > > > >> > > > >> >> > tokens
> > > > > > >> > > > >> >> > >> >> to
> > > > > > >> > > > >> >> > >> >> > > > Kafka
> > > > > > >> > > > >> >> > >> >> > > > > >> > 0.10.1?
> > > > > > >> > > > >> >> > >> >> > > > > >> > With the community working
> on a
> > > > > release
> > > > > > >> every
> > > > > > >> > > 3
> > > > > > >> > > > >> month,
> > > > > > >> > > > >> >> > this
> > > > > > >> > > > >> >> > >> >> is not
> > > > > > >> > > > >> >> > >> >> > > > a huge
> > > > > > >> > > > >> >> > >> >> > > > > >> > delay.
> > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > >> > > > >> >> > >> >> > > > > >> > Gwen
> > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11
> > PM,
> > > > > Ashish
> > > > > > >> Singh
> > > > > > >> > > <
> > > > > > >> > > > >> >> > >> >> > > asingh@cloudera.com>
> > > > > > >> > > > >> >> > >> >> > > > wrote:
> > > > > > >> > > > >> >> > >> >> > > > > >> > > Parth,
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > > Thanks again for the
> awesome
> > > > write
> > > > > > up.
> > > > > > >> > > > Following
> > > > > > >> > > > >> our
> > > > > > >> > > > >> >> > >> >> discussion
> > > > > > >> > > > >> >> > >> >> > > > from the
> > > > > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I think it will be
> > easier
> > > > to
> > > > > > >> compare
> > > > > > >> > > > >> various
> > > > > > >> > > > >> >> > >> >> alternatives
> > > > > > >> > > > >> >> > >> >> > > > if they
> > > > > > >> > > > >> >> > >> >> > > > > >> > are
> > > > > > >> > > > >> >> > >> >> > > > > >> > > listed together. I am
> stating
> > > > > below a
> > > > > > >> few
> > > > > > >> > > > >> >> > alternatives along
> > > > > > >> > > > >> >> > >> >> > > with
> > > > > > >> > > > >> >> > >> >> > > > a the
> > > > > > >> > > > >> >> > >> >> > > > > >> > > current proposal.
> > > > > > >> > > > >> >> > >> >> > > > > >> > > (Current proposal) Store
> > > > Delegation
> > > > > > >> Token,
> > > > > > >> > > DT,
> > > > > > >> > > > >> on ZK.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates
> > > with a
> > > > > > >> broker.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > > > > authenticated,
> > > > > > >> it
> > > > > > >> > > will
> > > > > > >> > > > >> make a
> > > > > > >> > > > >> >> > broker
> > > > > > >> > > > >> >> > >> >> side
> > > > > > >> > > > >> >> > >> >> > > > call to
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation
> token.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker generates
> a
> > > > shared
> > > > > > >> secret
> > > > > > >> > > > based
> > > > > > >> > > > >> on
> > > > > > >> > > > >> >> > >> >> > > HMAC-SHA256(a
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    Password/Secret shared
> > > between
> > > > > all
> > > > > > >> > > brokers,
> > > > > > >> > > > >> >> > randomly
> > > > > > >> > > > >> >> > >> >> > > generated
> > > > > > >> > > > >> >> > >> >> > > > > >> > tokenId).
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Broker stores this
> > token
> > > in
> > > > > its
> > > > > > >> in
> > > > > > >> > > > memory
> > > > > > >> > > > >> cache.
> > > > > > >> > > > >> >> > >> >> Broker
> > > > > > >> > > > >> >> > >> >> > > > also stores
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    the DelegationToken
> > without
> > > > the
> > > > > > >> hmac in
> > > > > > >> > > the
> > > > > > >> > > > >> >> > zookeeper.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. All brokers will
> have a
> > > > cache
> > > > > > >> backed
> > > > > > >> > > by
> > > > > > >> > > > >> >> > zookeeper so
> > > > > > >> > > > >> >> > >> >> they
> > > > > > >> > > > >> >> > >> >> > > > will all
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    get notified whenever a
> > new
> > > > > token
> > > > > > is
> > > > > > >> > > > >> generated and
> > > > > > >> > > > >> >> > they
> > > > > > >> > > > >> >> > >> >> will
> > > > > > >> > > > >> >> > >> >> > > > update
> > > > > > >> > > > >> >> > >> >> > > > > >> > their
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    local cache whenever
> token
> > > > state
> > > > > > >> changes.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Broker returns the
> > token
> > > to
> > > > > > >> Client.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Probable race
> > condition,
> > > > > client
> > > > > > >> tries
> > > > > > >> > > to
> > > > > > >> > > > >> >> > authenticate
> > > > > > >> > > > >> >> > >> >> with
> > > > > > >> > > > >> >> > >> >> > > > a broker
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    that is yet to be
> updated
> > > with
> > > > > the
> > > > > > >> newly
> > > > > > >> > > > >> generated
> > > > > > >> > > > >> >> > DT.
> > > > > > >> > > > >> >> > >> >> This
> > > > > > >> > > > >> >> > >> >> > > can
> > > > > > >> > > > >> >> > >> >> > > > > >> > probably be
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    dealt with making
> > dtRequest
> > > > > block
> > > > > > >> until
> > > > > > >> > > all
> > > > > > >> > > > >> >> > brokers have
> > > > > > >> > > > >> >> > >> >> > > > updated
> > > > > > >> > > > >> >> > >> >> > > > > >> > their DT
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    cache. Zk barrier or
> > similar
> > > > > > >> mechanism
> > > > > > >> > > can
> > > > > > >> > > > be
> > > > > > >> > > > >> used.
> > > > > > >> > > > >> >> > >> >> However,
> > > > > > >> > > > >> >> > >> >> > > > all such
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    mechanisms will increase
> > > > > > complexity.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Using a static secret
> > key
> > > > > from
> > > > > > >> config
> > > > > > >> > > > >> file. Will
> > > > > > >> > > > >> >> > >> >> require
> > > > > > >> > > > >> >> > >> >> > > yet
> > > > > > >> > > > >> >> > >> >> > > > > >> > another
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    config and uses a static
> > > > secret
> > > > > > >> key. It
> > > > > > >> > > is
> > > > > > >> > > > >> advised
> > > > > > >> > > > >> >> > to
> > > > > > >> > > > >> >> > >> >> rotate
> > > > > > >> > > > >> >> > >> >> > > > secret
> > > > > > >> > > > >> >> > >> >> > > > > >> > keys
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    periodically. This can
> be
> > > > > avoided
> > > > > > >> with
> > > > > > >> > > > >> controller
> > > > > > >> > > > >> >> > >> >> generating
> > > > > > >> > > > >> >> > >> >> > > > > >> > secretKey and
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    passing to brokers
> > > > periodically.
> > > > > > >> However,
> > > > > > >> > > > >> this will
> > > > > > >> > > > >> >> > >> >> require
> > > > > > >> > > > >> >> > >> >> > > > brokers to
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    maintain certain counts
> of
> > > > > > >> secretKeys.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 1) Have
> > controller
> > > > > > >> generate
> > > > > > >> > > > >> delegation
> > > > > > >> > > > >> >> > token.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates
> > > with a
> > > > > > >> broker.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > > > > authenticated,
> > > > > > >> it
> > > > > > >> > > will
> > > > > > >> > > > >> make a
> > > > > > >> > > > >> >> > broker
> > > > > > >> > > > >> >> > >> >> side
> > > > > > >> > > > >> >> > >> >> > > > call to
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation
> token.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. Broker forwards the
> > > request
> > > > > to
> > > > > > >> > > > controller.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Controller generates
> a
> > DT
> > > > and
> > > > > > >> > > broadcasts
> > > > > > >> > > > >> to all
> > > > > > >> > > > >> >> > >> >> brokers.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    5. Broker stores this
> > token
> > > in
> > > > > its
> > > > > > >> memory
> > > > > > >> > > > >> cache.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Controller responds
> to
> > > > > broker’s
> > > > > > >> DT
> > > > > > >> > > req.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    7. Broker returns the
> > token
> > > to
> > > > > > >> Client.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. We will have to add
> new
> > > > APIs
> > > > > to
> > > > > > >> > > support
> > > > > > >> > > > >> >> > controller
> > > > > > >> > > > >> >> > >> >> pushing
> > > > > > >> > > > >> >> > >> >> > > > tokens
> > > > > > >> > > > >> >> > >> >> > > > > >> > to
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    brokers on top of the
> > > minimal
> > > > > APIs
> > > > > > >> that
> > > > > > >> > > are
> > > > > > >> > > > >> >> > currently
> > > > > > >> > > > >> >> > >> >> > > proposed.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. We will also have to
> > add
> > > > APIs
> > > > > > to
> > > > > > >> > > support
> > > > > > >> > > > >> the
> > > > > > >> > > > >> >> > >> >> bootstrapping
> > > > > > >> > > > >> >> > >> >> > > > case,
> > > > > > >> > > > >> >> > >> >> > > > > >> > i.e,
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    when a new broker comes
> up
> > > it
> > > > > will
> > > > > > >> have
> > > > > > >> > > to
> > > > > > >> > > > >> get all
> > > > > > >> > > > >> >> > >> >> delegation
> > > > > > >> > > > >> >> > >> >> > > > tokens
> > > > > > >> > > > >> >> > >> >> > > > > >> > from
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    the controller.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. In catastrophic
> > failures
> > > > > where
> > > > > > >> all
> > > > > > >> > > > brokers
> > > > > > >> > > > >> go
> > > > > > >> > > > >> >> > down,
> > > > > > >> > > > >> >> > >> >> the
> > > > > > >> > > > >> >> > >> >> > > > tokens will
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if servers
> > are
> > > > > > >> restarted as
> > > > > > >> > > > >> tokens
> > > > > > >> > > > >> >> > are not
> > > > > > >> > > > >> >> > >> >> > > > persisted
> > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this happens, then
> > there
> > > > are
> > > > > > more
> > > > > > >> > > > important
> > > > > > >> > > > >> >> > things to
> > > > > > >> > > > >> >> > >> >> > > worry
> > > > > > >> > > > >> >> > >> >> > > > about
> > > > > > >> > > > >> >> > >> >> > > > > >> > and
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
> > > > > > >> re-authenticate.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 2) Do not
> > > distribute
> > > > > DT
> > > > > > to
> > > > > > >> > > other
> > > > > > >> > > > >> brokers
> > > > > > >> > > > >> >> > at
> > > > > > >> > > > >> >> > >> >> all.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates
> > > with a
> > > > > > >> broker.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > > > > authenticated,
> > > > > > >> it
> > > > > > >> > > will
> > > > > > >> > > > >> make a
> > > > > > >> > > > >> >> > broker
> > > > > > >> > > > >> >> > >> >> side
> > > > > > >> > > > >> >> > >> >> > > > call to
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation
> token.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker generates
> DT
> > > of
> > > > > > form,
> > > > > > >> > > [hmac +
> > > > > > >> > > > >> (owner,
> > > > > > >> > > > >> >> > >> >> renewer,
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    maxLifeTime, id, hmac,
> > > > > > >> expirationTime)]
> > > > > > >> > > and
> > > > > > >> > > > >> passes
> > > > > > >> > > > >> >> > back
> > > > > > >> > > > >> >> > >> >> this
> > > > > > >> > > > >> >> > >> >> > > > DT to
> > > > > > >> > > > >> >> > >> >> > > > > >> > client.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    hmac is generated via
> > > > > > >> {HMAC-SHA256(owner,
> > > > > > >> > > > >> renewer,
> > > > > > >> > > > >> >> > >> >> > > > maxLifeTime, id,
> > > > > > >> > > > >> >> > >> >> > > > > >> > hmac,
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    expirationTime) using
> > > > > SecretKey}.
> > > > > > >> Note
> > > > > > >> > > that
> > > > > > >> > > > >> all
> > > > > > >> > > > >> >> > brokers
> > > > > > >> > > > >> >> > >> >> have
> > > > > > >> > > > >> >> > >> >> > > > this
> > > > > > >> > > > >> >> > >> >> > > > > >> > SecretKey.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Client then goes to
> any
> > > > > broker
> > > > > > >> and to
> > > > > > >> > > > >> >> > authenticate
> > > > > > >> > > > >> >> > >> >> sends
> > > > > > >> > > > >> >> > >> >> > > > the DT.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    Broker recalculates hmac
> > > using
> > > > > > >> (owner,
> > > > > > >> > > > >> renewer,
> > > > > > >> > > > >> >> > >> >> maxLifeTime,
> > > > > > >> > > > >> >> > >> >> > > > id, hmac,
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    expirationTime) info
> from
> > DT
> > > > and
> > > > > > its
> > > > > > >> > > > >> SecretKey. If
> > > > > > >> > > > >> >> > it
> > > > > > >> > > > >> >> > >> >> matches
> > > > > > >> > > > >> >> > >> >> > > > with
> > > > > > >> > > > >> >> > >> >> > > > > >> > hmac of
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    DT, client is
> > authenticated.
> > > > > Yes,
> > > > > > >> it will
> > > > > > >> > > > do
> > > > > > >> > > > >> other
> > > > > > >> > > > >> >> > >> >> obvious
> > > > > > >> > > > >> >> > >> >> > > > checks of
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    timestamp expiry and
> such.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > > Note that secret key will
> be
> > > > > > generated
> > > > > > >> by
> > > > > > >> > > > >> controller
> > > > > > >> > > > >> >> > and
> > > > > > >> > > > >> >> > >> >> passed
> > > > > > >> > > > >> >> > >> >> > > to
> > > > > > >> > > > >> >> > >> >> > > > > >> > brokers
> > > > > > >> > > > >> >> > >> >> > > > > >> > > periodically.
> > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    1. How to delete a DT?
> > Yes,
> > > > that
> > > > > > is
> > > > > > >> a
> > > > > > >> > > > downside
> > > > > > >> > > > >> >> > here.
> > > > > > >> > > > >> >> > >> >> However,
> > > > > > >> > > > >> >> > >> >> > > > this can
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    be handled with brokers
> > > > > > maintaining
> > > > > > >> a
> > > > > > >> > > > >> blacklist of
> > > > > > >> > > > >> >> > DTs,
> > > > > > >> > > > >> >> > >> >> DTs
> > > > > > >> > > > >> >> > >> >> > > > from this
> > > > > > >> > > > >> >> > >> >> > > > > >> > list
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    can be removed after
> > expiry.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    2. In catastrophic
> > failures
> > > > > where
> > > > > > >> all
> > > > > > >> > > > brokers
> > > > > > >> > > > >> go
> > > > > > >> > > > >> >> > down,
> > > > > > >> > > > >> >> > >> >> the
> > > > > > >> > > > >> >> > >> >> > > > tokens will
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if servers
> > are
> > > > > > >> restarted as
> > > > > > >> > > > >> tokens
> > > > > > >> > > > >> >> > are not
> > > > > > >> > > > >> >> > >> >> > > > persisted
> > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    If this happens, then
> > there
> > > > are
> > > > > > more
> > > > > > >> > > > important
> > > > > > >> > > > >> >> > things to
> > > > > > >> > > > >> >> > >> >> > > worry
> > > > > > >> > > > >> >> > >> >> > > > about
> > > > > > >> > > > >> >> > >> >> > > > > >> > and
> > > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
> > > > > > >> re-authenticate.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at
> 1:58
> > > PM,
> > > > > > Parth
> > > > > > >> > > > >> Brahmbhatt <
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> pbrahmbhatt@hortonworks.com>
> > > > > wrote:
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > >> Hi,
> > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > >> > > > >> >> > >> >> > > > > >> > >> I have filed KIP-48 so we
> > can
> > > > > offer
> > > > > > >> hadoop
> > > > > > >> > > > like
> > > > > > >> > > > >> >> > delegation
> > > > > > >> > > > >> >> > >> >> > > > tokens in
> > > > > > >> > > > >> >> > >> >> > > > > >> > >> kafka. You can review the
> > > design
> > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > >> > > > >> >> > >> >> > > >
> > > > > > >> > > > >> >> > >> >> > >
> > > > > > >> > > > >> >> > >> >>
> > > > > > >> > > > >> >> >
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > > > > > >> > > > >> >> > >> >> > > > > >> > .
> > > > > > >> > > > >> >> > >> >> > > > > >> > >> This KIP depends on KIP-43
> > and
> > > > we
> > > > > > >> have also
> > > > > > >> > > > >> >> > discussed an
> > > > > > >> > > > >> >> > >> >> > > > alternative to
> > > > > > >> > > > >> >> > >> >> > > > > >> > >> proposed design here<
> > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > >> > > > >> >> > >> >> > > >
> > > > > > >> > > > >> >> > >> >> > >
> > > > > > >> > > > >> >> > >> >>
> > > > > > >> > > > >> >> >
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > > > > > >> > > > >> >> > >> >> > > > > >> > >> >.
> > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > >> > > > >> >> > >> >> > > > > >> > >> Thanks
> > > > > > >> > > > >> >> > >> >> > > > > >> > >> Parth
> > > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > > --
> > > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > > >> > > > >> >> > >> >> > > > > >> > > Regards,
> > > > > > >> > > > >> >> > >> >> > > > > >> > > Ashish
> > > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > > >> > > > >> >> > >> >> > > >
> > > > > > >> > > > >> >> > >> >> > >
> > > > > > >> > > > >> >> > >> >> > >
> > > > > > >> > > > >> >> > >> >> > >
> > > > > > >> > > > >> >> > >> >> > > --
> > > > > > >> > > > >> >> > >> >> > >
> > > > > > >> > > > >> >> > >> >> > > Regards,
> > > > > > >> > > > >> >> > >> >> > > Ashish
> > > > > > >> > > > >> >> > >> >> > >
> > > > > > >> > > > >> >> > >> >>
> > > > > > >> > > > >> >> >
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Rajini
> > >
> >
>
>
>
> --
> Regards,
>
> Rajini
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Rajini Sivaram <ra...@googlemail.com>]

Ismael,

I have created a JIRA (https://issues.apache.org/jira/browse/KAFKA-3751)
for adding SCRAM as a SASL mechanism. Would that need another KIP? If
KIP-48 will use this mechanism, can this just be a JIRA that gets reviewed
when the PR is ready?

Thank you,

Rajini

On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <is...@juma.me.uk> wrote:

> Thanks Rajini, SCRAM seems like a good candidate.
>
> Gwen had independently mentioned this as a SASL mechanism that might be
> useful for Kafka and I have been meaning to check it in more detail. Good
> to know that you are willing to contribute an implementation. Maybe we
> should file a separate JIRA for this?
>
> Ismael
>
> On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram <
> rajinisivaram@googlemail.com> wrote:
>
> > SCRAM (Salted Challenge Response Authentication Mechanism) is a better
> > mechanism than Digest-MD5. Java doesn't come with a built-in SCRAM
> > SaslServer or SaslClient, but I will be happy to add support in Kafka
> since
> > it would be a useful mechanism to support anyway.
> > https://tools.ietf.org/html/rfc7677 describes the protocol for
> > SCRAM-SHA-256.
> >
> > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Parth,
> > >
> > > Thanks for the explanation. A couple of more questions.
> > >
> > > 110. What does getDelegationTokenAs mean?
> > >
> > > 111. What's the typical rate of getting and renewing delegation tokens?
> > > That may have an impact on whether they should be directed to the
> > > controller.
> > >
> > > Jun
> > >
> > > On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt <
> > > brahmbhatt.parth@gmail.com> wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > Thanks for reviewing.
> > > >
> > > > * We could add a Cluster action to add acls on who can request
> > delegation
> > > > tokens. I don't see the use case for that yet but down the line when
> we
> > > > start supporting getDelegationTokenAs it will be necessary.
> > > > * Yes we recommend tokens to be only used/distributed over secure
> > > channels.
> > > > * Depending on what design we end up choosing Invalidation will be
> > > > responsibility of every broker or controller.
> > > > * I am not sure if I documented somewhere that invalidation will
> > directly
> > > > go through zookeeper but that is not the intent. Invalidation will
> > either
> > > > be request based or due to expiration. No direct zookeeper
> interaction
> > > from
> > > > any client.
> > > > * "Broker also stores the DelegationToken without the hmac in the
> > > > zookeeper." : Sorry about the confusion. The sole purpose of
> zookeeper
> > in
> > > > this design is as distribution channel for tokens between all brokers
> > > and a
> > > > layer that ensures only tokens that were generated by making a
> request
> > > to a
> > > > broker will be accepted (more on this in second paragraph). The token
> > > > consists of few elements (owner, renewer, uuid , expiration, hmac) ,
> > one
> > > of
> > > > which is the finally generated hmac but hmac it self is derivable if
> > you
> > > > have all the other elements of the token + secret key to generate
> hmac.
> > > > Given zookeeper does not provide SSL support we do not want the
> entire
> > > > token to be wire transferred to zookeeper as that will be an insecure
> > > wire
> > > > transfer. Instead we only store all the other elements of a
> delegation
> > > > tokens. Brokers can read these elements and because they also have
> > access
> > > > to secret key they will be able to generate hmac on their end.
> > > >
> > > > One of the alternative proposed is to avoid zookeeper altogether. A
> > > Client
> > > > will call broker with required information (owner, renwer,
> expiration)
> > > and
> > > > get back (signed hmac, uuid). Broker won't store this in zookeeper.
> > From
> > > > this point a client can contact any broker with all the delegation
> > token
> > > > info (owner, rewner, expiration, hmac, uuid) the borker will
> regenerate
> > > the
> > > > hmac and as long as it matches with hmac presented by client , broker
> > > will
> > > > allow the request to authenticate.  Only problem with this approach
> is
> > if
> > > > the secret key is compromised any client can now generate random
> tokens
> > > and
> > > > they will still be able to authenticate as any user they like. with
> > > > zookeeper we guarantee that only tokens acquired via a broker (using
> > some
> > > > auth scheme other than delegation token) will be accepted. We need to
> > > > discuss which proposal makes more sense and we can go over it in
> > > tomorrow's
> > > > meeting.
> > > >
> > > > Also, can you forward the invite to me?
> > > >
> > > > Thanks
> > > > Parth
> > > >
> > > >
> > > >
> > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <ju...@confluent.io> wrote:
> > > >
> > > > > Thanks for the KIP. A few comments.
> > > > >
> > > > > 100. This potentially can be useful for Kafka Connect and Kafka
> rest
> > > > proxy
> > > > > where a worker agent will need to run a task on behalf of a client.
> > We
> > > > will
> > > > > likely need to change how those services use Kafka clients
> > > > > (producer/consumer). Instead of a shared client per worker, we will
> > > need
> > > > a
> > > > > client per user task since the authentication happens at the
> > connection
> > > > > level. For Kafka Connect, the renewer will be the workers. So, we
> > > > probably
> > > > > need to allow multiple renewers. For Kafka rest proxy, the renewer
> > can
> > > > > probably just be the creator of the token.
> > > > >
> > > > > 101. Do we need new acl on who can request delegation tokens?
> > > > >
> > > > > 102. Do we recommend people to send delegation tokens in an
> encrypted
> > > > > channel?
> > > > >
> > > > > 103. Who is responsible for expiring tokens, every broker?
> > > > >
> > > > > 104. For invalidating tokens, would it be better to do it in a
> > request
> > > > > instead of going to ZK directly?
> > > > >
> > > > > 105. The terminology of client in the wiki sometimes refers to the
> > end
> > > > > client and some other times refers to the client using the
> delegation
> > > > > tokens. It would be useful to distinguish between the two.
> > > > >
> > > > > 106. Could you explain the sentence "Broker also stores the
> > > > DelegationToken
> > > > > without the hmac in the zookeeper." a bit more? I thought the
> > > delegation
> > > > > token is the hmac.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <ju...@confluent.io> wrote:
> > > > >
> > > > > > Hi, Harsha,
> > > > > >
> > > > > > Just sent out a KIP meeting invite. We can discuss this in the
> > > meeting
> > > > > > tomorrow.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <ka...@harsha.io> wrote:
> > > > > >
> > > > > >> Hi All,
> > > > > >>            Can we have a KIP meeting around this. The KIP is up
> > for
> > > > > >>            sometime and if there are any questions lets quickly
> > hash
> > > > out
> > > > > >>            details.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Harsha
> > > > > >>
> > > > > >> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt wrote:
> > > > > >> > That is what the hadoop echo system uses so no good reason
> > really.
> > > > We
> > > > > >> > could
> > > > > >> > change it to whatever is the newest recommended standard is.
> > > > > >> >
> > > > > >> > Thanks
> > > > > >> > Parth
> > > > > >> >
> > > > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <
> ismael@juma.me.uk
> > >
> > > > > wrote:
> > > > > >> >
> > > > > >> > > Hi Parth,
> > > > > >> > >
> > > > > >> > > Thanks for the KIP. I only started reviewing this and may
> have
> > > > > >> additional
> > > > > >> > > questions later. The immediate question that came to mind is
> > our
> > > > > >> choice of
> > > > > >> > > "DIGEST-MD5" even though it's marked as OBSOLETE in the IANA
> > > > > Registry
> > > > > >> of
> > > > > >> > > SASL mechanisms and the original RFC (2831) has been moved
> to
> > > > > Historic
> > > > > >> > > status:
> > > > > >> > >
> > > > > >> > > https://tools.ietf.org/html/rfc6331
> > > > > >> > >
> > > > >
> > http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> > > > > >> > >
> > > > > >> > > What is the reasoning behind that choice?
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > Ismael
> > > > > >> > >
> > > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <
> > > gwen@confluent.io
> > > > >
> > > > > >> wrote:
> > > > > >> > >
> > > > > >> > > > Also comments inline :)
> > > > > >> > > >
> > > > > >> > > > > * I want to emphasize that even though delegation tokens
> > > are a
> > > > > >> Hadoop
> > > > > >> > > > > innovation, I feel very strongly about not adding
> > dependency
> > > > on
> > > > > >> Hadoop
> > > > > >> > > > > when implementing delegation tokens for Kafka. The KIP
> > > doesn't
> > > > > >> imply
> > > > > >> > > > > such dependency, but if you can clarify...
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > *No hadoop dependency.*
> > > > > >> > > >
> > > > > >> > > > Yay! Just add this to the KIP so no one will read the KIP
> > and
> > > > > panic
> > > > > >> > > > three weeks before the next release...
> > > > > >> > > >
> > > > > >> > > > > * Can we get delegation token at any time after
> > > > authenticating?
> > > > > >> only
> > > > > >> > > > > immediately after?
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > *As long as you are authenticated you can get delegation
> > > > tokens.
> > > > > >> We
> > > > > >> > > need
> > > > > >> > > > to
> > > > > >> > > > > discuss if a client authenticated using delegation
> token,
> > > can
> > > > > also
> > > > > >> > > > acquire
> > > > > >> > > > > delegation token again or not. Also there is the
> question
> > of
> > > > do
> > > > > we
> > > > > >> > > allow
> > > > > >> > > > > anyone to acquire delegation token or we want specific
> > ACLs
> > > (I
> > > > > >> think
> > > > > >> > > its
> > > > > >> > > > an
> > > > > >> > > > > overkill.)*
> > > > > >> > > >
> > > > > >> > > > I agree that ACLs is an overkill.
> > > > > >> > > >
> > > > > >> > > > I think we are debating two options: Either require
> Kerberos
> > > > auth
> > > > > >> for
> > > > > >> > > > renewal or require non-owners to renew.
> > > > > >> > > > I *think* the latter is simpler (it basically require a
> "job
> > > > > master"
> > > > > >> > > > to take responsibility for the renewal, it will have its
> own
> > > > > >> identity
> > > > > >> > > > anyway and I think this is the correct design pattern
> > anyway.
> > > > For
> > > > > >> > > > storm, I'd expect Nimbus to coordinate renewals?), but it
> is
> > > > hard
> > > > > to
> > > > > >> > > > debate simplicity without looking at the code changes
> > > required.
> > > > If
> > > > > >> you
> > > > > >> > > > have a draft of how the "require Kerberos" will look in
> > Kafka
> > > > > code,
> > > > > >> > > > I'll be happy to take a look.
> > > > > >> > > >
> > > > > >> > > > > * My understanding is that tokens will propagate via ZK
> > but
> > > > > >> without
> > > > > >> > > > > additional changes to UpdateMetadata protocol, correct?
> > > > Clients
> > > > > >> > > > > currently don't retry on SASL auth failure (IIRC), but
> > since
> > > > the
> > > > > >> > > > > tokens propagate between brokers asynch, we will need to
> > > > retry a
> > > > > >> bit
> > > > > >> > > > > to avoid clients failing auth due to timing issues.
> > > > > >> > > > >
> > > > > >> > > > > *I am considering 2 alternatives right now. The current
> > > > > documented
> > > > > >> > > > approach
> > > > > >> > > > > is zookeeper based and it does not require any changes
> to
> > > > > >> > > UpdateMetadata
> > > > > >> > > > > protocol. An alternative approach can remove zookeeper
> > > > > dependency
> > > > > >> as
> > > > > >> > > well
> > > > > >> > > > > but we can discuss that in KIP discussion call.*
> > > > > >> > > >
> > > > > >> > > > Oooh! Sounds interesting. Do you want to ping Jun to
> > arrange a
> > > > > call?
> > > > > >> > > >
> > > > > >> > > > > * I liked Ashish's suggestion of having just the
> > controller
> > > > > issue
> > > > > >> the
> > > > > >> > > > > delegation tokens, to avoid syncing a shared secret. Not
> > > sure
> > > > if
> > > > > >> we
> > > > > >> > > > > want to continue the discussion here or on the wiki. I
> > think
> > > > > that
> > > > > >> we
> > > > > >> > > > > can decouple the problem of "token distribution" from
> > > "shared
> > > > > >> secret
> > > > > >> > > > > distribution" and use the controller as the only token
> > > > generator
> > > > > >> to
> > > > > >> > > > > solve the second issue, while still using ZK async to
> > > > distribute
> > > > > >> > > > > tokens.
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > *As mentioned in the previous Email I am fine with that
> > > > approach
> > > > > >> as
> > > > > >> > > long
> > > > > >> > > > as
> > > > > >> > > > > we agree that the extra complexity of adding/updating
> APIS
> > > > adds
> > > > > >> enough
> > > > > >> > > > > value. The advantage with the controller approach is
> > secret
> > > > > >> rotation
> > > > > >> > > can
> > > > > >> > > > be
> > > > > >> > > > > automated,frequent and would not require deployment. *
> > > > > >> > > >
> > > > > >> > > > Can you detail the extra complexity (or point me to the
> > email
> > > I
> > > > > >> > > > missed?) - which Apis are required?
> > > > > >> > > > As far as I can tell, clients can already find the
> > controller
> > > > from
> > > > > >> > > > metadata. I'm a bit more concerned about controller load.
> > > > > >> > > >
> > > > > >> > > > >
> > > > > >> > > > > * While I like the idea of forcing kerberos auth for
> > > renwal, I
> > > > > >> think
> > > > > >> > > > > it mixes the transport layer the the request content in
> a
> > > > pretty
> > > > > >> ugly
> > > > > >> > > > > way. Perhaps limiting renewer to non-owner is better.
> > > > > >> > > > >
> > > > > >> > > > > *I feel this is a necessary evil. While this will make
> the
> > > > kafka
> > > > > >> code
> > > > > >> > > > > pretty straight forward , forcing  renewer to non-owner
> > > pushes
> > > > > >> the code
> > > > > >> > > > > ugliness to client and makes it even harder to
> > integrate.  *
> > > > > >> > > >
> > > > > >> > > > As mentioned before, I don't think the "renewal by other"
> > > > approach
> > > > > >> is
> > > > > >> > > > that ugly for the clients we expect to use delegation
> tokens
> > > > since
> > > > > >> > > > they will have an app-master of some sort who requested
> the
> > > > token
> > > > > to
> > > > > >> > > > begin with.
> > > > > >> > > >
> > > > > >> > > > >
> > > > > >> > > > > The response for my question on how multiple identities
> > will
> > > > be
> > > > > >> > > > > handled wasn't super clear to me - AFAIK, we don't
> > > > authenticate
> > > > > >> each
> > > > > >> > > > > request, we authenticate connections.
> > > > > >> > > > >
> > > > > >> > > > > *We authenticate connections, and only when they are
> being
> > > > > >> established.
> > > > > >> > > > Let
> > > > > >> > > > > me try to phrase this as a question, in absence of
> > > delegation
> > > > > >> tokens if
> > > > > >> > > > we
> > > > > >> > > > > had to support the use case using user TGT's how would
> we
> > do
> > > > it?
> > > > > >> My
> > > > > >> > > point
> > > > > >> > > > > was it would be no different with delegation tokens. The
> > use
> > > > > case
> > > > > >> you
> > > > > >> > > are
> > > > > >> > > > > describing seems more like impersonation.*
> > > > > >> > > >
> > > > > >> > > > Yeah, I thought that one of the things that delegation
> > tokens
> > > > > >> handled.
> > > > > >> > > > Maybe I got it wrong :)
> > > > > >> > > >
> > > > > >> > > > Thanks for the detailed answers.
> > > > > >> > > >
> > > > > >> > > > Gwen
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > > Thanks
> > > > > >> > > > > Parth
> > > > > >> > > > >
> > > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <
> > > > > gwen@confluent.io
> > > > > >> >
> > > > > >> > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > >> Hi Parth and Harsha,
> > > > > >> > > > >>
> > > > > >> > > > >> Few more comments:
> > > > > >> > > > >>
> > > > > >> > > > >> * The API / RequestResponse section doesn't seem to
> have
> > > good
> > > > > >> > > > >> description of the changes to the Kafka Protocol.
> Sounds
> > > like
> > > > > >> you are
> > > > > >> > > > >> proposing new DelegationTokenRequest and
> > RenewTokenRequest
> > > > (and
> > > > > >> > > > >> matching responses), without detailing the contents of
> > the
> > > > > >> requests
> > > > > >> > > > >> and responses? Or rather, you show the class interface,
> > but
> > > > not
> > > > > >> the
> > > > > >> > > > >> underlying protocol. This could be seen as an
> > > implementation
> > > > > >> detail,
> > > > > >> > > > >> but since the binary protocol is what we provide to
> > > non-Java
> > > > > >> clients,
> > > > > >> > > > >> we need to show the changes there.
> > > > > >> > > > >>
> > > > > >> > > > >> * getDelegationToken sounds like
> > > > delegationTokenRequestHandler?
> > > > > >> Is it
> > > > > >> > > > >> planned to be part of KafkaApi? or Client? Its
> unclear...
> > > > > >> > > > >>
> > > > > >> > > > >> * I want to emphasize that even though delegation
> tokens
> > > are
> > > > a
> > > > > >> Hadoop
> > > > > >> > > > >> innovation, I feel very strongly about not adding
> > > dependency
> > > > on
> > > > > >> Hadoop
> > > > > >> > > > >> when implementing delegation tokens for Kafka. The KIP
> > > > doesn't
> > > > > >> imply
> > > > > >> > > > >> such dependency, but if you can clarify...
> > > > > >> > > > >>
> > > > > >> > > > >> * Can we get delegation token at any time after
> > > > authenticating?
> > > > > >> only
> > > > > >> > > > >> immediately after?
> > > > > >> > > > >>
> > > > > >> > > > >> * My understanding is that tokens will propagate via ZK
> > but
> > > > > >> without
> > > > > >> > > > >> additional changes to UpdateMetadata protocol, correct?
> > > > Clients
> > > > > >> > > > >> currently don't retry on SASL auth failure (IIRC), but
> > > since
> > > > > the
> > > > > >> > > > >> tokens propagate between brokers asynch, we will need
> to
> > > > retry
> > > > > a
> > > > > >> bit
> > > > > >> > > > >> to avoid clients failing auth due to timing issues.
> > > > > >> > > > >>
> > > > > >> > > > >> * Strongly agreeing on clients not touching ZK directly
> > :)
> > > > > >> > > > >>
> > > > > >> > > > >> * I liked Ashish's suggestion of having just the
> > controller
> > > > > >> issue the
> > > > > >> > > > >> delegation tokens, to avoid syncing a shared secret.
> Not
> > > sure
> > > > > if
> > > > > >> we
> > > > > >> > > > >> want to continue the discussion here or on the wiki. I
> > > think
> > > > > >> that we
> > > > > >> > > > >> can decouple the problem of "token distribution" from
> > > "shared
> > > > > >> secret
> > > > > >> > > > >> distribution" and use the controller as the only token
> > > > > generator
> > > > > >> to
> > > > > >> > > > >> solve the second issue, while still using ZK async to
> > > > > distribute
> > > > > >> > > > >> tokens.
> > > > > >> > > > >>
> > > > > >> > > > >> * I am also uncomfortable with infinite lifetime of
> > tokens
> > > > (and
> > > > > >> hoped
> > > > > >> > > > >> to hear from others in the community) - but having the
> > > option
> > > > > and
> > > > > >> > > > >> documenting it as less secure, allows users to
> configure
> > > > their
> > > > > >> system
> > > > > >> > > > >> as the wish.
> > > > > >> > > > >>
> > > > > >> > > > >> * While I like the idea of forcing kerberos auth for
> > > renwal,
> > > > I
> > > > > >> think
> > > > > >> > > > >> it mixes the transport layer the the request content
> in a
> > > > > pretty
> > > > > >> ugly
> > > > > >> > > > >> way. Perhaps limiting renewer to non-owner is better.
> > > > > >> > > > >>
> > > > > >> > > > >> Things I'd still like to see:
> > > > > >> > > > >>
> > > > > >> > > > >> * More detailed explanation on what we plan to do for
> the
> > > > java
> > > > > >> clients
> > > > > >> > > > >> specifically - new configuration? new APIs?
> > > > > >> > > > >> The response for my question on how multiple identities
> > > will
> > > > be
> > > > > >> > > > >> handled wasn't super clear to me - AFAIK, we don't
> > > > authenticate
> > > > > >> each
> > > > > >> > > > >> request, we authenticate connections.
> > > > > >> > > > >>
> > > > > >> > > > >> * Alternatives: Delegation tokens are only used in the
> > > Hadoop
> > > > > >> > > > >> ecosystem. I'm wondering if there are alternatives in
> > other
> > > > > >> ecosystems
> > > > > >> > > > >> (Mesos? Tachyon? Cassandra?) and whether there are some
> > > > > >> advantages
> > > > > >> > > > >> there.
> > > > > >> > > > >>
> > > > > >> > > > >> Gwen
> > > > > >> > > > >>
> > > > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <
> kafka@harsha.io
> > >
> > > > > wrote:
> > > > > >> > > > >> > Hi Gwen,
> > > > > >> > > > >> >            Can you look at Parth's last reply. Does
> it
> > > > answer
> > > > > >> your
> > > > > >> > > > >> >            concerns.
> > > > > >> > > > >> >
> > > > > >> > > > >> > Thanks,
> > > > > >> > > > >> > Harsha
> > > > > >> > > > >> >
> > > > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt
> > wrote:
> > > > > >> > > > >> >> Thanks for reviewing Gwen. The wiki already has
> > details
> > > on
> > > > > >> token
> > > > > >> > > > >> >> expiration
> > > > > >> > > > >> >> under token acquisition process
> > > > > >> > > > >> >> <
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> > > > > >> > > > >> >.
> > > > > >> > > > >> >> Current proposal is that tokens will expire based
> on a
> > > > > server
> > > > > >> side
> > > > > >> > > > >> >> configuration (default 24 hours) unless renewed.
> > Renewal
> > > > is
> > > > > >> only
> > > > > >> > > > allowed
> > > > > >> > > > >> >> until the max life time of token. Alternatively we
> > could
> > > > > also
> > > > > >> make
> > > > > >> > > > that
> > > > > >> > > > >> >> an
> > > > > >> > > > >> >> optional param and the server side default can serve
> > as
> > > > the
> > > > > >> upper
> > > > > >> > > > bound.
> > > > > >> > > > >> >>
> > > > > >> > > > >> >> To your second point it will be done exactly the
> same
> > > way
> > > > we
> > > > > >> would
> > > > > >> > > > >> >> support
> > > > > >> > > > >> >> multiple keytabs. The calling client will have to
> put
> > > the
> > > > > >> tokens it
> > > > > >> > > > >> wants
> > > > > >> > > > >> >> to use in the subject instance and call
> > produce/consume
> > > > > inside
> > > > > >> > > > >> >> subject.doas. Each caller will have to keep track of
> > its
> > > > own
> > > > > >> > > > subject. I
> > > > > >> > > > >> >> will have to look at the code to see if we support
> > this
> > > > > >> feature
> > > > > >> > > right
> > > > > >> > > > >> now
> > > > > >> > > > >> >> but my understanding is delegation token shouldn't
> > need
> > > > any
> > > > > >> special
> > > > > >> > > > >> >> treatment as its just another type of Credential in
> > the
> > > > > >> subject.
> > > > > >> > > > >> >>
> > > > > >> > > > >> >> I would also like to know what is your opinion about
> > > > > infinite
> > > > > >> > > renewal
> > > > > >> > > > >> (my
> > > > > >> > > > >> >> recommendation is to not support this), tokens
> > renewing
> > > > them
> > > > > >> > > self(my
> > > > > >> > > > >> >> recommendation is to not support this) and most
> > > > importantly
> > > > > >> your
> > > > > >> > > > choice
> > > > > >> > > > >> >> between the alternatives listed on this thread
> > > > > >> > > > >> >> <
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >>
> > > > >
> > > >
> > >
> >
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> > > > > >> > > > >> >
> > > > > >> > > > >> >> ( I am leaning towards the alternative-2 minus
> > > controller
> > > > > >> > > > distributing
> > > > > >> > > > >> >> secret). Thanks again for reviewing.
> > > > > >> > > > >> >>
> > > > > >> > > > >> >> Thanks
> > > > > >> > > > >> >> Parth
> > > > > >> > > > >> >>
> > > > > >> > > > >> >>
> > > > > >> > > > >> >>
> > > > > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <
> > > > > >> gwen@confluent.io>
> > > > > >> > > > wrote:
> > > > > >> > > > >> >>
> > > > > >> > > > >> >> > Harsha,
> > > > > >> > > > >> >> >
> > > > > >> > > > >> >> > I was thinking of the Rest Proxy. I didn't see
> your
> > > > design
> > > > > >> yet,
> > > > > >> > > > but in
> > > > > >> > > > >> >> > our proxy, we have a set of producers, which will
> > > serve
> > > > > >> multiple
> > > > > >> > > > users
> > > > > >> > > > >> >> > going through the proxy. Since these users will
> have
> > > > > >> different
> > > > > >> > > > >> >> > privileges, they'll need to authenticate
> separately,
> > > and
> > > > > >> can't
> > > > > >> > > > share a
> > > > > >> > > > >> >> > token.
> > > > > >> > > > >> >> >
> > > > > >> > > > >> >> > Am I missing anything?
> > > > > >> > > > >> >> >
> > > > > >> > > > >> >> > Gwen
> > > > > >> > > > >> >> >
> > > > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <
> > > kafka@harsha.io
> > > > >
> > > > > >> wrote:
> > > > > >> > > > >> >> > > Gwen,
> > > > > >> > > > >> >> > >            On your second point. Can you
> describe
> > a
> > > > > >> usecase
> > > > > >> > > where
> > > > > >> > > > >> >> > >            mutliple clients ended up sharing a
> > > > producer
> > > > > >> and
> > > > > >> > > even
> > > > > >> > > > if
> > > > > >> > > > >> they
> > > > > >> > > > >> >> > >            do why can't they not use single
> token
> > > that
> > > > > >> producer
> > > > > >> > > > >> >> > >            captures. Why would we need multiple
> > > > clients
> > > > > >> with
> > > > > >> > > > >> different
> > > > > >> > > > >> >> > >            tokens sharing a single instance of
> > > > producer.
> > > > > >> Also
> > > > > >> > > in
> > > > > >> > > > >> this
> > > > > >> > > > >> >> > >            case other clients have access all
> the
> > > > tokens
> > > > > >> no?
> > > > > >> > > > >> >> > >
> > > > > >> > > > >> >> > > Thanks,
> > > > > >> > > > >> >> > > Harsha
> > > > > >> > > > >> >> > >
> > > > > >> > > > >> >> > >
> > > > > >> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira
> > > wrote:
> > > > > >> > > > >> >> > >> Sorry for the delay:
> > > > > >> > > > >> >> > >>
> > > > > >> > > > >> >> > >> Two questions that we didn't see in the wiki:
> > > > > >> > > > >> >> > >> 1. Is there an expiration for delegation
> tokens?
> > > > > >> Renewal? How
> > > > > >> > > > do we
> > > > > >> > > > >> >> > >> revoke them?
> > > > > >> > > > >> >> > >> 2. If we want to use delegation tokens for
> > "do-as"
> > > > > (say,
> > > > > >> > > submit
> > > > > >> > > > >> Storm
> > > > > >> > > > >> >> > >> job as my user), we will need a producer for
> > every
> > > > job
> > > > > >> (we
> > > > > >> > > can't
> > > > > >> > > > >> share
> > > > > >> > > > >> >> > >> them between multiple jobs running on same
> node),
> > > > since
> > > > > >> we
> > > > > >> > > only
> > > > > >> > > > >> >> > >> authenticate when connecting. Is there a plan
> to
> > > > change
> > > > > >> this
> > > > > >> > > for
> > > > > >> > > > >> >> > >> delegation tokens, in order to allow multiple
> > users
> > > > > with
> > > > > >> > > > different
> > > > > >> > > > >> >> > >> tokens to share a client?
> > > > > >> > > > >> >> > >>
> > > > > >> > > > >> >> > >> Gwen
> > > > > >> > > > >> >> > >>
> > > > > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth
> brahmbhatt
> > > > > >> > > > >> >> > >> <br...@gmail.com> wrote:
> > > > > >> > > > >> >> > >> > Bumping this up one more time, can other
> > > committers
> > > > > >> review?
> > > > > >> > > > >> >> > >> >
> > > > > >> > > > >> >> > >> > Thanks
> > > > > >> > > > >> >> > >> > Parth
> > > > > >> > > > >> >> > >> >
> > > > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <
> > > > > >> kafka@harsha.io>
> > > > > >> > > > wrote:
> > > > > >> > > > >> >> > >> >
> > > > > >> > > > >> >> > >> >> Parth,
> > > > > >> > > > >> >> > >> >>           Overall current design looks good
> to
> > > > me. I
> > > > > >> am +1
> > > > > >> > > on
> > > > > >> > > > >> the
> > > > > >> > > > >> >> > KIP.
> > > > > >> > > > >> >> > >> >>
> > > > > >> > > > >> >> > >> >> Gwen , Jun can you review this as well.
> > > > > >> > > > >> >> > >> >>
> > > > > >> > > > >> >> > >> >> -Harsha
> > > > > >> > > > >> >> > >> >>
> > > > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth
> > > > brahmbhatt
> > > > > >> wrote:
> > > > > >> > > > >> >> > >> >> > Thanks for review Jitendra.
> > > > > >> > > > >> >> > >> >> >
> > > > > >> > > > >> >> > >> >> > I don't like the idea of infinite lifetime
> > > but I
> > > > > >> see the
> > > > > >> > > > >> Streaming
> > > > > >> > > > >> >> > use
> > > > > >> > > > >> >> > >> >> > case. Even for Streaming use case I was
> > hoping
> > > > > >> there will
> > > > > >> > > > be
> > > > > >> > > > >> some
> > > > > >> > > > >> >> > notion
> > > > > >> > > > >> >> > >> >> > of
> > > > > >> > > > >> >> > >> >> > master/driver that can get new delegation
> > > tokens
> > > > > at
> > > > > >> fixed
> > > > > >> > > > >> interval
> > > > > >> > > > >> >> > and
> > > > > >> > > > >> >> > >> >> > distribute to workers. If that is not the
> > case
> > > > for
> > > > > >> we can
> > > > > >> > > > >> discuss
> > > > > >> > > > >> >> > >> >> > delegation tokens renewing them self and
> the
> > > > > >> security
> > > > > >> > > > >> implications
> > > > > >> > > > >> >> > of the
> > > > > >> > > > >> >> > >> >> > same.
> > > > > >> > > > >> >> > >> >> >
> > > > > >> > > > >> >> > >> >> > I did not want clients to fetch tokens
> from
> > > > > >> zookeeper,
> > > > > >> > > > >> overall I
> > > > > >> > > > >> >> > think
> > > > > >> > > > >> >> > >> >> > its
> > > > > >> > > > >> >> > >> >> > better if clients don't rely on our
> metadata
> > > > store
> > > > > >> and I
> > > > > >> > > > >> think we
> > > > > >> > > > >> >> > are
> > > > > >> > > > >> >> > >> >> > moving in that direction with all the
> KIP-4
> > > > > >> improvements.
> > > > > >> > > > I
> > > > > >> > > > >> chose
> > > > > >> > > > >> >> > >> >> > zookeeper as in this case the client will
> > > still
> > > > > >> just talk
> > > > > >> > > > to
> > > > > >> > > > >> >> > broker , its
> > > > > >> > > > >> >> > >> >> > the brokers that will use zookeeper which
> we
> > > > > >> already do
> > > > > >> > > > for a
> > > > > >> > > > >> lot
> > > > > >> > > > >> >> > of
> > > > > >> > > > >> >> > >> >> > other
> > > > > >> > > > >> >> > >> >> > usecases + ease of development + and the
> > > ability
> > > > > so
> > > > > >> > > tokens
> > > > > >> > > > >> will
> > > > > >> > > > >> >> > survive
> > > > > >> > > > >> >> > >> >> > even a rolling restart/cluster failure.
> if a
> > > > > >> majority
> > > > > >> > > > agrees
> > > > > >> > > > >> the
> > > > > >> > > > >> >> > added
> > > > > >> > > > >> >> > >> >> > complexity to have controller forwarding
> > keys
> > > to
> > > > > all
> > > > > >> > > > broker is
> > > > > >> > > > >> >> > justified
> > > > > >> > > > >> >> > >> >> > as
> > > > > >> > > > >> >> > >> >> > it provides tighter security , I am fine
> > with
> > > > that
> > > > > >> option
> > > > > >> > > > too.
> > > > > >> > > > >> >> > >> >> >
> > > > > >> > > > >> >> > >> >> > Given zookeeper does not support SSL we
> can
> > > not
> > > > > >> store
> > > > > >> > > > master
> > > > > >> > > > >> keys
> > > > > >> > > > >> >> > in
> > > > > >> > > > >> >> > >> >> > zookeeper as master keys will be exposed
> on
> > > > wire.
> > > > > To
> > > > > >> > > > support
> > > > > >> > > > >> >> > rotation
> > > > > >> > > > >> >> > >> >> > without affecting current clients is
> > > something I
> > > > > >> need to
> > > > > >> > > > put
> > > > > >> > > > >> more
> > > > > >> > > > >> >> > thought
> > > > > >> > > > >> >> > >> >> > in. My current proposal assumes the
> rotation
> > > > will
> > > > > >> > > > invalidate
> > > > > >> > > > >> all
> > > > > >> > > > >> >> > current
> > > > > >> > > > >> >> > >> >> > tokens.
> > > > > >> > > > >> >> > >> >> >
> > > > > >> > > > >> >> > >> >> > I request committers to also review and
> post
> > > > their
> > > > > >> > > comments
> > > > > >> > > > >> so we
> > > > > >> > > > >> >> > can
> > > > > >> > > > >> >> > >> >> > make
> > > > > >> > > > >> >> > >> >> > progress on this KIP.
> > > > > >> > > > >> >> > >> >> >
> > > > > >> > > > >> >> > >> >> > Thanks
> > > > > >> > > > >> >> > >> >> > Parth
> > > > > >> > > > >> >> > >> >> >
> > > > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish
> > Singh
> > > <
> > > > > >> > > > >> asingh@cloudera.com
> > > > > >> > > > >> >> > >
> > > > > >> > > > >> >> > >> >> > wrote:
> > > > > >> > > > >> >> > >> >> >
> > > > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM,
> Harsha <
> > > > > >> > > > kafka@harsha.io>
> > > > > >> > > > >> >> > wrote:
> > > > > >> > > > >> >> > >> >> > >
> > > > > >> > > > >> >> > >> >> > > > Unifying the two discussion threads on
> > > this
> > > > > KIP.
> > > > > >> > > > >> >> > >> >> > > >
> > > > > >> > > > >> >> > >> >> > > > Here is the response from Jitendra
> > > > > >> > > > >> >> > >> >> > > >
> > > > > >> > > > >> >> > >> >> > > > "The need for a large number of
> clients
> > > that
> > > > > are
> > > > > >> > > > running
> > > > > >> > > > >> all
> > > > > >> > > > >> >> > over the
> > > > > >> > > > >> >> > >> >> > > > cluster that authenticate with Kafka
> > > > brokers,
> > > > > >> is very
> > > > > >> > > > >> similar
> > > > > >> > > > >> >> > to the
> > > > > >> > > > >> >> > >> >> > > > Hadoop use case of large number of
> tasks
> > > > > running
> > > > > >> > > across
> > > > > >> > > > >> the
> > > > > >> > > > >> >> > cluster
> > > > > >> > > > >> >> > >> >> that
> > > > > >> > > > >> >> > >> >> > > > need authentication to Hdfs Namenode.
> > > > > >> Therefore, the
> > > > > >> > > > >> >> > delegation token
> > > > > >> > > > >> >> > >> >> > > > approach does seem like a good fit for
> > > this
> > > > > use
> > > > > >> case
> > > > > >> > > > as we
> > > > > >> > > > >> >> > have seen
> > > > > >> > > > >> >> > >> >> it
> > > > > >> > > > >> >> > >> >> > > > working at large scale in HDFS and
> YARN.
> > > > > >> > > > >> >> > >> >> > > >
> > > > > >> > > > >> >> > >> >> > > >   The proposed design is very much
> > inline
> > > > with
> > > > > >> Hadoop
> > > > > >> > > > >> >> > approach. A few
> > > > > >> > > > >> >> > >> >> > > >   comments:
> > > > > >> > > > >> >> > >> >> > > >
> > > > > >> > > > >> >> > >> >> > > > 1) Why do you guys want to allow
> > infinite
> > > > > >> renewable
> > > > > >> > > > >> lifetime
> > > > > >> > > > >> >> > for a
> > > > > >> > > > >> >> > >> >> > > > token? HDFS restricts a token to a max
> > > life
> > > > > time
> > > > > >> > > > (default
> > > > > >> > > > >> 7
> > > > > >> > > > >> >> > days).  A
> > > > > >> > > > >> >> > >> >> > > > token's vulnerability is believed to
> > > > increase
> > > > > >> with
> > > > > >> > > > time.
> > > > > >> > > > >> >> > >> >> > > >
> > > > > >> > > > >> >> > >> >> > > I agree that having infinite lifetime
> > might
> > > > not
> > > > > >> be the
> > > > > >> > > > best
> > > > > >> > > > >> idea.
> > > > > >> > > > >> >> > >> >> > >
> > > > > >> > > > >> >> > >> >> > > >
> > > > > >> > > > >> >> > >> >> > > > 2) As I understand the tokens are
> stored
> > > in
> > > > > >> zookeeper
> > > > > >> > > > as
> > > > > >> > > > >> well,
> > > > > >> > > > >> >> > and
> > > > > >> > > > >> >> > >> >> can
> > > > > >> > > > >> >> > >> >> > > > be updated there. This is clever as it
> > can
> > > > > allow
> > > > > >> > > > >> replacing the
> > > > > >> > > > >> >> > tokens
> > > > > >> > > > >> >> > >> >> > > > once they run out of max life time,
> and
> > > > > clients
> > > > > >> can
> > > > > >> > > > >> download
> > > > > >> > > > >> >> > new
> > > > > >> > > > >> >> > >> >> tokens
> > > > > >> > > > >> >> > >> >> > > > from zookeeper. It shouldn't be a big
> > load
> > > > on
> > > > > >> > > zookeeper
> > > > > >> > > > >> as a
> > > > > >> > > > >> >> > client
> > > > > >> > > > >> >> > >> >> will
> > > > > >> > > > >> >> > >> >> > > > need to get a new token once in
> several
> > > > days.
> > > > > >> In this
> > > > > >> > > > >> approach
> > > > > >> > > > >> >> > you
> > > > > >> > > > >> >> > >> >> don't
> > > > > >> > > > >> >> > >> >> > > > need infinite lifetime on the token
> even
> > > for
> > > > > >> long
> > > > > >> > > > running
> > > > > >> > > > >> >> > clients.
> > > > > >> > > > >> >> > >> >> > > >
> > > > > >> > > > >> >> > >> >> > > > 3) The token password are generated
> > using
> > > a
> > > > > >> master
> > > > > >> > > key.
> > > > > >> > > > >> The
> > > > > >> > > > >> >> > master
> > > > > >> > > > >> >> > >> >> key
> > > > > >> > > > >> >> > >> >> > > > should also be periodically changed.
> In
> > > > > Hadoop,
> > > > > >> the
> > > > > >> > > > >> default
> > > > > >> > > > >> >> > renewal
> > > > > >> > > > >> >> > >> >> > > > period is 1 day.?
> > > > > >> > > > >> >> > >> >> > > >
> > > > > >> > > > >> >> > >> >> > > IIUC, this will require brokers
> > maintaining
> > > a
> > > > > >> list of X
> > > > > >> > > > most
> > > > > >> > > > >> >> > recent
> > > > > >> > > > >> >> > >> >> master
> > > > > >> > > > >> >> > >> >> > > keys. This list will have to be
> persisted
> > > > > >> somewhere, as
> > > > > >> > > > if a
> > > > > >> > > > >> >> > broker
> > > > > >> > > > >> >> > >> >> goes
> > > > > >> > > > >> >> > >> >> > > down it will have to get that list again
> > and
> > > > > >> storing
> > > > > >> > > > master
> > > > > >> > > > >> keys
> > > > > >> > > > >> >> > on ZK
> > > > > >> > > > >> >> > >> >> is
> > > > > >> > > > >> >> > >> >> > > not the best idea. However, if a broker
> > goes
> > > > > down
> > > > > >> then
> > > > > >> > > we
> > > > > >> > > > >> have
> > > > > >> > > > >> >> > much
> > > > > >> > > > >> >> > >> >> bigger
> > > > > >> > > > >> >> > >> >> > > issue to deal with and client can always
> > > > > >> > > re-authenticate
> > > > > >> > > > is
> > > > > >> > > > >> such
> > > > > >> > > > >> >> > >> >> events.
> > > > > >> > > > >> >> > >> >> > >
> > > > > >> > > > >> >> > >> >> > > Did you happen to take a look at other
> > > > > >> alternatives
> > > > > >> > > this
> > > > > >> > > > >> list has
> > > > > >> > > > >> >> > >> >> > > suggested?
> > > > > >> > > > >> >> > >> >> > >
> > > > > >> > > > >> >> > >> >> > > >
> > > > > >> > > > >> >> > >> >> > > > Thanks for a thorough proposal, great
> > > work!"
> > > > > >> > > > >> >> > >> >> > > >
> > > > > >> > > > >> >> > >> >> > > >
> > > > > >> > > > >> >> > >> >> > > >
> > > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen
> > > > Shapira
> > > > > >> wrote:
> > > > > >> > > > >> >> > >> >> > > > > Makes sense to me. Thanks!
> > > > > >> > > > >> >> > >> >> > > > >
> > > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM,
> > Harsha <
> > > > > >> > > > kafka@harsha.io
> > > > > >> > > > >> >
> > > > > >> > > > >> >> > wrote:
> > > > > >> > > > >> >> > >> >> > > > > > It doesn't need any release
> vehicle
> > > but
> > > > > >> still the
> > > > > >> > > > >> work can
> > > > > >> > > > >> >> > move
> > > > > >> > > > >> >> > >> >> > > > forward.
> > > > > >> > > > >> >> > >> >> > > > > > If anyone is interested in the KIP
> > > > please
> > > > > >> do the
> > > > > >> > > > >> review and
> > > > > >> > > > >> >> > >> >> provide
> > > > > >> > > > >> >> > >> >> > > the
> > > > > >> > > > >> >> > >> >> > > > > > comments.
> > > > > >> > > > >> >> > >> >> > > > > >
> > > > > >> > > > >> >> > >> >> > > > > > -Harsha
> > > > > >> > > > >> >> > >> >> > > > > >
> > > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM,
> > > Ismael
> > > > > >> Juma
> > > > > >> > > > wrote:
> > > > > >> > > > >> >> > >> >> > > > > >> I agree that it would be good to
> > have
> > > > > more
> > > > > >> time
> > > > > >> > > to
> > > > > >> > > > >> review
> > > > > >> > > > >> >> > and
> > > > > >> > > > >> >> > >> >> > > discuss
> > > > > >> > > > >> >> > >> >> > > > > >> KIP-48.
> > > > > >> > > > >> >> > >> >> > > > > >>
> > > > > >> > > > >> >> > >> >> > > > > >> Ismael
> > > > > >> > > > >> >> > >> >> > > > > >>
> > > > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM,
> > Gwen
> > > > > >> Shapira <
> > > > > >> > > > >> >> > >> >> gwen@confluent.io>
> > > > > >> > > > >> >> > >> >> > > > wrote:
> > > > > >> > > > >> >> > >> >> > > > > >>
> > > > > >> > > > >> >> > >> >> > > > > >> > Hi Team,
> > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > >> > > > >> >> > >> >> > > > > >> > Since KIP-48 depends on KIP-43,
> > > which
> > > > > is
> > > > > >> > > > already a
> > > > > >> > > > >> bit
> > > > > >> > > > >> >> > of a
> > > > > >> > > > >> >> > >> >> risk
> > > > > >> > > > >> >> > >> >> > > for
> > > > > >> > > > >> >> > >> >> > > > > >> > the next release - any chance
> we
> > > can
> > > > > >> delay
> > > > > >> > > > >> delegation
> > > > > >> > > > >> >> > tokens
> > > > > >> > > > >> >> > >> >> to
> > > > > >> > > > >> >> > >> >> > > > Kafka
> > > > > >> > > > >> >> > >> >> > > > > >> > 0.10.1?
> > > > > >> > > > >> >> > >> >> > > > > >> > With the community working on a
> > > > release
> > > > > >> every
> > > > > >> > > 3
> > > > > >> > > > >> month,
> > > > > >> > > > >> >> > this
> > > > > >> > > > >> >> > >> >> is not
> > > > > >> > > > >> >> > >> >> > > > a huge
> > > > > >> > > > >> >> > >> >> > > > > >> > delay.
> > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > >> > > > >> >> > >> >> > > > > >> > Gwen
> > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11
> PM,
> > > > Ashish
> > > > > >> Singh
> > > > > >> > > <
> > > > > >> > > > >> >> > >> >> > > asingh@cloudera.com>
> > > > > >> > > > >> >> > >> >> > > > wrote:
> > > > > >> > > > >> >> > >> >> > > > > >> > > Parth,
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > > Thanks again for the awesome
> > > write
> > > > > up.
> > > > > >> > > > Following
> > > > > >> > > > >> our
> > > > > >> > > > >> >> > >> >> discussion
> > > > > >> > > > >> >> > >> >> > > > from the
> > > > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I think it will be
> easier
> > > to
> > > > > >> compare
> > > > > >> > > > >> various
> > > > > >> > > > >> >> > >> >> alternatives
> > > > > >> > > > >> >> > >> >> > > > if they
> > > > > >> > > > >> >> > >> >> > > > > >> > are
> > > > > >> > > > >> >> > >> >> > > > > >> > > listed together. I am stating
> > > > below a
> > > > > >> few
> > > > > >> > > > >> >> > alternatives along
> > > > > >> > > > >> >> > >> >> > > with
> > > > > >> > > > >> >> > >> >> > > > a the
> > > > > >> > > > >> >> > >> >> > > > > >> > > current proposal.
> > > > > >> > > > >> >> > >> >> > > > > >> > > (Current proposal) Store
> > > Delegation
> > > > > >> Token,
> > > > > >> > > DT,
> > > > > >> > > > >> on ZK.
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates
> > with a
> > > > > >> broker.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > > > authenticated,
> > > > > >> it
> > > > > >> > > will
> > > > > >> > > > >> make a
> > > > > >> > > > >> >> > broker
> > > > > >> > > > >> >> > >> >> side
> > > > > >> > > > >> >> > >> >> > > > call to
> > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker generates a
> > > shared
> > > > > >> secret
> > > > > >> > > > based
> > > > > >> > > > >> on
> > > > > >> > > > >> >> > >> >> > > HMAC-SHA256(a
> > > > > >> > > > >> >> > >> >> > > > > >> > >    Password/Secret shared
> > between
> > > > all
> > > > > >> > > brokers,
> > > > > >> > > > >> >> > randomly
> > > > > >> > > > >> >> > >> >> > > generated
> > > > > >> > > > >> >> > >> >> > > > > >> > tokenId).
> > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Broker stores this
> token
> > in
> > > > its
> > > > > >> in
> > > > > >> > > > memory
> > > > > >> > > > >> cache.
> > > > > >> > > > >> >> > >> >> Broker
> > > > > >> > > > >> >> > >> >> > > > also stores
> > > > > >> > > > >> >> > >> >> > > > > >> > >    the DelegationToken
> without
> > > the
> > > > > >> hmac in
> > > > > >> > > the
> > > > > >> > > > >> >> > zookeeper.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    5. All brokers will have a
> > > cache
> > > > > >> backed
> > > > > >> > > by
> > > > > >> > > > >> >> > zookeeper so
> > > > > >> > > > >> >> > >> >> they
> > > > > >> > > > >> >> > >> >> > > > will all
> > > > > >> > > > >> >> > >> >> > > > > >> > >    get notified whenever a
> new
> > > > token
> > > > > is
> > > > > >> > > > >> generated and
> > > > > >> > > > >> >> > they
> > > > > >> > > > >> >> > >> >> will
> > > > > >> > > > >> >> > >> >> > > > update
> > > > > >> > > > >> >> > >> >> > > > > >> > their
> > > > > >> > > > >> >> > >> >> > > > > >> > >    local cache whenever token
> > > state
> > > > > >> changes.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Broker returns the
> token
> > to
> > > > > >> Client.
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Probable race
> condition,
> > > > client
> > > > > >> tries
> > > > > >> > > to
> > > > > >> > > > >> >> > authenticate
> > > > > >> > > > >> >> > >> >> with
> > > > > >> > > > >> >> > >> >> > > > a broker
> > > > > >> > > > >> >> > >> >> > > > > >> > >    that is yet to be updated
> > with
> > > > the
> > > > > >> newly
> > > > > >> > > > >> generated
> > > > > >> > > > >> >> > DT.
> > > > > >> > > > >> >> > >> >> This
> > > > > >> > > > >> >> > >> >> > > can
> > > > > >> > > > >> >> > >> >> > > > > >> > probably be
> > > > > >> > > > >> >> > >> >> > > > > >> > >    dealt with making
> dtRequest
> > > > block
> > > > > >> until
> > > > > >> > > all
> > > > > >> > > > >> >> > brokers have
> > > > > >> > > > >> >> > >> >> > > > updated
> > > > > >> > > > >> >> > >> >> > > > > >> > their DT
> > > > > >> > > > >> >> > >> >> > > > > >> > >    cache. Zk barrier or
> similar
> > > > > >> mechanism
> > > > > >> > > can
> > > > > >> > > > be
> > > > > >> > > > >> used.
> > > > > >> > > > >> >> > >> >> However,
> > > > > >> > > > >> >> > >> >> > > > all such
> > > > > >> > > > >> >> > >> >> > > > > >> > >    mechanisms will increase
> > > > > complexity.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Using a static secret
> key
> > > > from
> > > > > >> config
> > > > > >> > > > >> file. Will
> > > > > >> > > > >> >> > >> >> require
> > > > > >> > > > >> >> > >> >> > > yet
> > > > > >> > > > >> >> > >> >> > > > > >> > another
> > > > > >> > > > >> >> > >> >> > > > > >> > >    config and uses a static
> > > secret
> > > > > >> key. It
> > > > > >> > > is
> > > > > >> > > > >> advised
> > > > > >> > > > >> >> > to
> > > > > >> > > > >> >> > >> >> rotate
> > > > > >> > > > >> >> > >> >> > > > secret
> > > > > >> > > > >> >> > >> >> > > > > >> > keys
> > > > > >> > > > >> >> > >> >> > > > > >> > >    periodically. This can be
> > > > avoided
> > > > > >> with
> > > > > >> > > > >> controller
> > > > > >> > > > >> >> > >> >> generating
> > > > > >> > > > >> >> > >> >> > > > > >> > secretKey and
> > > > > >> > > > >> >> > >> >> > > > > >> > >    passing to brokers
> > > periodically.
> > > > > >> However,
> > > > > >> > > > >> this will
> > > > > >> > > > >> >> > >> >> require
> > > > > >> > > > >> >> > >> >> > > > brokers to
> > > > > >> > > > >> >> > >> >> > > > > >> > >    maintain certain counts of
> > > > > >> secretKeys.
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 1) Have
> controller
> > > > > >> generate
> > > > > >> > > > >> delegation
> > > > > >> > > > >> >> > token.
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates
> > with a
> > > > > >> broker.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > > > authenticated,
> > > > > >> it
> > > > > >> > > will
> > > > > >> > > > >> make a
> > > > > >> > > > >> >> > broker
> > > > > >> > > > >> >> > >> >> side
> > > > > >> > > > >> >> > >> >> > > > call to
> > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    3. Broker forwards the
> > request
> > > > to
> > > > > >> > > > controller.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Controller generates a
> DT
> > > and
> > > > > >> > > broadcasts
> > > > > >> > > > >> to all
> > > > > >> > > > >> >> > >> >> brokers.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    5. Broker stores this
> token
> > in
> > > > its
> > > > > >> memory
> > > > > >> > > > >> cache.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    6. Controller responds to
> > > > broker’s
> > > > > >> DT
> > > > > >> > > req.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    7. Broker returns the
> token
> > to
> > > > > >> Client.
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > >    1. We will have to add new
> > > APIs
> > > > to
> > > > > >> > > support
> > > > > >> > > > >> >> > controller
> > > > > >> > > > >> >> > >> >> pushing
> > > > > >> > > > >> >> > >> >> > > > tokens
> > > > > >> > > > >> >> > >> >> > > > > >> > to
> > > > > >> > > > >> >> > >> >> > > > > >> > >    brokers on top of the
> > minimal
> > > > APIs
> > > > > >> that
> > > > > >> > > are
> > > > > >> > > > >> >> > currently
> > > > > >> > > > >> >> > >> >> > > proposed.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    2. We will also have to
> add
> > > APIs
> > > > > to
> > > > > >> > > support
> > > > > >> > > > >> the
> > > > > >> > > > >> >> > >> >> bootstrapping
> > > > > >> > > > >> >> > >> >> > > > case,
> > > > > >> > > > >> >> > >> >> > > > > >> > i.e,
> > > > > >> > > > >> >> > >> >> > > > > >> > >    when a new broker comes up
> > it
> > > > will
> > > > > >> have
> > > > > >> > > to
> > > > > >> > > > >> get all
> > > > > >> > > > >> >> > >> >> delegation
> > > > > >> > > > >> >> > >> >> > > > tokens
> > > > > >> > > > >> >> > >> >> > > > > >> > from
> > > > > >> > > > >> >> > >> >> > > > > >> > >    the controller.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    3. In catastrophic
> failures
> > > > where
> > > > > >> all
> > > > > >> > > > brokers
> > > > > >> > > > >> go
> > > > > >> > > > >> >> > down,
> > > > > >> > > > >> >> > >> >> the
> > > > > >> > > > >> >> > >> >> > > > tokens will
> > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if servers
> are
> > > > > >> restarted as
> > > > > >> > > > >> tokens
> > > > > >> > > > >> >> > are not
> > > > > >> > > > >> >> > >> >> > > > persisted
> > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    If this happens, then
> there
> > > are
> > > > > more
> > > > > >> > > > important
> > > > > >> > > > >> >> > things to
> > > > > >> > > > >> >> > >> >> > > worry
> > > > > >> > > > >> >> > >> >> > > > about
> > > > > >> > > > >> >> > >> >> > > > > >> > and
> > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
> > > > > >> re-authenticate.
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 2) Do not
> > distribute
> > > > DT
> > > > > to
> > > > > >> > > other
> > > > > >> > > > >> brokers
> > > > > >> > > > >> >> > at
> > > > > >> > > > >> >> > >> >> all.
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates
> > with a
> > > > > >> broker.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > > > authenticated,
> > > > > >> it
> > > > > >> > > will
> > > > > >> > > > >> make a
> > > > > >> > > > >> >> > broker
> > > > > >> > > > >> >> > >> >> side
> > > > > >> > > > >> >> > >> >> > > > call to
> > > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker generates DT
> > of
> > > > > form,
> > > > > >> > > [hmac +
> > > > > >> > > > >> (owner,
> > > > > >> > > > >> >> > >> >> renewer,
> > > > > >> > > > >> >> > >> >> > > > > >> > >    maxLifeTime, id, hmac,
> > > > > >> expirationTime)]
> > > > > >> > > and
> > > > > >> > > > >> passes
> > > > > >> > > > >> >> > back
> > > > > >> > > > >> >> > >> >> this
> > > > > >> > > > >> >> > >> >> > > > DT to
> > > > > >> > > > >> >> > >> >> > > > > >> > client.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    hmac is generated via
> > > > > >> {HMAC-SHA256(owner,
> > > > > >> > > > >> renewer,
> > > > > >> > > > >> >> > >> >> > > > maxLifeTime, id,
> > > > > >> > > > >> >> > >> >> > > > > >> > hmac,
> > > > > >> > > > >> >> > >> >> > > > > >> > >    expirationTime) using
> > > > SecretKey}.
> > > > > >> Note
> > > > > >> > > that
> > > > > >> > > > >> all
> > > > > >> > > > >> >> > brokers
> > > > > >> > > > >> >> > >> >> have
> > > > > >> > > > >> >> > >> >> > > > this
> > > > > >> > > > >> >> > >> >> > > > > >> > SecretKey.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    4. Client then goes to any
> > > > broker
> > > > > >> and to
> > > > > >> > > > >> >> > authenticate
> > > > > >> > > > >> >> > >> >> sends
> > > > > >> > > > >> >> > >> >> > > > the DT.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    Broker recalculates hmac
> > using
> > > > > >> (owner,
> > > > > >> > > > >> renewer,
> > > > > >> > > > >> >> > >> >> maxLifeTime,
> > > > > >> > > > >> >> > >> >> > > > id, hmac,
> > > > > >> > > > >> >> > >> >> > > > > >> > >    expirationTime) info from
> DT
> > > and
> > > > > its
> > > > > >> > > > >> SecretKey. If
> > > > > >> > > > >> >> > it
> > > > > >> > > > >> >> > >> >> matches
> > > > > >> > > > >> >> > >> >> > > > with
> > > > > >> > > > >> >> > >> >> > > > > >> > hmac of
> > > > > >> > > > >> >> > >> >> > > > > >> > >    DT, client is
> authenticated.
> > > > Yes,
> > > > > >> it will
> > > > > >> > > > do
> > > > > >> > > > >> other
> > > > > >> > > > >> >> > >> >> obvious
> > > > > >> > > > >> >> > >> >> > > > checks of
> > > > > >> > > > >> >> > >> >> > > > > >> > >    timestamp expiry and such.
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > > Note that secret key will be
> > > > > generated
> > > > > >> by
> > > > > >> > > > >> controller
> > > > > >> > > > >> >> > and
> > > > > >> > > > >> >> > >> >> passed
> > > > > >> > > > >> >> > >> >> > > to
> > > > > >> > > > >> >> > >> >> > > > > >> > brokers
> > > > > >> > > > >> >> > >> >> > > > > >> > > periodically.
> > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > >    1. How to delete a DT?
> Yes,
> > > that
> > > > > is
> > > > > >> a
> > > > > >> > > > downside
> > > > > >> > > > >> >> > here.
> > > > > >> > > > >> >> > >> >> However,
> > > > > >> > > > >> >> > >> >> > > > this can
> > > > > >> > > > >> >> > >> >> > > > > >> > >    be handled with brokers
> > > > > maintaining
> > > > > >> a
> > > > > >> > > > >> blacklist of
> > > > > >> > > > >> >> > DTs,
> > > > > >> > > > >> >> > >> >> DTs
> > > > > >> > > > >> >> > >> >> > > > from this
> > > > > >> > > > >> >> > >> >> > > > > >> > list
> > > > > >> > > > >> >> > >> >> > > > > >> > >    can be removed after
> expiry.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    2. In catastrophic
> failures
> > > > where
> > > > > >> all
> > > > > >> > > > brokers
> > > > > >> > > > >> go
> > > > > >> > > > >> >> > down,
> > > > > >> > > > >> >> > >> >> the
> > > > > >> > > > >> >> > >> >> > > > tokens will
> > > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if servers
> are
> > > > > >> restarted as
> > > > > >> > > > >> tokens
> > > > > >> > > > >> >> > are not
> > > > > >> > > > >> >> > >> >> > > > persisted
> > > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > > > >> > > > >> >> > >> >> > > > > >> > >    If this happens, then
> there
> > > are
> > > > > more
> > > > > >> > > > important
> > > > > >> > > > >> >> > things to
> > > > > >> > > > >> >> > >> >> > > worry
> > > > > >> > > > >> >> > >> >> > > > about
> > > > > >> > > > >> >> > >> >> > > > > >> > and
> > > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
> > > > > >> re-authenticate.
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58
> > PM,
> > > > > Parth
> > > > > >> > > > >> Brahmbhatt <
> > > > > >> > > > >> >> > >> >> > > > > >> > > pbrahmbhatt@hortonworks.com>
> > > > wrote:
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > >> Hi,
> > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > >> > > > >> >> > >> >> > > > > >> > >> I have filed KIP-48 so we
> can
> > > > offer
> > > > > >> hadoop
> > > > > >> > > > like
> > > > > >> > > > >> >> > delegation
> > > > > >> > > > >> >> > >> >> > > > tokens in
> > > > > >> > > > >> >> > >> >> > > > > >> > >> kafka. You can review the
> > design
> > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > >> > > > >> >> > >> >> > > >
> > > > > >> > > > >> >> > >> >> > >
> > > > > >> > > > >> >> > >> >>
> > > > > >> > > > >> >> >
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > > > > >> > > > >> >> > >> >> > > > > >> > .
> > > > > >> > > > >> >> > >> >> > > > > >> > >> This KIP depends on KIP-43
> and
> > > we
> > > > > >> have also
> > > > > >> > > > >> >> > discussed an
> > > > > >> > > > >> >> > >> >> > > > alternative to
> > > > > >> > > > >> >> > >> >> > > > > >> > >> proposed design here<
> > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > >> > > > >> >> > >> >> > > >
> > > > > >> > > > >> >> > >> >> > >
> > > > > >> > > > >> >> > >> >>
> > > > > >> > > > >> >> >
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > > > > >> > > > >> >> > >> >> > > > > >> > >> >.
> > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > >> > > > >> >> > >> >> > > > > >> > >> Thanks
> > > > > >> > > > >> >> > >> >> > > > > >> > >> Parth
> > > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > > --
> > > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > > >> > > > >> >> > >> >> > > > > >> > > Regards,
> > > > > >> > > > >> >> > >> >> > > > > >> > > Ashish
> > > > > >> > > > >> >> > >> >> > > > > >> >
> > > > > >> > > > >> >> > >> >> > > >
> > > > > >> > > > >> >> > >> >> > >
> > > > > >> > > > >> >> > >> >> > >
> > > > > >> > > > >> >> > >> >> > >
> > > > > >> > > > >> >> > >> >> > > --
> > > > > >> > > > >> >> > >> >> > >
> > > > > >> > > > >> >> > >> >> > > Regards,
> > > > > >> > > > >> >> > >> >> > > Ashish
> > > > > >> > > > >> >> > >> >> > >
> > > > > >> > > > >> >> > >> >>
> > > > > >> > > > >> >> >
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >>
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>



-- 
Regards,

Rajini

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Ismael Juma <is...@juma.me.uk>]

Thanks Rajini, SCRAM seems like a good candidate.

Gwen had independently mentioned this as a SASL mechanism that might be
useful for Kafka and I have been meaning to check it in more detail. Good
to know that you are willing to contribute an implementation. Maybe we
should file a separate JIRA for this?

Ismael

On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram <
rajinisivaram@googlemail.com> wrote:

> SCRAM (Salted Challenge Response Authentication Mechanism) is a better
> mechanism than Digest-MD5. Java doesn't come with a built-in SCRAM
> SaslServer or SaslClient, but I will be happy to add support in Kafka since
> it would be a useful mechanism to support anyway.
> https://tools.ietf.org/html/rfc7677 describes the protocol for
> SCRAM-SHA-256.
>
> On Tue, May 24, 2016 at 2:37 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > Parth,
> >
> > Thanks for the explanation. A couple of more questions.
> >
> > 110. What does getDelegationTokenAs mean?
> >
> > 111. What's the typical rate of getting and renewing delegation tokens?
> > That may have an impact on whether they should be directed to the
> > controller.
> >
> > Jun
> >
> > On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt <
> > brahmbhatt.parth@gmail.com> wrote:
> >
> > > Hi Jun,
> > >
> > > Thanks for reviewing.
> > >
> > > * We could add a Cluster action to add acls on who can request
> delegation
> > > tokens. I don't see the use case for that yet but down the line when we
> > > start supporting getDelegationTokenAs it will be necessary.
> > > * Yes we recommend tokens to be only used/distributed over secure
> > channels.
> > > * Depending on what design we end up choosing Invalidation will be
> > > responsibility of every broker or controller.
> > > * I am not sure if I documented somewhere that invalidation will
> directly
> > > go through zookeeper but that is not the intent. Invalidation will
> either
> > > be request based or due to expiration. No direct zookeeper interaction
> > from
> > > any client.
> > > * "Broker also stores the DelegationToken without the hmac in the
> > > zookeeper." : Sorry about the confusion. The sole purpose of zookeeper
> in
> > > this design is as distribution channel for tokens between all brokers
> > and a
> > > layer that ensures only tokens that were generated by making a request
> > to a
> > > broker will be accepted (more on this in second paragraph). The token
> > > consists of few elements (owner, renewer, uuid , expiration, hmac) ,
> one
> > of
> > > which is the finally generated hmac but hmac it self is derivable if
> you
> > > have all the other elements of the token + secret key to generate hmac.
> > > Given zookeeper does not provide SSL support we do not want the entire
> > > token to be wire transferred to zookeeper as that will be an insecure
> > wire
> > > transfer. Instead we only store all the other elements of a delegation
> > > tokens. Brokers can read these elements and because they also have
> access
> > > to secret key they will be able to generate hmac on their end.
> > >
> > > One of the alternative proposed is to avoid zookeeper altogether. A
> > Client
> > > will call broker with required information (owner, renwer, expiration)
> > and
> > > get back (signed hmac, uuid). Broker won't store this in zookeeper.
> From
> > > this point a client can contact any broker with all the delegation
> token
> > > info (owner, rewner, expiration, hmac, uuid) the borker will regenerate
> > the
> > > hmac and as long as it matches with hmac presented by client , broker
> > will
> > > allow the request to authenticate.  Only problem with this approach is
> if
> > > the secret key is compromised any client can now generate random tokens
> > and
> > > they will still be able to authenticate as any user they like. with
> > > zookeeper we guarantee that only tokens acquired via a broker (using
> some
> > > auth scheme other than delegation token) will be accepted. We need to
> > > discuss which proposal makes more sense and we can go over it in
> > tomorrow's
> > > meeting.
> > >
> > > Also, can you forward the invite to me?
> > >
> > > Thanks
> > > Parth
> > >
> > >
> > >
> > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Thanks for the KIP. A few comments.
> > > >
> > > > 100. This potentially can be useful for Kafka Connect and Kafka rest
> > > proxy
> > > > where a worker agent will need to run a task on behalf of a client.
> We
> > > will
> > > > likely need to change how those services use Kafka clients
> > > > (producer/consumer). Instead of a shared client per worker, we will
> > need
> > > a
> > > > client per user task since the authentication happens at the
> connection
> > > > level. For Kafka Connect, the renewer will be the workers. So, we
> > > probably
> > > > need to allow multiple renewers. For Kafka rest proxy, the renewer
> can
> > > > probably just be the creator of the token.
> > > >
> > > > 101. Do we need new acl on who can request delegation tokens?
> > > >
> > > > 102. Do we recommend people to send delegation tokens in an encrypted
> > > > channel?
> > > >
> > > > 103. Who is responsible for expiring tokens, every broker?
> > > >
> > > > 104. For invalidating tokens, would it be better to do it in a
> request
> > > > instead of going to ZK directly?
> > > >
> > > > 105. The terminology of client in the wiki sometimes refers to the
> end
> > > > client and some other times refers to the client using the delegation
> > > > tokens. It would be useful to distinguish between the two.
> > > >
> > > > 106. Could you explain the sentence "Broker also stores the
> > > DelegationToken
> > > > without the hmac in the zookeeper." a bit more? I thought the
> > delegation
> > > > token is the hmac.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <ju...@confluent.io> wrote:
> > > >
> > > > > Hi, Harsha,
> > > > >
> > > > > Just sent out a KIP meeting invite. We can discuss this in the
> > meeting
> > > > > tomorrow.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <ka...@harsha.io> wrote:
> > > > >
> > > > >> Hi All,
> > > > >>            Can we have a KIP meeting around this. The KIP is up
> for
> > > > >>            sometime and if there are any questions lets quickly
> hash
> > > out
> > > > >>            details.
> > > > >>
> > > > >> Thanks,
> > > > >> Harsha
> > > > >>
> > > > >> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt wrote:
> > > > >> > That is what the hadoop echo system uses so no good reason
> really.
> > > We
> > > > >> > could
> > > > >> > change it to whatever is the newest recommended standard is.
> > > > >> >
> > > > >> > Thanks
> > > > >> > Parth
> > > > >> >
> > > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <ismael@juma.me.uk
> >
> > > > wrote:
> > > > >> >
> > > > >> > > Hi Parth,
> > > > >> > >
> > > > >> > > Thanks for the KIP. I only started reviewing this and may have
> > > > >> additional
> > > > >> > > questions later. The immediate question that came to mind is
> our
> > > > >> choice of
> > > > >> > > "DIGEST-MD5" even though it's marked as OBSOLETE in the IANA
> > > > Registry
> > > > >> of
> > > > >> > > SASL mechanisms and the original RFC (2831) has been moved to
> > > > Historic
> > > > >> > > status:
> > > > >> > >
> > > > >> > > https://tools.ietf.org/html/rfc6331
> > > > >> > >
> > > >
> http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> > > > >> > >
> > > > >> > > What is the reasoning behind that choice?
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Ismael
> > > > >> > >
> > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <
> > gwen@confluent.io
> > > >
> > > > >> wrote:
> > > > >> > >
> > > > >> > > > Also comments inline :)
> > > > >> > > >
> > > > >> > > > > * I want to emphasize that even though delegation tokens
> > are a
> > > > >> Hadoop
> > > > >> > > > > innovation, I feel very strongly about not adding
> dependency
> > > on
> > > > >> Hadoop
> > > > >> > > > > when implementing delegation tokens for Kafka. The KIP
> > doesn't
> > > > >> imply
> > > > >> > > > > such dependency, but if you can clarify...
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > *No hadoop dependency.*
> > > > >> > > >
> > > > >> > > > Yay! Just add this to the KIP so no one will read the KIP
> and
> > > > panic
> > > > >> > > > three weeks before the next release...
> > > > >> > > >
> > > > >> > > > > * Can we get delegation token at any time after
> > > authenticating?
> > > > >> only
> > > > >> > > > > immediately after?
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > *As long as you are authenticated you can get delegation
> > > tokens.
> > > > >> We
> > > > >> > > need
> > > > >> > > > to
> > > > >> > > > > discuss if a client authenticated using delegation token,
> > can
> > > > also
> > > > >> > > > acquire
> > > > >> > > > > delegation token again or not. Also there is the question
> of
> > > do
> > > > we
> > > > >> > > allow
> > > > >> > > > > anyone to acquire delegation token or we want specific
> ACLs
> > (I
> > > > >> think
> > > > >> > > its
> > > > >> > > > an
> > > > >> > > > > overkill.)*
> > > > >> > > >
> > > > >> > > > I agree that ACLs is an overkill.
> > > > >> > > >
> > > > >> > > > I think we are debating two options: Either require Kerberos
> > > auth
> > > > >> for
> > > > >> > > > renewal or require non-owners to renew.
> > > > >> > > > I *think* the latter is simpler (it basically require a "job
> > > > master"
> > > > >> > > > to take responsibility for the renewal, it will have its own
> > > > >> identity
> > > > >> > > > anyway and I think this is the correct design pattern
> anyway.
> > > For
> > > > >> > > > storm, I'd expect Nimbus to coordinate renewals?), but it is
> > > hard
> > > > to
> > > > >> > > > debate simplicity without looking at the code changes
> > required.
> > > If
> > > > >> you
> > > > >> > > > have a draft of how the "require Kerberos" will look in
> Kafka
> > > > code,
> > > > >> > > > I'll be happy to take a look.
> > > > >> > > >
> > > > >> > > > > * My understanding is that tokens will propagate via ZK
> but
> > > > >> without
> > > > >> > > > > additional changes to UpdateMetadata protocol, correct?
> > > Clients
> > > > >> > > > > currently don't retry on SASL auth failure (IIRC), but
> since
> > > the
> > > > >> > > > > tokens propagate between brokers asynch, we will need to
> > > retry a
> > > > >> bit
> > > > >> > > > > to avoid clients failing auth due to timing issues.
> > > > >> > > > >
> > > > >> > > > > *I am considering 2 alternatives right now. The current
> > > > documented
> > > > >> > > > approach
> > > > >> > > > > is zookeeper based and it does not require any changes to
> > > > >> > > UpdateMetadata
> > > > >> > > > > protocol. An alternative approach can remove zookeeper
> > > > dependency
> > > > >> as
> > > > >> > > well
> > > > >> > > > > but we can discuss that in KIP discussion call.*
> > > > >> > > >
> > > > >> > > > Oooh! Sounds interesting. Do you want to ping Jun to
> arrange a
> > > > call?
> > > > >> > > >
> > > > >> > > > > * I liked Ashish's suggestion of having just the
> controller
> > > > issue
> > > > >> the
> > > > >> > > > > delegation tokens, to avoid syncing a shared secret. Not
> > sure
> > > if
> > > > >> we
> > > > >> > > > > want to continue the discussion here or on the wiki. I
> think
> > > > that
> > > > >> we
> > > > >> > > > > can decouple the problem of "token distribution" from
> > "shared
> > > > >> secret
> > > > >> > > > > distribution" and use the controller as the only token
> > > generator
> > > > >> to
> > > > >> > > > > solve the second issue, while still using ZK async to
> > > distribute
> > > > >> > > > > tokens.
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > *As mentioned in the previous Email I am fine with that
> > > approach
> > > > >> as
> > > > >> > > long
> > > > >> > > > as
> > > > >> > > > > we agree that the extra complexity of adding/updating APIS
> > > adds
> > > > >> enough
> > > > >> > > > > value. The advantage with the controller approach is
> secret
> > > > >> rotation
> > > > >> > > can
> > > > >> > > > be
> > > > >> > > > > automated,frequent and would not require deployment. *
> > > > >> > > >
> > > > >> > > > Can you detail the extra complexity (or point me to the
> email
> > I
> > > > >> > > > missed?) - which Apis are required?
> > > > >> > > > As far as I can tell, clients can already find the
> controller
> > > from
> > > > >> > > > metadata. I'm a bit more concerned about controller load.
> > > > >> > > >
> > > > >> > > > >
> > > > >> > > > > * While I like the idea of forcing kerberos auth for
> > renwal, I
> > > > >> think
> > > > >> > > > > it mixes the transport layer the the request content in a
> > > pretty
> > > > >> ugly
> > > > >> > > > > way. Perhaps limiting renewer to non-owner is better.
> > > > >> > > > >
> > > > >> > > > > *I feel this is a necessary evil. While this will make the
> > > kafka
> > > > >> code
> > > > >> > > > > pretty straight forward , forcing  renewer to non-owner
> > pushes
> > > > >> the code
> > > > >> > > > > ugliness to client and makes it even harder to
> integrate.  *
> > > > >> > > >
> > > > >> > > > As mentioned before, I don't think the "renewal by other"
> > > approach
> > > > >> is
> > > > >> > > > that ugly for the clients we expect to use delegation tokens
> > > since
> > > > >> > > > they will have an app-master of some sort who requested the
> > > token
> > > > to
> > > > >> > > > begin with.
> > > > >> > > >
> > > > >> > > > >
> > > > >> > > > > The response for my question on how multiple identities
> will
> > > be
> > > > >> > > > > handled wasn't super clear to me - AFAIK, we don't
> > > authenticate
> > > > >> each
> > > > >> > > > > request, we authenticate connections.
> > > > >> > > > >
> > > > >> > > > > *We authenticate connections, and only when they are being
> > > > >> established.
> > > > >> > > > Let
> > > > >> > > > > me try to phrase this as a question, in absence of
> > delegation
> > > > >> tokens if
> > > > >> > > > we
> > > > >> > > > > had to support the use case using user TGT's how would we
> do
> > > it?
> > > > >> My
> > > > >> > > point
> > > > >> > > > > was it would be no different with delegation tokens. The
> use
> > > > case
> > > > >> you
> > > > >> > > are
> > > > >> > > > > describing seems more like impersonation.*
> > > > >> > > >
> > > > >> > > > Yeah, I thought that one of the things that delegation
> tokens
> > > > >> handled.
> > > > >> > > > Maybe I got it wrong :)
> > > > >> > > >
> > > > >> > > > Thanks for the detailed answers.
> > > > >> > > >
> > > > >> > > > Gwen
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > > Thanks
> > > > >> > > > > Parth
> > > > >> > > > >
> > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <
> > > > gwen@confluent.io
> > > > >> >
> > > > >> > > > wrote:
> > > > >> > > > >
> > > > >> > > > >> Hi Parth and Harsha,
> > > > >> > > > >>
> > > > >> > > > >> Few more comments:
> > > > >> > > > >>
> > > > >> > > > >> * The API / RequestResponse section doesn't seem to have
> > good
> > > > >> > > > >> description of the changes to the Kafka Protocol. Sounds
> > like
> > > > >> you are
> > > > >> > > > >> proposing new DelegationTokenRequest and
> RenewTokenRequest
> > > (and
> > > > >> > > > >> matching responses), without detailing the contents of
> the
> > > > >> requests
> > > > >> > > > >> and responses? Or rather, you show the class interface,
> but
> > > not
> > > > >> the
> > > > >> > > > >> underlying protocol. This could be seen as an
> > implementation
> > > > >> detail,
> > > > >> > > > >> but since the binary protocol is what we provide to
> > non-Java
> > > > >> clients,
> > > > >> > > > >> we need to show the changes there.
> > > > >> > > > >>
> > > > >> > > > >> * getDelegationToken sounds like
> > > delegationTokenRequestHandler?
> > > > >> Is it
> > > > >> > > > >> planned to be part of KafkaApi? or Client? Its unclear...
> > > > >> > > > >>
> > > > >> > > > >> * I want to emphasize that even though delegation tokens
> > are
> > > a
> > > > >> Hadoop
> > > > >> > > > >> innovation, I feel very strongly about not adding
> > dependency
> > > on
> > > > >> Hadoop
> > > > >> > > > >> when implementing delegation tokens for Kafka. The KIP
> > > doesn't
> > > > >> imply
> > > > >> > > > >> such dependency, but if you can clarify...
> > > > >> > > > >>
> > > > >> > > > >> * Can we get delegation token at any time after
> > > authenticating?
> > > > >> only
> > > > >> > > > >> immediately after?
> > > > >> > > > >>
> > > > >> > > > >> * My understanding is that tokens will propagate via ZK
> but
> > > > >> without
> > > > >> > > > >> additional changes to UpdateMetadata protocol, correct?
> > > Clients
> > > > >> > > > >> currently don't retry on SASL auth failure (IIRC), but
> > since
> > > > the
> > > > >> > > > >> tokens propagate between brokers asynch, we will need to
> > > retry
> > > > a
> > > > >> bit
> > > > >> > > > >> to avoid clients failing auth due to timing issues.
> > > > >> > > > >>
> > > > >> > > > >> * Strongly agreeing on clients not touching ZK directly
> :)
> > > > >> > > > >>
> > > > >> > > > >> * I liked Ashish's suggestion of having just the
> controller
> > > > >> issue the
> > > > >> > > > >> delegation tokens, to avoid syncing a shared secret. Not
> > sure
> > > > if
> > > > >> we
> > > > >> > > > >> want to continue the discussion here or on the wiki. I
> > think
> > > > >> that we
> > > > >> > > > >> can decouple the problem of "token distribution" from
> > "shared
> > > > >> secret
> > > > >> > > > >> distribution" and use the controller as the only token
> > > > generator
> > > > >> to
> > > > >> > > > >> solve the second issue, while still using ZK async to
> > > > distribute
> > > > >> > > > >> tokens.
> > > > >> > > > >>
> > > > >> > > > >> * I am also uncomfortable with infinite lifetime of
> tokens
> > > (and
> > > > >> hoped
> > > > >> > > > >> to hear from others in the community) - but having the
> > option
> > > > and
> > > > >> > > > >> documenting it as less secure, allows users to configure
> > > their
> > > > >> system
> > > > >> > > > >> as the wish.
> > > > >> > > > >>
> > > > >> > > > >> * While I like the idea of forcing kerberos auth for
> > renwal,
> > > I
> > > > >> think
> > > > >> > > > >> it mixes the transport layer the the request content in a
> > > > pretty
> > > > >> ugly
> > > > >> > > > >> way. Perhaps limiting renewer to non-owner is better.
> > > > >> > > > >>
> > > > >> > > > >> Things I'd still like to see:
> > > > >> > > > >>
> > > > >> > > > >> * More detailed explanation on what we plan to do for the
> > > java
> > > > >> clients
> > > > >> > > > >> specifically - new configuration? new APIs?
> > > > >> > > > >> The response for my question on how multiple identities
> > will
> > > be
> > > > >> > > > >> handled wasn't super clear to me - AFAIK, we don't
> > > authenticate
> > > > >> each
> > > > >> > > > >> request, we authenticate connections.
> > > > >> > > > >>
> > > > >> > > > >> * Alternatives: Delegation tokens are only used in the
> > Hadoop
> > > > >> > > > >> ecosystem. I'm wondering if there are alternatives in
> other
> > > > >> ecosystems
> > > > >> > > > >> (Mesos? Tachyon? Cassandra?) and whether there are some
> > > > >> advantages
> > > > >> > > > >> there.
> > > > >> > > > >>
> > > > >> > > > >> Gwen
> > > > >> > > > >>
> > > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <kafka@harsha.io
> >
> > > > wrote:
> > > > >> > > > >> > Hi Gwen,
> > > > >> > > > >> >            Can you look at Parth's last reply. Does it
> > > answer
> > > > >> your
> > > > >> > > > >> >            concerns.
> > > > >> > > > >> >
> > > > >> > > > >> > Thanks,
> > > > >> > > > >> > Harsha
> > > > >> > > > >> >
> > > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt
> wrote:
> > > > >> > > > >> >> Thanks for reviewing Gwen. The wiki already has
> details
> > on
> > > > >> token
> > > > >> > > > >> >> expiration
> > > > >> > > > >> >> under token acquisition process
> > > > >> > > > >> >> <
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> > > > >> > > > >> >.
> > > > >> > > > >> >> Current proposal is that tokens will expire based on a
> > > > server
> > > > >> side
> > > > >> > > > >> >> configuration (default 24 hours) unless renewed.
> Renewal
> > > is
> > > > >> only
> > > > >> > > > allowed
> > > > >> > > > >> >> until the max life time of token. Alternatively we
> could
> > > > also
> > > > >> make
> > > > >> > > > that
> > > > >> > > > >> >> an
> > > > >> > > > >> >> optional param and the server side default can serve
> as
> > > the
> > > > >> upper
> > > > >> > > > bound.
> > > > >> > > > >> >>
> > > > >> > > > >> >> To your second point it will be done exactly the same
> > way
> > > we
> > > > >> would
> > > > >> > > > >> >> support
> > > > >> > > > >> >> multiple keytabs. The calling client will have to put
> > the
> > > > >> tokens it
> > > > >> > > > >> wants
> > > > >> > > > >> >> to use in the subject instance and call
> produce/consume
> > > > inside
> > > > >> > > > >> >> subject.doas. Each caller will have to keep track of
> its
> > > own
> > > > >> > > > subject. I
> > > > >> > > > >> >> will have to look at the code to see if we support
> this
> > > > >> feature
> > > > >> > > right
> > > > >> > > > >> now
> > > > >> > > > >> >> but my understanding is delegation token shouldn't
> need
> > > any
> > > > >> special
> > > > >> > > > >> >> treatment as its just another type of Credential in
> the
> > > > >> subject.
> > > > >> > > > >> >>
> > > > >> > > > >> >> I would also like to know what is your opinion about
> > > > infinite
> > > > >> > > renewal
> > > > >> > > > >> (my
> > > > >> > > > >> >> recommendation is to not support this), tokens
> renewing
> > > them
> > > > >> > > self(my
> > > > >> > > > >> >> recommendation is to not support this) and most
> > > importantly
> > > > >> your
> > > > >> > > > choice
> > > > >> > > > >> >> between the alternatives listed on this thread
> > > > >> > > > >> >> <
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >>
> > > >
> > >
> >
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> > > > >> > > > >> >
> > > > >> > > > >> >> ( I am leaning towards the alternative-2 minus
> > controller
> > > > >> > > > distributing
> > > > >> > > > >> >> secret). Thanks again for reviewing.
> > > > >> > > > >> >>
> > > > >> > > > >> >> Thanks
> > > > >> > > > >> >> Parth
> > > > >> > > > >> >>
> > > > >> > > > >> >>
> > > > >> > > > >> >>
> > > > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <
> > > > >> gwen@confluent.io>
> > > > >> > > > wrote:
> > > > >> > > > >> >>
> > > > >> > > > >> >> > Harsha,
> > > > >> > > > >> >> >
> > > > >> > > > >> >> > I was thinking of the Rest Proxy. I didn't see your
> > > design
> > > > >> yet,
> > > > >> > > > but in
> > > > >> > > > >> >> > our proxy, we have a set of producers, which will
> > serve
> > > > >> multiple
> > > > >> > > > users
> > > > >> > > > >> >> > going through the proxy. Since these users will have
> > > > >> different
> > > > >> > > > >> >> > privileges, they'll need to authenticate separately,
> > and
> > > > >> can't
> > > > >> > > > share a
> > > > >> > > > >> >> > token.
> > > > >> > > > >> >> >
> > > > >> > > > >> >> > Am I missing anything?
> > > > >> > > > >> >> >
> > > > >> > > > >> >> > Gwen
> > > > >> > > > >> >> >
> > > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <
> > kafka@harsha.io
> > > >
> > > > >> wrote:
> > > > >> > > > >> >> > > Gwen,
> > > > >> > > > >> >> > >            On your second point. Can you describe
> a
> > > > >> usecase
> > > > >> > > where
> > > > >> > > > >> >> > >            mutliple clients ended up sharing a
> > > producer
> > > > >> and
> > > > >> > > even
> > > > >> > > > if
> > > > >> > > > >> they
> > > > >> > > > >> >> > >            do why can't they not use single token
> > that
> > > > >> producer
> > > > >> > > > >> >> > >            captures. Why would we need multiple
> > > clients
> > > > >> with
> > > > >> > > > >> different
> > > > >> > > > >> >> > >            tokens sharing a single instance of
> > > producer.
> > > > >> Also
> > > > >> > > in
> > > > >> > > > >> this
> > > > >> > > > >> >> > >            case other clients have access all the
> > > tokens
> > > > >> no?
> > > > >> > > > >> >> > >
> > > > >> > > > >> >> > > Thanks,
> > > > >> > > > >> >> > > Harsha
> > > > >> > > > >> >> > >
> > > > >> > > > >> >> > >
> > > > >> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira
> > wrote:
> > > > >> > > > >> >> > >> Sorry for the delay:
> > > > >> > > > >> >> > >>
> > > > >> > > > >> >> > >> Two questions that we didn't see in the wiki:
> > > > >> > > > >> >> > >> 1. Is there an expiration for delegation tokens?
> > > > >> Renewal? How
> > > > >> > > > do we
> > > > >> > > > >> >> > >> revoke them?
> > > > >> > > > >> >> > >> 2. If we want to use delegation tokens for
> "do-as"
> > > > (say,
> > > > >> > > submit
> > > > >> > > > >> Storm
> > > > >> > > > >> >> > >> job as my user), we will need a producer for
> every
> > > job
> > > > >> (we
> > > > >> > > can't
> > > > >> > > > >> share
> > > > >> > > > >> >> > >> them between multiple jobs running on same node),
> > > since
> > > > >> we
> > > > >> > > only
> > > > >> > > > >> >> > >> authenticate when connecting. Is there a plan to
> > > change
> > > > >> this
> > > > >> > > for
> > > > >> > > > >> >> > >> delegation tokens, in order to allow multiple
> users
> > > > with
> > > > >> > > > different
> > > > >> > > > >> >> > >> tokens to share a client?
> > > > >> > > > >> >> > >>
> > > > >> > > > >> >> > >> Gwen
> > > > >> > > > >> >> > >>
> > > > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
> > > > >> > > > >> >> > >> <br...@gmail.com> wrote:
> > > > >> > > > >> >> > >> > Bumping this up one more time, can other
> > committers
> > > > >> review?
> > > > >> > > > >> >> > >> >
> > > > >> > > > >> >> > >> > Thanks
> > > > >> > > > >> >> > >> > Parth
> > > > >> > > > >> >> > >> >
> > > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <
> > > > >> kafka@harsha.io>
> > > > >> > > > wrote:
> > > > >> > > > >> >> > >> >
> > > > >> > > > >> >> > >> >> Parth,
> > > > >> > > > >> >> > >> >>           Overall current design looks good to
> > > me. I
> > > > >> am +1
> > > > >> > > on
> > > > >> > > > >> the
> > > > >> > > > >> >> > KIP.
> > > > >> > > > >> >> > >> >>
> > > > >> > > > >> >> > >> >> Gwen , Jun can you review this as well.
> > > > >> > > > >> >> > >> >>
> > > > >> > > > >> >> > >> >> -Harsha
> > > > >> > > > >> >> > >> >>
> > > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth
> > > brahmbhatt
> > > > >> wrote:
> > > > >> > > > >> >> > >> >> > Thanks for review Jitendra.
> > > > >> > > > >> >> > >> >> >
> > > > >> > > > >> >> > >> >> > I don't like the idea of infinite lifetime
> > but I
> > > > >> see the
> > > > >> > > > >> Streaming
> > > > >> > > > >> >> > use
> > > > >> > > > >> >> > >> >> > case. Even for Streaming use case I was
> hoping
> > > > >> there will
> > > > >> > > > be
> > > > >> > > > >> some
> > > > >> > > > >> >> > notion
> > > > >> > > > >> >> > >> >> > of
> > > > >> > > > >> >> > >> >> > master/driver that can get new delegation
> > tokens
> > > > at
> > > > >> fixed
> > > > >> > > > >> interval
> > > > >> > > > >> >> > and
> > > > >> > > > >> >> > >> >> > distribute to workers. If that is not the
> case
> > > for
> > > > >> we can
> > > > >> > > > >> discuss
> > > > >> > > > >> >> > >> >> > delegation tokens renewing them self and the
> > > > >> security
> > > > >> > > > >> implications
> > > > >> > > > >> >> > of the
> > > > >> > > > >> >> > >> >> > same.
> > > > >> > > > >> >> > >> >> >
> > > > >> > > > >> >> > >> >> > I did not want clients to fetch tokens from
> > > > >> zookeeper,
> > > > >> > > > >> overall I
> > > > >> > > > >> >> > think
> > > > >> > > > >> >> > >> >> > its
> > > > >> > > > >> >> > >> >> > better if clients don't rely on our metadata
> > > store
> > > > >> and I
> > > > >> > > > >> think we
> > > > >> > > > >> >> > are
> > > > >> > > > >> >> > >> >> > moving in that direction with all the KIP-4
> > > > >> improvements.
> > > > >> > > > I
> > > > >> > > > >> chose
> > > > >> > > > >> >> > >> >> > zookeeper as in this case the client will
> > still
> > > > >> just talk
> > > > >> > > > to
> > > > >> > > > >> >> > broker , its
> > > > >> > > > >> >> > >> >> > the brokers that will use zookeeper which we
> > > > >> already do
> > > > >> > > > for a
> > > > >> > > > >> lot
> > > > >> > > > >> >> > of
> > > > >> > > > >> >> > >> >> > other
> > > > >> > > > >> >> > >> >> > usecases + ease of development + and the
> > ability
> > > > so
> > > > >> > > tokens
> > > > >> > > > >> will
> > > > >> > > > >> >> > survive
> > > > >> > > > >> >> > >> >> > even a rolling restart/cluster failure. if a
> > > > >> majority
> > > > >> > > > agrees
> > > > >> > > > >> the
> > > > >> > > > >> >> > added
> > > > >> > > > >> >> > >> >> > complexity to have controller forwarding
> keys
> > to
> > > > all
> > > > >> > > > broker is
> > > > >> > > > >> >> > justified
> > > > >> > > > >> >> > >> >> > as
> > > > >> > > > >> >> > >> >> > it provides tighter security , I am fine
> with
> > > that
> > > > >> option
> > > > >> > > > too.
> > > > >> > > > >> >> > >> >> >
> > > > >> > > > >> >> > >> >> > Given zookeeper does not support SSL we can
> > not
> > > > >> store
> > > > >> > > > master
> > > > >> > > > >> keys
> > > > >> > > > >> >> > in
> > > > >> > > > >> >> > >> >> > zookeeper as master keys will be exposed on
> > > wire.
> > > > To
> > > > >> > > > support
> > > > >> > > > >> >> > rotation
> > > > >> > > > >> >> > >> >> > without affecting current clients is
> > something I
> > > > >> need to
> > > > >> > > > put
> > > > >> > > > >> more
> > > > >> > > > >> >> > thought
> > > > >> > > > >> >> > >> >> > in. My current proposal assumes the rotation
> > > will
> > > > >> > > > invalidate
> > > > >> > > > >> all
> > > > >> > > > >> >> > current
> > > > >> > > > >> >> > >> >> > tokens.
> > > > >> > > > >> >> > >> >> >
> > > > >> > > > >> >> > >> >> > I request committers to also review and post
> > > their
> > > > >> > > comments
> > > > >> > > > >> so we
> > > > >> > > > >> >> > can
> > > > >> > > > >> >> > >> >> > make
> > > > >> > > > >> >> > >> >> > progress on this KIP.
> > > > >> > > > >> >> > >> >> >
> > > > >> > > > >> >> > >> >> > Thanks
> > > > >> > > > >> >> > >> >> > Parth
> > > > >> > > > >> >> > >> >> >
> > > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish
> Singh
> > <
> > > > >> > > > >> asingh@cloudera.com
> > > > >> > > > >> >> > >
> > > > >> > > > >> >> > >> >> > wrote:
> > > > >> > > > >> >> > >> >> >
> > > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <
> > > > >> > > > kafka@harsha.io>
> > > > >> > > > >> >> > wrote:
> > > > >> > > > >> >> > >> >> > >
> > > > >> > > > >> >> > >> >> > > > Unifying the two discussion threads on
> > this
> > > > KIP.
> > > > >> > > > >> >> > >> >> > > >
> > > > >> > > > >> >> > >> >> > > > Here is the response from Jitendra
> > > > >> > > > >> >> > >> >> > > >
> > > > >> > > > >> >> > >> >> > > > "The need for a large number of clients
> > that
> > > > are
> > > > >> > > > running
> > > > >> > > > >> all
> > > > >> > > > >> >> > over the
> > > > >> > > > >> >> > >> >> > > > cluster that authenticate with Kafka
> > > brokers,
> > > > >> is very
> > > > >> > > > >> similar
> > > > >> > > > >> >> > to the
> > > > >> > > > >> >> > >> >> > > > Hadoop use case of large number of tasks
> > > > running
> > > > >> > > across
> > > > >> > > > >> the
> > > > >> > > > >> >> > cluster
> > > > >> > > > >> >> > >> >> that
> > > > >> > > > >> >> > >> >> > > > need authentication to Hdfs Namenode.
> > > > >> Therefore, the
> > > > >> > > > >> >> > delegation token
> > > > >> > > > >> >> > >> >> > > > approach does seem like a good fit for
> > this
> > > > use
> > > > >> case
> > > > >> > > > as we
> > > > >> > > > >> >> > have seen
> > > > >> > > > >> >> > >> >> it
> > > > >> > > > >> >> > >> >> > > > working at large scale in HDFS and YARN.
> > > > >> > > > >> >> > >> >> > > >
> > > > >> > > > >> >> > >> >> > > >   The proposed design is very much
> inline
> > > with
> > > > >> Hadoop
> > > > >> > > > >> >> > approach. A few
> > > > >> > > > >> >> > >> >> > > >   comments:
> > > > >> > > > >> >> > >> >> > > >
> > > > >> > > > >> >> > >> >> > > > 1) Why do you guys want to allow
> infinite
> > > > >> renewable
> > > > >> > > > >> lifetime
> > > > >> > > > >> >> > for a
> > > > >> > > > >> >> > >> >> > > > token? HDFS restricts a token to a max
> > life
> > > > time
> > > > >> > > > (default
> > > > >> > > > >> 7
> > > > >> > > > >> >> > days).  A
> > > > >> > > > >> >> > >> >> > > > token's vulnerability is believed to
> > > increase
> > > > >> with
> > > > >> > > > time.
> > > > >> > > > >> >> > >> >> > > >
> > > > >> > > > >> >> > >> >> > > I agree that having infinite lifetime
> might
> > > not
> > > > >> be the
> > > > >> > > > best
> > > > >> > > > >> idea.
> > > > >> > > > >> >> > >> >> > >
> > > > >> > > > >> >> > >> >> > > >
> > > > >> > > > >> >> > >> >> > > > 2) As I understand the tokens are stored
> > in
> > > > >> zookeeper
> > > > >> > > > as
> > > > >> > > > >> well,
> > > > >> > > > >> >> > and
> > > > >> > > > >> >> > >> >> can
> > > > >> > > > >> >> > >> >> > > > be updated there. This is clever as it
> can
> > > > allow
> > > > >> > > > >> replacing the
> > > > >> > > > >> >> > tokens
> > > > >> > > > >> >> > >> >> > > > once they run out of max life time, and
> > > > clients
> > > > >> can
> > > > >> > > > >> download
> > > > >> > > > >> >> > new
> > > > >> > > > >> >> > >> >> tokens
> > > > >> > > > >> >> > >> >> > > > from zookeeper. It shouldn't be a big
> load
> > > on
> > > > >> > > zookeeper
> > > > >> > > > >> as a
> > > > >> > > > >> >> > client
> > > > >> > > > >> >> > >> >> will
> > > > >> > > > >> >> > >> >> > > > need to get a new token once in several
> > > days.
> > > > >> In this
> > > > >> > > > >> approach
> > > > >> > > > >> >> > you
> > > > >> > > > >> >> > >> >> don't
> > > > >> > > > >> >> > >> >> > > > need infinite lifetime on the token even
> > for
> > > > >> long
> > > > >> > > > running
> > > > >> > > > >> >> > clients.
> > > > >> > > > >> >> > >> >> > > >
> > > > >> > > > >> >> > >> >> > > > 3) The token password are generated
> using
> > a
> > > > >> master
> > > > >> > > key.
> > > > >> > > > >> The
> > > > >> > > > >> >> > master
> > > > >> > > > >> >> > >> >> key
> > > > >> > > > >> >> > >> >> > > > should also be periodically changed. In
> > > > Hadoop,
> > > > >> the
> > > > >> > > > >> default
> > > > >> > > > >> >> > renewal
> > > > >> > > > >> >> > >> >> > > > period is 1 day.?
> > > > >> > > > >> >> > >> >> > > >
> > > > >> > > > >> >> > >> >> > > IIUC, this will require brokers
> maintaining
> > a
> > > > >> list of X
> > > > >> > > > most
> > > > >> > > > >> >> > recent
> > > > >> > > > >> >> > >> >> master
> > > > >> > > > >> >> > >> >> > > keys. This list will have to be persisted
> > > > >> somewhere, as
> > > > >> > > > if a
> > > > >> > > > >> >> > broker
> > > > >> > > > >> >> > >> >> goes
> > > > >> > > > >> >> > >> >> > > down it will have to get that list again
> and
> > > > >> storing
> > > > >> > > > master
> > > > >> > > > >> keys
> > > > >> > > > >> >> > on ZK
> > > > >> > > > >> >> > >> >> is
> > > > >> > > > >> >> > >> >> > > not the best idea. However, if a broker
> goes
> > > > down
> > > > >> then
> > > > >> > > we
> > > > >> > > > >> have
> > > > >> > > > >> >> > much
> > > > >> > > > >> >> > >> >> bigger
> > > > >> > > > >> >> > >> >> > > issue to deal with and client can always
> > > > >> > > re-authenticate
> > > > >> > > > is
> > > > >> > > > >> such
> > > > >> > > > >> >> > >> >> events.
> > > > >> > > > >> >> > >> >> > >
> > > > >> > > > >> >> > >> >> > > Did you happen to take a look at other
> > > > >> alternatives
> > > > >> > > this
> > > > >> > > > >> list has
> > > > >> > > > >> >> > >> >> > > suggested?
> > > > >> > > > >> >> > >> >> > >
> > > > >> > > > >> >> > >> >> > > >
> > > > >> > > > >> >> > >> >> > > > Thanks for a thorough proposal, great
> > work!"
> > > > >> > > > >> >> > >> >> > > >
> > > > >> > > > >> >> > >> >> > > >
> > > > >> > > > >> >> > >> >> > > >
> > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen
> > > Shapira
> > > > >> wrote:
> > > > >> > > > >> >> > >> >> > > > > Makes sense to me. Thanks!
> > > > >> > > > >> >> > >> >> > > > >
> > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM,
> Harsha <
> > > > >> > > > kafka@harsha.io
> > > > >> > > > >> >
> > > > >> > > > >> >> > wrote:
> > > > >> > > > >> >> > >> >> > > > > > It doesn't need any release vehicle
> > but
> > > > >> still the
> > > > >> > > > >> work can
> > > > >> > > > >> >> > move
> > > > >> > > > >> >> > >> >> > > > forward.
> > > > >> > > > >> >> > >> >> > > > > > If anyone is interested in the KIP
> > > please
> > > > >> do the
> > > > >> > > > >> review and
> > > > >> > > > >> >> > >> >> provide
> > > > >> > > > >> >> > >> >> > > the
> > > > >> > > > >> >> > >> >> > > > > > comments.
> > > > >> > > > >> >> > >> >> > > > > >
> > > > >> > > > >> >> > >> >> > > > > > -Harsha
> > > > >> > > > >> >> > >> >> > > > > >
> > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM,
> > Ismael
> > > > >> Juma
> > > > >> > > > wrote:
> > > > >> > > > >> >> > >> >> > > > > >> I agree that it would be good to
> have
> > > > more
> > > > >> time
> > > > >> > > to
> > > > >> > > > >> review
> > > > >> > > > >> >> > and
> > > > >> > > > >> >> > >> >> > > discuss
> > > > >> > > > >> >> > >> >> > > > > >> KIP-48.
> > > > >> > > > >> >> > >> >> > > > > >>
> > > > >> > > > >> >> > >> >> > > > > >> Ismael
> > > > >> > > > >> >> > >> >> > > > > >>
> > > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM,
> Gwen
> > > > >> Shapira <
> > > > >> > > > >> >> > >> >> gwen@confluent.io>
> > > > >> > > > >> >> > >> >> > > > wrote:
> > > > >> > > > >> >> > >> >> > > > > >>
> > > > >> > > > >> >> > >> >> > > > > >> > Hi Team,
> > > > >> > > > >> >> > >> >> > > > > >> >
> > > > >> > > > >> >> > >> >> > > > > >> > Since KIP-48 depends on KIP-43,
> > which
> > > > is
> > > > >> > > > already a
> > > > >> > > > >> bit
> > > > >> > > > >> >> > of a
> > > > >> > > > >> >> > >> >> risk
> > > > >> > > > >> >> > >> >> > > for
> > > > >> > > > >> >> > >> >> > > > > >> > the next release - any chance we
> > can
> > > > >> delay
> > > > >> > > > >> delegation
> > > > >> > > > >> >> > tokens
> > > > >> > > > >> >> > >> >> to
> > > > >> > > > >> >> > >> >> > > > Kafka
> > > > >> > > > >> >> > >> >> > > > > >> > 0.10.1?
> > > > >> > > > >> >> > >> >> > > > > >> > With the community working on a
> > > release
> > > > >> every
> > > > >> > > 3
> > > > >> > > > >> month,
> > > > >> > > > >> >> > this
> > > > >> > > > >> >> > >> >> is not
> > > > >> > > > >> >> > >> >> > > > a huge
> > > > >> > > > >> >> > >> >> > > > > >> > delay.
> > > > >> > > > >> >> > >> >> > > > > >> >
> > > > >> > > > >> >> > >> >> > > > > >> > Gwen
> > > > >> > > > >> >> > >> >> > > > > >> >
> > > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM,
> > > Ashish
> > > > >> Singh
> > > > >> > > <
> > > > >> > > > >> >> > >> >> > > asingh@cloudera.com>
> > > > >> > > > >> >> > >> >> > > > wrote:
> > > > >> > > > >> >> > >> >> > > > > >> > > Parth,
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > > Thanks again for the awesome
> > write
> > > > up.
> > > > >> > > > Following
> > > > >> > > > >> our
> > > > >> > > > >> >> > >> >> discussion
> > > > >> > > > >> >> > >> >> > > > from the
> > > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I think it will be easier
> > to
> > > > >> compare
> > > > >> > > > >> various
> > > > >> > > > >> >> > >> >> alternatives
> > > > >> > > > >> >> > >> >> > > > if they
> > > > >> > > > >> >> > >> >> > > > > >> > are
> > > > >> > > > >> >> > >> >> > > > > >> > > listed together. I am stating
> > > below a
> > > > >> few
> > > > >> > > > >> >> > alternatives along
> > > > >> > > > >> >> > >> >> > > with
> > > > >> > > > >> >> > >> >> > > > a the
> > > > >> > > > >> >> > >> >> > > > > >> > > current proposal.
> > > > >> > > > >> >> > >> >> > > > > >> > > (Current proposal) Store
> > Delegation
> > > > >> Token,
> > > > >> > > DT,
> > > > >> > > > >> on ZK.
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates
> with a
> > > > >> broker.
> > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > > authenticated,
> > > > >> it
> > > > >> > > will
> > > > >> > > > >> make a
> > > > >> > > > >> >> > broker
> > > > >> > > > >> >> > >> >> side
> > > > >> > > > >> >> > >> >> > > > call to
> > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker generates a
> > shared
> > > > >> secret
> > > > >> > > > based
> > > > >> > > > >> on
> > > > >> > > > >> >> > >> >> > > HMAC-SHA256(a
> > > > >> > > > >> >> > >> >> > > > > >> > >    Password/Secret shared
> between
> > > all
> > > > >> > > brokers,
> > > > >> > > > >> >> > randomly
> > > > >> > > > >> >> > >> >> > > generated
> > > > >> > > > >> >> > >> >> > > > > >> > tokenId).
> > > > >> > > > >> >> > >> >> > > > > >> > >    4. Broker stores this token
> in
> > > its
> > > > >> in
> > > > >> > > > memory
> > > > >> > > > >> cache.
> > > > >> > > > >> >> > >> >> Broker
> > > > >> > > > >> >> > >> >> > > > also stores
> > > > >> > > > >> >> > >> >> > > > > >> > >    the DelegationToken without
> > the
> > > > >> hmac in
> > > > >> > > the
> > > > >> > > > >> >> > zookeeper.
> > > > >> > > > >> >> > >> >> > > > > >> > >    5. All brokers will have a
> > cache
> > > > >> backed
> > > > >> > > by
> > > > >> > > > >> >> > zookeeper so
> > > > >> > > > >> >> > >> >> they
> > > > >> > > > >> >> > >> >> > > > will all
> > > > >> > > > >> >> > >> >> > > > > >> > >    get notified whenever a new
> > > token
> > > > is
> > > > >> > > > >> generated and
> > > > >> > > > >> >> > they
> > > > >> > > > >> >> > >> >> will
> > > > >> > > > >> >> > >> >> > > > update
> > > > >> > > > >> >> > >> >> > > > > >> > their
> > > > >> > > > >> >> > >> >> > > > > >> > >    local cache whenever token
> > state
> > > > >> changes.
> > > > >> > > > >> >> > >> >> > > > > >> > >    6. Broker returns the token
> to
> > > > >> Client.
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > >    1. Probable race condition,
> > > client
> > > > >> tries
> > > > >> > > to
> > > > >> > > > >> >> > authenticate
> > > > >> > > > >> >> > >> >> with
> > > > >> > > > >> >> > >> >> > > > a broker
> > > > >> > > > >> >> > >> >> > > > > >> > >    that is yet to be updated
> with
> > > the
> > > > >> newly
> > > > >> > > > >> generated
> > > > >> > > > >> >> > DT.
> > > > >> > > > >> >> > >> >> This
> > > > >> > > > >> >> > >> >> > > can
> > > > >> > > > >> >> > >> >> > > > > >> > probably be
> > > > >> > > > >> >> > >> >> > > > > >> > >    dealt with making dtRequest
> > > block
> > > > >> until
> > > > >> > > all
> > > > >> > > > >> >> > brokers have
> > > > >> > > > >> >> > >> >> > > > updated
> > > > >> > > > >> >> > >> >> > > > > >> > their DT
> > > > >> > > > >> >> > >> >> > > > > >> > >    cache. Zk barrier or similar
> > > > >> mechanism
> > > > >> > > can
> > > > >> > > > be
> > > > >> > > > >> used.
> > > > >> > > > >> >> > >> >> However,
> > > > >> > > > >> >> > >> >> > > > all such
> > > > >> > > > >> >> > >> >> > > > > >> > >    mechanisms will increase
> > > > complexity.
> > > > >> > > > >> >> > >> >> > > > > >> > >    2. Using a static secret key
> > > from
> > > > >> config
> > > > >> > > > >> file. Will
> > > > >> > > > >> >> > >> >> require
> > > > >> > > > >> >> > >> >> > > yet
> > > > >> > > > >> >> > >> >> > > > > >> > another
> > > > >> > > > >> >> > >> >> > > > > >> > >    config and uses a static
> > secret
> > > > >> key. It
> > > > >> > > is
> > > > >> > > > >> advised
> > > > >> > > > >> >> > to
> > > > >> > > > >> >> > >> >> rotate
> > > > >> > > > >> >> > >> >> > > > secret
> > > > >> > > > >> >> > >> >> > > > > >> > keys
> > > > >> > > > >> >> > >> >> > > > > >> > >    periodically. This can be
> > > avoided
> > > > >> with
> > > > >> > > > >> controller
> > > > >> > > > >> >> > >> >> generating
> > > > >> > > > >> >> > >> >> > > > > >> > secretKey and
> > > > >> > > > >> >> > >> >> > > > > >> > >    passing to brokers
> > periodically.
> > > > >> However,
> > > > >> > > > >> this will
> > > > >> > > > >> >> > >> >> require
> > > > >> > > > >> >> > >> >> > > > brokers to
> > > > >> > > > >> >> > >> >> > > > > >> > >    maintain certain counts of
> > > > >> secretKeys.
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 1) Have controller
> > > > >> generate
> > > > >> > > > >> delegation
> > > > >> > > > >> >> > token.
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates
> with a
> > > > >> broker.
> > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > > authenticated,
> > > > >> it
> > > > >> > > will
> > > > >> > > > >> make a
> > > > >> > > > >> >> > broker
> > > > >> > > > >> >> > >> >> side
> > > > >> > > > >> >> > >> >> > > > call to
> > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > > > >> > > > >> >> > >> >> > > > > >> > >    3. Broker forwards the
> request
> > > to
> > > > >> > > > controller.
> > > > >> > > > >> >> > >> >> > > > > >> > >    4. Controller generates a DT
> > and
> > > > >> > > broadcasts
> > > > >> > > > >> to all
> > > > >> > > > >> >> > >> >> brokers.
> > > > >> > > > >> >> > >> >> > > > > >> > >    5. Broker stores this token
> in
> > > its
> > > > >> memory
> > > > >> > > > >> cache.
> > > > >> > > > >> >> > >> >> > > > > >> > >    6. Controller responds to
> > > broker’s
> > > > >> DT
> > > > >> > > req.
> > > > >> > > > >> >> > >> >> > > > > >> > >    7. Broker returns the token
> to
> > > > >> Client.
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > >    1. We will have to add new
> > APIs
> > > to
> > > > >> > > support
> > > > >> > > > >> >> > controller
> > > > >> > > > >> >> > >> >> pushing
> > > > >> > > > >> >> > >> >> > > > tokens
> > > > >> > > > >> >> > >> >> > > > > >> > to
> > > > >> > > > >> >> > >> >> > > > > >> > >    brokers on top of the
> minimal
> > > APIs
> > > > >> that
> > > > >> > > are
> > > > >> > > > >> >> > currently
> > > > >> > > > >> >> > >> >> > > proposed.
> > > > >> > > > >> >> > >> >> > > > > >> > >    2. We will also have to add
> > APIs
> > > > to
> > > > >> > > support
> > > > >> > > > >> the
> > > > >> > > > >> >> > >> >> bootstrapping
> > > > >> > > > >> >> > >> >> > > > case,
> > > > >> > > > >> >> > >> >> > > > > >> > i.e,
> > > > >> > > > >> >> > >> >> > > > > >> > >    when a new broker comes up
> it
> > > will
> > > > >> have
> > > > >> > > to
> > > > >> > > > >> get all
> > > > >> > > > >> >> > >> >> delegation
> > > > >> > > > >> >> > >> >> > > > tokens
> > > > >> > > > >> >> > >> >> > > > > >> > from
> > > > >> > > > >> >> > >> >> > > > > >> > >    the controller.
> > > > >> > > > >> >> > >> >> > > > > >> > >    3. In catastrophic failures
> > > where
> > > > >> all
> > > > >> > > > brokers
> > > > >> > > > >> go
> > > > >> > > > >> >> > down,
> > > > >> > > > >> >> > >> >> the
> > > > >> > > > >> >> > >> >> > > > tokens will
> > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if servers are
> > > > >> restarted as
> > > > >> > > > >> tokens
> > > > >> > > > >> >> > are not
> > > > >> > > > >> >> > >> >> > > > persisted
> > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > > >> > > > >> >> > >> >> > > > > >> > >    If this happens, then there
> > are
> > > > more
> > > > >> > > > important
> > > > >> > > > >> >> > things to
> > > > >> > > > >> >> > >> >> > > worry
> > > > >> > > > >> >> > >> >> > > > about
> > > > >> > > > >> >> > >> >> > > > > >> > and
> > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
> > > > >> re-authenticate.
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 2) Do not
> distribute
> > > DT
> > > > to
> > > > >> > > other
> > > > >> > > > >> brokers
> > > > >> > > > >> >> > at
> > > > >> > > > >> >> > >> >> all.
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates
> with a
> > > > >> broker.
> > > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > > authenticated,
> > > > >> it
> > > > >> > > will
> > > > >> > > > >> make a
> > > > >> > > > >> >> > broker
> > > > >> > > > >> >> > >> >> side
> > > > >> > > > >> >> > >> >> > > > call to
> > > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker generates DT
> of
> > > > form,
> > > > >> > > [hmac +
> > > > >> > > > >> (owner,
> > > > >> > > > >> >> > >> >> renewer,
> > > > >> > > > >> >> > >> >> > > > > >> > >    maxLifeTime, id, hmac,
> > > > >> expirationTime)]
> > > > >> > > and
> > > > >> > > > >> passes
> > > > >> > > > >> >> > back
> > > > >> > > > >> >> > >> >> this
> > > > >> > > > >> >> > >> >> > > > DT to
> > > > >> > > > >> >> > >> >> > > > > >> > client.
> > > > >> > > > >> >> > >> >> > > > > >> > >    hmac is generated via
> > > > >> {HMAC-SHA256(owner,
> > > > >> > > > >> renewer,
> > > > >> > > > >> >> > >> >> > > > maxLifeTime, id,
> > > > >> > > > >> >> > >> >> > > > > >> > hmac,
> > > > >> > > > >> >> > >> >> > > > > >> > >    expirationTime) using
> > > SecretKey}.
> > > > >> Note
> > > > >> > > that
> > > > >> > > > >> all
> > > > >> > > > >> >> > brokers
> > > > >> > > > >> >> > >> >> have
> > > > >> > > > >> >> > >> >> > > > this
> > > > >> > > > >> >> > >> >> > > > > >> > SecretKey.
> > > > >> > > > >> >> > >> >> > > > > >> > >    4. Client then goes to any
> > > broker
> > > > >> and to
> > > > >> > > > >> >> > authenticate
> > > > >> > > > >> >> > >> >> sends
> > > > >> > > > >> >> > >> >> > > > the DT.
> > > > >> > > > >> >> > >> >> > > > > >> > >    Broker recalculates hmac
> using
> > > > >> (owner,
> > > > >> > > > >> renewer,
> > > > >> > > > >> >> > >> >> maxLifeTime,
> > > > >> > > > >> >> > >> >> > > > id, hmac,
> > > > >> > > > >> >> > >> >> > > > > >> > >    expirationTime) info from DT
> > and
> > > > its
> > > > >> > > > >> SecretKey. If
> > > > >> > > > >> >> > it
> > > > >> > > > >> >> > >> >> matches
> > > > >> > > > >> >> > >> >> > > > with
> > > > >> > > > >> >> > >> >> > > > > >> > hmac of
> > > > >> > > > >> >> > >> >> > > > > >> > >    DT, client is authenticated.
> > > Yes,
> > > > >> it will
> > > > >> > > > do
> > > > >> > > > >> other
> > > > >> > > > >> >> > >> >> obvious
> > > > >> > > > >> >> > >> >> > > > checks of
> > > > >> > > > >> >> > >> >> > > > > >> > >    timestamp expiry and such.
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > > Note that secret key will be
> > > > generated
> > > > >> by
> > > > >> > > > >> controller
> > > > >> > > > >> >> > and
> > > > >> > > > >> >> > >> >> passed
> > > > >> > > > >> >> > >> >> > > to
> > > > >> > > > >> >> > >> >> > > > > >> > brokers
> > > > >> > > > >> >> > >> >> > > > > >> > > periodically.
> > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > >    1. How to delete a DT? Yes,
> > that
> > > > is
> > > > >> a
> > > > >> > > > downside
> > > > >> > > > >> >> > here.
> > > > >> > > > >> >> > >> >> However,
> > > > >> > > > >> >> > >> >> > > > this can
> > > > >> > > > >> >> > >> >> > > > > >> > >    be handled with brokers
> > > > maintaining
> > > > >> a
> > > > >> > > > >> blacklist of
> > > > >> > > > >> >> > DTs,
> > > > >> > > > >> >> > >> >> DTs
> > > > >> > > > >> >> > >> >> > > > from this
> > > > >> > > > >> >> > >> >> > > > > >> > list
> > > > >> > > > >> >> > >> >> > > > > >> > >    can be removed after expiry.
> > > > >> > > > >> >> > >> >> > > > > >> > >    2. In catastrophic failures
> > > where
> > > > >> all
> > > > >> > > > brokers
> > > > >> > > > >> go
> > > > >> > > > >> >> > down,
> > > > >> > > > >> >> > >> >> the
> > > > >> > > > >> >> > >> >> > > > tokens will
> > > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if servers are
> > > > >> restarted as
> > > > >> > > > >> tokens
> > > > >> > > > >> >> > are not
> > > > >> > > > >> >> > >> >> > > > persisted
> > > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > > >> > > > >> >> > >> >> > > > > >> > >    If this happens, then there
> > are
> > > > more
> > > > >> > > > important
> > > > >> > > > >> >> > things to
> > > > >> > > > >> >> > >> >> > > worry
> > > > >> > > > >> >> > >> >> > > > about
> > > > >> > > > >> >> > >> >> > > > > >> > and
> > > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
> > > > >> re-authenticate.
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58
> PM,
> > > > Parth
> > > > >> > > > >> Brahmbhatt <
> > > > >> > > > >> >> > >> >> > > > > >> > > pbrahmbhatt@hortonworks.com>
> > > wrote:
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > >> Hi,
> > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > >> > > > >> >> > >> >> > > > > >> > >> I have filed KIP-48 so we can
> > > offer
> > > > >> hadoop
> > > > >> > > > like
> > > > >> > > > >> >> > delegation
> > > > >> > > > >> >> > >> >> > > > tokens in
> > > > >> > > > >> >> > >> >> > > > > >> > >> kafka. You can review the
> design
> > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > >> > > > >> >> > >> >> > > > > >> >
> > > > >> > > > >> >> > >> >> > > >
> > > > >> > > > >> >> > >> >> > >
> > > > >> > > > >> >> > >> >>
> > > > >> > > > >> >> >
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > > > >> > > > >> >> > >> >> > > > > >> > .
> > > > >> > > > >> >> > >> >> > > > > >> > >> This KIP depends on KIP-43 and
> > we
> > > > >> have also
> > > > >> > > > >> >> > discussed an
> > > > >> > > > >> >> > >> >> > > > alternative to
> > > > >> > > > >> >> > >> >> > > > > >> > >> proposed design here<
> > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > >> > > > >> >> > >> >> > > > > >> >
> > > > >> > > > >> >> > >> >> > > >
> > > > >> > > > >> >> > >> >> > >
> > > > >> > > > >> >> > >> >>
> > > > >> > > > >> >> >
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > > > >> > > > >> >> > >> >> > > > > >> > >> >.
> > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > >> > > > >> >> > >> >> > > > > >> > >> Thanks
> > > > >> > > > >> >> > >> >> > > > > >> > >> Parth
> > > > >> > > > >> >> > >> >> > > > > >> > >>
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > > --
> > > > >> > > > >> >> > >> >> > > > > >> > >
> > > > >> > > > >> >> > >> >> > > > > >> > > Regards,
> > > > >> > > > >> >> > >> >> > > > > >> > > Ashish
> > > > >> > > > >> >> > >> >> > > > > >> >
> > > > >> > > > >> >> > >> >> > > >
> > > > >> > > > >> >> > >> >> > >
> > > > >> > > > >> >> > >> >> > >
> > > > >> > > > >> >> > >> >> > >
> > > > >> > > > >> >> > >> >> > > --
> > > > >> > > > >> >> > >> >> > >
> > > > >> > > > >> >> > >> >> > > Regards,
> > > > >> > > > >> >> > >> >> > > Ashish
> > > > >> > > > >> >> > >> >> > >
> > > > >> > > > >> >> > >> >>
> > > > >> > > > >> >> >
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >>
> > > > >>
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Regards,
>
> Rajini
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Rajini Sivaram <ra...@googlemail.com>]

SCRAM (Salted Challenge Response Authentication Mechanism) is a better
mechanism than Digest-MD5. Java doesn't come with a built-in SCRAM
SaslServer or SaslClient, but I will be happy to add support in Kafka since
it would be a useful mechanism to support anyway.
https://tools.ietf.org/html/rfc7677 describes the protocol for
SCRAM-SHA-256.

On Tue, May 24, 2016 at 2:37 AM, Jun Rao <ju...@confluent.io> wrote:

> Parth,
>
> Thanks for the explanation. A couple of more questions.
>
> 110. What does getDelegationTokenAs mean?
>
> 111. What's the typical rate of getting and renewing delegation tokens?
> That may have an impact on whether they should be directed to the
> controller.
>
> Jun
>
> On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt <
> brahmbhatt.parth@gmail.com> wrote:
>
> > Hi Jun,
> >
> > Thanks for reviewing.
> >
> > * We could add a Cluster action to add acls on who can request delegation
> > tokens. I don't see the use case for that yet but down the line when we
> > start supporting getDelegationTokenAs it will be necessary.
> > * Yes we recommend tokens to be only used/distributed over secure
> channels.
> > * Depending on what design we end up choosing Invalidation will be
> > responsibility of every broker or controller.
> > * I am not sure if I documented somewhere that invalidation will directly
> > go through zookeeper but that is not the intent. Invalidation will either
> > be request based or due to expiration. No direct zookeeper interaction
> from
> > any client.
> > * "Broker also stores the DelegationToken without the hmac in the
> > zookeeper." : Sorry about the confusion. The sole purpose of zookeeper in
> > this design is as distribution channel for tokens between all brokers
> and a
> > layer that ensures only tokens that were generated by making a request
> to a
> > broker will be accepted (more on this in second paragraph). The token
> > consists of few elements (owner, renewer, uuid , expiration, hmac) , one
> of
> > which is the finally generated hmac but hmac it self is derivable if you
> > have all the other elements of the token + secret key to generate hmac.
> > Given zookeeper does not provide SSL support we do not want the entire
> > token to be wire transferred to zookeeper as that will be an insecure
> wire
> > transfer. Instead we only store all the other elements of a delegation
> > tokens. Brokers can read these elements and because they also have access
> > to secret key they will be able to generate hmac on their end.
> >
> > One of the alternative proposed is to avoid zookeeper altogether. A
> Client
> > will call broker with required information (owner, renwer, expiration)
> and
> > get back (signed hmac, uuid). Broker won't store this in zookeeper. From
> > this point a client can contact any broker with all the delegation token
> > info (owner, rewner, expiration, hmac, uuid) the borker will regenerate
> the
> > hmac and as long as it matches with hmac presented by client , broker
> will
> > allow the request to authenticate.  Only problem with this approach is if
> > the secret key is compromised any client can now generate random tokens
> and
> > they will still be able to authenticate as any user they like. with
> > zookeeper we guarantee that only tokens acquired via a broker (using some
> > auth scheme other than delegation token) will be accepted. We need to
> > discuss which proposal makes more sense and we can go over it in
> tomorrow's
> > meeting.
> >
> > Also, can you forward the invite to me?
> >
> > Thanks
> > Parth
> >
> >
> >
> > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Thanks for the KIP. A few comments.
> > >
> > > 100. This potentially can be useful for Kafka Connect and Kafka rest
> > proxy
> > > where a worker agent will need to run a task on behalf of a client. We
> > will
> > > likely need to change how those services use Kafka clients
> > > (producer/consumer). Instead of a shared client per worker, we will
> need
> > a
> > > client per user task since the authentication happens at the connection
> > > level. For Kafka Connect, the renewer will be the workers. So, we
> > probably
> > > need to allow multiple renewers. For Kafka rest proxy, the renewer can
> > > probably just be the creator of the token.
> > >
> > > 101. Do we need new acl on who can request delegation tokens?
> > >
> > > 102. Do we recommend people to send delegation tokens in an encrypted
> > > channel?
> > >
> > > 103. Who is responsible for expiring tokens, every broker?
> > >
> > > 104. For invalidating tokens, would it be better to do it in a request
> > > instead of going to ZK directly?
> > >
> > > 105. The terminology of client in the wiki sometimes refers to the end
> > > client and some other times refers to the client using the delegation
> > > tokens. It would be useful to distinguish between the two.
> > >
> > > 106. Could you explain the sentence "Broker also stores the
> > DelegationToken
> > > without the hmac in the zookeeper." a bit more? I thought the
> delegation
> > > token is the hmac.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Hi, Harsha,
> > > >
> > > > Just sent out a KIP meeting invite. We can discuss this in the
> meeting
> > > > tomorrow.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <ka...@harsha.io> wrote:
> > > >
> > > >> Hi All,
> > > >>            Can we have a KIP meeting around this. The KIP is up for
> > > >>            sometime and if there are any questions lets quickly hash
> > out
> > > >>            details.
> > > >>
> > > >> Thanks,
> > > >> Harsha
> > > >>
> > > >> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt wrote:
> > > >> > That is what the hadoop echo system uses so no good reason really.
> > We
> > > >> > could
> > > >> > change it to whatever is the newest recommended standard is.
> > > >> >
> > > >> > Thanks
> > > >> > Parth
> > > >> >
> > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <is...@juma.me.uk>
> > > wrote:
> > > >> >
> > > >> > > Hi Parth,
> > > >> > >
> > > >> > > Thanks for the KIP. I only started reviewing this and may have
> > > >> additional
> > > >> > > questions later. The immediate question that came to mind is our
> > > >> choice of
> > > >> > > "DIGEST-MD5" even though it's marked as OBSOLETE in the IANA
> > > Registry
> > > >> of
> > > >> > > SASL mechanisms and the original RFC (2831) has been moved to
> > > Historic
> > > >> > > status:
> > > >> > >
> > > >> > > https://tools.ietf.org/html/rfc6331
> > > >> > >
> > > http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> > > >> > >
> > > >> > > What is the reasoning behind that choice?
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Ismael
> > > >> > >
> > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <
> gwen@confluent.io
> > >
> > > >> wrote:
> > > >> > >
> > > >> > > > Also comments inline :)
> > > >> > > >
> > > >> > > > > * I want to emphasize that even though delegation tokens
> are a
> > > >> Hadoop
> > > >> > > > > innovation, I feel very strongly about not adding dependency
> > on
> > > >> Hadoop
> > > >> > > > > when implementing delegation tokens for Kafka. The KIP
> doesn't
> > > >> imply
> > > >> > > > > such dependency, but if you can clarify...
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > *No hadoop dependency.*
> > > >> > > >
> > > >> > > > Yay! Just add this to the KIP so no one will read the KIP and
> > > panic
> > > >> > > > three weeks before the next release...
> > > >> > > >
> > > >> > > > > * Can we get delegation token at any time after
> > authenticating?
> > > >> only
> > > >> > > > > immediately after?
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > *As long as you are authenticated you can get delegation
> > tokens.
> > > >> We
> > > >> > > need
> > > >> > > > to
> > > >> > > > > discuss if a client authenticated using delegation token,
> can
> > > also
> > > >> > > > acquire
> > > >> > > > > delegation token again or not. Also there is the question of
> > do
> > > we
> > > >> > > allow
> > > >> > > > > anyone to acquire delegation token or we want specific ACLs
> (I
> > > >> think
> > > >> > > its
> > > >> > > > an
> > > >> > > > > overkill.)*
> > > >> > > >
> > > >> > > > I agree that ACLs is an overkill.
> > > >> > > >
> > > >> > > > I think we are debating two options: Either require Kerberos
> > auth
> > > >> for
> > > >> > > > renewal or require non-owners to renew.
> > > >> > > > I *think* the latter is simpler (it basically require a "job
> > > master"
> > > >> > > > to take responsibility for the renewal, it will have its own
> > > >> identity
> > > >> > > > anyway and I think this is the correct design pattern anyway.
> > For
> > > >> > > > storm, I'd expect Nimbus to coordinate renewals?), but it is
> > hard
> > > to
> > > >> > > > debate simplicity without looking at the code changes
> required.
> > If
> > > >> you
> > > >> > > > have a draft of how the "require Kerberos" will look in Kafka
> > > code,
> > > >> > > > I'll be happy to take a look.
> > > >> > > >
> > > >> > > > > * My understanding is that tokens will propagate via ZK but
> > > >> without
> > > >> > > > > additional changes to UpdateMetadata protocol, correct?
> > Clients
> > > >> > > > > currently don't retry on SASL auth failure (IIRC), but since
> > the
> > > >> > > > > tokens propagate between brokers asynch, we will need to
> > retry a
> > > >> bit
> > > >> > > > > to avoid clients failing auth due to timing issues.
> > > >> > > > >
> > > >> > > > > *I am considering 2 alternatives right now. The current
> > > documented
> > > >> > > > approach
> > > >> > > > > is zookeeper based and it does not require any changes to
> > > >> > > UpdateMetadata
> > > >> > > > > protocol. An alternative approach can remove zookeeper
> > > dependency
> > > >> as
> > > >> > > well
> > > >> > > > > but we can discuss that in KIP discussion call.*
> > > >> > > >
> > > >> > > > Oooh! Sounds interesting. Do you want to ping Jun to arrange a
> > > call?
> > > >> > > >
> > > >> > > > > * I liked Ashish's suggestion of having just the controller
> > > issue
> > > >> the
> > > >> > > > > delegation tokens, to avoid syncing a shared secret. Not
> sure
> > if
> > > >> we
> > > >> > > > > want to continue the discussion here or on the wiki. I think
> > > that
> > > >> we
> > > >> > > > > can decouple the problem of "token distribution" from
> "shared
> > > >> secret
> > > >> > > > > distribution" and use the controller as the only token
> > generator
> > > >> to
> > > >> > > > > solve the second issue, while still using ZK async to
> > distribute
> > > >> > > > > tokens.
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > *As mentioned in the previous Email I am fine with that
> > approach
> > > >> as
> > > >> > > long
> > > >> > > > as
> > > >> > > > > we agree that the extra complexity of adding/updating APIS
> > adds
> > > >> enough
> > > >> > > > > value. The advantage with the controller approach is secret
> > > >> rotation
> > > >> > > can
> > > >> > > > be
> > > >> > > > > automated,frequent and would not require deployment. *
> > > >> > > >
> > > >> > > > Can you detail the extra complexity (or point me to the email
> I
> > > >> > > > missed?) - which Apis are required?
> > > >> > > > As far as I can tell, clients can already find the controller
> > from
> > > >> > > > metadata. I'm a bit more concerned about controller load.
> > > >> > > >
> > > >> > > > >
> > > >> > > > > * While I like the idea of forcing kerberos auth for
> renwal, I
> > > >> think
> > > >> > > > > it mixes the transport layer the the request content in a
> > pretty
> > > >> ugly
> > > >> > > > > way. Perhaps limiting renewer to non-owner is better.
> > > >> > > > >
> > > >> > > > > *I feel this is a necessary evil. While this will make the
> > kafka
> > > >> code
> > > >> > > > > pretty straight forward , forcing  renewer to non-owner
> pushes
> > > >> the code
> > > >> > > > > ugliness to client and makes it even harder to integrate.  *
> > > >> > > >
> > > >> > > > As mentioned before, I don't think the "renewal by other"
> > approach
> > > >> is
> > > >> > > > that ugly for the clients we expect to use delegation tokens
> > since
> > > >> > > > they will have an app-master of some sort who requested the
> > token
> > > to
> > > >> > > > begin with.
> > > >> > > >
> > > >> > > > >
> > > >> > > > > The response for my question on how multiple identities will
> > be
> > > >> > > > > handled wasn't super clear to me - AFAIK, we don't
> > authenticate
> > > >> each
> > > >> > > > > request, we authenticate connections.
> > > >> > > > >
> > > >> > > > > *We authenticate connections, and only when they are being
> > > >> established.
> > > >> > > > Let
> > > >> > > > > me try to phrase this as a question, in absence of
> delegation
> > > >> tokens if
> > > >> > > > we
> > > >> > > > > had to support the use case using user TGT's how would we do
> > it?
> > > >> My
> > > >> > > point
> > > >> > > > > was it would be no different with delegation tokens. The use
> > > case
> > > >> you
> > > >> > > are
> > > >> > > > > describing seems more like impersonation.*
> > > >> > > >
> > > >> > > > Yeah, I thought that one of the things that delegation tokens
> > > >> handled.
> > > >> > > > Maybe I got it wrong :)
> > > >> > > >
> > > >> > > > Thanks for the detailed answers.
> > > >> > > >
> > > >> > > > Gwen
> > > >> > > >
> > > >> > > >
> > > >> > > > > Thanks
> > > >> > > > > Parth
> > > >> > > > >
> > > >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <
> > > gwen@confluent.io
> > > >> >
> > > >> > > > wrote:
> > > >> > > > >
> > > >> > > > >> Hi Parth and Harsha,
> > > >> > > > >>
> > > >> > > > >> Few more comments:
> > > >> > > > >>
> > > >> > > > >> * The API / RequestResponse section doesn't seem to have
> good
> > > >> > > > >> description of the changes to the Kafka Protocol. Sounds
> like
> > > >> you are
> > > >> > > > >> proposing new DelegationTokenRequest and RenewTokenRequest
> > (and
> > > >> > > > >> matching responses), without detailing the contents of the
> > > >> requests
> > > >> > > > >> and responses? Or rather, you show the class interface, but
> > not
> > > >> the
> > > >> > > > >> underlying protocol. This could be seen as an
> implementation
> > > >> detail,
> > > >> > > > >> but since the binary protocol is what we provide to
> non-Java
> > > >> clients,
> > > >> > > > >> we need to show the changes there.
> > > >> > > > >>
> > > >> > > > >> * getDelegationToken sounds like
> > delegationTokenRequestHandler?
> > > >> Is it
> > > >> > > > >> planned to be part of KafkaApi? or Client? Its unclear...
> > > >> > > > >>
> > > >> > > > >> * I want to emphasize that even though delegation tokens
> are
> > a
> > > >> Hadoop
> > > >> > > > >> innovation, I feel very strongly about not adding
> dependency
> > on
> > > >> Hadoop
> > > >> > > > >> when implementing delegation tokens for Kafka. The KIP
> > doesn't
> > > >> imply
> > > >> > > > >> such dependency, but if you can clarify...
> > > >> > > > >>
> > > >> > > > >> * Can we get delegation token at any time after
> > authenticating?
> > > >> only
> > > >> > > > >> immediately after?
> > > >> > > > >>
> > > >> > > > >> * My understanding is that tokens will propagate via ZK but
> > > >> without
> > > >> > > > >> additional changes to UpdateMetadata protocol, correct?
> > Clients
> > > >> > > > >> currently don't retry on SASL auth failure (IIRC), but
> since
> > > the
> > > >> > > > >> tokens propagate between brokers asynch, we will need to
> > retry
> > > a
> > > >> bit
> > > >> > > > >> to avoid clients failing auth due to timing issues.
> > > >> > > > >>
> > > >> > > > >> * Strongly agreeing on clients not touching ZK directly :)
> > > >> > > > >>
> > > >> > > > >> * I liked Ashish's suggestion of having just the controller
> > > >> issue the
> > > >> > > > >> delegation tokens, to avoid syncing a shared secret. Not
> sure
> > > if
> > > >> we
> > > >> > > > >> want to continue the discussion here or on the wiki. I
> think
> > > >> that we
> > > >> > > > >> can decouple the problem of "token distribution" from
> "shared
> > > >> secret
> > > >> > > > >> distribution" and use the controller as the only token
> > > generator
> > > >> to
> > > >> > > > >> solve the second issue, while still using ZK async to
> > > distribute
> > > >> > > > >> tokens.
> > > >> > > > >>
> > > >> > > > >> * I am also uncomfortable with infinite lifetime of tokens
> > (and
> > > >> hoped
> > > >> > > > >> to hear from others in the community) - but having the
> option
> > > and
> > > >> > > > >> documenting it as less secure, allows users to configure
> > their
> > > >> system
> > > >> > > > >> as the wish.
> > > >> > > > >>
> > > >> > > > >> * While I like the idea of forcing kerberos auth for
> renwal,
> > I
> > > >> think
> > > >> > > > >> it mixes the transport layer the the request content in a
> > > pretty
> > > >> ugly
> > > >> > > > >> way. Perhaps limiting renewer to non-owner is better.
> > > >> > > > >>
> > > >> > > > >> Things I'd still like to see:
> > > >> > > > >>
> > > >> > > > >> * More detailed explanation on what we plan to do for the
> > java
> > > >> clients
> > > >> > > > >> specifically - new configuration? new APIs?
> > > >> > > > >> The response for my question on how multiple identities
> will
> > be
> > > >> > > > >> handled wasn't super clear to me - AFAIK, we don't
> > authenticate
> > > >> each
> > > >> > > > >> request, we authenticate connections.
> > > >> > > > >>
> > > >> > > > >> * Alternatives: Delegation tokens are only used in the
> Hadoop
> > > >> > > > >> ecosystem. I'm wondering if there are alternatives in other
> > > >> ecosystems
> > > >> > > > >> (Mesos? Tachyon? Cassandra?) and whether there are some
> > > >> advantages
> > > >> > > > >> there.
> > > >> > > > >>
> > > >> > > > >> Gwen
> > > >> > > > >>
> > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <ka...@harsha.io>
> > > wrote:
> > > >> > > > >> > Hi Gwen,
> > > >> > > > >> >            Can you look at Parth's last reply. Does it
> > answer
> > > >> your
> > > >> > > > >> >            concerns.
> > > >> > > > >> >
> > > >> > > > >> > Thanks,
> > > >> > > > >> > Harsha
> > > >> > > > >> >
> > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt wrote:
> > > >> > > > >> >> Thanks for reviewing Gwen. The wiki already has details
> on
> > > >> token
> > > >> > > > >> >> expiration
> > > >> > > > >> >> under token acquisition process
> > > >> > > > >> >> <
> > > >> > > > >>
> > > >> > > >
> > > >> > >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> > > >> > > > >> >.
> > > >> > > > >> >> Current proposal is that tokens will expire based on a
> > > server
> > > >> side
> > > >> > > > >> >> configuration (default 24 hours) unless renewed. Renewal
> > is
> > > >> only
> > > >> > > > allowed
> > > >> > > > >> >> until the max life time of token. Alternatively we could
> > > also
> > > >> make
> > > >> > > > that
> > > >> > > > >> >> an
> > > >> > > > >> >> optional param and the server side default can serve as
> > the
> > > >> upper
> > > >> > > > bound.
> > > >> > > > >> >>
> > > >> > > > >> >> To your second point it will be done exactly the same
> way
> > we
> > > >> would
> > > >> > > > >> >> support
> > > >> > > > >> >> multiple keytabs. The calling client will have to put
> the
> > > >> tokens it
> > > >> > > > >> wants
> > > >> > > > >> >> to use in the subject instance and call produce/consume
> > > inside
> > > >> > > > >> >> subject.doas. Each caller will have to keep track of its
> > own
> > > >> > > > subject. I
> > > >> > > > >> >> will have to look at the code to see if we support this
> > > >> feature
> > > >> > > right
> > > >> > > > >> now
> > > >> > > > >> >> but my understanding is delegation token shouldn't need
> > any
> > > >> special
> > > >> > > > >> >> treatment as its just another type of Credential in the
> > > >> subject.
> > > >> > > > >> >>
> > > >> > > > >> >> I would also like to know what is your opinion about
> > > infinite
> > > >> > > renewal
> > > >> > > > >> (my
> > > >> > > > >> >> recommendation is to not support this), tokens renewing
> > them
> > > >> > > self(my
> > > >> > > > >> >> recommendation is to not support this) and most
> > importantly
> > > >> your
> > > >> > > > choice
> > > >> > > > >> >> between the alternatives listed on this thread
> > > >> > > > >> >> <
> > > >> > > > >>
> > > >> > > >
> > > >> > >
> > > >>
> > >
> >
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> > > >> > > > >> >
> > > >> > > > >> >> ( I am leaning towards the alternative-2 minus
> controller
> > > >> > > > distributing
> > > >> > > > >> >> secret). Thanks again for reviewing.
> > > >> > > > >> >>
> > > >> > > > >> >> Thanks
> > > >> > > > >> >> Parth
> > > >> > > > >> >>
> > > >> > > > >> >>
> > > >> > > > >> >>
> > > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <
> > > >> gwen@confluent.io>
> > > >> > > > wrote:
> > > >> > > > >> >>
> > > >> > > > >> >> > Harsha,
> > > >> > > > >> >> >
> > > >> > > > >> >> > I was thinking of the Rest Proxy. I didn't see your
> > design
> > > >> yet,
> > > >> > > > but in
> > > >> > > > >> >> > our proxy, we have a set of producers, which will
> serve
> > > >> multiple
> > > >> > > > users
> > > >> > > > >> >> > going through the proxy. Since these users will have
> > > >> different
> > > >> > > > >> >> > privileges, they'll need to authenticate separately,
> and
> > > >> can't
> > > >> > > > share a
> > > >> > > > >> >> > token.
> > > >> > > > >> >> >
> > > >> > > > >> >> > Am I missing anything?
> > > >> > > > >> >> >
> > > >> > > > >> >> > Gwen
> > > >> > > > >> >> >
> > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <
> kafka@harsha.io
> > >
> > > >> wrote:
> > > >> > > > >> >> > > Gwen,
> > > >> > > > >> >> > >            On your second point. Can you describe a
> > > >> usecase
> > > >> > > where
> > > >> > > > >> >> > >            mutliple clients ended up sharing a
> > producer
> > > >> and
> > > >> > > even
> > > >> > > > if
> > > >> > > > >> they
> > > >> > > > >> >> > >            do why can't they not use single token
> that
> > > >> producer
> > > >> > > > >> >> > >            captures. Why would we need multiple
> > clients
> > > >> with
> > > >> > > > >> different
> > > >> > > > >> >> > >            tokens sharing a single instance of
> > producer.
> > > >> Also
> > > >> > > in
> > > >> > > > >> this
> > > >> > > > >> >> > >            case other clients have access all the
> > tokens
> > > >> no?
> > > >> > > > >> >> > >
> > > >> > > > >> >> > > Thanks,
> > > >> > > > >> >> > > Harsha
> > > >> > > > >> >> > >
> > > >> > > > >> >> > >
> > > >> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira
> wrote:
> > > >> > > > >> >> > >> Sorry for the delay:
> > > >> > > > >> >> > >>
> > > >> > > > >> >> > >> Two questions that we didn't see in the wiki:
> > > >> > > > >> >> > >> 1. Is there an expiration for delegation tokens?
> > > >> Renewal? How
> > > >> > > > do we
> > > >> > > > >> >> > >> revoke them?
> > > >> > > > >> >> > >> 2. If we want to use delegation tokens for "do-as"
> > > (say,
> > > >> > > submit
> > > >> > > > >> Storm
> > > >> > > > >> >> > >> job as my user), we will need a producer for every
> > job
> > > >> (we
> > > >> > > can't
> > > >> > > > >> share
> > > >> > > > >> >> > >> them between multiple jobs running on same node),
> > since
> > > >> we
> > > >> > > only
> > > >> > > > >> >> > >> authenticate when connecting. Is there a plan to
> > change
> > > >> this
> > > >> > > for
> > > >> > > > >> >> > >> delegation tokens, in order to allow multiple users
> > > with
> > > >> > > > different
> > > >> > > > >> >> > >> tokens to share a client?
> > > >> > > > >> >> > >>
> > > >> > > > >> >> > >> Gwen
> > > >> > > > >> >> > >>
> > > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
> > > >> > > > >> >> > >> <br...@gmail.com> wrote:
> > > >> > > > >> >> > >> > Bumping this up one more time, can other
> committers
> > > >> review?
> > > >> > > > >> >> > >> >
> > > >> > > > >> >> > >> > Thanks
> > > >> > > > >> >> > >> > Parth
> > > >> > > > >> >> > >> >
> > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <
> > > >> kafka@harsha.io>
> > > >> > > > wrote:
> > > >> > > > >> >> > >> >
> > > >> > > > >> >> > >> >> Parth,
> > > >> > > > >> >> > >> >>           Overall current design looks good to
> > me. I
> > > >> am +1
> > > >> > > on
> > > >> > > > >> the
> > > >> > > > >> >> > KIP.
> > > >> > > > >> >> > >> >>
> > > >> > > > >> >> > >> >> Gwen , Jun can you review this as well.
> > > >> > > > >> >> > >> >>
> > > >> > > > >> >> > >> >> -Harsha
> > > >> > > > >> >> > >> >>
> > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth
> > brahmbhatt
> > > >> wrote:
> > > >> > > > >> >> > >> >> > Thanks for review Jitendra.
> > > >> > > > >> >> > >> >> >
> > > >> > > > >> >> > >> >> > I don't like the idea of infinite lifetime
> but I
> > > >> see the
> > > >> > > > >> Streaming
> > > >> > > > >> >> > use
> > > >> > > > >> >> > >> >> > case. Even for Streaming use case I was hoping
> > > >> there will
> > > >> > > > be
> > > >> > > > >> some
> > > >> > > > >> >> > notion
> > > >> > > > >> >> > >> >> > of
> > > >> > > > >> >> > >> >> > master/driver that can get new delegation
> tokens
> > > at
> > > >> fixed
> > > >> > > > >> interval
> > > >> > > > >> >> > and
> > > >> > > > >> >> > >> >> > distribute to workers. If that is not the case
> > for
> > > >> we can
> > > >> > > > >> discuss
> > > >> > > > >> >> > >> >> > delegation tokens renewing them self and the
> > > >> security
> > > >> > > > >> implications
> > > >> > > > >> >> > of the
> > > >> > > > >> >> > >> >> > same.
> > > >> > > > >> >> > >> >> >
> > > >> > > > >> >> > >> >> > I did not want clients to fetch tokens from
> > > >> zookeeper,
> > > >> > > > >> overall I
> > > >> > > > >> >> > think
> > > >> > > > >> >> > >> >> > its
> > > >> > > > >> >> > >> >> > better if clients don't rely on our metadata
> > store
> > > >> and I
> > > >> > > > >> think we
> > > >> > > > >> >> > are
> > > >> > > > >> >> > >> >> > moving in that direction with all the KIP-4
> > > >> improvements.
> > > >> > > > I
> > > >> > > > >> chose
> > > >> > > > >> >> > >> >> > zookeeper as in this case the client will
> still
> > > >> just talk
> > > >> > > > to
> > > >> > > > >> >> > broker , its
> > > >> > > > >> >> > >> >> > the brokers that will use zookeeper which we
> > > >> already do
> > > >> > > > for a
> > > >> > > > >> lot
> > > >> > > > >> >> > of
> > > >> > > > >> >> > >> >> > other
> > > >> > > > >> >> > >> >> > usecases + ease of development + and the
> ability
> > > so
> > > >> > > tokens
> > > >> > > > >> will
> > > >> > > > >> >> > survive
> > > >> > > > >> >> > >> >> > even a rolling restart/cluster failure. if a
> > > >> majority
> > > >> > > > agrees
> > > >> > > > >> the
> > > >> > > > >> >> > added
> > > >> > > > >> >> > >> >> > complexity to have controller forwarding keys
> to
> > > all
> > > >> > > > broker is
> > > >> > > > >> >> > justified
> > > >> > > > >> >> > >> >> > as
> > > >> > > > >> >> > >> >> > it provides tighter security , I am fine with
> > that
> > > >> option
> > > >> > > > too.
> > > >> > > > >> >> > >> >> >
> > > >> > > > >> >> > >> >> > Given zookeeper does not support SSL we can
> not
> > > >> store
> > > >> > > > master
> > > >> > > > >> keys
> > > >> > > > >> >> > in
> > > >> > > > >> >> > >> >> > zookeeper as master keys will be exposed on
> > wire.
> > > To
> > > >> > > > support
> > > >> > > > >> >> > rotation
> > > >> > > > >> >> > >> >> > without affecting current clients is
> something I
> > > >> need to
> > > >> > > > put
> > > >> > > > >> more
> > > >> > > > >> >> > thought
> > > >> > > > >> >> > >> >> > in. My current proposal assumes the rotation
> > will
> > > >> > > > invalidate
> > > >> > > > >> all
> > > >> > > > >> >> > current
> > > >> > > > >> >> > >> >> > tokens.
> > > >> > > > >> >> > >> >> >
> > > >> > > > >> >> > >> >> > I request committers to also review and post
> > their
> > > >> > > comments
> > > >> > > > >> so we
> > > >> > > > >> >> > can
> > > >> > > > >> >> > >> >> > make
> > > >> > > > >> >> > >> >> > progress on this KIP.
> > > >> > > > >> >> > >> >> >
> > > >> > > > >> >> > >> >> > Thanks
> > > >> > > > >> >> > >> >> > Parth
> > > >> > > > >> >> > >> >> >
> > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh
> <
> > > >> > > > >> asingh@cloudera.com
> > > >> > > > >> >> > >
> > > >> > > > >> >> > >> >> > wrote:
> > > >> > > > >> >> > >> >> >
> > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <
> > > >> > > > kafka@harsha.io>
> > > >> > > > >> >> > wrote:
> > > >> > > > >> >> > >> >> > >
> > > >> > > > >> >> > >> >> > > > Unifying the two discussion threads on
> this
> > > KIP.
> > > >> > > > >> >> > >> >> > > >
> > > >> > > > >> >> > >> >> > > > Here is the response from Jitendra
> > > >> > > > >> >> > >> >> > > >
> > > >> > > > >> >> > >> >> > > > "The need for a large number of clients
> that
> > > are
> > > >> > > > running
> > > >> > > > >> all
> > > >> > > > >> >> > over the
> > > >> > > > >> >> > >> >> > > > cluster that authenticate with Kafka
> > brokers,
> > > >> is very
> > > >> > > > >> similar
> > > >> > > > >> >> > to the
> > > >> > > > >> >> > >> >> > > > Hadoop use case of large number of tasks
> > > running
> > > >> > > across
> > > >> > > > >> the
> > > >> > > > >> >> > cluster
> > > >> > > > >> >> > >> >> that
> > > >> > > > >> >> > >> >> > > > need authentication to Hdfs Namenode.
> > > >> Therefore, the
> > > >> > > > >> >> > delegation token
> > > >> > > > >> >> > >> >> > > > approach does seem like a good fit for
> this
> > > use
> > > >> case
> > > >> > > > as we
> > > >> > > > >> >> > have seen
> > > >> > > > >> >> > >> >> it
> > > >> > > > >> >> > >> >> > > > working at large scale in HDFS and YARN.
> > > >> > > > >> >> > >> >> > > >
> > > >> > > > >> >> > >> >> > > >   The proposed design is very much inline
> > with
> > > >> Hadoop
> > > >> > > > >> >> > approach. A few
> > > >> > > > >> >> > >> >> > > >   comments:
> > > >> > > > >> >> > >> >> > > >
> > > >> > > > >> >> > >> >> > > > 1) Why do you guys want to allow infinite
> > > >> renewable
> > > >> > > > >> lifetime
> > > >> > > > >> >> > for a
> > > >> > > > >> >> > >> >> > > > token? HDFS restricts a token to a max
> life
> > > time
> > > >> > > > (default
> > > >> > > > >> 7
> > > >> > > > >> >> > days).  A
> > > >> > > > >> >> > >> >> > > > token's vulnerability is believed to
> > increase
> > > >> with
> > > >> > > > time.
> > > >> > > > >> >> > >> >> > > >
> > > >> > > > >> >> > >> >> > > I agree that having infinite lifetime might
> > not
> > > >> be the
> > > >> > > > best
> > > >> > > > >> idea.
> > > >> > > > >> >> > >> >> > >
> > > >> > > > >> >> > >> >> > > >
> > > >> > > > >> >> > >> >> > > > 2) As I understand the tokens are stored
> in
> > > >> zookeeper
> > > >> > > > as
> > > >> > > > >> well,
> > > >> > > > >> >> > and
> > > >> > > > >> >> > >> >> can
> > > >> > > > >> >> > >> >> > > > be updated there. This is clever as it can
> > > allow
> > > >> > > > >> replacing the
> > > >> > > > >> >> > tokens
> > > >> > > > >> >> > >> >> > > > once they run out of max life time, and
> > > clients
> > > >> can
> > > >> > > > >> download
> > > >> > > > >> >> > new
> > > >> > > > >> >> > >> >> tokens
> > > >> > > > >> >> > >> >> > > > from zookeeper. It shouldn't be a big load
> > on
> > > >> > > zookeeper
> > > >> > > > >> as a
> > > >> > > > >> >> > client
> > > >> > > > >> >> > >> >> will
> > > >> > > > >> >> > >> >> > > > need to get a new token once in several
> > days.
> > > >> In this
> > > >> > > > >> approach
> > > >> > > > >> >> > you
> > > >> > > > >> >> > >> >> don't
> > > >> > > > >> >> > >> >> > > > need infinite lifetime on the token even
> for
> > > >> long
> > > >> > > > running
> > > >> > > > >> >> > clients.
> > > >> > > > >> >> > >> >> > > >
> > > >> > > > >> >> > >> >> > > > 3) The token password are generated using
> a
> > > >> master
> > > >> > > key.
> > > >> > > > >> The
> > > >> > > > >> >> > master
> > > >> > > > >> >> > >> >> key
> > > >> > > > >> >> > >> >> > > > should also be periodically changed. In
> > > Hadoop,
> > > >> the
> > > >> > > > >> default
> > > >> > > > >> >> > renewal
> > > >> > > > >> >> > >> >> > > > period is 1 day.?
> > > >> > > > >> >> > >> >> > > >
> > > >> > > > >> >> > >> >> > > IIUC, this will require brokers maintaining
> a
> > > >> list of X
> > > >> > > > most
> > > >> > > > >> >> > recent
> > > >> > > > >> >> > >> >> master
> > > >> > > > >> >> > >> >> > > keys. This list will have to be persisted
> > > >> somewhere, as
> > > >> > > > if a
> > > >> > > > >> >> > broker
> > > >> > > > >> >> > >> >> goes
> > > >> > > > >> >> > >> >> > > down it will have to get that list again and
> > > >> storing
> > > >> > > > master
> > > >> > > > >> keys
> > > >> > > > >> >> > on ZK
> > > >> > > > >> >> > >> >> is
> > > >> > > > >> >> > >> >> > > not the best idea. However, if a broker goes
> > > down
> > > >> then
> > > >> > > we
> > > >> > > > >> have
> > > >> > > > >> >> > much
> > > >> > > > >> >> > >> >> bigger
> > > >> > > > >> >> > >> >> > > issue to deal with and client can always
> > > >> > > re-authenticate
> > > >> > > > is
> > > >> > > > >> such
> > > >> > > > >> >> > >> >> events.
> > > >> > > > >> >> > >> >> > >
> > > >> > > > >> >> > >> >> > > Did you happen to take a look at other
> > > >> alternatives
> > > >> > > this
> > > >> > > > >> list has
> > > >> > > > >> >> > >> >> > > suggested?
> > > >> > > > >> >> > >> >> > >
> > > >> > > > >> >> > >> >> > > >
> > > >> > > > >> >> > >> >> > > > Thanks for a thorough proposal, great
> work!"
> > > >> > > > >> >> > >> >> > > >
> > > >> > > > >> >> > >> >> > > >
> > > >> > > > >> >> > >> >> > > >
> > > >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen
> > Shapira
> > > >> wrote:
> > > >> > > > >> >> > >> >> > > > > Makes sense to me. Thanks!
> > > >> > > > >> >> > >> >> > > > >
> > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <
> > > >> > > > kafka@harsha.io
> > > >> > > > >> >
> > > >> > > > >> >> > wrote:
> > > >> > > > >> >> > >> >> > > > > > It doesn't need any release vehicle
> but
> > > >> still the
> > > >> > > > >> work can
> > > >> > > > >> >> > move
> > > >> > > > >> >> > >> >> > > > forward.
> > > >> > > > >> >> > >> >> > > > > > If anyone is interested in the KIP
> > please
> > > >> do the
> > > >> > > > >> review and
> > > >> > > > >> >> > >> >> provide
> > > >> > > > >> >> > >> >> > > the
> > > >> > > > >> >> > >> >> > > > > > comments.
> > > >> > > > >> >> > >> >> > > > > >
> > > >> > > > >> >> > >> >> > > > > > -Harsha
> > > >> > > > >> >> > >> >> > > > > >
> > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM,
> Ismael
> > > >> Juma
> > > >> > > > wrote:
> > > >> > > > >> >> > >> >> > > > > >> I agree that it would be good to have
> > > more
> > > >> time
> > > >> > > to
> > > >> > > > >> review
> > > >> > > > >> >> > and
> > > >> > > > >> >> > >> >> > > discuss
> > > >> > > > >> >> > >> >> > > > > >> KIP-48.
> > > >> > > > >> >> > >> >> > > > > >>
> > > >> > > > >> >> > >> >> > > > > >> Ismael
> > > >> > > > >> >> > >> >> > > > > >>
> > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen
> > > >> Shapira <
> > > >> > > > >> >> > >> >> gwen@confluent.io>
> > > >> > > > >> >> > >> >> > > > wrote:
> > > >> > > > >> >> > >> >> > > > > >>
> > > >> > > > >> >> > >> >> > > > > >> > Hi Team,
> > > >> > > > >> >> > >> >> > > > > >> >
> > > >> > > > >> >> > >> >> > > > > >> > Since KIP-48 depends on KIP-43,
> which
> > > is
> > > >> > > > already a
> > > >> > > > >> bit
> > > >> > > > >> >> > of a
> > > >> > > > >> >> > >> >> risk
> > > >> > > > >> >> > >> >> > > for
> > > >> > > > >> >> > >> >> > > > > >> > the next release - any chance we
> can
> > > >> delay
> > > >> > > > >> delegation
> > > >> > > > >> >> > tokens
> > > >> > > > >> >> > >> >> to
> > > >> > > > >> >> > >> >> > > > Kafka
> > > >> > > > >> >> > >> >> > > > > >> > 0.10.1?
> > > >> > > > >> >> > >> >> > > > > >> > With the community working on a
> > release
> > > >> every
> > > >> > > 3
> > > >> > > > >> month,
> > > >> > > > >> >> > this
> > > >> > > > >> >> > >> >> is not
> > > >> > > > >> >> > >> >> > > > a huge
> > > >> > > > >> >> > >> >> > > > > >> > delay.
> > > >> > > > >> >> > >> >> > > > > >> >
> > > >> > > > >> >> > >> >> > > > > >> > Gwen
> > > >> > > > >> >> > >> >> > > > > >> >
> > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM,
> > Ashish
> > > >> Singh
> > > >> > > <
> > > >> > > > >> >> > >> >> > > asingh@cloudera.com>
> > > >> > > > >> >> > >> >> > > > wrote:
> > > >> > > > >> >> > >> >> > > > > >> > > Parth,
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > > Thanks again for the awesome
> write
> > > up.
> > > >> > > > Following
> > > >> > > > >> our
> > > >> > > > >> >> > >> >> discussion
> > > >> > > > >> >> > >> >> > > > from the
> > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I think it will be easier
> to
> > > >> compare
> > > >> > > > >> various
> > > >> > > > >> >> > >> >> alternatives
> > > >> > > > >> >> > >> >> > > > if they
> > > >> > > > >> >> > >> >> > > > > >> > are
> > > >> > > > >> >> > >> >> > > > > >> > > listed together. I am stating
> > below a
> > > >> few
> > > >> > > > >> >> > alternatives along
> > > >> > > > >> >> > >> >> > > with
> > > >> > > > >> >> > >> >> > > > a the
> > > >> > > > >> >> > >> >> > > > > >> > > current proposal.
> > > >> > > > >> >> > >> >> > > > > >> > > (Current proposal) Store
> Delegation
> > > >> Token,
> > > >> > > DT,
> > > >> > > > >> on ZK.
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a
> > > >> broker.
> > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > authenticated,
> > > >> it
> > > >> > > will
> > > >> > > > >> make a
> > > >> > > > >> >> > broker
> > > >> > > > >> >> > >> >> side
> > > >> > > > >> >> > >> >> > > > call to
> > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker generates a
> shared
> > > >> secret
> > > >> > > > based
> > > >> > > > >> on
> > > >> > > > >> >> > >> >> > > HMAC-SHA256(a
> > > >> > > > >> >> > >> >> > > > > >> > >    Password/Secret shared between
> > all
> > > >> > > brokers,
> > > >> > > > >> >> > randomly
> > > >> > > > >> >> > >> >> > > generated
> > > >> > > > >> >> > >> >> > > > > >> > tokenId).
> > > >> > > > >> >> > >> >> > > > > >> > >    4. Broker stores this token in
> > its
> > > >> in
> > > >> > > > memory
> > > >> > > > >> cache.
> > > >> > > > >> >> > >> >> Broker
> > > >> > > > >> >> > >> >> > > > also stores
> > > >> > > > >> >> > >> >> > > > > >> > >    the DelegationToken without
> the
> > > >> hmac in
> > > >> > > the
> > > >> > > > >> >> > zookeeper.
> > > >> > > > >> >> > >> >> > > > > >> > >    5. All brokers will have a
> cache
> > > >> backed
> > > >> > > by
> > > >> > > > >> >> > zookeeper so
> > > >> > > > >> >> > >> >> they
> > > >> > > > >> >> > >> >> > > > will all
> > > >> > > > >> >> > >> >> > > > > >> > >    get notified whenever a new
> > token
> > > is
> > > >> > > > >> generated and
> > > >> > > > >> >> > they
> > > >> > > > >> >> > >> >> will
> > > >> > > > >> >> > >> >> > > > update
> > > >> > > > >> >> > >> >> > > > > >> > their
> > > >> > > > >> >> > >> >> > > > > >> > >    local cache whenever token
> state
> > > >> changes.
> > > >> > > > >> >> > >> >> > > > > >> > >    6. Broker returns the token to
> > > >> Client.
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > >    1. Probable race condition,
> > client
> > > >> tries
> > > >> > > to
> > > >> > > > >> >> > authenticate
> > > >> > > > >> >> > >> >> with
> > > >> > > > >> >> > >> >> > > > a broker
> > > >> > > > >> >> > >> >> > > > > >> > >    that is yet to be updated with
> > the
> > > >> newly
> > > >> > > > >> generated
> > > >> > > > >> >> > DT.
> > > >> > > > >> >> > >> >> This
> > > >> > > > >> >> > >> >> > > can
> > > >> > > > >> >> > >> >> > > > > >> > probably be
> > > >> > > > >> >> > >> >> > > > > >> > >    dealt with making dtRequest
> > block
> > > >> until
> > > >> > > all
> > > >> > > > >> >> > brokers have
> > > >> > > > >> >> > >> >> > > > updated
> > > >> > > > >> >> > >> >> > > > > >> > their DT
> > > >> > > > >> >> > >> >> > > > > >> > >    cache. Zk barrier or similar
> > > >> mechanism
> > > >> > > can
> > > >> > > > be
> > > >> > > > >> used.
> > > >> > > > >> >> > >> >> However,
> > > >> > > > >> >> > >> >> > > > all such
> > > >> > > > >> >> > >> >> > > > > >> > >    mechanisms will increase
> > > complexity.
> > > >> > > > >> >> > >> >> > > > > >> > >    2. Using a static secret key
> > from
> > > >> config
> > > >> > > > >> file. Will
> > > >> > > > >> >> > >> >> require
> > > >> > > > >> >> > >> >> > > yet
> > > >> > > > >> >> > >> >> > > > > >> > another
> > > >> > > > >> >> > >> >> > > > > >> > >    config and uses a static
> secret
> > > >> key. It
> > > >> > > is
> > > >> > > > >> advised
> > > >> > > > >> >> > to
> > > >> > > > >> >> > >> >> rotate
> > > >> > > > >> >> > >> >> > > > secret
> > > >> > > > >> >> > >> >> > > > > >> > keys
> > > >> > > > >> >> > >> >> > > > > >> > >    periodically. This can be
> > avoided
> > > >> with
> > > >> > > > >> controller
> > > >> > > > >> >> > >> >> generating
> > > >> > > > >> >> > >> >> > > > > >> > secretKey and
> > > >> > > > >> >> > >> >> > > > > >> > >    passing to brokers
> periodically.
> > > >> However,
> > > >> > > > >> this will
> > > >> > > > >> >> > >> >> require
> > > >> > > > >> >> > >> >> > > > brokers to
> > > >> > > > >> >> > >> >> > > > > >> > >    maintain certain counts of
> > > >> secretKeys.
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 1) Have controller
> > > >> generate
> > > >> > > > >> delegation
> > > >> > > > >> >> > token.
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a
> > > >> broker.
> > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > authenticated,
> > > >> it
> > > >> > > will
> > > >> > > > >> make a
> > > >> > > > >> >> > broker
> > > >> > > > >> >> > >> >> side
> > > >> > > > >> >> > >> >> > > > call to
> > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > > >> > > > >> >> > >> >> > > > > >> > >    3. Broker forwards the request
> > to
> > > >> > > > controller.
> > > >> > > > >> >> > >> >> > > > > >> > >    4. Controller generates a DT
> and
> > > >> > > broadcasts
> > > >> > > > >> to all
> > > >> > > > >> >> > >> >> brokers.
> > > >> > > > >> >> > >> >> > > > > >> > >    5. Broker stores this token in
> > its
> > > >> memory
> > > >> > > > >> cache.
> > > >> > > > >> >> > >> >> > > > > >> > >    6. Controller responds to
> > broker’s
> > > >> DT
> > > >> > > req.
> > > >> > > > >> >> > >> >> > > > > >> > >    7. Broker returns the token to
> > > >> Client.
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > >    1. We will have to add new
> APIs
> > to
> > > >> > > support
> > > >> > > > >> >> > controller
> > > >> > > > >> >> > >> >> pushing
> > > >> > > > >> >> > >> >> > > > tokens
> > > >> > > > >> >> > >> >> > > > > >> > to
> > > >> > > > >> >> > >> >> > > > > >> > >    brokers on top of the minimal
> > APIs
> > > >> that
> > > >> > > are
> > > >> > > > >> >> > currently
> > > >> > > > >> >> > >> >> > > proposed.
> > > >> > > > >> >> > >> >> > > > > >> > >    2. We will also have to add
> APIs
> > > to
> > > >> > > support
> > > >> > > > >> the
> > > >> > > > >> >> > >> >> bootstrapping
> > > >> > > > >> >> > >> >> > > > case,
> > > >> > > > >> >> > >> >> > > > > >> > i.e,
> > > >> > > > >> >> > >> >> > > > > >> > >    when a new broker comes up it
> > will
> > > >> have
> > > >> > > to
> > > >> > > > >> get all
> > > >> > > > >> >> > >> >> delegation
> > > >> > > > >> >> > >> >> > > > tokens
> > > >> > > > >> >> > >> >> > > > > >> > from
> > > >> > > > >> >> > >> >> > > > > >> > >    the controller.
> > > >> > > > >> >> > >> >> > > > > >> > >    3. In catastrophic failures
> > where
> > > >> all
> > > >> > > > brokers
> > > >> > > > >> go
> > > >> > > > >> >> > down,
> > > >> > > > >> >> > >> >> the
> > > >> > > > >> >> > >> >> > > > tokens will
> > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if servers are
> > > >> restarted as
> > > >> > > > >> tokens
> > > >> > > > >> >> > are not
> > > >> > > > >> >> > >> >> > > > persisted
> > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > >> > > > >> >> > >> >> > > > > >> > >    If this happens, then there
> are
> > > more
> > > >> > > > important
> > > >> > > > >> >> > things to
> > > >> > > > >> >> > >> >> > > worry
> > > >> > > > >> >> > >> >> > > > about
> > > >> > > > >> >> > >> >> > > > > >> > and
> > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
> > > >> re-authenticate.
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 2) Do not distribute
> > DT
> > > to
> > > >> > > other
> > > >> > > > >> brokers
> > > >> > > > >> >> > at
> > > >> > > > >> >> > >> >> all.
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a
> > > >> broker.
> > > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> > authenticated,
> > > >> it
> > > >> > > will
> > > >> > > > >> make a
> > > >> > > > >> >> > broker
> > > >> > > > >> >> > >> >> side
> > > >> > > > >> >> > >> >> > > > call to
> > > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > > >> > > > >> >> > >> >> > > > > >> > >    3. The broker generates DT of
> > > form,
> > > >> > > [hmac +
> > > >> > > > >> (owner,
> > > >> > > > >> >> > >> >> renewer,
> > > >> > > > >> >> > >> >> > > > > >> > >    maxLifeTime, id, hmac,
> > > >> expirationTime)]
> > > >> > > and
> > > >> > > > >> passes
> > > >> > > > >> >> > back
> > > >> > > > >> >> > >> >> this
> > > >> > > > >> >> > >> >> > > > DT to
> > > >> > > > >> >> > >> >> > > > > >> > client.
> > > >> > > > >> >> > >> >> > > > > >> > >    hmac is generated via
> > > >> {HMAC-SHA256(owner,
> > > >> > > > >> renewer,
> > > >> > > > >> >> > >> >> > > > maxLifeTime, id,
> > > >> > > > >> >> > >> >> > > > > >> > hmac,
> > > >> > > > >> >> > >> >> > > > > >> > >    expirationTime) using
> > SecretKey}.
> > > >> Note
> > > >> > > that
> > > >> > > > >> all
> > > >> > > > >> >> > brokers
> > > >> > > > >> >> > >> >> have
> > > >> > > > >> >> > >> >> > > > this
> > > >> > > > >> >> > >> >> > > > > >> > SecretKey.
> > > >> > > > >> >> > >> >> > > > > >> > >    4. Client then goes to any
> > broker
> > > >> and to
> > > >> > > > >> >> > authenticate
> > > >> > > > >> >> > >> >> sends
> > > >> > > > >> >> > >> >> > > > the DT.
> > > >> > > > >> >> > >> >> > > > > >> > >    Broker recalculates hmac using
> > > >> (owner,
> > > >> > > > >> renewer,
> > > >> > > > >> >> > >> >> maxLifeTime,
> > > >> > > > >> >> > >> >> > > > id, hmac,
> > > >> > > > >> >> > >> >> > > > > >> > >    expirationTime) info from DT
> and
> > > its
> > > >> > > > >> SecretKey. If
> > > >> > > > >> >> > it
> > > >> > > > >> >> > >> >> matches
> > > >> > > > >> >> > >> >> > > > with
> > > >> > > > >> >> > >> >> > > > > >> > hmac of
> > > >> > > > >> >> > >> >> > > > > >> > >    DT, client is authenticated.
> > Yes,
> > > >> it will
> > > >> > > > do
> > > >> > > > >> other
> > > >> > > > >> >> > >> >> obvious
> > > >> > > > >> >> > >> >> > > > checks of
> > > >> > > > >> >> > >> >> > > > > >> > >    timestamp expiry and such.
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > > Note that secret key will be
> > > generated
> > > >> by
> > > >> > > > >> controller
> > > >> > > > >> >> > and
> > > >> > > > >> >> > >> >> passed
> > > >> > > > >> >> > >> >> > > to
> > > >> > > > >> >> > >> >> > > > > >> > brokers
> > > >> > > > >> >> > >> >> > > > > >> > > periodically.
> > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > >    1. How to delete a DT? Yes,
> that
> > > is
> > > >> a
> > > >> > > > downside
> > > >> > > > >> >> > here.
> > > >> > > > >> >> > >> >> However,
> > > >> > > > >> >> > >> >> > > > this can
> > > >> > > > >> >> > >> >> > > > > >> > >    be handled with brokers
> > > maintaining
> > > >> a
> > > >> > > > >> blacklist of
> > > >> > > > >> >> > DTs,
> > > >> > > > >> >> > >> >> DTs
> > > >> > > > >> >> > >> >> > > > from this
> > > >> > > > >> >> > >> >> > > > > >> > list
> > > >> > > > >> >> > >> >> > > > > >> > >    can be removed after expiry.
> > > >> > > > >> >> > >> >> > > > > >> > >    2. In catastrophic failures
> > where
> > > >> all
> > > >> > > > brokers
> > > >> > > > >> go
> > > >> > > > >> >> > down,
> > > >> > > > >> >> > >> >> the
> > > >> > > > >> >> > >> >> > > > tokens will
> > > >> > > > >> >> > >> >> > > > > >> > >    be lost even if servers are
> > > >> restarted as
> > > >> > > > >> tokens
> > > >> > > > >> >> > are not
> > > >> > > > >> >> > >> >> > > > persisted
> > > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > > >> > > > >> >> > >> >> > > > > >> > >    If this happens, then there
> are
> > > more
> > > >> > > > important
> > > >> > > > >> >> > things to
> > > >> > > > >> >> > >> >> > > worry
> > > >> > > > >> >> > >> >> > > > about
> > > >> > > > >> >> > >> >> > > > > >> > and
> > > >> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
> > > >> re-authenticate.
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM,
> > > Parth
> > > >> > > > >> Brahmbhatt <
> > > >> > > > >> >> > >> >> > > > > >> > > pbrahmbhatt@hortonworks.com>
> > wrote:
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > >> Hi,
> > > >> > > > >> >> > >> >> > > > > >> > >>
> > > >> > > > >> >> > >> >> > > > > >> > >> I have filed KIP-48 so we can
> > offer
> > > >> hadoop
> > > >> > > > like
> > > >> > > > >> >> > delegation
> > > >> > > > >> >> > >> >> > > > tokens in
> > > >> > > > >> >> > >> >> > > > > >> > >> kafka. You can review the design
> > > >> > > > >> >> > >> >> > > > > >> > >>
> > > >> > > > >> >> > >> >> > > > > >> >
> > > >> > > > >> >> > >> >> > > >
> > > >> > > > >> >> > >> >> > >
> > > >> > > > >> >> > >> >>
> > > >> > > > >> >> >
> > > >> > > > >>
> > > >> > > >
> > > >> > >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > > >> > > > >> >> > >> >> > > > > >> > .
> > > >> > > > >> >> > >> >> > > > > >> > >> This KIP depends on KIP-43 and
> we
> > > >> have also
> > > >> > > > >> >> > discussed an
> > > >> > > > >> >> > >> >> > > > alternative to
> > > >> > > > >> >> > >> >> > > > > >> > >> proposed design here<
> > > >> > > > >> >> > >> >> > > > > >> > >>
> > > >> > > > >> >> > >> >> > > > > >> >
> > > >> > > > >> >> > >> >> > > >
> > > >> > > > >> >> > >> >> > >
> > > >> > > > >> >> > >> >>
> > > >> > > > >> >> >
> > > >> > > > >>
> > > >> > > >
> > > >> > >
> > > >>
> > >
> >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > > >> > > > >> >> > >> >> > > > > >> > >> >.
> > > >> > > > >> >> > >> >> > > > > >> > >>
> > > >> > > > >> >> > >> >> > > > > >> > >> Thanks
> > > >> > > > >> >> > >> >> > > > > >> > >> Parth
> > > >> > > > >> >> > >> >> > > > > >> > >>
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > > --
> > > >> > > > >> >> > >> >> > > > > >> > >
> > > >> > > > >> >> > >> >> > > > > >> > > Regards,
> > > >> > > > >> >> > >> >> > > > > >> > > Ashish
> > > >> > > > >> >> > >> >> > > > > >> >
> > > >> > > > >> >> > >> >> > > >
> > > >> > > > >> >> > >> >> > >
> > > >> > > > >> >> > >> >> > >
> > > >> > > > >> >> > >> >> > >
> > > >> > > > >> >> > >> >> > > --
> > > >> > > > >> >> > >> >> > >
> > > >> > > > >> >> > >> >> > > Regards,
> > > >> > > > >> >> > >> >> > > Ashish
> > > >> > > > >> >> > >> >> > >
> > > >> > > > >> >> > >> >>
> > > >> > > > >> >> >
> > > >> > > > >>
> > > >> > > >
> > > >> > >
> > > >>
> > > >>
> > > >
> > >
> >
>



-- 
Regards,

Rajini

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Jun Rao <ju...@confluent.io>]

Parth,

Thanks for the explanation. A couple of more questions.

110. What does getDelegationTokenAs mean?

111. What's the typical rate of getting and renewing delegation tokens?
That may have an impact on whether they should be directed to the
controller.

Jun

On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt <
brahmbhatt.parth@gmail.com> wrote:

> Hi Jun,
>
> Thanks for reviewing.
>
> * We could add a Cluster action to add acls on who can request delegation
> tokens. I don't see the use case for that yet but down the line when we
> start supporting getDelegationTokenAs it will be necessary.
> * Yes we recommend tokens to be only used/distributed over secure channels.
> * Depending on what design we end up choosing Invalidation will be
> responsibility of every broker or controller.
> * I am not sure if I documented somewhere that invalidation will directly
> go through zookeeper but that is not the intent. Invalidation will either
> be request based or due to expiration. No direct zookeeper interaction from
> any client.
> * "Broker also stores the DelegationToken without the hmac in the
> zookeeper." : Sorry about the confusion. The sole purpose of zookeeper in
> this design is as distribution channel for tokens between all brokers and a
> layer that ensures only tokens that were generated by making a request to a
> broker will be accepted (more on this in second paragraph). The token
> consists of few elements (owner, renewer, uuid , expiration, hmac) , one of
> which is the finally generated hmac but hmac it self is derivable if you
> have all the other elements of the token + secret key to generate hmac.
> Given zookeeper does not provide SSL support we do not want the entire
> token to be wire transferred to zookeeper as that will be an insecure wire
> transfer. Instead we only store all the other elements of a delegation
> tokens. Brokers can read these elements and because they also have access
> to secret key they will be able to generate hmac on their end.
>
> One of the alternative proposed is to avoid zookeeper altogether. A Client
> will call broker with required information (owner, renwer, expiration) and
> get back (signed hmac, uuid). Broker won't store this in zookeeper. From
> this point a client can contact any broker with all the delegation token
> info (owner, rewner, expiration, hmac, uuid) the borker will regenerate the
> hmac and as long as it matches with hmac presented by client , broker will
> allow the request to authenticate.  Only problem with this approach is if
> the secret key is compromised any client can now generate random tokens and
> they will still be able to authenticate as any user they like. with
> zookeeper we guarantee that only tokens acquired via a broker (using some
> auth scheme other than delegation token) will be accepted. We need to
> discuss which proposal makes more sense and we can go over it in tomorrow's
> meeting.
>
> Also, can you forward the invite to me?
>
> Thanks
> Parth
>
>
>
> On Mon, May 23, 2016 at 10:35 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > Thanks for the KIP. A few comments.
> >
> > 100. This potentially can be useful for Kafka Connect and Kafka rest
> proxy
> > where a worker agent will need to run a task on behalf of a client. We
> will
> > likely need to change how those services use Kafka clients
> > (producer/consumer). Instead of a shared client per worker, we will need
> a
> > client per user task since the authentication happens at the connection
> > level. For Kafka Connect, the renewer will be the workers. So, we
> probably
> > need to allow multiple renewers. For Kafka rest proxy, the renewer can
> > probably just be the creator of the token.
> >
> > 101. Do we need new acl on who can request delegation tokens?
> >
> > 102. Do we recommend people to send delegation tokens in an encrypted
> > channel?
> >
> > 103. Who is responsible for expiring tokens, every broker?
> >
> > 104. For invalidating tokens, would it be better to do it in a request
> > instead of going to ZK directly?
> >
> > 105. The terminology of client in the wiki sometimes refers to the end
> > client and some other times refers to the client using the delegation
> > tokens. It would be useful to distinguish between the two.
> >
> > 106. Could you explain the sentence "Broker also stores the
> DelegationToken
> > without the hmac in the zookeeper." a bit more? I thought the delegation
> > token is the hmac.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Hi, Harsha,
> > >
> > > Just sent out a KIP meeting invite. We can discuss this in the meeting
> > > tomorrow.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, May 19, 2016 at 8:47 AM, Harsha <ka...@harsha.io> wrote:
> > >
> > >> Hi All,
> > >>            Can we have a KIP meeting around this. The KIP is up for
> > >>            sometime and if there are any questions lets quickly hash
> out
> > >>            details.
> > >>
> > >> Thanks,
> > >> Harsha
> > >>
> > >> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt wrote:
> > >> > That is what the hadoop echo system uses so no good reason really.
> We
> > >> > could
> > >> > change it to whatever is the newest recommended standard is.
> > >> >
> > >> > Thanks
> > >> > Parth
> > >> >
> > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <is...@juma.me.uk>
> > wrote:
> > >> >
> > >> > > Hi Parth,
> > >> > >
> > >> > > Thanks for the KIP. I only started reviewing this and may have
> > >> additional
> > >> > > questions later. The immediate question that came to mind is our
> > >> choice of
> > >> > > "DIGEST-MD5" even though it's marked as OBSOLETE in the IANA
> > Registry
> > >> of
> > >> > > SASL mechanisms and the original RFC (2831) has been moved to
> > Historic
> > >> > > status:
> > >> > >
> > >> > > https://tools.ietf.org/html/rfc6331
> > >> > >
> > http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> > >> > >
> > >> > > What is the reasoning behind that choice?
> > >> > >
> > >> > > Thanks,
> > >> > > Ismael
> > >> > >
> > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <gwen@confluent.io
> >
> > >> wrote:
> > >> > >
> > >> > > > Also comments inline :)
> > >> > > >
> > >> > > > > * I want to emphasize that even though delegation tokens are a
> > >> Hadoop
> > >> > > > > innovation, I feel very strongly about not adding dependency
> on
> > >> Hadoop
> > >> > > > > when implementing delegation tokens for Kafka. The KIP doesn't
> > >> imply
> > >> > > > > such dependency, but if you can clarify...
> > >> > > > >
> > >> > > > >
> > >> > > > > *No hadoop dependency.*
> > >> > > >
> > >> > > > Yay! Just add this to the KIP so no one will read the KIP and
> > panic
> > >> > > > three weeks before the next release...
> > >> > > >
> > >> > > > > * Can we get delegation token at any time after
> authenticating?
> > >> only
> > >> > > > > immediately after?
> > >> > > > >
> > >> > > > >
> > >> > > > > *As long as you are authenticated you can get delegation
> tokens.
> > >> We
> > >> > > need
> > >> > > > to
> > >> > > > > discuss if a client authenticated using delegation token, can
> > also
> > >> > > > acquire
> > >> > > > > delegation token again or not. Also there is the question of
> do
> > we
> > >> > > allow
> > >> > > > > anyone to acquire delegation token or we want specific ACLs (I
> > >> think
> > >> > > its
> > >> > > > an
> > >> > > > > overkill.)*
> > >> > > >
> > >> > > > I agree that ACLs is an overkill.
> > >> > > >
> > >> > > > I think we are debating two options: Either require Kerberos
> auth
> > >> for
> > >> > > > renewal or require non-owners to renew.
> > >> > > > I *think* the latter is simpler (it basically require a "job
> > master"
> > >> > > > to take responsibility for the renewal, it will have its own
> > >> identity
> > >> > > > anyway and I think this is the correct design pattern anyway.
> For
> > >> > > > storm, I'd expect Nimbus to coordinate renewals?), but it is
> hard
> > to
> > >> > > > debate simplicity without looking at the code changes required.
> If
> > >> you
> > >> > > > have a draft of how the "require Kerberos" will look in Kafka
> > code,
> > >> > > > I'll be happy to take a look.
> > >> > > >
> > >> > > > > * My understanding is that tokens will propagate via ZK but
> > >> without
> > >> > > > > additional changes to UpdateMetadata protocol, correct?
> Clients
> > >> > > > > currently don't retry on SASL auth failure (IIRC), but since
> the
> > >> > > > > tokens propagate between brokers asynch, we will need to
> retry a
> > >> bit
> > >> > > > > to avoid clients failing auth due to timing issues.
> > >> > > > >
> > >> > > > > *I am considering 2 alternatives right now. The current
> > documented
> > >> > > > approach
> > >> > > > > is zookeeper based and it does not require any changes to
> > >> > > UpdateMetadata
> > >> > > > > protocol. An alternative approach can remove zookeeper
> > dependency
> > >> as
> > >> > > well
> > >> > > > > but we can discuss that in KIP discussion call.*
> > >> > > >
> > >> > > > Oooh! Sounds interesting. Do you want to ping Jun to arrange a
> > call?
> > >> > > >
> > >> > > > > * I liked Ashish's suggestion of having just the controller
> > issue
> > >> the
> > >> > > > > delegation tokens, to avoid syncing a shared secret. Not sure
> if
> > >> we
> > >> > > > > want to continue the discussion here or on the wiki. I think
> > that
> > >> we
> > >> > > > > can decouple the problem of "token distribution" from "shared
> > >> secret
> > >> > > > > distribution" and use the controller as the only token
> generator
> > >> to
> > >> > > > > solve the second issue, while still using ZK async to
> distribute
> > >> > > > > tokens.
> > >> > > > >
> > >> > > > >
> > >> > > > > *As mentioned in the previous Email I am fine with that
> approach
> > >> as
> > >> > > long
> > >> > > > as
> > >> > > > > we agree that the extra complexity of adding/updating APIS
> adds
> > >> enough
> > >> > > > > value. The advantage with the controller approach is secret
> > >> rotation
> > >> > > can
> > >> > > > be
> > >> > > > > automated,frequent and would not require deployment. *
> > >> > > >
> > >> > > > Can you detail the extra complexity (or point me to the email I
> > >> > > > missed?) - which Apis are required?
> > >> > > > As far as I can tell, clients can already find the controller
> from
> > >> > > > metadata. I'm a bit more concerned about controller load.
> > >> > > >
> > >> > > > >
> > >> > > > > * While I like the idea of forcing kerberos auth for renwal, I
> > >> think
> > >> > > > > it mixes the transport layer the the request content in a
> pretty
> > >> ugly
> > >> > > > > way. Perhaps limiting renewer to non-owner is better.
> > >> > > > >
> > >> > > > > *I feel this is a necessary evil. While this will make the
> kafka
> > >> code
> > >> > > > > pretty straight forward , forcing  renewer to non-owner pushes
> > >> the code
> > >> > > > > ugliness to client and makes it even harder to integrate.  *
> > >> > > >
> > >> > > > As mentioned before, I don't think the "renewal by other"
> approach
> > >> is
> > >> > > > that ugly for the clients we expect to use delegation tokens
> since
> > >> > > > they will have an app-master of some sort who requested the
> token
> > to
> > >> > > > begin with.
> > >> > > >
> > >> > > > >
> > >> > > > > The response for my question on how multiple identities will
> be
> > >> > > > > handled wasn't super clear to me - AFAIK, we don't
> authenticate
> > >> each
> > >> > > > > request, we authenticate connections.
> > >> > > > >
> > >> > > > > *We authenticate connections, and only when they are being
> > >> established.
> > >> > > > Let
> > >> > > > > me try to phrase this as a question, in absence of delegation
> > >> tokens if
> > >> > > > we
> > >> > > > > had to support the use case using user TGT's how would we do
> it?
> > >> My
> > >> > > point
> > >> > > > > was it would be no different with delegation tokens. The use
> > case
> > >> you
> > >> > > are
> > >> > > > > describing seems more like impersonation.*
> > >> > > >
> > >> > > > Yeah, I thought that one of the things that delegation tokens
> > >> handled.
> > >> > > > Maybe I got it wrong :)
> > >> > > >
> > >> > > > Thanks for the detailed answers.
> > >> > > >
> > >> > > > Gwen
> > >> > > >
> > >> > > >
> > >> > > > > Thanks
> > >> > > > > Parth
> > >> > > > >
> > >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <
> > gwen@confluent.io
> > >> >
> > >> > > > wrote:
> > >> > > > >
> > >> > > > >> Hi Parth and Harsha,
> > >> > > > >>
> > >> > > > >> Few more comments:
> > >> > > > >>
> > >> > > > >> * The API / RequestResponse section doesn't seem to have good
> > >> > > > >> description of the changes to the Kafka Protocol. Sounds like
> > >> you are
> > >> > > > >> proposing new DelegationTokenRequest and RenewTokenRequest
> (and
> > >> > > > >> matching responses), without detailing the contents of the
> > >> requests
> > >> > > > >> and responses? Or rather, you show the class interface, but
> not
> > >> the
> > >> > > > >> underlying protocol. This could be seen as an implementation
> > >> detail,
> > >> > > > >> but since the binary protocol is what we provide to non-Java
> > >> clients,
> > >> > > > >> we need to show the changes there.
> > >> > > > >>
> > >> > > > >> * getDelegationToken sounds like
> delegationTokenRequestHandler?
> > >> Is it
> > >> > > > >> planned to be part of KafkaApi? or Client? Its unclear...
> > >> > > > >>
> > >> > > > >> * I want to emphasize that even though delegation tokens are
> a
> > >> Hadoop
> > >> > > > >> innovation, I feel very strongly about not adding dependency
> on
> > >> Hadoop
> > >> > > > >> when implementing delegation tokens for Kafka. The KIP
> doesn't
> > >> imply
> > >> > > > >> such dependency, but if you can clarify...
> > >> > > > >>
> > >> > > > >> * Can we get delegation token at any time after
> authenticating?
> > >> only
> > >> > > > >> immediately after?
> > >> > > > >>
> > >> > > > >> * My understanding is that tokens will propagate via ZK but
> > >> without
> > >> > > > >> additional changes to UpdateMetadata protocol, correct?
> Clients
> > >> > > > >> currently don't retry on SASL auth failure (IIRC), but since
> > the
> > >> > > > >> tokens propagate between brokers asynch, we will need to
> retry
> > a
> > >> bit
> > >> > > > >> to avoid clients failing auth due to timing issues.
> > >> > > > >>
> > >> > > > >> * Strongly agreeing on clients not touching ZK directly :)
> > >> > > > >>
> > >> > > > >> * I liked Ashish's suggestion of having just the controller
> > >> issue the
> > >> > > > >> delegation tokens, to avoid syncing a shared secret. Not sure
> > if
> > >> we
> > >> > > > >> want to continue the discussion here or on the wiki. I think
> > >> that we
> > >> > > > >> can decouple the problem of "token distribution" from "shared
> > >> secret
> > >> > > > >> distribution" and use the controller as the only token
> > generator
> > >> to
> > >> > > > >> solve the second issue, while still using ZK async to
> > distribute
> > >> > > > >> tokens.
> > >> > > > >>
> > >> > > > >> * I am also uncomfortable with infinite lifetime of tokens
> (and
> > >> hoped
> > >> > > > >> to hear from others in the community) - but having the option
> > and
> > >> > > > >> documenting it as less secure, allows users to configure
> their
> > >> system
> > >> > > > >> as the wish.
> > >> > > > >>
> > >> > > > >> * While I like the idea of forcing kerberos auth for renwal,
> I
> > >> think
> > >> > > > >> it mixes the transport layer the the request content in a
> > pretty
> > >> ugly
> > >> > > > >> way. Perhaps limiting renewer to non-owner is better.
> > >> > > > >>
> > >> > > > >> Things I'd still like to see:
> > >> > > > >>
> > >> > > > >> * More detailed explanation on what we plan to do for the
> java
> > >> clients
> > >> > > > >> specifically - new configuration? new APIs?
> > >> > > > >> The response for my question on how multiple identities will
> be
> > >> > > > >> handled wasn't super clear to me - AFAIK, we don't
> authenticate
> > >> each
> > >> > > > >> request, we authenticate connections.
> > >> > > > >>
> > >> > > > >> * Alternatives: Delegation tokens are only used in the Hadoop
> > >> > > > >> ecosystem. I'm wondering if there are alternatives in other
> > >> ecosystems
> > >> > > > >> (Mesos? Tachyon? Cassandra?) and whether there are some
> > >> advantages
> > >> > > > >> there.
> > >> > > > >>
> > >> > > > >> Gwen
> > >> > > > >>
> > >> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <ka...@harsha.io>
> > wrote:
> > >> > > > >> > Hi Gwen,
> > >> > > > >> >            Can you look at Parth's last reply. Does it
> answer
> > >> your
> > >> > > > >> >            concerns.
> > >> > > > >> >
> > >> > > > >> > Thanks,
> > >> > > > >> > Harsha
> > >> > > > >> >
> > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt wrote:
> > >> > > > >> >> Thanks for reviewing Gwen. The wiki already has details on
> > >> token
> > >> > > > >> >> expiration
> > >> > > > >> >> under token acquisition process
> > >> > > > >> >> <
> > >> > > > >>
> > >> > > >
> > >> > >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> > >> > > > >> >.
> > >> > > > >> >> Current proposal is that tokens will expire based on a
> > server
> > >> side
> > >> > > > >> >> configuration (default 24 hours) unless renewed. Renewal
> is
> > >> only
> > >> > > > allowed
> > >> > > > >> >> until the max life time of token. Alternatively we could
> > also
> > >> make
> > >> > > > that
> > >> > > > >> >> an
> > >> > > > >> >> optional param and the server side default can serve as
> the
> > >> upper
> > >> > > > bound.
> > >> > > > >> >>
> > >> > > > >> >> To your second point it will be done exactly the same way
> we
> > >> would
> > >> > > > >> >> support
> > >> > > > >> >> multiple keytabs. The calling client will have to put the
> > >> tokens it
> > >> > > > >> wants
> > >> > > > >> >> to use in the subject instance and call produce/consume
> > inside
> > >> > > > >> >> subject.doas. Each caller will have to keep track of its
> own
> > >> > > > subject. I
> > >> > > > >> >> will have to look at the code to see if we support this
> > >> feature
> > >> > > right
> > >> > > > >> now
> > >> > > > >> >> but my understanding is delegation token shouldn't need
> any
> > >> special
> > >> > > > >> >> treatment as its just another type of Credential in the
> > >> subject.
> > >> > > > >> >>
> > >> > > > >> >> I would also like to know what is your opinion about
> > infinite
> > >> > > renewal
> > >> > > > >> (my
> > >> > > > >> >> recommendation is to not support this), tokens renewing
> them
> > >> > > self(my
> > >> > > > >> >> recommendation is to not support this) and most
> importantly
> > >> your
> > >> > > > choice
> > >> > > > >> >> between the alternatives listed on this thread
> > >> > > > >> >> <
> > >> > > > >>
> > >> > > >
> > >> > >
> > >>
> >
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> > >> > > > >> >
> > >> > > > >> >> ( I am leaning towards the alternative-2 minus controller
> > >> > > > distributing
> > >> > > > >> >> secret). Thanks again for reviewing.
> > >> > > > >> >>
> > >> > > > >> >> Thanks
> > >> > > > >> >> Parth
> > >> > > > >> >>
> > >> > > > >> >>
> > >> > > > >> >>
> > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <
> > >> gwen@confluent.io>
> > >> > > > wrote:
> > >> > > > >> >>
> > >> > > > >> >> > Harsha,
> > >> > > > >> >> >
> > >> > > > >> >> > I was thinking of the Rest Proxy. I didn't see your
> design
> > >> yet,
> > >> > > > but in
> > >> > > > >> >> > our proxy, we have a set of producers, which will serve
> > >> multiple
> > >> > > > users
> > >> > > > >> >> > going through the proxy. Since these users will have
> > >> different
> > >> > > > >> >> > privileges, they'll need to authenticate separately, and
> > >> can't
> > >> > > > share a
> > >> > > > >> >> > token.
> > >> > > > >> >> >
> > >> > > > >> >> > Am I missing anything?
> > >> > > > >> >> >
> > >> > > > >> >> > Gwen
> > >> > > > >> >> >
> > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <kafka@harsha.io
> >
> > >> wrote:
> > >> > > > >> >> > > Gwen,
> > >> > > > >> >> > >            On your second point. Can you describe a
> > >> usecase
> > >> > > where
> > >> > > > >> >> > >            mutliple clients ended up sharing a
> producer
> > >> and
> > >> > > even
> > >> > > > if
> > >> > > > >> they
> > >> > > > >> >> > >            do why can't they not use single token that
> > >> producer
> > >> > > > >> >> > >            captures. Why would we need multiple
> clients
> > >> with
> > >> > > > >> different
> > >> > > > >> >> > >            tokens sharing a single instance of
> producer.
> > >> Also
> > >> > > in
> > >> > > > >> this
> > >> > > > >> >> > >            case other clients have access all the
> tokens
> > >> no?
> > >> > > > >> >> > >
> > >> > > > >> >> > > Thanks,
> > >> > > > >> >> > > Harsha
> > >> > > > >> >> > >
> > >> > > > >> >> > >
> > >> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote:
> > >> > > > >> >> > >> Sorry for the delay:
> > >> > > > >> >> > >>
> > >> > > > >> >> > >> Two questions that we didn't see in the wiki:
> > >> > > > >> >> > >> 1. Is there an expiration for delegation tokens?
> > >> Renewal? How
> > >> > > > do we
> > >> > > > >> >> > >> revoke them?
> > >> > > > >> >> > >> 2. If we want to use delegation tokens for "do-as"
> > (say,
> > >> > > submit
> > >> > > > >> Storm
> > >> > > > >> >> > >> job as my user), we will need a producer for every
> job
> > >> (we
> > >> > > can't
> > >> > > > >> share
> > >> > > > >> >> > >> them between multiple jobs running on same node),
> since
> > >> we
> > >> > > only
> > >> > > > >> >> > >> authenticate when connecting. Is there a plan to
> change
> > >> this
> > >> > > for
> > >> > > > >> >> > >> delegation tokens, in order to allow multiple users
> > with
> > >> > > > different
> > >> > > > >> >> > >> tokens to share a client?
> > >> > > > >> >> > >>
> > >> > > > >> >> > >> Gwen
> > >> > > > >> >> > >>
> > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
> > >> > > > >> >> > >> <br...@gmail.com> wrote:
> > >> > > > >> >> > >> > Bumping this up one more time, can other committers
> > >> review?
> > >> > > > >> >> > >> >
> > >> > > > >> >> > >> > Thanks
> > >> > > > >> >> > >> > Parth
> > >> > > > >> >> > >> >
> > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <
> > >> kafka@harsha.io>
> > >> > > > wrote:
> > >> > > > >> >> > >> >
> > >> > > > >> >> > >> >> Parth,
> > >> > > > >> >> > >> >>           Overall current design looks good to
> me. I
> > >> am +1
> > >> > > on
> > >> > > > >> the
> > >> > > > >> >> > KIP.
> > >> > > > >> >> > >> >>
> > >> > > > >> >> > >> >> Gwen , Jun can you review this as well.
> > >> > > > >> >> > >> >>
> > >> > > > >> >> > >> >> -Harsha
> > >> > > > >> >> > >> >>
> > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth
> brahmbhatt
> > >> wrote:
> > >> > > > >> >> > >> >> > Thanks for review Jitendra.
> > >> > > > >> >> > >> >> >
> > >> > > > >> >> > >> >> > I don't like the idea of infinite lifetime but I
> > >> see the
> > >> > > > >> Streaming
> > >> > > > >> >> > use
> > >> > > > >> >> > >> >> > case. Even for Streaming use case I was hoping
> > >> there will
> > >> > > > be
> > >> > > > >> some
> > >> > > > >> >> > notion
> > >> > > > >> >> > >> >> > of
> > >> > > > >> >> > >> >> > master/driver that can get new delegation tokens
> > at
> > >> fixed
> > >> > > > >> interval
> > >> > > > >> >> > and
> > >> > > > >> >> > >> >> > distribute to workers. If that is not the case
> for
> > >> we can
> > >> > > > >> discuss
> > >> > > > >> >> > >> >> > delegation tokens renewing them self and the
> > >> security
> > >> > > > >> implications
> > >> > > > >> >> > of the
> > >> > > > >> >> > >> >> > same.
> > >> > > > >> >> > >> >> >
> > >> > > > >> >> > >> >> > I did not want clients to fetch tokens from
> > >> zookeeper,
> > >> > > > >> overall I
> > >> > > > >> >> > think
> > >> > > > >> >> > >> >> > its
> > >> > > > >> >> > >> >> > better if clients don't rely on our metadata
> store
> > >> and I
> > >> > > > >> think we
> > >> > > > >> >> > are
> > >> > > > >> >> > >> >> > moving in that direction with all the KIP-4
> > >> improvements.
> > >> > > > I
> > >> > > > >> chose
> > >> > > > >> >> > >> >> > zookeeper as in this case the client will still
> > >> just talk
> > >> > > > to
> > >> > > > >> >> > broker , its
> > >> > > > >> >> > >> >> > the brokers that will use zookeeper which we
> > >> already do
> > >> > > > for a
> > >> > > > >> lot
> > >> > > > >> >> > of
> > >> > > > >> >> > >> >> > other
> > >> > > > >> >> > >> >> > usecases + ease of development + and the ability
> > so
> > >> > > tokens
> > >> > > > >> will
> > >> > > > >> >> > survive
> > >> > > > >> >> > >> >> > even a rolling restart/cluster failure. if a
> > >> majority
> > >> > > > agrees
> > >> > > > >> the
> > >> > > > >> >> > added
> > >> > > > >> >> > >> >> > complexity to have controller forwarding keys to
> > all
> > >> > > > broker is
> > >> > > > >> >> > justified
> > >> > > > >> >> > >> >> > as
> > >> > > > >> >> > >> >> > it provides tighter security , I am fine with
> that
> > >> option
> > >> > > > too.
> > >> > > > >> >> > >> >> >
> > >> > > > >> >> > >> >> > Given zookeeper does not support SSL we can not
> > >> store
> > >> > > > master
> > >> > > > >> keys
> > >> > > > >> >> > in
> > >> > > > >> >> > >> >> > zookeeper as master keys will be exposed on
> wire.
> > To
> > >> > > > support
> > >> > > > >> >> > rotation
> > >> > > > >> >> > >> >> > without affecting current clients is something I
> > >> need to
> > >> > > > put
> > >> > > > >> more
> > >> > > > >> >> > thought
> > >> > > > >> >> > >> >> > in. My current proposal assumes the rotation
> will
> > >> > > > invalidate
> > >> > > > >> all
> > >> > > > >> >> > current
> > >> > > > >> >> > >> >> > tokens.
> > >> > > > >> >> > >> >> >
> > >> > > > >> >> > >> >> > I request committers to also review and post
> their
> > >> > > comments
> > >> > > > >> so we
> > >> > > > >> >> > can
> > >> > > > >> >> > >> >> > make
> > >> > > > >> >> > >> >> > progress on this KIP.
> > >> > > > >> >> > >> >> >
> > >> > > > >> >> > >> >> > Thanks
> > >> > > > >> >> > >> >> > Parth
> > >> > > > >> >> > >> >> >
> > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <
> > >> > > > >> asingh@cloudera.com
> > >> > > > >> >> > >
> > >> > > > >> >> > >> >> > wrote:
> > >> > > > >> >> > >> >> >
> > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <
> > >> > > > kafka@harsha.io>
> > >> > > > >> >> > wrote:
> > >> > > > >> >> > >> >> > >
> > >> > > > >> >> > >> >> > > > Unifying the two discussion threads on this
> > KIP.
> > >> > > > >> >> > >> >> > > >
> > >> > > > >> >> > >> >> > > > Here is the response from Jitendra
> > >> > > > >> >> > >> >> > > >
> > >> > > > >> >> > >> >> > > > "The need for a large number of clients that
> > are
> > >> > > > running
> > >> > > > >> all
> > >> > > > >> >> > over the
> > >> > > > >> >> > >> >> > > > cluster that authenticate with Kafka
> brokers,
> > >> is very
> > >> > > > >> similar
> > >> > > > >> >> > to the
> > >> > > > >> >> > >> >> > > > Hadoop use case of large number of tasks
> > running
> > >> > > across
> > >> > > > >> the
> > >> > > > >> >> > cluster
> > >> > > > >> >> > >> >> that
> > >> > > > >> >> > >> >> > > > need authentication to Hdfs Namenode.
> > >> Therefore, the
> > >> > > > >> >> > delegation token
> > >> > > > >> >> > >> >> > > > approach does seem like a good fit for this
> > use
> > >> case
> > >> > > > as we
> > >> > > > >> >> > have seen
> > >> > > > >> >> > >> >> it
> > >> > > > >> >> > >> >> > > > working at large scale in HDFS and YARN.
> > >> > > > >> >> > >> >> > > >
> > >> > > > >> >> > >> >> > > >   The proposed design is very much inline
> with
> > >> Hadoop
> > >> > > > >> >> > approach. A few
> > >> > > > >> >> > >> >> > > >   comments:
> > >> > > > >> >> > >> >> > > >
> > >> > > > >> >> > >> >> > > > 1) Why do you guys want to allow infinite
> > >> renewable
> > >> > > > >> lifetime
> > >> > > > >> >> > for a
> > >> > > > >> >> > >> >> > > > token? HDFS restricts a token to a max life
> > time
> > >> > > > (default
> > >> > > > >> 7
> > >> > > > >> >> > days).  A
> > >> > > > >> >> > >> >> > > > token's vulnerability is believed to
> increase
> > >> with
> > >> > > > time.
> > >> > > > >> >> > >> >> > > >
> > >> > > > >> >> > >> >> > > I agree that having infinite lifetime might
> not
> > >> be the
> > >> > > > best
> > >> > > > >> idea.
> > >> > > > >> >> > >> >> > >
> > >> > > > >> >> > >> >> > > >
> > >> > > > >> >> > >> >> > > > 2) As I understand the tokens are stored in
> > >> zookeeper
> > >> > > > as
> > >> > > > >> well,
> > >> > > > >> >> > and
> > >> > > > >> >> > >> >> can
> > >> > > > >> >> > >> >> > > > be updated there. This is clever as it can
> > allow
> > >> > > > >> replacing the
> > >> > > > >> >> > tokens
> > >> > > > >> >> > >> >> > > > once they run out of max life time, and
> > clients
> > >> can
> > >> > > > >> download
> > >> > > > >> >> > new
> > >> > > > >> >> > >> >> tokens
> > >> > > > >> >> > >> >> > > > from zookeeper. It shouldn't be a big load
> on
> > >> > > zookeeper
> > >> > > > >> as a
> > >> > > > >> >> > client
> > >> > > > >> >> > >> >> will
> > >> > > > >> >> > >> >> > > > need to get a new token once in several
> days.
> > >> In this
> > >> > > > >> approach
> > >> > > > >> >> > you
> > >> > > > >> >> > >> >> don't
> > >> > > > >> >> > >> >> > > > need infinite lifetime on the token even for
> > >> long
> > >> > > > running
> > >> > > > >> >> > clients.
> > >> > > > >> >> > >> >> > > >
> > >> > > > >> >> > >> >> > > > 3) The token password are generated using a
> > >> master
> > >> > > key.
> > >> > > > >> The
> > >> > > > >> >> > master
> > >> > > > >> >> > >> >> key
> > >> > > > >> >> > >> >> > > > should also be periodically changed. In
> > Hadoop,
> > >> the
> > >> > > > >> default
> > >> > > > >> >> > renewal
> > >> > > > >> >> > >> >> > > > period is 1 day.?
> > >> > > > >> >> > >> >> > > >
> > >> > > > >> >> > >> >> > > IIUC, this will require brokers maintaining a
> > >> list of X
> > >> > > > most
> > >> > > > >> >> > recent
> > >> > > > >> >> > >> >> master
> > >> > > > >> >> > >> >> > > keys. This list will have to be persisted
> > >> somewhere, as
> > >> > > > if a
> > >> > > > >> >> > broker
> > >> > > > >> >> > >> >> goes
> > >> > > > >> >> > >> >> > > down it will have to get that list again and
> > >> storing
> > >> > > > master
> > >> > > > >> keys
> > >> > > > >> >> > on ZK
> > >> > > > >> >> > >> >> is
> > >> > > > >> >> > >> >> > > not the best idea. However, if a broker goes
> > down
> > >> then
> > >> > > we
> > >> > > > >> have
> > >> > > > >> >> > much
> > >> > > > >> >> > >> >> bigger
> > >> > > > >> >> > >> >> > > issue to deal with and client can always
> > >> > > re-authenticate
> > >> > > > is
> > >> > > > >> such
> > >> > > > >> >> > >> >> events.
> > >> > > > >> >> > >> >> > >
> > >> > > > >> >> > >> >> > > Did you happen to take a look at other
> > >> alternatives
> > >> > > this
> > >> > > > >> list has
> > >> > > > >> >> > >> >> > > suggested?
> > >> > > > >> >> > >> >> > >
> > >> > > > >> >> > >> >> > > >
> > >> > > > >> >> > >> >> > > > Thanks for a thorough proposal, great work!"
> > >> > > > >> >> > >> >> > > >
> > >> > > > >> >> > >> >> > > >
> > >> > > > >> >> > >> >> > > >
> > >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen
> Shapira
> > >> wrote:
> > >> > > > >> >> > >> >> > > > > Makes sense to me. Thanks!
> > >> > > > >> >> > >> >> > > > >
> > >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <
> > >> > > > kafka@harsha.io
> > >> > > > >> >
> > >> > > > >> >> > wrote:
> > >> > > > >> >> > >> >> > > > > > It doesn't need any release vehicle but
> > >> still the
> > >> > > > >> work can
> > >> > > > >> >> > move
> > >> > > > >> >> > >> >> > > > forward.
> > >> > > > >> >> > >> >> > > > > > If anyone is interested in the KIP
> please
> > >> do the
> > >> > > > >> review and
> > >> > > > >> >> > >> >> provide
> > >> > > > >> >> > >> >> > > the
> > >> > > > >> >> > >> >> > > > > > comments.
> > >> > > > >> >> > >> >> > > > > >
> > >> > > > >> >> > >> >> > > > > > -Harsha
> > >> > > > >> >> > >> >> > > > > >
> > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael
> > >> Juma
> > >> > > > wrote:
> > >> > > > >> >> > >> >> > > > > >> I agree that it would be good to have
> > more
> > >> time
> > >> > > to
> > >> > > > >> review
> > >> > > > >> >> > and
> > >> > > > >> >> > >> >> > > discuss
> > >> > > > >> >> > >> >> > > > > >> KIP-48.
> > >> > > > >> >> > >> >> > > > > >>
> > >> > > > >> >> > >> >> > > > > >> Ismael
> > >> > > > >> >> > >> >> > > > > >>
> > >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen
> > >> Shapira <
> > >> > > > >> >> > >> >> gwen@confluent.io>
> > >> > > > >> >> > >> >> > > > wrote:
> > >> > > > >> >> > >> >> > > > > >>
> > >> > > > >> >> > >> >> > > > > >> > Hi Team,
> > >> > > > >> >> > >> >> > > > > >> >
> > >> > > > >> >> > >> >> > > > > >> > Since KIP-48 depends on KIP-43, which
> > is
> > >> > > > already a
> > >> > > > >> bit
> > >> > > > >> >> > of a
> > >> > > > >> >> > >> >> risk
> > >> > > > >> >> > >> >> > > for
> > >> > > > >> >> > >> >> > > > > >> > the next release - any chance we can
> > >> delay
> > >> > > > >> delegation
> > >> > > > >> >> > tokens
> > >> > > > >> >> > >> >> to
> > >> > > > >> >> > >> >> > > > Kafka
> > >> > > > >> >> > >> >> > > > > >> > 0.10.1?
> > >> > > > >> >> > >> >> > > > > >> > With the community working on a
> release
> > >> every
> > >> > > 3
> > >> > > > >> month,
> > >> > > > >> >> > this
> > >> > > > >> >> > >> >> is not
> > >> > > > >> >> > >> >> > > > a huge
> > >> > > > >> >> > >> >> > > > > >> > delay.
> > >> > > > >> >> > >> >> > > > > >> >
> > >> > > > >> >> > >> >> > > > > >> > Gwen
> > >> > > > >> >> > >> >> > > > > >> >
> > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM,
> Ashish
> > >> Singh
> > >> > > <
> > >> > > > >> >> > >> >> > > asingh@cloudera.com>
> > >> > > > >> >> > >> >> > > > wrote:
> > >> > > > >> >> > >> >> > > > > >> > > Parth,
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > > Thanks again for the awesome write
> > up.
> > >> > > > Following
> > >> > > > >> our
> > >> > > > >> >> > >> >> discussion
> > >> > > > >> >> > >> >> > > > from the
> > >> > > > >> >> > >> >> > > > > >> > > JIRA, I think it will be easier to
> > >> compare
> > >> > > > >> various
> > >> > > > >> >> > >> >> alternatives
> > >> > > > >> >> > >> >> > > > if they
> > >> > > > >> >> > >> >> > > > > >> > are
> > >> > > > >> >> > >> >> > > > > >> > > listed together. I am stating
> below a
> > >> few
> > >> > > > >> >> > alternatives along
> > >> > > > >> >> > >> >> > > with
> > >> > > > >> >> > >> >> > > > a the
> > >> > > > >> >> > >> >> > > > > >> > > current proposal.
> > >> > > > >> >> > >> >> > > > > >> > > (Current proposal) Store Delegation
> > >> Token,
> > >> > > DT,
> > >> > > > >> on ZK.
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a
> > >> broker.
> > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> authenticated,
> > >> it
> > >> > > will
> > >> > > > >> make a
> > >> > > > >> >> > broker
> > >> > > > >> >> > >> >> side
> > >> > > > >> >> > >> >> > > > call to
> > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > >> > > > >> >> > >> >> > > > > >> > >    3. The broker generates a shared
> > >> secret
> > >> > > > based
> > >> > > > >> on
> > >> > > > >> >> > >> >> > > HMAC-SHA256(a
> > >> > > > >> >> > >> >> > > > > >> > >    Password/Secret shared between
> all
> > >> > > brokers,
> > >> > > > >> >> > randomly
> > >> > > > >> >> > >> >> > > generated
> > >> > > > >> >> > >> >> > > > > >> > tokenId).
> > >> > > > >> >> > >> >> > > > > >> > >    4. Broker stores this token in
> its
> > >> in
> > >> > > > memory
> > >> > > > >> cache.
> > >> > > > >> >> > >> >> Broker
> > >> > > > >> >> > >> >> > > > also stores
> > >> > > > >> >> > >> >> > > > > >> > >    the DelegationToken without the
> > >> hmac in
> > >> > > the
> > >> > > > >> >> > zookeeper.
> > >> > > > >> >> > >> >> > > > > >> > >    5. All brokers will have a cache
> > >> backed
> > >> > > by
> > >> > > > >> >> > zookeeper so
> > >> > > > >> >> > >> >> they
> > >> > > > >> >> > >> >> > > > will all
> > >> > > > >> >> > >> >> > > > > >> > >    get notified whenever a new
> token
> > is
> > >> > > > >> generated and
> > >> > > > >> >> > they
> > >> > > > >> >> > >> >> will
> > >> > > > >> >> > >> >> > > > update
> > >> > > > >> >> > >> >> > > > > >> > their
> > >> > > > >> >> > >> >> > > > > >> > >    local cache whenever token state
> > >> changes.
> > >> > > > >> >> > >> >> > > > > >> > >    6. Broker returns the token to
> > >> Client.
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > >    1. Probable race condition,
> client
> > >> tries
> > >> > > to
> > >> > > > >> >> > authenticate
> > >> > > > >> >> > >> >> with
> > >> > > > >> >> > >> >> > > > a broker
> > >> > > > >> >> > >> >> > > > > >> > >    that is yet to be updated with
> the
> > >> newly
> > >> > > > >> generated
> > >> > > > >> >> > DT.
> > >> > > > >> >> > >> >> This
> > >> > > > >> >> > >> >> > > can
> > >> > > > >> >> > >> >> > > > > >> > probably be
> > >> > > > >> >> > >> >> > > > > >> > >    dealt with making dtRequest
> block
> > >> until
> > >> > > all
> > >> > > > >> >> > brokers have
> > >> > > > >> >> > >> >> > > > updated
> > >> > > > >> >> > >> >> > > > > >> > their DT
> > >> > > > >> >> > >> >> > > > > >> > >    cache. Zk barrier or similar
> > >> mechanism
> > >> > > can
> > >> > > > be
> > >> > > > >> used.
> > >> > > > >> >> > >> >> However,
> > >> > > > >> >> > >> >> > > > all such
> > >> > > > >> >> > >> >> > > > > >> > >    mechanisms will increase
> > complexity.
> > >> > > > >> >> > >> >> > > > > >> > >    2. Using a static secret key
> from
> > >> config
> > >> > > > >> file. Will
> > >> > > > >> >> > >> >> require
> > >> > > > >> >> > >> >> > > yet
> > >> > > > >> >> > >> >> > > > > >> > another
> > >> > > > >> >> > >> >> > > > > >> > >    config and uses a static secret
> > >> key. It
> > >> > > is
> > >> > > > >> advised
> > >> > > > >> >> > to
> > >> > > > >> >> > >> >> rotate
> > >> > > > >> >> > >> >> > > > secret
> > >> > > > >> >> > >> >> > > > > >> > keys
> > >> > > > >> >> > >> >> > > > > >> > >    periodically. This can be
> avoided
> > >> with
> > >> > > > >> controller
> > >> > > > >> >> > >> >> generating
> > >> > > > >> >> > >> >> > > > > >> > secretKey and
> > >> > > > >> >> > >> >> > > > > >> > >    passing to brokers periodically.
> > >> However,
> > >> > > > >> this will
> > >> > > > >> >> > >> >> require
> > >> > > > >> >> > >> >> > > > brokers to
> > >> > > > >> >> > >> >> > > > > >> > >    maintain certain counts of
> > >> secretKeys.
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > > (Alternative 1) Have controller
> > >> generate
> > >> > > > >> delegation
> > >> > > > >> >> > token.
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a
> > >> broker.
> > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> authenticated,
> > >> it
> > >> > > will
> > >> > > > >> make a
> > >> > > > >> >> > broker
> > >> > > > >> >> > >> >> side
> > >> > > > >> >> > >> >> > > > call to
> > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > >> > > > >> >> > >> >> > > > > >> > >    3. Broker forwards the request
> to
> > >> > > > controller.
> > >> > > > >> >> > >> >> > > > > >> > >    4. Controller generates a DT and
> > >> > > broadcasts
> > >> > > > >> to all
> > >> > > > >> >> > >> >> brokers.
> > >> > > > >> >> > >> >> > > > > >> > >    5. Broker stores this token in
> its
> > >> memory
> > >> > > > >> cache.
> > >> > > > >> >> > >> >> > > > > >> > >    6. Controller responds to
> broker’s
> > >> DT
> > >> > > req.
> > >> > > > >> >> > >> >> > > > > >> > >    7. Broker returns the token to
> > >> Client.
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > >    1. We will have to add new APIs
> to
> > >> > > support
> > >> > > > >> >> > controller
> > >> > > > >> >> > >> >> pushing
> > >> > > > >> >> > >> >> > > > tokens
> > >> > > > >> >> > >> >> > > > > >> > to
> > >> > > > >> >> > >> >> > > > > >> > >    brokers on top of the minimal
> APIs
> > >> that
> > >> > > are
> > >> > > > >> >> > currently
> > >> > > > >> >> > >> >> > > proposed.
> > >> > > > >> >> > >> >> > > > > >> > >    2. We will also have to add APIs
> > to
> > >> > > support
> > >> > > > >> the
> > >> > > > >> >> > >> >> bootstrapping
> > >> > > > >> >> > >> >> > > > case,
> > >> > > > >> >> > >> >> > > > > >> > i.e,
> > >> > > > >> >> > >> >> > > > > >> > >    when a new broker comes up it
> will
> > >> have
> > >> > > to
> > >> > > > >> get all
> > >> > > > >> >> > >> >> delegation
> > >> > > > >> >> > >> >> > > > tokens
> > >> > > > >> >> > >> >> > > > > >> > from
> > >> > > > >> >> > >> >> > > > > >> > >    the controller.
> > >> > > > >> >> > >> >> > > > > >> > >    3. In catastrophic failures
> where
> > >> all
> > >> > > > brokers
> > >> > > > >> go
> > >> > > > >> >> > down,
> > >> > > > >> >> > >> >> the
> > >> > > > >> >> > >> >> > > > tokens will
> > >> > > > >> >> > >> >> > > > > >> > >    be lost even if servers are
> > >> restarted as
> > >> > > > >> tokens
> > >> > > > >> >> > are not
> > >> > > > >> >> > >> >> > > > persisted
> > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > >> > > > >> >> > >> >> > > > > >> > >    If this happens, then there are
> > more
> > >> > > > important
> > >> > > > >> >> > things to
> > >> > > > >> >> > >> >> > > worry
> > >> > > > >> >> > >> >> > > > about
> > >> > > > >> >> > >> >> > > > > >> > and
> > >> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
> > >> re-authenticate.
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > > (Alternative 2) Do not distribute
> DT
> > to
> > >> > > other
> > >> > > > >> brokers
> > >> > > > >> >> > at
> > >> > > > >> >> > >> >> all.
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a
> > >> broker.
> > >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is
> authenticated,
> > >> it
> > >> > > will
> > >> > > > >> make a
> > >> > > > >> >> > broker
> > >> > > > >> >> > >> >> side
> > >> > > > >> >> > >> >> > > > call to
> > >> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > >> > > > >> >> > >> >> > > > > >> > >    3. The broker generates DT of
> > form,
> > >> > > [hmac +
> > >> > > > >> (owner,
> > >> > > > >> >> > >> >> renewer,
> > >> > > > >> >> > >> >> > > > > >> > >    maxLifeTime, id, hmac,
> > >> expirationTime)]
> > >> > > and
> > >> > > > >> passes
> > >> > > > >> >> > back
> > >> > > > >> >> > >> >> this
> > >> > > > >> >> > >> >> > > > DT to
> > >> > > > >> >> > >> >> > > > > >> > client.
> > >> > > > >> >> > >> >> > > > > >> > >    hmac is generated via
> > >> {HMAC-SHA256(owner,
> > >> > > > >> renewer,
> > >> > > > >> >> > >> >> > > > maxLifeTime, id,
> > >> > > > >> >> > >> >> > > > > >> > hmac,
> > >> > > > >> >> > >> >> > > > > >> > >    expirationTime) using
> SecretKey}.
> > >> Note
> > >> > > that
> > >> > > > >> all
> > >> > > > >> >> > brokers
> > >> > > > >> >> > >> >> have
> > >> > > > >> >> > >> >> > > > this
> > >> > > > >> >> > >> >> > > > > >> > SecretKey.
> > >> > > > >> >> > >> >> > > > > >> > >    4. Client then goes to any
> broker
> > >> and to
> > >> > > > >> >> > authenticate
> > >> > > > >> >> > >> >> sends
> > >> > > > >> >> > >> >> > > > the DT.
> > >> > > > >> >> > >> >> > > > > >> > >    Broker recalculates hmac using
> > >> (owner,
> > >> > > > >> renewer,
> > >> > > > >> >> > >> >> maxLifeTime,
> > >> > > > >> >> > >> >> > > > id, hmac,
> > >> > > > >> >> > >> >> > > > > >> > >    expirationTime) info from DT and
> > its
> > >> > > > >> SecretKey. If
> > >> > > > >> >> > it
> > >> > > > >> >> > >> >> matches
> > >> > > > >> >> > >> >> > > > with
> > >> > > > >> >> > >> >> > > > > >> > hmac of
> > >> > > > >> >> > >> >> > > > > >> > >    DT, client is authenticated.
> Yes,
> > >> it will
> > >> > > > do
> > >> > > > >> other
> > >> > > > >> >> > >> >> obvious
> > >> > > > >> >> > >> >> > > > checks of
> > >> > > > >> >> > >> >> > > > > >> > >    timestamp expiry and such.
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > > Note that secret key will be
> > generated
> > >> by
> > >> > > > >> controller
> > >> > > > >> >> > and
> > >> > > > >> >> > >> >> passed
> > >> > > > >> >> > >> >> > > to
> > >> > > > >> >> > >> >> > > > > >> > brokers
> > >> > > > >> >> > >> >> > > > > >> > > periodically.
> > >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > >    1. How to delete a DT? Yes, that
> > is
> > >> a
> > >> > > > downside
> > >> > > > >> >> > here.
> > >> > > > >> >> > >> >> However,
> > >> > > > >> >> > >> >> > > > this can
> > >> > > > >> >> > >> >> > > > > >> > >    be handled with brokers
> > maintaining
> > >> a
> > >> > > > >> blacklist of
> > >> > > > >> >> > DTs,
> > >> > > > >> >> > >> >> DTs
> > >> > > > >> >> > >> >> > > > from this
> > >> > > > >> >> > >> >> > > > > >> > list
> > >> > > > >> >> > >> >> > > > > >> > >    can be removed after expiry.
> > >> > > > >> >> > >> >> > > > > >> > >    2. In catastrophic failures
> where
> > >> all
> > >> > > > brokers
> > >> > > > >> go
> > >> > > > >> >> > down,
> > >> > > > >> >> > >> >> the
> > >> > > > >> >> > >> >> > > > tokens will
> > >> > > > >> >> > >> >> > > > > >> > >    be lost even if servers are
> > >> restarted as
> > >> > > > >> tokens
> > >> > > > >> >> > are not
> > >> > > > >> >> > >> >> > > > persisted
> > >> > > > >> >> > >> >> > > > > >> > anywhere.
> > >> > > > >> >> > >> >> > > > > >> > >    If this happens, then there are
> > more
> > >> > > > important
> > >> > > > >> >> > things to
> > >> > > > >> >> > >> >> > > worry
> > >> > > > >> >> > >> >> > > > about
> > >> > > > >> >> > >> >> > > > > >> > and
> > >> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
> > >> re-authenticate.
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM,
> > Parth
> > >> > > > >> Brahmbhatt <
> > >> > > > >> >> > >> >> > > > > >> > > pbrahmbhatt@hortonworks.com>
> wrote:
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > >> Hi,
> > >> > > > >> >> > >> >> > > > > >> > >>
> > >> > > > >> >> > >> >> > > > > >> > >> I have filed KIP-48 so we can
> offer
> > >> hadoop
> > >> > > > like
> > >> > > > >> >> > delegation
> > >> > > > >> >> > >> >> > > > tokens in
> > >> > > > >> >> > >> >> > > > > >> > >> kafka. You can review the design
> > >> > > > >> >> > >> >> > > > > >> > >>
> > >> > > > >> >> > >> >> > > > > >> >
> > >> > > > >> >> > >> >> > > >
> > >> > > > >> >> > >> >> > >
> > >> > > > >> >> > >> >>
> > >> > > > >> >> >
> > >> > > > >>
> > >> > > >
> > >> > >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > >> > > > >> >> > >> >> > > > > >> > .
> > >> > > > >> >> > >> >> > > > > >> > >> This KIP depends on KIP-43 and we
> > >> have also
> > >> > > > >> >> > discussed an
> > >> > > > >> >> > >> >> > > > alternative to
> > >> > > > >> >> > >> >> > > > > >> > >> proposed design here<
> > >> > > > >> >> > >> >> > > > > >> > >>
> > >> > > > >> >> > >> >> > > > > >> >
> > >> > > > >> >> > >> >> > > >
> > >> > > > >> >> > >> >> > >
> > >> > > > >> >> > >> >>
> > >> > > > >> >> >
> > >> > > > >>
> > >> > > >
> > >> > >
> > >>
> >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > >> > > > >> >> > >> >> > > > > >> > >> >.
> > >> > > > >> >> > >> >> > > > > >> > >>
> > >> > > > >> >> > >> >> > > > > >> > >> Thanks
> > >> > > > >> >> > >> >> > > > > >> > >> Parth
> > >> > > > >> >> > >> >> > > > > >> > >>
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > > --
> > >> > > > >> >> > >> >> > > > > >> > >
> > >> > > > >> >> > >> >> > > > > >> > > Regards,
> > >> > > > >> >> > >> >> > > > > >> > > Ashish
> > >> > > > >> >> > >> >> > > > > >> >
> > >> > > > >> >> > >> >> > > >
> > >> > > > >> >> > >> >> > >
> > >> > > > >> >> > >> >> > >
> > >> > > > >> >> > >> >> > >
> > >> > > > >> >> > >> >> > > --
> > >> > > > >> >> > >> >> > >
> > >> > > > >> >> > >> >> > > Regards,
> > >> > > > >> >> > >> >> > > Ashish
> > >> > > > >> >> > >> >> > >
> > >> > > > >> >> > >> >>
> > >> > > > >> >> >
> > >> > > > >>
> > >> > > >
> > >> > >
> > >>
> > >>
> > >
> >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[parth brahmbhatt <br...@gmail.com>]

Hi Jun,

Thanks for reviewing.

* We could add a Cluster action to add acls on who can request delegation
tokens. I don't see the use case for that yet but down the line when we
start supporting getDelegationTokenAs it will be necessary.
* Yes we recommend tokens to be only used/distributed over secure channels.
* Depending on what design we end up choosing Invalidation will be
responsibility of every broker or controller.
* I am not sure if I documented somewhere that invalidation will directly
go through zookeeper but that is not the intent. Invalidation will either
be request based or due to expiration. No direct zookeeper interaction from
any client.
* "Broker also stores the DelegationToken without the hmac in the
zookeeper." : Sorry about the confusion. The sole purpose of zookeeper in
this design is as distribution channel for tokens between all brokers and a
layer that ensures only tokens that were generated by making a request to a
broker will be accepted (more on this in second paragraph). The token
consists of few elements (owner, renewer, uuid , expiration, hmac) , one of
which is the finally generated hmac but hmac it self is derivable if you
have all the other elements of the token + secret key to generate hmac.
Given zookeeper does not provide SSL support we do not want the entire
token to be wire transferred to zookeeper as that will be an insecure wire
transfer. Instead we only store all the other elements of a delegation
tokens. Brokers can read these elements and because they also have access
to secret key they will be able to generate hmac on their end.

One of the alternative proposed is to avoid zookeeper altogether. A Client
will call broker with required information (owner, renwer, expiration) and
get back (signed hmac, uuid). Broker won't store this in zookeeper. From
this point a client can contact any broker with all the delegation token
info (owner, rewner, expiration, hmac, uuid) the borker will regenerate the
hmac and as long as it matches with hmac presented by client , broker will
allow the request to authenticate.  Only problem with this approach is if
the secret key is compromised any client can now generate random tokens and
they will still be able to authenticate as any user they like. with
zookeeper we guarantee that only tokens acquired via a broker (using some
auth scheme other than delegation token) will be accepted. We need to
discuss which proposal makes more sense and we can go over it in tomorrow's
meeting.

Also, can you forward the invite to me?

Thanks
Parth



On Mon, May 23, 2016 at 10:35 AM, Jun Rao <ju...@confluent.io> wrote:

> Thanks for the KIP. A few comments.
>
> 100. This potentially can be useful for Kafka Connect and Kafka rest proxy
> where a worker agent will need to run a task on behalf of a client. We will
> likely need to change how those services use Kafka clients
> (producer/consumer). Instead of a shared client per worker, we will need a
> client per user task since the authentication happens at the connection
> level. For Kafka Connect, the renewer will be the workers. So, we probably
> need to allow multiple renewers. For Kafka rest proxy, the renewer can
> probably just be the creator of the token.
>
> 101. Do we need new acl on who can request delegation tokens?
>
> 102. Do we recommend people to send delegation tokens in an encrypted
> channel?
>
> 103. Who is responsible for expiring tokens, every broker?
>
> 104. For invalidating tokens, would it be better to do it in a request
> instead of going to ZK directly?
>
> 105. The terminology of client in the wiki sometimes refers to the end
> client and some other times refers to the client using the delegation
> tokens. It would be useful to distinguish between the two.
>
> 106. Could you explain the sentence "Broker also stores the DelegationToken
> without the hmac in the zookeeper." a bit more? I thought the delegation
> token is the hmac.
>
> Thanks,
>
> Jun
>
>
> On Mon, May 23, 2016 at 9:22 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > Hi, Harsha,
> >
> > Just sent out a KIP meeting invite. We can discuss this in the meeting
> > tomorrow.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, May 19, 2016 at 8:47 AM, Harsha <ka...@harsha.io> wrote:
> >
> >> Hi All,
> >>            Can we have a KIP meeting around this. The KIP is up for
> >>            sometime and if there are any questions lets quickly hash out
> >>            details.
> >>
> >> Thanks,
> >> Harsha
> >>
> >> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt wrote:
> >> > That is what the hadoop echo system uses so no good reason really. We
> >> > could
> >> > change it to whatever is the newest recommended standard is.
> >> >
> >> > Thanks
> >> > Parth
> >> >
> >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <is...@juma.me.uk>
> wrote:
> >> >
> >> > > Hi Parth,
> >> > >
> >> > > Thanks for the KIP. I only started reviewing this and may have
> >> additional
> >> > > questions later. The immediate question that came to mind is our
> >> choice of
> >> > > "DIGEST-MD5" even though it's marked as OBSOLETE in the IANA
> Registry
> >> of
> >> > > SASL mechanisms and the original RFC (2831) has been moved to
> Historic
> >> > > status:
> >> > >
> >> > > https://tools.ietf.org/html/rfc6331
> >> > >
> http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> >> > >
> >> > > What is the reasoning behind that choice?
> >> > >
> >> > > Thanks,
> >> > > Ismael
> >> > >
> >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <gw...@confluent.io>
> >> wrote:
> >> > >
> >> > > > Also comments inline :)
> >> > > >
> >> > > > > * I want to emphasize that even though delegation tokens are a
> >> Hadoop
> >> > > > > innovation, I feel very strongly about not adding dependency on
> >> Hadoop
> >> > > > > when implementing delegation tokens for Kafka. The KIP doesn't
> >> imply
> >> > > > > such dependency, but if you can clarify...
> >> > > > >
> >> > > > >
> >> > > > > *No hadoop dependency.*
> >> > > >
> >> > > > Yay! Just add this to the KIP so no one will read the KIP and
> panic
> >> > > > three weeks before the next release...
> >> > > >
> >> > > > > * Can we get delegation token at any time after authenticating?
> >> only
> >> > > > > immediately after?
> >> > > > >
> >> > > > >
> >> > > > > *As long as you are authenticated you can get delegation tokens.
> >> We
> >> > > need
> >> > > > to
> >> > > > > discuss if a client authenticated using delegation token, can
> also
> >> > > > acquire
> >> > > > > delegation token again or not. Also there is the question of do
> we
> >> > > allow
> >> > > > > anyone to acquire delegation token or we want specific ACLs (I
> >> think
> >> > > its
> >> > > > an
> >> > > > > overkill.)*
> >> > > >
> >> > > > I agree that ACLs is an overkill.
> >> > > >
> >> > > > I think we are debating two options: Either require Kerberos auth
> >> for
> >> > > > renewal or require non-owners to renew.
> >> > > > I *think* the latter is simpler (it basically require a "job
> master"
> >> > > > to take responsibility for the renewal, it will have its own
> >> identity
> >> > > > anyway and I think this is the correct design pattern anyway. For
> >> > > > storm, I'd expect Nimbus to coordinate renewals?), but it is hard
> to
> >> > > > debate simplicity without looking at the code changes required. If
> >> you
> >> > > > have a draft of how the "require Kerberos" will look in Kafka
> code,
> >> > > > I'll be happy to take a look.
> >> > > >
> >> > > > > * My understanding is that tokens will propagate via ZK but
> >> without
> >> > > > > additional changes to UpdateMetadata protocol, correct? Clients
> >> > > > > currently don't retry on SASL auth failure (IIRC), but since the
> >> > > > > tokens propagate between brokers asynch, we will need to retry a
> >> bit
> >> > > > > to avoid clients failing auth due to timing issues.
> >> > > > >
> >> > > > > *I am considering 2 alternatives right now. The current
> documented
> >> > > > approach
> >> > > > > is zookeeper based and it does not require any changes to
> >> > > UpdateMetadata
> >> > > > > protocol. An alternative approach can remove zookeeper
> dependency
> >> as
> >> > > well
> >> > > > > but we can discuss that in KIP discussion call.*
> >> > > >
> >> > > > Oooh! Sounds interesting. Do you want to ping Jun to arrange a
> call?
> >> > > >
> >> > > > > * I liked Ashish's suggestion of having just the controller
> issue
> >> the
> >> > > > > delegation tokens, to avoid syncing a shared secret. Not sure if
> >> we
> >> > > > > want to continue the discussion here or on the wiki. I think
> that
> >> we
> >> > > > > can decouple the problem of "token distribution" from "shared
> >> secret
> >> > > > > distribution" and use the controller as the only token generator
> >> to
> >> > > > > solve the second issue, while still using ZK async to distribute
> >> > > > > tokens.
> >> > > > >
> >> > > > >
> >> > > > > *As mentioned in the previous Email I am fine with that approach
> >> as
> >> > > long
> >> > > > as
> >> > > > > we agree that the extra complexity of adding/updating APIS adds
> >> enough
> >> > > > > value. The advantage with the controller approach is secret
> >> rotation
> >> > > can
> >> > > > be
> >> > > > > automated,frequent and would not require deployment. *
> >> > > >
> >> > > > Can you detail the extra complexity (or point me to the email I
> >> > > > missed?) - which Apis are required?
> >> > > > As far as I can tell, clients can already find the controller from
> >> > > > metadata. I'm a bit more concerned about controller load.
> >> > > >
> >> > > > >
> >> > > > > * While I like the idea of forcing kerberos auth for renwal, I
> >> think
> >> > > > > it mixes the transport layer the the request content in a pretty
> >> ugly
> >> > > > > way. Perhaps limiting renewer to non-owner is better.
> >> > > > >
> >> > > > > *I feel this is a necessary evil. While this will make the kafka
> >> code
> >> > > > > pretty straight forward , forcing  renewer to non-owner pushes
> >> the code
> >> > > > > ugliness to client and makes it even harder to integrate.  *
> >> > > >
> >> > > > As mentioned before, I don't think the "renewal by other" approach
> >> is
> >> > > > that ugly for the clients we expect to use delegation tokens since
> >> > > > they will have an app-master of some sort who requested the token
> to
> >> > > > begin with.
> >> > > >
> >> > > > >
> >> > > > > The response for my question on how multiple identities will be
> >> > > > > handled wasn't super clear to me - AFAIK, we don't authenticate
> >> each
> >> > > > > request, we authenticate connections.
> >> > > > >
> >> > > > > *We authenticate connections, and only when they are being
> >> established.
> >> > > > Let
> >> > > > > me try to phrase this as a question, in absence of delegation
> >> tokens if
> >> > > > we
> >> > > > > had to support the use case using user TGT's how would we do it?
> >> My
> >> > > point
> >> > > > > was it would be no different with delegation tokens. The use
> case
> >> you
> >> > > are
> >> > > > > describing seems more like impersonation.*
> >> > > >
> >> > > > Yeah, I thought that one of the things that delegation tokens
> >> handled.
> >> > > > Maybe I got it wrong :)
> >> > > >
> >> > > > Thanks for the detailed answers.
> >> > > >
> >> > > > Gwen
> >> > > >
> >> > > >
> >> > > > > Thanks
> >> > > > > Parth
> >> > > > >
> >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <
> gwen@confluent.io
> >> >
> >> > > > wrote:
> >> > > > >
> >> > > > >> Hi Parth and Harsha,
> >> > > > >>
> >> > > > >> Few more comments:
> >> > > > >>
> >> > > > >> * The API / RequestResponse section doesn't seem to have good
> >> > > > >> description of the changes to the Kafka Protocol. Sounds like
> >> you are
> >> > > > >> proposing new DelegationTokenRequest and RenewTokenRequest (and
> >> > > > >> matching responses), without detailing the contents of the
> >> requests
> >> > > > >> and responses? Or rather, you show the class interface, but not
> >> the
> >> > > > >> underlying protocol. This could be seen as an implementation
> >> detail,
> >> > > > >> but since the binary protocol is what we provide to non-Java
> >> clients,
> >> > > > >> we need to show the changes there.
> >> > > > >>
> >> > > > >> * getDelegationToken sounds like delegationTokenRequestHandler?
> >> Is it
> >> > > > >> planned to be part of KafkaApi? or Client? Its unclear...
> >> > > > >>
> >> > > > >> * I want to emphasize that even though delegation tokens are a
> >> Hadoop
> >> > > > >> innovation, I feel very strongly about not adding dependency on
> >> Hadoop
> >> > > > >> when implementing delegation tokens for Kafka. The KIP doesn't
> >> imply
> >> > > > >> such dependency, but if you can clarify...
> >> > > > >>
> >> > > > >> * Can we get delegation token at any time after authenticating?
> >> only
> >> > > > >> immediately after?
> >> > > > >>
> >> > > > >> * My understanding is that tokens will propagate via ZK but
> >> without
> >> > > > >> additional changes to UpdateMetadata protocol, correct? Clients
> >> > > > >> currently don't retry on SASL auth failure (IIRC), but since
> the
> >> > > > >> tokens propagate between brokers asynch, we will need to retry
> a
> >> bit
> >> > > > >> to avoid clients failing auth due to timing issues.
> >> > > > >>
> >> > > > >> * Strongly agreeing on clients not touching ZK directly :)
> >> > > > >>
> >> > > > >> * I liked Ashish's suggestion of having just the controller
> >> issue the
> >> > > > >> delegation tokens, to avoid syncing a shared secret. Not sure
> if
> >> we
> >> > > > >> want to continue the discussion here or on the wiki. I think
> >> that we
> >> > > > >> can decouple the problem of "token distribution" from "shared
> >> secret
> >> > > > >> distribution" and use the controller as the only token
> generator
> >> to
> >> > > > >> solve the second issue, while still using ZK async to
> distribute
> >> > > > >> tokens.
> >> > > > >>
> >> > > > >> * I am also uncomfortable with infinite lifetime of tokens (and
> >> hoped
> >> > > > >> to hear from others in the community) - but having the option
> and
> >> > > > >> documenting it as less secure, allows users to configure their
> >> system
> >> > > > >> as the wish.
> >> > > > >>
> >> > > > >> * While I like the idea of forcing kerberos auth for renwal, I
> >> think
> >> > > > >> it mixes the transport layer the the request content in a
> pretty
> >> ugly
> >> > > > >> way. Perhaps limiting renewer to non-owner is better.
> >> > > > >>
> >> > > > >> Things I'd still like to see:
> >> > > > >>
> >> > > > >> * More detailed explanation on what we plan to do for the java
> >> clients
> >> > > > >> specifically - new configuration? new APIs?
> >> > > > >> The response for my question on how multiple identities will be
> >> > > > >> handled wasn't super clear to me - AFAIK, we don't authenticate
> >> each
> >> > > > >> request, we authenticate connections.
> >> > > > >>
> >> > > > >> * Alternatives: Delegation tokens are only used in the Hadoop
> >> > > > >> ecosystem. I'm wondering if there are alternatives in other
> >> ecosystems
> >> > > > >> (Mesos? Tachyon? Cassandra?) and whether there are some
> >> advantages
> >> > > > >> there.
> >> > > > >>
> >> > > > >> Gwen
> >> > > > >>
> >> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <ka...@harsha.io>
> wrote:
> >> > > > >> > Hi Gwen,
> >> > > > >> >            Can you look at Parth's last reply. Does it answer
> >> your
> >> > > > >> >            concerns.
> >> > > > >> >
> >> > > > >> > Thanks,
> >> > > > >> > Harsha
> >> > > > >> >
> >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt wrote:
> >> > > > >> >> Thanks for reviewing Gwen. The wiki already has details on
> >> token
> >> > > > >> >> expiration
> >> > > > >> >> under token acquisition process
> >> > > > >> >> <
> >> > > > >>
> >> > > >
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> >> > > > >> >.
> >> > > > >> >> Current proposal is that tokens will expire based on a
> server
> >> side
> >> > > > >> >> configuration (default 24 hours) unless renewed. Renewal is
> >> only
> >> > > > allowed
> >> > > > >> >> until the max life time of token. Alternatively we could
> also
> >> make
> >> > > > that
> >> > > > >> >> an
> >> > > > >> >> optional param and the server side default can serve as the
> >> upper
> >> > > > bound.
> >> > > > >> >>
> >> > > > >> >> To your second point it will be done exactly the same way we
> >> would
> >> > > > >> >> support
> >> > > > >> >> multiple keytabs. The calling client will have to put the
> >> tokens it
> >> > > > >> wants
> >> > > > >> >> to use in the subject instance and call produce/consume
> inside
> >> > > > >> >> subject.doas. Each caller will have to keep track of its own
> >> > > > subject. I
> >> > > > >> >> will have to look at the code to see if we support this
> >> feature
> >> > > right
> >> > > > >> now
> >> > > > >> >> but my understanding is delegation token shouldn't need any
> >> special
> >> > > > >> >> treatment as its just another type of Credential in the
> >> subject.
> >> > > > >> >>
> >> > > > >> >> I would also like to know what is your opinion about
> infinite
> >> > > renewal
> >> > > > >> (my
> >> > > > >> >> recommendation is to not support this), tokens renewing them
> >> > > self(my
> >> > > > >> >> recommendation is to not support this) and most importantly
> >> your
> >> > > > choice
> >> > > > >> >> between the alternatives listed on this thread
> >> > > > >> >> <
> >> > > > >>
> >> > > >
> >> > >
> >>
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> >> > > > >> >
> >> > > > >> >> ( I am leaning towards the alternative-2 minus controller
> >> > > > distributing
> >> > > > >> >> secret). Thanks again for reviewing.
> >> > > > >> >>
> >> > > > >> >> Thanks
> >> > > > >> >> Parth
> >> > > > >> >>
> >> > > > >> >>
> >> > > > >> >>
> >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <
> >> gwen@confluent.io>
> >> > > > wrote:
> >> > > > >> >>
> >> > > > >> >> > Harsha,
> >> > > > >> >> >
> >> > > > >> >> > I was thinking of the Rest Proxy. I didn't see your design
> >> yet,
> >> > > > but in
> >> > > > >> >> > our proxy, we have a set of producers, which will serve
> >> multiple
> >> > > > users
> >> > > > >> >> > going through the proxy. Since these users will have
> >> different
> >> > > > >> >> > privileges, they'll need to authenticate separately, and
> >> can't
> >> > > > share a
> >> > > > >> >> > token.
> >> > > > >> >> >
> >> > > > >> >> > Am I missing anything?
> >> > > > >> >> >
> >> > > > >> >> > Gwen
> >> > > > >> >> >
> >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io>
> >> wrote:
> >> > > > >> >> > > Gwen,
> >> > > > >> >> > >            On your second point. Can you describe a
> >> usecase
> >> > > where
> >> > > > >> >> > >            mutliple clients ended up sharing a producer
> >> and
> >> > > even
> >> > > > if
> >> > > > >> they
> >> > > > >> >> > >            do why can't they not use single token that
> >> producer
> >> > > > >> >> > >            captures. Why would we need multiple clients
> >> with
> >> > > > >> different
> >> > > > >> >> > >            tokens sharing a single instance of producer.
> >> Also
> >> > > in
> >> > > > >> this
> >> > > > >> >> > >            case other clients have access all the tokens
> >> no?
> >> > > > >> >> > >
> >> > > > >> >> > > Thanks,
> >> > > > >> >> > > Harsha
> >> > > > >> >> > >
> >> > > > >> >> > >
> >> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote:
> >> > > > >> >> > >> Sorry for the delay:
> >> > > > >> >> > >>
> >> > > > >> >> > >> Two questions that we didn't see in the wiki:
> >> > > > >> >> > >> 1. Is there an expiration for delegation tokens?
> >> Renewal? How
> >> > > > do we
> >> > > > >> >> > >> revoke them?
> >> > > > >> >> > >> 2. If we want to use delegation tokens for "do-as"
> (say,
> >> > > submit
> >> > > > >> Storm
> >> > > > >> >> > >> job as my user), we will need a producer for every job
> >> (we
> >> > > can't
> >> > > > >> share
> >> > > > >> >> > >> them between multiple jobs running on same node), since
> >> we
> >> > > only
> >> > > > >> >> > >> authenticate when connecting. Is there a plan to change
> >> this
> >> > > for
> >> > > > >> >> > >> delegation tokens, in order to allow multiple users
> with
> >> > > > different
> >> > > > >> >> > >> tokens to share a client?
> >> > > > >> >> > >>
> >> > > > >> >> > >> Gwen
> >> > > > >> >> > >>
> >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
> >> > > > >> >> > >> <br...@gmail.com> wrote:
> >> > > > >> >> > >> > Bumping this up one more time, can other committers
> >> review?
> >> > > > >> >> > >> >
> >> > > > >> >> > >> > Thanks
> >> > > > >> >> > >> > Parth
> >> > > > >> >> > >> >
> >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <
> >> kafka@harsha.io>
> >> > > > wrote:
> >> > > > >> >> > >> >
> >> > > > >> >> > >> >> Parth,
> >> > > > >> >> > >> >>           Overall current design looks good to me. I
> >> am +1
> >> > > on
> >> > > > >> the
> >> > > > >> >> > KIP.
> >> > > > >> >> > >> >>
> >> > > > >> >> > >> >> Gwen , Jun can you review this as well.
> >> > > > >> >> > >> >>
> >> > > > >> >> > >> >> -Harsha
> >> > > > >> >> > >> >>
> >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt
> >> wrote:
> >> > > > >> >> > >> >> > Thanks for review Jitendra.
> >> > > > >> >> > >> >> >
> >> > > > >> >> > >> >> > I don't like the idea of infinite lifetime but I
> >> see the
> >> > > > >> Streaming
> >> > > > >> >> > use
> >> > > > >> >> > >> >> > case. Even for Streaming use case I was hoping
> >> there will
> >> > > > be
> >> > > > >> some
> >> > > > >> >> > notion
> >> > > > >> >> > >> >> > of
> >> > > > >> >> > >> >> > master/driver that can get new delegation tokens
> at
> >> fixed
> >> > > > >> interval
> >> > > > >> >> > and
> >> > > > >> >> > >> >> > distribute to workers. If that is not the case for
> >> we can
> >> > > > >> discuss
> >> > > > >> >> > >> >> > delegation tokens renewing them self and the
> >> security
> >> > > > >> implications
> >> > > > >> >> > of the
> >> > > > >> >> > >> >> > same.
> >> > > > >> >> > >> >> >
> >> > > > >> >> > >> >> > I did not want clients to fetch tokens from
> >> zookeeper,
> >> > > > >> overall I
> >> > > > >> >> > think
> >> > > > >> >> > >> >> > its
> >> > > > >> >> > >> >> > better if clients don't rely on our metadata store
> >> and I
> >> > > > >> think we
> >> > > > >> >> > are
> >> > > > >> >> > >> >> > moving in that direction with all the KIP-4
> >> improvements.
> >> > > > I
> >> > > > >> chose
> >> > > > >> >> > >> >> > zookeeper as in this case the client will still
> >> just talk
> >> > > > to
> >> > > > >> >> > broker , its
> >> > > > >> >> > >> >> > the brokers that will use zookeeper which we
> >> already do
> >> > > > for a
> >> > > > >> lot
> >> > > > >> >> > of
> >> > > > >> >> > >> >> > other
> >> > > > >> >> > >> >> > usecases + ease of development + and the ability
> so
> >> > > tokens
> >> > > > >> will
> >> > > > >> >> > survive
> >> > > > >> >> > >> >> > even a rolling restart/cluster failure. if a
> >> majority
> >> > > > agrees
> >> > > > >> the
> >> > > > >> >> > added
> >> > > > >> >> > >> >> > complexity to have controller forwarding keys to
> all
> >> > > > broker is
> >> > > > >> >> > justified
> >> > > > >> >> > >> >> > as
> >> > > > >> >> > >> >> > it provides tighter security , I am fine with that
> >> option
> >> > > > too.
> >> > > > >> >> > >> >> >
> >> > > > >> >> > >> >> > Given zookeeper does not support SSL we can not
> >> store
> >> > > > master
> >> > > > >> keys
> >> > > > >> >> > in
> >> > > > >> >> > >> >> > zookeeper as master keys will be exposed on wire.
> To
> >> > > > support
> >> > > > >> >> > rotation
> >> > > > >> >> > >> >> > without affecting current clients is something I
> >> need to
> >> > > > put
> >> > > > >> more
> >> > > > >> >> > thought
> >> > > > >> >> > >> >> > in. My current proposal assumes the rotation will
> >> > > > invalidate
> >> > > > >> all
> >> > > > >> >> > current
> >> > > > >> >> > >> >> > tokens.
> >> > > > >> >> > >> >> >
> >> > > > >> >> > >> >> > I request committers to also review and post their
> >> > > comments
> >> > > > >> so we
> >> > > > >> >> > can
> >> > > > >> >> > >> >> > make
> >> > > > >> >> > >> >> > progress on this KIP.
> >> > > > >> >> > >> >> >
> >> > > > >> >> > >> >> > Thanks
> >> > > > >> >> > >> >> > Parth
> >> > > > >> >> > >> >> >
> >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <
> >> > > > >> asingh@cloudera.com
> >> > > > >> >> > >
> >> > > > >> >> > >> >> > wrote:
> >> > > > >> >> > >> >> >
> >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <
> >> > > > kafka@harsha.io>
> >> > > > >> >> > wrote:
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >> > > > Unifying the two discussion threads on this
> KIP.
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > > Here is the response from Jitendra
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > > "The need for a large number of clients that
> are
> >> > > > running
> >> > > > >> all
> >> > > > >> >> > over the
> >> > > > >> >> > >> >> > > > cluster that authenticate with Kafka brokers,
> >> is very
> >> > > > >> similar
> >> > > > >> >> > to the
> >> > > > >> >> > >> >> > > > Hadoop use case of large number of tasks
> running
> >> > > across
> >> > > > >> the
> >> > > > >> >> > cluster
> >> > > > >> >> > >> >> that
> >> > > > >> >> > >> >> > > > need authentication to Hdfs Namenode.
> >> Therefore, the
> >> > > > >> >> > delegation token
> >> > > > >> >> > >> >> > > > approach does seem like a good fit for this
> use
> >> case
> >> > > > as we
> >> > > > >> >> > have seen
> >> > > > >> >> > >> >> it
> >> > > > >> >> > >> >> > > > working at large scale in HDFS and YARN.
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > >   The proposed design is very much inline with
> >> Hadoop
> >> > > > >> >> > approach. A few
> >> > > > >> >> > >> >> > > >   comments:
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > > 1) Why do you guys want to allow infinite
> >> renewable
> >> > > > >> lifetime
> >> > > > >> >> > for a
> >> > > > >> >> > >> >> > > > token? HDFS restricts a token to a max life
> time
> >> > > > (default
> >> > > > >> 7
> >> > > > >> >> > days).  A
> >> > > > >> >> > >> >> > > > token's vulnerability is believed to increase
> >> with
> >> > > > time.
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > I agree that having infinite lifetime might not
> >> be the
> >> > > > best
> >> > > > >> idea.
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > > 2) As I understand the tokens are stored in
> >> zookeeper
> >> > > > as
> >> > > > >> well,
> >> > > > >> >> > and
> >> > > > >> >> > >> >> can
> >> > > > >> >> > >> >> > > > be updated there. This is clever as it can
> allow
> >> > > > >> replacing the
> >> > > > >> >> > tokens
> >> > > > >> >> > >> >> > > > once they run out of max life time, and
> clients
> >> can
> >> > > > >> download
> >> > > > >> >> > new
> >> > > > >> >> > >> >> tokens
> >> > > > >> >> > >> >> > > > from zookeeper. It shouldn't be a big load on
> >> > > zookeeper
> >> > > > >> as a
> >> > > > >> >> > client
> >> > > > >> >> > >> >> will
> >> > > > >> >> > >> >> > > > need to get a new token once in several days.
> >> In this
> >> > > > >> approach
> >> > > > >> >> > you
> >> > > > >> >> > >> >> don't
> >> > > > >> >> > >> >> > > > need infinite lifetime on the token even for
> >> long
> >> > > > running
> >> > > > >> >> > clients.
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > > 3) The token password are generated using a
> >> master
> >> > > key.
> >> > > > >> The
> >> > > > >> >> > master
> >> > > > >> >> > >> >> key
> >> > > > >> >> > >> >> > > > should also be periodically changed. In
> Hadoop,
> >> the
> >> > > > >> default
> >> > > > >> >> > renewal
> >> > > > >> >> > >> >> > > > period is 1 day.?
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > IIUC, this will require brokers maintaining a
> >> list of X
> >> > > > most
> >> > > > >> >> > recent
> >> > > > >> >> > >> >> master
> >> > > > >> >> > >> >> > > keys. This list will have to be persisted
> >> somewhere, as
> >> > > > if a
> >> > > > >> >> > broker
> >> > > > >> >> > >> >> goes
> >> > > > >> >> > >> >> > > down it will have to get that list again and
> >> storing
> >> > > > master
> >> > > > >> keys
> >> > > > >> >> > on ZK
> >> > > > >> >> > >> >> is
> >> > > > >> >> > >> >> > > not the best idea. However, if a broker goes
> down
> >> then
> >> > > we
> >> > > > >> have
> >> > > > >> >> > much
> >> > > > >> >> > >> >> bigger
> >> > > > >> >> > >> >> > > issue to deal with and client can always
> >> > > re-authenticate
> >> > > > is
> >> > > > >> such
> >> > > > >> >> > >> >> events.
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >> > > Did you happen to take a look at other
> >> alternatives
> >> > > this
> >> > > > >> list has
> >> > > > >> >> > >> >> > > suggested?
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > > Thanks for a thorough proposal, great work!"
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira
> >> wrote:
> >> > > > >> >> > >> >> > > > > Makes sense to me. Thanks!
> >> > > > >> >> > >> >> > > > >
> >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <
> >> > > > kafka@harsha.io
> >> > > > >> >
> >> > > > >> >> > wrote:
> >> > > > >> >> > >> >> > > > > > It doesn't need any release vehicle but
> >> still the
> >> > > > >> work can
> >> > > > >> >> > move
> >> > > > >> >> > >> >> > > > forward.
> >> > > > >> >> > >> >> > > > > > If anyone is interested in the KIP please
> >> do the
> >> > > > >> review and
> >> > > > >> >> > >> >> provide
> >> > > > >> >> > >> >> > > the
> >> > > > >> >> > >> >> > > > > > comments.
> >> > > > >> >> > >> >> > > > > >
> >> > > > >> >> > >> >> > > > > > -Harsha
> >> > > > >> >> > >> >> > > > > >
> >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael
> >> Juma
> >> > > > wrote:
> >> > > > >> >> > >> >> > > > > >> I agree that it would be good to have
> more
> >> time
> >> > > to
> >> > > > >> review
> >> > > > >> >> > and
> >> > > > >> >> > >> >> > > discuss
> >> > > > >> >> > >> >> > > > > >> KIP-48.
> >> > > > >> >> > >> >> > > > > >>
> >> > > > >> >> > >> >> > > > > >> Ismael
> >> > > > >> >> > >> >> > > > > >>
> >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen
> >> Shapira <
> >> > > > >> >> > >> >> gwen@confluent.io>
> >> > > > >> >> > >> >> > > > wrote:
> >> > > > >> >> > >> >> > > > > >>
> >> > > > >> >> > >> >> > > > > >> > Hi Team,
> >> > > > >> >> > >> >> > > > > >> >
> >> > > > >> >> > >> >> > > > > >> > Since KIP-48 depends on KIP-43, which
> is
> >> > > > already a
> >> > > > >> bit
> >> > > > >> >> > of a
> >> > > > >> >> > >> >> risk
> >> > > > >> >> > >> >> > > for
> >> > > > >> >> > >> >> > > > > >> > the next release - any chance we can
> >> delay
> >> > > > >> delegation
> >> > > > >> >> > tokens
> >> > > > >> >> > >> >> to
> >> > > > >> >> > >> >> > > > Kafka
> >> > > > >> >> > >> >> > > > > >> > 0.10.1?
> >> > > > >> >> > >> >> > > > > >> > With the community working on a release
> >> every
> >> > > 3
> >> > > > >> month,
> >> > > > >> >> > this
> >> > > > >> >> > >> >> is not
> >> > > > >> >> > >> >> > > > a huge
> >> > > > >> >> > >> >> > > > > >> > delay.
> >> > > > >> >> > >> >> > > > > >> >
> >> > > > >> >> > >> >> > > > > >> > Gwen
> >> > > > >> >> > >> >> > > > > >> >
> >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish
> >> Singh
> >> > > <
> >> > > > >> >> > >> >> > > asingh@cloudera.com>
> >> > > > >> >> > >> >> > > > wrote:
> >> > > > >> >> > >> >> > > > > >> > > Parth,
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > > Thanks again for the awesome write
> up.
> >> > > > Following
> >> > > > >> our
> >> > > > >> >> > >> >> discussion
> >> > > > >> >> > >> >> > > > from the
> >> > > > >> >> > >> >> > > > > >> > > JIRA, I think it will be easier to
> >> compare
> >> > > > >> various
> >> > > > >> >> > >> >> alternatives
> >> > > > >> >> > >> >> > > > if they
> >> > > > >> >> > >> >> > > > > >> > are
> >> > > > >> >> > >> >> > > > > >> > > listed together. I am stating below a
> >> few
> >> > > > >> >> > alternatives along
> >> > > > >> >> > >> >> > > with
> >> > > > >> >> > >> >> > > > a the
> >> > > > >> >> > >> >> > > > > >> > > current proposal.
> >> > > > >> >> > >> >> > > > > >> > > (Current proposal) Store Delegation
> >> Token,
> >> > > DT,
> >> > > > >> on ZK.
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a
> >> broker.
> >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is authenticated,
> >> it
> >> > > will
> >> > > > >> make a
> >> > > > >> >> > broker
> >> > > > >> >> > >> >> side
> >> > > > >> >> > >> >> > > > call to
> >> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> >> > > > >> >> > >> >> > > > > >> > >    3. The broker generates a shared
> >> secret
> >> > > > based
> >> > > > >> on
> >> > > > >> >> > >> >> > > HMAC-SHA256(a
> >> > > > >> >> > >> >> > > > > >> > >    Password/Secret shared between all
> >> > > brokers,
> >> > > > >> >> > randomly
> >> > > > >> >> > >> >> > > generated
> >> > > > >> >> > >> >> > > > > >> > tokenId).
> >> > > > >> >> > >> >> > > > > >> > >    4. Broker stores this token in its
> >> in
> >> > > > memory
> >> > > > >> cache.
> >> > > > >> >> > >> >> Broker
> >> > > > >> >> > >> >> > > > also stores
> >> > > > >> >> > >> >> > > > > >> > >    the DelegationToken without the
> >> hmac in
> >> > > the
> >> > > > >> >> > zookeeper.
> >> > > > >> >> > >> >> > > > > >> > >    5. All brokers will have a cache
> >> backed
> >> > > by
> >> > > > >> >> > zookeeper so
> >> > > > >> >> > >> >> they
> >> > > > >> >> > >> >> > > > will all
> >> > > > >> >> > >> >> > > > > >> > >    get notified whenever a new token
> is
> >> > > > >> generated and
> >> > > > >> >> > they
> >> > > > >> >> > >> >> will
> >> > > > >> >> > >> >> > > > update
> >> > > > >> >> > >> >> > > > > >> > their
> >> > > > >> >> > >> >> > > > > >> > >    local cache whenever token state
> >> changes.
> >> > > > >> >> > >> >> > > > > >> > >    6. Broker returns the token to
> >> Client.
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >    1. Probable race condition, client
> >> tries
> >> > > to
> >> > > > >> >> > authenticate
> >> > > > >> >> > >> >> with
> >> > > > >> >> > >> >> > > > a broker
> >> > > > >> >> > >> >> > > > > >> > >    that is yet to be updated with the
> >> newly
> >> > > > >> generated
> >> > > > >> >> > DT.
> >> > > > >> >> > >> >> This
> >> > > > >> >> > >> >> > > can
> >> > > > >> >> > >> >> > > > > >> > probably be
> >> > > > >> >> > >> >> > > > > >> > >    dealt with making dtRequest block
> >> until
> >> > > all
> >> > > > >> >> > brokers have
> >> > > > >> >> > >> >> > > > updated
> >> > > > >> >> > >> >> > > > > >> > their DT
> >> > > > >> >> > >> >> > > > > >> > >    cache. Zk barrier or similar
> >> mechanism
> >> > > can
> >> > > > be
> >> > > > >> used.
> >> > > > >> >> > >> >> However,
> >> > > > >> >> > >> >> > > > all such
> >> > > > >> >> > >> >> > > > > >> > >    mechanisms will increase
> complexity.
> >> > > > >> >> > >> >> > > > > >> > >    2. Using a static secret key from
> >> config
> >> > > > >> file. Will
> >> > > > >> >> > >> >> require
> >> > > > >> >> > >> >> > > yet
> >> > > > >> >> > >> >> > > > > >> > another
> >> > > > >> >> > >> >> > > > > >> > >    config and uses a static secret
> >> key. It
> >> > > is
> >> > > > >> advised
> >> > > > >> >> > to
> >> > > > >> >> > >> >> rotate
> >> > > > >> >> > >> >> > > > secret
> >> > > > >> >> > >> >> > > > > >> > keys
> >> > > > >> >> > >> >> > > > > >> > >    periodically. This can be avoided
> >> with
> >> > > > >> controller
> >> > > > >> >> > >> >> generating
> >> > > > >> >> > >> >> > > > > >> > secretKey and
> >> > > > >> >> > >> >> > > > > >> > >    passing to brokers periodically.
> >> However,
> >> > > > >> this will
> >> > > > >> >> > >> >> require
> >> > > > >> >> > >> >> > > > brokers to
> >> > > > >> >> > >> >> > > > > >> > >    maintain certain counts of
> >> secretKeys.
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > > (Alternative 1) Have controller
> >> generate
> >> > > > >> delegation
> >> > > > >> >> > token.
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a
> >> broker.
> >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is authenticated,
> >> it
> >> > > will
> >> > > > >> make a
> >> > > > >> >> > broker
> >> > > > >> >> > >> >> side
> >> > > > >> >> > >> >> > > > call to
> >> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> >> > > > >> >> > >> >> > > > > >> > >    3. Broker forwards the request to
> >> > > > controller.
> >> > > > >> >> > >> >> > > > > >> > >    4. Controller generates a DT and
> >> > > broadcasts
> >> > > > >> to all
> >> > > > >> >> > >> >> brokers.
> >> > > > >> >> > >> >> > > > > >> > >    5. Broker stores this token in its
> >> memory
> >> > > > >> cache.
> >> > > > >> >> > >> >> > > > > >> > >    6. Controller responds to broker’s
> >> DT
> >> > > req.
> >> > > > >> >> > >> >> > > > > >> > >    7. Broker returns the token to
> >> Client.
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >    1. We will have to add new APIs to
> >> > > support
> >> > > > >> >> > controller
> >> > > > >> >> > >> >> pushing
> >> > > > >> >> > >> >> > > > tokens
> >> > > > >> >> > >> >> > > > > >> > to
> >> > > > >> >> > >> >> > > > > >> > >    brokers on top of the minimal APIs
> >> that
> >> > > are
> >> > > > >> >> > currently
> >> > > > >> >> > >> >> > > proposed.
> >> > > > >> >> > >> >> > > > > >> > >    2. We will also have to add APIs
> to
> >> > > support
> >> > > > >> the
> >> > > > >> >> > >> >> bootstrapping
> >> > > > >> >> > >> >> > > > case,
> >> > > > >> >> > >> >> > > > > >> > i.e,
> >> > > > >> >> > >> >> > > > > >> > >    when a new broker comes up it will
> >> have
> >> > > to
> >> > > > >> get all
> >> > > > >> >> > >> >> delegation
> >> > > > >> >> > >> >> > > > tokens
> >> > > > >> >> > >> >> > > > > >> > from
> >> > > > >> >> > >> >> > > > > >> > >    the controller.
> >> > > > >> >> > >> >> > > > > >> > >    3. In catastrophic failures where
> >> all
> >> > > > brokers
> >> > > > >> go
> >> > > > >> >> > down,
> >> > > > >> >> > >> >> the
> >> > > > >> >> > >> >> > > > tokens will
> >> > > > >> >> > >> >> > > > > >> > >    be lost even if servers are
> >> restarted as
> >> > > > >> tokens
> >> > > > >> >> > are not
> >> > > > >> >> > >> >> > > > persisted
> >> > > > >> >> > >> >> > > > > >> > anywhere.
> >> > > > >> >> > >> >> > > > > >> > >    If this happens, then there are
> more
> >> > > > important
> >> > > > >> >> > things to
> >> > > > >> >> > >> >> > > worry
> >> > > > >> >> > >> >> > > > about
> >> > > > >> >> > >> >> > > > > >> > and
> >> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
> >> re-authenticate.
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > > (Alternative 2) Do not distribute DT
> to
> >> > > other
> >> > > > >> brokers
> >> > > > >> >> > at
> >> > > > >> >> > >> >> all.
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a
> >> broker.
> >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is authenticated,
> >> it
> >> > > will
> >> > > > >> make a
> >> > > > >> >> > broker
> >> > > > >> >> > >> >> side
> >> > > > >> >> > >> >> > > > call to
> >> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> >> > > > >> >> > >> >> > > > > >> > >    3. The broker generates DT of
> form,
> >> > > [hmac +
> >> > > > >> (owner,
> >> > > > >> >> > >> >> renewer,
> >> > > > >> >> > >> >> > > > > >> > >    maxLifeTime, id, hmac,
> >> expirationTime)]
> >> > > and
> >> > > > >> passes
> >> > > > >> >> > back
> >> > > > >> >> > >> >> this
> >> > > > >> >> > >> >> > > > DT to
> >> > > > >> >> > >> >> > > > > >> > client.
> >> > > > >> >> > >> >> > > > > >> > >    hmac is generated via
> >> {HMAC-SHA256(owner,
> >> > > > >> renewer,
> >> > > > >> >> > >> >> > > > maxLifeTime, id,
> >> > > > >> >> > >> >> > > > > >> > hmac,
> >> > > > >> >> > >> >> > > > > >> > >    expirationTime) using SecretKey}.
> >> Note
> >> > > that
> >> > > > >> all
> >> > > > >> >> > brokers
> >> > > > >> >> > >> >> have
> >> > > > >> >> > >> >> > > > this
> >> > > > >> >> > >> >> > > > > >> > SecretKey.
> >> > > > >> >> > >> >> > > > > >> > >    4. Client then goes to any broker
> >> and to
> >> > > > >> >> > authenticate
> >> > > > >> >> > >> >> sends
> >> > > > >> >> > >> >> > > > the DT.
> >> > > > >> >> > >> >> > > > > >> > >    Broker recalculates hmac using
> >> (owner,
> >> > > > >> renewer,
> >> > > > >> >> > >> >> maxLifeTime,
> >> > > > >> >> > >> >> > > > id, hmac,
> >> > > > >> >> > >> >> > > > > >> > >    expirationTime) info from DT and
> its
> >> > > > >> SecretKey. If
> >> > > > >> >> > it
> >> > > > >> >> > >> >> matches
> >> > > > >> >> > >> >> > > > with
> >> > > > >> >> > >> >> > > > > >> > hmac of
> >> > > > >> >> > >> >> > > > > >> > >    DT, client is authenticated. Yes,
> >> it will
> >> > > > do
> >> > > > >> other
> >> > > > >> >> > >> >> obvious
> >> > > > >> >> > >> >> > > > checks of
> >> > > > >> >> > >> >> > > > > >> > >    timestamp expiry and such.
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > > Note that secret key will be
> generated
> >> by
> >> > > > >> controller
> >> > > > >> >> > and
> >> > > > >> >> > >> >> passed
> >> > > > >> >> > >> >> > > to
> >> > > > >> >> > >> >> > > > > >> > brokers
> >> > > > >> >> > >> >> > > > > >> > > periodically.
> >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >    1. How to delete a DT? Yes, that
> is
> >> a
> >> > > > downside
> >> > > > >> >> > here.
> >> > > > >> >> > >> >> However,
> >> > > > >> >> > >> >> > > > this can
> >> > > > >> >> > >> >> > > > > >> > >    be handled with brokers
> maintaining
> >> a
> >> > > > >> blacklist of
> >> > > > >> >> > DTs,
> >> > > > >> >> > >> >> DTs
> >> > > > >> >> > >> >> > > > from this
> >> > > > >> >> > >> >> > > > > >> > list
> >> > > > >> >> > >> >> > > > > >> > >    can be removed after expiry.
> >> > > > >> >> > >> >> > > > > >> > >    2. In catastrophic failures where
> >> all
> >> > > > brokers
> >> > > > >> go
> >> > > > >> >> > down,
> >> > > > >> >> > >> >> the
> >> > > > >> >> > >> >> > > > tokens will
> >> > > > >> >> > >> >> > > > > >> > >    be lost even if servers are
> >> restarted as
> >> > > > >> tokens
> >> > > > >> >> > are not
> >> > > > >> >> > >> >> > > > persisted
> >> > > > >> >> > >> >> > > > > >> > anywhere.
> >> > > > >> >> > >> >> > > > > >> > >    If this happens, then there are
> more
> >> > > > important
> >> > > > >> >> > things to
> >> > > > >> >> > >> >> > > worry
> >> > > > >> >> > >> >> > > > about
> >> > > > >> >> > >> >> > > > > >> > and
> >> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
> >> re-authenticate.
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM,
> Parth
> >> > > > >> Brahmbhatt <
> >> > > > >> >> > >> >> > > > > >> > > pbrahmbhatt@hortonworks.com> wrote:
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >> Hi,
> >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > >> >> > >> >> > > > > >> > >> I have filed KIP-48 so we can offer
> >> hadoop
> >> > > > like
> >> > > > >> >> > delegation
> >> > > > >> >> > >> >> > > > tokens in
> >> > > > >> >> > >> >> > > > > >> > >> kafka. You can review the design
> >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > >> >> > >> >> > > > > >> >
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >>
> >> > > > >> >> >
> >> > > > >>
> >> > > >
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> >> > > > >> >> > >> >> > > > > >> > .
> >> > > > >> >> > >> >> > > > > >> > >> This KIP depends on KIP-43 and we
> >> have also
> >> > > > >> >> > discussed an
> >> > > > >> >> > >> >> > > > alternative to
> >> > > > >> >> > >> >> > > > > >> > >> proposed design here<
> >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > >> >> > >> >> > > > > >> >
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >>
> >> > > > >> >> >
> >> > > > >>
> >> > > >
> >> > >
> >>
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> >> > > > >> >> > >> >> > > > > >> > >> >.
> >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > >> >> > >> >> > > > > >> > >> Thanks
> >> > > > >> >> > >> >> > > > > >> > >> Parth
> >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > > --
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > > Regards,
> >> > > > >> >> > >> >> > > > > >> > > Ashish
> >> > > > >> >> > >> >> > > > > >> >
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >> > > --
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >> > > Regards,
> >> > > > >> >> > >> >> > > Ashish
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >>
> >> > > > >> >> >
> >> > > > >>
> >> > > >
> >> > >
> >>
> >>
> >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Gwen Shapira <gw...@confluent.io>]

Hi Jun,

Few of my answers below (since these are things we discussed, or that I
thought about)

On Mon, May 23, 2016 at 10:35 AM, Jun Rao <ju...@confluent.io> wrote:

> Thanks for the KIP. A few comments.
>
> 100. This potentially can be useful for Kafka Connect and Kafka rest proxy
> where a worker agent will need to run a task on behalf of a client. We will
> likely need to change how those services use Kafka clients
> (producer/consumer). Instead of a shared client per worker, we will need a
> client per user task since the authentication happens at the connection
> level. For Kafka Connect, the renewer will be the workers. So, we probably
> need to allow multiple renewers. For Kafka rest proxy, the renewer can
> probably just be the creator of the token.
>

For the connector, the token for each connect task with be created by the
connector that manages the task. It can also be responsible for renewing.


>
> 101. Do we need new acl on who can request delegation tokens?
>

We could, but I'd prefer not to have that. I can't see a use case of
preventing certain users from delegating, since they can't delegate more
than the privileges they already have.


>
> 102. Do we recommend people to send delegation tokens in an encrypted
> channel?
>


Definitely. But just like SASL/PLAIN, we can leave both options open.


>
> 103. Who is responsible for expiring tokens, every broker?
>

I think token validity can be checked when they are used, like in SASL?


>
> 104. For invalidating tokens, would it be better to do it in a request
> instead of going to ZK directly?
>

+1 to this.


>
> 105. The terminology of client in the wiki sometimes refers to the end
> client and some other times refers to the client using the delegation
> tokens. It would be useful to distinguish between the two.
>
> 106. Could you explain the sentence "Broker also stores the DelegationToken
> without the hmac in the zookeeper." a bit more? I thought the delegation
> token is the hmac.
>
> Thanks,
>
> Jun
>
>
> On Mon, May 23, 2016 at 9:22 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > Hi, Harsha,
> >
> > Just sent out a KIP meeting invite. We can discuss this in the meeting
> > tomorrow.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, May 19, 2016 at 8:47 AM, Harsha <ka...@harsha.io> wrote:
> >
> >> Hi All,
> >>            Can we have a KIP meeting around this. The KIP is up for
> >>            sometime and if there are any questions lets quickly hash out
> >>            details.
> >>
> >> Thanks,
> >> Harsha
> >>
> >> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt wrote:
> >> > That is what the hadoop echo system uses so no good reason really. We
> >> > could
> >> > change it to whatever is the newest recommended standard is.
> >> >
> >> > Thanks
> >> > Parth
> >> >
> >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <is...@juma.me.uk>
> wrote:
> >> >
> >> > > Hi Parth,
> >> > >
> >> > > Thanks for the KIP. I only started reviewing this and may have
> >> additional
> >> > > questions later. The immediate question that came to mind is our
> >> choice of
> >> > > "DIGEST-MD5" even though it's marked as OBSOLETE in the IANA
> Registry
> >> of
> >> > > SASL mechanisms and the original RFC (2831) has been moved to
> Historic
> >> > > status:
> >> > >
> >> > > https://tools.ietf.org/html/rfc6331
> >> > >
> http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> >> > >
> >> > > What is the reasoning behind that choice?
> >> > >
> >> > > Thanks,
> >> > > Ismael
> >> > >
> >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <gw...@confluent.io>
> >> wrote:
> >> > >
> >> > > > Also comments inline :)
> >> > > >
> >> > > > > * I want to emphasize that even though delegation tokens are a
> >> Hadoop
> >> > > > > innovation, I feel very strongly about not adding dependency on
> >> Hadoop
> >> > > > > when implementing delegation tokens for Kafka. The KIP doesn't
> >> imply
> >> > > > > such dependency, but if you can clarify...
> >> > > > >
> >> > > > >
> >> > > > > *No hadoop dependency.*
> >> > > >
> >> > > > Yay! Just add this to the KIP so no one will read the KIP and
> panic
> >> > > > three weeks before the next release...
> >> > > >
> >> > > > > * Can we get delegation token at any time after authenticating?
> >> only
> >> > > > > immediately after?
> >> > > > >
> >> > > > >
> >> > > > > *As long as you are authenticated you can get delegation tokens.
> >> We
> >> > > need
> >> > > > to
> >> > > > > discuss if a client authenticated using delegation token, can
> also
> >> > > > acquire
> >> > > > > delegation token again or not. Also there is the question of do
> we
> >> > > allow
> >> > > > > anyone to acquire delegation token or we want specific ACLs (I
> >> think
> >> > > its
> >> > > > an
> >> > > > > overkill.)*
> >> > > >
> >> > > > I agree that ACLs is an overkill.
> >> > > >
> >> > > > I think we are debating two options: Either require Kerberos auth
> >> for
> >> > > > renewal or require non-owners to renew.
> >> > > > I *think* the latter is simpler (it basically require a "job
> master"
> >> > > > to take responsibility for the renewal, it will have its own
> >> identity
> >> > > > anyway and I think this is the correct design pattern anyway. For
> >> > > > storm, I'd expect Nimbus to coordinate renewals?), but it is hard
> to
> >> > > > debate simplicity without looking at the code changes required. If
> >> you
> >> > > > have a draft of how the "require Kerberos" will look in Kafka
> code,
> >> > > > I'll be happy to take a look.
> >> > > >
> >> > > > > * My understanding is that tokens will propagate via ZK but
> >> without
> >> > > > > additional changes to UpdateMetadata protocol, correct? Clients
> >> > > > > currently don't retry on SASL auth failure (IIRC), but since the
> >> > > > > tokens propagate between brokers asynch, we will need to retry a
> >> bit
> >> > > > > to avoid clients failing auth due to timing issues.
> >> > > > >
> >> > > > > *I am considering 2 alternatives right now. The current
> documented
> >> > > > approach
> >> > > > > is zookeeper based and it does not require any changes to
> >> > > UpdateMetadata
> >> > > > > protocol. An alternative approach can remove zookeeper
> dependency
> >> as
> >> > > well
> >> > > > > but we can discuss that in KIP discussion call.*
> >> > > >
> >> > > > Oooh! Sounds interesting. Do you want to ping Jun to arrange a
> call?
> >> > > >
> >> > > > > * I liked Ashish's suggestion of having just the controller
> issue
> >> the
> >> > > > > delegation tokens, to avoid syncing a shared secret. Not sure if
> >> we
> >> > > > > want to continue the discussion here or on the wiki. I think
> that
> >> we
> >> > > > > can decouple the problem of "token distribution" from "shared
> >> secret
> >> > > > > distribution" and use the controller as the only token generator
> >> to
> >> > > > > solve the second issue, while still using ZK async to distribute
> >> > > > > tokens.
> >> > > > >
> >> > > > >
> >> > > > > *As mentioned in the previous Email I am fine with that approach
> >> as
> >> > > long
> >> > > > as
> >> > > > > we agree that the extra complexity of adding/updating APIS adds
> >> enough
> >> > > > > value. The advantage with the controller approach is secret
> >> rotation
> >> > > can
> >> > > > be
> >> > > > > automated,frequent and would not require deployment. *
> >> > > >
> >> > > > Can you detail the extra complexity (or point me to the email I
> >> > > > missed?) - which Apis are required?
> >> > > > As far as I can tell, clients can already find the controller from
> >> > > > metadata. I'm a bit more concerned about controller load.
> >> > > >
> >> > > > >
> >> > > > > * While I like the idea of forcing kerberos auth for renwal, I
> >> think
> >> > > > > it mixes the transport layer the the request content in a pretty
> >> ugly
> >> > > > > way. Perhaps limiting renewer to non-owner is better.
> >> > > > >
> >> > > > > *I feel this is a necessary evil. While this will make the kafka
> >> code
> >> > > > > pretty straight forward , forcing  renewer to non-owner pushes
> >> the code
> >> > > > > ugliness to client and makes it even harder to integrate.  *
> >> > > >
> >> > > > As mentioned before, I don't think the "renewal by other" approach
> >> is
> >> > > > that ugly for the clients we expect to use delegation tokens since
> >> > > > they will have an app-master of some sort who requested the token
> to
> >> > > > begin with.
> >> > > >
> >> > > > >
> >> > > > > The response for my question on how multiple identities will be
> >> > > > > handled wasn't super clear to me - AFAIK, we don't authenticate
> >> each
> >> > > > > request, we authenticate connections.
> >> > > > >
> >> > > > > *We authenticate connections, and only when they are being
> >> established.
> >> > > > Let
> >> > > > > me try to phrase this as a question, in absence of delegation
> >> tokens if
> >> > > > we
> >> > > > > had to support the use case using user TGT's how would we do it?
> >> My
> >> > > point
> >> > > > > was it would be no different with delegation tokens. The use
> case
> >> you
> >> > > are
> >> > > > > describing seems more like impersonation.*
> >> > > >
> >> > > > Yeah, I thought that one of the things that delegation tokens
> >> handled.
> >> > > > Maybe I got it wrong :)
> >> > > >
> >> > > > Thanks for the detailed answers.
> >> > > >
> >> > > > Gwen
> >> > > >
> >> > > >
> >> > > > > Thanks
> >> > > > > Parth
> >> > > > >
> >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <
> gwen@confluent.io
> >> >
> >> > > > wrote:
> >> > > > >
> >> > > > >> Hi Parth and Harsha,
> >> > > > >>
> >> > > > >> Few more comments:
> >> > > > >>
> >> > > > >> * The API / RequestResponse section doesn't seem to have good
> >> > > > >> description of the changes to the Kafka Protocol. Sounds like
> >> you are
> >> > > > >> proposing new DelegationTokenRequest and RenewTokenRequest (and
> >> > > > >> matching responses), without detailing the contents of the
> >> requests
> >> > > > >> and responses? Or rather, you show the class interface, but not
> >> the
> >> > > > >> underlying protocol. This could be seen as an implementation
> >> detail,
> >> > > > >> but since the binary protocol is what we provide to non-Java
> >> clients,
> >> > > > >> we need to show the changes there.
> >> > > > >>
> >> > > > >> * getDelegationToken sounds like delegationTokenRequestHandler?
> >> Is it
> >> > > > >> planned to be part of KafkaApi? or Client? Its unclear...
> >> > > > >>
> >> > > > >> * I want to emphasize that even though delegation tokens are a
> >> Hadoop
> >> > > > >> innovation, I feel very strongly about not adding dependency on
> >> Hadoop
> >> > > > >> when implementing delegation tokens for Kafka. The KIP doesn't
> >> imply
> >> > > > >> such dependency, but if you can clarify...
> >> > > > >>
> >> > > > >> * Can we get delegation token at any time after authenticating?
> >> only
> >> > > > >> immediately after?
> >> > > > >>
> >> > > > >> * My understanding is that tokens will propagate via ZK but
> >> without
> >> > > > >> additional changes to UpdateMetadata protocol, correct? Clients
> >> > > > >> currently don't retry on SASL auth failure (IIRC), but since
> the
> >> > > > >> tokens propagate between brokers asynch, we will need to retry
> a
> >> bit
> >> > > > >> to avoid clients failing auth due to timing issues.
> >> > > > >>
> >> > > > >> * Strongly agreeing on clients not touching ZK directly :)
> >> > > > >>
> >> > > > >> * I liked Ashish's suggestion of having just the controller
> >> issue the
> >> > > > >> delegation tokens, to avoid syncing a shared secret. Not sure
> if
> >> we
> >> > > > >> want to continue the discussion here or on the wiki. I think
> >> that we
> >> > > > >> can decouple the problem of "token distribution" from "shared
> >> secret
> >> > > > >> distribution" and use the controller as the only token
> generator
> >> to
> >> > > > >> solve the second issue, while still using ZK async to
> distribute
> >> > > > >> tokens.
> >> > > > >>
> >> > > > >> * I am also uncomfortable with infinite lifetime of tokens (and
> >> hoped
> >> > > > >> to hear from others in the community) - but having the option
> and
> >> > > > >> documenting it as less secure, allows users to configure their
> >> system
> >> > > > >> as the wish.
> >> > > > >>
> >> > > > >> * While I like the idea of forcing kerberos auth for renwal, I
> >> think
> >> > > > >> it mixes the transport layer the the request content in a
> pretty
> >> ugly
> >> > > > >> way. Perhaps limiting renewer to non-owner is better.
> >> > > > >>
> >> > > > >> Things I'd still like to see:
> >> > > > >>
> >> > > > >> * More detailed explanation on what we plan to do for the java
> >> clients
> >> > > > >> specifically - new configuration? new APIs?
> >> > > > >> The response for my question on how multiple identities will be
> >> > > > >> handled wasn't super clear to me - AFAIK, we don't authenticate
> >> each
> >> > > > >> request, we authenticate connections.
> >> > > > >>
> >> > > > >> * Alternatives: Delegation tokens are only used in the Hadoop
> >> > > > >> ecosystem. I'm wondering if there are alternatives in other
> >> ecosystems
> >> > > > >> (Mesos? Tachyon? Cassandra?) and whether there are some
> >> advantages
> >> > > > >> there.
> >> > > > >>
> >> > > > >> Gwen
> >> > > > >>
> >> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <ka...@harsha.io>
> wrote:
> >> > > > >> > Hi Gwen,
> >> > > > >> >            Can you look at Parth's last reply. Does it answer
> >> your
> >> > > > >> >            concerns.
> >> > > > >> >
> >> > > > >> > Thanks,
> >> > > > >> > Harsha
> >> > > > >> >
> >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt wrote:
> >> > > > >> >> Thanks for reviewing Gwen. The wiki already has details on
> >> token
> >> > > > >> >> expiration
> >> > > > >> >> under token acquisition process
> >> > > > >> >> <
> >> > > > >>
> >> > > >
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> >> > > > >> >.
> >> > > > >> >> Current proposal is that tokens will expire based on a
> server
> >> side
> >> > > > >> >> configuration (default 24 hours) unless renewed. Renewal is
> >> only
> >> > > > allowed
> >> > > > >> >> until the max life time of token. Alternatively we could
> also
> >> make
> >> > > > that
> >> > > > >> >> an
> >> > > > >> >> optional param and the server side default can serve as the
> >> upper
> >> > > > bound.
> >> > > > >> >>
> >> > > > >> >> To your second point it will be done exactly the same way we
> >> would
> >> > > > >> >> support
> >> > > > >> >> multiple keytabs. The calling client will have to put the
> >> tokens it
> >> > > > >> wants
> >> > > > >> >> to use in the subject instance and call produce/consume
> inside
> >> > > > >> >> subject.doas. Each caller will have to keep track of its own
> >> > > > subject. I
> >> > > > >> >> will have to look at the code to see if we support this
> >> feature
> >> > > right
> >> > > > >> now
> >> > > > >> >> but my understanding is delegation token shouldn't need any
> >> special
> >> > > > >> >> treatment as its just another type of Credential in the
> >> subject.
> >> > > > >> >>
> >> > > > >> >> I would also like to know what is your opinion about
> infinite
> >> > > renewal
> >> > > > >> (my
> >> > > > >> >> recommendation is to not support this), tokens renewing them
> >> > > self(my
> >> > > > >> >> recommendation is to not support this) and most importantly
> >> your
> >> > > > choice
> >> > > > >> >> between the alternatives listed on this thread
> >> > > > >> >> <
> >> > > > >>
> >> > > >
> >> > >
> >>
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> >> > > > >> >
> >> > > > >> >> ( I am leaning towards the alternative-2 minus controller
> >> > > > distributing
> >> > > > >> >> secret). Thanks again for reviewing.
> >> > > > >> >>
> >> > > > >> >> Thanks
> >> > > > >> >> Parth
> >> > > > >> >>
> >> > > > >> >>
> >> > > > >> >>
> >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <
> >> gwen@confluent.io>
> >> > > > wrote:
> >> > > > >> >>
> >> > > > >> >> > Harsha,
> >> > > > >> >> >
> >> > > > >> >> > I was thinking of the Rest Proxy. I didn't see your design
> >> yet,
> >> > > > but in
> >> > > > >> >> > our proxy, we have a set of producers, which will serve
> >> multiple
> >> > > > users
> >> > > > >> >> > going through the proxy. Since these users will have
> >> different
> >> > > > >> >> > privileges, they'll need to authenticate separately, and
> >> can't
> >> > > > share a
> >> > > > >> >> > token.
> >> > > > >> >> >
> >> > > > >> >> > Am I missing anything?
> >> > > > >> >> >
> >> > > > >> >> > Gwen
> >> > > > >> >> >
> >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io>
> >> wrote:
> >> > > > >> >> > > Gwen,
> >> > > > >> >> > >            On your second point. Can you describe a
> >> usecase
> >> > > where
> >> > > > >> >> > >            mutliple clients ended up sharing a producer
> >> and
> >> > > even
> >> > > > if
> >> > > > >> they
> >> > > > >> >> > >            do why can't they not use single token that
> >> producer
> >> > > > >> >> > >            captures. Why would we need multiple clients
> >> with
> >> > > > >> different
> >> > > > >> >> > >            tokens sharing a single instance of producer.
> >> Also
> >> > > in
> >> > > > >> this
> >> > > > >> >> > >            case other clients have access all the tokens
> >> no?
> >> > > > >> >> > >
> >> > > > >> >> > > Thanks,
> >> > > > >> >> > > Harsha
> >> > > > >> >> > >
> >> > > > >> >> > >
> >> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote:
> >> > > > >> >> > >> Sorry for the delay:
> >> > > > >> >> > >>
> >> > > > >> >> > >> Two questions that we didn't see in the wiki:
> >> > > > >> >> > >> 1. Is there an expiration for delegation tokens?
> >> Renewal? How
> >> > > > do we
> >> > > > >> >> > >> revoke them?
> >> > > > >> >> > >> 2. If we want to use delegation tokens for "do-as"
> (say,
> >> > > submit
> >> > > > >> Storm
> >> > > > >> >> > >> job as my user), we will need a producer for every job
> >> (we
> >> > > can't
> >> > > > >> share
> >> > > > >> >> > >> them between multiple jobs running on same node), since
> >> we
> >> > > only
> >> > > > >> >> > >> authenticate when connecting. Is there a plan to change
> >> this
> >> > > for
> >> > > > >> >> > >> delegation tokens, in order to allow multiple users
> with
> >> > > > different
> >> > > > >> >> > >> tokens to share a client?
> >> > > > >> >> > >>
> >> > > > >> >> > >> Gwen
> >> > > > >> >> > >>
> >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
> >> > > > >> >> > >> <br...@gmail.com> wrote:
> >> > > > >> >> > >> > Bumping this up one more time, can other committers
> >> review?
> >> > > > >> >> > >> >
> >> > > > >> >> > >> > Thanks
> >> > > > >> >> > >> > Parth
> >> > > > >> >> > >> >
> >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <
> >> kafka@harsha.io>
> >> > > > wrote:
> >> > > > >> >> > >> >
> >> > > > >> >> > >> >> Parth,
> >> > > > >> >> > >> >>           Overall current design looks good to me. I
> >> am +1
> >> > > on
> >> > > > >> the
> >> > > > >> >> > KIP.
> >> > > > >> >> > >> >>
> >> > > > >> >> > >> >> Gwen , Jun can you review this as well.
> >> > > > >> >> > >> >>
> >> > > > >> >> > >> >> -Harsha
> >> > > > >> >> > >> >>
> >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt
> >> wrote:
> >> > > > >> >> > >> >> > Thanks for review Jitendra.
> >> > > > >> >> > >> >> >
> >> > > > >> >> > >> >> > I don't like the idea of infinite lifetime but I
> >> see the
> >> > > > >> Streaming
> >> > > > >> >> > use
> >> > > > >> >> > >> >> > case. Even for Streaming use case I was hoping
> >> there will
> >> > > > be
> >> > > > >> some
> >> > > > >> >> > notion
> >> > > > >> >> > >> >> > of
> >> > > > >> >> > >> >> > master/driver that can get new delegation tokens
> at
> >> fixed
> >> > > > >> interval
> >> > > > >> >> > and
> >> > > > >> >> > >> >> > distribute to workers. If that is not the case for
> >> we can
> >> > > > >> discuss
> >> > > > >> >> > >> >> > delegation tokens renewing them self and the
> >> security
> >> > > > >> implications
> >> > > > >> >> > of the
> >> > > > >> >> > >> >> > same.
> >> > > > >> >> > >> >> >
> >> > > > >> >> > >> >> > I did not want clients to fetch tokens from
> >> zookeeper,
> >> > > > >> overall I
> >> > > > >> >> > think
> >> > > > >> >> > >> >> > its
> >> > > > >> >> > >> >> > better if clients don't rely on our metadata store
> >> and I
> >> > > > >> think we
> >> > > > >> >> > are
> >> > > > >> >> > >> >> > moving in that direction with all the KIP-4
> >> improvements.
> >> > > > I
> >> > > > >> chose
> >> > > > >> >> > >> >> > zookeeper as in this case the client will still
> >> just talk
> >> > > > to
> >> > > > >> >> > broker , its
> >> > > > >> >> > >> >> > the brokers that will use zookeeper which we
> >> already do
> >> > > > for a
> >> > > > >> lot
> >> > > > >> >> > of
> >> > > > >> >> > >> >> > other
> >> > > > >> >> > >> >> > usecases + ease of development + and the ability
> so
> >> > > tokens
> >> > > > >> will
> >> > > > >> >> > survive
> >> > > > >> >> > >> >> > even a rolling restart/cluster failure. if a
> >> majority
> >> > > > agrees
> >> > > > >> the
> >> > > > >> >> > added
> >> > > > >> >> > >> >> > complexity to have controller forwarding keys to
> all
> >> > > > broker is
> >> > > > >> >> > justified
> >> > > > >> >> > >> >> > as
> >> > > > >> >> > >> >> > it provides tighter security , I am fine with that
> >> option
> >> > > > too.
> >> > > > >> >> > >> >> >
> >> > > > >> >> > >> >> > Given zookeeper does not support SSL we can not
> >> store
> >> > > > master
> >> > > > >> keys
> >> > > > >> >> > in
> >> > > > >> >> > >> >> > zookeeper as master keys will be exposed on wire.
> To
> >> > > > support
> >> > > > >> >> > rotation
> >> > > > >> >> > >> >> > without affecting current clients is something I
> >> need to
> >> > > > put
> >> > > > >> more
> >> > > > >> >> > thought
> >> > > > >> >> > >> >> > in. My current proposal assumes the rotation will
> >> > > > invalidate
> >> > > > >> all
> >> > > > >> >> > current
> >> > > > >> >> > >> >> > tokens.
> >> > > > >> >> > >> >> >
> >> > > > >> >> > >> >> > I request committers to also review and post their
> >> > > comments
> >> > > > >> so we
> >> > > > >> >> > can
> >> > > > >> >> > >> >> > make
> >> > > > >> >> > >> >> > progress on this KIP.
> >> > > > >> >> > >> >> >
> >> > > > >> >> > >> >> > Thanks
> >> > > > >> >> > >> >> > Parth
> >> > > > >> >> > >> >> >
> >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <
> >> > > > >> asingh@cloudera.com
> >> > > > >> >> > >
> >> > > > >> >> > >> >> > wrote:
> >> > > > >> >> > >> >> >
> >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <
> >> > > > kafka@harsha.io>
> >> > > > >> >> > wrote:
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >> > > > Unifying the two discussion threads on this
> KIP.
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > > Here is the response from Jitendra
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > > "The need for a large number of clients that
> are
> >> > > > running
> >> > > > >> all
> >> > > > >> >> > over the
> >> > > > >> >> > >> >> > > > cluster that authenticate with Kafka brokers,
> >> is very
> >> > > > >> similar
> >> > > > >> >> > to the
> >> > > > >> >> > >> >> > > > Hadoop use case of large number of tasks
> running
> >> > > across
> >> > > > >> the
> >> > > > >> >> > cluster
> >> > > > >> >> > >> >> that
> >> > > > >> >> > >> >> > > > need authentication to Hdfs Namenode.
> >> Therefore, the
> >> > > > >> >> > delegation token
> >> > > > >> >> > >> >> > > > approach does seem like a good fit for this
> use
> >> case
> >> > > > as we
> >> > > > >> >> > have seen
> >> > > > >> >> > >> >> it
> >> > > > >> >> > >> >> > > > working at large scale in HDFS and YARN.
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > >   The proposed design is very much inline with
> >> Hadoop
> >> > > > >> >> > approach. A few
> >> > > > >> >> > >> >> > > >   comments:
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > > 1) Why do you guys want to allow infinite
> >> renewable
> >> > > > >> lifetime
> >> > > > >> >> > for a
> >> > > > >> >> > >> >> > > > token? HDFS restricts a token to a max life
> time
> >> > > > (default
> >> > > > >> 7
> >> > > > >> >> > days).  A
> >> > > > >> >> > >> >> > > > token's vulnerability is believed to increase
> >> with
> >> > > > time.
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > I agree that having infinite lifetime might not
> >> be the
> >> > > > best
> >> > > > >> idea.
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > > 2) As I understand the tokens are stored in
> >> zookeeper
> >> > > > as
> >> > > > >> well,
> >> > > > >> >> > and
> >> > > > >> >> > >> >> can
> >> > > > >> >> > >> >> > > > be updated there. This is clever as it can
> allow
> >> > > > >> replacing the
> >> > > > >> >> > tokens
> >> > > > >> >> > >> >> > > > once they run out of max life time, and
> clients
> >> can
> >> > > > >> download
> >> > > > >> >> > new
> >> > > > >> >> > >> >> tokens
> >> > > > >> >> > >> >> > > > from zookeeper. It shouldn't be a big load on
> >> > > zookeeper
> >> > > > >> as a
> >> > > > >> >> > client
> >> > > > >> >> > >> >> will
> >> > > > >> >> > >> >> > > > need to get a new token once in several days.
> >> In this
> >> > > > >> approach
> >> > > > >> >> > you
> >> > > > >> >> > >> >> don't
> >> > > > >> >> > >> >> > > > need infinite lifetime on the token even for
> >> long
> >> > > > running
> >> > > > >> >> > clients.
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > > 3) The token password are generated using a
> >> master
> >> > > key.
> >> > > > >> The
> >> > > > >> >> > master
> >> > > > >> >> > >> >> key
> >> > > > >> >> > >> >> > > > should also be periodically changed. In
> Hadoop,
> >> the
> >> > > > >> default
> >> > > > >> >> > renewal
> >> > > > >> >> > >> >> > > > period is 1 day.?
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > IIUC, this will require brokers maintaining a
> >> list of X
> >> > > > most
> >> > > > >> >> > recent
> >> > > > >> >> > >> >> master
> >> > > > >> >> > >> >> > > keys. This list will have to be persisted
> >> somewhere, as
> >> > > > if a
> >> > > > >> >> > broker
> >> > > > >> >> > >> >> goes
> >> > > > >> >> > >> >> > > down it will have to get that list again and
> >> storing
> >> > > > master
> >> > > > >> keys
> >> > > > >> >> > on ZK
> >> > > > >> >> > >> >> is
> >> > > > >> >> > >> >> > > not the best idea. However, if a broker goes
> down
> >> then
> >> > > we
> >> > > > >> have
> >> > > > >> >> > much
> >> > > > >> >> > >> >> bigger
> >> > > > >> >> > >> >> > > issue to deal with and client can always
> >> > > re-authenticate
> >> > > > is
> >> > > > >> such
> >> > > > >> >> > >> >> events.
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >> > > Did you happen to take a look at other
> >> alternatives
> >> > > this
> >> > > > >> list has
> >> > > > >> >> > >> >> > > suggested?
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > > Thanks for a thorough proposal, great work!"
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira
> >> wrote:
> >> > > > >> >> > >> >> > > > > Makes sense to me. Thanks!
> >> > > > >> >> > >> >> > > > >
> >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <
> >> > > > kafka@harsha.io
> >> > > > >> >
> >> > > > >> >> > wrote:
> >> > > > >> >> > >> >> > > > > > It doesn't need any release vehicle but
> >> still the
> >> > > > >> work can
> >> > > > >> >> > move
> >> > > > >> >> > >> >> > > > forward.
> >> > > > >> >> > >> >> > > > > > If anyone is interested in the KIP please
> >> do the
> >> > > > >> review and
> >> > > > >> >> > >> >> provide
> >> > > > >> >> > >> >> > > the
> >> > > > >> >> > >> >> > > > > > comments.
> >> > > > >> >> > >> >> > > > > >
> >> > > > >> >> > >> >> > > > > > -Harsha
> >> > > > >> >> > >> >> > > > > >
> >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael
> >> Juma
> >> > > > wrote:
> >> > > > >> >> > >> >> > > > > >> I agree that it would be good to have
> more
> >> time
> >> > > to
> >> > > > >> review
> >> > > > >> >> > and
> >> > > > >> >> > >> >> > > discuss
> >> > > > >> >> > >> >> > > > > >> KIP-48.
> >> > > > >> >> > >> >> > > > > >>
> >> > > > >> >> > >> >> > > > > >> Ismael
> >> > > > >> >> > >> >> > > > > >>
> >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen
> >> Shapira <
> >> > > > >> >> > >> >> gwen@confluent.io>
> >> > > > >> >> > >> >> > > > wrote:
> >> > > > >> >> > >> >> > > > > >>
> >> > > > >> >> > >> >> > > > > >> > Hi Team,
> >> > > > >> >> > >> >> > > > > >> >
> >> > > > >> >> > >> >> > > > > >> > Since KIP-48 depends on KIP-43, which
> is
> >> > > > already a
> >> > > > >> bit
> >> > > > >> >> > of a
> >> > > > >> >> > >> >> risk
> >> > > > >> >> > >> >> > > for
> >> > > > >> >> > >> >> > > > > >> > the next release - any chance we can
> >> delay
> >> > > > >> delegation
> >> > > > >> >> > tokens
> >> > > > >> >> > >> >> to
> >> > > > >> >> > >> >> > > > Kafka
> >> > > > >> >> > >> >> > > > > >> > 0.10.1?
> >> > > > >> >> > >> >> > > > > >> > With the community working on a release
> >> every
> >> > > 3
> >> > > > >> month,
> >> > > > >> >> > this
> >> > > > >> >> > >> >> is not
> >> > > > >> >> > >> >> > > > a huge
> >> > > > >> >> > >> >> > > > > >> > delay.
> >> > > > >> >> > >> >> > > > > >> >
> >> > > > >> >> > >> >> > > > > >> > Gwen
> >> > > > >> >> > >> >> > > > > >> >
> >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish
> >> Singh
> >> > > <
> >> > > > >> >> > >> >> > > asingh@cloudera.com>
> >> > > > >> >> > >> >> > > > wrote:
> >> > > > >> >> > >> >> > > > > >> > > Parth,
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > > Thanks again for the awesome write
> up.
> >> > > > Following
> >> > > > >> our
> >> > > > >> >> > >> >> discussion
> >> > > > >> >> > >> >> > > > from the
> >> > > > >> >> > >> >> > > > > >> > > JIRA, I think it will be easier to
> >> compare
> >> > > > >> various
> >> > > > >> >> > >> >> alternatives
> >> > > > >> >> > >> >> > > > if they
> >> > > > >> >> > >> >> > > > > >> > are
> >> > > > >> >> > >> >> > > > > >> > > listed together. I am stating below a
> >> few
> >> > > > >> >> > alternatives along
> >> > > > >> >> > >> >> > > with
> >> > > > >> >> > >> >> > > > a the
> >> > > > >> >> > >> >> > > > > >> > > current proposal.
> >> > > > >> >> > >> >> > > > > >> > > (Current proposal) Store Delegation
> >> Token,
> >> > > DT,
> >> > > > >> on ZK.
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a
> >> broker.
> >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is authenticated,
> >> it
> >> > > will
> >> > > > >> make a
> >> > > > >> >> > broker
> >> > > > >> >> > >> >> side
> >> > > > >> >> > >> >> > > > call to
> >> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> >> > > > >> >> > >> >> > > > > >> > >    3. The broker generates a shared
> >> secret
> >> > > > based
> >> > > > >> on
> >> > > > >> >> > >> >> > > HMAC-SHA256(a
> >> > > > >> >> > >> >> > > > > >> > >    Password/Secret shared between all
> >> > > brokers,
> >> > > > >> >> > randomly
> >> > > > >> >> > >> >> > > generated
> >> > > > >> >> > >> >> > > > > >> > tokenId).
> >> > > > >> >> > >> >> > > > > >> > >    4. Broker stores this token in its
> >> in
> >> > > > memory
> >> > > > >> cache.
> >> > > > >> >> > >> >> Broker
> >> > > > >> >> > >> >> > > > also stores
> >> > > > >> >> > >> >> > > > > >> > >    the DelegationToken without the
> >> hmac in
> >> > > the
> >> > > > >> >> > zookeeper.
> >> > > > >> >> > >> >> > > > > >> > >    5. All brokers will have a cache
> >> backed
> >> > > by
> >> > > > >> >> > zookeeper so
> >> > > > >> >> > >> >> they
> >> > > > >> >> > >> >> > > > will all
> >> > > > >> >> > >> >> > > > > >> > >    get notified whenever a new token
> is
> >> > > > >> generated and
> >> > > > >> >> > they
> >> > > > >> >> > >> >> will
> >> > > > >> >> > >> >> > > > update
> >> > > > >> >> > >> >> > > > > >> > their
> >> > > > >> >> > >> >> > > > > >> > >    local cache whenever token state
> >> changes.
> >> > > > >> >> > >> >> > > > > >> > >    6. Broker returns the token to
> >> Client.
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >    1. Probable race condition, client
> >> tries
> >> > > to
> >> > > > >> >> > authenticate
> >> > > > >> >> > >> >> with
> >> > > > >> >> > >> >> > > > a broker
> >> > > > >> >> > >> >> > > > > >> > >    that is yet to be updated with the
> >> newly
> >> > > > >> generated
> >> > > > >> >> > DT.
> >> > > > >> >> > >> >> This
> >> > > > >> >> > >> >> > > can
> >> > > > >> >> > >> >> > > > > >> > probably be
> >> > > > >> >> > >> >> > > > > >> > >    dealt with making dtRequest block
> >> until
> >> > > all
> >> > > > >> >> > brokers have
> >> > > > >> >> > >> >> > > > updated
> >> > > > >> >> > >> >> > > > > >> > their DT
> >> > > > >> >> > >> >> > > > > >> > >    cache. Zk barrier or similar
> >> mechanism
> >> > > can
> >> > > > be
> >> > > > >> used.
> >> > > > >> >> > >> >> However,
> >> > > > >> >> > >> >> > > > all such
> >> > > > >> >> > >> >> > > > > >> > >    mechanisms will increase
> complexity.
> >> > > > >> >> > >> >> > > > > >> > >    2. Using a static secret key from
> >> config
> >> > > > >> file. Will
> >> > > > >> >> > >> >> require
> >> > > > >> >> > >> >> > > yet
> >> > > > >> >> > >> >> > > > > >> > another
> >> > > > >> >> > >> >> > > > > >> > >    config and uses a static secret
> >> key. It
> >> > > is
> >> > > > >> advised
> >> > > > >> >> > to
> >> > > > >> >> > >> >> rotate
> >> > > > >> >> > >> >> > > > secret
> >> > > > >> >> > >> >> > > > > >> > keys
> >> > > > >> >> > >> >> > > > > >> > >    periodically. This can be avoided
> >> with
> >> > > > >> controller
> >> > > > >> >> > >> >> generating
> >> > > > >> >> > >> >> > > > > >> > secretKey and
> >> > > > >> >> > >> >> > > > > >> > >    passing to brokers periodically.
> >> However,
> >> > > > >> this will
> >> > > > >> >> > >> >> require
> >> > > > >> >> > >> >> > > > brokers to
> >> > > > >> >> > >> >> > > > > >> > >    maintain certain counts of
> >> secretKeys.
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > > (Alternative 1) Have controller
> >> generate
> >> > > > >> delegation
> >> > > > >> >> > token.
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a
> >> broker.
> >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is authenticated,
> >> it
> >> > > will
> >> > > > >> make a
> >> > > > >> >> > broker
> >> > > > >> >> > >> >> side
> >> > > > >> >> > >> >> > > > call to
> >> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> >> > > > >> >> > >> >> > > > > >> > >    3. Broker forwards the request to
> >> > > > controller.
> >> > > > >> >> > >> >> > > > > >> > >    4. Controller generates a DT and
> >> > > broadcasts
> >> > > > >> to all
> >> > > > >> >> > >> >> brokers.
> >> > > > >> >> > >> >> > > > > >> > >    5. Broker stores this token in its
> >> memory
> >> > > > >> cache.
> >> > > > >> >> > >> >> > > > > >> > >    6. Controller responds to broker’s
> >> DT
> >> > > req.
> >> > > > >> >> > >> >> > > > > >> > >    7. Broker returns the token to
> >> Client.
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >    1. We will have to add new APIs to
> >> > > support
> >> > > > >> >> > controller
> >> > > > >> >> > >> >> pushing
> >> > > > >> >> > >> >> > > > tokens
> >> > > > >> >> > >> >> > > > > >> > to
> >> > > > >> >> > >> >> > > > > >> > >    brokers on top of the minimal APIs
> >> that
> >> > > are
> >> > > > >> >> > currently
> >> > > > >> >> > >> >> > > proposed.
> >> > > > >> >> > >> >> > > > > >> > >    2. We will also have to add APIs
> to
> >> > > support
> >> > > > >> the
> >> > > > >> >> > >> >> bootstrapping
> >> > > > >> >> > >> >> > > > case,
> >> > > > >> >> > >> >> > > > > >> > i.e,
> >> > > > >> >> > >> >> > > > > >> > >    when a new broker comes up it will
> >> have
> >> > > to
> >> > > > >> get all
> >> > > > >> >> > >> >> delegation
> >> > > > >> >> > >> >> > > > tokens
> >> > > > >> >> > >> >> > > > > >> > from
> >> > > > >> >> > >> >> > > > > >> > >    the controller.
> >> > > > >> >> > >> >> > > > > >> > >    3. In catastrophic failures where
> >> all
> >> > > > brokers
> >> > > > >> go
> >> > > > >> >> > down,
> >> > > > >> >> > >> >> the
> >> > > > >> >> > >> >> > > > tokens will
> >> > > > >> >> > >> >> > > > > >> > >    be lost even if servers are
> >> restarted as
> >> > > > >> tokens
> >> > > > >> >> > are not
> >> > > > >> >> > >> >> > > > persisted
> >> > > > >> >> > >> >> > > > > >> > anywhere.
> >> > > > >> >> > >> >> > > > > >> > >    If this happens, then there are
> more
> >> > > > important
> >> > > > >> >> > things to
> >> > > > >> >> > >> >> > > worry
> >> > > > >> >> > >> >> > > > about
> >> > > > >> >> > >> >> > > > > >> > and
> >> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
> >> re-authenticate.
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > > (Alternative 2) Do not distribute DT
> to
> >> > > other
> >> > > > >> brokers
> >> > > > >> >> > at
> >> > > > >> >> > >> >> all.
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a
> >> broker.
> >> > > > >> >> > >> >> > > > > >> > >    2. Once a client is authenticated,
> >> it
> >> > > will
> >> > > > >> make a
> >> > > > >> >> > broker
> >> > > > >> >> > >> >> side
> >> > > > >> >> > >> >> > > > call to
> >> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> >> > > > >> >> > >> >> > > > > >> > >    3. The broker generates DT of
> form,
> >> > > [hmac +
> >> > > > >> (owner,
> >> > > > >> >> > >> >> renewer,
> >> > > > >> >> > >> >> > > > > >> > >    maxLifeTime, id, hmac,
> >> expirationTime)]
> >> > > and
> >> > > > >> passes
> >> > > > >> >> > back
> >> > > > >> >> > >> >> this
> >> > > > >> >> > >> >> > > > DT to
> >> > > > >> >> > >> >> > > > > >> > client.
> >> > > > >> >> > >> >> > > > > >> > >    hmac is generated via
> >> {HMAC-SHA256(owner,
> >> > > > >> renewer,
> >> > > > >> >> > >> >> > > > maxLifeTime, id,
> >> > > > >> >> > >> >> > > > > >> > hmac,
> >> > > > >> >> > >> >> > > > > >> > >    expirationTime) using SecretKey}.
> >> Note
> >> > > that
> >> > > > >> all
> >> > > > >> >> > brokers
> >> > > > >> >> > >> >> have
> >> > > > >> >> > >> >> > > > this
> >> > > > >> >> > >> >> > > > > >> > SecretKey.
> >> > > > >> >> > >> >> > > > > >> > >    4. Client then goes to any broker
> >> and to
> >> > > > >> >> > authenticate
> >> > > > >> >> > >> >> sends
> >> > > > >> >> > >> >> > > > the DT.
> >> > > > >> >> > >> >> > > > > >> > >    Broker recalculates hmac using
> >> (owner,
> >> > > > >> renewer,
> >> > > > >> >> > >> >> maxLifeTime,
> >> > > > >> >> > >> >> > > > id, hmac,
> >> > > > >> >> > >> >> > > > > >> > >    expirationTime) info from DT and
> its
> >> > > > >> SecretKey. If
> >> > > > >> >> > it
> >> > > > >> >> > >> >> matches
> >> > > > >> >> > >> >> > > > with
> >> > > > >> >> > >> >> > > > > >> > hmac of
> >> > > > >> >> > >> >> > > > > >> > >    DT, client is authenticated. Yes,
> >> it will
> >> > > > do
> >> > > > >> other
> >> > > > >> >> > >> >> obvious
> >> > > > >> >> > >> >> > > > checks of
> >> > > > >> >> > >> >> > > > > >> > >    timestamp expiry and such.
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > > Note that secret key will be
> generated
> >> by
> >> > > > >> controller
> >> > > > >> >> > and
> >> > > > >> >> > >> >> passed
> >> > > > >> >> > >> >> > > to
> >> > > > >> >> > >> >> > > > > >> > brokers
> >> > > > >> >> > >> >> > > > > >> > > periodically.
> >> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >    1. How to delete a DT? Yes, that
> is
> >> a
> >> > > > downside
> >> > > > >> >> > here.
> >> > > > >> >> > >> >> However,
> >> > > > >> >> > >> >> > > > this can
> >> > > > >> >> > >> >> > > > > >> > >    be handled with brokers
> maintaining
> >> a
> >> > > > >> blacklist of
> >> > > > >> >> > DTs,
> >> > > > >> >> > >> >> DTs
> >> > > > >> >> > >> >> > > > from this
> >> > > > >> >> > >> >> > > > > >> > list
> >> > > > >> >> > >> >> > > > > >> > >    can be removed after expiry.
> >> > > > >> >> > >> >> > > > > >> > >    2. In catastrophic failures where
> >> all
> >> > > > brokers
> >> > > > >> go
> >> > > > >> >> > down,
> >> > > > >> >> > >> >> the
> >> > > > >> >> > >> >> > > > tokens will
> >> > > > >> >> > >> >> > > > > >> > >    be lost even if servers are
> >> restarted as
> >> > > > >> tokens
> >> > > > >> >> > are not
> >> > > > >> >> > >> >> > > > persisted
> >> > > > >> >> > >> >> > > > > >> > anywhere.
> >> > > > >> >> > >> >> > > > > >> > >    If this happens, then there are
> more
> >> > > > important
> >> > > > >> >> > things to
> >> > > > >> >> > >> >> > > worry
> >> > > > >> >> > >> >> > > > about
> >> > > > >> >> > >> >> > > > > >> > and
> >> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
> >> re-authenticate.
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM,
> Parth
> >> > > > >> Brahmbhatt <
> >> > > > >> >> > >> >> > > > > >> > > pbrahmbhatt@hortonworks.com> wrote:
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >> Hi,
> >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > >> >> > >> >> > > > > >> > >> I have filed KIP-48 so we can offer
> >> hadoop
> >> > > > like
> >> > > > >> >> > delegation
> >> > > > >> >> > >> >> > > > tokens in
> >> > > > >> >> > >> >> > > > > >> > >> kafka. You can review the design
> >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > >> >> > >> >> > > > > >> >
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >>
> >> > > > >> >> >
> >> > > > >>
> >> > > >
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> >> > > > >> >> > >> >> > > > > >> > .
> >> > > > >> >> > >> >> > > > > >> > >> This KIP depends on KIP-43 and we
> >> have also
> >> > > > >> >> > discussed an
> >> > > > >> >> > >> >> > > > alternative to
> >> > > > >> >> > >> >> > > > > >> > >> proposed design here<
> >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > >> >> > >> >> > > > > >> >
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >>
> >> > > > >> >> >
> >> > > > >>
> >> > > >
> >> > >
> >>
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> >> > > > >> >> > >> >> > > > > >> > >> >.
> >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > >> >> > >> >> > > > > >> > >> Thanks
> >> > > > >> >> > >> >> > > > > >> > >> Parth
> >> > > > >> >> > >> >> > > > > >> > >>
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > > --
> >> > > > >> >> > >> >> > > > > >> > >
> >> > > > >> >> > >> >> > > > > >> > > Regards,
> >> > > > >> >> > >> >> > > > > >> > > Ashish
> >> > > > >> >> > >> >> > > > > >> >
> >> > > > >> >> > >> >> > > >
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >> > > --
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >> > > Regards,
> >> > > > >> >> > >> >> > > Ashish
> >> > > > >> >> > >> >> > >
> >> > > > >> >> > >> >>
> >> > > > >> >> >
> >> > > > >>
> >> > > >
> >> > >
> >>
> >>
> >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Jun Rao <ju...@confluent.io>]

Thanks for the KIP. A few comments.

100. This potentially can be useful for Kafka Connect and Kafka rest proxy
where a worker agent will need to run a task on behalf of a client. We will
likely need to change how those services use Kafka clients
(producer/consumer). Instead of a shared client per worker, we will need a
client per user task since the authentication happens at the connection
level. For Kafka Connect, the renewer will be the workers. So, we probably
need to allow multiple renewers. For Kafka rest proxy, the renewer can
probably just be the creator of the token.

101. Do we need new acl on who can request delegation tokens?

102. Do we recommend people to send delegation tokens in an encrypted
channel?

103. Who is responsible for expiring tokens, every broker?

104. For invalidating tokens, would it be better to do it in a request
instead of going to ZK directly?

105. The terminology of client in the wiki sometimes refers to the end
client and some other times refers to the client using the delegation
tokens. It would be useful to distinguish between the two.

106. Could you explain the sentence "Broker also stores the DelegationToken
without the hmac in the zookeeper." a bit more? I thought the delegation
token is the hmac.

Thanks,

Jun


On Mon, May 23, 2016 at 9:22 AM, Jun Rao <ju...@confluent.io> wrote:

> Hi, Harsha,
>
> Just sent out a KIP meeting invite. We can discuss this in the meeting
> tomorrow.
>
> Thanks,
>
> Jun
>
> On Thu, May 19, 2016 at 8:47 AM, Harsha <ka...@harsha.io> wrote:
>
>> Hi All,
>>            Can we have a KIP meeting around this. The KIP is up for
>>            sometime and if there are any questions lets quickly hash out
>>            details.
>>
>> Thanks,
>> Harsha
>>
>> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt wrote:
>> > That is what the hadoop echo system uses so no good reason really. We
>> > could
>> > change it to whatever is the newest recommended standard is.
>> >
>> > Thanks
>> > Parth
>> >
>> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <is...@juma.me.uk> wrote:
>> >
>> > > Hi Parth,
>> > >
>> > > Thanks for the KIP. I only started reviewing this and may have
>> additional
>> > > questions later. The immediate question that came to mind is our
>> choice of
>> > > "DIGEST-MD5" even though it's marked as OBSOLETE in the IANA Registry
>> of
>> > > SASL mechanisms and the original RFC (2831) has been moved to Historic
>> > > status:
>> > >
>> > > https://tools.ietf.org/html/rfc6331
>> > > http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
>> > >
>> > > What is the reasoning behind that choice?
>> > >
>> > > Thanks,
>> > > Ismael
>> > >
>> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <gw...@confluent.io>
>> wrote:
>> > >
>> > > > Also comments inline :)
>> > > >
>> > > > > * I want to emphasize that even though delegation tokens are a
>> Hadoop
>> > > > > innovation, I feel very strongly about not adding dependency on
>> Hadoop
>> > > > > when implementing delegation tokens for Kafka. The KIP doesn't
>> imply
>> > > > > such dependency, but if you can clarify...
>> > > > >
>> > > > >
>> > > > > *No hadoop dependency.*
>> > > >
>> > > > Yay! Just add this to the KIP so no one will read the KIP and panic
>> > > > three weeks before the next release...
>> > > >
>> > > > > * Can we get delegation token at any time after authenticating?
>> only
>> > > > > immediately after?
>> > > > >
>> > > > >
>> > > > > *As long as you are authenticated you can get delegation tokens.
>> We
>> > > need
>> > > > to
>> > > > > discuss if a client authenticated using delegation token, can also
>> > > > acquire
>> > > > > delegation token again or not. Also there is the question of do we
>> > > allow
>> > > > > anyone to acquire delegation token or we want specific ACLs (I
>> think
>> > > its
>> > > > an
>> > > > > overkill.)*
>> > > >
>> > > > I agree that ACLs is an overkill.
>> > > >
>> > > > I think we are debating two options: Either require Kerberos auth
>> for
>> > > > renewal or require non-owners to renew.
>> > > > I *think* the latter is simpler (it basically require a "job master"
>> > > > to take responsibility for the renewal, it will have its own
>> identity
>> > > > anyway and I think this is the correct design pattern anyway. For
>> > > > storm, I'd expect Nimbus to coordinate renewals?), but it is hard to
>> > > > debate simplicity without looking at the code changes required. If
>> you
>> > > > have a draft of how the "require Kerberos" will look in Kafka code,
>> > > > I'll be happy to take a look.
>> > > >
>> > > > > * My understanding is that tokens will propagate via ZK but
>> without
>> > > > > additional changes to UpdateMetadata protocol, correct? Clients
>> > > > > currently don't retry on SASL auth failure (IIRC), but since the
>> > > > > tokens propagate between brokers asynch, we will need to retry a
>> bit
>> > > > > to avoid clients failing auth due to timing issues.
>> > > > >
>> > > > > *I am considering 2 alternatives right now. The current documented
>> > > > approach
>> > > > > is zookeeper based and it does not require any changes to
>> > > UpdateMetadata
>> > > > > protocol. An alternative approach can remove zookeeper dependency
>> as
>> > > well
>> > > > > but we can discuss that in KIP discussion call.*
>> > > >
>> > > > Oooh! Sounds interesting. Do you want to ping Jun to arrange a call?
>> > > >
>> > > > > * I liked Ashish's suggestion of having just the controller issue
>> the
>> > > > > delegation tokens, to avoid syncing a shared secret. Not sure if
>> we
>> > > > > want to continue the discussion here or on the wiki. I think that
>> we
>> > > > > can decouple the problem of "token distribution" from "shared
>> secret
>> > > > > distribution" and use the controller as the only token generator
>> to
>> > > > > solve the second issue, while still using ZK async to distribute
>> > > > > tokens.
>> > > > >
>> > > > >
>> > > > > *As mentioned in the previous Email I am fine with that approach
>> as
>> > > long
>> > > > as
>> > > > > we agree that the extra complexity of adding/updating APIS adds
>> enough
>> > > > > value. The advantage with the controller approach is secret
>> rotation
>> > > can
>> > > > be
>> > > > > automated,frequent and would not require deployment. *
>> > > >
>> > > > Can you detail the extra complexity (or point me to the email I
>> > > > missed?) - which Apis are required?
>> > > > As far as I can tell, clients can already find the controller from
>> > > > metadata. I'm a bit more concerned about controller load.
>> > > >
>> > > > >
>> > > > > * While I like the idea of forcing kerberos auth for renwal, I
>> think
>> > > > > it mixes the transport layer the the request content in a pretty
>> ugly
>> > > > > way. Perhaps limiting renewer to non-owner is better.
>> > > > >
>> > > > > *I feel this is a necessary evil. While this will make the kafka
>> code
>> > > > > pretty straight forward , forcing  renewer to non-owner pushes
>> the code
>> > > > > ugliness to client and makes it even harder to integrate.  *
>> > > >
>> > > > As mentioned before, I don't think the "renewal by other" approach
>> is
>> > > > that ugly for the clients we expect to use delegation tokens since
>> > > > they will have an app-master of some sort who requested the token to
>> > > > begin with.
>> > > >
>> > > > >
>> > > > > The response for my question on how multiple identities will be
>> > > > > handled wasn't super clear to me - AFAIK, we don't authenticate
>> each
>> > > > > request, we authenticate connections.
>> > > > >
>> > > > > *We authenticate connections, and only when they are being
>> established.
>> > > > Let
>> > > > > me try to phrase this as a question, in absence of delegation
>> tokens if
>> > > > we
>> > > > > had to support the use case using user TGT's how would we do it?
>> My
>> > > point
>> > > > > was it would be no different with delegation tokens. The use case
>> you
>> > > are
>> > > > > describing seems more like impersonation.*
>> > > >
>> > > > Yeah, I thought that one of the things that delegation tokens
>> handled.
>> > > > Maybe I got it wrong :)
>> > > >
>> > > > Thanks for the detailed answers.
>> > > >
>> > > > Gwen
>> > > >
>> > > >
>> > > > > Thanks
>> > > > > Parth
>> > > > >
>> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <gwen@confluent.io
>> >
>> > > > wrote:
>> > > > >
>> > > > >> Hi Parth and Harsha,
>> > > > >>
>> > > > >> Few more comments:
>> > > > >>
>> > > > >> * The API / RequestResponse section doesn't seem to have good
>> > > > >> description of the changes to the Kafka Protocol. Sounds like
>> you are
>> > > > >> proposing new DelegationTokenRequest and RenewTokenRequest (and
>> > > > >> matching responses), without detailing the contents of the
>> requests
>> > > > >> and responses? Or rather, you show the class interface, but not
>> the
>> > > > >> underlying protocol. This could be seen as an implementation
>> detail,
>> > > > >> but since the binary protocol is what we provide to non-Java
>> clients,
>> > > > >> we need to show the changes there.
>> > > > >>
>> > > > >> * getDelegationToken sounds like delegationTokenRequestHandler?
>> Is it
>> > > > >> planned to be part of KafkaApi? or Client? Its unclear...
>> > > > >>
>> > > > >> * I want to emphasize that even though delegation tokens are a
>> Hadoop
>> > > > >> innovation, I feel very strongly about not adding dependency on
>> Hadoop
>> > > > >> when implementing delegation tokens for Kafka. The KIP doesn't
>> imply
>> > > > >> such dependency, but if you can clarify...
>> > > > >>
>> > > > >> * Can we get delegation token at any time after authenticating?
>> only
>> > > > >> immediately after?
>> > > > >>
>> > > > >> * My understanding is that tokens will propagate via ZK but
>> without
>> > > > >> additional changes to UpdateMetadata protocol, correct? Clients
>> > > > >> currently don't retry on SASL auth failure (IIRC), but since the
>> > > > >> tokens propagate between brokers asynch, we will need to retry a
>> bit
>> > > > >> to avoid clients failing auth due to timing issues.
>> > > > >>
>> > > > >> * Strongly agreeing on clients not touching ZK directly :)
>> > > > >>
>> > > > >> * I liked Ashish's suggestion of having just the controller
>> issue the
>> > > > >> delegation tokens, to avoid syncing a shared secret. Not sure if
>> we
>> > > > >> want to continue the discussion here or on the wiki. I think
>> that we
>> > > > >> can decouple the problem of "token distribution" from "shared
>> secret
>> > > > >> distribution" and use the controller as the only token generator
>> to
>> > > > >> solve the second issue, while still using ZK async to distribute
>> > > > >> tokens.
>> > > > >>
>> > > > >> * I am also uncomfortable with infinite lifetime of tokens (and
>> hoped
>> > > > >> to hear from others in the community) - but having the option and
>> > > > >> documenting it as less secure, allows users to configure their
>> system
>> > > > >> as the wish.
>> > > > >>
>> > > > >> * While I like the idea of forcing kerberos auth for renwal, I
>> think
>> > > > >> it mixes the transport layer the the request content in a pretty
>> ugly
>> > > > >> way. Perhaps limiting renewer to non-owner is better.
>> > > > >>
>> > > > >> Things I'd still like to see:
>> > > > >>
>> > > > >> * More detailed explanation on what we plan to do for the java
>> clients
>> > > > >> specifically - new configuration? new APIs?
>> > > > >> The response for my question on how multiple identities will be
>> > > > >> handled wasn't super clear to me - AFAIK, we don't authenticate
>> each
>> > > > >> request, we authenticate connections.
>> > > > >>
>> > > > >> * Alternatives: Delegation tokens are only used in the Hadoop
>> > > > >> ecosystem. I'm wondering if there are alternatives in other
>> ecosystems
>> > > > >> (Mesos? Tachyon? Cassandra?) and whether there are some
>> advantages
>> > > > >> there.
>> > > > >>
>> > > > >> Gwen
>> > > > >>
>> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <ka...@harsha.io> wrote:
>> > > > >> > Hi Gwen,
>> > > > >> >            Can you look at Parth's last reply. Does it answer
>> your
>> > > > >> >            concerns.
>> > > > >> >
>> > > > >> > Thanks,
>> > > > >> > Harsha
>> > > > >> >
>> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt wrote:
>> > > > >> >> Thanks for reviewing Gwen. The wiki already has details on
>> token
>> > > > >> >> expiration
>> > > > >> >> under token acquisition process
>> > > > >> >> <
>> > > > >>
>> > > >
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
>> > > > >> >.
>> > > > >> >> Current proposal is that tokens will expire based on a server
>> side
>> > > > >> >> configuration (default 24 hours) unless renewed. Renewal is
>> only
>> > > > allowed
>> > > > >> >> until the max life time of token. Alternatively we could also
>> make
>> > > > that
>> > > > >> >> an
>> > > > >> >> optional param and the server side default can serve as the
>> upper
>> > > > bound.
>> > > > >> >>
>> > > > >> >> To your second point it will be done exactly the same way we
>> would
>> > > > >> >> support
>> > > > >> >> multiple keytabs. The calling client will have to put the
>> tokens it
>> > > > >> wants
>> > > > >> >> to use in the subject instance and call produce/consume inside
>> > > > >> >> subject.doas. Each caller will have to keep track of its own
>> > > > subject. I
>> > > > >> >> will have to look at the code to see if we support this
>> feature
>> > > right
>> > > > >> now
>> > > > >> >> but my understanding is delegation token shouldn't need any
>> special
>> > > > >> >> treatment as its just another type of Credential in the
>> subject.
>> > > > >> >>
>> > > > >> >> I would also like to know what is your opinion about infinite
>> > > renewal
>> > > > >> (my
>> > > > >> >> recommendation is to not support this), tokens renewing them
>> > > self(my
>> > > > >> >> recommendation is to not support this) and most importantly
>> your
>> > > > choice
>> > > > >> >> between the alternatives listed on this thread
>> > > > >> >> <
>> > > > >>
>> > > >
>> > >
>> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
>> > > > >> >
>> > > > >> >> ( I am leaning towards the alternative-2 minus controller
>> > > > distributing
>> > > > >> >> secret). Thanks again for reviewing.
>> > > > >> >>
>> > > > >> >> Thanks
>> > > > >> >> Parth
>> > > > >> >>
>> > > > >> >>
>> > > > >> >>
>> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <
>> gwen@confluent.io>
>> > > > wrote:
>> > > > >> >>
>> > > > >> >> > Harsha,
>> > > > >> >> >
>> > > > >> >> > I was thinking of the Rest Proxy. I didn't see your design
>> yet,
>> > > > but in
>> > > > >> >> > our proxy, we have a set of producers, which will serve
>> multiple
>> > > > users
>> > > > >> >> > going through the proxy. Since these users will have
>> different
>> > > > >> >> > privileges, they'll need to authenticate separately, and
>> can't
>> > > > share a
>> > > > >> >> > token.
>> > > > >> >> >
>> > > > >> >> > Am I missing anything?
>> > > > >> >> >
>> > > > >> >> > Gwen
>> > > > >> >> >
>> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io>
>> wrote:
>> > > > >> >> > > Gwen,
>> > > > >> >> > >            On your second point. Can you describe a
>> usecase
>> > > where
>> > > > >> >> > >            mutliple clients ended up sharing a producer
>> and
>> > > even
>> > > > if
>> > > > >> they
>> > > > >> >> > >            do why can't they not use single token that
>> producer
>> > > > >> >> > >            captures. Why would we need multiple clients
>> with
>> > > > >> different
>> > > > >> >> > >            tokens sharing a single instance of producer.
>> Also
>> > > in
>> > > > >> this
>> > > > >> >> > >            case other clients have access all the tokens
>> no?
>> > > > >> >> > >
>> > > > >> >> > > Thanks,
>> > > > >> >> > > Harsha
>> > > > >> >> > >
>> > > > >> >> > >
>> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote:
>> > > > >> >> > >> Sorry for the delay:
>> > > > >> >> > >>
>> > > > >> >> > >> Two questions that we didn't see in the wiki:
>> > > > >> >> > >> 1. Is there an expiration for delegation tokens?
>> Renewal? How
>> > > > do we
>> > > > >> >> > >> revoke them?
>> > > > >> >> > >> 2. If we want to use delegation tokens for "do-as" (say,
>> > > submit
>> > > > >> Storm
>> > > > >> >> > >> job as my user), we will need a producer for every job
>> (we
>> > > can't
>> > > > >> share
>> > > > >> >> > >> them between multiple jobs running on same node), since
>> we
>> > > only
>> > > > >> >> > >> authenticate when connecting. Is there a plan to change
>> this
>> > > for
>> > > > >> >> > >> delegation tokens, in order to allow multiple users with
>> > > > different
>> > > > >> >> > >> tokens to share a client?
>> > > > >> >> > >>
>> > > > >> >> > >> Gwen
>> > > > >> >> > >>
>> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
>> > > > >> >> > >> <br...@gmail.com> wrote:
>> > > > >> >> > >> > Bumping this up one more time, can other committers
>> review?
>> > > > >> >> > >> >
>> > > > >> >> > >> > Thanks
>> > > > >> >> > >> > Parth
>> > > > >> >> > >> >
>> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <
>> kafka@harsha.io>
>> > > > wrote:
>> > > > >> >> > >> >
>> > > > >> >> > >> >> Parth,
>> > > > >> >> > >> >>           Overall current design looks good to me. I
>> am +1
>> > > on
>> > > > >> the
>> > > > >> >> > KIP.
>> > > > >> >> > >> >>
>> > > > >> >> > >> >> Gwen , Jun can you review this as well.
>> > > > >> >> > >> >>
>> > > > >> >> > >> >> -Harsha
>> > > > >> >> > >> >>
>> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt
>> wrote:
>> > > > >> >> > >> >> > Thanks for review Jitendra.
>> > > > >> >> > >> >> >
>> > > > >> >> > >> >> > I don't like the idea of infinite lifetime but I
>> see the
>> > > > >> Streaming
>> > > > >> >> > use
>> > > > >> >> > >> >> > case. Even for Streaming use case I was hoping
>> there will
>> > > > be
>> > > > >> some
>> > > > >> >> > notion
>> > > > >> >> > >> >> > of
>> > > > >> >> > >> >> > master/driver that can get new delegation tokens at
>> fixed
>> > > > >> interval
>> > > > >> >> > and
>> > > > >> >> > >> >> > distribute to workers. If that is not the case for
>> we can
>> > > > >> discuss
>> > > > >> >> > >> >> > delegation tokens renewing them self and the
>> security
>> > > > >> implications
>> > > > >> >> > of the
>> > > > >> >> > >> >> > same.
>> > > > >> >> > >> >> >
>> > > > >> >> > >> >> > I did not want clients to fetch tokens from
>> zookeeper,
>> > > > >> overall I
>> > > > >> >> > think
>> > > > >> >> > >> >> > its
>> > > > >> >> > >> >> > better if clients don't rely on our metadata store
>> and I
>> > > > >> think we
>> > > > >> >> > are
>> > > > >> >> > >> >> > moving in that direction with all the KIP-4
>> improvements.
>> > > > I
>> > > > >> chose
>> > > > >> >> > >> >> > zookeeper as in this case the client will still
>> just talk
>> > > > to
>> > > > >> >> > broker , its
>> > > > >> >> > >> >> > the brokers that will use zookeeper which we
>> already do
>> > > > for a
>> > > > >> lot
>> > > > >> >> > of
>> > > > >> >> > >> >> > other
>> > > > >> >> > >> >> > usecases + ease of development + and the ability so
>> > > tokens
>> > > > >> will
>> > > > >> >> > survive
>> > > > >> >> > >> >> > even a rolling restart/cluster failure. if a
>> majority
>> > > > agrees
>> > > > >> the
>> > > > >> >> > added
>> > > > >> >> > >> >> > complexity to have controller forwarding keys to all
>> > > > broker is
>> > > > >> >> > justified
>> > > > >> >> > >> >> > as
>> > > > >> >> > >> >> > it provides tighter security , I am fine with that
>> option
>> > > > too.
>> > > > >> >> > >> >> >
>> > > > >> >> > >> >> > Given zookeeper does not support SSL we can not
>> store
>> > > > master
>> > > > >> keys
>> > > > >> >> > in
>> > > > >> >> > >> >> > zookeeper as master keys will be exposed on wire. To
>> > > > support
>> > > > >> >> > rotation
>> > > > >> >> > >> >> > without affecting current clients is something I
>> need to
>> > > > put
>> > > > >> more
>> > > > >> >> > thought
>> > > > >> >> > >> >> > in. My current proposal assumes the rotation will
>> > > > invalidate
>> > > > >> all
>> > > > >> >> > current
>> > > > >> >> > >> >> > tokens.
>> > > > >> >> > >> >> >
>> > > > >> >> > >> >> > I request committers to also review and post their
>> > > comments
>> > > > >> so we
>> > > > >> >> > can
>> > > > >> >> > >> >> > make
>> > > > >> >> > >> >> > progress on this KIP.
>> > > > >> >> > >> >> >
>> > > > >> >> > >> >> > Thanks
>> > > > >> >> > >> >> > Parth
>> > > > >> >> > >> >> >
>> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <
>> > > > >> asingh@cloudera.com
>> > > > >> >> > >
>> > > > >> >> > >> >> > wrote:
>> > > > >> >> > >> >> >
>> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <
>> > > > kafka@harsha.io>
>> > > > >> >> > wrote:
>> > > > >> >> > >> >> > >
>> > > > >> >> > >> >> > > > Unifying the two discussion threads on this KIP.
>> > > > >> >> > >> >> > > >
>> > > > >> >> > >> >> > > > Here is the response from Jitendra
>> > > > >> >> > >> >> > > >
>> > > > >> >> > >> >> > > > "The need for a large number of clients that are
>> > > > running
>> > > > >> all
>> > > > >> >> > over the
>> > > > >> >> > >> >> > > > cluster that authenticate with Kafka brokers,
>> is very
>> > > > >> similar
>> > > > >> >> > to the
>> > > > >> >> > >> >> > > > Hadoop use case of large number of tasks running
>> > > across
>> > > > >> the
>> > > > >> >> > cluster
>> > > > >> >> > >> >> that
>> > > > >> >> > >> >> > > > need authentication to Hdfs Namenode.
>> Therefore, the
>> > > > >> >> > delegation token
>> > > > >> >> > >> >> > > > approach does seem like a good fit for this use
>> case
>> > > > as we
>> > > > >> >> > have seen
>> > > > >> >> > >> >> it
>> > > > >> >> > >> >> > > > working at large scale in HDFS and YARN.
>> > > > >> >> > >> >> > > >
>> > > > >> >> > >> >> > > >   The proposed design is very much inline with
>> Hadoop
>> > > > >> >> > approach. A few
>> > > > >> >> > >> >> > > >   comments:
>> > > > >> >> > >> >> > > >
>> > > > >> >> > >> >> > > > 1) Why do you guys want to allow infinite
>> renewable
>> > > > >> lifetime
>> > > > >> >> > for a
>> > > > >> >> > >> >> > > > token? HDFS restricts a token to a max life time
>> > > > (default
>> > > > >> 7
>> > > > >> >> > days).  A
>> > > > >> >> > >> >> > > > token's vulnerability is believed to increase
>> with
>> > > > time.
>> > > > >> >> > >> >> > > >
>> > > > >> >> > >> >> > > I agree that having infinite lifetime might not
>> be the
>> > > > best
>> > > > >> idea.
>> > > > >> >> > >> >> > >
>> > > > >> >> > >> >> > > >
>> > > > >> >> > >> >> > > > 2) As I understand the tokens are stored in
>> zookeeper
>> > > > as
>> > > > >> well,
>> > > > >> >> > and
>> > > > >> >> > >> >> can
>> > > > >> >> > >> >> > > > be updated there. This is clever as it can allow
>> > > > >> replacing the
>> > > > >> >> > tokens
>> > > > >> >> > >> >> > > > once they run out of max life time, and clients
>> can
>> > > > >> download
>> > > > >> >> > new
>> > > > >> >> > >> >> tokens
>> > > > >> >> > >> >> > > > from zookeeper. It shouldn't be a big load on
>> > > zookeeper
>> > > > >> as a
>> > > > >> >> > client
>> > > > >> >> > >> >> will
>> > > > >> >> > >> >> > > > need to get a new token once in several days.
>> In this
>> > > > >> approach
>> > > > >> >> > you
>> > > > >> >> > >> >> don't
>> > > > >> >> > >> >> > > > need infinite lifetime on the token even for
>> long
>> > > > running
>> > > > >> >> > clients.
>> > > > >> >> > >> >> > > >
>> > > > >> >> > >> >> > > > 3) The token password are generated using a
>> master
>> > > key.
>> > > > >> The
>> > > > >> >> > master
>> > > > >> >> > >> >> key
>> > > > >> >> > >> >> > > > should also be periodically changed. In Hadoop,
>> the
>> > > > >> default
>> > > > >> >> > renewal
>> > > > >> >> > >> >> > > > period is 1 day.?
>> > > > >> >> > >> >> > > >
>> > > > >> >> > >> >> > > IIUC, this will require brokers maintaining a
>> list of X
>> > > > most
>> > > > >> >> > recent
>> > > > >> >> > >> >> master
>> > > > >> >> > >> >> > > keys. This list will have to be persisted
>> somewhere, as
>> > > > if a
>> > > > >> >> > broker
>> > > > >> >> > >> >> goes
>> > > > >> >> > >> >> > > down it will have to get that list again and
>> storing
>> > > > master
>> > > > >> keys
>> > > > >> >> > on ZK
>> > > > >> >> > >> >> is
>> > > > >> >> > >> >> > > not the best idea. However, if a broker goes down
>> then
>> > > we
>> > > > >> have
>> > > > >> >> > much
>> > > > >> >> > >> >> bigger
>> > > > >> >> > >> >> > > issue to deal with and client can always
>> > > re-authenticate
>> > > > is
>> > > > >> such
>> > > > >> >> > >> >> events.
>> > > > >> >> > >> >> > >
>> > > > >> >> > >> >> > > Did you happen to take a look at other
>> alternatives
>> > > this
>> > > > >> list has
>> > > > >> >> > >> >> > > suggested?
>> > > > >> >> > >> >> > >
>> > > > >> >> > >> >> > > >
>> > > > >> >> > >> >> > > > Thanks for a thorough proposal, great work!"
>> > > > >> >> > >> >> > > >
>> > > > >> >> > >> >> > > >
>> > > > >> >> > >> >> > > >
>> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira
>> wrote:
>> > > > >> >> > >> >> > > > > Makes sense to me. Thanks!
>> > > > >> >> > >> >> > > > >
>> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <
>> > > > kafka@harsha.io
>> > > > >> >
>> > > > >> >> > wrote:
>> > > > >> >> > >> >> > > > > > It doesn't need any release vehicle but
>> still the
>> > > > >> work can
>> > > > >> >> > move
>> > > > >> >> > >> >> > > > forward.
>> > > > >> >> > >> >> > > > > > If anyone is interested in the KIP please
>> do the
>> > > > >> review and
>> > > > >> >> > >> >> provide
>> > > > >> >> > >> >> > > the
>> > > > >> >> > >> >> > > > > > comments.
>> > > > >> >> > >> >> > > > > >
>> > > > >> >> > >> >> > > > > > -Harsha
>> > > > >> >> > >> >> > > > > >
>> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael
>> Juma
>> > > > wrote:
>> > > > >> >> > >> >> > > > > >> I agree that it would be good to have more
>> time
>> > > to
>> > > > >> review
>> > > > >> >> > and
>> > > > >> >> > >> >> > > discuss
>> > > > >> >> > >> >> > > > > >> KIP-48.
>> > > > >> >> > >> >> > > > > >>
>> > > > >> >> > >> >> > > > > >> Ismael
>> > > > >> >> > >> >> > > > > >>
>> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen
>> Shapira <
>> > > > >> >> > >> >> gwen@confluent.io>
>> > > > >> >> > >> >> > > > wrote:
>> > > > >> >> > >> >> > > > > >>
>> > > > >> >> > >> >> > > > > >> > Hi Team,
>> > > > >> >> > >> >> > > > > >> >
>> > > > >> >> > >> >> > > > > >> > Since KIP-48 depends on KIP-43, which is
>> > > > already a
>> > > > >> bit
>> > > > >> >> > of a
>> > > > >> >> > >> >> risk
>> > > > >> >> > >> >> > > for
>> > > > >> >> > >> >> > > > > >> > the next release - any chance we can
>> delay
>> > > > >> delegation
>> > > > >> >> > tokens
>> > > > >> >> > >> >> to
>> > > > >> >> > >> >> > > > Kafka
>> > > > >> >> > >> >> > > > > >> > 0.10.1?
>> > > > >> >> > >> >> > > > > >> > With the community working on a release
>> every
>> > > 3
>> > > > >> month,
>> > > > >> >> > this
>> > > > >> >> > >> >> is not
>> > > > >> >> > >> >> > > > a huge
>> > > > >> >> > >> >> > > > > >> > delay.
>> > > > >> >> > >> >> > > > > >> >
>> > > > >> >> > >> >> > > > > >> > Gwen
>> > > > >> >> > >> >> > > > > >> >
>> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish
>> Singh
>> > > <
>> > > > >> >> > >> >> > > asingh@cloudera.com>
>> > > > >> >> > >> >> > > > wrote:
>> > > > >> >> > >> >> > > > > >> > > Parth,
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > > Thanks again for the awesome write up.
>> > > > Following
>> > > > >> our
>> > > > >> >> > >> >> discussion
>> > > > >> >> > >> >> > > > from the
>> > > > >> >> > >> >> > > > > >> > > JIRA, I think it will be easier to
>> compare
>> > > > >> various
>> > > > >> >> > >> >> alternatives
>> > > > >> >> > >> >> > > > if they
>> > > > >> >> > >> >> > > > > >> > are
>> > > > >> >> > >> >> > > > > >> > > listed together. I am stating below a
>> few
>> > > > >> >> > alternatives along
>> > > > >> >> > >> >> > > with
>> > > > >> >> > >> >> > > > a the
>> > > > >> >> > >> >> > > > > >> > > current proposal.
>> > > > >> >> > >> >> > > > > >> > > (Current proposal) Store Delegation
>> Token,
>> > > DT,
>> > > > >> on ZK.
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a
>> broker.
>> > > > >> >> > >> >> > > > > >> > >    2. Once a client is authenticated,
>> it
>> > > will
>> > > > >> make a
>> > > > >> >> > broker
>> > > > >> >> > >> >> side
>> > > > >> >> > >> >> > > > call to
>> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
>> > > > >> >> > >> >> > > > > >> > >    3. The broker generates a shared
>> secret
>> > > > based
>> > > > >> on
>> > > > >> >> > >> >> > > HMAC-SHA256(a
>> > > > >> >> > >> >> > > > > >> > >    Password/Secret shared between all
>> > > brokers,
>> > > > >> >> > randomly
>> > > > >> >> > >> >> > > generated
>> > > > >> >> > >> >> > > > > >> > tokenId).
>> > > > >> >> > >> >> > > > > >> > >    4. Broker stores this token in its
>> in
>> > > > memory
>> > > > >> cache.
>> > > > >> >> > >> >> Broker
>> > > > >> >> > >> >> > > > also stores
>> > > > >> >> > >> >> > > > > >> > >    the DelegationToken without the
>> hmac in
>> > > the
>> > > > >> >> > zookeeper.
>> > > > >> >> > >> >> > > > > >> > >    5. All brokers will have a cache
>> backed
>> > > by
>> > > > >> >> > zookeeper so
>> > > > >> >> > >> >> they
>> > > > >> >> > >> >> > > > will all
>> > > > >> >> > >> >> > > > > >> > >    get notified whenever a new token is
>> > > > >> generated and
>> > > > >> >> > they
>> > > > >> >> > >> >> will
>> > > > >> >> > >> >> > > > update
>> > > > >> >> > >> >> > > > > >> > their
>> > > > >> >> > >> >> > > > > >> > >    local cache whenever token state
>> changes.
>> > > > >> >> > >> >> > > > > >> > >    6. Broker returns the token to
>> Client.
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > >    1. Probable race condition, client
>> tries
>> > > to
>> > > > >> >> > authenticate
>> > > > >> >> > >> >> with
>> > > > >> >> > >> >> > > > a broker
>> > > > >> >> > >> >> > > > > >> > >    that is yet to be updated with the
>> newly
>> > > > >> generated
>> > > > >> >> > DT.
>> > > > >> >> > >> >> This
>> > > > >> >> > >> >> > > can
>> > > > >> >> > >> >> > > > > >> > probably be
>> > > > >> >> > >> >> > > > > >> > >    dealt with making dtRequest block
>> until
>> > > all
>> > > > >> >> > brokers have
>> > > > >> >> > >> >> > > > updated
>> > > > >> >> > >> >> > > > > >> > their DT
>> > > > >> >> > >> >> > > > > >> > >    cache. Zk barrier or similar
>> mechanism
>> > > can
>> > > > be
>> > > > >> used.
>> > > > >> >> > >> >> However,
>> > > > >> >> > >> >> > > > all such
>> > > > >> >> > >> >> > > > > >> > >    mechanisms will increase complexity.
>> > > > >> >> > >> >> > > > > >> > >    2. Using a static secret key from
>> config
>> > > > >> file. Will
>> > > > >> >> > >> >> require
>> > > > >> >> > >> >> > > yet
>> > > > >> >> > >> >> > > > > >> > another
>> > > > >> >> > >> >> > > > > >> > >    config and uses a static secret
>> key. It
>> > > is
>> > > > >> advised
>> > > > >> >> > to
>> > > > >> >> > >> >> rotate
>> > > > >> >> > >> >> > > > secret
>> > > > >> >> > >> >> > > > > >> > keys
>> > > > >> >> > >> >> > > > > >> > >    periodically. This can be avoided
>> with
>> > > > >> controller
>> > > > >> >> > >> >> generating
>> > > > >> >> > >> >> > > > > >> > secretKey and
>> > > > >> >> > >> >> > > > > >> > >    passing to brokers periodically.
>> However,
>> > > > >> this will
>> > > > >> >> > >> >> require
>> > > > >> >> > >> >> > > > brokers to
>> > > > >> >> > >> >> > > > > >> > >    maintain certain counts of
>> secretKeys.
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > > (Alternative 1) Have controller
>> generate
>> > > > >> delegation
>> > > > >> >> > token.
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a
>> broker.
>> > > > >> >> > >> >> > > > > >> > >    2. Once a client is authenticated,
>> it
>> > > will
>> > > > >> make a
>> > > > >> >> > broker
>> > > > >> >> > >> >> side
>> > > > >> >> > >> >> > > > call to
>> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
>> > > > >> >> > >> >> > > > > >> > >    3. Broker forwards the request to
>> > > > controller.
>> > > > >> >> > >> >> > > > > >> > >    4. Controller generates a DT and
>> > > broadcasts
>> > > > >> to all
>> > > > >> >> > >> >> brokers.
>> > > > >> >> > >> >> > > > > >> > >    5. Broker stores this token in its
>> memory
>> > > > >> cache.
>> > > > >> >> > >> >> > > > > >> > >    6. Controller responds to broker’s
>> DT
>> > > req.
>> > > > >> >> > >> >> > > > > >> > >    7. Broker returns the token to
>> Client.
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > >    1. We will have to add new APIs to
>> > > support
>> > > > >> >> > controller
>> > > > >> >> > >> >> pushing
>> > > > >> >> > >> >> > > > tokens
>> > > > >> >> > >> >> > > > > >> > to
>> > > > >> >> > >> >> > > > > >> > >    brokers on top of the minimal APIs
>> that
>> > > are
>> > > > >> >> > currently
>> > > > >> >> > >> >> > > proposed.
>> > > > >> >> > >> >> > > > > >> > >    2. We will also have to add APIs to
>> > > support
>> > > > >> the
>> > > > >> >> > >> >> bootstrapping
>> > > > >> >> > >> >> > > > case,
>> > > > >> >> > >> >> > > > > >> > i.e,
>> > > > >> >> > >> >> > > > > >> > >    when a new broker comes up it will
>> have
>> > > to
>> > > > >> get all
>> > > > >> >> > >> >> delegation
>> > > > >> >> > >> >> > > > tokens
>> > > > >> >> > >> >> > > > > >> > from
>> > > > >> >> > >> >> > > > > >> > >    the controller.
>> > > > >> >> > >> >> > > > > >> > >    3. In catastrophic failures where
>> all
>> > > > brokers
>> > > > >> go
>> > > > >> >> > down,
>> > > > >> >> > >> >> the
>> > > > >> >> > >> >> > > > tokens will
>> > > > >> >> > >> >> > > > > >> > >    be lost even if servers are
>> restarted as
>> > > > >> tokens
>> > > > >> >> > are not
>> > > > >> >> > >> >> > > > persisted
>> > > > >> >> > >> >> > > > > >> > anywhere.
>> > > > >> >> > >> >> > > > > >> > >    If this happens, then there are more
>> > > > important
>> > > > >> >> > things to
>> > > > >> >> > >> >> > > worry
>> > > > >> >> > >> >> > > > about
>> > > > >> >> > >> >> > > > > >> > and
>> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
>> re-authenticate.
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > > (Alternative 2) Do not distribute DT to
>> > > other
>> > > > >> brokers
>> > > > >> >> > at
>> > > > >> >> > >> >> all.
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a
>> broker.
>> > > > >> >> > >> >> > > > > >> > >    2. Once a client is authenticated,
>> it
>> > > will
>> > > > >> make a
>> > > > >> >> > broker
>> > > > >> >> > >> >> side
>> > > > >> >> > >> >> > > > call to
>> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
>> > > > >> >> > >> >> > > > > >> > >    3. The broker generates DT of form,
>> > > [hmac +
>> > > > >> (owner,
>> > > > >> >> > >> >> renewer,
>> > > > >> >> > >> >> > > > > >> > >    maxLifeTime, id, hmac,
>> expirationTime)]
>> > > and
>> > > > >> passes
>> > > > >> >> > back
>> > > > >> >> > >> >> this
>> > > > >> >> > >> >> > > > DT to
>> > > > >> >> > >> >> > > > > >> > client.
>> > > > >> >> > >> >> > > > > >> > >    hmac is generated via
>> {HMAC-SHA256(owner,
>> > > > >> renewer,
>> > > > >> >> > >> >> > > > maxLifeTime, id,
>> > > > >> >> > >> >> > > > > >> > hmac,
>> > > > >> >> > >> >> > > > > >> > >    expirationTime) using SecretKey}.
>> Note
>> > > that
>> > > > >> all
>> > > > >> >> > brokers
>> > > > >> >> > >> >> have
>> > > > >> >> > >> >> > > > this
>> > > > >> >> > >> >> > > > > >> > SecretKey.
>> > > > >> >> > >> >> > > > > >> > >    4. Client then goes to any broker
>> and to
>> > > > >> >> > authenticate
>> > > > >> >> > >> >> sends
>> > > > >> >> > >> >> > > > the DT.
>> > > > >> >> > >> >> > > > > >> > >    Broker recalculates hmac using
>> (owner,
>> > > > >> renewer,
>> > > > >> >> > >> >> maxLifeTime,
>> > > > >> >> > >> >> > > > id, hmac,
>> > > > >> >> > >> >> > > > > >> > >    expirationTime) info from DT and its
>> > > > >> SecretKey. If
>> > > > >> >> > it
>> > > > >> >> > >> >> matches
>> > > > >> >> > >> >> > > > with
>> > > > >> >> > >> >> > > > > >> > hmac of
>> > > > >> >> > >> >> > > > > >> > >    DT, client is authenticated. Yes,
>> it will
>> > > > do
>> > > > >> other
>> > > > >> >> > >> >> obvious
>> > > > >> >> > >> >> > > > checks of
>> > > > >> >> > >> >> > > > > >> > >    timestamp expiry and such.
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > > Note that secret key will be generated
>> by
>> > > > >> controller
>> > > > >> >> > and
>> > > > >> >> > >> >> passed
>> > > > >> >> > >> >> > > to
>> > > > >> >> > >> >> > > > > >> > brokers
>> > > > >> >> > >> >> > > > > >> > > periodically.
>> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > >    1. How to delete a DT? Yes, that is
>> a
>> > > > downside
>> > > > >> >> > here.
>> > > > >> >> > >> >> However,
>> > > > >> >> > >> >> > > > this can
>> > > > >> >> > >> >> > > > > >> > >    be handled with brokers maintaining
>> a
>> > > > >> blacklist of
>> > > > >> >> > DTs,
>> > > > >> >> > >> >> DTs
>> > > > >> >> > >> >> > > > from this
>> > > > >> >> > >> >> > > > > >> > list
>> > > > >> >> > >> >> > > > > >> > >    can be removed after expiry.
>> > > > >> >> > >> >> > > > > >> > >    2. In catastrophic failures where
>> all
>> > > > brokers
>> > > > >> go
>> > > > >> >> > down,
>> > > > >> >> > >> >> the
>> > > > >> >> > >> >> > > > tokens will
>> > > > >> >> > >> >> > > > > >> > >    be lost even if servers are
>> restarted as
>> > > > >> tokens
>> > > > >> >> > are not
>> > > > >> >> > >> >> > > > persisted
>> > > > >> >> > >> >> > > > > >> > anywhere.
>> > > > >> >> > >> >> > > > > >> > >    If this happens, then there are more
>> > > > important
>> > > > >> >> > things to
>> > > > >> >> > >> >> > > worry
>> > > > >> >> > >> >> > > > about
>> > > > >> >> > >> >> > > > > >> > and
>> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
>> re-authenticate.
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth
>> > > > >> Brahmbhatt <
>> > > > >> >> > >> >> > > > > >> > > pbrahmbhatt@hortonworks.com> wrote:
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > >> Hi,
>> > > > >> >> > >> >> > > > > >> > >>
>> > > > >> >> > >> >> > > > > >> > >> I have filed KIP-48 so we can offer
>> hadoop
>> > > > like
>> > > > >> >> > delegation
>> > > > >> >> > >> >> > > > tokens in
>> > > > >> >> > >> >> > > > > >> > >> kafka. You can review the design
>> > > > >> >> > >> >> > > > > >> > >>
>> > > > >> >> > >> >> > > > > >> >
>> > > > >> >> > >> >> > > >
>> > > > >> >> > >> >> > >
>> > > > >> >> > >> >>
>> > > > >> >> >
>> > > > >>
>> > > >
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
>> > > > >> >> > >> >> > > > > >> > .
>> > > > >> >> > >> >> > > > > >> > >> This KIP depends on KIP-43 and we
>> have also
>> > > > >> >> > discussed an
>> > > > >> >> > >> >> > > > alternative to
>> > > > >> >> > >> >> > > > > >> > >> proposed design here<
>> > > > >> >> > >> >> > > > > >> > >>
>> > > > >> >> > >> >> > > > > >> >
>> > > > >> >> > >> >> > > >
>> > > > >> >> > >> >> > >
>> > > > >> >> > >> >>
>> > > > >> >> >
>> > > > >>
>> > > >
>> > >
>> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
>> > > > >> >> > >> >> > > > > >> > >> >.
>> > > > >> >> > >> >> > > > > >> > >>
>> > > > >> >> > >> >> > > > > >> > >> Thanks
>> > > > >> >> > >> >> > > > > >> > >> Parth
>> > > > >> >> > >> >> > > > > >> > >>
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > > --
>> > > > >> >> > >> >> > > > > >> > >
>> > > > >> >> > >> >> > > > > >> > > Regards,
>> > > > >> >> > >> >> > > > > >> > > Ashish
>> > > > >> >> > >> >> > > > > >> >
>> > > > >> >> > >> >> > > >
>> > > > >> >> > >> >> > >
>> > > > >> >> > >> >> > >
>> > > > >> >> > >> >> > >
>> > > > >> >> > >> >> > > --
>> > > > >> >> > >> >> > >
>> > > > >> >> > >> >> > > Regards,
>> > > > >> >> > >> >> > > Ashish
>> > > > >> >> > >> >> > >
>> > > > >> >> > >> >>
>> > > > >> >> >
>> > > > >>
>> > > >
>> > >
>>
>>
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Jun Rao <ju...@confluent.io>]

Hi, Harsha,

Just sent out a KIP meeting invite. We can discuss this in the meeting
tomorrow.

Thanks,

Jun

On Thu, May 19, 2016 at 8:47 AM, Harsha <ka...@harsha.io> wrote:

> Hi All,
>            Can we have a KIP meeting around this. The KIP is up for
>            sometime and if there are any questions lets quickly hash out
>            details.
>
> Thanks,
> Harsha
>
> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt wrote:
> > That is what the hadoop echo system uses so no good reason really. We
> > could
> > change it to whatever is the newest recommended standard is.
> >
> > Thanks
> > Parth
> >
> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Parth,
> > >
> > > Thanks for the KIP. I only started reviewing this and may have
> additional
> > > questions later. The immediate question that came to mind is our
> choice of
> > > "DIGEST-MD5" even though it's marked as OBSOLETE in the IANA Registry
> of
> > > SASL mechanisms and the original RFC (2831) has been moved to Historic
> > > status:
> > >
> > > https://tools.ietf.org/html/rfc6331
> > > http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> > >
> > > What is the reasoning behind that choice?
> > >
> > > Thanks,
> > > Ismael
> > >
> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <gw...@confluent.io>
> wrote:
> > >
> > > > Also comments inline :)
> > > >
> > > > > * I want to emphasize that even though delegation tokens are a
> Hadoop
> > > > > innovation, I feel very strongly about not adding dependency on
> Hadoop
> > > > > when implementing delegation tokens for Kafka. The KIP doesn't
> imply
> > > > > such dependency, but if you can clarify...
> > > > >
> > > > >
> > > > > *No hadoop dependency.*
> > > >
> > > > Yay! Just add this to the KIP so no one will read the KIP and panic
> > > > three weeks before the next release...
> > > >
> > > > > * Can we get delegation token at any time after authenticating?
> only
> > > > > immediately after?
> > > > >
> > > > >
> > > > > *As long as you are authenticated you can get delegation tokens. We
> > > need
> > > > to
> > > > > discuss if a client authenticated using delegation token, can also
> > > > acquire
> > > > > delegation token again or not. Also there is the question of do we
> > > allow
> > > > > anyone to acquire delegation token or we want specific ACLs (I
> think
> > > its
> > > > an
> > > > > overkill.)*
> > > >
> > > > I agree that ACLs is an overkill.
> > > >
> > > > I think we are debating two options: Either require Kerberos auth for
> > > > renewal or require non-owners to renew.
> > > > I *think* the latter is simpler (it basically require a "job master"
> > > > to take responsibility for the renewal, it will have its own identity
> > > > anyway and I think this is the correct design pattern anyway. For
> > > > storm, I'd expect Nimbus to coordinate renewals?), but it is hard to
> > > > debate simplicity without looking at the code changes required. If
> you
> > > > have a draft of how the "require Kerberos" will look in Kafka code,
> > > > I'll be happy to take a look.
> > > >
> > > > > * My understanding is that tokens will propagate via ZK but without
> > > > > additional changes to UpdateMetadata protocol, correct? Clients
> > > > > currently don't retry on SASL auth failure (IIRC), but since the
> > > > > tokens propagate between brokers asynch, we will need to retry a
> bit
> > > > > to avoid clients failing auth due to timing issues.
> > > > >
> > > > > *I am considering 2 alternatives right now. The current documented
> > > > approach
> > > > > is zookeeper based and it does not require any changes to
> > > UpdateMetadata
> > > > > protocol. An alternative approach can remove zookeeper dependency
> as
> > > well
> > > > > but we can discuss that in KIP discussion call.*
> > > >
> > > > Oooh! Sounds interesting. Do you want to ping Jun to arrange a call?
> > > >
> > > > > * I liked Ashish's suggestion of having just the controller issue
> the
> > > > > delegation tokens, to avoid syncing a shared secret. Not sure if we
> > > > > want to continue the discussion here or on the wiki. I think that
> we
> > > > > can decouple the problem of "token distribution" from "shared
> secret
> > > > > distribution" and use the controller as the only token generator to
> > > > > solve the second issue, while still using ZK async to distribute
> > > > > tokens.
> > > > >
> > > > >
> > > > > *As mentioned in the previous Email I am fine with that approach as
> > > long
> > > > as
> > > > > we agree that the extra complexity of adding/updating APIS adds
> enough
> > > > > value. The advantage with the controller approach is secret
> rotation
> > > can
> > > > be
> > > > > automated,frequent and would not require deployment. *
> > > >
> > > > Can you detail the extra complexity (or point me to the email I
> > > > missed?) - which Apis are required?
> > > > As far as I can tell, clients can already find the controller from
> > > > metadata. I'm a bit more concerned about controller load.
> > > >
> > > > >
> > > > > * While I like the idea of forcing kerberos auth for renwal, I
> think
> > > > > it mixes the transport layer the the request content in a pretty
> ugly
> > > > > way. Perhaps limiting renewer to non-owner is better.
> > > > >
> > > > > *I feel this is a necessary evil. While this will make the kafka
> code
> > > > > pretty straight forward , forcing  renewer to non-owner pushes the
> code
> > > > > ugliness to client and makes it even harder to integrate.  *
> > > >
> > > > As mentioned before, I don't think the "renewal by other" approach is
> > > > that ugly for the clients we expect to use delegation tokens since
> > > > they will have an app-master of some sort who requested the token to
> > > > begin with.
> > > >
> > > > >
> > > > > The response for my question on how multiple identities will be
> > > > > handled wasn't super clear to me - AFAIK, we don't authenticate
> each
> > > > > request, we authenticate connections.
> > > > >
> > > > > *We authenticate connections, and only when they are being
> established.
> > > > Let
> > > > > me try to phrase this as a question, in absence of delegation
> tokens if
> > > > we
> > > > > had to support the use case using user TGT's how would we do it? My
> > > point
> > > > > was it would be no different with delegation tokens. The use case
> you
> > > are
> > > > > describing seems more like impersonation.*
> > > >
> > > > Yeah, I thought that one of the things that delegation tokens
> handled.
> > > > Maybe I got it wrong :)
> > > >
> > > > Thanks for the detailed answers.
> > > >
> > > > Gwen
> > > >
> > > >
> > > > > Thanks
> > > > > Parth
> > > > >
> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <gw...@confluent.io>
> > > > wrote:
> > > > >
> > > > >> Hi Parth and Harsha,
> > > > >>
> > > > >> Few more comments:
> > > > >>
> > > > >> * The API / RequestResponse section doesn't seem to have good
> > > > >> description of the changes to the Kafka Protocol. Sounds like you
> are
> > > > >> proposing new DelegationTokenRequest and RenewTokenRequest (and
> > > > >> matching responses), without detailing the contents of the
> requests
> > > > >> and responses? Or rather, you show the class interface, but not
> the
> > > > >> underlying protocol. This could be seen as an implementation
> detail,
> > > > >> but since the binary protocol is what we provide to non-Java
> clients,
> > > > >> we need to show the changes there.
> > > > >>
> > > > >> * getDelegationToken sounds like delegationTokenRequestHandler?
> Is it
> > > > >> planned to be part of KafkaApi? or Client? Its unclear...
> > > > >>
> > > > >> * I want to emphasize that even though delegation tokens are a
> Hadoop
> > > > >> innovation, I feel very strongly about not adding dependency on
> Hadoop
> > > > >> when implementing delegation tokens for Kafka. The KIP doesn't
> imply
> > > > >> such dependency, but if you can clarify...
> > > > >>
> > > > >> * Can we get delegation token at any time after authenticating?
> only
> > > > >> immediately after?
> > > > >>
> > > > >> * My understanding is that tokens will propagate via ZK but
> without
> > > > >> additional changes to UpdateMetadata protocol, correct? Clients
> > > > >> currently don't retry on SASL auth failure (IIRC), but since the
> > > > >> tokens propagate between brokers asynch, we will need to retry a
> bit
> > > > >> to avoid clients failing auth due to timing issues.
> > > > >>
> > > > >> * Strongly agreeing on clients not touching ZK directly :)
> > > > >>
> > > > >> * I liked Ashish's suggestion of having just the controller issue
> the
> > > > >> delegation tokens, to avoid syncing a shared secret. Not sure if
> we
> > > > >> want to continue the discussion here or on the wiki. I think that
> we
> > > > >> can decouple the problem of "token distribution" from "shared
> secret
> > > > >> distribution" and use the controller as the only token generator
> to
> > > > >> solve the second issue, while still using ZK async to distribute
> > > > >> tokens.
> > > > >>
> > > > >> * I am also uncomfortable with infinite lifetime of tokens (and
> hoped
> > > > >> to hear from others in the community) - but having the option and
> > > > >> documenting it as less secure, allows users to configure their
> system
> > > > >> as the wish.
> > > > >>
> > > > >> * While I like the idea of forcing kerberos auth for renwal, I
> think
> > > > >> it mixes the transport layer the the request content in a pretty
> ugly
> > > > >> way. Perhaps limiting renewer to non-owner is better.
> > > > >>
> > > > >> Things I'd still like to see:
> > > > >>
> > > > >> * More detailed explanation on what we plan to do for the java
> clients
> > > > >> specifically - new configuration? new APIs?
> > > > >> The response for my question on how multiple identities will be
> > > > >> handled wasn't super clear to me - AFAIK, we don't authenticate
> each
> > > > >> request, we authenticate connections.
> > > > >>
> > > > >> * Alternatives: Delegation tokens are only used in the Hadoop
> > > > >> ecosystem. I'm wondering if there are alternatives in other
> ecosystems
> > > > >> (Mesos? Tachyon? Cassandra?) and whether there are some advantages
> > > > >> there.
> > > > >>
> > > > >> Gwen
> > > > >>
> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <ka...@harsha.io> wrote:
> > > > >> > Hi Gwen,
> > > > >> >            Can you look at Parth's last reply. Does it answer
> your
> > > > >> >            concerns.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Harsha
> > > > >> >
> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt wrote:
> > > > >> >> Thanks for reviewing Gwen. The wiki already has details on
> token
> > > > >> >> expiration
> > > > >> >> under token acquisition process
> > > > >> >> <
> > > > >>
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> > > > >> >.
> > > > >> >> Current proposal is that tokens will expire based on a server
> side
> > > > >> >> configuration (default 24 hours) unless renewed. Renewal is
> only
> > > > allowed
> > > > >> >> until the max life time of token. Alternatively we could also
> make
> > > > that
> > > > >> >> an
> > > > >> >> optional param and the server side default can serve as the
> upper
> > > > bound.
> > > > >> >>
> > > > >> >> To your second point it will be done exactly the same way we
> would
> > > > >> >> support
> > > > >> >> multiple keytabs. The calling client will have to put the
> tokens it
> > > > >> wants
> > > > >> >> to use in the subject instance and call produce/consume inside
> > > > >> >> subject.doas. Each caller will have to keep track of its own
> > > > subject. I
> > > > >> >> will have to look at the code to see if we support this feature
> > > right
> > > > >> now
> > > > >> >> but my understanding is delegation token shouldn't need any
> special
> > > > >> >> treatment as its just another type of Credential in the
> subject.
> > > > >> >>
> > > > >> >> I would also like to know what is your opinion about infinite
> > > renewal
> > > > >> (my
> > > > >> >> recommendation is to not support this), tokens renewing them
> > > self(my
> > > > >> >> recommendation is to not support this) and most importantly
> your
> > > > choice
> > > > >> >> between the alternatives listed on this thread
> > > > >> >> <
> > > > >>
> > > >
> > >
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> > > > >> >
> > > > >> >> ( I am leaning towards the alternative-2 minus controller
> > > > distributing
> > > > >> >> secret). Thanks again for reviewing.
> > > > >> >>
> > > > >> >> Thanks
> > > > >> >> Parth
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <
> gwen@confluent.io>
> > > > wrote:
> > > > >> >>
> > > > >> >> > Harsha,
> > > > >> >> >
> > > > >> >> > I was thinking of the Rest Proxy. I didn't see your design
> yet,
> > > > but in
> > > > >> >> > our proxy, we have a set of producers, which will serve
> multiple
> > > > users
> > > > >> >> > going through the proxy. Since these users will have
> different
> > > > >> >> > privileges, they'll need to authenticate separately, and
> can't
> > > > share a
> > > > >> >> > token.
> > > > >> >> >
> > > > >> >> > Am I missing anything?
> > > > >> >> >
> > > > >> >> > Gwen
> > > > >> >> >
> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io>
> wrote:
> > > > >> >> > > Gwen,
> > > > >> >> > >            On your second point. Can you describe a usecase
> > > where
> > > > >> >> > >            mutliple clients ended up sharing a producer and
> > > even
> > > > if
> > > > >> they
> > > > >> >> > >            do why can't they not use single token that
> producer
> > > > >> >> > >            captures. Why would we need multiple clients
> with
> > > > >> different
> > > > >> >> > >            tokens sharing a single instance of producer.
> Also
> > > in
> > > > >> this
> > > > >> >> > >            case other clients have access all the tokens
> no?
> > > > >> >> > >
> > > > >> >> > > Thanks,
> > > > >> >> > > Harsha
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote:
> > > > >> >> > >> Sorry for the delay:
> > > > >> >> > >>
> > > > >> >> > >> Two questions that we didn't see in the wiki:
> > > > >> >> > >> 1. Is there an expiration for delegation tokens? Renewal?
> How
> > > > do we
> > > > >> >> > >> revoke them?
> > > > >> >> > >> 2. If we want to use delegation tokens for "do-as" (say,
> > > submit
> > > > >> Storm
> > > > >> >> > >> job as my user), we will need a producer for every job (we
> > > can't
> > > > >> share
> > > > >> >> > >> them between multiple jobs running on same node), since we
> > > only
> > > > >> >> > >> authenticate when connecting. Is there a plan to change
> this
> > > for
> > > > >> >> > >> delegation tokens, in order to allow multiple users with
> > > > different
> > > > >> >> > >> tokens to share a client?
> > > > >> >> > >>
> > > > >> >> > >> Gwen
> > > > >> >> > >>
> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
> > > > >> >> > >> <br...@gmail.com> wrote:
> > > > >> >> > >> > Bumping this up one more time, can other committers
> review?
> > > > >> >> > >> >
> > > > >> >> > >> > Thanks
> > > > >> >> > >> > Parth
> > > > >> >> > >> >
> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <
> kafka@harsha.io>
> > > > wrote:
> > > > >> >> > >> >
> > > > >> >> > >> >> Parth,
> > > > >> >> > >> >>           Overall current design looks good to me. I
> am +1
> > > on
> > > > >> the
> > > > >> >> > KIP.
> > > > >> >> > >> >>
> > > > >> >> > >> >> Gwen , Jun can you review this as well.
> > > > >> >> > >> >>
> > > > >> >> > >> >> -Harsha
> > > > >> >> > >> >>
> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt
> wrote:
> > > > >> >> > >> >> > Thanks for review Jitendra.
> > > > >> >> > >> >> >
> > > > >> >> > >> >> > I don't like the idea of infinite lifetime but I see
> the
> > > > >> Streaming
> > > > >> >> > use
> > > > >> >> > >> >> > case. Even for Streaming use case I was hoping there
> will
> > > > be
> > > > >> some
> > > > >> >> > notion
> > > > >> >> > >> >> > of
> > > > >> >> > >> >> > master/driver that can get new delegation tokens at
> fixed
> > > > >> interval
> > > > >> >> > and
> > > > >> >> > >> >> > distribute to workers. If that is not the case for
> we can
> > > > >> discuss
> > > > >> >> > >> >> > delegation tokens renewing them self and the security
> > > > >> implications
> > > > >> >> > of the
> > > > >> >> > >> >> > same.
> > > > >> >> > >> >> >
> > > > >> >> > >> >> > I did not want clients to fetch tokens from
> zookeeper,
> > > > >> overall I
> > > > >> >> > think
> > > > >> >> > >> >> > its
> > > > >> >> > >> >> > better if clients don't rely on our metadata store
> and I
> > > > >> think we
> > > > >> >> > are
> > > > >> >> > >> >> > moving in that direction with all the KIP-4
> improvements.
> > > > I
> > > > >> chose
> > > > >> >> > >> >> > zookeeper as in this case the client will still just
> talk
> > > > to
> > > > >> >> > broker , its
> > > > >> >> > >> >> > the brokers that will use zookeeper which we already
> do
> > > > for a
> > > > >> lot
> > > > >> >> > of
> > > > >> >> > >> >> > other
> > > > >> >> > >> >> > usecases + ease of development + and the ability so
> > > tokens
> > > > >> will
> > > > >> >> > survive
> > > > >> >> > >> >> > even a rolling restart/cluster failure. if a majority
> > > > agrees
> > > > >> the
> > > > >> >> > added
> > > > >> >> > >> >> > complexity to have controller forwarding keys to all
> > > > broker is
> > > > >> >> > justified
> > > > >> >> > >> >> > as
> > > > >> >> > >> >> > it provides tighter security , I am fine with that
> option
> > > > too.
> > > > >> >> > >> >> >
> > > > >> >> > >> >> > Given zookeeper does not support SSL we can not store
> > > > master
> > > > >> keys
> > > > >> >> > in
> > > > >> >> > >> >> > zookeeper as master keys will be exposed on wire. To
> > > > support
> > > > >> >> > rotation
> > > > >> >> > >> >> > without affecting current clients is something I
> need to
> > > > put
> > > > >> more
> > > > >> >> > thought
> > > > >> >> > >> >> > in. My current proposal assumes the rotation will
> > > > invalidate
> > > > >> all
> > > > >> >> > current
> > > > >> >> > >> >> > tokens.
> > > > >> >> > >> >> >
> > > > >> >> > >> >> > I request committers to also review and post their
> > > comments
> > > > >> so we
> > > > >> >> > can
> > > > >> >> > >> >> > make
> > > > >> >> > >> >> > progress on this KIP.
> > > > >> >> > >> >> >
> > > > >> >> > >> >> > Thanks
> > > > >> >> > >> >> > Parth
> > > > >> >> > >> >> >
> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <
> > > > >> asingh@cloudera.com
> > > > >> >> > >
> > > > >> >> > >> >> > wrote:
> > > > >> >> > >> >> >
> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <
> > > > kafka@harsha.io>
> > > > >> >> > wrote:
> > > > >> >> > >> >> > >
> > > > >> >> > >> >> > > > Unifying the two discussion threads on this KIP.
> > > > >> >> > >> >> > > >
> > > > >> >> > >> >> > > > Here is the response from Jitendra
> > > > >> >> > >> >> > > >
> > > > >> >> > >> >> > > > "The need for a large number of clients that are
> > > > running
> > > > >> all
> > > > >> >> > over the
> > > > >> >> > >> >> > > > cluster that authenticate with Kafka brokers, is
> very
> > > > >> similar
> > > > >> >> > to the
> > > > >> >> > >> >> > > > Hadoop use case of large number of tasks running
> > > across
> > > > >> the
> > > > >> >> > cluster
> > > > >> >> > >> >> that
> > > > >> >> > >> >> > > > need authentication to Hdfs Namenode. Therefore,
> the
> > > > >> >> > delegation token
> > > > >> >> > >> >> > > > approach does seem like a good fit for this use
> case
> > > > as we
> > > > >> >> > have seen
> > > > >> >> > >> >> it
> > > > >> >> > >> >> > > > working at large scale in HDFS and YARN.
> > > > >> >> > >> >> > > >
> > > > >> >> > >> >> > > >   The proposed design is very much inline with
> Hadoop
> > > > >> >> > approach. A few
> > > > >> >> > >> >> > > >   comments:
> > > > >> >> > >> >> > > >
> > > > >> >> > >> >> > > > 1) Why do you guys want to allow infinite
> renewable
> > > > >> lifetime
> > > > >> >> > for a
> > > > >> >> > >> >> > > > token? HDFS restricts a token to a max life time
> > > > (default
> > > > >> 7
> > > > >> >> > days).  A
> > > > >> >> > >> >> > > > token's vulnerability is believed to increase
> with
> > > > time.
> > > > >> >> > >> >> > > >
> > > > >> >> > >> >> > > I agree that having infinite lifetime might not be
> the
> > > > best
> > > > >> idea.
> > > > >> >> > >> >> > >
> > > > >> >> > >> >> > > >
> > > > >> >> > >> >> > > > 2) As I understand the tokens are stored in
> zookeeper
> > > > as
> > > > >> well,
> > > > >> >> > and
> > > > >> >> > >> >> can
> > > > >> >> > >> >> > > > be updated there. This is clever as it can allow
> > > > >> replacing the
> > > > >> >> > tokens
> > > > >> >> > >> >> > > > once they run out of max life time, and clients
> can
> > > > >> download
> > > > >> >> > new
> > > > >> >> > >> >> tokens
> > > > >> >> > >> >> > > > from zookeeper. It shouldn't be a big load on
> > > zookeeper
> > > > >> as a
> > > > >> >> > client
> > > > >> >> > >> >> will
> > > > >> >> > >> >> > > > need to get a new token once in several days. In
> this
> > > > >> approach
> > > > >> >> > you
> > > > >> >> > >> >> don't
> > > > >> >> > >> >> > > > need infinite lifetime on the token even for long
> > > > running
> > > > >> >> > clients.
> > > > >> >> > >> >> > > >
> > > > >> >> > >> >> > > > 3) The token password are generated using a
> master
> > > key.
> > > > >> The
> > > > >> >> > master
> > > > >> >> > >> >> key
> > > > >> >> > >> >> > > > should also be periodically changed. In Hadoop,
> the
> > > > >> default
> > > > >> >> > renewal
> > > > >> >> > >> >> > > > period is 1 day.?
> > > > >> >> > >> >> > > >
> > > > >> >> > >> >> > > IIUC, this will require brokers maintaining a list
> of X
> > > > most
> > > > >> >> > recent
> > > > >> >> > >> >> master
> > > > >> >> > >> >> > > keys. This list will have to be persisted
> somewhere, as
> > > > if a
> > > > >> >> > broker
> > > > >> >> > >> >> goes
> > > > >> >> > >> >> > > down it will have to get that list again and
> storing
> > > > master
> > > > >> keys
> > > > >> >> > on ZK
> > > > >> >> > >> >> is
> > > > >> >> > >> >> > > not the best idea. However, if a broker goes down
> then
> > > we
> > > > >> have
> > > > >> >> > much
> > > > >> >> > >> >> bigger
> > > > >> >> > >> >> > > issue to deal with and client can always
> > > re-authenticate
> > > > is
> > > > >> such
> > > > >> >> > >> >> events.
> > > > >> >> > >> >> > >
> > > > >> >> > >> >> > > Did you happen to take a look at other alternatives
> > > this
> > > > >> list has
> > > > >> >> > >> >> > > suggested?
> > > > >> >> > >> >> > >
> > > > >> >> > >> >> > > >
> > > > >> >> > >> >> > > > Thanks for a thorough proposal, great work!"
> > > > >> >> > >> >> > > >
> > > > >> >> > >> >> > > >
> > > > >> >> > >> >> > > >
> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira
> wrote:
> > > > >> >> > >> >> > > > > Makes sense to me. Thanks!
> > > > >> >> > >> >> > > > >
> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <
> > > > kafka@harsha.io
> > > > >> >
> > > > >> >> > wrote:
> > > > >> >> > >> >> > > > > > It doesn't need any release vehicle but
> still the
> > > > >> work can
> > > > >> >> > move
> > > > >> >> > >> >> > > > forward.
> > > > >> >> > >> >> > > > > > If anyone is interested in the KIP please do
> the
> > > > >> review and
> > > > >> >> > >> >> provide
> > > > >> >> > >> >> > > the
> > > > >> >> > >> >> > > > > > comments.
> > > > >> >> > >> >> > > > > >
> > > > >> >> > >> >> > > > > > -Harsha
> > > > >> >> > >> >> > > > > >
> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma
> > > > wrote:
> > > > >> >> > >> >> > > > > >> I agree that it would be good to have more
> time
> > > to
> > > > >> review
> > > > >> >> > and
> > > > >> >> > >> >> > > discuss
> > > > >> >> > >> >> > > > > >> KIP-48.
> > > > >> >> > >> >> > > > > >>
> > > > >> >> > >> >> > > > > >> Ismael
> > > > >> >> > >> >> > > > > >>
> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen
> Shapira <
> > > > >> >> > >> >> gwen@confluent.io>
> > > > >> >> > >> >> > > > wrote:
> > > > >> >> > >> >> > > > > >>
> > > > >> >> > >> >> > > > > >> > Hi Team,
> > > > >> >> > >> >> > > > > >> >
> > > > >> >> > >> >> > > > > >> > Since KIP-48 depends on KIP-43, which is
> > > > already a
> > > > >> bit
> > > > >> >> > of a
> > > > >> >> > >> >> risk
> > > > >> >> > >> >> > > for
> > > > >> >> > >> >> > > > > >> > the next release - any chance we can delay
> > > > >> delegation
> > > > >> >> > tokens
> > > > >> >> > >> >> to
> > > > >> >> > >> >> > > > Kafka
> > > > >> >> > >> >> > > > > >> > 0.10.1?
> > > > >> >> > >> >> > > > > >> > With the community working on a release
> every
> > > 3
> > > > >> month,
> > > > >> >> > this
> > > > >> >> > >> >> is not
> > > > >> >> > >> >> > > > a huge
> > > > >> >> > >> >> > > > > >> > delay.
> > > > >> >> > >> >> > > > > >> >
> > > > >> >> > >> >> > > > > >> > Gwen
> > > > >> >> > >> >> > > > > >> >
> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish
> Singh
> > > <
> > > > >> >> > >> >> > > asingh@cloudera.com>
> > > > >> >> > >> >> > > > wrote:
> > > > >> >> > >> >> > > > > >> > > Parth,
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > > Thanks again for the awesome write up.
> > > > Following
> > > > >> our
> > > > >> >> > >> >> discussion
> > > > >> >> > >> >> > > > from the
> > > > >> >> > >> >> > > > > >> > > JIRA, I think it will be easier to
> compare
> > > > >> various
> > > > >> >> > >> >> alternatives
> > > > >> >> > >> >> > > > if they
> > > > >> >> > >> >> > > > > >> > are
> > > > >> >> > >> >> > > > > >> > > listed together. I am stating below a
> few
> > > > >> >> > alternatives along
> > > > >> >> > >> >> > > with
> > > > >> >> > >> >> > > > a the
> > > > >> >> > >> >> > > > > >> > > current proposal.
> > > > >> >> > >> >> > > > > >> > > (Current proposal) Store Delegation
> Token,
> > > DT,
> > > > >> on ZK.
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a
> broker.
> > > > >> >> > >> >> > > > > >> > >    2. Once a client is authenticated, it
> > > will
> > > > >> make a
> > > > >> >> > broker
> > > > >> >> > >> >> side
> > > > >> >> > >> >> > > > call to
> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > > > >> >> > >> >> > > > > >> > >    3. The broker generates a shared
> secret
> > > > based
> > > > >> on
> > > > >> >> > >> >> > > HMAC-SHA256(a
> > > > >> >> > >> >> > > > > >> > >    Password/Secret shared between all
> > > brokers,
> > > > >> >> > randomly
> > > > >> >> > >> >> > > generated
> > > > >> >> > >> >> > > > > >> > tokenId).
> > > > >> >> > >> >> > > > > >> > >    4. Broker stores this token in its in
> > > > memory
> > > > >> cache.
> > > > >> >> > >> >> Broker
> > > > >> >> > >> >> > > > also stores
> > > > >> >> > >> >> > > > > >> > >    the DelegationToken without the hmac
> in
> > > the
> > > > >> >> > zookeeper.
> > > > >> >> > >> >> > > > > >> > >    5. All brokers will have a cache
> backed
> > > by
> > > > >> >> > zookeeper so
> > > > >> >> > >> >> they
> > > > >> >> > >> >> > > > will all
> > > > >> >> > >> >> > > > > >> > >    get notified whenever a new token is
> > > > >> generated and
> > > > >> >> > they
> > > > >> >> > >> >> will
> > > > >> >> > >> >> > > > update
> > > > >> >> > >> >> > > > > >> > their
> > > > >> >> > >> >> > > > > >> > >    local cache whenever token state
> changes.
> > > > >> >> > >> >> > > > > >> > >    6. Broker returns the token to
> Client.
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > >    1. Probable race condition, client
> tries
> > > to
> > > > >> >> > authenticate
> > > > >> >> > >> >> with
> > > > >> >> > >> >> > > > a broker
> > > > >> >> > >> >> > > > > >> > >    that is yet to be updated with the
> newly
> > > > >> generated
> > > > >> >> > DT.
> > > > >> >> > >> >> This
> > > > >> >> > >> >> > > can
> > > > >> >> > >> >> > > > > >> > probably be
> > > > >> >> > >> >> > > > > >> > >    dealt with making dtRequest block
> until
> > > all
> > > > >> >> > brokers have
> > > > >> >> > >> >> > > > updated
> > > > >> >> > >> >> > > > > >> > their DT
> > > > >> >> > >> >> > > > > >> > >    cache. Zk barrier or similar
> mechanism
> > > can
> > > > be
> > > > >> used.
> > > > >> >> > >> >> However,
> > > > >> >> > >> >> > > > all such
> > > > >> >> > >> >> > > > > >> > >    mechanisms will increase complexity.
> > > > >> >> > >> >> > > > > >> > >    2. Using a static secret key from
> config
> > > > >> file. Will
> > > > >> >> > >> >> require
> > > > >> >> > >> >> > > yet
> > > > >> >> > >> >> > > > > >> > another
> > > > >> >> > >> >> > > > > >> > >    config and uses a static secret key.
> It
> > > is
> > > > >> advised
> > > > >> >> > to
> > > > >> >> > >> >> rotate
> > > > >> >> > >> >> > > > secret
> > > > >> >> > >> >> > > > > >> > keys
> > > > >> >> > >> >> > > > > >> > >    periodically. This can be avoided
> with
> > > > >> controller
> > > > >> >> > >> >> generating
> > > > >> >> > >> >> > > > > >> > secretKey and
> > > > >> >> > >> >> > > > > >> > >    passing to brokers periodically.
> However,
> > > > >> this will
> > > > >> >> > >> >> require
> > > > >> >> > >> >> > > > brokers to
> > > > >> >> > >> >> > > > > >> > >    maintain certain counts of
> secretKeys.
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > > (Alternative 1) Have controller generate
> > > > >> delegation
> > > > >> >> > token.
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a
> broker.
> > > > >> >> > >> >> > > > > >> > >    2. Once a client is authenticated, it
> > > will
> > > > >> make a
> > > > >> >> > broker
> > > > >> >> > >> >> side
> > > > >> >> > >> >> > > > call to
> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > > > >> >> > >> >> > > > > >> > >    3. Broker forwards the request to
> > > > controller.
> > > > >> >> > >> >> > > > > >> > >    4. Controller generates a DT and
> > > broadcasts
> > > > >> to all
> > > > >> >> > >> >> brokers.
> > > > >> >> > >> >> > > > > >> > >    5. Broker stores this token in its
> memory
> > > > >> cache.
> > > > >> >> > >> >> > > > > >> > >    6. Controller responds to broker’s DT
> > > req.
> > > > >> >> > >> >> > > > > >> > >    7. Broker returns the token to
> Client.
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > >    1. We will have to add new APIs to
> > > support
> > > > >> >> > controller
> > > > >> >> > >> >> pushing
> > > > >> >> > >> >> > > > tokens
> > > > >> >> > >> >> > > > > >> > to
> > > > >> >> > >> >> > > > > >> > >    brokers on top of the minimal APIs
> that
> > > are
> > > > >> >> > currently
> > > > >> >> > >> >> > > proposed.
> > > > >> >> > >> >> > > > > >> > >    2. We will also have to add APIs to
> > > support
> > > > >> the
> > > > >> >> > >> >> bootstrapping
> > > > >> >> > >> >> > > > case,
> > > > >> >> > >> >> > > > > >> > i.e,
> > > > >> >> > >> >> > > > > >> > >    when a new broker comes up it will
> have
> > > to
> > > > >> get all
> > > > >> >> > >> >> delegation
> > > > >> >> > >> >> > > > tokens
> > > > >> >> > >> >> > > > > >> > from
> > > > >> >> > >> >> > > > > >> > >    the controller.
> > > > >> >> > >> >> > > > > >> > >    3. In catastrophic failures where all
> > > > brokers
> > > > >> go
> > > > >> >> > down,
> > > > >> >> > >> >> the
> > > > >> >> > >> >> > > > tokens will
> > > > >> >> > >> >> > > > > >> > >    be lost even if servers are
> restarted as
> > > > >> tokens
> > > > >> >> > are not
> > > > >> >> > >> >> > > > persisted
> > > > >> >> > >> >> > > > > >> > anywhere.
> > > > >> >> > >> >> > > > > >> > >    If this happens, then there are more
> > > > important
> > > > >> >> > things to
> > > > >> >> > >> >> > > worry
> > > > >> >> > >> >> > > > about
> > > > >> >> > >> >> > > > > >> > and
> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
> re-authenticate.
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > > (Alternative 2) Do not distribute DT to
> > > other
> > > > >> brokers
> > > > >> >> > at
> > > > >> >> > >> >> all.
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a
> broker.
> > > > >> >> > >> >> > > > > >> > >    2. Once a client is authenticated, it
> > > will
> > > > >> make a
> > > > >> >> > broker
> > > > >> >> > >> >> side
> > > > >> >> > >> >> > > > call to
> > > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > > > >> >> > >> >> > > > > >> > >    3. The broker generates DT of form,
> > > [hmac +
> > > > >> (owner,
> > > > >> >> > >> >> renewer,
> > > > >> >> > >> >> > > > > >> > >    maxLifeTime, id, hmac,
> expirationTime)]
> > > and
> > > > >> passes
> > > > >> >> > back
> > > > >> >> > >> >> this
> > > > >> >> > >> >> > > > DT to
> > > > >> >> > >> >> > > > > >> > client.
> > > > >> >> > >> >> > > > > >> > >    hmac is generated via
> {HMAC-SHA256(owner,
> > > > >> renewer,
> > > > >> >> > >> >> > > > maxLifeTime, id,
> > > > >> >> > >> >> > > > > >> > hmac,
> > > > >> >> > >> >> > > > > >> > >    expirationTime) using SecretKey}.
> Note
> > > that
> > > > >> all
> > > > >> >> > brokers
> > > > >> >> > >> >> have
> > > > >> >> > >> >> > > > this
> > > > >> >> > >> >> > > > > >> > SecretKey.
> > > > >> >> > >> >> > > > > >> > >    4. Client then goes to any broker
> and to
> > > > >> >> > authenticate
> > > > >> >> > >> >> sends
> > > > >> >> > >> >> > > > the DT.
> > > > >> >> > >> >> > > > > >> > >    Broker recalculates hmac using
> (owner,
> > > > >> renewer,
> > > > >> >> > >> >> maxLifeTime,
> > > > >> >> > >> >> > > > id, hmac,
> > > > >> >> > >> >> > > > > >> > >    expirationTime) info from DT and its
> > > > >> SecretKey. If
> > > > >> >> > it
> > > > >> >> > >> >> matches
> > > > >> >> > >> >> > > > with
> > > > >> >> > >> >> > > > > >> > hmac of
> > > > >> >> > >> >> > > > > >> > >    DT, client is authenticated. Yes, it
> will
> > > > do
> > > > >> other
> > > > >> >> > >> >> obvious
> > > > >> >> > >> >> > > > checks of
> > > > >> >> > >> >> > > > > >> > >    timestamp expiry and such.
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > > Note that secret key will be generated
> by
> > > > >> controller
> > > > >> >> > and
> > > > >> >> > >> >> passed
> > > > >> >> > >> >> > > to
> > > > >> >> > >> >> > > > > >> > brokers
> > > > >> >> > >> >> > > > > >> > > periodically.
> > > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > >    1. How to delete a DT? Yes, that is a
> > > > downside
> > > > >> >> > here.
> > > > >> >> > >> >> However,
> > > > >> >> > >> >> > > > this can
> > > > >> >> > >> >> > > > > >> > >    be handled with brokers maintaining a
> > > > >> blacklist of
> > > > >> >> > DTs,
> > > > >> >> > >> >> DTs
> > > > >> >> > >> >> > > > from this
> > > > >> >> > >> >> > > > > >> > list
> > > > >> >> > >> >> > > > > >> > >    can be removed after expiry.
> > > > >> >> > >> >> > > > > >> > >    2. In catastrophic failures where all
> > > > brokers
> > > > >> go
> > > > >> >> > down,
> > > > >> >> > >> >> the
> > > > >> >> > >> >> > > > tokens will
> > > > >> >> > >> >> > > > > >> > >    be lost even if servers are
> restarted as
> > > > >> tokens
> > > > >> >> > are not
> > > > >> >> > >> >> > > > persisted
> > > > >> >> > >> >> > > > > >> > anywhere.
> > > > >> >> > >> >> > > > > >> > >    If this happens, then there are more
> > > > important
> > > > >> >> > things to
> > > > >> >> > >> >> > > worry
> > > > >> >> > >> >> > > > about
> > > > >> >> > >> >> > > > > >> > and
> > > > >> >> > >> >> > > > > >> > >    maybe it is better to
> re-authenticate.
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth
> > > > >> Brahmbhatt <
> > > > >> >> > >> >> > > > > >> > > pbrahmbhatt@hortonworks.com> wrote:
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > >> Hi,
> > > > >> >> > >> >> > > > > >> > >>
> > > > >> >> > >> >> > > > > >> > >> I have filed KIP-48 so we can offer
> hadoop
> > > > like
> > > > >> >> > delegation
> > > > >> >> > >> >> > > > tokens in
> > > > >> >> > >> >> > > > > >> > >> kafka. You can review the design
> > > > >> >> > >> >> > > > > >> > >>
> > > > >> >> > >> >> > > > > >> >
> > > > >> >> > >> >> > > >
> > > > >> >> > >> >> > >
> > > > >> >> > >> >>
> > > > >> >> >
> > > > >>
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > > > >> >> > >> >> > > > > >> > .
> > > > >> >> > >> >> > > > > >> > >> This KIP depends on KIP-43 and we have
> also
> > > > >> >> > discussed an
> > > > >> >> > >> >> > > > alternative to
> > > > >> >> > >> >> > > > > >> > >> proposed design here<
> > > > >> >> > >> >> > > > > >> > >>
> > > > >> >> > >> >> > > > > >> >
> > > > >> >> > >> >> > > >
> > > > >> >> > >> >> > >
> > > > >> >> > >> >>
> > > > >> >> >
> > > > >>
> > > >
> > >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > > > >> >> > >> >> > > > > >> > >> >.
> > > > >> >> > >> >> > > > > >> > >>
> > > > >> >> > >> >> > > > > >> > >> Thanks
> > > > >> >> > >> >> > > > > >> > >> Parth
> > > > >> >> > >> >> > > > > >> > >>
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > > --
> > > > >> >> > >> >> > > > > >> > >
> > > > >> >> > >> >> > > > > >> > > Regards,
> > > > >> >> > >> >> > > > > >> > > Ashish
> > > > >> >> > >> >> > > > > >> >
> > > > >> >> > >> >> > > >
> > > > >> >> > >> >> > >
> > > > >> >> > >> >> > >
> > > > >> >> > >> >> > >
> > > > >> >> > >> >> > > --
> > > > >> >> > >> >> > >
> > > > >> >> > >> >> > > Regards,
> > > > >> >> > >> >> > > Ashish
> > > > >> >> > >> >> > >
> > > > >> >> > >> >>
> > > > >> >> >
> > > > >>
> > > >
> > >
>
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Harsha <ka...@harsha.io>]

Hi All,
           Can we have a KIP meeting around this. The KIP is up for
           sometime and if there are any questions lets quickly hash out
           details. 

Thanks,
Harsha

On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt wrote:
> That is what the hadoop echo system uses so no good reason really. We
> could
> change it to whatever is the newest recommended standard is.
> 
> Thanks
> Parth
> 
> On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <is...@juma.me.uk> wrote:
> 
> > Hi Parth,
> >
> > Thanks for the KIP. I only started reviewing this and may have additional
> > questions later. The immediate question that came to mind is our choice of
> > "DIGEST-MD5" even though it's marked as OBSOLETE in the IANA Registry of
> > SASL mechanisms and the original RFC (2831) has been moved to Historic
> > status:
> >
> > https://tools.ietf.org/html/rfc6331
> > http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> >
> > What is the reasoning behind that choice?
> >
> > Thanks,
> > Ismael
> >
> > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <gw...@confluent.io> wrote:
> >
> > > Also comments inline :)
> > >
> > > > * I want to emphasize that even though delegation tokens are a Hadoop
> > > > innovation, I feel very strongly about not adding dependency on Hadoop
> > > > when implementing delegation tokens for Kafka. The KIP doesn't imply
> > > > such dependency, but if you can clarify...
> > > >
> > > >
> > > > *No hadoop dependency.*
> > >
> > > Yay! Just add this to the KIP so no one will read the KIP and panic
> > > three weeks before the next release...
> > >
> > > > * Can we get delegation token at any time after authenticating? only
> > > > immediately after?
> > > >
> > > >
> > > > *As long as you are authenticated you can get delegation tokens. We
> > need
> > > to
> > > > discuss if a client authenticated using delegation token, can also
> > > acquire
> > > > delegation token again or not. Also there is the question of do we
> > allow
> > > > anyone to acquire delegation token or we want specific ACLs (I think
> > its
> > > an
> > > > overkill.)*
> > >
> > > I agree that ACLs is an overkill.
> > >
> > > I think we are debating two options: Either require Kerberos auth for
> > > renewal or require non-owners to renew.
> > > I *think* the latter is simpler (it basically require a "job master"
> > > to take responsibility for the renewal, it will have its own identity
> > > anyway and I think this is the correct design pattern anyway. For
> > > storm, I'd expect Nimbus to coordinate renewals?), but it is hard to
> > > debate simplicity without looking at the code changes required. If you
> > > have a draft of how the "require Kerberos" will look in Kafka code,
> > > I'll be happy to take a look.
> > >
> > > > * My understanding is that tokens will propagate via ZK but without
> > > > additional changes to UpdateMetadata protocol, correct? Clients
> > > > currently don't retry on SASL auth failure (IIRC), but since the
> > > > tokens propagate between brokers asynch, we will need to retry a bit
> > > > to avoid clients failing auth due to timing issues.
> > > >
> > > > *I am considering 2 alternatives right now. The current documented
> > > approach
> > > > is zookeeper based and it does not require any changes to
> > UpdateMetadata
> > > > protocol. An alternative approach can remove zookeeper dependency as
> > well
> > > > but we can discuss that in KIP discussion call.*
> > >
> > > Oooh! Sounds interesting. Do you want to ping Jun to arrange a call?
> > >
> > > > * I liked Ashish's suggestion of having just the controller issue the
> > > > delegation tokens, to avoid syncing a shared secret. Not sure if we
> > > > want to continue the discussion here or on the wiki. I think that we
> > > > can decouple the problem of "token distribution" from "shared secret
> > > > distribution" and use the controller as the only token generator to
> > > > solve the second issue, while still using ZK async to distribute
> > > > tokens.
> > > >
> > > >
> > > > *As mentioned in the previous Email I am fine with that approach as
> > long
> > > as
> > > > we agree that the extra complexity of adding/updating APIS adds enough
> > > > value. The advantage with the controller approach is secret rotation
> > can
> > > be
> > > > automated,frequent and would not require deployment. *
> > >
> > > Can you detail the extra complexity (or point me to the email I
> > > missed?) - which Apis are required?
> > > As far as I can tell, clients can already find the controller from
> > > metadata. I'm a bit more concerned about controller load.
> > >
> > > >
> > > > * While I like the idea of forcing kerberos auth for renwal, I think
> > > > it mixes the transport layer the the request content in a pretty ugly
> > > > way. Perhaps limiting renewer to non-owner is better.
> > > >
> > > > *I feel this is a necessary evil. While this will make the kafka code
> > > > pretty straight forward , forcing  renewer to non-owner pushes the code
> > > > ugliness to client and makes it even harder to integrate.  *
> > >
> > > As mentioned before, I don't think the "renewal by other" approach is
> > > that ugly for the clients we expect to use delegation tokens since
> > > they will have an app-master of some sort who requested the token to
> > > begin with.
> > >
> > > >
> > > > The response for my question on how multiple identities will be
> > > > handled wasn't super clear to me - AFAIK, we don't authenticate each
> > > > request, we authenticate connections.
> > > >
> > > > *We authenticate connections, and only when they are being established.
> > > Let
> > > > me try to phrase this as a question, in absence of delegation tokens if
> > > we
> > > > had to support the use case using user TGT's how would we do it? My
> > point
> > > > was it would be no different with delegation tokens. The use case you
> > are
> > > > describing seems more like impersonation.*
> > >
> > > Yeah, I thought that one of the things that delegation tokens handled.
> > > Maybe I got it wrong :)
> > >
> > > Thanks for the detailed answers.
> > >
> > > Gwen
> > >
> > >
> > > > Thanks
> > > > Parth
> > > >
> > > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <gw...@confluent.io>
> > > wrote:
> > > >
> > > >> Hi Parth and Harsha,
> > > >>
> > > >> Few more comments:
> > > >>
> > > >> * The API / RequestResponse section doesn't seem to have good
> > > >> description of the changes to the Kafka Protocol. Sounds like you are
> > > >> proposing new DelegationTokenRequest and RenewTokenRequest (and
> > > >> matching responses), without detailing the contents of the requests
> > > >> and responses? Or rather, you show the class interface, but not the
> > > >> underlying protocol. This could be seen as an implementation detail,
> > > >> but since the binary protocol is what we provide to non-Java clients,
> > > >> we need to show the changes there.
> > > >>
> > > >> * getDelegationToken sounds like delegationTokenRequestHandler? Is it
> > > >> planned to be part of KafkaApi? or Client? Its unclear...
> > > >>
> > > >> * I want to emphasize that even though delegation tokens are a Hadoop
> > > >> innovation, I feel very strongly about not adding dependency on Hadoop
> > > >> when implementing delegation tokens for Kafka. The KIP doesn't imply
> > > >> such dependency, but if you can clarify...
> > > >>
> > > >> * Can we get delegation token at any time after authenticating? only
> > > >> immediately after?
> > > >>
> > > >> * My understanding is that tokens will propagate via ZK but without
> > > >> additional changes to UpdateMetadata protocol, correct? Clients
> > > >> currently don't retry on SASL auth failure (IIRC), but since the
> > > >> tokens propagate between brokers asynch, we will need to retry a bit
> > > >> to avoid clients failing auth due to timing issues.
> > > >>
> > > >> * Strongly agreeing on clients not touching ZK directly :)
> > > >>
> > > >> * I liked Ashish's suggestion of having just the controller issue the
> > > >> delegation tokens, to avoid syncing a shared secret. Not sure if we
> > > >> want to continue the discussion here or on the wiki. I think that we
> > > >> can decouple the problem of "token distribution" from "shared secret
> > > >> distribution" and use the controller as the only token generator to
> > > >> solve the second issue, while still using ZK async to distribute
> > > >> tokens.
> > > >>
> > > >> * I am also uncomfortable with infinite lifetime of tokens (and hoped
> > > >> to hear from others in the community) - but having the option and
> > > >> documenting it as less secure, allows users to configure their system
> > > >> as the wish.
> > > >>
> > > >> * While I like the idea of forcing kerberos auth for renwal, I think
> > > >> it mixes the transport layer the the request content in a pretty ugly
> > > >> way. Perhaps limiting renewer to non-owner is better.
> > > >>
> > > >> Things I'd still like to see:
> > > >>
> > > >> * More detailed explanation on what we plan to do for the java clients
> > > >> specifically - new configuration? new APIs?
> > > >> The response for my question on how multiple identities will be
> > > >> handled wasn't super clear to me - AFAIK, we don't authenticate each
> > > >> request, we authenticate connections.
> > > >>
> > > >> * Alternatives: Delegation tokens are only used in the Hadoop
> > > >> ecosystem. I'm wondering if there are alternatives in other ecosystems
> > > >> (Mesos? Tachyon? Cassandra?) and whether there are some advantages
> > > >> there.
> > > >>
> > > >> Gwen
> > > >>
> > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <ka...@harsha.io> wrote:
> > > >> > Hi Gwen,
> > > >> >            Can you look at Parth's last reply. Does it answer your
> > > >> >            concerns.
> > > >> >
> > > >> > Thanks,
> > > >> > Harsha
> > > >> >
> > > >> > On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt wrote:
> > > >> >> Thanks for reviewing Gwen. The wiki already has details on token
> > > >> >> expiration
> > > >> >> under token acquisition process
> > > >> >> <
> > > >>
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> > > >> >.
> > > >> >> Current proposal is that tokens will expire based on a server side
> > > >> >> configuration (default 24 hours) unless renewed. Renewal is only
> > > allowed
> > > >> >> until the max life time of token. Alternatively we could also make
> > > that
> > > >> >> an
> > > >> >> optional param and the server side default can serve as the upper
> > > bound.
> > > >> >>
> > > >> >> To your second point it will be done exactly the same way we would
> > > >> >> support
> > > >> >> multiple keytabs. The calling client will have to put the tokens it
> > > >> wants
> > > >> >> to use in the subject instance and call produce/consume inside
> > > >> >> subject.doas. Each caller will have to keep track of its own
> > > subject. I
> > > >> >> will have to look at the code to see if we support this feature
> > right
> > > >> now
> > > >> >> but my understanding is delegation token shouldn't need any special
> > > >> >> treatment as its just another type of Credential in the subject.
> > > >> >>
> > > >> >> I would also like to know what is your opinion about infinite
> > renewal
> > > >> (my
> > > >> >> recommendation is to not support this), tokens renewing them
> > self(my
> > > >> >> recommendation is to not support this) and most importantly your
> > > choice
> > > >> >> between the alternatives listed on this thread
> > > >> >> <
> > > >>
> > >
> > http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> > > >> >
> > > >> >> ( I am leaning towards the alternative-2 minus controller
> > > distributing
> > > >> >> secret). Thanks again for reviewing.
> > > >> >>
> > > >> >> Thanks
> > > >> >> Parth
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <gw...@confluent.io>
> > > wrote:
> > > >> >>
> > > >> >> > Harsha,
> > > >> >> >
> > > >> >> > I was thinking of the Rest Proxy. I didn't see your design yet,
> > > but in
> > > >> >> > our proxy, we have a set of producers, which will serve multiple
> > > users
> > > >> >> > going through the proxy. Since these users will have different
> > > >> >> > privileges, they'll need to authenticate separately, and can't
> > > share a
> > > >> >> > token.
> > > >> >> >
> > > >> >> > Am I missing anything?
> > > >> >> >
> > > >> >> > Gwen
> > > >> >> >
> > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io> wrote:
> > > >> >> > > Gwen,
> > > >> >> > >            On your second point. Can you describe a usecase
> > where
> > > >> >> > >            mutliple clients ended up sharing a producer and
> > even
> > > if
> > > >> they
> > > >> >> > >            do why can't they not use single token that producer
> > > >> >> > >            captures. Why would we need multiple clients with
> > > >> different
> > > >> >> > >            tokens sharing a single instance of producer.  Also
> > in
> > > >> this
> > > >> >> > >            case other clients have access all the tokens no?
> > > >> >> > >
> > > >> >> > > Thanks,
> > > >> >> > > Harsha
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote:
> > > >> >> > >> Sorry for the delay:
> > > >> >> > >>
> > > >> >> > >> Two questions that we didn't see in the wiki:
> > > >> >> > >> 1. Is there an expiration for delegation tokens? Renewal? How
> > > do we
> > > >> >> > >> revoke them?
> > > >> >> > >> 2. If we want to use delegation tokens for "do-as" (say,
> > submit
> > > >> Storm
> > > >> >> > >> job as my user), we will need a producer for every job (we
> > can't
> > > >> share
> > > >> >> > >> them between multiple jobs running on same node), since we
> > only
> > > >> >> > >> authenticate when connecting. Is there a plan to change this
> > for
> > > >> >> > >> delegation tokens, in order to allow multiple users with
> > > different
> > > >> >> > >> tokens to share a client?
> > > >> >> > >>
> > > >> >> > >> Gwen
> > > >> >> > >>
> > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
> > > >> >> > >> <br...@gmail.com> wrote:
> > > >> >> > >> > Bumping this up one more time, can other committers review?
> > > >> >> > >> >
> > > >> >> > >> > Thanks
> > > >> >> > >> > Parth
> > > >> >> > >> >
> > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <ka...@harsha.io>
> > > wrote:
> > > >> >> > >> >
> > > >> >> > >> >> Parth,
> > > >> >> > >> >>           Overall current design looks good to me. I am +1
> > on
> > > >> the
> > > >> >> > KIP.
> > > >> >> > >> >>
> > > >> >> > >> >> Gwen , Jun can you review this as well.
> > > >> >> > >> >>
> > > >> >> > >> >> -Harsha
> > > >> >> > >> >>
> > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote:
> > > >> >> > >> >> > Thanks for review Jitendra.
> > > >> >> > >> >> >
> > > >> >> > >> >> > I don't like the idea of infinite lifetime but I see the
> > > >> Streaming
> > > >> >> > use
> > > >> >> > >> >> > case. Even for Streaming use case I was hoping there will
> > > be
> > > >> some
> > > >> >> > notion
> > > >> >> > >> >> > of
> > > >> >> > >> >> > master/driver that can get new delegation tokens at fixed
> > > >> interval
> > > >> >> > and
> > > >> >> > >> >> > distribute to workers. If that is not the case for we can
> > > >> discuss
> > > >> >> > >> >> > delegation tokens renewing them self and the security
> > > >> implications
> > > >> >> > of the
> > > >> >> > >> >> > same.
> > > >> >> > >> >> >
> > > >> >> > >> >> > I did not want clients to fetch tokens from zookeeper,
> > > >> overall I
> > > >> >> > think
> > > >> >> > >> >> > its
> > > >> >> > >> >> > better if clients don't rely on our metadata store and I
> > > >> think we
> > > >> >> > are
> > > >> >> > >> >> > moving in that direction with all the KIP-4 improvements.
> > > I
> > > >> chose
> > > >> >> > >> >> > zookeeper as in this case the client will still just talk
> > > to
> > > >> >> > broker , its
> > > >> >> > >> >> > the brokers that will use zookeeper which we already do
> > > for a
> > > >> lot
> > > >> >> > of
> > > >> >> > >> >> > other
> > > >> >> > >> >> > usecases + ease of development + and the ability so
> > tokens
> > > >> will
> > > >> >> > survive
> > > >> >> > >> >> > even a rolling restart/cluster failure. if a majority
> > > agrees
> > > >> the
> > > >> >> > added
> > > >> >> > >> >> > complexity to have controller forwarding keys to all
> > > broker is
> > > >> >> > justified
> > > >> >> > >> >> > as
> > > >> >> > >> >> > it provides tighter security , I am fine with that option
> > > too.
> > > >> >> > >> >> >
> > > >> >> > >> >> > Given zookeeper does not support SSL we can not store
> > > master
> > > >> keys
> > > >> >> > in
> > > >> >> > >> >> > zookeeper as master keys will be exposed on wire. To
> > > support
> > > >> >> > rotation
> > > >> >> > >> >> > without affecting current clients is something I need to
> > > put
> > > >> more
> > > >> >> > thought
> > > >> >> > >> >> > in. My current proposal assumes the rotation will
> > > invalidate
> > > >> all
> > > >> >> > current
> > > >> >> > >> >> > tokens.
> > > >> >> > >> >> >
> > > >> >> > >> >> > I request committers to also review and post their
> > comments
> > > >> so we
> > > >> >> > can
> > > >> >> > >> >> > make
> > > >> >> > >> >> > progress on this KIP.
> > > >> >> > >> >> >
> > > >> >> > >> >> > Thanks
> > > >> >> > >> >> > Parth
> > > >> >> > >> >> >
> > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <
> > > >> asingh@cloudera.com
> > > >> >> > >
> > > >> >> > >> >> > wrote:
> > > >> >> > >> >> >
> > > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <
> > > kafka@harsha.io>
> > > >> >> > wrote:
> > > >> >> > >> >> > >
> > > >> >> > >> >> > > > Unifying the two discussion threads on this KIP.
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > > Here is the response from Jitendra
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > > "The need for a large number of clients that are
> > > running
> > > >> all
> > > >> >> > over the
> > > >> >> > >> >> > > > cluster that authenticate with Kafka brokers, is very
> > > >> similar
> > > >> >> > to the
> > > >> >> > >> >> > > > Hadoop use case of large number of tasks running
> > across
> > > >> the
> > > >> >> > cluster
> > > >> >> > >> >> that
> > > >> >> > >> >> > > > need authentication to Hdfs Namenode. Therefore, the
> > > >> >> > delegation token
> > > >> >> > >> >> > > > approach does seem like a good fit for this use case
> > > as we
> > > >> >> > have seen
> > > >> >> > >> >> it
> > > >> >> > >> >> > > > working at large scale in HDFS and YARN.
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > >   The proposed design is very much inline with Hadoop
> > > >> >> > approach. A few
> > > >> >> > >> >> > > >   comments:
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > > 1) Why do you guys want to allow infinite renewable
> > > >> lifetime
> > > >> >> > for a
> > > >> >> > >> >> > > > token? HDFS restricts a token to a max life time
> > > (default
> > > >> 7
> > > >> >> > days).  A
> > > >> >> > >> >> > > > token's vulnerability is believed to increase with
> > > time.
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > I agree that having infinite lifetime might not be the
> > > best
> > > >> idea.
> > > >> >> > >> >> > >
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > > 2) As I understand the tokens are stored in zookeeper
> > > as
> > > >> well,
> > > >> >> > and
> > > >> >> > >> >> can
> > > >> >> > >> >> > > > be updated there. This is clever as it can allow
> > > >> replacing the
> > > >> >> > tokens
> > > >> >> > >> >> > > > once they run out of max life time, and clients can
> > > >> download
> > > >> >> > new
> > > >> >> > >> >> tokens
> > > >> >> > >> >> > > > from zookeeper. It shouldn't be a big load on
> > zookeeper
> > > >> as a
> > > >> >> > client
> > > >> >> > >> >> will
> > > >> >> > >> >> > > > need to get a new token once in several days. In this
> > > >> approach
> > > >> >> > you
> > > >> >> > >> >> don't
> > > >> >> > >> >> > > > need infinite lifetime on the token even for long
> > > running
> > > >> >> > clients.
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > > 3) The token password are generated using a master
> > key.
> > > >> The
> > > >> >> > master
> > > >> >> > >> >> key
> > > >> >> > >> >> > > > should also be periodically changed. In Hadoop, the
> > > >> default
> > > >> >> > renewal
> > > >> >> > >> >> > > > period is 1 day.?
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > IIUC, this will require brokers maintaining a list of X
> > > most
> > > >> >> > recent
> > > >> >> > >> >> master
> > > >> >> > >> >> > > keys. This list will have to be persisted somewhere, as
> > > if a
> > > >> >> > broker
> > > >> >> > >> >> goes
> > > >> >> > >> >> > > down it will have to get that list again and storing
> > > master
> > > >> keys
> > > >> >> > on ZK
> > > >> >> > >> >> is
> > > >> >> > >> >> > > not the best idea. However, if a broker goes down then
> > we
> > > >> have
> > > >> >> > much
> > > >> >> > >> >> bigger
> > > >> >> > >> >> > > issue to deal with and client can always
> > re-authenticate
> > > is
> > > >> such
> > > >> >> > >> >> events.
> > > >> >> > >> >> > >
> > > >> >> > >> >> > > Did you happen to take a look at other alternatives
> > this
> > > >> list has
> > > >> >> > >> >> > > suggested?
> > > >> >> > >> >> > >
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > > Thanks for a thorough proposal, great work!"
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
> > > >> >> > >> >> > > > > Makes sense to me. Thanks!
> > > >> >> > >> >> > > > >
> > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <
> > > kafka@harsha.io
> > > >> >
> > > >> >> > wrote:
> > > >> >> > >> >> > > > > > It doesn't need any release vehicle but still the
> > > >> work can
> > > >> >> > move
> > > >> >> > >> >> > > > forward.
> > > >> >> > >> >> > > > > > If anyone is interested in the KIP please do the
> > > >> review and
> > > >> >> > >> >> provide
> > > >> >> > >> >> > > the
> > > >> >> > >> >> > > > > > comments.
> > > >> >> > >> >> > > > > >
> > > >> >> > >> >> > > > > > -Harsha
> > > >> >> > >> >> > > > > >
> > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma
> > > wrote:
> > > >> >> > >> >> > > > > >> I agree that it would be good to have more time
> > to
> > > >> review
> > > >> >> > and
> > > >> >> > >> >> > > discuss
> > > >> >> > >> >> > > > > >> KIP-48.
> > > >> >> > >> >> > > > > >>
> > > >> >> > >> >> > > > > >> Ismael
> > > >> >> > >> >> > > > > >>
> > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <
> > > >> >> > >> >> gwen@confluent.io>
> > > >> >> > >> >> > > > wrote:
> > > >> >> > >> >> > > > > >>
> > > >> >> > >> >> > > > > >> > Hi Team,
> > > >> >> > >> >> > > > > >> >
> > > >> >> > >> >> > > > > >> > Since KIP-48 depends on KIP-43, which is
> > > already a
> > > >> bit
> > > >> >> > of a
> > > >> >> > >> >> risk
> > > >> >> > >> >> > > for
> > > >> >> > >> >> > > > > >> > the next release - any chance we can delay
> > > >> delegation
> > > >> >> > tokens
> > > >> >> > >> >> to
> > > >> >> > >> >> > > > Kafka
> > > >> >> > >> >> > > > > >> > 0.10.1?
> > > >> >> > >> >> > > > > >> > With the community working on a release every
> > 3
> > > >> month,
> > > >> >> > this
> > > >> >> > >> >> is not
> > > >> >> > >> >> > > > a huge
> > > >> >> > >> >> > > > > >> > delay.
> > > >> >> > >> >> > > > > >> >
> > > >> >> > >> >> > > > > >> > Gwen
> > > >> >> > >> >> > > > > >> >
> > > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh
> > <
> > > >> >> > >> >> > > asingh@cloudera.com>
> > > >> >> > >> >> > > > wrote:
> > > >> >> > >> >> > > > > >> > > Parth,
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > Thanks again for the awesome write up.
> > > Following
> > > >> our
> > > >> >> > >> >> discussion
> > > >> >> > >> >> > > > from the
> > > >> >> > >> >> > > > > >> > > JIRA, I think it will be easier to compare
> > > >> various
> > > >> >> > >> >> alternatives
> > > >> >> > >> >> > > > if they
> > > >> >> > >> >> > > > > >> > are
> > > >> >> > >> >> > > > > >> > > listed together. I am stating below a few
> > > >> >> > alternatives along
> > > >> >> > >> >> > > with
> > > >> >> > >> >> > > > a the
> > > >> >> > >> >> > > > > >> > > current proposal.
> > > >> >> > >> >> > > > > >> > > (Current proposal) Store Delegation Token,
> > DT,
> > > >> on ZK.
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a broker.
> > > >> >> > >> >> > > > > >> > >    2. Once a client is authenticated, it
> > will
> > > >> make a
> > > >> >> > broker
> > > >> >> > >> >> side
> > > >> >> > >> >> > > > call to
> > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > > >> >> > >> >> > > > > >> > >    3. The broker generates a shared secret
> > > based
> > > >> on
> > > >> >> > >> >> > > HMAC-SHA256(a
> > > >> >> > >> >> > > > > >> > >    Password/Secret shared between all
> > brokers,
> > > >> >> > randomly
> > > >> >> > >> >> > > generated
> > > >> >> > >> >> > > > > >> > tokenId).
> > > >> >> > >> >> > > > > >> > >    4. Broker stores this token in its in
> > > memory
> > > >> cache.
> > > >> >> > >> >> Broker
> > > >> >> > >> >> > > > also stores
> > > >> >> > >> >> > > > > >> > >    the DelegationToken without the hmac in
> > the
> > > >> >> > zookeeper.
> > > >> >> > >> >> > > > > >> > >    5. All brokers will have a cache backed
> > by
> > > >> >> > zookeeper so
> > > >> >> > >> >> they
> > > >> >> > >> >> > > > will all
> > > >> >> > >> >> > > > > >> > >    get notified whenever a new token is
> > > >> generated and
> > > >> >> > they
> > > >> >> > >> >> will
> > > >> >> > >> >> > > > update
> > > >> >> > >> >> > > > > >> > their
> > > >> >> > >> >> > > > > >> > >    local cache whenever token state changes.
> > > >> >> > >> >> > > > > >> > >    6. Broker returns the token to Client.
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > >    1. Probable race condition, client tries
> > to
> > > >> >> > authenticate
> > > >> >> > >> >> with
> > > >> >> > >> >> > > > a broker
> > > >> >> > >> >> > > > > >> > >    that is yet to be updated with the newly
> > > >> generated
> > > >> >> > DT.
> > > >> >> > >> >> This
> > > >> >> > >> >> > > can
> > > >> >> > >> >> > > > > >> > probably be
> > > >> >> > >> >> > > > > >> > >    dealt with making dtRequest block until
> > all
> > > >> >> > brokers have
> > > >> >> > >> >> > > > updated
> > > >> >> > >> >> > > > > >> > their DT
> > > >> >> > >> >> > > > > >> > >    cache. Zk barrier or similar mechanism
> > can
> > > be
> > > >> used.
> > > >> >> > >> >> However,
> > > >> >> > >> >> > > > all such
> > > >> >> > >> >> > > > > >> > >    mechanisms will increase complexity.
> > > >> >> > >> >> > > > > >> > >    2. Using a static secret key from config
> > > >> file. Will
> > > >> >> > >> >> require
> > > >> >> > >> >> > > yet
> > > >> >> > >> >> > > > > >> > another
> > > >> >> > >> >> > > > > >> > >    config and uses a static secret key. It
> > is
> > > >> advised
> > > >> >> > to
> > > >> >> > >> >> rotate
> > > >> >> > >> >> > > > secret
> > > >> >> > >> >> > > > > >> > keys
> > > >> >> > >> >> > > > > >> > >    periodically. This can be avoided with
> > > >> controller
> > > >> >> > >> >> generating
> > > >> >> > >> >> > > > > >> > secretKey and
> > > >> >> > >> >> > > > > >> > >    passing to brokers periodically. However,
> > > >> this will
> > > >> >> > >> >> require
> > > >> >> > >> >> > > > brokers to
> > > >> >> > >> >> > > > > >> > >    maintain certain counts of secretKeys.
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > (Alternative 1) Have controller generate
> > > >> delegation
> > > >> >> > token.
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a broker.
> > > >> >> > >> >> > > > > >> > >    2. Once a client is authenticated, it
> > will
> > > >> make a
> > > >> >> > broker
> > > >> >> > >> >> side
> > > >> >> > >> >> > > > call to
> > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > > >> >> > >> >> > > > > >> > >    3. Broker forwards the request to
> > > controller.
> > > >> >> > >> >> > > > > >> > >    4. Controller generates a DT and
> > broadcasts
> > > >> to all
> > > >> >> > >> >> brokers.
> > > >> >> > >> >> > > > > >> > >    5. Broker stores this token in its memory
> > > >> cache.
> > > >> >> > >> >> > > > > >> > >    6. Controller responds to broker’s DT
> > req.
> > > >> >> > >> >> > > > > >> > >    7. Broker returns the token to Client.
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > >    1. We will have to add new APIs to
> > support
> > > >> >> > controller
> > > >> >> > >> >> pushing
> > > >> >> > >> >> > > > tokens
> > > >> >> > >> >> > > > > >> > to
> > > >> >> > >> >> > > > > >> > >    brokers on top of the minimal APIs that
> > are
> > > >> >> > currently
> > > >> >> > >> >> > > proposed.
> > > >> >> > >> >> > > > > >> > >    2. We will also have to add APIs to
> > support
> > > >> the
> > > >> >> > >> >> bootstrapping
> > > >> >> > >> >> > > > case,
> > > >> >> > >> >> > > > > >> > i.e,
> > > >> >> > >> >> > > > > >> > >    when a new broker comes up it will have
> > to
> > > >> get all
> > > >> >> > >> >> delegation
> > > >> >> > >> >> > > > tokens
> > > >> >> > >> >> > > > > >> > from
> > > >> >> > >> >> > > > > >> > >    the controller.
> > > >> >> > >> >> > > > > >> > >    3. In catastrophic failures where all
> > > brokers
> > > >> go
> > > >> >> > down,
> > > >> >> > >> >> the
> > > >> >> > >> >> > > > tokens will
> > > >> >> > >> >> > > > > >> > >    be lost even if servers are restarted as
> > > >> tokens
> > > >> >> > are not
> > > >> >> > >> >> > > > persisted
> > > >> >> > >> >> > > > > >> > anywhere.
> > > >> >> > >> >> > > > > >> > >    If this happens, then there are more
> > > important
> > > >> >> > things to
> > > >> >> > >> >> > > worry
> > > >> >> > >> >> > > > about
> > > >> >> > >> >> > > > > >> > and
> > > >> >> > >> >> > > > > >> > >    maybe it is better to re-authenticate.
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > (Alternative 2) Do not distribute DT to
> > other
> > > >> brokers
> > > >> >> > at
> > > >> >> > >> >> all.
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > >    1. Client authenticates with a broker.
> > > >> >> > >> >> > > > > >> > >    2. Once a client is authenticated, it
> > will
> > > >> make a
> > > >> >> > broker
> > > >> >> > >> >> side
> > > >> >> > >> >> > > > call to
> > > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > > >> >> > >> >> > > > > >> > >    3. The broker generates DT of form,
> > [hmac +
> > > >> (owner,
> > > >> >> > >> >> renewer,
> > > >> >> > >> >> > > > > >> > >    maxLifeTime, id, hmac, expirationTime)]
> > and
> > > >> passes
> > > >> >> > back
> > > >> >> > >> >> this
> > > >> >> > >> >> > > > DT to
> > > >> >> > >> >> > > > > >> > client.
> > > >> >> > >> >> > > > > >> > >    hmac is generated via {HMAC-SHA256(owner,
> > > >> renewer,
> > > >> >> > >> >> > > > maxLifeTime, id,
> > > >> >> > >> >> > > > > >> > hmac,
> > > >> >> > >> >> > > > > >> > >    expirationTime) using SecretKey}. Note
> > that
> > > >> all
> > > >> >> > brokers
> > > >> >> > >> >> have
> > > >> >> > >> >> > > > this
> > > >> >> > >> >> > > > > >> > SecretKey.
> > > >> >> > >> >> > > > > >> > >    4. Client then goes to any broker and to
> > > >> >> > authenticate
> > > >> >> > >> >> sends
> > > >> >> > >> >> > > > the DT.
> > > >> >> > >> >> > > > > >> > >    Broker recalculates hmac using (owner,
> > > >> renewer,
> > > >> >> > >> >> maxLifeTime,
> > > >> >> > >> >> > > > id, hmac,
> > > >> >> > >> >> > > > > >> > >    expirationTime) info from DT and its
> > > >> SecretKey. If
> > > >> >> > it
> > > >> >> > >> >> matches
> > > >> >> > >> >> > > > with
> > > >> >> > >> >> > > > > >> > hmac of
> > > >> >> > >> >> > > > > >> > >    DT, client is authenticated. Yes, it will
> > > do
> > > >> other
> > > >> >> > >> >> obvious
> > > >> >> > >> >> > > > checks of
> > > >> >> > >> >> > > > > >> > >    timestamp expiry and such.
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > Note that secret key will be generated by
> > > >> controller
> > > >> >> > and
> > > >> >> > >> >> passed
> > > >> >> > >> >> > > to
> > > >> >> > >> >> > > > > >> > brokers
> > > >> >> > >> >> > > > > >> > > periodically.
> > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > >    1. How to delete a DT? Yes, that is a
> > > downside
> > > >> >> > here.
> > > >> >> > >> >> However,
> > > >> >> > >> >> > > > this can
> > > >> >> > >> >> > > > > >> > >    be handled with brokers maintaining a
> > > >> blacklist of
> > > >> >> > DTs,
> > > >> >> > >> >> DTs
> > > >> >> > >> >> > > > from this
> > > >> >> > >> >> > > > > >> > list
> > > >> >> > >> >> > > > > >> > >    can be removed after expiry.
> > > >> >> > >> >> > > > > >> > >    2. In catastrophic failures where all
> > > brokers
> > > >> go
> > > >> >> > down,
> > > >> >> > >> >> the
> > > >> >> > >> >> > > > tokens will
> > > >> >> > >> >> > > > > >> > >    be lost even if servers are restarted as
> > > >> tokens
> > > >> >> > are not
> > > >> >> > >> >> > > > persisted
> > > >> >> > >> >> > > > > >> > anywhere.
> > > >> >> > >> >> > > > > >> > >    If this happens, then there are more
> > > important
> > > >> >> > things to
> > > >> >> > >> >> > > worry
> > > >> >> > >> >> > > > about
> > > >> >> > >> >> > > > > >> > and
> > > >> >> > >> >> > > > > >> > >    maybe it is better to re-authenticate.
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth
> > > >> Brahmbhatt <
> > > >> >> > >> >> > > > > >> > > pbrahmbhatt@hortonworks.com> wrote:
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > >> Hi,
> > > >> >> > >> >> > > > > >> > >>
> > > >> >> > >> >> > > > > >> > >> I have filed KIP-48 so we can offer hadoop
> > > like
> > > >> >> > delegation
> > > >> >> > >> >> > > > tokens in
> > > >> >> > >> >> > > > > >> > >> kafka. You can review the design
> > > >> >> > >> >> > > > > >> > >>
> > > >> >> > >> >> > > > > >> >
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > >
> > > >> >> > >> >>
> > > >> >> >
> > > >>
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > > >> >> > >> >> > > > > >> > .
> > > >> >> > >> >> > > > > >> > >> This KIP depends on KIP-43 and we have also
> > > >> >> > discussed an
> > > >> >> > >> >> > > > alternative to
> > > >> >> > >> >> > > > > >> > >> proposed design here<
> > > >> >> > >> >> > > > > >> > >>
> > > >> >> > >> >> > > > > >> >
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > >
> > > >> >> > >> >>
> > > >> >> >
> > > >>
> > >
> > https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > > >> >> > >> >> > > > > >> > >> >.
> > > >> >> > >> >> > > > > >> > >>
> > > >> >> > >> >> > > > > >> > >> Thanks
> > > >> >> > >> >> > > > > >> > >> Parth
> > > >> >> > >> >> > > > > >> > >>
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > --
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > Regards,
> > > >> >> > >> >> > > > > >> > > Ashish
> > > >> >> > >> >> > > > > >> >
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > >
> > > >> >> > >> >> > >
> > > >> >> > >> >> > >
> > > >> >> > >> >> > > --
> > > >> >> > >> >> > >
> > > >> >> > >> >> > > Regards,
> > > >> >> > >> >> > > Ashish
> > > >> >> > >> >> > >
> > > >> >> > >> >>
> > > >> >> >
> > > >>
> > >
> >

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[parth brahmbhatt <br...@gmail.com>]

That is what the hadoop echo system uses so no good reason really. We could
change it to whatever is the newest recommended standard is.

Thanks
Parth

On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <is...@juma.me.uk> wrote:

> Hi Parth,
>
> Thanks for the KIP. I only started reviewing this and may have additional
> questions later. The immediate question that came to mind is our choice of
> "DIGEST-MD5" even though it's marked as OBSOLETE in the IANA Registry of
> SASL mechanisms and the original RFC (2831) has been moved to Historic
> status:
>
> https://tools.ietf.org/html/rfc6331
> http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
>
> What is the reasoning behind that choice?
>
> Thanks,
> Ismael
>
> On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <gw...@confluent.io> wrote:
>
> > Also comments inline :)
> >
> > > * I want to emphasize that even though delegation tokens are a Hadoop
> > > innovation, I feel very strongly about not adding dependency on Hadoop
> > > when implementing delegation tokens for Kafka. The KIP doesn't imply
> > > such dependency, but if you can clarify...
> > >
> > >
> > > *No hadoop dependency.*
> >
> > Yay! Just add this to the KIP so no one will read the KIP and panic
> > three weeks before the next release...
> >
> > > * Can we get delegation token at any time after authenticating? only
> > > immediately after?
> > >
> > >
> > > *As long as you are authenticated you can get delegation tokens. We
> need
> > to
> > > discuss if a client authenticated using delegation token, can also
> > acquire
> > > delegation token again or not. Also there is the question of do we
> allow
> > > anyone to acquire delegation token or we want specific ACLs (I think
> its
> > an
> > > overkill.)*
> >
> > I agree that ACLs is an overkill.
> >
> > I think we are debating two options: Either require Kerberos auth for
> > renewal or require non-owners to renew.
> > I *think* the latter is simpler (it basically require a "job master"
> > to take responsibility for the renewal, it will have its own identity
> > anyway and I think this is the correct design pattern anyway. For
> > storm, I'd expect Nimbus to coordinate renewals?), but it is hard to
> > debate simplicity without looking at the code changes required. If you
> > have a draft of how the "require Kerberos" will look in Kafka code,
> > I'll be happy to take a look.
> >
> > > * My understanding is that tokens will propagate via ZK but without
> > > additional changes to UpdateMetadata protocol, correct? Clients
> > > currently don't retry on SASL auth failure (IIRC), but since the
> > > tokens propagate between brokers asynch, we will need to retry a bit
> > > to avoid clients failing auth due to timing issues.
> > >
> > > *I am considering 2 alternatives right now. The current documented
> > approach
> > > is zookeeper based and it does not require any changes to
> UpdateMetadata
> > > protocol. An alternative approach can remove zookeeper dependency as
> well
> > > but we can discuss that in KIP discussion call.*
> >
> > Oooh! Sounds interesting. Do you want to ping Jun to arrange a call?
> >
> > > * I liked Ashish's suggestion of having just the controller issue the
> > > delegation tokens, to avoid syncing a shared secret. Not sure if we
> > > want to continue the discussion here or on the wiki. I think that we
> > > can decouple the problem of "token distribution" from "shared secret
> > > distribution" and use the controller as the only token generator to
> > > solve the second issue, while still using ZK async to distribute
> > > tokens.
> > >
> > >
> > > *As mentioned in the previous Email I am fine with that approach as
> long
> > as
> > > we agree that the extra complexity of adding/updating APIS adds enough
> > > value. The advantage with the controller approach is secret rotation
> can
> > be
> > > automated,frequent and would not require deployment. *
> >
> > Can you detail the extra complexity (or point me to the email I
> > missed?) - which Apis are required?
> > As far as I can tell, clients can already find the controller from
> > metadata. I'm a bit more concerned about controller load.
> >
> > >
> > > * While I like the idea of forcing kerberos auth for renwal, I think
> > > it mixes the transport layer the the request content in a pretty ugly
> > > way. Perhaps limiting renewer to non-owner is better.
> > >
> > > *I feel this is a necessary evil. While this will make the kafka code
> > > pretty straight forward , forcing  renewer to non-owner pushes the code
> > > ugliness to client and makes it even harder to integrate.  *
> >
> > As mentioned before, I don't think the "renewal by other" approach is
> > that ugly for the clients we expect to use delegation tokens since
> > they will have an app-master of some sort who requested the token to
> > begin with.
> >
> > >
> > > The response for my question on how multiple identities will be
> > > handled wasn't super clear to me - AFAIK, we don't authenticate each
> > > request, we authenticate connections.
> > >
> > > *We authenticate connections, and only when they are being established.
> > Let
> > > me try to phrase this as a question, in absence of delegation tokens if
> > we
> > > had to support the use case using user TGT's how would we do it? My
> point
> > > was it would be no different with delegation tokens. The use case you
> are
> > > describing seems more like impersonation.*
> >
> > Yeah, I thought that one of the things that delegation tokens handled.
> > Maybe I got it wrong :)
> >
> > Thanks for the detailed answers.
> >
> > Gwen
> >
> >
> > > Thanks
> > > Parth
> > >
> > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <gw...@confluent.io>
> > wrote:
> > >
> > >> Hi Parth and Harsha,
> > >>
> > >> Few more comments:
> > >>
> > >> * The API / RequestResponse section doesn't seem to have good
> > >> description of the changes to the Kafka Protocol. Sounds like you are
> > >> proposing new DelegationTokenRequest and RenewTokenRequest (and
> > >> matching responses), without detailing the contents of the requests
> > >> and responses? Or rather, you show the class interface, but not the
> > >> underlying protocol. This could be seen as an implementation detail,
> > >> but since the binary protocol is what we provide to non-Java clients,
> > >> we need to show the changes there.
> > >>
> > >> * getDelegationToken sounds like delegationTokenRequestHandler? Is it
> > >> planned to be part of KafkaApi? or Client? Its unclear...
> > >>
> > >> * I want to emphasize that even though delegation tokens are a Hadoop
> > >> innovation, I feel very strongly about not adding dependency on Hadoop
> > >> when implementing delegation tokens for Kafka. The KIP doesn't imply
> > >> such dependency, but if you can clarify...
> > >>
> > >> * Can we get delegation token at any time after authenticating? only
> > >> immediately after?
> > >>
> > >> * My understanding is that tokens will propagate via ZK but without
> > >> additional changes to UpdateMetadata protocol, correct? Clients
> > >> currently don't retry on SASL auth failure (IIRC), but since the
> > >> tokens propagate between brokers asynch, we will need to retry a bit
> > >> to avoid clients failing auth due to timing issues.
> > >>
> > >> * Strongly agreeing on clients not touching ZK directly :)
> > >>
> > >> * I liked Ashish's suggestion of having just the controller issue the
> > >> delegation tokens, to avoid syncing a shared secret. Not sure if we
> > >> want to continue the discussion here or on the wiki. I think that we
> > >> can decouple the problem of "token distribution" from "shared secret
> > >> distribution" and use the controller as the only token generator to
> > >> solve the second issue, while still using ZK async to distribute
> > >> tokens.
> > >>
> > >> * I am also uncomfortable with infinite lifetime of tokens (and hoped
> > >> to hear from others in the community) - but having the option and
> > >> documenting it as less secure, allows users to configure their system
> > >> as the wish.
> > >>
> > >> * While I like the idea of forcing kerberos auth for renwal, I think
> > >> it mixes the transport layer the the request content in a pretty ugly
> > >> way. Perhaps limiting renewer to non-owner is better.
> > >>
> > >> Things I'd still like to see:
> > >>
> > >> * More detailed explanation on what we plan to do for the java clients
> > >> specifically - new configuration? new APIs?
> > >> The response for my question on how multiple identities will be
> > >> handled wasn't super clear to me - AFAIK, we don't authenticate each
> > >> request, we authenticate connections.
> > >>
> > >> * Alternatives: Delegation tokens are only used in the Hadoop
> > >> ecosystem. I'm wondering if there are alternatives in other ecosystems
> > >> (Mesos? Tachyon? Cassandra?) and whether there are some advantages
> > >> there.
> > >>
> > >> Gwen
> > >>
> > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <ka...@harsha.io> wrote:
> > >> > Hi Gwen,
> > >> >            Can you look at Parth's last reply. Does it answer your
> > >> >            concerns.
> > >> >
> > >> > Thanks,
> > >> > Harsha
> > >> >
> > >> > On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt wrote:
> > >> >> Thanks for reviewing Gwen. The wiki already has details on token
> > >> >> expiration
> > >> >> under token acquisition process
> > >> >> <
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> > >> >.
> > >> >> Current proposal is that tokens will expire based on a server side
> > >> >> configuration (default 24 hours) unless renewed. Renewal is only
> > allowed
> > >> >> until the max life time of token. Alternatively we could also make
> > that
> > >> >> an
> > >> >> optional param and the server side default can serve as the upper
> > bound.
> > >> >>
> > >> >> To your second point it will be done exactly the same way we would
> > >> >> support
> > >> >> multiple keytabs. The calling client will have to put the tokens it
> > >> wants
> > >> >> to use in the subject instance and call produce/consume inside
> > >> >> subject.doas. Each caller will have to keep track of its own
> > subject. I
> > >> >> will have to look at the code to see if we support this feature
> right
> > >> now
> > >> >> but my understanding is delegation token shouldn't need any special
> > >> >> treatment as its just another type of Credential in the subject.
> > >> >>
> > >> >> I would also like to know what is your opinion about infinite
> renewal
> > >> (my
> > >> >> recommendation is to not support this), tokens renewing them
> self(my
> > >> >> recommendation is to not support this) and most importantly your
> > choice
> > >> >> between the alternatives listed on this thread
> > >> >> <
> > >>
> >
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> > >> >
> > >> >> ( I am leaning towards the alternative-2 minus controller
> > distributing
> > >> >> secret). Thanks again for reviewing.
> > >> >>
> > >> >> Thanks
> > >> >> Parth
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <gw...@confluent.io>
> > wrote:
> > >> >>
> > >> >> > Harsha,
> > >> >> >
> > >> >> > I was thinking of the Rest Proxy. I didn't see your design yet,
> > but in
> > >> >> > our proxy, we have a set of producers, which will serve multiple
> > users
> > >> >> > going through the proxy. Since these users will have different
> > >> >> > privileges, they'll need to authenticate separately, and can't
> > share a
> > >> >> > token.
> > >> >> >
> > >> >> > Am I missing anything?
> > >> >> >
> > >> >> > Gwen
> > >> >> >
> > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io> wrote:
> > >> >> > > Gwen,
> > >> >> > >            On your second point. Can you describe a usecase
> where
> > >> >> > >            mutliple clients ended up sharing a producer and
> even
> > if
> > >> they
> > >> >> > >            do why can't they not use single token that producer
> > >> >> > >            captures. Why would we need multiple clients with
> > >> different
> > >> >> > >            tokens sharing a single instance of producer.  Also
> in
> > >> this
> > >> >> > >            case other clients have access all the tokens no?
> > >> >> > >
> > >> >> > > Thanks,
> > >> >> > > Harsha
> > >> >> > >
> > >> >> > >
> > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote:
> > >> >> > >> Sorry for the delay:
> > >> >> > >>
> > >> >> > >> Two questions that we didn't see in the wiki:
> > >> >> > >> 1. Is there an expiration for delegation tokens? Renewal? How
> > do we
> > >> >> > >> revoke them?
> > >> >> > >> 2. If we want to use delegation tokens for "do-as" (say,
> submit
> > >> Storm
> > >> >> > >> job as my user), we will need a producer for every job (we
> can't
> > >> share
> > >> >> > >> them between multiple jobs running on same node), since we
> only
> > >> >> > >> authenticate when connecting. Is there a plan to change this
> for
> > >> >> > >> delegation tokens, in order to allow multiple users with
> > different
> > >> >> > >> tokens to share a client?
> > >> >> > >>
> > >> >> > >> Gwen
> > >> >> > >>
> > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
> > >> >> > >> <br...@gmail.com> wrote:
> > >> >> > >> > Bumping this up one more time, can other committers review?
> > >> >> > >> >
> > >> >> > >> > Thanks
> > >> >> > >> > Parth
> > >> >> > >> >
> > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <ka...@harsha.io>
> > wrote:
> > >> >> > >> >
> > >> >> > >> >> Parth,
> > >> >> > >> >>           Overall current design looks good to me. I am +1
> on
> > >> the
> > >> >> > KIP.
> > >> >> > >> >>
> > >> >> > >> >> Gwen , Jun can you review this as well.
> > >> >> > >> >>
> > >> >> > >> >> -Harsha
> > >> >> > >> >>
> > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote:
> > >> >> > >> >> > Thanks for review Jitendra.
> > >> >> > >> >> >
> > >> >> > >> >> > I don't like the idea of infinite lifetime but I see the
> > >> Streaming
> > >> >> > use
> > >> >> > >> >> > case. Even for Streaming use case I was hoping there will
> > be
> > >> some
> > >> >> > notion
> > >> >> > >> >> > of
> > >> >> > >> >> > master/driver that can get new delegation tokens at fixed
> > >> interval
> > >> >> > and
> > >> >> > >> >> > distribute to workers. If that is not the case for we can
> > >> discuss
> > >> >> > >> >> > delegation tokens renewing them self and the security
> > >> implications
> > >> >> > of the
> > >> >> > >> >> > same.
> > >> >> > >> >> >
> > >> >> > >> >> > I did not want clients to fetch tokens from zookeeper,
> > >> overall I
> > >> >> > think
> > >> >> > >> >> > its
> > >> >> > >> >> > better if clients don't rely on our metadata store and I
> > >> think we
> > >> >> > are
> > >> >> > >> >> > moving in that direction with all the KIP-4 improvements.
> > I
> > >> chose
> > >> >> > >> >> > zookeeper as in this case the client will still just talk
> > to
> > >> >> > broker , its
> > >> >> > >> >> > the brokers that will use zookeeper which we already do
> > for a
> > >> lot
> > >> >> > of
> > >> >> > >> >> > other
> > >> >> > >> >> > usecases + ease of development + and the ability so
> tokens
> > >> will
> > >> >> > survive
> > >> >> > >> >> > even a rolling restart/cluster failure. if a majority
> > agrees
> > >> the
> > >> >> > added
> > >> >> > >> >> > complexity to have controller forwarding keys to all
> > broker is
> > >> >> > justified
> > >> >> > >> >> > as
> > >> >> > >> >> > it provides tighter security , I am fine with that option
> > too.
> > >> >> > >> >> >
> > >> >> > >> >> > Given zookeeper does not support SSL we can not store
> > master
> > >> keys
> > >> >> > in
> > >> >> > >> >> > zookeeper as master keys will be exposed on wire. To
> > support
> > >> >> > rotation
> > >> >> > >> >> > without affecting current clients is something I need to
> > put
> > >> more
> > >> >> > thought
> > >> >> > >> >> > in. My current proposal assumes the rotation will
> > invalidate
> > >> all
> > >> >> > current
> > >> >> > >> >> > tokens.
> > >> >> > >> >> >
> > >> >> > >> >> > I request committers to also review and post their
> comments
> > >> so we
> > >> >> > can
> > >> >> > >> >> > make
> > >> >> > >> >> > progress on this KIP.
> > >> >> > >> >> >
> > >> >> > >> >> > Thanks
> > >> >> > >> >> > Parth
> > >> >> > >> >> >
> > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <
> > >> asingh@cloudera.com
> > >> >> > >
> > >> >> > >> >> > wrote:
> > >> >> > >> >> >
> > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <
> > kafka@harsha.io>
> > >> >> > wrote:
> > >> >> > >> >> > >
> > >> >> > >> >> > > > Unifying the two discussion threads on this KIP.
> > >> >> > >> >> > > >
> > >> >> > >> >> > > > Here is the response from Jitendra
> > >> >> > >> >> > > >
> > >> >> > >> >> > > > "The need for a large number of clients that are
> > running
> > >> all
> > >> >> > over the
> > >> >> > >> >> > > > cluster that authenticate with Kafka brokers, is very
> > >> similar
> > >> >> > to the
> > >> >> > >> >> > > > Hadoop use case of large number of tasks running
> across
> > >> the
> > >> >> > cluster
> > >> >> > >> >> that
> > >> >> > >> >> > > > need authentication to Hdfs Namenode. Therefore, the
> > >> >> > delegation token
> > >> >> > >> >> > > > approach does seem like a good fit for this use case
> > as we
> > >> >> > have seen
> > >> >> > >> >> it
> > >> >> > >> >> > > > working at large scale in HDFS and YARN.
> > >> >> > >> >> > > >
> > >> >> > >> >> > > >   The proposed design is very much inline with Hadoop
> > >> >> > approach. A few
> > >> >> > >> >> > > >   comments:
> > >> >> > >> >> > > >
> > >> >> > >> >> > > > 1) Why do you guys want to allow infinite renewable
> > >> lifetime
> > >> >> > for a
> > >> >> > >> >> > > > token? HDFS restricts a token to a max life time
> > (default
> > >> 7
> > >> >> > days).  A
> > >> >> > >> >> > > > token's vulnerability is believed to increase with
> > time.
> > >> >> > >> >> > > >
> > >> >> > >> >> > > I agree that having infinite lifetime might not be the
> > best
> > >> idea.
> > >> >> > >> >> > >
> > >> >> > >> >> > > >
> > >> >> > >> >> > > > 2) As I understand the tokens are stored in zookeeper
> > as
> > >> well,
> > >> >> > and
> > >> >> > >> >> can
> > >> >> > >> >> > > > be updated there. This is clever as it can allow
> > >> replacing the
> > >> >> > tokens
> > >> >> > >> >> > > > once they run out of max life time, and clients can
> > >> download
> > >> >> > new
> > >> >> > >> >> tokens
> > >> >> > >> >> > > > from zookeeper. It shouldn't be a big load on
> zookeeper
> > >> as a
> > >> >> > client
> > >> >> > >> >> will
> > >> >> > >> >> > > > need to get a new token once in several days. In this
> > >> approach
> > >> >> > you
> > >> >> > >> >> don't
> > >> >> > >> >> > > > need infinite lifetime on the token even for long
> > running
> > >> >> > clients.
> > >> >> > >> >> > > >
> > >> >> > >> >> > > > 3) The token password are generated using a master
> key.
> > >> The
> > >> >> > master
> > >> >> > >> >> key
> > >> >> > >> >> > > > should also be periodically changed. In Hadoop, the
> > >> default
> > >> >> > renewal
> > >> >> > >> >> > > > period is 1 day.?
> > >> >> > >> >> > > >
> > >> >> > >> >> > > IIUC, this will require brokers maintaining a list of X
> > most
> > >> >> > recent
> > >> >> > >> >> master
> > >> >> > >> >> > > keys. This list will have to be persisted somewhere, as
> > if a
> > >> >> > broker
> > >> >> > >> >> goes
> > >> >> > >> >> > > down it will have to get that list again and storing
> > master
> > >> keys
> > >> >> > on ZK
> > >> >> > >> >> is
> > >> >> > >> >> > > not the best idea. However, if a broker goes down then
> we
> > >> have
> > >> >> > much
> > >> >> > >> >> bigger
> > >> >> > >> >> > > issue to deal with and client can always
> re-authenticate
> > is
> > >> such
> > >> >> > >> >> events.
> > >> >> > >> >> > >
> > >> >> > >> >> > > Did you happen to take a look at other alternatives
> this
> > >> list has
> > >> >> > >> >> > > suggested?
> > >> >> > >> >> > >
> > >> >> > >> >> > > >
> > >> >> > >> >> > > > Thanks for a thorough proposal, great work!"
> > >> >> > >> >> > > >
> > >> >> > >> >> > > >
> > >> >> > >> >> > > >
> > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
> > >> >> > >> >> > > > > Makes sense to me. Thanks!
> > >> >> > >> >> > > > >
> > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <
> > kafka@harsha.io
> > >> >
> > >> >> > wrote:
> > >> >> > >> >> > > > > > It doesn't need any release vehicle but still the
> > >> work can
> > >> >> > move
> > >> >> > >> >> > > > forward.
> > >> >> > >> >> > > > > > If anyone is interested in the KIP please do the
> > >> review and
> > >> >> > >> >> provide
> > >> >> > >> >> > > the
> > >> >> > >> >> > > > > > comments.
> > >> >> > >> >> > > > > >
> > >> >> > >> >> > > > > > -Harsha
> > >> >> > >> >> > > > > >
> > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma
> > wrote:
> > >> >> > >> >> > > > > >> I agree that it would be good to have more time
> to
> > >> review
> > >> >> > and
> > >> >> > >> >> > > discuss
> > >> >> > >> >> > > > > >> KIP-48.
> > >> >> > >> >> > > > > >>
> > >> >> > >> >> > > > > >> Ismael
> > >> >> > >> >> > > > > >>
> > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <
> > >> >> > >> >> gwen@confluent.io>
> > >> >> > >> >> > > > wrote:
> > >> >> > >> >> > > > > >>
> > >> >> > >> >> > > > > >> > Hi Team,
> > >> >> > >> >> > > > > >> >
> > >> >> > >> >> > > > > >> > Since KIP-48 depends on KIP-43, which is
> > already a
> > >> bit
> > >> >> > of a
> > >> >> > >> >> risk
> > >> >> > >> >> > > for
> > >> >> > >> >> > > > > >> > the next release - any chance we can delay
> > >> delegation
> > >> >> > tokens
> > >> >> > >> >> to
> > >> >> > >> >> > > > Kafka
> > >> >> > >> >> > > > > >> > 0.10.1?
> > >> >> > >> >> > > > > >> > With the community working on a release every
> 3
> > >> month,
> > >> >> > this
> > >> >> > >> >> is not
> > >> >> > >> >> > > > a huge
> > >> >> > >> >> > > > > >> > delay.
> > >> >> > >> >> > > > > >> >
> > >> >> > >> >> > > > > >> > Gwen
> > >> >> > >> >> > > > > >> >
> > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh
> <
> > >> >> > >> >> > > asingh@cloudera.com>
> > >> >> > >> >> > > > wrote:
> > >> >> > >> >> > > > > >> > > Parth,
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > > Thanks again for the awesome write up.
> > Following
> > >> our
> > >> >> > >> >> discussion
> > >> >> > >> >> > > > from the
> > >> >> > >> >> > > > > >> > > JIRA, I think it will be easier to compare
> > >> various
> > >> >> > >> >> alternatives
> > >> >> > >> >> > > > if they
> > >> >> > >> >> > > > > >> > are
> > >> >> > >> >> > > > > >> > > listed together. I am stating below a few
> > >> >> > alternatives along
> > >> >> > >> >> > > with
> > >> >> > >> >> > > > a the
> > >> >> > >> >> > > > > >> > > current proposal.
> > >> >> > >> >> > > > > >> > > (Current proposal) Store Delegation Token,
> DT,
> > >> on ZK.
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > >    1. Client authenticates with a broker.
> > >> >> > >> >> > > > > >> > >    2. Once a client is authenticated, it
> will
> > >> make a
> > >> >> > broker
> > >> >> > >> >> side
> > >> >> > >> >> > > > call to
> > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > >> >> > >> >> > > > > >> > >    3. The broker generates a shared secret
> > based
> > >> on
> > >> >> > >> >> > > HMAC-SHA256(a
> > >> >> > >> >> > > > > >> > >    Password/Secret shared between all
> brokers,
> > >> >> > randomly
> > >> >> > >> >> > > generated
> > >> >> > >> >> > > > > >> > tokenId).
> > >> >> > >> >> > > > > >> > >    4. Broker stores this token in its in
> > memory
> > >> cache.
> > >> >> > >> >> Broker
> > >> >> > >> >> > > > also stores
> > >> >> > >> >> > > > > >> > >    the DelegationToken without the hmac in
> the
> > >> >> > zookeeper.
> > >> >> > >> >> > > > > >> > >    5. All brokers will have a cache backed
> by
> > >> >> > zookeeper so
> > >> >> > >> >> they
> > >> >> > >> >> > > > will all
> > >> >> > >> >> > > > > >> > >    get notified whenever a new token is
> > >> generated and
> > >> >> > they
> > >> >> > >> >> will
> > >> >> > >> >> > > > update
> > >> >> > >> >> > > > > >> > their
> > >> >> > >> >> > > > > >> > >    local cache whenever token state changes.
> > >> >> > >> >> > > > > >> > >    6. Broker returns the token to Client.
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > >    1. Probable race condition, client tries
> to
> > >> >> > authenticate
> > >> >> > >> >> with
> > >> >> > >> >> > > > a broker
> > >> >> > >> >> > > > > >> > >    that is yet to be updated with the newly
> > >> generated
> > >> >> > DT.
> > >> >> > >> >> This
> > >> >> > >> >> > > can
> > >> >> > >> >> > > > > >> > probably be
> > >> >> > >> >> > > > > >> > >    dealt with making dtRequest block until
> all
> > >> >> > brokers have
> > >> >> > >> >> > > > updated
> > >> >> > >> >> > > > > >> > their DT
> > >> >> > >> >> > > > > >> > >    cache. Zk barrier or similar mechanism
> can
> > be
> > >> used.
> > >> >> > >> >> However,
> > >> >> > >> >> > > > all such
> > >> >> > >> >> > > > > >> > >    mechanisms will increase complexity.
> > >> >> > >> >> > > > > >> > >    2. Using a static secret key from config
> > >> file. Will
> > >> >> > >> >> require
> > >> >> > >> >> > > yet
> > >> >> > >> >> > > > > >> > another
> > >> >> > >> >> > > > > >> > >    config and uses a static secret key. It
> is
> > >> advised
> > >> >> > to
> > >> >> > >> >> rotate
> > >> >> > >> >> > > > secret
> > >> >> > >> >> > > > > >> > keys
> > >> >> > >> >> > > > > >> > >    periodically. This can be avoided with
> > >> controller
> > >> >> > >> >> generating
> > >> >> > >> >> > > > > >> > secretKey and
> > >> >> > >> >> > > > > >> > >    passing to brokers periodically. However,
> > >> this will
> > >> >> > >> >> require
> > >> >> > >> >> > > > brokers to
> > >> >> > >> >> > > > > >> > >    maintain certain counts of secretKeys.
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > > (Alternative 1) Have controller generate
> > >> delegation
> > >> >> > token.
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > >    1. Client authenticates with a broker.
> > >> >> > >> >> > > > > >> > >    2. Once a client is authenticated, it
> will
> > >> make a
> > >> >> > broker
> > >> >> > >> >> side
> > >> >> > >> >> > > > call to
> > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > >> >> > >> >> > > > > >> > >    3. Broker forwards the request to
> > controller.
> > >> >> > >> >> > > > > >> > >    4. Controller generates a DT and
> broadcasts
> > >> to all
> > >> >> > >> >> brokers.
> > >> >> > >> >> > > > > >> > >    5. Broker stores this token in its memory
> > >> cache.
> > >> >> > >> >> > > > > >> > >    6. Controller responds to broker’s DT
> req.
> > >> >> > >> >> > > > > >> > >    7. Broker returns the token to Client.
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > >    1. We will have to add new APIs to
> support
> > >> >> > controller
> > >> >> > >> >> pushing
> > >> >> > >> >> > > > tokens
> > >> >> > >> >> > > > > >> > to
> > >> >> > >> >> > > > > >> > >    brokers on top of the minimal APIs that
> are
> > >> >> > currently
> > >> >> > >> >> > > proposed.
> > >> >> > >> >> > > > > >> > >    2. We will also have to add APIs to
> support
> > >> the
> > >> >> > >> >> bootstrapping
> > >> >> > >> >> > > > case,
> > >> >> > >> >> > > > > >> > i.e,
> > >> >> > >> >> > > > > >> > >    when a new broker comes up it will have
> to
> > >> get all
> > >> >> > >> >> delegation
> > >> >> > >> >> > > > tokens
> > >> >> > >> >> > > > > >> > from
> > >> >> > >> >> > > > > >> > >    the controller.
> > >> >> > >> >> > > > > >> > >    3. In catastrophic failures where all
> > brokers
> > >> go
> > >> >> > down,
> > >> >> > >> >> the
> > >> >> > >> >> > > > tokens will
> > >> >> > >> >> > > > > >> > >    be lost even if servers are restarted as
> > >> tokens
> > >> >> > are not
> > >> >> > >> >> > > > persisted
> > >> >> > >> >> > > > > >> > anywhere.
> > >> >> > >> >> > > > > >> > >    If this happens, then there are more
> > important
> > >> >> > things to
> > >> >> > >> >> > > worry
> > >> >> > >> >> > > > about
> > >> >> > >> >> > > > > >> > and
> > >> >> > >> >> > > > > >> > >    maybe it is better to re-authenticate.
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > > (Alternative 2) Do not distribute DT to
> other
> > >> brokers
> > >> >> > at
> > >> >> > >> >> all.
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > >    1. Client authenticates with a broker.
> > >> >> > >> >> > > > > >> > >    2. Once a client is authenticated, it
> will
> > >> make a
> > >> >> > broker
> > >> >> > >> >> side
> > >> >> > >> >> > > > call to
> > >> >> > >> >> > > > > >> > >    issue a delegation token.
> > >> >> > >> >> > > > > >> > >    3. The broker generates DT of form,
> [hmac +
> > >> (owner,
> > >> >> > >> >> renewer,
> > >> >> > >> >> > > > > >> > >    maxLifeTime, id, hmac, expirationTime)]
> and
> > >> passes
> > >> >> > back
> > >> >> > >> >> this
> > >> >> > >> >> > > > DT to
> > >> >> > >> >> > > > > >> > client.
> > >> >> > >> >> > > > > >> > >    hmac is generated via {HMAC-SHA256(owner,
> > >> renewer,
> > >> >> > >> >> > > > maxLifeTime, id,
> > >> >> > >> >> > > > > >> > hmac,
> > >> >> > >> >> > > > > >> > >    expirationTime) using SecretKey}. Note
> that
> > >> all
> > >> >> > brokers
> > >> >> > >> >> have
> > >> >> > >> >> > > > this
> > >> >> > >> >> > > > > >> > SecretKey.
> > >> >> > >> >> > > > > >> > >    4. Client then goes to any broker and to
> > >> >> > authenticate
> > >> >> > >> >> sends
> > >> >> > >> >> > > > the DT.
> > >> >> > >> >> > > > > >> > >    Broker recalculates hmac using (owner,
> > >> renewer,
> > >> >> > >> >> maxLifeTime,
> > >> >> > >> >> > > > id, hmac,
> > >> >> > >> >> > > > > >> > >    expirationTime) info from DT and its
> > >> SecretKey. If
> > >> >> > it
> > >> >> > >> >> matches
> > >> >> > >> >> > > > with
> > >> >> > >> >> > > > > >> > hmac of
> > >> >> > >> >> > > > > >> > >    DT, client is authenticated. Yes, it will
> > do
> > >> other
> > >> >> > >> >> obvious
> > >> >> > >> >> > > > checks of
> > >> >> > >> >> > > > > >> > >    timestamp expiry and such.
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > > Note that secret key will be generated by
> > >> controller
> > >> >> > and
> > >> >> > >> >> passed
> > >> >> > >> >> > > to
> > >> >> > >> >> > > > > >> > brokers
> > >> >> > >> >> > > > > >> > > periodically.
> > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > >    1. How to delete a DT? Yes, that is a
> > downside
> > >> >> > here.
> > >> >> > >> >> However,
> > >> >> > >> >> > > > this can
> > >> >> > >> >> > > > > >> > >    be handled with brokers maintaining a
> > >> blacklist of
> > >> >> > DTs,
> > >> >> > >> >> DTs
> > >> >> > >> >> > > > from this
> > >> >> > >> >> > > > > >> > list
> > >> >> > >> >> > > > > >> > >    can be removed after expiry.
> > >> >> > >> >> > > > > >> > >    2. In catastrophic failures where all
> > brokers
> > >> go
> > >> >> > down,
> > >> >> > >> >> the
> > >> >> > >> >> > > > tokens will
> > >> >> > >> >> > > > > >> > >    be lost even if servers are restarted as
> > >> tokens
> > >> >> > are not
> > >> >> > >> >> > > > persisted
> > >> >> > >> >> > > > > >> > anywhere.
> > >> >> > >> >> > > > > >> > >    If this happens, then there are more
> > important
> > >> >> > things to
> > >> >> > >> >> > > worry
> > >> >> > >> >> > > > about
> > >> >> > >> >> > > > > >> > and
> > >> >> > >> >> > > > > >> > >    maybe it is better to re-authenticate.
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth
> > >> Brahmbhatt <
> > >> >> > >> >> > > > > >> > > pbrahmbhatt@hortonworks.com> wrote:
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > >> Hi,
> > >> >> > >> >> > > > > >> > >>
> > >> >> > >> >> > > > > >> > >> I have filed KIP-48 so we can offer hadoop
> > like
> > >> >> > delegation
> > >> >> > >> >> > > > tokens in
> > >> >> > >> >> > > > > >> > >> kafka. You can review the design
> > >> >> > >> >> > > > > >> > >>
> > >> >> > >> >> > > > > >> >
> > >> >> > >> >> > > >
> > >> >> > >> >> > >
> > >> >> > >> >>
> > >> >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > >> >> > >> >> > > > > >> > .
> > >> >> > >> >> > > > > >> > >> This KIP depends on KIP-43 and we have also
> > >> >> > discussed an
> > >> >> > >> >> > > > alternative to
> > >> >> > >> >> > > > > >> > >> proposed design here<
> > >> >> > >> >> > > > > >> > >>
> > >> >> > >> >> > > > > >> >
> > >> >> > >> >> > > >
> > >> >> > >> >> > >
> > >> >> > >> >>
> > >> >> >
> > >>
> >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > >> >> > >> >> > > > > >> > >> >.
> > >> >> > >> >> > > > > >> > >>
> > >> >> > >> >> > > > > >> > >> Thanks
> > >> >> > >> >> > > > > >> > >> Parth
> > >> >> > >> >> > > > > >> > >>
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > > --
> > >> >> > >> >> > > > > >> > >
> > >> >> > >> >> > > > > >> > > Regards,
> > >> >> > >> >> > > > > >> > > Ashish
> > >> >> > >> >> > > > > >> >
> > >> >> > >> >> > > >
> > >> >> > >> >> > >
> > >> >> > >> >> > >
> > >> >> > >> >> > >
> > >> >> > >> >> > > --
> > >> >> > >> >> > >
> > >> >> > >> >> > > Regards,
> > >> >> > >> >> > > Ashish
> > >> >> > >> >> > >
> > >> >> > >> >>
> > >> >> >
> > >>
> >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Ismael Juma <is...@juma.me.uk>]

Hi Parth,

Thanks for the KIP. I only started reviewing this and may have additional
questions later. The immediate question that came to mind is our choice of
"DIGEST-MD5" even though it's marked as OBSOLETE in the IANA Registry of
SASL mechanisms and the original RFC (2831) has been moved to Historic
status:

https://tools.ietf.org/html/rfc6331
http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml

What is the reasoning behind that choice?

Thanks,
Ismael

On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <gw...@confluent.io> wrote:

> Also comments inline :)
>
> > * I want to emphasize that even though delegation tokens are a Hadoop
> > innovation, I feel very strongly about not adding dependency on Hadoop
> > when implementing delegation tokens for Kafka. The KIP doesn't imply
> > such dependency, but if you can clarify...
> >
> >
> > *No hadoop dependency.*
>
> Yay! Just add this to the KIP so no one will read the KIP and panic
> three weeks before the next release...
>
> > * Can we get delegation token at any time after authenticating? only
> > immediately after?
> >
> >
> > *As long as you are authenticated you can get delegation tokens. We need
> to
> > discuss if a client authenticated using delegation token, can also
> acquire
> > delegation token again or not. Also there is the question of do we allow
> > anyone to acquire delegation token or we want specific ACLs (I think its
> an
> > overkill.)*
>
> I agree that ACLs is an overkill.
>
> I think we are debating two options: Either require Kerberos auth for
> renewal or require non-owners to renew.
> I *think* the latter is simpler (it basically require a "job master"
> to take responsibility for the renewal, it will have its own identity
> anyway and I think this is the correct design pattern anyway. For
> storm, I'd expect Nimbus to coordinate renewals?), but it is hard to
> debate simplicity without looking at the code changes required. If you
> have a draft of how the "require Kerberos" will look in Kafka code,
> I'll be happy to take a look.
>
> > * My understanding is that tokens will propagate via ZK but without
> > additional changes to UpdateMetadata protocol, correct? Clients
> > currently don't retry on SASL auth failure (IIRC), but since the
> > tokens propagate between brokers asynch, we will need to retry a bit
> > to avoid clients failing auth due to timing issues.
> >
> > *I am considering 2 alternatives right now. The current documented
> approach
> > is zookeeper based and it does not require any changes to UpdateMetadata
> > protocol. An alternative approach can remove zookeeper dependency as well
> > but we can discuss that in KIP discussion call.*
>
> Oooh! Sounds interesting. Do you want to ping Jun to arrange a call?
>
> > * I liked Ashish's suggestion of having just the controller issue the
> > delegation tokens, to avoid syncing a shared secret. Not sure if we
> > want to continue the discussion here or on the wiki. I think that we
> > can decouple the problem of "token distribution" from "shared secret
> > distribution" and use the controller as the only token generator to
> > solve the second issue, while still using ZK async to distribute
> > tokens.
> >
> >
> > *As mentioned in the previous Email I am fine with that approach as long
> as
> > we agree that the extra complexity of adding/updating APIS adds enough
> > value. The advantage with the controller approach is secret rotation can
> be
> > automated,frequent and would not require deployment. *
>
> Can you detail the extra complexity (or point me to the email I
> missed?) - which Apis are required?
> As far as I can tell, clients can already find the controller from
> metadata. I'm a bit more concerned about controller load.
>
> >
> > * While I like the idea of forcing kerberos auth for renwal, I think
> > it mixes the transport layer the the request content in a pretty ugly
> > way. Perhaps limiting renewer to non-owner is better.
> >
> > *I feel this is a necessary evil. While this will make the kafka code
> > pretty straight forward , forcing  renewer to non-owner pushes the code
> > ugliness to client and makes it even harder to integrate.  *
>
> As mentioned before, I don't think the "renewal by other" approach is
> that ugly for the clients we expect to use delegation tokens since
> they will have an app-master of some sort who requested the token to
> begin with.
>
> >
> > The response for my question on how multiple identities will be
> > handled wasn't super clear to me - AFAIK, we don't authenticate each
> > request, we authenticate connections.
> >
> > *We authenticate connections, and only when they are being established.
> Let
> > me try to phrase this as a question, in absence of delegation tokens if
> we
> > had to support the use case using user TGT's how would we do it? My point
> > was it would be no different with delegation tokens. The use case you are
> > describing seems more like impersonation.*
>
> Yeah, I thought that one of the things that delegation tokens handled.
> Maybe I got it wrong :)
>
> Thanks for the detailed answers.
>
> Gwen
>
>
> > Thanks
> > Parth
> >
> > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <gw...@confluent.io>
> wrote:
> >
> >> Hi Parth and Harsha,
> >>
> >> Few more comments:
> >>
> >> * The API / RequestResponse section doesn't seem to have good
> >> description of the changes to the Kafka Protocol. Sounds like you are
> >> proposing new DelegationTokenRequest and RenewTokenRequest (and
> >> matching responses), without detailing the contents of the requests
> >> and responses? Or rather, you show the class interface, but not the
> >> underlying protocol. This could be seen as an implementation detail,
> >> but since the binary protocol is what we provide to non-Java clients,
> >> we need to show the changes there.
> >>
> >> * getDelegationToken sounds like delegationTokenRequestHandler? Is it
> >> planned to be part of KafkaApi? or Client? Its unclear...
> >>
> >> * I want to emphasize that even though delegation tokens are a Hadoop
> >> innovation, I feel very strongly about not adding dependency on Hadoop
> >> when implementing delegation tokens for Kafka. The KIP doesn't imply
> >> such dependency, but if you can clarify...
> >>
> >> * Can we get delegation token at any time after authenticating? only
> >> immediately after?
> >>
> >> * My understanding is that tokens will propagate via ZK but without
> >> additional changes to UpdateMetadata protocol, correct? Clients
> >> currently don't retry on SASL auth failure (IIRC), but since the
> >> tokens propagate between brokers asynch, we will need to retry a bit
> >> to avoid clients failing auth due to timing issues.
> >>
> >> * Strongly agreeing on clients not touching ZK directly :)
> >>
> >> * I liked Ashish's suggestion of having just the controller issue the
> >> delegation tokens, to avoid syncing a shared secret. Not sure if we
> >> want to continue the discussion here or on the wiki. I think that we
> >> can decouple the problem of "token distribution" from "shared secret
> >> distribution" and use the controller as the only token generator to
> >> solve the second issue, while still using ZK async to distribute
> >> tokens.
> >>
> >> * I am also uncomfortable with infinite lifetime of tokens (and hoped
> >> to hear from others in the community) - but having the option and
> >> documenting it as less secure, allows users to configure their system
> >> as the wish.
> >>
> >> * While I like the idea of forcing kerberos auth for renwal, I think
> >> it mixes the transport layer the the request content in a pretty ugly
> >> way. Perhaps limiting renewer to non-owner is better.
> >>
> >> Things I'd still like to see:
> >>
> >> * More detailed explanation on what we plan to do for the java clients
> >> specifically - new configuration? new APIs?
> >> The response for my question on how multiple identities will be
> >> handled wasn't super clear to me - AFAIK, we don't authenticate each
> >> request, we authenticate connections.
> >>
> >> * Alternatives: Delegation tokens are only used in the Hadoop
> >> ecosystem. I'm wondering if there are alternatives in other ecosystems
> >> (Mesos? Tachyon? Cassandra?) and whether there are some advantages
> >> there.
> >>
> >> Gwen
> >>
> >> On Thu, May 12, 2016 at 1:05 PM, Harsha <ka...@harsha.io> wrote:
> >> > Hi Gwen,
> >> >            Can you look at Parth's last reply. Does it answer your
> >> >            concerns.
> >> >
> >> > Thanks,
> >> > Harsha
> >> >
> >> > On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt wrote:
> >> >> Thanks for reviewing Gwen. The wiki already has details on token
> >> >> expiration
> >> >> under token acquisition process
> >> >> <
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> >> >.
> >> >> Current proposal is that tokens will expire based on a server side
> >> >> configuration (default 24 hours) unless renewed. Renewal is only
> allowed
> >> >> until the max life time of token. Alternatively we could also make
> that
> >> >> an
> >> >> optional param and the server side default can serve as the upper
> bound.
> >> >>
> >> >> To your second point it will be done exactly the same way we would
> >> >> support
> >> >> multiple keytabs. The calling client will have to put the tokens it
> >> wants
> >> >> to use in the subject instance and call produce/consume inside
> >> >> subject.doas. Each caller will have to keep track of its own
> subject. I
> >> >> will have to look at the code to see if we support this feature right
> >> now
> >> >> but my understanding is delegation token shouldn't need any special
> >> >> treatment as its just another type of Credential in the subject.
> >> >>
> >> >> I would also like to know what is your opinion about infinite renewal
> >> (my
> >> >> recommendation is to not support this), tokens renewing them self(my
> >> >> recommendation is to not support this) and most importantly your
> choice
> >> >> between the alternatives listed on this thread
> >> >> <
> >>
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> >> >
> >> >> ( I am leaning towards the alternative-2 minus controller
> distributing
> >> >> secret). Thanks again for reviewing.
> >> >>
> >> >> Thanks
> >> >> Parth
> >> >>
> >> >>
> >> >>
> >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <gw...@confluent.io>
> wrote:
> >> >>
> >> >> > Harsha,
> >> >> >
> >> >> > I was thinking of the Rest Proxy. I didn't see your design yet,
> but in
> >> >> > our proxy, we have a set of producers, which will serve multiple
> users
> >> >> > going through the proxy. Since these users will have different
> >> >> > privileges, they'll need to authenticate separately, and can't
> share a
> >> >> > token.
> >> >> >
> >> >> > Am I missing anything?
> >> >> >
> >> >> > Gwen
> >> >> >
> >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io> wrote:
> >> >> > > Gwen,
> >> >> > >            On your second point. Can you describe a usecase where
> >> >> > >            mutliple clients ended up sharing a producer and even
> if
> >> they
> >> >> > >            do why can't they not use single token that producer
> >> >> > >            captures. Why would we need multiple clients with
> >> different
> >> >> > >            tokens sharing a single instance of producer.  Also in
> >> this
> >> >> > >            case other clients have access all the tokens no?
> >> >> > >
> >> >> > > Thanks,
> >> >> > > Harsha
> >> >> > >
> >> >> > >
> >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote:
> >> >> > >> Sorry for the delay:
> >> >> > >>
> >> >> > >> Two questions that we didn't see in the wiki:
> >> >> > >> 1. Is there an expiration for delegation tokens? Renewal? How
> do we
> >> >> > >> revoke them?
> >> >> > >> 2. If we want to use delegation tokens for "do-as" (say, submit
> >> Storm
> >> >> > >> job as my user), we will need a producer for every job (we can't
> >> share
> >> >> > >> them between multiple jobs running on same node), since we only
> >> >> > >> authenticate when connecting. Is there a plan to change this for
> >> >> > >> delegation tokens, in order to allow multiple users with
> different
> >> >> > >> tokens to share a client?
> >> >> > >>
> >> >> > >> Gwen
> >> >> > >>
> >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
> >> >> > >> <br...@gmail.com> wrote:
> >> >> > >> > Bumping this up one more time, can other committers review?
> >> >> > >> >
> >> >> > >> > Thanks
> >> >> > >> > Parth
> >> >> > >> >
> >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <ka...@harsha.io>
> wrote:
> >> >> > >> >
> >> >> > >> >> Parth,
> >> >> > >> >>           Overall current design looks good to me. I am +1 on
> >> the
> >> >> > KIP.
> >> >> > >> >>
> >> >> > >> >> Gwen , Jun can you review this as well.
> >> >> > >> >>
> >> >> > >> >> -Harsha
> >> >> > >> >>
> >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote:
> >> >> > >> >> > Thanks for review Jitendra.
> >> >> > >> >> >
> >> >> > >> >> > I don't like the idea of infinite lifetime but I see the
> >> Streaming
> >> >> > use
> >> >> > >> >> > case. Even for Streaming use case I was hoping there will
> be
> >> some
> >> >> > notion
> >> >> > >> >> > of
> >> >> > >> >> > master/driver that can get new delegation tokens at fixed
> >> interval
> >> >> > and
> >> >> > >> >> > distribute to workers. If that is not the case for we can
> >> discuss
> >> >> > >> >> > delegation tokens renewing them self and the security
> >> implications
> >> >> > of the
> >> >> > >> >> > same.
> >> >> > >> >> >
> >> >> > >> >> > I did not want clients to fetch tokens from zookeeper,
> >> overall I
> >> >> > think
> >> >> > >> >> > its
> >> >> > >> >> > better if clients don't rely on our metadata store and I
> >> think we
> >> >> > are
> >> >> > >> >> > moving in that direction with all the KIP-4 improvements.
> I
> >> chose
> >> >> > >> >> > zookeeper as in this case the client will still just talk
> to
> >> >> > broker , its
> >> >> > >> >> > the brokers that will use zookeeper which we already do
> for a
> >> lot
> >> >> > of
> >> >> > >> >> > other
> >> >> > >> >> > usecases + ease of development + and the ability so tokens
> >> will
> >> >> > survive
> >> >> > >> >> > even a rolling restart/cluster failure. if a majority
> agrees
> >> the
> >> >> > added
> >> >> > >> >> > complexity to have controller forwarding keys to all
> broker is
> >> >> > justified
> >> >> > >> >> > as
> >> >> > >> >> > it provides tighter security , I am fine with that option
> too.
> >> >> > >> >> >
> >> >> > >> >> > Given zookeeper does not support SSL we can not store
> master
> >> keys
> >> >> > in
> >> >> > >> >> > zookeeper as master keys will be exposed on wire. To
> support
> >> >> > rotation
> >> >> > >> >> > without affecting current clients is something I need to
> put
> >> more
> >> >> > thought
> >> >> > >> >> > in. My current proposal assumes the rotation will
> invalidate
> >> all
> >> >> > current
> >> >> > >> >> > tokens.
> >> >> > >> >> >
> >> >> > >> >> > I request committers to also review and post their comments
> >> so we
> >> >> > can
> >> >> > >> >> > make
> >> >> > >> >> > progress on this KIP.
> >> >> > >> >> >
> >> >> > >> >> > Thanks
> >> >> > >> >> > Parth
> >> >> > >> >> >
> >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <
> >> asingh@cloudera.com
> >> >> > >
> >> >> > >> >> > wrote:
> >> >> > >> >> >
> >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <
> kafka@harsha.io>
> >> >> > wrote:
> >> >> > >> >> > >
> >> >> > >> >> > > > Unifying the two discussion threads on this KIP.
> >> >> > >> >> > > >
> >> >> > >> >> > > > Here is the response from Jitendra
> >> >> > >> >> > > >
> >> >> > >> >> > > > "The need for a large number of clients that are
> running
> >> all
> >> >> > over the
> >> >> > >> >> > > > cluster that authenticate with Kafka brokers, is very
> >> similar
> >> >> > to the
> >> >> > >> >> > > > Hadoop use case of large number of tasks running across
> >> the
> >> >> > cluster
> >> >> > >> >> that
> >> >> > >> >> > > > need authentication to Hdfs Namenode. Therefore, the
> >> >> > delegation token
> >> >> > >> >> > > > approach does seem like a good fit for this use case
> as we
> >> >> > have seen
> >> >> > >> >> it
> >> >> > >> >> > > > working at large scale in HDFS and YARN.
> >> >> > >> >> > > >
> >> >> > >> >> > > >   The proposed design is very much inline with Hadoop
> >> >> > approach. A few
> >> >> > >> >> > > >   comments:
> >> >> > >> >> > > >
> >> >> > >> >> > > > 1) Why do you guys want to allow infinite renewable
> >> lifetime
> >> >> > for a
> >> >> > >> >> > > > token? HDFS restricts a token to a max life time
> (default
> >> 7
> >> >> > days).  A
> >> >> > >> >> > > > token's vulnerability is believed to increase with
> time.
> >> >> > >> >> > > >
> >> >> > >> >> > > I agree that having infinite lifetime might not be the
> best
> >> idea.
> >> >> > >> >> > >
> >> >> > >> >> > > >
> >> >> > >> >> > > > 2) As I understand the tokens are stored in zookeeper
> as
> >> well,
> >> >> > and
> >> >> > >> >> can
> >> >> > >> >> > > > be updated there. This is clever as it can allow
> >> replacing the
> >> >> > tokens
> >> >> > >> >> > > > once they run out of max life time, and clients can
> >> download
> >> >> > new
> >> >> > >> >> tokens
> >> >> > >> >> > > > from zookeeper. It shouldn't be a big load on zookeeper
> >> as a
> >> >> > client
> >> >> > >> >> will
> >> >> > >> >> > > > need to get a new token once in several days. In this
> >> approach
> >> >> > you
> >> >> > >> >> don't
> >> >> > >> >> > > > need infinite lifetime on the token even for long
> running
> >> >> > clients.
> >> >> > >> >> > > >
> >> >> > >> >> > > > 3) The token password are generated using a master key.
> >> The
> >> >> > master
> >> >> > >> >> key
> >> >> > >> >> > > > should also be periodically changed. In Hadoop, the
> >> default
> >> >> > renewal
> >> >> > >> >> > > > period is 1 day.?
> >> >> > >> >> > > >
> >> >> > >> >> > > IIUC, this will require brokers maintaining a list of X
> most
> >> >> > recent
> >> >> > >> >> master
> >> >> > >> >> > > keys. This list will have to be persisted somewhere, as
> if a
> >> >> > broker
> >> >> > >> >> goes
> >> >> > >> >> > > down it will have to get that list again and storing
> master
> >> keys
> >> >> > on ZK
> >> >> > >> >> is
> >> >> > >> >> > > not the best idea. However, if a broker goes down then we
> >> have
> >> >> > much
> >> >> > >> >> bigger
> >> >> > >> >> > > issue to deal with and client can always re-authenticate
> is
> >> such
> >> >> > >> >> events.
> >> >> > >> >> > >
> >> >> > >> >> > > Did you happen to take a look at other alternatives this
> >> list has
> >> >> > >> >> > > suggested?
> >> >> > >> >> > >
> >> >> > >> >> > > >
> >> >> > >> >> > > > Thanks for a thorough proposal, great work!"
> >> >> > >> >> > > >
> >> >> > >> >> > > >
> >> >> > >> >> > > >
> >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
> >> >> > >> >> > > > > Makes sense to me. Thanks!
> >> >> > >> >> > > > >
> >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <
> kafka@harsha.io
> >> >
> >> >> > wrote:
> >> >> > >> >> > > > > > It doesn't need any release vehicle but still the
> >> work can
> >> >> > move
> >> >> > >> >> > > > forward.
> >> >> > >> >> > > > > > If anyone is interested in the KIP please do the
> >> review and
> >> >> > >> >> provide
> >> >> > >> >> > > the
> >> >> > >> >> > > > > > comments.
> >> >> > >> >> > > > > >
> >> >> > >> >> > > > > > -Harsha
> >> >> > >> >> > > > > >
> >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma
> wrote:
> >> >> > >> >> > > > > >> I agree that it would be good to have more time to
> >> review
> >> >> > and
> >> >> > >> >> > > discuss
> >> >> > >> >> > > > > >> KIP-48.
> >> >> > >> >> > > > > >>
> >> >> > >> >> > > > > >> Ismael
> >> >> > >> >> > > > > >>
> >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <
> >> >> > >> >> gwen@confluent.io>
> >> >> > >> >> > > > wrote:
> >> >> > >> >> > > > > >>
> >> >> > >> >> > > > > >> > Hi Team,
> >> >> > >> >> > > > > >> >
> >> >> > >> >> > > > > >> > Since KIP-48 depends on KIP-43, which is
> already a
> >> bit
> >> >> > of a
> >> >> > >> >> risk
> >> >> > >> >> > > for
> >> >> > >> >> > > > > >> > the next release - any chance we can delay
> >> delegation
> >> >> > tokens
> >> >> > >> >> to
> >> >> > >> >> > > > Kafka
> >> >> > >> >> > > > > >> > 0.10.1?
> >> >> > >> >> > > > > >> > With the community working on a release every 3
> >> month,
> >> >> > this
> >> >> > >> >> is not
> >> >> > >> >> > > > a huge
> >> >> > >> >> > > > > >> > delay.
> >> >> > >> >> > > > > >> >
> >> >> > >> >> > > > > >> > Gwen
> >> >> > >> >> > > > > >> >
> >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh <
> >> >> > >> >> > > asingh@cloudera.com>
> >> >> > >> >> > > > wrote:
> >> >> > >> >> > > > > >> > > Parth,
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > > Thanks again for the awesome write up.
> Following
> >> our
> >> >> > >> >> discussion
> >> >> > >> >> > > > from the
> >> >> > >> >> > > > > >> > > JIRA, I think it will be easier to compare
> >> various
> >> >> > >> >> alternatives
> >> >> > >> >> > > > if they
> >> >> > >> >> > > > > >> > are
> >> >> > >> >> > > > > >> > > listed together. I am stating below a few
> >> >> > alternatives along
> >> >> > >> >> > > with
> >> >> > >> >> > > > a the
> >> >> > >> >> > > > > >> > > current proposal.
> >> >> > >> >> > > > > >> > > (Current proposal) Store Delegation Token, DT,
> >> on ZK.
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > >    1. Client authenticates with a broker.
> >> >> > >> >> > > > > >> > >    2. Once a client is authenticated, it will
> >> make a
> >> >> > broker
> >> >> > >> >> side
> >> >> > >> >> > > > call to
> >> >> > >> >> > > > > >> > >    issue a delegation token.
> >> >> > >> >> > > > > >> > >    3. The broker generates a shared secret
> based
> >> on
> >> >> > >> >> > > HMAC-SHA256(a
> >> >> > >> >> > > > > >> > >    Password/Secret shared between all brokers,
> >> >> > randomly
> >> >> > >> >> > > generated
> >> >> > >> >> > > > > >> > tokenId).
> >> >> > >> >> > > > > >> > >    4. Broker stores this token in its in
> memory
> >> cache.
> >> >> > >> >> Broker
> >> >> > >> >> > > > also stores
> >> >> > >> >> > > > > >> > >    the DelegationToken without the hmac in the
> >> >> > zookeeper.
> >> >> > >> >> > > > > >> > >    5. All brokers will have a cache backed by
> >> >> > zookeeper so
> >> >> > >> >> they
> >> >> > >> >> > > > will all
> >> >> > >> >> > > > > >> > >    get notified whenever a new token is
> >> generated and
> >> >> > they
> >> >> > >> >> will
> >> >> > >> >> > > > update
> >> >> > >> >> > > > > >> > their
> >> >> > >> >> > > > > >> > >    local cache whenever token state changes.
> >> >> > >> >> > > > > >> > >    6. Broker returns the token to Client.
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > > Probable issues and fixes
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > >    1. Probable race condition, client tries to
> >> >> > authenticate
> >> >> > >> >> with
> >> >> > >> >> > > > a broker
> >> >> > >> >> > > > > >> > >    that is yet to be updated with the newly
> >> generated
> >> >> > DT.
> >> >> > >> >> This
> >> >> > >> >> > > can
> >> >> > >> >> > > > > >> > probably be
> >> >> > >> >> > > > > >> > >    dealt with making dtRequest block until all
> >> >> > brokers have
> >> >> > >> >> > > > updated
> >> >> > >> >> > > > > >> > their DT
> >> >> > >> >> > > > > >> > >    cache. Zk barrier or similar mechanism can
> be
> >> used.
> >> >> > >> >> However,
> >> >> > >> >> > > > all such
> >> >> > >> >> > > > > >> > >    mechanisms will increase complexity.
> >> >> > >> >> > > > > >> > >    2. Using a static secret key from config
> >> file. Will
> >> >> > >> >> require
> >> >> > >> >> > > yet
> >> >> > >> >> > > > > >> > another
> >> >> > >> >> > > > > >> > >    config and uses a static secret key. It is
> >> advised
> >> >> > to
> >> >> > >> >> rotate
> >> >> > >> >> > > > secret
> >> >> > >> >> > > > > >> > keys
> >> >> > >> >> > > > > >> > >    periodically. This can be avoided with
> >> controller
> >> >> > >> >> generating
> >> >> > >> >> > > > > >> > secretKey and
> >> >> > >> >> > > > > >> > >    passing to brokers periodically. However,
> >> this will
> >> >> > >> >> require
> >> >> > >> >> > > > brokers to
> >> >> > >> >> > > > > >> > >    maintain certain counts of secretKeys.
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > > (Alternative 1) Have controller generate
> >> delegation
> >> >> > token.
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > >    1. Client authenticates with a broker.
> >> >> > >> >> > > > > >> > >    2. Once a client is authenticated, it will
> >> make a
> >> >> > broker
> >> >> > >> >> side
> >> >> > >> >> > > > call to
> >> >> > >> >> > > > > >> > >    issue a delegation token.
> >> >> > >> >> > > > > >> > >    3. Broker forwards the request to
> controller.
> >> >> > >> >> > > > > >> > >    4. Controller generates a DT and broadcasts
> >> to all
> >> >> > >> >> brokers.
> >> >> > >> >> > > > > >> > >    5. Broker stores this token in its memory
> >> cache.
> >> >> > >> >> > > > > >> > >    6. Controller responds to broker’s DT req.
> >> >> > >> >> > > > > >> > >    7. Broker returns the token to Client.
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > > Probable issues and fixes
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > >    1. We will have to add new APIs to support
> >> >> > controller
> >> >> > >> >> pushing
> >> >> > >> >> > > > tokens
> >> >> > >> >> > > > > >> > to
> >> >> > >> >> > > > > >> > >    brokers on top of the minimal APIs that are
> >> >> > currently
> >> >> > >> >> > > proposed.
> >> >> > >> >> > > > > >> > >    2. We will also have to add APIs to support
> >> the
> >> >> > >> >> bootstrapping
> >> >> > >> >> > > > case,
> >> >> > >> >> > > > > >> > i.e,
> >> >> > >> >> > > > > >> > >    when a new broker comes up it will have to
> >> get all
> >> >> > >> >> delegation
> >> >> > >> >> > > > tokens
> >> >> > >> >> > > > > >> > from
> >> >> > >> >> > > > > >> > >    the controller.
> >> >> > >> >> > > > > >> > >    3. In catastrophic failures where all
> brokers
> >> go
> >> >> > down,
> >> >> > >> >> the
> >> >> > >> >> > > > tokens will
> >> >> > >> >> > > > > >> > >    be lost even if servers are restarted as
> >> tokens
> >> >> > are not
> >> >> > >> >> > > > persisted
> >> >> > >> >> > > > > >> > anywhere.
> >> >> > >> >> > > > > >> > >    If this happens, then there are more
> important
> >> >> > things to
> >> >> > >> >> > > worry
> >> >> > >> >> > > > about
> >> >> > >> >> > > > > >> > and
> >> >> > >> >> > > > > >> > >    maybe it is better to re-authenticate.
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > > (Alternative 2) Do not distribute DT to other
> >> brokers
> >> >> > at
> >> >> > >> >> all.
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > >    1. Client authenticates with a broker.
> >> >> > >> >> > > > > >> > >    2. Once a client is authenticated, it will
> >> make a
> >> >> > broker
> >> >> > >> >> side
> >> >> > >> >> > > > call to
> >> >> > >> >> > > > > >> > >    issue a delegation token.
> >> >> > >> >> > > > > >> > >    3. The broker generates DT of form, [hmac +
> >> (owner,
> >> >> > >> >> renewer,
> >> >> > >> >> > > > > >> > >    maxLifeTime, id, hmac, expirationTime)] and
> >> passes
> >> >> > back
> >> >> > >> >> this
> >> >> > >> >> > > > DT to
> >> >> > >> >> > > > > >> > client.
> >> >> > >> >> > > > > >> > >    hmac is generated via {HMAC-SHA256(owner,
> >> renewer,
> >> >> > >> >> > > > maxLifeTime, id,
> >> >> > >> >> > > > > >> > hmac,
> >> >> > >> >> > > > > >> > >    expirationTime) using SecretKey}. Note that
> >> all
> >> >> > brokers
> >> >> > >> >> have
> >> >> > >> >> > > > this
> >> >> > >> >> > > > > >> > SecretKey.
> >> >> > >> >> > > > > >> > >    4. Client then goes to any broker and to
> >> >> > authenticate
> >> >> > >> >> sends
> >> >> > >> >> > > > the DT.
> >> >> > >> >> > > > > >> > >    Broker recalculates hmac using (owner,
> >> renewer,
> >> >> > >> >> maxLifeTime,
> >> >> > >> >> > > > id, hmac,
> >> >> > >> >> > > > > >> > >    expirationTime) info from DT and its
> >> SecretKey. If
> >> >> > it
> >> >> > >> >> matches
> >> >> > >> >> > > > with
> >> >> > >> >> > > > > >> > hmac of
> >> >> > >> >> > > > > >> > >    DT, client is authenticated. Yes, it will
> do
> >> other
> >> >> > >> >> obvious
> >> >> > >> >> > > > checks of
> >> >> > >> >> > > > > >> > >    timestamp expiry and such.
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > > Note that secret key will be generated by
> >> controller
> >> >> > and
> >> >> > >> >> passed
> >> >> > >> >> > > to
> >> >> > >> >> > > > > >> > brokers
> >> >> > >> >> > > > > >> > > periodically.
> >> >> > >> >> > > > > >> > > Probable issues and fixes
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > >    1. How to delete a DT? Yes, that is a
> downside
> >> >> > here.
> >> >> > >> >> However,
> >> >> > >> >> > > > this can
> >> >> > >> >> > > > > >> > >    be handled with brokers maintaining a
> >> blacklist of
> >> >> > DTs,
> >> >> > >> >> DTs
> >> >> > >> >> > > > from this
> >> >> > >> >> > > > > >> > list
> >> >> > >> >> > > > > >> > >    can be removed after expiry.
> >> >> > >> >> > > > > >> > >    2. In catastrophic failures where all
> brokers
> >> go
> >> >> > down,
> >> >> > >> >> the
> >> >> > >> >> > > > tokens will
> >> >> > >> >> > > > > >> > >    be lost even if servers are restarted as
> >> tokens
> >> >> > are not
> >> >> > >> >> > > > persisted
> >> >> > >> >> > > > > >> > anywhere.
> >> >> > >> >> > > > > >> > >    If this happens, then there are more
> important
> >> >> > things to
> >> >> > >> >> > > worry
> >> >> > >> >> > > > about
> >> >> > >> >> > > > > >> > and
> >> >> > >> >> > > > > >> > >    maybe it is better to re-authenticate.
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth
> >> Brahmbhatt <
> >> >> > >> >> > > > > >> > > pbrahmbhatt@hortonworks.com> wrote:
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > >> Hi,
> >> >> > >> >> > > > > >> > >>
> >> >> > >> >> > > > > >> > >> I have filed KIP-48 so we can offer hadoop
> like
> >> >> > delegation
> >> >> > >> >> > > > tokens in
> >> >> > >> >> > > > > >> > >> kafka. You can review the design
> >> >> > >> >> > > > > >> > >>
> >> >> > >> >> > > > > >> >
> >> >> > >> >> > > >
> >> >> > >> >> > >
> >> >> > >> >>
> >> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> >> >> > >> >> > > > > >> > .
> >> >> > >> >> > > > > >> > >> This KIP depends on KIP-43 and we have also
> >> >> > discussed an
> >> >> > >> >> > > > alternative to
> >> >> > >> >> > > > > >> > >> proposed design here<
> >> >> > >> >> > > > > >> > >>
> >> >> > >> >> > > > > >> >
> >> >> > >> >> > > >
> >> >> > >> >> > >
> >> >> > >> >>
> >> >> >
> >>
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> >> >> > >> >> > > > > >> > >> >.
> >> >> > >> >> > > > > >> > >>
> >> >> > >> >> > > > > >> > >> Thanks
> >> >> > >> >> > > > > >> > >> Parth
> >> >> > >> >> > > > > >> > >>
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > > --
> >> >> > >> >> > > > > >> > >
> >> >> > >> >> > > > > >> > > Regards,
> >> >> > >> >> > > > > >> > > Ashish
> >> >> > >> >> > > > > >> >
> >> >> > >> >> > > >
> >> >> > >> >> > >
> >> >> > >> >> > >
> >> >> > >> >> > >
> >> >> > >> >> > > --
> >> >> > >> >> > >
> >> >> > >> >> > > Regards,
> >> >> > >> >> > > Ashish
> >> >> > >> >> > >
> >> >> > >> >>
> >> >> >
> >>
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Gwen Shapira <gw...@confluent.io>]

Also comments inline :)

> * I want to emphasize that even though delegation tokens are a Hadoop
> innovation, I feel very strongly about not adding dependency on Hadoop
> when implementing delegation tokens for Kafka. The KIP doesn't imply
> such dependency, but if you can clarify...
>
>
> *No hadoop dependency.*

Yay! Just add this to the KIP so no one will read the KIP and panic
three weeks before the next release...

> * Can we get delegation token at any time after authenticating? only
> immediately after?
>
>
> *As long as you are authenticated you can get delegation tokens. We need to
> discuss if a client authenticated using delegation token, can also acquire
> delegation token again or not. Also there is the question of do we allow
> anyone to acquire delegation token or we want specific ACLs (I think its an
> overkill.)*

I agree that ACLs is an overkill.

I think we are debating two options: Either require Kerberos auth for
renewal or require non-owners to renew.
I *think* the latter is simpler (it basically require a "job master"
to take responsibility for the renewal, it will have its own identity
anyway and I think this is the correct design pattern anyway. For
storm, I'd expect Nimbus to coordinate renewals?), but it is hard to
debate simplicity without looking at the code changes required. If you
have a draft of how the "require Kerberos" will look in Kafka code,
I'll be happy to take a look.

> * My understanding is that tokens will propagate via ZK but without
> additional changes to UpdateMetadata protocol, correct? Clients
> currently don't retry on SASL auth failure (IIRC), but since the
> tokens propagate between brokers asynch, we will need to retry a bit
> to avoid clients failing auth due to timing issues.
>
> *I am considering 2 alternatives right now. The current documented approach
> is zookeeper based and it does not require any changes to UpdateMetadata
> protocol. An alternative approach can remove zookeeper dependency as well
> but we can discuss that in KIP discussion call.*

Oooh! Sounds interesting. Do you want to ping Jun to arrange a call?

> * I liked Ashish's suggestion of having just the controller issue the
> delegation tokens, to avoid syncing a shared secret. Not sure if we
> want to continue the discussion here or on the wiki. I think that we
> can decouple the problem of "token distribution" from "shared secret
> distribution" and use the controller as the only token generator to
> solve the second issue, while still using ZK async to distribute
> tokens.
>
>
> *As mentioned in the previous Email I am fine with that approach as long as
> we agree that the extra complexity of adding/updating APIS adds enough
> value. The advantage with the controller approach is secret rotation can be
> automated,frequent and would not require deployment. *

Can you detail the extra complexity (or point me to the email I
missed?) - which Apis are required?
As far as I can tell, clients can already find the controller from
metadata. I'm a bit more concerned about controller load.

>
> * While I like the idea of forcing kerberos auth for renwal, I think
> it mixes the transport layer the the request content in a pretty ugly
> way. Perhaps limiting renewer to non-owner is better.
>
> *I feel this is a necessary evil. While this will make the kafka code
> pretty straight forward , forcing  renewer to non-owner pushes the code
> ugliness to client and makes it even harder to integrate.  *

As mentioned before, I don't think the "renewal by other" approach is
that ugly for the clients we expect to use delegation tokens since
they will have an app-master of some sort who requested the token to
begin with.

>
> The response for my question on how multiple identities will be
> handled wasn't super clear to me - AFAIK, we don't authenticate each
> request, we authenticate connections.
>
> *We authenticate connections, and only when they are being established. Let
> me try to phrase this as a question, in absence of delegation tokens if we
> had to support the use case using user TGT's how would we do it? My point
> was it would be no different with delegation tokens. The use case you are
> describing seems more like impersonation.*

Yeah, I thought that one of the things that delegation tokens handled.
Maybe I got it wrong :)

Thanks for the detailed answers.

Gwen


> Thanks
> Parth
>
> On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <gw...@confluent.io> wrote:
>
>> Hi Parth and Harsha,
>>
>> Few more comments:
>>
>> * The API / RequestResponse section doesn't seem to have good
>> description of the changes to the Kafka Protocol. Sounds like you are
>> proposing new DelegationTokenRequest and RenewTokenRequest (and
>> matching responses), without detailing the contents of the requests
>> and responses? Or rather, you show the class interface, but not the
>> underlying protocol. This could be seen as an implementation detail,
>> but since the binary protocol is what we provide to non-Java clients,
>> we need to show the changes there.
>>
>> * getDelegationToken sounds like delegationTokenRequestHandler? Is it
>> planned to be part of KafkaApi? or Client? Its unclear...
>>
>> * I want to emphasize that even though delegation tokens are a Hadoop
>> innovation, I feel very strongly about not adding dependency on Hadoop
>> when implementing delegation tokens for Kafka. The KIP doesn't imply
>> such dependency, but if you can clarify...
>>
>> * Can we get delegation token at any time after authenticating? only
>> immediately after?
>>
>> * My understanding is that tokens will propagate via ZK but without
>> additional changes to UpdateMetadata protocol, correct? Clients
>> currently don't retry on SASL auth failure (IIRC), but since the
>> tokens propagate between brokers asynch, we will need to retry a bit
>> to avoid clients failing auth due to timing issues.
>>
>> * Strongly agreeing on clients not touching ZK directly :)
>>
>> * I liked Ashish's suggestion of having just the controller issue the
>> delegation tokens, to avoid syncing a shared secret. Not sure if we
>> want to continue the discussion here or on the wiki. I think that we
>> can decouple the problem of "token distribution" from "shared secret
>> distribution" and use the controller as the only token generator to
>> solve the second issue, while still using ZK async to distribute
>> tokens.
>>
>> * I am also uncomfortable with infinite lifetime of tokens (and hoped
>> to hear from others in the community) - but having the option and
>> documenting it as less secure, allows users to configure their system
>> as the wish.
>>
>> * While I like the idea of forcing kerberos auth for renwal, I think
>> it mixes the transport layer the the request content in a pretty ugly
>> way. Perhaps limiting renewer to non-owner is better.
>>
>> Things I'd still like to see:
>>
>> * More detailed explanation on what we plan to do for the java clients
>> specifically - new configuration? new APIs?
>> The response for my question on how multiple identities will be
>> handled wasn't super clear to me - AFAIK, we don't authenticate each
>> request, we authenticate connections.
>>
>> * Alternatives: Delegation tokens are only used in the Hadoop
>> ecosystem. I'm wondering if there are alternatives in other ecosystems
>> (Mesos? Tachyon? Cassandra?) and whether there are some advantages
>> there.
>>
>> Gwen
>>
>> On Thu, May 12, 2016 at 1:05 PM, Harsha <ka...@harsha.io> wrote:
>> > Hi Gwen,
>> >            Can you look at Parth's last reply. Does it answer your
>> >            concerns.
>> >
>> > Thanks,
>> > Harsha
>> >
>> > On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt wrote:
>> >> Thanks for reviewing Gwen. The wiki already has details on token
>> >> expiration
>> >> under token acquisition process
>> >> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
>> >.
>> >> Current proposal is that tokens will expire based on a server side
>> >> configuration (default 24 hours) unless renewed. Renewal is only allowed
>> >> until the max life time of token. Alternatively we could also make that
>> >> an
>> >> optional param and the server side default can serve as the upper bound.
>> >>
>> >> To your second point it will be done exactly the same way we would
>> >> support
>> >> multiple keytabs. The calling client will have to put the tokens it
>> wants
>> >> to use in the subject instance and call produce/consume inside
>> >> subject.doas. Each caller will have to keep track of its own subject. I
>> >> will have to look at the code to see if we support this feature right
>> now
>> >> but my understanding is delegation token shouldn't need any special
>> >> treatment as its just another type of Credential in the subject.
>> >>
>> >> I would also like to know what is your opinion about infinite renewal
>> (my
>> >> recommendation is to not support this), tokens renewing them self(my
>> >> recommendation is to not support this) and most importantly your choice
>> >> between the alternatives listed on this thread
>> >> <
>> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
>> >
>> >> ( I am leaning towards the alternative-2 minus controller distributing
>> >> secret). Thanks again for reviewing.
>> >>
>> >> Thanks
>> >> Parth
>> >>
>> >>
>> >>
>> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <gw...@confluent.io> wrote:
>> >>
>> >> > Harsha,
>> >> >
>> >> > I was thinking of the Rest Proxy. I didn't see your design yet, but in
>> >> > our proxy, we have a set of producers, which will serve multiple users
>> >> > going through the proxy. Since these users will have different
>> >> > privileges, they'll need to authenticate separately, and can't share a
>> >> > token.
>> >> >
>> >> > Am I missing anything?
>> >> >
>> >> > Gwen
>> >> >
>> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io> wrote:
>> >> > > Gwen,
>> >> > >            On your second point. Can you describe a usecase where
>> >> > >            mutliple clients ended up sharing a producer and even if
>> they
>> >> > >            do why can't they not use single token that producer
>> >> > >            captures. Why would we need multiple clients with
>> different
>> >> > >            tokens sharing a single instance of producer.  Also in
>> this
>> >> > >            case other clients have access all the tokens no?
>> >> > >
>> >> > > Thanks,
>> >> > > Harsha
>> >> > >
>> >> > >
>> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote:
>> >> > >> Sorry for the delay:
>> >> > >>
>> >> > >> Two questions that we didn't see in the wiki:
>> >> > >> 1. Is there an expiration for delegation tokens? Renewal? How do we
>> >> > >> revoke them?
>> >> > >> 2. If we want to use delegation tokens for "do-as" (say, submit
>> Storm
>> >> > >> job as my user), we will need a producer for every job (we can't
>> share
>> >> > >> them between multiple jobs running on same node), since we only
>> >> > >> authenticate when connecting. Is there a plan to change this for
>> >> > >> delegation tokens, in order to allow multiple users with different
>> >> > >> tokens to share a client?
>> >> > >>
>> >> > >> Gwen
>> >> > >>
>> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
>> >> > >> <br...@gmail.com> wrote:
>> >> > >> > Bumping this up one more time, can other committers review?
>> >> > >> >
>> >> > >> > Thanks
>> >> > >> > Parth
>> >> > >> >
>> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <ka...@harsha.io> wrote:
>> >> > >> >
>> >> > >> >> Parth,
>> >> > >> >>           Overall current design looks good to me. I am +1 on
>> the
>> >> > KIP.
>> >> > >> >>
>> >> > >> >> Gwen , Jun can you review this as well.
>> >> > >> >>
>> >> > >> >> -Harsha
>> >> > >> >>
>> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote:
>> >> > >> >> > Thanks for review Jitendra.
>> >> > >> >> >
>> >> > >> >> > I don't like the idea of infinite lifetime but I see the
>> Streaming
>> >> > use
>> >> > >> >> > case. Even for Streaming use case I was hoping there will be
>> some
>> >> > notion
>> >> > >> >> > of
>> >> > >> >> > master/driver that can get new delegation tokens at fixed
>> interval
>> >> > and
>> >> > >> >> > distribute to workers. If that is not the case for we can
>> discuss
>> >> > >> >> > delegation tokens renewing them self and the security
>> implications
>> >> > of the
>> >> > >> >> > same.
>> >> > >> >> >
>> >> > >> >> > I did not want clients to fetch tokens from zookeeper,
>> overall I
>> >> > think
>> >> > >> >> > its
>> >> > >> >> > better if clients don't rely on our metadata store and I
>> think we
>> >> > are
>> >> > >> >> > moving in that direction with all the KIP-4 improvements.  I
>> chose
>> >> > >> >> > zookeeper as in this case the client will still just talk to
>> >> > broker , its
>> >> > >> >> > the brokers that will use zookeeper which we already do for a
>> lot
>> >> > of
>> >> > >> >> > other
>> >> > >> >> > usecases + ease of development + and the ability so tokens
>> will
>> >> > survive
>> >> > >> >> > even a rolling restart/cluster failure. if a majority agrees
>> the
>> >> > added
>> >> > >> >> > complexity to have controller forwarding keys to all broker is
>> >> > justified
>> >> > >> >> > as
>> >> > >> >> > it provides tighter security , I am fine with that option too.
>> >> > >> >> >
>> >> > >> >> > Given zookeeper does not support SSL we can not store master
>> keys
>> >> > in
>> >> > >> >> > zookeeper as master keys will be exposed on wire. To support
>> >> > rotation
>> >> > >> >> > without affecting current clients is something I need to put
>> more
>> >> > thought
>> >> > >> >> > in. My current proposal assumes the rotation will invalidate
>> all
>> >> > current
>> >> > >> >> > tokens.
>> >> > >> >> >
>> >> > >> >> > I request committers to also review and post their comments
>> so we
>> >> > can
>> >> > >> >> > make
>> >> > >> >> > progress on this KIP.
>> >> > >> >> >
>> >> > >> >> > Thanks
>> >> > >> >> > Parth
>> >> > >> >> >
>> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <
>> asingh@cloudera.com
>> >> > >
>> >> > >> >> > wrote:
>> >> > >> >> >
>> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <ka...@harsha.io>
>> >> > wrote:
>> >> > >> >> > >
>> >> > >> >> > > > Unifying the two discussion threads on this KIP.
>> >> > >> >> > > >
>> >> > >> >> > > > Here is the response from Jitendra
>> >> > >> >> > > >
>> >> > >> >> > > > "The need for a large number of clients that are running
>> all
>> >> > over the
>> >> > >> >> > > > cluster that authenticate with Kafka brokers, is very
>> similar
>> >> > to the
>> >> > >> >> > > > Hadoop use case of large number of tasks running across
>> the
>> >> > cluster
>> >> > >> >> that
>> >> > >> >> > > > need authentication to Hdfs Namenode. Therefore, the
>> >> > delegation token
>> >> > >> >> > > > approach does seem like a good fit for this use case as we
>> >> > have seen
>> >> > >> >> it
>> >> > >> >> > > > working at large scale in HDFS and YARN.
>> >> > >> >> > > >
>> >> > >> >> > > >   The proposed design is very much inline with Hadoop
>> >> > approach. A few
>> >> > >> >> > > >   comments:
>> >> > >> >> > > >
>> >> > >> >> > > > 1) Why do you guys want to allow infinite renewable
>> lifetime
>> >> > for a
>> >> > >> >> > > > token? HDFS restricts a token to a max life time (default
>> 7
>> >> > days).  A
>> >> > >> >> > > > token's vulnerability is believed to increase with time.
>> >> > >> >> > > >
>> >> > >> >> > > I agree that having infinite lifetime might not be the best
>> idea.
>> >> > >> >> > >
>> >> > >> >> > > >
>> >> > >> >> > > > 2) As I understand the tokens are stored in zookeeper as
>> well,
>> >> > and
>> >> > >> >> can
>> >> > >> >> > > > be updated there. This is clever as it can allow
>> replacing the
>> >> > tokens
>> >> > >> >> > > > once they run out of max life time, and clients can
>> download
>> >> > new
>> >> > >> >> tokens
>> >> > >> >> > > > from zookeeper. It shouldn't be a big load on zookeeper
>> as a
>> >> > client
>> >> > >> >> will
>> >> > >> >> > > > need to get a new token once in several days. In this
>> approach
>> >> > you
>> >> > >> >> don't
>> >> > >> >> > > > need infinite lifetime on the token even for long running
>> >> > clients.
>> >> > >> >> > > >
>> >> > >> >> > > > 3) The token password are generated using a master key.
>> The
>> >> > master
>> >> > >> >> key
>> >> > >> >> > > > should also be periodically changed. In Hadoop, the
>> default
>> >> > renewal
>> >> > >> >> > > > period is 1 day.?
>> >> > >> >> > > >
>> >> > >> >> > > IIUC, this will require brokers maintaining a list of X most
>> >> > recent
>> >> > >> >> master
>> >> > >> >> > > keys. This list will have to be persisted somewhere, as if a
>> >> > broker
>> >> > >> >> goes
>> >> > >> >> > > down it will have to get that list again and storing master
>> keys
>> >> > on ZK
>> >> > >> >> is
>> >> > >> >> > > not the best idea. However, if a broker goes down then we
>> have
>> >> > much
>> >> > >> >> bigger
>> >> > >> >> > > issue to deal with and client can always re-authenticate is
>> such
>> >> > >> >> events.
>> >> > >> >> > >
>> >> > >> >> > > Did you happen to take a look at other alternatives this
>> list has
>> >> > >> >> > > suggested?
>> >> > >> >> > >
>> >> > >> >> > > >
>> >> > >> >> > > > Thanks for a thorough proposal, great work!"
>> >> > >> >> > > >
>> >> > >> >> > > >
>> >> > >> >> > > >
>> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
>> >> > >> >> > > > > Makes sense to me. Thanks!
>> >> > >> >> > > > >
>> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <kafka@harsha.io
>> >
>> >> > wrote:
>> >> > >> >> > > > > > It doesn't need any release vehicle but still the
>> work can
>> >> > move
>> >> > >> >> > > > forward.
>> >> > >> >> > > > > > If anyone is interested in the KIP please do the
>> review and
>> >> > >> >> provide
>> >> > >> >> > > the
>> >> > >> >> > > > > > comments.
>> >> > >> >> > > > > >
>> >> > >> >> > > > > > -Harsha
>> >> > >> >> > > > > >
>> >> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma wrote:
>> >> > >> >> > > > > >> I agree that it would be good to have more time to
>> review
>> >> > and
>> >> > >> >> > > discuss
>> >> > >> >> > > > > >> KIP-48.
>> >> > >> >> > > > > >>
>> >> > >> >> > > > > >> Ismael
>> >> > >> >> > > > > >>
>> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <
>> >> > >> >> gwen@confluent.io>
>> >> > >> >> > > > wrote:
>> >> > >> >> > > > > >>
>> >> > >> >> > > > > >> > Hi Team,
>> >> > >> >> > > > > >> >
>> >> > >> >> > > > > >> > Since KIP-48 depends on KIP-43, which is already a
>> bit
>> >> > of a
>> >> > >> >> risk
>> >> > >> >> > > for
>> >> > >> >> > > > > >> > the next release - any chance we can delay
>> delegation
>> >> > tokens
>> >> > >> >> to
>> >> > >> >> > > > Kafka
>> >> > >> >> > > > > >> > 0.10.1?
>> >> > >> >> > > > > >> > With the community working on a release every 3
>> month,
>> >> > this
>> >> > >> >> is not
>> >> > >> >> > > > a huge
>> >> > >> >> > > > > >> > delay.
>> >> > >> >> > > > > >> >
>> >> > >> >> > > > > >> > Gwen
>> >> > >> >> > > > > >> >
>> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh <
>> >> > >> >> > > asingh@cloudera.com>
>> >> > >> >> > > > wrote:
>> >> > >> >> > > > > >> > > Parth,
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > > Thanks again for the awesome write up. Following
>> our
>> >> > >> >> discussion
>> >> > >> >> > > > from the
>> >> > >> >> > > > > >> > > JIRA, I think it will be easier to compare
>> various
>> >> > >> >> alternatives
>> >> > >> >> > > > if they
>> >> > >> >> > > > > >> > are
>> >> > >> >> > > > > >> > > listed together. I am stating below a few
>> >> > alternatives along
>> >> > >> >> > > with
>> >> > >> >> > > > a the
>> >> > >> >> > > > > >> > > current proposal.
>> >> > >> >> > > > > >> > > (Current proposal) Store Delegation Token, DT,
>> on ZK.
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > >    1. Client authenticates with a broker.
>> >> > >> >> > > > > >> > >    2. Once a client is authenticated, it will
>> make a
>> >> > broker
>> >> > >> >> side
>> >> > >> >> > > > call to
>> >> > >> >> > > > > >> > >    issue a delegation token.
>> >> > >> >> > > > > >> > >    3. The broker generates a shared secret based
>> on
>> >> > >> >> > > HMAC-SHA256(a
>> >> > >> >> > > > > >> > >    Password/Secret shared between all brokers,
>> >> > randomly
>> >> > >> >> > > generated
>> >> > >> >> > > > > >> > tokenId).
>> >> > >> >> > > > > >> > >    4. Broker stores this token in its in memory
>> cache.
>> >> > >> >> Broker
>> >> > >> >> > > > also stores
>> >> > >> >> > > > > >> > >    the DelegationToken without the hmac in the
>> >> > zookeeper.
>> >> > >> >> > > > > >> > >    5. All brokers will have a cache backed by
>> >> > zookeeper so
>> >> > >> >> they
>> >> > >> >> > > > will all
>> >> > >> >> > > > > >> > >    get notified whenever a new token is
>> generated and
>> >> > they
>> >> > >> >> will
>> >> > >> >> > > > update
>> >> > >> >> > > > > >> > their
>> >> > >> >> > > > > >> > >    local cache whenever token state changes.
>> >> > >> >> > > > > >> > >    6. Broker returns the token to Client.
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > > Probable issues and fixes
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > >    1. Probable race condition, client tries to
>> >> > authenticate
>> >> > >> >> with
>> >> > >> >> > > > a broker
>> >> > >> >> > > > > >> > >    that is yet to be updated with the newly
>> generated
>> >> > DT.
>> >> > >> >> This
>> >> > >> >> > > can
>> >> > >> >> > > > > >> > probably be
>> >> > >> >> > > > > >> > >    dealt with making dtRequest block until all
>> >> > brokers have
>> >> > >> >> > > > updated
>> >> > >> >> > > > > >> > their DT
>> >> > >> >> > > > > >> > >    cache. Zk barrier or similar mechanism can be
>> used.
>> >> > >> >> However,
>> >> > >> >> > > > all such
>> >> > >> >> > > > > >> > >    mechanisms will increase complexity.
>> >> > >> >> > > > > >> > >    2. Using a static secret key from config
>> file. Will
>> >> > >> >> require
>> >> > >> >> > > yet
>> >> > >> >> > > > > >> > another
>> >> > >> >> > > > > >> > >    config and uses a static secret key. It is
>> advised
>> >> > to
>> >> > >> >> rotate
>> >> > >> >> > > > secret
>> >> > >> >> > > > > >> > keys
>> >> > >> >> > > > > >> > >    periodically. This can be avoided with
>> controller
>> >> > >> >> generating
>> >> > >> >> > > > > >> > secretKey and
>> >> > >> >> > > > > >> > >    passing to brokers periodically. However,
>> this will
>> >> > >> >> require
>> >> > >> >> > > > brokers to
>> >> > >> >> > > > > >> > >    maintain certain counts of secretKeys.
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > > (Alternative 1) Have controller generate
>> delegation
>> >> > token.
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > >    1. Client authenticates with a broker.
>> >> > >> >> > > > > >> > >    2. Once a client is authenticated, it will
>> make a
>> >> > broker
>> >> > >> >> side
>> >> > >> >> > > > call to
>> >> > >> >> > > > > >> > >    issue a delegation token.
>> >> > >> >> > > > > >> > >    3. Broker forwards the request to controller.
>> >> > >> >> > > > > >> > >    4. Controller generates a DT and broadcasts
>> to all
>> >> > >> >> brokers.
>> >> > >> >> > > > > >> > >    5. Broker stores this token in its memory
>> cache.
>> >> > >> >> > > > > >> > >    6. Controller responds to broker’s DT req.
>> >> > >> >> > > > > >> > >    7. Broker returns the token to Client.
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > > Probable issues and fixes
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > >    1. We will have to add new APIs to support
>> >> > controller
>> >> > >> >> pushing
>> >> > >> >> > > > tokens
>> >> > >> >> > > > > >> > to
>> >> > >> >> > > > > >> > >    brokers on top of the minimal APIs that are
>> >> > currently
>> >> > >> >> > > proposed.
>> >> > >> >> > > > > >> > >    2. We will also have to add APIs to support
>> the
>> >> > >> >> bootstrapping
>> >> > >> >> > > > case,
>> >> > >> >> > > > > >> > i.e,
>> >> > >> >> > > > > >> > >    when a new broker comes up it will have to
>> get all
>> >> > >> >> delegation
>> >> > >> >> > > > tokens
>> >> > >> >> > > > > >> > from
>> >> > >> >> > > > > >> > >    the controller.
>> >> > >> >> > > > > >> > >    3. In catastrophic failures where all brokers
>> go
>> >> > down,
>> >> > >> >> the
>> >> > >> >> > > > tokens will
>> >> > >> >> > > > > >> > >    be lost even if servers are restarted as
>> tokens
>> >> > are not
>> >> > >> >> > > > persisted
>> >> > >> >> > > > > >> > anywhere.
>> >> > >> >> > > > > >> > >    If this happens, then there are more important
>> >> > things to
>> >> > >> >> > > worry
>> >> > >> >> > > > about
>> >> > >> >> > > > > >> > and
>> >> > >> >> > > > > >> > >    maybe it is better to re-authenticate.
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > > (Alternative 2) Do not distribute DT to other
>> brokers
>> >> > at
>> >> > >> >> all.
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > >    1. Client authenticates with a broker.
>> >> > >> >> > > > > >> > >    2. Once a client is authenticated, it will
>> make a
>> >> > broker
>> >> > >> >> side
>> >> > >> >> > > > call to
>> >> > >> >> > > > > >> > >    issue a delegation token.
>> >> > >> >> > > > > >> > >    3. The broker generates DT of form, [hmac +
>> (owner,
>> >> > >> >> renewer,
>> >> > >> >> > > > > >> > >    maxLifeTime, id, hmac, expirationTime)] and
>> passes
>> >> > back
>> >> > >> >> this
>> >> > >> >> > > > DT to
>> >> > >> >> > > > > >> > client.
>> >> > >> >> > > > > >> > >    hmac is generated via {HMAC-SHA256(owner,
>> renewer,
>> >> > >> >> > > > maxLifeTime, id,
>> >> > >> >> > > > > >> > hmac,
>> >> > >> >> > > > > >> > >    expirationTime) using SecretKey}. Note that
>> all
>> >> > brokers
>> >> > >> >> have
>> >> > >> >> > > > this
>> >> > >> >> > > > > >> > SecretKey.
>> >> > >> >> > > > > >> > >    4. Client then goes to any broker and to
>> >> > authenticate
>> >> > >> >> sends
>> >> > >> >> > > > the DT.
>> >> > >> >> > > > > >> > >    Broker recalculates hmac using (owner,
>> renewer,
>> >> > >> >> maxLifeTime,
>> >> > >> >> > > > id, hmac,
>> >> > >> >> > > > > >> > >    expirationTime) info from DT and its
>> SecretKey. If
>> >> > it
>> >> > >> >> matches
>> >> > >> >> > > > with
>> >> > >> >> > > > > >> > hmac of
>> >> > >> >> > > > > >> > >    DT, client is authenticated. Yes, it will do
>> other
>> >> > >> >> obvious
>> >> > >> >> > > > checks of
>> >> > >> >> > > > > >> > >    timestamp expiry and such.
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > > Note that secret key will be generated by
>> controller
>> >> > and
>> >> > >> >> passed
>> >> > >> >> > > to
>> >> > >> >> > > > > >> > brokers
>> >> > >> >> > > > > >> > > periodically.
>> >> > >> >> > > > > >> > > Probable issues and fixes
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > >    1. How to delete a DT? Yes, that is a downside
>> >> > here.
>> >> > >> >> However,
>> >> > >> >> > > > this can
>> >> > >> >> > > > > >> > >    be handled with brokers maintaining a
>> blacklist of
>> >> > DTs,
>> >> > >> >> DTs
>> >> > >> >> > > > from this
>> >> > >> >> > > > > >> > list
>> >> > >> >> > > > > >> > >    can be removed after expiry.
>> >> > >> >> > > > > >> > >    2. In catastrophic failures where all brokers
>> go
>> >> > down,
>> >> > >> >> the
>> >> > >> >> > > > tokens will
>> >> > >> >> > > > > >> > >    be lost even if servers are restarted as
>> tokens
>> >> > are not
>> >> > >> >> > > > persisted
>> >> > >> >> > > > > >> > anywhere.
>> >> > >> >> > > > > >> > >    If this happens, then there are more important
>> >> > things to
>> >> > >> >> > > worry
>> >> > >> >> > > > about
>> >> > >> >> > > > > >> > and
>> >> > >> >> > > > > >> > >    maybe it is better to re-authenticate.
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth
>> Brahmbhatt <
>> >> > >> >> > > > > >> > > pbrahmbhatt@hortonworks.com> wrote:
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > >> Hi,
>> >> > >> >> > > > > >> > >>
>> >> > >> >> > > > > >> > >> I have filed KIP-48 so we can offer hadoop like
>> >> > delegation
>> >> > >> >> > > > tokens in
>> >> > >> >> > > > > >> > >> kafka. You can review the design
>> >> > >> >> > > > > >> > >>
>> >> > >> >> > > > > >> >
>> >> > >> >> > > >
>> >> > >> >> > >
>> >> > >> >>
>> >> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
>> >> > >> >> > > > > >> > .
>> >> > >> >> > > > > >> > >> This KIP depends on KIP-43 and we have also
>> >> > discussed an
>> >> > >> >> > > > alternative to
>> >> > >> >> > > > > >> > >> proposed design here<
>> >> > >> >> > > > > >> > >>
>> >> > >> >> > > > > >> >
>> >> > >> >> > > >
>> >> > >> >> > >
>> >> > >> >>
>> >> >
>> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
>> >> > >> >> > > > > >> > >> >.
>> >> > >> >> > > > > >> > >>
>> >> > >> >> > > > > >> > >> Thanks
>> >> > >> >> > > > > >> > >> Parth
>> >> > >> >> > > > > >> > >>
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > > --
>> >> > >> >> > > > > >> > >
>> >> > >> >> > > > > >> > > Regards,
>> >> > >> >> > > > > >> > > Ashish
>> >> > >> >> > > > > >> >
>> >> > >> >> > > >
>> >> > >> >> > >
>> >> > >> >> > >
>> >> > >> >> > >
>> >> > >> >> > > --
>> >> > >> >> > >
>> >> > >> >> > > Regards,
>> >> > >> >> > > Ashish
>> >> > >> >> > >
>> >> > >> >>
>> >> >
>>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[parth brahmbhatt <br...@gmail.com>]

My comments are inline.

* The API / RequestResponse section doesn't seem to have good
description of the changes to the Kafka Protocol. Sounds like you are
proposing new DelegationTokenRequest and RenewTokenRequest (and
matching responses), without detailing the contents of the requests
and responses? Or rather, you show the class interface, but not the
underlying protocol. This could be seen as an implementation detail,
but since the binary protocol is what we provide to non-Java clients,
we need to show the changes there.

*I will add the binary protocol details.*

* getDelegationToken sounds like delegationTokenRequestHandler? Is it
planned to be part of KafkaApi? or Client? Its unclear...


*It will be part of KafkaAPI.*
* I want to emphasize that even though delegation tokens are a Hadoop
innovation, I feel very strongly about not adding dependency on Hadoop
when implementing delegation tokens for Kafka. The KIP doesn't imply
such dependency, but if you can clarify...


*No hadoop dependency.*
* Can we get delegation token at any time after authenticating? only
immediately after?


*As long as you are authenticated you can get delegation tokens. We need to
discuss if a client authenticated using delegation token, can also acquire
delegation token again or not. Also there is the question of do we allow
anyone to acquire delegation token or we want specific ACLs (I think its an
overkill.)*
* My understanding is that tokens will propagate via ZK but without
additional changes to UpdateMetadata protocol, correct? Clients
currently don't retry on SASL auth failure (IIRC), but since the
tokens propagate between brokers asynch, we will need to retry a bit
to avoid clients failing auth due to timing issues.

*I am considering 2 alternatives right now. The current documented approach
is zookeeper based and it does not require any changes to UpdateMetadata
protocol. An alternative approach can remove zookeeper dependency as well
but we can discuss that in KIP discussion call.*

* Strongly agreeing on clients not touching ZK directly :)

* I liked Ashish's suggestion of having just the controller issue the
delegation tokens, to avoid syncing a shared secret. Not sure if we
want to continue the discussion here or on the wiki. I think that we
can decouple the problem of "token distribution" from "shared secret
distribution" and use the controller as the only token generator to
solve the second issue, while still using ZK async to distribute
tokens.


*As mentioned in the previous Email I am fine with that approach as long as
we agree that the extra complexity of adding/updating APIS adds enough
value. The advantage with the controller approach is secret rotation can be
automated,frequent and would not require deployment. *

* I am also uncomfortable with infinite lifetime of tokens (and hoped
to hear from others in the community) - but having the option and
documenting it as less secure, allows users to configure their system
as the wish.


* While I like the idea of forcing kerberos auth for renwal, I think
it mixes the transport layer the the request content in a pretty ugly
way. Perhaps limiting renewer to non-owner is better.

*I feel this is a necessary evil. While this will make the kafka code
pretty straight forward , forcing  renewer to non-owner pushes the code
ugliness to client and makes it even harder to integrate.  *

Things I'd still like to see:

* More detailed explanation on what we plan to do for the java clients
specifically - new configuration? new APIs?

*Will add client side details.*

The response for my question on how multiple identities will be
handled wasn't super clear to me - AFAIK, we don't authenticate each
request, we authenticate connections.

*We authenticate connections, and only when they are being established. Let
me try to phrase this as a question, in absence of delegation tokens if we
had to support the use case using user TGT's how would we do it? My point
was it would be no different with delegation tokens. The use case you are
describing seems more like impersonation.*

* Alternatives: Delegation tokens are only used in the Hadoop
ecosystem. I'm wondering if there are alternatives in other ecosystems
(Mesos? Tachyon? Cassandra?) and whether there are some advantages
there.

*I have not evaluated those options, if someone else has expertise in it
feel free to chime in.*

Thanks
Parth

On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <gw...@confluent.io> wrote:

> Hi Parth and Harsha,
>
> Few more comments:
>
> * The API / RequestResponse section doesn't seem to have good
> description of the changes to the Kafka Protocol. Sounds like you are
> proposing new DelegationTokenRequest and RenewTokenRequest (and
> matching responses), without detailing the contents of the requests
> and responses? Or rather, you show the class interface, but not the
> underlying protocol. This could be seen as an implementation detail,
> but since the binary protocol is what we provide to non-Java clients,
> we need to show the changes there.
>
> * getDelegationToken sounds like delegationTokenRequestHandler? Is it
> planned to be part of KafkaApi? or Client? Its unclear...
>
> * I want to emphasize that even though delegation tokens are a Hadoop
> innovation, I feel very strongly about not adding dependency on Hadoop
> when implementing delegation tokens for Kafka. The KIP doesn't imply
> such dependency, but if you can clarify...
>
> * Can we get delegation token at any time after authenticating? only
> immediately after?
>
> * My understanding is that tokens will propagate via ZK but without
> additional changes to UpdateMetadata protocol, correct? Clients
> currently don't retry on SASL auth failure (IIRC), but since the
> tokens propagate between brokers asynch, we will need to retry a bit
> to avoid clients failing auth due to timing issues.
>
> * Strongly agreeing on clients not touching ZK directly :)
>
> * I liked Ashish's suggestion of having just the controller issue the
> delegation tokens, to avoid syncing a shared secret. Not sure if we
> want to continue the discussion here or on the wiki. I think that we
> can decouple the problem of "token distribution" from "shared secret
> distribution" and use the controller as the only token generator to
> solve the second issue, while still using ZK async to distribute
> tokens.
>
> * I am also uncomfortable with infinite lifetime of tokens (and hoped
> to hear from others in the community) - but having the option and
> documenting it as less secure, allows users to configure their system
> as the wish.
>
> * While I like the idea of forcing kerberos auth for renwal, I think
> it mixes the transport layer the the request content in a pretty ugly
> way. Perhaps limiting renewer to non-owner is better.
>
> Things I'd still like to see:
>
> * More detailed explanation on what we plan to do for the java clients
> specifically - new configuration? new APIs?
> The response for my question on how multiple identities will be
> handled wasn't super clear to me - AFAIK, we don't authenticate each
> request, we authenticate connections.
>
> * Alternatives: Delegation tokens are only used in the Hadoop
> ecosystem. I'm wondering if there are alternatives in other ecosystems
> (Mesos? Tachyon? Cassandra?) and whether there are some advantages
> there.
>
> Gwen
>
> On Thu, May 12, 2016 at 1:05 PM, Harsha <ka...@harsha.io> wrote:
> > Hi Gwen,
> >            Can you look at Parth's last reply. Does it answer your
> >            concerns.
> >
> > Thanks,
> > Harsha
> >
> > On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt wrote:
> >> Thanks for reviewing Gwen. The wiki already has details on token
> >> expiration
> >> under token acquisition process
> >> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> >.
> >> Current proposal is that tokens will expire based on a server side
> >> configuration (default 24 hours) unless renewed. Renewal is only allowed
> >> until the max life time of token. Alternatively we could also make that
> >> an
> >> optional param and the server side default can serve as the upper bound.
> >>
> >> To your second point it will be done exactly the same way we would
> >> support
> >> multiple keytabs. The calling client will have to put the tokens it
> wants
> >> to use in the subject instance and call produce/consume inside
> >> subject.doas. Each caller will have to keep track of its own subject. I
> >> will have to look at the code to see if we support this feature right
> now
> >> but my understanding is delegation token shouldn't need any special
> >> treatment as its just another type of Credential in the subject.
> >>
> >> I would also like to know what is your opinion about infinite renewal
> (my
> >> recommendation is to not support this), tokens renewing them self(my
> >> recommendation is to not support this) and most importantly your choice
> >> between the alternatives listed on this thread
> >> <
> http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> >
> >> ( I am leaning towards the alternative-2 minus controller distributing
> >> secret). Thanks again for reviewing.
> >>
> >> Thanks
> >> Parth
> >>
> >>
> >>
> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <gw...@confluent.io> wrote:
> >>
> >> > Harsha,
> >> >
> >> > I was thinking of the Rest Proxy. I didn't see your design yet, but in
> >> > our proxy, we have a set of producers, which will serve multiple users
> >> > going through the proxy. Since these users will have different
> >> > privileges, they'll need to authenticate separately, and can't share a
> >> > token.
> >> >
> >> > Am I missing anything?
> >> >
> >> > Gwen
> >> >
> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io> wrote:
> >> > > Gwen,
> >> > >            On your second point. Can you describe a usecase where
> >> > >            mutliple clients ended up sharing a producer and even if
> they
> >> > >            do why can't they not use single token that producer
> >> > >            captures. Why would we need multiple clients with
> different
> >> > >            tokens sharing a single instance of producer.  Also in
> this
> >> > >            case other clients have access all the tokens no?
> >> > >
> >> > > Thanks,
> >> > > Harsha
> >> > >
> >> > >
> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote:
> >> > >> Sorry for the delay:
> >> > >>
> >> > >> Two questions that we didn't see in the wiki:
> >> > >> 1. Is there an expiration for delegation tokens? Renewal? How do we
> >> > >> revoke them?
> >> > >> 2. If we want to use delegation tokens for "do-as" (say, submit
> Storm
> >> > >> job as my user), we will need a producer for every job (we can't
> share
> >> > >> them between multiple jobs running on same node), since we only
> >> > >> authenticate when connecting. Is there a plan to change this for
> >> > >> delegation tokens, in order to allow multiple users with different
> >> > >> tokens to share a client?
> >> > >>
> >> > >> Gwen
> >> > >>
> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
> >> > >> <br...@gmail.com> wrote:
> >> > >> > Bumping this up one more time, can other committers review?
> >> > >> >
> >> > >> > Thanks
> >> > >> > Parth
> >> > >> >
> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <ka...@harsha.io> wrote:
> >> > >> >
> >> > >> >> Parth,
> >> > >> >>           Overall current design looks good to me. I am +1 on
> the
> >> > KIP.
> >> > >> >>
> >> > >> >> Gwen , Jun can you review this as well.
> >> > >> >>
> >> > >> >> -Harsha
> >> > >> >>
> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote:
> >> > >> >> > Thanks for review Jitendra.
> >> > >> >> >
> >> > >> >> > I don't like the idea of infinite lifetime but I see the
> Streaming
> >> > use
> >> > >> >> > case. Even for Streaming use case I was hoping there will be
> some
> >> > notion
> >> > >> >> > of
> >> > >> >> > master/driver that can get new delegation tokens at fixed
> interval
> >> > and
> >> > >> >> > distribute to workers. If that is not the case for we can
> discuss
> >> > >> >> > delegation tokens renewing them self and the security
> implications
> >> > of the
> >> > >> >> > same.
> >> > >> >> >
> >> > >> >> > I did not want clients to fetch tokens from zookeeper,
> overall I
> >> > think
> >> > >> >> > its
> >> > >> >> > better if clients don't rely on our metadata store and I
> think we
> >> > are
> >> > >> >> > moving in that direction with all the KIP-4 improvements.  I
> chose
> >> > >> >> > zookeeper as in this case the client will still just talk to
> >> > broker , its
> >> > >> >> > the brokers that will use zookeeper which we already do for a
> lot
> >> > of
> >> > >> >> > other
> >> > >> >> > usecases + ease of development + and the ability so tokens
> will
> >> > survive
> >> > >> >> > even a rolling restart/cluster failure. if a majority agrees
> the
> >> > added
> >> > >> >> > complexity to have controller forwarding keys to all broker is
> >> > justified
> >> > >> >> > as
> >> > >> >> > it provides tighter security , I am fine with that option too.
> >> > >> >> >
> >> > >> >> > Given zookeeper does not support SSL we can not store master
> keys
> >> > in
> >> > >> >> > zookeeper as master keys will be exposed on wire. To support
> >> > rotation
> >> > >> >> > without affecting current clients is something I need to put
> more
> >> > thought
> >> > >> >> > in. My current proposal assumes the rotation will invalidate
> all
> >> > current
> >> > >> >> > tokens.
> >> > >> >> >
> >> > >> >> > I request committers to also review and post their comments
> so we
> >> > can
> >> > >> >> > make
> >> > >> >> > progress on this KIP.
> >> > >> >> >
> >> > >> >> > Thanks
> >> > >> >> > Parth
> >> > >> >> >
> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <
> asingh@cloudera.com
> >> > >
> >> > >> >> > wrote:
> >> > >> >> >
> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <ka...@harsha.io>
> >> > wrote:
> >> > >> >> > >
> >> > >> >> > > > Unifying the two discussion threads on this KIP.
> >> > >> >> > > >
> >> > >> >> > > > Here is the response from Jitendra
> >> > >> >> > > >
> >> > >> >> > > > "The need for a large number of clients that are running
> all
> >> > over the
> >> > >> >> > > > cluster that authenticate with Kafka brokers, is very
> similar
> >> > to the
> >> > >> >> > > > Hadoop use case of large number of tasks running across
> the
> >> > cluster
> >> > >> >> that
> >> > >> >> > > > need authentication to Hdfs Namenode. Therefore, the
> >> > delegation token
> >> > >> >> > > > approach does seem like a good fit for this use case as we
> >> > have seen
> >> > >> >> it
> >> > >> >> > > > working at large scale in HDFS and YARN.
> >> > >> >> > > >
> >> > >> >> > > >   The proposed design is very much inline with Hadoop
> >> > approach. A few
> >> > >> >> > > >   comments:
> >> > >> >> > > >
> >> > >> >> > > > 1) Why do you guys want to allow infinite renewable
> lifetime
> >> > for a
> >> > >> >> > > > token? HDFS restricts a token to a max life time (default
> 7
> >> > days).  A
> >> > >> >> > > > token's vulnerability is believed to increase with time.
> >> > >> >> > > >
> >> > >> >> > > I agree that having infinite lifetime might not be the best
> idea.
> >> > >> >> > >
> >> > >> >> > > >
> >> > >> >> > > > 2) As I understand the tokens are stored in zookeeper as
> well,
> >> > and
> >> > >> >> can
> >> > >> >> > > > be updated there. This is clever as it can allow
> replacing the
> >> > tokens
> >> > >> >> > > > once they run out of max life time, and clients can
> download
> >> > new
> >> > >> >> tokens
> >> > >> >> > > > from zookeeper. It shouldn't be a big load on zookeeper
> as a
> >> > client
> >> > >> >> will
> >> > >> >> > > > need to get a new token once in several days. In this
> approach
> >> > you
> >> > >> >> don't
> >> > >> >> > > > need infinite lifetime on the token even for long running
> >> > clients.
> >> > >> >> > > >
> >> > >> >> > > > 3) The token password are generated using a master key.
> The
> >> > master
> >> > >> >> key
> >> > >> >> > > > should also be periodically changed. In Hadoop, the
> default
> >> > renewal
> >> > >> >> > > > period is 1 day.?
> >> > >> >> > > >
> >> > >> >> > > IIUC, this will require brokers maintaining a list of X most
> >> > recent
> >> > >> >> master
> >> > >> >> > > keys. This list will have to be persisted somewhere, as if a
> >> > broker
> >> > >> >> goes
> >> > >> >> > > down it will have to get that list again and storing master
> keys
> >> > on ZK
> >> > >> >> is
> >> > >> >> > > not the best idea. However, if a broker goes down then we
> have
> >> > much
> >> > >> >> bigger
> >> > >> >> > > issue to deal with and client can always re-authenticate is
> such
> >> > >> >> events.
> >> > >> >> > >
> >> > >> >> > > Did you happen to take a look at other alternatives this
> list has
> >> > >> >> > > suggested?
> >> > >> >> > >
> >> > >> >> > > >
> >> > >> >> > > > Thanks for a thorough proposal, great work!"
> >> > >> >> > > >
> >> > >> >> > > >
> >> > >> >> > > >
> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
> >> > >> >> > > > > Makes sense to me. Thanks!
> >> > >> >> > > > >
> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <kafka@harsha.io
> >
> >> > wrote:
> >> > >> >> > > > > > It doesn't need any release vehicle but still the
> work can
> >> > move
> >> > >> >> > > > forward.
> >> > >> >> > > > > > If anyone is interested in the KIP please do the
> review and
> >> > >> >> provide
> >> > >> >> > > the
> >> > >> >> > > > > > comments.
> >> > >> >> > > > > >
> >> > >> >> > > > > > -Harsha
> >> > >> >> > > > > >
> >> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma wrote:
> >> > >> >> > > > > >> I agree that it would be good to have more time to
> review
> >> > and
> >> > >> >> > > discuss
> >> > >> >> > > > > >> KIP-48.
> >> > >> >> > > > > >>
> >> > >> >> > > > > >> Ismael
> >> > >> >> > > > > >>
> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <
> >> > >> >> gwen@confluent.io>
> >> > >> >> > > > wrote:
> >> > >> >> > > > > >>
> >> > >> >> > > > > >> > Hi Team,
> >> > >> >> > > > > >> >
> >> > >> >> > > > > >> > Since KIP-48 depends on KIP-43, which is already a
> bit
> >> > of a
> >> > >> >> risk
> >> > >> >> > > for
> >> > >> >> > > > > >> > the next release - any chance we can delay
> delegation
> >> > tokens
> >> > >> >> to
> >> > >> >> > > > Kafka
> >> > >> >> > > > > >> > 0.10.1?
> >> > >> >> > > > > >> > With the community working on a release every 3
> month,
> >> > this
> >> > >> >> is not
> >> > >> >> > > > a huge
> >> > >> >> > > > > >> > delay.
> >> > >> >> > > > > >> >
> >> > >> >> > > > > >> > Gwen
> >> > >> >> > > > > >> >
> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh <
> >> > >> >> > > asingh@cloudera.com>
> >> > >> >> > > > wrote:
> >> > >> >> > > > > >> > > Parth,
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > > Thanks again for the awesome write up. Following
> our
> >> > >> >> discussion
> >> > >> >> > > > from the
> >> > >> >> > > > > >> > > JIRA, I think it will be easier to compare
> various
> >> > >> >> alternatives
> >> > >> >> > > > if they
> >> > >> >> > > > > >> > are
> >> > >> >> > > > > >> > > listed together. I am stating below a few
> >> > alternatives along
> >> > >> >> > > with
> >> > >> >> > > > a the
> >> > >> >> > > > > >> > > current proposal.
> >> > >> >> > > > > >> > > (Current proposal) Store Delegation Token, DT,
> on ZK.
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > >    1. Client authenticates with a broker.
> >> > >> >> > > > > >> > >    2. Once a client is authenticated, it will
> make a
> >> > broker
> >> > >> >> side
> >> > >> >> > > > call to
> >> > >> >> > > > > >> > >    issue a delegation token.
> >> > >> >> > > > > >> > >    3. The broker generates a shared secret based
> on
> >> > >> >> > > HMAC-SHA256(a
> >> > >> >> > > > > >> > >    Password/Secret shared between all brokers,
> >> > randomly
> >> > >> >> > > generated
> >> > >> >> > > > > >> > tokenId).
> >> > >> >> > > > > >> > >    4. Broker stores this token in its in memory
> cache.
> >> > >> >> Broker
> >> > >> >> > > > also stores
> >> > >> >> > > > > >> > >    the DelegationToken without the hmac in the
> >> > zookeeper.
> >> > >> >> > > > > >> > >    5. All brokers will have a cache backed by
> >> > zookeeper so
> >> > >> >> they
> >> > >> >> > > > will all
> >> > >> >> > > > > >> > >    get notified whenever a new token is
> generated and
> >> > they
> >> > >> >> will
> >> > >> >> > > > update
> >> > >> >> > > > > >> > their
> >> > >> >> > > > > >> > >    local cache whenever token state changes.
> >> > >> >> > > > > >> > >    6. Broker returns the token to Client.
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > > Probable issues and fixes
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > >    1. Probable race condition, client tries to
> >> > authenticate
> >> > >> >> with
> >> > >> >> > > > a broker
> >> > >> >> > > > > >> > >    that is yet to be updated with the newly
> generated
> >> > DT.
> >> > >> >> This
> >> > >> >> > > can
> >> > >> >> > > > > >> > probably be
> >> > >> >> > > > > >> > >    dealt with making dtRequest block until all
> >> > brokers have
> >> > >> >> > > > updated
> >> > >> >> > > > > >> > their DT
> >> > >> >> > > > > >> > >    cache. Zk barrier or similar mechanism can be
> used.
> >> > >> >> However,
> >> > >> >> > > > all such
> >> > >> >> > > > > >> > >    mechanisms will increase complexity.
> >> > >> >> > > > > >> > >    2. Using a static secret key from config
> file. Will
> >> > >> >> require
> >> > >> >> > > yet
> >> > >> >> > > > > >> > another
> >> > >> >> > > > > >> > >    config and uses a static secret key. It is
> advised
> >> > to
> >> > >> >> rotate
> >> > >> >> > > > secret
> >> > >> >> > > > > >> > keys
> >> > >> >> > > > > >> > >    periodically. This can be avoided with
> controller
> >> > >> >> generating
> >> > >> >> > > > > >> > secretKey and
> >> > >> >> > > > > >> > >    passing to brokers periodically. However,
> this will
> >> > >> >> require
> >> > >> >> > > > brokers to
> >> > >> >> > > > > >> > >    maintain certain counts of secretKeys.
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > > (Alternative 1) Have controller generate
> delegation
> >> > token.
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > >    1. Client authenticates with a broker.
> >> > >> >> > > > > >> > >    2. Once a client is authenticated, it will
> make a
> >> > broker
> >> > >> >> side
> >> > >> >> > > > call to
> >> > >> >> > > > > >> > >    issue a delegation token.
> >> > >> >> > > > > >> > >    3. Broker forwards the request to controller.
> >> > >> >> > > > > >> > >    4. Controller generates a DT and broadcasts
> to all
> >> > >> >> brokers.
> >> > >> >> > > > > >> > >    5. Broker stores this token in its memory
> cache.
> >> > >> >> > > > > >> > >    6. Controller responds to broker’s DT req.
> >> > >> >> > > > > >> > >    7. Broker returns the token to Client.
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > > Probable issues and fixes
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > >    1. We will have to add new APIs to support
> >> > controller
> >> > >> >> pushing
> >> > >> >> > > > tokens
> >> > >> >> > > > > >> > to
> >> > >> >> > > > > >> > >    brokers on top of the minimal APIs that are
> >> > currently
> >> > >> >> > > proposed.
> >> > >> >> > > > > >> > >    2. We will also have to add APIs to support
> the
> >> > >> >> bootstrapping
> >> > >> >> > > > case,
> >> > >> >> > > > > >> > i.e,
> >> > >> >> > > > > >> > >    when a new broker comes up it will have to
> get all
> >> > >> >> delegation
> >> > >> >> > > > tokens
> >> > >> >> > > > > >> > from
> >> > >> >> > > > > >> > >    the controller.
> >> > >> >> > > > > >> > >    3. In catastrophic failures where all brokers
> go
> >> > down,
> >> > >> >> the
> >> > >> >> > > > tokens will
> >> > >> >> > > > > >> > >    be lost even if servers are restarted as
> tokens
> >> > are not
> >> > >> >> > > > persisted
> >> > >> >> > > > > >> > anywhere.
> >> > >> >> > > > > >> > >    If this happens, then there are more important
> >> > things to
> >> > >> >> > > worry
> >> > >> >> > > > about
> >> > >> >> > > > > >> > and
> >> > >> >> > > > > >> > >    maybe it is better to re-authenticate.
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > > (Alternative 2) Do not distribute DT to other
> brokers
> >> > at
> >> > >> >> all.
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > >    1. Client authenticates with a broker.
> >> > >> >> > > > > >> > >    2. Once a client is authenticated, it will
> make a
> >> > broker
> >> > >> >> side
> >> > >> >> > > > call to
> >> > >> >> > > > > >> > >    issue a delegation token.
> >> > >> >> > > > > >> > >    3. The broker generates DT of form, [hmac +
> (owner,
> >> > >> >> renewer,
> >> > >> >> > > > > >> > >    maxLifeTime, id, hmac, expirationTime)] and
> passes
> >> > back
> >> > >> >> this
> >> > >> >> > > > DT to
> >> > >> >> > > > > >> > client.
> >> > >> >> > > > > >> > >    hmac is generated via {HMAC-SHA256(owner,
> renewer,
> >> > >> >> > > > maxLifeTime, id,
> >> > >> >> > > > > >> > hmac,
> >> > >> >> > > > > >> > >    expirationTime) using SecretKey}. Note that
> all
> >> > brokers
> >> > >> >> have
> >> > >> >> > > > this
> >> > >> >> > > > > >> > SecretKey.
> >> > >> >> > > > > >> > >    4. Client then goes to any broker and to
> >> > authenticate
> >> > >> >> sends
> >> > >> >> > > > the DT.
> >> > >> >> > > > > >> > >    Broker recalculates hmac using (owner,
> renewer,
> >> > >> >> maxLifeTime,
> >> > >> >> > > > id, hmac,
> >> > >> >> > > > > >> > >    expirationTime) info from DT and its
> SecretKey. If
> >> > it
> >> > >> >> matches
> >> > >> >> > > > with
> >> > >> >> > > > > >> > hmac of
> >> > >> >> > > > > >> > >    DT, client is authenticated. Yes, it will do
> other
> >> > >> >> obvious
> >> > >> >> > > > checks of
> >> > >> >> > > > > >> > >    timestamp expiry and such.
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > > Note that secret key will be generated by
> controller
> >> > and
> >> > >> >> passed
> >> > >> >> > > to
> >> > >> >> > > > > >> > brokers
> >> > >> >> > > > > >> > > periodically.
> >> > >> >> > > > > >> > > Probable issues and fixes
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > >    1. How to delete a DT? Yes, that is a downside
> >> > here.
> >> > >> >> However,
> >> > >> >> > > > this can
> >> > >> >> > > > > >> > >    be handled with brokers maintaining a
> blacklist of
> >> > DTs,
> >> > >> >> DTs
> >> > >> >> > > > from this
> >> > >> >> > > > > >> > list
> >> > >> >> > > > > >> > >    can be removed after expiry.
> >> > >> >> > > > > >> > >    2. In catastrophic failures where all brokers
> go
> >> > down,
> >> > >> >> the
> >> > >> >> > > > tokens will
> >> > >> >> > > > > >> > >    be lost even if servers are restarted as
> tokens
> >> > are not
> >> > >> >> > > > persisted
> >> > >> >> > > > > >> > anywhere.
> >> > >> >> > > > > >> > >    If this happens, then there are more important
> >> > things to
> >> > >> >> > > worry
> >> > >> >> > > > about
> >> > >> >> > > > > >> > and
> >> > >> >> > > > > >> > >    maybe it is better to re-authenticate.
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth
> Brahmbhatt <
> >> > >> >> > > > > >> > > pbrahmbhatt@hortonworks.com> wrote:
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > >> Hi,
> >> > >> >> > > > > >> > >>
> >> > >> >> > > > > >> > >> I have filed KIP-48 so we can offer hadoop like
> >> > delegation
> >> > >> >> > > > tokens in
> >> > >> >> > > > > >> > >> kafka. You can review the design
> >> > >> >> > > > > >> > >>
> >> > >> >> > > > > >> >
> >> > >> >> > > >
> >> > >> >> > >
> >> > >> >>
> >> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> >> > >> >> > > > > >> > .
> >> > >> >> > > > > >> > >> This KIP depends on KIP-43 and we have also
> >> > discussed an
> >> > >> >> > > > alternative to
> >> > >> >> > > > > >> > >> proposed design here<
> >> > >> >> > > > > >> > >>
> >> > >> >> > > > > >> >
> >> > >> >> > > >
> >> > >> >> > >
> >> > >> >>
> >> >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> >> > >> >> > > > > >> > >> >.
> >> > >> >> > > > > >> > >>
> >> > >> >> > > > > >> > >> Thanks
> >> > >> >> > > > > >> > >> Parth
> >> > >> >> > > > > >> > >>
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > > --
> >> > >> >> > > > > >> > >
> >> > >> >> > > > > >> > > Regards,
> >> > >> >> > > > > >> > > Ashish
> >> > >> >> > > > > >> >
> >> > >> >> > > >
> >> > >> >> > >
> >> > >> >> > >
> >> > >> >> > >
> >> > >> >> > > --
> >> > >> >> > >
> >> > >> >> > > Regards,
> >> > >> >> > > Ashish
> >> > >> >> > >
> >> > >> >>
> >> >
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Gwen Shapira <gw...@confluent.io>]

Hi Parth and Harsha,

Few more comments:

* The API / RequestResponse section doesn't seem to have good
description of the changes to the Kafka Protocol. Sounds like you are
proposing new DelegationTokenRequest and RenewTokenRequest (and
matching responses), without detailing the contents of the requests
and responses? Or rather, you show the class interface, but not the
underlying protocol. This could be seen as an implementation detail,
but since the binary protocol is what we provide to non-Java clients,
we need to show the changes there.

* getDelegationToken sounds like delegationTokenRequestHandler? Is it
planned to be part of KafkaApi? or Client? Its unclear...

* I want to emphasize that even though delegation tokens are a Hadoop
innovation, I feel very strongly about not adding dependency on Hadoop
when implementing delegation tokens for Kafka. The KIP doesn't imply
such dependency, but if you can clarify...

* Can we get delegation token at any time after authenticating? only
immediately after?

* My understanding is that tokens will propagate via ZK but without
additional changes to UpdateMetadata protocol, correct? Clients
currently don't retry on SASL auth failure (IIRC), but since the
tokens propagate between brokers asynch, we will need to retry a bit
to avoid clients failing auth due to timing issues.

* Strongly agreeing on clients not touching ZK directly :)

* I liked Ashish's suggestion of having just the controller issue the
delegation tokens, to avoid syncing a shared secret. Not sure if we
want to continue the discussion here or on the wiki. I think that we
can decouple the problem of "token distribution" from "shared secret
distribution" and use the controller as the only token generator to
solve the second issue, while still using ZK async to distribute
tokens.

* I am also uncomfortable with infinite lifetime of tokens (and hoped
to hear from others in the community) - but having the option and
documenting it as less secure, allows users to configure their system
as the wish.

* While I like the idea of forcing kerberos auth for renwal, I think
it mixes the transport layer the the request content in a pretty ugly
way. Perhaps limiting renewer to non-owner is better.

Things I'd still like to see:

* More detailed explanation on what we plan to do for the java clients
specifically - new configuration? new APIs?
The response for my question on how multiple identities will be
handled wasn't super clear to me - AFAIK, we don't authenticate each
request, we authenticate connections.

* Alternatives: Delegation tokens are only used in the Hadoop
ecosystem. I'm wondering if there are alternatives in other ecosystems
(Mesos? Tachyon? Cassandra?) and whether there are some advantages
there.

Gwen

On Thu, May 12, 2016 at 1:05 PM, Harsha <ka...@harsha.io> wrote:
> Hi Gwen,
>            Can you look at Parth's last reply. Does it answer your
>            concerns.
>
> Thanks,
> Harsha
>
> On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt wrote:
>> Thanks for reviewing Gwen. The wiki already has details on token
>> expiration
>> under token acquisition process
>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition>.
>> Current proposal is that tokens will expire based on a server side
>> configuration (default 24 hours) unless renewed. Renewal is only allowed
>> until the max life time of token. Alternatively we could also make that
>> an
>> optional param and the server side default can serve as the upper bound.
>>
>> To your second point it will be done exactly the same way we would
>> support
>> multiple keytabs. The calling client will have to put the tokens it wants
>> to use in the subject instance and call produce/consume inside
>> subject.doas. Each caller will have to keep track of its own subject. I
>> will have to look at the code to see if we support this feature right now
>> but my understanding is delegation token shouldn't need any special
>> treatment as its just another type of Credential in the subject.
>>
>> I would also like to know what is your opinion about infinite renewal (my
>> recommendation is to not support this), tokens renewing them self(my
>> recommendation is to not support this) and most importantly your choice
>> between the alternatives listed on this thread
>> <http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism>
>> ( I am leaning towards the alternative-2 minus controller distributing
>> secret). Thanks again for reviewing.
>>
>> Thanks
>> Parth
>>
>>
>>
>> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <gw...@confluent.io> wrote:
>>
>> > Harsha,
>> >
>> > I was thinking of the Rest Proxy. I didn't see your design yet, but in
>> > our proxy, we have a set of producers, which will serve multiple users
>> > going through the proxy. Since these users will have different
>> > privileges, they'll need to authenticate separately, and can't share a
>> > token.
>> >
>> > Am I missing anything?
>> >
>> > Gwen
>> >
>> > On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io> wrote:
>> > > Gwen,
>> > >            On your second point. Can you describe a usecase where
>> > >            mutliple clients ended up sharing a producer and even if they
>> > >            do why can't they not use single token that producer
>> > >            captures. Why would we need multiple clients with different
>> > >            tokens sharing a single instance of producer.  Also in this
>> > >            case other clients have access all the tokens no?
>> > >
>> > > Thanks,
>> > > Harsha
>> > >
>> > >
>> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote:
>> > >> Sorry for the delay:
>> > >>
>> > >> Two questions that we didn't see in the wiki:
>> > >> 1. Is there an expiration for delegation tokens? Renewal? How do we
>> > >> revoke them?
>> > >> 2. If we want to use delegation tokens for "do-as" (say, submit Storm
>> > >> job as my user), we will need a producer for every job (we can't share
>> > >> them between multiple jobs running on same node), since we only
>> > >> authenticate when connecting. Is there a plan to change this for
>> > >> delegation tokens, in order to allow multiple users with different
>> > >> tokens to share a client?
>> > >>
>> > >> Gwen
>> > >>
>> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
>> > >> <br...@gmail.com> wrote:
>> > >> > Bumping this up one more time, can other committers review?
>> > >> >
>> > >> > Thanks
>> > >> > Parth
>> > >> >
>> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <ka...@harsha.io> wrote:
>> > >> >
>> > >> >> Parth,
>> > >> >>           Overall current design looks good to me. I am +1 on the
>> > KIP.
>> > >> >>
>> > >> >> Gwen , Jun can you review this as well.
>> > >> >>
>> > >> >> -Harsha
>> > >> >>
>> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote:
>> > >> >> > Thanks for review Jitendra.
>> > >> >> >
>> > >> >> > I don't like the idea of infinite lifetime but I see the Streaming
>> > use
>> > >> >> > case. Even for Streaming use case I was hoping there will be some
>> > notion
>> > >> >> > of
>> > >> >> > master/driver that can get new delegation tokens at fixed interval
>> > and
>> > >> >> > distribute to workers. If that is not the case for we can discuss
>> > >> >> > delegation tokens renewing them self and the security implications
>> > of the
>> > >> >> > same.
>> > >> >> >
>> > >> >> > I did not want clients to fetch tokens from zookeeper, overall I
>> > think
>> > >> >> > its
>> > >> >> > better if clients don't rely on our metadata store and I think we
>> > are
>> > >> >> > moving in that direction with all the KIP-4 improvements.  I chose
>> > >> >> > zookeeper as in this case the client will still just talk to
>> > broker , its
>> > >> >> > the brokers that will use zookeeper which we already do for a lot
>> > of
>> > >> >> > other
>> > >> >> > usecases + ease of development + and the ability so tokens will
>> > survive
>> > >> >> > even a rolling restart/cluster failure. if a majority agrees the
>> > added
>> > >> >> > complexity to have controller forwarding keys to all broker is
>> > justified
>> > >> >> > as
>> > >> >> > it provides tighter security , I am fine with that option too.
>> > >> >> >
>> > >> >> > Given zookeeper does not support SSL we can not store master keys
>> > in
>> > >> >> > zookeeper as master keys will be exposed on wire. To support
>> > rotation
>> > >> >> > without affecting current clients is something I need to put more
>> > thought
>> > >> >> > in. My current proposal assumes the rotation will invalidate all
>> > current
>> > >> >> > tokens.
>> > >> >> >
>> > >> >> > I request committers to also review and post their comments so we
>> > can
>> > >> >> > make
>> > >> >> > progress on this KIP.
>> > >> >> >
>> > >> >> > Thanks
>> > >> >> > Parth
>> > >> >> >
>> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <asingh@cloudera.com
>> > >
>> > >> >> > wrote:
>> > >> >> >
>> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <ka...@harsha.io>
>> > wrote:
>> > >> >> > >
>> > >> >> > > > Unifying the two discussion threads on this KIP.
>> > >> >> > > >
>> > >> >> > > > Here is the response from Jitendra
>> > >> >> > > >
>> > >> >> > > > "The need for a large number of clients that are running all
>> > over the
>> > >> >> > > > cluster that authenticate with Kafka brokers, is very similar
>> > to the
>> > >> >> > > > Hadoop use case of large number of tasks running across the
>> > cluster
>> > >> >> that
>> > >> >> > > > need authentication to Hdfs Namenode. Therefore, the
>> > delegation token
>> > >> >> > > > approach does seem like a good fit for this use case as we
>> > have seen
>> > >> >> it
>> > >> >> > > > working at large scale in HDFS and YARN.
>> > >> >> > > >
>> > >> >> > > >   The proposed design is very much inline with Hadoop
>> > approach. A few
>> > >> >> > > >   comments:
>> > >> >> > > >
>> > >> >> > > > 1) Why do you guys want to allow infinite renewable lifetime
>> > for a
>> > >> >> > > > token? HDFS restricts a token to a max life time (default 7
>> > days).  A
>> > >> >> > > > token's vulnerability is believed to increase with time.
>> > >> >> > > >
>> > >> >> > > I agree that having infinite lifetime might not be the best idea.
>> > >> >> > >
>> > >> >> > > >
>> > >> >> > > > 2) As I understand the tokens are stored in zookeeper as well,
>> > and
>> > >> >> can
>> > >> >> > > > be updated there. This is clever as it can allow replacing the
>> > tokens
>> > >> >> > > > once they run out of max life time, and clients can download
>> > new
>> > >> >> tokens
>> > >> >> > > > from zookeeper. It shouldn't be a big load on zookeeper as a
>> > client
>> > >> >> will
>> > >> >> > > > need to get a new token once in several days. In this approach
>> > you
>> > >> >> don't
>> > >> >> > > > need infinite lifetime on the token even for long running
>> > clients.
>> > >> >> > > >
>> > >> >> > > > 3) The token password are generated using a master key. The
>> > master
>> > >> >> key
>> > >> >> > > > should also be periodically changed. In Hadoop, the default
>> > renewal
>> > >> >> > > > period is 1 day.?
>> > >> >> > > >
>> > >> >> > > IIUC, this will require brokers maintaining a list of X most
>> > recent
>> > >> >> master
>> > >> >> > > keys. This list will have to be persisted somewhere, as if a
>> > broker
>> > >> >> goes
>> > >> >> > > down it will have to get that list again and storing master keys
>> > on ZK
>> > >> >> is
>> > >> >> > > not the best idea. However, if a broker goes down then we have
>> > much
>> > >> >> bigger
>> > >> >> > > issue to deal with and client can always re-authenticate is such
>> > >> >> events.
>> > >> >> > >
>> > >> >> > > Did you happen to take a look at other alternatives this list has
>> > >> >> > > suggested?
>> > >> >> > >
>> > >> >> > > >
>> > >> >> > > > Thanks for a thorough proposal, great work!"
>> > >> >> > > >
>> > >> >> > > >
>> > >> >> > > >
>> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
>> > >> >> > > > > Makes sense to me. Thanks!
>> > >> >> > > > >
>> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <ka...@harsha.io>
>> > wrote:
>> > >> >> > > > > > It doesn't need any release vehicle but still the work can
>> > move
>> > >> >> > > > forward.
>> > >> >> > > > > > If anyone is interested in the KIP please do the review and
>> > >> >> provide
>> > >> >> > > the
>> > >> >> > > > > > comments.
>> > >> >> > > > > >
>> > >> >> > > > > > -Harsha
>> > >> >> > > > > >
>> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma wrote:
>> > >> >> > > > > >> I agree that it would be good to have more time to review
>> > and
>> > >> >> > > discuss
>> > >> >> > > > > >> KIP-48.
>> > >> >> > > > > >>
>> > >> >> > > > > >> Ismael
>> > >> >> > > > > >>
>> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <
>> > >> >> gwen@confluent.io>
>> > >> >> > > > wrote:
>> > >> >> > > > > >>
>> > >> >> > > > > >> > Hi Team,
>> > >> >> > > > > >> >
>> > >> >> > > > > >> > Since KIP-48 depends on KIP-43, which is already a bit
>> > of a
>> > >> >> risk
>> > >> >> > > for
>> > >> >> > > > > >> > the next release - any chance we can delay delegation
>> > tokens
>> > >> >> to
>> > >> >> > > > Kafka
>> > >> >> > > > > >> > 0.10.1?
>> > >> >> > > > > >> > With the community working on a release every 3 month,
>> > this
>> > >> >> is not
>> > >> >> > > > a huge
>> > >> >> > > > > >> > delay.
>> > >> >> > > > > >> >
>> > >> >> > > > > >> > Gwen
>> > >> >> > > > > >> >
>> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh <
>> > >> >> > > asingh@cloudera.com>
>> > >> >> > > > wrote:
>> > >> >> > > > > >> > > Parth,
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > > Thanks again for the awesome write up. Following our
>> > >> >> discussion
>> > >> >> > > > from the
>> > >> >> > > > > >> > > JIRA, I think it will be easier to compare various
>> > >> >> alternatives
>> > >> >> > > > if they
>> > >> >> > > > > >> > are
>> > >> >> > > > > >> > > listed together. I am stating below a few
>> > alternatives along
>> > >> >> > > with
>> > >> >> > > > a the
>> > >> >> > > > > >> > > current proposal.
>> > >> >> > > > > >> > > (Current proposal) Store Delegation Token, DT, on ZK.
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > >    1. Client authenticates with a broker.
>> > >> >> > > > > >> > >    2. Once a client is authenticated, it will make a
>> > broker
>> > >> >> side
>> > >> >> > > > call to
>> > >> >> > > > > >> > >    issue a delegation token.
>> > >> >> > > > > >> > >    3. The broker generates a shared secret based on
>> > >> >> > > HMAC-SHA256(a
>> > >> >> > > > > >> > >    Password/Secret shared between all brokers,
>> > randomly
>> > >> >> > > generated
>> > >> >> > > > > >> > tokenId).
>> > >> >> > > > > >> > >    4. Broker stores this token in its in memory cache.
>> > >> >> Broker
>> > >> >> > > > also stores
>> > >> >> > > > > >> > >    the DelegationToken without the hmac in the
>> > zookeeper.
>> > >> >> > > > > >> > >    5. All brokers will have a cache backed by
>> > zookeeper so
>> > >> >> they
>> > >> >> > > > will all
>> > >> >> > > > > >> > >    get notified whenever a new token is generated and
>> > they
>> > >> >> will
>> > >> >> > > > update
>> > >> >> > > > > >> > their
>> > >> >> > > > > >> > >    local cache whenever token state changes.
>> > >> >> > > > > >> > >    6. Broker returns the token to Client.
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > > Probable issues and fixes
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > >    1. Probable race condition, client tries to
>> > authenticate
>> > >> >> with
>> > >> >> > > > a broker
>> > >> >> > > > > >> > >    that is yet to be updated with the newly generated
>> > DT.
>> > >> >> This
>> > >> >> > > can
>> > >> >> > > > > >> > probably be
>> > >> >> > > > > >> > >    dealt with making dtRequest block until all
>> > brokers have
>> > >> >> > > > updated
>> > >> >> > > > > >> > their DT
>> > >> >> > > > > >> > >    cache. Zk barrier or similar mechanism can be used.
>> > >> >> However,
>> > >> >> > > > all such
>> > >> >> > > > > >> > >    mechanisms will increase complexity.
>> > >> >> > > > > >> > >    2. Using a static secret key from config file. Will
>> > >> >> require
>> > >> >> > > yet
>> > >> >> > > > > >> > another
>> > >> >> > > > > >> > >    config and uses a static secret key. It is advised
>> > to
>> > >> >> rotate
>> > >> >> > > > secret
>> > >> >> > > > > >> > keys
>> > >> >> > > > > >> > >    periodically. This can be avoided with controller
>> > >> >> generating
>> > >> >> > > > > >> > secretKey and
>> > >> >> > > > > >> > >    passing to brokers periodically. However, this will
>> > >> >> require
>> > >> >> > > > brokers to
>> > >> >> > > > > >> > >    maintain certain counts of secretKeys.
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > > (Alternative 1) Have controller generate delegation
>> > token.
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > >    1. Client authenticates with a broker.
>> > >> >> > > > > >> > >    2. Once a client is authenticated, it will make a
>> > broker
>> > >> >> side
>> > >> >> > > > call to
>> > >> >> > > > > >> > >    issue a delegation token.
>> > >> >> > > > > >> > >    3. Broker forwards the request to controller.
>> > >> >> > > > > >> > >    4. Controller generates a DT and broadcasts to all
>> > >> >> brokers.
>> > >> >> > > > > >> > >    5. Broker stores this token in its memory cache.
>> > >> >> > > > > >> > >    6. Controller responds to broker’s DT req.
>> > >> >> > > > > >> > >    7. Broker returns the token to Client.
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > > Probable issues and fixes
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > >    1. We will have to add new APIs to support
>> > controller
>> > >> >> pushing
>> > >> >> > > > tokens
>> > >> >> > > > > >> > to
>> > >> >> > > > > >> > >    brokers on top of the minimal APIs that are
>> > currently
>> > >> >> > > proposed.
>> > >> >> > > > > >> > >    2. We will also have to add APIs to support the
>> > >> >> bootstrapping
>> > >> >> > > > case,
>> > >> >> > > > > >> > i.e,
>> > >> >> > > > > >> > >    when a new broker comes up it will have to get all
>> > >> >> delegation
>> > >> >> > > > tokens
>> > >> >> > > > > >> > from
>> > >> >> > > > > >> > >    the controller.
>> > >> >> > > > > >> > >    3. In catastrophic failures where all brokers go
>> > down,
>> > >> >> the
>> > >> >> > > > tokens will
>> > >> >> > > > > >> > >    be lost even if servers are restarted as tokens
>> > are not
>> > >> >> > > > persisted
>> > >> >> > > > > >> > anywhere.
>> > >> >> > > > > >> > >    If this happens, then there are more important
>> > things to
>> > >> >> > > worry
>> > >> >> > > > about
>> > >> >> > > > > >> > and
>> > >> >> > > > > >> > >    maybe it is better to re-authenticate.
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > > (Alternative 2) Do not distribute DT to other brokers
>> > at
>> > >> >> all.
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > >    1. Client authenticates with a broker.
>> > >> >> > > > > >> > >    2. Once a client is authenticated, it will make a
>> > broker
>> > >> >> side
>> > >> >> > > > call to
>> > >> >> > > > > >> > >    issue a delegation token.
>> > >> >> > > > > >> > >    3. The broker generates DT of form, [hmac + (owner,
>> > >> >> renewer,
>> > >> >> > > > > >> > >    maxLifeTime, id, hmac, expirationTime)] and passes
>> > back
>> > >> >> this
>> > >> >> > > > DT to
>> > >> >> > > > > >> > client.
>> > >> >> > > > > >> > >    hmac is generated via {HMAC-SHA256(owner, renewer,
>> > >> >> > > > maxLifeTime, id,
>> > >> >> > > > > >> > hmac,
>> > >> >> > > > > >> > >    expirationTime) using SecretKey}. Note that all
>> > brokers
>> > >> >> have
>> > >> >> > > > this
>> > >> >> > > > > >> > SecretKey.
>> > >> >> > > > > >> > >    4. Client then goes to any broker and to
>> > authenticate
>> > >> >> sends
>> > >> >> > > > the DT.
>> > >> >> > > > > >> > >    Broker recalculates hmac using (owner, renewer,
>> > >> >> maxLifeTime,
>> > >> >> > > > id, hmac,
>> > >> >> > > > > >> > >    expirationTime) info from DT and its SecretKey. If
>> > it
>> > >> >> matches
>> > >> >> > > > with
>> > >> >> > > > > >> > hmac of
>> > >> >> > > > > >> > >    DT, client is authenticated. Yes, it will do other
>> > >> >> obvious
>> > >> >> > > > checks of
>> > >> >> > > > > >> > >    timestamp expiry and such.
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > > Note that secret key will be generated by controller
>> > and
>> > >> >> passed
>> > >> >> > > to
>> > >> >> > > > > >> > brokers
>> > >> >> > > > > >> > > periodically.
>> > >> >> > > > > >> > > Probable issues and fixes
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > >    1. How to delete a DT? Yes, that is a downside
>> > here.
>> > >> >> However,
>> > >> >> > > > this can
>> > >> >> > > > > >> > >    be handled with brokers maintaining a blacklist of
>> > DTs,
>> > >> >> DTs
>> > >> >> > > > from this
>> > >> >> > > > > >> > list
>> > >> >> > > > > >> > >    can be removed after expiry.
>> > >> >> > > > > >> > >    2. In catastrophic failures where all brokers go
>> > down,
>> > >> >> the
>> > >> >> > > > tokens will
>> > >> >> > > > > >> > >    be lost even if servers are restarted as tokens
>> > are not
>> > >> >> > > > persisted
>> > >> >> > > > > >> > anywhere.
>> > >> >> > > > > >> > >    If this happens, then there are more important
>> > things to
>> > >> >> > > worry
>> > >> >> > > > about
>> > >> >> > > > > >> > and
>> > >> >> > > > > >> > >    maybe it is better to re-authenticate.
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth Brahmbhatt <
>> > >> >> > > > > >> > > pbrahmbhatt@hortonworks.com> wrote:
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > >> Hi,
>> > >> >> > > > > >> > >>
>> > >> >> > > > > >> > >> I have filed KIP-48 so we can offer hadoop like
>> > delegation
>> > >> >> > > > tokens in
>> > >> >> > > > > >> > >> kafka. You can review the design
>> > >> >> > > > > >> > >>
>> > >> >> > > > > >> >
>> > >> >> > > >
>> > >> >> > >
>> > >> >>
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
>> > >> >> > > > > >> > .
>> > >> >> > > > > >> > >> This KIP depends on KIP-43 and we have also
>> > discussed an
>> > >> >> > > > alternative to
>> > >> >> > > > > >> > >> proposed design here<
>> > >> >> > > > > >> > >>
>> > >> >> > > > > >> >
>> > >> >> > > >
>> > >> >> > >
>> > >> >>
>> > https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
>> > >> >> > > > > >> > >> >.
>> > >> >> > > > > >> > >>
>> > >> >> > > > > >> > >> Thanks
>> > >> >> > > > > >> > >> Parth
>> > >> >> > > > > >> > >>
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > > --
>> > >> >> > > > > >> > >
>> > >> >> > > > > >> > > Regards,
>> > >> >> > > > > >> > > Ashish
>> > >> >> > > > > >> >
>> > >> >> > > >
>> > >> >> > >
>> > >> >> > >
>> > >> >> > >
>> > >> >> > > --
>> > >> >> > >
>> > >> >> > > Regards,
>> > >> >> > > Ashish
>> > >> >> > >
>> > >> >>
>> >

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Harsha <ka...@harsha.io>]

Hi Gwen,
           Can you look at Parth's last reply. Does it answer your
           concerns.

Thanks,
Harsha

On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt wrote:
> Thanks for reviewing Gwen. The wiki already has details on token
> expiration
> under token acquisition process
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition>.
> Current proposal is that tokens will expire based on a server side
> configuration (default 24 hours) unless renewed. Renewal is only allowed
> until the max life time of token. Alternatively we could also make that
> an
> optional param and the server side default can serve as the upper bound.
> 
> To your second point it will be done exactly the same way we would
> support
> multiple keytabs. The calling client will have to put the tokens it wants
> to use in the subject instance and call produce/consume inside
> subject.doas. Each caller will have to keep track of its own subject. I
> will have to look at the code to see if we support this feature right now
> but my understanding is delegation token shouldn't need any special
> treatment as its just another type of Credential in the subject.
> 
> I would also like to know what is your opinion about infinite renewal (my
> recommendation is to not support this), tokens renewing them self(my
> recommendation is to not support this) and most importantly your choice
> between the alternatives listed on this thread
> <http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism>
> ( I am leaning towards the alternative-2 minus controller distributing
> secret). Thanks again for reviewing.
> 
> Thanks
> Parth
> 
> 
> 
> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <gw...@confluent.io> wrote:
> 
> > Harsha,
> >
> > I was thinking of the Rest Proxy. I didn't see your design yet, but in
> > our proxy, we have a set of producers, which will serve multiple users
> > going through the proxy. Since these users will have different
> > privileges, they'll need to authenticate separately, and can't share a
> > token.
> >
> > Am I missing anything?
> >
> > Gwen
> >
> > On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io> wrote:
> > > Gwen,
> > >            On your second point. Can you describe a usecase where
> > >            mutliple clients ended up sharing a producer and even if they
> > >            do why can't they not use single token that producer
> > >            captures. Why would we need multiple clients with different
> > >            tokens sharing a single instance of producer.  Also in this
> > >            case other clients have access all the tokens no?
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote:
> > >> Sorry for the delay:
> > >>
> > >> Two questions that we didn't see in the wiki:
> > >> 1. Is there an expiration for delegation tokens? Renewal? How do we
> > >> revoke them?
> > >> 2. If we want to use delegation tokens for "do-as" (say, submit Storm
> > >> job as my user), we will need a producer for every job (we can't share
> > >> them between multiple jobs running on same node), since we only
> > >> authenticate when connecting. Is there a plan to change this for
> > >> delegation tokens, in order to allow multiple users with different
> > >> tokens to share a client?
> > >>
> > >> Gwen
> > >>
> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
> > >> <br...@gmail.com> wrote:
> > >> > Bumping this up one more time, can other committers review?
> > >> >
> > >> > Thanks
> > >> > Parth
> > >> >
> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <ka...@harsha.io> wrote:
> > >> >
> > >> >> Parth,
> > >> >>           Overall current design looks good to me. I am +1 on the
> > KIP.
> > >> >>
> > >> >> Gwen , Jun can you review this as well.
> > >> >>
> > >> >> -Harsha
> > >> >>
> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote:
> > >> >> > Thanks for review Jitendra.
> > >> >> >
> > >> >> > I don't like the idea of infinite lifetime but I see the Streaming
> > use
> > >> >> > case. Even for Streaming use case I was hoping there will be some
> > notion
> > >> >> > of
> > >> >> > master/driver that can get new delegation tokens at fixed interval
> > and
> > >> >> > distribute to workers. If that is not the case for we can discuss
> > >> >> > delegation tokens renewing them self and the security implications
> > of the
> > >> >> > same.
> > >> >> >
> > >> >> > I did not want clients to fetch tokens from zookeeper, overall I
> > think
> > >> >> > its
> > >> >> > better if clients don't rely on our metadata store and I think we
> > are
> > >> >> > moving in that direction with all the KIP-4 improvements.  I chose
> > >> >> > zookeeper as in this case the client will still just talk to
> > broker , its
> > >> >> > the brokers that will use zookeeper which we already do for a lot
> > of
> > >> >> > other
> > >> >> > usecases + ease of development + and the ability so tokens will
> > survive
> > >> >> > even a rolling restart/cluster failure. if a majority agrees the
> > added
> > >> >> > complexity to have controller forwarding keys to all broker is
> > justified
> > >> >> > as
> > >> >> > it provides tighter security , I am fine with that option too.
> > >> >> >
> > >> >> > Given zookeeper does not support SSL we can not store master keys
> > in
> > >> >> > zookeeper as master keys will be exposed on wire. To support
> > rotation
> > >> >> > without affecting current clients is something I need to put more
> > thought
> > >> >> > in. My current proposal assumes the rotation will invalidate all
> > current
> > >> >> > tokens.
> > >> >> >
> > >> >> > I request committers to also review and post their comments so we
> > can
> > >> >> > make
> > >> >> > progress on this KIP.
> > >> >> >
> > >> >> > Thanks
> > >> >> > Parth
> > >> >> >
> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <asingh@cloudera.com
> > >
> > >> >> > wrote:
> > >> >> >
> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <ka...@harsha.io>
> > wrote:
> > >> >> > >
> > >> >> > > > Unifying the two discussion threads on this KIP.
> > >> >> > > >
> > >> >> > > > Here is the response from Jitendra
> > >> >> > > >
> > >> >> > > > "The need for a large number of clients that are running all
> > over the
> > >> >> > > > cluster that authenticate with Kafka brokers, is very similar
> > to the
> > >> >> > > > Hadoop use case of large number of tasks running across the
> > cluster
> > >> >> that
> > >> >> > > > need authentication to Hdfs Namenode. Therefore, the
> > delegation token
> > >> >> > > > approach does seem like a good fit for this use case as we
> > have seen
> > >> >> it
> > >> >> > > > working at large scale in HDFS and YARN.
> > >> >> > > >
> > >> >> > > >   The proposed design is very much inline with Hadoop
> > approach. A few
> > >> >> > > >   comments:
> > >> >> > > >
> > >> >> > > > 1) Why do you guys want to allow infinite renewable lifetime
> > for a
> > >> >> > > > token? HDFS restricts a token to a max life time (default 7
> > days).  A
> > >> >> > > > token's vulnerability is believed to increase with time.
> > >> >> > > >
> > >> >> > > I agree that having infinite lifetime might not be the best idea.
> > >> >> > >
> > >> >> > > >
> > >> >> > > > 2) As I understand the tokens are stored in zookeeper as well,
> > and
> > >> >> can
> > >> >> > > > be updated there. This is clever as it can allow replacing the
> > tokens
> > >> >> > > > once they run out of max life time, and clients can download
> > new
> > >> >> tokens
> > >> >> > > > from zookeeper. It shouldn't be a big load on zookeeper as a
> > client
> > >> >> will
> > >> >> > > > need to get a new token once in several days. In this approach
> > you
> > >> >> don't
> > >> >> > > > need infinite lifetime on the token even for long running
> > clients.
> > >> >> > > >
> > >> >> > > > 3) The token password are generated using a master key. The
> > master
> > >> >> key
> > >> >> > > > should also be periodically changed. In Hadoop, the default
> > renewal
> > >> >> > > > period is 1 day.?
> > >> >> > > >
> > >> >> > > IIUC, this will require brokers maintaining a list of X most
> > recent
> > >> >> master
> > >> >> > > keys. This list will have to be persisted somewhere, as if a
> > broker
> > >> >> goes
> > >> >> > > down it will have to get that list again and storing master keys
> > on ZK
> > >> >> is
> > >> >> > > not the best idea. However, if a broker goes down then we have
> > much
> > >> >> bigger
> > >> >> > > issue to deal with and client can always re-authenticate is such
> > >> >> events.
> > >> >> > >
> > >> >> > > Did you happen to take a look at other alternatives this list has
> > >> >> > > suggested?
> > >> >> > >
> > >> >> > > >
> > >> >> > > > Thanks for a thorough proposal, great work!"
> > >> >> > > >
> > >> >> > > >
> > >> >> > > >
> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
> > >> >> > > > > Makes sense to me. Thanks!
> > >> >> > > > >
> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <ka...@harsha.io>
> > wrote:
> > >> >> > > > > > It doesn't need any release vehicle but still the work can
> > move
> > >> >> > > > forward.
> > >> >> > > > > > If anyone is interested in the KIP please do the review and
> > >> >> provide
> > >> >> > > the
> > >> >> > > > > > comments.
> > >> >> > > > > >
> > >> >> > > > > > -Harsha
> > >> >> > > > > >
> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma wrote:
> > >> >> > > > > >> I agree that it would be good to have more time to review
> > and
> > >> >> > > discuss
> > >> >> > > > > >> KIP-48.
> > >> >> > > > > >>
> > >> >> > > > > >> Ismael
> > >> >> > > > > >>
> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <
> > >> >> gwen@confluent.io>
> > >> >> > > > wrote:
> > >> >> > > > > >>
> > >> >> > > > > >> > Hi Team,
> > >> >> > > > > >> >
> > >> >> > > > > >> > Since KIP-48 depends on KIP-43, which is already a bit
> > of a
> > >> >> risk
> > >> >> > > for
> > >> >> > > > > >> > the next release - any chance we can delay delegation
> > tokens
> > >> >> to
> > >> >> > > > Kafka
> > >> >> > > > > >> > 0.10.1?
> > >> >> > > > > >> > With the community working on a release every 3 month,
> > this
> > >> >> is not
> > >> >> > > > a huge
> > >> >> > > > > >> > delay.
> > >> >> > > > > >> >
> > >> >> > > > > >> > Gwen
> > >> >> > > > > >> >
> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh <
> > >> >> > > asingh@cloudera.com>
> > >> >> > > > wrote:
> > >> >> > > > > >> > > Parth,
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > Thanks again for the awesome write up. Following our
> > >> >> discussion
> > >> >> > > > from the
> > >> >> > > > > >> > > JIRA, I think it will be easier to compare various
> > >> >> alternatives
> > >> >> > > > if they
> > >> >> > > > > >> > are
> > >> >> > > > > >> > > listed together. I am stating below a few
> > alternatives along
> > >> >> > > with
> > >> >> > > > a the
> > >> >> > > > > >> > > current proposal.
> > >> >> > > > > >> > > (Current proposal) Store Delegation Token, DT, on ZK.
> > >> >> > > > > >> > >
> > >> >> > > > > >> > >    1. Client authenticates with a broker.
> > >> >> > > > > >> > >    2. Once a client is authenticated, it will make a
> > broker
> > >> >> side
> > >> >> > > > call to
> > >> >> > > > > >> > >    issue a delegation token.
> > >> >> > > > > >> > >    3. The broker generates a shared secret based on
> > >> >> > > HMAC-SHA256(a
> > >> >> > > > > >> > >    Password/Secret shared between all brokers,
> > randomly
> > >> >> > > generated
> > >> >> > > > > >> > tokenId).
> > >> >> > > > > >> > >    4. Broker stores this token in its in memory cache.
> > >> >> Broker
> > >> >> > > > also stores
> > >> >> > > > > >> > >    the DelegationToken without the hmac in the
> > zookeeper.
> > >> >> > > > > >> > >    5. All brokers will have a cache backed by
> > zookeeper so
> > >> >> they
> > >> >> > > > will all
> > >> >> > > > > >> > >    get notified whenever a new token is generated and
> > they
> > >> >> will
> > >> >> > > > update
> > >> >> > > > > >> > their
> > >> >> > > > > >> > >    local cache whenever token state changes.
> > >> >> > > > > >> > >    6. Broker returns the token to Client.
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > Probable issues and fixes
> > >> >> > > > > >> > >
> > >> >> > > > > >> > >    1. Probable race condition, client tries to
> > authenticate
> > >> >> with
> > >> >> > > > a broker
> > >> >> > > > > >> > >    that is yet to be updated with the newly generated
> > DT.
> > >> >> This
> > >> >> > > can
> > >> >> > > > > >> > probably be
> > >> >> > > > > >> > >    dealt with making dtRequest block until all
> > brokers have
> > >> >> > > > updated
> > >> >> > > > > >> > their DT
> > >> >> > > > > >> > >    cache. Zk barrier or similar mechanism can be used.
> > >> >> However,
> > >> >> > > > all such
> > >> >> > > > > >> > >    mechanisms will increase complexity.
> > >> >> > > > > >> > >    2. Using a static secret key from config file. Will
> > >> >> require
> > >> >> > > yet
> > >> >> > > > > >> > another
> > >> >> > > > > >> > >    config and uses a static secret key. It is advised
> > to
> > >> >> rotate
> > >> >> > > > secret
> > >> >> > > > > >> > keys
> > >> >> > > > > >> > >    periodically. This can be avoided with controller
> > >> >> generating
> > >> >> > > > > >> > secretKey and
> > >> >> > > > > >> > >    passing to brokers periodically. However, this will
> > >> >> require
> > >> >> > > > brokers to
> > >> >> > > > > >> > >    maintain certain counts of secretKeys.
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > (Alternative 1) Have controller generate delegation
> > token.
> > >> >> > > > > >> > >
> > >> >> > > > > >> > >    1. Client authenticates with a broker.
> > >> >> > > > > >> > >    2. Once a client is authenticated, it will make a
> > broker
> > >> >> side
> > >> >> > > > call to
> > >> >> > > > > >> > >    issue a delegation token.
> > >> >> > > > > >> > >    3. Broker forwards the request to controller.
> > >> >> > > > > >> > >    4. Controller generates a DT and broadcasts to all
> > >> >> brokers.
> > >> >> > > > > >> > >    5. Broker stores this token in its memory cache.
> > >> >> > > > > >> > >    6. Controller responds to broker’s DT req.
> > >> >> > > > > >> > >    7. Broker returns the token to Client.
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > Probable issues and fixes
> > >> >> > > > > >> > >
> > >> >> > > > > >> > >    1. We will have to add new APIs to support
> > controller
> > >> >> pushing
> > >> >> > > > tokens
> > >> >> > > > > >> > to
> > >> >> > > > > >> > >    brokers on top of the minimal APIs that are
> > currently
> > >> >> > > proposed.
> > >> >> > > > > >> > >    2. We will also have to add APIs to support the
> > >> >> bootstrapping
> > >> >> > > > case,
> > >> >> > > > > >> > i.e,
> > >> >> > > > > >> > >    when a new broker comes up it will have to get all
> > >> >> delegation
> > >> >> > > > tokens
> > >> >> > > > > >> > from
> > >> >> > > > > >> > >    the controller.
> > >> >> > > > > >> > >    3. In catastrophic failures where all brokers go
> > down,
> > >> >> the
> > >> >> > > > tokens will
> > >> >> > > > > >> > >    be lost even if servers are restarted as tokens
> > are not
> > >> >> > > > persisted
> > >> >> > > > > >> > anywhere.
> > >> >> > > > > >> > >    If this happens, then there are more important
> > things to
> > >> >> > > worry
> > >> >> > > > about
> > >> >> > > > > >> > and
> > >> >> > > > > >> > >    maybe it is better to re-authenticate.
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > (Alternative 2) Do not distribute DT to other brokers
> > at
> > >> >> all.
> > >> >> > > > > >> > >
> > >> >> > > > > >> > >    1. Client authenticates with a broker.
> > >> >> > > > > >> > >    2. Once a client is authenticated, it will make a
> > broker
> > >> >> side
> > >> >> > > > call to
> > >> >> > > > > >> > >    issue a delegation token.
> > >> >> > > > > >> > >    3. The broker generates DT of form, [hmac + (owner,
> > >> >> renewer,
> > >> >> > > > > >> > >    maxLifeTime, id, hmac, expirationTime)] and passes
> > back
> > >> >> this
> > >> >> > > > DT to
> > >> >> > > > > >> > client.
> > >> >> > > > > >> > >    hmac is generated via {HMAC-SHA256(owner, renewer,
> > >> >> > > > maxLifeTime, id,
> > >> >> > > > > >> > hmac,
> > >> >> > > > > >> > >    expirationTime) using SecretKey}. Note that all
> > brokers
> > >> >> have
> > >> >> > > > this
> > >> >> > > > > >> > SecretKey.
> > >> >> > > > > >> > >    4. Client then goes to any broker and to
> > authenticate
> > >> >> sends
> > >> >> > > > the DT.
> > >> >> > > > > >> > >    Broker recalculates hmac using (owner, renewer,
> > >> >> maxLifeTime,
> > >> >> > > > id, hmac,
> > >> >> > > > > >> > >    expirationTime) info from DT and its SecretKey. If
> > it
> > >> >> matches
> > >> >> > > > with
> > >> >> > > > > >> > hmac of
> > >> >> > > > > >> > >    DT, client is authenticated. Yes, it will do other
> > >> >> obvious
> > >> >> > > > checks of
> > >> >> > > > > >> > >    timestamp expiry and such.
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > Note that secret key will be generated by controller
> > and
> > >> >> passed
> > >> >> > > to
> > >> >> > > > > >> > brokers
> > >> >> > > > > >> > > periodically.
> > >> >> > > > > >> > > Probable issues and fixes
> > >> >> > > > > >> > >
> > >> >> > > > > >> > >    1. How to delete a DT? Yes, that is a downside
> > here.
> > >> >> However,
> > >> >> > > > this can
> > >> >> > > > > >> > >    be handled with brokers maintaining a blacklist of
> > DTs,
> > >> >> DTs
> > >> >> > > > from this
> > >> >> > > > > >> > list
> > >> >> > > > > >> > >    can be removed after expiry.
> > >> >> > > > > >> > >    2. In catastrophic failures where all brokers go
> > down,
> > >> >> the
> > >> >> > > > tokens will
> > >> >> > > > > >> > >    be lost even if servers are restarted as tokens
> > are not
> > >> >> > > > persisted
> > >> >> > > > > >> > anywhere.
> > >> >> > > > > >> > >    If this happens, then there are more important
> > things to
> > >> >> > > worry
> > >> >> > > > about
> > >> >> > > > > >> > and
> > >> >> > > > > >> > >    maybe it is better to re-authenticate.
> > >> >> > > > > >> > >
> > >> >> > > > > >> > >
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth Brahmbhatt <
> > >> >> > > > > >> > > pbrahmbhatt@hortonworks.com> wrote:
> > >> >> > > > > >> > >
> > >> >> > > > > >> > >> Hi,
> > >> >> > > > > >> > >>
> > >> >> > > > > >> > >> I have filed KIP-48 so we can offer hadoop like
> > delegation
> > >> >> > > > tokens in
> > >> >> > > > > >> > >> kafka. You can review the design
> > >> >> > > > > >> > >>
> > >> >> > > > > >> >
> > >> >> > > >
> > >> >> > >
> > >> >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > >> >> > > > > >> > .
> > >> >> > > > > >> > >> This KIP depends on KIP-43 and we have also
> > discussed an
> > >> >> > > > alternative to
> > >> >> > > > > >> > >> proposed design here<
> > >> >> > > > > >> > >>
> > >> >> > > > > >> >
> > >> >> > > >
> > >> >> > >
> > >> >>
> > https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > >> >> > > > > >> > >> >.
> > >> >> > > > > >> > >>
> > >> >> > > > > >> > >> Thanks
> > >> >> > > > > >> > >> Parth
> > >> >> > > > > >> > >>
> > >> >> > > > > >> > >
> > >> >> > > > > >> > >
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > --
> > >> >> > > > > >> > >
> > >> >> > > > > >> > > Regards,
> > >> >> > > > > >> > > Ashish
> > >> >> > > > > >> >
> > >> >> > > >
> > >> >> > >
> > >> >> > >
> > >> >> > >
> > >> >> > > --
> > >> >> > >
> > >> >> > > Regards,
> > >> >> > > Ashish
> > >> >> > >
> > >> >>
> >

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[parth brahmbhatt <br...@gmail.com>]

Thanks for reviewing Gwen. The wiki already has details on token expiration
under token acquisition process
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition>.
Current proposal is that tokens will expire based on a server side
configuration (default 24 hours) unless renewed. Renewal is only allowed
until the max life time of token. Alternatively we could also make that an
optional param and the server side default can serve as the upper bound.

To your second point it will be done exactly the same way we would support
multiple keytabs. The calling client will have to put the tokens it wants
to use in the subject instance and call produce/consume inside
subject.doas. Each caller will have to keep track of its own subject. I
will have to look at the code to see if we support this feature right now
but my understanding is delegation token shouldn't need any special
treatment as its just another type of Credential in the subject.

I would also like to know what is your opinion about infinite renewal (my
recommendation is to not support this), tokens renewing them self(my
recommendation is to not support this) and most importantly your choice
between the alternatives listed on this thread
<http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism>
( I am leaning towards the alternative-2 minus controller distributing
secret). Thanks again for reviewing.

Thanks
Parth



On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <gw...@confluent.io> wrote:

> Harsha,
>
> I was thinking of the Rest Proxy. I didn't see your design yet, but in
> our proxy, we have a set of producers, which will serve multiple users
> going through the proxy. Since these users will have different
> privileges, they'll need to authenticate separately, and can't share a
> token.
>
> Am I missing anything?
>
> Gwen
>
> On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io> wrote:
> > Gwen,
> >            On your second point. Can you describe a usecase where
> >            mutliple clients ended up sharing a producer and even if they
> >            do why can't they not use single token that producer
> >            captures. Why would we need multiple clients with different
> >            tokens sharing a single instance of producer.  Also in this
> >            case other clients have access all the tokens no?
> >
> > Thanks,
> > Harsha
> >
> >
> > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote:
> >> Sorry for the delay:
> >>
> >> Two questions that we didn't see in the wiki:
> >> 1. Is there an expiration for delegation tokens? Renewal? How do we
> >> revoke them?
> >> 2. If we want to use delegation tokens for "do-as" (say, submit Storm
> >> job as my user), we will need a producer for every job (we can't share
> >> them between multiple jobs running on same node), since we only
> >> authenticate when connecting. Is there a plan to change this for
> >> delegation tokens, in order to allow multiple users with different
> >> tokens to share a client?
> >>
> >> Gwen
> >>
> >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
> >> <br...@gmail.com> wrote:
> >> > Bumping this up one more time, can other committers review?
> >> >
> >> > Thanks
> >> > Parth
> >> >
> >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <ka...@harsha.io> wrote:
> >> >
> >> >> Parth,
> >> >>           Overall current design looks good to me. I am +1 on the
> KIP.
> >> >>
> >> >> Gwen , Jun can you review this as well.
> >> >>
> >> >> -Harsha
> >> >>
> >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote:
> >> >> > Thanks for review Jitendra.
> >> >> >
> >> >> > I don't like the idea of infinite lifetime but I see the Streaming
> use
> >> >> > case. Even for Streaming use case I was hoping there will be some
> notion
> >> >> > of
> >> >> > master/driver that can get new delegation tokens at fixed interval
> and
> >> >> > distribute to workers. If that is not the case for we can discuss
> >> >> > delegation tokens renewing them self and the security implications
> of the
> >> >> > same.
> >> >> >
> >> >> > I did not want clients to fetch tokens from zookeeper, overall I
> think
> >> >> > its
> >> >> > better if clients don't rely on our metadata store and I think we
> are
> >> >> > moving in that direction with all the KIP-4 improvements.  I chose
> >> >> > zookeeper as in this case the client will still just talk to
> broker , its
> >> >> > the brokers that will use zookeeper which we already do for a lot
> of
> >> >> > other
> >> >> > usecases + ease of development + and the ability so tokens will
> survive
> >> >> > even a rolling restart/cluster failure. if a majority agrees the
> added
> >> >> > complexity to have controller forwarding keys to all broker is
> justified
> >> >> > as
> >> >> > it provides tighter security , I am fine with that option too.
> >> >> >
> >> >> > Given zookeeper does not support SSL we can not store master keys
> in
> >> >> > zookeeper as master keys will be exposed on wire. To support
> rotation
> >> >> > without affecting current clients is something I need to put more
> thought
> >> >> > in. My current proposal assumes the rotation will invalidate all
> current
> >> >> > tokens.
> >> >> >
> >> >> > I request committers to also review and post their comments so we
> can
> >> >> > make
> >> >> > progress on this KIP.
> >> >> >
> >> >> > Thanks
> >> >> > Parth
> >> >> >
> >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <asingh@cloudera.com
> >
> >> >> > wrote:
> >> >> >
> >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <ka...@harsha.io>
> wrote:
> >> >> > >
> >> >> > > > Unifying the two discussion threads on this KIP.
> >> >> > > >
> >> >> > > > Here is the response from Jitendra
> >> >> > > >
> >> >> > > > "The need for a large number of clients that are running all
> over the
> >> >> > > > cluster that authenticate with Kafka brokers, is very similar
> to the
> >> >> > > > Hadoop use case of large number of tasks running across the
> cluster
> >> >> that
> >> >> > > > need authentication to Hdfs Namenode. Therefore, the
> delegation token
> >> >> > > > approach does seem like a good fit for this use case as we
> have seen
> >> >> it
> >> >> > > > working at large scale in HDFS and YARN.
> >> >> > > >
> >> >> > > >   The proposed design is very much inline with Hadoop
> approach. A few
> >> >> > > >   comments:
> >> >> > > >
> >> >> > > > 1) Why do you guys want to allow infinite renewable lifetime
> for a
> >> >> > > > token? HDFS restricts a token to a max life time (default 7
> days).  A
> >> >> > > > token's vulnerability is believed to increase with time.
> >> >> > > >
> >> >> > > I agree that having infinite lifetime might not be the best idea.
> >> >> > >
> >> >> > > >
> >> >> > > > 2) As I understand the tokens are stored in zookeeper as well,
> and
> >> >> can
> >> >> > > > be updated there. This is clever as it can allow replacing the
> tokens
> >> >> > > > once they run out of max life time, and clients can download
> new
> >> >> tokens
> >> >> > > > from zookeeper. It shouldn't be a big load on zookeeper as a
> client
> >> >> will
> >> >> > > > need to get a new token once in several days. In this approach
> you
> >> >> don't
> >> >> > > > need infinite lifetime on the token even for long running
> clients.
> >> >> > > >
> >> >> > > > 3) The token password are generated using a master key. The
> master
> >> >> key
> >> >> > > > should also be periodically changed. In Hadoop, the default
> renewal
> >> >> > > > period is 1 day.?
> >> >> > > >
> >> >> > > IIUC, this will require brokers maintaining a list of X most
> recent
> >> >> master
> >> >> > > keys. This list will have to be persisted somewhere, as if a
> broker
> >> >> goes
> >> >> > > down it will have to get that list again and storing master keys
> on ZK
> >> >> is
> >> >> > > not the best idea. However, if a broker goes down then we have
> much
> >> >> bigger
> >> >> > > issue to deal with and client can always re-authenticate is such
> >> >> events.
> >> >> > >
> >> >> > > Did you happen to take a look at other alternatives this list has
> >> >> > > suggested?
> >> >> > >
> >> >> > > >
> >> >> > > > Thanks for a thorough proposal, great work!"
> >> >> > > >
> >> >> > > >
> >> >> > > >
> >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
> >> >> > > > > Makes sense to me. Thanks!
> >> >> > > > >
> >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <ka...@harsha.io>
> wrote:
> >> >> > > > > > It doesn't need any release vehicle but still the work can
> move
> >> >> > > > forward.
> >> >> > > > > > If anyone is interested in the KIP please do the review and
> >> >> provide
> >> >> > > the
> >> >> > > > > > comments.
> >> >> > > > > >
> >> >> > > > > > -Harsha
> >> >> > > > > >
> >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma wrote:
> >> >> > > > > >> I agree that it would be good to have more time to review
> and
> >> >> > > discuss
> >> >> > > > > >> KIP-48.
> >> >> > > > > >>
> >> >> > > > > >> Ismael
> >> >> > > > > >>
> >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <
> >> >> gwen@confluent.io>
> >> >> > > > wrote:
> >> >> > > > > >>
> >> >> > > > > >> > Hi Team,
> >> >> > > > > >> >
> >> >> > > > > >> > Since KIP-48 depends on KIP-43, which is already a bit
> of a
> >> >> risk
> >> >> > > for
> >> >> > > > > >> > the next release - any chance we can delay delegation
> tokens
> >> >> to
> >> >> > > > Kafka
> >> >> > > > > >> > 0.10.1?
> >> >> > > > > >> > With the community working on a release every 3 month,
> this
> >> >> is not
> >> >> > > > a huge
> >> >> > > > > >> > delay.
> >> >> > > > > >> >
> >> >> > > > > >> > Gwen
> >> >> > > > > >> >
> >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh <
> >> >> > > asingh@cloudera.com>
> >> >> > > > wrote:
> >> >> > > > > >> > > Parth,
> >> >> > > > > >> > >
> >> >> > > > > >> > > Thanks again for the awesome write up. Following our
> >> >> discussion
> >> >> > > > from the
> >> >> > > > > >> > > JIRA, I think it will be easier to compare various
> >> >> alternatives
> >> >> > > > if they
> >> >> > > > > >> > are
> >> >> > > > > >> > > listed together. I am stating below a few
> alternatives along
> >> >> > > with
> >> >> > > > a the
> >> >> > > > > >> > > current proposal.
> >> >> > > > > >> > > (Current proposal) Store Delegation Token, DT, on ZK.
> >> >> > > > > >> > >
> >> >> > > > > >> > >    1. Client authenticates with a broker.
> >> >> > > > > >> > >    2. Once a client is authenticated, it will make a
> broker
> >> >> side
> >> >> > > > call to
> >> >> > > > > >> > >    issue a delegation token.
> >> >> > > > > >> > >    3. The broker generates a shared secret based on
> >> >> > > HMAC-SHA256(a
> >> >> > > > > >> > >    Password/Secret shared between all brokers,
> randomly
> >> >> > > generated
> >> >> > > > > >> > tokenId).
> >> >> > > > > >> > >    4. Broker stores this token in its in memory cache.
> >> >> Broker
> >> >> > > > also stores
> >> >> > > > > >> > >    the DelegationToken without the hmac in the
> zookeeper.
> >> >> > > > > >> > >    5. All brokers will have a cache backed by
> zookeeper so
> >> >> they
> >> >> > > > will all
> >> >> > > > > >> > >    get notified whenever a new token is generated and
> they
> >> >> will
> >> >> > > > update
> >> >> > > > > >> > their
> >> >> > > > > >> > >    local cache whenever token state changes.
> >> >> > > > > >> > >    6. Broker returns the token to Client.
> >> >> > > > > >> > >
> >> >> > > > > >> > > Probable issues and fixes
> >> >> > > > > >> > >
> >> >> > > > > >> > >    1. Probable race condition, client tries to
> authenticate
> >> >> with
> >> >> > > > a broker
> >> >> > > > > >> > >    that is yet to be updated with the newly generated
> DT.
> >> >> This
> >> >> > > can
> >> >> > > > > >> > probably be
> >> >> > > > > >> > >    dealt with making dtRequest block until all
> brokers have
> >> >> > > > updated
> >> >> > > > > >> > their DT
> >> >> > > > > >> > >    cache. Zk barrier or similar mechanism can be used.
> >> >> However,
> >> >> > > > all such
> >> >> > > > > >> > >    mechanisms will increase complexity.
> >> >> > > > > >> > >    2. Using a static secret key from config file. Will
> >> >> require
> >> >> > > yet
> >> >> > > > > >> > another
> >> >> > > > > >> > >    config and uses a static secret key. It is advised
> to
> >> >> rotate
> >> >> > > > secret
> >> >> > > > > >> > keys
> >> >> > > > > >> > >    periodically. This can be avoided with controller
> >> >> generating
> >> >> > > > > >> > secretKey and
> >> >> > > > > >> > >    passing to brokers periodically. However, this will
> >> >> require
> >> >> > > > brokers to
> >> >> > > > > >> > >    maintain certain counts of secretKeys.
> >> >> > > > > >> > >
> >> >> > > > > >> > > (Alternative 1) Have controller generate delegation
> token.
> >> >> > > > > >> > >
> >> >> > > > > >> > >    1. Client authenticates with a broker.
> >> >> > > > > >> > >    2. Once a client is authenticated, it will make a
> broker
> >> >> side
> >> >> > > > call to
> >> >> > > > > >> > >    issue a delegation token.
> >> >> > > > > >> > >    3. Broker forwards the request to controller.
> >> >> > > > > >> > >    4. Controller generates a DT and broadcasts to all
> >> >> brokers.
> >> >> > > > > >> > >    5. Broker stores this token in its memory cache.
> >> >> > > > > >> > >    6. Controller responds to broker’s DT req.
> >> >> > > > > >> > >    7. Broker returns the token to Client.
> >> >> > > > > >> > >
> >> >> > > > > >> > > Probable issues and fixes
> >> >> > > > > >> > >
> >> >> > > > > >> > >    1. We will have to add new APIs to support
> controller
> >> >> pushing
> >> >> > > > tokens
> >> >> > > > > >> > to
> >> >> > > > > >> > >    brokers on top of the minimal APIs that are
> currently
> >> >> > > proposed.
> >> >> > > > > >> > >    2. We will also have to add APIs to support the
> >> >> bootstrapping
> >> >> > > > case,
> >> >> > > > > >> > i.e,
> >> >> > > > > >> > >    when a new broker comes up it will have to get all
> >> >> delegation
> >> >> > > > tokens
> >> >> > > > > >> > from
> >> >> > > > > >> > >    the controller.
> >> >> > > > > >> > >    3. In catastrophic failures where all brokers go
> down,
> >> >> the
> >> >> > > > tokens will
> >> >> > > > > >> > >    be lost even if servers are restarted as tokens
> are not
> >> >> > > > persisted
> >> >> > > > > >> > anywhere.
> >> >> > > > > >> > >    If this happens, then there are more important
> things to
> >> >> > > worry
> >> >> > > > about
> >> >> > > > > >> > and
> >> >> > > > > >> > >    maybe it is better to re-authenticate.
> >> >> > > > > >> > >
> >> >> > > > > >> > > (Alternative 2) Do not distribute DT to other brokers
> at
> >> >> all.
> >> >> > > > > >> > >
> >> >> > > > > >> > >    1. Client authenticates with a broker.
> >> >> > > > > >> > >    2. Once a client is authenticated, it will make a
> broker
> >> >> side
> >> >> > > > call to
> >> >> > > > > >> > >    issue a delegation token.
> >> >> > > > > >> > >    3. The broker generates DT of form, [hmac + (owner,
> >> >> renewer,
> >> >> > > > > >> > >    maxLifeTime, id, hmac, expirationTime)] and passes
> back
> >> >> this
> >> >> > > > DT to
> >> >> > > > > >> > client.
> >> >> > > > > >> > >    hmac is generated via {HMAC-SHA256(owner, renewer,
> >> >> > > > maxLifeTime, id,
> >> >> > > > > >> > hmac,
> >> >> > > > > >> > >    expirationTime) using SecretKey}. Note that all
> brokers
> >> >> have
> >> >> > > > this
> >> >> > > > > >> > SecretKey.
> >> >> > > > > >> > >    4. Client then goes to any broker and to
> authenticate
> >> >> sends
> >> >> > > > the DT.
> >> >> > > > > >> > >    Broker recalculates hmac using (owner, renewer,
> >> >> maxLifeTime,
> >> >> > > > id, hmac,
> >> >> > > > > >> > >    expirationTime) info from DT and its SecretKey. If
> it
> >> >> matches
> >> >> > > > with
> >> >> > > > > >> > hmac of
> >> >> > > > > >> > >    DT, client is authenticated. Yes, it will do other
> >> >> obvious
> >> >> > > > checks of
> >> >> > > > > >> > >    timestamp expiry and such.
> >> >> > > > > >> > >
> >> >> > > > > >> > > Note that secret key will be generated by controller
> and
> >> >> passed
> >> >> > > to
> >> >> > > > > >> > brokers
> >> >> > > > > >> > > periodically.
> >> >> > > > > >> > > Probable issues and fixes
> >> >> > > > > >> > >
> >> >> > > > > >> > >    1. How to delete a DT? Yes, that is a downside
> here.
> >> >> However,
> >> >> > > > this can
> >> >> > > > > >> > >    be handled with brokers maintaining a blacklist of
> DTs,
> >> >> DTs
> >> >> > > > from this
> >> >> > > > > >> > list
> >> >> > > > > >> > >    can be removed after expiry.
> >> >> > > > > >> > >    2. In catastrophic failures where all brokers go
> down,
> >> >> the
> >> >> > > > tokens will
> >> >> > > > > >> > >    be lost even if servers are restarted as tokens
> are not
> >> >> > > > persisted
> >> >> > > > > >> > anywhere.
> >> >> > > > > >> > >    If this happens, then there are more important
> things to
> >> >> > > worry
> >> >> > > > about
> >> >> > > > > >> > and
> >> >> > > > > >> > >    maybe it is better to re-authenticate.
> >> >> > > > > >> > >
> >> >> > > > > >> > >
> >> >> > > > > >> > >
> >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth Brahmbhatt <
> >> >> > > > > >> > > pbrahmbhatt@hortonworks.com> wrote:
> >> >> > > > > >> > >
> >> >> > > > > >> > >> Hi,
> >> >> > > > > >> > >>
> >> >> > > > > >> > >> I have filed KIP-48 so we can offer hadoop like
> delegation
> >> >> > > > tokens in
> >> >> > > > > >> > >> kafka. You can review the design
> >> >> > > > > >> > >>
> >> >> > > > > >> >
> >> >> > > >
> >> >> > >
> >> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> >> >> > > > > >> > .
> >> >> > > > > >> > >> This KIP depends on KIP-43 and we have also
> discussed an
> >> >> > > > alternative to
> >> >> > > > > >> > >> proposed design here<
> >> >> > > > > >> > >>
> >> >> > > > > >> >
> >> >> > > >
> >> >> > >
> >> >>
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> >> >> > > > > >> > >> >.
> >> >> > > > > >> > >>
> >> >> > > > > >> > >> Thanks
> >> >> > > > > >> > >> Parth
> >> >> > > > > >> > >>
> >> >> > > > > >> > >
> >> >> > > > > >> > >
> >> >> > > > > >> > >
> >> >> > > > > >> > > --
> >> >> > > > > >> > >
> >> >> > > > > >> > > Regards,
> >> >> > > > > >> > > Ashish
> >> >> > > > > >> >
> >> >> > > >
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > > --
> >> >> > >
> >> >> > > Regards,
> >> >> > > Ashish
> >> >> > >
> >> >>
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Gwen Shapira <gw...@confluent.io>]

Harsha,

I was thinking of the Rest Proxy. I didn't see your design yet, but in
our proxy, we have a set of producers, which will serve multiple users
going through the proxy. Since these users will have different
privileges, they'll need to authenticate separately, and can't share a
token.

Am I missing anything?

Gwen

On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io> wrote:
> Gwen,
>            On your second point. Can you describe a usecase where
>            mutliple clients ended up sharing a producer and even if they
>            do why can't they not use single token that producer
>            captures. Why would we need multiple clients with different
>            tokens sharing a single instance of producer.  Also in this
>            case other clients have access all the tokens no?
>
> Thanks,
> Harsha
>
>
> On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote:
>> Sorry for the delay:
>>
>> Two questions that we didn't see in the wiki:
>> 1. Is there an expiration for delegation tokens? Renewal? How do we
>> revoke them?
>> 2. If we want to use delegation tokens for "do-as" (say, submit Storm
>> job as my user), we will need a producer for every job (we can't share
>> them between multiple jobs running on same node), since we only
>> authenticate when connecting. Is there a plan to change this for
>> delegation tokens, in order to allow multiple users with different
>> tokens to share a client?
>>
>> Gwen
>>
>> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
>> <br...@gmail.com> wrote:
>> > Bumping this up one more time, can other committers review?
>> >
>> > Thanks
>> > Parth
>> >
>> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <ka...@harsha.io> wrote:
>> >
>> >> Parth,
>> >>           Overall current design looks good to me. I am +1 on the KIP.
>> >>
>> >> Gwen , Jun can you review this as well.
>> >>
>> >> -Harsha
>> >>
>> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote:
>> >> > Thanks for review Jitendra.
>> >> >
>> >> > I don't like the idea of infinite lifetime but I see the Streaming use
>> >> > case. Even for Streaming use case I was hoping there will be some notion
>> >> > of
>> >> > master/driver that can get new delegation tokens at fixed interval and
>> >> > distribute to workers. If that is not the case for we can discuss
>> >> > delegation tokens renewing them self and the security implications of the
>> >> > same.
>> >> >
>> >> > I did not want clients to fetch tokens from zookeeper, overall I think
>> >> > its
>> >> > better if clients don't rely on our metadata store and I think we are
>> >> > moving in that direction with all the KIP-4 improvements.  I chose
>> >> > zookeeper as in this case the client will still just talk to broker , its
>> >> > the brokers that will use zookeeper which we already do for a lot of
>> >> > other
>> >> > usecases + ease of development + and the ability so tokens will survive
>> >> > even a rolling restart/cluster failure. if a majority agrees the added
>> >> > complexity to have controller forwarding keys to all broker is justified
>> >> > as
>> >> > it provides tighter security , I am fine with that option too.
>> >> >
>> >> > Given zookeeper does not support SSL we can not store master keys in
>> >> > zookeeper as master keys will be exposed on wire. To support rotation
>> >> > without affecting current clients is something I need to put more thought
>> >> > in. My current proposal assumes the rotation will invalidate all current
>> >> > tokens.
>> >> >
>> >> > I request committers to also review and post their comments so we can
>> >> > make
>> >> > progress on this KIP.
>> >> >
>> >> > Thanks
>> >> > Parth
>> >> >
>> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <as...@cloudera.com>
>> >> > wrote:
>> >> >
>> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <ka...@harsha.io> wrote:
>> >> > >
>> >> > > > Unifying the two discussion threads on this KIP.
>> >> > > >
>> >> > > > Here is the response from Jitendra
>> >> > > >
>> >> > > > "The need for a large number of clients that are running all over the
>> >> > > > cluster that authenticate with Kafka brokers, is very similar to the
>> >> > > > Hadoop use case of large number of tasks running across the cluster
>> >> that
>> >> > > > need authentication to Hdfs Namenode. Therefore, the delegation token
>> >> > > > approach does seem like a good fit for this use case as we have seen
>> >> it
>> >> > > > working at large scale in HDFS and YARN.
>> >> > > >
>> >> > > >   The proposed design is very much inline with Hadoop approach. A few
>> >> > > >   comments:
>> >> > > >
>> >> > > > 1) Why do you guys want to allow infinite renewable lifetime for a
>> >> > > > token? HDFS restricts a token to a max life time (default 7 days).  A
>> >> > > > token's vulnerability is believed to increase with time.
>> >> > > >
>> >> > > I agree that having infinite lifetime might not be the best idea.
>> >> > >
>> >> > > >
>> >> > > > 2) As I understand the tokens are stored in zookeeper as well, and
>> >> can
>> >> > > > be updated there. This is clever as it can allow replacing the tokens
>> >> > > > once they run out of max life time, and clients can download new
>> >> tokens
>> >> > > > from zookeeper. It shouldn't be a big load on zookeeper as a client
>> >> will
>> >> > > > need to get a new token once in several days. In this approach you
>> >> don't
>> >> > > > need infinite lifetime on the token even for long running clients.
>> >> > > >
>> >> > > > 3) The token password are generated using a master key. The master
>> >> key
>> >> > > > should also be periodically changed. In Hadoop, the default renewal
>> >> > > > period is 1 day.?
>> >> > > >
>> >> > > IIUC, this will require brokers maintaining a list of X most recent
>> >> master
>> >> > > keys. This list will have to be persisted somewhere, as if a broker
>> >> goes
>> >> > > down it will have to get that list again and storing master keys on ZK
>> >> is
>> >> > > not the best idea. However, if a broker goes down then we have much
>> >> bigger
>> >> > > issue to deal with and client can always re-authenticate is such
>> >> events.
>> >> > >
>> >> > > Did you happen to take a look at other alternatives this list has
>> >> > > suggested?
>> >> > >
>> >> > > >
>> >> > > > Thanks for a thorough proposal, great work!"
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
>> >> > > > > Makes sense to me. Thanks!
>> >> > > > >
>> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <ka...@harsha.io> wrote:
>> >> > > > > > It doesn't need any release vehicle but still the work can move
>> >> > > > forward.
>> >> > > > > > If anyone is interested in the KIP please do the review and
>> >> provide
>> >> > > the
>> >> > > > > > comments.
>> >> > > > > >
>> >> > > > > > -Harsha
>> >> > > > > >
>> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma wrote:
>> >> > > > > >> I agree that it would be good to have more time to review and
>> >> > > discuss
>> >> > > > > >> KIP-48.
>> >> > > > > >>
>> >> > > > > >> Ismael
>> >> > > > > >>
>> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <
>> >> gwen@confluent.io>
>> >> > > > wrote:
>> >> > > > > >>
>> >> > > > > >> > Hi Team,
>> >> > > > > >> >
>> >> > > > > >> > Since KIP-48 depends on KIP-43, which is already a bit of a
>> >> risk
>> >> > > for
>> >> > > > > >> > the next release - any chance we can delay delegation tokens
>> >> to
>> >> > > > Kafka
>> >> > > > > >> > 0.10.1?
>> >> > > > > >> > With the community working on a release every 3 month, this
>> >> is not
>> >> > > > a huge
>> >> > > > > >> > delay.
>> >> > > > > >> >
>> >> > > > > >> > Gwen
>> >> > > > > >> >
>> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh <
>> >> > > asingh@cloudera.com>
>> >> > > > wrote:
>> >> > > > > >> > > Parth,
>> >> > > > > >> > >
>> >> > > > > >> > > Thanks again for the awesome write up. Following our
>> >> discussion
>> >> > > > from the
>> >> > > > > >> > > JIRA, I think it will be easier to compare various
>> >> alternatives
>> >> > > > if they
>> >> > > > > >> > are
>> >> > > > > >> > > listed together. I am stating below a few alternatives along
>> >> > > with
>> >> > > > a the
>> >> > > > > >> > > current proposal.
>> >> > > > > >> > > (Current proposal) Store Delegation Token, DT, on ZK.
>> >> > > > > >> > >
>> >> > > > > >> > >    1. Client authenticates with a broker.
>> >> > > > > >> > >    2. Once a client is authenticated, it will make a broker
>> >> side
>> >> > > > call to
>> >> > > > > >> > >    issue a delegation token.
>> >> > > > > >> > >    3. The broker generates a shared secret based on
>> >> > > HMAC-SHA256(a
>> >> > > > > >> > >    Password/Secret shared between all brokers, randomly
>> >> > > generated
>> >> > > > > >> > tokenId).
>> >> > > > > >> > >    4. Broker stores this token in its in memory cache.
>> >> Broker
>> >> > > > also stores
>> >> > > > > >> > >    the DelegationToken without the hmac in the zookeeper.
>> >> > > > > >> > >    5. All brokers will have a cache backed by zookeeper so
>> >> they
>> >> > > > will all
>> >> > > > > >> > >    get notified whenever a new token is generated and they
>> >> will
>> >> > > > update
>> >> > > > > >> > their
>> >> > > > > >> > >    local cache whenever token state changes.
>> >> > > > > >> > >    6. Broker returns the token to Client.
>> >> > > > > >> > >
>> >> > > > > >> > > Probable issues and fixes
>> >> > > > > >> > >
>> >> > > > > >> > >    1. Probable race condition, client tries to authenticate
>> >> with
>> >> > > > a broker
>> >> > > > > >> > >    that is yet to be updated with the newly generated DT.
>> >> This
>> >> > > can
>> >> > > > > >> > probably be
>> >> > > > > >> > >    dealt with making dtRequest block until all brokers have
>> >> > > > updated
>> >> > > > > >> > their DT
>> >> > > > > >> > >    cache. Zk barrier or similar mechanism can be used.
>> >> However,
>> >> > > > all such
>> >> > > > > >> > >    mechanisms will increase complexity.
>> >> > > > > >> > >    2. Using a static secret key from config file. Will
>> >> require
>> >> > > yet
>> >> > > > > >> > another
>> >> > > > > >> > >    config and uses a static secret key. It is advised to
>> >> rotate
>> >> > > > secret
>> >> > > > > >> > keys
>> >> > > > > >> > >    periodically. This can be avoided with controller
>> >> generating
>> >> > > > > >> > secretKey and
>> >> > > > > >> > >    passing to brokers periodically. However, this will
>> >> require
>> >> > > > brokers to
>> >> > > > > >> > >    maintain certain counts of secretKeys.
>> >> > > > > >> > >
>> >> > > > > >> > > (Alternative 1) Have controller generate delegation token.
>> >> > > > > >> > >
>> >> > > > > >> > >    1. Client authenticates with a broker.
>> >> > > > > >> > >    2. Once a client is authenticated, it will make a broker
>> >> side
>> >> > > > call to
>> >> > > > > >> > >    issue a delegation token.
>> >> > > > > >> > >    3. Broker forwards the request to controller.
>> >> > > > > >> > >    4. Controller generates a DT and broadcasts to all
>> >> brokers.
>> >> > > > > >> > >    5. Broker stores this token in its memory cache.
>> >> > > > > >> > >    6. Controller responds to broker’s DT req.
>> >> > > > > >> > >    7. Broker returns the token to Client.
>> >> > > > > >> > >
>> >> > > > > >> > > Probable issues and fixes
>> >> > > > > >> > >
>> >> > > > > >> > >    1. We will have to add new APIs to support controller
>> >> pushing
>> >> > > > tokens
>> >> > > > > >> > to
>> >> > > > > >> > >    brokers on top of the minimal APIs that are currently
>> >> > > proposed.
>> >> > > > > >> > >    2. We will also have to add APIs to support the
>> >> bootstrapping
>> >> > > > case,
>> >> > > > > >> > i.e,
>> >> > > > > >> > >    when a new broker comes up it will have to get all
>> >> delegation
>> >> > > > tokens
>> >> > > > > >> > from
>> >> > > > > >> > >    the controller.
>> >> > > > > >> > >    3. In catastrophic failures where all brokers go down,
>> >> the
>> >> > > > tokens will
>> >> > > > > >> > >    be lost even if servers are restarted as tokens are not
>> >> > > > persisted
>> >> > > > > >> > anywhere.
>> >> > > > > >> > >    If this happens, then there are more important things to
>> >> > > worry
>> >> > > > about
>> >> > > > > >> > and
>> >> > > > > >> > >    maybe it is better to re-authenticate.
>> >> > > > > >> > >
>> >> > > > > >> > > (Alternative 2) Do not distribute DT to other brokers at
>> >> all.
>> >> > > > > >> > >
>> >> > > > > >> > >    1. Client authenticates with a broker.
>> >> > > > > >> > >    2. Once a client is authenticated, it will make a broker
>> >> side
>> >> > > > call to
>> >> > > > > >> > >    issue a delegation token.
>> >> > > > > >> > >    3. The broker generates DT of form, [hmac + (owner,
>> >> renewer,
>> >> > > > > >> > >    maxLifeTime, id, hmac, expirationTime)] and passes back
>> >> this
>> >> > > > DT to
>> >> > > > > >> > client.
>> >> > > > > >> > >    hmac is generated via {HMAC-SHA256(owner, renewer,
>> >> > > > maxLifeTime, id,
>> >> > > > > >> > hmac,
>> >> > > > > >> > >    expirationTime) using SecretKey}. Note that all brokers
>> >> have
>> >> > > > this
>> >> > > > > >> > SecretKey.
>> >> > > > > >> > >    4. Client then goes to any broker and to authenticate
>> >> sends
>> >> > > > the DT.
>> >> > > > > >> > >    Broker recalculates hmac using (owner, renewer,
>> >> maxLifeTime,
>> >> > > > id, hmac,
>> >> > > > > >> > >    expirationTime) info from DT and its SecretKey. If it
>> >> matches
>> >> > > > with
>> >> > > > > >> > hmac of
>> >> > > > > >> > >    DT, client is authenticated. Yes, it will do other
>> >> obvious
>> >> > > > checks of
>> >> > > > > >> > >    timestamp expiry and such.
>> >> > > > > >> > >
>> >> > > > > >> > > Note that secret key will be generated by controller and
>> >> passed
>> >> > > to
>> >> > > > > >> > brokers
>> >> > > > > >> > > periodically.
>> >> > > > > >> > > Probable issues and fixes
>> >> > > > > >> > >
>> >> > > > > >> > >    1. How to delete a DT? Yes, that is a downside here.
>> >> However,
>> >> > > > this can
>> >> > > > > >> > >    be handled with brokers maintaining a blacklist of DTs,
>> >> DTs
>> >> > > > from this
>> >> > > > > >> > list
>> >> > > > > >> > >    can be removed after expiry.
>> >> > > > > >> > >    2. In catastrophic failures where all brokers go down,
>> >> the
>> >> > > > tokens will
>> >> > > > > >> > >    be lost even if servers are restarted as tokens are not
>> >> > > > persisted
>> >> > > > > >> > anywhere.
>> >> > > > > >> > >    If this happens, then there are more important things to
>> >> > > worry
>> >> > > > about
>> >> > > > > >> > and
>> >> > > > > >> > >    maybe it is better to re-authenticate.
>> >> > > > > >> > >
>> >> > > > > >> > >
>> >> > > > > >> > >
>> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth Brahmbhatt <
>> >> > > > > >> > > pbrahmbhatt@hortonworks.com> wrote:
>> >> > > > > >> > >
>> >> > > > > >> > >> Hi,
>> >> > > > > >> > >>
>> >> > > > > >> > >> I have filed KIP-48 so we can offer hadoop like delegation
>> >> > > > tokens in
>> >> > > > > >> > >> kafka. You can review the design
>> >> > > > > >> > >>
>> >> > > > > >> >
>> >> > > >
>> >> > >
>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
>> >> > > > > >> > .
>> >> > > > > >> > >> This KIP depends on KIP-43 and we have also discussed an
>> >> > > > alternative to
>> >> > > > > >> > >> proposed design here<
>> >> > > > > >> > >>
>> >> > > > > >> >
>> >> > > >
>> >> > >
>> >> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
>> >> > > > > >> > >> >.
>> >> > > > > >> > >>
>> >> > > > > >> > >> Thanks
>> >> > > > > >> > >> Parth
>> >> > > > > >> > >>
>> >> > > > > >> > >
>> >> > > > > >> > >
>> >> > > > > >> > >
>> >> > > > > >> > > --
>> >> > > > > >> > >
>> >> > > > > >> > > Regards,
>> >> > > > > >> > > Ashish
>> >> > > > > >> >
>> >> > > >
>> >> > >
>> >> > >
>> >> > >
>> >> > > --
>> >> > >
>> >> > > Regards,
>> >> > > Ashish
>> >> > >
>> >>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Harsha <ka...@harsha.io>]

Gwen,
           On your second point. Can you describe a usecase where
           mutliple clients ended up sharing a producer and even if they
           do why can't they not use single token that producer
           captures. Why would we need multiple clients with different
           tokens sharing a single instance of producer.  Also in this
           case other clients have access all the tokens no?

Thanks,
Harsha
            

On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote:
> Sorry for the delay:
> 
> Two questions that we didn't see in the wiki:
> 1. Is there an expiration for delegation tokens? Renewal? How do we
> revoke them?
> 2. If we want to use delegation tokens for "do-as" (say, submit Storm
> job as my user), we will need a producer for every job (we can't share
> them between multiple jobs running on same node), since we only
> authenticate when connecting. Is there a plan to change this for
> delegation tokens, in order to allow multiple users with different
> tokens to share a client?
> 
> Gwen
> 
> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
> <br...@gmail.com> wrote:
> > Bumping this up one more time, can other committers review?
> >
> > Thanks
> > Parth
> >
> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <ka...@harsha.io> wrote:
> >
> >> Parth,
> >>           Overall current design looks good to me. I am +1 on the KIP.
> >>
> >> Gwen , Jun can you review this as well.
> >>
> >> -Harsha
> >>
> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote:
> >> > Thanks for review Jitendra.
> >> >
> >> > I don't like the idea of infinite lifetime but I see the Streaming use
> >> > case. Even for Streaming use case I was hoping there will be some notion
> >> > of
> >> > master/driver that can get new delegation tokens at fixed interval and
> >> > distribute to workers. If that is not the case for we can discuss
> >> > delegation tokens renewing them self and the security implications of the
> >> > same.
> >> >
> >> > I did not want clients to fetch tokens from zookeeper, overall I think
> >> > its
> >> > better if clients don't rely on our metadata store and I think we are
> >> > moving in that direction with all the KIP-4 improvements.  I chose
> >> > zookeeper as in this case the client will still just talk to broker , its
> >> > the brokers that will use zookeeper which we already do for a lot of
> >> > other
> >> > usecases + ease of development + and the ability so tokens will survive
> >> > even a rolling restart/cluster failure. if a majority agrees the added
> >> > complexity to have controller forwarding keys to all broker is justified
> >> > as
> >> > it provides tighter security , I am fine with that option too.
> >> >
> >> > Given zookeeper does not support SSL we can not store master keys in
> >> > zookeeper as master keys will be exposed on wire. To support rotation
> >> > without affecting current clients is something I need to put more thought
> >> > in. My current proposal assumes the rotation will invalidate all current
> >> > tokens.
> >> >
> >> > I request committers to also review and post their comments so we can
> >> > make
> >> > progress on this KIP.
> >> >
> >> > Thanks
> >> > Parth
> >> >
> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <as...@cloudera.com>
> >> > wrote:
> >> >
> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <ka...@harsha.io> wrote:
> >> > >
> >> > > > Unifying the two discussion threads on this KIP.
> >> > > >
> >> > > > Here is the response from Jitendra
> >> > > >
> >> > > > "The need for a large number of clients that are running all over the
> >> > > > cluster that authenticate with Kafka brokers, is very similar to the
> >> > > > Hadoop use case of large number of tasks running across the cluster
> >> that
> >> > > > need authentication to Hdfs Namenode. Therefore, the delegation token
> >> > > > approach does seem like a good fit for this use case as we have seen
> >> it
> >> > > > working at large scale in HDFS and YARN.
> >> > > >
> >> > > >   The proposed design is very much inline with Hadoop approach. A few
> >> > > >   comments:
> >> > > >
> >> > > > 1) Why do you guys want to allow infinite renewable lifetime for a
> >> > > > token? HDFS restricts a token to a max life time (default 7 days).  A
> >> > > > token's vulnerability is believed to increase with time.
> >> > > >
> >> > > I agree that having infinite lifetime might not be the best idea.
> >> > >
> >> > > >
> >> > > > 2) As I understand the tokens are stored in zookeeper as well, and
> >> can
> >> > > > be updated there. This is clever as it can allow replacing the tokens
> >> > > > once they run out of max life time, and clients can download new
> >> tokens
> >> > > > from zookeeper. It shouldn't be a big load on zookeeper as a client
> >> will
> >> > > > need to get a new token once in several days. In this approach you
> >> don't
> >> > > > need infinite lifetime on the token even for long running clients.
> >> > > >
> >> > > > 3) The token password are generated using a master key. The master
> >> key
> >> > > > should also be periodically changed. In Hadoop, the default renewal
> >> > > > period is 1 day.?
> >> > > >
> >> > > IIUC, this will require brokers maintaining a list of X most recent
> >> master
> >> > > keys. This list will have to be persisted somewhere, as if a broker
> >> goes
> >> > > down it will have to get that list again and storing master keys on ZK
> >> is
> >> > > not the best idea. However, if a broker goes down then we have much
> >> bigger
> >> > > issue to deal with and client can always re-authenticate is such
> >> events.
> >> > >
> >> > > Did you happen to take a look at other alternatives this list has
> >> > > suggested?
> >> > >
> >> > > >
> >> > > > Thanks for a thorough proposal, great work!"
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
> >> > > > > Makes sense to me. Thanks!
> >> > > > >
> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <ka...@harsha.io> wrote:
> >> > > > > > It doesn't need any release vehicle but still the work can move
> >> > > > forward.
> >> > > > > > If anyone is interested in the KIP please do the review and
> >> provide
> >> > > the
> >> > > > > > comments.
> >> > > > > >
> >> > > > > > -Harsha
> >> > > > > >
> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma wrote:
> >> > > > > >> I agree that it would be good to have more time to review and
> >> > > discuss
> >> > > > > >> KIP-48.
> >> > > > > >>
> >> > > > > >> Ismael
> >> > > > > >>
> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <
> >> gwen@confluent.io>
> >> > > > wrote:
> >> > > > > >>
> >> > > > > >> > Hi Team,
> >> > > > > >> >
> >> > > > > >> > Since KIP-48 depends on KIP-43, which is already a bit of a
> >> risk
> >> > > for
> >> > > > > >> > the next release - any chance we can delay delegation tokens
> >> to
> >> > > > Kafka
> >> > > > > >> > 0.10.1?
> >> > > > > >> > With the community working on a release every 3 month, this
> >> is not
> >> > > > a huge
> >> > > > > >> > delay.
> >> > > > > >> >
> >> > > > > >> > Gwen
> >> > > > > >> >
> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh <
> >> > > asingh@cloudera.com>
> >> > > > wrote:
> >> > > > > >> > > Parth,
> >> > > > > >> > >
> >> > > > > >> > > Thanks again for the awesome write up. Following our
> >> discussion
> >> > > > from the
> >> > > > > >> > > JIRA, I think it will be easier to compare various
> >> alternatives
> >> > > > if they
> >> > > > > >> > are
> >> > > > > >> > > listed together. I am stating below a few alternatives along
> >> > > with
> >> > > > a the
> >> > > > > >> > > current proposal.
> >> > > > > >> > > (Current proposal) Store Delegation Token, DT, on ZK.
> >> > > > > >> > >
> >> > > > > >> > >    1. Client authenticates with a broker.
> >> > > > > >> > >    2. Once a client is authenticated, it will make a broker
> >> side
> >> > > > call to
> >> > > > > >> > >    issue a delegation token.
> >> > > > > >> > >    3. The broker generates a shared secret based on
> >> > > HMAC-SHA256(a
> >> > > > > >> > >    Password/Secret shared between all brokers, randomly
> >> > > generated
> >> > > > > >> > tokenId).
> >> > > > > >> > >    4. Broker stores this token in its in memory cache.
> >> Broker
> >> > > > also stores
> >> > > > > >> > >    the DelegationToken without the hmac in the zookeeper.
> >> > > > > >> > >    5. All brokers will have a cache backed by zookeeper so
> >> they
> >> > > > will all
> >> > > > > >> > >    get notified whenever a new token is generated and they
> >> will
> >> > > > update
> >> > > > > >> > their
> >> > > > > >> > >    local cache whenever token state changes.
> >> > > > > >> > >    6. Broker returns the token to Client.
> >> > > > > >> > >
> >> > > > > >> > > Probable issues and fixes
> >> > > > > >> > >
> >> > > > > >> > >    1. Probable race condition, client tries to authenticate
> >> with
> >> > > > a broker
> >> > > > > >> > >    that is yet to be updated with the newly generated DT.
> >> This
> >> > > can
> >> > > > > >> > probably be
> >> > > > > >> > >    dealt with making dtRequest block until all brokers have
> >> > > > updated
> >> > > > > >> > their DT
> >> > > > > >> > >    cache. Zk barrier or similar mechanism can be used.
> >> However,
> >> > > > all such
> >> > > > > >> > >    mechanisms will increase complexity.
> >> > > > > >> > >    2. Using a static secret key from config file. Will
> >> require
> >> > > yet
> >> > > > > >> > another
> >> > > > > >> > >    config and uses a static secret key. It is advised to
> >> rotate
> >> > > > secret
> >> > > > > >> > keys
> >> > > > > >> > >    periodically. This can be avoided with controller
> >> generating
> >> > > > > >> > secretKey and
> >> > > > > >> > >    passing to brokers periodically. However, this will
> >> require
> >> > > > brokers to
> >> > > > > >> > >    maintain certain counts of secretKeys.
> >> > > > > >> > >
> >> > > > > >> > > (Alternative 1) Have controller generate delegation token.
> >> > > > > >> > >
> >> > > > > >> > >    1. Client authenticates with a broker.
> >> > > > > >> > >    2. Once a client is authenticated, it will make a broker
> >> side
> >> > > > call to
> >> > > > > >> > >    issue a delegation token.
> >> > > > > >> > >    3. Broker forwards the request to controller.
> >> > > > > >> > >    4. Controller generates a DT and broadcasts to all
> >> brokers.
> >> > > > > >> > >    5. Broker stores this token in its memory cache.
> >> > > > > >> > >    6. Controller responds to broker’s DT req.
> >> > > > > >> > >    7. Broker returns the token to Client.
> >> > > > > >> > >
> >> > > > > >> > > Probable issues and fixes
> >> > > > > >> > >
> >> > > > > >> > >    1. We will have to add new APIs to support controller
> >> pushing
> >> > > > tokens
> >> > > > > >> > to
> >> > > > > >> > >    brokers on top of the minimal APIs that are currently
> >> > > proposed.
> >> > > > > >> > >    2. We will also have to add APIs to support the
> >> bootstrapping
> >> > > > case,
> >> > > > > >> > i.e,
> >> > > > > >> > >    when a new broker comes up it will have to get all
> >> delegation
> >> > > > tokens
> >> > > > > >> > from
> >> > > > > >> > >    the controller.
> >> > > > > >> > >    3. In catastrophic failures where all brokers go down,
> >> the
> >> > > > tokens will
> >> > > > > >> > >    be lost even if servers are restarted as tokens are not
> >> > > > persisted
> >> > > > > >> > anywhere.
> >> > > > > >> > >    If this happens, then there are more important things to
> >> > > worry
> >> > > > about
> >> > > > > >> > and
> >> > > > > >> > >    maybe it is better to re-authenticate.
> >> > > > > >> > >
> >> > > > > >> > > (Alternative 2) Do not distribute DT to other brokers at
> >> all.
> >> > > > > >> > >
> >> > > > > >> > >    1. Client authenticates with a broker.
> >> > > > > >> > >    2. Once a client is authenticated, it will make a broker
> >> side
> >> > > > call to
> >> > > > > >> > >    issue a delegation token.
> >> > > > > >> > >    3. The broker generates DT of form, [hmac + (owner,
> >> renewer,
> >> > > > > >> > >    maxLifeTime, id, hmac, expirationTime)] and passes back
> >> this
> >> > > > DT to
> >> > > > > >> > client.
> >> > > > > >> > >    hmac is generated via {HMAC-SHA256(owner, renewer,
> >> > > > maxLifeTime, id,
> >> > > > > >> > hmac,
> >> > > > > >> > >    expirationTime) using SecretKey}. Note that all brokers
> >> have
> >> > > > this
> >> > > > > >> > SecretKey.
> >> > > > > >> > >    4. Client then goes to any broker and to authenticate
> >> sends
> >> > > > the DT.
> >> > > > > >> > >    Broker recalculates hmac using (owner, renewer,
> >> maxLifeTime,
> >> > > > id, hmac,
> >> > > > > >> > >    expirationTime) info from DT and its SecretKey. If it
> >> matches
> >> > > > with
> >> > > > > >> > hmac of
> >> > > > > >> > >    DT, client is authenticated. Yes, it will do other
> >> obvious
> >> > > > checks of
> >> > > > > >> > >    timestamp expiry and such.
> >> > > > > >> > >
> >> > > > > >> > > Note that secret key will be generated by controller and
> >> passed
> >> > > to
> >> > > > > >> > brokers
> >> > > > > >> > > periodically.
> >> > > > > >> > > Probable issues and fixes
> >> > > > > >> > >
> >> > > > > >> > >    1. How to delete a DT? Yes, that is a downside here.
> >> However,
> >> > > > this can
> >> > > > > >> > >    be handled with brokers maintaining a blacklist of DTs,
> >> DTs
> >> > > > from this
> >> > > > > >> > list
> >> > > > > >> > >    can be removed after expiry.
> >> > > > > >> > >    2. In catastrophic failures where all brokers go down,
> >> the
> >> > > > tokens will
> >> > > > > >> > >    be lost even if servers are restarted as tokens are not
> >> > > > persisted
> >> > > > > >> > anywhere.
> >> > > > > >> > >    If this happens, then there are more important things to
> >> > > worry
> >> > > > about
> >> > > > > >> > and
> >> > > > > >> > >    maybe it is better to re-authenticate.
> >> > > > > >> > >
> >> > > > > >> > >
> >> > > > > >> > >
> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth Brahmbhatt <
> >> > > > > >> > > pbrahmbhatt@hortonworks.com> wrote:
> >> > > > > >> > >
> >> > > > > >> > >> Hi,
> >> > > > > >> > >>
> >> > > > > >> > >> I have filed KIP-48 so we can offer hadoop like delegation
> >> > > > tokens in
> >> > > > > >> > >> kafka. You can review the design
> >> > > > > >> > >>
> >> > > > > >> >
> >> > > >
> >> > >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> >> > > > > >> > .
> >> > > > > >> > >> This KIP depends on KIP-43 and we have also discussed an
> >> > > > alternative to
> >> > > > > >> > >> proposed design here<
> >> > > > > >> > >>
> >> > > > > >> >
> >> > > >
> >> > >
> >> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> >> > > > > >> > >> >.
> >> > > > > >> > >>
> >> > > > > >> > >> Thanks
> >> > > > > >> > >> Parth
> >> > > > > >> > >>
> >> > > > > >> > >
> >> > > > > >> > >
> >> > > > > >> > >
> >> > > > > >> > > --
> >> > > > > >> > >
> >> > > > > >> > > Regards,
> >> > > > > >> > > Ashish
> >> > > > > >> >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > >
> >> > > Regards,
> >> > > Ashish
> >> > >
> >>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Gwen Shapira <gw...@confluent.io>]

Sorry for the delay:

Two questions that we didn't see in the wiki:
1. Is there an expiration for delegation tokens? Renewal? How do we revoke them?
2. If we want to use delegation tokens for "do-as" (say, submit Storm
job as my user), we will need a producer for every job (we can't share
them between multiple jobs running on same node), since we only
authenticate when connecting. Is there a plan to change this for
delegation tokens, in order to allow multiple users with different
tokens to share a client?

Gwen

On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
<br...@gmail.com> wrote:
> Bumping this up one more time, can other committers review?
>
> Thanks
> Parth
>
> On Tue, Apr 26, 2016 at 9:07 AM, Harsha <ka...@harsha.io> wrote:
>
>> Parth,
>>           Overall current design looks good to me. I am +1 on the KIP.
>>
>> Gwen , Jun can you review this as well.
>>
>> -Harsha
>>
>> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote:
>> > Thanks for review Jitendra.
>> >
>> > I don't like the idea of infinite lifetime but I see the Streaming use
>> > case. Even for Streaming use case I was hoping there will be some notion
>> > of
>> > master/driver that can get new delegation tokens at fixed interval and
>> > distribute to workers. If that is not the case for we can discuss
>> > delegation tokens renewing them self and the security implications of the
>> > same.
>> >
>> > I did not want clients to fetch tokens from zookeeper, overall I think
>> > its
>> > better if clients don't rely on our metadata store and I think we are
>> > moving in that direction with all the KIP-4 improvements.  I chose
>> > zookeeper as in this case the client will still just talk to broker , its
>> > the brokers that will use zookeeper which we already do for a lot of
>> > other
>> > usecases + ease of development + and the ability so tokens will survive
>> > even a rolling restart/cluster failure. if a majority agrees the added
>> > complexity to have controller forwarding keys to all broker is justified
>> > as
>> > it provides tighter security , I am fine with that option too.
>> >
>> > Given zookeeper does not support SSL we can not store master keys in
>> > zookeeper as master keys will be exposed on wire. To support rotation
>> > without affecting current clients is something I need to put more thought
>> > in. My current proposal assumes the rotation will invalidate all current
>> > tokens.
>> >
>> > I request committers to also review and post their comments so we can
>> > make
>> > progress on this KIP.
>> >
>> > Thanks
>> > Parth
>> >
>> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <as...@cloudera.com>
>> > wrote:
>> >
>> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <ka...@harsha.io> wrote:
>> > >
>> > > > Unifying the two discussion threads on this KIP.
>> > > >
>> > > > Here is the response from Jitendra
>> > > >
>> > > > "The need for a large number of clients that are running all over the
>> > > > cluster that authenticate with Kafka brokers, is very similar to the
>> > > > Hadoop use case of large number of tasks running across the cluster
>> that
>> > > > need authentication to Hdfs Namenode. Therefore, the delegation token
>> > > > approach does seem like a good fit for this use case as we have seen
>> it
>> > > > working at large scale in HDFS and YARN.
>> > > >
>> > > >   The proposed design is very much inline with Hadoop approach. A few
>> > > >   comments:
>> > > >
>> > > > 1) Why do you guys want to allow infinite renewable lifetime for a
>> > > > token? HDFS restricts a token to a max life time (default 7 days).  A
>> > > > token's vulnerability is believed to increase with time.
>> > > >
>> > > I agree that having infinite lifetime might not be the best idea.
>> > >
>> > > >
>> > > > 2) As I understand the tokens are stored in zookeeper as well, and
>> can
>> > > > be updated there. This is clever as it can allow replacing the tokens
>> > > > once they run out of max life time, and clients can download new
>> tokens
>> > > > from zookeeper. It shouldn't be a big load on zookeeper as a client
>> will
>> > > > need to get a new token once in several days. In this approach you
>> don't
>> > > > need infinite lifetime on the token even for long running clients.
>> > > >
>> > > > 3) The token password are generated using a master key. The master
>> key
>> > > > should also be periodically changed. In Hadoop, the default renewal
>> > > > period is 1 day.?
>> > > >
>> > > IIUC, this will require brokers maintaining a list of X most recent
>> master
>> > > keys. This list will have to be persisted somewhere, as if a broker
>> goes
>> > > down it will have to get that list again and storing master keys on ZK
>> is
>> > > not the best idea. However, if a broker goes down then we have much
>> bigger
>> > > issue to deal with and client can always re-authenticate is such
>> events.
>> > >
>> > > Did you happen to take a look at other alternatives this list has
>> > > suggested?
>> > >
>> > > >
>> > > > Thanks for a thorough proposal, great work!"
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
>> > > > > Makes sense to me. Thanks!
>> > > > >
>> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <ka...@harsha.io> wrote:
>> > > > > > It doesn't need any release vehicle but still the work can move
>> > > > forward.
>> > > > > > If anyone is interested in the KIP please do the review and
>> provide
>> > > the
>> > > > > > comments.
>> > > > > >
>> > > > > > -Harsha
>> > > > > >
>> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma wrote:
>> > > > > >> I agree that it would be good to have more time to review and
>> > > discuss
>> > > > > >> KIP-48.
>> > > > > >>
>> > > > > >> Ismael
>> > > > > >>
>> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <
>> gwen@confluent.io>
>> > > > wrote:
>> > > > > >>
>> > > > > >> > Hi Team,
>> > > > > >> >
>> > > > > >> > Since KIP-48 depends on KIP-43, which is already a bit of a
>> risk
>> > > for
>> > > > > >> > the next release - any chance we can delay delegation tokens
>> to
>> > > > Kafka
>> > > > > >> > 0.10.1?
>> > > > > >> > With the community working on a release every 3 month, this
>> is not
>> > > > a huge
>> > > > > >> > delay.
>> > > > > >> >
>> > > > > >> > Gwen
>> > > > > >> >
>> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh <
>> > > asingh@cloudera.com>
>> > > > wrote:
>> > > > > >> > > Parth,
>> > > > > >> > >
>> > > > > >> > > Thanks again for the awesome write up. Following our
>> discussion
>> > > > from the
>> > > > > >> > > JIRA, I think it will be easier to compare various
>> alternatives
>> > > > if they
>> > > > > >> > are
>> > > > > >> > > listed together. I am stating below a few alternatives along
>> > > with
>> > > > a the
>> > > > > >> > > current proposal.
>> > > > > >> > > (Current proposal) Store Delegation Token, DT, on ZK.
>> > > > > >> > >
>> > > > > >> > >    1. Client authenticates with a broker.
>> > > > > >> > >    2. Once a client is authenticated, it will make a broker
>> side
>> > > > call to
>> > > > > >> > >    issue a delegation token.
>> > > > > >> > >    3. The broker generates a shared secret based on
>> > > HMAC-SHA256(a
>> > > > > >> > >    Password/Secret shared between all brokers, randomly
>> > > generated
>> > > > > >> > tokenId).
>> > > > > >> > >    4. Broker stores this token in its in memory cache.
>> Broker
>> > > > also stores
>> > > > > >> > >    the DelegationToken without the hmac in the zookeeper.
>> > > > > >> > >    5. All brokers will have a cache backed by zookeeper so
>> they
>> > > > will all
>> > > > > >> > >    get notified whenever a new token is generated and they
>> will
>> > > > update
>> > > > > >> > their
>> > > > > >> > >    local cache whenever token state changes.
>> > > > > >> > >    6. Broker returns the token to Client.
>> > > > > >> > >
>> > > > > >> > > Probable issues and fixes
>> > > > > >> > >
>> > > > > >> > >    1. Probable race condition, client tries to authenticate
>> with
>> > > > a broker
>> > > > > >> > >    that is yet to be updated with the newly generated DT.
>> This
>> > > can
>> > > > > >> > probably be
>> > > > > >> > >    dealt with making dtRequest block until all brokers have
>> > > > updated
>> > > > > >> > their DT
>> > > > > >> > >    cache. Zk barrier or similar mechanism can be used.
>> However,
>> > > > all such
>> > > > > >> > >    mechanisms will increase complexity.
>> > > > > >> > >    2. Using a static secret key from config file. Will
>> require
>> > > yet
>> > > > > >> > another
>> > > > > >> > >    config and uses a static secret key. It is advised to
>> rotate
>> > > > secret
>> > > > > >> > keys
>> > > > > >> > >    periodically. This can be avoided with controller
>> generating
>> > > > > >> > secretKey and
>> > > > > >> > >    passing to brokers periodically. However, this will
>> require
>> > > > brokers to
>> > > > > >> > >    maintain certain counts of secretKeys.
>> > > > > >> > >
>> > > > > >> > > (Alternative 1) Have controller generate delegation token.
>> > > > > >> > >
>> > > > > >> > >    1. Client authenticates with a broker.
>> > > > > >> > >    2. Once a client is authenticated, it will make a broker
>> side
>> > > > call to
>> > > > > >> > >    issue a delegation token.
>> > > > > >> > >    3. Broker forwards the request to controller.
>> > > > > >> > >    4. Controller generates a DT and broadcasts to all
>> brokers.
>> > > > > >> > >    5. Broker stores this token in its memory cache.
>> > > > > >> > >    6. Controller responds to broker’s DT req.
>> > > > > >> > >    7. Broker returns the token to Client.
>> > > > > >> > >
>> > > > > >> > > Probable issues and fixes
>> > > > > >> > >
>> > > > > >> > >    1. We will have to add new APIs to support controller
>> pushing
>> > > > tokens
>> > > > > >> > to
>> > > > > >> > >    brokers on top of the minimal APIs that are currently
>> > > proposed.
>> > > > > >> > >    2. We will also have to add APIs to support the
>> bootstrapping
>> > > > case,
>> > > > > >> > i.e,
>> > > > > >> > >    when a new broker comes up it will have to get all
>> delegation
>> > > > tokens
>> > > > > >> > from
>> > > > > >> > >    the controller.
>> > > > > >> > >    3. In catastrophic failures where all brokers go down,
>> the
>> > > > tokens will
>> > > > > >> > >    be lost even if servers are restarted as tokens are not
>> > > > persisted
>> > > > > >> > anywhere.
>> > > > > >> > >    If this happens, then there are more important things to
>> > > worry
>> > > > about
>> > > > > >> > and
>> > > > > >> > >    maybe it is better to re-authenticate.
>> > > > > >> > >
>> > > > > >> > > (Alternative 2) Do not distribute DT to other brokers at
>> all.
>> > > > > >> > >
>> > > > > >> > >    1. Client authenticates with a broker.
>> > > > > >> > >    2. Once a client is authenticated, it will make a broker
>> side
>> > > > call to
>> > > > > >> > >    issue a delegation token.
>> > > > > >> > >    3. The broker generates DT of form, [hmac + (owner,
>> renewer,
>> > > > > >> > >    maxLifeTime, id, hmac, expirationTime)] and passes back
>> this
>> > > > DT to
>> > > > > >> > client.
>> > > > > >> > >    hmac is generated via {HMAC-SHA256(owner, renewer,
>> > > > maxLifeTime, id,
>> > > > > >> > hmac,
>> > > > > >> > >    expirationTime) using SecretKey}. Note that all brokers
>> have
>> > > > this
>> > > > > >> > SecretKey.
>> > > > > >> > >    4. Client then goes to any broker and to authenticate
>> sends
>> > > > the DT.
>> > > > > >> > >    Broker recalculates hmac using (owner, renewer,
>> maxLifeTime,
>> > > > id, hmac,
>> > > > > >> > >    expirationTime) info from DT and its SecretKey. If it
>> matches
>> > > > with
>> > > > > >> > hmac of
>> > > > > >> > >    DT, client is authenticated. Yes, it will do other
>> obvious
>> > > > checks of
>> > > > > >> > >    timestamp expiry and such.
>> > > > > >> > >
>> > > > > >> > > Note that secret key will be generated by controller and
>> passed
>> > > to
>> > > > > >> > brokers
>> > > > > >> > > periodically.
>> > > > > >> > > Probable issues and fixes
>> > > > > >> > >
>> > > > > >> > >    1. How to delete a DT? Yes, that is a downside here.
>> However,
>> > > > this can
>> > > > > >> > >    be handled with brokers maintaining a blacklist of DTs,
>> DTs
>> > > > from this
>> > > > > >> > list
>> > > > > >> > >    can be removed after expiry.
>> > > > > >> > >    2. In catastrophic failures where all brokers go down,
>> the
>> > > > tokens will
>> > > > > >> > >    be lost even if servers are restarted as tokens are not
>> > > > persisted
>> > > > > >> > anywhere.
>> > > > > >> > >    If this happens, then there are more important things to
>> > > worry
>> > > > about
>> > > > > >> > and
>> > > > > >> > >    maybe it is better to re-authenticate.
>> > > > > >> > >
>> > > > > >> > >
>> > > > > >> > >
>> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth Brahmbhatt <
>> > > > > >> > > pbrahmbhatt@hortonworks.com> wrote:
>> > > > > >> > >
>> > > > > >> > >> Hi,
>> > > > > >> > >>
>> > > > > >> > >> I have filed KIP-48 so we can offer hadoop like delegation
>> > > > tokens in
>> > > > > >> > >> kafka. You can review the design
>> > > > > >> > >>
>> > > > > >> >
>> > > >
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
>> > > > > >> > .
>> > > > > >> > >> This KIP depends on KIP-43 and we have also discussed an
>> > > > alternative to
>> > > > > >> > >> proposed design here<
>> > > > > >> > >>
>> > > > > >> >
>> > > >
>> > >
>> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
>> > > > > >> > >> >.
>> > > > > >> > >>
>> > > > > >> > >> Thanks
>> > > > > >> > >> Parth
>> > > > > >> > >>
>> > > > > >> > >
>> > > > > >> > >
>> > > > > >> > >
>> > > > > >> > > --
>> > > > > >> > >
>> > > > > >> > > Regards,
>> > > > > >> > > Ashish
>> > > > > >> >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > Regards,
>> > > Ashish
>> > >
>>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Giampaolo Trapasso <gi...@radicalbit.io>]

Hi,
I'm really interested in this KIP too.

Giampaolo



2016-05-03 18:12 GMT+02:00 parth brahmbhatt <br...@gmail.com>:

> Bumping this up one more time, can other committers review?
>
> Thanks
> Parth
>
> On Tue, Apr 26, 2016 at 9:07 AM, Harsha <ka...@harsha.io> wrote:
>
> > Parth,
> >           Overall current design looks good to me. I am +1 on the KIP.
> >
> > Gwen , Jun can you review this as well.
> >
> > -Harsha
> >
> > On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote:
> > > Thanks for review Jitendra.
> > >
> > > I don't like the idea of infinite lifetime but I see the Streaming use
> > > case. Even for Streaming use case I was hoping there will be some
> notion
> > > of
> > > master/driver that can get new delegation tokens at fixed interval and
> > > distribute to workers. If that is not the case for we can discuss
> > > delegation tokens renewing them self and the security implications of
> the
> > > same.
> > >
> > > I did not want clients to fetch tokens from zookeeper, overall I think
> > > its
> > > better if clients don't rely on our metadata store and I think we are
> > > moving in that direction with all the KIP-4 improvements.  I chose
> > > zookeeper as in this case the client will still just talk to broker ,
> its
> > > the brokers that will use zookeeper which we already do for a lot of
> > > other
> > > usecases + ease of development + and the ability so tokens will survive
> > > even a rolling restart/cluster failure. if a majority agrees the added
> > > complexity to have controller forwarding keys to all broker is
> justified
> > > as
> > > it provides tighter security , I am fine with that option too.
> > >
> > > Given zookeeper does not support SSL we can not store master keys in
> > > zookeeper as master keys will be exposed on wire. To support rotation
> > > without affecting current clients is something I need to put more
> thought
> > > in. My current proposal assumes the rotation will invalidate all
> current
> > > tokens.
> > >
> > > I request committers to also review and post their comments so we can
> > > make
> > > progress on this KIP.
> > >
> > > Thanks
> > > Parth
> > >
> > > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <as...@cloudera.com>
> > > wrote:
> > >
> > > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <ka...@harsha.io> wrote:
> > > >
> > > > > Unifying the two discussion threads on this KIP.
> > > > >
> > > > > Here is the response from Jitendra
> > > > >
> > > > > "The need for a large number of clients that are running all over
> the
> > > > > cluster that authenticate with Kafka brokers, is very similar to
> the
> > > > > Hadoop use case of large number of tasks running across the cluster
> > that
> > > > > need authentication to Hdfs Namenode. Therefore, the delegation
> token
> > > > > approach does seem like a good fit for this use case as we have
> seen
> > it
> > > > > working at large scale in HDFS and YARN.
> > > > >
> > > > >   The proposed design is very much inline with Hadoop approach. A
> few
> > > > >   comments:
> > > > >
> > > > > 1) Why do you guys want to allow infinite renewable lifetime for a
> > > > > token? HDFS restricts a token to a max life time (default 7
> days).  A
> > > > > token's vulnerability is believed to increase with time.
> > > > >
> > > > I agree that having infinite lifetime might not be the best idea.
> > > >
> > > > >
> > > > > 2) As I understand the tokens are stored in zookeeper as well, and
> > can
> > > > > be updated there. This is clever as it can allow replacing the
> tokens
> > > > > once they run out of max life time, and clients can download new
> > tokens
> > > > > from zookeeper. It shouldn't be a big load on zookeeper as a client
> > will
> > > > > need to get a new token once in several days. In this approach you
> > don't
> > > > > need infinite lifetime on the token even for long running clients.
> > > > >
> > > > > 3) The token password are generated using a master key. The master
> > key
> > > > > should also be periodically changed. In Hadoop, the default renewal
> > > > > period is 1 day.?
> > > > >
> > > > IIUC, this will require brokers maintaining a list of X most recent
> > master
> > > > keys. This list will have to be persisted somewhere, as if a broker
> > goes
> > > > down it will have to get that list again and storing master keys on
> ZK
> > is
> > > > not the best idea. However, if a broker goes down then we have much
> > bigger
> > > > issue to deal with and client can always re-authenticate is such
> > events.
> > > >
> > > > Did you happen to take a look at other alternatives this list has
> > > > suggested?
> > > >
> > > > >
> > > > > Thanks for a thorough proposal, great work!"
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
> > > > > > Makes sense to me. Thanks!
> > > > > >
> > > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <ka...@harsha.io> wrote:
> > > > > > > It doesn't need any release vehicle but still the work can move
> > > > > forward.
> > > > > > > If anyone is interested in the KIP please do the review and
> > provide
> > > > the
> > > > > > > comments.
> > > > > > >
> > > > > > > -Harsha
> > > > > > >
> > > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma wrote:
> > > > > > >> I agree that it would be good to have more time to review and
> > > > discuss
> > > > > > >> KIP-48.
> > > > > > >>
> > > > > > >> Ismael
> > > > > > >>
> > > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <
> > gwen@confluent.io>
> > > > > wrote:
> > > > > > >>
> > > > > > >> > Hi Team,
> > > > > > >> >
> > > > > > >> > Since KIP-48 depends on KIP-43, which is already a bit of a
> > risk
> > > > for
> > > > > > >> > the next release - any chance we can delay delegation tokens
> > to
> > > > > Kafka
> > > > > > >> > 0.10.1?
> > > > > > >> > With the community working on a release every 3 month, this
> > is not
> > > > > a huge
> > > > > > >> > delay.
> > > > > > >> >
> > > > > > >> > Gwen
> > > > > > >> >
> > > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh <
> > > > asingh@cloudera.com>
> > > > > wrote:
> > > > > > >> > > Parth,
> > > > > > >> > >
> > > > > > >> > > Thanks again for the awesome write up. Following our
> > discussion
> > > > > from the
> > > > > > >> > > JIRA, I think it will be easier to compare various
> > alternatives
> > > > > if they
> > > > > > >> > are
> > > > > > >> > > listed together. I am stating below a few alternatives
> along
> > > > with
> > > > > a the
> > > > > > >> > > current proposal.
> > > > > > >> > > (Current proposal) Store Delegation Token, DT, on ZK.
> > > > > > >> > >
> > > > > > >> > >    1. Client authenticates with a broker.
> > > > > > >> > >    2. Once a client is authenticated, it will make a
> broker
> > side
> > > > > call to
> > > > > > >> > >    issue a delegation token.
> > > > > > >> > >    3. The broker generates a shared secret based on
> > > > HMAC-SHA256(a
> > > > > > >> > >    Password/Secret shared between all brokers, randomly
> > > > generated
> > > > > > >> > tokenId).
> > > > > > >> > >    4. Broker stores this token in its in memory cache.
> > Broker
> > > > > also stores
> > > > > > >> > >    the DelegationToken without the hmac in the zookeeper.
> > > > > > >> > >    5. All brokers will have a cache backed by zookeeper so
> > they
> > > > > will all
> > > > > > >> > >    get notified whenever a new token is generated and they
> > will
> > > > > update
> > > > > > >> > their
> > > > > > >> > >    local cache whenever token state changes.
> > > > > > >> > >    6. Broker returns the token to Client.
> > > > > > >> > >
> > > > > > >> > > Probable issues and fixes
> > > > > > >> > >
> > > > > > >> > >    1. Probable race condition, client tries to
> authenticate
> > with
> > > > > a broker
> > > > > > >> > >    that is yet to be updated with the newly generated DT.
> > This
> > > > can
> > > > > > >> > probably be
> > > > > > >> > >    dealt with making dtRequest block until all brokers
> have
> > > > > updated
> > > > > > >> > their DT
> > > > > > >> > >    cache. Zk barrier or similar mechanism can be used.
> > However,
> > > > > all such
> > > > > > >> > >    mechanisms will increase complexity.
> > > > > > >> > >    2. Using a static secret key from config file. Will
> > require
> > > > yet
> > > > > > >> > another
> > > > > > >> > >    config and uses a static secret key. It is advised to
> > rotate
> > > > > secret
> > > > > > >> > keys
> > > > > > >> > >    periodically. This can be avoided with controller
> > generating
> > > > > > >> > secretKey and
> > > > > > >> > >    passing to brokers periodically. However, this will
> > require
> > > > > brokers to
> > > > > > >> > >    maintain certain counts of secretKeys.
> > > > > > >> > >
> > > > > > >> > > (Alternative 1) Have controller generate delegation token.
> > > > > > >> > >
> > > > > > >> > >    1. Client authenticates with a broker.
> > > > > > >> > >    2. Once a client is authenticated, it will make a
> broker
> > side
> > > > > call to
> > > > > > >> > >    issue a delegation token.
> > > > > > >> > >    3. Broker forwards the request to controller.
> > > > > > >> > >    4. Controller generates a DT and broadcasts to all
> > brokers.
> > > > > > >> > >    5. Broker stores this token in its memory cache.
> > > > > > >> > >    6. Controller responds to broker’s DT req.
> > > > > > >> > >    7. Broker returns the token to Client.
> > > > > > >> > >
> > > > > > >> > > Probable issues and fixes
> > > > > > >> > >
> > > > > > >> > >    1. We will have to add new APIs to support controller
> > pushing
> > > > > tokens
> > > > > > >> > to
> > > > > > >> > >    brokers on top of the minimal APIs that are currently
> > > > proposed.
> > > > > > >> > >    2. We will also have to add APIs to support the
> > bootstrapping
> > > > > case,
> > > > > > >> > i.e,
> > > > > > >> > >    when a new broker comes up it will have to get all
> > delegation
> > > > > tokens
> > > > > > >> > from
> > > > > > >> > >    the controller.
> > > > > > >> > >    3. In catastrophic failures where all brokers go down,
> > the
> > > > > tokens will
> > > > > > >> > >    be lost even if servers are restarted as tokens are not
> > > > > persisted
> > > > > > >> > anywhere.
> > > > > > >> > >    If this happens, then there are more important things
> to
> > > > worry
> > > > > about
> > > > > > >> > and
> > > > > > >> > >    maybe it is better to re-authenticate.
> > > > > > >> > >
> > > > > > >> > > (Alternative 2) Do not distribute DT to other brokers at
> > all.
> > > > > > >> > >
> > > > > > >> > >    1. Client authenticates with a broker.
> > > > > > >> > >    2. Once a client is authenticated, it will make a
> broker
> > side
> > > > > call to
> > > > > > >> > >    issue a delegation token.
> > > > > > >> > >    3. The broker generates DT of form, [hmac + (owner,
> > renewer,
> > > > > > >> > >    maxLifeTime, id, hmac, expirationTime)] and passes back
> > this
> > > > > DT to
> > > > > > >> > client.
> > > > > > >> > >    hmac is generated via {HMAC-SHA256(owner, renewer,
> > > > > maxLifeTime, id,
> > > > > > >> > hmac,
> > > > > > >> > >    expirationTime) using SecretKey}. Note that all brokers
> > have
> > > > > this
> > > > > > >> > SecretKey.
> > > > > > >> > >    4. Client then goes to any broker and to authenticate
> > sends
> > > > > the DT.
> > > > > > >> > >    Broker recalculates hmac using (owner, renewer,
> > maxLifeTime,
> > > > > id, hmac,
> > > > > > >> > >    expirationTime) info from DT and its SecretKey. If it
> > matches
> > > > > with
> > > > > > >> > hmac of
> > > > > > >> > >    DT, client is authenticated. Yes, it will do other
> > obvious
> > > > > checks of
> > > > > > >> > >    timestamp expiry and such.
> > > > > > >> > >
> > > > > > >> > > Note that secret key will be generated by controller and
> > passed
> > > > to
> > > > > > >> > brokers
> > > > > > >> > > periodically.
> > > > > > >> > > Probable issues and fixes
> > > > > > >> > >
> > > > > > >> > >    1. How to delete a DT? Yes, that is a downside here.
> > However,
> > > > > this can
> > > > > > >> > >    be handled with brokers maintaining a blacklist of DTs,
> > DTs
> > > > > from this
> > > > > > >> > list
> > > > > > >> > >    can be removed after expiry.
> > > > > > >> > >    2. In catastrophic failures where all brokers go down,
> > the
> > > > > tokens will
> > > > > > >> > >    be lost even if servers are restarted as tokens are not
> > > > > persisted
> > > > > > >> > anywhere.
> > > > > > >> > >    If this happens, then there are more important things
> to
> > > > worry
> > > > > about
> > > > > > >> > and
> > > > > > >> > >    maybe it is better to re-authenticate.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth Brahmbhatt <
> > > > > > >> > > pbrahmbhatt@hortonworks.com> wrote:
> > > > > > >> > >
> > > > > > >> > >> Hi,
> > > > > > >> > >>
> > > > > > >> > >> I have filed KIP-48 so we can offer hadoop like
> delegation
> > > > > tokens in
> > > > > > >> > >> kafka. You can review the design
> > > > > > >> > >>
> > > > > > >> >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > > > > > >> > .
> > > > > > >> > >> This KIP depends on KIP-43 and we have also discussed an
> > > > > alternative to
> > > > > > >> > >> proposed design here<
> > > > > > >> > >>
> > > > > > >> >
> > > > >
> > > >
> >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > > > > > >> > >> >.
> > > > > > >> > >>
> > > > > > >> > >> Thanks
> > > > > > >> > >> Parth
> > > > > > >> > >>
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > --
> > > > > > >> > >
> > > > > > >> > > Regards,
> > > > > > >> > > Ashish
> > > > > > >> >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Regards,
> > > > Ashish
> > > >
> >
>