You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Rajini Sivaram <ra...@googlemail.com> on 2016/10/04 13:43:36 UTC
[DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Hi all,
I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
mechanisms to Kafka:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-84%3A+Support+SASL+SCRAM+mechanisms
Comments and suggestions are welcome.
Thank you...
Regards,
Rajini
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Rajini Sivaram <ra...@googlemail.com>.
I have created KIP-86 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-86%3A+Configurable+SASL+callback+handlers)
to enable pluggable credential providers.
On Mon, Oct 10, 2016 at 7:40 PM, Rajini Sivaram <
rajinisivaram@googlemail.com> wrote:
> Gwen,
>
> Thank you, will raise a separate KIP for a pluggable interface.
>
> On Mon, Oct 10, 2016 at 5:55 PM, Gwen Shapira <gw...@confluent.io> wrote:
>
>> I think it is fine to break the password store to an interface in a
>> separate KIP. I actually love the idea of smaller KIPs dealing with
>> more specific functionality. I just wasn't clear why it was rejected.
>>
>> Thank you for clarifying. I'm happy with current proposal.
>>
>> Gwen
>>
>> On Mon, Oct 10, 2016 at 2:17 AM, Rajini Sivaram
>> <ra...@googlemail.com> wrote:
>> > Gwen,
>> >
>> > Thank you for reviewing the KIP.
>> >
>> > There has been interest in making the password verification in
>> SASL/PLAIN
>> > more pluggable. So I think it makes sense to have a pluggable interface
>> > that can be adopted for any SASL mechanism rather than just SCRAM. With
>> the
>> > current proposal, you can plugin another Scram SaslServer implementation
>> > with a different password store. This is similar to the current
>> SASL/PLAIN
>> > implementation.
>> >
>> > I agree that it will be good to make password stores more pluggable
>> rather
>> > than require users to override the whole SaslServer. I was going to look
>> > into this later, but I can do it as part of this KIP. Will update the
>> KIP
>> > with a pluggable interface.
>> >
>> > Thank you,
>> >
>> > Rajini
>> >
>> >
>> > On Fri, Oct 7, 2016 at 11:37 PM, Gwen Shapira <gw...@confluent.io>
>> wrote:
>> >
>> >> Can you talk more about rejecting the option of making the password
>> >> store pluggable? I am a bit uncomfortable with making ZK the one and
>> >> only password store...
>> >>
>> >> On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram
>> >> <ra...@googlemail.com> wrote:
>> >> > Hi all,
>> >> >
>> >> > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
>> >> > mechanisms to Kafka:
>> >> >
>> >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> >> 84%3A+Support+SASL+SCRAM+mechanisms
>> >> >
>> >> >
>> >> > Comments and suggestions are welcome.
>> >> >
>> >> > Thank you...
>> >> >
>> >> > Regards,
>> >> >
>> >> > Rajini
>> >>
>> >>
>> >>
>> >> --
>> >> Gwen Shapira
>> >> Product Manager | Confluent
>> >> 650.450.2760 | @gwenshap
>> >> Follow us: Twitter | blog
>> >>
>> >
>> >
>> >
>> > --
>> > Regards,
>> >
>> > Rajini
>>
>>
>>
>> --
>> Gwen Shapira
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>>
>
>
>
> --
> Regards,
>
> Rajini
>
--
Regards,
Rajini
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Rajini Sivaram <ra...@googlemail.com>.
Gwen,
Thank you, will raise a separate KIP for a pluggable interface.
On Mon, Oct 10, 2016 at 5:55 PM, Gwen Shapira <gw...@confluent.io> wrote:
> I think it is fine to break the password store to an interface in a
> separate KIP. I actually love the idea of smaller KIPs dealing with
> more specific functionality. I just wasn't clear why it was rejected.
>
> Thank you for clarifying. I'm happy with current proposal.
>
> Gwen
>
> On Mon, Oct 10, 2016 at 2:17 AM, Rajini Sivaram
> <ra...@googlemail.com> wrote:
> > Gwen,
> >
> > Thank you for reviewing the KIP.
> >
> > There has been interest in making the password verification in SASL/PLAIN
> > more pluggable. So I think it makes sense to have a pluggable interface
> > that can be adopted for any SASL mechanism rather than just SCRAM. With
> the
> > current proposal, you can plugin another Scram SaslServer implementation
> > with a different password store. This is similar to the current
> SASL/PLAIN
> > implementation.
> >
> > I agree that it will be good to make password stores more pluggable
> rather
> > than require users to override the whole SaslServer. I was going to look
> > into this later, but I can do it as part of this KIP. Will update the KIP
> > with a pluggable interface.
> >
> > Thank you,
> >
> > Rajini
> >
> >
> > On Fri, Oct 7, 2016 at 11:37 PM, Gwen Shapira <gw...@confluent.io> wrote:
> >
> >> Can you talk more about rejecting the option of making the password
> >> store pluggable? I am a bit uncomfortable with making ZK the one and
> >> only password store...
> >>
> >> On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram
> >> <ra...@googlemail.com> wrote:
> >> > Hi all,
> >> >
> >> > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
> >> > mechanisms to Kafka:
> >> >
> >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> 84%3A+Support+SASL+SCRAM+mechanisms
> >> >
> >> >
> >> > Comments and suggestions are welcome.
> >> >
> >> > Thank you...
> >> >
> >> > Regards,
> >> >
> >> > Rajini
> >>
> >>
> >>
> >> --
> >> Gwen Shapira
> >> Product Manager | Confluent
> >> 650.450.2760 | @gwenshap
> >> Follow us: Twitter | blog
> >>
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>
--
Regards,
Rajini
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Gwen Shapira <gw...@confluent.io>.
I think it is fine to break the password store to an interface in a
separate KIP. I actually love the idea of smaller KIPs dealing with
more specific functionality. I just wasn't clear why it was rejected.
Thank you for clarifying. I'm happy with current proposal.
Gwen
On Mon, Oct 10, 2016 at 2:17 AM, Rajini Sivaram
<ra...@googlemail.com> wrote:
> Gwen,
>
> Thank you for reviewing the KIP.
>
> There has been interest in making the password verification in SASL/PLAIN
> more pluggable. So I think it makes sense to have a pluggable interface
> that can be adopted for any SASL mechanism rather than just SCRAM. With the
> current proposal, you can plugin another Scram SaslServer implementation
> with a different password store. This is similar to the current SASL/PLAIN
> implementation.
>
> I agree that it will be good to make password stores more pluggable rather
> than require users to override the whole SaslServer. I was going to look
> into this later, but I can do it as part of this KIP. Will update the KIP
> with a pluggable interface.
>
> Thank you,
>
> Rajini
>
>
> On Fri, Oct 7, 2016 at 11:37 PM, Gwen Shapira <gw...@confluent.io> wrote:
>
>> Can you talk more about rejecting the option of making the password
>> store pluggable? I am a bit uncomfortable with making ZK the one and
>> only password store...
>>
>> On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram
>> <ra...@googlemail.com> wrote:
>> > Hi all,
>> >
>> > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
>> > mechanisms to Kafka:
>> >
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 84%3A+Support+SASL+SCRAM+mechanisms
>> >
>> >
>> > Comments and suggestions are welcome.
>> >
>> > Thank you...
>> >
>> > Regards,
>> >
>> > Rajini
>>
>>
>>
>> --
>> Gwen Shapira
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>>
>
>
>
> --
> Regards,
>
> Rajini
--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Rajini Sivaram <ra...@googlemail.com>.
Gwen,
Thank you for reviewing the KIP.
There has been interest in making the password verification in SASL/PLAIN
more pluggable. So I think it makes sense to have a pluggable interface
that can be adopted for any SASL mechanism rather than just SCRAM. With the
current proposal, you can plugin another Scram SaslServer implementation
with a different password store. This is similar to the current SASL/PLAIN
implementation.
I agree that it will be good to make password stores more pluggable rather
than require users to override the whole SaslServer. I was going to look
into this later, but I can do it as part of this KIP. Will update the KIP
with a pluggable interface.
Thank you,
Rajini
On Fri, Oct 7, 2016 at 11:37 PM, Gwen Shapira <gw...@confluent.io> wrote:
> Can you talk more about rejecting the option of making the password
> store pluggable? I am a bit uncomfortable with making ZK the one and
> only password store...
>
> On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram
> <ra...@googlemail.com> wrote:
> > Hi all,
> >
> > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
> > mechanisms to Kafka:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 84%3A+Support+SASL+SCRAM+mechanisms
> >
> >
> > Comments and suggestions are welcome.
> >
> > Thank you...
> >
> > Regards,
> >
> > Rajini
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>
--
Regards,
Rajini
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Gwen Shapira <gw...@confluent.io>.
Can you talk more about rejecting the option of making the password
store pluggable? I am a bit uncomfortable with making ZK the one and
only password store...
On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram
<ra...@googlemail.com> wrote:
> Hi all,
>
> I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
> mechanisms to Kafka:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-84%3A+Support+SASL+SCRAM+mechanisms
>
>
> Comments and suggestions are welcome.
>
> Thank you...
>
> Regards,
>
> Rajini
--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Roger Hoover <ro...@gmail.com>.
Yes. Thank you, Ismael.
On Wed, Feb 8, 2017 at 2:30 AM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Roger,
>
> Sorry for the delay. SCRAM is specified by:
>
> https://tools.ietf.org/html/rfc5802
>
> The following quote is relevant:
>
> A SCRAM mechanism name is a string "SCRAM-" followed by the
> > uppercased name of the underlying hash function taken from the IANA
> > "Hash Function Textual Names" registry (see http://www.iana.org),
> > optionally followed by the suffix "-PLUS" (see below)
>
>
> And:
>
> "md2" 1.2.840.113549.2.2 [RFC3279]
> > "md5" 1.2.840.113549.2.5 [RFC3279]
> > "sha-1" 1.3.14.3.2.26 [RFC3279]
> > "sha-224" 2.16.840.1.101.3.4.2.4 [RFC4055]
> > "sha-256" 2.16.840.1.101.3.4.2.1 [RFC4055]
> > "sha-384" 2.16.840.1.101.3.4.2.2 [RFC4055]
> > "sha-512" 2.16.840.1.101.3.4.2.3 [RFC4055]
>
>
> https://www.iana.org/assignments/hash-function-
> text-names/hash-function-text-names.xhtml
>
> As you see, bcrypt is not an option for the current spec. The naming of the
> mechanisms would be a bit misleading if support for bcrypt was added
> (SCRAM-PKBDF2-SHA512, SCRAM-BCRYPT*, etc. would be better).
>
> Does that make sense?
>
> Ismael
>
> On Tue, Jan 24, 2017 at 7:26 PM, Roger Hoover <ro...@gmail.com>
> wrote:
>
> > Thanks, Ismael. Just curious, why does it not make sense to do bcrypt
> > it in the context of SCRAM?
> >
> > On Mon, Jan 23, 2017 at 3:54 PM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Roger,
> > >
> > > SCRAM uses the PBKDF2 mechanism, here's a comparison between PBKDF2 and
> > > bcrypt:
> > >
> > > http://security.stackexchange.com/questions/4781/do-any-secu
> > > rity-experts-recommend-bcrypt-for-password-storage/6415#6415
> > >
> > > It may be worth supporting bcrypt, but not sure it would make sense to
> do
> > > it in the context of SCRAM.
> > >
> > > A minor correction: the KIP includes SCRAM-SHA-256 and SCRAM-SHA-512
> (not
> > > SCRAM-SHA-1).
> > >
> > > Ismael
> > >
> > > On Mon, Jan 23, 2017 at 10:49 PM, Roger Hoover <roger.hoover@gmail.com
> >
> > > wrote:
> > >
> > > > Sorry for the late question but is there a reason to choose SHA-1 and
> > > > SHA-256 instead of bcrypt?
> > > >
> > > > https://codahale.com/how-to-safely-store-a-password/
> > > >
> > > > On Fri, Nov 11, 2016 at 5:30 AM, Rajini Sivaram <
> > > > rajinisivaram@googlemail.com> wrote:
> > > >
> > > > > I think all the comments and suggestions on this thread have now
> been
> > > > > incorporated into the KIP. If there are no objections, I will start
> > the
> > > > > voting process on Monday.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > > > On Tue, Nov 8, 2016 at 9:20 PM, Rajini Sivaram <
> > > > > rajinisivaram@googlemail.com
> > > > > > wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > > Have added a sub-section on delegation token support to the KIP.
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > > On Tue, Nov 8, 2016 at 8:07 PM, Jun Rao <ju...@confluent.io>
> wrote:
> > > > > >
> > > > > >> Hi, Rajini,
> > > > > >>
> > > > > >> That makes sense. Could you document this potential future
> > extension
> > > > in
> > > > > >> the
> > > > > >> KIP?
> > > > > >>
> > > > > >> Jun
> > > > > >>
> > > > > >> On Tue, Nov 8, 2016 at 11:17 AM, Rajini Sivaram <
> > > > > >> rajinisivaram@googlemail.com> wrote:
> > > > > >>
> > > > > >> > Jun,
> > > > > >> >
> > > > > >> > 11. SCRAM messages have an optional extensions field which is
> a
> > > list
> > > > > of
> > > > > >> > key=value pairs. We can add an extension key to the first
> client
> > > > > >> message to
> > > > > >> > indicate delegation token. Broker can then obtain credentials
> > and
> > > > > >> principal
> > > > > >> > using a different code path for delegation tokens.
> > > > > >> >
> > > > > >> > On Tue, Nov 8, 2016 at 6:38 PM, Jun Rao <ju...@confluent.io>
> > wrote:
> > > > > >> >
> > > > > >> > > Magnus,
> > > > > >> > >
> > > > > >> > > Thanks for the input. If you don't feel strongly the need to
> > > bump
> > > > up
> > > > > >> the
> > > > > >> > > version of SaslHandshake, we can leave the version
> unchanged.
> > > > > >> > >
> > > > > >> > > Rajini,
> > > > > >> > >
> > > > > >> > > 11. Yes, we could send the HMAC as the SCRAM password for
> the
> > > > > >> delegation
> > > > > >> > > token. Do we need something to indicate that this SCRAM
> token
> > is
> > > > > >> special
> > > > > >> > > (i.e., delegation token) so that we can generate the correct
> > > > > >> > > KafkaPrincipal? The delegation token logic can be added
> > later. I
> > > > am
> > > > > >> > asking
> > > > > >> > > just so that we have enough in the design of SCRAM to add
> the
> > > > > >> delegation
> > > > > >> > > token logic later.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > >
> > > > > >> > > Jun
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Tue, Nov 8, 2016 at 4:42 AM, Rajini Sivaram <
> > > > > >> > > rajinisivaram@googlemail.com
> > > > > >> > > > wrote:
> > > > > >> > >
> > > > > >> > > > Hi Jun,
> > > > > >> > > >
> > > > > >> > > > 10. *s=<salt>* and *i=<iterations>* come from the SCRAM
> > > standard
> > > > > >> (they
> > > > > >> > > are
> > > > > >> > > > transferred during SCRAM auth). Scram messages look like
> > (for
> > > > > >> example)
> > > > > >> > > > *r=<nonce>,s=<salt>,i=<iterations>*. StoredKey and
> > ServerKey
> > > > and
> > > > > >> not
> > > > > >> > > > transferred in SCRAM messages, so I picked two keys that
> are
> > > > > unused
> > > > > >> in
> > > > > >> > > > SCRAM.
> > > > > >> > > >
> > > > > >> > > > 11. SCRAM (like DIGEST-MD5 or PLAIN) uses a shared
> > > > secret/password
> > > > > >> for
> > > > > >> > > > authentication along with a username and an optional
> > > > > >> authorization-id.
> > > > > >> > > > Kafka uses the username as the identity (Kafka principal)
> > for
> > > > > >> > > > authentication and authorization. KIP-48 doesn't mention
> > > > > >> KafkaPrincipal
> > > > > >> > > in
> > > > > >> > > > the section "Authentication using Token", but a delegation
> > > token
> > > > > is
> > > > > >> > > > associated with a Kafka principal. Since delegation tokens
> > are
> > > > > >> acquired
> > > > > >> > > on
> > > > > >> > > > behalf of a KafkaPrincipal and the principal is included
> in
> > > the
> > > > > >> token
> > > > > >> > as
> > > > > >> > > > the token owner, clients authenticating with delegation
> > > tokens
> > > > > >> could
> > > > > >> > use
> > > > > >> > > > the token owner as username and the token HMAC as shared
> > > > > >> > secret/password.
> > > > > >> > > >
> > > > > >> > > > If necessary, any other form of token identifier may be
> used
> > > as
> > > > > >> > username
> > > > > >> > > as
> > > > > >> > > > well as long as it contains sufficient information for the
> > > > broker
> > > > > to
> > > > > >> > > > retrieve/compute the principal and HMAC for
> authentication.
> > > The
> > > > > >> server
> > > > > >> > > > callback handler can be updated when delegation tokens are
> > > > > >> implemented
> > > > > >> > to
> > > > > >> > > > generate Kafka principal accordingly.
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On Tue, Nov 8, 2016 at 1:03 AM, Jun Rao <jun@confluent.io
> >
> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi, Rajini,
> > > > > >> > > > >
> > > > > >> > > > > A couple of other questions on the KIP.
> > > > > >> > > > >
> > > > > >> > > > > 10. For the config values stored in ZK, are those keys
> (s,
> > > t,
> > > > k,
> > > > > >> i,
> > > > > >> > > etc)
> > > > > >> > > > > stored under scram-sha-256 standard?
> > > > > >> > > > >
> > > > > >> > > > > 11. Could KIP-48 (delegation token) use this KIP to send
> > > > > >> delegation
> > > > > >> > > > tokens?
> > > > > >> > > > > In KIP-48, the client sends a HMAC as the delegation
> token
> > > to
> > > > > the
> > > > > >> > > server.
> > > > > >> > > > > Not sure how this gets mapped to the username/password
> in
> > > this
> > > > > >> KIP.
> > > > > >> > > > >
> > > > > >> > > > > Thanks,
> > > > > >> > > > >
> > > > > >> > > > > Jun
> > > > > >> > > > >
> > > > > >> > > > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > > > > >> > > > > rajinisivaram@googlemail.com
> > > > > >> > > > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Hi all,
> > > > > >> > > > > >
> > > > > >> > > > > > I have just created KIP-84 to add SCRAM-SHA-1 and
> > > > > SCRAM-SHA-256
> > > > > >> > SASL
> > > > > >> > > > > > mechanisms to Kafka:
> > > > > >> > > > > >
> > > > > >> > > > > > https://cwiki.apache.org/
> confluence/display/KAFKA/KIP-
> > > > > >> > > > > > 84%3A+Support+SASL+SCRAM+mechanisms
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > Comments and suggestions are welcome.
> > > > > >> > > > > >
> > > > > >> > > > > > Thank you...
> > > > > >> > > > > >
> > > > > >> > > > > > Regards,
> > > > > >> > > > > >
> > > > > >> > > > > > Rajini
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > --
> > > > > >> > > > Regards,
> > > > > >> > > >
> > > > > >> > > > Rajini
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > --
> > > > > >> > Regards,
> > > > > >> >
> > > > > >> > Rajini
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Roger,
Sorry for the delay. SCRAM is specified by:
https://tools.ietf.org/html/rfc5802
The following quote is relevant:
A SCRAM mechanism name is a string "SCRAM-" followed by the
> uppercased name of the underlying hash function taken from the IANA
> "Hash Function Textual Names" registry (see http://www.iana.org),
> optionally followed by the suffix "-PLUS" (see below)
And:
"md2" 1.2.840.113549.2.2 [RFC3279]
> "md5" 1.2.840.113549.2.5 [RFC3279]
> "sha-1" 1.3.14.3.2.26 [RFC3279]
> "sha-224" 2.16.840.1.101.3.4.2.4 [RFC4055]
> "sha-256" 2.16.840.1.101.3.4.2.1 [RFC4055]
> "sha-384" 2.16.840.1.101.3.4.2.2 [RFC4055]
> "sha-512" 2.16.840.1.101.3.4.2.3 [RFC4055]
https://www.iana.org/assignments/hash-function-text-names/hash-function-text-names.xhtml
As you see, bcrypt is not an option for the current spec. The naming of the
mechanisms would be a bit misleading if support for bcrypt was added
(SCRAM-PKBDF2-SHA512, SCRAM-BCRYPT*, etc. would be better).
Does that make sense?
Ismael
On Tue, Jan 24, 2017 at 7:26 PM, Roger Hoover <ro...@gmail.com>
wrote:
> Thanks, Ismael. Just curious, why does it not make sense to do bcrypt
> it in the context of SCRAM?
>
> On Mon, Jan 23, 2017 at 3:54 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Roger,
> >
> > SCRAM uses the PBKDF2 mechanism, here's a comparison between PBKDF2 and
> > bcrypt:
> >
> > http://security.stackexchange.com/questions/4781/do-any-secu
> > rity-experts-recommend-bcrypt-for-password-storage/6415#6415
> >
> > It may be worth supporting bcrypt, but not sure it would make sense to do
> > it in the context of SCRAM.
> >
> > A minor correction: the KIP includes SCRAM-SHA-256 and SCRAM-SHA-512 (not
> > SCRAM-SHA-1).
> >
> > Ismael
> >
> > On Mon, Jan 23, 2017 at 10:49 PM, Roger Hoover <ro...@gmail.com>
> > wrote:
> >
> > > Sorry for the late question but is there a reason to choose SHA-1 and
> > > SHA-256 instead of bcrypt?
> > >
> > > https://codahale.com/how-to-safely-store-a-password/
> > >
> > > On Fri, Nov 11, 2016 at 5:30 AM, Rajini Sivaram <
> > > rajinisivaram@googlemail.com> wrote:
> > >
> > > > I think all the comments and suggestions on this thread have now been
> > > > incorporated into the KIP. If there are no objections, I will start
> the
> > > > voting process on Monday.
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > > > On Tue, Nov 8, 2016 at 9:20 PM, Rajini Sivaram <
> > > > rajinisivaram@googlemail.com
> > > > > wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > > Have added a sub-section on delegation token support to the KIP.
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Rajini
> > > > >
> > > > > On Tue, Nov 8, 2016 at 8:07 PM, Jun Rao <ju...@confluent.io> wrote:
> > > > >
> > > > >> Hi, Rajini,
> > > > >>
> > > > >> That makes sense. Could you document this potential future
> extension
> > > in
> > > > >> the
> > > > >> KIP?
> > > > >>
> > > > >> Jun
> > > > >>
> > > > >> On Tue, Nov 8, 2016 at 11:17 AM, Rajini Sivaram <
> > > > >> rajinisivaram@googlemail.com> wrote:
> > > > >>
> > > > >> > Jun,
> > > > >> >
> > > > >> > 11. SCRAM messages have an optional extensions field which is a
> > list
> > > > of
> > > > >> > key=value pairs. We can add an extension key to the first client
> > > > >> message to
> > > > >> > indicate delegation token. Broker can then obtain credentials
> and
> > > > >> principal
> > > > >> > using a different code path for delegation tokens.
> > > > >> >
> > > > >> > On Tue, Nov 8, 2016 at 6:38 PM, Jun Rao <ju...@confluent.io>
> wrote:
> > > > >> >
> > > > >> > > Magnus,
> > > > >> > >
> > > > >> > > Thanks for the input. If you don't feel strongly the need to
> > bump
> > > up
> > > > >> the
> > > > >> > > version of SaslHandshake, we can leave the version unchanged.
> > > > >> > >
> > > > >> > > Rajini,
> > > > >> > >
> > > > >> > > 11. Yes, we could send the HMAC as the SCRAM password for the
> > > > >> delegation
> > > > >> > > token. Do we need something to indicate that this SCRAM token
> is
> > > > >> special
> > > > >> > > (i.e., delegation token) so that we can generate the correct
> > > > >> > > KafkaPrincipal? The delegation token logic can be added
> later. I
> > > am
> > > > >> > asking
> > > > >> > > just so that we have enough in the design of SCRAM to add the
> > > > >> delegation
> > > > >> > > token logic later.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > >
> > > > >> > > Jun
> > > > >> > >
> > > > >> > >
> > > > >> > > On Tue, Nov 8, 2016 at 4:42 AM, Rajini Sivaram <
> > > > >> > > rajinisivaram@googlemail.com
> > > > >> > > > wrote:
> > > > >> > >
> > > > >> > > > Hi Jun,
> > > > >> > > >
> > > > >> > > > 10. *s=<salt>* and *i=<iterations>* come from the SCRAM
> > standard
> > > > >> (they
> > > > >> > > are
> > > > >> > > > transferred during SCRAM auth). Scram messages look like
> (for
> > > > >> example)
> > > > >> > > > *r=<nonce>,s=<salt>,i=<iterations>*. StoredKey and
> ServerKey
> > > and
> > > > >> not
> > > > >> > > > transferred in SCRAM messages, so I picked two keys that are
> > > > unused
> > > > >> in
> > > > >> > > > SCRAM.
> > > > >> > > >
> > > > >> > > > 11. SCRAM (like DIGEST-MD5 or PLAIN) uses a shared
> > > secret/password
> > > > >> for
> > > > >> > > > authentication along with a username and an optional
> > > > >> authorization-id.
> > > > >> > > > Kafka uses the username as the identity (Kafka principal)
> for
> > > > >> > > > authentication and authorization. KIP-48 doesn't mention
> > > > >> KafkaPrincipal
> > > > >> > > in
> > > > >> > > > the section "Authentication using Token", but a delegation
> > token
> > > > is
> > > > >> > > > associated with a Kafka principal. Since delegation tokens
> are
> > > > >> acquired
> > > > >> > > on
> > > > >> > > > behalf of a KafkaPrincipal and the principal is included in
> > the
> > > > >> token
> > > > >> > as
> > > > >> > > > the token owner, clients authenticating with delegation
> > tokens
> > > > >> could
> > > > >> > use
> > > > >> > > > the token owner as username and the token HMAC as shared
> > > > >> > secret/password.
> > > > >> > > >
> > > > >> > > > If necessary, any other form of token identifier may be used
> > as
> > > > >> > username
> > > > >> > > as
> > > > >> > > > well as long as it contains sufficient information for the
> > > broker
> > > > to
> > > > >> > > > retrieve/compute the principal and HMAC for authentication.
> > The
> > > > >> server
> > > > >> > > > callback handler can be updated when delegation tokens are
> > > > >> implemented
> > > > >> > to
> > > > >> > > > generate Kafka principal accordingly.
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Tue, Nov 8, 2016 at 1:03 AM, Jun Rao <ju...@confluent.io>
> > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi, Rajini,
> > > > >> > > > >
> > > > >> > > > > A couple of other questions on the KIP.
> > > > >> > > > >
> > > > >> > > > > 10. For the config values stored in ZK, are those keys (s,
> > t,
> > > k,
> > > > >> i,
> > > > >> > > etc)
> > > > >> > > > > stored under scram-sha-256 standard?
> > > > >> > > > >
> > > > >> > > > > 11. Could KIP-48 (delegation token) use this KIP to send
> > > > >> delegation
> > > > >> > > > tokens?
> > > > >> > > > > In KIP-48, the client sends a HMAC as the delegation token
> > to
> > > > the
> > > > >> > > server.
> > > > >> > > > > Not sure how this gets mapped to the username/password in
> > this
> > > > >> KIP.
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > >
> > > > >> > > > > Jun
> > > > >> > > > >
> > > > >> > > > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > > > >> > > > > rajinisivaram@googlemail.com
> > > > >> > > > > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Hi all,
> > > > >> > > > > >
> > > > >> > > > > > I have just created KIP-84 to add SCRAM-SHA-1 and
> > > > SCRAM-SHA-256
> > > > >> > SASL
> > > > >> > > > > > mechanisms to Kafka:
> > > > >> > > > > >
> > > > >> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >> > > > > > 84%3A+Support+SASL+SCRAM+mechanisms
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > Comments and suggestions are welcome.
> > > > >> > > > > >
> > > > >> > > > > > Thank you...
> > > > >> > > > > >
> > > > >> > > > > > Regards,
> > > > >> > > > > >
> > > > >> > > > > > Rajini
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > --
> > > > >> > > > Regards,
> > > > >> > > >
> > > > >> > > > Rajini
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> > Regards,
> > > > >> >
> > > > >> > Rajini
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Roger Hoover <ro...@gmail.com>.
Thanks, Ismael. Just curious, why does it not make sense to do bcrypt
it in the context of SCRAM?
On Mon, Jan 23, 2017 at 3:54 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Roger,
>
> SCRAM uses the PBKDF2 mechanism, here's a comparison between PBKDF2 and
> bcrypt:
>
> http://security.stackexchange.com/questions/4781/do-any-secu
> rity-experts-recommend-bcrypt-for-password-storage/6415#6415
>
> It may be worth supporting bcrypt, but not sure it would make sense to do
> it in the context of SCRAM.
>
> A minor correction: the KIP includes SCRAM-SHA-256 and SCRAM-SHA-512 (not
> SCRAM-SHA-1).
>
> Ismael
>
> On Mon, Jan 23, 2017 at 10:49 PM, Roger Hoover <ro...@gmail.com>
> wrote:
>
> > Sorry for the late question but is there a reason to choose SHA-1 and
> > SHA-256 instead of bcrypt?
> >
> > https://codahale.com/how-to-safely-store-a-password/
> >
> > On Fri, Nov 11, 2016 at 5:30 AM, Rajini Sivaram <
> > rajinisivaram@googlemail.com> wrote:
> >
> > > I think all the comments and suggestions on this thread have now been
> > > incorporated into the KIP. If there are no objections, I will start the
> > > voting process on Monday.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Tue, Nov 8, 2016 at 9:20 PM, Rajini Sivaram <
> > > rajinisivaram@googlemail.com
> > > > wrote:
> > >
> > > > Jun,
> > > >
> > > > Have added a sub-section on delegation token support to the KIP.
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> > > > On Tue, Nov 8, 2016 at 8:07 PM, Jun Rao <ju...@confluent.io> wrote:
> > > >
> > > >> Hi, Rajini,
> > > >>
> > > >> That makes sense. Could you document this potential future extension
> > in
> > > >> the
> > > >> KIP?
> > > >>
> > > >> Jun
> > > >>
> > > >> On Tue, Nov 8, 2016 at 11:17 AM, Rajini Sivaram <
> > > >> rajinisivaram@googlemail.com> wrote:
> > > >>
> > > >> > Jun,
> > > >> >
> > > >> > 11. SCRAM messages have an optional extensions field which is a
> list
> > > of
> > > >> > key=value pairs. We can add an extension key to the first client
> > > >> message to
> > > >> > indicate delegation token. Broker can then obtain credentials and
> > > >> principal
> > > >> > using a different code path for delegation tokens.
> > > >> >
> > > >> > On Tue, Nov 8, 2016 at 6:38 PM, Jun Rao <ju...@confluent.io> wrote:
> > > >> >
> > > >> > > Magnus,
> > > >> > >
> > > >> > > Thanks for the input. If you don't feel strongly the need to
> bump
> > up
> > > >> the
> > > >> > > version of SaslHandshake, we can leave the version unchanged.
> > > >> > >
> > > >> > > Rajini,
> > > >> > >
> > > >> > > 11. Yes, we could send the HMAC as the SCRAM password for the
> > > >> delegation
> > > >> > > token. Do we need something to indicate that this SCRAM token is
> > > >> special
> > > >> > > (i.e., delegation token) so that we can generate the correct
> > > >> > > KafkaPrincipal? The delegation token logic can be added later. I
> > am
> > > >> > asking
> > > >> > > just so that we have enough in the design of SCRAM to add the
> > > >> delegation
> > > >> > > token logic later.
> > > >> > >
> > > >> > > Thanks,
> > > >> > >
> > > >> > > Jun
> > > >> > >
> > > >> > >
> > > >> > > On Tue, Nov 8, 2016 at 4:42 AM, Rajini Sivaram <
> > > >> > > rajinisivaram@googlemail.com
> > > >> > > > wrote:
> > > >> > >
> > > >> > > > Hi Jun,
> > > >> > > >
> > > >> > > > 10. *s=<salt>* and *i=<iterations>* come from the SCRAM
> standard
> > > >> (they
> > > >> > > are
> > > >> > > > transferred during SCRAM auth). Scram messages look like (for
> > > >> example)
> > > >> > > > *r=<nonce>,s=<salt>,i=<iterations>*. StoredKey and ServerKey
> > and
> > > >> not
> > > >> > > > transferred in SCRAM messages, so I picked two keys that are
> > > unused
> > > >> in
> > > >> > > > SCRAM.
> > > >> > > >
> > > >> > > > 11. SCRAM (like DIGEST-MD5 or PLAIN) uses a shared
> > secret/password
> > > >> for
> > > >> > > > authentication along with a username and an optional
> > > >> authorization-id.
> > > >> > > > Kafka uses the username as the identity (Kafka principal) for
> > > >> > > > authentication and authorization. KIP-48 doesn't mention
> > > >> KafkaPrincipal
> > > >> > > in
> > > >> > > > the section "Authentication using Token", but a delegation
> token
> > > is
> > > >> > > > associated with a Kafka principal. Since delegation tokens are
> > > >> acquired
> > > >> > > on
> > > >> > > > behalf of a KafkaPrincipal and the principal is included in
> the
> > > >> token
> > > >> > as
> > > >> > > > the token owner, clients authenticating with delegation
> tokens
> > > >> could
> > > >> > use
> > > >> > > > the token owner as username and the token HMAC as shared
> > > >> > secret/password.
> > > >> > > >
> > > >> > > > If necessary, any other form of token identifier may be used
> as
> > > >> > username
> > > >> > > as
> > > >> > > > well as long as it contains sufficient information for the
> > broker
> > > to
> > > >> > > > retrieve/compute the principal and HMAC for authentication.
> The
> > > >> server
> > > >> > > > callback handler can be updated when delegation tokens are
> > > >> implemented
> > > >> > to
> > > >> > > > generate Kafka principal accordingly.
> > > >> > > >
> > > >> > > >
> > > >> > > > On Tue, Nov 8, 2016 at 1:03 AM, Jun Rao <ju...@confluent.io>
> > wrote:
> > > >> > > >
> > > >> > > > > Hi, Rajini,
> > > >> > > > >
> > > >> > > > > A couple of other questions on the KIP.
> > > >> > > > >
> > > >> > > > > 10. For the config values stored in ZK, are those keys (s,
> t,
> > k,
> > > >> i,
> > > >> > > etc)
> > > >> > > > > stored under scram-sha-256 standard?
> > > >> > > > >
> > > >> > > > > 11. Could KIP-48 (delegation token) use this KIP to send
> > > >> delegation
> > > >> > > > tokens?
> > > >> > > > > In KIP-48, the client sends a HMAC as the delegation token
> to
> > > the
> > > >> > > server.
> > > >> > > > > Not sure how this gets mapped to the username/password in
> this
> > > >> KIP.
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > >
> > > >> > > > > Jun
> > > >> > > > >
> > > >> > > > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > > >> > > > > rajinisivaram@googlemail.com
> > > >> > > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Hi all,
> > > >> > > > > >
> > > >> > > > > > I have just created KIP-84 to add SCRAM-SHA-1 and
> > > SCRAM-SHA-256
> > > >> > SASL
> > > >> > > > > > mechanisms to Kafka:
> > > >> > > > > >
> > > >> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >> > > > > > 84%3A+Support+SASL+SCRAM+mechanisms
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > Comments and suggestions are welcome.
> > > >> > > > > >
> > > >> > > > > > Thank you...
> > > >> > > > > >
> > > >> > > > > > Regards,
> > > >> > > > > >
> > > >> > > > > > Rajini
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > --
> > > >> > > > Regards,
> > > >> > > >
> > > >> > > > Rajini
> > > >> > > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Regards,
> > > >> >
> > > >> > Rajini
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Rajini
> > >
> >
>
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Roger,
SCRAM uses the PBKDF2 mechanism, here's a comparison between PBKDF2 and
bcrypt:
http://security.stackexchange.com/questions/4781/do-any-secu
rity-experts-recommend-bcrypt-for-password-storage/6415#6415
It may be worth supporting bcrypt, but not sure it would make sense to do
it in the context of SCRAM.
A minor correction: the KIP includes SCRAM-SHA-256 and SCRAM-SHA-512 (not
SCRAM-SHA-1).
Ismael
On Mon, Jan 23, 2017 at 10:49 PM, Roger Hoover <ro...@gmail.com>
wrote:
> Sorry for the late question but is there a reason to choose SHA-1 and
> SHA-256 instead of bcrypt?
>
> https://codahale.com/how-to-safely-store-a-password/
>
> On Fri, Nov 11, 2016 at 5:30 AM, Rajini Sivaram <
> rajinisivaram@googlemail.com> wrote:
>
> > I think all the comments and suggestions on this thread have now been
> > incorporated into the KIP. If there are no objections, I will start the
> > voting process on Monday.
> >
> > Regards,
> >
> > Rajini
> >
> > On Tue, Nov 8, 2016 at 9:20 PM, Rajini Sivaram <
> > rajinisivaram@googlemail.com
> > > wrote:
> >
> > > Jun,
> > >
> > > Have added a sub-section on delegation token support to the KIP.
> > >
> > > Thank you,
> > >
> > > Rajini
> > >
> > > On Tue, Nov 8, 2016 at 8:07 PM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > >> Hi, Rajini,
> > >>
> > >> That makes sense. Could you document this potential future extension
> in
> > >> the
> > >> KIP?
> > >>
> > >> Jun
> > >>
> > >> On Tue, Nov 8, 2016 at 11:17 AM, Rajini Sivaram <
> > >> rajinisivaram@googlemail.com> wrote:
> > >>
> > >> > Jun,
> > >> >
> > >> > 11. SCRAM messages have an optional extensions field which is a list
> > of
> > >> > key=value pairs. We can add an extension key to the first client
> > >> message to
> > >> > indicate delegation token. Broker can then obtain credentials and
> > >> principal
> > >> > using a different code path for delegation tokens.
> > >> >
> > >> > On Tue, Nov 8, 2016 at 6:38 PM, Jun Rao <ju...@confluent.io> wrote:
> > >> >
> > >> > > Magnus,
> > >> > >
> > >> > > Thanks for the input. If you don't feel strongly the need to bump
> up
> > >> the
> > >> > > version of SaslHandshake, we can leave the version unchanged.
> > >> > >
> > >> > > Rajini,
> > >> > >
> > >> > > 11. Yes, we could send the HMAC as the SCRAM password for the
> > >> delegation
> > >> > > token. Do we need something to indicate that this SCRAM token is
> > >> special
> > >> > > (i.e., delegation token) so that we can generate the correct
> > >> > > KafkaPrincipal? The delegation token logic can be added later. I
> am
> > >> > asking
> > >> > > just so that we have enough in the design of SCRAM to add the
> > >> delegation
> > >> > > token logic later.
> > >> > >
> > >> > > Thanks,
> > >> > >
> > >> > > Jun
> > >> > >
> > >> > >
> > >> > > On Tue, Nov 8, 2016 at 4:42 AM, Rajini Sivaram <
> > >> > > rajinisivaram@googlemail.com
> > >> > > > wrote:
> > >> > >
> > >> > > > Hi Jun,
> > >> > > >
> > >> > > > 10. *s=<salt>* and *i=<iterations>* come from the SCRAM standard
> > >> (they
> > >> > > are
> > >> > > > transferred during SCRAM auth). Scram messages look like (for
> > >> example)
> > >> > > > *r=<nonce>,s=<salt>,i=<iterations>*. StoredKey and ServerKey
> and
> > >> not
> > >> > > > transferred in SCRAM messages, so I picked two keys that are
> > unused
> > >> in
> > >> > > > SCRAM.
> > >> > > >
> > >> > > > 11. SCRAM (like DIGEST-MD5 or PLAIN) uses a shared
> secret/password
> > >> for
> > >> > > > authentication along with a username and an optional
> > >> authorization-id.
> > >> > > > Kafka uses the username as the identity (Kafka principal) for
> > >> > > > authentication and authorization. KIP-48 doesn't mention
> > >> KafkaPrincipal
> > >> > > in
> > >> > > > the section "Authentication using Token", but a delegation token
> > is
> > >> > > > associated with a Kafka principal. Since delegation tokens are
> > >> acquired
> > >> > > on
> > >> > > > behalf of a KafkaPrincipal and the principal is included in the
> > >> token
> > >> > as
> > >> > > > the token owner, clients authenticating with delegation tokens
> > >> could
> > >> > use
> > >> > > > the token owner as username and the token HMAC as shared
> > >> > secret/password.
> > >> > > >
> > >> > > > If necessary, any other form of token identifier may be used as
> > >> > username
> > >> > > as
> > >> > > > well as long as it contains sufficient information for the
> broker
> > to
> > >> > > > retrieve/compute the principal and HMAC for authentication. The
> > >> server
> > >> > > > callback handler can be updated when delegation tokens are
> > >> implemented
> > >> > to
> > >> > > > generate Kafka principal accordingly.
> > >> > > >
> > >> > > >
> > >> > > > On Tue, Nov 8, 2016 at 1:03 AM, Jun Rao <ju...@confluent.io>
> wrote:
> > >> > > >
> > >> > > > > Hi, Rajini,
> > >> > > > >
> > >> > > > > A couple of other questions on the KIP.
> > >> > > > >
> > >> > > > > 10. For the config values stored in ZK, are those keys (s, t,
> k,
> > >> i,
> > >> > > etc)
> > >> > > > > stored under scram-sha-256 standard?
> > >> > > > >
> > >> > > > > 11. Could KIP-48 (delegation token) use this KIP to send
> > >> delegation
> > >> > > > tokens?
> > >> > > > > In KIP-48, the client sends a HMAC as the delegation token to
> > the
> > >> > > server.
> > >> > > > > Not sure how this gets mapped to the username/password in this
> > >> KIP.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > >
> > >> > > > > Jun
> > >> > > > >
> > >> > > > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > >> > > > > rajinisivaram@googlemail.com
> > >> > > > > > wrote:
> > >> > > > >
> > >> > > > > > Hi all,
> > >> > > > > >
> > >> > > > > > I have just created KIP-84 to add SCRAM-SHA-1 and
> > SCRAM-SHA-256
> > >> > SASL
> > >> > > > > > mechanisms to Kafka:
> > >> > > > > >
> > >> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >> > > > > > 84%3A+Support+SASL+SCRAM+mechanisms
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Comments and suggestions are welcome.
> > >> > > > > >
> > >> > > > > > Thank you...
> > >> > > > > >
> > >> > > > > > Regards,
> > >> > > > > >
> > >> > > > > > Rajini
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > --
> > >> > > > Regards,
> > >> > > >
> > >> > > > Rajini
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Regards,
> > >> >
> > >> > Rajini
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Rajini
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Roger Hoover <ro...@gmail.com>.
Sorry for the late question but is there a reason to choose SHA-1 and
SHA-256 instead of bcrypt?
https://codahale.com/how-to-safely-store-a-password/
On Fri, Nov 11, 2016 at 5:30 AM, Rajini Sivaram <
rajinisivaram@googlemail.com> wrote:
> I think all the comments and suggestions on this thread have now been
> incorporated into the KIP. If there are no objections, I will start the
> voting process on Monday.
>
> Regards,
>
> Rajini
>
> On Tue, Nov 8, 2016 at 9:20 PM, Rajini Sivaram <
> rajinisivaram@googlemail.com
> > wrote:
>
> > Jun,
> >
> > Have added a sub-section on delegation token support to the KIP.
> >
> > Thank you,
> >
> > Rajini
> >
> > On Tue, Nov 8, 2016 at 8:07 PM, Jun Rao <ju...@confluent.io> wrote:
> >
> >> Hi, Rajini,
> >>
> >> That makes sense. Could you document this potential future extension in
> >> the
> >> KIP?
> >>
> >> Jun
> >>
> >> On Tue, Nov 8, 2016 at 11:17 AM, Rajini Sivaram <
> >> rajinisivaram@googlemail.com> wrote:
> >>
> >> > Jun,
> >> >
> >> > 11. SCRAM messages have an optional extensions field which is a list
> of
> >> > key=value pairs. We can add an extension key to the first client
> >> message to
> >> > indicate delegation token. Broker can then obtain credentials and
> >> principal
> >> > using a different code path for delegation tokens.
> >> >
> >> > On Tue, Nov 8, 2016 at 6:38 PM, Jun Rao <ju...@confluent.io> wrote:
> >> >
> >> > > Magnus,
> >> > >
> >> > > Thanks for the input. If you don't feel strongly the need to bump up
> >> the
> >> > > version of SaslHandshake, we can leave the version unchanged.
> >> > >
> >> > > Rajini,
> >> > >
> >> > > 11. Yes, we could send the HMAC as the SCRAM password for the
> >> delegation
> >> > > token. Do we need something to indicate that this SCRAM token is
> >> special
> >> > > (i.e., delegation token) so that we can generate the correct
> >> > > KafkaPrincipal? The delegation token logic can be added later. I am
> >> > asking
> >> > > just so that we have enough in the design of SCRAM to add the
> >> delegation
> >> > > token logic later.
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Jun
> >> > >
> >> > >
> >> > > On Tue, Nov 8, 2016 at 4:42 AM, Rajini Sivaram <
> >> > > rajinisivaram@googlemail.com
> >> > > > wrote:
> >> > >
> >> > > > Hi Jun,
> >> > > >
> >> > > > 10. *s=<salt>* and *i=<iterations>* come from the SCRAM standard
> >> (they
> >> > > are
> >> > > > transferred during SCRAM auth). Scram messages look like (for
> >> example)
> >> > > > *r=<nonce>,s=<salt>,i=<iterations>*. StoredKey and ServerKey and
> >> not
> >> > > > transferred in SCRAM messages, so I picked two keys that are
> unused
> >> in
> >> > > > SCRAM.
> >> > > >
> >> > > > 11. SCRAM (like DIGEST-MD5 or PLAIN) uses a shared secret/password
> >> for
> >> > > > authentication along with a username and an optional
> >> authorization-id.
> >> > > > Kafka uses the username as the identity (Kafka principal) for
> >> > > > authentication and authorization. KIP-48 doesn't mention
> >> KafkaPrincipal
> >> > > in
> >> > > > the section "Authentication using Token", but a delegation token
> is
> >> > > > associated with a Kafka principal. Since delegation tokens are
> >> acquired
> >> > > on
> >> > > > behalf of a KafkaPrincipal and the principal is included in the
> >> token
> >> > as
> >> > > > the token owner, clients authenticating with delegation tokens
> >> could
> >> > use
> >> > > > the token owner as username and the token HMAC as shared
> >> > secret/password.
> >> > > >
> >> > > > If necessary, any other form of token identifier may be used as
> >> > username
> >> > > as
> >> > > > well as long as it contains sufficient information for the broker
> to
> >> > > > retrieve/compute the principal and HMAC for authentication. The
> >> server
> >> > > > callback handler can be updated when delegation tokens are
> >> implemented
> >> > to
> >> > > > generate Kafka principal accordingly.
> >> > > >
> >> > > >
> >> > > > On Tue, Nov 8, 2016 at 1:03 AM, Jun Rao <ju...@confluent.io> wrote:
> >> > > >
> >> > > > > Hi, Rajini,
> >> > > > >
> >> > > > > A couple of other questions on the KIP.
> >> > > > >
> >> > > > > 10. For the config values stored in ZK, are those keys (s, t, k,
> >> i,
> >> > > etc)
> >> > > > > stored under scram-sha-256 standard?
> >> > > > >
> >> > > > > 11. Could KIP-48 (delegation token) use this KIP to send
> >> delegation
> >> > > > tokens?
> >> > > > > In KIP-48, the client sends a HMAC as the delegation token to
> the
> >> > > server.
> >> > > > > Not sure how this gets mapped to the username/password in this
> >> KIP.
> >> > > > >
> >> > > > > Thanks,
> >> > > > >
> >> > > > > Jun
> >> > > > >
> >> > > > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> >> > > > > rajinisivaram@googlemail.com
> >> > > > > > wrote:
> >> > > > >
> >> > > > > > Hi all,
> >> > > > > >
> >> > > > > > I have just created KIP-84 to add SCRAM-SHA-1 and
> SCRAM-SHA-256
> >> > SASL
> >> > > > > > mechanisms to Kafka:
> >> > > > > >
> >> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> > > > > > 84%3A+Support+SASL+SCRAM+mechanisms
> >> > > > > >
> >> > > > > >
> >> > > > > > Comments and suggestions are welcome.
> >> > > > > >
> >> > > > > > Thank you...
> >> > > > > >
> >> > > > > > Regards,
> >> > > > > >
> >> > > > > > Rajini
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Regards,
> >> > > >
> >> > > > Rajini
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Regards,
> >> >
> >> > Rajini
> >> >
> >>
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>
>
>
> --
> Regards,
>
> Rajini
>
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Rajini Sivaram <ra...@googlemail.com>.
I think all the comments and suggestions on this thread have now been
incorporated into the KIP. If there are no objections, I will start the
voting process on Monday.
Regards,
Rajini
On Tue, Nov 8, 2016 at 9:20 PM, Rajini Sivaram <rajinisivaram@googlemail.com
> wrote:
> Jun,
>
> Have added a sub-section on delegation token support to the KIP.
>
> Thank you,
>
> Rajini
>
> On Tue, Nov 8, 2016 at 8:07 PM, Jun Rao <ju...@confluent.io> wrote:
>
>> Hi, Rajini,
>>
>> That makes sense. Could you document this potential future extension in
>> the
>> KIP?
>>
>> Jun
>>
>> On Tue, Nov 8, 2016 at 11:17 AM, Rajini Sivaram <
>> rajinisivaram@googlemail.com> wrote:
>>
>> > Jun,
>> >
>> > 11. SCRAM messages have an optional extensions field which is a list of
>> > key=value pairs. We can add an extension key to the first client
>> message to
>> > indicate delegation token. Broker can then obtain credentials and
>> principal
>> > using a different code path for delegation tokens.
>> >
>> > On Tue, Nov 8, 2016 at 6:38 PM, Jun Rao <ju...@confluent.io> wrote:
>> >
>> > > Magnus,
>> > >
>> > > Thanks for the input. If you don't feel strongly the need to bump up
>> the
>> > > version of SaslHandshake, we can leave the version unchanged.
>> > >
>> > > Rajini,
>> > >
>> > > 11. Yes, we could send the HMAC as the SCRAM password for the
>> delegation
>> > > token. Do we need something to indicate that this SCRAM token is
>> special
>> > > (i.e., delegation token) so that we can generate the correct
>> > > KafkaPrincipal? The delegation token logic can be added later. I am
>> > asking
>> > > just so that we have enough in the design of SCRAM to add the
>> delegation
>> > > token logic later.
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > >
>> > > On Tue, Nov 8, 2016 at 4:42 AM, Rajini Sivaram <
>> > > rajinisivaram@googlemail.com
>> > > > wrote:
>> > >
>> > > > Hi Jun,
>> > > >
>> > > > 10. *s=<salt>* and *i=<iterations>* come from the SCRAM standard
>> (they
>> > > are
>> > > > transferred during SCRAM auth). Scram messages look like (for
>> example)
>> > > > *r=<nonce>,s=<salt>,i=<iterations>*. StoredKey and ServerKey and
>> not
>> > > > transferred in SCRAM messages, so I picked two keys that are unused
>> in
>> > > > SCRAM.
>> > > >
>> > > > 11. SCRAM (like DIGEST-MD5 or PLAIN) uses a shared secret/password
>> for
>> > > > authentication along with a username and an optional
>> authorization-id.
>> > > > Kafka uses the username as the identity (Kafka principal) for
>> > > > authentication and authorization. KIP-48 doesn't mention
>> KafkaPrincipal
>> > > in
>> > > > the section "Authentication using Token", but a delegation token is
>> > > > associated with a Kafka principal. Since delegation tokens are
>> acquired
>> > > on
>> > > > behalf of a KafkaPrincipal and the principal is included in the
>> token
>> > as
>> > > > the token owner, clients authenticating with delegation tokens
>> could
>> > use
>> > > > the token owner as username and the token HMAC as shared
>> > secret/password.
>> > > >
>> > > > If necessary, any other form of token identifier may be used as
>> > username
>> > > as
>> > > > well as long as it contains sufficient information for the broker to
>> > > > retrieve/compute the principal and HMAC for authentication. The
>> server
>> > > > callback handler can be updated when delegation tokens are
>> implemented
>> > to
>> > > > generate Kafka principal accordingly.
>> > > >
>> > > >
>> > > > On Tue, Nov 8, 2016 at 1:03 AM, Jun Rao <ju...@confluent.io> wrote:
>> > > >
>> > > > > Hi, Rajini,
>> > > > >
>> > > > > A couple of other questions on the KIP.
>> > > > >
>> > > > > 10. For the config values stored in ZK, are those keys (s, t, k,
>> i,
>> > > etc)
>> > > > > stored under scram-sha-256 standard?
>> > > > >
>> > > > > 11. Could KIP-48 (delegation token) use this KIP to send
>> delegation
>> > > > tokens?
>> > > > > In KIP-48, the client sends a HMAC as the delegation token to the
>> > > server.
>> > > > > Not sure how this gets mapped to the username/password in this
>> KIP.
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > Jun
>> > > > >
>> > > > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
>> > > > > rajinisivaram@googlemail.com
>> > > > > > wrote:
>> > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256
>> > SASL
>> > > > > > mechanisms to Kafka:
>> > > > > >
>> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > > > 84%3A+Support+SASL+SCRAM+mechanisms
>> > > > > >
>> > > > > >
>> > > > > > Comments and suggestions are welcome.
>> > > > > >
>> > > > > > Thank you...
>> > > > > >
>> > > > > > Regards,
>> > > > > >
>> > > > > > Rajini
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Regards,
>> > > >
>> > > > Rajini
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Regards,
>> >
>> > Rajini
>> >
>>
>
>
>
> --
> Regards,
>
> Rajini
>
--
Regards,
Rajini
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Rajini Sivaram <ra...@googlemail.com>.
Jun,
Have added a sub-section on delegation token support to the KIP.
Thank you,
Rajini
On Tue, Nov 8, 2016 at 8:07 PM, Jun Rao <ju...@confluent.io> wrote:
> Hi, Rajini,
>
> That makes sense. Could you document this potential future extension in the
> KIP?
>
> Jun
>
> On Tue, Nov 8, 2016 at 11:17 AM, Rajini Sivaram <
> rajinisivaram@googlemail.com> wrote:
>
> > Jun,
> >
> > 11. SCRAM messages have an optional extensions field which is a list of
> > key=value pairs. We can add an extension key to the first client message
> to
> > indicate delegation token. Broker can then obtain credentials and
> principal
> > using a different code path for delegation tokens.
> >
> > On Tue, Nov 8, 2016 at 6:38 PM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Magnus,
> > >
> > > Thanks for the input. If you don't feel strongly the need to bump up
> the
> > > version of SaslHandshake, we can leave the version unchanged.
> > >
> > > Rajini,
> > >
> > > 11. Yes, we could send the HMAC as the SCRAM password for the
> delegation
> > > token. Do we need something to indicate that this SCRAM token is
> special
> > > (i.e., delegation token) so that we can generate the correct
> > > KafkaPrincipal? The delegation token logic can be added later. I am
> > asking
> > > just so that we have enough in the design of SCRAM to add the
> delegation
> > > token logic later.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Tue, Nov 8, 2016 at 4:42 AM, Rajini Sivaram <
> > > rajinisivaram@googlemail.com
> > > > wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > 10. *s=<salt>* and *i=<iterations>* come from the SCRAM standard
> (they
> > > are
> > > > transferred during SCRAM auth). Scram messages look like (for
> example)
> > > > *r=<nonce>,s=<salt>,i=<iterations>*. StoredKey and ServerKey and not
> > > > transferred in SCRAM messages, so I picked two keys that are unused
> in
> > > > SCRAM.
> > > >
> > > > 11. SCRAM (like DIGEST-MD5 or PLAIN) uses a shared secret/password
> for
> > > > authentication along with a username and an optional
> authorization-id.
> > > > Kafka uses the username as the identity (Kafka principal) for
> > > > authentication and authorization. KIP-48 doesn't mention
> KafkaPrincipal
> > > in
> > > > the section "Authentication using Token", but a delegation token is
> > > > associated with a Kafka principal. Since delegation tokens are
> acquired
> > > on
> > > > behalf of a KafkaPrincipal and the principal is included in the token
> > as
> > > > the token owner, clients authenticating with delegation tokens could
> > use
> > > > the token owner as username and the token HMAC as shared
> > secret/password.
> > > >
> > > > If necessary, any other form of token identifier may be used as
> > username
> > > as
> > > > well as long as it contains sufficient information for the broker to
> > > > retrieve/compute the principal and HMAC for authentication. The
> server
> > > > callback handler can be updated when delegation tokens are
> implemented
> > to
> > > > generate Kafka principal accordingly.
> > > >
> > > >
> > > > On Tue, Nov 8, 2016 at 1:03 AM, Jun Rao <ju...@confluent.io> wrote:
> > > >
> > > > > Hi, Rajini,
> > > > >
> > > > > A couple of other questions on the KIP.
> > > > >
> > > > > 10. For the config values stored in ZK, are those keys (s, t, k, i,
> > > etc)
> > > > > stored under scram-sha-256 standard?
> > > > >
> > > > > 11. Could KIP-48 (delegation token) use this KIP to send delegation
> > > > tokens?
> > > > > In KIP-48, the client sends a HMAC as the delegation token to the
> > > server.
> > > > > Not sure how this gets mapped to the username/password in this KIP.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > > > > rajinisivaram@googlemail.com
> > > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256
> > SASL
> > > > > > mechanisms to Kafka:
> > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 84%3A+Support+SASL+SCRAM+mechanisms
> > > > > >
> > > > > >
> > > > > > Comments and suggestions are welcome.
> > > > > >
> > > > > > Thank you...
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>
--
Regards,
Rajini
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Jun Rao <ju...@confluent.io>.
Hi, Rajini,
That makes sense. Could you document this potential future extension in the
KIP?
Jun
On Tue, Nov 8, 2016 at 11:17 AM, Rajini Sivaram <
rajinisivaram@googlemail.com> wrote:
> Jun,
>
> 11. SCRAM messages have an optional extensions field which is a list of
> key=value pairs. We can add an extension key to the first client message to
> indicate delegation token. Broker can then obtain credentials and principal
> using a different code path for delegation tokens.
>
> On Tue, Nov 8, 2016 at 6:38 PM, Jun Rao <ju...@confluent.io> wrote:
>
> > Magnus,
> >
> > Thanks for the input. If you don't feel strongly the need to bump up the
> > version of SaslHandshake, we can leave the version unchanged.
> >
> > Rajini,
> >
> > 11. Yes, we could send the HMAC as the SCRAM password for the delegation
> > token. Do we need something to indicate that this SCRAM token is special
> > (i.e., delegation token) so that we can generate the correct
> > KafkaPrincipal? The delegation token logic can be added later. I am
> asking
> > just so that we have enough in the design of SCRAM to add the delegation
> > token logic later.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Nov 8, 2016 at 4:42 AM, Rajini Sivaram <
> > rajinisivaram@googlemail.com
> > > wrote:
> >
> > > Hi Jun,
> > >
> > > 10. *s=<salt>* and *i=<iterations>* come from the SCRAM standard (they
> > are
> > > transferred during SCRAM auth). Scram messages look like (for example)
> > > *r=<nonce>,s=<salt>,i=<iterations>*. StoredKey and ServerKey and not
> > > transferred in SCRAM messages, so I picked two keys that are unused in
> > > SCRAM.
> > >
> > > 11. SCRAM (like DIGEST-MD5 or PLAIN) uses a shared secret/password for
> > > authentication along with a username and an optional authorization-id.
> > > Kafka uses the username as the identity (Kafka principal) for
> > > authentication and authorization. KIP-48 doesn't mention KafkaPrincipal
> > in
> > > the section "Authentication using Token", but a delegation token is
> > > associated with a Kafka principal. Since delegation tokens are acquired
> > on
> > > behalf of a KafkaPrincipal and the principal is included in the token
> as
> > > the token owner, clients authenticating with delegation tokens could
> use
> > > the token owner as username and the token HMAC as shared
> secret/password.
> > >
> > > If necessary, any other form of token identifier may be used as
> username
> > as
> > > well as long as it contains sufficient information for the broker to
> > > retrieve/compute the principal and HMAC for authentication. The server
> > > callback handler can be updated when delegation tokens are implemented
> to
> > > generate Kafka principal accordingly.
> > >
> > >
> > > On Tue, Nov 8, 2016 at 1:03 AM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Hi, Rajini,
> > > >
> > > > A couple of other questions on the KIP.
> > > >
> > > > 10. For the config values stored in ZK, are those keys (s, t, k, i,
> > etc)
> > > > stored under scram-sha-256 standard?
> > > >
> > > > 11. Could KIP-48 (delegation token) use this KIP to send delegation
> > > tokens?
> > > > In KIP-48, the client sends a HMAC as the delegation token to the
> > server.
> > > > Not sure how this gets mapped to the username/password in this KIP.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > > > rajinisivaram@googlemail.com
> > > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256
> SASL
> > > > > mechanisms to Kafka:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 84%3A+Support+SASL+SCRAM+mechanisms
> > > > >
> > > > >
> > > > > Comments and suggestions are welcome.
> > > > >
> > > > > Thank you...
> > > > >
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Rajini
> > >
> >
>
>
>
> --
> Regards,
>
> Rajini
>
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Rajini Sivaram <ra...@googlemail.com>.
Jun,
11. SCRAM messages have an optional extensions field which is a list of
key=value pairs. We can add an extension key to the first client message to
indicate delegation token. Broker can then obtain credentials and principal
using a different code path for delegation tokens.
On Tue, Nov 8, 2016 at 6:38 PM, Jun Rao <ju...@confluent.io> wrote:
> Magnus,
>
> Thanks for the input. If you don't feel strongly the need to bump up the
> version of SaslHandshake, we can leave the version unchanged.
>
> Rajini,
>
> 11. Yes, we could send the HMAC as the SCRAM password for the delegation
> token. Do we need something to indicate that this SCRAM token is special
> (i.e., delegation token) so that we can generate the correct
> KafkaPrincipal? The delegation token logic can be added later. I am asking
> just so that we have enough in the design of SCRAM to add the delegation
> token logic later.
>
> Thanks,
>
> Jun
>
>
> On Tue, Nov 8, 2016 at 4:42 AM, Rajini Sivaram <
> rajinisivaram@googlemail.com
> > wrote:
>
> > Hi Jun,
> >
> > 10. *s=<salt>* and *i=<iterations>* come from the SCRAM standard (they
> are
> > transferred during SCRAM auth). Scram messages look like (for example)
> > *r=<nonce>,s=<salt>,i=<iterations>*. StoredKey and ServerKey and not
> > transferred in SCRAM messages, so I picked two keys that are unused in
> > SCRAM.
> >
> > 11. SCRAM (like DIGEST-MD5 or PLAIN) uses a shared secret/password for
> > authentication along with a username and an optional authorization-id.
> > Kafka uses the username as the identity (Kafka principal) for
> > authentication and authorization. KIP-48 doesn't mention KafkaPrincipal
> in
> > the section "Authentication using Token", but a delegation token is
> > associated with a Kafka principal. Since delegation tokens are acquired
> on
> > behalf of a KafkaPrincipal and the principal is included in the token as
> > the token owner, clients authenticating with delegation tokens could use
> > the token owner as username and the token HMAC as shared secret/password.
> >
> > If necessary, any other form of token identifier may be used as username
> as
> > well as long as it contains sufficient information for the broker to
> > retrieve/compute the principal and HMAC for authentication. The server
> > callback handler can be updated when delegation tokens are implemented to
> > generate Kafka principal accordingly.
> >
> >
> > On Tue, Nov 8, 2016 at 1:03 AM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Hi, Rajini,
> > >
> > > A couple of other questions on the KIP.
> > >
> > > 10. For the config values stored in ZK, are those keys (s, t, k, i,
> etc)
> > > stored under scram-sha-256 standard?
> > >
> > > 11. Could KIP-48 (delegation token) use this KIP to send delegation
> > tokens?
> > > In KIP-48, the client sends a HMAC as the delegation token to the
> server.
> > > Not sure how this gets mapped to the username/password in this KIP.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > > rajinisivaram@googlemail.com
> > > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
> > > > mechanisms to Kafka:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 84%3A+Support+SASL+SCRAM+mechanisms
> > > >
> > > >
> > > > Comments and suggestions are welcome.
> > > >
> > > > Thank you...
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>
--
Regards,
Rajini
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Jun Rao <ju...@confluent.io>.
Magnus,
Thanks for the input. If you don't feel strongly the need to bump up the
version of SaslHandshake, we can leave the version unchanged.
Rajini,
11. Yes, we could send the HMAC as the SCRAM password for the delegation
token. Do we need something to indicate that this SCRAM token is special
(i.e., delegation token) so that we can generate the correct
KafkaPrincipal? The delegation token logic can be added later. I am asking
just so that we have enough in the design of SCRAM to add the delegation
token logic later.
Thanks,
Jun
On Tue, Nov 8, 2016 at 4:42 AM, Rajini Sivaram <rajinisivaram@googlemail.com
> wrote:
> Hi Jun,
>
> 10. *s=<salt>* and *i=<iterations>* come from the SCRAM standard (they are
> transferred during SCRAM auth). Scram messages look like (for example)
> *r=<nonce>,s=<salt>,i=<iterations>*. StoredKey and ServerKey and not
> transferred in SCRAM messages, so I picked two keys that are unused in
> SCRAM.
>
> 11. SCRAM (like DIGEST-MD5 or PLAIN) uses a shared secret/password for
> authentication along with a username and an optional authorization-id.
> Kafka uses the username as the identity (Kafka principal) for
> authentication and authorization. KIP-48 doesn't mention KafkaPrincipal in
> the section "Authentication using Token", but a delegation token is
> associated with a Kafka principal. Since delegation tokens are acquired on
> behalf of a KafkaPrincipal and the principal is included in the token as
> the token owner, clients authenticating with delegation tokens could use
> the token owner as username and the token HMAC as shared secret/password.
>
> If necessary, any other form of token identifier may be used as username as
> well as long as it contains sufficient information for the broker to
> retrieve/compute the principal and HMAC for authentication. The server
> callback handler can be updated when delegation tokens are implemented to
> generate Kafka principal accordingly.
>
>
> On Tue, Nov 8, 2016 at 1:03 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > Hi, Rajini,
> >
> > A couple of other questions on the KIP.
> >
> > 10. For the config values stored in ZK, are those keys (s, t, k, i, etc)
> > stored under scram-sha-256 standard?
> >
> > 11. Could KIP-48 (delegation token) use this KIP to send delegation
> tokens?
> > In KIP-48, the client sends a HMAC as the delegation token to the server.
> > Not sure how this gets mapped to the username/password in this KIP.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > rajinisivaram@googlemail.com
> > > wrote:
> >
> > > Hi all,
> > >
> > > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
> > > mechanisms to Kafka:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 84%3A+Support+SASL+SCRAM+mechanisms
> > >
> > >
> > > Comments and suggestions are welcome.
> > >
> > > Thank you...
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> >
>
>
>
> --
> Regards,
>
> Rajini
>
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Magnus Edenhill <ma...@edenhill.se>.
Hey,
I'm probably going to argue against bumping the SaslHandshake version, it
is redundant to the existing SaslHandshakeResponse.enabled_mechanisms field.
With SaslHandshake version bump:
* Client performs ApiVersionRequest
* If SaslHandshake >= v1: use SCRAM
* If SaslHandshake < v1: use some fallback mechanism (PLAIN, GSSAPI, ..)
* Send SaslHandshakeRequest with selected mechanism
* Broker returns Ok, or error if the mechanism is unsupported or disabled.
* On error, use some fallback mechanism. [*]
Without SaslHandshake version bump:
* Client sends SaslHandshake mechanism=SCRAM
* Broker returns Ok, or error if the mechanism is unsupported or disabled.
* On error, use some fallback mechanism [*]
This goes to show that in both scenarios the client will need some logic to
handle an unsupported/disabled mechanism,
so bumping the SaslHandshake version doesn't really help much (unless the
API version reflects the configured authenticator).
[*]: I dont think any client implements auth mechanism fallback and I have
yet to see a real-world use-case for it,
so if the mechanism configured on the client isnt available on the broker
that is typically a terminal error.
And for error reporting to the user:
all it needs to say is that the client's configured mechanism is not
available on the broker, but these enabled_mechanisms are.
My two cents,
Magnus
2016-11-08 13:42 GMT+01:00 Rajini Sivaram <ra...@googlemail.com>:
> Hi Jun,
>
> 10. *s=<salt>* and *i=<iterations>* come from the SCRAM standard (they are
> transferred during SCRAM auth). Scram messages look like (for example)
> *r=<nonce>,s=<salt>,i=<iterations>*. StoredKey and ServerKey and not
> transferred in SCRAM messages, so I picked two keys that are unused in
> SCRAM.
>
> 11. SCRAM (like DIGEST-MD5 or PLAIN) uses a shared secret/password for
> authentication along with a username and an optional authorization-id.
> Kafka uses the username as the identity (Kafka principal) for
> authentication and authorization. KIP-48 doesn't mention KafkaPrincipal in
> the section "Authentication using Token", but a delegation token is
> associated with a Kafka principal. Since delegation tokens are acquired on
> behalf of a KafkaPrincipal and the principal is included in the token as
> the token owner, clients authenticating with delegation tokens could use
> the token owner as username and the token HMAC as shared secret/password.
>
> If necessary, any other form of token identifier may be used as username as
> well as long as it contains sufficient information for the broker to
> retrieve/compute the principal and HMAC for authentication. The server
> callback handler can be updated when delegation tokens are implemented to
> generate Kafka principal accordingly.
>
>
> On Tue, Nov 8, 2016 at 1:03 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > Hi, Rajini,
> >
> > A couple of other questions on the KIP.
> >
> > 10. For the config values stored in ZK, are those keys (s, t, k, i, etc)
> > stored under scram-sha-256 standard?
> >
> > 11. Could KIP-48 (delegation token) use this KIP to send delegation
> tokens?
> > In KIP-48, the client sends a HMAC as the delegation token to the server.
> > Not sure how this gets mapped to the username/password in this KIP.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > rajinisivaram@googlemail.com
> > > wrote:
> >
> > > Hi all,
> > >
> > > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
> > > mechanisms to Kafka:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 84%3A+Support+SASL+SCRAM+mechanisms
> > >
> > >
> > > Comments and suggestions are welcome.
> > >
> > > Thank you...
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> >
>
>
>
> --
> Regards,
>
> Rajini
>
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Rajini Sivaram <ra...@googlemail.com>.
Hi Jun,
10. *s=<salt>* and *i=<iterations>* come from the SCRAM standard (they are
transferred during SCRAM auth). Scram messages look like (for example)
*r=<nonce>,s=<salt>,i=<iterations>*. StoredKey and ServerKey and not
transferred in SCRAM messages, so I picked two keys that are unused in
SCRAM.
11. SCRAM (like DIGEST-MD5 or PLAIN) uses a shared secret/password for
authentication along with a username and an optional authorization-id.
Kafka uses the username as the identity (Kafka principal) for
authentication and authorization. KIP-48 doesn't mention KafkaPrincipal in
the section "Authentication using Token", but a delegation token is
associated with a Kafka principal. Since delegation tokens are acquired on
behalf of a KafkaPrincipal and the principal is included in the token as
the token owner, clients authenticating with delegation tokens could use
the token owner as username and the token HMAC as shared secret/password.
If necessary, any other form of token identifier may be used as username as
well as long as it contains sufficient information for the broker to
retrieve/compute the principal and HMAC for authentication. The server
callback handler can be updated when delegation tokens are implemented to
generate Kafka principal accordingly.
On Tue, Nov 8, 2016 at 1:03 AM, Jun Rao <ju...@confluent.io> wrote:
> Hi, Rajini,
>
> A couple of other questions on the KIP.
>
> 10. For the config values stored in ZK, are those keys (s, t, k, i, etc)
> stored under scram-sha-256 standard?
>
> 11. Could KIP-48 (delegation token) use this KIP to send delegation tokens?
> In KIP-48, the client sends a HMAC as the delegation token to the server.
> Not sure how this gets mapped to the username/password in this KIP.
>
> Thanks,
>
> Jun
>
> On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> rajinisivaram@googlemail.com
> > wrote:
>
> > Hi all,
> >
> > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
> > mechanisms to Kafka:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 84%3A+Support+SASL+SCRAM+mechanisms
> >
> >
> > Comments and suggestions are welcome.
> >
> > Thank you...
> >
> > Regards,
> >
> > Rajini
> >
>
--
Regards,
Rajini
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Jun Rao <ju...@confluent.io>.
Hi, Rajini,
A couple of other questions on the KIP.
10. For the config values stored in ZK, are those keys (s, t, k, i, etc)
stored under scram-sha-256 standard?
11. Could KIP-48 (delegation token) use this KIP to send delegation tokens?
In KIP-48, the client sends a HMAC as the delegation token to the server.
Not sure how this gets mapped to the username/password in this KIP.
Thanks,
Jun
On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <rajinisivaram@googlemail.com
> wrote:
> Hi all,
>
> I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
> mechanisms to Kafka:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 84%3A+Support+SASL+SCRAM+mechanisms
>
>
> Comments and suggestions are welcome.
>
> Thank you...
>
> Regards,
>
> Rajini
>
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Jun Rao <ju...@confluent.io>.
I think Magnus brought up the point of bumping up the version of
SaslHandshakeRequest
during the discussion of KIP-35. A broker may be capable of supporting a
list of SASL mechanisms X and may be enabling a list of SASL mechanisms Y.
Y will be a subset of X. Now, from a client's perspective, it may be useful
to differentiate the two cases. For example, the client could at least give
different error messages. If a broker is not capable of supporting a SASL
mechanism, the error could say "please upgrade the broker to a particular
version". If the broker is capable, but not enabled with a particular SASL
mechanism, the error could just say "please enable this SASL mechanism in
the broker". We will be able to distinguish these two cases if we bump up
the version of SaslHandshakeRequest every time we add the support for a new
SASL mechanism. Not sure how critical this is, but that's my understanding
from the KIP-35 discussion. Perhaps Magnus could chime in.
Ewen,
I am not sure that I get your comment in the brackets. ApiVersionRequest
happens before SaslHandshakeRequest and doesn't need SASL authentication to
be completed.
Thanks,
Jun
On Thu, Nov 3, 2016 at 8:18 AM, Harsha Chintalapani <ka...@harsha.io> wrote:
> Agree with Rajini on not incrementing the protocol version. As brokers are
> returning the list of supported mechanisms don't think it warrants a
> protocol version bump.
>
> Thanks,
> Harsha
>
> On Thu, Nov 3, 2016 at 7:59 AM Rajini Sivaram <
> rajinisivaram@googlemail.com>
> wrote:
>
> > Ismael,
> >
> > 2) I left out MD5 because it is insecure, but thought of keeping SHA-1
> > since it is supported by most services that support SCRAM today. Since
> > there is no actual requirement to support SHA-1 and we don't really want
> to
> > store credentials in Zookeeper using SHA-1, it makes sense not to support
> > SHA-1. Will update the KIP. Thanks.
> >
> > On Thu, Nov 3, 2016 at 1:25 PM, Ismael Juma <is...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > I'm currently with limited internet access so I have not done a proper
> > > review of the KIP, my apologies. A couple of thoughts:
> > >
> > > 1. I agree with Rajini and I don't see the value in bumping the
> protocol
> > > version in this case. As Rajini said, the enabled mechanisms are
> > > independent of the broker version. If and when we have a feature API,
> we
> > > can consider exposing it that way.
> > >
> > > 2. Do we really want to support sha1? The security of it has been
> > > questioned and the general recommendation is to avoid it these days.
> > >
> > > Ismael
> > >
> > > On 3 Nov 2016 6:51 am, "Rajini Sivaram" <ra...@googlemail.com>
> > > wrote:
> > >
> > > > I don't have a strong objection to bumping up SaslHandshakeRequest
> > > version,
> > > > though I am still not convinced of the usefulness. I do agree that
> > KIP-35
> > > > should be standard way to determine client/broker compatibility. But
> I
> > am
> > > > not sure ApiVersionsRequest is a good fit for clients to choose a
> SASL
> > > > mechanism.
> > > >
> > > > 1. SaslHandshakeRequest is the only way a client can figure out
> if a
> > > > SASL mechanism is actually enabled in the broker. The fact that
> > broker
> > > > version n supports SCRAM doesn't imply that a broker of version n
> > > > actually
> > > > has the mechanism enabled. Since enabling a mechanism involves
> some
> > > > effort
> > > > like installing an authentication server and/or configuring
> > > credentials,
> > > > SASL mechanisms are a feature of a broker installation very unlike
> > > > versions. As you said, "features" would have worked better.
> > > > 2. New SASL mechanisms can be added to older Kafka broker
> versions.
> > > With
> > > > some tweaking, the SCRAM implementation from KIP-84 can be enabled
> > in
> > > a
> > > > 0.10.0 broker without changing any broker code. KIP-86 would make
> > this
> > > > even
> > > > easier, but it is already possible to add new or custom mechanisms
> > to
> > > > existing broker versions. A client using ApiVersionsRequest to
> > choose
> > > a
> > > > SASL mechanism is only checking when a mechanism was included in a
> > > > broker
> > > > release, not the "capability" of a broker to support the
> mechanism.
> > I
> > > am
> > > > not sure we should encourage clients to choose mechanisms based on
> > > > versions.
> > > > 3. Clients need additional configuration based on the chosen
> > > mechanism.
> > > > One of the reasons I couldn't see any value in using
> > > ApiVersionsRequest
> > > > in
> > > > the Java client was because clients are configured with a single
> > SASL
> > > > mechanism and a JAAS configuration corresponding to that
> mechanism.
> > > If a
> > > > client wants to choose between Kerberos and SCRAM, the client
> would
> > > need
> > > > keytab/principal for kerberos and username/password for SCRAM.
> > Clients
> > > > that
> > > > possess multiple credentials without knowing what the broker
> > supports
> > > -
> > > > that sounds like a rather unusual scenario.
> > > > 4. For some mechanisms, clients may want to choose mechanism at
> > > runtime.
> > > > For example, a client is configured with username/ password and
> > wants
> > > to
> > > > choose between SCRAM-SHA-1/SCRAM-SHA-256/PLAIN depending on which
> > > > mechanisms are enabled in the broker. To choose between
> SCRAM-SHA-1
> > > and
> > > > SCRAM-SHA-256, clients have to use SaslHandshakeRequest since they
> > > were
> > > > added in the same version. To choose between PLAIN and
> > SCRAM-SHA-256,
> > > a
> > > > versions based approach would work, but wouldn't it be better for
> > > > clients
> > > > to rely on just SaslHandshakeRequest rather than
> > > > ApiVersionsRequest+SaslHandshakeRequest so that the solution
> works
> > in
> > > > all
> > > > scenarios?
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Nov 3, 2016 at 4:33 AM, Ewen Cheslack-Postava <
> > ewen@confluent.io
> > > >
> > > > wrote:
> > > >
> > > > > I think the bump isn't strictly required, but if the client is
> KIP-35
> > > > > aware, it can proactively choose a compatible SASL mechanism based
> on
> > > its
> > > > > initial ApiVersionRequest and avoid an extra connection round trip
> > when
> > > > > there are client/broker version differences. Without this, a newer
> > > client
> > > > > would have to do 2 set of requests since the first SASL mechanism
> > might
> > > > not
> > > > > be compatible.
> > > > >
> > > > > I don't think this is a deal breaker, but I do think it would be
> good
> > > to
> > > > > just standardize on KIP-35 as the way we figure out client/broker
> > > > > compatibility. The SASL stuff happened in parallel (maybe before?)
> > > KIP-35
> > > > > and ended up with its own mechanism, but I'm in favor of trying to
> > > > simplify
> > > > > everything by centralizing those considerations into a single API
> > call.
> > > > (By
> > > > > the way, dredging up now ancient history in the KIP-35 discussion,
> > this
> > > > is
> > > > > also why "features" vs "API version" is relevant. If we wanted to
> > > > configure
> > > > > a newer broker to disable SASL mechanisms we no longer want to
> allow
> > > use
> > > > > of, this isn't really possible to express via API versions unless
> we
> > > also
> > > > > explicitly add an API version that doesn't support that mechanism
> > > whereas
> > > > > features would make this easier to toggle on/off. The
> > > > SaslHandshakeRequest
> > > > > probably makes it easier to keep thing secure compared to the
> current
> > > > state
> > > > > of ApiVersionRequest).
> > > > >
> > > > > -Ewen
> > > > >
> > > > > On Tue, Nov 1, 2016 at 2:09 PM, Rajini Sivaram <
> > > > > rajinisivaram@googlemail.com
> > > > > > wrote:
> > > > >
> > > > > > Gwen,
> > > > > >
> > > > > > I had thought the same too and hence I am assuming that Java
> > clients
> > > > > could
> > > > > > simply use SaslHandshakeRequest. SaslHandshakeRequest returns the
> > > list
> > > > of
> > > > > > mechanisms enabled in the broker. I think Jun's point was that by
> > > > > > incrementing the version of SaslHandshakeRequest, clients can use
> > > > > > ApiVersionsRequest to figure out the mechanisms the broker is
> > capable
> > > > of
> > > > > > supporting and use that information to choose a mechanism to send
> > in
> > > > > > SaslHandshakeRequest. Not sure how useful this actually is, so
> will
> > > > wait
> > > > > > for Jun's response.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Nov 1, 2016 at 8:18 PM, Gwen Shapira <gw...@confluent.io>
> > > > wrote:
> > > > > >
> > > > > > > Wait, I thought SaslHandshakeResponse includes a list of
> > mechanisms
> > > > > > > supported, so I'm not sure why we need to bump the version?
> > > > > > >
> > > > > > > I expect clients will send SaslHandshakeRequest_V0, see which
> > > > > mechanisms
> > > > > > > are supported and make a call based on that? Which means KIP-35
> > is
> > > > not
> > > > > > > required in that case? Am I missing something?
> > > > > > >
> > > > > > > On Tue, Nov 1, 2016 at 1:07 PM, Rajini Sivaram <
> > > > > > > rajinisivaram@googlemail.com
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > Jun,
> > > > > > > >
> > > > > > > > I have added the following text to the KIP. Does this match
> > your
> > > > > > > > expectation?
> > > > > > > >
> > > > > > > > *SaslHandshakeRequest version will be increased from 0 to 1
> so
> > > that
> > > > > > > clients
> > > > > > > > can determine if the broker is capable of supporting SCRAM
> > > > mechanisms
> > > > > > > using
> > > > > > > > ApiVersionsRequest. Java clients will not be updated to use
> > > > > > > > ApiVersionsRequest to choose SASL mechanism under this KIP.
> > Java
> > > > > > clients
> > > > > > > > will continue to use their configured SASL mechanism and will
> > > fail
> > > > > > > > connection if the requested mechanism is not enabled in the
> > > > broker.*
> > > > > > > >
> > > > > > > > Thank you,
> > > > > > > >
> > > > > > > > Rajini
> > > > > > > >
> > > > > > > > On Tue, Nov 1, 2016 at 4:54 PM, Jun Rao <ju...@confluent.io>
> > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Rajini,
> > > > > > > > >
> > > > > > > > > One more thing. It seems that we should bump up the version
> > of
> > > > > > > > > SaslHandshakeRequest? This way, the client can figure out
> > which
> > > > > SASL
> > > > > > > > > mechanisms the broker is capable of supporting through
> > > > > > > ApiVersionRequest.
> > > > > > > > > We discussed this briefly as part of KIP-43.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Nov 1, 2016 at 7:41 AM, Rajini Sivaram <
> > > > > > > > > rajinisivaram@googlemail.com
> > > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > If there are no more comments, I will start vote on this
> > KIP
> > > > > later
> > > > > > > this
> > > > > > > > > > week. In the meantime, please feel free to post any
> > feedback
> > > or
> > > > > > > > > > suggestions. Initial implementation is here:
> > > > > > > > > > https://github.com/apache/kafka/pull/2086.
> > > > > > > > > >
> > > > > > > > > > Thank you,
> > > > > > > > > >
> > > > > > > > > > Rajini
> > > > > > > > > >
> > > > > > > > > > On Thu, Oct 27, 2016 at 11:18 AM, Rajini Sivaram <
> > > > > > > > > > rajinisivaram@googlemail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Jun,
> > > > > > > > > > >
> > > > > > > > > > > 4) Agree, it does make the implementation simpler.
> > Updated
> > > > KIP.
> > > > > > > > > > > 5) Thank you, that looks neater. Updated KIP.
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Oct 26, 2016 at 6:59 PM, Jun Rao <
> > jun@confluent.io
> > > >
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi, Rajini,
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks for the reply.
> > > > > > > > > > >>
> > > > > > > > > > >> 4. Implementation wise, it seems to me that it's
> simpler
> > > to
> > > > > read
> > > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > >> cache than reading directly from ZK since the config
> > > manager
> > > > > > > already
> > > > > > > > > > >> propagates all config changes through ZK. Also, it's
> > > > probably
> > > > > a
> > > > > > > good
> > > > > > > > > > idea
> > > > > > > > > > >> to limit the places in the code base that directly
> > > accesses
> > > > > ZK.
> > > > > > > > > > >>
> > > > > > > > > > >> 5. Yes, it seems that it makes sense to add the new
> > SCRAM
> > > > > > > > > configurations
> > > > > > > > > > >> to
> > > > > > > > > > >> the existing /config/users/<encoded-user>. I am not
> sure
> > > how
> > > > > one
> > > > > > > > would
> > > > > > > > > > >> remove the SCRAM configurations in the example though
> > > since
> > > > > the
> > > > > > > > > > properties
> > > > > > > > > > >> specified in add-config is not the ones actually being
> > > > stored.
> > > > > > An
> > > > > > > > > > >> alternative is to doing sth like the following. It may
> > > still
> > > > > > feel
> > > > > > > a
> > > > > > > > > bit
> > > > > > > > > > >> weird and I am wondering if there is a clearer
> approach.
> > > > > > > > > > >>
> > > > > > > > > > >> bin/kafka-configs.sh --zookeeper localhost:2181
> --alter
> > > > > > > --add-config
> > > > > > > > > > >> 'scram_sha-256=[password=
> alice-secret,iterations=4096],
> > > > > > > scram_sha-1=
> > > > > > > > > > >> [password=alice-secret,iterations=4096]'
> --entity-type
> > > > users
> > > > > > > > > > >> --entity-name
> > > > > > > > > > >> alice
> > > > > > > > > > >>
> > > > > > > > > > >> bin/kafka-configs.sh --zookeeper localhost:2181
> --alter
> > > > > > > > > --delete-config
> > > > > > > > > > >> 'scram_sha-256,scram_sha-1' --entity-type users
> > > > --entity-name
> > > > > > > alice
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks,
> > > > > > > > > > >>
> > > > > > > > > > >> Jun
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> On Wed, Oct 26, 2016 at 4:35 AM, Rajini Sivaram <
> > > > > > > > > > >> rajinisivaram@googlemail.com> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > Hi Jun,
> > > > > > > > > > >> >
> > > > > > > > > > >> > Thank you for reviewing the KIP. Answers below:
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > 1. Yes, agree, Updated KIP.
> > > > > > > > > > >> > 2. User specifies a password and iteration count.
> > > > > > > > kaka-configs.sh
> > > > > > > > > > >> > generates a random salt and then generates
> > StoredKey
> > > > and
> > > > > > > > > ServerKey
> > > > > > > > > > >> for
> > > > > > > > > > >> > that
> > > > > > > > > > >> > password using the same message formatter
> > > > implementation
> > > > > > used
> > > > > > > > for
> > > > > > > > > > >> SCRAM
> > > > > > > > > > >> > authentication. I have added some more detail to
> > the
> > > > KIP
> > > > > (
> > > > > > > > > > >> > https://cwiki.apache.org/
> > > confluence/display/KAFKA/KIP-
> > > > > > > > > > >> > 84%3A+Support+SASL+SCRAM+mechanisms#KIP-84:
> > > > > > > > > > SupportSASLSCRAMmechanisms-
> > > > > > > > > > >> > Tools).
> > > > > > > > > > >> > Does that answer the question?
> > > > > > > > > > >> > 3. I started off thinking just one
> (SCRAM-SHA-256)
> > > and
> > > > > then
> > > > > > > > > thought
> > > > > > > > > > >> > another one is required to make sure that the
> > > > > > implementation
> > > > > > > > can
> > > > > > > > > > cope
> > > > > > > > > > >> > with
> > > > > > > > > > >> > multiple SCRAM mechanisms. But actually you are
> > > right,
> > > > we
> > > > > > can
> > > > > > > > > > support
> > > > > > > > > > >> > all.
> > > > > > > > > > >> > I haven't added the old md2/md5 mechanisms that
> > > aren't
> > > > > very
> > > > > > > > > secure,
> > > > > > > > > > >> but
> > > > > > > > > > >> > I
> > > > > > > > > > >> > have included all the SHA algorithms.
> > > > > > > > > > >> > 4. Since credentials are only required when a
> > > > connection
> > > > > is
> > > > > > > > made,
> > > > > > > > > > it
> > > > > > > > > > >> > feels like we can just read the latest value from
> > ZK
> > > > > rather
> > > > > > > > than
> > > > > > > > > > >> cache
> > > > > > > > > > >> > all
> > > > > > > > > > >> > users and keep them updated. Having said that, we
> > can
> > > > > > always
> > > > > > > > add
> > > > > > > > > > >> caching
> > > > > > > > > > >> > later if we find that the overhead of reading
> from
> > ZK
> > > > > every
> > > > > > > > time
> > > > > > > > > is
> > > > > > > > > > >> too
> > > > > > > > > > >> > expensive. Since caching doesn't change any
> > > externals,
> > > > > this
> > > > > > > can
> > > > > > > > > be
> > > > > > > > > > >> done
> > > > > > > > > > >> > in
> > > > > > > > > > >> > a JIRA later - would that be ok?
> > > > > > > > > > >> > 5. Thanks, updated. I have changed the property
> > names
> > > > to
> > > > > > > > include
> > > > > > > > > > >> > mechanism. To avoid four separate properties per
> > > > > mechanism
> > > > > > in
> > > > > > > > > ZK, I
> > > > > > > > > > >> have
> > > > > > > > > > >> > changed the format to use a single property
> > > (lower-case
> > > > > > > > mechanism
> > > > > > > > > > >> name)
> > > > > > > > > > >> > with four values concatenated in a format similar
> > to
> > > > > SCRAM
> > > > > > > > > > messages.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Do you think storing SCRAM credentials in
> > > > > > > > > /config/users/<encoded-user>
> > > > > > > > > > >> > along with existing quota properties is okay? Or
> > should
> > > > they
> > > > > > be
> > > > > > > > > under
> > > > > > > > > > a
> > > > > > > > > > >> > different path (eg. /config/credentials/<encoded-
> > > user>)?
> > > > Or
> > > > > > > > should
> > > > > > > > > > >> they be
> > > > > > > > > > >> > on a completely different path like ACLs with a
> > separate
> > > > > tool
> > > > > > > > > instead
> > > > > > > > > > of
> > > > > > > > > > >> > reusing kaka-configs.sh?
> > > > > > > > > > >> >
> > > > > > > > > > >> > Thank you,
> > > > > > > > > > >> >
> > > > > > > > > > >> > Rajini
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Tue, Oct 25, 2016 at 11:55 PM, Jun Rao <
> > > > jun@confluent.io
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > > Hi, Rajini,
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Thanks for the proposal. Looks good overall and
> > seems
> > > > > quite
> > > > > > > > useful
> > > > > > > > > > >> (e.g.
> > > > > > > > > > >> > > for supporting delegation tokens). A few
> > > > > comments/questions
> > > > > > > > below.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > 1. For the ZK data format change, should we use
> the
> > > same
> > > > > > > > > convention
> > > > > > > > > > >> as in
> > > > > > > > > > >> > > KIP-55 to use encoded user name (i.e.,
> > > > > > > > > > /config/users/<encoded-user1>)
> > > > > > > > > > >> ?
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > 2. For tooling, could you describe how user
> > typically
> > > > > > > generates
> > > > > > > > > > >> > > scam_server_key and scram_stored_key to be used by
> > > > > > > > > kafka-config.sh?
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > 3. Is there a particular reason to only support
> sha1
> > > and
> > > > > > > sha128?
> > > > > > > > > > >> Should
> > > > > > > > > > >> > we
> > > > > > > > > > >> > > support more hashes listed below in the future?
> > > > > > > > > > >> > > http://www.iana.org/assignments/hash-function-
> > > > > > > > > > >> > > text-names/hash-function-text-names.xhtml
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > 4. Is there a reason not to cache user credentials
> > in
> > > > the
> > > > > > > > broker?
> > > > > > > > > > The
> > > > > > > > > > >> > > dynamic config mechanism already supports loading
> > > > configs
> > > > > > into
> > > > > > > > > > >> broker's
> > > > > > > > > > >> > > cache. Checking credentials from broker's cache is
> > > more
> > > > > > > > efficient
> > > > > > > > > > than
> > > > > > > > > > >> > > reading from ZK each time.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > 5. Typo "scram_iteration-4096" (= instead of -).
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Thanks,
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Jun
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > > > > > > > > > >> > > rajinisivaram@googlemail.com
> > > > > > > > > > >> > > > wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > Hi all,
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > I have just created KIP-84 to add SCRAM-SHA-1
> and
> > > > > > > > SCRAM-SHA-256
> > > > > > > > > > SASL
> > > > > > > > > > >> > > > mechanisms to Kafka:
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > https://cwiki.apache.org/
> > > > confluence/display/KAFKA/KIP-
> > > > > > > > > > >> > > > 84%3A+Support+SASL+SCRAM+mechanisms
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Comments and suggestions are welcome.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Thank you...
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Regards,
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Rajini
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > --
> > > > > > > > > > >> > Regards,
> > > > > > > > > > >> >
> > > > > > > > > > >> > Rajini
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Regards,
> > > > > > > > > > >
> > > > > > > > > > > Rajini
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Regards,
> > > > > > > > > >
> > > > > > > > > > Rajini
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Rajini
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Gwen Shapira*
> > > > > > > Product Manager | Confluent
> > > > > > > 650.450.2760 <(650)%20450-2760> | @gwenshap
> > > > > > > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> > > > > > > <http://www.confluent.io/blog>
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks,
> > > > > Ewen
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Harsha Chintalapani <ka...@harsha.io>.
Agree with Rajini on not incrementing the protocol version. As brokers are
returning the list of supported mechanisms don't think it warrants a
protocol version bump.
Thanks,
Harsha
On Thu, Nov 3, 2016 at 7:59 AM Rajini Sivaram <ra...@googlemail.com>
wrote:
> Ismael,
>
> 2) I left out MD5 because it is insecure, but thought of keeping SHA-1
> since it is supported by most services that support SCRAM today. Since
> there is no actual requirement to support SHA-1 and we don't really want to
> store credentials in Zookeeper using SHA-1, it makes sense not to support
> SHA-1. Will update the KIP. Thanks.
>
> On Thu, Nov 3, 2016 at 1:25 PM, Ismael Juma <is...@gmail.com> wrote:
>
> > Hi all,
> >
> > I'm currently with limited internet access so I have not done a proper
> > review of the KIP, my apologies. A couple of thoughts:
> >
> > 1. I agree with Rajini and I don't see the value in bumping the protocol
> > version in this case. As Rajini said, the enabled mechanisms are
> > independent of the broker version. If and when we have a feature API, we
> > can consider exposing it that way.
> >
> > 2. Do we really want to support sha1? The security of it has been
> > questioned and the general recommendation is to avoid it these days.
> >
> > Ismael
> >
> > On 3 Nov 2016 6:51 am, "Rajini Sivaram" <ra...@googlemail.com>
> > wrote:
> >
> > > I don't have a strong objection to bumping up SaslHandshakeRequest
> > version,
> > > though I am still not convinced of the usefulness. I do agree that
> KIP-35
> > > should be standard way to determine client/broker compatibility. But I
> am
> > > not sure ApiVersionsRequest is a good fit for clients to choose a SASL
> > > mechanism.
> > >
> > > 1. SaslHandshakeRequest is the only way a client can figure out if a
> > > SASL mechanism is actually enabled in the broker. The fact that
> broker
> > > version n supports SCRAM doesn't imply that a broker of version n
> > > actually
> > > has the mechanism enabled. Since enabling a mechanism involves some
> > > effort
> > > like installing an authentication server and/or configuring
> > credentials,
> > > SASL mechanisms are a feature of a broker installation very unlike
> > > versions. As you said, "features" would have worked better.
> > > 2. New SASL mechanisms can be added to older Kafka broker versions.
> > With
> > > some tweaking, the SCRAM implementation from KIP-84 can be enabled
> in
> > a
> > > 0.10.0 broker without changing any broker code. KIP-86 would make
> this
> > > even
> > > easier, but it is already possible to add new or custom mechanisms
> to
> > > existing broker versions. A client using ApiVersionsRequest to
> choose
> > a
> > > SASL mechanism is only checking when a mechanism was included in a
> > > broker
> > > release, not the "capability" of a broker to support the mechanism.
> I
> > am
> > > not sure we should encourage clients to choose mechanisms based on
> > > versions.
> > > 3. Clients need additional configuration based on the chosen
> > mechanism.
> > > One of the reasons I couldn't see any value in using
> > ApiVersionsRequest
> > > in
> > > the Java client was because clients are configured with a single
> SASL
> > > mechanism and a JAAS configuration corresponding to that mechanism.
> > If a
> > > client wants to choose between Kerberos and SCRAM, the client would
> > need
> > > keytab/principal for kerberos and username/password for SCRAM.
> Clients
> > > that
> > > possess multiple credentials without knowing what the broker
> supports
> > -
> > > that sounds like a rather unusual scenario.
> > > 4. For some mechanisms, clients may want to choose mechanism at
> > runtime.
> > > For example, a client is configured with username/ password and
> wants
> > to
> > > choose between SCRAM-SHA-1/SCRAM-SHA-256/PLAIN depending on which
> > > mechanisms are enabled in the broker. To choose between SCRAM-SHA-1
> > and
> > > SCRAM-SHA-256, clients have to use SaslHandshakeRequest since they
> > were
> > > added in the same version. To choose between PLAIN and
> SCRAM-SHA-256,
> > a
> > > versions based approach would work, but wouldn't it be better for
> > > clients
> > > to rely on just SaslHandshakeRequest rather than
> > > ApiVersionsRequest+SaslHandshakeRequest so that the solution works
> in
> > > all
> > > scenarios?
> > >
> > >
> > >
> > >
> > > On Thu, Nov 3, 2016 at 4:33 AM, Ewen Cheslack-Postava <
> ewen@confluent.io
> > >
> > > wrote:
> > >
> > > > I think the bump isn't strictly required, but if the client is KIP-35
> > > > aware, it can proactively choose a compatible SASL mechanism based on
> > its
> > > > initial ApiVersionRequest and avoid an extra connection round trip
> when
> > > > there are client/broker version differences. Without this, a newer
> > client
> > > > would have to do 2 set of requests since the first SASL mechanism
> might
> > > not
> > > > be compatible.
> > > >
> > > > I don't think this is a deal breaker, but I do think it would be good
> > to
> > > > just standardize on KIP-35 as the way we figure out client/broker
> > > > compatibility. The SASL stuff happened in parallel (maybe before?)
> > KIP-35
> > > > and ended up with its own mechanism, but I'm in favor of trying to
> > > simplify
> > > > everything by centralizing those considerations into a single API
> call.
> > > (By
> > > > the way, dredging up now ancient history in the KIP-35 discussion,
> this
> > > is
> > > > also why "features" vs "API version" is relevant. If we wanted to
> > > configure
> > > > a newer broker to disable SASL mechanisms we no longer want to allow
> > use
> > > > of, this isn't really possible to express via API versions unless we
> > also
> > > > explicitly add an API version that doesn't support that mechanism
> > whereas
> > > > features would make this easier to toggle on/off. The
> > > SaslHandshakeRequest
> > > > probably makes it easier to keep thing secure compared to the current
> > > state
> > > > of ApiVersionRequest).
> > > >
> > > > -Ewen
> > > >
> > > > On Tue, Nov 1, 2016 at 2:09 PM, Rajini Sivaram <
> > > > rajinisivaram@googlemail.com
> > > > > wrote:
> > > >
> > > > > Gwen,
> > > > >
> > > > > I had thought the same too and hence I am assuming that Java
> clients
> > > > could
> > > > > simply use SaslHandshakeRequest. SaslHandshakeRequest returns the
> > list
> > > of
> > > > > mechanisms enabled in the broker. I think Jun's point was that by
> > > > > incrementing the version of SaslHandshakeRequest, clients can use
> > > > > ApiVersionsRequest to figure out the mechanisms the broker is
> capable
> > > of
> > > > > supporting and use that information to choose a mechanism to send
> in
> > > > > SaslHandshakeRequest. Not sure how useful this actually is, so will
> > > wait
> > > > > for Jun's response.
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Nov 1, 2016 at 8:18 PM, Gwen Shapira <gw...@confluent.io>
> > > wrote:
> > > > >
> > > > > > Wait, I thought SaslHandshakeResponse includes a list of
> mechanisms
> > > > > > supported, so I'm not sure why we need to bump the version?
> > > > > >
> > > > > > I expect clients will send SaslHandshakeRequest_V0, see which
> > > > mechanisms
> > > > > > are supported and make a call based on that? Which means KIP-35
> is
> > > not
> > > > > > required in that case? Am I missing something?
> > > > > >
> > > > > > On Tue, Nov 1, 2016 at 1:07 PM, Rajini Sivaram <
> > > > > > rajinisivaram@googlemail.com
> > > > > > > wrote:
> > > > > >
> > > > > > > Jun,
> > > > > > >
> > > > > > > I have added the following text to the KIP. Does this match
> your
> > > > > > > expectation?
> > > > > > >
> > > > > > > *SaslHandshakeRequest version will be increased from 0 to 1 so
> > that
> > > > > > clients
> > > > > > > can determine if the broker is capable of supporting SCRAM
> > > mechanisms
> > > > > > using
> > > > > > > ApiVersionsRequest. Java clients will not be updated to use
> > > > > > > ApiVersionsRequest to choose SASL mechanism under this KIP.
> Java
> > > > > clients
> > > > > > > will continue to use their configured SASL mechanism and will
> > fail
> > > > > > > connection if the requested mechanism is not enabled in the
> > > broker.*
> > > > > > >
> > > > > > > Thank you,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > > > On Tue, Nov 1, 2016 at 4:54 PM, Jun Rao <ju...@confluent.io>
> > wrote:
> > > > > > >
> > > > > > > > Hi, Rajini,
> > > > > > > >
> > > > > > > > One more thing. It seems that we should bump up the version
> of
> > > > > > > > SaslHandshakeRequest? This way, the client can figure out
> which
> > > > SASL
> > > > > > > > mechanisms the broker is capable of supporting through
> > > > > > ApiVersionRequest.
> > > > > > > > We discussed this briefly as part of KIP-43.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Nov 1, 2016 at 7:41 AM, Rajini Sivaram <
> > > > > > > > rajinisivaram@googlemail.com
> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > If there are no more comments, I will start vote on this
> KIP
> > > > later
> > > > > > this
> > > > > > > > > week. In the meantime, please feel free to post any
> feedback
> > or
> > > > > > > > > suggestions. Initial implementation is here:
> > > > > > > > > https://github.com/apache/kafka/pull/2086.
> > > > > > > > >
> > > > > > > > > Thank you,
> > > > > > > > >
> > > > > > > > > Rajini
> > > > > > > > >
> > > > > > > > > On Thu, Oct 27, 2016 at 11:18 AM, Rajini Sivaram <
> > > > > > > > > rajinisivaram@googlemail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Jun,
> > > > > > > > > >
> > > > > > > > > > 4) Agree, it does make the implementation simpler.
> Updated
> > > KIP.
> > > > > > > > > > 5) Thank you, that looks neater. Updated KIP.
> > > > > > > > > >
> > > > > > > > > > On Wed, Oct 26, 2016 at 6:59 PM, Jun Rao <
> jun@confluent.io
> > >
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi, Rajini,
> > > > > > > > > >>
> > > > > > > > > >> Thanks for the reply.
> > > > > > > > > >>
> > > > > > > > > >> 4. Implementation wise, it seems to me that it's simpler
> > to
> > > > read
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > >> cache than reading directly from ZK since the config
> > manager
> > > > > > already
> > > > > > > > > >> propagates all config changes through ZK. Also, it's
> > > probably
> > > > a
> > > > > > good
> > > > > > > > > idea
> > > > > > > > > >> to limit the places in the code base that directly
> > accesses
> > > > ZK.
> > > > > > > > > >>
> > > > > > > > > >> 5. Yes, it seems that it makes sense to add the new
> SCRAM
> > > > > > > > configurations
> > > > > > > > > >> to
> > > > > > > > > >> the existing /config/users/<encoded-user>. I am not sure
> > how
> > > > one
> > > > > > > would
> > > > > > > > > >> remove the SCRAM configurations in the example though
> > since
> > > > the
> > > > > > > > > properties
> > > > > > > > > >> specified in add-config is not the ones actually being
> > > stored.
> > > > > An
> > > > > > > > > >> alternative is to doing sth like the following. It may
> > still
> > > > > feel
> > > > > > a
> > > > > > > > bit
> > > > > > > > > >> weird and I am wondering if there is a clearer approach.
> > > > > > > > > >>
> > > > > > > > > >> bin/kafka-configs.sh --zookeeper localhost:2181 --alter
> > > > > > --add-config
> > > > > > > > > >> 'scram_sha-256=[password=alice-secret,iterations=4096],
> > > > > > scram_sha-1=
> > > > > > > > > >> [password=alice-secret,iterations=4096]' --entity-type
> > > users
> > > > > > > > > >> --entity-name
> > > > > > > > > >> alice
> > > > > > > > > >>
> > > > > > > > > >> bin/kafka-configs.sh --zookeeper localhost:2181 --alter
> > > > > > > > --delete-config
> > > > > > > > > >> 'scram_sha-256,scram_sha-1' --entity-type users
> > > --entity-name
> > > > > > alice
> > > > > > > > > >>
> > > > > > > > > >> Thanks,
> > > > > > > > > >>
> > > > > > > > > >> Jun
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> On Wed, Oct 26, 2016 at 4:35 AM, Rajini Sivaram <
> > > > > > > > > >> rajinisivaram@googlemail.com> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Hi Jun,
> > > > > > > > > >> >
> > > > > > > > > >> > Thank you for reviewing the KIP. Answers below:
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > 1. Yes, agree, Updated KIP.
> > > > > > > > > >> > 2. User specifies a password and iteration count.
> > > > > > > kaka-configs.sh
> > > > > > > > > >> > generates a random salt and then generates
> StoredKey
> > > and
> > > > > > > > ServerKey
> > > > > > > > > >> for
> > > > > > > > > >> > that
> > > > > > > > > >> > password using the same message formatter
> > > implementation
> > > > > used
> > > > > > > for
> > > > > > > > > >> SCRAM
> > > > > > > > > >> > authentication. I have added some more detail to
> the
> > > KIP
> > > > (
> > > > > > > > > >> > https://cwiki.apache.org/
> > confluence/display/KAFKA/KIP-
> > > > > > > > > >> > 84%3A+Support+SASL+SCRAM+mechanisms#KIP-84:
> > > > > > > > > SupportSASLSCRAMmechanisms-
> > > > > > > > > >> > Tools).
> > > > > > > > > >> > Does that answer the question?
> > > > > > > > > >> > 3. I started off thinking just one (SCRAM-SHA-256)
> > and
> > > > then
> > > > > > > > thought
> > > > > > > > > >> > another one is required to make sure that the
> > > > > implementation
> > > > > > > can
> > > > > > > > > cope
> > > > > > > > > >> > with
> > > > > > > > > >> > multiple SCRAM mechanisms. But actually you are
> > right,
> > > we
> > > > > can
> > > > > > > > > support
> > > > > > > > > >> > all.
> > > > > > > > > >> > I haven't added the old md2/md5 mechanisms that
> > aren't
> > > > very
> > > > > > > > secure,
> > > > > > > > > >> but
> > > > > > > > > >> > I
> > > > > > > > > >> > have included all the SHA algorithms.
> > > > > > > > > >> > 4. Since credentials are only required when a
> > > connection
> > > > is
> > > > > > > made,
> > > > > > > > > it
> > > > > > > > > >> > feels like we can just read the latest value from
> ZK
> > > > rather
> > > > > > > than
> > > > > > > > > >> cache
> > > > > > > > > >> > all
> > > > > > > > > >> > users and keep them updated. Having said that, we
> can
> > > > > always
> > > > > > > add
> > > > > > > > > >> caching
> > > > > > > > > >> > later if we find that the overhead of reading from
> ZK
> > > > every
> > > > > > > time
> > > > > > > > is
> > > > > > > > > >> too
> > > > > > > > > >> > expensive. Since caching doesn't change any
> > externals,
> > > > this
> > > > > > can
> > > > > > > > be
> > > > > > > > > >> done
> > > > > > > > > >> > in
> > > > > > > > > >> > a JIRA later - would that be ok?
> > > > > > > > > >> > 5. Thanks, updated. I have changed the property
> names
> > > to
> > > > > > > include
> > > > > > > > > >> > mechanism. To avoid four separate properties per
> > > > mechanism
> > > > > in
> > > > > > > > ZK, I
> > > > > > > > > >> have
> > > > > > > > > >> > changed the format to use a single property
> > (lower-case
> > > > > > > mechanism
> > > > > > > > > >> name)
> > > > > > > > > >> > with four values concatenated in a format similar
> to
> > > > SCRAM
> > > > > > > > > messages.
> > > > > > > > > >> >
> > > > > > > > > >> > Do you think storing SCRAM credentials in
> > > > > > > > /config/users/<encoded-user>
> > > > > > > > > >> > along with existing quota properties is okay? Or
> should
> > > they
> > > > > be
> > > > > > > > under
> > > > > > > > > a
> > > > > > > > > >> > different path (eg. /config/credentials/<encoded-
> > user>)?
> > > Or
> > > > > > > should
> > > > > > > > > >> they be
> > > > > > > > > >> > on a completely different path like ACLs with a
> separate
> > > > tool
> > > > > > > > instead
> > > > > > > > > of
> > > > > > > > > >> > reusing kaka-configs.sh?
> > > > > > > > > >> >
> > > > > > > > > >> > Thank you,
> > > > > > > > > >> >
> > > > > > > > > >> > Rajini
> > > > > > > > > >> >
> > > > > > > > > >> > On Tue, Oct 25, 2016 at 11:55 PM, Jun Rao <
> > > jun@confluent.io
> > > > >
> > > > > > > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > > Hi, Rajini,
> > > > > > > > > >> > >
> > > > > > > > > >> > > Thanks for the proposal. Looks good overall and
> seems
> > > > quite
> > > > > > > useful
> > > > > > > > > >> (e.g.
> > > > > > > > > >> > > for supporting delegation tokens). A few
> > > > comments/questions
> > > > > > > below.
> > > > > > > > > >> > >
> > > > > > > > > >> > > 1. For the ZK data format change, should we use the
> > same
> > > > > > > > convention
> > > > > > > > > >> as in
> > > > > > > > > >> > > KIP-55 to use encoded user name (i.e.,
> > > > > > > > > /config/users/<encoded-user1>)
> > > > > > > > > >> ?
> > > > > > > > > >> > >
> > > > > > > > > >> > > 2. For tooling, could you describe how user
> typically
> > > > > > generates
> > > > > > > > > >> > > scam_server_key and scram_stored_key to be used by
> > > > > > > > kafka-config.sh?
> > > > > > > > > >> > >
> > > > > > > > > >> > > 3. Is there a particular reason to only support sha1
> > and
> > > > > > sha128?
> > > > > > > > > >> Should
> > > > > > > > > >> > we
> > > > > > > > > >> > > support more hashes listed below in the future?
> > > > > > > > > >> > > http://www.iana.org/assignments/hash-function-
> > > > > > > > > >> > > text-names/hash-function-text-names.xhtml
> > > > > > > > > >> > >
> > > > > > > > > >> > > 4. Is there a reason not to cache user credentials
> in
> > > the
> > > > > > > broker?
> > > > > > > > > The
> > > > > > > > > >> > > dynamic config mechanism already supports loading
> > > configs
> > > > > into
> > > > > > > > > >> broker's
> > > > > > > > > >> > > cache. Checking credentials from broker's cache is
> > more
> > > > > > > efficient
> > > > > > > > > than
> > > > > > > > > >> > > reading from ZK each time.
> > > > > > > > > >> > >
> > > > > > > > > >> > > 5. Typo "scram_iteration-4096" (= instead of -).
> > > > > > > > > >> > >
> > > > > > > > > >> > > Thanks,
> > > > > > > > > >> > >
> > > > > > > > > >> > > Jun
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > > > > > > > > >> > > rajinisivaram@googlemail.com
> > > > > > > > > >> > > > wrote:
> > > > > > > > > >> > >
> > > > > > > > > >> > > > Hi all,
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > I have just created KIP-84 to add SCRAM-SHA-1 and
> > > > > > > SCRAM-SHA-256
> > > > > > > > > SASL
> > > > > > > > > >> > > > mechanisms to Kafka:
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > https://cwiki.apache.org/
> > > confluence/display/KAFKA/KIP-
> > > > > > > > > >> > > > 84%3A+Support+SASL+SCRAM+mechanisms
> > > > > > > > > >> > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Comments and suggestions are welcome.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Thank you...
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Regards,
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Rajini
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > --
> > > > > > > > > >> > Regards,
> > > > > > > > > >> >
> > > > > > > > > >> > Rajini
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Regards,
> > > > > > > > > >
> > > > > > > > > > Rajini
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Regards,
> > > > > > > > >
> > > > > > > > > Rajini
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Gwen Shapira*
> > > > > > Product Manager | Confluent
> > > > > > 650.450.2760 <(650)%20450-2760> | @gwenshap
> > > > > > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> > > > > > <http://www.confluent.io/blog>
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Ewen
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Rajini
> > >
> >
>
>
>
> --
> Regards,
>
> Rajini
>
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Rajini Sivaram <ra...@googlemail.com>.
Ismael,
2) I left out MD5 because it is insecure, but thought of keeping SHA-1
since it is supported by most services that support SCRAM today. Since
there is no actual requirement to support SHA-1 and we don't really want to
store credentials in Zookeeper using SHA-1, it makes sense not to support
SHA-1. Will update the KIP. Thanks.
On Thu, Nov 3, 2016 at 1:25 PM, Ismael Juma <is...@gmail.com> wrote:
> Hi all,
>
> I'm currently with limited internet access so I have not done a proper
> review of the KIP, my apologies. A couple of thoughts:
>
> 1. I agree with Rajini and I don't see the value in bumping the protocol
> version in this case. As Rajini said, the enabled mechanisms are
> independent of the broker version. If and when we have a feature API, we
> can consider exposing it that way.
>
> 2. Do we really want to support sha1? The security of it has been
> questioned and the general recommendation is to avoid it these days.
>
> Ismael
>
> On 3 Nov 2016 6:51 am, "Rajini Sivaram" <ra...@googlemail.com>
> wrote:
>
> > I don't have a strong objection to bumping up SaslHandshakeRequest
> version,
> > though I am still not convinced of the usefulness. I do agree that KIP-35
> > should be standard way to determine client/broker compatibility. But I am
> > not sure ApiVersionsRequest is a good fit for clients to choose a SASL
> > mechanism.
> >
> > 1. SaslHandshakeRequest is the only way a client can figure out if a
> > SASL mechanism is actually enabled in the broker. The fact that broker
> > version n supports SCRAM doesn't imply that a broker of version n
> > actually
> > has the mechanism enabled. Since enabling a mechanism involves some
> > effort
> > like installing an authentication server and/or configuring
> credentials,
> > SASL mechanisms are a feature of a broker installation very unlike
> > versions. As you said, "features" would have worked better.
> > 2. New SASL mechanisms can be added to older Kafka broker versions.
> With
> > some tweaking, the SCRAM implementation from KIP-84 can be enabled in
> a
> > 0.10.0 broker without changing any broker code. KIP-86 would make this
> > even
> > easier, but it is already possible to add new or custom mechanisms to
> > existing broker versions. A client using ApiVersionsRequest to choose
> a
> > SASL mechanism is only checking when a mechanism was included in a
> > broker
> > release, not the "capability" of a broker to support the mechanism. I
> am
> > not sure we should encourage clients to choose mechanisms based on
> > versions.
> > 3. Clients need additional configuration based on the chosen
> mechanism.
> > One of the reasons I couldn't see any value in using
> ApiVersionsRequest
> > in
> > the Java client was because clients are configured with a single SASL
> > mechanism and a JAAS configuration corresponding to that mechanism.
> If a
> > client wants to choose between Kerberos and SCRAM, the client would
> need
> > keytab/principal for kerberos and username/password for SCRAM. Clients
> > that
> > possess multiple credentials without knowing what the broker supports
> -
> > that sounds like a rather unusual scenario.
> > 4. For some mechanisms, clients may want to choose mechanism at
> runtime.
> > For example, a client is configured with username/ password and wants
> to
> > choose between SCRAM-SHA-1/SCRAM-SHA-256/PLAIN depending on which
> > mechanisms are enabled in the broker. To choose between SCRAM-SHA-1
> and
> > SCRAM-SHA-256, clients have to use SaslHandshakeRequest since they
> were
> > added in the same version. To choose between PLAIN and SCRAM-SHA-256,
> a
> > versions based approach would work, but wouldn't it be better for
> > clients
> > to rely on just SaslHandshakeRequest rather than
> > ApiVersionsRequest+SaslHandshakeRequest so that the solution works in
> > all
> > scenarios?
> >
> >
> >
> >
> > On Thu, Nov 3, 2016 at 4:33 AM, Ewen Cheslack-Postava <ewen@confluent.io
> >
> > wrote:
> >
> > > I think the bump isn't strictly required, but if the client is KIP-35
> > > aware, it can proactively choose a compatible SASL mechanism based on
> its
> > > initial ApiVersionRequest and avoid an extra connection round trip when
> > > there are client/broker version differences. Without this, a newer
> client
> > > would have to do 2 set of requests since the first SASL mechanism might
> > not
> > > be compatible.
> > >
> > > I don't think this is a deal breaker, but I do think it would be good
> to
> > > just standardize on KIP-35 as the way we figure out client/broker
> > > compatibility. The SASL stuff happened in parallel (maybe before?)
> KIP-35
> > > and ended up with its own mechanism, but I'm in favor of trying to
> > simplify
> > > everything by centralizing those considerations into a single API call.
> > (By
> > > the way, dredging up now ancient history in the KIP-35 discussion, this
> > is
> > > also why "features" vs "API version" is relevant. If we wanted to
> > configure
> > > a newer broker to disable SASL mechanisms we no longer want to allow
> use
> > > of, this isn't really possible to express via API versions unless we
> also
> > > explicitly add an API version that doesn't support that mechanism
> whereas
> > > features would make this easier to toggle on/off. The
> > SaslHandshakeRequest
> > > probably makes it easier to keep thing secure compared to the current
> > state
> > > of ApiVersionRequest).
> > >
> > > -Ewen
> > >
> > > On Tue, Nov 1, 2016 at 2:09 PM, Rajini Sivaram <
> > > rajinisivaram@googlemail.com
> > > > wrote:
> > >
> > > > Gwen,
> > > >
> > > > I had thought the same too and hence I am assuming that Java clients
> > > could
> > > > simply use SaslHandshakeRequest. SaslHandshakeRequest returns the
> list
> > of
> > > > mechanisms enabled in the broker. I think Jun's point was that by
> > > > incrementing the version of SaslHandshakeRequest, clients can use
> > > > ApiVersionsRequest to figure out the mechanisms the broker is capable
> > of
> > > > supporting and use that information to choose a mechanism to send in
> > > > SaslHandshakeRequest. Not sure how useful this actually is, so will
> > wait
> > > > for Jun's response.
> > > >
> > > >
> > > >
> > > > On Tue, Nov 1, 2016 at 8:18 PM, Gwen Shapira <gw...@confluent.io>
> > wrote:
> > > >
> > > > > Wait, I thought SaslHandshakeResponse includes a list of mechanisms
> > > > > supported, so I'm not sure why we need to bump the version?
> > > > >
> > > > > I expect clients will send SaslHandshakeRequest_V0, see which
> > > mechanisms
> > > > > are supported and make a call based on that? Which means KIP-35 is
> > not
> > > > > required in that case? Am I missing something?
> > > > >
> > > > > On Tue, Nov 1, 2016 at 1:07 PM, Rajini Sivaram <
> > > > > rajinisivaram@googlemail.com
> > > > > > wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > > I have added the following text to the KIP. Does this match your
> > > > > > expectation?
> > > > > >
> > > > > > *SaslHandshakeRequest version will be increased from 0 to 1 so
> that
> > > > > clients
> > > > > > can determine if the broker is capable of supporting SCRAM
> > mechanisms
> > > > > using
> > > > > > ApiVersionsRequest. Java clients will not be updated to use
> > > > > > ApiVersionsRequest to choose SASL mechanism under this KIP. Java
> > > > clients
> > > > > > will continue to use their configured SASL mechanism and will
> fail
> > > > > > connection if the requested mechanism is not enabled in the
> > broker.*
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > > On Tue, Nov 1, 2016 at 4:54 PM, Jun Rao <ju...@confluent.io>
> wrote:
> > > > > >
> > > > > > > Hi, Rajini,
> > > > > > >
> > > > > > > One more thing. It seems that we should bump up the version of
> > > > > > > SaslHandshakeRequest? This way, the client can figure out which
> > > SASL
> > > > > > > mechanisms the broker is capable of supporting through
> > > > > ApiVersionRequest.
> > > > > > > We discussed this briefly as part of KIP-43.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 1, 2016 at 7:41 AM, Rajini Sivaram <
> > > > > > > rajinisivaram@googlemail.com
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > If there are no more comments, I will start vote on this KIP
> > > later
> > > > > this
> > > > > > > > week. In the meantime, please feel free to post any feedback
> or
> > > > > > > > suggestions. Initial implementation is here:
> > > > > > > > https://github.com/apache/kafka/pull/2086.
> > > > > > > >
> > > > > > > > Thank you,
> > > > > > > >
> > > > > > > > Rajini
> > > > > > > >
> > > > > > > > On Thu, Oct 27, 2016 at 11:18 AM, Rajini Sivaram <
> > > > > > > > rajinisivaram@googlemail.com> wrote:
> > > > > > > >
> > > > > > > > > Jun,
> > > > > > > > >
> > > > > > > > > 4) Agree, it does make the implementation simpler. Updated
> > KIP.
> > > > > > > > > 5) Thank you, that looks neater. Updated KIP.
> > > > > > > > >
> > > > > > > > > On Wed, Oct 26, 2016 at 6:59 PM, Jun Rao <jun@confluent.io
> >
> > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi, Rajini,
> > > > > > > > >>
> > > > > > > > >> Thanks for the reply.
> > > > > > > > >>
> > > > > > > > >> 4. Implementation wise, it seems to me that it's simpler
> to
> > > read
> > > > > > from
> > > > > > > > the
> > > > > > > > >> cache than reading directly from ZK since the config
> manager
> > > > > already
> > > > > > > > >> propagates all config changes through ZK. Also, it's
> > probably
> > > a
> > > > > good
> > > > > > > > idea
> > > > > > > > >> to limit the places in the code base that directly
> accesses
> > > ZK.
> > > > > > > > >>
> > > > > > > > >> 5. Yes, it seems that it makes sense to add the new SCRAM
> > > > > > > configurations
> > > > > > > > >> to
> > > > > > > > >> the existing /config/users/<encoded-user>. I am not sure
> how
> > > one
> > > > > > would
> > > > > > > > >> remove the SCRAM configurations in the example though
> since
> > > the
> > > > > > > > properties
> > > > > > > > >> specified in add-config is not the ones actually being
> > stored.
> > > > An
> > > > > > > > >> alternative is to doing sth like the following. It may
> still
> > > > feel
> > > > > a
> > > > > > > bit
> > > > > > > > >> weird and I am wondering if there is a clearer approach.
> > > > > > > > >>
> > > > > > > > >> bin/kafka-configs.sh --zookeeper localhost:2181 --alter
> > > > > --add-config
> > > > > > > > >> 'scram_sha-256=[password=alice-secret,iterations=4096],
> > > > > scram_sha-1=
> > > > > > > > >> [password=alice-secret,iterations=4096]' --entity-type
> > users
> > > > > > > > >> --entity-name
> > > > > > > > >> alice
> > > > > > > > >>
> > > > > > > > >> bin/kafka-configs.sh --zookeeper localhost:2181 --alter
> > > > > > > --delete-config
> > > > > > > > >> 'scram_sha-256,scram_sha-1' --entity-type users
> > --entity-name
> > > > > alice
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >>
> > > > > > > > >> Jun
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Wed, Oct 26, 2016 at 4:35 AM, Rajini Sivaram <
> > > > > > > > >> rajinisivaram@googlemail.com> wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi Jun,
> > > > > > > > >> >
> > > > > > > > >> > Thank you for reviewing the KIP. Answers below:
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > 1. Yes, agree, Updated KIP.
> > > > > > > > >> > 2. User specifies a password and iteration count.
> > > > > > kaka-configs.sh
> > > > > > > > >> > generates a random salt and then generates StoredKey
> > and
> > > > > > > ServerKey
> > > > > > > > >> for
> > > > > > > > >> > that
> > > > > > > > >> > password using the same message formatter
> > implementation
> > > > used
> > > > > > for
> > > > > > > > >> SCRAM
> > > > > > > > >> > authentication. I have added some more detail to the
> > KIP
> > > (
> > > > > > > > >> > https://cwiki.apache.org/
> confluence/display/KAFKA/KIP-
> > > > > > > > >> > 84%3A+Support+SASL+SCRAM+mechanisms#KIP-84:
> > > > > > > > SupportSASLSCRAMmechanisms-
> > > > > > > > >> > Tools).
> > > > > > > > >> > Does that answer the question?
> > > > > > > > >> > 3. I started off thinking just one (SCRAM-SHA-256)
> and
> > > then
> > > > > > > thought
> > > > > > > > >> > another one is required to make sure that the
> > > > implementation
> > > > > > can
> > > > > > > > cope
> > > > > > > > >> > with
> > > > > > > > >> > multiple SCRAM mechanisms. But actually you are
> right,
> > we
> > > > can
> > > > > > > > support
> > > > > > > > >> > all.
> > > > > > > > >> > I haven't added the old md2/md5 mechanisms that
> aren't
> > > very
> > > > > > > secure,
> > > > > > > > >> but
> > > > > > > > >> > I
> > > > > > > > >> > have included all the SHA algorithms.
> > > > > > > > >> > 4. Since credentials are only required when a
> > connection
> > > is
> > > > > > made,
> > > > > > > > it
> > > > > > > > >> > feels like we can just read the latest value from ZK
> > > rather
> > > > > > than
> > > > > > > > >> cache
> > > > > > > > >> > all
> > > > > > > > >> > users and keep them updated. Having said that, we can
> > > > always
> > > > > > add
> > > > > > > > >> caching
> > > > > > > > >> > later if we find that the overhead of reading from ZK
> > > every
> > > > > > time
> > > > > > > is
> > > > > > > > >> too
> > > > > > > > >> > expensive. Since caching doesn't change any
> externals,
> > > this
> > > > > can
> > > > > > > be
> > > > > > > > >> done
> > > > > > > > >> > in
> > > > > > > > >> > a JIRA later - would that be ok?
> > > > > > > > >> > 5. Thanks, updated. I have changed the property names
> > to
> > > > > > include
> > > > > > > > >> > mechanism. To avoid four separate properties per
> > > mechanism
> > > > in
> > > > > > > ZK, I
> > > > > > > > >> have
> > > > > > > > >> > changed the format to use a single property
> (lower-case
> > > > > > mechanism
> > > > > > > > >> name)
> > > > > > > > >> > with four values concatenated in a format similar to
> > > SCRAM
> > > > > > > > messages.
> > > > > > > > >> >
> > > > > > > > >> > Do you think storing SCRAM credentials in
> > > > > > > /config/users/<encoded-user>
> > > > > > > > >> > along with existing quota properties is okay? Or should
> > they
> > > > be
> > > > > > > under
> > > > > > > > a
> > > > > > > > >> > different path (eg. /config/credentials/<encoded-
> user>)?
> > Or
> > > > > > should
> > > > > > > > >> they be
> > > > > > > > >> > on a completely different path like ACLs with a separate
> > > tool
> > > > > > > instead
> > > > > > > > of
> > > > > > > > >> > reusing kaka-configs.sh?
> > > > > > > > >> >
> > > > > > > > >> > Thank you,
> > > > > > > > >> >
> > > > > > > > >> > Rajini
> > > > > > > > >> >
> > > > > > > > >> > On Tue, Oct 25, 2016 at 11:55 PM, Jun Rao <
> > jun@confluent.io
> > > >
> > > > > > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Hi, Rajini,
> > > > > > > > >> > >
> > > > > > > > >> > > Thanks for the proposal. Looks good overall and seems
> > > quite
> > > > > > useful
> > > > > > > > >> (e.g.
> > > > > > > > >> > > for supporting delegation tokens). A few
> > > comments/questions
> > > > > > below.
> > > > > > > > >> > >
> > > > > > > > >> > > 1. For the ZK data format change, should we use the
> same
> > > > > > > convention
> > > > > > > > >> as in
> > > > > > > > >> > > KIP-55 to use encoded user name (i.e.,
> > > > > > > > /config/users/<encoded-user1>)
> > > > > > > > >> ?
> > > > > > > > >> > >
> > > > > > > > >> > > 2. For tooling, could you describe how user typically
> > > > > generates
> > > > > > > > >> > > scam_server_key and scram_stored_key to be used by
> > > > > > > kafka-config.sh?
> > > > > > > > >> > >
> > > > > > > > >> > > 3. Is there a particular reason to only support sha1
> and
> > > > > sha128?
> > > > > > > > >> Should
> > > > > > > > >> > we
> > > > > > > > >> > > support more hashes listed below in the future?
> > > > > > > > >> > > http://www.iana.org/assignments/hash-function-
> > > > > > > > >> > > text-names/hash-function-text-names.xhtml
> > > > > > > > >> > >
> > > > > > > > >> > > 4. Is there a reason not to cache user credentials in
> > the
> > > > > > broker?
> > > > > > > > The
> > > > > > > > >> > > dynamic config mechanism already supports loading
> > configs
> > > > into
> > > > > > > > >> broker's
> > > > > > > > >> > > cache. Checking credentials from broker's cache is
> more
> > > > > > efficient
> > > > > > > > than
> > > > > > > > >> > > reading from ZK each time.
> > > > > > > > >> > >
> > > > > > > > >> > > 5. Typo "scram_iteration-4096" (= instead of -).
> > > > > > > > >> > >
> > > > > > > > >> > > Thanks,
> > > > > > > > >> > >
> > > > > > > > >> > > Jun
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > > > > > > > >> > > rajinisivaram@googlemail.com
> > > > > > > > >> > > > wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > > Hi all,
> > > > > > > > >> > > >
> > > > > > > > >> > > > I have just created KIP-84 to add SCRAM-SHA-1 and
> > > > > > SCRAM-SHA-256
> > > > > > > > SASL
> > > > > > > > >> > > > mechanisms to Kafka:
> > > > > > > > >> > > >
> > > > > > > > >> > > > https://cwiki.apache.org/
> > confluence/display/KAFKA/KIP-
> > > > > > > > >> > > > 84%3A+Support+SASL+SCRAM+mechanisms
> > > > > > > > >> > > >
> > > > > > > > >> > > >
> > > > > > > > >> > > > Comments and suggestions are welcome.
> > > > > > > > >> > > >
> > > > > > > > >> > > > Thank you...
> > > > > > > > >> > > >
> > > > > > > > >> > > > Regards,
> > > > > > > > >> > > >
> > > > > > > > >> > > > Rajini
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > --
> > > > > > > > >> > Regards,
> > > > > > > > >> >
> > > > > > > > >> > Rajini
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Regards,
> > > > > > > > >
> > > > > > > > > Rajini
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Rajini
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Gwen Shapira*
> > > > > Product Manager | Confluent
> > > > > 650.450.2760 | @gwenshap
> > > > > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> > > > > <http://www.confluent.io/blog>
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Ewen
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>
--
Regards,
Rajini
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Ismael Juma <is...@gmail.com>.
Hi all,
I'm currently with limited internet access so I have not done a proper
review of the KIP, my apologies. A couple of thoughts:
1. I agree with Rajini and I don't see the value in bumping the protocol
version in this case. As Rajini said, the enabled mechanisms are
independent of the broker version. If and when we have a feature API, we
can consider exposing it that way.
2. Do we really want to support sha1? The security of it has been
questioned and the general recommendation is to avoid it these days.
Ismael
On 3 Nov 2016 6:51 am, "Rajini Sivaram" <ra...@googlemail.com>
wrote:
> I don't have a strong objection to bumping up SaslHandshakeRequest version,
> though I am still not convinced of the usefulness. I do agree that KIP-35
> should be standard way to determine client/broker compatibility. But I am
> not sure ApiVersionsRequest is a good fit for clients to choose a SASL
> mechanism.
>
> 1. SaslHandshakeRequest is the only way a client can figure out if a
> SASL mechanism is actually enabled in the broker. The fact that broker
> version n supports SCRAM doesn't imply that a broker of version n
> actually
> has the mechanism enabled. Since enabling a mechanism involves some
> effort
> like installing an authentication server and/or configuring credentials,
> SASL mechanisms are a feature of a broker installation very unlike
> versions. As you said, "features" would have worked better.
> 2. New SASL mechanisms can be added to older Kafka broker versions. With
> some tweaking, the SCRAM implementation from KIP-84 can be enabled in a
> 0.10.0 broker without changing any broker code. KIP-86 would make this
> even
> easier, but it is already possible to add new or custom mechanisms to
> existing broker versions. A client using ApiVersionsRequest to choose a
> SASL mechanism is only checking when a mechanism was included in a
> broker
> release, not the "capability" of a broker to support the mechanism. I am
> not sure we should encourage clients to choose mechanisms based on
> versions.
> 3. Clients need additional configuration based on the chosen mechanism.
> One of the reasons I couldn't see any value in using ApiVersionsRequest
> in
> the Java client was because clients are configured with a single SASL
> mechanism and a JAAS configuration corresponding to that mechanism. If a
> client wants to choose between Kerberos and SCRAM, the client would need
> keytab/principal for kerberos and username/password for SCRAM. Clients
> that
> possess multiple credentials without knowing what the broker supports -
> that sounds like a rather unusual scenario.
> 4. For some mechanisms, clients may want to choose mechanism at runtime.
> For example, a client is configured with username/ password and wants to
> choose between SCRAM-SHA-1/SCRAM-SHA-256/PLAIN depending on which
> mechanisms are enabled in the broker. To choose between SCRAM-SHA-1 and
> SCRAM-SHA-256, clients have to use SaslHandshakeRequest since they were
> added in the same version. To choose between PLAIN and SCRAM-SHA-256, a
> versions based approach would work, but wouldn't it be better for
> clients
> to rely on just SaslHandshakeRequest rather than
> ApiVersionsRequest+SaslHandshakeRequest so that the solution works in
> all
> scenarios?
>
>
>
>
> On Thu, Nov 3, 2016 at 4:33 AM, Ewen Cheslack-Postava <ew...@confluent.io>
> wrote:
>
> > I think the bump isn't strictly required, but if the client is KIP-35
> > aware, it can proactively choose a compatible SASL mechanism based on its
> > initial ApiVersionRequest and avoid an extra connection round trip when
> > there are client/broker version differences. Without this, a newer client
> > would have to do 2 set of requests since the first SASL mechanism might
> not
> > be compatible.
> >
> > I don't think this is a deal breaker, but I do think it would be good to
> > just standardize on KIP-35 as the way we figure out client/broker
> > compatibility. The SASL stuff happened in parallel (maybe before?) KIP-35
> > and ended up with its own mechanism, but I'm in favor of trying to
> simplify
> > everything by centralizing those considerations into a single API call.
> (By
> > the way, dredging up now ancient history in the KIP-35 discussion, this
> is
> > also why "features" vs "API version" is relevant. If we wanted to
> configure
> > a newer broker to disable SASL mechanisms we no longer want to allow use
> > of, this isn't really possible to express via API versions unless we also
> > explicitly add an API version that doesn't support that mechanism whereas
> > features would make this easier to toggle on/off. The
> SaslHandshakeRequest
> > probably makes it easier to keep thing secure compared to the current
> state
> > of ApiVersionRequest).
> >
> > -Ewen
> >
> > On Tue, Nov 1, 2016 at 2:09 PM, Rajini Sivaram <
> > rajinisivaram@googlemail.com
> > > wrote:
> >
> > > Gwen,
> > >
> > > I had thought the same too and hence I am assuming that Java clients
> > could
> > > simply use SaslHandshakeRequest. SaslHandshakeRequest returns the list
> of
> > > mechanisms enabled in the broker. I think Jun's point was that by
> > > incrementing the version of SaslHandshakeRequest, clients can use
> > > ApiVersionsRequest to figure out the mechanisms the broker is capable
> of
> > > supporting and use that information to choose a mechanism to send in
> > > SaslHandshakeRequest. Not sure how useful this actually is, so will
> wait
> > > for Jun's response.
> > >
> > >
> > >
> > > On Tue, Nov 1, 2016 at 8:18 PM, Gwen Shapira <gw...@confluent.io>
> wrote:
> > >
> > > > Wait, I thought SaslHandshakeResponse includes a list of mechanisms
> > > > supported, so I'm not sure why we need to bump the version?
> > > >
> > > > I expect clients will send SaslHandshakeRequest_V0, see which
> > mechanisms
> > > > are supported and make a call based on that? Which means KIP-35 is
> not
> > > > required in that case? Am I missing something?
> > > >
> > > > On Tue, Nov 1, 2016 at 1:07 PM, Rajini Sivaram <
> > > > rajinisivaram@googlemail.com
> > > > > wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > > I have added the following text to the KIP. Does this match your
> > > > > expectation?
> > > > >
> > > > > *SaslHandshakeRequest version will be increased from 0 to 1 so that
> > > > clients
> > > > > can determine if the broker is capable of supporting SCRAM
> mechanisms
> > > > using
> > > > > ApiVersionsRequest. Java clients will not be updated to use
> > > > > ApiVersionsRequest to choose SASL mechanism under this KIP. Java
> > > clients
> > > > > will continue to use their configured SASL mechanism and will fail
> > > > > connection if the requested mechanism is not enabled in the
> broker.*
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Rajini
> > > > >
> > > > > On Tue, Nov 1, 2016 at 4:54 PM, Jun Rao <ju...@confluent.io> wrote:
> > > > >
> > > > > > Hi, Rajini,
> > > > > >
> > > > > > One more thing. It seems that we should bump up the version of
> > > > > > SaslHandshakeRequest? This way, the client can figure out which
> > SASL
> > > > > > mechanisms the broker is capable of supporting through
> > > > ApiVersionRequest.
> > > > > > We discussed this briefly as part of KIP-43.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Nov 1, 2016 at 7:41 AM, Rajini Sivaram <
> > > > > > rajinisivaram@googlemail.com
> > > > > > > wrote:
> > > > > >
> > > > > > > If there are no more comments, I will start vote on this KIP
> > later
> > > > this
> > > > > > > week. In the meantime, please feel free to post any feedback or
> > > > > > > suggestions. Initial implementation is here:
> > > > > > > https://github.com/apache/kafka/pull/2086.
> > > > > > >
> > > > > > > Thank you,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > > > On Thu, Oct 27, 2016 at 11:18 AM, Rajini Sivaram <
> > > > > > > rajinisivaram@googlemail.com> wrote:
> > > > > > >
> > > > > > > > Jun,
> > > > > > > >
> > > > > > > > 4) Agree, it does make the implementation simpler. Updated
> KIP.
> > > > > > > > 5) Thank you, that looks neater. Updated KIP.
> > > > > > > >
> > > > > > > > On Wed, Oct 26, 2016 at 6:59 PM, Jun Rao <ju...@confluent.io>
> > > wrote:
> > > > > > > >
> > > > > > > >> Hi, Rajini,
> > > > > > > >>
> > > > > > > >> Thanks for the reply.
> > > > > > > >>
> > > > > > > >> 4. Implementation wise, it seems to me that it's simpler to
> > read
> > > > > from
> > > > > > > the
> > > > > > > >> cache than reading directly from ZK since the config manager
> > > > already
> > > > > > > >> propagates all config changes through ZK. Also, it's
> probably
> > a
> > > > good
> > > > > > > idea
> > > > > > > >> to limit the places in the code base that directly accesses
> > ZK.
> > > > > > > >>
> > > > > > > >> 5. Yes, it seems that it makes sense to add the new SCRAM
> > > > > > configurations
> > > > > > > >> to
> > > > > > > >> the existing /config/users/<encoded-user>. I am not sure how
> > one
> > > > > would
> > > > > > > >> remove the SCRAM configurations in the example though since
> > the
> > > > > > > properties
> > > > > > > >> specified in add-config is not the ones actually being
> stored.
> > > An
> > > > > > > >> alternative is to doing sth like the following. It may still
> > > feel
> > > > a
> > > > > > bit
> > > > > > > >> weird and I am wondering if there is a clearer approach.
> > > > > > > >>
> > > > > > > >> bin/kafka-configs.sh --zookeeper localhost:2181 --alter
> > > > --add-config
> > > > > > > >> 'scram_sha-256=[password=alice-secret,iterations=4096],
> > > > scram_sha-1=
> > > > > > > >> [password=alice-secret,iterations=4096]' --entity-type
> users
> > > > > > > >> --entity-name
> > > > > > > >> alice
> > > > > > > >>
> > > > > > > >> bin/kafka-configs.sh --zookeeper localhost:2181 --alter
> > > > > > --delete-config
> > > > > > > >> 'scram_sha-256,scram_sha-1' --entity-type users
> --entity-name
> > > > alice
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >>
> > > > > > > >> Jun
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Wed, Oct 26, 2016 at 4:35 AM, Rajini Sivaram <
> > > > > > > >> rajinisivaram@googlemail.com> wrote:
> > > > > > > >>
> > > > > > > >> > Hi Jun,
> > > > > > > >> >
> > > > > > > >> > Thank you for reviewing the KIP. Answers below:
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > 1. Yes, agree, Updated KIP.
> > > > > > > >> > 2. User specifies a password and iteration count.
> > > > > kaka-configs.sh
> > > > > > > >> > generates a random salt and then generates StoredKey
> and
> > > > > > ServerKey
> > > > > > > >> for
> > > > > > > >> > that
> > > > > > > >> > password using the same message formatter
> implementation
> > > used
> > > > > for
> > > > > > > >> SCRAM
> > > > > > > >> > authentication. I have added some more detail to the
> KIP
> > (
> > > > > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > >> > 84%3A+Support+SASL+SCRAM+mechanisms#KIP-84:
> > > > > > > SupportSASLSCRAMmechanisms-
> > > > > > > >> > Tools).
> > > > > > > >> > Does that answer the question?
> > > > > > > >> > 3. I started off thinking just one (SCRAM-SHA-256) and
> > then
> > > > > > thought
> > > > > > > >> > another one is required to make sure that the
> > > implementation
> > > > > can
> > > > > > > cope
> > > > > > > >> > with
> > > > > > > >> > multiple SCRAM mechanisms. But actually you are right,
> we
> > > can
> > > > > > > support
> > > > > > > >> > all.
> > > > > > > >> > I haven't added the old md2/md5 mechanisms that aren't
> > very
> > > > > > secure,
> > > > > > > >> but
> > > > > > > >> > I
> > > > > > > >> > have included all the SHA algorithms.
> > > > > > > >> > 4. Since credentials are only required when a
> connection
> > is
> > > > > made,
> > > > > > > it
> > > > > > > >> > feels like we can just read the latest value from ZK
> > rather
> > > > > than
> > > > > > > >> cache
> > > > > > > >> > all
> > > > > > > >> > users and keep them updated. Having said that, we can
> > > always
> > > > > add
> > > > > > > >> caching
> > > > > > > >> > later if we find that the overhead of reading from ZK
> > every
> > > > > time
> > > > > > is
> > > > > > > >> too
> > > > > > > >> > expensive. Since caching doesn't change any externals,
> > this
> > > > can
> > > > > > be
> > > > > > > >> done
> > > > > > > >> > in
> > > > > > > >> > a JIRA later - would that be ok?
> > > > > > > >> > 5. Thanks, updated. I have changed the property names
> to
> > > > > include
> > > > > > > >> > mechanism. To avoid four separate properties per
> > mechanism
> > > in
> > > > > > ZK, I
> > > > > > > >> have
> > > > > > > >> > changed the format to use a single property (lower-case
> > > > > mechanism
> > > > > > > >> name)
> > > > > > > >> > with four values concatenated in a format similar to
> > SCRAM
> > > > > > > messages.
> > > > > > > >> >
> > > > > > > >> > Do you think storing SCRAM credentials in
> > > > > > /config/users/<encoded-user>
> > > > > > > >> > along with existing quota properties is okay? Or should
> they
> > > be
> > > > > > under
> > > > > > > a
> > > > > > > >> > different path (eg. /config/credentials/<encoded-user>)?
> Or
> > > > > should
> > > > > > > >> they be
> > > > > > > >> > on a completely different path like ACLs with a separate
> > tool
> > > > > > instead
> > > > > > > of
> > > > > > > >> > reusing kaka-configs.sh?
> > > > > > > >> >
> > > > > > > >> > Thank you,
> > > > > > > >> >
> > > > > > > >> > Rajini
> > > > > > > >> >
> > > > > > > >> > On Tue, Oct 25, 2016 at 11:55 PM, Jun Rao <
> jun@confluent.io
> > >
> > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > > Hi, Rajini,
> > > > > > > >> > >
> > > > > > > >> > > Thanks for the proposal. Looks good overall and seems
> > quite
> > > > > useful
> > > > > > > >> (e.g.
> > > > > > > >> > > for supporting delegation tokens). A few
> > comments/questions
> > > > > below.
> > > > > > > >> > >
> > > > > > > >> > > 1. For the ZK data format change, should we use the same
> > > > > > convention
> > > > > > > >> as in
> > > > > > > >> > > KIP-55 to use encoded user name (i.e.,
> > > > > > > /config/users/<encoded-user1>)
> > > > > > > >> ?
> > > > > > > >> > >
> > > > > > > >> > > 2. For tooling, could you describe how user typically
> > > > generates
> > > > > > > >> > > scam_server_key and scram_stored_key to be used by
> > > > > > kafka-config.sh?
> > > > > > > >> > >
> > > > > > > >> > > 3. Is there a particular reason to only support sha1 and
> > > > sha128?
> > > > > > > >> Should
> > > > > > > >> > we
> > > > > > > >> > > support more hashes listed below in the future?
> > > > > > > >> > > http://www.iana.org/assignments/hash-function-
> > > > > > > >> > > text-names/hash-function-text-names.xhtml
> > > > > > > >> > >
> > > > > > > >> > > 4. Is there a reason not to cache user credentials in
> the
> > > > > broker?
> > > > > > > The
> > > > > > > >> > > dynamic config mechanism already supports loading
> configs
> > > into
> > > > > > > >> broker's
> > > > > > > >> > > cache. Checking credentials from broker's cache is more
> > > > > efficient
> > > > > > > than
> > > > > > > >> > > reading from ZK each time.
> > > > > > > >> > >
> > > > > > > >> > > 5. Typo "scram_iteration-4096" (= instead of -).
> > > > > > > >> > >
> > > > > > > >> > > Thanks,
> > > > > > > >> > >
> > > > > > > >> > > Jun
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > > > > > > >> > > rajinisivaram@googlemail.com
> > > > > > > >> > > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Hi all,
> > > > > > > >> > > >
> > > > > > > >> > > > I have just created KIP-84 to add SCRAM-SHA-1 and
> > > > > SCRAM-SHA-256
> > > > > > > SASL
> > > > > > > >> > > > mechanisms to Kafka:
> > > > > > > >> > > >
> > > > > > > >> > > > https://cwiki.apache.org/
> confluence/display/KAFKA/KIP-
> > > > > > > >> > > > 84%3A+Support+SASL+SCRAM+mechanisms
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > > Comments and suggestions are welcome.
> > > > > > > >> > > >
> > > > > > > >> > > > Thank you...
> > > > > > > >> > > >
> > > > > > > >> > > > Regards,
> > > > > > > >> > > >
> > > > > > > >> > > > Rajini
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > --
> > > > > > > >> > Regards,
> > > > > > > >> >
> > > > > > > >> > Rajini
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Rajini
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Gwen Shapira*
> > > > Product Manager | Confluent
> > > > 650.450.2760 | @gwenshap
> > > > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> > > > <http://www.confluent.io/blog>
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Rajini
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>
>
>
> --
> Regards,
>
> Rajini
>
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Rajini Sivaram <ra...@googlemail.com>.
I don't have a strong objection to bumping up SaslHandshakeRequest version,
though I am still not convinced of the usefulness. I do agree that KIP-35
should be standard way to determine client/broker compatibility. But I am
not sure ApiVersionsRequest is a good fit for clients to choose a SASL
mechanism.
1. SaslHandshakeRequest is the only way a client can figure out if a
SASL mechanism is actually enabled in the broker. The fact that broker
version n supports SCRAM doesn't imply that a broker of version n actually
has the mechanism enabled. Since enabling a mechanism involves some effort
like installing an authentication server and/or configuring credentials,
SASL mechanisms are a feature of a broker installation very unlike
versions. As you said, "features" would have worked better.
2. New SASL mechanisms can be added to older Kafka broker versions. With
some tweaking, the SCRAM implementation from KIP-84 can be enabled in a
0.10.0 broker without changing any broker code. KIP-86 would make this even
easier, but it is already possible to add new or custom mechanisms to
existing broker versions. A client using ApiVersionsRequest to choose a
SASL mechanism is only checking when a mechanism was included in a broker
release, not the "capability" of a broker to support the mechanism. I am
not sure we should encourage clients to choose mechanisms based on versions.
3. Clients need additional configuration based on the chosen mechanism.
One of the reasons I couldn't see any value in using ApiVersionsRequest in
the Java client was because clients are configured with a single SASL
mechanism and a JAAS configuration corresponding to that mechanism. If a
client wants to choose between Kerberos and SCRAM, the client would need
keytab/principal for kerberos and username/password for SCRAM. Clients that
possess multiple credentials without knowing what the broker supports -
that sounds like a rather unusual scenario.
4. For some mechanisms, clients may want to choose mechanism at runtime.
For example, a client is configured with username/ password and wants to
choose between SCRAM-SHA-1/SCRAM-SHA-256/PLAIN depending on which
mechanisms are enabled in the broker. To choose between SCRAM-SHA-1 and
SCRAM-SHA-256, clients have to use SaslHandshakeRequest since they were
added in the same version. To choose between PLAIN and SCRAM-SHA-256, a
versions based approach would work, but wouldn't it be better for clients
to rely on just SaslHandshakeRequest rather than
ApiVersionsRequest+SaslHandshakeRequest so that the solution works in all
scenarios?
On Thu, Nov 3, 2016 at 4:33 AM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> I think the bump isn't strictly required, but if the client is KIP-35
> aware, it can proactively choose a compatible SASL mechanism based on its
> initial ApiVersionRequest and avoid an extra connection round trip when
> there are client/broker version differences. Without this, a newer client
> would have to do 2 set of requests since the first SASL mechanism might not
> be compatible.
>
> I don't think this is a deal breaker, but I do think it would be good to
> just standardize on KIP-35 as the way we figure out client/broker
> compatibility. The SASL stuff happened in parallel (maybe before?) KIP-35
> and ended up with its own mechanism, but I'm in favor of trying to simplify
> everything by centralizing those considerations into a single API call. (By
> the way, dredging up now ancient history in the KIP-35 discussion, this is
> also why "features" vs "API version" is relevant. If we wanted to configure
> a newer broker to disable SASL mechanisms we no longer want to allow use
> of, this isn't really possible to express via API versions unless we also
> explicitly add an API version that doesn't support that mechanism whereas
> features would make this easier to toggle on/off. The SaslHandshakeRequest
> probably makes it easier to keep thing secure compared to the current state
> of ApiVersionRequest).
>
> -Ewen
>
> On Tue, Nov 1, 2016 at 2:09 PM, Rajini Sivaram <
> rajinisivaram@googlemail.com
> > wrote:
>
> > Gwen,
> >
> > I had thought the same too and hence I am assuming that Java clients
> could
> > simply use SaslHandshakeRequest. SaslHandshakeRequest returns the list of
> > mechanisms enabled in the broker. I think Jun's point was that by
> > incrementing the version of SaslHandshakeRequest, clients can use
> > ApiVersionsRequest to figure out the mechanisms the broker is capable of
> > supporting and use that information to choose a mechanism to send in
> > SaslHandshakeRequest. Not sure how useful this actually is, so will wait
> > for Jun's response.
> >
> >
> >
> > On Tue, Nov 1, 2016 at 8:18 PM, Gwen Shapira <gw...@confluent.io> wrote:
> >
> > > Wait, I thought SaslHandshakeResponse includes a list of mechanisms
> > > supported, so I'm not sure why we need to bump the version?
> > >
> > > I expect clients will send SaslHandshakeRequest_V0, see which
> mechanisms
> > > are supported and make a call based on that? Which means KIP-35 is not
> > > required in that case? Am I missing something?
> > >
> > > On Tue, Nov 1, 2016 at 1:07 PM, Rajini Sivaram <
> > > rajinisivaram@googlemail.com
> > > > wrote:
> > >
> > > > Jun,
> > > >
> > > > I have added the following text to the KIP. Does this match your
> > > > expectation?
> > > >
> > > > *SaslHandshakeRequest version will be increased from 0 to 1 so that
> > > clients
> > > > can determine if the broker is capable of supporting SCRAM mechanisms
> > > using
> > > > ApiVersionsRequest. Java clients will not be updated to use
> > > > ApiVersionsRequest to choose SASL mechanism under this KIP. Java
> > clients
> > > > will continue to use their configured SASL mechanism and will fail
> > > > connection if the requested mechanism is not enabled in the broker.*
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> > > > On Tue, Nov 1, 2016 at 4:54 PM, Jun Rao <ju...@confluent.io> wrote:
> > > >
> > > > > Hi, Rajini,
> > > > >
> > > > > One more thing. It seems that we should bump up the version of
> > > > > SaslHandshakeRequest? This way, the client can figure out which
> SASL
> > > > > mechanisms the broker is capable of supporting through
> > > ApiVersionRequest.
> > > > > We discussed this briefly as part of KIP-43.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Nov 1, 2016 at 7:41 AM, Rajini Sivaram <
> > > > > rajinisivaram@googlemail.com
> > > > > > wrote:
> > > > >
> > > > > > If there are no more comments, I will start vote on this KIP
> later
> > > this
> > > > > > week. In the meantime, please feel free to post any feedback or
> > > > > > suggestions. Initial implementation is here:
> > > > > > https://github.com/apache/kafka/pull/2086.
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > > On Thu, Oct 27, 2016 at 11:18 AM, Rajini Sivaram <
> > > > > > rajinisivaram@googlemail.com> wrote:
> > > > > >
> > > > > > > Jun,
> > > > > > >
> > > > > > > 4) Agree, it does make the implementation simpler. Updated KIP.
> > > > > > > 5) Thank you, that looks neater. Updated KIP.
> > > > > > >
> > > > > > > On Wed, Oct 26, 2016 at 6:59 PM, Jun Rao <ju...@confluent.io>
> > wrote:
> > > > > > >
> > > > > > >> Hi, Rajini,
> > > > > > >>
> > > > > > >> Thanks for the reply.
> > > > > > >>
> > > > > > >> 4. Implementation wise, it seems to me that it's simpler to
> read
> > > > from
> > > > > > the
> > > > > > >> cache than reading directly from ZK since the config manager
> > > already
> > > > > > >> propagates all config changes through ZK. Also, it's probably
> a
> > > good
> > > > > > idea
> > > > > > >> to limit the places in the code base that directly accesses
> ZK.
> > > > > > >>
> > > > > > >> 5. Yes, it seems that it makes sense to add the new SCRAM
> > > > > configurations
> > > > > > >> to
> > > > > > >> the existing /config/users/<encoded-user>. I am not sure how
> one
> > > > would
> > > > > > >> remove the SCRAM configurations in the example though since
> the
> > > > > > properties
> > > > > > >> specified in add-config is not the ones actually being stored.
> > An
> > > > > > >> alternative is to doing sth like the following. It may still
> > feel
> > > a
> > > > > bit
> > > > > > >> weird and I am wondering if there is a clearer approach.
> > > > > > >>
> > > > > > >> bin/kafka-configs.sh --zookeeper localhost:2181 --alter
> > > --add-config
> > > > > > >> 'scram_sha-256=[password=alice-secret,iterations=4096],
> > > scram_sha-1=
> > > > > > >> [password=alice-secret,iterations=4096]' --entity-type users
> > > > > > >> --entity-name
> > > > > > >> alice
> > > > > > >>
> > > > > > >> bin/kafka-configs.sh --zookeeper localhost:2181 --alter
> > > > > --delete-config
> > > > > > >> 'scram_sha-256,scram_sha-1' --entity-type users --entity-name
> > > alice
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >>
> > > > > > >> Jun
> > > > > > >>
> > > > > > >>
> > > > > > >> On Wed, Oct 26, 2016 at 4:35 AM, Rajini Sivaram <
> > > > > > >> rajinisivaram@googlemail.com> wrote:
> > > > > > >>
> > > > > > >> > Hi Jun,
> > > > > > >> >
> > > > > > >> > Thank you for reviewing the KIP. Answers below:
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > 1. Yes, agree, Updated KIP.
> > > > > > >> > 2. User specifies a password and iteration count.
> > > > kaka-configs.sh
> > > > > > >> > generates a random salt and then generates StoredKey and
> > > > > ServerKey
> > > > > > >> for
> > > > > > >> > that
> > > > > > >> > password using the same message formatter implementation
> > used
> > > > for
> > > > > > >> SCRAM
> > > > > > >> > authentication. I have added some more detail to the KIP
> (
> > > > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > >> > 84%3A+Support+SASL+SCRAM+mechanisms#KIP-84:
> > > > > > SupportSASLSCRAMmechanisms-
> > > > > > >> > Tools).
> > > > > > >> > Does that answer the question?
> > > > > > >> > 3. I started off thinking just one (SCRAM-SHA-256) and
> then
> > > > > thought
> > > > > > >> > another one is required to make sure that the
> > implementation
> > > > can
> > > > > > cope
> > > > > > >> > with
> > > > > > >> > multiple SCRAM mechanisms. But actually you are right, we
> > can
> > > > > > support
> > > > > > >> > all.
> > > > > > >> > I haven't added the old md2/md5 mechanisms that aren't
> very
> > > > > secure,
> > > > > > >> but
> > > > > > >> > I
> > > > > > >> > have included all the SHA algorithms.
> > > > > > >> > 4. Since credentials are only required when a connection
> is
> > > > made,
> > > > > > it
> > > > > > >> > feels like we can just read the latest value from ZK
> rather
> > > > than
> > > > > > >> cache
> > > > > > >> > all
> > > > > > >> > users and keep them updated. Having said that, we can
> > always
> > > > add
> > > > > > >> caching
> > > > > > >> > later if we find that the overhead of reading from ZK
> every
> > > > time
> > > > > is
> > > > > > >> too
> > > > > > >> > expensive. Since caching doesn't change any externals,
> this
> > > can
> > > > > be
> > > > > > >> done
> > > > > > >> > in
> > > > > > >> > a JIRA later - would that be ok?
> > > > > > >> > 5. Thanks, updated. I have changed the property names to
> > > > include
> > > > > > >> > mechanism. To avoid four separate properties per
> mechanism
> > in
> > > > > ZK, I
> > > > > > >> have
> > > > > > >> > changed the format to use a single property (lower-case
> > > > mechanism
> > > > > > >> name)
> > > > > > >> > with four values concatenated in a format similar to
> SCRAM
> > > > > > messages.
> > > > > > >> >
> > > > > > >> > Do you think storing SCRAM credentials in
> > > > > /config/users/<encoded-user>
> > > > > > >> > along with existing quota properties is okay? Or should they
> > be
> > > > > under
> > > > > > a
> > > > > > >> > different path (eg. /config/credentials/<encoded-user>)? Or
> > > > should
> > > > > > >> they be
> > > > > > >> > on a completely different path like ACLs with a separate
> tool
> > > > > instead
> > > > > > of
> > > > > > >> > reusing kaka-configs.sh?
> > > > > > >> >
> > > > > > >> > Thank you,
> > > > > > >> >
> > > > > > >> > Rajini
> > > > > > >> >
> > > > > > >> > On Tue, Oct 25, 2016 at 11:55 PM, Jun Rao <jun@confluent.io
> >
> > > > wrote:
> > > > > > >> >
> > > > > > >> > > Hi, Rajini,
> > > > > > >> > >
> > > > > > >> > > Thanks for the proposal. Looks good overall and seems
> quite
> > > > useful
> > > > > > >> (e.g.
> > > > > > >> > > for supporting delegation tokens). A few
> comments/questions
> > > > below.
> > > > > > >> > >
> > > > > > >> > > 1. For the ZK data format change, should we use the same
> > > > > convention
> > > > > > >> as in
> > > > > > >> > > KIP-55 to use encoded user name (i.e.,
> > > > > > /config/users/<encoded-user1>)
> > > > > > >> ?
> > > > > > >> > >
> > > > > > >> > > 2. For tooling, could you describe how user typically
> > > generates
> > > > > > >> > > scam_server_key and scram_stored_key to be used by
> > > > > kafka-config.sh?
> > > > > > >> > >
> > > > > > >> > > 3. Is there a particular reason to only support sha1 and
> > > sha128?
> > > > > > >> Should
> > > > > > >> > we
> > > > > > >> > > support more hashes listed below in the future?
> > > > > > >> > > http://www.iana.org/assignments/hash-function-
> > > > > > >> > > text-names/hash-function-text-names.xhtml
> > > > > > >> > >
> > > > > > >> > > 4. Is there a reason not to cache user credentials in the
> > > > broker?
> > > > > > The
> > > > > > >> > > dynamic config mechanism already supports loading configs
> > into
> > > > > > >> broker's
> > > > > > >> > > cache. Checking credentials from broker's cache is more
> > > > efficient
> > > > > > than
> > > > > > >> > > reading from ZK each time.
> > > > > > >> > >
> > > > > > >> > > 5. Typo "scram_iteration-4096" (= instead of -).
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > >
> > > > > > >> > > Jun
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > > > > > >> > > rajinisivaram@googlemail.com
> > > > > > >> > > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Hi all,
> > > > > > >> > > >
> > > > > > >> > > > I have just created KIP-84 to add SCRAM-SHA-1 and
> > > > SCRAM-SHA-256
> > > > > > SASL
> > > > > > >> > > > mechanisms to Kafka:
> > > > > > >> > > >
> > > > > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > >> > > > 84%3A+Support+SASL+SCRAM+mechanisms
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > Comments and suggestions are welcome.
> > > > > > >> > > >
> > > > > > >> > > > Thank you...
> > > > > > >> > > >
> > > > > > >> > > > Regards,
> > > > > > >> > > >
> > > > > > >> > > > Rajini
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > --
> > > > > > >> > Regards,
> > > > > > >> >
> > > > > > >> > Rajini
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> > >
> > >
> > > --
> > > *Gwen Shapira*
> > > Product Manager | Confluent
> > > 650.450.2760 | @gwenshap
> > > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> > > <http://www.confluent.io/blog>
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>
>
>
> --
> Thanks,
> Ewen
>
--
Regards,
Rajini
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
I think the bump isn't strictly required, but if the client is KIP-35
aware, it can proactively choose a compatible SASL mechanism based on its
initial ApiVersionRequest and avoid an extra connection round trip when
there are client/broker version differences. Without this, a newer client
would have to do 2 set of requests since the first SASL mechanism might not
be compatible.
I don't think this is a deal breaker, but I do think it would be good to
just standardize on KIP-35 as the way we figure out client/broker
compatibility. The SASL stuff happened in parallel (maybe before?) KIP-35
and ended up with its own mechanism, but I'm in favor of trying to simplify
everything by centralizing those considerations into a single API call. (By
the way, dredging up now ancient history in the KIP-35 discussion, this is
also why "features" vs "API version" is relevant. If we wanted to configure
a newer broker to disable SASL mechanisms we no longer want to allow use
of, this isn't really possible to express via API versions unless we also
explicitly add an API version that doesn't support that mechanism whereas
features would make this easier to toggle on/off. The SaslHandshakeRequest
probably makes it easier to keep thing secure compared to the current state
of ApiVersionRequest).
-Ewen
On Tue, Nov 1, 2016 at 2:09 PM, Rajini Sivaram <rajinisivaram@googlemail.com
> wrote:
> Gwen,
>
> I had thought the same too and hence I am assuming that Java clients could
> simply use SaslHandshakeRequest. SaslHandshakeRequest returns the list of
> mechanisms enabled in the broker. I think Jun's point was that by
> incrementing the version of SaslHandshakeRequest, clients can use
> ApiVersionsRequest to figure out the mechanisms the broker is capable of
> supporting and use that information to choose a mechanism to send in
> SaslHandshakeRequest. Not sure how useful this actually is, so will wait
> for Jun's response.
>
>
>
> On Tue, Nov 1, 2016 at 8:18 PM, Gwen Shapira <gw...@confluent.io> wrote:
>
> > Wait, I thought SaslHandshakeResponse includes a list of mechanisms
> > supported, so I'm not sure why we need to bump the version?
> >
> > I expect clients will send SaslHandshakeRequest_V0, see which mechanisms
> > are supported and make a call based on that? Which means KIP-35 is not
> > required in that case? Am I missing something?
> >
> > On Tue, Nov 1, 2016 at 1:07 PM, Rajini Sivaram <
> > rajinisivaram@googlemail.com
> > > wrote:
> >
> > > Jun,
> > >
> > > I have added the following text to the KIP. Does this match your
> > > expectation?
> > >
> > > *SaslHandshakeRequest version will be increased from 0 to 1 so that
> > clients
> > > can determine if the broker is capable of supporting SCRAM mechanisms
> > using
> > > ApiVersionsRequest. Java clients will not be updated to use
> > > ApiVersionsRequest to choose SASL mechanism under this KIP. Java
> clients
> > > will continue to use their configured SASL mechanism and will fail
> > > connection if the requested mechanism is not enabled in the broker.*
> > >
> > > Thank you,
> > >
> > > Rajini
> > >
> > > On Tue, Nov 1, 2016 at 4:54 PM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Hi, Rajini,
> > > >
> > > > One more thing. It seems that we should bump up the version of
> > > > SaslHandshakeRequest? This way, the client can figure out which SASL
> > > > mechanisms the broker is capable of supporting through
> > ApiVersionRequest.
> > > > We discussed this briefly as part of KIP-43.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > >
> > > > On Tue, Nov 1, 2016 at 7:41 AM, Rajini Sivaram <
> > > > rajinisivaram@googlemail.com
> > > > > wrote:
> > > >
> > > > > If there are no more comments, I will start vote on this KIP later
> > this
> > > > > week. In the meantime, please feel free to post any feedback or
> > > > > suggestions. Initial implementation is here:
> > > > > https://github.com/apache/kafka/pull/2086.
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Rajini
> > > > >
> > > > > On Thu, Oct 27, 2016 at 11:18 AM, Rajini Sivaram <
> > > > > rajinisivaram@googlemail.com> wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > > 4) Agree, it does make the implementation simpler. Updated KIP.
> > > > > > 5) Thank you, that looks neater. Updated KIP.
> > > > > >
> > > > > > On Wed, Oct 26, 2016 at 6:59 PM, Jun Rao <ju...@confluent.io>
> wrote:
> > > > > >
> > > > > >> Hi, Rajini,
> > > > > >>
> > > > > >> Thanks for the reply.
> > > > > >>
> > > > > >> 4. Implementation wise, it seems to me that it's simpler to read
> > > from
> > > > > the
> > > > > >> cache than reading directly from ZK since the config manager
> > already
> > > > > >> propagates all config changes through ZK. Also, it's probably a
> > good
> > > > > idea
> > > > > >> to limit the places in the code base that directly accesses ZK.
> > > > > >>
> > > > > >> 5. Yes, it seems that it makes sense to add the new SCRAM
> > > > configurations
> > > > > >> to
> > > > > >> the existing /config/users/<encoded-user>. I am not sure how one
> > > would
> > > > > >> remove the SCRAM configurations in the example though since the
> > > > > properties
> > > > > >> specified in add-config is not the ones actually being stored.
> An
> > > > > >> alternative is to doing sth like the following. It may still
> feel
> > a
> > > > bit
> > > > > >> weird and I am wondering if there is a clearer approach.
> > > > > >>
> > > > > >> bin/kafka-configs.sh --zookeeper localhost:2181 --alter
> > --add-config
> > > > > >> 'scram_sha-256=[password=alice-secret,iterations=4096],
> > scram_sha-1=
> > > > > >> [password=alice-secret,iterations=4096]' --entity-type users
> > > > > >> --entity-name
> > > > > >> alice
> > > > > >>
> > > > > >> bin/kafka-configs.sh --zookeeper localhost:2181 --alter
> > > > --delete-config
> > > > > >> 'scram_sha-256,scram_sha-1' --entity-type users --entity-name
> > alice
> > > > > >>
> > > > > >> Thanks,
> > > > > >>
> > > > > >> Jun
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Oct 26, 2016 at 4:35 AM, Rajini Sivaram <
> > > > > >> rajinisivaram@googlemail.com> wrote:
> > > > > >>
> > > > > >> > Hi Jun,
> > > > > >> >
> > > > > >> > Thank you for reviewing the KIP. Answers below:
> > > > > >> >
> > > > > >> >
> > > > > >> > 1. Yes, agree, Updated KIP.
> > > > > >> > 2. User specifies a password and iteration count.
> > > kaka-configs.sh
> > > > > >> > generates a random salt and then generates StoredKey and
> > > > ServerKey
> > > > > >> for
> > > > > >> > that
> > > > > >> > password using the same message formatter implementation
> used
> > > for
> > > > > >> SCRAM
> > > > > >> > authentication. I have added some more detail to the KIP (
> > > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > >> > 84%3A+Support+SASL+SCRAM+mechanisms#KIP-84:
> > > > > SupportSASLSCRAMmechanisms-
> > > > > >> > Tools).
> > > > > >> > Does that answer the question?
> > > > > >> > 3. I started off thinking just one (SCRAM-SHA-256) and then
> > > > thought
> > > > > >> > another one is required to make sure that the
> implementation
> > > can
> > > > > cope
> > > > > >> > with
> > > > > >> > multiple SCRAM mechanisms. But actually you are right, we
> can
> > > > > support
> > > > > >> > all.
> > > > > >> > I haven't added the old md2/md5 mechanisms that aren't very
> > > > secure,
> > > > > >> but
> > > > > >> > I
> > > > > >> > have included all the SHA algorithms.
> > > > > >> > 4. Since credentials are only required when a connection is
> > > made,
> > > > > it
> > > > > >> > feels like we can just read the latest value from ZK rather
> > > than
> > > > > >> cache
> > > > > >> > all
> > > > > >> > users and keep them updated. Having said that, we can
> always
> > > add
> > > > > >> caching
> > > > > >> > later if we find that the overhead of reading from ZK every
> > > time
> > > > is
> > > > > >> too
> > > > > >> > expensive. Since caching doesn't change any externals, this
> > can
> > > > be
> > > > > >> done
> > > > > >> > in
> > > > > >> > a JIRA later - would that be ok?
> > > > > >> > 5. Thanks, updated. I have changed the property names to
> > > include
> > > > > >> > mechanism. To avoid four separate properties per mechanism
> in
> > > > ZK, I
> > > > > >> have
> > > > > >> > changed the format to use a single property (lower-case
> > > mechanism
> > > > > >> name)
> > > > > >> > with four values concatenated in a format similar to SCRAM
> > > > > messages.
> > > > > >> >
> > > > > >> > Do you think storing SCRAM credentials in
> > > > /config/users/<encoded-user>
> > > > > >> > along with existing quota properties is okay? Or should they
> be
> > > > under
> > > > > a
> > > > > >> > different path (eg. /config/credentials/<encoded-user>)? Or
> > > should
> > > > > >> they be
> > > > > >> > on a completely different path like ACLs with a separate tool
> > > > instead
> > > > > of
> > > > > >> > reusing kaka-configs.sh?
> > > > > >> >
> > > > > >> > Thank you,
> > > > > >> >
> > > > > >> > Rajini
> > > > > >> >
> > > > > >> > On Tue, Oct 25, 2016 at 11:55 PM, Jun Rao <ju...@confluent.io>
> > > wrote:
> > > > > >> >
> > > > > >> > > Hi, Rajini,
> > > > > >> > >
> > > > > >> > > Thanks for the proposal. Looks good overall and seems quite
> > > useful
> > > > > >> (e.g.
> > > > > >> > > for supporting delegation tokens). A few comments/questions
> > > below.
> > > > > >> > >
> > > > > >> > > 1. For the ZK data format change, should we use the same
> > > > convention
> > > > > >> as in
> > > > > >> > > KIP-55 to use encoded user name (i.e.,
> > > > > /config/users/<encoded-user1>)
> > > > > >> ?
> > > > > >> > >
> > > > > >> > > 2. For tooling, could you describe how user typically
> > generates
> > > > > >> > > scam_server_key and scram_stored_key to be used by
> > > > kafka-config.sh?
> > > > > >> > >
> > > > > >> > > 3. Is there a particular reason to only support sha1 and
> > sha128?
> > > > > >> Should
> > > > > >> > we
> > > > > >> > > support more hashes listed below in the future?
> > > > > >> > > http://www.iana.org/assignments/hash-function-
> > > > > >> > > text-names/hash-function-text-names.xhtml
> > > > > >> > >
> > > > > >> > > 4. Is there a reason not to cache user credentials in the
> > > broker?
> > > > > The
> > > > > >> > > dynamic config mechanism already supports loading configs
> into
> > > > > >> broker's
> > > > > >> > > cache. Checking credentials from broker's cache is more
> > > efficient
> > > > > than
> > > > > >> > > reading from ZK each time.
> > > > > >> > >
> > > > > >> > > 5. Typo "scram_iteration-4096" (= instead of -).
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > >
> > > > > >> > > Jun
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > > > > >> > > rajinisivaram@googlemail.com
> > > > > >> > > > wrote:
> > > > > >> > >
> > > > > >> > > > Hi all,
> > > > > >> > > >
> > > > > >> > > > I have just created KIP-84 to add SCRAM-SHA-1 and
> > > SCRAM-SHA-256
> > > > > SASL
> > > > > >> > > > mechanisms to Kafka:
> > > > > >> > > >
> > > > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > >> > > > 84%3A+Support+SASL+SCRAM+mechanisms
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > Comments and suggestions are welcome.
> > > > > >> > > >
> > > > > >> > > > Thank you...
> > > > > >> > > >
> > > > > >> > > > Regards,
> > > > > >> > > >
> > > > > >> > > > Rajini
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > --
> > > > > >> > Regards,
> > > > > >> >
> > > > > >> > Rajini
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Rajini
> > >
> >
> >
> >
> > --
> > *Gwen Shapira*
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> > <http://www.confluent.io/blog>
> >
>
>
>
> --
> Regards,
>
> Rajini
>
--
Thanks,
Ewen
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Rajini Sivaram <ra...@googlemail.com>.
Gwen,
I had thought the same too and hence I am assuming that Java clients could
simply use SaslHandshakeRequest. SaslHandshakeRequest returns the list of
mechanisms enabled in the broker. I think Jun's point was that by
incrementing the version of SaslHandshakeRequest, clients can use
ApiVersionsRequest to figure out the mechanisms the broker is capable of
supporting and use that information to choose a mechanism to send in
SaslHandshakeRequest. Not sure how useful this actually is, so will wait
for Jun's response.
On Tue, Nov 1, 2016 at 8:18 PM, Gwen Shapira <gw...@confluent.io> wrote:
> Wait, I thought SaslHandshakeResponse includes a list of mechanisms
> supported, so I'm not sure why we need to bump the version?
>
> I expect clients will send SaslHandshakeRequest_V0, see which mechanisms
> are supported and make a call based on that? Which means KIP-35 is not
> required in that case? Am I missing something?
>
> On Tue, Nov 1, 2016 at 1:07 PM, Rajini Sivaram <
> rajinisivaram@googlemail.com
> > wrote:
>
> > Jun,
> >
> > I have added the following text to the KIP. Does this match your
> > expectation?
> >
> > *SaslHandshakeRequest version will be increased from 0 to 1 so that
> clients
> > can determine if the broker is capable of supporting SCRAM mechanisms
> using
> > ApiVersionsRequest. Java clients will not be updated to use
> > ApiVersionsRequest to choose SASL mechanism under this KIP. Java clients
> > will continue to use their configured SASL mechanism and will fail
> > connection if the requested mechanism is not enabled in the broker.*
> >
> > Thank you,
> >
> > Rajini
> >
> > On Tue, Nov 1, 2016 at 4:54 PM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Hi, Rajini,
> > >
> > > One more thing. It seems that we should bump up the version of
> > > SaslHandshakeRequest? This way, the client can figure out which SASL
> > > mechanisms the broker is capable of supporting through
> ApiVersionRequest.
> > > We discussed this briefly as part of KIP-43.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > >
> > > On Tue, Nov 1, 2016 at 7:41 AM, Rajini Sivaram <
> > > rajinisivaram@googlemail.com
> > > > wrote:
> > >
> > > > If there are no more comments, I will start vote on this KIP later
> this
> > > > week. In the meantime, please feel free to post any feedback or
> > > > suggestions. Initial implementation is here:
> > > > https://github.com/apache/kafka/pull/2086.
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> > > > On Thu, Oct 27, 2016 at 11:18 AM, Rajini Sivaram <
> > > > rajinisivaram@googlemail.com> wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > > 4) Agree, it does make the implementation simpler. Updated KIP.
> > > > > 5) Thank you, that looks neater. Updated KIP.
> > > > >
> > > > > On Wed, Oct 26, 2016 at 6:59 PM, Jun Rao <ju...@confluent.io> wrote:
> > > > >
> > > > >> Hi, Rajini,
> > > > >>
> > > > >> Thanks for the reply.
> > > > >>
> > > > >> 4. Implementation wise, it seems to me that it's simpler to read
> > from
> > > > the
> > > > >> cache than reading directly from ZK since the config manager
> already
> > > > >> propagates all config changes through ZK. Also, it's probably a
> good
> > > > idea
> > > > >> to limit the places in the code base that directly accesses ZK.
> > > > >>
> > > > >> 5. Yes, it seems that it makes sense to add the new SCRAM
> > > configurations
> > > > >> to
> > > > >> the existing /config/users/<encoded-user>. I am not sure how one
> > would
> > > > >> remove the SCRAM configurations in the example though since the
> > > > properties
> > > > >> specified in add-config is not the ones actually being stored. An
> > > > >> alternative is to doing sth like the following. It may still feel
> a
> > > bit
> > > > >> weird and I am wondering if there is a clearer approach.
> > > > >>
> > > > >> bin/kafka-configs.sh --zookeeper localhost:2181 --alter
> --add-config
> > > > >> 'scram_sha-256=[password=alice-secret,iterations=4096],
> scram_sha-1=
> > > > >> [password=alice-secret,iterations=4096]' --entity-type users
> > > > >> --entity-name
> > > > >> alice
> > > > >>
> > > > >> bin/kafka-configs.sh --zookeeper localhost:2181 --alter
> > > --delete-config
> > > > >> 'scram_sha-256,scram_sha-1' --entity-type users --entity-name
> alice
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Jun
> > > > >>
> > > > >>
> > > > >> On Wed, Oct 26, 2016 at 4:35 AM, Rajini Sivaram <
> > > > >> rajinisivaram@googlemail.com> wrote:
> > > > >>
> > > > >> > Hi Jun,
> > > > >> >
> > > > >> > Thank you for reviewing the KIP. Answers below:
> > > > >> >
> > > > >> >
> > > > >> > 1. Yes, agree, Updated KIP.
> > > > >> > 2. User specifies a password and iteration count.
> > kaka-configs.sh
> > > > >> > generates a random salt and then generates StoredKey and
> > > ServerKey
> > > > >> for
> > > > >> > that
> > > > >> > password using the same message formatter implementation used
> > for
> > > > >> SCRAM
> > > > >> > authentication. I have added some more detail to the KIP (
> > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >> > 84%3A+Support+SASL+SCRAM+mechanisms#KIP-84:
> > > > SupportSASLSCRAMmechanisms-
> > > > >> > Tools).
> > > > >> > Does that answer the question?
> > > > >> > 3. I started off thinking just one (SCRAM-SHA-256) and then
> > > thought
> > > > >> > another one is required to make sure that the implementation
> > can
> > > > cope
> > > > >> > with
> > > > >> > multiple SCRAM mechanisms. But actually you are right, we can
> > > > support
> > > > >> > all.
> > > > >> > I haven't added the old md2/md5 mechanisms that aren't very
> > > secure,
> > > > >> but
> > > > >> > I
> > > > >> > have included all the SHA algorithms.
> > > > >> > 4. Since credentials are only required when a connection is
> > made,
> > > > it
> > > > >> > feels like we can just read the latest value from ZK rather
> > than
> > > > >> cache
> > > > >> > all
> > > > >> > users and keep them updated. Having said that, we can always
> > add
> > > > >> caching
> > > > >> > later if we find that the overhead of reading from ZK every
> > time
> > > is
> > > > >> too
> > > > >> > expensive. Since caching doesn't change any externals, this
> can
> > > be
> > > > >> done
> > > > >> > in
> > > > >> > a JIRA later - would that be ok?
> > > > >> > 5. Thanks, updated. I have changed the property names to
> > include
> > > > >> > mechanism. To avoid four separate properties per mechanism in
> > > ZK, I
> > > > >> have
> > > > >> > changed the format to use a single property (lower-case
> > mechanism
> > > > >> name)
> > > > >> > with four values concatenated in a format similar to SCRAM
> > > > messages.
> > > > >> >
> > > > >> > Do you think storing SCRAM credentials in
> > > /config/users/<encoded-user>
> > > > >> > along with existing quota properties is okay? Or should they be
> > > under
> > > > a
> > > > >> > different path (eg. /config/credentials/<encoded-user>)? Or
> > should
> > > > >> they be
> > > > >> > on a completely different path like ACLs with a separate tool
> > > instead
> > > > of
> > > > >> > reusing kaka-configs.sh?
> > > > >> >
> > > > >> > Thank you,
> > > > >> >
> > > > >> > Rajini
> > > > >> >
> > > > >> > On Tue, Oct 25, 2016 at 11:55 PM, Jun Rao <ju...@confluent.io>
> > wrote:
> > > > >> >
> > > > >> > > Hi, Rajini,
> > > > >> > >
> > > > >> > > Thanks for the proposal. Looks good overall and seems quite
> > useful
> > > > >> (e.g.
> > > > >> > > for supporting delegation tokens). A few comments/questions
> > below.
> > > > >> > >
> > > > >> > > 1. For the ZK data format change, should we use the same
> > > convention
> > > > >> as in
> > > > >> > > KIP-55 to use encoded user name (i.e.,
> > > > /config/users/<encoded-user1>)
> > > > >> ?
> > > > >> > >
> > > > >> > > 2. For tooling, could you describe how user typically
> generates
> > > > >> > > scam_server_key and scram_stored_key to be used by
> > > kafka-config.sh?
> > > > >> > >
> > > > >> > > 3. Is there a particular reason to only support sha1 and
> sha128?
> > > > >> Should
> > > > >> > we
> > > > >> > > support more hashes listed below in the future?
> > > > >> > > http://www.iana.org/assignments/hash-function-
> > > > >> > > text-names/hash-function-text-names.xhtml
> > > > >> > >
> > > > >> > > 4. Is there a reason not to cache user credentials in the
> > broker?
> > > > The
> > > > >> > > dynamic config mechanism already supports loading configs into
> > > > >> broker's
> > > > >> > > cache. Checking credentials from broker's cache is more
> > efficient
> > > > than
> > > > >> > > reading from ZK each time.
> > > > >> > >
> > > > >> > > 5. Typo "scram_iteration-4096" (= instead of -).
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > >
> > > > >> > > Jun
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > > > >> > > rajinisivaram@googlemail.com
> > > > >> > > > wrote:
> > > > >> > >
> > > > >> > > > Hi all,
> > > > >> > > >
> > > > >> > > > I have just created KIP-84 to add SCRAM-SHA-1 and
> > SCRAM-SHA-256
> > > > SASL
> > > > >> > > > mechanisms to Kafka:
> > > > >> > > >
> > > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >> > > > 84%3A+Support+SASL+SCRAM+mechanisms
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > Comments and suggestions are welcome.
> > > > >> > > >
> > > > >> > > > Thank you...
> > > > >> > > >
> > > > >> > > > Regards,
> > > > >> > > >
> > > > >> > > > Rajini
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> > Regards,
> > > > >> >
> > > > >> > Rajini
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> <http://www.confluent.io/blog>
>
--
Regards,
Rajini
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Gwen Shapira <gw...@confluent.io>.
Wait, I thought SaslHandshakeResponse includes a list of mechanisms
supported, so I'm not sure why we need to bump the version?
I expect clients will send SaslHandshakeRequest_V0, see which mechanisms
are supported and make a call based on that? Which means KIP-35 is not
required in that case? Am I missing something?
On Tue, Nov 1, 2016 at 1:07 PM, Rajini Sivaram <rajinisivaram@googlemail.com
> wrote:
> Jun,
>
> I have added the following text to the KIP. Does this match your
> expectation?
>
> *SaslHandshakeRequest version will be increased from 0 to 1 so that clients
> can determine if the broker is capable of supporting SCRAM mechanisms using
> ApiVersionsRequest. Java clients will not be updated to use
> ApiVersionsRequest to choose SASL mechanism under this KIP. Java clients
> will continue to use their configured SASL mechanism and will fail
> connection if the requested mechanism is not enabled in the broker.*
>
> Thank you,
>
> Rajini
>
> On Tue, Nov 1, 2016 at 4:54 PM, Jun Rao <ju...@confluent.io> wrote:
>
> > Hi, Rajini,
> >
> > One more thing. It seems that we should bump up the version of
> > SaslHandshakeRequest? This way, the client can figure out which SASL
> > mechanisms the broker is capable of supporting through ApiVersionRequest.
> > We discussed this briefly as part of KIP-43.
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Tue, Nov 1, 2016 at 7:41 AM, Rajini Sivaram <
> > rajinisivaram@googlemail.com
> > > wrote:
> >
> > > If there are no more comments, I will start vote on this KIP later this
> > > week. In the meantime, please feel free to post any feedback or
> > > suggestions. Initial implementation is here:
> > > https://github.com/apache/kafka/pull/2086.
> > >
> > > Thank you,
> > >
> > > Rajini
> > >
> > > On Thu, Oct 27, 2016 at 11:18 AM, Rajini Sivaram <
> > > rajinisivaram@googlemail.com> wrote:
> > >
> > > > Jun,
> > > >
> > > > 4) Agree, it does make the implementation simpler. Updated KIP.
> > > > 5) Thank you, that looks neater. Updated KIP.
> > > >
> > > > On Wed, Oct 26, 2016 at 6:59 PM, Jun Rao <ju...@confluent.io> wrote:
> > > >
> > > >> Hi, Rajini,
> > > >>
> > > >> Thanks for the reply.
> > > >>
> > > >> 4. Implementation wise, it seems to me that it's simpler to read
> from
> > > the
> > > >> cache than reading directly from ZK since the config manager already
> > > >> propagates all config changes through ZK. Also, it's probably a good
> > > idea
> > > >> to limit the places in the code base that directly accesses ZK.
> > > >>
> > > >> 5. Yes, it seems that it makes sense to add the new SCRAM
> > configurations
> > > >> to
> > > >> the existing /config/users/<encoded-user>. I am not sure how one
> would
> > > >> remove the SCRAM configurations in the example though since the
> > > properties
> > > >> specified in add-config is not the ones actually being stored. An
> > > >> alternative is to doing sth like the following. It may still feel a
> > bit
> > > >> weird and I am wondering if there is a clearer approach.
> > > >>
> > > >> bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config
> > > >> 'scram_sha-256=[password=alice-secret,iterations=4096],scram_sha-1=
> > > >> [password=alice-secret,iterations=4096]' --entity-type users
> > > >> --entity-name
> > > >> alice
> > > >>
> > > >> bin/kafka-configs.sh --zookeeper localhost:2181 --alter
> > --delete-config
> > > >> 'scram_sha-256,scram_sha-1' --entity-type users --entity-name alice
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Jun
> > > >>
> > > >>
> > > >> On Wed, Oct 26, 2016 at 4:35 AM, Rajini Sivaram <
> > > >> rajinisivaram@googlemail.com> wrote:
> > > >>
> > > >> > Hi Jun,
> > > >> >
> > > >> > Thank you for reviewing the KIP. Answers below:
> > > >> >
> > > >> >
> > > >> > 1. Yes, agree, Updated KIP.
> > > >> > 2. User specifies a password and iteration count.
> kaka-configs.sh
> > > >> > generates a random salt and then generates StoredKey and
> > ServerKey
> > > >> for
> > > >> > that
> > > >> > password using the same message formatter implementation used
> for
> > > >> SCRAM
> > > >> > authentication. I have added some more detail to the KIP (
> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >> > 84%3A+Support+SASL+SCRAM+mechanisms#KIP-84:
> > > SupportSASLSCRAMmechanisms-
> > > >> > Tools).
> > > >> > Does that answer the question?
> > > >> > 3. I started off thinking just one (SCRAM-SHA-256) and then
> > thought
> > > >> > another one is required to make sure that the implementation
> can
> > > cope
> > > >> > with
> > > >> > multiple SCRAM mechanisms. But actually you are right, we can
> > > support
> > > >> > all.
> > > >> > I haven't added the old md2/md5 mechanisms that aren't very
> > secure,
> > > >> but
> > > >> > I
> > > >> > have included all the SHA algorithms.
> > > >> > 4. Since credentials are only required when a connection is
> made,
> > > it
> > > >> > feels like we can just read the latest value from ZK rather
> than
> > > >> cache
> > > >> > all
> > > >> > users and keep them updated. Having said that, we can always
> add
> > > >> caching
> > > >> > later if we find that the overhead of reading from ZK every
> time
> > is
> > > >> too
> > > >> > expensive. Since caching doesn't change any externals, this can
> > be
> > > >> done
> > > >> > in
> > > >> > a JIRA later - would that be ok?
> > > >> > 5. Thanks, updated. I have changed the property names to
> include
> > > >> > mechanism. To avoid four separate properties per mechanism in
> > ZK, I
> > > >> have
> > > >> > changed the format to use a single property (lower-case
> mechanism
> > > >> name)
> > > >> > with four values concatenated in a format similar to SCRAM
> > > messages.
> > > >> >
> > > >> > Do you think storing SCRAM credentials in
> > /config/users/<encoded-user>
> > > >> > along with existing quota properties is okay? Or should they be
> > under
> > > a
> > > >> > different path (eg. /config/credentials/<encoded-user>)? Or
> should
> > > >> they be
> > > >> > on a completely different path like ACLs with a separate tool
> > instead
> > > of
> > > >> > reusing kaka-configs.sh?
> > > >> >
> > > >> > Thank you,
> > > >> >
> > > >> > Rajini
> > > >> >
> > > >> > On Tue, Oct 25, 2016 at 11:55 PM, Jun Rao <ju...@confluent.io>
> wrote:
> > > >> >
> > > >> > > Hi, Rajini,
> > > >> > >
> > > >> > > Thanks for the proposal. Looks good overall and seems quite
> useful
> > > >> (e.g.
> > > >> > > for supporting delegation tokens). A few comments/questions
> below.
> > > >> > >
> > > >> > > 1. For the ZK data format change, should we use the same
> > convention
> > > >> as in
> > > >> > > KIP-55 to use encoded user name (i.e.,
> > > /config/users/<encoded-user1>)
> > > >> ?
> > > >> > >
> > > >> > > 2. For tooling, could you describe how user typically generates
> > > >> > > scam_server_key and scram_stored_key to be used by
> > kafka-config.sh?
> > > >> > >
> > > >> > > 3. Is there a particular reason to only support sha1 and sha128?
> > > >> Should
> > > >> > we
> > > >> > > support more hashes listed below in the future?
> > > >> > > http://www.iana.org/assignments/hash-function-
> > > >> > > text-names/hash-function-text-names.xhtml
> > > >> > >
> > > >> > > 4. Is there a reason not to cache user credentials in the
> broker?
> > > The
> > > >> > > dynamic config mechanism already supports loading configs into
> > > >> broker's
> > > >> > > cache. Checking credentials from broker's cache is more
> efficient
> > > than
> > > >> > > reading from ZK each time.
> > > >> > >
> > > >> > > 5. Typo "scram_iteration-4096" (= instead of -).
> > > >> > >
> > > >> > > Thanks,
> > > >> > >
> > > >> > > Jun
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > > >> > > rajinisivaram@googlemail.com
> > > >> > > > wrote:
> > > >> > >
> > > >> > > > Hi all,
> > > >> > > >
> > > >> > > > I have just created KIP-84 to add SCRAM-SHA-1 and
> SCRAM-SHA-256
> > > SASL
> > > >> > > > mechanisms to Kafka:
> > > >> > > >
> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >> > > > 84%3A+Support+SASL+SCRAM+mechanisms
> > > >> > > >
> > > >> > > >
> > > >> > > > Comments and suggestions are welcome.
> > > >> > > >
> > > >> > > > Thank you...
> > > >> > > >
> > > >> > > > Regards,
> > > >> > > >
> > > >> > > > Rajini
> > > >> > > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Regards,
> > > >> >
> > > >> > Rajini
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Rajini
> > >
> >
>
>
>
> --
> Regards,
>
> Rajini
>
--
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
<http://www.confluent.io/blog>
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Rajini Sivaram <ra...@googlemail.com>.
Jun,
I have added the following text to the KIP. Does this match your
expectation?
*SaslHandshakeRequest version will be increased from 0 to 1 so that clients
can determine if the broker is capable of supporting SCRAM mechanisms using
ApiVersionsRequest. Java clients will not be updated to use
ApiVersionsRequest to choose SASL mechanism under this KIP. Java clients
will continue to use their configured SASL mechanism and will fail
connection if the requested mechanism is not enabled in the broker.*
Thank you,
Rajini
On Tue, Nov 1, 2016 at 4:54 PM, Jun Rao <ju...@confluent.io> wrote:
> Hi, Rajini,
>
> One more thing. It seems that we should bump up the version of
> SaslHandshakeRequest? This way, the client can figure out which SASL
> mechanisms the broker is capable of supporting through ApiVersionRequest.
> We discussed this briefly as part of KIP-43.
>
> Thanks,
>
> Jun
>
>
>
> On Tue, Nov 1, 2016 at 7:41 AM, Rajini Sivaram <
> rajinisivaram@googlemail.com
> > wrote:
>
> > If there are no more comments, I will start vote on this KIP later this
> > week. In the meantime, please feel free to post any feedback or
> > suggestions. Initial implementation is here:
> > https://github.com/apache/kafka/pull/2086.
> >
> > Thank you,
> >
> > Rajini
> >
> > On Thu, Oct 27, 2016 at 11:18 AM, Rajini Sivaram <
> > rajinisivaram@googlemail.com> wrote:
> >
> > > Jun,
> > >
> > > 4) Agree, it does make the implementation simpler. Updated KIP.
> > > 5) Thank you, that looks neater. Updated KIP.
> > >
> > > On Wed, Oct 26, 2016 at 6:59 PM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > >> Hi, Rajini,
> > >>
> > >> Thanks for the reply.
> > >>
> > >> 4. Implementation wise, it seems to me that it's simpler to read from
> > the
> > >> cache than reading directly from ZK since the config manager already
> > >> propagates all config changes through ZK. Also, it's probably a good
> > idea
> > >> to limit the places in the code base that directly accesses ZK.
> > >>
> > >> 5. Yes, it seems that it makes sense to add the new SCRAM
> configurations
> > >> to
> > >> the existing /config/users/<encoded-user>. I am not sure how one would
> > >> remove the SCRAM configurations in the example though since the
> > properties
> > >> specified in add-config is not the ones actually being stored. An
> > >> alternative is to doing sth like the following. It may still feel a
> bit
> > >> weird and I am wondering if there is a clearer approach.
> > >>
> > >> bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config
> > >> 'scram_sha-256=[password=alice-secret,iterations=4096],scram_sha-1=
> > >> [password=alice-secret,iterations=4096]' --entity-type users
> > >> --entity-name
> > >> alice
> > >>
> > >> bin/kafka-configs.sh --zookeeper localhost:2181 --alter
> --delete-config
> > >> 'scram_sha-256,scram_sha-1' --entity-type users --entity-name alice
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >>
> > >> On Wed, Oct 26, 2016 at 4:35 AM, Rajini Sivaram <
> > >> rajinisivaram@googlemail.com> wrote:
> > >>
> > >> > Hi Jun,
> > >> >
> > >> > Thank you for reviewing the KIP. Answers below:
> > >> >
> > >> >
> > >> > 1. Yes, agree, Updated KIP.
> > >> > 2. User specifies a password and iteration count. kaka-configs.sh
> > >> > generates a random salt and then generates StoredKey and
> ServerKey
> > >> for
> > >> > that
> > >> > password using the same message formatter implementation used for
> > >> SCRAM
> > >> > authentication. I have added some more detail to the KIP (
> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >> > 84%3A+Support+SASL+SCRAM+mechanisms#KIP-84:
> > SupportSASLSCRAMmechanisms-
> > >> > Tools).
> > >> > Does that answer the question?
> > >> > 3. I started off thinking just one (SCRAM-SHA-256) and then
> thought
> > >> > another one is required to make sure that the implementation can
> > cope
> > >> > with
> > >> > multiple SCRAM mechanisms. But actually you are right, we can
> > support
> > >> > all.
> > >> > I haven't added the old md2/md5 mechanisms that aren't very
> secure,
> > >> but
> > >> > I
> > >> > have included all the SHA algorithms.
> > >> > 4. Since credentials are only required when a connection is made,
> > it
> > >> > feels like we can just read the latest value from ZK rather than
> > >> cache
> > >> > all
> > >> > users and keep them updated. Having said that, we can always add
> > >> caching
> > >> > later if we find that the overhead of reading from ZK every time
> is
> > >> too
> > >> > expensive. Since caching doesn't change any externals, this can
> be
> > >> done
> > >> > in
> > >> > a JIRA later - would that be ok?
> > >> > 5. Thanks, updated. I have changed the property names to include
> > >> > mechanism. To avoid four separate properties per mechanism in
> ZK, I
> > >> have
> > >> > changed the format to use a single property (lower-case mechanism
> > >> name)
> > >> > with four values concatenated in a format similar to SCRAM
> > messages.
> > >> >
> > >> > Do you think storing SCRAM credentials in
> /config/users/<encoded-user>
> > >> > along with existing quota properties is okay? Or should they be
> under
> > a
> > >> > different path (eg. /config/credentials/<encoded-user>)? Or should
> > >> they be
> > >> > on a completely different path like ACLs with a separate tool
> instead
> > of
> > >> > reusing kaka-configs.sh?
> > >> >
> > >> > Thank you,
> > >> >
> > >> > Rajini
> > >> >
> > >> > On Tue, Oct 25, 2016 at 11:55 PM, Jun Rao <ju...@confluent.io> wrote:
> > >> >
> > >> > > Hi, Rajini,
> > >> > >
> > >> > > Thanks for the proposal. Looks good overall and seems quite useful
> > >> (e.g.
> > >> > > for supporting delegation tokens). A few comments/questions below.
> > >> > >
> > >> > > 1. For the ZK data format change, should we use the same
> convention
> > >> as in
> > >> > > KIP-55 to use encoded user name (i.e.,
> > /config/users/<encoded-user1>)
> > >> ?
> > >> > >
> > >> > > 2. For tooling, could you describe how user typically generates
> > >> > > scam_server_key and scram_stored_key to be used by
> kafka-config.sh?
> > >> > >
> > >> > > 3. Is there a particular reason to only support sha1 and sha128?
> > >> Should
> > >> > we
> > >> > > support more hashes listed below in the future?
> > >> > > http://www.iana.org/assignments/hash-function-
> > >> > > text-names/hash-function-text-names.xhtml
> > >> > >
> > >> > > 4. Is there a reason not to cache user credentials in the broker?
> > The
> > >> > > dynamic config mechanism already supports loading configs into
> > >> broker's
> > >> > > cache. Checking credentials from broker's cache is more efficient
> > than
> > >> > > reading from ZK each time.
> > >> > >
> > >> > > 5. Typo "scram_iteration-4096" (= instead of -).
> > >> > >
> > >> > > Thanks,
> > >> > >
> > >> > > Jun
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > >> > > rajinisivaram@googlemail.com
> > >> > > > wrote:
> > >> > >
> > >> > > > Hi all,
> > >> > > >
> > >> > > > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256
> > SASL
> > >> > > > mechanisms to Kafka:
> > >> > > >
> > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >> > > > 84%3A+Support+SASL+SCRAM+mechanisms
> > >> > > >
> > >> > > >
> > >> > > > Comments and suggestions are welcome.
> > >> > > >
> > >> > > > Thank you...
> > >> > > >
> > >> > > > Regards,
> > >> > > >
> > >> > > > Rajini
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Regards,
> > >> >
> > >> > Rajini
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Rajini
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>
--
Regards,
Rajini
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Jun Rao <ju...@confluent.io>.
Hi, Rajini,
One more thing. It seems that we should bump up the version of
SaslHandshakeRequest? This way, the client can figure out which SASL
mechanisms the broker is capable of supporting through ApiVersionRequest.
We discussed this briefly as part of KIP-43.
Thanks,
Jun
On Tue, Nov 1, 2016 at 7:41 AM, Rajini Sivaram <rajinisivaram@googlemail.com
> wrote:
> If there are no more comments, I will start vote on this KIP later this
> week. In the meantime, please feel free to post any feedback or
> suggestions. Initial implementation is here:
> https://github.com/apache/kafka/pull/2086.
>
> Thank you,
>
> Rajini
>
> On Thu, Oct 27, 2016 at 11:18 AM, Rajini Sivaram <
> rajinisivaram@googlemail.com> wrote:
>
> > Jun,
> >
> > 4) Agree, it does make the implementation simpler. Updated KIP.
> > 5) Thank you, that looks neater. Updated KIP.
> >
> > On Wed, Oct 26, 2016 at 6:59 PM, Jun Rao <ju...@confluent.io> wrote:
> >
> >> Hi, Rajini,
> >>
> >> Thanks for the reply.
> >>
> >> 4. Implementation wise, it seems to me that it's simpler to read from
> the
> >> cache than reading directly from ZK since the config manager already
> >> propagates all config changes through ZK. Also, it's probably a good
> idea
> >> to limit the places in the code base that directly accesses ZK.
> >>
> >> 5. Yes, it seems that it makes sense to add the new SCRAM configurations
> >> to
> >> the existing /config/users/<encoded-user>. I am not sure how one would
> >> remove the SCRAM configurations in the example though since the
> properties
> >> specified in add-config is not the ones actually being stored. An
> >> alternative is to doing sth like the following. It may still feel a bit
> >> weird and I am wondering if there is a clearer approach.
> >>
> >> bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config
> >> 'scram_sha-256=[password=alice-secret,iterations=4096],scram_sha-1=
> >> [password=alice-secret,iterations=4096]' --entity-type users
> >> --entity-name
> >> alice
> >>
> >> bin/kafka-configs.sh --zookeeper localhost:2181 --alter --delete-config
> >> 'scram_sha-256,scram_sha-1' --entity-type users --entity-name alice
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >> On Wed, Oct 26, 2016 at 4:35 AM, Rajini Sivaram <
> >> rajinisivaram@googlemail.com> wrote:
> >>
> >> > Hi Jun,
> >> >
> >> > Thank you for reviewing the KIP. Answers below:
> >> >
> >> >
> >> > 1. Yes, agree, Updated KIP.
> >> > 2. User specifies a password and iteration count. kaka-configs.sh
> >> > generates a random salt and then generates StoredKey and ServerKey
> >> for
> >> > that
> >> > password using the same message formatter implementation used for
> >> SCRAM
> >> > authentication. I have added some more detail to the KIP (
> >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> > 84%3A+Support+SASL+SCRAM+mechanisms#KIP-84:
> SupportSASLSCRAMmechanisms-
> >> > Tools).
> >> > Does that answer the question?
> >> > 3. I started off thinking just one (SCRAM-SHA-256) and then thought
> >> > another one is required to make sure that the implementation can
> cope
> >> > with
> >> > multiple SCRAM mechanisms. But actually you are right, we can
> support
> >> > all.
> >> > I haven't added the old md2/md5 mechanisms that aren't very secure,
> >> but
> >> > I
> >> > have included all the SHA algorithms.
> >> > 4. Since credentials are only required when a connection is made,
> it
> >> > feels like we can just read the latest value from ZK rather than
> >> cache
> >> > all
> >> > users and keep them updated. Having said that, we can always add
> >> caching
> >> > later if we find that the overhead of reading from ZK every time is
> >> too
> >> > expensive. Since caching doesn't change any externals, this can be
> >> done
> >> > in
> >> > a JIRA later - would that be ok?
> >> > 5. Thanks, updated. I have changed the property names to include
> >> > mechanism. To avoid four separate properties per mechanism in ZK, I
> >> have
> >> > changed the format to use a single property (lower-case mechanism
> >> name)
> >> > with four values concatenated in a format similar to SCRAM
> messages.
> >> >
> >> > Do you think storing SCRAM credentials in /config/users/<encoded-user>
> >> > along with existing quota properties is okay? Or should they be under
> a
> >> > different path (eg. /config/credentials/<encoded-user>)? Or should
> >> they be
> >> > on a completely different path like ACLs with a separate tool instead
> of
> >> > reusing kaka-configs.sh?
> >> >
> >> > Thank you,
> >> >
> >> > Rajini
> >> >
> >> > On Tue, Oct 25, 2016 at 11:55 PM, Jun Rao <ju...@confluent.io> wrote:
> >> >
> >> > > Hi, Rajini,
> >> > >
> >> > > Thanks for the proposal. Looks good overall and seems quite useful
> >> (e.g.
> >> > > for supporting delegation tokens). A few comments/questions below.
> >> > >
> >> > > 1. For the ZK data format change, should we use the same convention
> >> as in
> >> > > KIP-55 to use encoded user name (i.e.,
> /config/users/<encoded-user1>)
> >> ?
> >> > >
> >> > > 2. For tooling, could you describe how user typically generates
> >> > > scam_server_key and scram_stored_key to be used by kafka-config.sh?
> >> > >
> >> > > 3. Is there a particular reason to only support sha1 and sha128?
> >> Should
> >> > we
> >> > > support more hashes listed below in the future?
> >> > > http://www.iana.org/assignments/hash-function-
> >> > > text-names/hash-function-text-names.xhtml
> >> > >
> >> > > 4. Is there a reason not to cache user credentials in the broker?
> The
> >> > > dynamic config mechanism already supports loading configs into
> >> broker's
> >> > > cache. Checking credentials from broker's cache is more efficient
> than
> >> > > reading from ZK each time.
> >> > >
> >> > > 5. Typo "scram_iteration-4096" (= instead of -).
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Jun
> >> > >
> >> > >
> >> > >
> >> > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> >> > > rajinisivaram@googlemail.com
> >> > > > wrote:
> >> > >
> >> > > > Hi all,
> >> > > >
> >> > > > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256
> SASL
> >> > > > mechanisms to Kafka:
> >> > > >
> >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> > > > 84%3A+Support+SASL+SCRAM+mechanisms
> >> > > >
> >> > > >
> >> > > > Comments and suggestions are welcome.
> >> > > >
> >> > > > Thank you...
> >> > > >
> >> > > > Regards,
> >> > > >
> >> > > > Rajini
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Regards,
> >> >
> >> > Rajini
> >> >
> >>
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>
>
>
> --
> Regards,
>
> Rajini
>
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Rajini Sivaram <ra...@googlemail.com>.
If there are no more comments, I will start vote on this KIP later this
week. In the meantime, please feel free to post any feedback or
suggestions. Initial implementation is here:
https://github.com/apache/kafka/pull/2086.
Thank you,
Rajini
On Thu, Oct 27, 2016 at 11:18 AM, Rajini Sivaram <
rajinisivaram@googlemail.com> wrote:
> Jun,
>
> 4) Agree, it does make the implementation simpler. Updated KIP.
> 5) Thank you, that looks neater. Updated KIP.
>
> On Wed, Oct 26, 2016 at 6:59 PM, Jun Rao <ju...@confluent.io> wrote:
>
>> Hi, Rajini,
>>
>> Thanks for the reply.
>>
>> 4. Implementation wise, it seems to me that it's simpler to read from the
>> cache than reading directly from ZK since the config manager already
>> propagates all config changes through ZK. Also, it's probably a good idea
>> to limit the places in the code base that directly accesses ZK.
>>
>> 5. Yes, it seems that it makes sense to add the new SCRAM configurations
>> to
>> the existing /config/users/<encoded-user>. I am not sure how one would
>> remove the SCRAM configurations in the example though since the properties
>> specified in add-config is not the ones actually being stored. An
>> alternative is to doing sth like the following. It may still feel a bit
>> weird and I am wondering if there is a clearer approach.
>>
>> bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config
>> 'scram_sha-256=[password=alice-secret,iterations=4096],scram_sha-1=
>> [password=alice-secret,iterations=4096]' --entity-type users
>> --entity-name
>> alice
>>
>> bin/kafka-configs.sh --zookeeper localhost:2181 --alter --delete-config
>> 'scram_sha-256,scram_sha-1' --entity-type users --entity-name alice
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Wed, Oct 26, 2016 at 4:35 AM, Rajini Sivaram <
>> rajinisivaram@googlemail.com> wrote:
>>
>> > Hi Jun,
>> >
>> > Thank you for reviewing the KIP. Answers below:
>> >
>> >
>> > 1. Yes, agree, Updated KIP.
>> > 2. User specifies a password and iteration count. kaka-configs.sh
>> > generates a random salt and then generates StoredKey and ServerKey
>> for
>> > that
>> > password using the same message formatter implementation used for
>> SCRAM
>> > authentication. I have added some more detail to the KIP (
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > 84%3A+Support+SASL+SCRAM+mechanisms#KIP-84:SupportSASLSCRAMmechanisms-
>> > Tools).
>> > Does that answer the question?
>> > 3. I started off thinking just one (SCRAM-SHA-256) and then thought
>> > another one is required to make sure that the implementation can cope
>> > with
>> > multiple SCRAM mechanisms. But actually you are right, we can support
>> > all.
>> > I haven't added the old md2/md5 mechanisms that aren't very secure,
>> but
>> > I
>> > have included all the SHA algorithms.
>> > 4. Since credentials are only required when a connection is made, it
>> > feels like we can just read the latest value from ZK rather than
>> cache
>> > all
>> > users and keep them updated. Having said that, we can always add
>> caching
>> > later if we find that the overhead of reading from ZK every time is
>> too
>> > expensive. Since caching doesn't change any externals, this can be
>> done
>> > in
>> > a JIRA later - would that be ok?
>> > 5. Thanks, updated. I have changed the property names to include
>> > mechanism. To avoid four separate properties per mechanism in ZK, I
>> have
>> > changed the format to use a single property (lower-case mechanism
>> name)
>> > with four values concatenated in a format similar to SCRAM messages.
>> >
>> > Do you think storing SCRAM credentials in /config/users/<encoded-user>
>> > along with existing quota properties is okay? Or should they be under a
>> > different path (eg. /config/credentials/<encoded-user>)? Or should
>> they be
>> > on a completely different path like ACLs with a separate tool instead of
>> > reusing kaka-configs.sh?
>> >
>> > Thank you,
>> >
>> > Rajini
>> >
>> > On Tue, Oct 25, 2016 at 11:55 PM, Jun Rao <ju...@confluent.io> wrote:
>> >
>> > > Hi, Rajini,
>> > >
>> > > Thanks for the proposal. Looks good overall and seems quite useful
>> (e.g.
>> > > for supporting delegation tokens). A few comments/questions below.
>> > >
>> > > 1. For the ZK data format change, should we use the same convention
>> as in
>> > > KIP-55 to use encoded user name (i.e., /config/users/<encoded-user1>)
>> ?
>> > >
>> > > 2. For tooling, could you describe how user typically generates
>> > > scam_server_key and scram_stored_key to be used by kafka-config.sh?
>> > >
>> > > 3. Is there a particular reason to only support sha1 and sha128?
>> Should
>> > we
>> > > support more hashes listed below in the future?
>> > > http://www.iana.org/assignments/hash-function-
>> > > text-names/hash-function-text-names.xhtml
>> > >
>> > > 4. Is there a reason not to cache user credentials in the broker? The
>> > > dynamic config mechanism already supports loading configs into
>> broker's
>> > > cache. Checking credentials from broker's cache is more efficient than
>> > > reading from ZK each time.
>> > >
>> > > 5. Typo "scram_iteration-4096" (= instead of -).
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > >
>> > >
>> > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
>> > > rajinisivaram@googlemail.com
>> > > > wrote:
>> > >
>> > > > Hi all,
>> > > >
>> > > > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
>> > > > mechanisms to Kafka:
>> > > >
>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > 84%3A+Support+SASL+SCRAM+mechanisms
>> > > >
>> > > >
>> > > > Comments and suggestions are welcome.
>> > > >
>> > > > Thank you...
>> > > >
>> > > > Regards,
>> > > >
>> > > > Rajini
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Regards,
>> >
>> > Rajini
>> >
>>
>
>
>
> --
> Regards,
>
> Rajini
>
--
Regards,
Rajini
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Rajini Sivaram <ra...@googlemail.com>.
Jun,
4) Agree, it does make the implementation simpler. Updated KIP.
5) Thank you, that looks neater. Updated KIP.
On Wed, Oct 26, 2016 at 6:59 PM, Jun Rao <ju...@confluent.io> wrote:
> Hi, Rajini,
>
> Thanks for the reply.
>
> 4. Implementation wise, it seems to me that it's simpler to read from the
> cache than reading directly from ZK since the config manager already
> propagates all config changes through ZK. Also, it's probably a good idea
> to limit the places in the code base that directly accesses ZK.
>
> 5. Yes, it seems that it makes sense to add the new SCRAM configurations to
> the existing /config/users/<encoded-user>. I am not sure how one would
> remove the SCRAM configurations in the example though since the properties
> specified in add-config is not the ones actually being stored. An
> alternative is to doing sth like the following. It may still feel a bit
> weird and I am wondering if there is a clearer approach.
>
> bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config
> 'scram_sha-256=[password=alice-secret,iterations=4096],scram_sha-1=
> [password=alice-secret,iterations=4096]' --entity-type users --entity-name
> alice
>
> bin/kafka-configs.sh --zookeeper localhost:2181 --alter --delete-config
> 'scram_sha-256,scram_sha-1' --entity-type users --entity-name alice
>
> Thanks,
>
> Jun
>
>
> On Wed, Oct 26, 2016 at 4:35 AM, Rajini Sivaram <
> rajinisivaram@googlemail.com> wrote:
>
> > Hi Jun,
> >
> > Thank you for reviewing the KIP. Answers below:
> >
> >
> > 1. Yes, agree, Updated KIP.
> > 2. User specifies a password and iteration count. kaka-configs.sh
> > generates a random salt and then generates StoredKey and ServerKey for
> > that
> > password using the same message formatter implementation used for
> SCRAM
> > authentication. I have added some more detail to the KIP (
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 84%3A+Support+SASL+SCRAM+mechanisms#KIP-84:SupportSASLSCRAMmechanisms-
> > Tools).
> > Does that answer the question?
> > 3. I started off thinking just one (SCRAM-SHA-256) and then thought
> > another one is required to make sure that the implementation can cope
> > with
> > multiple SCRAM mechanisms. But actually you are right, we can support
> > all.
> > I haven't added the old md2/md5 mechanisms that aren't very secure,
> but
> > I
> > have included all the SHA algorithms.
> > 4. Since credentials are only required when a connection is made, it
> > feels like we can just read the latest value from ZK rather than cache
> > all
> > users and keep them updated. Having said that, we can always add
> caching
> > later if we find that the overhead of reading from ZK every time is
> too
> > expensive. Since caching doesn't change any externals, this can be
> done
> > in
> > a JIRA later - would that be ok?
> > 5. Thanks, updated. I have changed the property names to include
> > mechanism. To avoid four separate properties per mechanism in ZK, I
> have
> > changed the format to use a single property (lower-case mechanism
> name)
> > with four values concatenated in a format similar to SCRAM messages.
> >
> > Do you think storing SCRAM credentials in /config/users/<encoded-user>
> > along with existing quota properties is okay? Or should they be under a
> > different path (eg. /config/credentials/<encoded-user>)? Or should they
> be
> > on a completely different path like ACLs with a separate tool instead of
> > reusing kaka-configs.sh?
> >
> > Thank you,
> >
> > Rajini
> >
> > On Tue, Oct 25, 2016 at 11:55 PM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Hi, Rajini,
> > >
> > > Thanks for the proposal. Looks good overall and seems quite useful
> (e.g.
> > > for supporting delegation tokens). A few comments/questions below.
> > >
> > > 1. For the ZK data format change, should we use the same convention as
> in
> > > KIP-55 to use encoded user name (i.e., /config/users/<encoded-user1>)?
> > >
> > > 2. For tooling, could you describe how user typically generates
> > > scam_server_key and scram_stored_key to be used by kafka-config.sh?
> > >
> > > 3. Is there a particular reason to only support sha1 and sha128? Should
> > we
> > > support more hashes listed below in the future?
> > > http://www.iana.org/assignments/hash-function-
> > > text-names/hash-function-text-names.xhtml
> > >
> > > 4. Is there a reason not to cache user credentials in the broker? The
> > > dynamic config mechanism already supports loading configs into broker's
> > > cache. Checking credentials from broker's cache is more efficient than
> > > reading from ZK each time.
> > >
> > > 5. Typo "scram_iteration-4096" (= instead of -).
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > >
> > > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > > rajinisivaram@googlemail.com
> > > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
> > > > mechanisms to Kafka:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 84%3A+Support+SASL+SCRAM+mechanisms
> > > >
> > > >
> > > > Comments and suggestions are welcome.
> > > >
> > > > Thank you...
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>
--
Regards,
Rajini
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Jun Rao <ju...@confluent.io>.
Hi, Rajini,
Thanks for the reply.
4. Implementation wise, it seems to me that it's simpler to read from the
cache than reading directly from ZK since the config manager already
propagates all config changes through ZK. Also, it's probably a good idea
to limit the places in the code base that directly accesses ZK.
5. Yes, it seems that it makes sense to add the new SCRAM configurations to
the existing /config/users/<encoded-user>. I am not sure how one would
remove the SCRAM configurations in the example though since the properties
specified in add-config is not the ones actually being stored. An
alternative is to doing sth like the following. It may still feel a bit
weird and I am wondering if there is a clearer approach.
bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config
'scram_sha-256=[password=alice-secret,iterations=4096],scram_sha-1=
[password=alice-secret,iterations=4096]' --entity-type users --entity-name
alice
bin/kafka-configs.sh --zookeeper localhost:2181 --alter --delete-config
'scram_sha-256,scram_sha-1' --entity-type users --entity-name alice
Thanks,
Jun
On Wed, Oct 26, 2016 at 4:35 AM, Rajini Sivaram <
rajinisivaram@googlemail.com> wrote:
> Hi Jun,
>
> Thank you for reviewing the KIP. Answers below:
>
>
> 1. Yes, agree, Updated KIP.
> 2. User specifies a password and iteration count. kaka-configs.sh
> generates a random salt and then generates StoredKey and ServerKey for
> that
> password using the same message formatter implementation used for SCRAM
> authentication. I have added some more detail to the KIP (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 84%3A+Support+SASL+SCRAM+mechanisms#KIP-84:SupportSASLSCRAMmechanisms-
> Tools).
> Does that answer the question?
> 3. I started off thinking just one (SCRAM-SHA-256) and then thought
> another one is required to make sure that the implementation can cope
> with
> multiple SCRAM mechanisms. But actually you are right, we can support
> all.
> I haven't added the old md2/md5 mechanisms that aren't very secure, but
> I
> have included all the SHA algorithms.
> 4. Since credentials are only required when a connection is made, it
> feels like we can just read the latest value from ZK rather than cache
> all
> users and keep them updated. Having said that, we can always add caching
> later if we find that the overhead of reading from ZK every time is too
> expensive. Since caching doesn't change any externals, this can be done
> in
> a JIRA later - would that be ok?
> 5. Thanks, updated. I have changed the property names to include
> mechanism. To avoid four separate properties per mechanism in ZK, I have
> changed the format to use a single property (lower-case mechanism name)
> with four values concatenated in a format similar to SCRAM messages.
>
> Do you think storing SCRAM credentials in /config/users/<encoded-user>
> along with existing quota properties is okay? Or should they be under a
> different path (eg. /config/credentials/<encoded-user>)? Or should they be
> on a completely different path like ACLs with a separate tool instead of
> reusing kaka-configs.sh?
>
> Thank you,
>
> Rajini
>
> On Tue, Oct 25, 2016 at 11:55 PM, Jun Rao <ju...@confluent.io> wrote:
>
> > Hi, Rajini,
> >
> > Thanks for the proposal. Looks good overall and seems quite useful (e.g.
> > for supporting delegation tokens). A few comments/questions below.
> >
> > 1. For the ZK data format change, should we use the same convention as in
> > KIP-55 to use encoded user name (i.e., /config/users/<encoded-user1>)?
> >
> > 2. For tooling, could you describe how user typically generates
> > scam_server_key and scram_stored_key to be used by kafka-config.sh?
> >
> > 3. Is there a particular reason to only support sha1 and sha128? Should
> we
> > support more hashes listed below in the future?
> > http://www.iana.org/assignments/hash-function-
> > text-names/hash-function-text-names.xhtml
> >
> > 4. Is there a reason not to cache user credentials in the broker? The
> > dynamic config mechanism already supports loading configs into broker's
> > cache. Checking credentials from broker's cache is more efficient than
> > reading from ZK each time.
> >
> > 5. Typo "scram_iteration-4096" (= instead of -).
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> > rajinisivaram@googlemail.com
> > > wrote:
> >
> > > Hi all,
> > >
> > > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
> > > mechanisms to Kafka:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 84%3A+Support+SASL+SCRAM+mechanisms
> > >
> > >
> > > Comments and suggestions are welcome.
> > >
> > > Thank you...
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> >
>
>
>
> --
> Regards,
>
> Rajini
>
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Rajini Sivaram <ra...@googlemail.com>.
Hi Jun,
Thank you for reviewing the KIP. Answers below:
1. Yes, agree, Updated KIP.
2. User specifies a password and iteration count. kaka-configs.sh
generates a random salt and then generates StoredKey and ServerKey for that
password using the same message formatter implementation used for SCRAM
authentication. I have added some more detail to the KIP (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-84%3A+Support+SASL+SCRAM+mechanisms#KIP-84:SupportSASLSCRAMmechanisms-Tools).
Does that answer the question?
3. I started off thinking just one (SCRAM-SHA-256) and then thought
another one is required to make sure that the implementation can cope with
multiple SCRAM mechanisms. But actually you are right, we can support all.
I haven't added the old md2/md5 mechanisms that aren't very secure, but I
have included all the SHA algorithms.
4. Since credentials are only required when a connection is made, it
feels like we can just read the latest value from ZK rather than cache all
users and keep them updated. Having said that, we can always add caching
later if we find that the overhead of reading from ZK every time is too
expensive. Since caching doesn't change any externals, this can be done in
a JIRA later - would that be ok?
5. Thanks, updated. I have changed the property names to include
mechanism. To avoid four separate properties per mechanism in ZK, I have
changed the format to use a single property (lower-case mechanism name)
with four values concatenated in a format similar to SCRAM messages.
Do you think storing SCRAM credentials in /config/users/<encoded-user>
along with existing quota properties is okay? Or should they be under a
different path (eg. /config/credentials/<encoded-user>)? Or should they be
on a completely different path like ACLs with a separate tool instead of
reusing kaka-configs.sh?
Thank you,
Rajini
On Tue, Oct 25, 2016 at 11:55 PM, Jun Rao <ju...@confluent.io> wrote:
> Hi, Rajini,
>
> Thanks for the proposal. Looks good overall and seems quite useful (e.g.
> for supporting delegation tokens). A few comments/questions below.
>
> 1. For the ZK data format change, should we use the same convention as in
> KIP-55 to use encoded user name (i.e., /config/users/<encoded-user1>)?
>
> 2. For tooling, could you describe how user typically generates
> scam_server_key and scram_stored_key to be used by kafka-config.sh?
>
> 3. Is there a particular reason to only support sha1 and sha128? Should we
> support more hashes listed below in the future?
> http://www.iana.org/assignments/hash-function-
> text-names/hash-function-text-names.xhtml
>
> 4. Is there a reason not to cache user credentials in the broker? The
> dynamic config mechanism already supports loading configs into broker's
> cache. Checking credentials from broker's cache is more efficient than
> reading from ZK each time.
>
> 5. Typo "scram_iteration-4096" (= instead of -).
>
> Thanks,
>
> Jun
>
>
>
> On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <
> rajinisivaram@googlemail.com
> > wrote:
>
> > Hi all,
> >
> > I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
> > mechanisms to Kafka:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 84%3A+Support+SASL+SCRAM+mechanisms
> >
> >
> > Comments and suggestions are welcome.
> >
> > Thank you...
> >
> > Regards,
> >
> > Rajini
> >
>
--
Regards,
Rajini
Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms
Posted by Jun Rao <ju...@confluent.io>.
Hi, Rajini,
Thanks for the proposal. Looks good overall and seems quite useful (e.g.
for supporting delegation tokens). A few comments/questions below.
1. For the ZK data format change, should we use the same convention as in
KIP-55 to use encoded user name (i.e., /config/users/<encoded-user1>)?
2. For tooling, could you describe how user typically generates
scam_server_key and scram_stored_key to be used by kafka-config.sh?
3. Is there a particular reason to only support sha1 and sha128? Should we
support more hashes listed below in the future?
http://www.iana.org/assignments/hash-function-text-names/hash-function-text-names.xhtml
4. Is there a reason not to cache user credentials in the broker? The
dynamic config mechanism already supports loading configs into broker's
cache. Checking credentials from broker's cache is more efficient than
reading from ZK each time.
5. Typo "scram_iteration-4096" (= instead of -).
Thanks,
Jun
On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram <rajinisivaram@googlemail.com
> wrote:
> Hi all,
>
> I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
> mechanisms to Kafka:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 84%3A+Support+SASL+SCRAM+mechanisms
>
>
> Comments and suggestions are welcome.
>
> Thank you...
>
> Regards,
>
> Rajini
>