You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Magnus Edenhill <ma...@edenhill.se> on 2016/05/03 00:41:59 UTC
Re: [DISCUSS] KIP-43: Kafka SASL enhancements
Rajini,
I think I found a small documentation error on the KIP-43 wiki page, it
says the SASL framing size is int16, but I believe it should be int32.
Can you verify?
Regards,
Magnus
2016-04-25 15:38 GMT+02:00 Rajini Sivaram <ra...@googlemail.com>:
> Magnus,
>
> I have updated KIP-43 to include a section with the handshake
> request/response format. Have also added some more text to distinguish the
> actual authentication flow from the Kafka handshake/request flow.
>
> Thank you,
>
> Rajini
>
>
> On Mon, Apr 25, 2016 at 3:41 AM, Magnus Edenhill <ma...@edenhill.se>
> wrote:
>
> > Rajini,
> >
> > the KIP wiki is a bit unclear on the protocol changes.
> > Could you document the proposed Kafka protocol requests&responses in the
> > standard format (as on "A guide to the Kafka protocol").
> > This information should also be added to that page when the KIP is
> > accepted.
> > I think it would also be good to clarify what SASL handshake means, if
> that
> > is the Kafka-leved SASL mechanism handshake or the opaque SASL data
> > handshake performed by the SASL libraries.
> >
> > Thanks,
> > Magnus
> >
> > 2016-04-19 17:20 GMT-07:00 Jun Rao <ju...@confluent.io>:
> >
> > > Just to close the loop on this. Discussed with Magnus offline on how
> > KIP-43
> > > and KIP-35 can play together. We agreed upon the following proposal.
> > >
> > > On a SASL port,
> > >
> > > client sends:
> > >
> > > ApiVersionRequest (optional), SaslHandshakeRequest, SASL tokens
> (size
> > > delimited as being done now), regular api requests
> > >
> > > client receives:
> > >
> > > ApiVersionResponse (optional), SaslHandshakeResponse, SASL tokens
> > (size
> > > delimited as being done now), regular api responses
> > >
> > > The format of SaslHandshakeRequest is what's currently described in
> > > KIP-43. There
> > > will be some minor tweaks on ApiVersionResponse, which Magnus will
> follow
> > > up in the KIP-35 thread itself.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, Apr 13, 2016 at 5:59 AM, Rajini Sivaram <
> > > rajinisivaram@googlemail.com> wrote:
> > >
> > > > I have updated the PR (https://github.com/apache/kafka/pull/812) and
> > > > KIP-43
> > > > to use standard Kafka format for the new request/response added by
> > > KIP-43.
> > > > I haven't changed the overall structure of the Java code. Feedback is
> > > > appreciated.
> > > >
> > > > Thanks,
> > > >
> > > > Rajini
> > > >
> > > > On Tue, Apr 12, 2016 at 3:52 PM, Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > >
> > > > > Hi Jun,
> > > > >
> > > > > Comments inline.
> > > > >
> > > > > On Mon, Apr 11, 2016 at 1:57 AM, Jun Rao <ju...@confluent.io> wrote:
> > > > >
> > > > > > Yes, that should be fine right? Since the new api key will start
> > with
> > > > a 0
> > > > > > byte, it actually guarantees that it's different from 0x60 (1st
> > byte
> > > in
> > > > > the
> > > > > > old protocol) even if we change the request version id in the
> > future.
> > > > >
> > > > >
> > > > > Yes, this is true. Also, the GSS API library will throw an
> exception
> > if
> > > > the
> > > > > first byte is not 0x60 (for the case where newer clients connect to
> > > older
> > > > > brokers):
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/frohoff/jdk8u-dev-jdk/blob/master/src/share/classes/sun/security/jgss/GSSHeader.java#L97
> > > > >
> > > > >
> > > > > And the DEFECTIVE_TOKEN status code is specified in both RFC
> 2743[1]
> > > and
> > > > > RFC 5653[2]. Section 3.1 of RFC 2743 specifies that the token tag
> > > > consists
> > > > > of the following elements, in order:
> > > > >
> > > > > 1. 0x60 -- Tag for [APPLICATION 0] SEQUENCE; indicates that
> > > > > -- constructed form, definite length encoding follows.
> > > > >
> > > > > 2. Token length octets ...
> > > > >
> > > > >
> > > > > Ismael
> > > > >
> > > > > [1] Generic Security Service Application Program Interface Version
> 2,
> > > > > Update 1: https://tools.ietf.org/html/rfc2743
> > > > > [2] Generic Security Service API Version 2: Java Bindings Update:
> > > > > https://tools.ietf.org/html/rfc5653
> > > > >
> > > > > Ismael
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> >
>
>
>
> --
> Regards,
>
> Rajini
>
Re: [DISCUSS] KIP-43: Kafka SASL enhancements
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Gwen,
Very good question! There's a PR here:
https://github.com/apache/kafka/pull/1232
:)
Ismael
On Tue, May 3, 2016 at 5:20 PM, Gwen Shapira <gw...@confluent.io> wrote:
> Are we planning on updating the security section in Kafka documentation?
>
> On Tue, May 3, 2016 at 12:18 AM, Rajini Sivaram
> <ra...@googlemail.com> wrote:
> > Magnus,
> >
> > Yes, you are absolutely right. I have fixed the wiki page. Thank you for
> > pointing it out.
> >
> > Regards,
> >
> > Rajini
> >
> > On Mon, May 2, 2016 at 11:41 PM, Magnus Edenhill <ma...@edenhill.se>
> wrote:
> >
> >> Rajini,
> >>
> >> I think I found a small documentation error on the KIP-43 wiki page, it
> >> says the SASL framing size is int16, but I believe it should be int32.
> >>
> >> Can you verify?
> >>
> >> Regards,
> >> Magnus
> >>
> >>
> >> 2016-04-25 15:38 GMT+02:00 Rajini Sivaram <rajinisivaram@googlemail.com
> >:
> >>
> >> > Magnus,
> >> >
> >> > I have updated KIP-43 to include a section with the handshake
> >> > request/response format. Have also added some more text to distinguish
> >> the
> >> > actual authentication flow from the Kafka handshake/request flow.
> >> >
> >> > Thank you,
> >> >
> >> > Rajini
> >> >
> >> >
> >> > On Mon, Apr 25, 2016 at 3:41 AM, Magnus Edenhill <ma...@edenhill.se>
> >> > wrote:
> >> >
> >> > > Rajini,
> >> > >
> >> > > the KIP wiki is a bit unclear on the protocol changes.
> >> > > Could you document the proposed Kafka protocol requests&responses in
> >> the
> >> > > standard format (as on "A guide to the Kafka protocol").
> >> > > This information should also be added to that page when the KIP is
> >> > > accepted.
> >> > > I think it would also be good to clarify what SASL handshake means,
> if
> >> > that
> >> > > is the Kafka-leved SASL mechanism handshake or the opaque SASL data
> >> > > handshake performed by the SASL libraries.
> >> > >
> >> > > Thanks,
> >> > > Magnus
> >> > >
> >> > > 2016-04-19 17:20 GMT-07:00 Jun Rao <ju...@confluent.io>:
> >> > >
> >> > > > Just to close the loop on this. Discussed with Magnus offline on
> how
> >> > > KIP-43
> >> > > > and KIP-35 can play together. We agreed upon the following
> proposal.
> >> > > >
> >> > > > On a SASL port,
> >> > > >
> >> > > > client sends:
> >> > > >
> >> > > > ApiVersionRequest (optional), SaslHandshakeRequest, SASL
> tokens
> >> > (size
> >> > > > delimited as being done now), regular api requests
> >> > > >
> >> > > > client receives:
> >> > > >
> >> > > > ApiVersionResponse (optional), SaslHandshakeResponse, SASL
> tokens
> >> > > (size
> >> > > > delimited as being done now), regular api responses
> >> > > >
> >> > > > The format of SaslHandshakeRequest is what's currently described
> in
> >> > > > KIP-43. There
> >> > > > will be some minor tweaks on ApiVersionResponse, which Magnus will
> >> > follow
> >> > > > up in the KIP-35 thread itself.
> >> > > >
> >> > > > Thanks,
> >> > > >
> >> > > > Jun
> >> > > >
> >> > > >
> >> > > > On Wed, Apr 13, 2016 at 5:59 AM, Rajini Sivaram <
> >> > > > rajinisivaram@googlemail.com> wrote:
> >> > > >
> >> > > > > I have updated the PR (https://github.com/apache/kafka/pull/812
> )
> >> and
> >> > > > > KIP-43
> >> > > > > to use standard Kafka format for the new request/response added
> by
> >> > > > KIP-43.
> >> > > > > I haven't changed the overall structure of the Java code.
> Feedback
> >> is
> >> > > > > appreciated.
> >> > > > >
> >> > > > > Thanks,
> >> > > > >
> >> > > > > Rajini
> >> > > > >
> >> > > > > On Tue, Apr 12, 2016 at 3:52 PM, Ismael Juma <ismael@juma.me.uk
> >
> >> > > wrote:
> >> > > > >
> >> > > > > > Hi Jun,
> >> > > > > >
> >> > > > > > Comments inline.
> >> > > > > >
> >> > > > > > On Mon, Apr 11, 2016 at 1:57 AM, Jun Rao <ju...@confluent.io>
> >> wrote:
> >> > > > > >
> >> > > > > > > Yes, that should be fine right? Since the new api key will
> >> start
> >> > > with
> >> > > > > a 0
> >> > > > > > > byte, it actually guarantees that it's different from 0x60
> (1st
> >> > > byte
> >> > > > in
> >> > > > > > the
> >> > > > > > > old protocol) even if we change the request version id in
> the
> >> > > future.
> >> > > > > >
> >> > > > > >
> >> > > > > > Yes, this is true. Also, the GSS API library will throw an
> >> > exception
> >> > > if
> >> > > > > the
> >> > > > > > first byte is not 0x60 (for the case where newer clients
> connect
> >> to
> >> > > > older
> >> > > > > > brokers):
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/frohoff/jdk8u-dev-jdk/blob/master/src/share/classes/sun/security/jgss/GSSHeader.java#L97
> >> > > > > >
> >> > > > > >
> >> > > > > > And the DEFECTIVE_TOKEN status code is specified in both RFC
> >> > 2743[1]
> >> > > > and
> >> > > > > > RFC 5653[2]. Section 3.1 of RFC 2743 specifies that the token
> tag
> >> > > > > consists
> >> > > > > > of the following elements, in order:
> >> > > > > >
> >> > > > > > 1. 0x60 -- Tag for [APPLICATION 0] SEQUENCE; indicates that
> >> > > > > > -- constructed form, definite length encoding follows.
> >> > > > > >
> >> > > > > > 2. Token length octets ...
> >> > > > > >
> >> > > > > >
> >> > > > > > Ismael
> >> > > > > >
> >> > > > > > [1] Generic Security Service Application Program Interface
> >> Version
> >> > 2,
> >> > > > > > Update 1: https://tools.ietf.org/html/rfc2743
> >> > > > > > [2] Generic Security Service API Version 2: Java Bindings
> Update:
> >> > > > > > https://tools.ietf.org/html/rfc5653
> >> > > > > >
> >> > > > > > Ismael
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > Regards,
> >> > > > >
> >> > > > > Rajini
> >> > > > >
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Regards,
> >> >
> >> > Rajini
> >> >
> >>
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
>
Re: [DISCUSS] KIP-43: Kafka SASL enhancements
Posted by Gwen Shapira <gw...@confluent.io>.
Are we planning on updating the security section in Kafka documentation?
On Tue, May 3, 2016 at 12:18 AM, Rajini Sivaram
<ra...@googlemail.com> wrote:
> Magnus,
>
> Yes, you are absolutely right. I have fixed the wiki page. Thank you for
> pointing it out.
>
> Regards,
>
> Rajini
>
> On Mon, May 2, 2016 at 11:41 PM, Magnus Edenhill <ma...@edenhill.se> wrote:
>
>> Rajini,
>>
>> I think I found a small documentation error on the KIP-43 wiki page, it
>> says the SASL framing size is int16, but I believe it should be int32.
>>
>> Can you verify?
>>
>> Regards,
>> Magnus
>>
>>
>> 2016-04-25 15:38 GMT+02:00 Rajini Sivaram <ra...@googlemail.com>:
>>
>> > Magnus,
>> >
>> > I have updated KIP-43 to include a section with the handshake
>> > request/response format. Have also added some more text to distinguish
>> the
>> > actual authentication flow from the Kafka handshake/request flow.
>> >
>> > Thank you,
>> >
>> > Rajini
>> >
>> >
>> > On Mon, Apr 25, 2016 at 3:41 AM, Magnus Edenhill <ma...@edenhill.se>
>> > wrote:
>> >
>> > > Rajini,
>> > >
>> > > the KIP wiki is a bit unclear on the protocol changes.
>> > > Could you document the proposed Kafka protocol requests&responses in
>> the
>> > > standard format (as on "A guide to the Kafka protocol").
>> > > This information should also be added to that page when the KIP is
>> > > accepted.
>> > > I think it would also be good to clarify what SASL handshake means, if
>> > that
>> > > is the Kafka-leved SASL mechanism handshake or the opaque SASL data
>> > > handshake performed by the SASL libraries.
>> > >
>> > > Thanks,
>> > > Magnus
>> > >
>> > > 2016-04-19 17:20 GMT-07:00 Jun Rao <ju...@confluent.io>:
>> > >
>> > > > Just to close the loop on this. Discussed with Magnus offline on how
>> > > KIP-43
>> > > > and KIP-35 can play together. We agreed upon the following proposal.
>> > > >
>> > > > On a SASL port,
>> > > >
>> > > > client sends:
>> > > >
>> > > > ApiVersionRequest (optional), SaslHandshakeRequest, SASL tokens
>> > (size
>> > > > delimited as being done now), regular api requests
>> > > >
>> > > > client receives:
>> > > >
>> > > > ApiVersionResponse (optional), SaslHandshakeResponse, SASL tokens
>> > > (size
>> > > > delimited as being done now), regular api responses
>> > > >
>> > > > The format of SaslHandshakeRequest is what's currently described in
>> > > > KIP-43. There
>> > > > will be some minor tweaks on ApiVersionResponse, which Magnus will
>> > follow
>> > > > up in the KIP-35 thread itself.
>> > > >
>> > > > Thanks,
>> > > >
>> > > > Jun
>> > > >
>> > > >
>> > > > On Wed, Apr 13, 2016 at 5:59 AM, Rajini Sivaram <
>> > > > rajinisivaram@googlemail.com> wrote:
>> > > >
>> > > > > I have updated the PR (https://github.com/apache/kafka/pull/812)
>> and
>> > > > > KIP-43
>> > > > > to use standard Kafka format for the new request/response added by
>> > > > KIP-43.
>> > > > > I haven't changed the overall structure of the Java code. Feedback
>> is
>> > > > > appreciated.
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > Rajini
>> > > > >
>> > > > > On Tue, Apr 12, 2016 at 3:52 PM, Ismael Juma <is...@juma.me.uk>
>> > > wrote:
>> > > > >
>> > > > > > Hi Jun,
>> > > > > >
>> > > > > > Comments inline.
>> > > > > >
>> > > > > > On Mon, Apr 11, 2016 at 1:57 AM, Jun Rao <ju...@confluent.io>
>> wrote:
>> > > > > >
>> > > > > > > Yes, that should be fine right? Since the new api key will
>> start
>> > > with
>> > > > > a 0
>> > > > > > > byte, it actually guarantees that it's different from 0x60 (1st
>> > > byte
>> > > > in
>> > > > > > the
>> > > > > > > old protocol) even if we change the request version id in the
>> > > future.
>> > > > > >
>> > > > > >
>> > > > > > Yes, this is true. Also, the GSS API library will throw an
>> > exception
>> > > if
>> > > > > the
>> > > > > > first byte is not 0x60 (for the case where newer clients connect
>> to
>> > > > older
>> > > > > > brokers):
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/frohoff/jdk8u-dev-jdk/blob/master/src/share/classes/sun/security/jgss/GSSHeader.java#L97
>> > > > > >
>> > > > > >
>> > > > > > And the DEFECTIVE_TOKEN status code is specified in both RFC
>> > 2743[1]
>> > > > and
>> > > > > > RFC 5653[2]. Section 3.1 of RFC 2743 specifies that the token tag
>> > > > > consists
>> > > > > > of the following elements, in order:
>> > > > > >
>> > > > > > 1. 0x60 -- Tag for [APPLICATION 0] SEQUENCE; indicates that
>> > > > > > -- constructed form, definite length encoding follows.
>> > > > > >
>> > > > > > 2. Token length octets ...
>> > > > > >
>> > > > > >
>> > > > > > Ismael
>> > > > > >
>> > > > > > [1] Generic Security Service Application Program Interface
>> Version
>> > 2,
>> > > > > > Update 1: https://tools.ietf.org/html/rfc2743
>> > > > > > [2] Generic Security Service API Version 2: Java Bindings Update:
>> > > > > > https://tools.ietf.org/html/rfc5653
>> > > > > >
>> > > > > > Ismael
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Regards,
>> > > > >
>> > > > > Rajini
>> > > > >
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Regards,
>> >
>> > Rajini
>> >
>>
>
>
>
> --
> Regards,
>
> Rajini
Re: [DISCUSS] KIP-43: Kafka SASL enhancements
Posted by Rajini Sivaram <ra...@googlemail.com>.
Magnus,
Yes, you are absolutely right. I have fixed the wiki page. Thank you for
pointing it out.
Regards,
Rajini
On Mon, May 2, 2016 at 11:41 PM, Magnus Edenhill <ma...@edenhill.se> wrote:
> Rajini,
>
> I think I found a small documentation error on the KIP-43 wiki page, it
> says the SASL framing size is int16, but I believe it should be int32.
>
> Can you verify?
>
> Regards,
> Magnus
>
>
> 2016-04-25 15:38 GMT+02:00 Rajini Sivaram <ra...@googlemail.com>:
>
> > Magnus,
> >
> > I have updated KIP-43 to include a section with the handshake
> > request/response format. Have also added some more text to distinguish
> the
> > actual authentication flow from the Kafka handshake/request flow.
> >
> > Thank you,
> >
> > Rajini
> >
> >
> > On Mon, Apr 25, 2016 at 3:41 AM, Magnus Edenhill <ma...@edenhill.se>
> > wrote:
> >
> > > Rajini,
> > >
> > > the KIP wiki is a bit unclear on the protocol changes.
> > > Could you document the proposed Kafka protocol requests&responses in
> the
> > > standard format (as on "A guide to the Kafka protocol").
> > > This information should also be added to that page when the KIP is
> > > accepted.
> > > I think it would also be good to clarify what SASL handshake means, if
> > that
> > > is the Kafka-leved SASL mechanism handshake or the opaque SASL data
> > > handshake performed by the SASL libraries.
> > >
> > > Thanks,
> > > Magnus
> > >
> > > 2016-04-19 17:20 GMT-07:00 Jun Rao <ju...@confluent.io>:
> > >
> > > > Just to close the loop on this. Discussed with Magnus offline on how
> > > KIP-43
> > > > and KIP-35 can play together. We agreed upon the following proposal.
> > > >
> > > > On a SASL port,
> > > >
> > > > client sends:
> > > >
> > > > ApiVersionRequest (optional), SaslHandshakeRequest, SASL tokens
> > (size
> > > > delimited as being done now), regular api requests
> > > >
> > > > client receives:
> > > >
> > > > ApiVersionResponse (optional), SaslHandshakeResponse, SASL tokens
> > > (size
> > > > delimited as being done now), regular api responses
> > > >
> > > > The format of SaslHandshakeRequest is what's currently described in
> > > > KIP-43. There
> > > > will be some minor tweaks on ApiVersionResponse, which Magnus will
> > follow
> > > > up in the KIP-35 thread itself.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Wed, Apr 13, 2016 at 5:59 AM, Rajini Sivaram <
> > > > rajinisivaram@googlemail.com> wrote:
> > > >
> > > > > I have updated the PR (https://github.com/apache/kafka/pull/812)
> and
> > > > > KIP-43
> > > > > to use standard Kafka format for the new request/response added by
> > > > KIP-43.
> > > > > I haven't changed the overall structure of the Java code. Feedback
> is
> > > > > appreciated.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Rajini
> > > > >
> > > > > On Tue, Apr 12, 2016 at 3:52 PM, Ismael Juma <is...@juma.me.uk>
> > > wrote:
> > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > Comments inline.
> > > > > >
> > > > > > On Mon, Apr 11, 2016 at 1:57 AM, Jun Rao <ju...@confluent.io>
> wrote:
> > > > > >
> > > > > > > Yes, that should be fine right? Since the new api key will
> start
> > > with
> > > > > a 0
> > > > > > > byte, it actually guarantees that it's different from 0x60 (1st
> > > byte
> > > > in
> > > > > > the
> > > > > > > old protocol) even if we change the request version id in the
> > > future.
> > > > > >
> > > > > >
> > > > > > Yes, this is true. Also, the GSS API library will throw an
> > exception
> > > if
> > > > > the
> > > > > > first byte is not 0x60 (for the case where newer clients connect
> to
> > > > older
> > > > > > brokers):
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/frohoff/jdk8u-dev-jdk/blob/master/src/share/classes/sun/security/jgss/GSSHeader.java#L97
> > > > > >
> > > > > >
> > > > > > And the DEFECTIVE_TOKEN status code is specified in both RFC
> > 2743[1]
> > > > and
> > > > > > RFC 5653[2]. Section 3.1 of RFC 2743 specifies that the token tag
> > > > > consists
> > > > > > of the following elements, in order:
> > > > > >
> > > > > > 1. 0x60 -- Tag for [APPLICATION 0] SEQUENCE; indicates that
> > > > > > -- constructed form, definite length encoding follows.
> > > > > >
> > > > > > 2. Token length octets ...
> > > > > >
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > [1] Generic Security Service Application Program Interface
> Version
> > 2,
> > > > > > Update 1: https://tools.ietf.org/html/rfc2743
> > > > > > [2] Generic Security Service API Version 2: Java Bindings Update:
> > > > > > https://tools.ietf.org/html/rfc5653
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>
--
Regards,
Rajini