You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Ismael Juma <is...@juma.me.uk> on 2016/12/21 20:56:17 UTC
KIP-103: Separation of Internal and External traffic
Hi all,
We've posted "KIP-103: Separation of Internal and External traffic" for
discussion:
*https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+traffic
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+traffic>*
Please take a look. Your feedback is appreciated.
Thanks,
Ismael
Re: KIP-103: Separation of Internal and External traffic
Posted by Ismael Juma <is...@juma.me.uk>.
I updated the KIP to cover point 1 as well.
If there are no additional comments, I intend to start a vote thread within
a day or two.
Thanks,
Ismael
On Wed, Jan 4, 2017 at 11:30 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Thanks for the review Jun. Replies follow.
>
> 1. That's a very good point. Adding a prefix to the JAAS entry name with a
> fallback to the name without the prefix is consistent with how the other
> configs are handled, so I'll update the KIP to mention that.
>
> 2. That was a typo, thanks for catching it. I didn't mean to include
> `listener_security_protocol_map` in `UpdateMetadataRequest` as it's not
> needed (as you said). Fixed.
>
> Ismael
>
> On Wed, Jan 4, 2017 at 10:13 PM, Jun Rao <ju...@confluent.io> wrote:
>
>> Hi, Ismael,
>>
>> Thanks for the proposal. Looks good to me overall. Just a couple of minor
>> comments.
>>
>> 1. Sasl also has some properties provided through the login context in the
>> jaas file. Do we want to extend that to allow different login context for
>> different protocol labels on the server side (e.g. Label1KafkaServer,
>> Label2KafkaServer)? This doesn't have to be implemented right away though,
>> as long as we have a plan on how to extend it in the future.
>>
>> 2. In the UpdateMetadataRequest protocol, could you describe what's in
>> listener_security_protocol_map?
>> Also, since we include protocol_label in endpoints, do we really need to
>> include listener_security_protocol_map?
>>
>> Jun
>>
>>
>>
>> On Wed, Jan 4, 2017 at 4:34 AM, Ismael Juma <is...@juma.me.uk> wrote:
>>
>> > Hi Colin,
>> >
>> > Thanks for the feedback. It's a good question regarding the name
>> `protocol
>> > label`. It was an easy starting name given that the security protocol
>> was
>> > replaced by a label in the listener string. However, I agree that it's
>> > perhaps not as clear as it could be. Maybe `listener key` would be a
>> better
>> > name? It makes it clear that it should be unique in a listeners list and
>> > that it's used to associate a listener to something else (like a
>> security
>> > protocol). Thoughts?
>> >
>> > Ismael
>> >
>> > On Wed, Jan 4, 2017 at 12:29 AM, Colin McCabe <cm...@apache.org>
>> wrote:
>> >
>> > > Good idea. It would be really nice to be able to constrain
>> replication
>> > > traffic to a specific interface or use different security settings.
>> > >
>> > > I'm having a little trouble understanding the "protocol label"
>> concept.
>> > > Clearly protocol labels map to protocols, but they also seem to
>> identify
>> > > particular types of traffic. Would it be more appropriate to call
>> these
>> > > "traffic types" or "endpoint types"? Or am I misunderstanding the
>> > > proposal?
>> > >
>> > > cheers,
>> > > Colin
>> > >
>> > >
>> > > On Thu, Dec 22, 2016, at 08:00, Ismael Juma wrote:
>> > > > I've updated the KIP to:
>> > > >
>> > > > 1. Include the ability to set different security configs depending
>> on
>> > the
>> > > > protocol label.
>> > > > 2. Include the mapping from protocol label to security protocol in
>> ZK
>> > and
>> > > > UpdateMetadataRequest.
>> > > > 3. More items in the "Rejected Alternatives" section.
>> > > > 4. Take into account old ZooKeeper-based consumers.
>> > > >
>> > > > Feedback is appreciated as always.
>> > > >
>> > > > I'm particularly interested in people's opinions on the config
>> format
>> > as
>> > > > I
>> > > > am still unsure when it comes to the proposed format versus the
>> first
>> > > > rejected alternative.
>> > > >
>> > > > Ismael
>> > > >
>> > > > On Wed, Dec 21, 2016 at 11:37 PM, Ismael Juma <is...@juma.me.uk>
>> > wrote:
>> > > >
>> > > > > Thanks Rajini.
>> > > > >
>> > > > > I agree that it's worth thinking about what a fully configurable
>> > label
>> > > > > would look like. I'll update the KIP.
>> > > > >
>> > > > > Ismael
>> > > > >
>> > > > > On 21 Dec 2016 10:53 pm, "Rajini Sivaram" <
>> rajinisivaram@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > Hi Ismael,
>> > > > >
>> > > > > Thank you for the KIP. This is a very useful change.
>> > > > >
>> > > > > Once you allow multiple interfaces with the same security
>> protocol,
>> > you
>> > > > > will soon also need to be able to configure protocol-specific
>> > > properties
>> > > > > for each of the interfaces. To use SSL on internal and external
>> > > networks,
>> > > > > you would almost definitely want different keystores with
>> different
>> > > > > hostname/IP addresses. Similarly for SASL, you might want to
>> enable
>> > > > > different mechanisms, use a different authentication server etc.
>> This
>> > > is
>> > > > > listed under future work.But it may be worth thinking about what a
>> > > fully
>> > > > > configurable 'label' looks like. Would every property now become a
>> > > list/map
>> > > > > like listeners - you would then end up with maps of lists for some
>> > > > > properties. It will good if all properties corresponding to a
>> label
>> > > > > including listener and advertised.listener are configured
>> > consistently
>> > > - if
>> > > > > that is possible,
>> > > > >
>> > > > >
>> > > > > On Wed, Dec 21, 2016 at 8:56 PM, Ismael Juma <is...@juma.me.uk>
>> > > wrote:
>> > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > We've posted "KIP-103: Separation of Internal and External
>> traffic"
>> > > for
>> > > > > > discussion:
>> > > > > >
>> > > > > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > > > 103%3A+Separation+of+Internal+and+External+traffic
>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > > > 103%3A+Separation+of+Internal+and+External+traffic>*
>> > > > > >
>> > > > > > Please take a look. Your feedback is appreciated.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Ismael
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > >
>> >
>>
>
>
Re: KIP-103: Separation of Internal and External traffic
Posted by Ismael Juma <is...@juma.me.uk>.
Thanks for the review Jun. Replies follow.
1. That's a very good point. Adding a prefix to the JAAS entry name with a
fallback to the name without the prefix is consistent with how the other
configs are handled, so I'll update the KIP to mention that.
2. That was a typo, thanks for catching it. I didn't mean to include
`listener_security_protocol_map` in `UpdateMetadataRequest` as it's not
needed (as you said). Fixed.
Ismael
On Wed, Jan 4, 2017 at 10:13 PM, Jun Rao <ju...@confluent.io> wrote:
> Hi, Ismael,
>
> Thanks for the proposal. Looks good to me overall. Just a couple of minor
> comments.
>
> 1. Sasl also has some properties provided through the login context in the
> jaas file. Do we want to extend that to allow different login context for
> different protocol labels on the server side (e.g. Label1KafkaServer,
> Label2KafkaServer)? This doesn't have to be implemented right away though,
> as long as we have a plan on how to extend it in the future.
>
> 2. In the UpdateMetadataRequest protocol, could you describe what's in
> listener_security_protocol_map?
> Also, since we include protocol_label in endpoints, do we really need to
> include listener_security_protocol_map?
>
> Jun
>
>
>
> On Wed, Jan 4, 2017 at 4:34 AM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Colin,
> >
> > Thanks for the feedback. It's a good question regarding the name
> `protocol
> > label`. It was an easy starting name given that the security protocol was
> > replaced by a label in the listener string. However, I agree that it's
> > perhaps not as clear as it could be. Maybe `listener key` would be a
> better
> > name? It makes it clear that it should be unique in a listeners list and
> > that it's used to associate a listener to something else (like a security
> > protocol). Thoughts?
> >
> > Ismael
> >
> > On Wed, Jan 4, 2017 at 12:29 AM, Colin McCabe <cm...@apache.org>
> wrote:
> >
> > > Good idea. It would be really nice to be able to constrain replication
> > > traffic to a specific interface or use different security settings.
> > >
> > > I'm having a little trouble understanding the "protocol label" concept.
> > > Clearly protocol labels map to protocols, but they also seem to
> identify
> > > particular types of traffic. Would it be more appropriate to call
> these
> > > "traffic types" or "endpoint types"? Or am I misunderstanding the
> > > proposal?
> > >
> > > cheers,
> > > Colin
> > >
> > >
> > > On Thu, Dec 22, 2016, at 08:00, Ismael Juma wrote:
> > > > I've updated the KIP to:
> > > >
> > > > 1. Include the ability to set different security configs depending on
> > the
> > > > protocol label.
> > > > 2. Include the mapping from protocol label to security protocol in ZK
> > and
> > > > UpdateMetadataRequest.
> > > > 3. More items in the "Rejected Alternatives" section.
> > > > 4. Take into account old ZooKeeper-based consumers.
> > > >
> > > > Feedback is appreciated as always.
> > > >
> > > > I'm particularly interested in people's opinions on the config format
> > as
> > > > I
> > > > am still unsure when it comes to the proposed format versus the first
> > > > rejected alternative.
> > > >
> > > > Ismael
> > > >
> > > > On Wed, Dec 21, 2016 at 11:37 PM, Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > >
> > > > > Thanks Rajini.
> > > > >
> > > > > I agree that it's worth thinking about what a fully configurable
> > label
> > > > > would look like. I'll update the KIP.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On 21 Dec 2016 10:53 pm, "Rajini Sivaram" <rajinisivaram@gmail.com
> >
> > > wrote:
> > > > >
> > > > > Hi Ismael,
> > > > >
> > > > > Thank you for the KIP. This is a very useful change.
> > > > >
> > > > > Once you allow multiple interfaces with the same security protocol,
> > you
> > > > > will soon also need to be able to configure protocol-specific
> > > properties
> > > > > for each of the interfaces. To use SSL on internal and external
> > > networks,
> > > > > you would almost definitely want different keystores with different
> > > > > hostname/IP addresses. Similarly for SASL, you might want to enable
> > > > > different mechanisms, use a different authentication server etc.
> This
> > > is
> > > > > listed under future work.But it may be worth thinking about what a
> > > fully
> > > > > configurable 'label' looks like. Would every property now become a
> > > list/map
> > > > > like listeners - you would then end up with maps of lists for some
> > > > > properties. It will good if all properties corresponding to a
> label
> > > > > including listener and advertised.listener are configured
> > consistently
> > > - if
> > > > > that is possible,
> > > > >
> > > > >
> > > > > On Wed, Dec 21, 2016 at 8:56 PM, Ismael Juma <is...@juma.me.uk>
> > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We've posted "KIP-103: Separation of Internal and External
> traffic"
> > > for
> > > > > > discussion:
> > > > > >
> > > > > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 103%3A+Separation+of+Internal+and+External+traffic
> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 103%3A+Separation+of+Internal+and+External+traffic>*
> > > > > >
> > > > > > Please take a look. Your feedback is appreciated.
> > > > > >
> > > > > > Thanks,
> > > > > > Ismael
> > > > > >
> > > > >
> > > > >
> > > > >
> > >
> >
>
Re: KIP-103: Separation of Internal and External traffic
Posted by Jun Rao <ju...@confluent.io>.
Hi, Ismael,
Thanks for the proposal. Looks good to me overall. Just a couple of minor
comments.
1. Sasl also has some properties provided through the login context in the
jaas file. Do we want to extend that to allow different login context for
different protocol labels on the server side (e.g. Label1KafkaServer,
Label2KafkaServer)? This doesn't have to be implemented right away though,
as long as we have a plan on how to extend it in the future.
2. In the UpdateMetadataRequest protocol, could you describe what's in
listener_security_protocol_map?
Also, since we include protocol_label in endpoints, do we really need to
include listener_security_protocol_map?
Jun
On Wed, Jan 4, 2017 at 4:34 AM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Colin,
>
> Thanks for the feedback. It's a good question regarding the name `protocol
> label`. It was an easy starting name given that the security protocol was
> replaced by a label in the listener string. However, I agree that it's
> perhaps not as clear as it could be. Maybe `listener key` would be a better
> name? It makes it clear that it should be unique in a listeners list and
> that it's used to associate a listener to something else (like a security
> protocol). Thoughts?
>
> Ismael
>
> On Wed, Jan 4, 2017 at 12:29 AM, Colin McCabe <cm...@apache.org> wrote:
>
> > Good idea. It would be really nice to be able to constrain replication
> > traffic to a specific interface or use different security settings.
> >
> > I'm having a little trouble understanding the "protocol label" concept.
> > Clearly protocol labels map to protocols, but they also seem to identify
> > particular types of traffic. Would it be more appropriate to call these
> > "traffic types" or "endpoint types"? Or am I misunderstanding the
> > proposal?
> >
> > cheers,
> > Colin
> >
> >
> > On Thu, Dec 22, 2016, at 08:00, Ismael Juma wrote:
> > > I've updated the KIP to:
> > >
> > > 1. Include the ability to set different security configs depending on
> the
> > > protocol label.
> > > 2. Include the mapping from protocol label to security protocol in ZK
> and
> > > UpdateMetadataRequest.
> > > 3. More items in the "Rejected Alternatives" section.
> > > 4. Take into account old ZooKeeper-based consumers.
> > >
> > > Feedback is appreciated as always.
> > >
> > > I'm particularly interested in people's opinions on the config format
> as
> > > I
> > > am still unsure when it comes to the proposed format versus the first
> > > rejected alternative.
> > >
> > > Ismael
> > >
> > > On Wed, Dec 21, 2016 at 11:37 PM, Ismael Juma <is...@juma.me.uk>
> wrote:
> > >
> > > > Thanks Rajini.
> > > >
> > > > I agree that it's worth thinking about what a fully configurable
> label
> > > > would look like. I'll update the KIP.
> > > >
> > > > Ismael
> > > >
> > > > On 21 Dec 2016 10:53 pm, "Rajini Sivaram" <ra...@gmail.com>
> > wrote:
> > > >
> > > > Hi Ismael,
> > > >
> > > > Thank you for the KIP. This is a very useful change.
> > > >
> > > > Once you allow multiple interfaces with the same security protocol,
> you
> > > > will soon also need to be able to configure protocol-specific
> > properties
> > > > for each of the interfaces. To use SSL on internal and external
> > networks,
> > > > you would almost definitely want different keystores with different
> > > > hostname/IP addresses. Similarly for SASL, you might want to enable
> > > > different mechanisms, use a different authentication server etc. This
> > is
> > > > listed under future work.But it may be worth thinking about what a
> > fully
> > > > configurable 'label' looks like. Would every property now become a
> > list/map
> > > > like listeners - you would then end up with maps of lists for some
> > > > properties. It will good if all properties corresponding to a label
> > > > including listener and advertised.listener are configured
> consistently
> > - if
> > > > that is possible,
> > > >
> > > >
> > > > On Wed, Dec 21, 2016 at 8:56 PM, Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > We've posted "KIP-103: Separation of Internal and External traffic"
> > for
> > > > > discussion:
> > > > >
> > > > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 103%3A+Separation+of+Internal+and+External+traffic
> > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 103%3A+Separation+of+Internal+and+External+traffic>*
> > > > >
> > > > > Please take a look. Your feedback is appreciated.
> > > > >
> > > > > Thanks,
> > > > > Ismael
> > > > >
> > > >
> > > >
> > > >
> >
>
Re: KIP-103: Separation of Internal and External traffic
Posted by Roger Hoover <ro...@gmail.com>.
Thank you, Ismael.
Sent from my iPhone
> On Jan 6, 2017, at 4:46 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> Thanks Roger. I asked around and it seems like `listener name` is what
> people found most intuitive in the context of configs. So, I have updated
> the KIP to use that.
>
> Ismael
>
>> On Fri, Jan 6, 2017 at 9:42 PM, Roger Hoover <ro...@gmail.com> wrote:
>>
>> Ismael,
>>
>> Listener id would also convey uniqueness but I'm ok with listener key as
>> well since it fits with the use of the term "map" in other properties.
>>
>> My initially feeling against the word key was that it seemed more natural
>> in documentation about Kafka allowing multiple listener (even with the
>> same protocol) and listeners are identified by name or ID. It seemed a
>> little more awkward to talk about listeners having keys as identifiers.
>> That fact that the listener ID is used as a key in config maps is secondary.
>>
>> Your suggestion for removing the protocol prefix makes sense. Listeners
>> must have a protocol but that ZooKeeper field is only meant to hold the
>> listener ID.
>>
>> Cheers,
>>
>> Roger
>>
>> Sent from my iPhone
>>
>>> On Jan 6, 2017, at 12:24 PM, Ismael Juma <is...@juma.me.uk> wrote:
>>>
>>> Hi Roger,
>>>
>>> I think `listener_key` makes it clear that it has to be unique per
>>> listener, so I prefer it a little over `listener_name`. Since the
>> existing
>>> config is called `listeners` instead of `protocol.listeners`, maybe we
>>> don't need the protocol prefix?
>>>
>>> Ismael
>>>
>>>> On Fri, Jan 6, 2017 at 7:48 PM, Roger Hoover <ro...@gmail.com>
>> wrote:
>>>>
>>>> Maybe it's clearer to to say protocol_listener_name? The proposed
>> config
>>>> allows you to name each listener and refer to their names in various
>>>> places.
>>>>
>>>>
>>>>
>>>>> On Wed, Jan 4, 2017 at 4:34 AM, Ismael Juma <is...@juma.me.uk> wrote:
>>>>>
>>>>> Hi Colin,
>>>>>
>>>>> Thanks for the feedback. It's a good question regarding the name
>>>> `protocol
>>>>> label`. It was an easy starting name given that the security protocol
>> was
>>>>> replaced by a label in the listener string. However, I agree that it's
>>>>> perhaps not as clear as it could be. Maybe `listener key` would be a
>>>> better
>>>>> name? It makes it clear that it should be unique in a listeners list
>> and
>>>>> that it's used to associate a listener to something else (like a
>> security
>>>>> protocol). Thoughts?
>>>>>
>>>>> Ismael
>>>>>
>>>>> On Wed, Jan 4, 2017 at 12:29 AM, Colin McCabe <cm...@apache.org>
>>>> wrote:
>>>>>
>>>>>> Good idea. It would be really nice to be able to constrain
>> replication
>>>>>> traffic to a specific interface or use different security settings.
>>>>>>
>>>>>> I'm having a little trouble understanding the "protocol label"
>> concept.
>>>>>> Clearly protocol labels map to protocols, but they also seem to
>>>> identify
>>>>>> particular types of traffic. Would it be more appropriate to call
>>>> these
>>>>>> "traffic types" or "endpoint types"? Or am I misunderstanding the
>>>>>> proposal?
>>>>>>
>>>>>> cheers,
>>>>>> Colin
>>>>>>
>>>>>>
>>>>>>> On Thu, Dec 22, 2016, at 08:00, Ismael Juma wrote:
>>>>>>> I've updated the KIP to:
>>>>>>>
>>>>>>> 1. Include the ability to set different security configs depending on
>>>>> the
>>>>>>> protocol label.
>>>>>>> 2. Include the mapping from protocol label to security protocol in ZK
>>>>> and
>>>>>>> UpdateMetadataRequest.
>>>>>>> 3. More items in the "Rejected Alternatives" section.
>>>>>>> 4. Take into account old ZooKeeper-based consumers.
>>>>>>>
>>>>>>> Feedback is appreciated as always.
>>>>>>>
>>>>>>> I'm particularly interested in people's opinions on the config format
>>>>> as
>>>>>>> I
>>>>>>> am still unsure when it comes to the proposed format versus the first
>>>>>>> rejected alternative.
>>>>>>>
>>>>>>> Ismael
>>>>>>>
>>>>>>> On Wed, Dec 21, 2016 at 11:37 PM, Ismael Juma <is...@juma.me.uk>
>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks Rajini.
>>>>>>>>
>>>>>>>> I agree that it's worth thinking about what a fully configurable
>>>>> label
>>>>>>>> would look like. I'll update the KIP.
>>>>>>>>
>>>>>>>> Ismael
>>>>>>>>
>>>>>>>> On 21 Dec 2016 10:53 pm, "Rajini Sivaram" <rajinisivaram@gmail.com
>>>>>
>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi Ismael,
>>>>>>>>
>>>>>>>> Thank you for the KIP. This is a very useful change.
>>>>>>>>
>>>>>>>> Once you allow multiple interfaces with the same security protocol,
>>>>> you
>>>>>>>> will soon also need to be able to configure protocol-specific
>>>>>> properties
>>>>>>>> for each of the interfaces. To use SSL on internal and external
>>>>>> networks,
>>>>>>>> you would almost definitely want different keystores with different
>>>>>>>> hostname/IP addresses. Similarly for SASL, you might want to enable
>>>>>>>> different mechanisms, use a different authentication server etc.
>>>> This
>>>>>> is
>>>>>>>> listed under future work.But it may be worth thinking about what a
>>>>>> fully
>>>>>>>> configurable 'label' looks like. Would every property now become a
>>>>>> list/map
>>>>>>>> like listeners - you would then end up with maps of lists for some
>>>>>>>> properties. It will good if all properties corresponding to a
>>>> label
>>>>>>>> including listener and advertised.listener are configured
>>>>> consistently
>>>>>> - if
>>>>>>>> that is possible,
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Dec 21, 2016 at 8:56 PM, Ismael Juma <is...@juma.me.uk>
>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> We've posted "KIP-103: Separation of Internal and External
>>>> traffic"
>>>>>> for
>>>>>>>>> discussion:
>>>>>>>>>
>>>>>>>>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>>> 103%3A+Separation+of+Internal+and+External+traffic
>>>>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>>> 103%3A+Separation+of+Internal+and+External+traffic>*
>>>>>>>>>
>>>>>>>>> Please take a look. Your feedback is appreciated.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Ismael
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>
Re: KIP-103: Separation of Internal and External traffic
Posted by Ismael Juma <is...@juma.me.uk>.
Thanks Roger. I asked around and it seems like `listener name` is what
people found most intuitive in the context of configs. So, I have updated
the KIP to use that.
Ismael
On Fri, Jan 6, 2017 at 9:42 PM, Roger Hoover <ro...@gmail.com> wrote:
> Ismael,
>
> Listener id would also convey uniqueness but I'm ok with listener key as
> well since it fits with the use of the term "map" in other properties.
>
> My initially feeling against the word key was that it seemed more natural
> in documentation about Kafka allowing multiple listener (even with the
> same protocol) and listeners are identified by name or ID. It seemed a
> little more awkward to talk about listeners having keys as identifiers.
> That fact that the listener ID is used as a key in config maps is secondary.
>
> Your suggestion for removing the protocol prefix makes sense. Listeners
> must have a protocol but that ZooKeeper field is only meant to hold the
> listener ID.
>
> Cheers,
>
> Roger
>
> Sent from my iPhone
>
> > On Jan 6, 2017, at 12:24 PM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > Hi Roger,
> >
> > I think `listener_key` makes it clear that it has to be unique per
> > listener, so I prefer it a little over `listener_name`. Since the
> existing
> > config is called `listeners` instead of `protocol.listeners`, maybe we
> > don't need the protocol prefix?
> >
> > Ismael
> >
> >> On Fri, Jan 6, 2017 at 7:48 PM, Roger Hoover <ro...@gmail.com>
> wrote:
> >>
> >> Maybe it's clearer to to say protocol_listener_name? The proposed
> config
> >> allows you to name each listener and refer to their names in various
> >> places.
> >>
> >>
> >>
> >>> On Wed, Jan 4, 2017 at 4:34 AM, Ismael Juma <is...@juma.me.uk> wrote:
> >>>
> >>> Hi Colin,
> >>>
> >>> Thanks for the feedback. It's a good question regarding the name
> >> `protocol
> >>> label`. It was an easy starting name given that the security protocol
> was
> >>> replaced by a label in the listener string. However, I agree that it's
> >>> perhaps not as clear as it could be. Maybe `listener key` would be a
> >> better
> >>> name? It makes it clear that it should be unique in a listeners list
> and
> >>> that it's used to associate a listener to something else (like a
> security
> >>> protocol). Thoughts?
> >>>
> >>> Ismael
> >>>
> >>> On Wed, Jan 4, 2017 at 12:29 AM, Colin McCabe <cm...@apache.org>
> >> wrote:
> >>>
> >>>> Good idea. It would be really nice to be able to constrain
> replication
> >>>> traffic to a specific interface or use different security settings.
> >>>>
> >>>> I'm having a little trouble understanding the "protocol label"
> concept.
> >>>> Clearly protocol labels map to protocols, but they also seem to
> >> identify
> >>>> particular types of traffic. Would it be more appropriate to call
> >> these
> >>>> "traffic types" or "endpoint types"? Or am I misunderstanding the
> >>>> proposal?
> >>>>
> >>>> cheers,
> >>>> Colin
> >>>>
> >>>>
> >>>>> On Thu, Dec 22, 2016, at 08:00, Ismael Juma wrote:
> >>>>> I've updated the KIP to:
> >>>>>
> >>>>> 1. Include the ability to set different security configs depending on
> >>> the
> >>>>> protocol label.
> >>>>> 2. Include the mapping from protocol label to security protocol in ZK
> >>> and
> >>>>> UpdateMetadataRequest.
> >>>>> 3. More items in the "Rejected Alternatives" section.
> >>>>> 4. Take into account old ZooKeeper-based consumers.
> >>>>>
> >>>>> Feedback is appreciated as always.
> >>>>>
> >>>>> I'm particularly interested in people's opinions on the config format
> >>> as
> >>>>> I
> >>>>> am still unsure when it comes to the proposed format versus the first
> >>>>> rejected alternative.
> >>>>>
> >>>>> Ismael
> >>>>>
> >>>>> On Wed, Dec 21, 2016 at 11:37 PM, Ismael Juma <is...@juma.me.uk>
> >>> wrote:
> >>>>>
> >>>>>> Thanks Rajini.
> >>>>>>
> >>>>>> I agree that it's worth thinking about what a fully configurable
> >>> label
> >>>>>> would look like. I'll update the KIP.
> >>>>>>
> >>>>>> Ismael
> >>>>>>
> >>>>>> On 21 Dec 2016 10:53 pm, "Rajini Sivaram" <rajinisivaram@gmail.com
> >>>
> >>>> wrote:
> >>>>>>
> >>>>>> Hi Ismael,
> >>>>>>
> >>>>>> Thank you for the KIP. This is a very useful change.
> >>>>>>
> >>>>>> Once you allow multiple interfaces with the same security protocol,
> >>> you
> >>>>>> will soon also need to be able to configure protocol-specific
> >>>> properties
> >>>>>> for each of the interfaces. To use SSL on internal and external
> >>>> networks,
> >>>>>> you would almost definitely want different keystores with different
> >>>>>> hostname/IP addresses. Similarly for SASL, you might want to enable
> >>>>>> different mechanisms, use a different authentication server etc.
> >> This
> >>>> is
> >>>>>> listed under future work.But it may be worth thinking about what a
> >>>> fully
> >>>>>> configurable 'label' looks like. Would every property now become a
> >>>> list/map
> >>>>>> like listeners - you would then end up with maps of lists for some
> >>>>>> properties. It will good if all properties corresponding to a
> >> label
> >>>>>> including listener and advertised.listener are configured
> >>> consistently
> >>>> - if
> >>>>>> that is possible,
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Dec 21, 2016 at 8:56 PM, Ismael Juma <is...@juma.me.uk>
> >>>> wrote:
> >>>>>>
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> We've posted "KIP-103: Separation of Internal and External
> >> traffic"
> >>>> for
> >>>>>>> discussion:
> >>>>>>>
> >>>>>>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>>>> 103%3A+Separation+of+Internal+and+External+traffic
> >>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>>>> 103%3A+Separation+of+Internal+and+External+traffic>*
> >>>>>>>
> >>>>>>> Please take a look. Your feedback is appreciated.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Ismael
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>
> >>
>
Re: KIP-103: Separation of Internal and External traffic
Posted by Roger Hoover <ro...@gmail.com>.
Ismael,
Listener id would also convey uniqueness but I'm ok with listener key as well since it fits with the use of the term "map" in other properties.
My initially feeling against the word key was that it seemed more natural in documentation about Kafka allowing multiple listener (even with the same protocol) and listeners are identified by name or ID. It seemed a little more awkward to talk about listeners having keys as identifiers. That fact that the listener ID is used as a key in config maps is secondary.
Your suggestion for removing the protocol prefix makes sense. Listeners must have a protocol but that ZooKeeper field is only meant to hold the listener ID.
Cheers,
Roger
Sent from my iPhone
> On Jan 6, 2017, at 12:24 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> Hi Roger,
>
> I think `listener_key` makes it clear that it has to be unique per
> listener, so I prefer it a little over `listener_name`. Since the existing
> config is called `listeners` instead of `protocol.listeners`, maybe we
> don't need the protocol prefix?
>
> Ismael
>
>> On Fri, Jan 6, 2017 at 7:48 PM, Roger Hoover <ro...@gmail.com> wrote:
>>
>> Maybe it's clearer to to say protocol_listener_name? The proposed config
>> allows you to name each listener and refer to their names in various
>> places.
>>
>>
>>
>>> On Wed, Jan 4, 2017 at 4:34 AM, Ismael Juma <is...@juma.me.uk> wrote:
>>>
>>> Hi Colin,
>>>
>>> Thanks for the feedback. It's a good question regarding the name
>> `protocol
>>> label`. It was an easy starting name given that the security protocol was
>>> replaced by a label in the listener string. However, I agree that it's
>>> perhaps not as clear as it could be. Maybe `listener key` would be a
>> better
>>> name? It makes it clear that it should be unique in a listeners list and
>>> that it's used to associate a listener to something else (like a security
>>> protocol). Thoughts?
>>>
>>> Ismael
>>>
>>> On Wed, Jan 4, 2017 at 12:29 AM, Colin McCabe <cm...@apache.org>
>> wrote:
>>>
>>>> Good idea. It would be really nice to be able to constrain replication
>>>> traffic to a specific interface or use different security settings.
>>>>
>>>> I'm having a little trouble understanding the "protocol label" concept.
>>>> Clearly protocol labels map to protocols, but they also seem to
>> identify
>>>> particular types of traffic. Would it be more appropriate to call
>> these
>>>> "traffic types" or "endpoint types"? Or am I misunderstanding the
>>>> proposal?
>>>>
>>>> cheers,
>>>> Colin
>>>>
>>>>
>>>>> On Thu, Dec 22, 2016, at 08:00, Ismael Juma wrote:
>>>>> I've updated the KIP to:
>>>>>
>>>>> 1. Include the ability to set different security configs depending on
>>> the
>>>>> protocol label.
>>>>> 2. Include the mapping from protocol label to security protocol in ZK
>>> and
>>>>> UpdateMetadataRequest.
>>>>> 3. More items in the "Rejected Alternatives" section.
>>>>> 4. Take into account old ZooKeeper-based consumers.
>>>>>
>>>>> Feedback is appreciated as always.
>>>>>
>>>>> I'm particularly interested in people's opinions on the config format
>>> as
>>>>> I
>>>>> am still unsure when it comes to the proposed format versus the first
>>>>> rejected alternative.
>>>>>
>>>>> Ismael
>>>>>
>>>>> On Wed, Dec 21, 2016 at 11:37 PM, Ismael Juma <is...@juma.me.uk>
>>> wrote:
>>>>>
>>>>>> Thanks Rajini.
>>>>>>
>>>>>> I agree that it's worth thinking about what a fully configurable
>>> label
>>>>>> would look like. I'll update the KIP.
>>>>>>
>>>>>> Ismael
>>>>>>
>>>>>> On 21 Dec 2016 10:53 pm, "Rajini Sivaram" <rajinisivaram@gmail.com
>>>
>>>> wrote:
>>>>>>
>>>>>> Hi Ismael,
>>>>>>
>>>>>> Thank you for the KIP. This is a very useful change.
>>>>>>
>>>>>> Once you allow multiple interfaces with the same security protocol,
>>> you
>>>>>> will soon also need to be able to configure protocol-specific
>>>> properties
>>>>>> for each of the interfaces. To use SSL on internal and external
>>>> networks,
>>>>>> you would almost definitely want different keystores with different
>>>>>> hostname/IP addresses. Similarly for SASL, you might want to enable
>>>>>> different mechanisms, use a different authentication server etc.
>> This
>>>> is
>>>>>> listed under future work.But it may be worth thinking about what a
>>>> fully
>>>>>> configurable 'label' looks like. Would every property now become a
>>>> list/map
>>>>>> like listeners - you would then end up with maps of lists for some
>>>>>> properties. It will good if all properties corresponding to a
>> label
>>>>>> including listener and advertised.listener are configured
>>> consistently
>>>> - if
>>>>>> that is possible,
>>>>>>
>>>>>>
>>>>>> On Wed, Dec 21, 2016 at 8:56 PM, Ismael Juma <is...@juma.me.uk>
>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> We've posted "KIP-103: Separation of Internal and External
>> traffic"
>>>> for
>>>>>>> discussion:
>>>>>>>
>>>>>>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>> 103%3A+Separation+of+Internal+and+External+traffic
>>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>> 103%3A+Separation+of+Internal+and+External+traffic>*
>>>>>>>
>>>>>>> Please take a look. Your feedback is appreciated.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Ismael
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>>
Re: KIP-103: Separation of Internal and External traffic
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Roger,
I think `listener_key` makes it clear that it has to be unique per
listener, so I prefer it a little over `listener_name`. Since the existing
config is called `listeners` instead of `protocol.listeners`, maybe we
don't need the protocol prefix?
Ismael
On Fri, Jan 6, 2017 at 7:48 PM, Roger Hoover <ro...@gmail.com> wrote:
> Maybe it's clearer to to say protocol_listener_name? The proposed config
> allows you to name each listener and refer to their names in various
> places.
>
>
>
> On Wed, Jan 4, 2017 at 4:34 AM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Colin,
> >
> > Thanks for the feedback. It's a good question regarding the name
> `protocol
> > label`. It was an easy starting name given that the security protocol was
> > replaced by a label in the listener string. However, I agree that it's
> > perhaps not as clear as it could be. Maybe `listener key` would be a
> better
> > name? It makes it clear that it should be unique in a listeners list and
> > that it's used to associate a listener to something else (like a security
> > protocol). Thoughts?
> >
> > Ismael
> >
> > On Wed, Jan 4, 2017 at 12:29 AM, Colin McCabe <cm...@apache.org>
> wrote:
> >
> > > Good idea. It would be really nice to be able to constrain replication
> > > traffic to a specific interface or use different security settings.
> > >
> > > I'm having a little trouble understanding the "protocol label" concept.
> > > Clearly protocol labels map to protocols, but they also seem to
> identify
> > > particular types of traffic. Would it be more appropriate to call
> these
> > > "traffic types" or "endpoint types"? Or am I misunderstanding the
> > > proposal?
> > >
> > > cheers,
> > > Colin
> > >
> > >
> > > On Thu, Dec 22, 2016, at 08:00, Ismael Juma wrote:
> > > > I've updated the KIP to:
> > > >
> > > > 1. Include the ability to set different security configs depending on
> > the
> > > > protocol label.
> > > > 2. Include the mapping from protocol label to security protocol in ZK
> > and
> > > > UpdateMetadataRequest.
> > > > 3. More items in the "Rejected Alternatives" section.
> > > > 4. Take into account old ZooKeeper-based consumers.
> > > >
> > > > Feedback is appreciated as always.
> > > >
> > > > I'm particularly interested in people's opinions on the config format
> > as
> > > > I
> > > > am still unsure when it comes to the proposed format versus the first
> > > > rejected alternative.
> > > >
> > > > Ismael
> > > >
> > > > On Wed, Dec 21, 2016 at 11:37 PM, Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > >
> > > > > Thanks Rajini.
> > > > >
> > > > > I agree that it's worth thinking about what a fully configurable
> > label
> > > > > would look like. I'll update the KIP.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On 21 Dec 2016 10:53 pm, "Rajini Sivaram" <rajinisivaram@gmail.com
> >
> > > wrote:
> > > > >
> > > > > Hi Ismael,
> > > > >
> > > > > Thank you for the KIP. This is a very useful change.
> > > > >
> > > > > Once you allow multiple interfaces with the same security protocol,
> > you
> > > > > will soon also need to be able to configure protocol-specific
> > > properties
> > > > > for each of the interfaces. To use SSL on internal and external
> > > networks,
> > > > > you would almost definitely want different keystores with different
> > > > > hostname/IP addresses. Similarly for SASL, you might want to enable
> > > > > different mechanisms, use a different authentication server etc.
> This
> > > is
> > > > > listed under future work.But it may be worth thinking about what a
> > > fully
> > > > > configurable 'label' looks like. Would every property now become a
> > > list/map
> > > > > like listeners - you would then end up with maps of lists for some
> > > > > properties. It will good if all properties corresponding to a
> label
> > > > > including listener and advertised.listener are configured
> > consistently
> > > - if
> > > > > that is possible,
> > > > >
> > > > >
> > > > > On Wed, Dec 21, 2016 at 8:56 PM, Ismael Juma <is...@juma.me.uk>
> > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We've posted "KIP-103: Separation of Internal and External
> traffic"
> > > for
> > > > > > discussion:
> > > > > >
> > > > > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 103%3A+Separation+of+Internal+and+External+traffic
> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 103%3A+Separation+of+Internal+and+External+traffic>*
> > > > > >
> > > > > > Please take a look. Your feedback is appreciated.
> > > > > >
> > > > > > Thanks,
> > > > > > Ismael
> > > > > >
> > > > >
> > > > >
> > > > >
> > >
> >
>
Re: KIP-103: Separation of Internal and External traffic
Posted by Roger Hoover <ro...@gmail.com>.
Maybe it's clearer to to say protocol_listener_name? The proposed config
allows you to name each listener and refer to their names in various places.
On Wed, Jan 4, 2017 at 4:34 AM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Colin,
>
> Thanks for the feedback. It's a good question regarding the name `protocol
> label`. It was an easy starting name given that the security protocol was
> replaced by a label in the listener string. However, I agree that it's
> perhaps not as clear as it could be. Maybe `listener key` would be a better
> name? It makes it clear that it should be unique in a listeners list and
> that it's used to associate a listener to something else (like a security
> protocol). Thoughts?
>
> Ismael
>
> On Wed, Jan 4, 2017 at 12:29 AM, Colin McCabe <cm...@apache.org> wrote:
>
> > Good idea. It would be really nice to be able to constrain replication
> > traffic to a specific interface or use different security settings.
> >
> > I'm having a little trouble understanding the "protocol label" concept.
> > Clearly protocol labels map to protocols, but they also seem to identify
> > particular types of traffic. Would it be more appropriate to call these
> > "traffic types" or "endpoint types"? Or am I misunderstanding the
> > proposal?
> >
> > cheers,
> > Colin
> >
> >
> > On Thu, Dec 22, 2016, at 08:00, Ismael Juma wrote:
> > > I've updated the KIP to:
> > >
> > > 1. Include the ability to set different security configs depending on
> the
> > > protocol label.
> > > 2. Include the mapping from protocol label to security protocol in ZK
> and
> > > UpdateMetadataRequest.
> > > 3. More items in the "Rejected Alternatives" section.
> > > 4. Take into account old ZooKeeper-based consumers.
> > >
> > > Feedback is appreciated as always.
> > >
> > > I'm particularly interested in people's opinions on the config format
> as
> > > I
> > > am still unsure when it comes to the proposed format versus the first
> > > rejected alternative.
> > >
> > > Ismael
> > >
> > > On Wed, Dec 21, 2016 at 11:37 PM, Ismael Juma <is...@juma.me.uk>
> wrote:
> > >
> > > > Thanks Rajini.
> > > >
> > > > I agree that it's worth thinking about what a fully configurable
> label
> > > > would look like. I'll update the KIP.
> > > >
> > > > Ismael
> > > >
> > > > On 21 Dec 2016 10:53 pm, "Rajini Sivaram" <ra...@gmail.com>
> > wrote:
> > > >
> > > > Hi Ismael,
> > > >
> > > > Thank you for the KIP. This is a very useful change.
> > > >
> > > > Once you allow multiple interfaces with the same security protocol,
> you
> > > > will soon also need to be able to configure protocol-specific
> > properties
> > > > for each of the interfaces. To use SSL on internal and external
> > networks,
> > > > you would almost definitely want different keystores with different
> > > > hostname/IP addresses. Similarly for SASL, you might want to enable
> > > > different mechanisms, use a different authentication server etc. This
> > is
> > > > listed under future work.But it may be worth thinking about what a
> > fully
> > > > configurable 'label' looks like. Would every property now become a
> > list/map
> > > > like listeners - you would then end up with maps of lists for some
> > > > properties. It will good if all properties corresponding to a label
> > > > including listener and advertised.listener are configured
> consistently
> > - if
> > > > that is possible,
> > > >
> > > >
> > > > On Wed, Dec 21, 2016 at 8:56 PM, Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > We've posted "KIP-103: Separation of Internal and External traffic"
> > for
> > > > > discussion:
> > > > >
> > > > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 103%3A+Separation+of+Internal+and+External+traffic
> > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 103%3A+Separation+of+Internal+and+External+traffic>*
> > > > >
> > > > > Please take a look. Your feedback is appreciated.
> > > > >
> > > > > Thanks,
> > > > > Ismael
> > > > >
> > > >
> > > >
> > > >
> >
>
Re: KIP-103: Separation of Internal and External traffic
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Colin,
Thanks for the feedback. It's a good question regarding the name `protocol
label`. It was an easy starting name given that the security protocol was
replaced by a label in the listener string. However, I agree that it's
perhaps not as clear as it could be. Maybe `listener key` would be a better
name? It makes it clear that it should be unique in a listeners list and
that it's used to associate a listener to something else (like a security
protocol). Thoughts?
Ismael
On Wed, Jan 4, 2017 at 12:29 AM, Colin McCabe <cm...@apache.org> wrote:
> Good idea. It would be really nice to be able to constrain replication
> traffic to a specific interface or use different security settings.
>
> I'm having a little trouble understanding the "protocol label" concept.
> Clearly protocol labels map to protocols, but they also seem to identify
> particular types of traffic. Would it be more appropriate to call these
> "traffic types" or "endpoint types"? Or am I misunderstanding the
> proposal?
>
> cheers,
> Colin
>
>
> On Thu, Dec 22, 2016, at 08:00, Ismael Juma wrote:
> > I've updated the KIP to:
> >
> > 1. Include the ability to set different security configs depending on the
> > protocol label.
> > 2. Include the mapping from protocol label to security protocol in ZK and
> > UpdateMetadataRequest.
> > 3. More items in the "Rejected Alternatives" section.
> > 4. Take into account old ZooKeeper-based consumers.
> >
> > Feedback is appreciated as always.
> >
> > I'm particularly interested in people's opinions on the config format as
> > I
> > am still unsure when it comes to the proposed format versus the first
> > rejected alternative.
> >
> > Ismael
> >
> > On Wed, Dec 21, 2016 at 11:37 PM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Thanks Rajini.
> > >
> > > I agree that it's worth thinking about what a fully configurable label
> > > would look like. I'll update the KIP.
> > >
> > > Ismael
> > >
> > > On 21 Dec 2016 10:53 pm, "Rajini Sivaram" <ra...@gmail.com>
> wrote:
> > >
> > > Hi Ismael,
> > >
> > > Thank you for the KIP. This is a very useful change.
> > >
> > > Once you allow multiple interfaces with the same security protocol, you
> > > will soon also need to be able to configure protocol-specific
> properties
> > > for each of the interfaces. To use SSL on internal and external
> networks,
> > > you would almost definitely want different keystores with different
> > > hostname/IP addresses. Similarly for SASL, you might want to enable
> > > different mechanisms, use a different authentication server etc. This
> is
> > > listed under future work.But it may be worth thinking about what a
> fully
> > > configurable 'label' looks like. Would every property now become a
> list/map
> > > like listeners - you would then end up with maps of lists for some
> > > properties. It will good if all properties corresponding to a label
> > > including listener and advertised.listener are configured consistently
> - if
> > > that is possible,
> > >
> > >
> > > On Wed, Dec 21, 2016 at 8:56 PM, Ismael Juma <is...@juma.me.uk>
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > We've posted "KIP-103: Separation of Internal and External traffic"
> for
> > > > discussion:
> > > >
> > > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 103%3A+Separation+of+Internal+and+External+traffic
> > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 103%3A+Separation+of+Internal+and+External+traffic>*
> > > >
> > > > Please take a look. Your feedback is appreciated.
> > > >
> > > > Thanks,
> > > > Ismael
> > > >
> > >
> > >
> > >
>
Re: KIP-103: Separation of Internal and External traffic
Posted by Colin McCabe <cm...@apache.org>.
Good idea. It would be really nice to be able to constrain replication
traffic to a specific interface or use different security settings.
I'm having a little trouble understanding the "protocol label" concept.
Clearly protocol labels map to protocols, but they also seem to identify
particular types of traffic. Would it be more appropriate to call these
"traffic types" or "endpoint types"? Or am I misunderstanding the
proposal?
cheers,
Colin
On Thu, Dec 22, 2016, at 08:00, Ismael Juma wrote:
> I've updated the KIP to:
>
> 1. Include the ability to set different security configs depending on the
> protocol label.
> 2. Include the mapping from protocol label to security protocol in ZK and
> UpdateMetadataRequest.
> 3. More items in the "Rejected Alternatives" section.
> 4. Take into account old ZooKeeper-based consumers.
>
> Feedback is appreciated as always.
>
> I'm particularly interested in people's opinions on the config format as
> I
> am still unsure when it comes to the proposed format versus the first
> rejected alternative.
>
> Ismael
>
> On Wed, Dec 21, 2016 at 11:37 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Thanks Rajini.
> >
> > I agree that it's worth thinking about what a fully configurable label
> > would look like. I'll update the KIP.
> >
> > Ismael
> >
> > On 21 Dec 2016 10:53 pm, "Rajini Sivaram" <ra...@gmail.com> wrote:
> >
> > Hi Ismael,
> >
> > Thank you for the KIP. This is a very useful change.
> >
> > Once you allow multiple interfaces with the same security protocol, you
> > will soon also need to be able to configure protocol-specific properties
> > for each of the interfaces. To use SSL on internal and external networks,
> > you would almost definitely want different keystores with different
> > hostname/IP addresses. Similarly for SASL, you might want to enable
> > different mechanisms, use a different authentication server etc. This is
> > listed under future work.But it may be worth thinking about what a fully
> > configurable 'label' looks like. Would every property now become a list/map
> > like listeners - you would then end up with maps of lists for some
> > properties. It will good if all properties corresponding to a label
> > including listener and advertised.listener are configured consistently - if
> > that is possible,
> >
> >
> > On Wed, Dec 21, 2016 at 8:56 PM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi all,
> > >
> > > We've posted "KIP-103: Separation of Internal and External traffic" for
> > > discussion:
> > >
> > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 103%3A+Separation+of+Internal+and+External+traffic
> > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 103%3A+Separation+of+Internal+and+External+traffic>*
> > >
> > > Please take a look. Your feedback is appreciated.
> > >
> > > Thanks,
> > > Ismael
> > >
> >
> >
> >
Re: KIP-103: Separation of Internal and External traffic
Posted by Ismael Juma <is...@juma.me.uk>.
I've updated the KIP to:
1. Include the ability to set different security configs depending on the
protocol label.
2. Include the mapping from protocol label to security protocol in ZK and
UpdateMetadataRequest.
3. More items in the "Rejected Alternatives" section.
4. Take into account old ZooKeeper-based consumers.
Feedback is appreciated as always.
I'm particularly interested in people's opinions on the config format as I
am still unsure when it comes to the proposed format versus the first
rejected alternative.
Ismael
On Wed, Dec 21, 2016 at 11:37 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Thanks Rajini.
>
> I agree that it's worth thinking about what a fully configurable label
> would look like. I'll update the KIP.
>
> Ismael
>
> On 21 Dec 2016 10:53 pm, "Rajini Sivaram" <ra...@gmail.com> wrote:
>
> Hi Ismael,
>
> Thank you for the KIP. This is a very useful change.
>
> Once you allow multiple interfaces with the same security protocol, you
> will soon also need to be able to configure protocol-specific properties
> for each of the interfaces. To use SSL on internal and external networks,
> you would almost definitely want different keystores with different
> hostname/IP addresses. Similarly for SASL, you might want to enable
> different mechanisms, use a different authentication server etc. This is
> listed under future work.But it may be worth thinking about what a fully
> configurable 'label' looks like. Would every property now become a list/map
> like listeners - you would then end up with maps of lists for some
> properties. It will good if all properties corresponding to a label
> including listener and advertised.listener are configured consistently - if
> that is possible,
>
>
> On Wed, Dec 21, 2016 at 8:56 PM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi all,
> >
> > We've posted "KIP-103: Separation of Internal and External traffic" for
> > discussion:
> >
> > *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 103%3A+Separation+of+Internal+and+External+traffic
> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 103%3A+Separation+of+Internal+and+External+traffic>*
> >
> > Please take a look. Your feedback is appreciated.
> >
> > Thanks,
> > Ismael
> >
>
>
>
Re: KIP-103: Separation of Internal and External traffic
Posted by Ismael Juma <is...@juma.me.uk>.
Thanks Rajini.
I agree that it's worth thinking about what a fully configurable label
would look like. I'll update the KIP.
Ismael
On 21 Dec 2016 10:53 pm, "Rajini Sivaram" <ra...@gmail.com> wrote:
Hi Ismael,
Thank you for the KIP. This is a very useful change.
Once you allow multiple interfaces with the same security protocol, you
will soon also need to be able to configure protocol-specific properties
for each of the interfaces. To use SSL on internal and external networks,
you would almost definitely want different keystores with different
hostname/IP addresses. Similarly for SASL, you might want to enable
different mechanisms, use a different authentication server etc. This is
listed under future work.But it may be worth thinking about what a fully
configurable 'label' looks like. Would every property now become a list/map
like listeners - you would then end up with maps of lists for some
properties. It will good if all properties corresponding to a label
including listener and advertised.listener are configured consistently - if
that is possible,
On Wed, Dec 21, 2016 at 8:56 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi all,
>
> We've posted "KIP-103: Separation of Internal and External traffic" for
> discussion:
>
> *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 103%3A+Separation+of+Internal+and+External+traffic
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 103%3A+Separation+of+Internal+and+External+traffic>*
>
> Please take a look. Your feedback is appreciated.
>
> Thanks,
> Ismael
>
Re: KIP-103: Separation of Internal and External traffic
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Ismael,
Thank you for the KIP. This is a very useful change.
Once you allow multiple interfaces with the same security protocol, you
will soon also need to be able to configure protocol-specific properties
for each of the interfaces. To use SSL on internal and external networks,
you would almost definitely want different keystores with different
hostname/IP addresses. Similarly for SASL, you might want to enable
different mechanisms, use a different authentication server etc. This is
listed under future work.But it may be worth thinking about what a fully
configurable 'label' looks like. Would every property now become a list/map
like listeners - you would then end up with maps of lists for some
properties. It will good if all properties corresponding to a label
including listener and advertised.listener are configured consistently - if
that is possible,
On Wed, Dec 21, 2016 at 8:56 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi all,
>
> We've posted "KIP-103: Separation of Internal and External traffic" for
> discussion:
>
> *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 103%3A+Separation+of+Internal+and+External+traffic
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 103%3A+Separation+of+Internal+and+External+traffic>*
>
> Please take a look. Your feedback is appreciated.
>
> Thanks,
> Ismael
>