You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Edoardo Comar <ed...@gmail.com> on 2018/09/11 14:00:58 UTC
Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS resolved
IP addresses
Hi all,
after some time we updated KIP-302 to reuse the config key introduced by
KIP-235, with a different value to avoid conflicts between the two.
Also clarified the use of multiple IPs only of the same type (IPv4/IPv6).
We'd appreciate a further review and discussion.
Thanks!
Edo
On Fri, 25 May 2018 at 12:36, Edoardo Comar <ed...@gmail.com> wrote:
> Hi Jonathan,
> I'm ok with an expandable enum for the config that could be extended
> in the future.
> It is marginally better than multiple potentially conflicting config
> entries.
>
> Though as I think the change for KIP-302 is independent from KIP-235
> and they do not conflict,
> when we'll look back at it post 2.0 we may see if it is more valuable
> to shoehorn its config in an expanded enum or not
>
> thanks,
> Edo
>
> On 24 May 2018 at 16:50, Skrzypek, Jonathan <Jo...@gs.com>
> wrote:
> > Hi,
> >
> > As Rajini suggested in the thread for KIP 235 (attached), we could try
> to have an enum that would drive what does the client expands/resolves.
> >
> > I suggest a client config called client.dns.lookup with different values
> possible :
> >
> > - no : no dns lookup
> > - hostnames.only : perform dns lookup on both bootstrap.servers and
> advertised listeners
> > - canonical.hostnames.only : perform dns lookup on both
> bootstrap.servers and advertised listeners
> > - bootstrap.hostnames.only : perform dns lookup on bootstrap.servers
> list and expand it
> > - bootstrap.canonical.hostnames.only : perform dns lookup on
> bootstrap.servers list and expand it
> > - advertised.listeners.hostnames.only : perform dns lookup on advertised
> listeners
> > - advertised.listeners.canonical.hostnames.only : perform dns lookup on
> advertised listeners
> >
> > I realize this is a bit heavy but this gives users the ability to pick
> and choose.
> > I didn't include a setting to mix hostnames and canonical hostnames as
> I'm not sure there would be a valid use case.
> >
> > Alternatively, to have less possible values, we could have 2 parameters :
> >
> > - dns.lookup.type with values : hostname / canonical.host.name
> > - dns.lookup.behaviour : bootstrap.servers, advertised.listeners, both
> >
> > Thoughts ?
> >
> > Jonathan Skrzypek
> >
> >
> > -----Original Message-----
> > From: Edoardo Comar [mailto:edocomar@gmail.com]
> > Sent: 17 May 2018 23:50
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS
> resolved IP addresses
> >
> > Hi Jonathan,
> >
> >> A solution might be to expose to users the choice of using hostname or
> canonical host name on both sides.
> >> Say having one setting that collapses functionalities from both KIPs
> (bootstrap expansion + advertised lookup)
> >> and an additional parameter that defines how the resolution is
> performed, using getCanonicalHostName() or not.
> >
> > thanks sounds to me *less* simple than independent config options, sorry.
> >
> > I would like to say once again that by itself KIP-302 only speeds up
> > the client behavior that can happen anyway when the client restarts
> > multiple times,
> > as every time there is no guarantee that - in presence of multiple A
> > DNS records - the same IP is returned. Attempting to use additiona IPs
> > if the first fail just makes client recovery faster.
> >
> > cheers
> > Edo
> >
> > On 17 May 2018 at 12:12, Skrzypek, Jonathan <Jo...@gs.com>
> wrote:
> >> Yes, makes sense.
> >> You mentioned multiple times you see no overlap and no issue with your
> KIP, and that they solve different use cases.
> >>
> >> Appreciate you have an existing use case that would work, but we need
> to make sure this isn't confusing to users and that any combination will
> always work, across security protocols.
> >>
> >> A solution might be to expose to users the choice of using hostname or
> canonical host name on both sides.
> >> Say having one setting that collapses functionalities from both KIPs
> (bootstrap expansion + advertised lookup) and an additional parameter that
> defines how the resolution is performed, using getCanonicalHostName() or
> not.
> >>
> >> Maybe that gives less flexibility as users wouldn't be able to decide
> to only perform DNS lookup on bootstrap.servers or on advertised listeners.
> >> But this would ensure consistency so that a user can decide to use
> cnames or not (depending on their certificates and Kerberos principals in
> their environment) and it would work.
> >>
> >> Jonathan Skrzypek
> >>
> >> -----Original Message-----
> >> From: Edoardo Comar [mailto:edocomar@gmail.com]
> >> Sent: 16 May 2018 21:59
> >> To: dev@kafka.apache.org
> >> Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS
> resolved IP addresses
> >>
> >> Hi Jonathan,
> >> I am afraid that may not work for everybody.
> >>
> >> It would not work for us.
> >> With our current DNS, my Kafka clients are perfectly happy to use any
> IPs -
> >> DNS has multiple A records for the 'myhostname.mydomain' used for
> >> bootstrap and advertised listeners.
> >> The hosts all serve TLS certificates that include
> >> 'myhostname.mydomain' and the clients are happy.
> >>
> >> However, applying getCanonicalHostName to those IPs would return
> >> hostnames that would not match the TLS certificates.
> >>
> >> So once again I believe your solution and ours solve different use
> cases.
> >>
> >> cheers
> >> Edo
> >>
> >> On 16 May 2018 at 18:29, Skrzypek, Jonathan <Jo...@gs.com>
> wrote:
> >>> I think there are combinations that will break SASL and SSL auth.
> >>> Could the trick be to have a single parameter that triggers dns
> resolve both for bootstrap and advertised listeners, both using
> getCanonicalHostName() ?
> >>>
> >>> Jonathan Skrzypek
> >>>
> >>> -----Original Message-----
> >>> From: Edoardo Comar [mailto:edocomar@gmail.com]
> >>> Sent: 16 May 2018 17:03
> >>> To: dev@kafka.apache.org
> >>> Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS
> resolved IP addresses
> >>>
> >>> Hi Rajini,
> >>>
> >>> In your example KIP-302 would attempt to connect to the first address
> >>> returned, let's say
> >>>
> >>> www.apache.org/195.154.151.36
> >>>
> >>> then, only if that fails, will in turn try the remaining:
> >>>
> >>> www.apache.org/40.79.78.1
> >>> www.apache.org/140.211.11.105
> >>> www.apache.org/2001:bc8:2142:300:0:0:0:0
> >>>
> >>> You're right to say that we expect certificates served by those
> >>> endpoints to be valid for "www.apache.org"
> >>>
> >>> Without KIP-302, only one would be attempted.
> >>> Which is the first one, that can change every time
> >>> (typically changes on every Java process restart,
> >>> but may change also any time InetAddress.getAllByName it's invoked
> >>> depending on the caching).
> >>>
> >>> The behavioral change that KIP-302 may introduce is that in the
> example above,
> >>> also an IPv6 connection may be attempted after some IPv4 ones.
> >>>
> >>> InetAddress.getAllByName() implementation uses a system property
> >>> "java.net.preferIPv6Addresses"
> >>> to decide which type of address to return first (default is still IPv4
> >>> in java 10)
> >>>
> >>> We will amend the KIP and PR so that the loop only uses IPs of the
> >>> same type as the first one returned.
> >>>
> >>> A part from the above, KIP 302 does not seem to change any existing
> >>> client behaviour, as any one of multiple IP addresses (of a given
> >>> v4/v6 type) can currently be picked.
> >>> We're happy however to keep the looping behavior optional with the
> >>> discussed config property, disabled by default.
> >>>
> >>> As for KIP-235 that may introduce new hostnames in the bootstrap list
> >>> (the current PR rewrites the bootstrap list)
> >>> and we fail to see the conflict with KIP-302, whatever the set of
> >>> configs chosen.
> >>>
> >>> We'd be happy to try understand what we are missing in a KIP call :-)
> >>>
> >>> cheers
> >>> Edo
> >>>
> >>> On 15 May 2018 at 16:58, Rajini Sivaram <ra...@gmail.com>
> wrote:
> >>>> Hi Edo,
> >>>>
> >>>> I agree that KIP-235 and KIP-302 address different scenarios. And I
> agree
> >>>> that each one is not sufficient in itself to address both the
> scenarios.
> >>>> But I also think that they conflict and hence they need to be looked
> at
> >>>> together and perhaps use a single config.
> >>>>
> >>>> As an example:
> >>>>
> >>>> If I run:
> >>>>
> >>>> for (InetAddress address : InetAddress.getAllByName("www.apache.org"))
> {
> >>>> System.out.printf("HostName %s canonicalHostName %s IP %s\n",
> >>>> address.getHostName(), address.getCanonicalHostName(),
> >>>> address.getHostAddress());
> >>>> }
> >>>>
> >>>> I get:
> >>>>
> >>>> HostName www.apache.org canonicalHostName tlp-eu-west.apache.org IP
> >>>> 195.154.151.36
> >>>> HostName www.apache.org canonicalHostName 40.79.78.1 IP 40.79.78.1
> >>>> HostName www.apache.org canonicalHostName themis.apache.org IP
> >>>> 140.211.11.105
> >>>> HostName www.apache.org canonicalHostName 2001:bc8:2142:300:0:0:0:0
> IP
> >>>> 2001:bc8:2142:300:0:0:0:0
> >>>>
> >>>>
> >>>> If www.apache.org is used as a bootstrap address, KIP-302 would
> connect to (
> >>>> www.apache.org/195.154.151.36 and www.apache.org/140.211.11.105)
> while
> >>>> KIP-235 would connect to (tlp-eu-west.apache.org/195.154.151.3. and
> >>>> themis.apache.org/140.211.11.105). This is a significant difference
> not
> >>>> just for Kerberos, but for any secure environment where hostname is
> >>>> verified to prevent man-in-the-middle attacks. In your case, I
> presume you
> >>>> would have SSL certificates with the equivalent of www.apache.org on
> both
> >>>> the load balancers. In Jonathan's case, I presume he has Kerberos
> >>>> principals for the equivalent of tlp-eu-west.apache.org and
> >>>> themis.apache.org. We would want to support both scenarios
> regardless of
> >>>> the security protocol, just need to come up with configuration
> options that
> >>>> don't conflict.
> >>>>
> >>>>
> >>>> On Mon, May 14, 2018 at 5:24 PM, Edoardo Comar <ed...@gmail.com>
> wrote:
> >>>>
> >>>>> Thanks Rajini
> >>>>>
> >>>>> I still don't see the overlap between the two KIPS
> >>>>>
> >>>>> KIP-235 allows an expansion of hostnames on the bootstrap list.
> >>>>>
> >>>>> KIP-302 allows alternative IPs to be used to attempt a connection
> >>>>> (either at bootstrap and when processing the MetadataResponse) to a
> >>>>> given hostname.
> >>>>>
> >>>>> A use case would be that of active/passive LB fronting the brokers.
> >>>>>
> >>>>> Arguably, if Java honored the DNS-set TTL, and the TTL was low and on
> >>>>> subsequent requests, the order of IPs returned by the
> >>>>> InetAddress.getAllByName() was random, we may not need such an
> >>>>> enhancement.
> >>>>> In practice, a Java client can get stuck on a "bad" IP forever if it
> >>>>> only relies on the first IP returned.
> >>>>>
> >>>>> HTH,
> >>>>> Edo
> >>>>>
> >>>>> On 14 May 2018 at 16:23, Rajini Sivaram <ra...@gmail.com>
> wrote:
> >>>>> > Hi Edo,
> >>>>> >
> >>>>> > Thanks for the KIP. I think it will be good to include a diagram
> to make
> >>>>> it
> >>>>> > easier to distinguish this scenario from that of KIP-235 without
> reading
> >>>>> > the PR.
> >>>>> >
> >>>>> > It may be worth considering if KIP-235 and this KIP could use a
> single
> >>>>> > config name with different values instead of two boolean configs:
> >>>>> >
> >>>>> > bootstrap.reverse.dns.lookup = true/false
> >>>>> > enable.all.dns.ips = true/false
> >>>>> >
> >>>>> > Not all values of (bootstrap.reverse.dns.lookup,
> enable.all.dns.ips) seem
> >>>>> > to make sense. And not all scenarios are handled. Even if we use
> multiple
> >>>>> > configs, it seems to me that we may want to name them differently.
> >>>>> >
> >>>>> > The possible combinations are:
> >>>>> >
> >>>>> > 1) Bootstrap
> >>>>> >
> >>>>> > a) No lookup
> >>>>> > b) Use all DNS entries with host name
> >>>>> > c) Use all DNS entries with canonical host name
> >>>>> >
> >>>>> > 2) Advertised listeners
> >>>>> >
> >>>>> > a) No lookup
> >>>>> > b) Use all DNS entries with host name
> >>>>> > c) Use all DNS entries with canonical host name
> >>>>> >
> >>>>> > The combinations that are enabled by the two boolean configs (
> >>>>> > bootstrap.reverse.dns.lookup, enable.all.dns.ips) are:
> >>>>> >
> >>>>> > - (false, false) => (1a, 2a)
> >>>>> > - (true, false) => (1c, 2a)
> >>>>> > - (false, true) => (1b, 2b)
> >>>>> > - (true, true) => (??, 2b)
> >>>>> >
> >>>>> > It will be good if we can clearly identify which combinations we
> want to
> >>>>> > support and the scenarios where they may be useful. Perhaps (1a,
> 2a),
> >>>>> (1c,
> >>>>> > 2a), (1b, 2b) and (1c, 2c) are useful?
> >>>>> >
> >>>>> >
> >>>>> > On Mon, May 14, 2018 at 2:58 PM, Skrzypek, Jonathan <
> >>>>> > Jonathan.Skrzypek@gs.com> wrote:
> >>>>> >
> >>>>> >> Ah, apologies didn't see there was already a decent amount of
> discussion
> >>>>> >> on this in the PR.
> >>>>> >>
> >>>>> >> This kind of sounds related to the environment you're running to
> me.
> >>>>> >> What is the rationale behind using the advertised listeners to do
> your
> >>>>> >> load balancing advertisement rather than a top level alias that
> has
> >>>>> >> everything ?
> >>>>> >>
> >>>>> >> It sounds like in your case there is a mismatch between
> >>>>> bootstrap.servers
> >>>>> >> and advertised.listeners, and you want advertised.listeners to
> take
> >>>>> >> precedence and have the client iterate over what is returned by
> the
> >>>>> broker.
> >>>>> >> So the extra parameter doesn't only have to do with DNS but it's
> also
> >>>>> >> appending from the broker, maybe the parameter name should
> reflect this
> >>>>> ?
> >>>>> >>
> >>>>> >> Jonathan Skrzypek
> >>>>> >>
> >>>>> >>
> >>>>> >> -----Original Message-----
> >>>>> >> From: Skrzypek, Jonathan [Tech]
> >>>>> >> Sent: 14 May 2018 14:46
> >>>>> >> To: dev@kafka.apache.org
> >>>>> >> Subject: RE: [DISCUSS] KIP-302 - Enable Kafka clients to use all
> DNS
> >>>>> >> resolved IP addresses
> >>>>> >>
> >>>>> >> Hi,
> >>>>> >>
> >>>>> >> I see you noted the similarities with KIP-235.
> >>>>> >> But KIP-235 might also solve what this KIP is trying to achieve.
> >>>>> >>
> >>>>> >> When parsing bootstrap.servers, KIP-235 has the client add all
> >>>>> underlying
> >>>>> >> hostnames and IPs.
> >>>>> >> And this happens before hitting the NetworkClient.
> >>>>> >>
> >>>>> >> So to me the client will try every single endpoint behind any
> >>>>> >> bootstrap.servers record.
> >>>>> >>
> >>>>> >> See
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_kafka_pull_4485_commits_24757eb7b0&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaCbPOM0NM1EHo-E&m=_ud9m_JZJ87C7eGsKcmzgJgDpNQDIIv5R4i_7VlhkLc&s=TqaiA9uW_myYO6FN-gKPfPlioxZR6DhnlBTpEj5M2aQ&e=
> >>>>> >>
> 6bcf8c7d7649c85232c52b5d54f0e4#diff-89ef153462e64c250a21bd324ae1a851
> >>>>> >> which calls getAllByName like you suggested
> >>>>> >>
> >>>>> >> Jonathan Skrzypek
> >>>>> >>
> >>>>> >>
> >>>>> >> -----Original Message-----
> >>>>> >> From: Edoardo Comar [mailto:edocomar@gmail.com]
> >>>>> >> Sent: 14 May 2018 14:17
> >>>>> >> To: dev@kafka.apache.org
> >>>>> >> Subject: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS
> >>>>> resolved
> >>>>> >> IP addresses
> >>>>> >>
> >>>>> >> Hi all,
> >>>>> >>
> >>>>> >> We just opened a KIP to add support for the client to use all IPs
> >>>>> returned
> >>>>> >> by DNS for the brokers
> >>>>> >>
> >>>>> >> The details are here -
> >>>>> >>
> >>>>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.a
> >>>>> >> pache.org_confluence_display_KAFKA_KIP-2D302-2B-2D-2BEnable-
> >>>>> >> 2BKafka-2Bclients-2Bto-2Buse-2Ball-2BDNS-2Bresolved-2BIP-
> >>>>> >> 2Baddresses&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxL
> >>>>> >> xfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaCbPOM0NM1EHo-E&m=EJafF
> >>>>> >> l1clRyolgtcu2uCc4_cIOJnlxb1r1n-D2Dti4k&s=C-UZ6KUG7JFiPD_
> >>>>> >> CnHczDOVqH9-XC5f_OFkw4BTNrI4&e=
> >>>>> >>
> >>>>> >> The JIRA and provisional PR (where the discussion lead to the
> creation
> >>>>> of
> >>>>> >> this KIP) are :
> >>>>> >>
> >>>>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> >>>>> >> apache.org_jira_browse_KAFKA-2D6863&d=DwIBaQ&c=7563p3e2zaQw0
> >>>>> >> AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6
> >>>>> >> eaCbPOM0NM1EHo-E&m=EJafFl1clRyolgtcu2uCc4_cIOJnlxb1r1n-
> >>>>> >> D2Dti4k&s=3Puqs5iYoPsw6hARQr6gvokdFE-H5USMiNVGOUtNkJI&e=
> >>>>> >>
> >>>>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> >>>>> >> com_apache_kafka_pull_4987&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgy
> >>>>> >> agb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaC
> >>>>> >> bPOM0NM1EHo-E&m=EJafFl1clRyolgtcu2uCc4_cIOJnlxb1r1n-D2Dti4k&
> >>>>> >> s=Hqn5dOgQy4-MHTIJLE49O8bNomry3SoGq9OVoHU-CRA&e=
> >>>>> >>
> >>>>> >> Looking forward to the community's feedback.
> >>>>> >> It would be amazing to have it voted by May 22nd :-) :-)
> >>>>> >>
> >>>>> >> Edoardo & Mickael
> >>>>> >>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> "When the people fear their government, there is tyranny; when the
> >>>>> government fears the people, there is liberty." [Thomas Jefferson]
> >>>>>
> >>>
> >>>
> >>>
> >>> --
> >>> "When the people fear their government, there is tyranny; when the
> >>> government fears the people, there is liberty." [Thomas Jefferson]
> >>
> >>
> >>
> >> --
> >> "When the people fear their government, there is tyranny; when the
> >> government fears the people, there is liberty." [Thomas Jefferson]
> >
> >
> >
> > --
> > "When the people fear their government, there is tyranny; when the
> > government fears the people, there is liberty." [Thomas Jefferson]
> >
> > ________________________________
> >
> > Your Personal Data: We may collect and process information about you
> that may be subject to data protection laws. For more information about how
> we use and disclose your personal data, how we protect your information,
> our legal basis to use your information, your rights and who you can
> contact, please refer to: www.gs.com/privacy-notices<
> http://www.gs.com/privacy-notices>
> >
> >
> > ---------- Forwarded message ----------
> > From: Rajini Sivaram <ra...@gmail.com>
> > To: "Skrzypek, Jonathan" <Jo...@ln.email.gs.com>, dev <
> dev@kafka.apache.org>, Ismael Juma <is...@juma.me.uk>
> > Cc:
> > Bcc:
> > Date: Tue, 22 May 2018 15:05:07 +0000
> > Subject: Re: FW: [VOTE] KIP-235 Add DNS alias support for secured
> connection
> > Hi Jonathan,
> >
> > I think it would make sense to convert the config in this KIP into an
> enum so that we can add more variations later on. But since KIP-302 is
> still under discussion, it is not clear what the config name should be.
> Since today is the KIP deadline and the implementation itself is
> straightforward, it would make sense to progress with this one for 2.0.0 if
> we can get one more binding vote.
> >
> > Ismael, do you have time to take a look at KIP-235 today?
> >
> > Thanks,
> >
> > Rajini
> >
> >
> > On Tue, May 22, 2018 at 3:45 PM, Skrzypek, Jonathan <
> Jonathan.Skrzypek@gs.com> wrote:
> >>
> >> Hello Rajini,
> >>
> >> What do you think should be the next step here ?
> >>
> >>
> >> Jonathan Skrzypek
> >>
> >> -----Original Message-----
> >> From: Skrzypek, Jonathan [Tech]
> >> Sent: 21 May 2018 10:51
> >> To: 'dev'
> >> Subject: RE: [VOTE] KIP-235 Add DNS alias support for secured connection
> >>
> >> Hi,
> >>
> >> What would be the next step here ?
> >> I know there's a discussion going on around KIP-302, but I'm also
> conscious that the 2.0.0 deadline for KIPs is tomorrow.
> >> I've opened this KIP in January and discussions have been productive
> with an end solution I had the impression was reasonable, so I am keen to
> see it make it the next release.
> >>
> >>
> >> Jonathan Skrzypek
> >>
> >> -----Original Message-----
> >> From: Skrzypek, Jonathan [Tech]
> >> Sent: 14 May 2018 13:48
> >> To: dev
> >> Subject: RE: [VOTE] KIP-235 Add DNS alias support for secured connection
> >>
> >> Sure, I modified the KIP to add more details
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-235%3A+Add+DNS+alias+support+for+secured+connection
> >>
> >>
> >> Jonathan Skrzypek
> >>
> >>
> >> -----Original Message-----
> >> From: Ismael Juma [mailto:ismael@juma.me.uk]
> >> Sent: 14 May 2018 11:53
> >> To: dev
> >> Subject: Re: [VOTE] KIP-235 Add DNS alias support for secured connection
> >>
> >> Thanks for the KIP, Jonathan. It would be helpful to have more detail on
> >> how SSL authentication could be broken if the new behaviour is the
> default.
> >> I know this was discussed in the mailing list thread, but it's
> important to
> >> include it in the KIP since it's the main reason why a new config is
> needed
> >> (and configs should be avoided whenever we can just do the right thing).
> >>
> >> Ismael
> >>
> >> On Fri, Mar 23, 2018 at 12:05 PM Skrzypek, Jonathan <
> >> Jonathan.Skrzypek@gs.com> wrote:
> >>
> >> > Hi,
> >> >
> >> > I would like to start a vote for KIP-235
> >> >
> >> >
> >> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D235-253A-2BAdd-2BDNS-2Balias-2Bsupport-2Bfor-2Bsecured-2Bconnection&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaCbPOM0NM1EHo-E&m=FM_uCHnnO2dqxWC0bi7_QOJKfKmQI80-Xduvb-URWOw&s=RpGkijfK-WHcU0s8ZtMXEkIr69QraJhYKaGSC9V_rnI&e=
> >> >
> >> > This is a proposition to add an option for reverse dns lookup of
> >> > bootstrap.servers hosts, allowing the use of dns aliases on clusters
> using
> >> > SASL authentication.
> >> >
> >> >
> >> >
> >> >
> >>
> >> ________________________________
> >>
> >> Your Personal Data: We may collect and process information about you
> that may be subject to data protection laws. For more information about how
> we use and disclose your personal data, how we protect your information,
> our legal basis to use your information, your rights and who you can
> contact, please refer to: www.gs.com/privacy-notices<
> http://www.gs.com/privacy-notices>
> >
> >
> >
>
>
>
> --
> "When the people fear their government, there is tyranny; when the
> government fears the people, there is liberty." [Thomas Jefferson]
>
--
"When the people fear their government, there is tyranny; when the
government fears the people, there is liberty." [Thomas Jefferson]
Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS resolved IP
addresses
Posted by Edoardo Comar <EC...@uk.ibm.com>.
Yes I should, should't I ? :-)
--------------------------------------------------
Edoardo Comar
IBM Message Hub
IBM UK Ltd, Hursley Park, SO21 2JN
From: Rajini Sivaram <ra...@gmail.com>
To: dev <de...@kafka.apache.org>
Date: 18/09/2018 15:33
Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all
DNS resolved IP addresses
Hi Edo,
The KIP looks good, perhaps you can start vote on this to get it in time
for 2.1.0?
Regards,
Rajini
On Tue, Sep 18, 2018 at 1:58 PM, Eno Thereska <en...@gmail.com>
wrote:
> Got it, thanks!
>
> Eno
>
> On Tue, Sep 18, 2018 at 1:41 PM, Edoardo Comar <EC...@uk.ibm.com>
wrote:
>
> > Hi Eno,
> >
> > we implement network separation and the machines fronting the brokers
> > (call them LBs for simplicity) will route the connection to the
correct
> > broker based on the TLS SNI. We register in the DNS multiple A records
> > (the IPs of all LBs) for each the brokers' hostnames.
> > As long as all the brokers are up, the cluster is fully functioning
even
> > if just one of the LBs is up.
> >
> > HTH,
> > Edo
> > --------------------------------------------------
> >
> > Edoardo Comar
> >
> > IBM Message Hub
> >
> > IBM UK Ltd, Hursley Park, SO21 2JN
> >
> >
> >
> > From: Eno Thereska <en...@gmail.com>
> > To: dev@kafka.apache.org
> > Date: 18/09/2018 10:24
> > Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use
all
> > DNS resolved IP addresses
> >
> >
> >
> > Hi folks,
> >
> > Could you expand the motivation a bit? When would it make sense to use
an
> > LB in front of Kafka brokers? A client needs to access each broker
> > directly
> > to consume the data in that broker and cannot be redirected to another
> > broker. What exact scenario are you seeing that needs this KIP?
> >
> > Thanks
> > Eno
> >
> > On Tue, Sep 18, 2018 at 9:52 AM, Mickael Maison <
> mickael.maison@gmail.com>
> > wrote:
> >
> > > Bumping this thread
> > >
> > > It's a relatively small change that would help cloud environments
with
> > > load balancers fronting brokers
> > > On Tue, Sep 11, 2018 at 3:01 PM Edoardo Comar <ed...@gmail.com>
> > wrote:
> > > >
> > > > Hi all,
> > > > after some time we updated KIP-302 to reuse the config key
introduced
> > by
> > > > KIP-235, with a different value to avoid conflicts between the
two.
> > > > Also clarified the use of multiple IPs only of the same type
> > (IPv4/IPv6).
> > > >
> > > > We'd appreciate a further review and discussion.
> > > > Thanks!
> > > > Edo
> > > >
> > > >
> > > > On Fri, 25 May 2018 at 12:36, Edoardo Comar <ed...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Jonathan,
> > > > > I'm ok with an expandable enum for the config that could be
> extended
> > > > > in the future.
> > > > > It is marginally better than multiple potentially conflicting
> config
> > > > > entries.
> > > > >
> > > > > Though as I think the change for KIP-302 is independent from
> KIP-235
> > > > > and they do not conflict,
> > > > > when we'll look back at it post 2.0 we may see if it is more
> > valuable
> > > > > to shoehorn its config in an expanded enum or not
> > > > >
> > > > > thanks,
> > > > > Edo
> > > > >
> > > > > On 24 May 2018 at 16:50, Skrzypek, Jonathan
> > <Jo...@gs.com>
> > > > > wrote:
> > > > > > Hi,
> > > > > >
> > > > > > As Rajini suggested in the thread for KIP 235 (attached), we
> could
> > > try
> > > > > to have an enum that would drive what does the client
> > expands/resolves.
> > > > > >
> > > > > > I suggest a client config called client.dns.lookup with
different
> > > values
> > > > > possible :
> > > > > >
> > > > > > - no : no dns lookup
> > > > > > - hostnames.only : perform dns lookup on both
bootstrap.servers
> > and
> > > > > advertised listeners
> > > > > > - canonical.hostnames.only : perform dns lookup on both
> > > > > bootstrap.servers and advertised listeners
> > > > > > - bootstrap.hostnames.only : perform dns lookup on
> > bootstrap.servers
> > > > > list and expand it
> > > > > > - bootstrap.canonical.hostnames.only : perform dns lookup on
> > > > > bootstrap.servers list and expand it
> > > > > > - advertised.listeners.hostnames.only : perform dns lookup on
> > > advertised
> > > > > listeners
> > > > > > - advertised.listeners.canonical.hostnames.only : perform dns
> > > lookup on
> > > > > advertised listeners
> > > > > >
> > > > > > I realize this is a bit heavy but this gives users the ability
to
> > > pick
> > > > > and choose.
> > > > > > I didn't include a setting to mix hostnames and canonical
> > hostnames
> > > as
> > > > > I'm not sure there would be a valid use case.
> > > > > >
> > > > > > Alternatively, to have less possible values, we could have 2
> > > parameters :
> > > > > >
> > > > > > - dns.lookup.type with values : hostname / canonical.host.name
> > > > > > - dns.lookup.behaviour : bootstrap.servers,
advertised.listeners,
> > > both
> > > > > >
> > > > > > Thoughts ?
> > > > > >
> > > > > > Jonathan Skrzypek
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > > > > Sent: 17 May 2018 23:50
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use
all
> > DNS
> > > > > resolved IP addresses
> > > > > >
> > > > > > Hi Jonathan,
> > > > > >
> > > > > >> A solution might be to expose to users the choice of using
> > hostname
> > > or
> > > > > canonical host name on both sides.
> > > > > >> Say having one setting that collapses functionalities from
both
> > KIPs
> > > > > (bootstrap expansion + advertised lookup)
> > > > > >> and an additional parameter that defines how the resolution
is
> > > > > performed, using getCanonicalHostName() or not.
> > > > > >
> > > > > > thanks sounds to me *less* simple than independent config
> options,
> > > sorry.
> > > > > >
> > > > > > I would like to say once again that by itself KIP-302 only
> speeds
> > up
> > > > > > the client behavior that can happen anyway when the client
> > restarts
> > > > > > multiple times,
> > > > > > as every time there is no guarantee that - in presence of
> multiple
> > A
> > > > > > DNS records - the same IP is returned. Attempting to use
> additiona
> > > IPs
> > > > > > if the first fail just makes client recovery faster.
> > > > > >
> > > > > > cheers
> > > > > > Edo
> > > > > >
> > > > > > On 17 May 2018 at 12:12, Skrzypek, Jonathan <
> > > Jonathan.Skrzypek@gs.com>
> > > > > wrote:
> > > > > >> Yes, makes sense.
> > > > > >> You mentioned multiple times you see no overlap and no issue
> with
> > > your
> > > > > KIP, and that they solve different use cases.
> > > > > >>
> > > > > >> Appreciate you have an existing use case that would work, but
we
> > > need
> > > > > to make sure this isn't confusing to users and that any
combination
> > > will
> > > > > always work, across security protocols.
> > > > > >>
> > > > > >> A solution might be to expose to users the choice of using
> > hostname
> > > or
> > > > > canonical host name on both sides.
> > > > > >> Say having one setting that collapses functionalities from
both
> > KIPs
> > > > > (bootstrap expansion + advertised lookup) and an additional
> > parameter
> > > that
> > > > > defines how the resolution is performed, using
> > getCanonicalHostName()
> > > or
> > > > > not.
> > > > > >>
> > > > > >> Maybe that gives less flexibility as users wouldn't be able
to
> > > decide
> > > > > to only perform DNS lookup on bootstrap.servers or on advertised
> > > listeners.
> > > > > >> But this would ensure consistency so that a user can decide
to
> > use
> > > > > cnames or not (depending on their certificates and Kerberos
> > principals
> > > in
> > > > > their environment) and it would work.
> > > > > >>
> > > > > >> Jonathan Skrzypek
> > > > > >>
> > > > > >> -----Original Message-----
> > > > > >> From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > > > >> Sent: 16 May 2018 21:59
> > > > > >> To: dev@kafka.apache.org
> > > > > >> Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use
all
> > DNS
> > > > > resolved IP addresses
> > > > > >>
> > > > > >> Hi Jonathan,
> > > > > >> I am afraid that may not work for everybody.
> > > > > >>
> > > > > >> It would not work for us.
> > > > > >> With our current DNS, my Kafka clients are perfectly happy to
> use
> > > any
> > > > > IPs -
> > > > > >> DNS has multiple A records for the 'myhostname.mydomain' used
> for
> > > > > >> bootstrap and advertised listeners.
> > > > > >> The hosts all serve TLS certificates that include
> > > > > >> 'myhostname.mydomain' and the clients are happy.
> > > > > >>
> > > > > >> However, applying getCanonicalHostName to those IPs would
return
> > > > > >> hostnames that would not match the TLS certificates.
> > > > > >>
> > > > > >> So once again I believe your solution and ours solve
different
> > use
> > > > > cases.
> > > > > >>
> > > > > >> cheers
> > > > > >> Edo
> > > > > >>
> > > > > >> On 16 May 2018 at 18:29, Skrzypek, Jonathan <
> > > Jonathan.Skrzypek@gs.com>
> > > > > wrote:
> > > > > >>> I think there are combinations that will break SASL and SSL
> > auth.
> > > > > >>> Could the trick be to have a single parameter that triggers
dns
> > > > > resolve both for bootstrap and advertised listeners, both using
> > > > > getCanonicalHostName() ?
> > > > > >>>
> > > > > >>> Jonathan Skrzypek
> > > > > >>>
> > > > > >>> -----Original Message-----
> > > > > >>> From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > > > >>> Sent: 16 May 2018 17:03
> > > > > >>> To: dev@kafka.apache.org
> > > > > >>> Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use
> all
> > > DNS
> > > > > resolved IP addresses
> > > > > >>>
> > > > > >>> Hi Rajini,
> > > > > >>>
> > > > > >>> In your example KIP-302 would attempt to connect to the
first
> > > address
> > > > > >>> returned, let's say
> > > > > >>>
> > > > > >>> www.apache.org/195.154.151.36
> > > > > >>>
> > > > > >>> then, only if that fails, will in turn try the remaining:
> > > > > >>>
> > > > > >>> www.apache.org/40.79.78.1
> > > > > >>> www.apache.org/140.211.11.105
> > > > > >>> www.apache.org/2001:bc8:2142:300:0:0:0:0
> > > > > >>>
> > > > > >>> You're right to say that we expect certificates served by
those
> > > > > >>> endpoints to be valid for "www.apache.org"
> > > > > >>>
> > > > > >>> Without KIP-302, only one would be attempted.
> > > > > >>> Which is the first one, that can change every time
> > > > > >>> (typically changes on every Java process restart,
> > > > > >>> but may change also any time InetAddress.getAllByName it's
> > invoked
> > > > > >>> depending on the caching).
> > > > > >>>
> > > > > >>> The behavioral change that KIP-302 may introduce is that in
the
> > > > > example above,
> > > > > >>> also an IPv6 connection may be attempted after some IPv4
ones.
> > > > > >>>
> > > > > >>> InetAddress.getAllByName() implementation uses a system
> property
> > > > > >>> "java.net.preferIPv6Addresses"
> > > > > >>> to decide which type of address to return first (default is
> > still
> > > IPv4
> > > > > >>> in java 10)
> > > > > >>>
> > > > > >>> We will amend the KIP and PR so that the loop only uses IPs
of
> > the
> > > > > >>> same type as the first one returned.
> > > > > >>>
> > > > > >>> A part from the above, KIP 302 does not seem to change any
> > existing
> > > > > >>> client behaviour, as any one of multiple IP addresses (of a
> > given
> > > > > >>> v4/v6 type) can currently be picked.
> > > > > >>> We're happy however to keep the looping behavior optional
with
> > the
> > > > > >>> discussed config property, disabled by default.
> > > > > >>>
> > > > > >>> As for KIP-235 that may introduce new hostnames in the
> bootstrap
> > > list
> > > > > >>> (the current PR rewrites the bootstrap list)
> > > > > >>> and we fail to see the conflict with KIP-302, whatever the
set
> > of
> > > > > >>> configs chosen.
> > > > > >>>
> > > > > >>> We'd be happy to try understand what we are missing in a KIP
> > call
> > > :-)
> > > > > >>>
> > > > > >>> cheers
> > > > > >>> Edo
> > > > > >>>
> > > > > >>> On 15 May 2018 at 16:58, Rajini Sivaram
> > <ra...@gmail.com>
> > > > > wrote:
> > > > > >>>> Hi Edo,
> > > > > >>>>
> > > > > >>>> I agree that KIP-235 and KIP-302 address different
scenarios.
> > And
> > > I
> > > > > agree
> > > > > >>>> that each one is not sufficient in itself to address both
the
> > > > > scenarios.
> > > > > >>>> But I also think that they conflict and hence they need to
be
> > > looked
> > > > > at
> > > > > >>>> together and perhaps use a single config.
> > > > > >>>>
> > > > > >>>> As an example:
> > > > > >>>>
> > > > > >>>> If I run:
> > > > > >>>>
> > > > > >>>> for (InetAddress address : InetAddress.getAllByName("www.
> > > apache.org"))
> > > > > {
> > > > > >>>> System.out.printf("HostName %s canonicalHostName %s IP
> > %s\n",
> > > > > >>>> address.getHostName(),
> > address.getCanonicalHostName()
> > > ,
> > > > > >>>> address.getHostAddress());
> > > > > >>>> }
> > > > > >>>>
> > > > > >>>> I get:
> > > > > >>>>
> > > > > >>>> HostName www.apache.org canonicalHostName
> > tlp-eu-west.apache.org
> > > IP
> > > > > >>>> 195.154.151.36
> > > > > >>>> HostName www.apache.org canonicalHostName 40.79.78.1 IP
> > > 40.79.78.1
> > > > > >>>> HostName www.apache.org canonicalHostName themis.apache.org
> IP
> > > > > >>>> 140.211.11.105
> > > > > >>>> HostName www.apache.org canonicalHostName
> > > 2001:bc8:2142:300:0:0:0:0
> > > > > IP
> > > > > >>>> 2001:bc8:2142:300:0:0:0:0
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> If www.apache.org is used as a bootstrap address, KIP-302
> would
> > > > > connect to (
> > > > > >>>> www.apache.org/195.154.151.36 and
> > www.apache.org/140.211.11.105)
> > > > > while
> > > > > >>>> KIP-235 would connect to (tlp-eu-west.apache.org/195.
> 154.151.3.
> > > and
> > > > > >>>> themis.apache.org/140.211.11.105). This is a significant
> > > difference
> > > > > not
> > > > > >>>> just for Kerberos, but for any secure environment where
> > hostname
> > > is
> > > > > >>>> verified to prevent man-in-the-middle attacks. In your
case, I
> > > > > presume you
> > > > > >>>> would have SSL certificates with the equivalent of
> > www.apache.org
> > > on
> > > > > both
> > > > > >>>> the load balancers. In Jonathan's case, I presume he has
> > Kerberos
> > > > > >>>> principals for the equivalent of tlp-eu-west.apache.org and
> > > > > >>>> themis.apache.org. We would want to support both scenarios
> > > > > regardless of
> > > > > >>>> the security protocol, just need to come up with
configuration
> > > > > options that
> > > > > >>>> don't conflict.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> On Mon, May 14, 2018 at 5:24 PM, Edoardo Comar <
> > > edocomar@gmail.com>
> > > > > wrote:
> > > > > >>>>
> > > > > >>>>> Thanks Rajini
> > > > > >>>>>
> > > > > >>>>> I still don't see the overlap between the two KIPS
> > > > > >>>>>
> > > > > >>>>> KIP-235 allows an expansion of hostnames on the bootstrap
> > list.
> > > > > >>>>>
> > > > > >>>>> KIP-302 allows alternative IPs to be used to attempt a
> > connection
> > > > > >>>>> (either at bootstrap and when processing the
> MetadataResponse)
> > > to a
> > > > > >>>>> given hostname.
> > > > > >>>>>
> > > > > >>>>> A use case would be that of active/passive LB fronting the
> > > brokers.
> > > > > >>>>>
> > > > > >>>>> Arguably, if Java honored the DNS-set TTL, and the TTL was
> low
> > > and on
> > > > > >>>>> subsequent requests, the order of IPs returned by the
> > > > > >>>>> InetAddress.getAllByName() was random, we may not need
such
> an
> > > > > >>>>> enhancement.
> > > > > >>>>> In practice, a Java client can get stuck on a "bad" IP
> forever
> > > if it
> > > > > >>>>> only relies on the first IP returned.
> > > > > >>>>>
> > > > > >>>>> HTH,
> > > > > >>>>> Edo
> > > > > >>>>>
> > > > > >>>>> On 14 May 2018 at 16:23, Rajini Sivaram
> > <rajinisivaram@gmail.com
> > > >
> > > > > wrote:
> > > > > >>>>> > Hi Edo,
> > > > > >>>>> >
> > > > > >>>>> > Thanks for the KIP. I think it will be good to include a
> > > diagram
> > > > > to make
> > > > > >>>>> it
> > > > > >>>>> > easier to distinguish this scenario from that of KIP-235
> > > without
> > > > > reading
> > > > > >>>>> > the PR.
> > > > > >>>>> >
> > > > > >>>>> > It may be worth considering if KIP-235 and this KIP
could
> > use a
> > > > > single
> > > > > >>>>> > config name with different values instead of two boolean
> > > configs:
> > > > > >>>>> >
> > > > > >>>>> > bootstrap.reverse.dns.lookup = true/false
> > > > > >>>>> > enable.all.dns.ips = true/false
> > > > > >>>>> >
> > > > > >>>>> > Not all values of (bootstrap.reverse.dns.lookup,
> > > > > enable.all.dns.ips) seem
> > > > > >>>>> > to make sense. And not all scenarios are handled. Even
if
> we
> > > use
> > > > > multiple
> > > > > >>>>> > configs, it seems to me that we may want to name them
> > > differently.
> > > > > >>>>> >
> > > > > >>>>> > The possible combinations are:
> > > > > >>>>> >
> > > > > >>>>> > 1) Bootstrap
> > > > > >>>>> >
> > > > > >>>>> > a) No lookup
> > > > > >>>>> > b) Use all DNS entries with host name
> > > > > >>>>> > c) Use all DNS entries with canonical host name
> > > > > >>>>> >
> > > > > >>>>> > 2) Advertised listeners
> > > > > >>>>> >
> > > > > >>>>> > a) No lookup
> > > > > >>>>> > b) Use all DNS entries with host name
> > > > > >>>>> > c) Use all DNS entries with canonical host name
> > > > > >>>>> >
> > > > > >>>>> > The combinations that are enabled by the two boolean
> configs
> > (
> > > > > >>>>> > bootstrap.reverse.dns.lookup, enable.all.dns.ips) are:
> > > > > >>>>> >
> > > > > >>>>> > - (false, false) => (1a, 2a)
> > > > > >>>>> > - (true, false) => (1c, 2a)
> > > > > >>>>> > - (false, true) => (1b, 2b)
> > > > > >>>>> > - (true, true) => (??, 2b)
> > > > > >>>>> >
> > > > > >>>>> > It will be good if we can clearly identify which
> > combinations
> > > we
> > > > > want to
> > > > > >>>>> > support and the scenarios where they may be useful.
Perhaps
> > > (1a,
> > > > > 2a),
> > > > > >>>>> (1c,
> > > > > >>>>> > 2a), (1b, 2b) and (1c, 2c) are useful?
> > > > > >>>>> >
> > > > > >>>>> >
> > > > > >>>>> > On Mon, May 14, 2018 at 2:58 PM, Skrzypek, Jonathan <
> > > > > >>>>> > Jonathan.Skrzypek@gs.com> wrote:
> > > > > >>>>> >
> > > > > >>>>> >> Ah, apologies didn't see there was already a decent
amount
> > of
> > > > > discussion
> > > > > >>>>> >> on this in the PR.
> > > > > >>>>> >>
> > > > > >>>>> >> This kind of sounds related to the environment you're
> > running
> > > to
> > > > > me.
> > > > > >>>>> >> What is the rationale behind using the advertised
> listeners
> > > to do
> > > > > your
> > > > > >>>>> >> load balancing advertisement rather than a top level
alias
> > > that
> > > > > has
> > > > > >>>>> >> everything ?
> > > > > >>>>> >>
> > > > > >>>>> >> It sounds like in your case there is a mismatch between
> > > > > >>>>> bootstrap.servers
> > > > > >>>>> >> and advertised.listeners, and you want
> advertised.listeners
> > to
> > > > > take
> > > > > >>>>> >> precedence and have the client iterate over what is
> > returned
> > > by
> > > > > the
> > > > > >>>>> broker.
> > > > > >>>>> >> So the extra parameter doesn't only have to do with DNS
> but
> > > it's
> > > > > also
> > > > > >>>>> >> appending from the broker, maybe the parameter name
should
> > > > > reflect this
> > > > > >>>>> ?
> > > > > >>>>> >>
> > > > > >>>>> >> Jonathan Skrzypek
> > > > > >>>>> >>
> > > > > >>>>> >>
> > > > > >>>>> >> -----Original Message-----
> > > > > >>>>> >> From: Skrzypek, Jonathan [Tech]
> > > > > >>>>> >> Sent: 14 May 2018 14:46
> > > > > >>>>> >> To: dev@kafka.apache.org
> > > > > >>>>> >> Subject: RE: [DISCUSS] KIP-302 - Enable Kafka clients
to
> > use
> > > all
> > > > > DNS
> > > > > >>>>> >> resolved IP addresses
> > > > > >>>>> >>
> > > > > >>>>> >> Hi,
> > > > > >>>>> >>
> > > > > >>>>> >> I see you noted the similarities with KIP-235.
> > > > > >>>>> >> But KIP-235 might also solve what this KIP is trying to
> > > achieve.
> > > > > >>>>> >>
> > > > > >>>>> >> When parsing bootstrap.servers, KIP-235 has the client
add
> > all
> > > > > >>>>> underlying
> > > > > >>>>> >> hostnames and IPs.
> > > > > >>>>> >> And this happens before hitting the NetworkClient.
> > > > > >>>>> >>
> > > > > >>>>> >> So to me the client will try every single endpoint
behind
> > any
> > > > > >>>>> >> bootstrap.servers record.
> > > > > >>>>> >>
> > > > > >>>>> >> See
> > > > > INVALID URI REMOVED.
> > > com_apache_kafka_pull_4485_commits_24757eb7b0&d=DwIBaQ&c=
> > > 7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_
> QFAPdxGlafmDu9_
> > > r6eaCbPOM0NM1EHo-E&m=_ud9m_JZJ87C7eGsKcmzgJgDpNQDIIv5R4i_
> > > 7VlhkLc&s=TqaiA9uW_myYO6FN-gKPfPlioxZR6DhnlBTpEj5M2aQ&e=
> > > > > >>>>> >>
> > > > >
6bcf8c7d7649c85232c52b5d54f0e4#diff-89ef153462e64c250a21bd324ae1a8
> 51
> > > > > >>>>> >> which calls getAllByName like you suggested
> > > > > >>>>> >>
> > > > > >>>>> >> Jonathan Skrzypek
> > > > > >>>>> >>
> > > > > >>>>> >>
> > > > > >>>>> >> -----Original Message-----
> > > > > >>>>> >> From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > > > >>>>> >> Sent: 14 May 2018 14:17
> > > > > >>>>> >> To: dev@kafka.apache.org
> > > > > >>>>> >> Subject: [DISCUSS] KIP-302 - Enable Kafka clients to
use
> > all
> > > DNS
> > > > > >>>>> resolved
> > > > > >>>>> >> IP addresses
> > > > > >>>>> >>
> > > > > >>>>> >> Hi all,
> > > > > >>>>> >>
> > > > > >>>>> >> We just opened a KIP to add support for the client to
use
> > all
> > > IPs
> > > > > >>>>> returned
> > > > > >>>>> >> by DNS for the brokers
> > > > > >>>>> >>
> > > > > >>>>> >> The details are here -
> > > > > >>>>> >>
> > > > > >>>>> >>
> > INVALID URI REMOVED
> > > > > >>>>> >>
> > pache.org_confluence_display_KAFKA_KIP-2D302-2B-2D-2BEnable-
> > > > > >>>>> >>
2BKafka-2Bclients-2Bto-2Buse-2Ball-2BDNS-2Bresolved-2BIP-
> > > > > >>>>> >>
> > 2Baddresses&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxL
> > > > > >>>>> >>
> > xfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaCbPOM0NM1EHo-E&m=EJafF
> > > > > >>>>> >>
l1clRyolgtcu2uCc4_cIOJnlxb1r1n-D2Dti4k&s=C-UZ6KUG7JFiPD_
> > > > > >>>>> >> CnHczDOVqH9-XC5f_OFkw4BTNrI4&e=
> > > > > >>>>> >>
> > > > > >>>>> >> The JIRA and provisional PR (where the discussion lead
to
> > the
> > > > > creation
> > > > > >>>>> of
> > > > > >>>>> >> this KIP) are :
> > > > > >>>>> >>
> > > > > >>>>> >> INVALID URI REMOVED
> > .
> > > > > >>>>> >>
> > apache.org_jira_browse_KAFKA-2D6863&d=DwIBaQ&c=7563p3e2zaQw0
> > > > > >>>>> >>
> > AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6
> > > > > >>>>> >> eaCbPOM0NM1EHo-E&m=EJafFl1clRyolgtcu2uCc4_cIOJnlxb1r1n-
> > > > > >>>>> >>
D2Dti4k&s=3Puqs5iYoPsw6hARQr6gvokdFE-H5USMiNVGOUtNkJI&e=
> > > > > >>>>> >>
> > > > > >>>>> >> INVALID URI REMOVED
> > .
> > > > > >>>>> >>
> > com_apache_kafka_pull_4987&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgy
> > > > > >>>>> >> agb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaC
> > > > > >>>>> >>
> > bPOM0NM1EHo-E&m=EJafFl1clRyolgtcu2uCc4_cIOJnlxb1r1n-D2Dti4k&
> > > > > >>>>> >> s=Hqn5dOgQy4-MHTIJLE49O8bNomry3SoGq9OVoHU-CRA&e=
> > > > > >>>>> >>
> > > > > >>>>> >> Looking forward to the community's feedback.
> > > > > >>>>> >> It would be amazing to have it voted by May 22nd :-)
:-)
> > > > > >>>>> >>
> > > > > >>>>> >> Edoardo & Mickael
> > > > > >>>>> >>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> --
> > > > > >>>>> "When the people fear their government, there is tyranny;
> when
> > > the
> > > > > >>>>> government fears the people, there is liberty." [Thomas
> > > Jefferson]
> > > > > >>>>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>> --
> > > > > >>> "When the people fear their government, there is tyranny;
when
> > the
> > > > > >>> government fears the people, there is liberty." [Thomas
> > Jefferson]
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> "When the people fear their government, there is tyranny;
when
> > the
> > > > > >> government fears the people, there is liberty." [Thomas
> > Jefferson]
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > "When the people fear their government, there is tyranny; when
> the
> > > > > > government fears the people, there is liberty." [Thomas
> Jefferson]
> > > > > >
> > > > > > ________________________________
> > > > > >
> > > > > > Your Personal Data: We may collect and process information
about
> > you
> > > > > that may be subject to data protection laws. For more
information
> > > about how
> > > > > we use and disclose your personal data, how we protect your
> > > information,
> > > > > our legal basis to use your information, your rights and who you
> can
> > > > > contact, please refer to: www.gs.com/privacy-notices<
> > > > >
> >
http://www.gs.com/privacy-notices
> > >
> > > > > >
> > > > > >
> > > > > > ---------- Forwarded message ----------
> > > > > > From: Rajini Sivaram <ra...@gmail.com>
> > > > > > To: "Skrzypek, Jonathan" <Jo...@ln.email.gs.com>,
> dev
> > <
> > > > > dev@kafka.apache.org>, Ismael Juma <is...@juma.me.uk>
> > > > > > Cc:
> > > > > > Bcc:
> > > > > > Date: Tue, 22 May 2018 15:05:07 +0000
> > > > > > Subject: Re: FW: [VOTE] KIP-235 Add DNS alias support for
secured
> > > > > connection
> > > > > > Hi Jonathan,
> > > > > >
> > > > > > I think it would make sense to convert the config in this KIP
> into
> > an
> > > > > enum so that we can add more variations later on. But since
KIP-302
> > is
> > > > > still under discussion, it is not clear what the config name
should
> > be.
> > > > > Since today is the KIP deadline and the implementation itself is
> > > > > straightforward, it would make sense to progress with this one
for
> > > 2.0.0 if
> > > > > we can get one more binding vote.
> > > > > >
> > > > > > Ismael, do you have time to take a look at KIP-235 today?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > >
> > > > > > On Tue, May 22, 2018 at 3:45 PM, Skrzypek, Jonathan <
> > > > > Jonathan.Skrzypek@gs.com> wrote:
> > > > > >>
> > > > > >> Hello Rajini,
> > > > > >>
> > > > > >> What do you think should be the next step here ?
> > > > > >>
> > > > > >>
> > > > > >> Jonathan Skrzypek
> > > > > >>
> > > > > >> -----Original Message-----
> > > > > >> From: Skrzypek, Jonathan [Tech]
> > > > > >> Sent: 21 May 2018 10:51
> > > > > >> To: 'dev'
> > > > > >> Subject: RE: [VOTE] KIP-235 Add DNS alias support for secured
> > > connection
> > > > > >>
> > > > > >> Hi,
> > > > > >>
> > > > > >> What would be the next step here ?
> > > > > >> I know there's a discussion going on around KIP-302, but I'm
> also
> > > > > conscious that the 2.0.0 deadline for KIPs is tomorrow.
> > > > > >> I've opened this KIP in January and discussions have been
> > productive
> > > > > with an end solution I had the impression was reasonable, so I
am
> > keen
> > > to
> > > > > see it make it the next release.
> > > > > >>
> > > > > >>
> > > > > >> Jonathan Skrzypek
> > > > > >>
> > > > > >> -----Original Message-----
> > > > > >> From: Skrzypek, Jonathan [Tech]
> > > > > >> Sent: 14 May 2018 13:48
> > > > > >> To: dev
> > > > > >> Subject: RE: [VOTE] KIP-235 Add DNS alias support for secured
> > > connection
> > > > > >>
> > > > > >> Sure, I modified the KIP to add more details
> > > > > >>
> > > > > >>
> > > > >
> >
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >
> > > 235%3A+Add+DNS+alias+support+for+secured+connection
> > > > > >>
> > > > > >>
> > > > > >> Jonathan Skrzypek
> > > > > >>
> > > > > >>
> > > > > >> -----Original Message-----
> > > > > >> From: Ismael Juma [mailto:ismael@juma.me.uk]
> > > > > >> Sent: 14 May 2018 11:53
> > > > > >> To: dev
> > > > > >> Subject: Re: [VOTE] KIP-235 Add DNS alias support for secured
> > > connection
> > > > > >>
> > > > > >> Thanks for the KIP, Jonathan. It would be helpful to have
more
> > > detail on
> > > > > >> how SSL authentication could be broken if the new behaviour
is
> > the
> > > > > default.
> > > > > >> I know this was discussed in the mailing list thread, but
it's
> > > > > important to
> > > > > >> include it in the KIP since it's the main reason why a new
> config
> > is
> > > > > needed
> > > > > >> (and configs should be avoided whenever we can just do the
right
> > > thing).
> > > > > >>
> > > > > >> Ismael
> > > > > >>
> > > > > >> On Fri, Mar 23, 2018 at 12:05 PM Skrzypek, Jonathan <
> > > > > >> Jonathan.Skrzypek@gs.com> wrote:
> > > > > >>
> > > > > >> > Hi,
> > > > > >> >
> > > > > >> > I would like to start a vote for KIP-235
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > INVALID URI REMOVED.
> > > apache.org_confluence_display_KAFKA_KIP-2D235-253A-2BAdd-
> > > 2BDNS-2Balias-2Bsupport-2Bfor-2Bsecured-2Bconnection&d=DwIBaQ&c=
> > > 7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_
> QFAPdxGlafmDu9_
> > > r6eaCbPOM0NM1EHo-E&m=FM_uCHnnO2dqxWC0bi7_QOJKfKmQI80-
> > > Xduvb-URWOw&s=RpGkijfK-WHcU0s8ZtMXEkIr69QraJhYKaGSC9V_rnI&e=
> > > > > >> >
> > > > > >> > This is a proposition to add an option for reverse dns
lookup
> > of
> > > > > >> > bootstrap.servers hosts, allowing the use of dns aliases on
> > > clusters
> > > > > using
> > > > > >> > SASL authentication.
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >> ________________________________
> > > > > >>
> > > > > >> Your Personal Data: We may collect and process information
about
> > you
> > > > > that may be subject to data protection laws. For more
information
> > > about how
> > > > > we use and disclose your personal data, how we protect your
> > > information,
> > > > > our legal basis to use your information, your rights and who you
> can
> > > > > contact, please refer to: www.gs.com/privacy-notices<
> > > > >
> >
http://www.gs.com/privacy-notices
> > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > "When the people fear their government, there is tyranny; when
the
> > > > > government fears the people, there is liberty." [Thomas
Jefferson]
> > > > >
> > > >
> > > >
> > > > --
> > > > "When the people fear their government, there is tyranny; when the
> > > > government fears the people, there is liberty." [Thomas Jefferson]
> > >
> >
> >
> >
> > Unless stated otherwise above:
> > IBM United Kingdom Limited - Registered in England and Wales with
number
> > 741598.
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> 3AU
> >
>
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS resolved
IP addresses
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Edo,
The KIP looks good, perhaps you can start vote on this to get it in time
for 2.1.0?
Regards,
Rajini
On Tue, Sep 18, 2018 at 1:58 PM, Eno Thereska <en...@gmail.com>
wrote:
> Got it, thanks!
>
> Eno
>
> On Tue, Sep 18, 2018 at 1:41 PM, Edoardo Comar <EC...@uk.ibm.com> wrote:
>
> > Hi Eno,
> >
> > we implement network separation and the machines fronting the brokers
> > (call them LBs for simplicity) will route the connection to the correct
> > broker based on the TLS SNI. We register in the DNS multiple A records
> > (the IPs of all LBs) for each the brokers' hostnames.
> > As long as all the brokers are up, the cluster is fully functioning even
> > if just one of the LBs is up.
> >
> > HTH,
> > Edo
> > --------------------------------------------------
> >
> > Edoardo Comar
> >
> > IBM Message Hub
> >
> > IBM UK Ltd, Hursley Park, SO21 2JN
> >
> >
> >
> > From: Eno Thereska <en...@gmail.com>
> > To: dev@kafka.apache.org
> > Date: 18/09/2018 10:24
> > Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all
> > DNS resolved IP addresses
> >
> >
> >
> > Hi folks,
> >
> > Could you expand the motivation a bit? When would it make sense to use an
> > LB in front of Kafka brokers? A client needs to access each broker
> > directly
> > to consume the data in that broker and cannot be redirected to another
> > broker. What exact scenario are you seeing that needs this KIP?
> >
> > Thanks
> > Eno
> >
> > On Tue, Sep 18, 2018 at 9:52 AM, Mickael Maison <
> mickael.maison@gmail.com>
> > wrote:
> >
> > > Bumping this thread
> > >
> > > It's a relatively small change that would help cloud environments with
> > > load balancers fronting brokers
> > > On Tue, Sep 11, 2018 at 3:01 PM Edoardo Comar <ed...@gmail.com>
> > wrote:
> > > >
> > > > Hi all,
> > > > after some time we updated KIP-302 to reuse the config key introduced
> > by
> > > > KIP-235, with a different value to avoid conflicts between the two.
> > > > Also clarified the use of multiple IPs only of the same type
> > (IPv4/IPv6).
> > > >
> > > > We'd appreciate a further review and discussion.
> > > > Thanks!
> > > > Edo
> > > >
> > > >
> > > > On Fri, 25 May 2018 at 12:36, Edoardo Comar <ed...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Jonathan,
> > > > > I'm ok with an expandable enum for the config that could be
> extended
> > > > > in the future.
> > > > > It is marginally better than multiple potentially conflicting
> config
> > > > > entries.
> > > > >
> > > > > Though as I think the change for KIP-302 is independent from
> KIP-235
> > > > > and they do not conflict,
> > > > > when we'll look back at it post 2.0 we may see if it is more
> > valuable
> > > > > to shoehorn its config in an expanded enum or not
> > > > >
> > > > > thanks,
> > > > > Edo
> > > > >
> > > > > On 24 May 2018 at 16:50, Skrzypek, Jonathan
> > <Jo...@gs.com>
> > > > > wrote:
> > > > > > Hi,
> > > > > >
> > > > > > As Rajini suggested in the thread for KIP 235 (attached), we
> could
> > > try
> > > > > to have an enum that would drive what does the client
> > expands/resolves.
> > > > > >
> > > > > > I suggest a client config called client.dns.lookup with different
> > > values
> > > > > possible :
> > > > > >
> > > > > > - no : no dns lookup
> > > > > > - hostnames.only : perform dns lookup on both bootstrap.servers
> > and
> > > > > advertised listeners
> > > > > > - canonical.hostnames.only : perform dns lookup on both
> > > > > bootstrap.servers and advertised listeners
> > > > > > - bootstrap.hostnames.only : perform dns lookup on
> > bootstrap.servers
> > > > > list and expand it
> > > > > > - bootstrap.canonical.hostnames.only : perform dns lookup on
> > > > > bootstrap.servers list and expand it
> > > > > > - advertised.listeners.hostnames.only : perform dns lookup on
> > > advertised
> > > > > listeners
> > > > > > - advertised.listeners.canonical.hostnames.only : perform dns
> > > lookup on
> > > > > advertised listeners
> > > > > >
> > > > > > I realize this is a bit heavy but this gives users the ability to
> > > pick
> > > > > and choose.
> > > > > > I didn't include a setting to mix hostnames and canonical
> > hostnames
> > > as
> > > > > I'm not sure there would be a valid use case.
> > > > > >
> > > > > > Alternatively, to have less possible values, we could have 2
> > > parameters :
> > > > > >
> > > > > > - dns.lookup.type with values : hostname / canonical.host.name
> > > > > > - dns.lookup.behaviour : bootstrap.servers, advertised.listeners,
> > > both
> > > > > >
> > > > > > Thoughts ?
> > > > > >
> > > > > > Jonathan Skrzypek
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > > > > Sent: 17 May 2018 23:50
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all
> > DNS
> > > > > resolved IP addresses
> > > > > >
> > > > > > Hi Jonathan,
> > > > > >
> > > > > >> A solution might be to expose to users the choice of using
> > hostname
> > > or
> > > > > canonical host name on both sides.
> > > > > >> Say having one setting that collapses functionalities from both
> > KIPs
> > > > > (bootstrap expansion + advertised lookup)
> > > > > >> and an additional parameter that defines how the resolution is
> > > > > performed, using getCanonicalHostName() or not.
> > > > > >
> > > > > > thanks sounds to me *less* simple than independent config
> options,
> > > sorry.
> > > > > >
> > > > > > I would like to say once again that by itself KIP-302 only
> speeds
> > up
> > > > > > the client behavior that can happen anyway when the client
> > restarts
> > > > > > multiple times,
> > > > > > as every time there is no guarantee that - in presence of
> multiple
> > A
> > > > > > DNS records - the same IP is returned. Attempting to use
> additiona
> > > IPs
> > > > > > if the first fail just makes client recovery faster.
> > > > > >
> > > > > > cheers
> > > > > > Edo
> > > > > >
> > > > > > On 17 May 2018 at 12:12, Skrzypek, Jonathan <
> > > Jonathan.Skrzypek@gs.com>
> > > > > wrote:
> > > > > >> Yes, makes sense.
> > > > > >> You mentioned multiple times you see no overlap and no issue
> with
> > > your
> > > > > KIP, and that they solve different use cases.
> > > > > >>
> > > > > >> Appreciate you have an existing use case that would work, but we
> > > need
> > > > > to make sure this isn't confusing to users and that any combination
> > > will
> > > > > always work, across security protocols.
> > > > > >>
> > > > > >> A solution might be to expose to users the choice of using
> > hostname
> > > or
> > > > > canonical host name on both sides.
> > > > > >> Say having one setting that collapses functionalities from both
> > KIPs
> > > > > (bootstrap expansion + advertised lookup) and an additional
> > parameter
> > > that
> > > > > defines how the resolution is performed, using
> > getCanonicalHostName()
> > > or
> > > > > not.
> > > > > >>
> > > > > >> Maybe that gives less flexibility as users wouldn't be able to
> > > decide
> > > > > to only perform DNS lookup on bootstrap.servers or on advertised
> > > listeners.
> > > > > >> But this would ensure consistency so that a user can decide to
> > use
> > > > > cnames or not (depending on their certificates and Kerberos
> > principals
> > > in
> > > > > their environment) and it would work.
> > > > > >>
> > > > > >> Jonathan Skrzypek
> > > > > >>
> > > > > >> -----Original Message-----
> > > > > >> From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > > > >> Sent: 16 May 2018 21:59
> > > > > >> To: dev@kafka.apache.org
> > > > > >> Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all
> > DNS
> > > > > resolved IP addresses
> > > > > >>
> > > > > >> Hi Jonathan,
> > > > > >> I am afraid that may not work for everybody.
> > > > > >>
> > > > > >> It would not work for us.
> > > > > >> With our current DNS, my Kafka clients are perfectly happy to
> use
> > > any
> > > > > IPs -
> > > > > >> DNS has multiple A records for the 'myhostname.mydomain' used
> for
> > > > > >> bootstrap and advertised listeners.
> > > > > >> The hosts all serve TLS certificates that include
> > > > > >> 'myhostname.mydomain' and the clients are happy.
> > > > > >>
> > > > > >> However, applying getCanonicalHostName to those IPs would return
> > > > > >> hostnames that would not match the TLS certificates.
> > > > > >>
> > > > > >> So once again I believe your solution and ours solve different
> > use
> > > > > cases.
> > > > > >>
> > > > > >> cheers
> > > > > >> Edo
> > > > > >>
> > > > > >> On 16 May 2018 at 18:29, Skrzypek, Jonathan <
> > > Jonathan.Skrzypek@gs.com>
> > > > > wrote:
> > > > > >>> I think there are combinations that will break SASL and SSL
> > auth.
> > > > > >>> Could the trick be to have a single parameter that triggers dns
> > > > > resolve both for bootstrap and advertised listeners, both using
> > > > > getCanonicalHostName() ?
> > > > > >>>
> > > > > >>> Jonathan Skrzypek
> > > > > >>>
> > > > > >>> -----Original Message-----
> > > > > >>> From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > > > >>> Sent: 16 May 2018 17:03
> > > > > >>> To: dev@kafka.apache.org
> > > > > >>> Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use
> all
> > > DNS
> > > > > resolved IP addresses
> > > > > >>>
> > > > > >>> Hi Rajini,
> > > > > >>>
> > > > > >>> In your example KIP-302 would attempt to connect to the first
> > > address
> > > > > >>> returned, let's say
> > > > > >>>
> > > > > >>> www.apache.org/195.154.151.36
> > > > > >>>
> > > > > >>> then, only if that fails, will in turn try the remaining:
> > > > > >>>
> > > > > >>> www.apache.org/40.79.78.1
> > > > > >>> www.apache.org/140.211.11.105
> > > > > >>> www.apache.org/2001:bc8:2142:300:0:0:0:0
> > > > > >>>
> > > > > >>> You're right to say that we expect certificates served by those
> > > > > >>> endpoints to be valid for "www.apache.org"
> > > > > >>>
> > > > > >>> Without KIP-302, only one would be attempted.
> > > > > >>> Which is the first one, that can change every time
> > > > > >>> (typically changes on every Java process restart,
> > > > > >>> but may change also any time InetAddress.getAllByName it's
> > invoked
> > > > > >>> depending on the caching).
> > > > > >>>
> > > > > >>> The behavioral change that KIP-302 may introduce is that in the
> > > > > example above,
> > > > > >>> also an IPv6 connection may be attempted after some IPv4 ones.
> > > > > >>>
> > > > > >>> InetAddress.getAllByName() implementation uses a system
> property
> > > > > >>> "java.net.preferIPv6Addresses"
> > > > > >>> to decide which type of address to return first (default is
> > still
> > > IPv4
> > > > > >>> in java 10)
> > > > > >>>
> > > > > >>> We will amend the KIP and PR so that the loop only uses IPs of
> > the
> > > > > >>> same type as the first one returned.
> > > > > >>>
> > > > > >>> A part from the above, KIP 302 does not seem to change any
> > existing
> > > > > >>> client behaviour, as any one of multiple IP addresses (of a
> > given
> > > > > >>> v4/v6 type) can currently be picked.
> > > > > >>> We're happy however to keep the looping behavior optional with
> > the
> > > > > >>> discussed config property, disabled by default.
> > > > > >>>
> > > > > >>> As for KIP-235 that may introduce new hostnames in the
> bootstrap
> > > list
> > > > > >>> (the current PR rewrites the bootstrap list)
> > > > > >>> and we fail to see the conflict with KIP-302, whatever the set
> > of
> > > > > >>> configs chosen.
> > > > > >>>
> > > > > >>> We'd be happy to try understand what we are missing in a KIP
> > call
> > > :-)
> > > > > >>>
> > > > > >>> cheers
> > > > > >>> Edo
> > > > > >>>
> > > > > >>> On 15 May 2018 at 16:58, Rajini Sivaram
> > <ra...@gmail.com>
> > > > > wrote:
> > > > > >>>> Hi Edo,
> > > > > >>>>
> > > > > >>>> I agree that KIP-235 and KIP-302 address different scenarios.
> > And
> > > I
> > > > > agree
> > > > > >>>> that each one is not sufficient in itself to address both the
> > > > > scenarios.
> > > > > >>>> But I also think that they conflict and hence they need to be
> > > looked
> > > > > at
> > > > > >>>> together and perhaps use a single config.
> > > > > >>>>
> > > > > >>>> As an example:
> > > > > >>>>
> > > > > >>>> If I run:
> > > > > >>>>
> > > > > >>>> for (InetAddress address : InetAddress.getAllByName("www.
> > > apache.org"))
> > > > > {
> > > > > >>>> System.out.printf("HostName %s canonicalHostName %s IP
> > %s\n",
> > > > > >>>> address.getHostName(),
> > address.getCanonicalHostName()
> > > ,
> > > > > >>>> address.getHostAddress());
> > > > > >>>> }
> > > > > >>>>
> > > > > >>>> I get:
> > > > > >>>>
> > > > > >>>> HostName www.apache.org canonicalHostName
> > tlp-eu-west.apache.org
> > > IP
> > > > > >>>> 195.154.151.36
> > > > > >>>> HostName www.apache.org canonicalHostName 40.79.78.1 IP
> > > 40.79.78.1
> > > > > >>>> HostName www.apache.org canonicalHostName themis.apache.org
> IP
> > > > > >>>> 140.211.11.105
> > > > > >>>> HostName www.apache.org canonicalHostName
> > > 2001:bc8:2142:300:0:0:0:0
> > > > > IP
> > > > > >>>> 2001:bc8:2142:300:0:0:0:0
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> If www.apache.org is used as a bootstrap address, KIP-302
> would
> > > > > connect to (
> > > > > >>>> www.apache.org/195.154.151.36 and
> > www.apache.org/140.211.11.105)
> > > > > while
> > > > > >>>> KIP-235 would connect to (tlp-eu-west.apache.org/195.
> 154.151.3.
> > > and
> > > > > >>>> themis.apache.org/140.211.11.105). This is a significant
> > > difference
> > > > > not
> > > > > >>>> just for Kerberos, but for any secure environment where
> > hostname
> > > is
> > > > > >>>> verified to prevent man-in-the-middle attacks. In your case, I
> > > > > presume you
> > > > > >>>> would have SSL certificates with the equivalent of
> > www.apache.org
> > > on
> > > > > both
> > > > > >>>> the load balancers. In Jonathan's case, I presume he has
> > Kerberos
> > > > > >>>> principals for the equivalent of tlp-eu-west.apache.org and
> > > > > >>>> themis.apache.org. We would want to support both scenarios
> > > > > regardless of
> > > > > >>>> the security protocol, just need to come up with configuration
> > > > > options that
> > > > > >>>> don't conflict.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> On Mon, May 14, 2018 at 5:24 PM, Edoardo Comar <
> > > edocomar@gmail.com>
> > > > > wrote:
> > > > > >>>>
> > > > > >>>>> Thanks Rajini
> > > > > >>>>>
> > > > > >>>>> I still don't see the overlap between the two KIPS
> > > > > >>>>>
> > > > > >>>>> KIP-235 allows an expansion of hostnames on the bootstrap
> > list.
> > > > > >>>>>
> > > > > >>>>> KIP-302 allows alternative IPs to be used to attempt a
> > connection
> > > > > >>>>> (either at bootstrap and when processing the
> MetadataResponse)
> > > to a
> > > > > >>>>> given hostname.
> > > > > >>>>>
> > > > > >>>>> A use case would be that of active/passive LB fronting the
> > > brokers.
> > > > > >>>>>
> > > > > >>>>> Arguably, if Java honored the DNS-set TTL, and the TTL was
> low
> > > and on
> > > > > >>>>> subsequent requests, the order of IPs returned by the
> > > > > >>>>> InetAddress.getAllByName() was random, we may not need such
> an
> > > > > >>>>> enhancement.
> > > > > >>>>> In practice, a Java client can get stuck on a "bad" IP
> forever
> > > if it
> > > > > >>>>> only relies on the first IP returned.
> > > > > >>>>>
> > > > > >>>>> HTH,
> > > > > >>>>> Edo
> > > > > >>>>>
> > > > > >>>>> On 14 May 2018 at 16:23, Rajini Sivaram
> > <rajinisivaram@gmail.com
> > > >
> > > > > wrote:
> > > > > >>>>> > Hi Edo,
> > > > > >>>>> >
> > > > > >>>>> > Thanks for the KIP. I think it will be good to include a
> > > diagram
> > > > > to make
> > > > > >>>>> it
> > > > > >>>>> > easier to distinguish this scenario from that of KIP-235
> > > without
> > > > > reading
> > > > > >>>>> > the PR.
> > > > > >>>>> >
> > > > > >>>>> > It may be worth considering if KIP-235 and this KIP could
> > use a
> > > > > single
> > > > > >>>>> > config name with different values instead of two boolean
> > > configs:
> > > > > >>>>> >
> > > > > >>>>> > bootstrap.reverse.dns.lookup = true/false
> > > > > >>>>> > enable.all.dns.ips = true/false
> > > > > >>>>> >
> > > > > >>>>> > Not all values of (bootstrap.reverse.dns.lookup,
> > > > > enable.all.dns.ips) seem
> > > > > >>>>> > to make sense. And not all scenarios are handled. Even if
> we
> > > use
> > > > > multiple
> > > > > >>>>> > configs, it seems to me that we may want to name them
> > > differently.
> > > > > >>>>> >
> > > > > >>>>> > The possible combinations are:
> > > > > >>>>> >
> > > > > >>>>> > 1) Bootstrap
> > > > > >>>>> >
> > > > > >>>>> > a) No lookup
> > > > > >>>>> > b) Use all DNS entries with host name
> > > > > >>>>> > c) Use all DNS entries with canonical host name
> > > > > >>>>> >
> > > > > >>>>> > 2) Advertised listeners
> > > > > >>>>> >
> > > > > >>>>> > a) No lookup
> > > > > >>>>> > b) Use all DNS entries with host name
> > > > > >>>>> > c) Use all DNS entries with canonical host name
> > > > > >>>>> >
> > > > > >>>>> > The combinations that are enabled by the two boolean
> configs
> > (
> > > > > >>>>> > bootstrap.reverse.dns.lookup, enable.all.dns.ips) are:
> > > > > >>>>> >
> > > > > >>>>> > - (false, false) => (1a, 2a)
> > > > > >>>>> > - (true, false) => (1c, 2a)
> > > > > >>>>> > - (false, true) => (1b, 2b)
> > > > > >>>>> > - (true, true) => (??, 2b)
> > > > > >>>>> >
> > > > > >>>>> > It will be good if we can clearly identify which
> > combinations
> > > we
> > > > > want to
> > > > > >>>>> > support and the scenarios where they may be useful. Perhaps
> > > (1a,
> > > > > 2a),
> > > > > >>>>> (1c,
> > > > > >>>>> > 2a), (1b, 2b) and (1c, 2c) are useful?
> > > > > >>>>> >
> > > > > >>>>> >
> > > > > >>>>> > On Mon, May 14, 2018 at 2:58 PM, Skrzypek, Jonathan <
> > > > > >>>>> > Jonathan.Skrzypek@gs.com> wrote:
> > > > > >>>>> >
> > > > > >>>>> >> Ah, apologies didn't see there was already a decent amount
> > of
> > > > > discussion
> > > > > >>>>> >> on this in the PR.
> > > > > >>>>> >>
> > > > > >>>>> >> This kind of sounds related to the environment you're
> > running
> > > to
> > > > > me.
> > > > > >>>>> >> What is the rationale behind using the advertised
> listeners
> > > to do
> > > > > your
> > > > > >>>>> >> load balancing advertisement rather than a top level alias
> > > that
> > > > > has
> > > > > >>>>> >> everything ?
> > > > > >>>>> >>
> > > > > >>>>> >> It sounds like in your case there is a mismatch between
> > > > > >>>>> bootstrap.servers
> > > > > >>>>> >> and advertised.listeners, and you want
> advertised.listeners
> > to
> > > > > take
> > > > > >>>>> >> precedence and have the client iterate over what is
> > returned
> > > by
> > > > > the
> > > > > >>>>> broker.
> > > > > >>>>> >> So the extra parameter doesn't only have to do with DNS
> but
> > > it's
> > > > > also
> > > > > >>>>> >> appending from the broker, maybe the parameter name should
> > > > > reflect this
> > > > > >>>>> ?
> > > > > >>>>> >>
> > > > > >>>>> >> Jonathan Skrzypek
> > > > > >>>>> >>
> > > > > >>>>> >>
> > > > > >>>>> >> -----Original Message-----
> > > > > >>>>> >> From: Skrzypek, Jonathan [Tech]
> > > > > >>>>> >> Sent: 14 May 2018 14:46
> > > > > >>>>> >> To: dev@kafka.apache.org
> > > > > >>>>> >> Subject: RE: [DISCUSS] KIP-302 - Enable Kafka clients to
> > use
> > > all
> > > > > DNS
> > > > > >>>>> >> resolved IP addresses
> > > > > >>>>> >>
> > > > > >>>>> >> Hi,
> > > > > >>>>> >>
> > > > > >>>>> >> I see you noted the similarities with KIP-235.
> > > > > >>>>> >> But KIP-235 might also solve what this KIP is trying to
> > > achieve.
> > > > > >>>>> >>
> > > > > >>>>> >> When parsing bootstrap.servers, KIP-235 has the client add
> > all
> > > > > >>>>> underlying
> > > > > >>>>> >> hostnames and IPs.
> > > > > >>>>> >> And this happens before hitting the NetworkClient.
> > > > > >>>>> >>
> > > > > >>>>> >> So to me the client will try every single endpoint behind
> > any
> > > > > >>>>> >> bootstrap.servers record.
> > > > > >>>>> >>
> > > > > >>>>> >> See
> > > > > INVALID URI REMOVED.
> > > com_apache_kafka_pull_4485_commits_24757eb7b0&d=DwIBaQ&c=
> > > 7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_
> QFAPdxGlafmDu9_
> > > r6eaCbPOM0NM1EHo-E&m=_ud9m_JZJ87C7eGsKcmzgJgDpNQDIIv5R4i_
> > > 7VlhkLc&s=TqaiA9uW_myYO6FN-gKPfPlioxZR6DhnlBTpEj5M2aQ&e=
> > > > > >>>>> >>
> > > > > 6bcf8c7d7649c85232c52b5d54f0e4#diff-89ef153462e64c250a21bd324ae1a8
> 51
> > > > > >>>>> >> which calls getAllByName like you suggested
> > > > > >>>>> >>
> > > > > >>>>> >> Jonathan Skrzypek
> > > > > >>>>> >>
> > > > > >>>>> >>
> > > > > >>>>> >> -----Original Message-----
> > > > > >>>>> >> From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > > > >>>>> >> Sent: 14 May 2018 14:17
> > > > > >>>>> >> To: dev@kafka.apache.org
> > > > > >>>>> >> Subject: [DISCUSS] KIP-302 - Enable Kafka clients to use
> > all
> > > DNS
> > > > > >>>>> resolved
> > > > > >>>>> >> IP addresses
> > > > > >>>>> >>
> > > > > >>>>> >> Hi all,
> > > > > >>>>> >>
> > > > > >>>>> >> We just opened a KIP to add support for the client to use
> > all
> > > IPs
> > > > > >>>>> returned
> > > > > >>>>> >> by DNS for the brokers
> > > > > >>>>> >>
> > > > > >>>>> >> The details are here -
> > > > > >>>>> >>
> > > > > >>>>> >>
> > INVALID URI REMOVED
> > > > > >>>>> >>
> > pache.org_confluence_display_KAFKA_KIP-2D302-2B-2D-2BEnable-
> > > > > >>>>> >> 2BKafka-2Bclients-2Bto-2Buse-2Ball-2BDNS-2Bresolved-2BIP-
> > > > > >>>>> >>
> > 2Baddresses&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxL
> > > > > >>>>> >>
> > xfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaCbPOM0NM1EHo-E&m=EJafF
> > > > > >>>>> >> l1clRyolgtcu2uCc4_cIOJnlxb1r1n-D2Dti4k&s=C-UZ6KUG7JFiPD_
> > > > > >>>>> >> CnHczDOVqH9-XC5f_OFkw4BTNrI4&e=
> > > > > >>>>> >>
> > > > > >>>>> >> The JIRA and provisional PR (where the discussion lead to
> > the
> > > > > creation
> > > > > >>>>> of
> > > > > >>>>> >> this KIP) are :
> > > > > >>>>> >>
> > > > > >>>>> >> INVALID URI REMOVED
> > .
> > > > > >>>>> >>
> > apache.org_jira_browse_KAFKA-2D6863&d=DwIBaQ&c=7563p3e2zaQw0
> > > > > >>>>> >>
> > AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6
> > > > > >>>>> >> eaCbPOM0NM1EHo-E&m=EJafFl1clRyolgtcu2uCc4_cIOJnlxb1r1n-
> > > > > >>>>> >> D2Dti4k&s=3Puqs5iYoPsw6hARQr6gvokdFE-H5USMiNVGOUtNkJI&e=
> > > > > >>>>> >>
> > > > > >>>>> >> INVALID URI REMOVED
> > .
> > > > > >>>>> >>
> > com_apache_kafka_pull_4987&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgy
> > > > > >>>>> >> agb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaC
> > > > > >>>>> >>
> > bPOM0NM1EHo-E&m=EJafFl1clRyolgtcu2uCc4_cIOJnlxb1r1n-D2Dti4k&
> > > > > >>>>> >> s=Hqn5dOgQy4-MHTIJLE49O8bNomry3SoGq9OVoHU-CRA&e=
> > > > > >>>>> >>
> > > > > >>>>> >> Looking forward to the community's feedback.
> > > > > >>>>> >> It would be amazing to have it voted by May 22nd :-) :-)
> > > > > >>>>> >>
> > > > > >>>>> >> Edoardo & Mickael
> > > > > >>>>> >>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> --
> > > > > >>>>> "When the people fear their government, there is tyranny;
> when
> > > the
> > > > > >>>>> government fears the people, there is liberty." [Thomas
> > > Jefferson]
> > > > > >>>>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>> --
> > > > > >>> "When the people fear their government, there is tyranny; when
> > the
> > > > > >>> government fears the people, there is liberty." [Thomas
> > Jefferson]
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> "When the people fear their government, there is tyranny; when
> > the
> > > > > >> government fears the people, there is liberty." [Thomas
> > Jefferson]
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > "When the people fear their government, there is tyranny; when
> the
> > > > > > government fears the people, there is liberty." [Thomas
> Jefferson]
> > > > > >
> > > > > > ________________________________
> > > > > >
> > > > > > Your Personal Data: We may collect and process information about
> > you
> > > > > that may be subject to data protection laws. For more information
> > > about how
> > > > > we use and disclose your personal data, how we protect your
> > > information,
> > > > > our legal basis to use your information, your rights and who you
> can
> > > > > contact, please refer to: www.gs.com/privacy-notices<
> > > > >
> > http://www.gs.com/privacy-notices
> > >
> > > > > >
> > > > > >
> > > > > > ---------- Forwarded message ----------
> > > > > > From: Rajini Sivaram <ra...@gmail.com>
> > > > > > To: "Skrzypek, Jonathan" <Jo...@ln.email.gs.com>,
> dev
> > <
> > > > > dev@kafka.apache.org>, Ismael Juma <is...@juma.me.uk>
> > > > > > Cc:
> > > > > > Bcc:
> > > > > > Date: Tue, 22 May 2018 15:05:07 +0000
> > > > > > Subject: Re: FW: [VOTE] KIP-235 Add DNS alias support for secured
> > > > > connection
> > > > > > Hi Jonathan,
> > > > > >
> > > > > > I think it would make sense to convert the config in this KIP
> into
> > an
> > > > > enum so that we can add more variations later on. But since KIP-302
> > is
> > > > > still under discussion, it is not clear what the config name should
> > be.
> > > > > Since today is the KIP deadline and the implementation itself is
> > > > > straightforward, it would make sense to progress with this one for
> > > 2.0.0 if
> > > > > we can get one more binding vote.
> > > > > >
> > > > > > Ismael, do you have time to take a look at KIP-235 today?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > >
> > > > > > On Tue, May 22, 2018 at 3:45 PM, Skrzypek, Jonathan <
> > > > > Jonathan.Skrzypek@gs.com> wrote:
> > > > > >>
> > > > > >> Hello Rajini,
> > > > > >>
> > > > > >> What do you think should be the next step here ?
> > > > > >>
> > > > > >>
> > > > > >> Jonathan Skrzypek
> > > > > >>
> > > > > >> -----Original Message-----
> > > > > >> From: Skrzypek, Jonathan [Tech]
> > > > > >> Sent: 21 May 2018 10:51
> > > > > >> To: 'dev'
> > > > > >> Subject: RE: [VOTE] KIP-235 Add DNS alias support for secured
> > > connection
> > > > > >>
> > > > > >> Hi,
> > > > > >>
> > > > > >> What would be the next step here ?
> > > > > >> I know there's a discussion going on around KIP-302, but I'm
> also
> > > > > conscious that the 2.0.0 deadline for KIPs is tomorrow.
> > > > > >> I've opened this KIP in January and discussions have been
> > productive
> > > > > with an end solution I had the impression was reasonable, so I am
> > keen
> > > to
> > > > > see it make it the next release.
> > > > > >>
> > > > > >>
> > > > > >> Jonathan Skrzypek
> > > > > >>
> > > > > >> -----Original Message-----
> > > > > >> From: Skrzypek, Jonathan [Tech]
> > > > > >> Sent: 14 May 2018 13:48
> > > > > >> To: dev
> > > > > >> Subject: RE: [VOTE] KIP-235 Add DNS alias support for secured
> > > connection
> > > > > >>
> > > > > >> Sure, I modified the KIP to add more details
> > > > > >>
> > > > > >>
> > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >
> > > 235%3A+Add+DNS+alias+support+for+secured+connection
> > > > > >>
> > > > > >>
> > > > > >> Jonathan Skrzypek
> > > > > >>
> > > > > >>
> > > > > >> -----Original Message-----
> > > > > >> From: Ismael Juma [mailto:ismael@juma.me.uk]
> > > > > >> Sent: 14 May 2018 11:53
> > > > > >> To: dev
> > > > > >> Subject: Re: [VOTE] KIP-235 Add DNS alias support for secured
> > > connection
> > > > > >>
> > > > > >> Thanks for the KIP, Jonathan. It would be helpful to have more
> > > detail on
> > > > > >> how SSL authentication could be broken if the new behaviour is
> > the
> > > > > default.
> > > > > >> I know this was discussed in the mailing list thread, but it's
> > > > > important to
> > > > > >> include it in the KIP since it's the main reason why a new
> config
> > is
> > > > > needed
> > > > > >> (and configs should be avoided whenever we can just do the right
> > > thing).
> > > > > >>
> > > > > >> Ismael
> > > > > >>
> > > > > >> On Fri, Mar 23, 2018 at 12:05 PM Skrzypek, Jonathan <
> > > > > >> Jonathan.Skrzypek@gs.com> wrote:
> > > > > >>
> > > > > >> > Hi,
> > > > > >> >
> > > > > >> > I would like to start a vote for KIP-235
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > INVALID URI REMOVED.
> > > apache.org_confluence_display_KAFKA_KIP-2D235-253A-2BAdd-
> > > 2BDNS-2Balias-2Bsupport-2Bfor-2Bsecured-2Bconnection&d=DwIBaQ&c=
> > > 7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_
> QFAPdxGlafmDu9_
> > > r6eaCbPOM0NM1EHo-E&m=FM_uCHnnO2dqxWC0bi7_QOJKfKmQI80-
> > > Xduvb-URWOw&s=RpGkijfK-WHcU0s8ZtMXEkIr69QraJhYKaGSC9V_rnI&e=
> > > > > >> >
> > > > > >> > This is a proposition to add an option for reverse dns lookup
> > of
> > > > > >> > bootstrap.servers hosts, allowing the use of dns aliases on
> > > clusters
> > > > > using
> > > > > >> > SASL authentication.
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >> ________________________________
> > > > > >>
> > > > > >> Your Personal Data: We may collect and process information about
> > you
> > > > > that may be subject to data protection laws. For more information
> > > about how
> > > > > we use and disclose your personal data, how we protect your
> > > information,
> > > > > our legal basis to use your information, your rights and who you
> can
> > > > > contact, please refer to: www.gs.com/privacy-notices<
> > > > >
> > http://www.gs.com/privacy-notices
> > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > "When the people fear their government, there is tyranny; when the
> > > > > government fears the people, there is liberty." [Thomas Jefferson]
> > > > >
> > > >
> > > >
> > > > --
> > > > "When the people fear their government, there is tyranny; when the
> > > > government fears the people, there is liberty." [Thomas Jefferson]
> > >
> >
> >
> >
> > Unless stated otherwise above:
> > IBM United Kingdom Limited - Registered in England and Wales with number
> > 741598.
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> 3AU
> >
>
Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS resolved
IP addresses
Posted by Eno Thereska <en...@gmail.com>.
Got it, thanks!
Eno
On Tue, Sep 18, 2018 at 1:41 PM, Edoardo Comar <EC...@uk.ibm.com> wrote:
> Hi Eno,
>
> we implement network separation and the machines fronting the brokers
> (call them LBs for simplicity) will route the connection to the correct
> broker based on the TLS SNI. We register in the DNS multiple A records
> (the IPs of all LBs) for each the brokers' hostnames.
> As long as all the brokers are up, the cluster is fully functioning even
> if just one of the LBs is up.
>
> HTH,
> Edo
> --------------------------------------------------
>
> Edoardo Comar
>
> IBM Message Hub
>
> IBM UK Ltd, Hursley Park, SO21 2JN
>
>
>
> From: Eno Thereska <en...@gmail.com>
> To: dev@kafka.apache.org
> Date: 18/09/2018 10:24
> Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all
> DNS resolved IP addresses
>
>
>
> Hi folks,
>
> Could you expand the motivation a bit? When would it make sense to use an
> LB in front of Kafka brokers? A client needs to access each broker
> directly
> to consume the data in that broker and cannot be redirected to another
> broker. What exact scenario are you seeing that needs this KIP?
>
> Thanks
> Eno
>
> On Tue, Sep 18, 2018 at 9:52 AM, Mickael Maison <mi...@gmail.com>
> wrote:
>
> > Bumping this thread
> >
> > It's a relatively small change that would help cloud environments with
> > load balancers fronting brokers
> > On Tue, Sep 11, 2018 at 3:01 PM Edoardo Comar <ed...@gmail.com>
> wrote:
> > >
> > > Hi all,
> > > after some time we updated KIP-302 to reuse the config key introduced
> by
> > > KIP-235, with a different value to avoid conflicts between the two.
> > > Also clarified the use of multiple IPs only of the same type
> (IPv4/IPv6).
> > >
> > > We'd appreciate a further review and discussion.
> > > Thanks!
> > > Edo
> > >
> > >
> > > On Fri, 25 May 2018 at 12:36, Edoardo Comar <ed...@gmail.com>
> wrote:
> > >
> > > > Hi Jonathan,
> > > > I'm ok with an expandable enum for the config that could be extended
> > > > in the future.
> > > > It is marginally better than multiple potentially conflicting config
> > > > entries.
> > > >
> > > > Though as I think the change for KIP-302 is independent from KIP-235
> > > > and they do not conflict,
> > > > when we'll look back at it post 2.0 we may see if it is more
> valuable
> > > > to shoehorn its config in an expanded enum or not
> > > >
> > > > thanks,
> > > > Edo
> > > >
> > > > On 24 May 2018 at 16:50, Skrzypek, Jonathan
> <Jo...@gs.com>
> > > > wrote:
> > > > > Hi,
> > > > >
> > > > > As Rajini suggested in the thread for KIP 235 (attached), we could
> > try
> > > > to have an enum that would drive what does the client
> expands/resolves.
> > > > >
> > > > > I suggest a client config called client.dns.lookup with different
> > values
> > > > possible :
> > > > >
> > > > > - no : no dns lookup
> > > > > - hostnames.only : perform dns lookup on both bootstrap.servers
> and
> > > > advertised listeners
> > > > > - canonical.hostnames.only : perform dns lookup on both
> > > > bootstrap.servers and advertised listeners
> > > > > - bootstrap.hostnames.only : perform dns lookup on
> bootstrap.servers
> > > > list and expand it
> > > > > - bootstrap.canonical.hostnames.only : perform dns lookup on
> > > > bootstrap.servers list and expand it
> > > > > - advertised.listeners.hostnames.only : perform dns lookup on
> > advertised
> > > > listeners
> > > > > - advertised.listeners.canonical.hostnames.only : perform dns
> > lookup on
> > > > advertised listeners
> > > > >
> > > > > I realize this is a bit heavy but this gives users the ability to
> > pick
> > > > and choose.
> > > > > I didn't include a setting to mix hostnames and canonical
> hostnames
> > as
> > > > I'm not sure there would be a valid use case.
> > > > >
> > > > > Alternatively, to have less possible values, we could have 2
> > parameters :
> > > > >
> > > > > - dns.lookup.type with values : hostname / canonical.host.name
> > > > > - dns.lookup.behaviour : bootstrap.servers, advertised.listeners,
> > both
> > > > >
> > > > > Thoughts ?
> > > > >
> > > > > Jonathan Skrzypek
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > > > Sent: 17 May 2018 23:50
> > > > > To: dev@kafka.apache.org
> > > > > Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all
> DNS
> > > > resolved IP addresses
> > > > >
> > > > > Hi Jonathan,
> > > > >
> > > > >> A solution might be to expose to users the choice of using
> hostname
> > or
> > > > canonical host name on both sides.
> > > > >> Say having one setting that collapses functionalities from both
> KIPs
> > > > (bootstrap expansion + advertised lookup)
> > > > >> and an additional parameter that defines how the resolution is
> > > > performed, using getCanonicalHostName() or not.
> > > > >
> > > > > thanks sounds to me *less* simple than independent config options,
> > sorry.
> > > > >
> > > > > I would like to say once again that by itself KIP-302 only speeds
> up
> > > > > the client behavior that can happen anyway when the client
> restarts
> > > > > multiple times,
> > > > > as every time there is no guarantee that - in presence of multiple
> A
> > > > > DNS records - the same IP is returned. Attempting to use additiona
> > IPs
> > > > > if the first fail just makes client recovery faster.
> > > > >
> > > > > cheers
> > > > > Edo
> > > > >
> > > > > On 17 May 2018 at 12:12, Skrzypek, Jonathan <
> > Jonathan.Skrzypek@gs.com>
> > > > wrote:
> > > > >> Yes, makes sense.
> > > > >> You mentioned multiple times you see no overlap and no issue with
> > your
> > > > KIP, and that they solve different use cases.
> > > > >>
> > > > >> Appreciate you have an existing use case that would work, but we
> > need
> > > > to make sure this isn't confusing to users and that any combination
> > will
> > > > always work, across security protocols.
> > > > >>
> > > > >> A solution might be to expose to users the choice of using
> hostname
> > or
> > > > canonical host name on both sides.
> > > > >> Say having one setting that collapses functionalities from both
> KIPs
> > > > (bootstrap expansion + advertised lookup) and an additional
> parameter
> > that
> > > > defines how the resolution is performed, using
> getCanonicalHostName()
> > or
> > > > not.
> > > > >>
> > > > >> Maybe that gives less flexibility as users wouldn't be able to
> > decide
> > > > to only perform DNS lookup on bootstrap.servers or on advertised
> > listeners.
> > > > >> But this would ensure consistency so that a user can decide to
> use
> > > > cnames or not (depending on their certificates and Kerberos
> principals
> > in
> > > > their environment) and it would work.
> > > > >>
> > > > >> Jonathan Skrzypek
> > > > >>
> > > > >> -----Original Message-----
> > > > >> From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > > >> Sent: 16 May 2018 21:59
> > > > >> To: dev@kafka.apache.org
> > > > >> Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all
> DNS
> > > > resolved IP addresses
> > > > >>
> > > > >> Hi Jonathan,
> > > > >> I am afraid that may not work for everybody.
> > > > >>
> > > > >> It would not work for us.
> > > > >> With our current DNS, my Kafka clients are perfectly happy to use
> > any
> > > > IPs -
> > > > >> DNS has multiple A records for the 'myhostname.mydomain' used for
> > > > >> bootstrap and advertised listeners.
> > > > >> The hosts all serve TLS certificates that include
> > > > >> 'myhostname.mydomain' and the clients are happy.
> > > > >>
> > > > >> However, applying getCanonicalHostName to those IPs would return
> > > > >> hostnames that would not match the TLS certificates.
> > > > >>
> > > > >> So once again I believe your solution and ours solve different
> use
> > > > cases.
> > > > >>
> > > > >> cheers
> > > > >> Edo
> > > > >>
> > > > >> On 16 May 2018 at 18:29, Skrzypek, Jonathan <
> > Jonathan.Skrzypek@gs.com>
> > > > wrote:
> > > > >>> I think there are combinations that will break SASL and SSL
> auth.
> > > > >>> Could the trick be to have a single parameter that triggers dns
> > > > resolve both for bootstrap and advertised listeners, both using
> > > > getCanonicalHostName() ?
> > > > >>>
> > > > >>> Jonathan Skrzypek
> > > > >>>
> > > > >>> -----Original Message-----
> > > > >>> From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > > >>> Sent: 16 May 2018 17:03
> > > > >>> To: dev@kafka.apache.org
> > > > >>> Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all
> > DNS
> > > > resolved IP addresses
> > > > >>>
> > > > >>> Hi Rajini,
> > > > >>>
> > > > >>> In your example KIP-302 would attempt to connect to the first
> > address
> > > > >>> returned, let's say
> > > > >>>
> > > > >>> www.apache.org/195.154.151.36
> > > > >>>
> > > > >>> then, only if that fails, will in turn try the remaining:
> > > > >>>
> > > > >>> www.apache.org/40.79.78.1
> > > > >>> www.apache.org/140.211.11.105
> > > > >>> www.apache.org/2001:bc8:2142:300:0:0:0:0
> > > > >>>
> > > > >>> You're right to say that we expect certificates served by those
> > > > >>> endpoints to be valid for "www.apache.org"
> > > > >>>
> > > > >>> Without KIP-302, only one would be attempted.
> > > > >>> Which is the first one, that can change every time
> > > > >>> (typically changes on every Java process restart,
> > > > >>> but may change also any time InetAddress.getAllByName it's
> invoked
> > > > >>> depending on the caching).
> > > > >>>
> > > > >>> The behavioral change that KIP-302 may introduce is that in the
> > > > example above,
> > > > >>> also an IPv6 connection may be attempted after some IPv4 ones.
> > > > >>>
> > > > >>> InetAddress.getAllByName() implementation uses a system property
> > > > >>> "java.net.preferIPv6Addresses"
> > > > >>> to decide which type of address to return first (default is
> still
> > IPv4
> > > > >>> in java 10)
> > > > >>>
> > > > >>> We will amend the KIP and PR so that the loop only uses IPs of
> the
> > > > >>> same type as the first one returned.
> > > > >>>
> > > > >>> A part from the above, KIP 302 does not seem to change any
> existing
> > > > >>> client behaviour, as any one of multiple IP addresses (of a
> given
> > > > >>> v4/v6 type) can currently be picked.
> > > > >>> We're happy however to keep the looping behavior optional with
> the
> > > > >>> discussed config property, disabled by default.
> > > > >>>
> > > > >>> As for KIP-235 that may introduce new hostnames in the bootstrap
> > list
> > > > >>> (the current PR rewrites the bootstrap list)
> > > > >>> and we fail to see the conflict with KIP-302, whatever the set
> of
> > > > >>> configs chosen.
> > > > >>>
> > > > >>> We'd be happy to try understand what we are missing in a KIP
> call
> > :-)
> > > > >>>
> > > > >>> cheers
> > > > >>> Edo
> > > > >>>
> > > > >>> On 15 May 2018 at 16:58, Rajini Sivaram
> <ra...@gmail.com>
> > > > wrote:
> > > > >>>> Hi Edo,
> > > > >>>>
> > > > >>>> I agree that KIP-235 and KIP-302 address different scenarios.
> And
> > I
> > > > agree
> > > > >>>> that each one is not sufficient in itself to address both the
> > > > scenarios.
> > > > >>>> But I also think that they conflict and hence they need to be
> > looked
> > > > at
> > > > >>>> together and perhaps use a single config.
> > > > >>>>
> > > > >>>> As an example:
> > > > >>>>
> > > > >>>> If I run:
> > > > >>>>
> > > > >>>> for (InetAddress address : InetAddress.getAllByName("www.
> > apache.org"))
> > > > {
> > > > >>>> System.out.printf("HostName %s canonicalHostName %s IP
> %s\n",
> > > > >>>> address.getHostName(),
> address.getCanonicalHostName()
> > ,
> > > > >>>> address.getHostAddress());
> > > > >>>> }
> > > > >>>>
> > > > >>>> I get:
> > > > >>>>
> > > > >>>> HostName www.apache.org canonicalHostName
> tlp-eu-west.apache.org
> > IP
> > > > >>>> 195.154.151.36
> > > > >>>> HostName www.apache.org canonicalHostName 40.79.78.1 IP
> > 40.79.78.1
> > > > >>>> HostName www.apache.org canonicalHostName themis.apache.org IP
> > > > >>>> 140.211.11.105
> > > > >>>> HostName www.apache.org canonicalHostName
> > 2001:bc8:2142:300:0:0:0:0
> > > > IP
> > > > >>>> 2001:bc8:2142:300:0:0:0:0
> > > > >>>>
> > > > >>>>
> > > > >>>> If www.apache.org is used as a bootstrap address, KIP-302 would
> > > > connect to (
> > > > >>>> www.apache.org/195.154.151.36 and
> www.apache.org/140.211.11.105)
> > > > while
> > > > >>>> KIP-235 would connect to (tlp-eu-west.apache.org/195.154.151.3.
> > and
> > > > >>>> themis.apache.org/140.211.11.105). This is a significant
> > difference
> > > > not
> > > > >>>> just for Kerberos, but for any secure environment where
> hostname
> > is
> > > > >>>> verified to prevent man-in-the-middle attacks. In your case, I
> > > > presume you
> > > > >>>> would have SSL certificates with the equivalent of
> www.apache.org
> > on
> > > > both
> > > > >>>> the load balancers. In Jonathan's case, I presume he has
> Kerberos
> > > > >>>> principals for the equivalent of tlp-eu-west.apache.org and
> > > > >>>> themis.apache.org. We would want to support both scenarios
> > > > regardless of
> > > > >>>> the security protocol, just need to come up with configuration
> > > > options that
> > > > >>>> don't conflict.
> > > > >>>>
> > > > >>>>
> > > > >>>> On Mon, May 14, 2018 at 5:24 PM, Edoardo Comar <
> > edocomar@gmail.com>
> > > > wrote:
> > > > >>>>
> > > > >>>>> Thanks Rajini
> > > > >>>>>
> > > > >>>>> I still don't see the overlap between the two KIPS
> > > > >>>>>
> > > > >>>>> KIP-235 allows an expansion of hostnames on the bootstrap
> list.
> > > > >>>>>
> > > > >>>>> KIP-302 allows alternative IPs to be used to attempt a
> connection
> > > > >>>>> (either at bootstrap and when processing the MetadataResponse)
> > to a
> > > > >>>>> given hostname.
> > > > >>>>>
> > > > >>>>> A use case would be that of active/passive LB fronting the
> > brokers.
> > > > >>>>>
> > > > >>>>> Arguably, if Java honored the DNS-set TTL, and the TTL was low
> > and on
> > > > >>>>> subsequent requests, the order of IPs returned by the
> > > > >>>>> InetAddress.getAllByName() was random, we may not need such an
> > > > >>>>> enhancement.
> > > > >>>>> In practice, a Java client can get stuck on a "bad" IP forever
> > if it
> > > > >>>>> only relies on the first IP returned.
> > > > >>>>>
> > > > >>>>> HTH,
> > > > >>>>> Edo
> > > > >>>>>
> > > > >>>>> On 14 May 2018 at 16:23, Rajini Sivaram
> <rajinisivaram@gmail.com
> > >
> > > > wrote:
> > > > >>>>> > Hi Edo,
> > > > >>>>> >
> > > > >>>>> > Thanks for the KIP. I think it will be good to include a
> > diagram
> > > > to make
> > > > >>>>> it
> > > > >>>>> > easier to distinguish this scenario from that of KIP-235
> > without
> > > > reading
> > > > >>>>> > the PR.
> > > > >>>>> >
> > > > >>>>> > It may be worth considering if KIP-235 and this KIP could
> use a
> > > > single
> > > > >>>>> > config name with different values instead of two boolean
> > configs:
> > > > >>>>> >
> > > > >>>>> > bootstrap.reverse.dns.lookup = true/false
> > > > >>>>> > enable.all.dns.ips = true/false
> > > > >>>>> >
> > > > >>>>> > Not all values of (bootstrap.reverse.dns.lookup,
> > > > enable.all.dns.ips) seem
> > > > >>>>> > to make sense. And not all scenarios are handled. Even if we
> > use
> > > > multiple
> > > > >>>>> > configs, it seems to me that we may want to name them
> > differently.
> > > > >>>>> >
> > > > >>>>> > The possible combinations are:
> > > > >>>>> >
> > > > >>>>> > 1) Bootstrap
> > > > >>>>> >
> > > > >>>>> > a) No lookup
> > > > >>>>> > b) Use all DNS entries with host name
> > > > >>>>> > c) Use all DNS entries with canonical host name
> > > > >>>>> >
> > > > >>>>> > 2) Advertised listeners
> > > > >>>>> >
> > > > >>>>> > a) No lookup
> > > > >>>>> > b) Use all DNS entries with host name
> > > > >>>>> > c) Use all DNS entries with canonical host name
> > > > >>>>> >
> > > > >>>>> > The combinations that are enabled by the two boolean configs
> (
> > > > >>>>> > bootstrap.reverse.dns.lookup, enable.all.dns.ips) are:
> > > > >>>>> >
> > > > >>>>> > - (false, false) => (1a, 2a)
> > > > >>>>> > - (true, false) => (1c, 2a)
> > > > >>>>> > - (false, true) => (1b, 2b)
> > > > >>>>> > - (true, true) => (??, 2b)
> > > > >>>>> >
> > > > >>>>> > It will be good if we can clearly identify which
> combinations
> > we
> > > > want to
> > > > >>>>> > support and the scenarios where they may be useful. Perhaps
> > (1a,
> > > > 2a),
> > > > >>>>> (1c,
> > > > >>>>> > 2a), (1b, 2b) and (1c, 2c) are useful?
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> > On Mon, May 14, 2018 at 2:58 PM, Skrzypek, Jonathan <
> > > > >>>>> > Jonathan.Skrzypek@gs.com> wrote:
> > > > >>>>> >
> > > > >>>>> >> Ah, apologies didn't see there was already a decent amount
> of
> > > > discussion
> > > > >>>>> >> on this in the PR.
> > > > >>>>> >>
> > > > >>>>> >> This kind of sounds related to the environment you're
> running
> > to
> > > > me.
> > > > >>>>> >> What is the rationale behind using the advertised listeners
> > to do
> > > > your
> > > > >>>>> >> load balancing advertisement rather than a top level alias
> > that
> > > > has
> > > > >>>>> >> everything ?
> > > > >>>>> >>
> > > > >>>>> >> It sounds like in your case there is a mismatch between
> > > > >>>>> bootstrap.servers
> > > > >>>>> >> and advertised.listeners, and you want advertised.listeners
> to
> > > > take
> > > > >>>>> >> precedence and have the client iterate over what is
> returned
> > by
> > > > the
> > > > >>>>> broker.
> > > > >>>>> >> So the extra parameter doesn't only have to do with DNS but
> > it's
> > > > also
> > > > >>>>> >> appending from the broker, maybe the parameter name should
> > > > reflect this
> > > > >>>>> ?
> > > > >>>>> >>
> > > > >>>>> >> Jonathan Skrzypek
> > > > >>>>> >>
> > > > >>>>> >>
> > > > >>>>> >> -----Original Message-----
> > > > >>>>> >> From: Skrzypek, Jonathan [Tech]
> > > > >>>>> >> Sent: 14 May 2018 14:46
> > > > >>>>> >> To: dev@kafka.apache.org
> > > > >>>>> >> Subject: RE: [DISCUSS] KIP-302 - Enable Kafka clients to
> use
> > all
> > > > DNS
> > > > >>>>> >> resolved IP addresses
> > > > >>>>> >>
> > > > >>>>> >> Hi,
> > > > >>>>> >>
> > > > >>>>> >> I see you noted the similarities with KIP-235.
> > > > >>>>> >> But KIP-235 might also solve what this KIP is trying to
> > achieve.
> > > > >>>>> >>
> > > > >>>>> >> When parsing bootstrap.servers, KIP-235 has the client add
> all
> > > > >>>>> underlying
> > > > >>>>> >> hostnames and IPs.
> > > > >>>>> >> And this happens before hitting the NetworkClient.
> > > > >>>>> >>
> > > > >>>>> >> So to me the client will try every single endpoint behind
> any
> > > > >>>>> >> bootstrap.servers record.
> > > > >>>>> >>
> > > > >>>>> >> See
> > > > INVALID URI REMOVED.
> > com_apache_kafka_pull_4485_commits_24757eb7b0&d=DwIBaQ&c=
> > 7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_
> > r6eaCbPOM0NM1EHo-E&m=_ud9m_JZJ87C7eGsKcmzgJgDpNQDIIv5R4i_
> > 7VlhkLc&s=TqaiA9uW_myYO6FN-gKPfPlioxZR6DhnlBTpEj5M2aQ&e=
> > > > >>>>> >>
> > > > 6bcf8c7d7649c85232c52b5d54f0e4#diff-89ef153462e64c250a21bd324ae1a851
> > > > >>>>> >> which calls getAllByName like you suggested
> > > > >>>>> >>
> > > > >>>>> >> Jonathan Skrzypek
> > > > >>>>> >>
> > > > >>>>> >>
> > > > >>>>> >> -----Original Message-----
> > > > >>>>> >> From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > > >>>>> >> Sent: 14 May 2018 14:17
> > > > >>>>> >> To: dev@kafka.apache.org
> > > > >>>>> >> Subject: [DISCUSS] KIP-302 - Enable Kafka clients to use
> all
> > DNS
> > > > >>>>> resolved
> > > > >>>>> >> IP addresses
> > > > >>>>> >>
> > > > >>>>> >> Hi all,
> > > > >>>>> >>
> > > > >>>>> >> We just opened a KIP to add support for the client to use
> all
> > IPs
> > > > >>>>> returned
> > > > >>>>> >> by DNS for the brokers
> > > > >>>>> >>
> > > > >>>>> >> The details are here -
> > > > >>>>> >>
> > > > >>>>> >>
> INVALID URI REMOVED
> > > > >>>>> >>
> pache.org_confluence_display_KAFKA_KIP-2D302-2B-2D-2BEnable-
> > > > >>>>> >> 2BKafka-2Bclients-2Bto-2Buse-2Ball-2BDNS-2Bresolved-2BIP-
> > > > >>>>> >>
> 2Baddresses&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxL
> > > > >>>>> >>
> xfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaCbPOM0NM1EHo-E&m=EJafF
> > > > >>>>> >> l1clRyolgtcu2uCc4_cIOJnlxb1r1n-D2Dti4k&s=C-UZ6KUG7JFiPD_
> > > > >>>>> >> CnHczDOVqH9-XC5f_OFkw4BTNrI4&e=
> > > > >>>>> >>
> > > > >>>>> >> The JIRA and provisional PR (where the discussion lead to
> the
> > > > creation
> > > > >>>>> of
> > > > >>>>> >> this KIP) are :
> > > > >>>>> >>
> > > > >>>>> >> INVALID URI REMOVED
> .
> > > > >>>>> >>
> apache.org_jira_browse_KAFKA-2D6863&d=DwIBaQ&c=7563p3e2zaQw0
> > > > >>>>> >>
> AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6
> > > > >>>>> >> eaCbPOM0NM1EHo-E&m=EJafFl1clRyolgtcu2uCc4_cIOJnlxb1r1n-
> > > > >>>>> >> D2Dti4k&s=3Puqs5iYoPsw6hARQr6gvokdFE-H5USMiNVGOUtNkJI&e=
> > > > >>>>> >>
> > > > >>>>> >> INVALID URI REMOVED
> .
> > > > >>>>> >>
> com_apache_kafka_pull_4987&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgy
> > > > >>>>> >> agb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaC
> > > > >>>>> >>
> bPOM0NM1EHo-E&m=EJafFl1clRyolgtcu2uCc4_cIOJnlxb1r1n-D2Dti4k&
> > > > >>>>> >> s=Hqn5dOgQy4-MHTIJLE49O8bNomry3SoGq9OVoHU-CRA&e=
> > > > >>>>> >>
> > > > >>>>> >> Looking forward to the community's feedback.
> > > > >>>>> >> It would be amazing to have it voted by May 22nd :-) :-)
> > > > >>>>> >>
> > > > >>>>> >> Edoardo & Mickael
> > > > >>>>> >>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> --
> > > > >>>>> "When the people fear their government, there is tyranny; when
> > the
> > > > >>>>> government fears the people, there is liberty." [Thomas
> > Jefferson]
> > > > >>>>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> "When the people fear their government, there is tyranny; when
> the
> > > > >>> government fears the people, there is liberty." [Thomas
> Jefferson]
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> "When the people fear their government, there is tyranny; when
> the
> > > > >> government fears the people, there is liberty." [Thomas
> Jefferson]
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > "When the people fear their government, there is tyranny; when the
> > > > > government fears the people, there is liberty." [Thomas Jefferson]
> > > > >
> > > > > ________________________________
> > > > >
> > > > > Your Personal Data: We may collect and process information about
> you
> > > > that may be subject to data protection laws. For more information
> > about how
> > > > we use and disclose your personal data, how we protect your
> > information,
> > > > our legal basis to use your information, your rights and who you can
> > > > contact, please refer to: www.gs.com/privacy-notices<
> > > >
> http://www.gs.com/privacy-notices
> >
> > > > >
> > > > >
> > > > > ---------- Forwarded message ----------
> > > > > From: Rajini Sivaram <ra...@gmail.com>
> > > > > To: "Skrzypek, Jonathan" <Jo...@ln.email.gs.com>, dev
> <
> > > > dev@kafka.apache.org>, Ismael Juma <is...@juma.me.uk>
> > > > > Cc:
> > > > > Bcc:
> > > > > Date: Tue, 22 May 2018 15:05:07 +0000
> > > > > Subject: Re: FW: [VOTE] KIP-235 Add DNS alias support for secured
> > > > connection
> > > > > Hi Jonathan,
> > > > >
> > > > > I think it would make sense to convert the config in this KIP into
> an
> > > > enum so that we can add more variations later on. But since KIP-302
> is
> > > > still under discussion, it is not clear what the config name should
> be.
> > > > Since today is the KIP deadline and the implementation itself is
> > > > straightforward, it would make sense to progress with this one for
> > 2.0.0 if
> > > > we can get one more binding vote.
> > > > >
> > > > > Ismael, do you have time to take a look at KIP-235 today?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Rajini
> > > > >
> > > > >
> > > > > On Tue, May 22, 2018 at 3:45 PM, Skrzypek, Jonathan <
> > > > Jonathan.Skrzypek@gs.com> wrote:
> > > > >>
> > > > >> Hello Rajini,
> > > > >>
> > > > >> What do you think should be the next step here ?
> > > > >>
> > > > >>
> > > > >> Jonathan Skrzypek
> > > > >>
> > > > >> -----Original Message-----
> > > > >> From: Skrzypek, Jonathan [Tech]
> > > > >> Sent: 21 May 2018 10:51
> > > > >> To: 'dev'
> > > > >> Subject: RE: [VOTE] KIP-235 Add DNS alias support for secured
> > connection
> > > > >>
> > > > >> Hi,
> > > > >>
> > > > >> What would be the next step here ?
> > > > >> I know there's a discussion going on around KIP-302, but I'm also
> > > > conscious that the 2.0.0 deadline for KIPs is tomorrow.
> > > > >> I've opened this KIP in January and discussions have been
> productive
> > > > with an end solution I had the impression was reasonable, so I am
> keen
> > to
> > > > see it make it the next release.
> > > > >>
> > > > >>
> > > > >> Jonathan Skrzypek
> > > > >>
> > > > >> -----Original Message-----
> > > > >> From: Skrzypek, Jonathan [Tech]
> > > > >> Sent: 14 May 2018 13:48
> > > > >> To: dev
> > > > >> Subject: RE: [VOTE] KIP-235 Add DNS alias support for secured
> > connection
> > > > >>
> > > > >> Sure, I modified the KIP to add more details
> > > > >>
> > > > >>
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> > 235%3A+Add+DNS+alias+support+for+secured+connection
> > > > >>
> > > > >>
> > > > >> Jonathan Skrzypek
> > > > >>
> > > > >>
> > > > >> -----Original Message-----
> > > > >> From: Ismael Juma [mailto:ismael@juma.me.uk]
> > > > >> Sent: 14 May 2018 11:53
> > > > >> To: dev
> > > > >> Subject: Re: [VOTE] KIP-235 Add DNS alias support for secured
> > connection
> > > > >>
> > > > >> Thanks for the KIP, Jonathan. It would be helpful to have more
> > detail on
> > > > >> how SSL authentication could be broken if the new behaviour is
> the
> > > > default.
> > > > >> I know this was discussed in the mailing list thread, but it's
> > > > important to
> > > > >> include it in the KIP since it's the main reason why a new config
> is
> > > > needed
> > > > >> (and configs should be avoided whenever we can just do the right
> > thing).
> > > > >>
> > > > >> Ismael
> > > > >>
> > > > >> On Fri, Mar 23, 2018 at 12:05 PM Skrzypek, Jonathan <
> > > > >> Jonathan.Skrzypek@gs.com> wrote:
> > > > >>
> > > > >> > Hi,
> > > > >> >
> > > > >> > I would like to start a vote for KIP-235
> > > > >> >
> > > > >> >
> > > > >> >
> > > > INVALID URI REMOVED.
> > apache.org_confluence_display_KAFKA_KIP-2D235-253A-2BAdd-
> > 2BDNS-2Balias-2Bsupport-2Bfor-2Bsecured-2Bconnection&d=DwIBaQ&c=
> > 7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_
> > r6eaCbPOM0NM1EHo-E&m=FM_uCHnnO2dqxWC0bi7_QOJKfKmQI80-
> > Xduvb-URWOw&s=RpGkijfK-WHcU0s8ZtMXEkIr69QraJhYKaGSC9V_rnI&e=
> > > > >> >
> > > > >> > This is a proposition to add an option for reverse dns lookup
> of
> > > > >> > bootstrap.servers hosts, allowing the use of dns aliases on
> > clusters
> > > > using
> > > > >> > SASL authentication.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >>
> > > > >> ________________________________
> > > > >>
> > > > >> Your Personal Data: We may collect and process information about
> you
> > > > that may be subject to data protection laws. For more information
> > about how
> > > > we use and disclose your personal data, how we protect your
> > information,
> > > > our legal basis to use your information, your rights and who you can
> > > > contact, please refer to: www.gs.com/privacy-notices<
> > > >
> http://www.gs.com/privacy-notices
> >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > "When the people fear their government, there is tyranny; when the
> > > > government fears the people, there is liberty." [Thomas Jefferson]
> > > >
> > >
> > >
> > > --
> > > "When the people fear their government, there is tyranny; when the
> > > government fears the people, there is liberty." [Thomas Jefferson]
> >
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS resolved IP
addresses
Posted by Edoardo Comar <EC...@uk.ibm.com>.
Hi Eno,
we implement network separation and the machines fronting the brokers
(call them LBs for simplicity) will route the connection to the correct
broker based on the TLS SNI. We register in the DNS multiple A records
(the IPs of all LBs) for each the brokers' hostnames.
As long as all the brokers are up, the cluster is fully functioning even
if just one of the LBs is up.
HTH,
Edo
--------------------------------------------------
Edoardo Comar
IBM Message Hub
IBM UK Ltd, Hursley Park, SO21 2JN
From: Eno Thereska <en...@gmail.com>
To: dev@kafka.apache.org
Date: 18/09/2018 10:24
Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all
DNS resolved IP addresses
Hi folks,
Could you expand the motivation a bit? When would it make sense to use an
LB in front of Kafka brokers? A client needs to access each broker
directly
to consume the data in that broker and cannot be redirected to another
broker. What exact scenario are you seeing that needs this KIP?
Thanks
Eno
On Tue, Sep 18, 2018 at 9:52 AM, Mickael Maison <mi...@gmail.com>
wrote:
> Bumping this thread
>
> It's a relatively small change that would help cloud environments with
> load balancers fronting brokers
> On Tue, Sep 11, 2018 at 3:01 PM Edoardo Comar <ed...@gmail.com>
wrote:
> >
> > Hi all,
> > after some time we updated KIP-302 to reuse the config key introduced
by
> > KIP-235, with a different value to avoid conflicts between the two.
> > Also clarified the use of multiple IPs only of the same type
(IPv4/IPv6).
> >
> > We'd appreciate a further review and discussion.
> > Thanks!
> > Edo
> >
> >
> > On Fri, 25 May 2018 at 12:36, Edoardo Comar <ed...@gmail.com>
wrote:
> >
> > > Hi Jonathan,
> > > I'm ok with an expandable enum for the config that could be extended
> > > in the future.
> > > It is marginally better than multiple potentially conflicting config
> > > entries.
> > >
> > > Though as I think the change for KIP-302 is independent from KIP-235
> > > and they do not conflict,
> > > when we'll look back at it post 2.0 we may see if it is more
valuable
> > > to shoehorn its config in an expanded enum or not
> > >
> > > thanks,
> > > Edo
> > >
> > > On 24 May 2018 at 16:50, Skrzypek, Jonathan
<Jo...@gs.com>
> > > wrote:
> > > > Hi,
> > > >
> > > > As Rajini suggested in the thread for KIP 235 (attached), we could
> try
> > > to have an enum that would drive what does the client
expands/resolves.
> > > >
> > > > I suggest a client config called client.dns.lookup with different
> values
> > > possible :
> > > >
> > > > - no : no dns lookup
> > > > - hostnames.only : perform dns lookup on both bootstrap.servers
and
> > > advertised listeners
> > > > - canonical.hostnames.only : perform dns lookup on both
> > > bootstrap.servers and advertised listeners
> > > > - bootstrap.hostnames.only : perform dns lookup on
bootstrap.servers
> > > list and expand it
> > > > - bootstrap.canonical.hostnames.only : perform dns lookup on
> > > bootstrap.servers list and expand it
> > > > - advertised.listeners.hostnames.only : perform dns lookup on
> advertised
> > > listeners
> > > > - advertised.listeners.canonical.hostnames.only : perform dns
> lookup on
> > > advertised listeners
> > > >
> > > > I realize this is a bit heavy but this gives users the ability to
> pick
> > > and choose.
> > > > I didn't include a setting to mix hostnames and canonical
hostnames
> as
> > > I'm not sure there would be a valid use case.
> > > >
> > > > Alternatively, to have less possible values, we could have 2
> parameters :
> > > >
> > > > - dns.lookup.type with values : hostname / canonical.host.name
> > > > - dns.lookup.behaviour : bootstrap.servers, advertised.listeners,
> both
> > > >
> > > > Thoughts ?
> > > >
> > > > Jonathan Skrzypek
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > > Sent: 17 May 2018 23:50
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all
DNS
> > > resolved IP addresses
> > > >
> > > > Hi Jonathan,
> > > >
> > > >> A solution might be to expose to users the choice of using
hostname
> or
> > > canonical host name on both sides.
> > > >> Say having one setting that collapses functionalities from both
KIPs
> > > (bootstrap expansion + advertised lookup)
> > > >> and an additional parameter that defines how the resolution is
> > > performed, using getCanonicalHostName() or not.
> > > >
> > > > thanks sounds to me *less* simple than independent config options,
> sorry.
> > > >
> > > > I would like to say once again that by itself KIP-302 only speeds
up
> > > > the client behavior that can happen anyway when the client
restarts
> > > > multiple times,
> > > > as every time there is no guarantee that - in presence of multiple
A
> > > > DNS records - the same IP is returned. Attempting to use additiona
> IPs
> > > > if the first fail just makes client recovery faster.
> > > >
> > > > cheers
> > > > Edo
> > > >
> > > > On 17 May 2018 at 12:12, Skrzypek, Jonathan <
> Jonathan.Skrzypek@gs.com>
> > > wrote:
> > > >> Yes, makes sense.
> > > >> You mentioned multiple times you see no overlap and no issue with
> your
> > > KIP, and that they solve different use cases.
> > > >>
> > > >> Appreciate you have an existing use case that would work, but we
> need
> > > to make sure this isn't confusing to users and that any combination
> will
> > > always work, across security protocols.
> > > >>
> > > >> A solution might be to expose to users the choice of using
hostname
> or
> > > canonical host name on both sides.
> > > >> Say having one setting that collapses functionalities from both
KIPs
> > > (bootstrap expansion + advertised lookup) and an additional
parameter
> that
> > > defines how the resolution is performed, using
getCanonicalHostName()
> or
> > > not.
> > > >>
> > > >> Maybe that gives less flexibility as users wouldn't be able to
> decide
> > > to only perform DNS lookup on bootstrap.servers or on advertised
> listeners.
> > > >> But this would ensure consistency so that a user can decide to
use
> > > cnames or not (depending on their certificates and Kerberos
principals
> in
> > > their environment) and it would work.
> > > >>
> > > >> Jonathan Skrzypek
> > > >>
> > > >> -----Original Message-----
> > > >> From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > >> Sent: 16 May 2018 21:59
> > > >> To: dev@kafka.apache.org
> > > >> Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all
DNS
> > > resolved IP addresses
> > > >>
> > > >> Hi Jonathan,
> > > >> I am afraid that may not work for everybody.
> > > >>
> > > >> It would not work for us.
> > > >> With our current DNS, my Kafka clients are perfectly happy to use
> any
> > > IPs -
> > > >> DNS has multiple A records for the 'myhostname.mydomain' used for
> > > >> bootstrap and advertised listeners.
> > > >> The hosts all serve TLS certificates that include
> > > >> 'myhostname.mydomain' and the clients are happy.
> > > >>
> > > >> However, applying getCanonicalHostName to those IPs would return
> > > >> hostnames that would not match the TLS certificates.
> > > >>
> > > >> So once again I believe your solution and ours solve different
use
> > > cases.
> > > >>
> > > >> cheers
> > > >> Edo
> > > >>
> > > >> On 16 May 2018 at 18:29, Skrzypek, Jonathan <
> Jonathan.Skrzypek@gs.com>
> > > wrote:
> > > >>> I think there are combinations that will break SASL and SSL
auth.
> > > >>> Could the trick be to have a single parameter that triggers dns
> > > resolve both for bootstrap and advertised listeners, both using
> > > getCanonicalHostName() ?
> > > >>>
> > > >>> Jonathan Skrzypek
> > > >>>
> > > >>> -----Original Message-----
> > > >>> From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > >>> Sent: 16 May 2018 17:03
> > > >>> To: dev@kafka.apache.org
> > > >>> Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all
> DNS
> > > resolved IP addresses
> > > >>>
> > > >>> Hi Rajini,
> > > >>>
> > > >>> In your example KIP-302 would attempt to connect to the first
> address
> > > >>> returned, let's say
> > > >>>
> > > >>> www.apache.org/195.154.151.36
> > > >>>
> > > >>> then, only if that fails, will in turn try the remaining:
> > > >>>
> > > >>> www.apache.org/40.79.78.1
> > > >>> www.apache.org/140.211.11.105
> > > >>> www.apache.org/2001:bc8:2142:300:0:0:0:0
> > > >>>
> > > >>> You're right to say that we expect certificates served by those
> > > >>> endpoints to be valid for "www.apache.org"
> > > >>>
> > > >>> Without KIP-302, only one would be attempted.
> > > >>> Which is the first one, that can change every time
> > > >>> (typically changes on every Java process restart,
> > > >>> but may change also any time InetAddress.getAllByName it's
invoked
> > > >>> depending on the caching).
> > > >>>
> > > >>> The behavioral change that KIP-302 may introduce is that in the
> > > example above,
> > > >>> also an IPv6 connection may be attempted after some IPv4 ones.
> > > >>>
> > > >>> InetAddress.getAllByName() implementation uses a system property
> > > >>> "java.net.preferIPv6Addresses"
> > > >>> to decide which type of address to return first (default is
still
> IPv4
> > > >>> in java 10)
> > > >>>
> > > >>> We will amend the KIP and PR so that the loop only uses IPs of
the
> > > >>> same type as the first one returned.
> > > >>>
> > > >>> A part from the above, KIP 302 does not seem to change any
existing
> > > >>> client behaviour, as any one of multiple IP addresses (of a
given
> > > >>> v4/v6 type) can currently be picked.
> > > >>> We're happy however to keep the looping behavior optional with
the
> > > >>> discussed config property, disabled by default.
> > > >>>
> > > >>> As for KIP-235 that may introduce new hostnames in the bootstrap
> list
> > > >>> (the current PR rewrites the bootstrap list)
> > > >>> and we fail to see the conflict with KIP-302, whatever the set
of
> > > >>> configs chosen.
> > > >>>
> > > >>> We'd be happy to try understand what we are missing in a KIP
call
> :-)
> > > >>>
> > > >>> cheers
> > > >>> Edo
> > > >>>
> > > >>> On 15 May 2018 at 16:58, Rajini Sivaram
<ra...@gmail.com>
> > > wrote:
> > > >>>> Hi Edo,
> > > >>>>
> > > >>>> I agree that KIP-235 and KIP-302 address different scenarios.
And
> I
> > > agree
> > > >>>> that each one is not sufficient in itself to address both the
> > > scenarios.
> > > >>>> But I also think that they conflict and hence they need to be
> looked
> > > at
> > > >>>> together and perhaps use a single config.
> > > >>>>
> > > >>>> As an example:
> > > >>>>
> > > >>>> If I run:
> > > >>>>
> > > >>>> for (InetAddress address : InetAddress.getAllByName("www.
> apache.org"))
> > > {
> > > >>>> System.out.printf("HostName %s canonicalHostName %s IP
%s\n",
> > > >>>> address.getHostName(),
address.getCanonicalHostName()
> ,
> > > >>>> address.getHostAddress());
> > > >>>> }
> > > >>>>
> > > >>>> I get:
> > > >>>>
> > > >>>> HostName www.apache.org canonicalHostName
tlp-eu-west.apache.org
> IP
> > > >>>> 195.154.151.36
> > > >>>> HostName www.apache.org canonicalHostName 40.79.78.1 IP
> 40.79.78.1
> > > >>>> HostName www.apache.org canonicalHostName themis.apache.org IP
> > > >>>> 140.211.11.105
> > > >>>> HostName www.apache.org canonicalHostName
> 2001:bc8:2142:300:0:0:0:0
> > > IP
> > > >>>> 2001:bc8:2142:300:0:0:0:0
> > > >>>>
> > > >>>>
> > > >>>> If www.apache.org is used as a bootstrap address, KIP-302 would
> > > connect to (
> > > >>>> www.apache.org/195.154.151.36 and
www.apache.org/140.211.11.105)
> > > while
> > > >>>> KIP-235 would connect to (tlp-eu-west.apache.org/195.154.151.3.
> and
> > > >>>> themis.apache.org/140.211.11.105). This is a significant
> difference
> > > not
> > > >>>> just for Kerberos, but for any secure environment where
hostname
> is
> > > >>>> verified to prevent man-in-the-middle attacks. In your case, I
> > > presume you
> > > >>>> would have SSL certificates with the equivalent of
www.apache.org
> on
> > > both
> > > >>>> the load balancers. In Jonathan's case, I presume he has
Kerberos
> > > >>>> principals for the equivalent of tlp-eu-west.apache.org and
> > > >>>> themis.apache.org. We would want to support both scenarios
> > > regardless of
> > > >>>> the security protocol, just need to come up with configuration
> > > options that
> > > >>>> don't conflict.
> > > >>>>
> > > >>>>
> > > >>>> On Mon, May 14, 2018 at 5:24 PM, Edoardo Comar <
> edocomar@gmail.com>
> > > wrote:
> > > >>>>
> > > >>>>> Thanks Rajini
> > > >>>>>
> > > >>>>> I still don't see the overlap between the two KIPS
> > > >>>>>
> > > >>>>> KIP-235 allows an expansion of hostnames on the bootstrap
list.
> > > >>>>>
> > > >>>>> KIP-302 allows alternative IPs to be used to attempt a
connection
> > > >>>>> (either at bootstrap and when processing the MetadataResponse)
> to a
> > > >>>>> given hostname.
> > > >>>>>
> > > >>>>> A use case would be that of active/passive LB fronting the
> brokers.
> > > >>>>>
> > > >>>>> Arguably, if Java honored the DNS-set TTL, and the TTL was low
> and on
> > > >>>>> subsequent requests, the order of IPs returned by the
> > > >>>>> InetAddress.getAllByName() was random, we may not need such an
> > > >>>>> enhancement.
> > > >>>>> In practice, a Java client can get stuck on a "bad" IP forever
> if it
> > > >>>>> only relies on the first IP returned.
> > > >>>>>
> > > >>>>> HTH,
> > > >>>>> Edo
> > > >>>>>
> > > >>>>> On 14 May 2018 at 16:23, Rajini Sivaram
<rajinisivaram@gmail.com
> >
> > > wrote:
> > > >>>>> > Hi Edo,
> > > >>>>> >
> > > >>>>> > Thanks for the KIP. I think it will be good to include a
> diagram
> > > to make
> > > >>>>> it
> > > >>>>> > easier to distinguish this scenario from that of KIP-235
> without
> > > reading
> > > >>>>> > the PR.
> > > >>>>> >
> > > >>>>> > It may be worth considering if KIP-235 and this KIP could
use a
> > > single
> > > >>>>> > config name with different values instead of two boolean
> configs:
> > > >>>>> >
> > > >>>>> > bootstrap.reverse.dns.lookup = true/false
> > > >>>>> > enable.all.dns.ips = true/false
> > > >>>>> >
> > > >>>>> > Not all values of (bootstrap.reverse.dns.lookup,
> > > enable.all.dns.ips) seem
> > > >>>>> > to make sense. And not all scenarios are handled. Even if we
> use
> > > multiple
> > > >>>>> > configs, it seems to me that we may want to name them
> differently.
> > > >>>>> >
> > > >>>>> > The possible combinations are:
> > > >>>>> >
> > > >>>>> > 1) Bootstrap
> > > >>>>> >
> > > >>>>> > a) No lookup
> > > >>>>> > b) Use all DNS entries with host name
> > > >>>>> > c) Use all DNS entries with canonical host name
> > > >>>>> >
> > > >>>>> > 2) Advertised listeners
> > > >>>>> >
> > > >>>>> > a) No lookup
> > > >>>>> > b) Use all DNS entries with host name
> > > >>>>> > c) Use all DNS entries with canonical host name
> > > >>>>> >
> > > >>>>> > The combinations that are enabled by the two boolean configs
(
> > > >>>>> > bootstrap.reverse.dns.lookup, enable.all.dns.ips) are:
> > > >>>>> >
> > > >>>>> > - (false, false) => (1a, 2a)
> > > >>>>> > - (true, false) => (1c, 2a)
> > > >>>>> > - (false, true) => (1b, 2b)
> > > >>>>> > - (true, true) => (??, 2b)
> > > >>>>> >
> > > >>>>> > It will be good if we can clearly identify which
combinations
> we
> > > want to
> > > >>>>> > support and the scenarios where they may be useful. Perhaps
> (1a,
> > > 2a),
> > > >>>>> (1c,
> > > >>>>> > 2a), (1b, 2b) and (1c, 2c) are useful?
> > > >>>>> >
> > > >>>>> >
> > > >>>>> > On Mon, May 14, 2018 at 2:58 PM, Skrzypek, Jonathan <
> > > >>>>> > Jonathan.Skrzypek@gs.com> wrote:
> > > >>>>> >
> > > >>>>> >> Ah, apologies didn't see there was already a decent amount
of
> > > discussion
> > > >>>>> >> on this in the PR.
> > > >>>>> >>
> > > >>>>> >> This kind of sounds related to the environment you're
running
> to
> > > me.
> > > >>>>> >> What is the rationale behind using the advertised listeners
> to do
> > > your
> > > >>>>> >> load balancing advertisement rather than a top level alias
> that
> > > has
> > > >>>>> >> everything ?
> > > >>>>> >>
> > > >>>>> >> It sounds like in your case there is a mismatch between
> > > >>>>> bootstrap.servers
> > > >>>>> >> and advertised.listeners, and you want advertised.listeners
to
> > > take
> > > >>>>> >> precedence and have the client iterate over what is
returned
> by
> > > the
> > > >>>>> broker.
> > > >>>>> >> So the extra parameter doesn't only have to do with DNS but
> it's
> > > also
> > > >>>>> >> appending from the broker, maybe the parameter name should
> > > reflect this
> > > >>>>> ?
> > > >>>>> >>
> > > >>>>> >> Jonathan Skrzypek
> > > >>>>> >>
> > > >>>>> >>
> > > >>>>> >> -----Original Message-----
> > > >>>>> >> From: Skrzypek, Jonathan [Tech]
> > > >>>>> >> Sent: 14 May 2018 14:46
> > > >>>>> >> To: dev@kafka.apache.org
> > > >>>>> >> Subject: RE: [DISCUSS] KIP-302 - Enable Kafka clients to
use
> all
> > > DNS
> > > >>>>> >> resolved IP addresses
> > > >>>>> >>
> > > >>>>> >> Hi,
> > > >>>>> >>
> > > >>>>> >> I see you noted the similarities with KIP-235.
> > > >>>>> >> But KIP-235 might also solve what this KIP is trying to
> achieve.
> > > >>>>> >>
> > > >>>>> >> When parsing bootstrap.servers, KIP-235 has the client add
all
> > > >>>>> underlying
> > > >>>>> >> hostnames and IPs.
> > > >>>>> >> And this happens before hitting the NetworkClient.
> > > >>>>> >>
> > > >>>>> >> So to me the client will try every single endpoint behind
any
> > > >>>>> >> bootstrap.servers record.
> > > >>>>> >>
> > > >>>>> >> See
> > > INVALID URI REMOVED.
> com_apache_kafka_pull_4485_commits_24757eb7b0&d=DwIBaQ&c=
> 7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_
> r6eaCbPOM0NM1EHo-E&m=_ud9m_JZJ87C7eGsKcmzgJgDpNQDIIv5R4i_
> 7VlhkLc&s=TqaiA9uW_myYO6FN-gKPfPlioxZR6DhnlBTpEj5M2aQ&e=
> > > >>>>> >>
> > > 6bcf8c7d7649c85232c52b5d54f0e4#diff-89ef153462e64c250a21bd324ae1a851
> > > >>>>> >> which calls getAllByName like you suggested
> > > >>>>> >>
> > > >>>>> >> Jonathan Skrzypek
> > > >>>>> >>
> > > >>>>> >>
> > > >>>>> >> -----Original Message-----
> > > >>>>> >> From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > >>>>> >> Sent: 14 May 2018 14:17
> > > >>>>> >> To: dev@kafka.apache.org
> > > >>>>> >> Subject: [DISCUSS] KIP-302 - Enable Kafka clients to use
all
> DNS
> > > >>>>> resolved
> > > >>>>> >> IP addresses
> > > >>>>> >>
> > > >>>>> >> Hi all,
> > > >>>>> >>
> > > >>>>> >> We just opened a KIP to add support for the client to use
all
> IPs
> > > >>>>> returned
> > > >>>>> >> by DNS for the brokers
> > > >>>>> >>
> > > >>>>> >> The details are here -
> > > >>>>> >>
> > > >>>>> >>
INVALID URI REMOVED
> > > >>>>> >>
pache.org_confluence_display_KAFKA_KIP-2D302-2B-2D-2BEnable-
> > > >>>>> >> 2BKafka-2Bclients-2Bto-2Buse-2Ball-2BDNS-2Bresolved-2BIP-
> > > >>>>> >>
2Baddresses&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxL
> > > >>>>> >>
xfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaCbPOM0NM1EHo-E&m=EJafF
> > > >>>>> >> l1clRyolgtcu2uCc4_cIOJnlxb1r1n-D2Dti4k&s=C-UZ6KUG7JFiPD_
> > > >>>>> >> CnHczDOVqH9-XC5f_OFkw4BTNrI4&e=
> > > >>>>> >>
> > > >>>>> >> The JIRA and provisional PR (where the discussion lead to
the
> > > creation
> > > >>>>> of
> > > >>>>> >> this KIP) are :
> > > >>>>> >>
> > > >>>>> >> INVALID URI REMOVED
.
> > > >>>>> >>
apache.org_jira_browse_KAFKA-2D6863&d=DwIBaQ&c=7563p3e2zaQw0
> > > >>>>> >>
AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6
> > > >>>>> >> eaCbPOM0NM1EHo-E&m=EJafFl1clRyolgtcu2uCc4_cIOJnlxb1r1n-
> > > >>>>> >> D2Dti4k&s=3Puqs5iYoPsw6hARQr6gvokdFE-H5USMiNVGOUtNkJI&e=
> > > >>>>> >>
> > > >>>>> >> INVALID URI REMOVED
.
> > > >>>>> >>
com_apache_kafka_pull_4987&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgy
> > > >>>>> >> agb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaC
> > > >>>>> >>
bPOM0NM1EHo-E&m=EJafFl1clRyolgtcu2uCc4_cIOJnlxb1r1n-D2Dti4k&
> > > >>>>> >> s=Hqn5dOgQy4-MHTIJLE49O8bNomry3SoGq9OVoHU-CRA&e=
> > > >>>>> >>
> > > >>>>> >> Looking forward to the community's feedback.
> > > >>>>> >> It would be amazing to have it voted by May 22nd :-) :-)
> > > >>>>> >>
> > > >>>>> >> Edoardo & Mickael
> > > >>>>> >>
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> --
> > > >>>>> "When the people fear their government, there is tyranny; when
> the
> > > >>>>> government fears the people, there is liberty." [Thomas
> Jefferson]
> > > >>>>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> "When the people fear their government, there is tyranny; when
the
> > > >>> government fears the people, there is liberty." [Thomas
Jefferson]
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> "When the people fear their government, there is tyranny; when
the
> > > >> government fears the people, there is liberty." [Thomas
Jefferson]
> > > >
> > > >
> > > >
> > > > --
> > > > "When the people fear their government, there is tyranny; when the
> > > > government fears the people, there is liberty." [Thomas Jefferson]
> > > >
> > > > ________________________________
> > > >
> > > > Your Personal Data: We may collect and process information about
you
> > > that may be subject to data protection laws. For more information
> about how
> > > we use and disclose your personal data, how we protect your
> information,
> > > our legal basis to use your information, your rights and who you can
> > > contact, please refer to: www.gs.com/privacy-notices<
> > >
http://www.gs.com/privacy-notices
>
> > > >
> > > >
> > > > ---------- Forwarded message ----------
> > > > From: Rajini Sivaram <ra...@gmail.com>
> > > > To: "Skrzypek, Jonathan" <Jo...@ln.email.gs.com>, dev
<
> > > dev@kafka.apache.org>, Ismael Juma <is...@juma.me.uk>
> > > > Cc:
> > > > Bcc:
> > > > Date: Tue, 22 May 2018 15:05:07 +0000
> > > > Subject: Re: FW: [VOTE] KIP-235 Add DNS alias support for secured
> > > connection
> > > > Hi Jonathan,
> > > >
> > > > I think it would make sense to convert the config in this KIP into
an
> > > enum so that we can add more variations later on. But since KIP-302
is
> > > still under discussion, it is not clear what the config name should
be.
> > > Since today is the KIP deadline and the implementation itself is
> > > straightforward, it would make sense to progress with this one for
> 2.0.0 if
> > > we can get one more binding vote.
> > > >
> > > > Ismael, do you have time to take a look at KIP-235 today?
> > > >
> > > > Thanks,
> > > >
> > > > Rajini
> > > >
> > > >
> > > > On Tue, May 22, 2018 at 3:45 PM, Skrzypek, Jonathan <
> > > Jonathan.Skrzypek@gs.com> wrote:
> > > >>
> > > >> Hello Rajini,
> > > >>
> > > >> What do you think should be the next step here ?
> > > >>
> > > >>
> > > >> Jonathan Skrzypek
> > > >>
> > > >> -----Original Message-----
> > > >> From: Skrzypek, Jonathan [Tech]
> > > >> Sent: 21 May 2018 10:51
> > > >> To: 'dev'
> > > >> Subject: RE: [VOTE] KIP-235 Add DNS alias support for secured
> connection
> > > >>
> > > >> Hi,
> > > >>
> > > >> What would be the next step here ?
> > > >> I know there's a discussion going on around KIP-302, but I'm also
> > > conscious that the 2.0.0 deadline for KIPs is tomorrow.
> > > >> I've opened this KIP in January and discussions have been
productive
> > > with an end solution I had the impression was reasonable, so I am
keen
> to
> > > see it make it the next release.
> > > >>
> > > >>
> > > >> Jonathan Skrzypek
> > > >>
> > > >> -----Original Message-----
> > > >> From: Skrzypek, Jonathan [Tech]
> > > >> Sent: 14 May 2018 13:48
> > > >> To: dev
> > > >> Subject: RE: [VOTE] KIP-235 Add DNS alias support for secured
> connection
> > > >>
> > > >> Sure, I modified the KIP to add more details
> > > >>
> > > >>
> > >
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 235%3A+Add+DNS+alias+support+for+secured+connection
> > > >>
> > > >>
> > > >> Jonathan Skrzypek
> > > >>
> > > >>
> > > >> -----Original Message-----
> > > >> From: Ismael Juma [mailto:ismael@juma.me.uk]
> > > >> Sent: 14 May 2018 11:53
> > > >> To: dev
> > > >> Subject: Re: [VOTE] KIP-235 Add DNS alias support for secured
> connection
> > > >>
> > > >> Thanks for the KIP, Jonathan. It would be helpful to have more
> detail on
> > > >> how SSL authentication could be broken if the new behaviour is
the
> > > default.
> > > >> I know this was discussed in the mailing list thread, but it's
> > > important to
> > > >> include it in the KIP since it's the main reason why a new config
is
> > > needed
> > > >> (and configs should be avoided whenever we can just do the right
> thing).
> > > >>
> > > >> Ismael
> > > >>
> > > >> On Fri, Mar 23, 2018 at 12:05 PM Skrzypek, Jonathan <
> > > >> Jonathan.Skrzypek@gs.com> wrote:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > I would like to start a vote for KIP-235
> > > >> >
> > > >> >
> > > >> >
> > > INVALID URI REMOVED.
> apache.org_confluence_display_KAFKA_KIP-2D235-253A-2BAdd-
> 2BDNS-2Balias-2Bsupport-2Bfor-2Bsecured-2Bconnection&d=DwIBaQ&c=
> 7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_
> r6eaCbPOM0NM1EHo-E&m=FM_uCHnnO2dqxWC0bi7_QOJKfKmQI80-
> Xduvb-URWOw&s=RpGkijfK-WHcU0s8ZtMXEkIr69QraJhYKaGSC9V_rnI&e=
> > > >> >
> > > >> > This is a proposition to add an option for reverse dns lookup
of
> > > >> > bootstrap.servers hosts, allowing the use of dns aliases on
> clusters
> > > using
> > > >> > SASL authentication.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >> ________________________________
> > > >>
> > > >> Your Personal Data: We may collect and process information about
you
> > > that may be subject to data protection laws. For more information
> about how
> > > we use and disclose your personal data, how we protect your
> information,
> > > our legal basis to use your information, your rights and who you can
> > > contact, please refer to: www.gs.com/privacy-notices<
> > >
http://www.gs.com/privacy-notices
>
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > > "When the people fear their government, there is tyranny; when the
> > > government fears the people, there is liberty." [Thomas Jefferson]
> > >
> >
> >
> > --
> > "When the people fear their government, there is tyranny; when the
> > government fears the people, there is liberty." [Thomas Jefferson]
>
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS resolved
IP addresses
Posted by Eno Thereska <en...@gmail.com>.
Hi folks,
Could you expand the motivation a bit? When would it make sense to use an
LB in front of Kafka brokers? A client needs to access each broker directly
to consume the data in that broker and cannot be redirected to another
broker. What exact scenario are you seeing that needs this KIP?
Thanks
Eno
On Tue, Sep 18, 2018 at 9:52 AM, Mickael Maison <mi...@gmail.com>
wrote:
> Bumping this thread
>
> It's a relatively small change that would help cloud environments with
> load balancers fronting brokers
> On Tue, Sep 11, 2018 at 3:01 PM Edoardo Comar <ed...@gmail.com> wrote:
> >
> > Hi all,
> > after some time we updated KIP-302 to reuse the config key introduced by
> > KIP-235, with a different value to avoid conflicts between the two.
> > Also clarified the use of multiple IPs only of the same type (IPv4/IPv6).
> >
> > We'd appreciate a further review and discussion.
> > Thanks!
> > Edo
> >
> >
> > On Fri, 25 May 2018 at 12:36, Edoardo Comar <ed...@gmail.com> wrote:
> >
> > > Hi Jonathan,
> > > I'm ok with an expandable enum for the config that could be extended
> > > in the future.
> > > It is marginally better than multiple potentially conflicting config
> > > entries.
> > >
> > > Though as I think the change for KIP-302 is independent from KIP-235
> > > and they do not conflict,
> > > when we'll look back at it post 2.0 we may see if it is more valuable
> > > to shoehorn its config in an expanded enum or not
> > >
> > > thanks,
> > > Edo
> > >
> > > On 24 May 2018 at 16:50, Skrzypek, Jonathan <Jo...@gs.com>
> > > wrote:
> > > > Hi,
> > > >
> > > > As Rajini suggested in the thread for KIP 235 (attached), we could
> try
> > > to have an enum that would drive what does the client expands/resolves.
> > > >
> > > > I suggest a client config called client.dns.lookup with different
> values
> > > possible :
> > > >
> > > > - no : no dns lookup
> > > > - hostnames.only : perform dns lookup on both bootstrap.servers and
> > > advertised listeners
> > > > - canonical.hostnames.only : perform dns lookup on both
> > > bootstrap.servers and advertised listeners
> > > > - bootstrap.hostnames.only : perform dns lookup on bootstrap.servers
> > > list and expand it
> > > > - bootstrap.canonical.hostnames.only : perform dns lookup on
> > > bootstrap.servers list and expand it
> > > > - advertised.listeners.hostnames.only : perform dns lookup on
> advertised
> > > listeners
> > > > - advertised.listeners.canonical.hostnames.only : perform dns
> lookup on
> > > advertised listeners
> > > >
> > > > I realize this is a bit heavy but this gives users the ability to
> pick
> > > and choose.
> > > > I didn't include a setting to mix hostnames and canonical hostnames
> as
> > > I'm not sure there would be a valid use case.
> > > >
> > > > Alternatively, to have less possible values, we could have 2
> parameters :
> > > >
> > > > - dns.lookup.type with values : hostname / canonical.host.name
> > > > - dns.lookup.behaviour : bootstrap.servers, advertised.listeners,
> both
> > > >
> > > > Thoughts ?
> > > >
> > > > Jonathan Skrzypek
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > > Sent: 17 May 2018 23:50
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS
> > > resolved IP addresses
> > > >
> > > > Hi Jonathan,
> > > >
> > > >> A solution might be to expose to users the choice of using hostname
> or
> > > canonical host name on both sides.
> > > >> Say having one setting that collapses functionalities from both KIPs
> > > (bootstrap expansion + advertised lookup)
> > > >> and an additional parameter that defines how the resolution is
> > > performed, using getCanonicalHostName() or not.
> > > >
> > > > thanks sounds to me *less* simple than independent config options,
> sorry.
> > > >
> > > > I would like to say once again that by itself KIP-302 only speeds up
> > > > the client behavior that can happen anyway when the client restarts
> > > > multiple times,
> > > > as every time there is no guarantee that - in presence of multiple A
> > > > DNS records - the same IP is returned. Attempting to use additiona
> IPs
> > > > if the first fail just makes client recovery faster.
> > > >
> > > > cheers
> > > > Edo
> > > >
> > > > On 17 May 2018 at 12:12, Skrzypek, Jonathan <
> Jonathan.Skrzypek@gs.com>
> > > wrote:
> > > >> Yes, makes sense.
> > > >> You mentioned multiple times you see no overlap and no issue with
> your
> > > KIP, and that they solve different use cases.
> > > >>
> > > >> Appreciate you have an existing use case that would work, but we
> need
> > > to make sure this isn't confusing to users and that any combination
> will
> > > always work, across security protocols.
> > > >>
> > > >> A solution might be to expose to users the choice of using hostname
> or
> > > canonical host name on both sides.
> > > >> Say having one setting that collapses functionalities from both KIPs
> > > (bootstrap expansion + advertised lookup) and an additional parameter
> that
> > > defines how the resolution is performed, using getCanonicalHostName()
> or
> > > not.
> > > >>
> > > >> Maybe that gives less flexibility as users wouldn't be able to
> decide
> > > to only perform DNS lookup on bootstrap.servers or on advertised
> listeners.
> > > >> But this would ensure consistency so that a user can decide to use
> > > cnames or not (depending on their certificates and Kerberos principals
> in
> > > their environment) and it would work.
> > > >>
> > > >> Jonathan Skrzypek
> > > >>
> > > >> -----Original Message-----
> > > >> From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > >> Sent: 16 May 2018 21:59
> > > >> To: dev@kafka.apache.org
> > > >> Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS
> > > resolved IP addresses
> > > >>
> > > >> Hi Jonathan,
> > > >> I am afraid that may not work for everybody.
> > > >>
> > > >> It would not work for us.
> > > >> With our current DNS, my Kafka clients are perfectly happy to use
> any
> > > IPs -
> > > >> DNS has multiple A records for the 'myhostname.mydomain' used for
> > > >> bootstrap and advertised listeners.
> > > >> The hosts all serve TLS certificates that include
> > > >> 'myhostname.mydomain' and the clients are happy.
> > > >>
> > > >> However, applying getCanonicalHostName to those IPs would return
> > > >> hostnames that would not match the TLS certificates.
> > > >>
> > > >> So once again I believe your solution and ours solve different use
> > > cases.
> > > >>
> > > >> cheers
> > > >> Edo
> > > >>
> > > >> On 16 May 2018 at 18:29, Skrzypek, Jonathan <
> Jonathan.Skrzypek@gs.com>
> > > wrote:
> > > >>> I think there are combinations that will break SASL and SSL auth.
> > > >>> Could the trick be to have a single parameter that triggers dns
> > > resolve both for bootstrap and advertised listeners, both using
> > > getCanonicalHostName() ?
> > > >>>
> > > >>> Jonathan Skrzypek
> > > >>>
> > > >>> -----Original Message-----
> > > >>> From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > >>> Sent: 16 May 2018 17:03
> > > >>> To: dev@kafka.apache.org
> > > >>> Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all
> DNS
> > > resolved IP addresses
> > > >>>
> > > >>> Hi Rajini,
> > > >>>
> > > >>> In your example KIP-302 would attempt to connect to the first
> address
> > > >>> returned, let's say
> > > >>>
> > > >>> www.apache.org/195.154.151.36
> > > >>>
> > > >>> then, only if that fails, will in turn try the remaining:
> > > >>>
> > > >>> www.apache.org/40.79.78.1
> > > >>> www.apache.org/140.211.11.105
> > > >>> www.apache.org/2001:bc8:2142:300:0:0:0:0
> > > >>>
> > > >>> You're right to say that we expect certificates served by those
> > > >>> endpoints to be valid for "www.apache.org"
> > > >>>
> > > >>> Without KIP-302, only one would be attempted.
> > > >>> Which is the first one, that can change every time
> > > >>> (typically changes on every Java process restart,
> > > >>> but may change also any time InetAddress.getAllByName it's invoked
> > > >>> depending on the caching).
> > > >>>
> > > >>> The behavioral change that KIP-302 may introduce is that in the
> > > example above,
> > > >>> also an IPv6 connection may be attempted after some IPv4 ones.
> > > >>>
> > > >>> InetAddress.getAllByName() implementation uses a system property
> > > >>> "java.net.preferIPv6Addresses"
> > > >>> to decide which type of address to return first (default is still
> IPv4
> > > >>> in java 10)
> > > >>>
> > > >>> We will amend the KIP and PR so that the loop only uses IPs of the
> > > >>> same type as the first one returned.
> > > >>>
> > > >>> A part from the above, KIP 302 does not seem to change any existing
> > > >>> client behaviour, as any one of multiple IP addresses (of a given
> > > >>> v4/v6 type) can currently be picked.
> > > >>> We're happy however to keep the looping behavior optional with the
> > > >>> discussed config property, disabled by default.
> > > >>>
> > > >>> As for KIP-235 that may introduce new hostnames in the bootstrap
> list
> > > >>> (the current PR rewrites the bootstrap list)
> > > >>> and we fail to see the conflict with KIP-302, whatever the set of
> > > >>> configs chosen.
> > > >>>
> > > >>> We'd be happy to try understand what we are missing in a KIP call
> :-)
> > > >>>
> > > >>> cheers
> > > >>> Edo
> > > >>>
> > > >>> On 15 May 2018 at 16:58, Rajini Sivaram <ra...@gmail.com>
> > > wrote:
> > > >>>> Hi Edo,
> > > >>>>
> > > >>>> I agree that KIP-235 and KIP-302 address different scenarios. And
> I
> > > agree
> > > >>>> that each one is not sufficient in itself to address both the
> > > scenarios.
> > > >>>> But I also think that they conflict and hence they need to be
> looked
> > > at
> > > >>>> together and perhaps use a single config.
> > > >>>>
> > > >>>> As an example:
> > > >>>>
> > > >>>> If I run:
> > > >>>>
> > > >>>> for (InetAddress address : InetAddress.getAllByName("www.
> apache.org"))
> > > {
> > > >>>> System.out.printf("HostName %s canonicalHostName %s IP %s\n",
> > > >>>> address.getHostName(), address.getCanonicalHostName()
> ,
> > > >>>> address.getHostAddress());
> > > >>>> }
> > > >>>>
> > > >>>> I get:
> > > >>>>
> > > >>>> HostName www.apache.org canonicalHostName tlp-eu-west.apache.org
> IP
> > > >>>> 195.154.151.36
> > > >>>> HostName www.apache.org canonicalHostName 40.79.78.1 IP
> 40.79.78.1
> > > >>>> HostName www.apache.org canonicalHostName themis.apache.org IP
> > > >>>> 140.211.11.105
> > > >>>> HostName www.apache.org canonicalHostName
> 2001:bc8:2142:300:0:0:0:0
> > > IP
> > > >>>> 2001:bc8:2142:300:0:0:0:0
> > > >>>>
> > > >>>>
> > > >>>> If www.apache.org is used as a bootstrap address, KIP-302 would
> > > connect to (
> > > >>>> www.apache.org/195.154.151.36 and www.apache.org/140.211.11.105)
> > > while
> > > >>>> KIP-235 would connect to (tlp-eu-west.apache.org/195.154.151.3.
> and
> > > >>>> themis.apache.org/140.211.11.105). This is a significant
> difference
> > > not
> > > >>>> just for Kerberos, but for any secure environment where hostname
> is
> > > >>>> verified to prevent man-in-the-middle attacks. In your case, I
> > > presume you
> > > >>>> would have SSL certificates with the equivalent of www.apache.org
> on
> > > both
> > > >>>> the load balancers. In Jonathan's case, I presume he has Kerberos
> > > >>>> principals for the equivalent of tlp-eu-west.apache.org and
> > > >>>> themis.apache.org. We would want to support both scenarios
> > > regardless of
> > > >>>> the security protocol, just need to come up with configuration
> > > options that
> > > >>>> don't conflict.
> > > >>>>
> > > >>>>
> > > >>>> On Mon, May 14, 2018 at 5:24 PM, Edoardo Comar <
> edocomar@gmail.com>
> > > wrote:
> > > >>>>
> > > >>>>> Thanks Rajini
> > > >>>>>
> > > >>>>> I still don't see the overlap between the two KIPS
> > > >>>>>
> > > >>>>> KIP-235 allows an expansion of hostnames on the bootstrap list.
> > > >>>>>
> > > >>>>> KIP-302 allows alternative IPs to be used to attempt a connection
> > > >>>>> (either at bootstrap and when processing the MetadataResponse)
> to a
> > > >>>>> given hostname.
> > > >>>>>
> > > >>>>> A use case would be that of active/passive LB fronting the
> brokers.
> > > >>>>>
> > > >>>>> Arguably, if Java honored the DNS-set TTL, and the TTL was low
> and on
> > > >>>>> subsequent requests, the order of IPs returned by the
> > > >>>>> InetAddress.getAllByName() was random, we may not need such an
> > > >>>>> enhancement.
> > > >>>>> In practice, a Java client can get stuck on a "bad" IP forever
> if it
> > > >>>>> only relies on the first IP returned.
> > > >>>>>
> > > >>>>> HTH,
> > > >>>>> Edo
> > > >>>>>
> > > >>>>> On 14 May 2018 at 16:23, Rajini Sivaram <rajinisivaram@gmail.com
> >
> > > wrote:
> > > >>>>> > Hi Edo,
> > > >>>>> >
> > > >>>>> > Thanks for the KIP. I think it will be good to include a
> diagram
> > > to make
> > > >>>>> it
> > > >>>>> > easier to distinguish this scenario from that of KIP-235
> without
> > > reading
> > > >>>>> > the PR.
> > > >>>>> >
> > > >>>>> > It may be worth considering if KIP-235 and this KIP could use a
> > > single
> > > >>>>> > config name with different values instead of two boolean
> configs:
> > > >>>>> >
> > > >>>>> > bootstrap.reverse.dns.lookup = true/false
> > > >>>>> > enable.all.dns.ips = true/false
> > > >>>>> >
> > > >>>>> > Not all values of (bootstrap.reverse.dns.lookup,
> > > enable.all.dns.ips) seem
> > > >>>>> > to make sense. And not all scenarios are handled. Even if we
> use
> > > multiple
> > > >>>>> > configs, it seems to me that we may want to name them
> differently.
> > > >>>>> >
> > > >>>>> > The possible combinations are:
> > > >>>>> >
> > > >>>>> > 1) Bootstrap
> > > >>>>> >
> > > >>>>> > a) No lookup
> > > >>>>> > b) Use all DNS entries with host name
> > > >>>>> > c) Use all DNS entries with canonical host name
> > > >>>>> >
> > > >>>>> > 2) Advertised listeners
> > > >>>>> >
> > > >>>>> > a) No lookup
> > > >>>>> > b) Use all DNS entries with host name
> > > >>>>> > c) Use all DNS entries with canonical host name
> > > >>>>> >
> > > >>>>> > The combinations that are enabled by the two boolean configs (
> > > >>>>> > bootstrap.reverse.dns.lookup, enable.all.dns.ips) are:
> > > >>>>> >
> > > >>>>> > - (false, false) => (1a, 2a)
> > > >>>>> > - (true, false) => (1c, 2a)
> > > >>>>> > - (false, true) => (1b, 2b)
> > > >>>>> > - (true, true) => (??, 2b)
> > > >>>>> >
> > > >>>>> > It will be good if we can clearly identify which combinations
> we
> > > want to
> > > >>>>> > support and the scenarios where they may be useful. Perhaps
> (1a,
> > > 2a),
> > > >>>>> (1c,
> > > >>>>> > 2a), (1b, 2b) and (1c, 2c) are useful?
> > > >>>>> >
> > > >>>>> >
> > > >>>>> > On Mon, May 14, 2018 at 2:58 PM, Skrzypek, Jonathan <
> > > >>>>> > Jonathan.Skrzypek@gs.com> wrote:
> > > >>>>> >
> > > >>>>> >> Ah, apologies didn't see there was already a decent amount of
> > > discussion
> > > >>>>> >> on this in the PR.
> > > >>>>> >>
> > > >>>>> >> This kind of sounds related to the environment you're running
> to
> > > me.
> > > >>>>> >> What is the rationale behind using the advertised listeners
> to do
> > > your
> > > >>>>> >> load balancing advertisement rather than a top level alias
> that
> > > has
> > > >>>>> >> everything ?
> > > >>>>> >>
> > > >>>>> >> It sounds like in your case there is a mismatch between
> > > >>>>> bootstrap.servers
> > > >>>>> >> and advertised.listeners, and you want advertised.listeners to
> > > take
> > > >>>>> >> precedence and have the client iterate over what is returned
> by
> > > the
> > > >>>>> broker.
> > > >>>>> >> So the extra parameter doesn't only have to do with DNS but
> it's
> > > also
> > > >>>>> >> appending from the broker, maybe the parameter name should
> > > reflect this
> > > >>>>> ?
> > > >>>>> >>
> > > >>>>> >> Jonathan Skrzypek
> > > >>>>> >>
> > > >>>>> >>
> > > >>>>> >> -----Original Message-----
> > > >>>>> >> From: Skrzypek, Jonathan [Tech]
> > > >>>>> >> Sent: 14 May 2018 14:46
> > > >>>>> >> To: dev@kafka.apache.org
> > > >>>>> >> Subject: RE: [DISCUSS] KIP-302 - Enable Kafka clients to use
> all
> > > DNS
> > > >>>>> >> resolved IP addresses
> > > >>>>> >>
> > > >>>>> >> Hi,
> > > >>>>> >>
> > > >>>>> >> I see you noted the similarities with KIP-235.
> > > >>>>> >> But KIP-235 might also solve what this KIP is trying to
> achieve.
> > > >>>>> >>
> > > >>>>> >> When parsing bootstrap.servers, KIP-235 has the client add all
> > > >>>>> underlying
> > > >>>>> >> hostnames and IPs.
> > > >>>>> >> And this happens before hitting the NetworkClient.
> > > >>>>> >>
> > > >>>>> >> So to me the client will try every single endpoint behind any
> > > >>>>> >> bootstrap.servers record.
> > > >>>>> >>
> > > >>>>> >> See
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> com_apache_kafka_pull_4485_commits_24757eb7b0&d=DwIBaQ&c=
> 7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_
> r6eaCbPOM0NM1EHo-E&m=_ud9m_JZJ87C7eGsKcmzgJgDpNQDIIv5R4i_
> 7VlhkLc&s=TqaiA9uW_myYO6FN-gKPfPlioxZR6DhnlBTpEj5M2aQ&e=
> > > >>>>> >>
> > > 6bcf8c7d7649c85232c52b5d54f0e4#diff-89ef153462e64c250a21bd324ae1a851
> > > >>>>> >> which calls getAllByName like you suggested
> > > >>>>> >>
> > > >>>>> >> Jonathan Skrzypek
> > > >>>>> >>
> > > >>>>> >>
> > > >>>>> >> -----Original Message-----
> > > >>>>> >> From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > >>>>> >> Sent: 14 May 2018 14:17
> > > >>>>> >> To: dev@kafka.apache.org
> > > >>>>> >> Subject: [DISCUSS] KIP-302 - Enable Kafka clients to use all
> DNS
> > > >>>>> resolved
> > > >>>>> >> IP addresses
> > > >>>>> >>
> > > >>>>> >> Hi all,
> > > >>>>> >>
> > > >>>>> >> We just opened a KIP to add support for the client to use all
> IPs
> > > >>>>> returned
> > > >>>>> >> by DNS for the brokers
> > > >>>>> >>
> > > >>>>> >> The details are here -
> > > >>>>> >>
> > > >>>>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.a
> > > >>>>> >> pache.org_confluence_display_KAFKA_KIP-2D302-2B-2D-2BEnable-
> > > >>>>> >> 2BKafka-2Bclients-2Bto-2Buse-2Ball-2BDNS-2Bresolved-2BIP-
> > > >>>>> >> 2Baddresses&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxL
> > > >>>>> >> xfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaCbPOM0NM1EHo-E&m=EJafF
> > > >>>>> >> l1clRyolgtcu2uCc4_cIOJnlxb1r1n-D2Dti4k&s=C-UZ6KUG7JFiPD_
> > > >>>>> >> CnHczDOVqH9-XC5f_OFkw4BTNrI4&e=
> > > >>>>> >>
> > > >>>>> >> The JIRA and provisional PR (where the discussion lead to the
> > > creation
> > > >>>>> of
> > > >>>>> >> this KIP) are :
> > > >>>>> >>
> > > >>>>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> > > >>>>> >> apache.org_jira_browse_KAFKA-2D6863&d=DwIBaQ&c=7563p3e2zaQw0
> > > >>>>> >> AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6
> > > >>>>> >> eaCbPOM0NM1EHo-E&m=EJafFl1clRyolgtcu2uCc4_cIOJnlxb1r1n-
> > > >>>>> >> D2Dti4k&s=3Puqs5iYoPsw6hARQr6gvokdFE-H5USMiNVGOUtNkJI&e=
> > > >>>>> >>
> > > >>>>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> > > >>>>> >> com_apache_kafka_pull_4987&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgy
> > > >>>>> >> agb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaC
> > > >>>>> >> bPOM0NM1EHo-E&m=EJafFl1clRyolgtcu2uCc4_cIOJnlxb1r1n-D2Dti4k&
> > > >>>>> >> s=Hqn5dOgQy4-MHTIJLE49O8bNomry3SoGq9OVoHU-CRA&e=
> > > >>>>> >>
> > > >>>>> >> Looking forward to the community's feedback.
> > > >>>>> >> It would be amazing to have it voted by May 22nd :-) :-)
> > > >>>>> >>
> > > >>>>> >> Edoardo & Mickael
> > > >>>>> >>
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> --
> > > >>>>> "When the people fear their government, there is tyranny; when
> the
> > > >>>>> government fears the people, there is liberty." [Thomas
> Jefferson]
> > > >>>>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> "When the people fear their government, there is tyranny; when the
> > > >>> government fears the people, there is liberty." [Thomas Jefferson]
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> "When the people fear their government, there is tyranny; when the
> > > >> government fears the people, there is liberty." [Thomas Jefferson]
> > > >
> > > >
> > > >
> > > > --
> > > > "When the people fear their government, there is tyranny; when the
> > > > government fears the people, there is liberty." [Thomas Jefferson]
> > > >
> > > > ________________________________
> > > >
> > > > Your Personal Data: We may collect and process information about you
> > > that may be subject to data protection laws. For more information
> about how
> > > we use and disclose your personal data, how we protect your
> information,
> > > our legal basis to use your information, your rights and who you can
> > > contact, please refer to: www.gs.com/privacy-notices<
> > > http://www.gs.com/privacy-notices>
> > > >
> > > >
> > > > ---------- Forwarded message ----------
> > > > From: Rajini Sivaram <ra...@gmail.com>
> > > > To: "Skrzypek, Jonathan" <Jo...@ln.email.gs.com>, dev <
> > > dev@kafka.apache.org>, Ismael Juma <is...@juma.me.uk>
> > > > Cc:
> > > > Bcc:
> > > > Date: Tue, 22 May 2018 15:05:07 +0000
> > > > Subject: Re: FW: [VOTE] KIP-235 Add DNS alias support for secured
> > > connection
> > > > Hi Jonathan,
> > > >
> > > > I think it would make sense to convert the config in this KIP into an
> > > enum so that we can add more variations later on. But since KIP-302 is
> > > still under discussion, it is not clear what the config name should be.
> > > Since today is the KIP deadline and the implementation itself is
> > > straightforward, it would make sense to progress with this one for
> 2.0.0 if
> > > we can get one more binding vote.
> > > >
> > > > Ismael, do you have time to take a look at KIP-235 today?
> > > >
> > > > Thanks,
> > > >
> > > > Rajini
> > > >
> > > >
> > > > On Tue, May 22, 2018 at 3:45 PM, Skrzypek, Jonathan <
> > > Jonathan.Skrzypek@gs.com> wrote:
> > > >>
> > > >> Hello Rajini,
> > > >>
> > > >> What do you think should be the next step here ?
> > > >>
> > > >>
> > > >> Jonathan Skrzypek
> > > >>
> > > >> -----Original Message-----
> > > >> From: Skrzypek, Jonathan [Tech]
> > > >> Sent: 21 May 2018 10:51
> > > >> To: 'dev'
> > > >> Subject: RE: [VOTE] KIP-235 Add DNS alias support for secured
> connection
> > > >>
> > > >> Hi,
> > > >>
> > > >> What would be the next step here ?
> > > >> I know there's a discussion going on around KIP-302, but I'm also
> > > conscious that the 2.0.0 deadline for KIPs is tomorrow.
> > > >> I've opened this KIP in January and discussions have been productive
> > > with an end solution I had the impression was reasonable, so I am keen
> to
> > > see it make it the next release.
> > > >>
> > > >>
> > > >> Jonathan Skrzypek
> > > >>
> > > >> -----Original Message-----
> > > >> From: Skrzypek, Jonathan [Tech]
> > > >> Sent: 14 May 2018 13:48
> > > >> To: dev
> > > >> Subject: RE: [VOTE] KIP-235 Add DNS alias support for secured
> connection
> > > >>
> > > >> Sure, I modified the KIP to add more details
> > > >>
> > > >>
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 235%3A+Add+DNS+alias+support+for+secured+connection
> > > >>
> > > >>
> > > >> Jonathan Skrzypek
> > > >>
> > > >>
> > > >> -----Original Message-----
> > > >> From: Ismael Juma [mailto:ismael@juma.me.uk]
> > > >> Sent: 14 May 2018 11:53
> > > >> To: dev
> > > >> Subject: Re: [VOTE] KIP-235 Add DNS alias support for secured
> connection
> > > >>
> > > >> Thanks for the KIP, Jonathan. It would be helpful to have more
> detail on
> > > >> how SSL authentication could be broken if the new behaviour is the
> > > default.
> > > >> I know this was discussed in the mailing list thread, but it's
> > > important to
> > > >> include it in the KIP since it's the main reason why a new config is
> > > needed
> > > >> (and configs should be avoided whenever we can just do the right
> thing).
> > > >>
> > > >> Ismael
> > > >>
> > > >> On Fri, Mar 23, 2018 at 12:05 PM Skrzypek, Jonathan <
> > > >> Jonathan.Skrzypek@gs.com> wrote:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > I would like to start a vote for KIP-235
> > > >> >
> > > >> >
> > > >> >
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
> apache.org_confluence_display_KAFKA_KIP-2D235-253A-2BAdd-
> 2BDNS-2Balias-2Bsupport-2Bfor-2Bsecured-2Bconnection&d=DwIBaQ&c=
> 7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_
> r6eaCbPOM0NM1EHo-E&m=FM_uCHnnO2dqxWC0bi7_QOJKfKmQI80-
> Xduvb-URWOw&s=RpGkijfK-WHcU0s8ZtMXEkIr69QraJhYKaGSC9V_rnI&e=
> > > >> >
> > > >> > This is a proposition to add an option for reverse dns lookup of
> > > >> > bootstrap.servers hosts, allowing the use of dns aliases on
> clusters
> > > using
> > > >> > SASL authentication.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >> ________________________________
> > > >>
> > > >> Your Personal Data: We may collect and process information about you
> > > that may be subject to data protection laws. For more information
> about how
> > > we use and disclose your personal data, how we protect your
> information,
> > > our legal basis to use your information, your rights and who you can
> > > contact, please refer to: www.gs.com/privacy-notices<
> > > http://www.gs.com/privacy-notices>
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > > "When the people fear their government, there is tyranny; when the
> > > government fears the people, there is liberty." [Thomas Jefferson]
> > >
> >
> >
> > --
> > "When the people fear their government, there is tyranny; when the
> > government fears the people, there is liberty." [Thomas Jefferson]
>
Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS resolved
IP addresses
Posted by Mickael Maison <mi...@gmail.com>.
Bumping this thread
It's a relatively small change that would help cloud environments with
load balancers fronting brokers
On Tue, Sep 11, 2018 at 3:01 PM Edoardo Comar <ed...@gmail.com> wrote:
>
> Hi all,
> after some time we updated KIP-302 to reuse the config key introduced by
> KIP-235, with a different value to avoid conflicts between the two.
> Also clarified the use of multiple IPs only of the same type (IPv4/IPv6).
>
> We'd appreciate a further review and discussion.
> Thanks!
> Edo
>
>
> On Fri, 25 May 2018 at 12:36, Edoardo Comar <ed...@gmail.com> wrote:
>
> > Hi Jonathan,
> > I'm ok with an expandable enum for the config that could be extended
> > in the future.
> > It is marginally better than multiple potentially conflicting config
> > entries.
> >
> > Though as I think the change for KIP-302 is independent from KIP-235
> > and they do not conflict,
> > when we'll look back at it post 2.0 we may see if it is more valuable
> > to shoehorn its config in an expanded enum or not
> >
> > thanks,
> > Edo
> >
> > On 24 May 2018 at 16:50, Skrzypek, Jonathan <Jo...@gs.com>
> > wrote:
> > > Hi,
> > >
> > > As Rajini suggested in the thread for KIP 235 (attached), we could try
> > to have an enum that would drive what does the client expands/resolves.
> > >
> > > I suggest a client config called client.dns.lookup with different values
> > possible :
> > >
> > > - no : no dns lookup
> > > - hostnames.only : perform dns lookup on both bootstrap.servers and
> > advertised listeners
> > > - canonical.hostnames.only : perform dns lookup on both
> > bootstrap.servers and advertised listeners
> > > - bootstrap.hostnames.only : perform dns lookup on bootstrap.servers
> > list and expand it
> > > - bootstrap.canonical.hostnames.only : perform dns lookup on
> > bootstrap.servers list and expand it
> > > - advertised.listeners.hostnames.only : perform dns lookup on advertised
> > listeners
> > > - advertised.listeners.canonical.hostnames.only : perform dns lookup on
> > advertised listeners
> > >
> > > I realize this is a bit heavy but this gives users the ability to pick
> > and choose.
> > > I didn't include a setting to mix hostnames and canonical hostnames as
> > I'm not sure there would be a valid use case.
> > >
> > > Alternatively, to have less possible values, we could have 2 parameters :
> > >
> > > - dns.lookup.type with values : hostname / canonical.host.name
> > > - dns.lookup.behaviour : bootstrap.servers, advertised.listeners, both
> > >
> > > Thoughts ?
> > >
> > > Jonathan Skrzypek
> > >
> > >
> > > -----Original Message-----
> > > From: Edoardo Comar [mailto:edocomar@gmail.com]
> > > Sent: 17 May 2018 23:50
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS
> > resolved IP addresses
> > >
> > > Hi Jonathan,
> > >
> > >> A solution might be to expose to users the choice of using hostname or
> > canonical host name on both sides.
> > >> Say having one setting that collapses functionalities from both KIPs
> > (bootstrap expansion + advertised lookup)
> > >> and an additional parameter that defines how the resolution is
> > performed, using getCanonicalHostName() or not.
> > >
> > > thanks sounds to me *less* simple than independent config options, sorry.
> > >
> > > I would like to say once again that by itself KIP-302 only speeds up
> > > the client behavior that can happen anyway when the client restarts
> > > multiple times,
> > > as every time there is no guarantee that - in presence of multiple A
> > > DNS records - the same IP is returned. Attempting to use additiona IPs
> > > if the first fail just makes client recovery faster.
> > >
> > > cheers
> > > Edo
> > >
> > > On 17 May 2018 at 12:12, Skrzypek, Jonathan <Jo...@gs.com>
> > wrote:
> > >> Yes, makes sense.
> > >> You mentioned multiple times you see no overlap and no issue with your
> > KIP, and that they solve different use cases.
> > >>
> > >> Appreciate you have an existing use case that would work, but we need
> > to make sure this isn't confusing to users and that any combination will
> > always work, across security protocols.
> > >>
> > >> A solution might be to expose to users the choice of using hostname or
> > canonical host name on both sides.
> > >> Say having one setting that collapses functionalities from both KIPs
> > (bootstrap expansion + advertised lookup) and an additional parameter that
> > defines how the resolution is performed, using getCanonicalHostName() or
> > not.
> > >>
> > >> Maybe that gives less flexibility as users wouldn't be able to decide
> > to only perform DNS lookup on bootstrap.servers or on advertised listeners.
> > >> But this would ensure consistency so that a user can decide to use
> > cnames or not (depending on their certificates and Kerberos principals in
> > their environment) and it would work.
> > >>
> > >> Jonathan Skrzypek
> > >>
> > >> -----Original Message-----
> > >> From: Edoardo Comar [mailto:edocomar@gmail.com]
> > >> Sent: 16 May 2018 21:59
> > >> To: dev@kafka.apache.org
> > >> Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS
> > resolved IP addresses
> > >>
> > >> Hi Jonathan,
> > >> I am afraid that may not work for everybody.
> > >>
> > >> It would not work for us.
> > >> With our current DNS, my Kafka clients are perfectly happy to use any
> > IPs -
> > >> DNS has multiple A records for the 'myhostname.mydomain' used for
> > >> bootstrap and advertised listeners.
> > >> The hosts all serve TLS certificates that include
> > >> 'myhostname.mydomain' and the clients are happy.
> > >>
> > >> However, applying getCanonicalHostName to those IPs would return
> > >> hostnames that would not match the TLS certificates.
> > >>
> > >> So once again I believe your solution and ours solve different use
> > cases.
> > >>
> > >> cheers
> > >> Edo
> > >>
> > >> On 16 May 2018 at 18:29, Skrzypek, Jonathan <Jo...@gs.com>
> > wrote:
> > >>> I think there are combinations that will break SASL and SSL auth.
> > >>> Could the trick be to have a single parameter that triggers dns
> > resolve both for bootstrap and advertised listeners, both using
> > getCanonicalHostName() ?
> > >>>
> > >>> Jonathan Skrzypek
> > >>>
> > >>> -----Original Message-----
> > >>> From: Edoardo Comar [mailto:edocomar@gmail.com]
> > >>> Sent: 16 May 2018 17:03
> > >>> To: dev@kafka.apache.org
> > >>> Subject: Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS
> > resolved IP addresses
> > >>>
> > >>> Hi Rajini,
> > >>>
> > >>> In your example KIP-302 would attempt to connect to the first address
> > >>> returned, let's say
> > >>>
> > >>> www.apache.org/195.154.151.36
> > >>>
> > >>> then, only if that fails, will in turn try the remaining:
> > >>>
> > >>> www.apache.org/40.79.78.1
> > >>> www.apache.org/140.211.11.105
> > >>> www.apache.org/2001:bc8:2142:300:0:0:0:0
> > >>>
> > >>> You're right to say that we expect certificates served by those
> > >>> endpoints to be valid for "www.apache.org"
> > >>>
> > >>> Without KIP-302, only one would be attempted.
> > >>> Which is the first one, that can change every time
> > >>> (typically changes on every Java process restart,
> > >>> but may change also any time InetAddress.getAllByName it's invoked
> > >>> depending on the caching).
> > >>>
> > >>> The behavioral change that KIP-302 may introduce is that in the
> > example above,
> > >>> also an IPv6 connection may be attempted after some IPv4 ones.
> > >>>
> > >>> InetAddress.getAllByName() implementation uses a system property
> > >>> "java.net.preferIPv6Addresses"
> > >>> to decide which type of address to return first (default is still IPv4
> > >>> in java 10)
> > >>>
> > >>> We will amend the KIP and PR so that the loop only uses IPs of the
> > >>> same type as the first one returned.
> > >>>
> > >>> A part from the above, KIP 302 does not seem to change any existing
> > >>> client behaviour, as any one of multiple IP addresses (of a given
> > >>> v4/v6 type) can currently be picked.
> > >>> We're happy however to keep the looping behavior optional with the
> > >>> discussed config property, disabled by default.
> > >>>
> > >>> As for KIP-235 that may introduce new hostnames in the bootstrap list
> > >>> (the current PR rewrites the bootstrap list)
> > >>> and we fail to see the conflict with KIP-302, whatever the set of
> > >>> configs chosen.
> > >>>
> > >>> We'd be happy to try understand what we are missing in a KIP call :-)
> > >>>
> > >>> cheers
> > >>> Edo
> > >>>
> > >>> On 15 May 2018 at 16:58, Rajini Sivaram <ra...@gmail.com>
> > wrote:
> > >>>> Hi Edo,
> > >>>>
> > >>>> I agree that KIP-235 and KIP-302 address different scenarios. And I
> > agree
> > >>>> that each one is not sufficient in itself to address both the
> > scenarios.
> > >>>> But I also think that they conflict and hence they need to be looked
> > at
> > >>>> together and perhaps use a single config.
> > >>>>
> > >>>> As an example:
> > >>>>
> > >>>> If I run:
> > >>>>
> > >>>> for (InetAddress address : InetAddress.getAllByName("www.apache.org"))
> > {
> > >>>> System.out.printf("HostName %s canonicalHostName %s IP %s\n",
> > >>>> address.getHostName(), address.getCanonicalHostName(),
> > >>>> address.getHostAddress());
> > >>>> }
> > >>>>
> > >>>> I get:
> > >>>>
> > >>>> HostName www.apache.org canonicalHostName tlp-eu-west.apache.org IP
> > >>>> 195.154.151.36
> > >>>> HostName www.apache.org canonicalHostName 40.79.78.1 IP 40.79.78.1
> > >>>> HostName www.apache.org canonicalHostName themis.apache.org IP
> > >>>> 140.211.11.105
> > >>>> HostName www.apache.org canonicalHostName 2001:bc8:2142:300:0:0:0:0
> > IP
> > >>>> 2001:bc8:2142:300:0:0:0:0
> > >>>>
> > >>>>
> > >>>> If www.apache.org is used as a bootstrap address, KIP-302 would
> > connect to (
> > >>>> www.apache.org/195.154.151.36 and www.apache.org/140.211.11.105)
> > while
> > >>>> KIP-235 would connect to (tlp-eu-west.apache.org/195.154.151.3. and
> > >>>> themis.apache.org/140.211.11.105). This is a significant difference
> > not
> > >>>> just for Kerberos, but for any secure environment where hostname is
> > >>>> verified to prevent man-in-the-middle attacks. In your case, I
> > presume you
> > >>>> would have SSL certificates with the equivalent of www.apache.org on
> > both
> > >>>> the load balancers. In Jonathan's case, I presume he has Kerberos
> > >>>> principals for the equivalent of tlp-eu-west.apache.org and
> > >>>> themis.apache.org. We would want to support both scenarios
> > regardless of
> > >>>> the security protocol, just need to come up with configuration
> > options that
> > >>>> don't conflict.
> > >>>>
> > >>>>
> > >>>> On Mon, May 14, 2018 at 5:24 PM, Edoardo Comar <ed...@gmail.com>
> > wrote:
> > >>>>
> > >>>>> Thanks Rajini
> > >>>>>
> > >>>>> I still don't see the overlap between the two KIPS
> > >>>>>
> > >>>>> KIP-235 allows an expansion of hostnames on the bootstrap list.
> > >>>>>
> > >>>>> KIP-302 allows alternative IPs to be used to attempt a connection
> > >>>>> (either at bootstrap and when processing the MetadataResponse) to a
> > >>>>> given hostname.
> > >>>>>
> > >>>>> A use case would be that of active/passive LB fronting the brokers.
> > >>>>>
> > >>>>> Arguably, if Java honored the DNS-set TTL, and the TTL was low and on
> > >>>>> subsequent requests, the order of IPs returned by the
> > >>>>> InetAddress.getAllByName() was random, we may not need such an
> > >>>>> enhancement.
> > >>>>> In practice, a Java client can get stuck on a "bad" IP forever if it
> > >>>>> only relies on the first IP returned.
> > >>>>>
> > >>>>> HTH,
> > >>>>> Edo
> > >>>>>
> > >>>>> On 14 May 2018 at 16:23, Rajini Sivaram <ra...@gmail.com>
> > wrote:
> > >>>>> > Hi Edo,
> > >>>>> >
> > >>>>> > Thanks for the KIP. I think it will be good to include a diagram
> > to make
> > >>>>> it
> > >>>>> > easier to distinguish this scenario from that of KIP-235 without
> > reading
> > >>>>> > the PR.
> > >>>>> >
> > >>>>> > It may be worth considering if KIP-235 and this KIP could use a
> > single
> > >>>>> > config name with different values instead of two boolean configs:
> > >>>>> >
> > >>>>> > bootstrap.reverse.dns.lookup = true/false
> > >>>>> > enable.all.dns.ips = true/false
> > >>>>> >
> > >>>>> > Not all values of (bootstrap.reverse.dns.lookup,
> > enable.all.dns.ips) seem
> > >>>>> > to make sense. And not all scenarios are handled. Even if we use
> > multiple
> > >>>>> > configs, it seems to me that we may want to name them differently.
> > >>>>> >
> > >>>>> > The possible combinations are:
> > >>>>> >
> > >>>>> > 1) Bootstrap
> > >>>>> >
> > >>>>> > a) No lookup
> > >>>>> > b) Use all DNS entries with host name
> > >>>>> > c) Use all DNS entries with canonical host name
> > >>>>> >
> > >>>>> > 2) Advertised listeners
> > >>>>> >
> > >>>>> > a) No lookup
> > >>>>> > b) Use all DNS entries with host name
> > >>>>> > c) Use all DNS entries with canonical host name
> > >>>>> >
> > >>>>> > The combinations that are enabled by the two boolean configs (
> > >>>>> > bootstrap.reverse.dns.lookup, enable.all.dns.ips) are:
> > >>>>> >
> > >>>>> > - (false, false) => (1a, 2a)
> > >>>>> > - (true, false) => (1c, 2a)
> > >>>>> > - (false, true) => (1b, 2b)
> > >>>>> > - (true, true) => (??, 2b)
> > >>>>> >
> > >>>>> > It will be good if we can clearly identify which combinations we
> > want to
> > >>>>> > support and the scenarios where they may be useful. Perhaps (1a,
> > 2a),
> > >>>>> (1c,
> > >>>>> > 2a), (1b, 2b) and (1c, 2c) are useful?
> > >>>>> >
> > >>>>> >
> > >>>>> > On Mon, May 14, 2018 at 2:58 PM, Skrzypek, Jonathan <
> > >>>>> > Jonathan.Skrzypek@gs.com> wrote:
> > >>>>> >
> > >>>>> >> Ah, apologies didn't see there was already a decent amount of
> > discussion
> > >>>>> >> on this in the PR.
> > >>>>> >>
> > >>>>> >> This kind of sounds related to the environment you're running to
> > me.
> > >>>>> >> What is the rationale behind using the advertised listeners to do
> > your
> > >>>>> >> load balancing advertisement rather than a top level alias that
> > has
> > >>>>> >> everything ?
> > >>>>> >>
> > >>>>> >> It sounds like in your case there is a mismatch between
> > >>>>> bootstrap.servers
> > >>>>> >> and advertised.listeners, and you want advertised.listeners to
> > take
> > >>>>> >> precedence and have the client iterate over what is returned by
> > the
> > >>>>> broker.
> > >>>>> >> So the extra parameter doesn't only have to do with DNS but it's
> > also
> > >>>>> >> appending from the broker, maybe the parameter name should
> > reflect this
> > >>>>> ?
> > >>>>> >>
> > >>>>> >> Jonathan Skrzypek
> > >>>>> >>
> > >>>>> >>
> > >>>>> >> -----Original Message-----
> > >>>>> >> From: Skrzypek, Jonathan [Tech]
> > >>>>> >> Sent: 14 May 2018 14:46
> > >>>>> >> To: dev@kafka.apache.org
> > >>>>> >> Subject: RE: [DISCUSS] KIP-302 - Enable Kafka clients to use all
> > DNS
> > >>>>> >> resolved IP addresses
> > >>>>> >>
> > >>>>> >> Hi,
> > >>>>> >>
> > >>>>> >> I see you noted the similarities with KIP-235.
> > >>>>> >> But KIP-235 might also solve what this KIP is trying to achieve.
> > >>>>> >>
> > >>>>> >> When parsing bootstrap.servers, KIP-235 has the client add all
> > >>>>> underlying
> > >>>>> >> hostnames and IPs.
> > >>>>> >> And this happens before hitting the NetworkClient.
> > >>>>> >>
> > >>>>> >> So to me the client will try every single endpoint behind any
> > >>>>> >> bootstrap.servers record.
> > >>>>> >>
> > >>>>> >> See
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_kafka_pull_4485_commits_24757eb7b0&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaCbPOM0NM1EHo-E&m=_ud9m_JZJ87C7eGsKcmzgJgDpNQDIIv5R4i_7VlhkLc&s=TqaiA9uW_myYO6FN-gKPfPlioxZR6DhnlBTpEj5M2aQ&e=
> > >>>>> >>
> > 6bcf8c7d7649c85232c52b5d54f0e4#diff-89ef153462e64c250a21bd324ae1a851
> > >>>>> >> which calls getAllByName like you suggested
> > >>>>> >>
> > >>>>> >> Jonathan Skrzypek
> > >>>>> >>
> > >>>>> >>
> > >>>>> >> -----Original Message-----
> > >>>>> >> From: Edoardo Comar [mailto:edocomar@gmail.com]
> > >>>>> >> Sent: 14 May 2018 14:17
> > >>>>> >> To: dev@kafka.apache.org
> > >>>>> >> Subject: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS
> > >>>>> resolved
> > >>>>> >> IP addresses
> > >>>>> >>
> > >>>>> >> Hi all,
> > >>>>> >>
> > >>>>> >> We just opened a KIP to add support for the client to use all IPs
> > >>>>> returned
> > >>>>> >> by DNS for the brokers
> > >>>>> >>
> > >>>>> >> The details are here -
> > >>>>> >>
> > >>>>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.a
> > >>>>> >> pache.org_confluence_display_KAFKA_KIP-2D302-2B-2D-2BEnable-
> > >>>>> >> 2BKafka-2Bclients-2Bto-2Buse-2Ball-2BDNS-2Bresolved-2BIP-
> > >>>>> >> 2Baddresses&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxL
> > >>>>> >> xfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaCbPOM0NM1EHo-E&m=EJafF
> > >>>>> >> l1clRyolgtcu2uCc4_cIOJnlxb1r1n-D2Dti4k&s=C-UZ6KUG7JFiPD_
> > >>>>> >> CnHczDOVqH9-XC5f_OFkw4BTNrI4&e=
> > >>>>> >>
> > >>>>> >> The JIRA and provisional PR (where the discussion lead to the
> > creation
> > >>>>> of
> > >>>>> >> this KIP) are :
> > >>>>> >>
> > >>>>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> > >>>>> >> apache.org_jira_browse_KAFKA-2D6863&d=DwIBaQ&c=7563p3e2zaQw0
> > >>>>> >> AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6
> > >>>>> >> eaCbPOM0NM1EHo-E&m=EJafFl1clRyolgtcu2uCc4_cIOJnlxb1r1n-
> > >>>>> >> D2Dti4k&s=3Puqs5iYoPsw6hARQr6gvokdFE-H5USMiNVGOUtNkJI&e=
> > >>>>> >>
> > >>>>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> > >>>>> >> com_apache_kafka_pull_4987&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgy
> > >>>>> >> agb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaC
> > >>>>> >> bPOM0NM1EHo-E&m=EJafFl1clRyolgtcu2uCc4_cIOJnlxb1r1n-D2Dti4k&
> > >>>>> >> s=Hqn5dOgQy4-MHTIJLE49O8bNomry3SoGq9OVoHU-CRA&e=
> > >>>>> >>
> > >>>>> >> Looking forward to the community's feedback.
> > >>>>> >> It would be amazing to have it voted by May 22nd :-) :-)
> > >>>>> >>
> > >>>>> >> Edoardo & Mickael
> > >>>>> >>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> "When the people fear their government, there is tyranny; when the
> > >>>>> government fears the people, there is liberty." [Thomas Jefferson]
> > >>>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> "When the people fear their government, there is tyranny; when the
> > >>> government fears the people, there is liberty." [Thomas Jefferson]
> > >>
> > >>
> > >>
> > >> --
> > >> "When the people fear their government, there is tyranny; when the
> > >> government fears the people, there is liberty." [Thomas Jefferson]
> > >
> > >
> > >
> > > --
> > > "When the people fear their government, there is tyranny; when the
> > > government fears the people, there is liberty." [Thomas Jefferson]
> > >
> > > ________________________________
> > >
> > > Your Personal Data: We may collect and process information about you
> > that may be subject to data protection laws. For more information about how
> > we use and disclose your personal data, how we protect your information,
> > our legal basis to use your information, your rights and who you can
> > contact, please refer to: www.gs.com/privacy-notices<
> > http://www.gs.com/privacy-notices>
> > >
> > >
> > > ---------- Forwarded message ----------
> > > From: Rajini Sivaram <ra...@gmail.com>
> > > To: "Skrzypek, Jonathan" <Jo...@ln.email.gs.com>, dev <
> > dev@kafka.apache.org>, Ismael Juma <is...@juma.me.uk>
> > > Cc:
> > > Bcc:
> > > Date: Tue, 22 May 2018 15:05:07 +0000
> > > Subject: Re: FW: [VOTE] KIP-235 Add DNS alias support for secured
> > connection
> > > Hi Jonathan,
> > >
> > > I think it would make sense to convert the config in this KIP into an
> > enum so that we can add more variations later on. But since KIP-302 is
> > still under discussion, it is not clear what the config name should be.
> > Since today is the KIP deadline and the implementation itself is
> > straightforward, it would make sense to progress with this one for 2.0.0 if
> > we can get one more binding vote.
> > >
> > > Ismael, do you have time to take a look at KIP-235 today?
> > >
> > > Thanks,
> > >
> > > Rajini
> > >
> > >
> > > On Tue, May 22, 2018 at 3:45 PM, Skrzypek, Jonathan <
> > Jonathan.Skrzypek@gs.com> wrote:
> > >>
> > >> Hello Rajini,
> > >>
> > >> What do you think should be the next step here ?
> > >>
> > >>
> > >> Jonathan Skrzypek
> > >>
> > >> -----Original Message-----
> > >> From: Skrzypek, Jonathan [Tech]
> > >> Sent: 21 May 2018 10:51
> > >> To: 'dev'
> > >> Subject: RE: [VOTE] KIP-235 Add DNS alias support for secured connection
> > >>
> > >> Hi,
> > >>
> > >> What would be the next step here ?
> > >> I know there's a discussion going on around KIP-302, but I'm also
> > conscious that the 2.0.0 deadline for KIPs is tomorrow.
> > >> I've opened this KIP in January and discussions have been productive
> > with an end solution I had the impression was reasonable, so I am keen to
> > see it make it the next release.
> > >>
> > >>
> > >> Jonathan Skrzypek
> > >>
> > >> -----Original Message-----
> > >> From: Skrzypek, Jonathan [Tech]
> > >> Sent: 14 May 2018 13:48
> > >> To: dev
> > >> Subject: RE: [VOTE] KIP-235 Add DNS alias support for secured connection
> > >>
> > >> Sure, I modified the KIP to add more details
> > >>
> > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-235%3A+Add+DNS+alias+support+for+secured+connection
> > >>
> > >>
> > >> Jonathan Skrzypek
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Ismael Juma [mailto:ismael@juma.me.uk]
> > >> Sent: 14 May 2018 11:53
> > >> To: dev
> > >> Subject: Re: [VOTE] KIP-235 Add DNS alias support for secured connection
> > >>
> > >> Thanks for the KIP, Jonathan. It would be helpful to have more detail on
> > >> how SSL authentication could be broken if the new behaviour is the
> > default.
> > >> I know this was discussed in the mailing list thread, but it's
> > important to
> > >> include it in the KIP since it's the main reason why a new config is
> > needed
> > >> (and configs should be avoided whenever we can just do the right thing).
> > >>
> > >> Ismael
> > >>
> > >> On Fri, Mar 23, 2018 at 12:05 PM Skrzypek, Jonathan <
> > >> Jonathan.Skrzypek@gs.com> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > I would like to start a vote for KIP-235
> > >> >
> > >> >
> > >> >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D235-253A-2BAdd-2BDNS-2Balias-2Bsupport-2Bfor-2Bsecured-2Bconnection&d=DwIBaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=nNmJlu1rR_QFAPdxGlafmDu9_r6eaCbPOM0NM1EHo-E&m=FM_uCHnnO2dqxWC0bi7_QOJKfKmQI80-Xduvb-URWOw&s=RpGkijfK-WHcU0s8ZtMXEkIr69QraJhYKaGSC9V_rnI&e=
> > >> >
> > >> > This is a proposition to add an option for reverse dns lookup of
> > >> > bootstrap.servers hosts, allowing the use of dns aliases on clusters
> > using
> > >> > SASL authentication.
> > >> >
> > >> >
> > >> >
> > >> >
> > >>
> > >> ________________________________
> > >>
> > >> Your Personal Data: We may collect and process information about you
> > that may be subject to data protection laws. For more information about how
> > we use and disclose your personal data, how we protect your information,
> > our legal basis to use your information, your rights and who you can
> > contact, please refer to: www.gs.com/privacy-notices<
> > http://www.gs.com/privacy-notices>
> > >
> > >
> > >
> >
> >
> >
> > --
> > "When the people fear their government, there is tyranny; when the
> > government fears the people, there is liberty." [Thomas Jefferson]
> >
>
>
> --
> "When the people fear their government, there is tyranny; when the
> government fears the people, there is liberty." [Thomas Jefferson]