You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Ron Dagostino <rn...@gmail.com> on 2019/12/23 20:03:44 UTC
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Hi everyone. Let's get this discussion going again now that Kafka 2.4 with
Zookeeper 3.5.6 is out.
First, regarding the KIP number, the other KIP that was using this number
moved to KIP 534, so KIP 515 remains the correct number for this
discussion. I've updated the Kafka Improvement Proposal page to list this
KIP in the 515 slot, so we're all set there.
Regarding the actual issues under discussion, I think there are some things
we should clarify.
1) It is possible to use TLS connectivity to Zookeeper from Apache Kafka
2.4 -- the problem is that configuration information has to be passed via
system properties as "-D" command line options on the java invocation of
the broker, and those are not secure (anyone with access to the box can see
the command line used to invoke the broker); the configuration includes
sensitive information (e.g. a keystore password), so we need a secure
mechanism for passing the configuration values. I believe the real
motivation for this KIP is to harden the configuration mechanism for
Zookeeper TLS connectivity.
2) I believe the list of CLI tools that continue to use direct Zookeeper
connectivity in a non-deprecated fashion is:
a) zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
b)
kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
3) I believe Kafka.admin.ConfigCommand presents a conundrum because it
explicitly states in a comment that a supported use case is bootstrapping a
Kafka cluster with encrypted passwords in Zookeeper (see
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65).
This means it will be especially difficult to deprecate this particular
direct Zookeeper connectivity without a different storage mechanism for
dynamic configuration values being available (i.e. the self-managed quorum
referred to in KIP-500).
I think it would be easier and simpler to harden the Zookeeper TLS
configuration in both Kafka and in CLI tools compared to trying to
deprecate the direct Zookeeper connectivity in the above 3 CLI tools --
especially when ConfigCommand has no obvious short-term path to deprecation.
Regarding how to harden the TLS configuration, adding the appropriate
configs to Kafka is an obvious choice for the brokers. Not that these
values cannot be dynamically reconfigurable because the values would be
required to connect to Zookeeper via TLS in the first place.
Regarding how to harden the TLS configuration for the 3 remaining CLI
tools, it would be relatively straightforward to add support for a
"--zk-tls-config-file <String: Zookeeper TLS configuration system
properties file path>" option to each one.
Thoughts?
Ron
On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <pe...@gmail.com>
wrote:
> True, the number is duplicated.
>
> do you know how can we get the problem fix? I was not aware, I missed,
> sorry the need to add the KIP to the table.
>
> -- Pere
>
> Missatge de Jorge Esteban Quilcate Otoya <qu...@gmail.com> del
> dia
> dt., 3 de set. 2019 a les 13:43:
>
> > Hi Pere,
> >
> > Have you add your KIP to the list here
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> ?
> >
> > I found the KIP number assigned to another.
> >
> >
> >
> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <pe...@gmail.com>
> > wrote:
> >
> >> Thanks for your time Harsha,
> >> anyone else with comments? looking forward to hearing from you.
> >>
> >> Stupid question: when do you move from discussion to vote?
> >>
> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv., 30 d’ag.
> >> 2019 a les 21:59:
> >>
> >> > Thanks Pere. KIP looks good to me.
> >> > -Harsha
> >> >
> >> >
> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> >> pere.urbon@gmail.com>
> >> > wrote:
> >> >
> >> >> Not really,
> >> >> my idea is to keep the JAAS parameter, so people don't see major
> >> >> changes. But if you pass a properties file, then this takes
> precedence
> >> over
> >> >> the other, with the idea that you can do sasl as well with the
> >> properties
> >> >> files.
> >> >>
> >> >> Makes sense?
> >> >>
> >> >> -- Pere
> >> >>
> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv., 30
> >> d’ag.
> >> >> 2019 a les 19:00:
> >> >>
> >> >>> Hi Pere,
> >> >>> Thanks for the KIP. Enabling SSL for zookeeper for Kafka
> >> makes
> >> >>> sense.
> >> >>> "The changes are planned to be introduced in a compatible way, by
> >> >>> keeping the current JAAS variable precedence."
> >> >>> Can you elaborate a bit here. If the user configures a JAAS file
> with
> >> >>> Client section it will take precedence over zookeeper SSL configs?
> >> >>>
> >> >>> Thanks,
> >> >>> Harsha
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> >> pere.urbon@gmail.com>
> >> >>> wrote:
> >> >>>
> >> >>>> Hi,
> >> >>>> quick question, I saw in another mail that 2.4 release is planned
> for
> >> >>>> September. I think it would be really awesome to have this for this
> >> >>>> release, do you think we can make it?
> >> >>>>
> >> >>>> -- Pere
> >> >>>>
> >> >>>> Missatge de Pere Urbón Bayes <pe...@gmail.com> del dia dj.,
> 29
> >> >>>> d’ag. 2019 a les 20:10:
> >> >>>>
> >> >>>> Hi,
> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm really
> need
> >> >>>> to the process. Looking forward to hearing from you and learn the
> >> best
> >> >>>> ropes here.
> >> >>>>
> >> >>>> I would like to propose this KIP-515 to enable the ZookeeperClients
> >> to
> >> >>>> take full advantage of the TLS communication in the new Zookeeper
> >> 3.5.5.
> >> >>>> Specially interesting it the Zookeeper Security Migration, that
> >> without
> >> >>>> this change will not work with TLS, disabling users to use ACLs
> when
> >> the
> >> >>>> Zookeeper cluster use TLS.
> >> >>>>
> >> >>>> link:
> >> >>>>
> >> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> >> >>>>
> >> >>>> Looking forward to hearing from you on this,
> >> >>>>
> >> >>>> /cheers
> >> >>>>
> >> >>>> --
> >> >>>> Pere Urbon-Bayes
> >> >>>> Software Architect
> >> >>>> http://www.purbon.com
> >> >>>> https://twitter.com/purbon
> >> >>>> https://www.linkedin.com/in/purbon/
> >> >>>>
> >> >>>> --
> >> >>>> Pere Urbon-Bayes
> >> >>>> Software Architect
> >> >>>> http://www.purbon.com
> >> >>>> https://twitter.com/purbon
> >> >>>> https://www.linkedin.com/in/purbon/
> >> >>>>
> >> >>>
> >> >>>
> >> >>
> >> >> --
> >> >> Pere Urbon-Bayes
> >> >> Software Architect
> >> >> http://www.purbon.com
> >> >> https://twitter.com/purbon
> >> >> https://www.linkedin.com/in/purbon/
> >> >>
> >> >
> >> >
> >>
> >> --
> >> Pere Urbon-Bayes
> >> Software Architect
> >> http://www.purbon.com
> >> https://twitter.com/purbon
> >> https://www.linkedin.com/in/purbon/
> >>
> >
>
> --
> Pere Urbon-Bayes
> Software Architect
> http://www.purbon.com
> https://twitter.com/purbon
> https://www.linkedin.com/in/purbon/
>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Ron Dagostino <rn...@gmail.com>.
Hi Rajini. Thanks for pointing out kafka-acls as another bootstrap use
case. I will update the KIP to mention it. (BTW, I sent an email on 1/8
stating that I had updated the KIP to reflect the current understanding and
discussion, but that email did not go through -- the KIP has undergone a
major rewrite, so interested parties should definitely take a look).
It is easy to support the bootstrap use case as I described previously, so
I think we should support it in both kafka-acls and kafka-configs.
Regarding SASL + SSL simultaneously, yes, the two can be used together in
ZooKeeper. I will be sure to state this in any documentation, and I will
update the KIP to fully describe how this works, including the principal
that is used and whether there is any way to transform the DN in the
certificate. Here is a quick summary:
1) ZooKeeper 3.5.6 requires a client certificate. There is a sever-side
config "ssl.clientAuth" that the code recognizes (case-insensitively:
want/need/none are the valid options), but this config has no effect in
3.5.6 (https://issues.apache.org/jira/browse/ZOOKEEPER-3674). I built from
source recently and confirmed that it worked in the 3.6 SNAPSHOT, but that
version is not yet released.
2) ZooKeeper will associate multiple identities with any session that
authenticates multiple ways (e.g. SSL and SASL). A client is authorized if
it has at least 1 of the identities present in an ACL.
3) The SSL identity is the DN, and that can only be changed by implementing
and using a custom ZK authentication provider that overrides the
method protected
String getClientId(X509Certificate clientCert).
Given the above, if clients want to use client certificate authentication
with SASL, then there really isn't any difficulty -- brokers and ZK
security migrator will be identified by both their respective DNs and the
SASL identity, and at least the SASL identity will be common to all clients
as it is today, so it continues to work fine.
If clients want to use client certificate authentication without SASL
(which is I believe what Pere intended in the original KIP discussion),
then either all brokers and ZK security migrator will have to present the
same DN (with different SANs for the different hostnames) or ZooKeeper will
have to be running with a custom authProvider that maps the different DNs
to a single, common identity (this second approach is more difficult and
seems to be less secure than the first one that uses SANs since it creates
the possibility of the DN mapping being leveraged in a malicious way).
Since I am catching up on emails I sent that did not go through, I'll also
include the following two comments here:
1) The ZooKeeper project's command line interface has no facility for
accepting client TLS configuration except via Java system properties. I've
opened ZooKeeper issue https://issues.apache.org/jira/browse/ZOOKEEPER-3689 for
this, but in the meantime we can address it in Kafka's
zookeeper-shell.{bat,sh} scripts by invoking a new, Kafka-specific subclass
of org.apache.zookeeper.ZooKeeperMain as opposed
to org.apache.zookeeper.ZooKeeperMain itself. We would simply look for a
"-zk-tls-config-file <file>" argument pair, create the TLS client config if
specified, and pass the arguments without the config information to the
superclass; then we would override the connectToZK(String) method to use
any created TLS client config. While the subclass unfortunately needs to
be in the same org.apache.zookeeper package due to the package-private
run() method in ZooKeeperMain that it has to call, it is working fine
(confirmed via a system test).
2) Regarding Thomas Zhou's question on 1/10 about changing the way client
certificates are loaded (
https://lists.apache.org/thread.html/r3bf6c58b052bf47bc99c360dff86a8bd38c35a89a7387ba2a5f16dee%40%3Cdev.kafka.apache.org%3E),
when I updated the KIP I included all of the ZooKeeper-supported
client configurations related to TLS (not just keystore/trsustore locations
and passwords). There is no intention to support anything beyond what
ZooKeeper supports, but the "zookeeper.ssl.context.supplier.class"
configuration is included and is useful for this particular use case.
Ron
On Tue, Jan 14, 2020 at 10:48 AM Rajini Sivaram <ra...@gmail.com>
wrote:
> Resending this note which wasn't delivered when it was sent on Jan 6th:
>
> Hi Ron,
>
> Thanks for taking over this KIP. Is it possible to use SASL/GSSAPI with SSL
> for ZooKeeper like we do in Kafka? If not, we need to describe that in the
> migration section. We should also document the principals used for
> authorization when SSL is used and any additional configs that may be used
> for transforming the DN from the certificate.
>
> For the CLI tools, the two that use ZooKeeper for bootstrapping are
> kafka-configs.sh and kafka-acls.sh. In both cases, we support direct
> connectivity to ZooKeeper in order to store configs/ACLs prior to starting
> brokers. Are we going to support SSL for both?
>
> Regards,
>
> Rajini
>
> On Tue, Jan 14, 2020 at 3:39 PM Ron Dagostino <rn...@gmail.com> wrote:
>
> > Hi Colin.
> >
> > <<< It seems like this [--zk-tls-config-file information] could just
> appear
> > in a configuration file, which all of these tools already accept (I
> think)
> >
> > ZkSecurityMigrator has no such property file facility; adding a
> > "--zk-tls-config-file" parameter is exactly for this purpose. If we add
> > that to ZkSecurityMigrator then it is trivial to add it to other commands
> > (the same code is simply reused; it ends up being just a few extra
> lines).
> > I do not see any parameter in the other two commands to adjust the ZK
> > connection; ConfigCommand accepts a "--command-config" flag, but
> according
> > to the code "This is used only with --bootstrap-server option for
> > describing and altering broker configs."
> >
> > I do agree there would be no need to add "--zk-tls-config-file" to
> > ReassignPartitionsCommand if its direct ZK connectivity is replaced in
> time
> > for the next release.
> >
> > ConfigCommand supports the "--bootstrap-server" option and will have its
> > direct ZooKeeper access formally deprecated as per KIP-555, but the
> special
> > use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> > prior to starting Kafka will still exist, so it feels like while
> > "--zk-tls-config-file" would never be used except for this use case it
> > could probably still be added for this particular situation.
> >
> > Ron
> >
> > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> > sent) did not go through. I am trying to get emails through now to move
> > this discussion forward.
> >
> > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > Hi everyone. I would like to make the following changes to the KIP.
> > > >
> > > > MOTIVATION:
> > > > Include a statement that it will be difficult in the short term to
> > > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > > (which
> > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
> cluster
> > > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > > case;
> > > > therefore it is in scope to be able to securely configure the CLI
> tools
> > > > that still leverage non-deprecated direct Zookeeper communication for
> > TLS
> > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > > zookeeper-security-migration.sh).
> > >
> > > Hi Ron,
> > >
> > > Thanks for the KIP.
> > >
> > > About deprecations:
> > >
> > > * zookeeper-security-migration: clearly, deprecating ZK access in this
> > one
> > > would not make sense, since it would defeat the whole point of the tool
> > :)
> > >
> > > * kafka-reassign-partitions: ZK access should be deprecated here.
> > KIP-455
> > > implementation has dragged a bit, but this should get done soon.
> > Certainly
> > > before the next release.
> > >
> > > * kafka-configs: I think ZK access should be deprecated here as well.
> I
> > > agree there is a super-special bootstrapping case here, but that should
> > > have its own tool, not use kafka-configs.
> > >
> > > I will post a separate KIP for this, though.
> > >
> > > >
> > > > GOALS:
> > > > Support the secure configuration of TLS-encrypted communication
> between
> > > > Zookeeper and:
> > > > a) Kafka brokers
> > > > b) The three CLI tools mentioned above that still support direct,
> > > > non-deprecated communication to Zookeeper
> > > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > > communication in CLI tools as part of this KIP; such work will occur
> in
> > > > future KIPs instead.
> > > >
> > > > PUBLIC INTERFACES:
> > > > 1) The following new broker configurations will be recognized.
> > > > zookeeper.client.secure (default value = false, for backwards
> > > > compatibility)
> > > > zookeeper.clientCnxnSocket
> > > > zookeeper.ssl.keyStore.location
> > > > zookeeper.ssl.keyStore.password
> > > > zookeeper.ssl.trustStore.location
> > > > zookeeper.ssl.trustStore.password
> > > > It will be an error for any of the last 5 values to be left
> unspecified
> > > if
> > > > zookeeper.client.secure is explicitly set to true.
> > > >
> > > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> > > supports
> > > > the ability to connect to a different Zookeeper instance than the one
> > the
> > > > brokers use. We therefore also add the following optional configs,
> > which
> > > > override the corresponding ones from above when present:
> > > > authorizer.zookeeper.client.secure
> > > > authorizer.zookeeper.clientCnxnSocket
> > > > authorizer.zookeeper.ssl.keyStore.location
> > > > authorizer.zookeeper.ssl.keyStore.password
> > > > authorizer.zookeeper.ssl.trustStore.location
> > > > authorizer.zookeeper.ssl.trustStore.password
> > > >
> > > > 3) The three CLI tools mentioned above will support a new
> > > --zk-tls-config-file
> > > > <String: Zookeeper TLS configuration file path>" option. The
> following
> > > > properties will be recognized in that file, and unrecognized
> properties
> > > > will be ignored to allow the possibility of pointing
> zk-tls-config-file
> > > at
> > > > the broker's config file.
> > > > zookeeper.client.secure (default value = false)
> > > > zookeeper.clientCnxnSocket
> > > > zookeeper.ssl.keyStore.location
> > > > zookeeper.ssl.keyStore.password
> > > > zookeeper.ssl.trustStore.location
> > > > zookeeper.ssl.trustStore.password
> > > > It will be an error for any of the last 5 values to be left
> unspecified
> > > if
> > > > zookeeper.client.secure is explicitly set to true.
> > >
> > > Do we really need a --zk-tls-config-file flag? It seems like this
> could
> > > just appear in a configuration file, which all of these tools already
> > > accept (I think).
> > >
> > > best,
> > > Colin
> > >
> > >
> > > >
> > > > Ron
> > > >
> > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rn...@gmail.com>
> > wrote:
> > > >
> > > > > Hi everyone. Let's get this discussion going again now that Kafka
> > 2.4
> > > > > with Zookeeper 3.5.6 is out.
> > > > >
> > > > > First, regarding the KIP number, the other KIP that was using this
> > > number
> > > > > moved to KIP 534, so KIP 515 remains the correct number for this
> > > > > discussion. I've updated the Kafka Improvement Proposal page to
> list
> > > this
> > > > > KIP in the 515 slot, so we're all set there.
> > > > >
> > > > > Regarding the actual issues under discussion, I think there are
> some
> > > > > things we should clarify.
> > > > >
> > > > > 1) It is possible to use TLS connectivity to Zookeeper from Apache
> > > Kafka
> > > > > 2.4 -- the problem is that configuration information has to be
> passed
> > > via
> > > > > system properties as "-D" command line options on the java
> invocation
> > > of
> > > > > the broker, and those are not secure (anyone with access to the box
> > > can see
> > > > > the command line used to invoke the broker); the configuration
> > includes
> > > > > sensitive information (e.g. a keystore password), so we need a
> secure
> > > > > mechanism for passing the configuration values. I believe the real
> > > > > motivation for this KIP is to harden the configuration mechanism
> for
> > > > > Zookeeper TLS connectivity.
> > > > >
> > > > > 2) I believe the list of CLI tools that continue to use direct
> > > Zookeeper
> > > > > connectivity in a non-deprecated fashion is:
> > > > > a)
> zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > > b)
> > > > >
> > >
> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > > > >
> > > > > 3) I believe Kafka.admin.ConfigCommand presents a conundrum because
> > it
> > > > > explicitly states in a comment that a supported use case is
> > > bootstrapping a
> > > > > Kafka cluster with encrypted passwords in Zookeeper (see
> > > > >
> > >
> >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > > ).
> > > > > This means it will be especially difficult to deprecate this
> > particular
> > > > > direct Zookeeper connectivity without a different storage mechanism
> > for
> > > > > dynamic configuration values being available (i.e. the self-managed
> > > quorum
> > > > > referred to in KIP-500).
> > > > >
> > > > > I think it would be easier and simpler to harden the Zookeeper TLS
> > > > > configuration in both Kafka and in CLI tools compared to trying to
> > > > > deprecate the direct Zookeeper connectivity in the above 3 CLI
> tools
> > --
> > > > > especially when ConfigCommand has no obvious short-term path to
> > > deprecation.
> > > > >
> > > > > Regarding how to harden the TLS configuration, adding the
> appropriate
> > > > > configs to Kafka is an obvious choice for the brokers. Not that
> > these
> > > > > values cannot be dynamically reconfigurable because the values
> would
> > be
> > > > > required to connect to Zookeeper via TLS in the first place.
> > > > >
> > > > > Regarding how to harden the TLS configuration for the 3 remaining
> CLI
> > > > > tools, it would be relatively straightforward to add support for a
> > > > > "--zk-tls-config-file <String: Zookeeper TLS configuration system
> > > > > properties file path>" option to each one.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Ron
> > > > >
> > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <
> > pere.urbon@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > >> True, the number is duplicated.
> > > > >>
> > > > >> do you know how can we get the problem fix? I was not aware, I
> > > missed,
> > > > >> sorry the need to add the KIP to the table.
> > > > >>
> > > > >> -- Pere
> > > > >>
> > > > >> Missatge de Jorge Esteban Quilcate Otoya <
> quilcate.jorge@gmail.com>
> > > del
> > > > >> dia
> > > > >> dt., 3 de set. 2019 a les 13:43:
> > > > >>
> > > > >> > Hi Pere,
> > > > >> >
> > > > >> > Have you add your KIP to the list here
> > > > >> >
> > > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > >> ?
> > > > >> >
> > > > >> > I found the KIP number assigned to another.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> > > pere.urbon@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> >> Thanks for your time Harsha,
> > > > >> >> anyone else with comments? looking forward to hearing from
> > you.
> > > > >> >>
> > > > >> >> Stupid question: when do you move from discussion to vote?
> > > > >> >>
> > > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv.,
> > 30
> > > > >> d’ag.
> > > > >> >> 2019 a les 21:59:
> > > > >> >>
> > > > >> >> > Thanks Pere. KIP looks good to me.
> > > > >> >> > -Harsha
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> > > > >> >> pere.urbon@gmail.com>
> > > > >> >> > wrote:
> > > > >> >> >
> > > > >> >> >> Not really,
> > > > >> >> >> my idea is to keep the JAAS parameter, so people don't see
> > > major
> > > > >> >> >> changes. But if you pass a properties file, then this takes
> > > > >> precedence
> > > > >> >> over
> > > > >> >> >> the other, with the idea that you can do sasl as well with
> the
> > > > >> >> properties
> > > > >> >> >> files.
> > > > >> >> >>
> > > > >> >> >> Makes sense?
> > > > >> >> >>
> > > > >> >> >> -- Pere
> > > > >> >> >>
> > > > >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> > dv.,
> > > 30
> > > > >> >> d’ag.
> > > > >> >> >> 2019 a les 19:00:
> > > > >> >> >>
> > > > >> >> >>> Hi Pere,
> > > > >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper
> for
> > > Kafka
> > > > >> >> makes
> > > > >> >> >>> sense.
> > > > >> >> >>> "The changes are planned to be introduced in a compatible
> > way,
> > > by
> > > > >> >> >>> keeping the current JAAS variable precedence."
> > > > >> >> >>> Can you elaborate a bit here. If the user configures a JAAS
> > > file
> > > > >> with
> > > > >> >> >>> Client section it will take precedence over zookeeper SSL
> > > configs?
> > > > >> >> >>>
> > > > >> >> >>> Thanks,
> > > > >> >> >>> Harsha
> > > > >> >> >>>
> > > > >> >> >>>
> > > > >> >> >>>
> > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> > > > >> >> pere.urbon@gmail.com>
> > > > >> >> >>> wrote:
> > > > >> >> >>>
> > > > >> >> >>>> Hi,
> > > > >> >> >>>> quick question, I saw in another mail that 2.4 release is
> > > planned
> > > > >> for
> > > > >> >> >>>> September. I think it would be really awesome to have this
> > for
> > > > >> this
> > > > >> >> >>>> release, do you think we can make it?
> > > > >> >> >>>>
> > > > >> >> >>>> -- Pere
> > > > >> >> >>>>
> > > > >> >> >>>> Missatge de Pere Urbón Bayes <pe...@gmail.com> del
> dia
> > > dj.,
> > > > >> 29
> > > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > > >> >> >>>>
> > > > >> >> >>>> Hi,
> > > > >> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm
> > > really
> > > > >> need
> > > > >> >> >>>> to the process. Looking forward to hearing from you and
> > learn
> > > the
> > > > >> >> best
> > > > >> >> >>>> ropes here.
> > > > >> >> >>>>
> > > > >> >> >>>> I would like to propose this KIP-515 to enable the
> > > > >> ZookeeperClients
> > > > >> >> to
> > > > >> >> >>>> take full advantage of the TLS communication in the new
> > > Zookeeper
> > > > >> >> 3.5.5.
> > > > >> >> >>>> Specially interesting it the Zookeeper Security Migration,
> > > that
> > > > >> >> without
> > > > >> >> >>>> this change will not work with TLS, disabling users to use
> > > ACLs
> > > > >> when
> > > > >> >> the
> > > > >> >> >>>> Zookeeper cluster use TLS.
> > > > >> >> >>>>
> > > > >> >> >>>> link:
> > > > >> >> >>>>
> > > > >> >> >>>>
> > > > >> >>
> > > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > > >> >> >>>>
> > > > >> >> >>>> Looking forward to hearing from you on this,
> > > > >> >> >>>>
> > > > >> >> >>>> /cheers
> > > > >> >> >>>>
> > > > >> >> >>>> --
> > > > >> >> >>>> Pere Urbon-Bayes
> > > > >> >> >>>> Software Architect
> > > > >> >> >>>> http://www.purbon.com
> > > > >> >> >>>> https://twitter.com/purbon
> > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > >> >> >>>>
> > > > >> >> >>>> --
> > > > >> >> >>>> Pere Urbon-Bayes
> > > > >> >> >>>> Software Architect
> > > > >> >> >>>> http://www.purbon.com
> > > > >> >> >>>> https://twitter.com/purbon
> > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > >> >> >>>>
> > > > >> >> >>>
> > > > >> >> >>>
> > > > >> >> >>
> > > > >> >> >> --
> > > > >> >> >> Pere Urbon-Bayes
> > > > >> >> >> Software Architect
> > > > >> >> >> http://www.purbon.com
> > > > >> >> >> https://twitter.com/purbon
> > > > >> >> >> https://www.linkedin.com/in/purbon/
> > > > >> >> >>
> > > > >> >> >
> > > > >> >> >
> > > > >> >>
> > > > >> >> --
> > > > >> >> Pere Urbon-Bayes
> > > > >> >> Software Architect
> > > > >> >> http://www.purbon.com
> > > > >> >> https://twitter.com/purbon
> > > > >> >> https://www.linkedin.com/in/purbon/
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >> --
> > > > >> Pere Urbon-Bayes
> > > > >> Software Architect
> > > > >> http://www.purbon.com
> > > > >> https://twitter.com/purbon
> > > > >> https://www.linkedin.com/in/purbon/
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Rajini Sivaram <ra...@gmail.com>.
Resending this note which wasn't delivered when it was sent on Jan 6th:
Hi Ron,
Thanks for taking over this KIP. Is it possible to use SASL/GSSAPI with SSL
for ZooKeeper like we do in Kafka? If not, we need to describe that in the
migration section. We should also document the principals used for
authorization when SSL is used and any additional configs that may be used
for transforming the DN from the certificate.
For the CLI tools, the two that use ZooKeeper for bootstrapping are
kafka-configs.sh and kafka-acls.sh. In both cases, we support direct
connectivity to ZooKeeper in order to store configs/ACLs prior to starting
brokers. Are we going to support SSL for both?
Regards,
Rajini
On Tue, Jan 14, 2020 at 3:39 PM Ron Dagostino <rn...@gmail.com> wrote:
> Hi Colin.
>
> <<< It seems like this [--zk-tls-config-file information] could just appear
> in a configuration file, which all of these tools already accept (I think)
>
> ZkSecurityMigrator has no such property file facility; adding a
> "--zk-tls-config-file" parameter is exactly for this purpose. If we add
> that to ZkSecurityMigrator then it is trivial to add it to other commands
> (the same code is simply reused; it ends up being just a few extra lines).
> I do not see any parameter in the other two commands to adjust the ZK
> connection; ConfigCommand accepts a "--command-config" flag, but according
> to the code "This is used only with --bootstrap-server option for
> describing and altering broker configs."
>
> I do agree there would be no need to add "--zk-tls-config-file" to
> ReassignPartitionsCommand if its direct ZK connectivity is replaced in time
> for the next release.
>
> ConfigCommand supports the "--bootstrap-server" option and will have its
> direct ZooKeeper access formally deprecated as per KIP-555, but the special
> use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> prior to starting Kafka will still exist, so it feels like while
> "--zk-tls-config-file" would never be used except for this use case it
> could probably still be added for this particular situation.
>
> Ron
>
> P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> sent) did not go through. I am trying to get emails through now to move
> this discussion forward.
>
> On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cm...@apache.org> wrote:
>
> > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > Hi everyone. I would like to make the following changes to the KIP.
> > >
> > > MOTIVATION:
> > > Include a statement that it will be difficult in the short term to
> > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > (which
> > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka cluster
> > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > case;
> > > therefore it is in scope to be able to securely configure the CLI tools
> > > that still leverage non-deprecated direct Zookeeper communication for
> TLS
> > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > zookeeper-security-migration.sh).
> >
> > Hi Ron,
> >
> > Thanks for the KIP.
> >
> > About deprecations:
> >
> > * zookeeper-security-migration: clearly, deprecating ZK access in this
> one
> > would not make sense, since it would defeat the whole point of the tool
> :)
> >
> > * kafka-reassign-partitions: ZK access should be deprecated here.
> KIP-455
> > implementation has dragged a bit, but this should get done soon.
> Certainly
> > before the next release.
> >
> > * kafka-configs: I think ZK access should be deprecated here as well. I
> > agree there is a super-special bootstrapping case here, but that should
> > have its own tool, not use kafka-configs.
> >
> > I will post a separate KIP for this, though.
> >
> > >
> > > GOALS:
> > > Support the secure configuration of TLS-encrypted communication between
> > > Zookeeper and:
> > > a) Kafka brokers
> > > b) The three CLI tools mentioned above that still support direct,
> > > non-deprecated communication to Zookeeper
> > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > communication in CLI tools as part of this KIP; such work will occur in
> > > future KIPs instead.
> > >
> > > PUBLIC INTERFACES:
> > > 1) The following new broker configurations will be recognized.
> > > zookeeper.client.secure (default value = false, for backwards
> > > compatibility)
> > > zookeeper.clientCnxnSocket
> > > zookeeper.ssl.keyStore.location
> > > zookeeper.ssl.keyStore.password
> > > zookeeper.ssl.trustStore.location
> > > zookeeper.ssl.trustStore.password
> > > It will be an error for any of the last 5 values to be left unspecified
> > if
> > > zookeeper.client.secure is explicitly set to true.
> > >
> > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> > supports
> > > the ability to connect to a different Zookeeper instance than the one
> the
> > > brokers use. We therefore also add the following optional configs,
> which
> > > override the corresponding ones from above when present:
> > > authorizer.zookeeper.client.secure
> > > authorizer.zookeeper.clientCnxnSocket
> > > authorizer.zookeeper.ssl.keyStore.location
> > > authorizer.zookeeper.ssl.keyStore.password
> > > authorizer.zookeeper.ssl.trustStore.location
> > > authorizer.zookeeper.ssl.trustStore.password
> > >
> > > 3) The three CLI tools mentioned above will support a new
> > --zk-tls-config-file
> > > <String: Zookeeper TLS configuration file path>" option. The following
> > > properties will be recognized in that file, and unrecognized properties
> > > will be ignored to allow the possibility of pointing zk-tls-config-file
> > at
> > > the broker's config file.
> > > zookeeper.client.secure (default value = false)
> > > zookeeper.clientCnxnSocket
> > > zookeeper.ssl.keyStore.location
> > > zookeeper.ssl.keyStore.password
> > > zookeeper.ssl.trustStore.location
> > > zookeeper.ssl.trustStore.password
> > > It will be an error for any of the last 5 values to be left unspecified
> > if
> > > zookeeper.client.secure is explicitly set to true.
> >
> > Do we really need a --zk-tls-config-file flag? It seems like this could
> > just appear in a configuration file, which all of these tools already
> > accept (I think).
> >
> > best,
> > Colin
> >
> >
> > >
> > > Ron
> > >
> > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rn...@gmail.com>
> wrote:
> > >
> > > > Hi everyone. Let's get this discussion going again now that Kafka
> 2.4
> > > > with Zookeeper 3.5.6 is out.
> > > >
> > > > First, regarding the KIP number, the other KIP that was using this
> > number
> > > > moved to KIP 534, so KIP 515 remains the correct number for this
> > > > discussion. I've updated the Kafka Improvement Proposal page to list
> > this
> > > > KIP in the 515 slot, so we're all set there.
> > > >
> > > > Regarding the actual issues under discussion, I think there are some
> > > > things we should clarify.
> > > >
> > > > 1) It is possible to use TLS connectivity to Zookeeper from Apache
> > Kafka
> > > > 2.4 -- the problem is that configuration information has to be passed
> > via
> > > > system properties as "-D" command line options on the java invocation
> > of
> > > > the broker, and those are not secure (anyone with access to the box
> > can see
> > > > the command line used to invoke the broker); the configuration
> includes
> > > > sensitive information (e.g. a keystore password), so we need a secure
> > > > mechanism for passing the configuration values. I believe the real
> > > > motivation for this KIP is to harden the configuration mechanism for
> > > > Zookeeper TLS connectivity.
> > > >
> > > > 2) I believe the list of CLI tools that continue to use direct
> > Zookeeper
> > > > connectivity in a non-deprecated fashion is:
> > > > a) zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > b)
> > > >
> > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > > >
> > > > 3) I believe Kafka.admin.ConfigCommand presents a conundrum because
> it
> > > > explicitly states in a comment that a supported use case is
> > bootstrapping a
> > > > Kafka cluster with encrypted passwords in Zookeeper (see
> > > >
> >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > ).
> > > > This means it will be especially difficult to deprecate this
> particular
> > > > direct Zookeeper connectivity without a different storage mechanism
> for
> > > > dynamic configuration values being available (i.e. the self-managed
> > quorum
> > > > referred to in KIP-500).
> > > >
> > > > I think it would be easier and simpler to harden the Zookeeper TLS
> > > > configuration in both Kafka and in CLI tools compared to trying to
> > > > deprecate the direct Zookeeper connectivity in the above 3 CLI tools
> --
> > > > especially when ConfigCommand has no obvious short-term path to
> > deprecation.
> > > >
> > > > Regarding how to harden the TLS configuration, adding the appropriate
> > > > configs to Kafka is an obvious choice for the brokers. Not that
> these
> > > > values cannot be dynamically reconfigurable because the values would
> be
> > > > required to connect to Zookeeper via TLS in the first place.
> > > >
> > > > Regarding how to harden the TLS configuration for the 3 remaining CLI
> > > > tools, it would be relatively straightforward to add support for a
> > > > "--zk-tls-config-file <String: Zookeeper TLS configuration system
> > > > properties file path>" option to each one.
> > > >
> > > > Thoughts?
> > > >
> > > > Ron
> > > >
> > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <
> pere.urbon@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> True, the number is duplicated.
> > > >>
> > > >> do you know how can we get the problem fix? I was not aware, I
> > missed,
> > > >> sorry the need to add the KIP to the table.
> > > >>
> > > >> -- Pere
> > > >>
> > > >> Missatge de Jorge Esteban Quilcate Otoya <qu...@gmail.com>
> > del
> > > >> dia
> > > >> dt., 3 de set. 2019 a les 13:43:
> > > >>
> > > >> > Hi Pere,
> > > >> >
> > > >> > Have you add your KIP to the list here
> > > >> >
> > > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > >> ?
> > > >> >
> > > >> > I found the KIP number assigned to another.
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> > pere.urbon@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> >> Thanks for your time Harsha,
> > > >> >> anyone else with comments? looking forward to hearing from
> you.
> > > >> >>
> > > >> >> Stupid question: when do you move from discussion to vote?
> > > >> >>
> > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv.,
> 30
> > > >> d’ag.
> > > >> >> 2019 a les 21:59:
> > > >> >>
> > > >> >> > Thanks Pere. KIP looks good to me.
> > > >> >> > -Harsha
> > > >> >> >
> > > >> >> >
> > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> > > >> >> pere.urbon@gmail.com>
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> >> Not really,
> > > >> >> >> my idea is to keep the JAAS parameter, so people don't see
> > major
> > > >> >> >> changes. But if you pass a properties file, then this takes
> > > >> precedence
> > > >> >> over
> > > >> >> >> the other, with the idea that you can do sasl as well with the
> > > >> >> properties
> > > >> >> >> files.
> > > >> >> >>
> > > >> >> >> Makes sense?
> > > >> >> >>
> > > >> >> >> -- Pere
> > > >> >> >>
> > > >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> dv.,
> > 30
> > > >> >> d’ag.
> > > >> >> >> 2019 a les 19:00:
> > > >> >> >>
> > > >> >> >>> Hi Pere,
> > > >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper for
> > Kafka
> > > >> >> makes
> > > >> >> >>> sense.
> > > >> >> >>> "The changes are planned to be introduced in a compatible
> way,
> > by
> > > >> >> >>> keeping the current JAAS variable precedence."
> > > >> >> >>> Can you elaborate a bit here. If the user configures a JAAS
> > file
> > > >> with
> > > >> >> >>> Client section it will take precedence over zookeeper SSL
> > configs?
> > > >> >> >>>
> > > >> >> >>> Thanks,
> > > >> >> >>> Harsha
> > > >> >> >>>
> > > >> >> >>>
> > > >> >> >>>
> > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> > > >> >> pere.urbon@gmail.com>
> > > >> >> >>> wrote:
> > > >> >> >>>
> > > >> >> >>>> Hi,
> > > >> >> >>>> quick question, I saw in another mail that 2.4 release is
> > planned
> > > >> for
> > > >> >> >>>> September. I think it would be really awesome to have this
> for
> > > >> this
> > > >> >> >>>> release, do you think we can make it?
> > > >> >> >>>>
> > > >> >> >>>> -- Pere
> > > >> >> >>>>
> > > >> >> >>>> Missatge de Pere Urbón Bayes <pe...@gmail.com> del dia
> > dj.,
> > > >> 29
> > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > >> >> >>>>
> > > >> >> >>>> Hi,
> > > >> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm
> > really
> > > >> need
> > > >> >> >>>> to the process. Looking forward to hearing from you and
> learn
> > the
> > > >> >> best
> > > >> >> >>>> ropes here.
> > > >> >> >>>>
> > > >> >> >>>> I would like to propose this KIP-515 to enable the
> > > >> ZookeeperClients
> > > >> >> to
> > > >> >> >>>> take full advantage of the TLS communication in the new
> > Zookeeper
> > > >> >> 3.5.5.
> > > >> >> >>>> Specially interesting it the Zookeeper Security Migration,
> > that
> > > >> >> without
> > > >> >> >>>> this change will not work with TLS, disabling users to use
> > ACLs
> > > >> when
> > > >> >> the
> > > >> >> >>>> Zookeeper cluster use TLS.
> > > >> >> >>>>
> > > >> >> >>>> link:
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >>
> > > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > >> >> >>>>
> > > >> >> >>>> Looking forward to hearing from you on this,
> > > >> >> >>>>
> > > >> >> >>>> /cheers
> > > >> >> >>>>
> > > >> >> >>>> --
> > > >> >> >>>> Pere Urbon-Bayes
> > > >> >> >>>> Software Architect
> > > >> >> >>>> http://www.purbon.com
> > > >> >> >>>> https://twitter.com/purbon
> > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > >> >> >>>>
> > > >> >> >>>> --
> > > >> >> >>>> Pere Urbon-Bayes
> > > >> >> >>>> Software Architect
> > > >> >> >>>> http://www.purbon.com
> > > >> >> >>>> https://twitter.com/purbon
> > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > >> >> >>>>
> > > >> >> >>>
> > > >> >> >>>
> > > >> >> >>
> > > >> >> >> --
> > > >> >> >> Pere Urbon-Bayes
> > > >> >> >> Software Architect
> > > >> >> >> http://www.purbon.com
> > > >> >> >> https://twitter.com/purbon
> > > >> >> >> https://www.linkedin.com/in/purbon/
> > > >> >> >>
> > > >> >> >
> > > >> >> >
> > > >> >>
> > > >> >> --
> > > >> >> Pere Urbon-Bayes
> > > >> >> Software Architect
> > > >> >> http://www.purbon.com
> > > >> >> https://twitter.com/purbon
> > > >> >> https://www.linkedin.com/in/purbon/
> > > >> >>
> > > >> >
> > > >>
> > > >> --
> > > >> Pere Urbon-Bayes
> > > >> Software Architect
> > > >> http://www.purbon.com
> > > >> https://twitter.com/purbon
> > > >> https://www.linkedin.com/in/purbon/
> > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Ron Dagostino <rn...@gmail.com>.
Hi Manikumar. Yes, you are correct: if the ZK Security Migrator session authenticates to ZooKeeper with multiple identities — SASL and certificate — then ACLs are applied authorizing both the SASL principal and the certificate DN.
Ron
> On Jan 15, 2020, at 6:33 AM, Manikumar <ma...@gmail.com> wrote:
>
> Hi Ron,
>
> Thanks for the KIP. KIP looks good to me.
>
> Am I correct in understanding that, when we run ZkSecurityMigrator with
> SASL + SSL, multiple identities will be added to the ACLs?
>
> Thanks,
>
> On Wed, Jan 15, 2020 at 1:19 AM Rajini Sivaram <ra...@gmail.com>
> wrote:
>
>> Hi Ron,
>>
>> Thanks for the detailed explanation, sounds good to me.
>>
>> A few more questions:
>>
>> 1) At the moment, all sensitive broker configs including
>> keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
>> starting up brokers. We are now adding SSL keystore/truststore passwords
>> for ZooKeeper client that cannot be stored in ZooKeeper since you need
>> these to connect to ZK. We should perhaps document that these configs can
>> be encrypted using KIP-421.
>>
>> 2) We can dynamically update keystores and trust stores used by brokers
>> without restarting the broker. Can we support this easily for ZK clients
>> created by the broker, for example by adding our own
>> `zookeeper.ssl.context.supplier.class`?
>>
>> 3) Looks like we are using config names that directly map to ZK configs.
>> Have we considered using equivalent Kafka config names with prefixes,
>> perhaps with inheritance from the non-prefixed names? Not sure if this is a
>> good idea, but perhaps worth documenting in Rejected Alternatives at least.
>>
>>
>>> On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <cm...@apache.org> wrote:
>>>
>>> Hi Ron,
>>>
>>> Thanks for the explanation. I guess thinking about it a little bit more,
>>> we should just add --zk-tls-config-file to all of these commands.
>>>
>>> We will be removing this option (plus ZK in general) from these commands
>>> in the next major release, but ZK is still supported in 2.5, so we should
>>> just do the logical thing. (The exception is ZkSecurityMigrator, which
>>> will stay).
>>>
>>> best,
>>> Colin
>>>
>>>
>>>> On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
>>>> Hi Colin.
>>>>
>>>> <<< It seems like this [--zk-tls-config-file information] could just
>>> appear
>>>> in a configuration file, which all of these tools already accept (I
>>> think)
>>>>
>>>> ZkSecurityMigrator has no such property file facility; adding a
>>>> "--zk-tls-config-file" parameter is exactly for this purpose. If we
>> add
>>>> that to ZkSecurityMigrator then it is trivial to add it to other
>> commands
>>>> (the same code is simply reused; it ends up being just a few extra
>>> lines).
>>>> I do not see any parameter in the other two commands to adjust the ZK
>>>> connection; ConfigCommand accepts a "--command-config" flag, but
>>> according
>>>> to the code "This is used only with --bootstrap-server option for
>>>> describing and altering broker configs."
>>>>
>>>> I do agree there would be no need to add "--zk-tls-config-file" to
>>>> ReassignPartitionsCommand if its direct ZK connectivity is replaced in
>>> time
>>>> for the next release.
>>>>
>>>> ConfigCommand supports the "--bootstrap-server" option and will have
>> its
>>>> direct ZooKeeper access formally deprecated as per KIP-555, but the
>>> special
>>>> use case of bootstrapping a ZooKeeper ensemble with encrypted
>> credentials
>>>> prior to starting Kafka will still exist, so it feels like while
>>>> "--zk-tls-config-file" would never be used except for this use case it
>>>> could probably still be added for this particular situation.
>>>>
>>>> Ron
>>>>
>>>> P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
>>>> sent) did not go through. I am trying to get emails through now to
>> move
>>>> this discussion forward.
>>>>
>>>> On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cm...@apache.org>
>> wrote:
>>>>
>>>>>> On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
>>>>>> Hi everyone. I would like to make the following changes to the
>> KIP.
>>>>>>
>>>>>> MOTIVATION:
>>>>>> Include a statement that it will be difficult in the short term to
>>>>>> deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
>>>>> (which
>>>>>> invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
>>> cluster
>>>>>> with encrypted passwords in Zookeeper is an explicitly-supported
>> use
>>>>> case;
>>>>>> therefore it is in scope to be able to securely configure the CLI
>>> tools
>>>>>> that still leverage non-deprecated direct Zookeeper communication
>>> for TLS
>>>>>> (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
>>>>>> zookeeper-security-migration.sh).
>>>>>
>>>>> Hi Ron,
>>>>>
>>>>> Thanks for the KIP.
>>>>>
>>>>> About deprecations:
>>>>>
>>>>> * zookeeper-security-migration: clearly, deprecating ZK access in
>> this
>>> one
>>>>> would not make sense, since it would defeat the whole point of the
>>> tool :)
>>>>>
>>>>> * kafka-reassign-partitions: ZK access should be deprecated here.
>>> KIP-455
>>>>> implementation has dragged a bit, but this should get done soon.
>>> Certainly
>>>>> before the next release.
>>>>>
>>>>> * kafka-configs: I think ZK access should be deprecated here as well.
>>> I
>>>>> agree there is a super-special bootstrapping case here, but that
>> should
>>>>> have its own tool, not use kafka-configs.
>>>>>
>>>>> I will post a separate KIP for this, though.
>>>>>
>>>>>>
>>>>>> GOALS:
>>>>>> Support the secure configuration of TLS-encrypted communication
>>> between
>>>>>> Zookeeper and:
>>>>>> a) Kafka brokers
>>>>>> b) The three CLI tools mentioned above that still support direct,
>>>>>> non-deprecated communication to Zookeeper
>>>>>> It is explicitly out-of-scope to deprecate any direct Zookeeper
>>>>>> communication in CLI tools as part of this KIP; such work will
>> occur
>>> in
>>>>>> future KIPs instead.
>>>>>>
>>>>>> PUBLIC INTERFACES:
>>>>>> 1) The following new broker configurations will be recognized.
>>>>>> zookeeper.client.secure (default value = false, for backwards
>>>>>> compatibility)
>>>>>> zookeeper.clientCnxnSocket
>>>>>> zookeeper.ssl.keyStore.location
>>>>>> zookeeper.ssl.keyStore.password
>>>>>> zookeeper.ssl.trustStore.location
>>>>>> zookeeper.ssl.trustStore.password
>>>>>> It will be an error for any of the last 5 values to be left
>>> unspecified
>>>>> if
>>>>>> zookeeper.client.secure is explicitly set to true.
>>>>>>
>>>>>> 2) In addition, the kafka.security.authorizer.AclAuthorizer class
>>>>> supports
>>>>>> the ability to connect to a different Zookeeper instance than the
>>> one the
>>>>>> brokers use. We therefore also add the following optional configs,
>>> which
>>>>>> override the corresponding ones from above when present:
>>>>>> authorizer.zookeeper.client.secure
>>>>>> authorizer.zookeeper.clientCnxnSocket
>>>>>> authorizer.zookeeper.ssl.keyStore.location
>>>>>> authorizer.zookeeper.ssl.keyStore.password
>>>>>> authorizer.zookeeper.ssl.trustStore.location
>>>>>> authorizer.zookeeper.ssl.trustStore.password
>>>>>>
>>>>>> 3) The three CLI tools mentioned above will support a new
>>>>> --zk-tls-config-file
>>>>>> <String: Zookeeper TLS configuration file path>" option. The
>>> following
>>>>>> properties will be recognized in that file, and unrecognized
>>> properties
>>>>>> will be ignored to allow the possibility of pointing
>>> zk-tls-config-file
>>>>> at
>>>>>> the broker's config file.
>>>>>> zookeeper.client.secure (default value = false)
>>>>>> zookeeper.clientCnxnSocket
>>>>>> zookeeper.ssl.keyStore.location
>>>>>> zookeeper.ssl.keyStore.password
>>>>>> zookeeper.ssl.trustStore.location
>>>>>> zookeeper.ssl.trustStore.password
>>>>>> It will be an error for any of the last 5 values to be left
>>> unspecified
>>>>> if
>>>>>> zookeeper.client.secure is explicitly set to true.
>>>>>
>>>>> Do we really need a --zk-tls-config-file flag? It seems like this
>>> could
>>>>> just appear in a configuration file, which all of these tools already
>>>>> accept (I think).
>>>>>
>>>>> best,
>>>>> Colin
>>>>>
>>>>>
>>>>>>
>>>>>> Ron
>>>>>>
>>>>>> On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rn...@gmail.com>
>>> wrote:
>>>>>>
>>>>>>> Hi everyone. Let's get this discussion going again now that
>> Kafka
>>> 2.4
>>>>>>> with Zookeeper 3.5.6 is out.
>>>>>>>
>>>>>>> First, regarding the KIP number, the other KIP that was using
>> this
>>>>> number
>>>>>>> moved to KIP 534, so KIP 515 remains the correct number for this
>>>>>>> discussion. I've updated the Kafka Improvement Proposal page to
>>> list
>>>>> this
>>>>>>> KIP in the 515 slot, so we're all set there.
>>>>>>>
>>>>>>> Regarding the actual issues under discussion, I think there are
>>> some
>>>>>>> things we should clarify.
>>>>>>>
>>>>>>> 1) It is possible to use TLS connectivity to Zookeeper from
>> Apache
>>>>> Kafka
>>>>>>> 2.4 -- the problem is that configuration information has to be
>>> passed
>>>>> via
>>>>>>> system properties as "-D" command line options on the java
>>> invocation
>>>>> of
>>>>>>> the broker, and those are not secure (anyone with access to the
>> box
>>>>> can see
>>>>>>> the command line used to invoke the broker); the configuration
>>> includes
>>>>>>> sensitive information (e.g. a keystore password), so we need a
>>> secure
>>>>>>> mechanism for passing the configuration values. I believe the
>> real
>>>>>>> motivation for this KIP is to harden the configuration mechanism
>>> for
>>>>>>> Zookeeper TLS connectivity.
>>>>>>>
>>>>>>> 2) I believe the list of CLI tools that continue to use direct
>>>>> Zookeeper
>>>>>>> connectivity in a non-deprecated fashion is:
>>>>>>> a)
>>> zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
>>>>>>> b)
>>>>>>>
>>>>>
>>> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
>>>>>>> c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
>>>>>>>
>>>>>>> 3) I believe Kafka.admin.ConfigCommand presents a conundrum
>>> because it
>>>>>>> explicitly states in a comment that a supported use case is
>>>>> bootstrapping a
>>>>>>> Kafka cluster with encrypted passwords in Zookeeper (see
>>>>>>>
>>>>>
>>>
>> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
>>>>> ).
>>>>>>> This means it will be especially difficult to deprecate this
>>> particular
>>>>>>> direct Zookeeper connectivity without a different storage
>>> mechanism for
>>>>>>> dynamic configuration values being available (i.e. the
>> self-managed
>>>>> quorum
>>>>>>> referred to in KIP-500).
>>>>>>>
>>>>>>> I think it would be easier and simpler to harden the Zookeeper
>> TLS
>>>>>>> configuration in both Kafka and in CLI tools compared to trying
>> to
>>>>>>> deprecate the direct Zookeeper connectivity in the above 3 CLI
>>> tools --
>>>>>>> especially when ConfigCommand has no obvious short-term path to
>>>>> deprecation.
>>>>>>>
>>>>>>> Regarding how to harden the TLS configuration, adding the
>>> appropriate
>>>>>>> configs to Kafka is an obvious choice for the brokers. Not that
>>> these
>>>>>>> values cannot be dynamically reconfigurable because the values
>>> would be
>>>>>>> required to connect to Zookeeper via TLS in the first place.
>>>>>>>
>>>>>>> Regarding how to harden the TLS configuration for the 3 remaining
>>> CLI
>>>>>>> tools, it would be relatively straightforward to add support for
>> a
>>>>>>> "--zk-tls-config-file <String: Zookeeper TLS configuration system
>>>>>>> properties file path>" option to each one.
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> Ron
>>>>>>>
>>>>>>> On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <
>>> pere.urbon@gmail.com
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> True, the number is duplicated.
>>>>>>>>
>>>>>>>> do you know how can we get the problem fix? I was not aware, I
>>>>> missed,
>>>>>>>> sorry the need to add the KIP to the table.
>>>>>>>>
>>>>>>>> -- Pere
>>>>>>>>
>>>>>>>> Missatge de Jorge Esteban Quilcate Otoya <
>>> quilcate.jorge@gmail.com>
>>>>> del
>>>>>>>> dia
>>>>>>>> dt., 3 de set. 2019 a les 13:43:
>>>>>>>>
>>>>>>>>> Hi Pere,
>>>>>>>>>
>>>>>>>>> Have you add your KIP to the list here
>>>>>>>>>
>>>>>>>>
>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>>>>>>>> ?
>>>>>>>>>
>>>>>>>>> I found the KIP number assigned to another.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
>>>>> pere.urbon@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks for your time Harsha,
>>>>>>>>>> anyone else with comments? looking forward to hearing from
>>> you.
>>>>>>>>>>
>>>>>>>>>> Stupid question: when do you move from discussion to vote?
>>>>>>>>>>
>>>>>>>>>> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
>>> dv., 30
>>>>>>>> d’ag.
>>>>>>>>>> 2019 a les 21:59:
>>>>>>>>>>
>>>>>>>>>>> Thanks Pere. KIP looks good to me.
>>>>>>>>>>> -Harsha
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
>>>>>>>>>> pere.urbon@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Not really,
>>>>>>>>>>>> my idea is to keep the JAAS parameter, so people don't
>> see
>>>>> major
>>>>>>>>>>>> changes. But if you pass a properties file, then this
>> takes
>>>>>>>> precedence
>>>>>>>>>> over
>>>>>>>>>>>> the other, with the idea that you can do sasl as well with
>>> the
>>>>>>>>>> properties
>>>>>>>>>>>> files.
>>>>>>>>>>>>
>>>>>>>>>>>> Makes sense?
>>>>>>>>>>>>
>>>>>>>>>>>> -- Pere
>>>>>>>>>>>>
>>>>>>>>>>>> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
>>> dv.,
>>>>> 30
>>>>>>>>>> d’ag.
>>>>>>>>>>>> 2019 a les 19:00:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Pere,
>>>>>>>>>>>>> Thanks for the KIP. Enabling SSL for zookeeper
>>> for
>>>>> Kafka
>>>>>>>>>> makes
>>>>>>>>>>>>> sense.
>>>>>>>>>>>>> "The changes are planned to be introduced in a compatible
>>> way,
>>>>> by
>>>>>>>>>>>>> keeping the current JAAS variable precedence."
>>>>>>>>>>>>> Can you elaborate a bit here. If the user configures a
>> JAAS
>>>>> file
>>>>>>>> with
>>>>>>>>>>>>> Client section it will take precedence over zookeeper SSL
>>>>> configs?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Harsha
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
>>>>>>>>>> pere.urbon@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> quick question, I saw in another mail that 2.4 release
>> is
>>>>> planned
>>>>>>>> for
>>>>>>>>>>>>>> September. I think it would be really awesome to have
>>> this for
>>>>>>>> this
>>>>>>>>>>>>>> release, do you think we can make it?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -- Pere
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Missatge de Pere Urbón Bayes <pe...@gmail.com> del
>>> dia
>>>>> dj.,
>>>>>>>> 29
>>>>>>>>>>>>>> d’ag. 2019 a les 20:10:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> this is my first KIP for a change in Apache Kafka, so
>> I'm
>>>>> really
>>>>>>>> need
>>>>>>>>>>>>>> to the process. Looking forward to hearing from you and
>>> learn
>>>>> the
>>>>>>>>>> best
>>>>>>>>>>>>>> ropes here.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I would like to propose this KIP-515 to enable the
>>>>>>>> ZookeeperClients
>>>>>>>>>> to
>>>>>>>>>>>>>> take full advantage of the TLS communication in the new
>>>>> Zookeeper
>>>>>>>>>> 3.5.5.
>>>>>>>>>>>>>> Specially interesting it the Zookeeper Security
>> Migration,
>>>>> that
>>>>>>>>>> without
>>>>>>>>>>>>>> this change will not work with TLS, disabling users to
>> use
>>>>> ACLs
>>>>>>>> when
>>>>>>>>>> the
>>>>>>>>>>>>>> Zookeeper cluster use TLS.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> link:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Looking forward to hearing from you on this,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /cheers
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>> http://www.purbon.com
>>>>>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>> http://www.purbon.com
>>>>>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>>>> Software Architect
>>>>>>>>>>>> http://www.purbon.com
>>>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>> Software Architect
>>>>>>>>>> http://www.purbon.com
>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Pere Urbon-Bayes
>>>>>>>> Software Architect
>>>>>>>> http://www.purbon.com
>>>>>>>> https://twitter.com/purbon
>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Manikumar <ma...@gmail.com>.
Hi Ron,
Thanks for the KIP. KIP looks good to me.
Am I correct in understanding that, when we run ZkSecurityMigrator with
SASL + SSL, multiple identities will be added to the ACLs?
Thanks,
On Wed, Jan 15, 2020 at 1:19 AM Rajini Sivaram <ra...@gmail.com>
wrote:
> Hi Ron,
>
> Thanks for the detailed explanation, sounds good to me.
>
> A few more questions:
>
> 1) At the moment, all sensitive broker configs including
> keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
> starting up brokers. We are now adding SSL keystore/truststore passwords
> for ZooKeeper client that cannot be stored in ZooKeeper since you need
> these to connect to ZK. We should perhaps document that these configs can
> be encrypted using KIP-421.
>
> 2) We can dynamically update keystores and trust stores used by brokers
> without restarting the broker. Can we support this easily for ZK clients
> created by the broker, for example by adding our own
> `zookeeper.ssl.context.supplier.class`?
>
> 3) Looks like we are using config names that directly map to ZK configs.
> Have we considered using equivalent Kafka config names with prefixes,
> perhaps with inheritance from the non-prefixed names? Not sure if this is a
> good idea, but perhaps worth documenting in Rejected Alternatives at least.
>
>
> On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <cm...@apache.org> wrote:
>
> > Hi Ron,
> >
> > Thanks for the explanation. I guess thinking about it a little bit more,
> > we should just add --zk-tls-config-file to all of these commands.
> >
> > We will be removing this option (plus ZK in general) from these commands
> > in the next major release, but ZK is still supported in 2.5, so we should
> > just do the logical thing. (The exception is ZkSecurityMigrator, which
> > will stay).
> >
> > best,
> > Colin
> >
> >
> > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > Hi Colin.
> > >
> > > <<< It seems like this [--zk-tls-config-file information] could just
> > appear
> > > in a configuration file, which all of these tools already accept (I
> > think)
> > >
> > > ZkSecurityMigrator has no such property file facility; adding a
> > > "--zk-tls-config-file" parameter is exactly for this purpose. If we
> add
> > > that to ZkSecurityMigrator then it is trivial to add it to other
> commands
> > > (the same code is simply reused; it ends up being just a few extra
> > lines).
> > > I do not see any parameter in the other two commands to adjust the ZK
> > > connection; ConfigCommand accepts a "--command-config" flag, but
> > according
> > > to the code "This is used only with --bootstrap-server option for
> > > describing and altering broker configs."
> > >
> > > I do agree there would be no need to add "--zk-tls-config-file" to
> > > ReassignPartitionsCommand if its direct ZK connectivity is replaced in
> > time
> > > for the next release.
> > >
> > > ConfigCommand supports the "--bootstrap-server" option and will have
> its
> > > direct ZooKeeper access formally deprecated as per KIP-555, but the
> > special
> > > use case of bootstrapping a ZooKeeper ensemble with encrypted
> credentials
> > > prior to starting Kafka will still exist, so it feels like while
> > > "--zk-tls-config-file" would never be used except for this use case it
> > > could probably still be added for this particular situation.
> > >
> > > Ron
> > >
> > > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> > > sent) did not go through. I am trying to get emails through now to
> move
> > > this discussion forward.
> > >
> > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cm...@apache.org>
> wrote:
> > >
> > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > > Hi everyone. I would like to make the following changes to the
> KIP.
> > > > >
> > > > > MOTIVATION:
> > > > > Include a statement that it will be difficult in the short term to
> > > > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > > > (which
> > > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
> > cluster
> > > > > with encrypted passwords in Zookeeper is an explicitly-supported
> use
> > > > case;
> > > > > therefore it is in scope to be able to securely configure the CLI
> > tools
> > > > > that still leverage non-deprecated direct Zookeeper communication
> > for TLS
> > > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > > > zookeeper-security-migration.sh).
> > > >
> > > > Hi Ron,
> > > >
> > > > Thanks for the KIP.
> > > >
> > > > About deprecations:
> > > >
> > > > * zookeeper-security-migration: clearly, deprecating ZK access in
> this
> > one
> > > > would not make sense, since it would defeat the whole point of the
> > tool :)
> > > >
> > > > * kafka-reassign-partitions: ZK access should be deprecated here.
> > KIP-455
> > > > implementation has dragged a bit, but this should get done soon.
> > Certainly
> > > > before the next release.
> > > >
> > > > * kafka-configs: I think ZK access should be deprecated here as well.
> > I
> > > > agree there is a super-special bootstrapping case here, but that
> should
> > > > have its own tool, not use kafka-configs.
> > > >
> > > > I will post a separate KIP for this, though.
> > > >
> > > > >
> > > > > GOALS:
> > > > > Support the secure configuration of TLS-encrypted communication
> > between
> > > > > Zookeeper and:
> > > > > a) Kafka brokers
> > > > > b) The three CLI tools mentioned above that still support direct,
> > > > > non-deprecated communication to Zookeeper
> > > > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > > > communication in CLI tools as part of this KIP; such work will
> occur
> > in
> > > > > future KIPs instead.
> > > > >
> > > > > PUBLIC INTERFACES:
> > > > > 1) The following new broker configurations will be recognized.
> > > > > zookeeper.client.secure (default value = false, for backwards
> > > > > compatibility)
> > > > > zookeeper.clientCnxnSocket
> > > > > zookeeper.ssl.keyStore.location
> > > > > zookeeper.ssl.keyStore.password
> > > > > zookeeper.ssl.trustStore.location
> > > > > zookeeper.ssl.trustStore.password
> > > > > It will be an error for any of the last 5 values to be left
> > unspecified
> > > > if
> > > > > zookeeper.client.secure is explicitly set to true.
> > > > >
> > > > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> > > > supports
> > > > > the ability to connect to a different Zookeeper instance than the
> > one the
> > > > > brokers use. We therefore also add the following optional configs,
> > which
> > > > > override the corresponding ones from above when present:
> > > > > authorizer.zookeeper.client.secure
> > > > > authorizer.zookeeper.clientCnxnSocket
> > > > > authorizer.zookeeper.ssl.keyStore.location
> > > > > authorizer.zookeeper.ssl.keyStore.password
> > > > > authorizer.zookeeper.ssl.trustStore.location
> > > > > authorizer.zookeeper.ssl.trustStore.password
> > > > >
> > > > > 3) The three CLI tools mentioned above will support a new
> > > > --zk-tls-config-file
> > > > > <String: Zookeeper TLS configuration file path>" option. The
> > following
> > > > > properties will be recognized in that file, and unrecognized
> > properties
> > > > > will be ignored to allow the possibility of pointing
> > zk-tls-config-file
> > > > at
> > > > > the broker's config file.
> > > > > zookeeper.client.secure (default value = false)
> > > > > zookeeper.clientCnxnSocket
> > > > > zookeeper.ssl.keyStore.location
> > > > > zookeeper.ssl.keyStore.password
> > > > > zookeeper.ssl.trustStore.location
> > > > > zookeeper.ssl.trustStore.password
> > > > > It will be an error for any of the last 5 values to be left
> > unspecified
> > > > if
> > > > > zookeeper.client.secure is explicitly set to true.
> > > >
> > > > Do we really need a --zk-tls-config-file flag? It seems like this
> > could
> > > > just appear in a configuration file, which all of these tools already
> > > > accept (I think).
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > >
> > > > > Ron
> > > > >
> > > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rn...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hi everyone. Let's get this discussion going again now that
> Kafka
> > 2.4
> > > > > > with Zookeeper 3.5.6 is out.
> > > > > >
> > > > > > First, regarding the KIP number, the other KIP that was using
> this
> > > > number
> > > > > > moved to KIP 534, so KIP 515 remains the correct number for this
> > > > > > discussion. I've updated the Kafka Improvement Proposal page to
> > list
> > > > this
> > > > > > KIP in the 515 slot, so we're all set there.
> > > > > >
> > > > > > Regarding the actual issues under discussion, I think there are
> > some
> > > > > > things we should clarify.
> > > > > >
> > > > > > 1) It is possible to use TLS connectivity to Zookeeper from
> Apache
> > > > Kafka
> > > > > > 2.4 -- the problem is that configuration information has to be
> > passed
> > > > via
> > > > > > system properties as "-D" command line options on the java
> > invocation
> > > > of
> > > > > > the broker, and those are not secure (anyone with access to the
> box
> > > > can see
> > > > > > the command line used to invoke the broker); the configuration
> > includes
> > > > > > sensitive information (e.g. a keystore password), so we need a
> > secure
> > > > > > mechanism for passing the configuration values. I believe the
> real
> > > > > > motivation for this KIP is to harden the configuration mechanism
> > for
> > > > > > Zookeeper TLS connectivity.
> > > > > >
> > > > > > 2) I believe the list of CLI tools that continue to use direct
> > > > Zookeeper
> > > > > > connectivity in a non-deprecated fashion is:
> > > > > > a)
> > zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > > > b)
> > > > > >
> > > >
> > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > > > > >
> > > > > > 3) I believe Kafka.admin.ConfigCommand presents a conundrum
> > because it
> > > > > > explicitly states in a comment that a supported use case is
> > > > bootstrapping a
> > > > > > Kafka cluster with encrypted passwords in Zookeeper (see
> > > > > >
> > > >
> >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > > > ).
> > > > > > This means it will be especially difficult to deprecate this
> > particular
> > > > > > direct Zookeeper connectivity without a different storage
> > mechanism for
> > > > > > dynamic configuration values being available (i.e. the
> self-managed
> > > > quorum
> > > > > > referred to in KIP-500).
> > > > > >
> > > > > > I think it would be easier and simpler to harden the Zookeeper
> TLS
> > > > > > configuration in both Kafka and in CLI tools compared to trying
> to
> > > > > > deprecate the direct Zookeeper connectivity in the above 3 CLI
> > tools --
> > > > > > especially when ConfigCommand has no obvious short-term path to
> > > > deprecation.
> > > > > >
> > > > > > Regarding how to harden the TLS configuration, adding the
> > appropriate
> > > > > > configs to Kafka is an obvious choice for the brokers. Not that
> > these
> > > > > > values cannot be dynamically reconfigurable because the values
> > would be
> > > > > > required to connect to Zookeeper via TLS in the first place.
> > > > > >
> > > > > > Regarding how to harden the TLS configuration for the 3 remaining
> > CLI
> > > > > > tools, it would be relatively straightforward to add support for
> a
> > > > > > "--zk-tls-config-file <String: Zookeeper TLS configuration system
> > > > > > properties file path>" option to each one.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > Ron
> > > > > >
> > > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <
> > pere.urbon@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> True, the number is duplicated.
> > > > > >>
> > > > > >> do you know how can we get the problem fix? I was not aware, I
> > > > missed,
> > > > > >> sorry the need to add the KIP to the table.
> > > > > >>
> > > > > >> -- Pere
> > > > > >>
> > > > > >> Missatge de Jorge Esteban Quilcate Otoya <
> > quilcate.jorge@gmail.com>
> > > > del
> > > > > >> dia
> > > > > >> dt., 3 de set. 2019 a les 13:43:
> > > > > >>
> > > > > >> > Hi Pere,
> > > > > >> >
> > > > > >> > Have you add your KIP to the list here
> > > > > >> >
> > > > > >>
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > >> ?
> > > > > >> >
> > > > > >> > I found the KIP number assigned to another.
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> > > > pere.urbon@gmail.com>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> >> Thanks for your time Harsha,
> > > > > >> >> anyone else with comments? looking forward to hearing from
> > you.
> > > > > >> >>
> > > > > >> >> Stupid question: when do you move from discussion to vote?
> > > > > >> >>
> > > > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> > dv., 30
> > > > > >> d’ag.
> > > > > >> >> 2019 a les 21:59:
> > > > > >> >>
> > > > > >> >> > Thanks Pere. KIP looks good to me.
> > > > > >> >> > -Harsha
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> > > > > >> >> pere.urbon@gmail.com>
> > > > > >> >> > wrote:
> > > > > >> >> >
> > > > > >> >> >> Not really,
> > > > > >> >> >> my idea is to keep the JAAS parameter, so people don't
> see
> > > > major
> > > > > >> >> >> changes. But if you pass a properties file, then this
> takes
> > > > > >> precedence
> > > > > >> >> over
> > > > > >> >> >> the other, with the idea that you can do sasl as well with
> > the
> > > > > >> >> properties
> > > > > >> >> >> files.
> > > > > >> >> >>
> > > > > >> >> >> Makes sense?
> > > > > >> >> >>
> > > > > >> >> >> -- Pere
> > > > > >> >> >>
> > > > > >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> > dv.,
> > > > 30
> > > > > >> >> d’ag.
> > > > > >> >> >> 2019 a les 19:00:
> > > > > >> >> >>
> > > > > >> >> >>> Hi Pere,
> > > > > >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper
> > for
> > > > Kafka
> > > > > >> >> makes
> > > > > >> >> >>> sense.
> > > > > >> >> >>> "The changes are planned to be introduced in a compatible
> > way,
> > > > by
> > > > > >> >> >>> keeping the current JAAS variable precedence."
> > > > > >> >> >>> Can you elaborate a bit here. If the user configures a
> JAAS
> > > > file
> > > > > >> with
> > > > > >> >> >>> Client section it will take precedence over zookeeper SSL
> > > > configs?
> > > > > >> >> >>>
> > > > > >> >> >>> Thanks,
> > > > > >> >> >>> Harsha
> > > > > >> >> >>>
> > > > > >> >> >>>
> > > > > >> >> >>>
> > > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> > > > > >> >> pere.urbon@gmail.com>
> > > > > >> >> >>> wrote:
> > > > > >> >> >>>
> > > > > >> >> >>>> Hi,
> > > > > >> >> >>>> quick question, I saw in another mail that 2.4 release
> is
> > > > planned
> > > > > >> for
> > > > > >> >> >>>> September. I think it would be really awesome to have
> > this for
> > > > > >> this
> > > > > >> >> >>>> release, do you think we can make it?
> > > > > >> >> >>>>
> > > > > >> >> >>>> -- Pere
> > > > > >> >> >>>>
> > > > > >> >> >>>> Missatge de Pere Urbón Bayes <pe...@gmail.com> del
> > dia
> > > > dj.,
> > > > > >> 29
> > > > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > > > >> >> >>>>
> > > > > >> >> >>>> Hi,
> > > > > >> >> >>>> this is my first KIP for a change in Apache Kafka, so
> I'm
> > > > really
> > > > > >> need
> > > > > >> >> >>>> to the process. Looking forward to hearing from you and
> > learn
> > > > the
> > > > > >> >> best
> > > > > >> >> >>>> ropes here.
> > > > > >> >> >>>>
> > > > > >> >> >>>> I would like to propose this KIP-515 to enable the
> > > > > >> ZookeeperClients
> > > > > >> >> to
> > > > > >> >> >>>> take full advantage of the TLS communication in the new
> > > > Zookeeper
> > > > > >> >> 3.5.5.
> > > > > >> >> >>>> Specially interesting it the Zookeeper Security
> Migration,
> > > > that
> > > > > >> >> without
> > > > > >> >> >>>> this change will not work with TLS, disabling users to
> use
> > > > ACLs
> > > > > >> when
> > > > > >> >> the
> > > > > >> >> >>>> Zookeeper cluster use TLS.
> > > > > >> >> >>>>
> > > > > >> >> >>>> link:
> > > > > >> >> >>>>
> > > > > >> >> >>>>
> > > > > >> >>
> > > > > >>
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > > > >> >> >>>>
> > > > > >> >> >>>> Looking forward to hearing from you on this,
> > > > > >> >> >>>>
> > > > > >> >> >>>> /cheers
> > > > > >> >> >>>>
> > > > > >> >> >>>> --
> > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > >> >> >>>> Software Architect
> > > > > >> >> >>>> http://www.purbon.com
> > > > > >> >> >>>> https://twitter.com/purbon
> > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > >> >> >>>>
> > > > > >> >> >>>> --
> > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > >> >> >>>> Software Architect
> > > > > >> >> >>>> http://www.purbon.com
> > > > > >> >> >>>> https://twitter.com/purbon
> > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > >> >> >>>>
> > > > > >> >> >>>
> > > > > >> >> >>>
> > > > > >> >> >>
> > > > > >> >> >> --
> > > > > >> >> >> Pere Urbon-Bayes
> > > > > >> >> >> Software Architect
> > > > > >> >> >> http://www.purbon.com
> > > > > >> >> >> https://twitter.com/purbon
> > > > > >> >> >> https://www.linkedin.com/in/purbon/
> > > > > >> >> >>
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >>
> > > > > >> >> --
> > > > > >> >> Pere Urbon-Bayes
> > > > > >> >> Software Architect
> > > > > >> >> http://www.purbon.com
> > > > > >> >> https://twitter.com/purbon
> > > > > >> >> https://www.linkedin.com/in/purbon/
> > > > > >> >>
> > > > > >> >
> > > > > >>
> > > > > >> --
> > > > > >> Pere Urbon-Bayes
> > > > > >> Software Architect
> > > > > >> http://www.purbon.com
> > > > > >> https://twitter.com/purbon
> > > > > >> https://www.linkedin.com/in/purbon/
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Ron Dagostino <rn...@gmail.com>.
Hi Rajini. Thanks for the feedback. I have now come around to the
idea of abandoning the ZooKeeper camelCase style in favor of being
consistent with the existing Kafka broker configs. Here's what I've
done:
#1) ZK style vs. Kafka style
I've updated the KIP to convert all of the following configs from ZK
camelCase style to Kafka style:
zookeeper.ssl.keyStore.location => zookeeper.ssl.keystore.location
zookeeper.ssl.keyStore.password => zookeeper.ssl.keystore.password
zookeeper.ssl.keyStore.type => zookeeper.ssl.keystore.type
zookeeper.ssl.trustStore.location => zookeeper.ssl.truststore.location
zookeeper.ssl.trustStore.password => zookeeper.ssl.truststore.password
zookeeper.ssl.trustStore.type => zookeeper.ssl.truststore.type
zookeeper.ssl.ciphersuites => zookeeper.ssl.cipher.suites
zookeeper.ssl.enabledProtocols => zookeeper.ssl.enabled.protocols
I've also renamed the following as you suggested (all similar broker
configs use the suffix ".enable" instead of ".enabled"):
zookeeper.client.secure => zookeeper.ssl.client.enable
Finally, to be consistent with Kafka's ".enable" suffix convention,
I've adjusted these two (we didn't discuss this, but it makes sense to
me; I'll assume it's fine if there are no further comments about it):
zookeeper.ssl.crl => zookeeper.ssl.crl.enable
zookeeper.ssl.ocsp => zookeeper.ssl.ocsp.enable
UNRESOLVED ITEM #1:
Regarding zookeeper.ssl.hostnameVerification and
ssl.endpoint.identification.algorithm, I'm not sure what the right
solution is here. "zookeeper.ssl.endpoint.identification.algorithm"
seems weird to me -- we aren't trying to distinguish between HTTPS and
LDAPS (which are the two valid endpoint identification algorithms
according to https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html).
Does Kafka provide the ability to disable hostname checking by
clearing this property from the default value of (case-insensitively)
HTTPS? If so then maybe there is a doc bug here to make that
explicit. So maybe we can explicitly clarify if that is how it works
(and I would submit a PR to amend the docs to state it if so); then we
can perhaps decide what to do here. I would think the two choices are
zookeeper.ssl.endpoint.identification.algorithm (string with the
default https or blank) or zookeeper.ssl.hostname.verify.enable (no
camel-case, boolean with the default being true).
#2) Config Inheritance
UNRESOLVED ITEM #2:
I think what you are saying is that for the following 9 configs, if
the "zookeeper." prefixed version is not provided then the
non-prefixed version should be used instead. Can you confirm? The
default values for the last two are an issue, but see #3 below. I
haven't updated the KIP yet -- I will wait for confirmation that these
9 configs are the ones to be inherited and the default value issue is
resolved.
[zookeeper.]ssl.keystore.{location,password,type}
[zookeeper.]ssl.truststore.{location,password,type}
[zookeeper.]ssl.cipher.suites
[zookeeper.]ssl.protocol
[zookeeper.]ssl.enabled.protocols
Note: The AclAuthorizer configs on the broker side do not use
inheritance at all -- if AclAuthorizer is pointed at a separate ZK
quorum then the configs must specify everything.
#3) Default values
UNRESOLVED ITEM #3:
I agree it is probably best to decouple disabling older TLS protocols
from making TLSv1.3 as the default if both can't be done --
independent of anything else, I think think in this case it's best to
tighten security a little bit now and then a bit more later on than to
wait until both changes can happen at the same time. However, as you
point out, if the defaults for Kafka's ssl.protocol and
ssl.enabled.protocols were to be amended to TLSv1.2 then it would also
allow the default values for ZK and Kafka to be the same for
ssl.protocol/ssl.enabled.protocols, and then inheritance works well
for these configs -- we probably shouldn't do inheritance for them
otherwise. (BTW, just for completeness, I will state that the
allowable values are defined in the "SSLContext Algorithms" section of
https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html).
#4) System Properties
It is not a problem to start with ZK-style system properties when the
user explicitly defines them. The evaluation algorithm will be:
1) Start with the ZK-style system property if it is defined
2) Use the explicitly-prefixed "zookeeper." property value if it is
defined (overriding any system property value)
3) Inherit any non-prefixed Kafka config value if #1 and #2 did not
result in a value
4) Use the default value if there is still no value
I've updated the KIP accordingly to reflect this evaluation order..
Ron
On Thu, Jan 16, 2020 at 6:06 AM Rajini Sivaram <ra...@gmail.com> wrote:
>
> Forgot to mention:
>
> Overriding system property with a differently named Kafka configuration
> option is something we already do for JAAS configs - we default to System
> property `java.security.auth.login.config` if `sasl.jaas.config` is not
> configured.
>
> On Thu, Jan 16, 2020 at 10:58 AM Rajini Sivaram <ra...@gmail.com>
> wrote:
>
> > Hi Ron/Colin,
> >
> > Thanks for the responses. I think we need to consider these cases:
> >
> > 1) The recommended approach: This is the one we document. Configs
> > consistent with Kafka make sense here. It seems a lot more confusing to
> > configure `ssl.cipher.suites`, `ssl.enabled.protocols`, `
> > zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols` with
> > multiple naming inconsistencies in server.properties. For users who have
> > never used ZK system properties, this is clearly simpler. Another config
> > to consider is `ssl.endpoint.identification.algorithm` which does the
> > same thing as `zookeeper.ssl.hostnameVerification` - if we are staying
> > consistent, we should use the Kafka format for this too. I would also
> > consider using something like `zookeeper.client.ssl.enabled` instead of `
> > zookeeper.client.secure` since it is difficult to tell what `secure`
> > means when you have options to configure Kerberos, ACLs and/or SSL.
> >
> > 2) Inheritance using prefixed configs as we do for other prefixed configs
> > in Kafka: I think it may be useful for configs like `ssl.protocol` and `
> > ssl.cipher.suites`.
> >
> > 3) Default values: Do we want to use Kafka defaults? The inconsistency we
> > have is `ssl.protocol/ssl.enabled.protocols` since we still have insecure
> > protocols enabled for Kafka. We have KIP-553 (
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956)
> > open to disable these. It is currently blocked since it also talks about
> > enabling TLSv1.3 by default and we haven't done sufficient testing for
> > that. Since it will be good to disable the older insecure protocols anyway,
> > perhaps we could do that without enabling TLSv1.3 by default in 2.5?
> >
> > 4) Backward compatibility for deployments which are using system
> > properties. We shouldn't override system properties with Kafka defaults or
> > inherited values. But we do want to override if user configures properties
> > explicitly. In the current approach, this was simpler since we just had to
> > set the configured values. But if we decide to inherit Kafka configs, then
> > we will need to check system properties and update accordingly. But that
> > should be doable?
> >
> > On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe <cm...@apache.org> wrote:
> >
> >> On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote:
> >> > Hi Colin. Two things come to mind with respect to ZooKeeper camelCase
> >> > style vs Kafka-style config names for ZooKeeper. First, I think it
> >> > would be desirable for the client configs and broker configs to be
> >> > interoperable. For example, it feels like it would be convenient to
> >> > be able to pass the broker's config file to the ZK Security Migrator
> >> > tool. I think whatever we use (ZooKeeper camelCase style or Kafka
> >> > style), I think we should use the same for both.
> >> >
> >> > The second thing to think about is the fact that ZooKeeper
> >> > configuration inherently uses Java system properties as a first pass.
> >> > So while we might switch to Kafka-style configs in the config file
> >> > (e.g. zookeeper.ssl.trust.store.location), the
> >> > org.apache.zookeeper.client.ZKClientConfig class (which is how TLS
> >> > configs are set) is going to look for the camelCase
> >> > zookeeper.ssl.trustStore.location system property and use any value
> >> > there. I think this could be very confusing for people. Granted, we
> >> > are doing this so that people don't have to use system properties, but
> >> > there is no way to turn off the use of system properties, so I think
> >> > there would be pretty substantial potential for confusion.
> >>
> >> I don't think we document or recommend that anyone use system properties
> >> to configure Zookeeper usage within Kafka. And I think the reason is
> >> exactly the issue you pointed out -- it wouldn't be consistent with our
> >> other configurations.
> >>
> >> best,
> >> Colin
> >>
> >> >
> >> > One idea is to migrate the system properties -- i.e. look to see if
> >> > zookeeper.ssl.trustStore.location is set and if it is clear out the
> >> > value there and move it under the zookeeper.ssl.trust.store.location
> >> > system property. (I said it was an idea -- not necessarily a good one
> >> > :-)
> >> >
> >> > Ron
> >> >
> >> > On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe <cm...@apache.org>
> >> wrote:
> >> > >
> >> > > On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
> >> > > > Thanks Colin and Rajini.
> >> > > >
> >> > > > I've updated the KIP to say that KIP-421 functionality is available
> >> to
> >> > > > encrypt sensitive configs like the ZK key store and trust store
> >> > > > passwords. (I've also made it clear that the configs are not
> >> > > > dynamically reconfigurable since dynamic values are stored in ZK and
> >> > > > the data is needed to get to ZK in the first place). Colin, let me
> >> > > > know if you think anything else needs to be said/done about it -- I
> >> > > > wasn't sure if your comment above implies that there are additional
> >> > > > direct actions that we need to take aside from this.
> >> > > >
> >> > > > I agree that making the brokers' ZooKeeper clients dynamic with
> >> > > > respect to key and trust stores (e.g. via
> >> > > > zookeeper.ssl.context.supplier.class) may increase the risk that
> >> this
> >> > > > KIP may not land in time for 2.5.
> >> > > >
> >> > > > Regarding the config names for ZK being different than the ZooKeeper
> >> > > > configs (e.g. camel-case keyStore/trustStore instead of
> >> > > > keystore/truststore, ciphersuites instead of cipher.suites,
> >> > > > enabledProtocols instead of enabled.protocols), I agree it is an
> >> > > > issue. I tried to mitigate it by putting warning text in the config
> >> > > > docs for these. Regarding configuration inheritance, I think what
> >> you
> >> > > > are saying is that for any configs where the broker has a clear
> >> > > > semantic equivalent, the broker could fall back to the broker value
> >> if
> >> > > > the ZK value is not given. The potential list is:
> >> > > >
> >> > > > zookeeper.ssl.keyStore.location => ssl.keystore.location
> >> > > > zookeeper.ssl.keyStore.password => ssl.keystore.password
> >> > > > zookeeper.ssl.keyStore.type => ssl.keystore.type
> >> > > > zookeeper.ssl.trustStore.location => ssl.truststore.location
> >> > > > zookeeper.ssl.trustStore.password => ssl.truststore.password
> >> > > > zookeeper.ssl.trustStore.type => ssl.truststore.type
> >> > > > zookeeper.ssl.ciphersuites => ssl.cipher.suites
> >> > > >
> >> > > > Note that there are two configs that look the same based on their
> >> key
> >> > > > names but whose allowable values differ:
> >> > > >
> >> > > > zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default:
> >> TLS)
> >> > > > zookeeper.ssl.enabledProtocols(Default: value of protocol property)
> >> =
> >> > > > ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
> >> > > >
> >> > > > Not sure what the right answer is, but hopefully this detail will
> >> help
> >> > > > get us there.
> >> > > >
> >> > > > Ron
> >> > >
> >> > > I think, on balance, I agree with Rajini that it would be nice to
> >> make the configs look more like Kafka configs. We don't have camel-case in
> >> configuration keys, so we should avoid it here.
> >> > >
> >> > > best,
> >> > > Colin
> >> > >
> >> > > >
> >> > > > On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <cm...@apache.org>
> >> wrote:
> >> > > > >
> >> > > > > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> >> > > > > > Hi Ron,
> >> > > > > >
> >> > > > > > Thanks for the detailed explanation, sounds good to me.
> >> > > > > >
> >> > > > > > A few more questions:
> >> > > > > >
> >> > > > > > 1) At the moment, all sensitive broker configs including
> >> > > > > > keystore/truststore passwords can be stored encrypted in
> >> ZooKeeper prior to
> >> > > > > > starting up brokers. We are now adding SSL keystore/truststore
> >> passwords
> >> > > > > > for ZooKeeper client that cannot be stored in ZooKeeper since
> >> you need
> >> > > > > > these to connect to ZK. We should perhaps document that these
> >> configs can
> >> > > > > > be encrypted using KIP-421.
> >> > > > >
> >> > > > > That's a good point. Previously, we didn't have ZK on-the-wire
> >> security support. However, now that we do, sending sensitive keystore and
> >> truststore passwords to ZK in cleartext should be deprecated in favor of
> >> using KIP-421.
> >> > > > >
> >> > > > > >
> >> > > > > > 2) We can dynamically update keystores and trust stores used by
> >> brokers
> >> > > > > > without restarting the broker. Can we support this easily for
> >> ZK clients
> >> > > > > > created by the broker, for example by adding our own
> >> > > > > > `zookeeper.ssl.context.supplier.class`?
> >> > > > > >
> >> > > > >
> >> > > > > Hmm. That might be better to consider in a follow-up, since the
> >> deadline for 2.5 KIPs is coming up?
> >> > > > >
> >> > > > > best,
> >> > > > > Colin
> >> > > > >
> >> > > > > > 3) Looks like we are using config names that directly map to ZK
> >> configs.
> >> > > > > > Have we considered using equivalent Kafka config names with
> >> prefixes,
> >> > > > > > perhaps with inheritance from the non-prefixed names? Not sure
> >> if this is a
> >> > > > > > good idea, but perhaps worth documenting in Rejected
> >> Alternatives at least.
> >> > > > > >
> >> > > > > >
> >> > > > > > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <
> >> cmccabe@apache.org> wrote:
> >> > > > > >
> >> > > > > > > Hi Ron,
> >> > > > > > >
> >> > > > > > > Thanks for the explanation. I guess thinking about it a
> >> little bit more,
> >> > > > > > > we should just add --zk-tls-config-file to all of these
> >> commands.
> >> > > > > > >
> >> > > > > > > We will be removing this option (plus ZK in general) from
> >> these commands
> >> > > > > > > in the next major release, but ZK is still supported in 2.5,
> >> so we should
> >> > > > > > > just do the logical thing. (The exception is
> >> ZkSecurityMigrator, which
> >> > > > > > > will stay).
> >> > > > > > >
> >> > > > > > > best,
> >> > > > > > > Colin
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> >> > > > > > > > Hi Colin.
> >> > > > > > > >
> >> > > > > > > > <<< It seems like this [--zk-tls-config-file information]
> >> could just
> >> > > > > > > appear
> >> > > > > > > > in a configuration file, which all of these tools already
> >> accept (I
> >> > > > > > > think)
> >> > > > > > > >
> >> > > > > > > > ZkSecurityMigrator has no such property file facility;
> >> adding a
> >> > > > > > > > "--zk-tls-config-file" parameter is exactly for this
> >> purpose. If we add
> >> > > > > > > > that to ZkSecurityMigrator then it is trivial to add it to
> >> other commands
> >> > > > > > > > (the same code is simply reused; it ends up being just a
> >> few extra
> >> > > > > > > lines).
> >> > > > > > > > I do not see any parameter in the other two commands to
> >> adjust the ZK
> >> > > > > > > > connection; ConfigCommand accepts a "--command-config"
> >> flag, but
> >> > > > > > > according
> >> > > > > > > > to the code "This is used only with --bootstrap-server
> >> option for
> >> > > > > > > > describing and altering broker configs."
> >> > > > > > > >
> >> > > > > > > > I do agree there would be no need to add
> >> "--zk-tls-config-file" to
> >> > > > > > > > ReassignPartitionsCommand if its direct ZK connectivity is
> >> replaced in
> >> > > > > > > time
> >> > > > > > > > for the next release.
> >> > > > > > > >
> >> > > > > > > > ConfigCommand supports the "--bootstrap-server" option and
> >> will have its
> >> > > > > > > > direct ZooKeeper access formally deprecated as per KIP-555,
> >> but the
> >> > > > > > > special
> >> > > > > > > > use case of bootstrapping a ZooKeeper ensemble with
> >> encrypted credentials
> >> > > > > > > > prior to starting Kafka will still exist, so it feels like
> >> while
> >> > > > > > > > "--zk-tls-config-file" would never be used except for this
> >> use case it
> >> > > > > > > > could probably still be added for this particular situation.
> >> > > > > > > >
> >> > > > > > > > Ron
> >> > > > > > > >
> >> > > > > > > > P.S. I responded on 1/6 but I just discovered that e, ail
> >> (and 3 more I
> >> > > > > > > > sent) did not go through. I am trying to get emails
> >> through now to move
> >> > > > > > > > this discussion forward.
> >> > > > > > > >
> >> > > > > > > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <
> >> cmccabe@apache.org> wrote:
> >> > > > > > > >
> >> > > > > > > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> >> > > > > > > > > > Hi everyone. I would like to make the following
> >> changes to the KIP.
> >> > > > > > > > > >
> >> > > > > > > > > > MOTIVATION:
> >> > > > > > > > > > Include a statement that it will be difficult in the
> >> short term to
> >> > > > > > > > > > deprecate direct Zookeeper communication in
> >> kafka-configs.{sh, bat}
> >> > > > > > > > > (which
> >> > > > > > > > > > invoke kafka.admin.ConfigCommand) because bootstrapping
> >> a Kafka
> >> > > > > > > cluster
> >> > > > > > > > > > with encrypted passwords in Zookeeper is an
> >> explicitly-supported use
> >> > > > > > > > > case;
> >> > > > > > > > > > therefore it is in scope to be able to securely
> >> configure the CLI
> >> > > > > > > tools
> >> > > > > > > > > > that still leverage non-deprecated direct Zookeeper
> >> communication
> >> > > > > > > for TLS
> >> > > > > > > > > > (the other 2 tools are kafka-reassign-partitions.{sh,
> >> bat} and
> >> > > > > > > > > > zookeeper-security-migration.sh).
> >> > > > > > > > >
> >> > > > > > > > > Hi Ron,
> >> > > > > > > > >
> >> > > > > > > > > Thanks for the KIP.
> >> > > > > > > > >
> >> > > > > > > > > About deprecations:
> >> > > > > > > > >
> >> > > > > > > > > * zookeeper-security-migration: clearly, deprecating ZK
> >> access in this
> >> > > > > > > one
> >> > > > > > > > > would not make sense, since it would defeat the whole
> >> point of the
> >> > > > > > > tool :)
> >> > > > > > > > >
> >> > > > > > > > > * kafka-reassign-partitions: ZK access should be
> >> deprecated here.
> >> > > > > > > KIP-455
> >> > > > > > > > > implementation has dragged a bit, but this should get
> >> done soon.
> >> > > > > > > Certainly
> >> > > > > > > > > before the next release.
> >> > > > > > > > >
> >> > > > > > > > > * kafka-configs: I think ZK access should be deprecated
> >> here as well.
> >> > > > > > > I
> >> > > > > > > > > agree there is a super-special bootstrapping case here,
> >> but that should
> >> > > > > > > > > have its own tool, not use kafka-configs.
> >> > > > > > > > >
> >> > > > > > > > > I will post a separate KIP for this, though.
> >> > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > GOALS:
> >> > > > > > > > > > Support the secure configuration of TLS-encrypted
> >> communication
> >> > > > > > > between
> >> > > > > > > > > > Zookeeper and:
> >> > > > > > > > > > a) Kafka brokers
> >> > > > > > > > > > b) The three CLI tools mentioned above that still
> >> support direct,
> >> > > > > > > > > > non-deprecated communication to Zookeeper
> >> > > > > > > > > > It is explicitly out-of-scope to deprecate any direct
> >> Zookeeper
> >> > > > > > > > > > communication in CLI tools as part of this KIP; such
> >> work will occur
> >> > > > > > > in
> >> > > > > > > > > > future KIPs instead.
> >> > > > > > > > > >
> >> > > > > > > > > > PUBLIC INTERFACES:
> >> > > > > > > > > > 1) The following new broker configurations will be
> >> recognized.
> >> > > > > > > > > > zookeeper.client.secure (default value = false, for
> >> backwards
> >> > > > > > > > > > compatibility)
> >> > > > > > > > > > zookeeper.clientCnxnSocket
> >> > > > > > > > > > zookeeper.ssl.keyStore.location
> >> > > > > > > > > > zookeeper.ssl.keyStore.password
> >> > > > > > > > > > zookeeper.ssl.trustStore.location
> >> > > > > > > > > > zookeeper.ssl.trustStore.password
> >> > > > > > > > > > It will be an error for any of the last 5 values to be
> >> left
> >> > > > > > > unspecified
> >> > > > > > > > > if
> >> > > > > > > > > > zookeeper.client.secure is explicitly set to true.
> >> > > > > > > > > >
> >> > > > > > > > > > 2) In addition, the
> >> kafka.security.authorizer.AclAuthorizer class
> >> > > > > > > > > supports
> >> > > > > > > > > > the ability to connect to a different Zookeeper
> >> instance than the
> >> > > > > > > one the
> >> > > > > > > > > > brokers use. We therefore also add the following
> >> optional configs,
> >> > > > > > > which
> >> > > > > > > > > > override the corresponding ones from above when present:
> >> > > > > > > > > > authorizer.zookeeper.client.secure
> >> > > > > > > > > > authorizer.zookeeper.clientCnxnSocket
> >> > > > > > > > > > authorizer.zookeeper.ssl.keyStore.location
> >> > > > > > > > > > authorizer.zookeeper.ssl.keyStore.password
> >> > > > > > > > > > authorizer.zookeeper.ssl.trustStore.location
> >> > > > > > > > > > authorizer.zookeeper.ssl.trustStore.password
> >> > > > > > > > > >
> >> > > > > > > > > > 3) The three CLI tools mentioned above will support a
> >> new
> >> > > > > > > > > --zk-tls-config-file
> >> > > > > > > > > > <String: Zookeeper TLS configuration file path>"
> >> option. The
> >> > > > > > > following
> >> > > > > > > > > > properties will be recognized in that file, and
> >> unrecognized
> >> > > > > > > properties
> >> > > > > > > > > > will be ignored to allow the possibility of pointing
> >> > > > > > > zk-tls-config-file
> >> > > > > > > > > at
> >> > > > > > > > > > the broker's config file.
> >> > > > > > > > > > zookeeper.client.secure (default value = false)
> >> > > > > > > > > > zookeeper.clientCnxnSocket
> >> > > > > > > > > > zookeeper.ssl.keyStore.location
> >> > > > > > > > > > zookeeper.ssl.keyStore.password
> >> > > > > > > > > > zookeeper.ssl.trustStore.location
> >> > > > > > > > > > zookeeper.ssl.trustStore.password
> >> > > > > > > > > > It will be an error for any of the last 5 values to be
> >> left
> >> > > > > > > unspecified
> >> > > > > > > > > if
> >> > > > > > > > > > zookeeper.client.secure is explicitly set to true.
> >> > > > > > > > >
> >> > > > > > > > > Do we really need a --zk-tls-config-file flag? It seems
> >> like this
> >> > > > > > > could
> >> > > > > > > > > just appear in a configuration file, which all of these
> >> tools already
> >> > > > > > > > > accept (I think).
> >> > > > > > > > >
> >> > > > > > > > > best,
> >> > > > > > > > > Colin
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Ron
> >> > > > > > > > > >
> >> > > > > > > > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <
> >> rndgstn@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi everyone. Let's get this discussion going again
> >> now that Kafka
> >> > > > > > > 2.4
> >> > > > > > > > > > > with Zookeeper 3.5.6 is out.
> >> > > > > > > > > > >
> >> > > > > > > > > > > First, regarding the KIP number, the other KIP that
> >> was using this
> >> > > > > > > > > number
> >> > > > > > > > > > > moved to KIP 534, so KIP 515 remains the correct
> >> number for this
> >> > > > > > > > > > > discussion. I've updated the Kafka Improvement
> >> Proposal page to
> >> > > > > > > list
> >> > > > > > > > > this
> >> > > > > > > > > > > KIP in the 515 slot, so we're all set there.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Regarding the actual issues under discussion, I think
> >> there are
> >> > > > > > > some
> >> > > > > > > > > > > things we should clarify.
> >> > > > > > > > > > >
> >> > > > > > > > > > > 1) It is possible to use TLS connectivity to
> >> Zookeeper from Apache
> >> > > > > > > > > Kafka
> >> > > > > > > > > > > 2.4 -- the problem is that configuration information
> >> has to be
> >> > > > > > > passed
> >> > > > > > > > > via
> >> > > > > > > > > > > system properties as "-D" command line options on the
> >> java
> >> > > > > > > invocation
> >> > > > > > > > > of
> >> > > > > > > > > > > the broker, and those are not secure (anyone with
> >> access to the box
> >> > > > > > > > > can see
> >> > > > > > > > > > > the command line used to invoke the broker); the
> >> configuration
> >> > > > > > > includes
> >> > > > > > > > > > > sensitive information (e.g. a keystore password), so
> >> we need a
> >> > > > > > > secure
> >> > > > > > > > > > > mechanism for passing the configuration values. I
> >> believe the real
> >> > > > > > > > > > > motivation for this KIP is to harden the
> >> configuration mechanism
> >> > > > > > > for
> >> > > > > > > > > > > Zookeeper TLS connectivity.
> >> > > > > > > > > > >
> >> > > > > > > > > > > 2) I believe the list of CLI tools that continue to
> >> use direct
> >> > > > > > > > > Zookeeper
> >> > > > > > > > > > > connectivity in a non-deprecated fashion is:
> >> > > > > > > > > > > a)
> >> > > > > > >
> >> zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> >> > > > > > > > > > > b)
> >> > > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > >
> >> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> >> > > > > > > > > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> >> > > > > > > > > > >
> >> > > > > > > > > > > 3) I believe Kafka.admin.ConfigCommand presents a
> >> conundrum
> >> > > > > > > because it
> >> > > > > > > > > > > explicitly states in a comment that a supported use
> >> case is
> >> > > > > > > > > bootstrapping a
> >> > > > > > > > > > > Kafka cluster with encrypted passwords in Zookeeper
> >> (see
> >> > > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > >
> >> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> >> > > > > > > > > ).
> >> > > > > > > > > > > This means it will be especially difficult to
> >> deprecate this
> >> > > > > > > particular
> >> > > > > > > > > > > direct Zookeeper connectivity without a different
> >> storage
> >> > > > > > > mechanism for
> >> > > > > > > > > > > dynamic configuration values being available (i.e.
> >> the self-managed
> >> > > > > > > > > quorum
> >> > > > > > > > > > > referred to in KIP-500).
> >> > > > > > > > > > >
> >> > > > > > > > > > > I think it would be easier and simpler to harden the
> >> Zookeeper TLS
> >> > > > > > > > > > > configuration in both Kafka and in CLI tools compared
> >> to trying to
> >> > > > > > > > > > > deprecate the direct Zookeeper connectivity in the
> >> above 3 CLI
> >> > > > > > > tools --
> >> > > > > > > > > > > especially when ConfigCommand has no obvious
> >> short-term path to
> >> > > > > > > > > deprecation.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Regarding how to harden the TLS configuration, adding
> >> the
> >> > > > > > > appropriate
> >> > > > > > > > > > > configs to Kafka is an obvious choice for the
> >> brokers. Not that
> >> > > > > > > these
> >> > > > > > > > > > > values cannot be dynamically reconfigurable because
> >> the values
> >> > > > > > > would be
> >> > > > > > > > > > > required to connect to Zookeeper via TLS in the first
> >> place.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Regarding how to harden the TLS configuration for the
> >> 3 remaining
> >> > > > > > > CLI
> >> > > > > > > > > > > tools, it would be relatively straightforward to add
> >> support for a
> >> > > > > > > > > > > "--zk-tls-config-file <String: Zookeeper TLS
> >> configuration system
> >> > > > > > > > > > > properties file path>" option to each one.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thoughts?
> >> > > > > > > > > > >
> >> > > > > > > > > > > Ron
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <
> >> > > > > > > pere.urbon@gmail.com
> >> > > > > > > > > >
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > >> True, the number is duplicated.
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> do you know how can we get the problem fix? I was
> >> not aware, I
> >> > > > > > > > > missed,
> >> > > > > > > > > > >> sorry the need to add the KIP to the table.
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> -- Pere
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> Missatge de Jorge Esteban Quilcate Otoya <
> >> > > > > > > quilcate.jorge@gmail.com>
> >> > > > > > > > > del
> >> > > > > > > > > > >> dia
> >> > > > > > > > > > >> dt., 3 de set. 2019 a les 13:43:
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> > Hi Pere,
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > Have you add your KIP to the list here
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > >
> >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >> > > > > > > > > > >> ?
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > I found the KIP number assigned to another.
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> >> > > > > > > > > pere.urbon@gmail.com>
> >> > > > > > > > > > >> > wrote:
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> >> Thanks for your time Harsha,
> >> > > > > > > > > > >> >> anyone else with comments? looking forward to
> >> hearing from
> >> > > > > > > you.
> >> > > > > > > > > > >> >>
> >> > > > > > > > > > >> >> Stupid question: when do you move from discussion
> >> to vote?
> >> > > > > > > > > > >> >>
> >> > > > > > > > > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io>
> >> del dia
> >> > > > > > > dv., 30
> >> > > > > > > > > > >> d’ag.
> >> > > > > > > > > > >> >> 2019 a les 21:59:
> >> > > > > > > > > > >> >>
> >> > > > > > > > > > >> >> > Thanks Pere. KIP looks good to me.
> >> > > > > > > > > > >> >> > -Harsha
> >> > > > > > > > > > >> >> >
> >> > > > > > > > > > >> >> >
> >> > > > > > > > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón
> >> Bayes <
> >> > > > > > > > > > >> >> pere.urbon@gmail.com>
> >> > > > > > > > > > >> >> > wrote:
> >> > > > > > > > > > >> >> >
> >> > > > > > > > > > >> >> >> Not really,
> >> > > > > > > > > > >> >> >> my idea is to keep the JAAS parameter, so
> >> people don't see
> >> > > > > > > > > major
> >> > > > > > > > > > >> >> >> changes. But if you pass a properties file,
> >> then this takes
> >> > > > > > > > > > >> precedence
> >> > > > > > > > > > >> >> over
> >> > > > > > > > > > >> >> >> the other, with the idea that you can do sasl
> >> as well with
> >> > > > > > > the
> >> > > > > > > > > > >> >> properties
> >> > > > > > > > > > >> >> >> files.
> >> > > > > > > > > > >> >> >>
> >> > > > > > > > > > >> >> >> Makes sense?
> >> > > > > > > > > > >> >> >>
> >> > > > > > > > > > >> >> >> -- Pere
> >> > > > > > > > > > >> >> >>
> >> > > > > > > > > > >> >> >> Missatge de Harsha Chintalapani <
> >> kafka@harsha.io> del dia
> >> > > > > > > dv.,
> >> > > > > > > > > 30
> >> > > > > > > > > > >> >> d’ag.
> >> > > > > > > > > > >> >> >> 2019 a les 19:00:
> >> > > > > > > > > > >> >> >>
> >> > > > > > > > > > >> >> >>> Hi Pere,
> >> > > > > > > > > > >> >> >>> Thanks for the KIP. Enabling SSL
> >> for zookeeper
> >> > > > > > > for
> >> > > > > > > > > Kafka
> >> > > > > > > > > > >> >> makes
> >> > > > > > > > > > >> >> >>> sense.
> >> > > > > > > > > > >> >> >>> "The changes are planned to be introduced in
> >> a compatible
> >> > > > > > > way,
> >> > > > > > > > > by
> >> > > > > > > > > > >> >> >>> keeping the current JAAS variable precedence."
> >> > > > > > > > > > >> >> >>> Can you elaborate a bit here. If the user
> >> configures a JAAS
> >> > > > > > > > > file
> >> > > > > > > > > > >> with
> >> > > > > > > > > > >> >> >>> Client section it will take precedence over
> >> zookeeper SSL
> >> > > > > > > > > configs?
> >> > > > > > > > > > >> >> >>>
> >> > > > > > > > > > >> >> >>> Thanks,
> >> > > > > > > > > > >> >> >>> Harsha
> >> > > > > > > > > > >> >> >>>
> >> > > > > > > > > > >> >> >>>
> >> > > > > > > > > > >> >> >>>
> >> > > > > > > > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón
> >> Bayes <
> >> > > > > > > > > > >> >> pere.urbon@gmail.com>
> >> > > > > > > > > > >> >> >>> wrote:
> >> > > > > > > > > > >> >> >>>
> >> > > > > > > > > > >> >> >>>> Hi,
> >> > > > > > > > > > >> >> >>>> quick question, I saw in another mail that
> >> 2.4 release is
> >> > > > > > > > > planned
> >> > > > > > > > > > >> for
> >> > > > > > > > > > >> >> >>>> September. I think it would be really
> >> awesome to have
> >> > > > > > > this for
> >> > > > > > > > > > >> this
> >> > > > > > > > > > >> >> >>>> release, do you think we can make it?
> >> > > > > > > > > > >> >> >>>>
> >> > > > > > > > > > >> >> >>>> -- Pere
> >> > > > > > > > > > >> >> >>>>
> >> > > > > > > > > > >> >> >>>> Missatge de Pere Urbón Bayes <
> >> pere.urbon@gmail.com> del
> >> > > > > > > dia
> >> > > > > > > > > dj.,
> >> > > > > > > > > > >> 29
> >> > > > > > > > > > >> >> >>>> d’ag. 2019 a les 20:10:
> >> > > > > > > > > > >> >> >>>>
> >> > > > > > > > > > >> >> >>>> Hi,
> >> > > > > > > > > > >> >> >>>> this is my first KIP for a change in Apache
> >> Kafka, so I'm
> >> > > > > > > > > really
> >> > > > > > > > > > >> need
> >> > > > > > > > > > >> >> >>>> to the process. Looking forward to hearing
> >> from you and
> >> > > > > > > learn
> >> > > > > > > > > the
> >> > > > > > > > > > >> >> best
> >> > > > > > > > > > >> >> >>>> ropes here.
> >> > > > > > > > > > >> >> >>>>
> >> > > > > > > > > > >> >> >>>> I would like to propose this KIP-515 to
> >> enable the
> >> > > > > > > > > > >> ZookeeperClients
> >> > > > > > > > > > >> >> to
> >> > > > > > > > > > >> >> >>>> take full advantage of the TLS communication
> >> in the new
> >> > > > > > > > > Zookeeper
> >> > > > > > > > > > >> >> 3.5.5.
> >> > > > > > > > > > >> >> >>>> Specially interesting it the Zookeeper
> >> Security Migration,
> >> > > > > > > > > that
> >> > > > > > > > > > >> >> without
> >> > > > > > > > > > >> >> >>>> this change will not work with TLS,
> >> disabling users to use
> >> > > > > > > > > ACLs
> >> > > > > > > > > > >> when
> >> > > > > > > > > > >> >> the
> >> > > > > > > > > > >> >> >>>> Zookeeper cluster use TLS.
> >> > > > > > > > > > >> >> >>>>
> >> > > > > > > > > > >> >> >>>> link:
> >> > > > > > > > > > >> >> >>>>
> >> > > > > > > > > > >> >> >>>>
> >> > > > > > > > > > >> >>
> >> > > > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> >> > > > > > > > > > >> >> >>>>
> >> > > > > > > > > > >> >> >>>> Looking forward to hearing from you on this,
> >> > > > > > > > > > >> >> >>>>
> >> > > > > > > > > > >> >> >>>> /cheers
> >> > > > > > > > > > >> >> >>>>
> >> > > > > > > > > > >> >> >>>> --
> >> > > > > > > > > > >> >> >>>> Pere Urbon-Bayes
> >> > > > > > > > > > >> >> >>>> Software Architect
> >> > > > > > > > > > >> >> >>>> http://www.purbon.com
> >> > > > > > > > > > >> >> >>>> https://twitter.com/purbon
> >> > > > > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> >> > > > > > > > > > >> >> >>>>
> >> > > > > > > > > > >> >> >>>> --
> >> > > > > > > > > > >> >> >>>> Pere Urbon-Bayes
> >> > > > > > > > > > >> >> >>>> Software Architect
> >> > > > > > > > > > >> >> >>>> http://www.purbon.com
> >> > > > > > > > > > >> >> >>>> https://twitter.com/purbon
> >> > > > > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> >> > > > > > > > > > >> >> >>>>
> >> > > > > > > > > > >> >> >>>
> >> > > > > > > > > > >> >> >>>
> >> > > > > > > > > > >> >> >>
> >> > > > > > > > > > >> >> >> --
> >> > > > > > > > > > >> >> >> Pere Urbon-Bayes
> >> > > > > > > > > > >> >> >> Software Architect
> >> > > > > > > > > > >> >> >> http://www.purbon.com
> >> > > > > > > > > > >> >> >> https://twitter.com/purbon
> >> > > > > > > > > > >> >> >> https://www.linkedin.com/in/purbon/
> >> > > > > > > > > > >> >> >>
> >> > > > > > > > > > >> >> >
> >> > > > > > > > > > >> >> >
> >> > > > > > > > > > >> >>
> >> > > > > > > > > > >> >> --
> >> > > > > > > > > > >> >> Pere Urbon-Bayes
> >> > > > > > > > > > >> >> Software Architect
> >> > > > > > > > > > >> >> http://www.purbon.com
> >> > > > > > > > > > >> >> https://twitter.com/purbon
> >> > > > > > > > > > >> >> https://www.linkedin.com/in/purbon/
> >> > > > > > > > > > >> >>
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> --
> >> > > > > > > > > > >> Pere Urbon-Bayes
> >> > > > > > > > > > >> Software Architect
> >> > > > > > > > > > >> http://www.purbon.com
> >> > > > > > > > > > >> https://twitter.com/purbon
> >> > > > > > > > > > >> https://www.linkedin.com/in/purbon/
> >> > > > > > > > > > >>
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > >
> >> >
> >>
> >
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Rajini Sivaram <ra...@gmail.com>.
Forgot to mention:
Overriding system property with a differently named Kafka configuration
option is something we already do for JAAS configs - we default to System
property `java.security.auth.login.config` if `sasl.jaas.config` is not
configured.
On Thu, Jan 16, 2020 at 10:58 AM Rajini Sivaram <ra...@gmail.com>
wrote:
> Hi Ron/Colin,
>
> Thanks for the responses. I think we need to consider these cases:
>
> 1) The recommended approach: This is the one we document. Configs
> consistent with Kafka make sense here. It seems a lot more confusing to
> configure `ssl.cipher.suites`, `ssl.enabled.protocols`, `
> zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols` with
> multiple naming inconsistencies in server.properties. For users who have
> never used ZK system properties, this is clearly simpler. Another config
> to consider is `ssl.endpoint.identification.algorithm` which does the
> same thing as `zookeeper.ssl.hostnameVerification` - if we are staying
> consistent, we should use the Kafka format for this too. I would also
> consider using something like `zookeeper.client.ssl.enabled` instead of `
> zookeeper.client.secure` since it is difficult to tell what `secure`
> means when you have options to configure Kerberos, ACLs and/or SSL.
>
> 2) Inheritance using prefixed configs as we do for other prefixed configs
> in Kafka: I think it may be useful for configs like `ssl.protocol` and `
> ssl.cipher.suites`.
>
> 3) Default values: Do we want to use Kafka defaults? The inconsistency we
> have is `ssl.protocol/ssl.enabled.protocols` since we still have insecure
> protocols enabled for Kafka. We have KIP-553 (
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956)
> open to disable these. It is currently blocked since it also talks about
> enabling TLSv1.3 by default and we haven't done sufficient testing for
> that. Since it will be good to disable the older insecure protocols anyway,
> perhaps we could do that without enabling TLSv1.3 by default in 2.5?
>
> 4) Backward compatibility for deployments which are using system
> properties. We shouldn't override system properties with Kafka defaults or
> inherited values. But we do want to override if user configures properties
> explicitly. In the current approach, this was simpler since we just had to
> set the configured values. But if we decide to inherit Kafka configs, then
> we will need to check system properties and update accordingly. But that
> should be doable?
>
> On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe <cm...@apache.org> wrote:
>
>> On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote:
>> > Hi Colin. Two things come to mind with respect to ZooKeeper camelCase
>> > style vs Kafka-style config names for ZooKeeper. First, I think it
>> > would be desirable for the client configs and broker configs to be
>> > interoperable. For example, it feels like it would be convenient to
>> > be able to pass the broker's config file to the ZK Security Migrator
>> > tool. I think whatever we use (ZooKeeper camelCase style or Kafka
>> > style), I think we should use the same for both.
>> >
>> > The second thing to think about is the fact that ZooKeeper
>> > configuration inherently uses Java system properties as a first pass.
>> > So while we might switch to Kafka-style configs in the config file
>> > (e.g. zookeeper.ssl.trust.store.location), the
>> > org.apache.zookeeper.client.ZKClientConfig class (which is how TLS
>> > configs are set) is going to look for the camelCase
>> > zookeeper.ssl.trustStore.location system property and use any value
>> > there. I think this could be very confusing for people. Granted, we
>> > are doing this so that people don't have to use system properties, but
>> > there is no way to turn off the use of system properties, so I think
>> > there would be pretty substantial potential for confusion.
>>
>> I don't think we document or recommend that anyone use system properties
>> to configure Zookeeper usage within Kafka. And I think the reason is
>> exactly the issue you pointed out -- it wouldn't be consistent with our
>> other configurations.
>>
>> best,
>> Colin
>>
>> >
>> > One idea is to migrate the system properties -- i.e. look to see if
>> > zookeeper.ssl.trustStore.location is set and if it is clear out the
>> > value there and move it under the zookeeper.ssl.trust.store.location
>> > system property. (I said it was an idea -- not necessarily a good one
>> > :-)
>> >
>> > Ron
>> >
>> > On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe <cm...@apache.org>
>> wrote:
>> > >
>> > > On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
>> > > > Thanks Colin and Rajini.
>> > > >
>> > > > I've updated the KIP to say that KIP-421 functionality is available
>> to
>> > > > encrypt sensitive configs like the ZK key store and trust store
>> > > > passwords. (I've also made it clear that the configs are not
>> > > > dynamically reconfigurable since dynamic values are stored in ZK and
>> > > > the data is needed to get to ZK in the first place). Colin, let me
>> > > > know if you think anything else needs to be said/done about it -- I
>> > > > wasn't sure if your comment above implies that there are additional
>> > > > direct actions that we need to take aside from this.
>> > > >
>> > > > I agree that making the brokers' ZooKeeper clients dynamic with
>> > > > respect to key and trust stores (e.g. via
>> > > > zookeeper.ssl.context.supplier.class) may increase the risk that
>> this
>> > > > KIP may not land in time for 2.5.
>> > > >
>> > > > Regarding the config names for ZK being different than the ZooKeeper
>> > > > configs (e.g. camel-case keyStore/trustStore instead of
>> > > > keystore/truststore, ciphersuites instead of cipher.suites,
>> > > > enabledProtocols instead of enabled.protocols), I agree it is an
>> > > > issue. I tried to mitigate it by putting warning text in the config
>> > > > docs for these. Regarding configuration inheritance, I think what
>> you
>> > > > are saying is that for any configs where the broker has a clear
>> > > > semantic equivalent, the broker could fall back to the broker value
>> if
>> > > > the ZK value is not given. The potential list is:
>> > > >
>> > > > zookeeper.ssl.keyStore.location => ssl.keystore.location
>> > > > zookeeper.ssl.keyStore.password => ssl.keystore.password
>> > > > zookeeper.ssl.keyStore.type => ssl.keystore.type
>> > > > zookeeper.ssl.trustStore.location => ssl.truststore.location
>> > > > zookeeper.ssl.trustStore.password => ssl.truststore.password
>> > > > zookeeper.ssl.trustStore.type => ssl.truststore.type
>> > > > zookeeper.ssl.ciphersuites => ssl.cipher.suites
>> > > >
>> > > > Note that there are two configs that look the same based on their
>> key
>> > > > names but whose allowable values differ:
>> > > >
>> > > > zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default:
>> TLS)
>> > > > zookeeper.ssl.enabledProtocols(Default: value of protocol property)
>> =
>> > > > ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
>> > > >
>> > > > Not sure what the right answer is, but hopefully this detail will
>> help
>> > > > get us there.
>> > > >
>> > > > Ron
>> > >
>> > > I think, on balance, I agree with Rajini that it would be nice to
>> make the configs look more like Kafka configs. We don't have camel-case in
>> configuration keys, so we should avoid it here.
>> > >
>> > > best,
>> > > Colin
>> > >
>> > > >
>> > > > On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <cm...@apache.org>
>> wrote:
>> > > > >
>> > > > > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
>> > > > > > Hi Ron,
>> > > > > >
>> > > > > > Thanks for the detailed explanation, sounds good to me.
>> > > > > >
>> > > > > > A few more questions:
>> > > > > >
>> > > > > > 1) At the moment, all sensitive broker configs including
>> > > > > > keystore/truststore passwords can be stored encrypted in
>> ZooKeeper prior to
>> > > > > > starting up brokers. We are now adding SSL keystore/truststore
>> passwords
>> > > > > > for ZooKeeper client that cannot be stored in ZooKeeper since
>> you need
>> > > > > > these to connect to ZK. We should perhaps document that these
>> configs can
>> > > > > > be encrypted using KIP-421.
>> > > > >
>> > > > > That's a good point. Previously, we didn't have ZK on-the-wire
>> security support. However, now that we do, sending sensitive keystore and
>> truststore passwords to ZK in cleartext should be deprecated in favor of
>> using KIP-421.
>> > > > >
>> > > > > >
>> > > > > > 2) We can dynamically update keystores and trust stores used by
>> brokers
>> > > > > > without restarting the broker. Can we support this easily for
>> ZK clients
>> > > > > > created by the broker, for example by adding our own
>> > > > > > `zookeeper.ssl.context.supplier.class`?
>> > > > > >
>> > > > >
>> > > > > Hmm. That might be better to consider in a follow-up, since the
>> deadline for 2.5 KIPs is coming up?
>> > > > >
>> > > > > best,
>> > > > > Colin
>> > > > >
>> > > > > > 3) Looks like we are using config names that directly map to ZK
>> configs.
>> > > > > > Have we considered using equivalent Kafka config names with
>> prefixes,
>> > > > > > perhaps with inheritance from the non-prefixed names? Not sure
>> if this is a
>> > > > > > good idea, but perhaps worth documenting in Rejected
>> Alternatives at least.
>> > > > > >
>> > > > > >
>> > > > > > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <
>> cmccabe@apache.org> wrote:
>> > > > > >
>> > > > > > > Hi Ron,
>> > > > > > >
>> > > > > > > Thanks for the explanation. I guess thinking about it a
>> little bit more,
>> > > > > > > we should just add --zk-tls-config-file to all of these
>> commands.
>> > > > > > >
>> > > > > > > We will be removing this option (plus ZK in general) from
>> these commands
>> > > > > > > in the next major release, but ZK is still supported in 2.5,
>> so we should
>> > > > > > > just do the logical thing. (The exception is
>> ZkSecurityMigrator, which
>> > > > > > > will stay).
>> > > > > > >
>> > > > > > > best,
>> > > > > > > Colin
>> > > > > > >
>> > > > > > >
>> > > > > > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
>> > > > > > > > Hi Colin.
>> > > > > > > >
>> > > > > > > > <<< It seems like this [--zk-tls-config-file information]
>> could just
>> > > > > > > appear
>> > > > > > > > in a configuration file, which all of these tools already
>> accept (I
>> > > > > > > think)
>> > > > > > > >
>> > > > > > > > ZkSecurityMigrator has no such property file facility;
>> adding a
>> > > > > > > > "--zk-tls-config-file" parameter is exactly for this
>> purpose. If we add
>> > > > > > > > that to ZkSecurityMigrator then it is trivial to add it to
>> other commands
>> > > > > > > > (the same code is simply reused; it ends up being just a
>> few extra
>> > > > > > > lines).
>> > > > > > > > I do not see any parameter in the other two commands to
>> adjust the ZK
>> > > > > > > > connection; ConfigCommand accepts a "--command-config"
>> flag, but
>> > > > > > > according
>> > > > > > > > to the code "This is used only with --bootstrap-server
>> option for
>> > > > > > > > describing and altering broker configs."
>> > > > > > > >
>> > > > > > > > I do agree there would be no need to add
>> "--zk-tls-config-file" to
>> > > > > > > > ReassignPartitionsCommand if its direct ZK connectivity is
>> replaced in
>> > > > > > > time
>> > > > > > > > for the next release.
>> > > > > > > >
>> > > > > > > > ConfigCommand supports the "--bootstrap-server" option and
>> will have its
>> > > > > > > > direct ZooKeeper access formally deprecated as per KIP-555,
>> but the
>> > > > > > > special
>> > > > > > > > use case of bootstrapping a ZooKeeper ensemble with
>> encrypted credentials
>> > > > > > > > prior to starting Kafka will still exist, so it feels like
>> while
>> > > > > > > > "--zk-tls-config-file" would never be used except for this
>> use case it
>> > > > > > > > could probably still be added for this particular situation.
>> > > > > > > >
>> > > > > > > > Ron
>> > > > > > > >
>> > > > > > > > P.S. I responded on 1/6 but I just discovered that e, ail
>> (and 3 more I
>> > > > > > > > sent) did not go through. I am trying to get emails
>> through now to move
>> > > > > > > > this discussion forward.
>> > > > > > > >
>> > > > > > > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <
>> cmccabe@apache.org> wrote:
>> > > > > > > >
>> > > > > > > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
>> > > > > > > > > > Hi everyone. I would like to make the following
>> changes to the KIP.
>> > > > > > > > > >
>> > > > > > > > > > MOTIVATION:
>> > > > > > > > > > Include a statement that it will be difficult in the
>> short term to
>> > > > > > > > > > deprecate direct Zookeeper communication in
>> kafka-configs.{sh, bat}
>> > > > > > > > > (which
>> > > > > > > > > > invoke kafka.admin.ConfigCommand) because bootstrapping
>> a Kafka
>> > > > > > > cluster
>> > > > > > > > > > with encrypted passwords in Zookeeper is an
>> explicitly-supported use
>> > > > > > > > > case;
>> > > > > > > > > > therefore it is in scope to be able to securely
>> configure the CLI
>> > > > > > > tools
>> > > > > > > > > > that still leverage non-deprecated direct Zookeeper
>> communication
>> > > > > > > for TLS
>> > > > > > > > > > (the other 2 tools are kafka-reassign-partitions.{sh,
>> bat} and
>> > > > > > > > > > zookeeper-security-migration.sh).
>> > > > > > > > >
>> > > > > > > > > Hi Ron,
>> > > > > > > > >
>> > > > > > > > > Thanks for the KIP.
>> > > > > > > > >
>> > > > > > > > > About deprecations:
>> > > > > > > > >
>> > > > > > > > > * zookeeper-security-migration: clearly, deprecating ZK
>> access in this
>> > > > > > > one
>> > > > > > > > > would not make sense, since it would defeat the whole
>> point of the
>> > > > > > > tool :)
>> > > > > > > > >
>> > > > > > > > > * kafka-reassign-partitions: ZK access should be
>> deprecated here.
>> > > > > > > KIP-455
>> > > > > > > > > implementation has dragged a bit, but this should get
>> done soon.
>> > > > > > > Certainly
>> > > > > > > > > before the next release.
>> > > > > > > > >
>> > > > > > > > > * kafka-configs: I think ZK access should be deprecated
>> here as well.
>> > > > > > > I
>> > > > > > > > > agree there is a super-special bootstrapping case here,
>> but that should
>> > > > > > > > > have its own tool, not use kafka-configs.
>> > > > > > > > >
>> > > > > > > > > I will post a separate KIP for this, though.
>> > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > GOALS:
>> > > > > > > > > > Support the secure configuration of TLS-encrypted
>> communication
>> > > > > > > between
>> > > > > > > > > > Zookeeper and:
>> > > > > > > > > > a) Kafka brokers
>> > > > > > > > > > b) The three CLI tools mentioned above that still
>> support direct,
>> > > > > > > > > > non-deprecated communication to Zookeeper
>> > > > > > > > > > It is explicitly out-of-scope to deprecate any direct
>> Zookeeper
>> > > > > > > > > > communication in CLI tools as part of this KIP; such
>> work will occur
>> > > > > > > in
>> > > > > > > > > > future KIPs instead.
>> > > > > > > > > >
>> > > > > > > > > > PUBLIC INTERFACES:
>> > > > > > > > > > 1) The following new broker configurations will be
>> recognized.
>> > > > > > > > > > zookeeper.client.secure (default value = false, for
>> backwards
>> > > > > > > > > > compatibility)
>> > > > > > > > > > zookeeper.clientCnxnSocket
>> > > > > > > > > > zookeeper.ssl.keyStore.location
>> > > > > > > > > > zookeeper.ssl.keyStore.password
>> > > > > > > > > > zookeeper.ssl.trustStore.location
>> > > > > > > > > > zookeeper.ssl.trustStore.password
>> > > > > > > > > > It will be an error for any of the last 5 values to be
>> left
>> > > > > > > unspecified
>> > > > > > > > > if
>> > > > > > > > > > zookeeper.client.secure is explicitly set to true.
>> > > > > > > > > >
>> > > > > > > > > > 2) In addition, the
>> kafka.security.authorizer.AclAuthorizer class
>> > > > > > > > > supports
>> > > > > > > > > > the ability to connect to a different Zookeeper
>> instance than the
>> > > > > > > one the
>> > > > > > > > > > brokers use. We therefore also add the following
>> optional configs,
>> > > > > > > which
>> > > > > > > > > > override the corresponding ones from above when present:
>> > > > > > > > > > authorizer.zookeeper.client.secure
>> > > > > > > > > > authorizer.zookeeper.clientCnxnSocket
>> > > > > > > > > > authorizer.zookeeper.ssl.keyStore.location
>> > > > > > > > > > authorizer.zookeeper.ssl.keyStore.password
>> > > > > > > > > > authorizer.zookeeper.ssl.trustStore.location
>> > > > > > > > > > authorizer.zookeeper.ssl.trustStore.password
>> > > > > > > > > >
>> > > > > > > > > > 3) The three CLI tools mentioned above will support a
>> new
>> > > > > > > > > --zk-tls-config-file
>> > > > > > > > > > <String: Zookeeper TLS configuration file path>"
>> option. The
>> > > > > > > following
>> > > > > > > > > > properties will be recognized in that file, and
>> unrecognized
>> > > > > > > properties
>> > > > > > > > > > will be ignored to allow the possibility of pointing
>> > > > > > > zk-tls-config-file
>> > > > > > > > > at
>> > > > > > > > > > the broker's config file.
>> > > > > > > > > > zookeeper.client.secure (default value = false)
>> > > > > > > > > > zookeeper.clientCnxnSocket
>> > > > > > > > > > zookeeper.ssl.keyStore.location
>> > > > > > > > > > zookeeper.ssl.keyStore.password
>> > > > > > > > > > zookeeper.ssl.trustStore.location
>> > > > > > > > > > zookeeper.ssl.trustStore.password
>> > > > > > > > > > It will be an error for any of the last 5 values to be
>> left
>> > > > > > > unspecified
>> > > > > > > > > if
>> > > > > > > > > > zookeeper.client.secure is explicitly set to true.
>> > > > > > > > >
>> > > > > > > > > Do we really need a --zk-tls-config-file flag? It seems
>> like this
>> > > > > > > could
>> > > > > > > > > just appear in a configuration file, which all of these
>> tools already
>> > > > > > > > > accept (I think).
>> > > > > > > > >
>> > > > > > > > > best,
>> > > > > > > > > Colin
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Ron
>> > > > > > > > > >
>> > > > > > > > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <
>> rndgstn@gmail.com>
>> > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi everyone. Let's get this discussion going again
>> now that Kafka
>> > > > > > > 2.4
>> > > > > > > > > > > with Zookeeper 3.5.6 is out.
>> > > > > > > > > > >
>> > > > > > > > > > > First, regarding the KIP number, the other KIP that
>> was using this
>> > > > > > > > > number
>> > > > > > > > > > > moved to KIP 534, so KIP 515 remains the correct
>> number for this
>> > > > > > > > > > > discussion. I've updated the Kafka Improvement
>> Proposal page to
>> > > > > > > list
>> > > > > > > > > this
>> > > > > > > > > > > KIP in the 515 slot, so we're all set there.
>> > > > > > > > > > >
>> > > > > > > > > > > Regarding the actual issues under discussion, I think
>> there are
>> > > > > > > some
>> > > > > > > > > > > things we should clarify.
>> > > > > > > > > > >
>> > > > > > > > > > > 1) It is possible to use TLS connectivity to
>> Zookeeper from Apache
>> > > > > > > > > Kafka
>> > > > > > > > > > > 2.4 -- the problem is that configuration information
>> has to be
>> > > > > > > passed
>> > > > > > > > > via
>> > > > > > > > > > > system properties as "-D" command line options on the
>> java
>> > > > > > > invocation
>> > > > > > > > > of
>> > > > > > > > > > > the broker, and those are not secure (anyone with
>> access to the box
>> > > > > > > > > can see
>> > > > > > > > > > > the command line used to invoke the broker); the
>> configuration
>> > > > > > > includes
>> > > > > > > > > > > sensitive information (e.g. a keystore password), so
>> we need a
>> > > > > > > secure
>> > > > > > > > > > > mechanism for passing the configuration values. I
>> believe the real
>> > > > > > > > > > > motivation for this KIP is to harden the
>> configuration mechanism
>> > > > > > > for
>> > > > > > > > > > > Zookeeper TLS connectivity.
>> > > > > > > > > > >
>> > > > > > > > > > > 2) I believe the list of CLI tools that continue to
>> use direct
>> > > > > > > > > Zookeeper
>> > > > > > > > > > > connectivity in a non-deprecated fashion is:
>> > > > > > > > > > > a)
>> > > > > > >
>> zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
>> > > > > > > > > > > b)
>> > > > > > > > > > >
>> > > > > > > > >
>> > > > > > >
>> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
>> > > > > > > > > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
>> > > > > > > > > > >
>> > > > > > > > > > > 3) I believe Kafka.admin.ConfigCommand presents a
>> conundrum
>> > > > > > > because it
>> > > > > > > > > > > explicitly states in a comment that a supported use
>> case is
>> > > > > > > > > bootstrapping a
>> > > > > > > > > > > Kafka cluster with encrypted passwords in Zookeeper
>> (see
>> > > > > > > > > > >
>> > > > > > > > >
>> > > > > > >
>> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
>> > > > > > > > > ).
>> > > > > > > > > > > This means it will be especially difficult to
>> deprecate this
>> > > > > > > particular
>> > > > > > > > > > > direct Zookeeper connectivity without a different
>> storage
>> > > > > > > mechanism for
>> > > > > > > > > > > dynamic configuration values being available (i.e.
>> the self-managed
>> > > > > > > > > quorum
>> > > > > > > > > > > referred to in KIP-500).
>> > > > > > > > > > >
>> > > > > > > > > > > I think it would be easier and simpler to harden the
>> Zookeeper TLS
>> > > > > > > > > > > configuration in both Kafka and in CLI tools compared
>> to trying to
>> > > > > > > > > > > deprecate the direct Zookeeper connectivity in the
>> above 3 CLI
>> > > > > > > tools --
>> > > > > > > > > > > especially when ConfigCommand has no obvious
>> short-term path to
>> > > > > > > > > deprecation.
>> > > > > > > > > > >
>> > > > > > > > > > > Regarding how to harden the TLS configuration, adding
>> the
>> > > > > > > appropriate
>> > > > > > > > > > > configs to Kafka is an obvious choice for the
>> brokers. Not that
>> > > > > > > these
>> > > > > > > > > > > values cannot be dynamically reconfigurable because
>> the values
>> > > > > > > would be
>> > > > > > > > > > > required to connect to Zookeeper via TLS in the first
>> place.
>> > > > > > > > > > >
>> > > > > > > > > > > Regarding how to harden the TLS configuration for the
>> 3 remaining
>> > > > > > > CLI
>> > > > > > > > > > > tools, it would be relatively straightforward to add
>> support for a
>> > > > > > > > > > > "--zk-tls-config-file <String: Zookeeper TLS
>> configuration system
>> > > > > > > > > > > properties file path>" option to each one.
>> > > > > > > > > > >
>> > > > > > > > > > > Thoughts?
>> > > > > > > > > > >
>> > > > > > > > > > > Ron
>> > > > > > > > > > >
>> > > > > > > > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <
>> > > > > > > pere.urbon@gmail.com
>> > > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > >> True, the number is duplicated.
>> > > > > > > > > > >>
>> > > > > > > > > > >> do you know how can we get the problem fix? I was
>> not aware, I
>> > > > > > > > > missed,
>> > > > > > > > > > >> sorry the need to add the KIP to the table.
>> > > > > > > > > > >>
>> > > > > > > > > > >> -- Pere
>> > > > > > > > > > >>
>> > > > > > > > > > >> Missatge de Jorge Esteban Quilcate Otoya <
>> > > > > > > quilcate.jorge@gmail.com>
>> > > > > > > > > del
>> > > > > > > > > > >> dia
>> > > > > > > > > > >> dt., 3 de set. 2019 a les 13:43:
>> > > > > > > > > > >>
>> > > > > > > > > > >> > Hi Pere,
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > Have you add your KIP to the list here
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > >
>> > > > > > >
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>> > > > > > > > > > >> ?
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > I found the KIP number assigned to another.
>> > > > > > > > > > >> >
>> > > > > > > > > > >> >
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
>> > > > > > > > > pere.urbon@gmail.com>
>> > > > > > > > > > >> > wrote:
>> > > > > > > > > > >> >
>> > > > > > > > > > >> >> Thanks for your time Harsha,
>> > > > > > > > > > >> >> anyone else with comments? looking forward to
>> hearing from
>> > > > > > > you.
>> > > > > > > > > > >> >>
>> > > > > > > > > > >> >> Stupid question: when do you move from discussion
>> to vote?
>> > > > > > > > > > >> >>
>> > > > > > > > > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io>
>> del dia
>> > > > > > > dv., 30
>> > > > > > > > > > >> d’ag.
>> > > > > > > > > > >> >> 2019 a les 21:59:
>> > > > > > > > > > >> >>
>> > > > > > > > > > >> >> > Thanks Pere. KIP looks good to me.
>> > > > > > > > > > >> >> > -Harsha
>> > > > > > > > > > >> >> >
>> > > > > > > > > > >> >> >
>> > > > > > > > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón
>> Bayes <
>> > > > > > > > > > >> >> pere.urbon@gmail.com>
>> > > > > > > > > > >> >> > wrote:
>> > > > > > > > > > >> >> >
>> > > > > > > > > > >> >> >> Not really,
>> > > > > > > > > > >> >> >> my idea is to keep the JAAS parameter, so
>> people don't see
>> > > > > > > > > major
>> > > > > > > > > > >> >> >> changes. But if you pass a properties file,
>> then this takes
>> > > > > > > > > > >> precedence
>> > > > > > > > > > >> >> over
>> > > > > > > > > > >> >> >> the other, with the idea that you can do sasl
>> as well with
>> > > > > > > the
>> > > > > > > > > > >> >> properties
>> > > > > > > > > > >> >> >> files.
>> > > > > > > > > > >> >> >>
>> > > > > > > > > > >> >> >> Makes sense?
>> > > > > > > > > > >> >> >>
>> > > > > > > > > > >> >> >> -- Pere
>> > > > > > > > > > >> >> >>
>> > > > > > > > > > >> >> >> Missatge de Harsha Chintalapani <
>> kafka@harsha.io> del dia
>> > > > > > > dv.,
>> > > > > > > > > 30
>> > > > > > > > > > >> >> d’ag.
>> > > > > > > > > > >> >> >> 2019 a les 19:00:
>> > > > > > > > > > >> >> >>
>> > > > > > > > > > >> >> >>> Hi Pere,
>> > > > > > > > > > >> >> >>> Thanks for the KIP. Enabling SSL
>> for zookeeper
>> > > > > > > for
>> > > > > > > > > Kafka
>> > > > > > > > > > >> >> makes
>> > > > > > > > > > >> >> >>> sense.
>> > > > > > > > > > >> >> >>> "The changes are planned to be introduced in
>> a compatible
>> > > > > > > way,
>> > > > > > > > > by
>> > > > > > > > > > >> >> >>> keeping the current JAAS variable precedence."
>> > > > > > > > > > >> >> >>> Can you elaborate a bit here. If the user
>> configures a JAAS
>> > > > > > > > > file
>> > > > > > > > > > >> with
>> > > > > > > > > > >> >> >>> Client section it will take precedence over
>> zookeeper SSL
>> > > > > > > > > configs?
>> > > > > > > > > > >> >> >>>
>> > > > > > > > > > >> >> >>> Thanks,
>> > > > > > > > > > >> >> >>> Harsha
>> > > > > > > > > > >> >> >>>
>> > > > > > > > > > >> >> >>>
>> > > > > > > > > > >> >> >>>
>> > > > > > > > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón
>> Bayes <
>> > > > > > > > > > >> >> pere.urbon@gmail.com>
>> > > > > > > > > > >> >> >>> wrote:
>> > > > > > > > > > >> >> >>>
>> > > > > > > > > > >> >> >>>> Hi,
>> > > > > > > > > > >> >> >>>> quick question, I saw in another mail that
>> 2.4 release is
>> > > > > > > > > planned
>> > > > > > > > > > >> for
>> > > > > > > > > > >> >> >>>> September. I think it would be really
>> awesome to have
>> > > > > > > this for
>> > > > > > > > > > >> this
>> > > > > > > > > > >> >> >>>> release, do you think we can make it?
>> > > > > > > > > > >> >> >>>>
>> > > > > > > > > > >> >> >>>> -- Pere
>> > > > > > > > > > >> >> >>>>
>> > > > > > > > > > >> >> >>>> Missatge de Pere Urbón Bayes <
>> pere.urbon@gmail.com> del
>> > > > > > > dia
>> > > > > > > > > dj.,
>> > > > > > > > > > >> 29
>> > > > > > > > > > >> >> >>>> d’ag. 2019 a les 20:10:
>> > > > > > > > > > >> >> >>>>
>> > > > > > > > > > >> >> >>>> Hi,
>> > > > > > > > > > >> >> >>>> this is my first KIP for a change in Apache
>> Kafka, so I'm
>> > > > > > > > > really
>> > > > > > > > > > >> need
>> > > > > > > > > > >> >> >>>> to the process. Looking forward to hearing
>> from you and
>> > > > > > > learn
>> > > > > > > > > the
>> > > > > > > > > > >> >> best
>> > > > > > > > > > >> >> >>>> ropes here.
>> > > > > > > > > > >> >> >>>>
>> > > > > > > > > > >> >> >>>> I would like to propose this KIP-515 to
>> enable the
>> > > > > > > > > > >> ZookeeperClients
>> > > > > > > > > > >> >> to
>> > > > > > > > > > >> >> >>>> take full advantage of the TLS communication
>> in the new
>> > > > > > > > > Zookeeper
>> > > > > > > > > > >> >> 3.5.5.
>> > > > > > > > > > >> >> >>>> Specially interesting it the Zookeeper
>> Security Migration,
>> > > > > > > > > that
>> > > > > > > > > > >> >> without
>> > > > > > > > > > >> >> >>>> this change will not work with TLS,
>> disabling users to use
>> > > > > > > > > ACLs
>> > > > > > > > > > >> when
>> > > > > > > > > > >> >> the
>> > > > > > > > > > >> >> >>>> Zookeeper cluster use TLS.
>> > > > > > > > > > >> >> >>>>
>> > > > > > > > > > >> >> >>>> link:
>> > > > > > > > > > >> >> >>>>
>> > > > > > > > > > >> >> >>>>
>> > > > > > > > > > >> >>
>> > > > > > > > > > >>
>> > > > > > > > >
>> > > > > > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
>> > > > > > > > > > >> >> >>>>
>> > > > > > > > > > >> >> >>>> Looking forward to hearing from you on this,
>> > > > > > > > > > >> >> >>>>
>> > > > > > > > > > >> >> >>>> /cheers
>> > > > > > > > > > >> >> >>>>
>> > > > > > > > > > >> >> >>>> --
>> > > > > > > > > > >> >> >>>> Pere Urbon-Bayes
>> > > > > > > > > > >> >> >>>> Software Architect
>> > > > > > > > > > >> >> >>>> http://www.purbon.com
>> > > > > > > > > > >> >> >>>> https://twitter.com/purbon
>> > > > > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
>> > > > > > > > > > >> >> >>>>
>> > > > > > > > > > >> >> >>>> --
>> > > > > > > > > > >> >> >>>> Pere Urbon-Bayes
>> > > > > > > > > > >> >> >>>> Software Architect
>> > > > > > > > > > >> >> >>>> http://www.purbon.com
>> > > > > > > > > > >> >> >>>> https://twitter.com/purbon
>> > > > > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
>> > > > > > > > > > >> >> >>>>
>> > > > > > > > > > >> >> >>>
>> > > > > > > > > > >> >> >>>
>> > > > > > > > > > >> >> >>
>> > > > > > > > > > >> >> >> --
>> > > > > > > > > > >> >> >> Pere Urbon-Bayes
>> > > > > > > > > > >> >> >> Software Architect
>> > > > > > > > > > >> >> >> http://www.purbon.com
>> > > > > > > > > > >> >> >> https://twitter.com/purbon
>> > > > > > > > > > >> >> >> https://www.linkedin.com/in/purbon/
>> > > > > > > > > > >> >> >>
>> > > > > > > > > > >> >> >
>> > > > > > > > > > >> >> >
>> > > > > > > > > > >> >>
>> > > > > > > > > > >> >> --
>> > > > > > > > > > >> >> Pere Urbon-Bayes
>> > > > > > > > > > >> >> Software Architect
>> > > > > > > > > > >> >> http://www.purbon.com
>> > > > > > > > > > >> >> https://twitter.com/purbon
>> > > > > > > > > > >> >> https://www.linkedin.com/in/purbon/
>> > > > > > > > > > >> >>
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > > >> --
>> > > > > > > > > > >> Pere Urbon-Bayes
>> > > > > > > > > > >> Software Architect
>> > > > > > > > > > >> http://www.purbon.com
>> > > > > > > > > > >> https://twitter.com/purbon
>> > > > > > > > > > >> https://www.linkedin.com/in/purbon/
>> > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> >
>>
>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Ron Dagostino <rn...@gmail.com>.
Hi Rajini. Thanks. Yes, I’ll update the KIP, and then I’ll start the vote.
Ron
> On Jan 20, 2020, at 6:30 AM, Rajini Sivaram <ra...@gmail.com> wrote:
>
> Hi Ron,
>
> Good point. Keystore and truststore configs (type/location/password) are
> dynamically reconfigurable. A few of them like protocol, enabled.protocols,
> cipher.suites and endpoint.identification.algorithm are not. But to keep it
> simple, it may be better not to inherit any of the SSL configs. Perhaps
> just document that in Rejected Alternatives?
>
>> On Sat, Jan 18, 2020 at 12:43 AM Ron Dagostino <rn...@gmail.com> wrote:
>>
>> Thanks, Rajini. Still not sure what the answer is, but I thought of a
>> couple more issues with config inheritance that I wanted to raise.
>> The first is a minor issue, but just to document it (and I will add it
>> to the KIP as well), ZooKeeper does not support a key password that
>> differs from the keystore password. So inheriting from Kafka's TLS
>> configs would only work if those configs use the same key and keystore
>> password. Again, not a big deal -- just something to document.
>>
>> However...
>>
>> I think there may be a fundamental problem with inheriting broker
>> configs for ZooKeeper access. The problem is that the broker's TLS
>> configs are dynamically reconfigurable, and the dynamic values
>> themselves are stored in ZooKeeper. It would be impossible to know
>> those values without connecting to ZooKeeper, and it would be
>> impossible to connect to ZooKeeper without the values. Given this, I
>> wonder if inheritance is impossible in this particular case.
>>
>> Ron
>>
>>> On Jan 17, 2020, at 5:39 PM, Rajini Sivaram <ra...@gmail.com>
>> wrote:
>>>
>>> Hi Ron,
>>>
>>> For Kafka, we moved from hostname verification disabled-by-default to
>>> enabled-by-default under
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-294+-+Enable+TLS+hostname+verification+by+default
>> .
>>> So we have tested empty String for disabling hostname verification and we
>>> know that it works, not only in test environments, but also in actual
>>> deployments. Basically, "ssl.endpoint.identification.algorithm=" in the
>>> file results in a config value of empty String and not setting it results
>>> in the default value of HTTPS since the value is null.I would probably do
>>> the same for the ZK config. If you are creating certificates for
>> ZooKeeper
>>> without hostnames, then you may be doing the same for Kafka too and in
>> both
>>> cases, the config would be an empty string. This is insecure anyway and
>> we
>>> would really expect this only in test environments. Not sure if it is
>> worth
>>> making an exception for this in terms of consistent config names.
>>>
>>>> On Fri, Jan 17, 2020 at 2:37 PM Ron Dagostino <rn...@gmail.com>
>> wrote:
>>>>
>>>> Hi again, Rajini. I've updated the KIP, and while doing it I became
>>>> concerned about using zookeeper.ssl.endpoint.identification.algorithm
>>>> for enabling/disabling hostname verification. The KIP reflects what
>>>> we decided, but upon reading it, I wonder if it is workable. Here's
>>>> what it says for this config:
>>>>
>>>> "Specifies whether to enable hostname verification in the ZooKeeper
>>>> TLS negotiation process, with (case-insensitively) "https" meaning
>>>> ZooKeeper hostname verification is enabled and an explicit blank value
>>>> meaning it is disabled (disabling it is only recommended for testing
>>>> purposes). Overrides any explicit "true" or "false" value set via the
>>>> <code>zookeeper.ssl.hostnameVerification</code> system property (true
>>>> implying https and false implying blank) and inherits the value of
>>>> <code>ssl.endpoint.identification.algorithm</code> (if any) if no
>>>> explicit value or non-value is set both here and via the system
>>>> property."
>>>>
>>>> I wonder if someone sets an explicit empty value
>>>> (zookeeper.ssl.endpoint.identification.algorithm=) in the config file,
>>>> will we be able to distinguish between that and a config file that
>>>> doesn't have it at all? And even if we can, this feels pretty forced
>>>> to me, like we are bending over backwards for consistency when
>>>> "zookeeper.ssl.hostname.verification.enable" is a natural fit. I
>>>> doubt this config is going to be used very much, so it really does
>>>> feel to me like we should just go with the natural fit and ditch the
>>>> compatibility and inheritance for this one config.
>>>>
>>>> Ron
>>>>
>>>>> On Fri, Jan 17, 2020 at 8:07 AM Ron Dagostino <rn...@gmail.com>
>> wrote:
>>>>>
>>>>> Hi Rajini. I’ll submit a documentation PR to clarify the Kafka
>> behavior
>>>> of #1 and will adopt the same config for ZK. I agree now we should
>> inherit
>>>> for AclAuthorizer too — I was just stuck on the “no inheritance” idea
>> more
>>>> than I realized, and while the ZK quorums are different that doesn’t
>>>> necessarily mean the same client identity wouldn’t be valid on both
>> (and if
>>>> not then the configs can still be overridden).
>>>>>
>>>>> I’ll update the KIP today and let this version soak for a day or two,
>>>> and then I’ll start a vote if no other issues/comments arise.
>>>>>
>>>>> Ron
>>>>>
>>>>>> On Jan 17, 2020, at 6:08 AM, Rajini Sivaram <ra...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>> Hi Ron,
>>>>>>
>>>>>> Unresolved item #1: Yes, Kafka disables hostname verification if
>>>>>> "ssl.endpoint.identification.algorithm" is an empty string.
>>>>>>
>>>>>> Unresolved item #2: Yes, those 9 plus hostname verification. If we do
>>>>>> decide to inherit Kafka configs, wouldn't we inherit these 10 in
>>>>>> AclAuthorizer as well?
>>>>>>
>>>>>> Unresolved item #3: Agree, following up on the discussion thread for
>>>>>> KIP-553.
>>>>>>
>>>>>>
>>>>>>> On Fri, Jan 17, 2020 at 3:29 AM Ron Dagostino <rn...@gmail.com>
>>>> wrote:
>>>>>>>
>>>>>>> Hi Colin. TLS to ZooKeeper will remain an opt-in. The configuration
>>>>>>> value that turns it on is zookeeper.ssl.client.enable (which used to
>>>>>>> be called zookeeper.client.secure before we renamed everything to be
>>>>>>> consistent with Kafka's config naming style and eliminate confusion).
>>>>>>> The default value for this is false. Everything else related to TLS
>>>>>>> has no effect unless this is explicitly set to true, and this value
>>>>>>> does not inherit from anywhere,
>>>>>>>
>>>>>>> Ron
>>>>>>>
>>>>>>>> On Thu, Jan 16, 2020 at 5:39 PM Colin McCabe <cm...@apache.org>
>>>> wrote:
>>>>>>>>
>>>>>>>>> On Thu, Jan 16, 2020, at 02:58, Rajini Sivaram wrote:
>>>>>>>>> Hi Ron/Colin,
>>>>>>>>>
>>>>>>>>> Thanks for the responses. I think we need to consider these cases:
>>>>>>>>>
>>>>>>>>> 1) The recommended approach: This is the one we document. Configs
>>>>>>>>> consistent with Kafka make sense here. It seems a lot more
>>>> confusing to
>>>>>>>>> configure `ssl.cipher.suites`, `ssl.enabled.protocols`, `
>>>>>>>>> zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols`
>>>> with
>>>>>>>>> multiple naming inconsistencies in server.properties. For users who
>>>>>>> have
>>>>>>>>> never used ZK system properties, this is clearly simpler. Another
>>>>>>> config to
>>>>>>>>> consider is `ssl.endpoint.identification.algorithm` which does the
>>>> same
>>>>>>>>> thing as `zookeeper.ssl.hostnameVerification` - if we are staying
>>>>>>>>> consistent, we should use the Kafka format for this too. I would
>>>> also
>>>>>>>>> consider using something like `zookeeper.client.ssl.enabled`
>> instead
>>>>>>> of `
>>>>>>>>> zookeeper.client.secure` since it is difficult to tell what
>> `secure`
>>>>>>> means
>>>>>>>>> when you have options to configure Kerberos, ACLs and/or SSL.
>>>>>>>>>
>>>>>>>>> 2) Inheritance using prefixed configs as we do for other prefixed
>>>>>>> configs
>>>>>>>>> in Kafka: I think it may be useful for configs like `ssl.protocol`
>>>> and
>>>>>>> `
>>>>>>>>> ssl.cipher.suites`.
>>>>>>>>>
>>>>>>>>> 3) Default values: Do we want to use Kafka defaults? The
>>>> inconsistency
>>>>>>> we
>>>>>>>>> have is `ssl.protocol/ssl.enabled.protocols` since we still have
>>>>>>> insecure
>>>>>>>>> protocols enabled for Kafka. We have KIP-553 (
>>>>>>>>>
>>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
>>>>>>> )
>>>>>>>>> open to disable these. It is currently blocked since it also talks
>>>>>>> about
>>>>>>>>> enabling TLSv1.3 by default and we haven't done sufficient testing
>>>> for
>>>>>>>>> that. Since it will be good to disable the older insecure protocols
>>>>>>> anyway,
>>>>>>>>> perhaps we could do that without enabling TLSv1.3 by default in
>> 2.5?
>>>>>>>>
>>>>>>>> Hi Rajini,
>>>>>>>>
>>>>>>>> Inheriting the SSL settings from Kafka would have the effect of
>>>> changing
>>>>>>> the defaults for almost everyone using SSL from non-SSL ZK to
>>>> SSL-based ZK,
>>>>>>> right? This seems to expand the scope of the KIP greatly, since it
>>>> moves
>>>>>>> it from being opt-in to opt-out. Are there any potential negative
>>>>>>> implications of this?
>>>>>>>>
>>>>>>>> best,
>>>>>>>> Colin
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> 4) Backward compatibility for deployments which are using system
>>>>>>>>> properties. We shouldn't override system properties with Kafka
>>>>>>> defaults or
>>>>>>>>> inherited values. But we do want to override if user configures
>>>>>>> properties
>>>>>>>>> explicitly. In the current approach, this was simpler since we just
>>>>>>> had to
>>>>>>>>> set the configured values. But if we decide to inherit Kafka
>>>> configs,
>>>>>>> then
>>>>>>>>> we will need to check system properties and update accordingly. But
>>>>>>> that
>>>>>>>>> should be doable?
>>>>>>>>>
>>>>>>>>> On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe <cm...@apache.org>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote:
>>>>>>>>>>> Hi Colin. Two things come to mind with respect to ZooKeeper
>>>>>>> camelCase
>>>>>>>>>>> style vs Kafka-style config names for ZooKeeper. First, I think
>>>> it
>>>>>>>>>>> would be desirable for the client configs and broker configs to
>> be
>>>>>>>>>>> interoperable. For example, it feels like it would be convenient
>>>>>>> to
>>>>>>>>>>> be able to pass the broker's config file to the ZK Security
>>>>>>> Migrator
>>>>>>>>>>> tool. I think whatever we use (ZooKeeper camelCase style or
>> Kafka
>>>>>>>>>>> style), I think we should use the same for both.
>>>>>>>>>>>
>>>>>>>>>>> The second thing to think about is the fact that ZooKeeper
>>>>>>>>>>> configuration inherently uses Java system properties as a first
>>>>>>> pass.
>>>>>>>>>>> So while we might switch to Kafka-style configs in the config
>> file
>>>>>>>>>>> (e.g. zookeeper.ssl.trust.store.location), the
>>>>>>>>>>> org.apache.zookeeper.client.ZKClientConfig class (which is how
>> TLS
>>>>>>>>>>> configs are set) is going to look for the camelCase
>>>>>>>>>>> zookeeper.ssl.trustStore.location system property and use any
>>>> value
>>>>>>>>>>> there. I think this could be very confusing for people.
>> Granted,
>>>>>>> we
>>>>>>>>>>> are doing this so that people don't have to use system
>> properties,
>>>>>>> but
>>>>>>>>>>> there is no way to turn off the use of system properties, so I
>>>>>>> think
>>>>>>>>>>> there would be pretty substantial potential for confusion.
>>>>>>>>>>
>>>>>>>>>> I don't think we document or recommend that anyone use system
>>>>>>> properties
>>>>>>>>>> to configure Zookeeper usage within Kafka. And I think the
>> reason
>>>>>>> is
>>>>>>>>>> exactly the issue you pointed out -- it wouldn't be consistent
>> with
>>>>>>> our
>>>>>>>>>> other configurations.
>>>>>>>>>>
>>>>>>>>>> best,
>>>>>>>>>> Colin
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> One idea is to migrate the system properties -- i.e. look to see
>>>> if
>>>>>>>>>>> zookeeper.ssl.trustStore.location is set and if it is clear out
>>>> the
>>>>>>>>>>> value there and move it under the
>>>>>>> zookeeper.ssl.trust.store.location
>>>>>>>>>>> system property. (I said it was an idea -- not necessarily a
>> good
>>>>>>> one
>>>>>>>>>>> :-)
>>>>>>>>>>>
>>>>>>>>>>> Ron
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe <cmccabe@apache.org
>>>
>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
>>>>>>>>>>>>> Thanks Colin and Rajini.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've updated the KIP to say that KIP-421 functionality is
>>>>>>> available
>>>>>>>>>> to
>>>>>>>>>>>>> encrypt sensitive configs like the ZK key store and trust store
>>>>>>>>>>>>> passwords. (I've also made it clear that the configs are not
>>>>>>>>>>>>> dynamically reconfigurable since dynamic values are stored in
>>>>>>> ZK and
>>>>>>>>>>>>> the data is needed to get to ZK in the first place). Colin,
>>>>>>> let me
>>>>>>>>>>>>> know if you think anything else needs to be said/done about it
>>>>>>> -- I
>>>>>>>>>>>>> wasn't sure if your comment above implies that there are
>>>>>>> additional
>>>>>>>>>>>>> direct actions that we need to take aside from this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I agree that making the brokers' ZooKeeper clients dynamic with
>>>>>>>>>>>>> respect to key and trust stores (e.g. via
>>>>>>>>>>>>> zookeeper.ssl.context.supplier.class) may increase the risk
>>>>>>> that this
>>>>>>>>>>>>> KIP may not land in time for 2.5.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regarding the config names for ZK being different than the
>>>>>>> ZooKeeper
>>>>>>>>>>>>> configs (e.g. camel-case keyStore/trustStore instead of
>>>>>>>>>>>>> keystore/truststore, ciphersuites instead of cipher.suites,
>>>>>>>>>>>>> enabledProtocols instead of enabled.protocols), I agree it is
>>>>>>> an
>>>>>>>>>>>>> issue. I tried to mitigate it by putting warning text in the
>>>>>>> config
>>>>>>>>>>>>> docs for these. Regarding configuration inheritance, I think
>>>>>>> what
>>>>>>>>>> you
>>>>>>>>>>>>> are saying is that for any configs where the broker has a clear
>>>>>>>>>>>>> semantic equivalent, the broker could fall back to the broker
>>>>>>> value
>>>>>>>>>> if
>>>>>>>>>>>>> the ZK value is not given. The potential list is:
>>>>>>>>>>>>>
>>>>>>>>>>>>> zookeeper.ssl.keyStore.location => ssl.keystore.location
>>>>>>>>>>>>> zookeeper.ssl.keyStore.password => ssl.keystore.password
>>>>>>>>>>>>> zookeeper.ssl.keyStore.type => ssl.keystore.type
>>>>>>>>>>>>> zookeeper.ssl.trustStore.location => ssl.truststore.location
>>>>>>>>>>>>> zookeeper.ssl.trustStore.password => ssl.truststore.password
>>>>>>>>>>>>> zookeeper.ssl.trustStore.type => ssl.truststore.type
>>>>>>>>>>>>> zookeeper.ssl.ciphersuites => ssl.cipher.suites
>>>>>>>>>>>>>
>>>>>>>>>>>>> Note that there are two configs that look the same based on
>>>>>>> their key
>>>>>>>>>>>>> names but whose allowable values differ:
>>>>>>>>>>>>>
>>>>>>>>>>>>> zookeeper.ssl.protocol(Default: TLSv1.2) =>
>>>>>>> ssl.protocol(Default:
>>>>>>>>>> TLS)
>>>>>>>>>>>>> zookeeper.ssl.enabledProtocols(Default: value of protocol
>>>>>>> property) =
>>>>>>>>>>>>> ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Not sure what the right answer is, but hopefully this detail
>>>>>>> will
>>>>>>>>>> help
>>>>>>>>>>>>> get us there.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ron
>>>>>>>>>>>>
>>>>>>>>>>>> I think, on balance, I agree with Rajini that it would be nice
>>>>>>> to make
>>>>>>>>>> the configs look more like Kafka configs. We don't have
>> camel-case
>>>>>>> in
>>>>>>>>>> configuration keys, so we should avoid it here.
>>>>>>>>>>>>
>>>>>>>>>>>> best,
>>>>>>>>>>>> Colin
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <
>>>>>>> cmccabe@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
>>>>>>>>>>>>>>> Hi Ron,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the detailed explanation, sounds good to me.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> A few more questions:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1) At the moment, all sensitive broker configs including
>>>>>>>>>>>>>>> keystore/truststore passwords can be stored encrypted in
>>>>>>>>>> ZooKeeper prior to
>>>>>>>>>>>>>>> starting up brokers. We are now adding SSL
>>>>>>> keystore/truststore
>>>>>>>>>> passwords
>>>>>>>>>>>>>>> for ZooKeeper client that cannot be stored in ZooKeeper
>>>>>>> since
>>>>>>>>>> you need
>>>>>>>>>>>>>>> these to connect to ZK. We should perhaps document that
>>>>>>> these
>>>>>>>>>> configs can
>>>>>>>>>>>>>>> be encrypted using KIP-421.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That's a good point. Previously, we didn't have ZK
>>>>>>> on-the-wire
>>>>>>>>>> security support. However, now that we do, sending sensitive
>>>>>>> keystore and
>>>>>>>>>> truststore passwords to ZK in cleartext should be deprecated in
>>>>>>> favor of
>>>>>>>>>> using KIP-421.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2) We can dynamically update keystores and trust stores
>>>>>>> used by
>>>>>>>>>> brokers
>>>>>>>>>>>>>>> without restarting the broker. Can we support this easily
>>>>>>> for ZK
>>>>>>>>>> clients
>>>>>>>>>>>>>>> created by the broker, for example by adding our own
>>>>>>>>>>>>>>> `zookeeper.ssl.context.supplier.class`?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hmm. That might be better to consider in a follow-up, since
>>>>>>> the
>>>>>>>>>> deadline for 2.5 KIPs is coming up?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> best,
>>>>>>>>>>>>>> Colin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 3) Looks like we are using config names that directly map
>>>>>>> to ZK
>>>>>>>>>> configs.
>>>>>>>>>>>>>>> Have we considered using equivalent Kafka config names with
>>>>>>>>>> prefixes,
>>>>>>>>>>>>>>> perhaps with inheritance from the non-prefixed names? Not
>>>>>>> sure
>>>>>>>>>> if this is a
>>>>>>>>>>>>>>> good idea, but perhaps worth documenting in Rejected
>>>>>>>>>> Alternatives at least.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <
>>>>>>> cmccabe@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Ron,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for the explanation. I guess thinking about it a
>>>>>>>>>> little bit more,
>>>>>>>>>>>>>>>> we should just add --zk-tls-config-file to all of these
>>>>>>>>>> commands.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We will be removing this option (plus ZK in general) from
>>>>>>>>>> these commands
>>>>>>>>>>>>>>>> in the next major release, but ZK is still supported in
>>>>>>> 2.5,
>>>>>>>>>> so we should
>>>>>>>>>>>>>>>> just do the logical thing. (The exception is
>>>>>>>>>> ZkSecurityMigrator, which
>>>>>>>>>>>>>>>> will stay).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> best,
>>>>>>>>>>>>>>>> Colin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
>>>>>>>>>>>>>>>>> Hi Colin.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> <<< It seems like this [--zk-tls-config-file
>>>>>>> information]
>>>>>>>>>> could just
>>>>>>>>>>>>>>>> appear
>>>>>>>>>>>>>>>>> in a configuration file, which all of these tools
>>>>>>> already
>>>>>>>>>> accept (I
>>>>>>>>>>>>>>>> think)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ZkSecurityMigrator has no such property file facility;
>>>>>>>>>> adding a
>>>>>>>>>>>>>>>>> "--zk-tls-config-file" parameter is exactly for this
>>>>>>>>>> purpose. If we add
>>>>>>>>>>>>>>>>> that to ZkSecurityMigrator then it is trivial to add
>>>>>>> it to
>>>>>>>>>> other commands
>>>>>>>>>>>>>>>>> (the same code is simply reused; it ends up being just
>>>>>>> a few
>>>>>>>>>> extra
>>>>>>>>>>>>>>>> lines).
>>>>>>>>>>>>>>>>> I do not see any parameter in the other two commands to
>>>>>>>>>> adjust the ZK
>>>>>>>>>>>>>>>>> connection; ConfigCommand accepts a "--command-config"
>>>>>>> flag,
>>>>>>>>>> but
>>>>>>>>>>>>>>>> according
>>>>>>>>>>>>>>>>> to the code "This is used only with --bootstrap-server
>>>>>>>>>> option for
>>>>>>>>>>>>>>>>> describing and altering broker configs."
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I do agree there would be no need to add
>>>>>>>>>> "--zk-tls-config-file" to
>>>>>>>>>>>>>>>>> ReassignPartitionsCommand if its direct ZK
>>>>>>> connectivity is
>>>>>>>>>> replaced in
>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>> for the next release.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ConfigCommand supports the "--bootstrap-server" option
>>>>>>> and
>>>>>>>>>> will have its
>>>>>>>>>>>>>>>>> direct ZooKeeper access formally deprecated as per
>>>>>>> KIP-555,
>>>>>>>>>> but the
>>>>>>>>>>>>>>>> special
>>>>>>>>>>>>>>>>> use case of bootstrapping a ZooKeeper ensemble with
>>>>>>>>>> encrypted credentials
>>>>>>>>>>>>>>>>> prior to starting Kafka will still exist, so it feels
>>>>>>> like
>>>>>>>>>> while
>>>>>>>>>>>>>>>>> "--zk-tls-config-file" would never be used except for
>>>>>>> this
>>>>>>>>>> use case it
>>>>>>>>>>>>>>>>> could probably still be added for this particular
>>>>>>> situation.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ron
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> P.S. I responded on 1/6 but I just discovered that e,
>>>>>>> ail
>>>>>>>>>> (and 3 more I
>>>>>>>>>>>>>>>>> sent) did not go through. I am trying to get emails
>>>>>>> through
>>>>>>>>>> now to move
>>>>>>>>>>>>>>>>> this discussion forward.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <
>>>>>>>>>> cmccabe@apache.org> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
>>>>>>>>>>>>>>>>>>> Hi everyone. I would like to make the following
>>>>>>> changes
>>>>>>>>>> to the KIP.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> MOTIVATION:
>>>>>>>>>>>>>>>>>>> Include a statement that it will be difficult in
>>>>>>> the
>>>>>>>>>> short term to
>>>>>>>>>>>>>>>>>>> deprecate direct Zookeeper communication in
>>>>>>>>>> kafka-configs.{sh, bat}
>>>>>>>>>>>>>>>>>> (which
>>>>>>>>>>>>>>>>>>> invoke kafka.admin.ConfigCommand) because
>>>>>>> bootstrapping
>>>>>>>>>> a Kafka
>>>>>>>>>>>>>>>> cluster
>>>>>>>>>>>>>>>>>>> with encrypted passwords in Zookeeper is an
>>>>>>>>>> explicitly-supported use
>>>>>>>>>>>>>>>>>> case;
>>>>>>>>>>>>>>>>>>> therefore it is in scope to be able to securely
>>>>>>>>>> configure the CLI
>>>>>>>>>>>>>>>> tools
>>>>>>>>>>>>>>>>>>> that still leverage non-deprecated direct Zookeeper
>>>>>>>>>> communication
>>>>>>>>>>>>>>>> for TLS
>>>>>>>>>>>>>>>>>>> (the other 2 tools are
>>>>>>> kafka-reassign-partitions.{sh,
>>>>>>>>>> bat} and
>>>>>>>>>>>>>>>>>>> zookeeper-security-migration.sh).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Ron,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> About deprecations:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * zookeeper-security-migration: clearly, deprecating
>>>>>>> ZK
>>>>>>>>>> access in this
>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>> would not make sense, since it would defeat the whole
>>>>>>>>>> point of the
>>>>>>>>>>>>>>>> tool :)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * kafka-reassign-partitions: ZK access should be
>>>>>>>>>> deprecated here.
>>>>>>>>>>>>>>>> KIP-455
>>>>>>>>>>>>>>>>>> implementation has dragged a bit, but this should
>>>>>>> get done
>>>>>>>>>> soon.
>>>>>>>>>>>>>>>> Certainly
>>>>>>>>>>>>>>>>>> before the next release.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * kafka-configs: I think ZK access should be
>>>>>>> deprecated
>>>>>>>>>> here as well.
>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>> agree there is a super-special bootstrapping case
>>>>>>> here,
>>>>>>>>>> but that should
>>>>>>>>>>>>>>>>>> have its own tool, not use kafka-configs.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I will post a separate KIP for this, though.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> GOALS:
>>>>>>>>>>>>>>>>>>> Support the secure configuration of TLS-encrypted
>>>>>>>>>> communication
>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>>>> Zookeeper and:
>>>>>>>>>>>>>>>>>>> a) Kafka brokers
>>>>>>>>>>>>>>>>>>> b) The three CLI tools mentioned above that still
>>>>>>>>>> support direct,
>>>>>>>>>>>>>>>>>>> non-deprecated communication to Zookeeper
>>>>>>>>>>>>>>>>>>> It is explicitly out-of-scope to deprecate any
>>>>>>> direct
>>>>>>>>>> Zookeeper
>>>>>>>>>>>>>>>>>>> communication in CLI tools as part of this KIP;
>>>>>>> such
>>>>>>>>>> work will occur
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> future KIPs instead.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> PUBLIC INTERFACES:
>>>>>>>>>>>>>>>>>>> 1) The following new broker configurations will be
>>>>>>>>>> recognized.
>>>>>>>>>>>>>>>>>>> zookeeper.client.secure (default value = false,
>>>>>>> for
>>>>>>>>>> backwards
>>>>>>>>>>>>>>>>>>> compatibility)
>>>>>>>>>>>>>>>>>>> zookeeper.clientCnxnSocket
>>>>>>>>>>>>>>>>>>> zookeeper.ssl.keyStore.location
>>>>>>>>>>>>>>>>>>> zookeeper.ssl.keyStore.password
>>>>>>>>>>>>>>>>>>> zookeeper.ssl.trustStore.location
>>>>>>>>>>>>>>>>>>> zookeeper.ssl.trustStore.password
>>>>>>>>>>>>>>>>>>> It will be an error for any of the last 5 values
>>>>>>> to be
>>>>>>>>>> left
>>>>>>>>>>>>>>>> unspecified
>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>> zookeeper.client.secure is explicitly set to true.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2) In addition, the
>>>>>>>>>> kafka.security.authorizer.AclAuthorizer class
>>>>>>>>>>>>>>>>>> supports
>>>>>>>>>>>>>>>>>>> the ability to connect to a different Zookeeper
>>>>>>> instance
>>>>>>>>>> than the
>>>>>>>>>>>>>>>> one the
>>>>>>>>>>>>>>>>>>> brokers use. We therefore also add the following
>>>>>>>>>> optional configs,
>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>> override the corresponding ones from above when
>>>>>>> present:
>>>>>>>>>>>>>>>>>>> authorizer.zookeeper.client.secure
>>>>>>>>>>>>>>>>>>> authorizer.zookeeper.clientCnxnSocket
>>>>>>>>>>>>>>>>>>> authorizer.zookeeper.ssl.keyStore.location
>>>>>>>>>>>>>>>>>>> authorizer.zookeeper.ssl.keyStore.password
>>>>>>>>>>>>>>>>>>> authorizer.zookeeper.ssl.trustStore.location
>>>>>>>>>>>>>>>>>>> authorizer.zookeeper.ssl.trustStore.password
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 3) The three CLI tools mentioned above will
>>>>>>> support a new
>>>>>>>>>>>>>>>>>> --zk-tls-config-file
>>>>>>>>>>>>>>>>>>> <String: Zookeeper TLS configuration file path>"
>>>>>>>>>> option. The
>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>> properties will be recognized in that file, and
>>>>>>>>>> unrecognized
>>>>>>>>>>>>>>>> properties
>>>>>>>>>>>>>>>>>>> will be ignored to allow the possibility of
>>>>>>> pointing
>>>>>>>>>>>>>>>> zk-tls-config-file
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> the broker's config file.
>>>>>>>>>>>>>>>>>>> zookeeper.client.secure (default value = false)
>>>>>>>>>>>>>>>>>>> zookeeper.clientCnxnSocket
>>>>>>>>>>>>>>>>>>> zookeeper.ssl.keyStore.location
>>>>>>>>>>>>>>>>>>> zookeeper.ssl.keyStore.password
>>>>>>>>>>>>>>>>>>> zookeeper.ssl.trustStore.location
>>>>>>>>>>>>>>>>>>> zookeeper.ssl.trustStore.password
>>>>>>>>>>>>>>>>>>> It will be an error for any of the last 5 values
>>>>>>> to be
>>>>>>>>>> left
>>>>>>>>>>>>>>>> unspecified
>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>> zookeeper.client.secure is explicitly set to true.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Do we really need a --zk-tls-config-file flag? It
>>>>>>> seems
>>>>>>>>>> like this
>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>> just appear in a configuration file, which all of
>>>>>>> these
>>>>>>>>>> tools already
>>>>>>>>>>>>>>>>>> accept (I think).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> best,
>>>>>>>>>>>>>>>>>> Colin
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Ron
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <
>>>>>>>>>> rndgstn@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi everyone. Let's get this discussion going
>>>>>>> again
>>>>>>>>>> now that Kafka
>>>>>>>>>>>>>>>> 2.4
>>>>>>>>>>>>>>>>>>>> with Zookeeper 3.5.6 is out.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> First, regarding the KIP number, the other KIP
>>>>>>> that
>>>>>>>>>> was using this
>>>>>>>>>>>>>>>>>> number
>>>>>>>>>>>>>>>>>>>> moved to KIP 534, so KIP 515 remains the correct
>>>>>>>>>> number for this
>>>>>>>>>>>>>>>>>>>> discussion. I've updated the Kafka Improvement
>>>>>>>>>> Proposal page to
>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> KIP in the 515 slot, so we're all set there.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regarding the actual issues under discussion, I
>>>>>>> think
>>>>>>>>>> there are
>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>> things we should clarify.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 1) It is possible to use TLS connectivity to
>>>>>>> Zookeeper
>>>>>>>>>> from Apache
>>>>>>>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>>>>>>> 2.4 -- the problem is that configuration
>>>>>>> information
>>>>>>>>>> has to be
>>>>>>>>>>>>>>>> passed
>>>>>>>>>>>>>>>>>> via
>>>>>>>>>>>>>>>>>>>> system properties as "-D" command line options
>>>>>>> on the
>>>>>>>>>> java
>>>>>>>>>>>>>>>> invocation
>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>> the broker, and those are not secure (anyone with
>>>>>>>>>> access to the box
>>>>>>>>>>>>>>>>>> can see
>>>>>>>>>>>>>>>>>>>> the command line used to invoke the broker); the
>>>>>>>>>> configuration
>>>>>>>>>>>>>>>> includes
>>>>>>>>>>>>>>>>>>>> sensitive information (e.g. a keystore
>>>>>>> password), so
>>>>>>>>>> we need a
>>>>>>>>>>>>>>>> secure
>>>>>>>>>>>>>>>>>>>> mechanism for passing the configuration values.
>>>>>>> I
>>>>>>>>>> believe the real
>>>>>>>>>>>>>>>>>>>> motivation for this KIP is to harden the
>>>>>>> configuration
>>>>>>>>>> mechanism
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>> Zookeeper TLS connectivity.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2) I believe the list of CLI tools that continue
>>>>>>> to
>>>>>>>>>> use direct
>>>>>>>>>>>>>>>>>> Zookeeper
>>>>>>>>>>>>>>>>>>>> connectivity in a non-deprecated fashion is:
>>>>>>>>>>>>>>>>>>>> a)
>>>>>>>>>>>>>>>>
>>>>>>> zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
>>>>>>>>>>>>>>>>>>>> b)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
>>>>>>>>>>>>>>>>>>>> c) kafka-configs.{sh,
>>>>>>> bat}/kafka.admin.ConfigCommand
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 3) I believe Kafka.admin.ConfigCommand presents a
>>>>>>>>>> conundrum
>>>>>>>>>>>>>>>> because it
>>>>>>>>>>>>>>>>>>>> explicitly states in a comment that a supported
>>>>>>> use
>>>>>>>>>> case is
>>>>>>>>>>>>>>>>>> bootstrapping a
>>>>>>>>>>>>>>>>>>>> Kafka cluster with encrypted passwords in
>>>>>>> Zookeeper
>>>>>>>>>> (see
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>
>> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
>>>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>>>>> This means it will be especially difficult to
>>>>>>>>>> deprecate this
>>>>>>>>>>>>>>>> particular
>>>>>>>>>>>>>>>>>>>> direct Zookeeper connectivity without a different
>>>>>>>>>> storage
>>>>>>>>>>>>>>>> mechanism for
>>>>>>>>>>>>>>>>>>>> dynamic configuration values being available
>>>>>>> (i.e. the
>>>>>>>>>> self-managed
>>>>>>>>>>>>>>>>>> quorum
>>>>>>>>>>>>>>>>>>>> referred to in KIP-500).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I think it would be easier and simpler to harden
>>>>>>> the
>>>>>>>>>> Zookeeper TLS
>>>>>>>>>>>>>>>>>>>> configuration in both Kafka and in CLI tools
>>>>>>> compared
>>>>>>>>>> to trying to
>>>>>>>>>>>>>>>>>>>> deprecate the direct Zookeeper connectivity in
>>>>>>> the
>>>>>>>>>> above 3 CLI
>>>>>>>>>>>>>>>> tools --
>>>>>>>>>>>>>>>>>>>> especially when ConfigCommand has no obvious
>>>>>>>>>> short-term path to
>>>>>>>>>>>>>>>>>> deprecation.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regarding how to harden the TLS configuration,
>>>>>>> adding
>>>>>>>>>> the
>>>>>>>>>>>>>>>> appropriate
>>>>>>>>>>>>>>>>>>>> configs to Kafka is an obvious choice for the
>>>>>>>>>> brokers. Not that
>>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>>> values cannot be dynamically reconfigurable
>>>>>>> because
>>>>>>>>>> the values
>>>>>>>>>>>>>>>> would be
>>>>>>>>>>>>>>>>>>>> required to connect to Zookeeper via TLS in the
>>>>>>> first
>>>>>>>>>> place.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regarding how to harden the TLS configuration
>>>>>>> for the
>>>>>>>>>> 3 remaining
>>>>>>>>>>>>>>>> CLI
>>>>>>>>>>>>>>>>>>>> tools, it would be relatively straightforward to
>>>>>>> add
>>>>>>>>>> support for a
>>>>>>>>>>>>>>>>>>>> "--zk-tls-config-file <String: Zookeeper TLS
>>>>>>>>>> configuration system
>>>>>>>>>>>>>>>>>>>> properties file path>" option to each one.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Ron
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes
>>>>>>> <
>>>>>>>>>>>>>>>> pere.urbon@gmail.com
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> True, the number is duplicated.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> do you know how can we get the problem fix? I
>>>>>>> was
>>>>>>>>>> not aware, I
>>>>>>>>>>>>>>>>>> missed,
>>>>>>>>>>>>>>>>>>>>> sorry the need to add the KIP to the table.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -- Pere
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Missatge de Jorge Esteban Quilcate Otoya <
>>>>>>>>>>>>>>>> quilcate.jorge@gmail.com>
>>>>>>>>>>>>>>>>>> del
>>>>>>>>>>>>>>>>>>>>> dia
>>>>>>>>>>>>>>>>>>>>> dt., 3 de set. 2019 a les 13:43:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi Pere,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Have you add your KIP to the list here
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>>>>>>>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I found the KIP number assigned to another.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón
>>>>>>> Bayes <
>>>>>>>>>>>>>>>>>> pere.urbon@gmail.com>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks for your time Harsha,
>>>>>>>>>>>>>>>>>>>>>>> anyone else with comments? looking
>>>>>>> forward to
>>>>>>>>>> hearing from
>>>>>>>>>>>>>>>> you.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Stupid question: when do you move from
>>>>>>> discussion
>>>>>>>>>> to vote?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Missatge de Harsha Chintalapani <
>>>>>>> kafka@harsha.io>
>>>>>>>>>> del dia
>>>>>>>>>>>>>>>> dv., 30
>>>>>>>>>>>>>>>>>>>>> d’ag.
>>>>>>>>>>>>>>>>>>>>>>> 2019 a les 21:59:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks Pere. KIP looks good to me.
>>>>>>>>>>>>>>>>>>>>>>>> -Harsha
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 30, 2019 at 10:05 AM, Pere
>>>>>>> Urbón
>>>>>>>>>> Bayes <
>>>>>>>>>>>>>>>>>>>>>>> pere.urbon@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Not really,
>>>>>>>>>>>>>>>>>>>>>>>>> my idea is to keep the JAAS parameter,
>>>>>>> so
>>>>>>>>>> people don't see
>>>>>>>>>>>>>>>>>> major
>>>>>>>>>>>>>>>>>>>>>>>>> changes. But if you pass a properties
>>>>>>> file,
>>>>>>>>>> then this takes
>>>>>>>>>>>>>>>>>>>>> precedence
>>>>>>>>>>>>>>>>>>>>>>> over
>>>>>>>>>>>>>>>>>>>>>>>>> the other, with the idea that you can do
>>>>>>> sasl
>>>>>>>>>> as well with
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> properties
>>>>>>>>>>>>>>>>>>>>>>>>> files.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Makes sense?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> -- Pere
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Missatge de Harsha Chintalapani <
>>>>>>>>>> kafka@harsha.io> del dia
>>>>>>>>>>>>>>>> dv.,
>>>>>>>>>>>>>>>>>> 30
>>>>>>>>>>>>>>>>>>>>>>> d’ag.
>>>>>>>>>>>>>>>>>>>>>>>>> 2019 a les 19:00:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Pere,
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. Enabling
>>>>>>> SSL for
>>>>>>>>>> zookeeper
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>>>>>>>>>> makes
>>>>>>>>>>>>>>>>>>>>>>>>>> sense.
>>>>>>>>>>>>>>>>>>>>>>>>>> "The changes are planned to be
>>>>>>> introduced in a
>>>>>>>>>> compatible
>>>>>>>>>>>>>>>> way,
>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>> keeping the current JAAS variable
>>>>>>> precedence."
>>>>>>>>>>>>>>>>>>>>>>>>>> Can you elaborate a bit here. If the user
>>>>>>>>>> configures a JAAS
>>>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>> Client section it will take precedence
>>>>>>> over
>>>>>>>>>> zookeeper SSL
>>>>>>>>>>>>>>>>>> configs?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> Harsha
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 30, 2019 at 7:50 AM, Pere
>>>>>>> Urbón
>>>>>>>>>> Bayes <
>>>>>>>>>>>>>>>>>>>>>>> pere.urbon@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>> quick question, I saw in another mail
>>>>>>> that
>>>>>>>>>> 2.4 release is
>>>>>>>>>>>>>>>>>> planned
>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>> September. I think it would be really
>>>>>>> awesome
>>>>>>>>>> to have
>>>>>>>>>>>>>>>> this for
>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>> release, do you think we can make it?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> -- Pere
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Missatge de Pere Urbón Bayes <
>>>>>>>>>> pere.urbon@gmail.com> del
>>>>>>>>>>>>>>>> dia
>>>>>>>>>>>>>>>>>> dj.,
>>>>>>>>>>>>>>>>>>>>> 29
>>>>>>>>>>>>>>>>>>>>>>>>>>> d’ag. 2019 a les 20:10:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>> this is my first KIP for a change in
>>>>>>> Apache
>>>>>>>>>> Kafka, so I'm
>>>>>>>>>>>>>>>>>> really
>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>>>>>> to the process. Looking forward to
>>>>>>> hearing
>>>>>>>>>> from you and
>>>>>>>>>>>>>>>> learn
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> best
>>>>>>>>>>>>>>>>>>>>>>>>>>> ropes here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to propose this KIP-515 to
>>>>>>>>>> enable the
>>>>>>>>>>>>>>>>>>>>> ZookeeperClients
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>> take full advantage of the TLS
>>>>>>> communication
>>>>>>>>>> in the new
>>>>>>>>>>>>>>>>>> Zookeeper
>>>>>>>>>>>>>>>>>>>>>>> 3.5.5.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Specially interesting it the Zookeeper
>>>>>>>>>> Security Migration,
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>>>>>>>>>>>>> this change will not work with TLS,
>>>>>>> disabling
>>>>>>>>>> users to use
>>>>>>>>>>>>>>>>>> ACLs
>>>>>>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> Zookeeper cluster use TLS.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> link:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to hearing from you on
>>>>>>> this,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> /cheers
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>>>>>>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>>>>>>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>>>>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
>>>>>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
>>>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>>>
>>
>>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Ron,
Good point. Keystore and truststore configs (type/location/password) are
dynamically reconfigurable. A few of them like protocol, enabled.protocols,
cipher.suites and endpoint.identification.algorithm are not. But to keep it
simple, it may be better not to inherit any of the SSL configs. Perhaps
just document that in Rejected Alternatives?
On Sat, Jan 18, 2020 at 12:43 AM Ron Dagostino <rn...@gmail.com> wrote:
> Thanks, Rajini. Still not sure what the answer is, but I thought of a
> couple more issues with config inheritance that I wanted to raise.
> The first is a minor issue, but just to document it (and I will add it
> to the KIP as well), ZooKeeper does not support a key password that
> differs from the keystore password. So inheriting from Kafka's TLS
> configs would only work if those configs use the same key and keystore
> password. Again, not a big deal -- just something to document.
>
> However...
>
> I think there may be a fundamental problem with inheriting broker
> configs for ZooKeeper access. The problem is that the broker's TLS
> configs are dynamically reconfigurable, and the dynamic values
> themselves are stored in ZooKeeper. It would be impossible to know
> those values without connecting to ZooKeeper, and it would be
> impossible to connect to ZooKeeper without the values. Given this, I
> wonder if inheritance is impossible in this particular case.
>
> Ron
>
> > On Jan 17, 2020, at 5:39 PM, Rajini Sivaram <ra...@gmail.com>
> wrote:
> >
> > Hi Ron,
> >
> > For Kafka, we moved from hostname verification disabled-by-default to
> > enabled-by-default under
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-294+-+Enable+TLS+hostname+verification+by+default
> .
> > So we have tested empty String for disabling hostname verification and we
> > know that it works, not only in test environments, but also in actual
> > deployments. Basically, "ssl.endpoint.identification.algorithm=" in the
> > file results in a config value of empty String and not setting it results
> > in the default value of HTTPS since the value is null.I would probably do
> > the same for the ZK config. If you are creating certificates for
> ZooKeeper
> > without hostnames, then you may be doing the same for Kafka too and in
> both
> > cases, the config would be an empty string. This is insecure anyway and
> we
> > would really expect this only in test environments. Not sure if it is
> worth
> > making an exception for this in terms of consistent config names.
> >
> >> On Fri, Jan 17, 2020 at 2:37 PM Ron Dagostino <rn...@gmail.com>
> wrote:
> >>
> >> Hi again, Rajini. I've updated the KIP, and while doing it I became
> >> concerned about using zookeeper.ssl.endpoint.identification.algorithm
> >> for enabling/disabling hostname verification. The KIP reflects what
> >> we decided, but upon reading it, I wonder if it is workable. Here's
> >> what it says for this config:
> >>
> >> "Specifies whether to enable hostname verification in the ZooKeeper
> >> TLS negotiation process, with (case-insensitively) "https" meaning
> >> ZooKeeper hostname verification is enabled and an explicit blank value
> >> meaning it is disabled (disabling it is only recommended for testing
> >> purposes). Overrides any explicit "true" or "false" value set via the
> >> <code>zookeeper.ssl.hostnameVerification</code> system property (true
> >> implying https and false implying blank) and inherits the value of
> >> <code>ssl.endpoint.identification.algorithm</code> (if any) if no
> >> explicit value or non-value is set both here and via the system
> >> property."
> >>
> >> I wonder if someone sets an explicit empty value
> >> (zookeeper.ssl.endpoint.identification.algorithm=) in the config file,
> >> will we be able to distinguish between that and a config file that
> >> doesn't have it at all? And even if we can, this feels pretty forced
> >> to me, like we are bending over backwards for consistency when
> >> "zookeeper.ssl.hostname.verification.enable" is a natural fit. I
> >> doubt this config is going to be used very much, so it really does
> >> feel to me like we should just go with the natural fit and ditch the
> >> compatibility and inheritance for this one config.
> >>
> >> Ron
> >>
> >>> On Fri, Jan 17, 2020 at 8:07 AM Ron Dagostino <rn...@gmail.com>
> wrote:
> >>>
> >>> Hi Rajini. I’ll submit a documentation PR to clarify the Kafka
> behavior
> >> of #1 and will adopt the same config for ZK. I agree now we should
> inherit
> >> for AclAuthorizer too — I was just stuck on the “no inheritance” idea
> more
> >> than I realized, and while the ZK quorums are different that doesn’t
> >> necessarily mean the same client identity wouldn’t be valid on both
> (and if
> >> not then the configs can still be overridden).
> >>>
> >>> I’ll update the KIP today and let this version soak for a day or two,
> >> and then I’ll start a vote if no other issues/comments arise.
> >>>
> >>> Ron
> >>>
> >>>> On Jan 17, 2020, at 6:08 AM, Rajini Sivaram <ra...@gmail.com>
> >> wrote:
> >>>>
> >>>> Hi Ron,
> >>>>
> >>>> Unresolved item #1: Yes, Kafka disables hostname verification if
> >>>> "ssl.endpoint.identification.algorithm" is an empty string.
> >>>>
> >>>> Unresolved item #2: Yes, those 9 plus hostname verification. If we do
> >>>> decide to inherit Kafka configs, wouldn't we inherit these 10 in
> >>>> AclAuthorizer as well?
> >>>>
> >>>> Unresolved item #3: Agree, following up on the discussion thread for
> >>>> KIP-553.
> >>>>
> >>>>
> >>>>> On Fri, Jan 17, 2020 at 3:29 AM Ron Dagostino <rn...@gmail.com>
> >> wrote:
> >>>>>
> >>>>> Hi Colin. TLS to ZooKeeper will remain an opt-in. The configuration
> >>>>> value that turns it on is zookeeper.ssl.client.enable (which used to
> >>>>> be called zookeeper.client.secure before we renamed everything to be
> >>>>> consistent with Kafka's config naming style and eliminate confusion).
> >>>>> The default value for this is false. Everything else related to TLS
> >>>>> has no effect unless this is explicitly set to true, and this value
> >>>>> does not inherit from anywhere,
> >>>>>
> >>>>> Ron
> >>>>>
> >>>>>> On Thu, Jan 16, 2020 at 5:39 PM Colin McCabe <cm...@apache.org>
> >> wrote:
> >>>>>>
> >>>>>>> On Thu, Jan 16, 2020, at 02:58, Rajini Sivaram wrote:
> >>>>>>> Hi Ron/Colin,
> >>>>>>>
> >>>>>>> Thanks for the responses. I think we need to consider these cases:
> >>>>>>>
> >>>>>>> 1) The recommended approach: This is the one we document. Configs
> >>>>>>> consistent with Kafka make sense here. It seems a lot more
> >> confusing to
> >>>>>>> configure `ssl.cipher.suites`, `ssl.enabled.protocols`, `
> >>>>>>> zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols`
> >> with
> >>>>>>> multiple naming inconsistencies in server.properties. For users who
> >>>>> have
> >>>>>>> never used ZK system properties, this is clearly simpler. Another
> >>>>> config to
> >>>>>>> consider is `ssl.endpoint.identification.algorithm` which does the
> >> same
> >>>>>>> thing as `zookeeper.ssl.hostnameVerification` - if we are staying
> >>>>>>> consistent, we should use the Kafka format for this too. I would
> >> also
> >>>>>>> consider using something like `zookeeper.client.ssl.enabled`
> instead
> >>>>> of `
> >>>>>>> zookeeper.client.secure` since it is difficult to tell what
> `secure`
> >>>>> means
> >>>>>>> when you have options to configure Kerberos, ACLs and/or SSL.
> >>>>>>>
> >>>>>>> 2) Inheritance using prefixed configs as we do for other prefixed
> >>>>> configs
> >>>>>>> in Kafka: I think it may be useful for configs like `ssl.protocol`
> >> and
> >>>>> `
> >>>>>>> ssl.cipher.suites`.
> >>>>>>>
> >>>>>>> 3) Default values: Do we want to use Kafka defaults? The
> >> inconsistency
> >>>>> we
> >>>>>>> have is `ssl.protocol/ssl.enabled.protocols` since we still have
> >>>>> insecure
> >>>>>>> protocols enabled for Kafka. We have KIP-553 (
> >>>>>>>
> >>>>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
> >>>>> )
> >>>>>>> open to disable these. It is currently blocked since it also talks
> >>>>> about
> >>>>>>> enabling TLSv1.3 by default and we haven't done sufficient testing
> >> for
> >>>>>>> that. Since it will be good to disable the older insecure protocols
> >>>>> anyway,
> >>>>>>> perhaps we could do that without enabling TLSv1.3 by default in
> 2.5?
> >>>>>>
> >>>>>> Hi Rajini,
> >>>>>>
> >>>>>> Inheriting the SSL settings from Kafka would have the effect of
> >> changing
> >>>>> the defaults for almost everyone using SSL from non-SSL ZK to
> >> SSL-based ZK,
> >>>>> right? This seems to expand the scope of the KIP greatly, since it
> >> moves
> >>>>> it from being opt-in to opt-out. Are there any potential negative
> >>>>> implications of this?
> >>>>>>
> >>>>>> best,
> >>>>>> Colin
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> 4) Backward compatibility for deployments which are using system
> >>>>>>> properties. We shouldn't override system properties with Kafka
> >>>>> defaults or
> >>>>>>> inherited values. But we do want to override if user configures
> >>>>> properties
> >>>>>>> explicitly. In the current approach, this was simpler since we just
> >>>>> had to
> >>>>>>> set the configured values. But if we decide to inherit Kafka
> >> configs,
> >>>>> then
> >>>>>>> we will need to check system properties and update accordingly. But
> >>>>> that
> >>>>>>> should be doable?
> >>>>>>>
> >>>>>>> On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe <cm...@apache.org>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>>> On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote:
> >>>>>>>>> Hi Colin. Two things come to mind with respect to ZooKeeper
> >>>>> camelCase
> >>>>>>>>> style vs Kafka-style config names for ZooKeeper. First, I think
> >> it
> >>>>>>>>> would be desirable for the client configs and broker configs to
> be
> >>>>>>>>> interoperable. For example, it feels like it would be convenient
> >>>>> to
> >>>>>>>>> be able to pass the broker's config file to the ZK Security
> >>>>> Migrator
> >>>>>>>>> tool. I think whatever we use (ZooKeeper camelCase style or
> Kafka
> >>>>>>>>> style), I think we should use the same for both.
> >>>>>>>>>
> >>>>>>>>> The second thing to think about is the fact that ZooKeeper
> >>>>>>>>> configuration inherently uses Java system properties as a first
> >>>>> pass.
> >>>>>>>>> So while we might switch to Kafka-style configs in the config
> file
> >>>>>>>>> (e.g. zookeeper.ssl.trust.store.location), the
> >>>>>>>>> org.apache.zookeeper.client.ZKClientConfig class (which is how
> TLS
> >>>>>>>>> configs are set) is going to look for the camelCase
> >>>>>>>>> zookeeper.ssl.trustStore.location system property and use any
> >> value
> >>>>>>>>> there. I think this could be very confusing for people.
> Granted,
> >>>>> we
> >>>>>>>>> are doing this so that people don't have to use system
> properties,
> >>>>> but
> >>>>>>>>> there is no way to turn off the use of system properties, so I
> >>>>> think
> >>>>>>>>> there would be pretty substantial potential for confusion.
> >>>>>>>>
> >>>>>>>> I don't think we document or recommend that anyone use system
> >>>>> properties
> >>>>>>>> to configure Zookeeper usage within Kafka. And I think the
> reason
> >>>>> is
> >>>>>>>> exactly the issue you pointed out -- it wouldn't be consistent
> with
> >>>>> our
> >>>>>>>> other configurations.
> >>>>>>>>
> >>>>>>>> best,
> >>>>>>>> Colin
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> One idea is to migrate the system properties -- i.e. look to see
> >> if
> >>>>>>>>> zookeeper.ssl.trustStore.location is set and if it is clear out
> >> the
> >>>>>>>>> value there and move it under the
> >>>>> zookeeper.ssl.trust.store.location
> >>>>>>>>> system property. (I said it was an idea -- not necessarily a
> good
> >>>>> one
> >>>>>>>>> :-)
> >>>>>>>>>
> >>>>>>>>> Ron
> >>>>>>>>>
> >>>>>>>>> On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe <cmccabe@apache.org
> >
> >>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
> >>>>>>>>>>> Thanks Colin and Rajini.
> >>>>>>>>>>>
> >>>>>>>>>>> I've updated the KIP to say that KIP-421 functionality is
> >>>>> available
> >>>>>>>> to
> >>>>>>>>>>> encrypt sensitive configs like the ZK key store and trust store
> >>>>>>>>>>> passwords. (I've also made it clear that the configs are not
> >>>>>>>>>>> dynamically reconfigurable since dynamic values are stored in
> >>>>> ZK and
> >>>>>>>>>>> the data is needed to get to ZK in the first place). Colin,
> >>>>> let me
> >>>>>>>>>>> know if you think anything else needs to be said/done about it
> >>>>> -- I
> >>>>>>>>>>> wasn't sure if your comment above implies that there are
> >>>>> additional
> >>>>>>>>>>> direct actions that we need to take aside from this.
> >>>>>>>>>>>
> >>>>>>>>>>> I agree that making the brokers' ZooKeeper clients dynamic with
> >>>>>>>>>>> respect to key and trust stores (e.g. via
> >>>>>>>>>>> zookeeper.ssl.context.supplier.class) may increase the risk
> >>>>> that this
> >>>>>>>>>>> KIP may not land in time for 2.5.
> >>>>>>>>>>>
> >>>>>>>>>>> Regarding the config names for ZK being different than the
> >>>>> ZooKeeper
> >>>>>>>>>>> configs (e.g. camel-case keyStore/trustStore instead of
> >>>>>>>>>>> keystore/truststore, ciphersuites instead of cipher.suites,
> >>>>>>>>>>> enabledProtocols instead of enabled.protocols), I agree it is
> >>>>> an
> >>>>>>>>>>> issue. I tried to mitigate it by putting warning text in the
> >>>>> config
> >>>>>>>>>>> docs for these. Regarding configuration inheritance, I think
> >>>>> what
> >>>>>>>> you
> >>>>>>>>>>> are saying is that for any configs where the broker has a clear
> >>>>>>>>>>> semantic equivalent, the broker could fall back to the broker
> >>>>> value
> >>>>>>>> if
> >>>>>>>>>>> the ZK value is not given. The potential list is:
> >>>>>>>>>>>
> >>>>>>>>>>> zookeeper.ssl.keyStore.location => ssl.keystore.location
> >>>>>>>>>>> zookeeper.ssl.keyStore.password => ssl.keystore.password
> >>>>>>>>>>> zookeeper.ssl.keyStore.type => ssl.keystore.type
> >>>>>>>>>>> zookeeper.ssl.trustStore.location => ssl.truststore.location
> >>>>>>>>>>> zookeeper.ssl.trustStore.password => ssl.truststore.password
> >>>>>>>>>>> zookeeper.ssl.trustStore.type => ssl.truststore.type
> >>>>>>>>>>> zookeeper.ssl.ciphersuites => ssl.cipher.suites
> >>>>>>>>>>>
> >>>>>>>>>>> Note that there are two configs that look the same based on
> >>>>> their key
> >>>>>>>>>>> names but whose allowable values differ:
> >>>>>>>>>>>
> >>>>>>>>>>> zookeeper.ssl.protocol(Default: TLSv1.2) =>
> >>>>> ssl.protocol(Default:
> >>>>>>>> TLS)
> >>>>>>>>>>> zookeeper.ssl.enabledProtocols(Default: value of protocol
> >>>>> property) =
> >>>>>>>>>>> ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
> >>>>>>>>>>>
> >>>>>>>>>>> Not sure what the right answer is, but hopefully this detail
> >>>>> will
> >>>>>>>> help
> >>>>>>>>>>> get us there.
> >>>>>>>>>>>
> >>>>>>>>>>> Ron
> >>>>>>>>>>
> >>>>>>>>>> I think, on balance, I agree with Rajini that it would be nice
> >>>>> to make
> >>>>>>>> the configs look more like Kafka configs. We don't have
> camel-case
> >>>>> in
> >>>>>>>> configuration keys, so we should avoid it here.
> >>>>>>>>>>
> >>>>>>>>>> best,
> >>>>>>>>>> Colin
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <
> >>>>> cmccabe@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> >>>>>>>>>>>>> Hi Ron,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the detailed explanation, sounds good to me.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> A few more questions:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 1) At the moment, all sensitive broker configs including
> >>>>>>>>>>>>> keystore/truststore passwords can be stored encrypted in
> >>>>>>>> ZooKeeper prior to
> >>>>>>>>>>>>> starting up brokers. We are now adding SSL
> >>>>> keystore/truststore
> >>>>>>>> passwords
> >>>>>>>>>>>>> for ZooKeeper client that cannot be stored in ZooKeeper
> >>>>> since
> >>>>>>>> you need
> >>>>>>>>>>>>> these to connect to ZK. We should perhaps document that
> >>>>> these
> >>>>>>>> configs can
> >>>>>>>>>>>>> be encrypted using KIP-421.
> >>>>>>>>>>>>
> >>>>>>>>>>>> That's a good point. Previously, we didn't have ZK
> >>>>> on-the-wire
> >>>>>>>> security support. However, now that we do, sending sensitive
> >>>>> keystore and
> >>>>>>>> truststore passwords to ZK in cleartext should be deprecated in
> >>>>> favor of
> >>>>>>>> using KIP-421.
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 2) We can dynamically update keystores and trust stores
> >>>>> used by
> >>>>>>>> brokers
> >>>>>>>>>>>>> without restarting the broker. Can we support this easily
> >>>>> for ZK
> >>>>>>>> clients
> >>>>>>>>>>>>> created by the broker, for example by adding our own
> >>>>>>>>>>>>> `zookeeper.ssl.context.supplier.class`?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hmm. That might be better to consider in a follow-up, since
> >>>>> the
> >>>>>>>> deadline for 2.5 KIPs is coming up?
> >>>>>>>>>>>>
> >>>>>>>>>>>> best,
> >>>>>>>>>>>> Colin
> >>>>>>>>>>>>
> >>>>>>>>>>>>> 3) Looks like we are using config names that directly map
> >>>>> to ZK
> >>>>>>>> configs.
> >>>>>>>>>>>>> Have we considered using equivalent Kafka config names with
> >>>>>>>> prefixes,
> >>>>>>>>>>>>> perhaps with inheritance from the non-prefixed names? Not
> >>>>> sure
> >>>>>>>> if this is a
> >>>>>>>>>>>>> good idea, but perhaps worth documenting in Rejected
> >>>>>>>> Alternatives at least.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <
> >>>>> cmccabe@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Ron,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the explanation. I guess thinking about it a
> >>>>>>>> little bit more,
> >>>>>>>>>>>>>> we should just add --zk-tls-config-file to all of these
> >>>>>>>> commands.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We will be removing this option (plus ZK in general) from
> >>>>>>>> these commands
> >>>>>>>>>>>>>> in the next major release, but ZK is still supported in
> >>>>> 2.5,
> >>>>>>>> so we should
> >>>>>>>>>>>>>> just do the logical thing. (The exception is
> >>>>>>>> ZkSecurityMigrator, which
> >>>>>>>>>>>>>> will stay).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> best,
> >>>>>>>>>>>>>> Colin
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> >>>>>>>>>>>>>>> Hi Colin.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> <<< It seems like this [--zk-tls-config-file
> >>>>> information]
> >>>>>>>> could just
> >>>>>>>>>>>>>> appear
> >>>>>>>>>>>>>>> in a configuration file, which all of these tools
> >>>>> already
> >>>>>>>> accept (I
> >>>>>>>>>>>>>> think)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ZkSecurityMigrator has no such property file facility;
> >>>>>>>> adding a
> >>>>>>>>>>>>>>> "--zk-tls-config-file" parameter is exactly for this
> >>>>>>>> purpose. If we add
> >>>>>>>>>>>>>>> that to ZkSecurityMigrator then it is trivial to add
> >>>>> it to
> >>>>>>>> other commands
> >>>>>>>>>>>>>>> (the same code is simply reused; it ends up being just
> >>>>> a few
> >>>>>>>> extra
> >>>>>>>>>>>>>> lines).
> >>>>>>>>>>>>>>> I do not see any parameter in the other two commands to
> >>>>>>>> adjust the ZK
> >>>>>>>>>>>>>>> connection; ConfigCommand accepts a "--command-config"
> >>>>> flag,
> >>>>>>>> but
> >>>>>>>>>>>>>> according
> >>>>>>>>>>>>>>> to the code "This is used only with --bootstrap-server
> >>>>>>>> option for
> >>>>>>>>>>>>>>> describing and altering broker configs."
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I do agree there would be no need to add
> >>>>>>>> "--zk-tls-config-file" to
> >>>>>>>>>>>>>>> ReassignPartitionsCommand if its direct ZK
> >>>>> connectivity is
> >>>>>>>> replaced in
> >>>>>>>>>>>>>> time
> >>>>>>>>>>>>>>> for the next release.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ConfigCommand supports the "--bootstrap-server" option
> >>>>> and
> >>>>>>>> will have its
> >>>>>>>>>>>>>>> direct ZooKeeper access formally deprecated as per
> >>>>> KIP-555,
> >>>>>>>> but the
> >>>>>>>>>>>>>> special
> >>>>>>>>>>>>>>> use case of bootstrapping a ZooKeeper ensemble with
> >>>>>>>> encrypted credentials
> >>>>>>>>>>>>>>> prior to starting Kafka will still exist, so it feels
> >>>>> like
> >>>>>>>> while
> >>>>>>>>>>>>>>> "--zk-tls-config-file" would never be used except for
> >>>>> this
> >>>>>>>> use case it
> >>>>>>>>>>>>>>> could probably still be added for this particular
> >>>>> situation.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Ron
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> P.S. I responded on 1/6 but I just discovered that e,
> >>>>> ail
> >>>>>>>> (and 3 more I
> >>>>>>>>>>>>>>> sent) did not go through. I am trying to get emails
> >>>>> through
> >>>>>>>> now to move
> >>>>>>>>>>>>>>> this discussion forward.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <
> >>>>>>>> cmccabe@apache.org> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> >>>>>>>>>>>>>>>>> Hi everyone. I would like to make the following
> >>>>> changes
> >>>>>>>> to the KIP.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> MOTIVATION:
> >>>>>>>>>>>>>>>>> Include a statement that it will be difficult in
> >>>>> the
> >>>>>>>> short term to
> >>>>>>>>>>>>>>>>> deprecate direct Zookeeper communication in
> >>>>>>>> kafka-configs.{sh, bat}
> >>>>>>>>>>>>>>>> (which
> >>>>>>>>>>>>>>>>> invoke kafka.admin.ConfigCommand) because
> >>>>> bootstrapping
> >>>>>>>> a Kafka
> >>>>>>>>>>>>>> cluster
> >>>>>>>>>>>>>>>>> with encrypted passwords in Zookeeper is an
> >>>>>>>> explicitly-supported use
> >>>>>>>>>>>>>>>> case;
> >>>>>>>>>>>>>>>>> therefore it is in scope to be able to securely
> >>>>>>>> configure the CLI
> >>>>>>>>>>>>>> tools
> >>>>>>>>>>>>>>>>> that still leverage non-deprecated direct Zookeeper
> >>>>>>>> communication
> >>>>>>>>>>>>>> for TLS
> >>>>>>>>>>>>>>>>> (the other 2 tools are
> >>>>> kafka-reassign-partitions.{sh,
> >>>>>>>> bat} and
> >>>>>>>>>>>>>>>>> zookeeper-security-migration.sh).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Ron,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for the KIP.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> About deprecations:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> * zookeeper-security-migration: clearly, deprecating
> >>>>> ZK
> >>>>>>>> access in this
> >>>>>>>>>>>>>> one
> >>>>>>>>>>>>>>>> would not make sense, since it would defeat the whole
> >>>>>>>> point of the
> >>>>>>>>>>>>>> tool :)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> * kafka-reassign-partitions: ZK access should be
> >>>>>>>> deprecated here.
> >>>>>>>>>>>>>> KIP-455
> >>>>>>>>>>>>>>>> implementation has dragged a bit, but this should
> >>>>> get done
> >>>>>>>> soon.
> >>>>>>>>>>>>>> Certainly
> >>>>>>>>>>>>>>>> before the next release.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> * kafka-configs: I think ZK access should be
> >>>>> deprecated
> >>>>>>>> here as well.
> >>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>> agree there is a super-special bootstrapping case
> >>>>> here,
> >>>>>>>> but that should
> >>>>>>>>>>>>>>>> have its own tool, not use kafka-configs.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I will post a separate KIP for this, though.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> GOALS:
> >>>>>>>>>>>>>>>>> Support the secure configuration of TLS-encrypted
> >>>>>>>> communication
> >>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>> Zookeeper and:
> >>>>>>>>>>>>>>>>> a) Kafka brokers
> >>>>>>>>>>>>>>>>> b) The three CLI tools mentioned above that still
> >>>>>>>> support direct,
> >>>>>>>>>>>>>>>>> non-deprecated communication to Zookeeper
> >>>>>>>>>>>>>>>>> It is explicitly out-of-scope to deprecate any
> >>>>> direct
> >>>>>>>> Zookeeper
> >>>>>>>>>>>>>>>>> communication in CLI tools as part of this KIP;
> >>>>> such
> >>>>>>>> work will occur
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>> future KIPs instead.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> PUBLIC INTERFACES:
> >>>>>>>>>>>>>>>>> 1) The following new broker configurations will be
> >>>>>>>> recognized.
> >>>>>>>>>>>>>>>>> zookeeper.client.secure (default value = false,
> >>>>> for
> >>>>>>>> backwards
> >>>>>>>>>>>>>>>>> compatibility)
> >>>>>>>>>>>>>>>>> zookeeper.clientCnxnSocket
> >>>>>>>>>>>>>>>>> zookeeper.ssl.keyStore.location
> >>>>>>>>>>>>>>>>> zookeeper.ssl.keyStore.password
> >>>>>>>>>>>>>>>>> zookeeper.ssl.trustStore.location
> >>>>>>>>>>>>>>>>> zookeeper.ssl.trustStore.password
> >>>>>>>>>>>>>>>>> It will be an error for any of the last 5 values
> >>>>> to be
> >>>>>>>> left
> >>>>>>>>>>>>>> unspecified
> >>>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>> zookeeper.client.secure is explicitly set to true.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 2) In addition, the
> >>>>>>>> kafka.security.authorizer.AclAuthorizer class
> >>>>>>>>>>>>>>>> supports
> >>>>>>>>>>>>>>>>> the ability to connect to a different Zookeeper
> >>>>> instance
> >>>>>>>> than the
> >>>>>>>>>>>>>> one the
> >>>>>>>>>>>>>>>>> brokers use. We therefore also add the following
> >>>>>>>> optional configs,
> >>>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>> override the corresponding ones from above when
> >>>>> present:
> >>>>>>>>>>>>>>>>> authorizer.zookeeper.client.secure
> >>>>>>>>>>>>>>>>> authorizer.zookeeper.clientCnxnSocket
> >>>>>>>>>>>>>>>>> authorizer.zookeeper.ssl.keyStore.location
> >>>>>>>>>>>>>>>>> authorizer.zookeeper.ssl.keyStore.password
> >>>>>>>>>>>>>>>>> authorizer.zookeeper.ssl.trustStore.location
> >>>>>>>>>>>>>>>>> authorizer.zookeeper.ssl.trustStore.password
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 3) The three CLI tools mentioned above will
> >>>>> support a new
> >>>>>>>>>>>>>>>> --zk-tls-config-file
> >>>>>>>>>>>>>>>>> <String: Zookeeper TLS configuration file path>"
> >>>>>>>> option. The
> >>>>>>>>>>>>>> following
> >>>>>>>>>>>>>>>>> properties will be recognized in that file, and
> >>>>>>>> unrecognized
> >>>>>>>>>>>>>> properties
> >>>>>>>>>>>>>>>>> will be ignored to allow the possibility of
> >>>>> pointing
> >>>>>>>>>>>>>> zk-tls-config-file
> >>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>> the broker's config file.
> >>>>>>>>>>>>>>>>> zookeeper.client.secure (default value = false)
> >>>>>>>>>>>>>>>>> zookeeper.clientCnxnSocket
> >>>>>>>>>>>>>>>>> zookeeper.ssl.keyStore.location
> >>>>>>>>>>>>>>>>> zookeeper.ssl.keyStore.password
> >>>>>>>>>>>>>>>>> zookeeper.ssl.trustStore.location
> >>>>>>>>>>>>>>>>> zookeeper.ssl.trustStore.password
> >>>>>>>>>>>>>>>>> It will be an error for any of the last 5 values
> >>>>> to be
> >>>>>>>> left
> >>>>>>>>>>>>>> unspecified
> >>>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>> zookeeper.client.secure is explicitly set to true.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Do we really need a --zk-tls-config-file flag? It
> >>>>> seems
> >>>>>>>> like this
> >>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>> just appear in a configuration file, which all of
> >>>>> these
> >>>>>>>> tools already
> >>>>>>>>>>>>>>>> accept (I think).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> best,
> >>>>>>>>>>>>>>>> Colin
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Ron
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <
> >>>>>>>> rndgstn@gmail.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi everyone. Let's get this discussion going
> >>>>> again
> >>>>>>>> now that Kafka
> >>>>>>>>>>>>>> 2.4
> >>>>>>>>>>>>>>>>>> with Zookeeper 3.5.6 is out.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> First, regarding the KIP number, the other KIP
> >>>>> that
> >>>>>>>> was using this
> >>>>>>>>>>>>>>>> number
> >>>>>>>>>>>>>>>>>> moved to KIP 534, so KIP 515 remains the correct
> >>>>>>>> number for this
> >>>>>>>>>>>>>>>>>> discussion. I've updated the Kafka Improvement
> >>>>>>>> Proposal page to
> >>>>>>>>>>>>>> list
> >>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>> KIP in the 515 slot, so we're all set there.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Regarding the actual issues under discussion, I
> >>>>> think
> >>>>>>>> there are
> >>>>>>>>>>>>>> some
> >>>>>>>>>>>>>>>>>> things we should clarify.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 1) It is possible to use TLS connectivity to
> >>>>> Zookeeper
> >>>>>>>> from Apache
> >>>>>>>>>>>>>>>> Kafka
> >>>>>>>>>>>>>>>>>> 2.4 -- the problem is that configuration
> >>>>> information
> >>>>>>>> has to be
> >>>>>>>>>>>>>> passed
> >>>>>>>>>>>>>>>> via
> >>>>>>>>>>>>>>>>>> system properties as "-D" command line options
> >>>>> on the
> >>>>>>>> java
> >>>>>>>>>>>>>> invocation
> >>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>> the broker, and those are not secure (anyone with
> >>>>>>>> access to the box
> >>>>>>>>>>>>>>>> can see
> >>>>>>>>>>>>>>>>>> the command line used to invoke the broker); the
> >>>>>>>> configuration
> >>>>>>>>>>>>>> includes
> >>>>>>>>>>>>>>>>>> sensitive information (e.g. a keystore
> >>>>> password), so
> >>>>>>>> we need a
> >>>>>>>>>>>>>> secure
> >>>>>>>>>>>>>>>>>> mechanism for passing the configuration values.
> >>>>> I
> >>>>>>>> believe the real
> >>>>>>>>>>>>>>>>>> motivation for this KIP is to harden the
> >>>>> configuration
> >>>>>>>> mechanism
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>> Zookeeper TLS connectivity.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 2) I believe the list of CLI tools that continue
> >>>>> to
> >>>>>>>> use direct
> >>>>>>>>>>>>>>>> Zookeeper
> >>>>>>>>>>>>>>>>>> connectivity in a non-deprecated fashion is:
> >>>>>>>>>>>>>>>>>> a)
> >>>>>>>>>>>>>>
> >>>>> zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> >>>>>>>>>>>>>>>>>> b)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>
> >>>>>
> >> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> >>>>>>>>>>>>>>>>>> c) kafka-configs.{sh,
> >>>>> bat}/kafka.admin.ConfigCommand
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 3) I believe Kafka.admin.ConfigCommand presents a
> >>>>>>>> conundrum
> >>>>>>>>>>>>>> because it
> >>>>>>>>>>>>>>>>>> explicitly states in a comment that a supported
> >>>>> use
> >>>>>>>> case is
> >>>>>>>>>>>>>>>> bootstrapping a
> >>>>>>>>>>>>>>>>>> Kafka cluster with encrypted passwords in
> >>>>> Zookeeper
> >>>>>>>> (see
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> >>>>>>>>>>>>>>>> ).
> >>>>>>>>>>>>>>>>>> This means it will be especially difficult to
> >>>>>>>> deprecate this
> >>>>>>>>>>>>>> particular
> >>>>>>>>>>>>>>>>>> direct Zookeeper connectivity without a different
> >>>>>>>> storage
> >>>>>>>>>>>>>> mechanism for
> >>>>>>>>>>>>>>>>>> dynamic configuration values being available
> >>>>> (i.e. the
> >>>>>>>> self-managed
> >>>>>>>>>>>>>>>> quorum
> >>>>>>>>>>>>>>>>>> referred to in KIP-500).
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I think it would be easier and simpler to harden
> >>>>> the
> >>>>>>>> Zookeeper TLS
> >>>>>>>>>>>>>>>>>> configuration in both Kafka and in CLI tools
> >>>>> compared
> >>>>>>>> to trying to
> >>>>>>>>>>>>>>>>>> deprecate the direct Zookeeper connectivity in
> >>>>> the
> >>>>>>>> above 3 CLI
> >>>>>>>>>>>>>> tools --
> >>>>>>>>>>>>>>>>>> especially when ConfigCommand has no obvious
> >>>>>>>> short-term path to
> >>>>>>>>>>>>>>>> deprecation.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Regarding how to harden the TLS configuration,
> >>>>> adding
> >>>>>>>> the
> >>>>>>>>>>>>>> appropriate
> >>>>>>>>>>>>>>>>>> configs to Kafka is an obvious choice for the
> >>>>>>>> brokers. Not that
> >>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>> values cannot be dynamically reconfigurable
> >>>>> because
> >>>>>>>> the values
> >>>>>>>>>>>>>> would be
> >>>>>>>>>>>>>>>>>> required to connect to Zookeeper via TLS in the
> >>>>> first
> >>>>>>>> place.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Regarding how to harden the TLS configuration
> >>>>> for the
> >>>>>>>> 3 remaining
> >>>>>>>>>>>>>> CLI
> >>>>>>>>>>>>>>>>>> tools, it would be relatively straightforward to
> >>>>> add
> >>>>>>>> support for a
> >>>>>>>>>>>>>>>>>> "--zk-tls-config-file <String: Zookeeper TLS
> >>>>>>>> configuration system
> >>>>>>>>>>>>>>>>>> properties file path>" option to each one.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thoughts?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Ron
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes
> >>>>> <
> >>>>>>>>>>>>>> pere.urbon@gmail.com
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> True, the number is duplicated.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> do you know how can we get the problem fix? I
> >>>>> was
> >>>>>>>> not aware, I
> >>>>>>>>>>>>>>>> missed,
> >>>>>>>>>>>>>>>>>>> sorry the need to add the KIP to the table.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -- Pere
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Missatge de Jorge Esteban Quilcate Otoya <
> >>>>>>>>>>>>>> quilcate.jorge@gmail.com>
> >>>>>>>>>>>>>>>> del
> >>>>>>>>>>>>>>>>>>> dia
> >>>>>>>>>>>>>>>>>>> dt., 3 de set. 2019 a les 13:43:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi Pere,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Have you add your KIP to the list here
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >>>>>>>>>>>>>>>>>>> ?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I found the KIP number assigned to another.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón
> >>>>> Bayes <
> >>>>>>>>>>>>>>>> pere.urbon@gmail.com>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for your time Harsha,
> >>>>>>>>>>>>>>>>>>>>> anyone else with comments? looking
> >>>>> forward to
> >>>>>>>> hearing from
> >>>>>>>>>>>>>> you.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Stupid question: when do you move from
> >>>>> discussion
> >>>>>>>> to vote?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Missatge de Harsha Chintalapani <
> >>>>> kafka@harsha.io>
> >>>>>>>> del dia
> >>>>>>>>>>>>>> dv., 30
> >>>>>>>>>>>>>>>>>>> d’ag.
> >>>>>>>>>>>>>>>>>>>>> 2019 a les 21:59:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks Pere. KIP looks good to me.
> >>>>>>>>>>>>>>>>>>>>>> -Harsha
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 30, 2019 at 10:05 AM, Pere
> >>>>> Urbón
> >>>>>>>> Bayes <
> >>>>>>>>>>>>>>>>>>>>> pere.urbon@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Not really,
> >>>>>>>>>>>>>>>>>>>>>>> my idea is to keep the JAAS parameter,
> >>>>> so
> >>>>>>>> people don't see
> >>>>>>>>>>>>>>>> major
> >>>>>>>>>>>>>>>>>>>>>>> changes. But if you pass a properties
> >>>>> file,
> >>>>>>>> then this takes
> >>>>>>>>>>>>>>>>>>> precedence
> >>>>>>>>>>>>>>>>>>>>> over
> >>>>>>>>>>>>>>>>>>>>>>> the other, with the idea that you can do
> >>>>> sasl
> >>>>>>>> as well with
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> properties
> >>>>>>>>>>>>>>>>>>>>>>> files.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Makes sense?
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> -- Pere
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Missatge de Harsha Chintalapani <
> >>>>>>>> kafka@harsha.io> del dia
> >>>>>>>>>>>>>> dv.,
> >>>>>>>>>>>>>>>> 30
> >>>>>>>>>>>>>>>>>>>>> d’ag.
> >>>>>>>>>>>>>>>>>>>>>>> 2019 a les 19:00:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi Pere,
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. Enabling
> >>>>> SSL for
> >>>>>>>> zookeeper
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> Kafka
> >>>>>>>>>>>>>>>>>>>>> makes
> >>>>>>>>>>>>>>>>>>>>>>>> sense.
> >>>>>>>>>>>>>>>>>>>>>>>> "The changes are planned to be
> >>>>> introduced in a
> >>>>>>>> compatible
> >>>>>>>>>>>>>> way,
> >>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>>> keeping the current JAAS variable
> >>>>> precedence."
> >>>>>>>>>>>>>>>>>>>>>>>> Can you elaborate a bit here. If the user
> >>>>>>>> configures a JAAS
> >>>>>>>>>>>>>>>> file
> >>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>> Client section it will take precedence
> >>>>> over
> >>>>>>>> zookeeper SSL
> >>>>>>>>>>>>>>>> configs?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>> Harsha
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 30, 2019 at 7:50 AM, Pere
> >>>>> Urbón
> >>>>>>>> Bayes <
> >>>>>>>>>>>>>>>>>>>>> pere.urbon@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>>>>>> quick question, I saw in another mail
> >>>>> that
> >>>>>>>> 2.4 release is
> >>>>>>>>>>>>>>>> planned
> >>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>> September. I think it would be really
> >>>>> awesome
> >>>>>>>> to have
> >>>>>>>>>>>>>> this for
> >>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>> release, do you think we can make it?
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> -- Pere
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Missatge de Pere Urbón Bayes <
> >>>>>>>> pere.urbon@gmail.com> del
> >>>>>>>>>>>>>> dia
> >>>>>>>>>>>>>>>> dj.,
> >>>>>>>>>>>>>>>>>>> 29
> >>>>>>>>>>>>>>>>>>>>>>>>> d’ag. 2019 a les 20:10:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>>>>>> this is my first KIP for a change in
> >>>>> Apache
> >>>>>>>> Kafka, so I'm
> >>>>>>>>>>>>>>>> really
> >>>>>>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>>>>>>>>> to the process. Looking forward to
> >>>>> hearing
> >>>>>>>> from you and
> >>>>>>>>>>>>>> learn
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> best
> >>>>>>>>>>>>>>>>>>>>>>>>> ropes here.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> I would like to propose this KIP-515 to
> >>>>>>>> enable the
> >>>>>>>>>>>>>>>>>>> ZookeeperClients
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> take full advantage of the TLS
> >>>>> communication
> >>>>>>>> in the new
> >>>>>>>>>>>>>>>> Zookeeper
> >>>>>>>>>>>>>>>>>>>>> 3.5.5.
> >>>>>>>>>>>>>>>>>>>>>>>>> Specially interesting it the Zookeeper
> >>>>>>>> Security Migration,
> >>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>> without
> >>>>>>>>>>>>>>>>>>>>>>>>> this change will not work with TLS,
> >>>>> disabling
> >>>>>>>> users to use
> >>>>>>>>>>>>>>>> ACLs
> >>>>>>>>>>>>>>>>>>> when
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> Zookeeper cluster use TLS.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> link:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>
> >>>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to hearing from you on
> >>>>> this,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> /cheers
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
> >>>>>>>>>>>>>>>>>>>>>>>>> Software Architect
> >>>>>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
> >>>>>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
> >>>>>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
> >>>>>>>>>>>>>>>>>>>>>>>>> Software Architect
> >>>>>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
> >>>>>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
> >>>>>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
> >>>>>>>>>>>>>>>>>>>>>>> Software Architect
> >>>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
> >>>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
> >>>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
> >>>>>>>>>>>>>>>>>>>>> Software Architect
> >>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
> >>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
> >>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
> >>>>>>>>>>>>>>>>>>> Software Architect
> >>>>>>>>>>>>>>>>>>> http://www.purbon.com
> >>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
> >>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>
> >>
>
>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Ron Dagostino <rn...@gmail.com>.
Thanks, Rajini. Still not sure what the answer is, but I thought of a
couple more issues with config inheritance that I wanted to raise.
The first is a minor issue, but just to document it (and I will add it
to the KIP as well), ZooKeeper does not support a key password that
differs from the keystore password. So inheriting from Kafka's TLS
configs would only work if those configs use the same key and keystore
password. Again, not a big deal -- just something to document.
However...
I think there may be a fundamental problem with inheriting broker
configs for ZooKeeper access. The problem is that the broker's TLS
configs are dynamically reconfigurable, and the dynamic values
themselves are stored in ZooKeeper. It would be impossible to know
those values without connecting to ZooKeeper, and it would be
impossible to connect to ZooKeeper without the values. Given this, I
wonder if inheritance is impossible in this particular case.
Ron
> On Jan 17, 2020, at 5:39 PM, Rajini Sivaram <ra...@gmail.com> wrote:
>
> Hi Ron,
>
> For Kafka, we moved from hostname verification disabled-by-default to
> enabled-by-default under
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-294+-+Enable+TLS+hostname+verification+by+default.
> So we have tested empty String for disabling hostname verification and we
> know that it works, not only in test environments, but also in actual
> deployments. Basically, "ssl.endpoint.identification.algorithm=" in the
> file results in a config value of empty String and not setting it results
> in the default value of HTTPS since the value is null.I would probably do
> the same for the ZK config. If you are creating certificates for ZooKeeper
> without hostnames, then you may be doing the same for Kafka too and in both
> cases, the config would be an empty string. This is insecure anyway and we
> would really expect this only in test environments. Not sure if it is worth
> making an exception for this in terms of consistent config names.
>
>> On Fri, Jan 17, 2020 at 2:37 PM Ron Dagostino <rn...@gmail.com> wrote:
>>
>> Hi again, Rajini. I've updated the KIP, and while doing it I became
>> concerned about using zookeeper.ssl.endpoint.identification.algorithm
>> for enabling/disabling hostname verification. The KIP reflects what
>> we decided, but upon reading it, I wonder if it is workable. Here's
>> what it says for this config:
>>
>> "Specifies whether to enable hostname verification in the ZooKeeper
>> TLS negotiation process, with (case-insensitively) "https" meaning
>> ZooKeeper hostname verification is enabled and an explicit blank value
>> meaning it is disabled (disabling it is only recommended for testing
>> purposes). Overrides any explicit "true" or "false" value set via the
>> <code>zookeeper.ssl.hostnameVerification</code> system property (true
>> implying https and false implying blank) and inherits the value of
>> <code>ssl.endpoint.identification.algorithm</code> (if any) if no
>> explicit value or non-value is set both here and via the system
>> property."
>>
>> I wonder if someone sets an explicit empty value
>> (zookeeper.ssl.endpoint.identification.algorithm=) in the config file,
>> will we be able to distinguish between that and a config file that
>> doesn't have it at all? And even if we can, this feels pretty forced
>> to me, like we are bending over backwards for consistency when
>> "zookeeper.ssl.hostname.verification.enable" is a natural fit. I
>> doubt this config is going to be used very much, so it really does
>> feel to me like we should just go with the natural fit and ditch the
>> compatibility and inheritance for this one config.
>>
>> Ron
>>
>>> On Fri, Jan 17, 2020 at 8:07 AM Ron Dagostino <rn...@gmail.com> wrote:
>>>
>>> Hi Rajini. I’ll submit a documentation PR to clarify the Kafka behavior
>> of #1 and will adopt the same config for ZK. I agree now we should inherit
>> for AclAuthorizer too — I was just stuck on the “no inheritance” idea more
>> than I realized, and while the ZK quorums are different that doesn’t
>> necessarily mean the same client identity wouldn’t be valid on both (and if
>> not then the configs can still be overridden).
>>>
>>> I’ll update the KIP today and let this version soak for a day or two,
>> and then I’ll start a vote if no other issues/comments arise.
>>>
>>> Ron
>>>
>>>> On Jan 17, 2020, at 6:08 AM, Rajini Sivaram <ra...@gmail.com>
>> wrote:
>>>>
>>>> Hi Ron,
>>>>
>>>> Unresolved item #1: Yes, Kafka disables hostname verification if
>>>> "ssl.endpoint.identification.algorithm" is an empty string.
>>>>
>>>> Unresolved item #2: Yes, those 9 plus hostname verification. If we do
>>>> decide to inherit Kafka configs, wouldn't we inherit these 10 in
>>>> AclAuthorizer as well?
>>>>
>>>> Unresolved item #3: Agree, following up on the discussion thread for
>>>> KIP-553.
>>>>
>>>>
>>>>> On Fri, Jan 17, 2020 at 3:29 AM Ron Dagostino <rn...@gmail.com>
>> wrote:
>>>>>
>>>>> Hi Colin. TLS to ZooKeeper will remain an opt-in. The configuration
>>>>> value that turns it on is zookeeper.ssl.client.enable (which used to
>>>>> be called zookeeper.client.secure before we renamed everything to be
>>>>> consistent with Kafka's config naming style and eliminate confusion).
>>>>> The default value for this is false. Everything else related to TLS
>>>>> has no effect unless this is explicitly set to true, and this value
>>>>> does not inherit from anywhere,
>>>>>
>>>>> Ron
>>>>>
>>>>>> On Thu, Jan 16, 2020 at 5:39 PM Colin McCabe <cm...@apache.org>
>> wrote:
>>>>>>
>>>>>>> On Thu, Jan 16, 2020, at 02:58, Rajini Sivaram wrote:
>>>>>>> Hi Ron/Colin,
>>>>>>>
>>>>>>> Thanks for the responses. I think we need to consider these cases:
>>>>>>>
>>>>>>> 1) The recommended approach: This is the one we document. Configs
>>>>>>> consistent with Kafka make sense here. It seems a lot more
>> confusing to
>>>>>>> configure `ssl.cipher.suites`, `ssl.enabled.protocols`, `
>>>>>>> zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols`
>> with
>>>>>>> multiple naming inconsistencies in server.properties. For users who
>>>>> have
>>>>>>> never used ZK system properties, this is clearly simpler. Another
>>>>> config to
>>>>>>> consider is `ssl.endpoint.identification.algorithm` which does the
>> same
>>>>>>> thing as `zookeeper.ssl.hostnameVerification` - if we are staying
>>>>>>> consistent, we should use the Kafka format for this too. I would
>> also
>>>>>>> consider using something like `zookeeper.client.ssl.enabled` instead
>>>>> of `
>>>>>>> zookeeper.client.secure` since it is difficult to tell what `secure`
>>>>> means
>>>>>>> when you have options to configure Kerberos, ACLs and/or SSL.
>>>>>>>
>>>>>>> 2) Inheritance using prefixed configs as we do for other prefixed
>>>>> configs
>>>>>>> in Kafka: I think it may be useful for configs like `ssl.protocol`
>> and
>>>>> `
>>>>>>> ssl.cipher.suites`.
>>>>>>>
>>>>>>> 3) Default values: Do we want to use Kafka defaults? The
>> inconsistency
>>>>> we
>>>>>>> have is `ssl.protocol/ssl.enabled.protocols` since we still have
>>>>> insecure
>>>>>>> protocols enabled for Kafka. We have KIP-553 (
>>>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
>>>>> )
>>>>>>> open to disable these. It is currently blocked since it also talks
>>>>> about
>>>>>>> enabling TLSv1.3 by default and we haven't done sufficient testing
>> for
>>>>>>> that. Since it will be good to disable the older insecure protocols
>>>>> anyway,
>>>>>>> perhaps we could do that without enabling TLSv1.3 by default in 2.5?
>>>>>>
>>>>>> Hi Rajini,
>>>>>>
>>>>>> Inheriting the SSL settings from Kafka would have the effect of
>> changing
>>>>> the defaults for almost everyone using SSL from non-SSL ZK to
>> SSL-based ZK,
>>>>> right? This seems to expand the scope of the KIP greatly, since it
>> moves
>>>>> it from being opt-in to opt-out. Are there any potential negative
>>>>> implications of this?
>>>>>>
>>>>>> best,
>>>>>> Colin
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> 4) Backward compatibility for deployments which are using system
>>>>>>> properties. We shouldn't override system properties with Kafka
>>>>> defaults or
>>>>>>> inherited values. But we do want to override if user configures
>>>>> properties
>>>>>>> explicitly. In the current approach, this was simpler since we just
>>>>> had to
>>>>>>> set the configured values. But if we decide to inherit Kafka
>> configs,
>>>>> then
>>>>>>> we will need to check system properties and update accordingly. But
>>>>> that
>>>>>>> should be doable?
>>>>>>>
>>>>>>> On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe <cm...@apache.org>
>>>>> wrote:
>>>>>>>
>>>>>>>>> On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote:
>>>>>>>>> Hi Colin. Two things come to mind with respect to ZooKeeper
>>>>> camelCase
>>>>>>>>> style vs Kafka-style config names for ZooKeeper. First, I think
>> it
>>>>>>>>> would be desirable for the client configs and broker configs to be
>>>>>>>>> interoperable. For example, it feels like it would be convenient
>>>>> to
>>>>>>>>> be able to pass the broker's config file to the ZK Security
>>>>> Migrator
>>>>>>>>> tool. I think whatever we use (ZooKeeper camelCase style or Kafka
>>>>>>>>> style), I think we should use the same for both.
>>>>>>>>>
>>>>>>>>> The second thing to think about is the fact that ZooKeeper
>>>>>>>>> configuration inherently uses Java system properties as a first
>>>>> pass.
>>>>>>>>> So while we might switch to Kafka-style configs in the config file
>>>>>>>>> (e.g. zookeeper.ssl.trust.store.location), the
>>>>>>>>> org.apache.zookeeper.client.ZKClientConfig class (which is how TLS
>>>>>>>>> configs are set) is going to look for the camelCase
>>>>>>>>> zookeeper.ssl.trustStore.location system property and use any
>> value
>>>>>>>>> there. I think this could be very confusing for people. Granted,
>>>>> we
>>>>>>>>> are doing this so that people don't have to use system properties,
>>>>> but
>>>>>>>>> there is no way to turn off the use of system properties, so I
>>>>> think
>>>>>>>>> there would be pretty substantial potential for confusion.
>>>>>>>>
>>>>>>>> I don't think we document or recommend that anyone use system
>>>>> properties
>>>>>>>> to configure Zookeeper usage within Kafka. And I think the reason
>>>>> is
>>>>>>>> exactly the issue you pointed out -- it wouldn't be consistent with
>>>>> our
>>>>>>>> other configurations.
>>>>>>>>
>>>>>>>> best,
>>>>>>>> Colin
>>>>>>>>
>>>>>>>>>
>>>>>>>>> One idea is to migrate the system properties -- i.e. look to see
>> if
>>>>>>>>> zookeeper.ssl.trustStore.location is set and if it is clear out
>> the
>>>>>>>>> value there and move it under the
>>>>> zookeeper.ssl.trust.store.location
>>>>>>>>> system property. (I said it was an idea -- not necessarily a good
>>>>> one
>>>>>>>>> :-)
>>>>>>>>>
>>>>>>>>> Ron
>>>>>>>>>
>>>>>>>>> On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe <cm...@apache.org>
>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
>>>>>>>>>>> Thanks Colin and Rajini.
>>>>>>>>>>>
>>>>>>>>>>> I've updated the KIP to say that KIP-421 functionality is
>>>>> available
>>>>>>>> to
>>>>>>>>>>> encrypt sensitive configs like the ZK key store and trust store
>>>>>>>>>>> passwords. (I've also made it clear that the configs are not
>>>>>>>>>>> dynamically reconfigurable since dynamic values are stored in
>>>>> ZK and
>>>>>>>>>>> the data is needed to get to ZK in the first place). Colin,
>>>>> let me
>>>>>>>>>>> know if you think anything else needs to be said/done about it
>>>>> -- I
>>>>>>>>>>> wasn't sure if your comment above implies that there are
>>>>> additional
>>>>>>>>>>> direct actions that we need to take aside from this.
>>>>>>>>>>>
>>>>>>>>>>> I agree that making the brokers' ZooKeeper clients dynamic with
>>>>>>>>>>> respect to key and trust stores (e.g. via
>>>>>>>>>>> zookeeper.ssl.context.supplier.class) may increase the risk
>>>>> that this
>>>>>>>>>>> KIP may not land in time for 2.5.
>>>>>>>>>>>
>>>>>>>>>>> Regarding the config names for ZK being different than the
>>>>> ZooKeeper
>>>>>>>>>>> configs (e.g. camel-case keyStore/trustStore instead of
>>>>>>>>>>> keystore/truststore, ciphersuites instead of cipher.suites,
>>>>>>>>>>> enabledProtocols instead of enabled.protocols), I agree it is
>>>>> an
>>>>>>>>>>> issue. I tried to mitigate it by putting warning text in the
>>>>> config
>>>>>>>>>>> docs for these. Regarding configuration inheritance, I think
>>>>> what
>>>>>>>> you
>>>>>>>>>>> are saying is that for any configs where the broker has a clear
>>>>>>>>>>> semantic equivalent, the broker could fall back to the broker
>>>>> value
>>>>>>>> if
>>>>>>>>>>> the ZK value is not given. The potential list is:
>>>>>>>>>>>
>>>>>>>>>>> zookeeper.ssl.keyStore.location => ssl.keystore.location
>>>>>>>>>>> zookeeper.ssl.keyStore.password => ssl.keystore.password
>>>>>>>>>>> zookeeper.ssl.keyStore.type => ssl.keystore.type
>>>>>>>>>>> zookeeper.ssl.trustStore.location => ssl.truststore.location
>>>>>>>>>>> zookeeper.ssl.trustStore.password => ssl.truststore.password
>>>>>>>>>>> zookeeper.ssl.trustStore.type => ssl.truststore.type
>>>>>>>>>>> zookeeper.ssl.ciphersuites => ssl.cipher.suites
>>>>>>>>>>>
>>>>>>>>>>> Note that there are two configs that look the same based on
>>>>> their key
>>>>>>>>>>> names but whose allowable values differ:
>>>>>>>>>>>
>>>>>>>>>>> zookeeper.ssl.protocol(Default: TLSv1.2) =>
>>>>> ssl.protocol(Default:
>>>>>>>> TLS)
>>>>>>>>>>> zookeeper.ssl.enabledProtocols(Default: value of protocol
>>>>> property) =
>>>>>>>>>>> ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
>>>>>>>>>>>
>>>>>>>>>>> Not sure what the right answer is, but hopefully this detail
>>>>> will
>>>>>>>> help
>>>>>>>>>>> get us there.
>>>>>>>>>>>
>>>>>>>>>>> Ron
>>>>>>>>>>
>>>>>>>>>> I think, on balance, I agree with Rajini that it would be nice
>>>>> to make
>>>>>>>> the configs look more like Kafka configs. We don't have camel-case
>>>>> in
>>>>>>>> configuration keys, so we should avoid it here.
>>>>>>>>>>
>>>>>>>>>> best,
>>>>>>>>>> Colin
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <
>>>>> cmccabe@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
>>>>>>>>>>>>> Hi Ron,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the detailed explanation, sounds good to me.
>>>>>>>>>>>>>
>>>>>>>>>>>>> A few more questions:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1) At the moment, all sensitive broker configs including
>>>>>>>>>>>>> keystore/truststore passwords can be stored encrypted in
>>>>>>>> ZooKeeper prior to
>>>>>>>>>>>>> starting up brokers. We are now adding SSL
>>>>> keystore/truststore
>>>>>>>> passwords
>>>>>>>>>>>>> for ZooKeeper client that cannot be stored in ZooKeeper
>>>>> since
>>>>>>>> you need
>>>>>>>>>>>>> these to connect to ZK. We should perhaps document that
>>>>> these
>>>>>>>> configs can
>>>>>>>>>>>>> be encrypted using KIP-421.
>>>>>>>>>>>>
>>>>>>>>>>>> That's a good point. Previously, we didn't have ZK
>>>>> on-the-wire
>>>>>>>> security support. However, now that we do, sending sensitive
>>>>> keystore and
>>>>>>>> truststore passwords to ZK in cleartext should be deprecated in
>>>>> favor of
>>>>>>>> using KIP-421.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2) We can dynamically update keystores and trust stores
>>>>> used by
>>>>>>>> brokers
>>>>>>>>>>>>> without restarting the broker. Can we support this easily
>>>>> for ZK
>>>>>>>> clients
>>>>>>>>>>>>> created by the broker, for example by adding our own
>>>>>>>>>>>>> `zookeeper.ssl.context.supplier.class`?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hmm. That might be better to consider in a follow-up, since
>>>>> the
>>>>>>>> deadline for 2.5 KIPs is coming up?
>>>>>>>>>>>>
>>>>>>>>>>>> best,
>>>>>>>>>>>> Colin
>>>>>>>>>>>>
>>>>>>>>>>>>> 3) Looks like we are using config names that directly map
>>>>> to ZK
>>>>>>>> configs.
>>>>>>>>>>>>> Have we considered using equivalent Kafka config names with
>>>>>>>> prefixes,
>>>>>>>>>>>>> perhaps with inheritance from the non-prefixed names? Not
>>>>> sure
>>>>>>>> if this is a
>>>>>>>>>>>>> good idea, but perhaps worth documenting in Rejected
>>>>>>>> Alternatives at least.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <
>>>>> cmccabe@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Ron,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the explanation. I guess thinking about it a
>>>>>>>> little bit more,
>>>>>>>>>>>>>> we should just add --zk-tls-config-file to all of these
>>>>>>>> commands.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We will be removing this option (plus ZK in general) from
>>>>>>>> these commands
>>>>>>>>>>>>>> in the next major release, but ZK is still supported in
>>>>> 2.5,
>>>>>>>> so we should
>>>>>>>>>>>>>> just do the logical thing. (The exception is
>>>>>>>> ZkSecurityMigrator, which
>>>>>>>>>>>>>> will stay).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> best,
>>>>>>>>>>>>>> Colin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
>>>>>>>>>>>>>>> Hi Colin.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> <<< It seems like this [--zk-tls-config-file
>>>>> information]
>>>>>>>> could just
>>>>>>>>>>>>>> appear
>>>>>>>>>>>>>>> in a configuration file, which all of these tools
>>>>> already
>>>>>>>> accept (I
>>>>>>>>>>>>>> think)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ZkSecurityMigrator has no such property file facility;
>>>>>>>> adding a
>>>>>>>>>>>>>>> "--zk-tls-config-file" parameter is exactly for this
>>>>>>>> purpose. If we add
>>>>>>>>>>>>>>> that to ZkSecurityMigrator then it is trivial to add
>>>>> it to
>>>>>>>> other commands
>>>>>>>>>>>>>>> (the same code is simply reused; it ends up being just
>>>>> a few
>>>>>>>> extra
>>>>>>>>>>>>>> lines).
>>>>>>>>>>>>>>> I do not see any parameter in the other two commands to
>>>>>>>> adjust the ZK
>>>>>>>>>>>>>>> connection; ConfigCommand accepts a "--command-config"
>>>>> flag,
>>>>>>>> but
>>>>>>>>>>>>>> according
>>>>>>>>>>>>>>> to the code "This is used only with --bootstrap-server
>>>>>>>> option for
>>>>>>>>>>>>>>> describing and altering broker configs."
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I do agree there would be no need to add
>>>>>>>> "--zk-tls-config-file" to
>>>>>>>>>>>>>>> ReassignPartitionsCommand if its direct ZK
>>>>> connectivity is
>>>>>>>> replaced in
>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>> for the next release.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ConfigCommand supports the "--bootstrap-server" option
>>>>> and
>>>>>>>> will have its
>>>>>>>>>>>>>>> direct ZooKeeper access formally deprecated as per
>>>>> KIP-555,
>>>>>>>> but the
>>>>>>>>>>>>>> special
>>>>>>>>>>>>>>> use case of bootstrapping a ZooKeeper ensemble with
>>>>>>>> encrypted credentials
>>>>>>>>>>>>>>> prior to starting Kafka will still exist, so it feels
>>>>> like
>>>>>>>> while
>>>>>>>>>>>>>>> "--zk-tls-config-file" would never be used except for
>>>>> this
>>>>>>>> use case it
>>>>>>>>>>>>>>> could probably still be added for this particular
>>>>> situation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ron
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> P.S. I responded on 1/6 but I just discovered that e,
>>>>> ail
>>>>>>>> (and 3 more I
>>>>>>>>>>>>>>> sent) did not go through. I am trying to get emails
>>>>> through
>>>>>>>> now to move
>>>>>>>>>>>>>>> this discussion forward.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <
>>>>>>>> cmccabe@apache.org> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
>>>>>>>>>>>>>>>>> Hi everyone. I would like to make the following
>>>>> changes
>>>>>>>> to the KIP.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> MOTIVATION:
>>>>>>>>>>>>>>>>> Include a statement that it will be difficult in
>>>>> the
>>>>>>>> short term to
>>>>>>>>>>>>>>>>> deprecate direct Zookeeper communication in
>>>>>>>> kafka-configs.{sh, bat}
>>>>>>>>>>>>>>>> (which
>>>>>>>>>>>>>>>>> invoke kafka.admin.ConfigCommand) because
>>>>> bootstrapping
>>>>>>>> a Kafka
>>>>>>>>>>>>>> cluster
>>>>>>>>>>>>>>>>> with encrypted passwords in Zookeeper is an
>>>>>>>> explicitly-supported use
>>>>>>>>>>>>>>>> case;
>>>>>>>>>>>>>>>>> therefore it is in scope to be able to securely
>>>>>>>> configure the CLI
>>>>>>>>>>>>>> tools
>>>>>>>>>>>>>>>>> that still leverage non-deprecated direct Zookeeper
>>>>>>>> communication
>>>>>>>>>>>>>> for TLS
>>>>>>>>>>>>>>>>> (the other 2 tools are
>>>>> kafka-reassign-partitions.{sh,
>>>>>>>> bat} and
>>>>>>>>>>>>>>>>> zookeeper-security-migration.sh).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Ron,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> About deprecations:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * zookeeper-security-migration: clearly, deprecating
>>>>> ZK
>>>>>>>> access in this
>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>> would not make sense, since it would defeat the whole
>>>>>>>> point of the
>>>>>>>>>>>>>> tool :)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * kafka-reassign-partitions: ZK access should be
>>>>>>>> deprecated here.
>>>>>>>>>>>>>> KIP-455
>>>>>>>>>>>>>>>> implementation has dragged a bit, but this should
>>>>> get done
>>>>>>>> soon.
>>>>>>>>>>>>>> Certainly
>>>>>>>>>>>>>>>> before the next release.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * kafka-configs: I think ZK access should be
>>>>> deprecated
>>>>>>>> here as well.
>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>> agree there is a super-special bootstrapping case
>>>>> here,
>>>>>>>> but that should
>>>>>>>>>>>>>>>> have its own tool, not use kafka-configs.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I will post a separate KIP for this, though.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> GOALS:
>>>>>>>>>>>>>>>>> Support the secure configuration of TLS-encrypted
>>>>>>>> communication
>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>> Zookeeper and:
>>>>>>>>>>>>>>>>> a) Kafka brokers
>>>>>>>>>>>>>>>>> b) The three CLI tools mentioned above that still
>>>>>>>> support direct,
>>>>>>>>>>>>>>>>> non-deprecated communication to Zookeeper
>>>>>>>>>>>>>>>>> It is explicitly out-of-scope to deprecate any
>>>>> direct
>>>>>>>> Zookeeper
>>>>>>>>>>>>>>>>> communication in CLI tools as part of this KIP;
>>>>> such
>>>>>>>> work will occur
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> future KIPs instead.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> PUBLIC INTERFACES:
>>>>>>>>>>>>>>>>> 1) The following new broker configurations will be
>>>>>>>> recognized.
>>>>>>>>>>>>>>>>> zookeeper.client.secure (default value = false,
>>>>> for
>>>>>>>> backwards
>>>>>>>>>>>>>>>>> compatibility)
>>>>>>>>>>>>>>>>> zookeeper.clientCnxnSocket
>>>>>>>>>>>>>>>>> zookeeper.ssl.keyStore.location
>>>>>>>>>>>>>>>>> zookeeper.ssl.keyStore.password
>>>>>>>>>>>>>>>>> zookeeper.ssl.trustStore.location
>>>>>>>>>>>>>>>>> zookeeper.ssl.trustStore.password
>>>>>>>>>>>>>>>>> It will be an error for any of the last 5 values
>>>>> to be
>>>>>>>> left
>>>>>>>>>>>>>> unspecified
>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>> zookeeper.client.secure is explicitly set to true.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2) In addition, the
>>>>>>>> kafka.security.authorizer.AclAuthorizer class
>>>>>>>>>>>>>>>> supports
>>>>>>>>>>>>>>>>> the ability to connect to a different Zookeeper
>>>>> instance
>>>>>>>> than the
>>>>>>>>>>>>>> one the
>>>>>>>>>>>>>>>>> brokers use. We therefore also add the following
>>>>>>>> optional configs,
>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>> override the corresponding ones from above when
>>>>> present:
>>>>>>>>>>>>>>>>> authorizer.zookeeper.client.secure
>>>>>>>>>>>>>>>>> authorizer.zookeeper.clientCnxnSocket
>>>>>>>>>>>>>>>>> authorizer.zookeeper.ssl.keyStore.location
>>>>>>>>>>>>>>>>> authorizer.zookeeper.ssl.keyStore.password
>>>>>>>>>>>>>>>>> authorizer.zookeeper.ssl.trustStore.location
>>>>>>>>>>>>>>>>> authorizer.zookeeper.ssl.trustStore.password
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 3) The three CLI tools mentioned above will
>>>>> support a new
>>>>>>>>>>>>>>>> --zk-tls-config-file
>>>>>>>>>>>>>>>>> <String: Zookeeper TLS configuration file path>"
>>>>>>>> option. The
>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>> properties will be recognized in that file, and
>>>>>>>> unrecognized
>>>>>>>>>>>>>> properties
>>>>>>>>>>>>>>>>> will be ignored to allow the possibility of
>>>>> pointing
>>>>>>>>>>>>>> zk-tls-config-file
>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> the broker's config file.
>>>>>>>>>>>>>>>>> zookeeper.client.secure (default value = false)
>>>>>>>>>>>>>>>>> zookeeper.clientCnxnSocket
>>>>>>>>>>>>>>>>> zookeeper.ssl.keyStore.location
>>>>>>>>>>>>>>>>> zookeeper.ssl.keyStore.password
>>>>>>>>>>>>>>>>> zookeeper.ssl.trustStore.location
>>>>>>>>>>>>>>>>> zookeeper.ssl.trustStore.password
>>>>>>>>>>>>>>>>> It will be an error for any of the last 5 values
>>>>> to be
>>>>>>>> left
>>>>>>>>>>>>>> unspecified
>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>> zookeeper.client.secure is explicitly set to true.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Do we really need a --zk-tls-config-file flag? It
>>>>> seems
>>>>>>>> like this
>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>> just appear in a configuration file, which all of
>>>>> these
>>>>>>>> tools already
>>>>>>>>>>>>>>>> accept (I think).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> best,
>>>>>>>>>>>>>>>> Colin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ron
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <
>>>>>>>> rndgstn@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi everyone. Let's get this discussion going
>>>>> again
>>>>>>>> now that Kafka
>>>>>>>>>>>>>> 2.4
>>>>>>>>>>>>>>>>>> with Zookeeper 3.5.6 is out.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> First, regarding the KIP number, the other KIP
>>>>> that
>>>>>>>> was using this
>>>>>>>>>>>>>>>> number
>>>>>>>>>>>>>>>>>> moved to KIP 534, so KIP 515 remains the correct
>>>>>>>> number for this
>>>>>>>>>>>>>>>>>> discussion. I've updated the Kafka Improvement
>>>>>>>> Proposal page to
>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> KIP in the 515 slot, so we're all set there.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regarding the actual issues under discussion, I
>>>>> think
>>>>>>>> there are
>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>> things we should clarify.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 1) It is possible to use TLS connectivity to
>>>>> Zookeeper
>>>>>>>> from Apache
>>>>>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>>>>> 2.4 -- the problem is that configuration
>>>>> information
>>>>>>>> has to be
>>>>>>>>>>>>>> passed
>>>>>>>>>>>>>>>> via
>>>>>>>>>>>>>>>>>> system properties as "-D" command line options
>>>>> on the
>>>>>>>> java
>>>>>>>>>>>>>> invocation
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> the broker, and those are not secure (anyone with
>>>>>>>> access to the box
>>>>>>>>>>>>>>>> can see
>>>>>>>>>>>>>>>>>> the command line used to invoke the broker); the
>>>>>>>> configuration
>>>>>>>>>>>>>> includes
>>>>>>>>>>>>>>>>>> sensitive information (e.g. a keystore
>>>>> password), so
>>>>>>>> we need a
>>>>>>>>>>>>>> secure
>>>>>>>>>>>>>>>>>> mechanism for passing the configuration values.
>>>>> I
>>>>>>>> believe the real
>>>>>>>>>>>>>>>>>> motivation for this KIP is to harden the
>>>>> configuration
>>>>>>>> mechanism
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> Zookeeper TLS connectivity.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2) I believe the list of CLI tools that continue
>>>>> to
>>>>>>>> use direct
>>>>>>>>>>>>>>>> Zookeeper
>>>>>>>>>>>>>>>>>> connectivity in a non-deprecated fashion is:
>>>>>>>>>>>>>>>>>> a)
>>>>>>>>>>>>>>
>>>>> zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
>>>>>>>>>>>>>>>>>> b)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>
>>>>>
>> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
>>>>>>>>>>>>>>>>>> c) kafka-configs.{sh,
>>>>> bat}/kafka.admin.ConfigCommand
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 3) I believe Kafka.admin.ConfigCommand presents a
>>>>>>>> conundrum
>>>>>>>>>>>>>> because it
>>>>>>>>>>>>>>>>>> explicitly states in a comment that a supported
>>>>> use
>>>>>>>> case is
>>>>>>>>>>>>>>>> bootstrapping a
>>>>>>>>>>>>>>>>>> Kafka cluster with encrypted passwords in
>>>>> Zookeeper
>>>>>>>> (see
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>
>>>>>
>> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>>> This means it will be especially difficult to
>>>>>>>> deprecate this
>>>>>>>>>>>>>> particular
>>>>>>>>>>>>>>>>>> direct Zookeeper connectivity without a different
>>>>>>>> storage
>>>>>>>>>>>>>> mechanism for
>>>>>>>>>>>>>>>>>> dynamic configuration values being available
>>>>> (i.e. the
>>>>>>>> self-managed
>>>>>>>>>>>>>>>> quorum
>>>>>>>>>>>>>>>>>> referred to in KIP-500).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I think it would be easier and simpler to harden
>>>>> the
>>>>>>>> Zookeeper TLS
>>>>>>>>>>>>>>>>>> configuration in both Kafka and in CLI tools
>>>>> compared
>>>>>>>> to trying to
>>>>>>>>>>>>>>>>>> deprecate the direct Zookeeper connectivity in
>>>>> the
>>>>>>>> above 3 CLI
>>>>>>>>>>>>>> tools --
>>>>>>>>>>>>>>>>>> especially when ConfigCommand has no obvious
>>>>>>>> short-term path to
>>>>>>>>>>>>>>>> deprecation.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regarding how to harden the TLS configuration,
>>>>> adding
>>>>>>>> the
>>>>>>>>>>>>>> appropriate
>>>>>>>>>>>>>>>>>> configs to Kafka is an obvious choice for the
>>>>>>>> brokers. Not that
>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>> values cannot be dynamically reconfigurable
>>>>> because
>>>>>>>> the values
>>>>>>>>>>>>>> would be
>>>>>>>>>>>>>>>>>> required to connect to Zookeeper via TLS in the
>>>>> first
>>>>>>>> place.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regarding how to harden the TLS configuration
>>>>> for the
>>>>>>>> 3 remaining
>>>>>>>>>>>>>> CLI
>>>>>>>>>>>>>>>>>> tools, it would be relatively straightforward to
>>>>> add
>>>>>>>> support for a
>>>>>>>>>>>>>>>>>> "--zk-tls-config-file <String: Zookeeper TLS
>>>>>>>> configuration system
>>>>>>>>>>>>>>>>>> properties file path>" option to each one.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Ron
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes
>>>>> <
>>>>>>>>>>>>>> pere.urbon@gmail.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> True, the number is duplicated.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> do you know how can we get the problem fix? I
>>>>> was
>>>>>>>> not aware, I
>>>>>>>>>>>>>>>> missed,
>>>>>>>>>>>>>>>>>>> sorry the need to add the KIP to the table.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -- Pere
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Missatge de Jorge Esteban Quilcate Otoya <
>>>>>>>>>>>>>> quilcate.jorge@gmail.com>
>>>>>>>>>>>>>>>> del
>>>>>>>>>>>>>>>>>>> dia
>>>>>>>>>>>>>>>>>>> dt., 3 de set. 2019 a les 13:43:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Pere,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Have you add your KIP to the list here
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>>>>>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I found the KIP number assigned to another.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón
>>>>> Bayes <
>>>>>>>>>>>>>>>> pere.urbon@gmail.com>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for your time Harsha,
>>>>>>>>>>>>>>>>>>>>> anyone else with comments? looking
>>>>> forward to
>>>>>>>> hearing from
>>>>>>>>>>>>>> you.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Stupid question: when do you move from
>>>>> discussion
>>>>>>>> to vote?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Missatge de Harsha Chintalapani <
>>>>> kafka@harsha.io>
>>>>>>>> del dia
>>>>>>>>>>>>>> dv., 30
>>>>>>>>>>>>>>>>>>> d’ag.
>>>>>>>>>>>>>>>>>>>>> 2019 a les 21:59:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks Pere. KIP looks good to me.
>>>>>>>>>>>>>>>>>>>>>> -Harsha
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 30, 2019 at 10:05 AM, Pere
>>>>> Urbón
>>>>>>>> Bayes <
>>>>>>>>>>>>>>>>>>>>> pere.urbon@gmail.com>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Not really,
>>>>>>>>>>>>>>>>>>>>>>> my idea is to keep the JAAS parameter,
>>>>> so
>>>>>>>> people don't see
>>>>>>>>>>>>>>>> major
>>>>>>>>>>>>>>>>>>>>>>> changes. But if you pass a properties
>>>>> file,
>>>>>>>> then this takes
>>>>>>>>>>>>>>>>>>> precedence
>>>>>>>>>>>>>>>>>>>>> over
>>>>>>>>>>>>>>>>>>>>>>> the other, with the idea that you can do
>>>>> sasl
>>>>>>>> as well with
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> properties
>>>>>>>>>>>>>>>>>>>>>>> files.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Makes sense?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> -- Pere
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Missatge de Harsha Chintalapani <
>>>>>>>> kafka@harsha.io> del dia
>>>>>>>>>>>>>> dv.,
>>>>>>>>>>>>>>>> 30
>>>>>>>>>>>>>>>>>>>>> d’ag.
>>>>>>>>>>>>>>>>>>>>>>> 2019 a les 19:00:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi Pere,
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. Enabling
>>>>> SSL for
>>>>>>>> zookeeper
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>>>>>>>> makes
>>>>>>>>>>>>>>>>>>>>>>>> sense.
>>>>>>>>>>>>>>>>>>>>>>>> "The changes are planned to be
>>>>> introduced in a
>>>>>>>> compatible
>>>>>>>>>>>>>> way,
>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>> keeping the current JAAS variable
>>>>> precedence."
>>>>>>>>>>>>>>>>>>>>>>>> Can you elaborate a bit here. If the user
>>>>>>>> configures a JAAS
>>>>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>> Client section it will take precedence
>>>>> over
>>>>>>>> zookeeper SSL
>>>>>>>>>>>>>>>> configs?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> Harsha
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 30, 2019 at 7:50 AM, Pere
>>>>> Urbón
>>>>>>>> Bayes <
>>>>>>>>>>>>>>>>>>>>> pere.urbon@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>> quick question, I saw in another mail
>>>>> that
>>>>>>>> 2.4 release is
>>>>>>>>>>>>>>>> planned
>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>> September. I think it would be really
>>>>> awesome
>>>>>>>> to have
>>>>>>>>>>>>>> this for
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>> release, do you think we can make it?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> -- Pere
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Missatge de Pere Urbón Bayes <
>>>>>>>> pere.urbon@gmail.com> del
>>>>>>>>>>>>>> dia
>>>>>>>>>>>>>>>> dj.,
>>>>>>>>>>>>>>>>>>> 29
>>>>>>>>>>>>>>>>>>>>>>>>> d’ag. 2019 a les 20:10:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>> this is my first KIP for a change in
>>>>> Apache
>>>>>>>> Kafka, so I'm
>>>>>>>>>>>>>>>> really
>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>>>> to the process. Looking forward to
>>>>> hearing
>>>>>>>> from you and
>>>>>>>>>>>>>> learn
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> best
>>>>>>>>>>>>>>>>>>>>>>>>> ropes here.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I would like to propose this KIP-515 to
>>>>>>>> enable the
>>>>>>>>>>>>>>>>>>> ZookeeperClients
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> take full advantage of the TLS
>>>>> communication
>>>>>>>> in the new
>>>>>>>>>>>>>>>> Zookeeper
>>>>>>>>>>>>>>>>>>>>> 3.5.5.
>>>>>>>>>>>>>>>>>>>>>>>>> Specially interesting it the Zookeeper
>>>>>>>> Security Migration,
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>>>>>>>>>>> this change will not work with TLS,
>>>>> disabling
>>>>>>>> users to use
>>>>>>>>>>>>>>>> ACLs
>>>>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> Zookeeper cluster use TLS.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> link:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to hearing from you on
>>>>> this,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> /cheers
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>>>>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
>>>>>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>>>>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
>>>>>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
>>>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>>>>>>> http://www.purbon.com
>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>
>>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Ron,
For Kafka, we moved from hostname verification disabled-by-default to
enabled-by-default under
https://cwiki.apache.org/confluence/display/KAFKA/KIP-294+-+Enable+TLS+hostname+verification+by+default.
So we have tested empty String for disabling hostname verification and we
know that it works, not only in test environments, but also in actual
deployments. Basically, "ssl.endpoint.identification.algorithm=" in the
file results in a config value of empty String and not setting it results
in the default value of HTTPS since the value is null.I would probably do
the same for the ZK config. If you are creating certificates for ZooKeeper
without hostnames, then you may be doing the same for Kafka too and in both
cases, the config would be an empty string. This is insecure anyway and we
would really expect this only in test environments. Not sure if it is worth
making an exception for this in terms of consistent config names.
On Fri, Jan 17, 2020 at 2:37 PM Ron Dagostino <rn...@gmail.com> wrote:
> Hi again, Rajini. I've updated the KIP, and while doing it I became
> concerned about using zookeeper.ssl.endpoint.identification.algorithm
> for enabling/disabling hostname verification. The KIP reflects what
> we decided, but upon reading it, I wonder if it is workable. Here's
> what it says for this config:
>
> "Specifies whether to enable hostname verification in the ZooKeeper
> TLS negotiation process, with (case-insensitively) "https" meaning
> ZooKeeper hostname verification is enabled and an explicit blank value
> meaning it is disabled (disabling it is only recommended for testing
> purposes). Overrides any explicit "true" or "false" value set via the
> <code>zookeeper.ssl.hostnameVerification</code> system property (true
> implying https and false implying blank) and inherits the value of
> <code>ssl.endpoint.identification.algorithm</code> (if any) if no
> explicit value or non-value is set both here and via the system
> property."
>
> I wonder if someone sets an explicit empty value
> (zookeeper.ssl.endpoint.identification.algorithm=) in the config file,
> will we be able to distinguish between that and a config file that
> doesn't have it at all? And even if we can, this feels pretty forced
> to me, like we are bending over backwards for consistency when
> "zookeeper.ssl.hostname.verification.enable" is a natural fit. I
> doubt this config is going to be used very much, so it really does
> feel to me like we should just go with the natural fit and ditch the
> compatibility and inheritance for this one config.
>
> Ron
>
> On Fri, Jan 17, 2020 at 8:07 AM Ron Dagostino <rn...@gmail.com> wrote:
> >
> > Hi Rajini. I’ll submit a documentation PR to clarify the Kafka behavior
> of #1 and will adopt the same config for ZK. I agree now we should inherit
> for AclAuthorizer too — I was just stuck on the “no inheritance” idea more
> than I realized, and while the ZK quorums are different that doesn’t
> necessarily mean the same client identity wouldn’t be valid on both (and if
> not then the configs can still be overridden).
> >
> > I’ll update the KIP today and let this version soak for a day or two,
> and then I’ll start a vote if no other issues/comments arise.
> >
> > Ron
> >
> > > On Jan 17, 2020, at 6:08 AM, Rajini Sivaram <ra...@gmail.com>
> wrote:
> > >
> > > Hi Ron,
> > >
> > > Unresolved item #1: Yes, Kafka disables hostname verification if
> > > "ssl.endpoint.identification.algorithm" is an empty string.
> > >
> > > Unresolved item #2: Yes, those 9 plus hostname verification. If we do
> > > decide to inherit Kafka configs, wouldn't we inherit these 10 in
> > > AclAuthorizer as well?
> > >
> > > Unresolved item #3: Agree, following up on the discussion thread for
> > > KIP-553.
> > >
> > >
> > >> On Fri, Jan 17, 2020 at 3:29 AM Ron Dagostino <rn...@gmail.com>
> wrote:
> > >>
> > >> Hi Colin. TLS to ZooKeeper will remain an opt-in. The configuration
> > >> value that turns it on is zookeeper.ssl.client.enable (which used to
> > >> be called zookeeper.client.secure before we renamed everything to be
> > >> consistent with Kafka's config naming style and eliminate confusion).
> > >> The default value for this is false. Everything else related to TLS
> > >> has no effect unless this is explicitly set to true, and this value
> > >> does not inherit from anywhere,
> > >>
> > >> Ron
> > >>
> > >>> On Thu, Jan 16, 2020 at 5:39 PM Colin McCabe <cm...@apache.org>
> wrote:
> > >>>
> > >>>> On Thu, Jan 16, 2020, at 02:58, Rajini Sivaram wrote:
> > >>>> Hi Ron/Colin,
> > >>>>
> > >>>> Thanks for the responses. I think we need to consider these cases:
> > >>>>
> > >>>> 1) The recommended approach: This is the one we document. Configs
> > >>>> consistent with Kafka make sense here. It seems a lot more
> confusing to
> > >>>> configure `ssl.cipher.suites`, `ssl.enabled.protocols`, `
> > >>>> zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols`
> with
> > >>>> multiple naming inconsistencies in server.properties. For users who
> > >> have
> > >>>> never used ZK system properties, this is clearly simpler. Another
> > >> config to
> > >>>> consider is `ssl.endpoint.identification.algorithm` which does the
> same
> > >>>> thing as `zookeeper.ssl.hostnameVerification` - if we are staying
> > >>>> consistent, we should use the Kafka format for this too. I would
> also
> > >>>> consider using something like `zookeeper.client.ssl.enabled` instead
> > >> of `
> > >>>> zookeeper.client.secure` since it is difficult to tell what `secure`
> > >> means
> > >>>> when you have options to configure Kerberos, ACLs and/or SSL.
> > >>>>
> > >>>> 2) Inheritance using prefixed configs as we do for other prefixed
> > >> configs
> > >>>> in Kafka: I think it may be useful for configs like `ssl.protocol`
> and
> > >> `
> > >>>> ssl.cipher.suites`.
> > >>>>
> > >>>> 3) Default values: Do we want to use Kafka defaults? The
> inconsistency
> > >> we
> > >>>> have is `ssl.protocol/ssl.enabled.protocols` since we still have
> > >> insecure
> > >>>> protocols enabled for Kafka. We have KIP-553 (
> > >>>>
> > >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
> > >> )
> > >>>> open to disable these. It is currently blocked since it also talks
> > >> about
> > >>>> enabling TLSv1.3 by default and we haven't done sufficient testing
> for
> > >>>> that. Since it will be good to disable the older insecure protocols
> > >> anyway,
> > >>>> perhaps we could do that without enabling TLSv1.3 by default in 2.5?
> > >>>
> > >>> Hi Rajini,
> > >>>
> > >>> Inheriting the SSL settings from Kafka would have the effect of
> changing
> > >> the defaults for almost everyone using SSL from non-SSL ZK to
> SSL-based ZK,
> > >> right? This seems to expand the scope of the KIP greatly, since it
> moves
> > >> it from being opt-in to opt-out. Are there any potential negative
> > >> implications of this?
> > >>>
> > >>> best,
> > >>> Colin
> > >>>
> > >>>
> > >>>>
> > >>>> 4) Backward compatibility for deployments which are using system
> > >>>> properties. We shouldn't override system properties with Kafka
> > >> defaults or
> > >>>> inherited values. But we do want to override if user configures
> > >> properties
> > >>>> explicitly. In the current approach, this was simpler since we just
> > >> had to
> > >>>> set the configured values. But if we decide to inherit Kafka
> configs,
> > >> then
> > >>>> we will need to check system properties and update accordingly. But
> > >> that
> > >>>> should be doable?
> > >>>>
> > >>>> On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe <cm...@apache.org>
> > >> wrote:
> > >>>>
> > >>>>>> On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote:
> > >>>>>> Hi Colin. Two things come to mind with respect to ZooKeeper
> > >> camelCase
> > >>>>>> style vs Kafka-style config names for ZooKeeper. First, I think
> it
> > >>>>>> would be desirable for the client configs and broker configs to be
> > >>>>>> interoperable. For example, it feels like it would be convenient
> > >> to
> > >>>>>> be able to pass the broker's config file to the ZK Security
> > >> Migrator
> > >>>>>> tool. I think whatever we use (ZooKeeper camelCase style or Kafka
> > >>>>>> style), I think we should use the same for both.
> > >>>>>>
> > >>>>>> The second thing to think about is the fact that ZooKeeper
> > >>>>>> configuration inherently uses Java system properties as a first
> > >> pass.
> > >>>>>> So while we might switch to Kafka-style configs in the config file
> > >>>>>> (e.g. zookeeper.ssl.trust.store.location), the
> > >>>>>> org.apache.zookeeper.client.ZKClientConfig class (which is how TLS
> > >>>>>> configs are set) is going to look for the camelCase
> > >>>>>> zookeeper.ssl.trustStore.location system property and use any
> value
> > >>>>>> there. I think this could be very confusing for people. Granted,
> > >> we
> > >>>>>> are doing this so that people don't have to use system properties,
> > >> but
> > >>>>>> there is no way to turn off the use of system properties, so I
> > >> think
> > >>>>>> there would be pretty substantial potential for confusion.
> > >>>>>
> > >>>>> I don't think we document or recommend that anyone use system
> > >> properties
> > >>>>> to configure Zookeeper usage within Kafka. And I think the reason
> > >> is
> > >>>>> exactly the issue you pointed out -- it wouldn't be consistent with
> > >> our
> > >>>>> other configurations.
> > >>>>>
> > >>>>> best,
> > >>>>> Colin
> > >>>>>
> > >>>>>>
> > >>>>>> One idea is to migrate the system properties -- i.e. look to see
> if
> > >>>>>> zookeeper.ssl.trustStore.location is set and if it is clear out
> the
> > >>>>>> value there and move it under the
> > >> zookeeper.ssl.trust.store.location
> > >>>>>> system property. (I said it was an idea -- not necessarily a good
> > >> one
> > >>>>>> :-)
> > >>>>>>
> > >>>>>> Ron
> > >>>>>>
> > >>>>>> On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe <cm...@apache.org>
> > >> wrote:
> > >>>>>>>
> > >>>>>>>> On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
> > >>>>>>>> Thanks Colin and Rajini.
> > >>>>>>>>
> > >>>>>>>> I've updated the KIP to say that KIP-421 functionality is
> > >> available
> > >>>>> to
> > >>>>>>>> encrypt sensitive configs like the ZK key store and trust store
> > >>>>>>>> passwords. (I've also made it clear that the configs are not
> > >>>>>>>> dynamically reconfigurable since dynamic values are stored in
> > >> ZK and
> > >>>>>>>> the data is needed to get to ZK in the first place). Colin,
> > >> let me
> > >>>>>>>> know if you think anything else needs to be said/done about it
> > >> -- I
> > >>>>>>>> wasn't sure if your comment above implies that there are
> > >> additional
> > >>>>>>>> direct actions that we need to take aside from this.
> > >>>>>>>>
> > >>>>>>>> I agree that making the brokers' ZooKeeper clients dynamic with
> > >>>>>>>> respect to key and trust stores (e.g. via
> > >>>>>>>> zookeeper.ssl.context.supplier.class) may increase the risk
> > >> that this
> > >>>>>>>> KIP may not land in time for 2.5.
> > >>>>>>>>
> > >>>>>>>> Regarding the config names for ZK being different than the
> > >> ZooKeeper
> > >>>>>>>> configs (e.g. camel-case keyStore/trustStore instead of
> > >>>>>>>> keystore/truststore, ciphersuites instead of cipher.suites,
> > >>>>>>>> enabledProtocols instead of enabled.protocols), I agree it is
> > >> an
> > >>>>>>>> issue. I tried to mitigate it by putting warning text in the
> > >> config
> > >>>>>>>> docs for these. Regarding configuration inheritance, I think
> > >> what
> > >>>>> you
> > >>>>>>>> are saying is that for any configs where the broker has a clear
> > >>>>>>>> semantic equivalent, the broker could fall back to the broker
> > >> value
> > >>>>> if
> > >>>>>>>> the ZK value is not given. The potential list is:
> > >>>>>>>>
> > >>>>>>>> zookeeper.ssl.keyStore.location => ssl.keystore.location
> > >>>>>>>> zookeeper.ssl.keyStore.password => ssl.keystore.password
> > >>>>>>>> zookeeper.ssl.keyStore.type => ssl.keystore.type
> > >>>>>>>> zookeeper.ssl.trustStore.location => ssl.truststore.location
> > >>>>>>>> zookeeper.ssl.trustStore.password => ssl.truststore.password
> > >>>>>>>> zookeeper.ssl.trustStore.type => ssl.truststore.type
> > >>>>>>>> zookeeper.ssl.ciphersuites => ssl.cipher.suites
> > >>>>>>>>
> > >>>>>>>> Note that there are two configs that look the same based on
> > >> their key
> > >>>>>>>> names but whose allowable values differ:
> > >>>>>>>>
> > >>>>>>>> zookeeper.ssl.protocol(Default: TLSv1.2) =>
> > >> ssl.protocol(Default:
> > >>>>> TLS)
> > >>>>>>>> zookeeper.ssl.enabledProtocols(Default: value of protocol
> > >> property) =
> > >>>>>>>> ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
> > >>>>>>>>
> > >>>>>>>> Not sure what the right answer is, but hopefully this detail
> > >> will
> > >>>>> help
> > >>>>>>>> get us there.
> > >>>>>>>>
> > >>>>>>>> Ron
> > >>>>>>>
> > >>>>>>> I think, on balance, I agree with Rajini that it would be nice
> > >> to make
> > >>>>> the configs look more like Kafka configs. We don't have camel-case
> > >> in
> > >>>>> configuration keys, so we should avoid it here.
> > >>>>>>>
> > >>>>>>> best,
> > >>>>>>> Colin
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <
> > >> cmccabe@apache.org>
> > >>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> > >>>>>>>>>> Hi Ron,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for the detailed explanation, sounds good to me.
> > >>>>>>>>>>
> > >>>>>>>>>> A few more questions:
> > >>>>>>>>>>
> > >>>>>>>>>> 1) At the moment, all sensitive broker configs including
> > >>>>>>>>>> keystore/truststore passwords can be stored encrypted in
> > >>>>> ZooKeeper prior to
> > >>>>>>>>>> starting up brokers. We are now adding SSL
> > >> keystore/truststore
> > >>>>> passwords
> > >>>>>>>>>> for ZooKeeper client that cannot be stored in ZooKeeper
> > >> since
> > >>>>> you need
> > >>>>>>>>>> these to connect to ZK. We should perhaps document that
> > >> these
> > >>>>> configs can
> > >>>>>>>>>> be encrypted using KIP-421.
> > >>>>>>>>>
> > >>>>>>>>> That's a good point. Previously, we didn't have ZK
> > >> on-the-wire
> > >>>>> security support. However, now that we do, sending sensitive
> > >> keystore and
> > >>>>> truststore passwords to ZK in cleartext should be deprecated in
> > >> favor of
> > >>>>> using KIP-421.
> > >>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> 2) We can dynamically update keystores and trust stores
> > >> used by
> > >>>>> brokers
> > >>>>>>>>>> without restarting the broker. Can we support this easily
> > >> for ZK
> > >>>>> clients
> > >>>>>>>>>> created by the broker, for example by adding our own
> > >>>>>>>>>> `zookeeper.ssl.context.supplier.class`?
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Hmm. That might be better to consider in a follow-up, since
> > >> the
> > >>>>> deadline for 2.5 KIPs is coming up?
> > >>>>>>>>>
> > >>>>>>>>> best,
> > >>>>>>>>> Colin
> > >>>>>>>>>
> > >>>>>>>>>> 3) Looks like we are using config names that directly map
> > >> to ZK
> > >>>>> configs.
> > >>>>>>>>>> Have we considered using equivalent Kafka config names with
> > >>>>> prefixes,
> > >>>>>>>>>> perhaps with inheritance from the non-prefixed names? Not
> > >> sure
> > >>>>> if this is a
> > >>>>>>>>>> good idea, but perhaps worth documenting in Rejected
> > >>>>> Alternatives at least.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <
> > >> cmccabe@apache.org>
> > >>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi Ron,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for the explanation. I guess thinking about it a
> > >>>>> little bit more,
> > >>>>>>>>>>> we should just add --zk-tls-config-file to all of these
> > >>>>> commands.
> > >>>>>>>>>>>
> > >>>>>>>>>>> We will be removing this option (plus ZK in general) from
> > >>>>> these commands
> > >>>>>>>>>>> in the next major release, but ZK is still supported in
> > >> 2.5,
> > >>>>> so we should
> > >>>>>>>>>>> just do the logical thing. (The exception is
> > >>>>> ZkSecurityMigrator, which
> > >>>>>>>>>>> will stay).
> > >>>>>>>>>>>
> > >>>>>>>>>>> best,
> > >>>>>>>>>>> Colin
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > >>>>>>>>>>>> Hi Colin.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> <<< It seems like this [--zk-tls-config-file
> > >> information]
> > >>>>> could just
> > >>>>>>>>>>> appear
> > >>>>>>>>>>>> in a configuration file, which all of these tools
> > >> already
> > >>>>> accept (I
> > >>>>>>>>>>> think)
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ZkSecurityMigrator has no such property file facility;
> > >>>>> adding a
> > >>>>>>>>>>>> "--zk-tls-config-file" parameter is exactly for this
> > >>>>> purpose. If we add
> > >>>>>>>>>>>> that to ZkSecurityMigrator then it is trivial to add
> > >> it to
> > >>>>> other commands
> > >>>>>>>>>>>> (the same code is simply reused; it ends up being just
> > >> a few
> > >>>>> extra
> > >>>>>>>>>>> lines).
> > >>>>>>>>>>>> I do not see any parameter in the other two commands to
> > >>>>> adjust the ZK
> > >>>>>>>>>>>> connection; ConfigCommand accepts a "--command-config"
> > >> flag,
> > >>>>> but
> > >>>>>>>>>>> according
> > >>>>>>>>>>>> to the code "This is used only with --bootstrap-server
> > >>>>> option for
> > >>>>>>>>>>>> describing and altering broker configs."
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I do agree there would be no need to add
> > >>>>> "--zk-tls-config-file" to
> > >>>>>>>>>>>> ReassignPartitionsCommand if its direct ZK
> > >> connectivity is
> > >>>>> replaced in
> > >>>>>>>>>>> time
> > >>>>>>>>>>>> for the next release.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ConfigCommand supports the "--bootstrap-server" option
> > >> and
> > >>>>> will have its
> > >>>>>>>>>>>> direct ZooKeeper access formally deprecated as per
> > >> KIP-555,
> > >>>>> but the
> > >>>>>>>>>>> special
> > >>>>>>>>>>>> use case of bootstrapping a ZooKeeper ensemble with
> > >>>>> encrypted credentials
> > >>>>>>>>>>>> prior to starting Kafka will still exist, so it feels
> > >> like
> > >>>>> while
> > >>>>>>>>>>>> "--zk-tls-config-file" would never be used except for
> > >> this
> > >>>>> use case it
> > >>>>>>>>>>>> could probably still be added for this particular
> > >> situation.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Ron
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> P.S. I responded on 1/6 but I just discovered that e,
> > >> ail
> > >>>>> (and 3 more I
> > >>>>>>>>>>>> sent) did not go through. I am trying to get emails
> > >> through
> > >>>>> now to move
> > >>>>>>>>>>>> this discussion forward.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <
> > >>>>> cmccabe@apache.org> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > >>>>>>>>>>>>>> Hi everyone. I would like to make the following
> > >> changes
> > >>>>> to the KIP.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> MOTIVATION:
> > >>>>>>>>>>>>>> Include a statement that it will be difficult in
> > >> the
> > >>>>> short term to
> > >>>>>>>>>>>>>> deprecate direct Zookeeper communication in
> > >>>>> kafka-configs.{sh, bat}
> > >>>>>>>>>>>>> (which
> > >>>>>>>>>>>>>> invoke kafka.admin.ConfigCommand) because
> > >> bootstrapping
> > >>>>> a Kafka
> > >>>>>>>>>>> cluster
> > >>>>>>>>>>>>>> with encrypted passwords in Zookeeper is an
> > >>>>> explicitly-supported use
> > >>>>>>>>>>>>> case;
> > >>>>>>>>>>>>>> therefore it is in scope to be able to securely
> > >>>>> configure the CLI
> > >>>>>>>>>>> tools
> > >>>>>>>>>>>>>> that still leverage non-deprecated direct Zookeeper
> > >>>>> communication
> > >>>>>>>>>>> for TLS
> > >>>>>>>>>>>>>> (the other 2 tools are
> > >> kafka-reassign-partitions.{sh,
> > >>>>> bat} and
> > >>>>>>>>>>>>>> zookeeper-security-migration.sh).
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi Ron,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks for the KIP.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> About deprecations:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> * zookeeper-security-migration: clearly, deprecating
> > >> ZK
> > >>>>> access in this
> > >>>>>>>>>>> one
> > >>>>>>>>>>>>> would not make sense, since it would defeat the whole
> > >>>>> point of the
> > >>>>>>>>>>> tool :)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> * kafka-reassign-partitions: ZK access should be
> > >>>>> deprecated here.
> > >>>>>>>>>>> KIP-455
> > >>>>>>>>>>>>> implementation has dragged a bit, but this should
> > >> get done
> > >>>>> soon.
> > >>>>>>>>>>> Certainly
> > >>>>>>>>>>>>> before the next release.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> * kafka-configs: I think ZK access should be
> > >> deprecated
> > >>>>> here as well.
> > >>>>>>>>>>> I
> > >>>>>>>>>>>>> agree there is a super-special bootstrapping case
> > >> here,
> > >>>>> but that should
> > >>>>>>>>>>>>> have its own tool, not use kafka-configs.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I will post a separate KIP for this, though.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> GOALS:
> > >>>>>>>>>>>>>> Support the secure configuration of TLS-encrypted
> > >>>>> communication
> > >>>>>>>>>>> between
> > >>>>>>>>>>>>>> Zookeeper and:
> > >>>>>>>>>>>>>> a) Kafka brokers
> > >>>>>>>>>>>>>> b) The three CLI tools mentioned above that still
> > >>>>> support direct,
> > >>>>>>>>>>>>>> non-deprecated communication to Zookeeper
> > >>>>>>>>>>>>>> It is explicitly out-of-scope to deprecate any
> > >> direct
> > >>>>> Zookeeper
> > >>>>>>>>>>>>>> communication in CLI tools as part of this KIP;
> > >> such
> > >>>>> work will occur
> > >>>>>>>>>>> in
> > >>>>>>>>>>>>>> future KIPs instead.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> PUBLIC INTERFACES:
> > >>>>>>>>>>>>>> 1) The following new broker configurations will be
> > >>>>> recognized.
> > >>>>>>>>>>>>>> zookeeper.client.secure (default value = false,
> > >> for
> > >>>>> backwards
> > >>>>>>>>>>>>>> compatibility)
> > >>>>>>>>>>>>>> zookeeper.clientCnxnSocket
> > >>>>>>>>>>>>>> zookeeper.ssl.keyStore.location
> > >>>>>>>>>>>>>> zookeeper.ssl.keyStore.password
> > >>>>>>>>>>>>>> zookeeper.ssl.trustStore.location
> > >>>>>>>>>>>>>> zookeeper.ssl.trustStore.password
> > >>>>>>>>>>>>>> It will be an error for any of the last 5 values
> > >> to be
> > >>>>> left
> > >>>>>>>>>>> unspecified
> > >>>>>>>>>>>>> if
> > >>>>>>>>>>>>>> zookeeper.client.secure is explicitly set to true.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 2) In addition, the
> > >>>>> kafka.security.authorizer.AclAuthorizer class
> > >>>>>>>>>>>>> supports
> > >>>>>>>>>>>>>> the ability to connect to a different Zookeeper
> > >> instance
> > >>>>> than the
> > >>>>>>>>>>> one the
> > >>>>>>>>>>>>>> brokers use. We therefore also add the following
> > >>>>> optional configs,
> > >>>>>>>>>>> which
> > >>>>>>>>>>>>>> override the corresponding ones from above when
> > >> present:
> > >>>>>>>>>>>>>> authorizer.zookeeper.client.secure
> > >>>>>>>>>>>>>> authorizer.zookeeper.clientCnxnSocket
> > >>>>>>>>>>>>>> authorizer.zookeeper.ssl.keyStore.location
> > >>>>>>>>>>>>>> authorizer.zookeeper.ssl.keyStore.password
> > >>>>>>>>>>>>>> authorizer.zookeeper.ssl.trustStore.location
> > >>>>>>>>>>>>>> authorizer.zookeeper.ssl.trustStore.password
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 3) The three CLI tools mentioned above will
> > >> support a new
> > >>>>>>>>>>>>> --zk-tls-config-file
> > >>>>>>>>>>>>>> <String: Zookeeper TLS configuration file path>"
> > >>>>> option. The
> > >>>>>>>>>>> following
> > >>>>>>>>>>>>>> properties will be recognized in that file, and
> > >>>>> unrecognized
> > >>>>>>>>>>> properties
> > >>>>>>>>>>>>>> will be ignored to allow the possibility of
> > >> pointing
> > >>>>>>>>>>> zk-tls-config-file
> > >>>>>>>>>>>>> at
> > >>>>>>>>>>>>>> the broker's config file.
> > >>>>>>>>>>>>>> zookeeper.client.secure (default value = false)
> > >>>>>>>>>>>>>> zookeeper.clientCnxnSocket
> > >>>>>>>>>>>>>> zookeeper.ssl.keyStore.location
> > >>>>>>>>>>>>>> zookeeper.ssl.keyStore.password
> > >>>>>>>>>>>>>> zookeeper.ssl.trustStore.location
> > >>>>>>>>>>>>>> zookeeper.ssl.trustStore.password
> > >>>>>>>>>>>>>> It will be an error for any of the last 5 values
> > >> to be
> > >>>>> left
> > >>>>>>>>>>> unspecified
> > >>>>>>>>>>>>> if
> > >>>>>>>>>>>>>> zookeeper.client.secure is explicitly set to true.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Do we really need a --zk-tls-config-file flag? It
> > >> seems
> > >>>>> like this
> > >>>>>>>>>>> could
> > >>>>>>>>>>>>> just appear in a configuration file, which all of
> > >> these
> > >>>>> tools already
> > >>>>>>>>>>>>> accept (I think).
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> best,
> > >>>>>>>>>>>>> Colin
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Ron
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <
> > >>>>> rndgstn@gmail.com>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi everyone. Let's get this discussion going
> > >> again
> > >>>>> now that Kafka
> > >>>>>>>>>>> 2.4
> > >>>>>>>>>>>>>>> with Zookeeper 3.5.6 is out.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> First, regarding the KIP number, the other KIP
> > >> that
> > >>>>> was using this
> > >>>>>>>>>>>>> number
> > >>>>>>>>>>>>>>> moved to KIP 534, so KIP 515 remains the correct
> > >>>>> number for this
> > >>>>>>>>>>>>>>> discussion. I've updated the Kafka Improvement
> > >>>>> Proposal page to
> > >>>>>>>>>>> list
> > >>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>> KIP in the 515 slot, so we're all set there.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Regarding the actual issues under discussion, I
> > >> think
> > >>>>> there are
> > >>>>>>>>>>> some
> > >>>>>>>>>>>>>>> things we should clarify.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 1) It is possible to use TLS connectivity to
> > >> Zookeeper
> > >>>>> from Apache
> > >>>>>>>>>>>>> Kafka
> > >>>>>>>>>>>>>>> 2.4 -- the problem is that configuration
> > >> information
> > >>>>> has to be
> > >>>>>>>>>>> passed
> > >>>>>>>>>>>>> via
> > >>>>>>>>>>>>>>> system properties as "-D" command line options
> > >> on the
> > >>>>> java
> > >>>>>>>>>>> invocation
> > >>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>> the broker, and those are not secure (anyone with
> > >>>>> access to the box
> > >>>>>>>>>>>>> can see
> > >>>>>>>>>>>>>>> the command line used to invoke the broker); the
> > >>>>> configuration
> > >>>>>>>>>>> includes
> > >>>>>>>>>>>>>>> sensitive information (e.g. a keystore
> > >> password), so
> > >>>>> we need a
> > >>>>>>>>>>> secure
> > >>>>>>>>>>>>>>> mechanism for passing the configuration values.
> > >> I
> > >>>>> believe the real
> > >>>>>>>>>>>>>>> motivation for this KIP is to harden the
> > >> configuration
> > >>>>> mechanism
> > >>>>>>>>>>> for
> > >>>>>>>>>>>>>>> Zookeeper TLS connectivity.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 2) I believe the list of CLI tools that continue
> > >> to
> > >>>>> use direct
> > >>>>>>>>>>>>> Zookeeper
> > >>>>>>>>>>>>>>> connectivity in a non-deprecated fashion is:
> > >>>>>>>>>>>>>>> a)
> > >>>>>>>>>>>
> > >> zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > >>>>>>>>>>>>>>> b)
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>
> > >>
> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > >>>>>>>>>>>>>>> c) kafka-configs.{sh,
> > >> bat}/kafka.admin.ConfigCommand
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 3) I believe Kafka.admin.ConfigCommand presents a
> > >>>>> conundrum
> > >>>>>>>>>>> because it
> > >>>>>>>>>>>>>>> explicitly states in a comment that a supported
> > >> use
> > >>>>> case is
> > >>>>>>>>>>>>> bootstrapping a
> > >>>>>>>>>>>>>>> Kafka cluster with encrypted passwords in
> > >> Zookeeper
> > >>>>> (see
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>
> > >>
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > >>>>>>>>>>>>> ).
> > >>>>>>>>>>>>>>> This means it will be especially difficult to
> > >>>>> deprecate this
> > >>>>>>>>>>> particular
> > >>>>>>>>>>>>>>> direct Zookeeper connectivity without a different
> > >>>>> storage
> > >>>>>>>>>>> mechanism for
> > >>>>>>>>>>>>>>> dynamic configuration values being available
> > >> (i.e. the
> > >>>>> self-managed
> > >>>>>>>>>>>>> quorum
> > >>>>>>>>>>>>>>> referred to in KIP-500).
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I think it would be easier and simpler to harden
> > >> the
> > >>>>> Zookeeper TLS
> > >>>>>>>>>>>>>>> configuration in both Kafka and in CLI tools
> > >> compared
> > >>>>> to trying to
> > >>>>>>>>>>>>>>> deprecate the direct Zookeeper connectivity in
> > >> the
> > >>>>> above 3 CLI
> > >>>>>>>>>>> tools --
> > >>>>>>>>>>>>>>> especially when ConfigCommand has no obvious
> > >>>>> short-term path to
> > >>>>>>>>>>>>> deprecation.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Regarding how to harden the TLS configuration,
> > >> adding
> > >>>>> the
> > >>>>>>>>>>> appropriate
> > >>>>>>>>>>>>>>> configs to Kafka is an obvious choice for the
> > >>>>> brokers. Not that
> > >>>>>>>>>>> these
> > >>>>>>>>>>>>>>> values cannot be dynamically reconfigurable
> > >> because
> > >>>>> the values
> > >>>>>>>>>>> would be
> > >>>>>>>>>>>>>>> required to connect to Zookeeper via TLS in the
> > >> first
> > >>>>> place.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Regarding how to harden the TLS configuration
> > >> for the
> > >>>>> 3 remaining
> > >>>>>>>>>>> CLI
> > >>>>>>>>>>>>>>> tools, it would be relatively straightforward to
> > >> add
> > >>>>> support for a
> > >>>>>>>>>>>>>>> "--zk-tls-config-file <String: Zookeeper TLS
> > >>>>> configuration system
> > >>>>>>>>>>>>>>> properties file path>" option to each one.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thoughts?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Ron
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes
> > >> <
> > >>>>>>>>>>> pere.urbon@gmail.com
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> True, the number is duplicated.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> do you know how can we get the problem fix? I
> > >> was
> > >>>>> not aware, I
> > >>>>>>>>>>>>> missed,
> > >>>>>>>>>>>>>>>> sorry the need to add the KIP to the table.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> -- Pere
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Missatge de Jorge Esteban Quilcate Otoya <
> > >>>>>>>>>>> quilcate.jorge@gmail.com>
> > >>>>>>>>>>>>> del
> > >>>>>>>>>>>>>>>> dia
> > >>>>>>>>>>>>>>>> dt., 3 de set. 2019 a les 13:43:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi Pere,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Have you add your KIP to the list here
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > >>>>>>>>>>>>>>>> ?
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I found the KIP number assigned to another.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón
> > >> Bayes <
> > >>>>>>>>>>>>> pere.urbon@gmail.com>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks for your time Harsha,
> > >>>>>>>>>>>>>>>>>> anyone else with comments? looking
> > >> forward to
> > >>>>> hearing from
> > >>>>>>>>>>> you.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Stupid question: when do you move from
> > >> discussion
> > >>>>> to vote?
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Missatge de Harsha Chintalapani <
> > >> kafka@harsha.io>
> > >>>>> del dia
> > >>>>>>>>>>> dv., 30
> > >>>>>>>>>>>>>>>> d’ag.
> > >>>>>>>>>>>>>>>>>> 2019 a les 21:59:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Thanks Pere. KIP looks good to me.
> > >>>>>>>>>>>>>>>>>>> -Harsha
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On Fri, Aug 30, 2019 at 10:05 AM, Pere
> > >> Urbón
> > >>>>> Bayes <
> > >>>>>>>>>>>>>>>>>> pere.urbon@gmail.com>
> > >>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Not really,
> > >>>>>>>>>>>>>>>>>>>> my idea is to keep the JAAS parameter,
> > >> so
> > >>>>> people don't see
> > >>>>>>>>>>>>> major
> > >>>>>>>>>>>>>>>>>>>> changes. But if you pass a properties
> > >> file,
> > >>>>> then this takes
> > >>>>>>>>>>>>>>>> precedence
> > >>>>>>>>>>>>>>>>>> over
> > >>>>>>>>>>>>>>>>>>>> the other, with the idea that you can do
> > >> sasl
> > >>>>> as well with
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> properties
> > >>>>>>>>>>>>>>>>>>>> files.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Makes sense?
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> -- Pere
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Missatge de Harsha Chintalapani <
> > >>>>> kafka@harsha.io> del dia
> > >>>>>>>>>>> dv.,
> > >>>>>>>>>>>>> 30
> > >>>>>>>>>>>>>>>>>> d’ag.
> > >>>>>>>>>>>>>>>>>>>> 2019 a les 19:00:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hi Pere,
> > >>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. Enabling
> > >> SSL for
> > >>>>> zookeeper
> > >>>>>>>>>>> for
> > >>>>>>>>>>>>> Kafka
> > >>>>>>>>>>>>>>>>>> makes
> > >>>>>>>>>>>>>>>>>>>>> sense.
> > >>>>>>>>>>>>>>>>>>>>> "The changes are planned to be
> > >> introduced in a
> > >>>>> compatible
> > >>>>>>>>>>> way,
> > >>>>>>>>>>>>> by
> > >>>>>>>>>>>>>>>>>>>>> keeping the current JAAS variable
> > >> precedence."
> > >>>>>>>>>>>>>>>>>>>>> Can you elaborate a bit here. If the user
> > >>>>> configures a JAAS
> > >>>>>>>>>>>>> file
> > >>>>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>>>>>> Client section it will take precedence
> > >> over
> > >>>>> zookeeper SSL
> > >>>>>>>>>>>>> configs?
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>> Harsha
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> On Fri, Aug 30, 2019 at 7:50 AM, Pere
> > >> Urbón
> > >>>>> Bayes <
> > >>>>>>>>>>>>>>>>>> pere.urbon@gmail.com>
> > >>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Hi,
> > >>>>>>>>>>>>>>>>>>>>>> quick question, I saw in another mail
> > >> that
> > >>>>> 2.4 release is
> > >>>>>>>>>>>>> planned
> > >>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>> September. I think it would be really
> > >> awesome
> > >>>>> to have
> > >>>>>>>>>>> this for
> > >>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>> release, do you think we can make it?
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> -- Pere
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Missatge de Pere Urbón Bayes <
> > >>>>> pere.urbon@gmail.com> del
> > >>>>>>>>>>> dia
> > >>>>>>>>>>>>> dj.,
> > >>>>>>>>>>>>>>>> 29
> > >>>>>>>>>>>>>>>>>>>>>> d’ag. 2019 a les 20:10:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Hi,
> > >>>>>>>>>>>>>>>>>>>>>> this is my first KIP for a change in
> > >> Apache
> > >>>>> Kafka, so I'm
> > >>>>>>>>>>>>> really
> > >>>>>>>>>>>>>>>> need
> > >>>>>>>>>>>>>>>>>>>>>> to the process. Looking forward to
> > >> hearing
> > >>>>> from you and
> > >>>>>>>>>>> learn
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> best
> > >>>>>>>>>>>>>>>>>>>>>> ropes here.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> I would like to propose this KIP-515 to
> > >>>>> enable the
> > >>>>>>>>>>>>>>>> ZookeeperClients
> > >>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>> take full advantage of the TLS
> > >> communication
> > >>>>> in the new
> > >>>>>>>>>>>>> Zookeeper
> > >>>>>>>>>>>>>>>>>> 3.5.5.
> > >>>>>>>>>>>>>>>>>>>>>> Specially interesting it the Zookeeper
> > >>>>> Security Migration,
> > >>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>> without
> > >>>>>>>>>>>>>>>>>>>>>> this change will not work with TLS,
> > >> disabling
> > >>>>> users to use
> > >>>>>>>>>>>>> ACLs
> > >>>>>>>>>>>>>>>> when
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>> Zookeeper cluster use TLS.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> link:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Looking forward to hearing from you on
> > >> this,
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> /cheers
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
> > >>>>>>>>>>>>>>>>>>>>>> Software Architect
> > >>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
> > >>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
> > >>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
> > >>>>>>>>>>>>>>>>>>>>>> Software Architect
> > >>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
> > >>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
> > >>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
> > >>>>>>>>>>>>>>>>>>>> Software Architect
> > >>>>>>>>>>>>>>>>>>>> http://www.purbon.com
> > >>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
> > >>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
> > >>>>>>>>>>>>>>>>>> Software Architect
> > >>>>>>>>>>>>>>>>>> http://www.purbon.com
> > >>>>>>>>>>>>>>>>>> https://twitter.com/purbon
> > >>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>>> Pere Urbon-Bayes
> > >>>>>>>>>>>>>>>> Software Architect
> > >>>>>>>>>>>>>>>> http://www.purbon.com
> > >>>>>>>>>>>>>>>> https://twitter.com/purbon
> > >>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> > >>
>
>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Ron Dagostino <rn...@gmail.com>.
Hi again, Rajini. I've updated the KIP, and while doing it I became
concerned about using zookeeper.ssl.endpoint.identification.algorithm
for enabling/disabling hostname verification. The KIP reflects what
we decided, but upon reading it, I wonder if it is workable. Here's
what it says for this config:
"Specifies whether to enable hostname verification in the ZooKeeper
TLS negotiation process, with (case-insensitively) "https" meaning
ZooKeeper hostname verification is enabled and an explicit blank value
meaning it is disabled (disabling it is only recommended for testing
purposes). Overrides any explicit "true" or "false" value set via the
<code>zookeeper.ssl.hostnameVerification</code> system property (true
implying https and false implying blank) and inherits the value of
<code>ssl.endpoint.identification.algorithm</code> (if any) if no
explicit value or non-value is set both here and via the system
property."
I wonder if someone sets an explicit empty value
(zookeeper.ssl.endpoint.identification.algorithm=) in the config file,
will we be able to distinguish between that and a config file that
doesn't have it at all? And even if we can, this feels pretty forced
to me, like we are bending over backwards for consistency when
"zookeeper.ssl.hostname.verification.enable" is a natural fit. I
doubt this config is going to be used very much, so it really does
feel to me like we should just go with the natural fit and ditch the
compatibility and inheritance for this one config.
Ron
On Fri, Jan 17, 2020 at 8:07 AM Ron Dagostino <rn...@gmail.com> wrote:
>
> Hi Rajini. I’ll submit a documentation PR to clarify the Kafka behavior of #1 and will adopt the same config for ZK. I agree now we should inherit for AclAuthorizer too — I was just stuck on the “no inheritance” idea more than I realized, and while the ZK quorums are different that doesn’t necessarily mean the same client identity wouldn’t be valid on both (and if not then the configs can still be overridden).
>
> I’ll update the KIP today and let this version soak for a day or two, and then I’ll start a vote if no other issues/comments arise.
>
> Ron
>
> > On Jan 17, 2020, at 6:08 AM, Rajini Sivaram <ra...@gmail.com> wrote:
> >
> > Hi Ron,
> >
> > Unresolved item #1: Yes, Kafka disables hostname verification if
> > "ssl.endpoint.identification.algorithm" is an empty string.
> >
> > Unresolved item #2: Yes, those 9 plus hostname verification. If we do
> > decide to inherit Kafka configs, wouldn't we inherit these 10 in
> > AclAuthorizer as well?
> >
> > Unresolved item #3: Agree, following up on the discussion thread for
> > KIP-553.
> >
> >
> >> On Fri, Jan 17, 2020 at 3:29 AM Ron Dagostino <rn...@gmail.com> wrote:
> >>
> >> Hi Colin. TLS to ZooKeeper will remain an opt-in. The configuration
> >> value that turns it on is zookeeper.ssl.client.enable (which used to
> >> be called zookeeper.client.secure before we renamed everything to be
> >> consistent with Kafka's config naming style and eliminate confusion).
> >> The default value for this is false. Everything else related to TLS
> >> has no effect unless this is explicitly set to true, and this value
> >> does not inherit from anywhere,
> >>
> >> Ron
> >>
> >>> On Thu, Jan 16, 2020 at 5:39 PM Colin McCabe <cm...@apache.org> wrote:
> >>>
> >>>> On Thu, Jan 16, 2020, at 02:58, Rajini Sivaram wrote:
> >>>> Hi Ron/Colin,
> >>>>
> >>>> Thanks for the responses. I think we need to consider these cases:
> >>>>
> >>>> 1) The recommended approach: This is the one we document. Configs
> >>>> consistent with Kafka make sense here. It seems a lot more confusing to
> >>>> configure `ssl.cipher.suites`, `ssl.enabled.protocols`, `
> >>>> zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols` with
> >>>> multiple naming inconsistencies in server.properties. For users who
> >> have
> >>>> never used ZK system properties, this is clearly simpler. Another
> >> config to
> >>>> consider is `ssl.endpoint.identification.algorithm` which does the same
> >>>> thing as `zookeeper.ssl.hostnameVerification` - if we are staying
> >>>> consistent, we should use the Kafka format for this too. I would also
> >>>> consider using something like `zookeeper.client.ssl.enabled` instead
> >> of `
> >>>> zookeeper.client.secure` since it is difficult to tell what `secure`
> >> means
> >>>> when you have options to configure Kerberos, ACLs and/or SSL.
> >>>>
> >>>> 2) Inheritance using prefixed configs as we do for other prefixed
> >> configs
> >>>> in Kafka: I think it may be useful for configs like `ssl.protocol` and
> >> `
> >>>> ssl.cipher.suites`.
> >>>>
> >>>> 3) Default values: Do we want to use Kafka defaults? The inconsistency
> >> we
> >>>> have is `ssl.protocol/ssl.enabled.protocols` since we still have
> >> insecure
> >>>> protocols enabled for Kafka. We have KIP-553 (
> >>>>
> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
> >> )
> >>>> open to disable these. It is currently blocked since it also talks
> >> about
> >>>> enabling TLSv1.3 by default and we haven't done sufficient testing for
> >>>> that. Since it will be good to disable the older insecure protocols
> >> anyway,
> >>>> perhaps we could do that without enabling TLSv1.3 by default in 2.5?
> >>>
> >>> Hi Rajini,
> >>>
> >>> Inheriting the SSL settings from Kafka would have the effect of changing
> >> the defaults for almost everyone using SSL from non-SSL ZK to SSL-based ZK,
> >> right? This seems to expand the scope of the KIP greatly, since it moves
> >> it from being opt-in to opt-out. Are there any potential negative
> >> implications of this?
> >>>
> >>> best,
> >>> Colin
> >>>
> >>>
> >>>>
> >>>> 4) Backward compatibility for deployments which are using system
> >>>> properties. We shouldn't override system properties with Kafka
> >> defaults or
> >>>> inherited values. But we do want to override if user configures
> >> properties
> >>>> explicitly. In the current approach, this was simpler since we just
> >> had to
> >>>> set the configured values. But if we decide to inherit Kafka configs,
> >> then
> >>>> we will need to check system properties and update accordingly. But
> >> that
> >>>> should be doable?
> >>>>
> >>>> On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe <cm...@apache.org>
> >> wrote:
> >>>>
> >>>>>> On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote:
> >>>>>> Hi Colin. Two things come to mind with respect to ZooKeeper
> >> camelCase
> >>>>>> style vs Kafka-style config names for ZooKeeper. First, I think it
> >>>>>> would be desirable for the client configs and broker configs to be
> >>>>>> interoperable. For example, it feels like it would be convenient
> >> to
> >>>>>> be able to pass the broker's config file to the ZK Security
> >> Migrator
> >>>>>> tool. I think whatever we use (ZooKeeper camelCase style or Kafka
> >>>>>> style), I think we should use the same for both.
> >>>>>>
> >>>>>> The second thing to think about is the fact that ZooKeeper
> >>>>>> configuration inherently uses Java system properties as a first
> >> pass.
> >>>>>> So while we might switch to Kafka-style configs in the config file
> >>>>>> (e.g. zookeeper.ssl.trust.store.location), the
> >>>>>> org.apache.zookeeper.client.ZKClientConfig class (which is how TLS
> >>>>>> configs are set) is going to look for the camelCase
> >>>>>> zookeeper.ssl.trustStore.location system property and use any value
> >>>>>> there. I think this could be very confusing for people. Granted,
> >> we
> >>>>>> are doing this so that people don't have to use system properties,
> >> but
> >>>>>> there is no way to turn off the use of system properties, so I
> >> think
> >>>>>> there would be pretty substantial potential for confusion.
> >>>>>
> >>>>> I don't think we document or recommend that anyone use system
> >> properties
> >>>>> to configure Zookeeper usage within Kafka. And I think the reason
> >> is
> >>>>> exactly the issue you pointed out -- it wouldn't be consistent with
> >> our
> >>>>> other configurations.
> >>>>>
> >>>>> best,
> >>>>> Colin
> >>>>>
> >>>>>>
> >>>>>> One idea is to migrate the system properties -- i.e. look to see if
> >>>>>> zookeeper.ssl.trustStore.location is set and if it is clear out the
> >>>>>> value there and move it under the
> >> zookeeper.ssl.trust.store.location
> >>>>>> system property. (I said it was an idea -- not necessarily a good
> >> one
> >>>>>> :-)
> >>>>>>
> >>>>>> Ron
> >>>>>>
> >>>>>> On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe <cm...@apache.org>
> >> wrote:
> >>>>>>>
> >>>>>>>> On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
> >>>>>>>> Thanks Colin and Rajini.
> >>>>>>>>
> >>>>>>>> I've updated the KIP to say that KIP-421 functionality is
> >> available
> >>>>> to
> >>>>>>>> encrypt sensitive configs like the ZK key store and trust store
> >>>>>>>> passwords. (I've also made it clear that the configs are not
> >>>>>>>> dynamically reconfigurable since dynamic values are stored in
> >> ZK and
> >>>>>>>> the data is needed to get to ZK in the first place). Colin,
> >> let me
> >>>>>>>> know if you think anything else needs to be said/done about it
> >> -- I
> >>>>>>>> wasn't sure if your comment above implies that there are
> >> additional
> >>>>>>>> direct actions that we need to take aside from this.
> >>>>>>>>
> >>>>>>>> I agree that making the brokers' ZooKeeper clients dynamic with
> >>>>>>>> respect to key and trust stores (e.g. via
> >>>>>>>> zookeeper.ssl.context.supplier.class) may increase the risk
> >> that this
> >>>>>>>> KIP may not land in time for 2.5.
> >>>>>>>>
> >>>>>>>> Regarding the config names for ZK being different than the
> >> ZooKeeper
> >>>>>>>> configs (e.g. camel-case keyStore/trustStore instead of
> >>>>>>>> keystore/truststore, ciphersuites instead of cipher.suites,
> >>>>>>>> enabledProtocols instead of enabled.protocols), I agree it is
> >> an
> >>>>>>>> issue. I tried to mitigate it by putting warning text in the
> >> config
> >>>>>>>> docs for these. Regarding configuration inheritance, I think
> >> what
> >>>>> you
> >>>>>>>> are saying is that for any configs where the broker has a clear
> >>>>>>>> semantic equivalent, the broker could fall back to the broker
> >> value
> >>>>> if
> >>>>>>>> the ZK value is not given. The potential list is:
> >>>>>>>>
> >>>>>>>> zookeeper.ssl.keyStore.location => ssl.keystore.location
> >>>>>>>> zookeeper.ssl.keyStore.password => ssl.keystore.password
> >>>>>>>> zookeeper.ssl.keyStore.type => ssl.keystore.type
> >>>>>>>> zookeeper.ssl.trustStore.location => ssl.truststore.location
> >>>>>>>> zookeeper.ssl.trustStore.password => ssl.truststore.password
> >>>>>>>> zookeeper.ssl.trustStore.type => ssl.truststore.type
> >>>>>>>> zookeeper.ssl.ciphersuites => ssl.cipher.suites
> >>>>>>>>
> >>>>>>>> Note that there are two configs that look the same based on
> >> their key
> >>>>>>>> names but whose allowable values differ:
> >>>>>>>>
> >>>>>>>> zookeeper.ssl.protocol(Default: TLSv1.2) =>
> >> ssl.protocol(Default:
> >>>>> TLS)
> >>>>>>>> zookeeper.ssl.enabledProtocols(Default: value of protocol
> >> property) =
> >>>>>>>> ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
> >>>>>>>>
> >>>>>>>> Not sure what the right answer is, but hopefully this detail
> >> will
> >>>>> help
> >>>>>>>> get us there.
> >>>>>>>>
> >>>>>>>> Ron
> >>>>>>>
> >>>>>>> I think, on balance, I agree with Rajini that it would be nice
> >> to make
> >>>>> the configs look more like Kafka configs. We don't have camel-case
> >> in
> >>>>> configuration keys, so we should avoid it here.
> >>>>>>>
> >>>>>>> best,
> >>>>>>> Colin
> >>>>>>>
> >>>>>>>>
> >>>>>>>> On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <
> >> cmccabe@apache.org>
> >>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> >>>>>>>>>> Hi Ron,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the detailed explanation, sounds good to me.
> >>>>>>>>>>
> >>>>>>>>>> A few more questions:
> >>>>>>>>>>
> >>>>>>>>>> 1) At the moment, all sensitive broker configs including
> >>>>>>>>>> keystore/truststore passwords can be stored encrypted in
> >>>>> ZooKeeper prior to
> >>>>>>>>>> starting up brokers. We are now adding SSL
> >> keystore/truststore
> >>>>> passwords
> >>>>>>>>>> for ZooKeeper client that cannot be stored in ZooKeeper
> >> since
> >>>>> you need
> >>>>>>>>>> these to connect to ZK. We should perhaps document that
> >> these
> >>>>> configs can
> >>>>>>>>>> be encrypted using KIP-421.
> >>>>>>>>>
> >>>>>>>>> That's a good point. Previously, we didn't have ZK
> >> on-the-wire
> >>>>> security support. However, now that we do, sending sensitive
> >> keystore and
> >>>>> truststore passwords to ZK in cleartext should be deprecated in
> >> favor of
> >>>>> using KIP-421.
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> 2) We can dynamically update keystores and trust stores
> >> used by
> >>>>> brokers
> >>>>>>>>>> without restarting the broker. Can we support this easily
> >> for ZK
> >>>>> clients
> >>>>>>>>>> created by the broker, for example by adding our own
> >>>>>>>>>> `zookeeper.ssl.context.supplier.class`?
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Hmm. That might be better to consider in a follow-up, since
> >> the
> >>>>> deadline for 2.5 KIPs is coming up?
> >>>>>>>>>
> >>>>>>>>> best,
> >>>>>>>>> Colin
> >>>>>>>>>
> >>>>>>>>>> 3) Looks like we are using config names that directly map
> >> to ZK
> >>>>> configs.
> >>>>>>>>>> Have we considered using equivalent Kafka config names with
> >>>>> prefixes,
> >>>>>>>>>> perhaps with inheritance from the non-prefixed names? Not
> >> sure
> >>>>> if this is a
> >>>>>>>>>> good idea, but perhaps worth documenting in Rejected
> >>>>> Alternatives at least.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <
> >> cmccabe@apache.org>
> >>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Ron,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the explanation. I guess thinking about it a
> >>>>> little bit more,
> >>>>>>>>>>> we should just add --zk-tls-config-file to all of these
> >>>>> commands.
> >>>>>>>>>>>
> >>>>>>>>>>> We will be removing this option (plus ZK in general) from
> >>>>> these commands
> >>>>>>>>>>> in the next major release, but ZK is still supported in
> >> 2.5,
> >>>>> so we should
> >>>>>>>>>>> just do the logical thing. (The exception is
> >>>>> ZkSecurityMigrator, which
> >>>>>>>>>>> will stay).
> >>>>>>>>>>>
> >>>>>>>>>>> best,
> >>>>>>>>>>> Colin
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> >>>>>>>>>>>> Hi Colin.
> >>>>>>>>>>>>
> >>>>>>>>>>>> <<< It seems like this [--zk-tls-config-file
> >> information]
> >>>>> could just
> >>>>>>>>>>> appear
> >>>>>>>>>>>> in a configuration file, which all of these tools
> >> already
> >>>>> accept (I
> >>>>>>>>>>> think)
> >>>>>>>>>>>>
> >>>>>>>>>>>> ZkSecurityMigrator has no such property file facility;
> >>>>> adding a
> >>>>>>>>>>>> "--zk-tls-config-file" parameter is exactly for this
> >>>>> purpose. If we add
> >>>>>>>>>>>> that to ZkSecurityMigrator then it is trivial to add
> >> it to
> >>>>> other commands
> >>>>>>>>>>>> (the same code is simply reused; it ends up being just
> >> a few
> >>>>> extra
> >>>>>>>>>>> lines).
> >>>>>>>>>>>> I do not see any parameter in the other two commands to
> >>>>> adjust the ZK
> >>>>>>>>>>>> connection; ConfigCommand accepts a "--command-config"
> >> flag,
> >>>>> but
> >>>>>>>>>>> according
> >>>>>>>>>>>> to the code "This is used only with --bootstrap-server
> >>>>> option for
> >>>>>>>>>>>> describing and altering broker configs."
> >>>>>>>>>>>>
> >>>>>>>>>>>> I do agree there would be no need to add
> >>>>> "--zk-tls-config-file" to
> >>>>>>>>>>>> ReassignPartitionsCommand if its direct ZK
> >> connectivity is
> >>>>> replaced in
> >>>>>>>>>>> time
> >>>>>>>>>>>> for the next release.
> >>>>>>>>>>>>
> >>>>>>>>>>>> ConfigCommand supports the "--bootstrap-server" option
> >> and
> >>>>> will have its
> >>>>>>>>>>>> direct ZooKeeper access formally deprecated as per
> >> KIP-555,
> >>>>> but the
> >>>>>>>>>>> special
> >>>>>>>>>>>> use case of bootstrapping a ZooKeeper ensemble with
> >>>>> encrypted credentials
> >>>>>>>>>>>> prior to starting Kafka will still exist, so it feels
> >> like
> >>>>> while
> >>>>>>>>>>>> "--zk-tls-config-file" would never be used except for
> >> this
> >>>>> use case it
> >>>>>>>>>>>> could probably still be added for this particular
> >> situation.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Ron
> >>>>>>>>>>>>
> >>>>>>>>>>>> P.S. I responded on 1/6 but I just discovered that e,
> >> ail
> >>>>> (and 3 more I
> >>>>>>>>>>>> sent) did not go through. I am trying to get emails
> >> through
> >>>>> now to move
> >>>>>>>>>>>> this discussion forward.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <
> >>>>> cmccabe@apache.org> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> >>>>>>>>>>>>>> Hi everyone. I would like to make the following
> >> changes
> >>>>> to the KIP.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> MOTIVATION:
> >>>>>>>>>>>>>> Include a statement that it will be difficult in
> >> the
> >>>>> short term to
> >>>>>>>>>>>>>> deprecate direct Zookeeper communication in
> >>>>> kafka-configs.{sh, bat}
> >>>>>>>>>>>>> (which
> >>>>>>>>>>>>>> invoke kafka.admin.ConfigCommand) because
> >> bootstrapping
> >>>>> a Kafka
> >>>>>>>>>>> cluster
> >>>>>>>>>>>>>> with encrypted passwords in Zookeeper is an
> >>>>> explicitly-supported use
> >>>>>>>>>>>>> case;
> >>>>>>>>>>>>>> therefore it is in scope to be able to securely
> >>>>> configure the CLI
> >>>>>>>>>>> tools
> >>>>>>>>>>>>>> that still leverage non-deprecated direct Zookeeper
> >>>>> communication
> >>>>>>>>>>> for TLS
> >>>>>>>>>>>>>> (the other 2 tools are
> >> kafka-reassign-partitions.{sh,
> >>>>> bat} and
> >>>>>>>>>>>>>> zookeeper-security-migration.sh).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Ron,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the KIP.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> About deprecations:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> * zookeeper-security-migration: clearly, deprecating
> >> ZK
> >>>>> access in this
> >>>>>>>>>>> one
> >>>>>>>>>>>>> would not make sense, since it would defeat the whole
> >>>>> point of the
> >>>>>>>>>>> tool :)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> * kafka-reassign-partitions: ZK access should be
> >>>>> deprecated here.
> >>>>>>>>>>> KIP-455
> >>>>>>>>>>>>> implementation has dragged a bit, but this should
> >> get done
> >>>>> soon.
> >>>>>>>>>>> Certainly
> >>>>>>>>>>>>> before the next release.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> * kafka-configs: I think ZK access should be
> >> deprecated
> >>>>> here as well.
> >>>>>>>>>>> I
> >>>>>>>>>>>>> agree there is a super-special bootstrapping case
> >> here,
> >>>>> but that should
> >>>>>>>>>>>>> have its own tool, not use kafka-configs.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I will post a separate KIP for this, though.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> GOALS:
> >>>>>>>>>>>>>> Support the secure configuration of TLS-encrypted
> >>>>> communication
> >>>>>>>>>>> between
> >>>>>>>>>>>>>> Zookeeper and:
> >>>>>>>>>>>>>> a) Kafka brokers
> >>>>>>>>>>>>>> b) The three CLI tools mentioned above that still
> >>>>> support direct,
> >>>>>>>>>>>>>> non-deprecated communication to Zookeeper
> >>>>>>>>>>>>>> It is explicitly out-of-scope to deprecate any
> >> direct
> >>>>> Zookeeper
> >>>>>>>>>>>>>> communication in CLI tools as part of this KIP;
> >> such
> >>>>> work will occur
> >>>>>>>>>>> in
> >>>>>>>>>>>>>> future KIPs instead.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> PUBLIC INTERFACES:
> >>>>>>>>>>>>>> 1) The following new broker configurations will be
> >>>>> recognized.
> >>>>>>>>>>>>>> zookeeper.client.secure (default value = false,
> >> for
> >>>>> backwards
> >>>>>>>>>>>>>> compatibility)
> >>>>>>>>>>>>>> zookeeper.clientCnxnSocket
> >>>>>>>>>>>>>> zookeeper.ssl.keyStore.location
> >>>>>>>>>>>>>> zookeeper.ssl.keyStore.password
> >>>>>>>>>>>>>> zookeeper.ssl.trustStore.location
> >>>>>>>>>>>>>> zookeeper.ssl.trustStore.password
> >>>>>>>>>>>>>> It will be an error for any of the last 5 values
> >> to be
> >>>>> left
> >>>>>>>>>>> unspecified
> >>>>>>>>>>>>> if
> >>>>>>>>>>>>>> zookeeper.client.secure is explicitly set to true.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2) In addition, the
> >>>>> kafka.security.authorizer.AclAuthorizer class
> >>>>>>>>>>>>> supports
> >>>>>>>>>>>>>> the ability to connect to a different Zookeeper
> >> instance
> >>>>> than the
> >>>>>>>>>>> one the
> >>>>>>>>>>>>>> brokers use. We therefore also add the following
> >>>>> optional configs,
> >>>>>>>>>>> which
> >>>>>>>>>>>>>> override the corresponding ones from above when
> >> present:
> >>>>>>>>>>>>>> authorizer.zookeeper.client.secure
> >>>>>>>>>>>>>> authorizer.zookeeper.clientCnxnSocket
> >>>>>>>>>>>>>> authorizer.zookeeper.ssl.keyStore.location
> >>>>>>>>>>>>>> authorizer.zookeeper.ssl.keyStore.password
> >>>>>>>>>>>>>> authorizer.zookeeper.ssl.trustStore.location
> >>>>>>>>>>>>>> authorizer.zookeeper.ssl.trustStore.password
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 3) The three CLI tools mentioned above will
> >> support a new
> >>>>>>>>>>>>> --zk-tls-config-file
> >>>>>>>>>>>>>> <String: Zookeeper TLS configuration file path>"
> >>>>> option. The
> >>>>>>>>>>> following
> >>>>>>>>>>>>>> properties will be recognized in that file, and
> >>>>> unrecognized
> >>>>>>>>>>> properties
> >>>>>>>>>>>>>> will be ignored to allow the possibility of
> >> pointing
> >>>>>>>>>>> zk-tls-config-file
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>> the broker's config file.
> >>>>>>>>>>>>>> zookeeper.client.secure (default value = false)
> >>>>>>>>>>>>>> zookeeper.clientCnxnSocket
> >>>>>>>>>>>>>> zookeeper.ssl.keyStore.location
> >>>>>>>>>>>>>> zookeeper.ssl.keyStore.password
> >>>>>>>>>>>>>> zookeeper.ssl.trustStore.location
> >>>>>>>>>>>>>> zookeeper.ssl.trustStore.password
> >>>>>>>>>>>>>> It will be an error for any of the last 5 values
> >> to be
> >>>>> left
> >>>>>>>>>>> unspecified
> >>>>>>>>>>>>> if
> >>>>>>>>>>>>>> zookeeper.client.secure is explicitly set to true.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Do we really need a --zk-tls-config-file flag? It
> >> seems
> >>>>> like this
> >>>>>>>>>>> could
> >>>>>>>>>>>>> just appear in a configuration file, which all of
> >> these
> >>>>> tools already
> >>>>>>>>>>>>> accept (I think).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> best,
> >>>>>>>>>>>>> Colin
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ron
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <
> >>>>> rndgstn@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi everyone. Let's get this discussion going
> >> again
> >>>>> now that Kafka
> >>>>>>>>>>> 2.4
> >>>>>>>>>>>>>>> with Zookeeper 3.5.6 is out.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> First, regarding the KIP number, the other KIP
> >> that
> >>>>> was using this
> >>>>>>>>>>>>> number
> >>>>>>>>>>>>>>> moved to KIP 534, so KIP 515 remains the correct
> >>>>> number for this
> >>>>>>>>>>>>>>> discussion. I've updated the Kafka Improvement
> >>>>> Proposal page to
> >>>>>>>>>>> list
> >>>>>>>>>>>>> this
> >>>>>>>>>>>>>>> KIP in the 515 slot, so we're all set there.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Regarding the actual issues under discussion, I
> >> think
> >>>>> there are
> >>>>>>>>>>> some
> >>>>>>>>>>>>>>> things we should clarify.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 1) It is possible to use TLS connectivity to
> >> Zookeeper
> >>>>> from Apache
> >>>>>>>>>>>>> Kafka
> >>>>>>>>>>>>>>> 2.4 -- the problem is that configuration
> >> information
> >>>>> has to be
> >>>>>>>>>>> passed
> >>>>>>>>>>>>> via
> >>>>>>>>>>>>>>> system properties as "-D" command line options
> >> on the
> >>>>> java
> >>>>>>>>>>> invocation
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>>> the broker, and those are not secure (anyone with
> >>>>> access to the box
> >>>>>>>>>>>>> can see
> >>>>>>>>>>>>>>> the command line used to invoke the broker); the
> >>>>> configuration
> >>>>>>>>>>> includes
> >>>>>>>>>>>>>>> sensitive information (e.g. a keystore
> >> password), so
> >>>>> we need a
> >>>>>>>>>>> secure
> >>>>>>>>>>>>>>> mechanism for passing the configuration values.
> >> I
> >>>>> believe the real
> >>>>>>>>>>>>>>> motivation for this KIP is to harden the
> >> configuration
> >>>>> mechanism
> >>>>>>>>>>> for
> >>>>>>>>>>>>>>> Zookeeper TLS connectivity.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2) I believe the list of CLI tools that continue
> >> to
> >>>>> use direct
> >>>>>>>>>>>>> Zookeeper
> >>>>>>>>>>>>>>> connectivity in a non-deprecated fashion is:
> >>>>>>>>>>>>>>> a)
> >>>>>>>>>>>
> >> zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> >>>>>>>>>>>>>>> b)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> >>>>>>>>>>>>>>> c) kafka-configs.{sh,
> >> bat}/kafka.admin.ConfigCommand
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 3) I believe Kafka.admin.ConfigCommand presents a
> >>>>> conundrum
> >>>>>>>>>>> because it
> >>>>>>>>>>>>>>> explicitly states in a comment that a supported
> >> use
> >>>>> case is
> >>>>>>>>>>>>> bootstrapping a
> >>>>>>>>>>>>>>> Kafka cluster with encrypted passwords in
> >> Zookeeper
> >>>>> (see
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> >>>>>>>>>>>>> ).
> >>>>>>>>>>>>>>> This means it will be especially difficult to
> >>>>> deprecate this
> >>>>>>>>>>> particular
> >>>>>>>>>>>>>>> direct Zookeeper connectivity without a different
> >>>>> storage
> >>>>>>>>>>> mechanism for
> >>>>>>>>>>>>>>> dynamic configuration values being available
> >> (i.e. the
> >>>>> self-managed
> >>>>>>>>>>>>> quorum
> >>>>>>>>>>>>>>> referred to in KIP-500).
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I think it would be easier and simpler to harden
> >> the
> >>>>> Zookeeper TLS
> >>>>>>>>>>>>>>> configuration in both Kafka and in CLI tools
> >> compared
> >>>>> to trying to
> >>>>>>>>>>>>>>> deprecate the direct Zookeeper connectivity in
> >> the
> >>>>> above 3 CLI
> >>>>>>>>>>> tools --
> >>>>>>>>>>>>>>> especially when ConfigCommand has no obvious
> >>>>> short-term path to
> >>>>>>>>>>>>> deprecation.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Regarding how to harden the TLS configuration,
> >> adding
> >>>>> the
> >>>>>>>>>>> appropriate
> >>>>>>>>>>>>>>> configs to Kafka is an obvious choice for the
> >>>>> brokers. Not that
> >>>>>>>>>>> these
> >>>>>>>>>>>>>>> values cannot be dynamically reconfigurable
> >> because
> >>>>> the values
> >>>>>>>>>>> would be
> >>>>>>>>>>>>>>> required to connect to Zookeeper via TLS in the
> >> first
> >>>>> place.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Regarding how to harden the TLS configuration
> >> for the
> >>>>> 3 remaining
> >>>>>>>>>>> CLI
> >>>>>>>>>>>>>>> tools, it would be relatively straightforward to
> >> add
> >>>>> support for a
> >>>>>>>>>>>>>>> "--zk-tls-config-file <String: Zookeeper TLS
> >>>>> configuration system
> >>>>>>>>>>>>>>> properties file path>" option to each one.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thoughts?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Ron
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes
> >> <
> >>>>>>>>>>> pere.urbon@gmail.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> True, the number is duplicated.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> do you know how can we get the problem fix? I
> >> was
> >>>>> not aware, I
> >>>>>>>>>>>>> missed,
> >>>>>>>>>>>>>>>> sorry the need to add the KIP to the table.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -- Pere
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Missatge de Jorge Esteban Quilcate Otoya <
> >>>>>>>>>>> quilcate.jorge@gmail.com>
> >>>>>>>>>>>>> del
> >>>>>>>>>>>>>>>> dia
> >>>>>>>>>>>>>>>> dt., 3 de set. 2019 a les 13:43:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Pere,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Have you add your KIP to the list here
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >>>>>>>>>>>>>>>> ?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I found the KIP number assigned to another.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón
> >> Bayes <
> >>>>>>>>>>>>> pere.urbon@gmail.com>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for your time Harsha,
> >>>>>>>>>>>>>>>>>> anyone else with comments? looking
> >> forward to
> >>>>> hearing from
> >>>>>>>>>>> you.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Stupid question: when do you move from
> >> discussion
> >>>>> to vote?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Missatge de Harsha Chintalapani <
> >> kafka@harsha.io>
> >>>>> del dia
> >>>>>>>>>>> dv., 30
> >>>>>>>>>>>>>>>> d’ag.
> >>>>>>>>>>>>>>>>>> 2019 a les 21:59:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks Pere. KIP looks good to me.
> >>>>>>>>>>>>>>>>>>> -Harsha
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Fri, Aug 30, 2019 at 10:05 AM, Pere
> >> Urbón
> >>>>> Bayes <
> >>>>>>>>>>>>>>>>>> pere.urbon@gmail.com>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Not really,
> >>>>>>>>>>>>>>>>>>>> my idea is to keep the JAAS parameter,
> >> so
> >>>>> people don't see
> >>>>>>>>>>>>> major
> >>>>>>>>>>>>>>>>>>>> changes. But if you pass a properties
> >> file,
> >>>>> then this takes
> >>>>>>>>>>>>>>>> precedence
> >>>>>>>>>>>>>>>>>> over
> >>>>>>>>>>>>>>>>>>>> the other, with the idea that you can do
> >> sasl
> >>>>> as well with
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> properties
> >>>>>>>>>>>>>>>>>>>> files.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Makes sense?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> -- Pere
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Missatge de Harsha Chintalapani <
> >>>>> kafka@harsha.io> del dia
> >>>>>>>>>>> dv.,
> >>>>>>>>>>>>> 30
> >>>>>>>>>>>>>>>>>> d’ag.
> >>>>>>>>>>>>>>>>>>>> 2019 a les 19:00:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi Pere,
> >>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. Enabling
> >> SSL for
> >>>>> zookeeper
> >>>>>>>>>>> for
> >>>>>>>>>>>>> Kafka
> >>>>>>>>>>>>>>>>>> makes
> >>>>>>>>>>>>>>>>>>>>> sense.
> >>>>>>>>>>>>>>>>>>>>> "The changes are planned to be
> >> introduced in a
> >>>>> compatible
> >>>>>>>>>>> way,
> >>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>> keeping the current JAAS variable
> >> precedence."
> >>>>>>>>>>>>>>>>>>>>> Can you elaborate a bit here. If the user
> >>>>> configures a JAAS
> >>>>>>>>>>>>> file
> >>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>> Client section it will take precedence
> >> over
> >>>>> zookeeper SSL
> >>>>>>>>>>>>> configs?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>> Harsha
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Fri, Aug 30, 2019 at 7:50 AM, Pere
> >> Urbón
> >>>>> Bayes <
> >>>>>>>>>>>>>>>>>> pere.urbon@gmail.com>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>>> quick question, I saw in another mail
> >> that
> >>>>> 2.4 release is
> >>>>>>>>>>>>> planned
> >>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>> September. I think it would be really
> >> awesome
> >>>>> to have
> >>>>>>>>>>> this for
> >>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>> release, do you think we can make it?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> -- Pere
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Missatge de Pere Urbón Bayes <
> >>>>> pere.urbon@gmail.com> del
> >>>>>>>>>>> dia
> >>>>>>>>>>>>> dj.,
> >>>>>>>>>>>>>>>> 29
> >>>>>>>>>>>>>>>>>>>>>> d’ag. 2019 a les 20:10:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>>> this is my first KIP for a change in
> >> Apache
> >>>>> Kafka, so I'm
> >>>>>>>>>>>>> really
> >>>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>>>>>> to the process. Looking forward to
> >> hearing
> >>>>> from you and
> >>>>>>>>>>> learn
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> best
> >>>>>>>>>>>>>>>>>>>>>> ropes here.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I would like to propose this KIP-515 to
> >>>>> enable the
> >>>>>>>>>>>>>>>> ZookeeperClients
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> take full advantage of the TLS
> >> communication
> >>>>> in the new
> >>>>>>>>>>>>> Zookeeper
> >>>>>>>>>>>>>>>>>> 3.5.5.
> >>>>>>>>>>>>>>>>>>>>>> Specially interesting it the Zookeeper
> >>>>> Security Migration,
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> without
> >>>>>>>>>>>>>>>>>>>>>> this change will not work with TLS,
> >> disabling
> >>>>> users to use
> >>>>>>>>>>>>> ACLs
> >>>>>>>>>>>>>>>> when
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> Zookeeper cluster use TLS.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> link:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Looking forward to hearing from you on
> >> this,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> /cheers
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
> >>>>>>>>>>>>>>>>>>>>>> Software Architect
> >>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
> >>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
> >>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
> >>>>>>>>>>>>>>>>>>>>>> Software Architect
> >>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
> >>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
> >>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
> >>>>>>>>>>>>>>>>>>>> Software Architect
> >>>>>>>>>>>>>>>>>>>> http://www.purbon.com
> >>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
> >>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
> >>>>>>>>>>>>>>>>>> Software Architect
> >>>>>>>>>>>>>>>>>> http://www.purbon.com
> >>>>>>>>>>>>>>>>>> https://twitter.com/purbon
> >>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>> Pere Urbon-Bayes
> >>>>>>>>>>>>>>>> Software Architect
> >>>>>>>>>>>>>>>> http://www.purbon.com
> >>>>>>>>>>>>>>>> https://twitter.com/purbon
> >>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Ron Dagostino <rn...@gmail.com>.
Hi Rajini. I’ll submit a documentation PR to clarify the Kafka behavior of #1 and will adopt the same config for ZK. I agree now we should inherit for AclAuthorizer too — I was just stuck on the “no inheritance” idea more than I realized, and while the ZK quorums are different that doesn’t necessarily mean the same client identity wouldn’t be valid on both (and if not then the configs can still be overridden).
I’ll update the KIP today and let this version soak for a day or two, and then I’ll start a vote if no other issues/comments arise.
Ron
> On Jan 17, 2020, at 6:08 AM, Rajini Sivaram <ra...@gmail.com> wrote:
>
> Hi Ron,
>
> Unresolved item #1: Yes, Kafka disables hostname verification if
> "ssl.endpoint.identification.algorithm" is an empty string.
>
> Unresolved item #2: Yes, those 9 plus hostname verification. If we do
> decide to inherit Kafka configs, wouldn't we inherit these 10 in
> AclAuthorizer as well?
>
> Unresolved item #3: Agree, following up on the discussion thread for
> KIP-553.
>
>
>> On Fri, Jan 17, 2020 at 3:29 AM Ron Dagostino <rn...@gmail.com> wrote:
>>
>> Hi Colin. TLS to ZooKeeper will remain an opt-in. The configuration
>> value that turns it on is zookeeper.ssl.client.enable (which used to
>> be called zookeeper.client.secure before we renamed everything to be
>> consistent with Kafka's config naming style and eliminate confusion).
>> The default value for this is false. Everything else related to TLS
>> has no effect unless this is explicitly set to true, and this value
>> does not inherit from anywhere,
>>
>> Ron
>>
>>> On Thu, Jan 16, 2020 at 5:39 PM Colin McCabe <cm...@apache.org> wrote:
>>>
>>>> On Thu, Jan 16, 2020, at 02:58, Rajini Sivaram wrote:
>>>> Hi Ron/Colin,
>>>>
>>>> Thanks for the responses. I think we need to consider these cases:
>>>>
>>>> 1) The recommended approach: This is the one we document. Configs
>>>> consistent with Kafka make sense here. It seems a lot more confusing to
>>>> configure `ssl.cipher.suites`, `ssl.enabled.protocols`, `
>>>> zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols` with
>>>> multiple naming inconsistencies in server.properties. For users who
>> have
>>>> never used ZK system properties, this is clearly simpler. Another
>> config to
>>>> consider is `ssl.endpoint.identification.algorithm` which does the same
>>>> thing as `zookeeper.ssl.hostnameVerification` - if we are staying
>>>> consistent, we should use the Kafka format for this too. I would also
>>>> consider using something like `zookeeper.client.ssl.enabled` instead
>> of `
>>>> zookeeper.client.secure` since it is difficult to tell what `secure`
>> means
>>>> when you have options to configure Kerberos, ACLs and/or SSL.
>>>>
>>>> 2) Inheritance using prefixed configs as we do for other prefixed
>> configs
>>>> in Kafka: I think it may be useful for configs like `ssl.protocol` and
>> `
>>>> ssl.cipher.suites`.
>>>>
>>>> 3) Default values: Do we want to use Kafka defaults? The inconsistency
>> we
>>>> have is `ssl.protocol/ssl.enabled.protocols` since we still have
>> insecure
>>>> protocols enabled for Kafka. We have KIP-553 (
>>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
>> )
>>>> open to disable these. It is currently blocked since it also talks
>> about
>>>> enabling TLSv1.3 by default and we haven't done sufficient testing for
>>>> that. Since it will be good to disable the older insecure protocols
>> anyway,
>>>> perhaps we could do that without enabling TLSv1.3 by default in 2.5?
>>>
>>> Hi Rajini,
>>>
>>> Inheriting the SSL settings from Kafka would have the effect of changing
>> the defaults for almost everyone using SSL from non-SSL ZK to SSL-based ZK,
>> right? This seems to expand the scope of the KIP greatly, since it moves
>> it from being opt-in to opt-out. Are there any potential negative
>> implications of this?
>>>
>>> best,
>>> Colin
>>>
>>>
>>>>
>>>> 4) Backward compatibility for deployments which are using system
>>>> properties. We shouldn't override system properties with Kafka
>> defaults or
>>>> inherited values. But we do want to override if user configures
>> properties
>>>> explicitly. In the current approach, this was simpler since we just
>> had to
>>>> set the configured values. But if we decide to inherit Kafka configs,
>> then
>>>> we will need to check system properties and update accordingly. But
>> that
>>>> should be doable?
>>>>
>>>> On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe <cm...@apache.org>
>> wrote:
>>>>
>>>>>> On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote:
>>>>>> Hi Colin. Two things come to mind with respect to ZooKeeper
>> camelCase
>>>>>> style vs Kafka-style config names for ZooKeeper. First, I think it
>>>>>> would be desirable for the client configs and broker configs to be
>>>>>> interoperable. For example, it feels like it would be convenient
>> to
>>>>>> be able to pass the broker's config file to the ZK Security
>> Migrator
>>>>>> tool. I think whatever we use (ZooKeeper camelCase style or Kafka
>>>>>> style), I think we should use the same for both.
>>>>>>
>>>>>> The second thing to think about is the fact that ZooKeeper
>>>>>> configuration inherently uses Java system properties as a first
>> pass.
>>>>>> So while we might switch to Kafka-style configs in the config file
>>>>>> (e.g. zookeeper.ssl.trust.store.location), the
>>>>>> org.apache.zookeeper.client.ZKClientConfig class (which is how TLS
>>>>>> configs are set) is going to look for the camelCase
>>>>>> zookeeper.ssl.trustStore.location system property and use any value
>>>>>> there. I think this could be very confusing for people. Granted,
>> we
>>>>>> are doing this so that people don't have to use system properties,
>> but
>>>>>> there is no way to turn off the use of system properties, so I
>> think
>>>>>> there would be pretty substantial potential for confusion.
>>>>>
>>>>> I don't think we document or recommend that anyone use system
>> properties
>>>>> to configure Zookeeper usage within Kafka. And I think the reason
>> is
>>>>> exactly the issue you pointed out -- it wouldn't be consistent with
>> our
>>>>> other configurations.
>>>>>
>>>>> best,
>>>>> Colin
>>>>>
>>>>>>
>>>>>> One idea is to migrate the system properties -- i.e. look to see if
>>>>>> zookeeper.ssl.trustStore.location is set and if it is clear out the
>>>>>> value there and move it under the
>> zookeeper.ssl.trust.store.location
>>>>>> system property. (I said it was an idea -- not necessarily a good
>> one
>>>>>> :-)
>>>>>>
>>>>>> Ron
>>>>>>
>>>>>> On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe <cm...@apache.org>
>> wrote:
>>>>>>>
>>>>>>>> On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
>>>>>>>> Thanks Colin and Rajini.
>>>>>>>>
>>>>>>>> I've updated the KIP to say that KIP-421 functionality is
>> available
>>>>> to
>>>>>>>> encrypt sensitive configs like the ZK key store and trust store
>>>>>>>> passwords. (I've also made it clear that the configs are not
>>>>>>>> dynamically reconfigurable since dynamic values are stored in
>> ZK and
>>>>>>>> the data is needed to get to ZK in the first place). Colin,
>> let me
>>>>>>>> know if you think anything else needs to be said/done about it
>> -- I
>>>>>>>> wasn't sure if your comment above implies that there are
>> additional
>>>>>>>> direct actions that we need to take aside from this.
>>>>>>>>
>>>>>>>> I agree that making the brokers' ZooKeeper clients dynamic with
>>>>>>>> respect to key and trust stores (e.g. via
>>>>>>>> zookeeper.ssl.context.supplier.class) may increase the risk
>> that this
>>>>>>>> KIP may not land in time for 2.5.
>>>>>>>>
>>>>>>>> Regarding the config names for ZK being different than the
>> ZooKeeper
>>>>>>>> configs (e.g. camel-case keyStore/trustStore instead of
>>>>>>>> keystore/truststore, ciphersuites instead of cipher.suites,
>>>>>>>> enabledProtocols instead of enabled.protocols), I agree it is
>> an
>>>>>>>> issue. I tried to mitigate it by putting warning text in the
>> config
>>>>>>>> docs for these. Regarding configuration inheritance, I think
>> what
>>>>> you
>>>>>>>> are saying is that for any configs where the broker has a clear
>>>>>>>> semantic equivalent, the broker could fall back to the broker
>> value
>>>>> if
>>>>>>>> the ZK value is not given. The potential list is:
>>>>>>>>
>>>>>>>> zookeeper.ssl.keyStore.location => ssl.keystore.location
>>>>>>>> zookeeper.ssl.keyStore.password => ssl.keystore.password
>>>>>>>> zookeeper.ssl.keyStore.type => ssl.keystore.type
>>>>>>>> zookeeper.ssl.trustStore.location => ssl.truststore.location
>>>>>>>> zookeeper.ssl.trustStore.password => ssl.truststore.password
>>>>>>>> zookeeper.ssl.trustStore.type => ssl.truststore.type
>>>>>>>> zookeeper.ssl.ciphersuites => ssl.cipher.suites
>>>>>>>>
>>>>>>>> Note that there are two configs that look the same based on
>> their key
>>>>>>>> names but whose allowable values differ:
>>>>>>>>
>>>>>>>> zookeeper.ssl.protocol(Default: TLSv1.2) =>
>> ssl.protocol(Default:
>>>>> TLS)
>>>>>>>> zookeeper.ssl.enabledProtocols(Default: value of protocol
>> property) =
>>>>>>>> ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
>>>>>>>>
>>>>>>>> Not sure what the right answer is, but hopefully this detail
>> will
>>>>> help
>>>>>>>> get us there.
>>>>>>>>
>>>>>>>> Ron
>>>>>>>
>>>>>>> I think, on balance, I agree with Rajini that it would be nice
>> to make
>>>>> the configs look more like Kafka configs. We don't have camel-case
>> in
>>>>> configuration keys, so we should avoid it here.
>>>>>>>
>>>>>>> best,
>>>>>>> Colin
>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <
>> cmccabe@apache.org>
>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
>>>>>>>>>> Hi Ron,
>>>>>>>>>>
>>>>>>>>>> Thanks for the detailed explanation, sounds good to me.
>>>>>>>>>>
>>>>>>>>>> A few more questions:
>>>>>>>>>>
>>>>>>>>>> 1) At the moment, all sensitive broker configs including
>>>>>>>>>> keystore/truststore passwords can be stored encrypted in
>>>>> ZooKeeper prior to
>>>>>>>>>> starting up brokers. We are now adding SSL
>> keystore/truststore
>>>>> passwords
>>>>>>>>>> for ZooKeeper client that cannot be stored in ZooKeeper
>> since
>>>>> you need
>>>>>>>>>> these to connect to ZK. We should perhaps document that
>> these
>>>>> configs can
>>>>>>>>>> be encrypted using KIP-421.
>>>>>>>>>
>>>>>>>>> That's a good point. Previously, we didn't have ZK
>> on-the-wire
>>>>> security support. However, now that we do, sending sensitive
>> keystore and
>>>>> truststore passwords to ZK in cleartext should be deprecated in
>> favor of
>>>>> using KIP-421.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2) We can dynamically update keystores and trust stores
>> used by
>>>>> brokers
>>>>>>>>>> without restarting the broker. Can we support this easily
>> for ZK
>>>>> clients
>>>>>>>>>> created by the broker, for example by adding our own
>>>>>>>>>> `zookeeper.ssl.context.supplier.class`?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hmm. That might be better to consider in a follow-up, since
>> the
>>>>> deadline for 2.5 KIPs is coming up?
>>>>>>>>>
>>>>>>>>> best,
>>>>>>>>> Colin
>>>>>>>>>
>>>>>>>>>> 3) Looks like we are using config names that directly map
>> to ZK
>>>>> configs.
>>>>>>>>>> Have we considered using equivalent Kafka config names with
>>>>> prefixes,
>>>>>>>>>> perhaps with inheritance from the non-prefixed names? Not
>> sure
>>>>> if this is a
>>>>>>>>>> good idea, but perhaps worth documenting in Rejected
>>>>> Alternatives at least.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <
>> cmccabe@apache.org>
>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Ron,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the explanation. I guess thinking about it a
>>>>> little bit more,
>>>>>>>>>>> we should just add --zk-tls-config-file to all of these
>>>>> commands.
>>>>>>>>>>>
>>>>>>>>>>> We will be removing this option (plus ZK in general) from
>>>>> these commands
>>>>>>>>>>> in the next major release, but ZK is still supported in
>> 2.5,
>>>>> so we should
>>>>>>>>>>> just do the logical thing. (The exception is
>>>>> ZkSecurityMigrator, which
>>>>>>>>>>> will stay).
>>>>>>>>>>>
>>>>>>>>>>> best,
>>>>>>>>>>> Colin
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
>>>>>>>>>>>> Hi Colin.
>>>>>>>>>>>>
>>>>>>>>>>>> <<< It seems like this [--zk-tls-config-file
>> information]
>>>>> could just
>>>>>>>>>>> appear
>>>>>>>>>>>> in a configuration file, which all of these tools
>> already
>>>>> accept (I
>>>>>>>>>>> think)
>>>>>>>>>>>>
>>>>>>>>>>>> ZkSecurityMigrator has no such property file facility;
>>>>> adding a
>>>>>>>>>>>> "--zk-tls-config-file" parameter is exactly for this
>>>>> purpose. If we add
>>>>>>>>>>>> that to ZkSecurityMigrator then it is trivial to add
>> it to
>>>>> other commands
>>>>>>>>>>>> (the same code is simply reused; it ends up being just
>> a few
>>>>> extra
>>>>>>>>>>> lines).
>>>>>>>>>>>> I do not see any parameter in the other two commands to
>>>>> adjust the ZK
>>>>>>>>>>>> connection; ConfigCommand accepts a "--command-config"
>> flag,
>>>>> but
>>>>>>>>>>> according
>>>>>>>>>>>> to the code "This is used only with --bootstrap-server
>>>>> option for
>>>>>>>>>>>> describing and altering broker configs."
>>>>>>>>>>>>
>>>>>>>>>>>> I do agree there would be no need to add
>>>>> "--zk-tls-config-file" to
>>>>>>>>>>>> ReassignPartitionsCommand if its direct ZK
>> connectivity is
>>>>> replaced in
>>>>>>>>>>> time
>>>>>>>>>>>> for the next release.
>>>>>>>>>>>>
>>>>>>>>>>>> ConfigCommand supports the "--bootstrap-server" option
>> and
>>>>> will have its
>>>>>>>>>>>> direct ZooKeeper access formally deprecated as per
>> KIP-555,
>>>>> but the
>>>>>>>>>>> special
>>>>>>>>>>>> use case of bootstrapping a ZooKeeper ensemble with
>>>>> encrypted credentials
>>>>>>>>>>>> prior to starting Kafka will still exist, so it feels
>> like
>>>>> while
>>>>>>>>>>>> "--zk-tls-config-file" would never be used except for
>> this
>>>>> use case it
>>>>>>>>>>>> could probably still be added for this particular
>> situation.
>>>>>>>>>>>>
>>>>>>>>>>>> Ron
>>>>>>>>>>>>
>>>>>>>>>>>> P.S. I responded on 1/6 but I just discovered that e,
>> ail
>>>>> (and 3 more I
>>>>>>>>>>>> sent) did not go through. I am trying to get emails
>> through
>>>>> now to move
>>>>>>>>>>>> this discussion forward.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <
>>>>> cmccabe@apache.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
>>>>>>>>>>>>>> Hi everyone. I would like to make the following
>> changes
>>>>> to the KIP.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> MOTIVATION:
>>>>>>>>>>>>>> Include a statement that it will be difficult in
>> the
>>>>> short term to
>>>>>>>>>>>>>> deprecate direct Zookeeper communication in
>>>>> kafka-configs.{sh, bat}
>>>>>>>>>>>>> (which
>>>>>>>>>>>>>> invoke kafka.admin.ConfigCommand) because
>> bootstrapping
>>>>> a Kafka
>>>>>>>>>>> cluster
>>>>>>>>>>>>>> with encrypted passwords in Zookeeper is an
>>>>> explicitly-supported use
>>>>>>>>>>>>> case;
>>>>>>>>>>>>>> therefore it is in scope to be able to securely
>>>>> configure the CLI
>>>>>>>>>>> tools
>>>>>>>>>>>>>> that still leverage non-deprecated direct Zookeeper
>>>>> communication
>>>>>>>>>>> for TLS
>>>>>>>>>>>>>> (the other 2 tools are
>> kafka-reassign-partitions.{sh,
>>>>> bat} and
>>>>>>>>>>>>>> zookeeper-security-migration.sh).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Ron,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>
>>>>>>>>>>>>> About deprecations:
>>>>>>>>>>>>>
>>>>>>>>>>>>> * zookeeper-security-migration: clearly, deprecating
>> ZK
>>>>> access in this
>>>>>>>>>>> one
>>>>>>>>>>>>> would not make sense, since it would defeat the whole
>>>>> point of the
>>>>>>>>>>> tool :)
>>>>>>>>>>>>>
>>>>>>>>>>>>> * kafka-reassign-partitions: ZK access should be
>>>>> deprecated here.
>>>>>>>>>>> KIP-455
>>>>>>>>>>>>> implementation has dragged a bit, but this should
>> get done
>>>>> soon.
>>>>>>>>>>> Certainly
>>>>>>>>>>>>> before the next release.
>>>>>>>>>>>>>
>>>>>>>>>>>>> * kafka-configs: I think ZK access should be
>> deprecated
>>>>> here as well.
>>>>>>>>>>> I
>>>>>>>>>>>>> agree there is a super-special bootstrapping case
>> here,
>>>>> but that should
>>>>>>>>>>>>> have its own tool, not use kafka-configs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I will post a separate KIP for this, though.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> GOALS:
>>>>>>>>>>>>>> Support the secure configuration of TLS-encrypted
>>>>> communication
>>>>>>>>>>> between
>>>>>>>>>>>>>> Zookeeper and:
>>>>>>>>>>>>>> a) Kafka brokers
>>>>>>>>>>>>>> b) The three CLI tools mentioned above that still
>>>>> support direct,
>>>>>>>>>>>>>> non-deprecated communication to Zookeeper
>>>>>>>>>>>>>> It is explicitly out-of-scope to deprecate any
>> direct
>>>>> Zookeeper
>>>>>>>>>>>>>> communication in CLI tools as part of this KIP;
>> such
>>>>> work will occur
>>>>>>>>>>> in
>>>>>>>>>>>>>> future KIPs instead.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> PUBLIC INTERFACES:
>>>>>>>>>>>>>> 1) The following new broker configurations will be
>>>>> recognized.
>>>>>>>>>>>>>> zookeeper.client.secure (default value = false,
>> for
>>>>> backwards
>>>>>>>>>>>>>> compatibility)
>>>>>>>>>>>>>> zookeeper.clientCnxnSocket
>>>>>>>>>>>>>> zookeeper.ssl.keyStore.location
>>>>>>>>>>>>>> zookeeper.ssl.keyStore.password
>>>>>>>>>>>>>> zookeeper.ssl.trustStore.location
>>>>>>>>>>>>>> zookeeper.ssl.trustStore.password
>>>>>>>>>>>>>> It will be an error for any of the last 5 values
>> to be
>>>>> left
>>>>>>>>>>> unspecified
>>>>>>>>>>>>> if
>>>>>>>>>>>>>> zookeeper.client.secure is explicitly set to true.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2) In addition, the
>>>>> kafka.security.authorizer.AclAuthorizer class
>>>>>>>>>>>>> supports
>>>>>>>>>>>>>> the ability to connect to a different Zookeeper
>> instance
>>>>> than the
>>>>>>>>>>> one the
>>>>>>>>>>>>>> brokers use. We therefore also add the following
>>>>> optional configs,
>>>>>>>>>>> which
>>>>>>>>>>>>>> override the corresponding ones from above when
>> present:
>>>>>>>>>>>>>> authorizer.zookeeper.client.secure
>>>>>>>>>>>>>> authorizer.zookeeper.clientCnxnSocket
>>>>>>>>>>>>>> authorizer.zookeeper.ssl.keyStore.location
>>>>>>>>>>>>>> authorizer.zookeeper.ssl.keyStore.password
>>>>>>>>>>>>>> authorizer.zookeeper.ssl.trustStore.location
>>>>>>>>>>>>>> authorizer.zookeeper.ssl.trustStore.password
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 3) The three CLI tools mentioned above will
>> support a new
>>>>>>>>>>>>> --zk-tls-config-file
>>>>>>>>>>>>>> <String: Zookeeper TLS configuration file path>"
>>>>> option. The
>>>>>>>>>>> following
>>>>>>>>>>>>>> properties will be recognized in that file, and
>>>>> unrecognized
>>>>>>>>>>> properties
>>>>>>>>>>>>>> will be ignored to allow the possibility of
>> pointing
>>>>>>>>>>> zk-tls-config-file
>>>>>>>>>>>>> at
>>>>>>>>>>>>>> the broker's config file.
>>>>>>>>>>>>>> zookeeper.client.secure (default value = false)
>>>>>>>>>>>>>> zookeeper.clientCnxnSocket
>>>>>>>>>>>>>> zookeeper.ssl.keyStore.location
>>>>>>>>>>>>>> zookeeper.ssl.keyStore.password
>>>>>>>>>>>>>> zookeeper.ssl.trustStore.location
>>>>>>>>>>>>>> zookeeper.ssl.trustStore.password
>>>>>>>>>>>>>> It will be an error for any of the last 5 values
>> to be
>>>>> left
>>>>>>>>>>> unspecified
>>>>>>>>>>>>> if
>>>>>>>>>>>>>> zookeeper.client.secure is explicitly set to true.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Do we really need a --zk-tls-config-file flag? It
>> seems
>>>>> like this
>>>>>>>>>>> could
>>>>>>>>>>>>> just appear in a configuration file, which all of
>> these
>>>>> tools already
>>>>>>>>>>>>> accept (I think).
>>>>>>>>>>>>>
>>>>>>>>>>>>> best,
>>>>>>>>>>>>> Colin
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ron
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <
>>>>> rndgstn@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi everyone. Let's get this discussion going
>> again
>>>>> now that Kafka
>>>>>>>>>>> 2.4
>>>>>>>>>>>>>>> with Zookeeper 3.5.6 is out.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> First, regarding the KIP number, the other KIP
>> that
>>>>> was using this
>>>>>>>>>>>>> number
>>>>>>>>>>>>>>> moved to KIP 534, so KIP 515 remains the correct
>>>>> number for this
>>>>>>>>>>>>>>> discussion. I've updated the Kafka Improvement
>>>>> Proposal page to
>>>>>>>>>>> list
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> KIP in the 515 slot, so we're all set there.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regarding the actual issues under discussion, I
>> think
>>>>> there are
>>>>>>>>>>> some
>>>>>>>>>>>>>>> things we should clarify.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1) It is possible to use TLS connectivity to
>> Zookeeper
>>>>> from Apache
>>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>> 2.4 -- the problem is that configuration
>> information
>>>>> has to be
>>>>>>>>>>> passed
>>>>>>>>>>>>> via
>>>>>>>>>>>>>>> system properties as "-D" command line options
>> on the
>>>>> java
>>>>>>>>>>> invocation
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> the broker, and those are not secure (anyone with
>>>>> access to the box
>>>>>>>>>>>>> can see
>>>>>>>>>>>>>>> the command line used to invoke the broker); the
>>>>> configuration
>>>>>>>>>>> includes
>>>>>>>>>>>>>>> sensitive information (e.g. a keystore
>> password), so
>>>>> we need a
>>>>>>>>>>> secure
>>>>>>>>>>>>>>> mechanism for passing the configuration values.
>> I
>>>>> believe the real
>>>>>>>>>>>>>>> motivation for this KIP is to harden the
>> configuration
>>>>> mechanism
>>>>>>>>>>> for
>>>>>>>>>>>>>>> Zookeeper TLS connectivity.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2) I believe the list of CLI tools that continue
>> to
>>>>> use direct
>>>>>>>>>>>>> Zookeeper
>>>>>>>>>>>>>>> connectivity in a non-deprecated fashion is:
>>>>>>>>>>>>>>> a)
>>>>>>>>>>>
>> zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
>>>>>>>>>>>>>>> b)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>
>> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
>>>>>>>>>>>>>>> c) kafka-configs.{sh,
>> bat}/kafka.admin.ConfigCommand
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 3) I believe Kafka.admin.ConfigCommand presents a
>>>>> conundrum
>>>>>>>>>>> because it
>>>>>>>>>>>>>>> explicitly states in a comment that a supported
>> use
>>>>> case is
>>>>>>>>>>>>> bootstrapping a
>>>>>>>>>>>>>>> Kafka cluster with encrypted passwords in
>> Zookeeper
>>>>> (see
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>
>> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>> This means it will be especially difficult to
>>>>> deprecate this
>>>>>>>>>>> particular
>>>>>>>>>>>>>>> direct Zookeeper connectivity without a different
>>>>> storage
>>>>>>>>>>> mechanism for
>>>>>>>>>>>>>>> dynamic configuration values being available
>> (i.e. the
>>>>> self-managed
>>>>>>>>>>>>> quorum
>>>>>>>>>>>>>>> referred to in KIP-500).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think it would be easier and simpler to harden
>> the
>>>>> Zookeeper TLS
>>>>>>>>>>>>>>> configuration in both Kafka and in CLI tools
>> compared
>>>>> to trying to
>>>>>>>>>>>>>>> deprecate the direct Zookeeper connectivity in
>> the
>>>>> above 3 CLI
>>>>>>>>>>> tools --
>>>>>>>>>>>>>>> especially when ConfigCommand has no obvious
>>>>> short-term path to
>>>>>>>>>>>>> deprecation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regarding how to harden the TLS configuration,
>> adding
>>>>> the
>>>>>>>>>>> appropriate
>>>>>>>>>>>>>>> configs to Kafka is an obvious choice for the
>>>>> brokers. Not that
>>>>>>>>>>> these
>>>>>>>>>>>>>>> values cannot be dynamically reconfigurable
>> because
>>>>> the values
>>>>>>>>>>> would be
>>>>>>>>>>>>>>> required to connect to Zookeeper via TLS in the
>> first
>>>>> place.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regarding how to harden the TLS configuration
>> for the
>>>>> 3 remaining
>>>>>>>>>>> CLI
>>>>>>>>>>>>>>> tools, it would be relatively straightforward to
>> add
>>>>> support for a
>>>>>>>>>>>>>>> "--zk-tls-config-file <String: Zookeeper TLS
>>>>> configuration system
>>>>>>>>>>>>>>> properties file path>" option to each one.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ron
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes
>> <
>>>>>>>>>>> pere.urbon@gmail.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> True, the number is duplicated.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> do you know how can we get the problem fix? I
>> was
>>>>> not aware, I
>>>>>>>>>>>>> missed,
>>>>>>>>>>>>>>>> sorry the need to add the KIP to the table.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -- Pere
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Missatge de Jorge Esteban Quilcate Otoya <
>>>>>>>>>>> quilcate.jorge@gmail.com>
>>>>>>>>>>>>> del
>>>>>>>>>>>>>>>> dia
>>>>>>>>>>>>>>>> dt., 3 de set. 2019 a les 13:43:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Pere,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Have you add your KIP to the list here
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I found the KIP number assigned to another.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón
>> Bayes <
>>>>>>>>>>>>> pere.urbon@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for your time Harsha,
>>>>>>>>>>>>>>>>>> anyone else with comments? looking
>> forward to
>>>>> hearing from
>>>>>>>>>>> you.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Stupid question: when do you move from
>> discussion
>>>>> to vote?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Missatge de Harsha Chintalapani <
>> kafka@harsha.io>
>>>>> del dia
>>>>>>>>>>> dv., 30
>>>>>>>>>>>>>>>> d’ag.
>>>>>>>>>>>>>>>>>> 2019 a les 21:59:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks Pere. KIP looks good to me.
>>>>>>>>>>>>>>>>>>> -Harsha
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Aug 30, 2019 at 10:05 AM, Pere
>> Urbón
>>>>> Bayes <
>>>>>>>>>>>>>>>>>> pere.urbon@gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Not really,
>>>>>>>>>>>>>>>>>>>> my idea is to keep the JAAS parameter,
>> so
>>>>> people don't see
>>>>>>>>>>>>> major
>>>>>>>>>>>>>>>>>>>> changes. But if you pass a properties
>> file,
>>>>> then this takes
>>>>>>>>>>>>>>>> precedence
>>>>>>>>>>>>>>>>>> over
>>>>>>>>>>>>>>>>>>>> the other, with the idea that you can do
>> sasl
>>>>> as well with
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> properties
>>>>>>>>>>>>>>>>>>>> files.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Makes sense?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -- Pere
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Missatge de Harsha Chintalapani <
>>>>> kafka@harsha.io> del dia
>>>>>>>>>>> dv.,
>>>>>>>>>>>>> 30
>>>>>>>>>>>>>>>>>> d’ag.
>>>>>>>>>>>>>>>>>>>> 2019 a les 19:00:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Pere,
>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. Enabling
>> SSL for
>>>>> zookeeper
>>>>>>>>>>> for
>>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>>>>> makes
>>>>>>>>>>>>>>>>>>>>> sense.
>>>>>>>>>>>>>>>>>>>>> "The changes are planned to be
>> introduced in a
>>>>> compatible
>>>>>>>>>>> way,
>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>> keeping the current JAAS variable
>> precedence."
>>>>>>>>>>>>>>>>>>>>> Can you elaborate a bit here. If the user
>>>>> configures a JAAS
>>>>>>>>>>>>> file
>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> Client section it will take precedence
>> over
>>>>> zookeeper SSL
>>>>>>>>>>>>> configs?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> Harsha
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 30, 2019 at 7:50 AM, Pere
>> Urbón
>>>>> Bayes <
>>>>>>>>>>>>>>>>>> pere.urbon@gmail.com>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>> quick question, I saw in another mail
>> that
>>>>> 2.4 release is
>>>>>>>>>>>>> planned
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>> September. I think it would be really
>> awesome
>>>>> to have
>>>>>>>>>>> this for
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>> release, do you think we can make it?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> -- Pere
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Missatge de Pere Urbón Bayes <
>>>>> pere.urbon@gmail.com> del
>>>>>>>>>>> dia
>>>>>>>>>>>>> dj.,
>>>>>>>>>>>>>>>> 29
>>>>>>>>>>>>>>>>>>>>>> d’ag. 2019 a les 20:10:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>> this is my first KIP for a change in
>> Apache
>>>>> Kafka, so I'm
>>>>>>>>>>>>> really
>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>> to the process. Looking forward to
>> hearing
>>>>> from you and
>>>>>>>>>>> learn
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> best
>>>>>>>>>>>>>>>>>>>>>> ropes here.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I would like to propose this KIP-515 to
>>>>> enable the
>>>>>>>>>>>>>>>> ZookeeperClients
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> take full advantage of the TLS
>> communication
>>>>> in the new
>>>>>>>>>>>>> Zookeeper
>>>>>>>>>>>>>>>>>> 3.5.5.
>>>>>>>>>>>>>>>>>>>>>> Specially interesting it the Zookeeper
>>>>> Security Migration,
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>>>>>>>> this change will not work with TLS,
>> disabling
>>>>> users to use
>>>>>>>>>>>>> ACLs
>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> Zookeeper cluster use TLS.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> link:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Looking forward to hearing from you on
>> this,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> /cheers
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
>>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
>>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>>>>>>>> http://www.purbon.com
>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>>>>>> http://www.purbon.com
>>>>>>>>>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Pere Urbon-Bayes
>>>>>>>>>>>>>>>> Software Architect
>>>>>>>>>>>>>>>> http://www.purbon.com
>>>>>>>>>>>>>>>> https://twitter.com/purbon
>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Ron,
Unresolved item #1: Yes, Kafka disables hostname verification if
"ssl.endpoint.identification.algorithm" is an empty string.
Unresolved item #2: Yes, those 9 plus hostname verification. If we do
decide to inherit Kafka configs, wouldn't we inherit these 10 in
AclAuthorizer as well?
Unresolved item #3: Agree, following up on the discussion thread for
KIP-553.
On Fri, Jan 17, 2020 at 3:29 AM Ron Dagostino <rn...@gmail.com> wrote:
> Hi Colin. TLS to ZooKeeper will remain an opt-in. The configuration
> value that turns it on is zookeeper.ssl.client.enable (which used to
> be called zookeeper.client.secure before we renamed everything to be
> consistent with Kafka's config naming style and eliminate confusion).
> The default value for this is false. Everything else related to TLS
> has no effect unless this is explicitly set to true, and this value
> does not inherit from anywhere,
>
> Ron
>
> On Thu, Jan 16, 2020 at 5:39 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > On Thu, Jan 16, 2020, at 02:58, Rajini Sivaram wrote:
> > > Hi Ron/Colin,
> > >
> > > Thanks for the responses. I think we need to consider these cases:
> > >
> > > 1) The recommended approach: This is the one we document. Configs
> > > consistent with Kafka make sense here. It seems a lot more confusing to
> > > configure `ssl.cipher.suites`, `ssl.enabled.protocols`, `
> > > zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols` with
> > > multiple naming inconsistencies in server.properties. For users who
> have
> > > never used ZK system properties, this is clearly simpler. Another
> config to
> > > consider is `ssl.endpoint.identification.algorithm` which does the same
> > > thing as `zookeeper.ssl.hostnameVerification` - if we are staying
> > > consistent, we should use the Kafka format for this too. I would also
> > > consider using something like `zookeeper.client.ssl.enabled` instead
> of `
> > > zookeeper.client.secure` since it is difficult to tell what `secure`
> means
> > > when you have options to configure Kerberos, ACLs and/or SSL.
> > >
> > > 2) Inheritance using prefixed configs as we do for other prefixed
> configs
> > > in Kafka: I think it may be useful for configs like `ssl.protocol` and
> `
> > > ssl.cipher.suites`.
> > >
> > > 3) Default values: Do we want to use Kafka defaults? The inconsistency
> we
> > > have is `ssl.protocol/ssl.enabled.protocols` since we still have
> insecure
> > > protocols enabled for Kafka. We have KIP-553 (
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
> )
> > > open to disable these. It is currently blocked since it also talks
> about
> > > enabling TLSv1.3 by default and we haven't done sufficient testing for
> > > that. Since it will be good to disable the older insecure protocols
> anyway,
> > > perhaps we could do that without enabling TLSv1.3 by default in 2.5?
> >
> > Hi Rajini,
> >
> > Inheriting the SSL settings from Kafka would have the effect of changing
> the defaults for almost everyone using SSL from non-SSL ZK to SSL-based ZK,
> right? This seems to expand the scope of the KIP greatly, since it moves
> it from being opt-in to opt-out. Are there any potential negative
> implications of this?
> >
> > best,
> > Colin
> >
> >
> > >
> > > 4) Backward compatibility for deployments which are using system
> > > properties. We shouldn't override system properties with Kafka
> defaults or
> > > inherited values. But we do want to override if user configures
> properties
> > > explicitly. In the current approach, this was simpler since we just
> had to
> > > set the configured values. But if we decide to inherit Kafka configs,
> then
> > > we will need to check system properties and update accordingly. But
> that
> > > should be doable?
> > >
> > > On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe <cm...@apache.org>
> wrote:
> > >
> > > > On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote:
> > > > > Hi Colin. Two things come to mind with respect to ZooKeeper
> camelCase
> > > > > style vs Kafka-style config names for ZooKeeper. First, I think it
> > > > > would be desirable for the client configs and broker configs to be
> > > > > interoperable. For example, it feels like it would be convenient
> to
> > > > > be able to pass the broker's config file to the ZK Security
> Migrator
> > > > > tool. I think whatever we use (ZooKeeper camelCase style or Kafka
> > > > > style), I think we should use the same for both.
> > > > >
> > > > > The second thing to think about is the fact that ZooKeeper
> > > > > configuration inherently uses Java system properties as a first
> pass.
> > > > > So while we might switch to Kafka-style configs in the config file
> > > > > (e.g. zookeeper.ssl.trust.store.location), the
> > > > > org.apache.zookeeper.client.ZKClientConfig class (which is how TLS
> > > > > configs are set) is going to look for the camelCase
> > > > > zookeeper.ssl.trustStore.location system property and use any value
> > > > > there. I think this could be very confusing for people. Granted,
> we
> > > > > are doing this so that people don't have to use system properties,
> but
> > > > > there is no way to turn off the use of system properties, so I
> think
> > > > > there would be pretty substantial potential for confusion.
> > > >
> > > > I don't think we document or recommend that anyone use system
> properties
> > > > to configure Zookeeper usage within Kafka. And I think the reason
> is
> > > > exactly the issue you pointed out -- it wouldn't be consistent with
> our
> > > > other configurations.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > >
> > > > > One idea is to migrate the system properties -- i.e. look to see if
> > > > > zookeeper.ssl.trustStore.location is set and if it is clear out the
> > > > > value there and move it under the
> zookeeper.ssl.trust.store.location
> > > > > system property. (I said it was an idea -- not necessarily a good
> one
> > > > > :-)
> > > > >
> > > > > Ron
> > > > >
> > > > > On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe <cm...@apache.org>
> wrote:
> > > > > >
> > > > > > On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
> > > > > > > Thanks Colin and Rajini.
> > > > > > >
> > > > > > > I've updated the KIP to say that KIP-421 functionality is
> available
> > > > to
> > > > > > > encrypt sensitive configs like the ZK key store and trust store
> > > > > > > passwords. (I've also made it clear that the configs are not
> > > > > > > dynamically reconfigurable since dynamic values are stored in
> ZK and
> > > > > > > the data is needed to get to ZK in the first place). Colin,
> let me
> > > > > > > know if you think anything else needs to be said/done about it
> -- I
> > > > > > > wasn't sure if your comment above implies that there are
> additional
> > > > > > > direct actions that we need to take aside from this.
> > > > > > >
> > > > > > > I agree that making the brokers' ZooKeeper clients dynamic with
> > > > > > > respect to key and trust stores (e.g. via
> > > > > > > zookeeper.ssl.context.supplier.class) may increase the risk
> that this
> > > > > > > KIP may not land in time for 2.5.
> > > > > > >
> > > > > > > Regarding the config names for ZK being different than the
> ZooKeeper
> > > > > > > configs (e.g. camel-case keyStore/trustStore instead of
> > > > > > > keystore/truststore, ciphersuites instead of cipher.suites,
> > > > > > > enabledProtocols instead of enabled.protocols), I agree it is
> an
> > > > > > > issue. I tried to mitigate it by putting warning text in the
> config
> > > > > > > docs for these. Regarding configuration inheritance, I think
> what
> > > > you
> > > > > > > are saying is that for any configs where the broker has a clear
> > > > > > > semantic equivalent, the broker could fall back to the broker
> value
> > > > if
> > > > > > > the ZK value is not given. The potential list is:
> > > > > > >
> > > > > > > zookeeper.ssl.keyStore.location => ssl.keystore.location
> > > > > > > zookeeper.ssl.keyStore.password => ssl.keystore.password
> > > > > > > zookeeper.ssl.keyStore.type => ssl.keystore.type
> > > > > > > zookeeper.ssl.trustStore.location => ssl.truststore.location
> > > > > > > zookeeper.ssl.trustStore.password => ssl.truststore.password
> > > > > > > zookeeper.ssl.trustStore.type => ssl.truststore.type
> > > > > > > zookeeper.ssl.ciphersuites => ssl.cipher.suites
> > > > > > >
> > > > > > > Note that there are two configs that look the same based on
> their key
> > > > > > > names but whose allowable values differ:
> > > > > > >
> > > > > > > zookeeper.ssl.protocol(Default: TLSv1.2) =>
> ssl.protocol(Default:
> > > > TLS)
> > > > > > > zookeeper.ssl.enabledProtocols(Default: value of protocol
> property) =
> > > > > > > ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
> > > > > > >
> > > > > > > Not sure what the right answer is, but hopefully this detail
> will
> > > > help
> > > > > > > get us there.
> > > > > > >
> > > > > > > Ron
> > > > > >
> > > > > > I think, on balance, I agree with Rajini that it would be nice
> to make
> > > > the configs look more like Kafka configs. We don't have camel-case
> in
> > > > configuration keys, so we should avoid it here.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > >
> > > > > > > On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <
> cmccabe@apache.org>
> > > > wrote:
> > > > > > > >
> > > > > > > > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> > > > > > > > > Hi Ron,
> > > > > > > > >
> > > > > > > > > Thanks for the detailed explanation, sounds good to me.
> > > > > > > > >
> > > > > > > > > A few more questions:
> > > > > > > > >
> > > > > > > > > 1) At the moment, all sensitive broker configs including
> > > > > > > > > keystore/truststore passwords can be stored encrypted in
> > > > ZooKeeper prior to
> > > > > > > > > starting up brokers. We are now adding SSL
> keystore/truststore
> > > > passwords
> > > > > > > > > for ZooKeeper client that cannot be stored in ZooKeeper
> since
> > > > you need
> > > > > > > > > these to connect to ZK. We should perhaps document that
> these
> > > > configs can
> > > > > > > > > be encrypted using KIP-421.
> > > > > > > >
> > > > > > > > That's a good point. Previously, we didn't have ZK
> on-the-wire
> > > > security support. However, now that we do, sending sensitive
> keystore and
> > > > truststore passwords to ZK in cleartext should be deprecated in
> favor of
> > > > using KIP-421.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > 2) We can dynamically update keystores and trust stores
> used by
> > > > brokers
> > > > > > > > > without restarting the broker. Can we support this easily
> for ZK
> > > > clients
> > > > > > > > > created by the broker, for example by adding our own
> > > > > > > > > `zookeeper.ssl.context.supplier.class`?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hmm. That might be better to consider in a follow-up, since
> the
> > > > deadline for 2.5 KIPs is coming up?
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > > > 3) Looks like we are using config names that directly map
> to ZK
> > > > configs.
> > > > > > > > > Have we considered using equivalent Kafka config names with
> > > > prefixes,
> > > > > > > > > perhaps with inheritance from the non-prefixed names? Not
> sure
> > > > if this is a
> > > > > > > > > good idea, but perhaps worth documenting in Rejected
> > > > Alternatives at least.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <
> cmccabe@apache.org>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Ron,
> > > > > > > > > >
> > > > > > > > > > Thanks for the explanation. I guess thinking about it a
> > > > little bit more,
> > > > > > > > > > we should just add --zk-tls-config-file to all of these
> > > > commands.
> > > > > > > > > >
> > > > > > > > > > We will be removing this option (plus ZK in general) from
> > > > these commands
> > > > > > > > > > in the next major release, but ZK is still supported in
> 2.5,
> > > > so we should
> > > > > > > > > > just do the logical thing. (The exception is
> > > > ZkSecurityMigrator, which
> > > > > > > > > > will stay).
> > > > > > > > > >
> > > > > > > > > > best,
> > > > > > > > > > Colin
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > > > > > > > > > Hi Colin.
> > > > > > > > > > >
> > > > > > > > > > > <<< It seems like this [--zk-tls-config-file
> information]
> > > > could just
> > > > > > > > > > appear
> > > > > > > > > > > in a configuration file, which all of these tools
> already
> > > > accept (I
> > > > > > > > > > think)
> > > > > > > > > > >
> > > > > > > > > > > ZkSecurityMigrator has no such property file facility;
> > > > adding a
> > > > > > > > > > > "--zk-tls-config-file" parameter is exactly for this
> > > > purpose. If we add
> > > > > > > > > > > that to ZkSecurityMigrator then it is trivial to add
> it to
> > > > other commands
> > > > > > > > > > > (the same code is simply reused; it ends up being just
> a few
> > > > extra
> > > > > > > > > > lines).
> > > > > > > > > > > I do not see any parameter in the other two commands to
> > > > adjust the ZK
> > > > > > > > > > > connection; ConfigCommand accepts a "--command-config"
> flag,
> > > > but
> > > > > > > > > > according
> > > > > > > > > > > to the code "This is used only with --bootstrap-server
> > > > option for
> > > > > > > > > > > describing and altering broker configs."
> > > > > > > > > > >
> > > > > > > > > > > I do agree there would be no need to add
> > > > "--zk-tls-config-file" to
> > > > > > > > > > > ReassignPartitionsCommand if its direct ZK
> connectivity is
> > > > replaced in
> > > > > > > > > > time
> > > > > > > > > > > for the next release.
> > > > > > > > > > >
> > > > > > > > > > > ConfigCommand supports the "--bootstrap-server" option
> and
> > > > will have its
> > > > > > > > > > > direct ZooKeeper access formally deprecated as per
> KIP-555,
> > > > but the
> > > > > > > > > > special
> > > > > > > > > > > use case of bootstrapping a ZooKeeper ensemble with
> > > > encrypted credentials
> > > > > > > > > > > prior to starting Kafka will still exist, so it feels
> like
> > > > while
> > > > > > > > > > > "--zk-tls-config-file" would never be used except for
> this
> > > > use case it
> > > > > > > > > > > could probably still be added for this particular
> situation.
> > > > > > > > > > >
> > > > > > > > > > > Ron
> > > > > > > > > > >
> > > > > > > > > > > P.S. I responded on 1/6 but I just discovered that e,
> ail
> > > > (and 3 more I
> > > > > > > > > > > sent) did not go through. I am trying to get emails
> through
> > > > now to move
> > > > > > > > > > > this discussion forward.
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <
> > > > cmccabe@apache.org> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > > > > > > > > > > Hi everyone. I would like to make the following
> changes
> > > > to the KIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > MOTIVATION:
> > > > > > > > > > > > > Include a statement that it will be difficult in
> the
> > > > short term to
> > > > > > > > > > > > > deprecate direct Zookeeper communication in
> > > > kafka-configs.{sh, bat}
> > > > > > > > > > > > (which
> > > > > > > > > > > > > invoke kafka.admin.ConfigCommand) because
> bootstrapping
> > > > a Kafka
> > > > > > > > > > cluster
> > > > > > > > > > > > > with encrypted passwords in Zookeeper is an
> > > > explicitly-supported use
> > > > > > > > > > > > case;
> > > > > > > > > > > > > therefore it is in scope to be able to securely
> > > > configure the CLI
> > > > > > > > > > tools
> > > > > > > > > > > > > that still leverage non-deprecated direct Zookeeper
> > > > communication
> > > > > > > > > > for TLS
> > > > > > > > > > > > > (the other 2 tools are
> kafka-reassign-partitions.{sh,
> > > > bat} and
> > > > > > > > > > > > > zookeeper-security-migration.sh).
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Ron,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the KIP.
> > > > > > > > > > > >
> > > > > > > > > > > > About deprecations:
> > > > > > > > > > > >
> > > > > > > > > > > > * zookeeper-security-migration: clearly, deprecating
> ZK
> > > > access in this
> > > > > > > > > > one
> > > > > > > > > > > > would not make sense, since it would defeat the whole
> > > > point of the
> > > > > > > > > > tool :)
> > > > > > > > > > > >
> > > > > > > > > > > > * kafka-reassign-partitions: ZK access should be
> > > > deprecated here.
> > > > > > > > > > KIP-455
> > > > > > > > > > > > implementation has dragged a bit, but this should
> get done
> > > > soon.
> > > > > > > > > > Certainly
> > > > > > > > > > > > before the next release.
> > > > > > > > > > > >
> > > > > > > > > > > > * kafka-configs: I think ZK access should be
> deprecated
> > > > here as well.
> > > > > > > > > > I
> > > > > > > > > > > > agree there is a super-special bootstrapping case
> here,
> > > > but that should
> > > > > > > > > > > > have its own tool, not use kafka-configs.
> > > > > > > > > > > >
> > > > > > > > > > > > I will post a separate KIP for this, though.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > GOALS:
> > > > > > > > > > > > > Support the secure configuration of TLS-encrypted
> > > > communication
> > > > > > > > > > between
> > > > > > > > > > > > > Zookeeper and:
> > > > > > > > > > > > > a) Kafka brokers
> > > > > > > > > > > > > b) The three CLI tools mentioned above that still
> > > > support direct,
> > > > > > > > > > > > > non-deprecated communication to Zookeeper
> > > > > > > > > > > > > It is explicitly out-of-scope to deprecate any
> direct
> > > > Zookeeper
> > > > > > > > > > > > > communication in CLI tools as part of this KIP;
> such
> > > > work will occur
> > > > > > > > > > in
> > > > > > > > > > > > > future KIPs instead.
> > > > > > > > > > > > >
> > > > > > > > > > > > > PUBLIC INTERFACES:
> > > > > > > > > > > > > 1) The following new broker configurations will be
> > > > recognized.
> > > > > > > > > > > > > zookeeper.client.secure (default value = false,
> for
> > > > backwards
> > > > > > > > > > > > > compatibility)
> > > > > > > > > > > > > zookeeper.clientCnxnSocket
> > > > > > > > > > > > > zookeeper.ssl.keyStore.location
> > > > > > > > > > > > > zookeeper.ssl.keyStore.password
> > > > > > > > > > > > > zookeeper.ssl.trustStore.location
> > > > > > > > > > > > > zookeeper.ssl.trustStore.password
> > > > > > > > > > > > > It will be an error for any of the last 5 values
> to be
> > > > left
> > > > > > > > > > unspecified
> > > > > > > > > > > > if
> > > > > > > > > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2) In addition, the
> > > > kafka.security.authorizer.AclAuthorizer class
> > > > > > > > > > > > supports
> > > > > > > > > > > > > the ability to connect to a different Zookeeper
> instance
> > > > than the
> > > > > > > > > > one the
> > > > > > > > > > > > > brokers use. We therefore also add the following
> > > > optional configs,
> > > > > > > > > > which
> > > > > > > > > > > > > override the corresponding ones from above when
> present:
> > > > > > > > > > > > > authorizer.zookeeper.client.secure
> > > > > > > > > > > > > authorizer.zookeeper.clientCnxnSocket
> > > > > > > > > > > > > authorizer.zookeeper.ssl.keyStore.location
> > > > > > > > > > > > > authorizer.zookeeper.ssl.keyStore.password
> > > > > > > > > > > > > authorizer.zookeeper.ssl.trustStore.location
> > > > > > > > > > > > > authorizer.zookeeper.ssl.trustStore.password
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3) The three CLI tools mentioned above will
> support a new
> > > > > > > > > > > > --zk-tls-config-file
> > > > > > > > > > > > > <String: Zookeeper TLS configuration file path>"
> > > > option. The
> > > > > > > > > > following
> > > > > > > > > > > > > properties will be recognized in that file, and
> > > > unrecognized
> > > > > > > > > > properties
> > > > > > > > > > > > > will be ignored to allow the possibility of
> pointing
> > > > > > > > > > zk-tls-config-file
> > > > > > > > > > > > at
> > > > > > > > > > > > > the broker's config file.
> > > > > > > > > > > > > zookeeper.client.secure (default value = false)
> > > > > > > > > > > > > zookeeper.clientCnxnSocket
> > > > > > > > > > > > > zookeeper.ssl.keyStore.location
> > > > > > > > > > > > > zookeeper.ssl.keyStore.password
> > > > > > > > > > > > > zookeeper.ssl.trustStore.location
> > > > > > > > > > > > > zookeeper.ssl.trustStore.password
> > > > > > > > > > > > > It will be an error for any of the last 5 values
> to be
> > > > left
> > > > > > > > > > unspecified
> > > > > > > > > > > > if
> > > > > > > > > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > > > > > > > >
> > > > > > > > > > > > Do we really need a --zk-tls-config-file flag? It
> seems
> > > > like this
> > > > > > > > > > could
> > > > > > > > > > > > just appear in a configuration file, which all of
> these
> > > > tools already
> > > > > > > > > > > > accept (I think).
> > > > > > > > > > > >
> > > > > > > > > > > > best,
> > > > > > > > > > > > Colin
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Ron
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <
> > > > rndgstn@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi everyone. Let's get this discussion going
> again
> > > > now that Kafka
> > > > > > > > > > 2.4
> > > > > > > > > > > > > > with Zookeeper 3.5.6 is out.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > First, regarding the KIP number, the other KIP
> that
> > > > was using this
> > > > > > > > > > > > number
> > > > > > > > > > > > > > moved to KIP 534, so KIP 515 remains the correct
> > > > number for this
> > > > > > > > > > > > > > discussion. I've updated the Kafka Improvement
> > > > Proposal page to
> > > > > > > > > > list
> > > > > > > > > > > > this
> > > > > > > > > > > > > > KIP in the 515 slot, so we're all set there.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding the actual issues under discussion, I
> think
> > > > there are
> > > > > > > > > > some
> > > > > > > > > > > > > > things we should clarify.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1) It is possible to use TLS connectivity to
> Zookeeper
> > > > from Apache
> > > > > > > > > > > > Kafka
> > > > > > > > > > > > > > 2.4 -- the problem is that configuration
> information
> > > > has to be
> > > > > > > > > > passed
> > > > > > > > > > > > via
> > > > > > > > > > > > > > system properties as "-D" command line options
> on the
> > > > java
> > > > > > > > > > invocation
> > > > > > > > > > > > of
> > > > > > > > > > > > > > the broker, and those are not secure (anyone with
> > > > access to the box
> > > > > > > > > > > > can see
> > > > > > > > > > > > > > the command line used to invoke the broker); the
> > > > configuration
> > > > > > > > > > includes
> > > > > > > > > > > > > > sensitive information (e.g. a keystore
> password), so
> > > > we need a
> > > > > > > > > > secure
> > > > > > > > > > > > > > mechanism for passing the configuration values.
> I
> > > > believe the real
> > > > > > > > > > > > > > motivation for this KIP is to harden the
> configuration
> > > > mechanism
> > > > > > > > > > for
> > > > > > > > > > > > > > Zookeeper TLS connectivity.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2) I believe the list of CLI tools that continue
> to
> > > > use direct
> > > > > > > > > > > > Zookeeper
> > > > > > > > > > > > > > connectivity in a non-deprecated fashion is:
> > > > > > > > > > > > > > a)
> > > > > > > > > >
> zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > > > > > > > > > > > b)
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > >
> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > > > > > > > > > > > c) kafka-configs.{sh,
> bat}/kafka.admin.ConfigCommand
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3) I believe Kafka.admin.ConfigCommand presents a
> > > > conundrum
> > > > > > > > > > because it
> > > > > > > > > > > > > > explicitly states in a comment that a supported
> use
> > > > case is
> > > > > > > > > > > > bootstrapping a
> > > > > > > > > > > > > > Kafka cluster with encrypted passwords in
> Zookeeper
> > > > (see
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > > > > > > > > > > > ).
> > > > > > > > > > > > > > This means it will be especially difficult to
> > > > deprecate this
> > > > > > > > > > particular
> > > > > > > > > > > > > > direct Zookeeper connectivity without a different
> > > > storage
> > > > > > > > > > mechanism for
> > > > > > > > > > > > > > dynamic configuration values being available
> (i.e. the
> > > > self-managed
> > > > > > > > > > > > quorum
> > > > > > > > > > > > > > referred to in KIP-500).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think it would be easier and simpler to harden
> the
> > > > Zookeeper TLS
> > > > > > > > > > > > > > configuration in both Kafka and in CLI tools
> compared
> > > > to trying to
> > > > > > > > > > > > > > deprecate the direct Zookeeper connectivity in
> the
> > > > above 3 CLI
> > > > > > > > > > tools --
> > > > > > > > > > > > > > especially when ConfigCommand has no obvious
> > > > short-term path to
> > > > > > > > > > > > deprecation.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding how to harden the TLS configuration,
> adding
> > > > the
> > > > > > > > > > appropriate
> > > > > > > > > > > > > > configs to Kafka is an obvious choice for the
> > > > brokers. Not that
> > > > > > > > > > these
> > > > > > > > > > > > > > values cannot be dynamically reconfigurable
> because
> > > > the values
> > > > > > > > > > would be
> > > > > > > > > > > > > > required to connect to Zookeeper via TLS in the
> first
> > > > place.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding how to harden the TLS configuration
> for the
> > > > 3 remaining
> > > > > > > > > > CLI
> > > > > > > > > > > > > > tools, it would be relatively straightforward to
> add
> > > > support for a
> > > > > > > > > > > > > > "--zk-tls-config-file <String: Zookeeper TLS
> > > > configuration system
> > > > > > > > > > > > > > properties file path>" option to each one.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thoughts?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Ron
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes
> <
> > > > > > > > > > pere.urbon@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> True, the number is duplicated.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> do you know how can we get the problem fix? I
> was
> > > > not aware, I
> > > > > > > > > > > > missed,
> > > > > > > > > > > > > >> sorry the need to add the KIP to the table.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> -- Pere
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Missatge de Jorge Esteban Quilcate Otoya <
> > > > > > > > > > quilcate.jorge@gmail.com>
> > > > > > > > > > > > del
> > > > > > > > > > > > > >> dia
> > > > > > > > > > > > > >> dt., 3 de set. 2019 a les 13:43:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> > Hi Pere,
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > Have you add your KIP to the list here
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > > > > > > > > > >> ?
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > I found the KIP number assigned to another.
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón
> Bayes <
> > > > > > > > > > > > pere.urbon@gmail.com>
> > > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >> Thanks for your time Harsha,
> > > > > > > > > > > > > >> >> anyone else with comments? looking
> forward to
> > > > hearing from
> > > > > > > > > > you.
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> Stupid question: when do you move from
> discussion
> > > > to vote?
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> Missatge de Harsha Chintalapani <
> kafka@harsha.io>
> > > > del dia
> > > > > > > > > > dv., 30
> > > > > > > > > > > > > >> d’ag.
> > > > > > > > > > > > > >> >> 2019 a les 21:59:
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> > Thanks Pere. KIP looks good to me.
> > > > > > > > > > > > > >> >> > -Harsha
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere
> Urbón
> > > > Bayes <
> > > > > > > > > > > > > >> >> pere.urbon@gmail.com>
> > > > > > > > > > > > > >> >> > wrote:
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> >> Not really,
> > > > > > > > > > > > > >> >> >> my idea is to keep the JAAS parameter,
> so
> > > > people don't see
> > > > > > > > > > > > major
> > > > > > > > > > > > > >> >> >> changes. But if you pass a properties
> file,
> > > > then this takes
> > > > > > > > > > > > > >> precedence
> > > > > > > > > > > > > >> >> over
> > > > > > > > > > > > > >> >> >> the other, with the idea that you can do
> sasl
> > > > as well with
> > > > > > > > > > the
> > > > > > > > > > > > > >> >> properties
> > > > > > > > > > > > > >> >> >> files.
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >> >> Makes sense?
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >> >> -- Pere
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >> >> Missatge de Harsha Chintalapani <
> > > > kafka@harsha.io> del dia
> > > > > > > > > > dv.,
> > > > > > > > > > > > 30
> > > > > > > > > > > > > >> >> d’ag.
> > > > > > > > > > > > > >> >> >> 2019 a les 19:00:
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >> >>> Hi Pere,
> > > > > > > > > > > > > >> >> >>> Thanks for the KIP. Enabling
> SSL for
> > > > zookeeper
> > > > > > > > > > for
> > > > > > > > > > > > Kafka
> > > > > > > > > > > > > >> >> makes
> > > > > > > > > > > > > >> >> >>> sense.
> > > > > > > > > > > > > >> >> >>> "The changes are planned to be
> introduced in a
> > > > compatible
> > > > > > > > > > way,
> > > > > > > > > > > > by
> > > > > > > > > > > > > >> >> >>> keeping the current JAAS variable
> precedence."
> > > > > > > > > > > > > >> >> >>> Can you elaborate a bit here. If the user
> > > > configures a JAAS
> > > > > > > > > > > > file
> > > > > > > > > > > > > >> with
> > > > > > > > > > > > > >> >> >>> Client section it will take precedence
> over
> > > > zookeeper SSL
> > > > > > > > > > > > configs?
> > > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > > >> >> >>> Thanks,
> > > > > > > > > > > > > >> >> >>> Harsha
> > > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere
> Urbón
> > > > Bayes <
> > > > > > > > > > > > > >> >> pere.urbon@gmail.com>
> > > > > > > > > > > > > >> >> >>> wrote:
> > > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > > >> >> >>>> Hi,
> > > > > > > > > > > > > >> >> >>>> quick question, I saw in another mail
> that
> > > > 2.4 release is
> > > > > > > > > > > > planned
> > > > > > > > > > > > > >> for
> > > > > > > > > > > > > >> >> >>>> September. I think it would be really
> awesome
> > > > to have
> > > > > > > > > > this for
> > > > > > > > > > > > > >> this
> > > > > > > > > > > > > >> >> >>>> release, do you think we can make it?
> > > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > > >> >> >>>> -- Pere
> > > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > > >> >> >>>> Missatge de Pere Urbón Bayes <
> > > > pere.urbon@gmail.com> del
> > > > > > > > > > dia
> > > > > > > > > > > > dj.,
> > > > > > > > > > > > > >> 29
> > > > > > > > > > > > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > > >> >> >>>> Hi,
> > > > > > > > > > > > > >> >> >>>> this is my first KIP for a change in
> Apache
> > > > Kafka, so I'm
> > > > > > > > > > > > really
> > > > > > > > > > > > > >> need
> > > > > > > > > > > > > >> >> >>>> to the process. Looking forward to
> hearing
> > > > from you and
> > > > > > > > > > learn
> > > > > > > > > > > > the
> > > > > > > > > > > > > >> >> best
> > > > > > > > > > > > > >> >> >>>> ropes here.
> > > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > > >> >> >>>> I would like to propose this KIP-515 to
> > > > enable the
> > > > > > > > > > > > > >> ZookeeperClients
> > > > > > > > > > > > > >> >> to
> > > > > > > > > > > > > >> >> >>>> take full advantage of the TLS
> communication
> > > > in the new
> > > > > > > > > > > > Zookeeper
> > > > > > > > > > > > > >> >> 3.5.5.
> > > > > > > > > > > > > >> >> >>>> Specially interesting it the Zookeeper
> > > > Security Migration,
> > > > > > > > > > > > that
> > > > > > > > > > > > > >> >> without
> > > > > > > > > > > > > >> >> >>>> this change will not work with TLS,
> disabling
> > > > users to use
> > > > > > > > > > > > ACLs
> > > > > > > > > > > > > >> when
> > > > > > > > > > > > > >> >> the
> > > > > > > > > > > > > >> >> >>>> Zookeeper cluster use TLS.
> > > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > > >> >> >>>> link:
> > > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > > >> >> >>>> Looking forward to hearing from you on
> this,
> > > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > > >> >> >>>> /cheers
> > > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > > >> >> >>>> --
> > > > > > > > > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > > > > > > > > >> >> >>>> Software Architect
> > > > > > > > > > > > > >> >> >>>> http://www.purbon.com
> > > > > > > > > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > > > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > > >> >> >>>> --
> > > > > > > > > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > > > > > > > > >> >> >>>> Software Architect
> > > > > > > > > > > > > >> >> >>>> http://www.purbon.com
> > > > > > > > > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > > > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >> >> --
> > > > > > > > > > > > > >> >> >> Pere Urbon-Bayes
> > > > > > > > > > > > > >> >> >> Software Architect
> > > > > > > > > > > > > >> >> >> http://www.purbon.com
> > > > > > > > > > > > > >> >> >> https://twitter.com/purbon
> > > > > > > > > > > > > >> >> >> https://www.linkedin.com/in/purbon/
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> --
> > > > > > > > > > > > > >> >> Pere Urbon-Bayes
> > > > > > > > > > > > > >> >> Software Architect
> > > > > > > > > > > > > >> >> http://www.purbon.com
> > > > > > > > > > > > > >> >> https://twitter.com/purbon
> > > > > > > > > > > > > >> >> https://www.linkedin.com/in/purbon/
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> --
> > > > > > > > > > > > > >> Pere Urbon-Bayes
> > > > > > > > > > > > > >> Software Architect
> > > > > > > > > > > > > >> http://www.purbon.com
> > > > > > > > > > > > > >> https://twitter.com/purbon
> > > > > > > > > > > > > >> https://www.linkedin.com/in/purbon/
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
>
>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Ron Dagostino <rn...@gmail.com>.
Hi Colin. TLS to ZooKeeper will remain an opt-in. The configuration
value that turns it on is zookeeper.ssl.client.enable (which used to
be called zookeeper.client.secure before we renamed everything to be
consistent with Kafka's config naming style and eliminate confusion).
The default value for this is false. Everything else related to TLS
has no effect unless this is explicitly set to true, and this value
does not inherit from anywhere,
Ron
On Thu, Jan 16, 2020 at 5:39 PM Colin McCabe <cm...@apache.org> wrote:
>
> On Thu, Jan 16, 2020, at 02:58, Rajini Sivaram wrote:
> > Hi Ron/Colin,
> >
> > Thanks for the responses. I think we need to consider these cases:
> >
> > 1) The recommended approach: This is the one we document. Configs
> > consistent with Kafka make sense here. It seems a lot more confusing to
> > configure `ssl.cipher.suites`, `ssl.enabled.protocols`, `
> > zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols` with
> > multiple naming inconsistencies in server.properties. For users who have
> > never used ZK system properties, this is clearly simpler. Another config to
> > consider is `ssl.endpoint.identification.algorithm` which does the same
> > thing as `zookeeper.ssl.hostnameVerification` - if we are staying
> > consistent, we should use the Kafka format for this too. I would also
> > consider using something like `zookeeper.client.ssl.enabled` instead of `
> > zookeeper.client.secure` since it is difficult to tell what `secure` means
> > when you have options to configure Kerberos, ACLs and/or SSL.
> >
> > 2) Inheritance using prefixed configs as we do for other prefixed configs
> > in Kafka: I think it may be useful for configs like `ssl.protocol` and `
> > ssl.cipher.suites`.
> >
> > 3) Default values: Do we want to use Kafka defaults? The inconsistency we
> > have is `ssl.protocol/ssl.enabled.protocols` since we still have insecure
> > protocols enabled for Kafka. We have KIP-553 (
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956)
> > open to disable these. It is currently blocked since it also talks about
> > enabling TLSv1.3 by default and we haven't done sufficient testing for
> > that. Since it will be good to disable the older insecure protocols anyway,
> > perhaps we could do that without enabling TLSv1.3 by default in 2.5?
>
> Hi Rajini,
>
> Inheriting the SSL settings from Kafka would have the effect of changing the defaults for almost everyone using SSL from non-SSL ZK to SSL-based ZK, right? This seems to expand the scope of the KIP greatly, since it moves it from being opt-in to opt-out. Are there any potential negative implications of this?
>
> best,
> Colin
>
>
> >
> > 4) Backward compatibility for deployments which are using system
> > properties. We shouldn't override system properties with Kafka defaults or
> > inherited values. But we do want to override if user configures properties
> > explicitly. In the current approach, this was simpler since we just had to
> > set the configured values. But if we decide to inherit Kafka configs, then
> > we will need to check system properties and update accordingly. But that
> > should be doable?
> >
> > On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote:
> > > > Hi Colin. Two things come to mind with respect to ZooKeeper camelCase
> > > > style vs Kafka-style config names for ZooKeeper. First, I think it
> > > > would be desirable for the client configs and broker configs to be
> > > > interoperable. For example, it feels like it would be convenient to
> > > > be able to pass the broker's config file to the ZK Security Migrator
> > > > tool. I think whatever we use (ZooKeeper camelCase style or Kafka
> > > > style), I think we should use the same for both.
> > > >
> > > > The second thing to think about is the fact that ZooKeeper
> > > > configuration inherently uses Java system properties as a first pass.
> > > > So while we might switch to Kafka-style configs in the config file
> > > > (e.g. zookeeper.ssl.trust.store.location), the
> > > > org.apache.zookeeper.client.ZKClientConfig class (which is how TLS
> > > > configs are set) is going to look for the camelCase
> > > > zookeeper.ssl.trustStore.location system property and use any value
> > > > there. I think this could be very confusing for people. Granted, we
> > > > are doing this so that people don't have to use system properties, but
> > > > there is no way to turn off the use of system properties, so I think
> > > > there would be pretty substantial potential for confusion.
> > >
> > > I don't think we document or recommend that anyone use system properties
> > > to configure Zookeeper usage within Kafka. And I think the reason is
> > > exactly the issue you pointed out -- it wouldn't be consistent with our
> > > other configurations.
> > >
> > > best,
> > > Colin
> > >
> > > >
> > > > One idea is to migrate the system properties -- i.e. look to see if
> > > > zookeeper.ssl.trustStore.location is set and if it is clear out the
> > > > value there and move it under the zookeeper.ssl.trust.store.location
> > > > system property. (I said it was an idea -- not necessarily a good one
> > > > :-)
> > > >
> > > > Ron
> > > >
> > > > On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe <cm...@apache.org> wrote:
> > > > >
> > > > > On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
> > > > > > Thanks Colin and Rajini.
> > > > > >
> > > > > > I've updated the KIP to say that KIP-421 functionality is available
> > > to
> > > > > > encrypt sensitive configs like the ZK key store and trust store
> > > > > > passwords. (I've also made it clear that the configs are not
> > > > > > dynamically reconfigurable since dynamic values are stored in ZK and
> > > > > > the data is needed to get to ZK in the first place). Colin, let me
> > > > > > know if you think anything else needs to be said/done about it -- I
> > > > > > wasn't sure if your comment above implies that there are additional
> > > > > > direct actions that we need to take aside from this.
> > > > > >
> > > > > > I agree that making the brokers' ZooKeeper clients dynamic with
> > > > > > respect to key and trust stores (e.g. via
> > > > > > zookeeper.ssl.context.supplier.class) may increase the risk that this
> > > > > > KIP may not land in time for 2.5.
> > > > > >
> > > > > > Regarding the config names for ZK being different than the ZooKeeper
> > > > > > configs (e.g. camel-case keyStore/trustStore instead of
> > > > > > keystore/truststore, ciphersuites instead of cipher.suites,
> > > > > > enabledProtocols instead of enabled.protocols), I agree it is an
> > > > > > issue. I tried to mitigate it by putting warning text in the config
> > > > > > docs for these. Regarding configuration inheritance, I think what
> > > you
> > > > > > are saying is that for any configs where the broker has a clear
> > > > > > semantic equivalent, the broker could fall back to the broker value
> > > if
> > > > > > the ZK value is not given. The potential list is:
> > > > > >
> > > > > > zookeeper.ssl.keyStore.location => ssl.keystore.location
> > > > > > zookeeper.ssl.keyStore.password => ssl.keystore.password
> > > > > > zookeeper.ssl.keyStore.type => ssl.keystore.type
> > > > > > zookeeper.ssl.trustStore.location => ssl.truststore.location
> > > > > > zookeeper.ssl.trustStore.password => ssl.truststore.password
> > > > > > zookeeper.ssl.trustStore.type => ssl.truststore.type
> > > > > > zookeeper.ssl.ciphersuites => ssl.cipher.suites
> > > > > >
> > > > > > Note that there are two configs that look the same based on their key
> > > > > > names but whose allowable values differ:
> > > > > >
> > > > > > zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default:
> > > TLS)
> > > > > > zookeeper.ssl.enabledProtocols(Default: value of protocol property) =
> > > > > > ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
> > > > > >
> > > > > > Not sure what the right answer is, but hopefully this detail will
> > > help
> > > > > > get us there.
> > > > > >
> > > > > > Ron
> > > > >
> > > > > I think, on balance, I agree with Rajini that it would be nice to make
> > > the configs look more like Kafka configs. We don't have camel-case in
> > > configuration keys, so we should avoid it here.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > >
> > > > > > On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <cm...@apache.org>
> > > wrote:
> > > > > > >
> > > > > > > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> > > > > > > > Hi Ron,
> > > > > > > >
> > > > > > > > Thanks for the detailed explanation, sounds good to me.
> > > > > > > >
> > > > > > > > A few more questions:
> > > > > > > >
> > > > > > > > 1) At the moment, all sensitive broker configs including
> > > > > > > > keystore/truststore passwords can be stored encrypted in
> > > ZooKeeper prior to
> > > > > > > > starting up brokers. We are now adding SSL keystore/truststore
> > > passwords
> > > > > > > > for ZooKeeper client that cannot be stored in ZooKeeper since
> > > you need
> > > > > > > > these to connect to ZK. We should perhaps document that these
> > > configs can
> > > > > > > > be encrypted using KIP-421.
> > > > > > >
> > > > > > > That's a good point. Previously, we didn't have ZK on-the-wire
> > > security support. However, now that we do, sending sensitive keystore and
> > > truststore passwords to ZK in cleartext should be deprecated in favor of
> > > using KIP-421.
> > > > > > >
> > > > > > > >
> > > > > > > > 2) We can dynamically update keystores and trust stores used by
> > > brokers
> > > > > > > > without restarting the broker. Can we support this easily for ZK
> > > clients
> > > > > > > > created by the broker, for example by adding our own
> > > > > > > > `zookeeper.ssl.context.supplier.class`?
> > > > > > > >
> > > > > > >
> > > > > > > Hmm. That might be better to consider in a follow-up, since the
> > > deadline for 2.5 KIPs is coming up?
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > > 3) Looks like we are using config names that directly map to ZK
> > > configs.
> > > > > > > > Have we considered using equivalent Kafka config names with
> > > prefixes,
> > > > > > > > perhaps with inheritance from the non-prefixed names? Not sure
> > > if this is a
> > > > > > > > good idea, but perhaps worth documenting in Rejected
> > > Alternatives at least.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <cm...@apache.org>
> > > wrote:
> > > > > > > >
> > > > > > > > > Hi Ron,
> > > > > > > > >
> > > > > > > > > Thanks for the explanation. I guess thinking about it a
> > > little bit more,
> > > > > > > > > we should just add --zk-tls-config-file to all of these
> > > commands.
> > > > > > > > >
> > > > > > > > > We will be removing this option (plus ZK in general) from
> > > these commands
> > > > > > > > > in the next major release, but ZK is still supported in 2.5,
> > > so we should
> > > > > > > > > just do the logical thing. (The exception is
> > > ZkSecurityMigrator, which
> > > > > > > > > will stay).
> > > > > > > > >
> > > > > > > > > best,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > > > > > > > > Hi Colin.
> > > > > > > > > >
> > > > > > > > > > <<< It seems like this [--zk-tls-config-file information]
> > > could just
> > > > > > > > > appear
> > > > > > > > > > in a configuration file, which all of these tools already
> > > accept (I
> > > > > > > > > think)
> > > > > > > > > >
> > > > > > > > > > ZkSecurityMigrator has no such property file facility;
> > > adding a
> > > > > > > > > > "--zk-tls-config-file" parameter is exactly for this
> > > purpose. If we add
> > > > > > > > > > that to ZkSecurityMigrator then it is trivial to add it to
> > > other commands
> > > > > > > > > > (the same code is simply reused; it ends up being just a few
> > > extra
> > > > > > > > > lines).
> > > > > > > > > > I do not see any parameter in the other two commands to
> > > adjust the ZK
> > > > > > > > > > connection; ConfigCommand accepts a "--command-config" flag,
> > > but
> > > > > > > > > according
> > > > > > > > > > to the code "This is used only with --bootstrap-server
> > > option for
> > > > > > > > > > describing and altering broker configs."
> > > > > > > > > >
> > > > > > > > > > I do agree there would be no need to add
> > > "--zk-tls-config-file" to
> > > > > > > > > > ReassignPartitionsCommand if its direct ZK connectivity is
> > > replaced in
> > > > > > > > > time
> > > > > > > > > > for the next release.
> > > > > > > > > >
> > > > > > > > > > ConfigCommand supports the "--bootstrap-server" option and
> > > will have its
> > > > > > > > > > direct ZooKeeper access formally deprecated as per KIP-555,
> > > but the
> > > > > > > > > special
> > > > > > > > > > use case of bootstrapping a ZooKeeper ensemble with
> > > encrypted credentials
> > > > > > > > > > prior to starting Kafka will still exist, so it feels like
> > > while
> > > > > > > > > > "--zk-tls-config-file" would never be used except for this
> > > use case it
> > > > > > > > > > could probably still be added for this particular situation.
> > > > > > > > > >
> > > > > > > > > > Ron
> > > > > > > > > >
> > > > > > > > > > P.S. I responded on 1/6 but I just discovered that e, ail
> > > (and 3 more I
> > > > > > > > > > sent) did not go through. I am trying to get emails through
> > > now to move
> > > > > > > > > > this discussion forward.
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <
> > > cmccabe@apache.org> wrote:
> > > > > > > > > >
> > > > > > > > > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > > > > > > > > > Hi everyone. I would like to make the following changes
> > > to the KIP.
> > > > > > > > > > > >
> > > > > > > > > > > > MOTIVATION:
> > > > > > > > > > > > Include a statement that it will be difficult in the
> > > short term to
> > > > > > > > > > > > deprecate direct Zookeeper communication in
> > > kafka-configs.{sh, bat}
> > > > > > > > > > > (which
> > > > > > > > > > > > invoke kafka.admin.ConfigCommand) because bootstrapping
> > > a Kafka
> > > > > > > > > cluster
> > > > > > > > > > > > with encrypted passwords in Zookeeper is an
> > > explicitly-supported use
> > > > > > > > > > > case;
> > > > > > > > > > > > therefore it is in scope to be able to securely
> > > configure the CLI
> > > > > > > > > tools
> > > > > > > > > > > > that still leverage non-deprecated direct Zookeeper
> > > communication
> > > > > > > > > for TLS
> > > > > > > > > > > > (the other 2 tools are kafka-reassign-partitions.{sh,
> > > bat} and
> > > > > > > > > > > > zookeeper-security-migration.sh).
> > > > > > > > > > >
> > > > > > > > > > > Hi Ron,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the KIP.
> > > > > > > > > > >
> > > > > > > > > > > About deprecations:
> > > > > > > > > > >
> > > > > > > > > > > * zookeeper-security-migration: clearly, deprecating ZK
> > > access in this
> > > > > > > > > one
> > > > > > > > > > > would not make sense, since it would defeat the whole
> > > point of the
> > > > > > > > > tool :)
> > > > > > > > > > >
> > > > > > > > > > > * kafka-reassign-partitions: ZK access should be
> > > deprecated here.
> > > > > > > > > KIP-455
> > > > > > > > > > > implementation has dragged a bit, but this should get done
> > > soon.
> > > > > > > > > Certainly
> > > > > > > > > > > before the next release.
> > > > > > > > > > >
> > > > > > > > > > > * kafka-configs: I think ZK access should be deprecated
> > > here as well.
> > > > > > > > > I
> > > > > > > > > > > agree there is a super-special bootstrapping case here,
> > > but that should
> > > > > > > > > > > have its own tool, not use kafka-configs.
> > > > > > > > > > >
> > > > > > > > > > > I will post a separate KIP for this, though.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > GOALS:
> > > > > > > > > > > > Support the secure configuration of TLS-encrypted
> > > communication
> > > > > > > > > between
> > > > > > > > > > > > Zookeeper and:
> > > > > > > > > > > > a) Kafka brokers
> > > > > > > > > > > > b) The three CLI tools mentioned above that still
> > > support direct,
> > > > > > > > > > > > non-deprecated communication to Zookeeper
> > > > > > > > > > > > It is explicitly out-of-scope to deprecate any direct
> > > Zookeeper
> > > > > > > > > > > > communication in CLI tools as part of this KIP; such
> > > work will occur
> > > > > > > > > in
> > > > > > > > > > > > future KIPs instead.
> > > > > > > > > > > >
> > > > > > > > > > > > PUBLIC INTERFACES:
> > > > > > > > > > > > 1) The following new broker configurations will be
> > > recognized.
> > > > > > > > > > > > zookeeper.client.secure (default value = false, for
> > > backwards
> > > > > > > > > > > > compatibility)
> > > > > > > > > > > > zookeeper.clientCnxnSocket
> > > > > > > > > > > > zookeeper.ssl.keyStore.location
> > > > > > > > > > > > zookeeper.ssl.keyStore.password
> > > > > > > > > > > > zookeeper.ssl.trustStore.location
> > > > > > > > > > > > zookeeper.ssl.trustStore.password
> > > > > > > > > > > > It will be an error for any of the last 5 values to be
> > > left
> > > > > > > > > unspecified
> > > > > > > > > > > if
> > > > > > > > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > > > > > > > >
> > > > > > > > > > > > 2) In addition, the
> > > kafka.security.authorizer.AclAuthorizer class
> > > > > > > > > > > supports
> > > > > > > > > > > > the ability to connect to a different Zookeeper instance
> > > than the
> > > > > > > > > one the
> > > > > > > > > > > > brokers use. We therefore also add the following
> > > optional configs,
> > > > > > > > > which
> > > > > > > > > > > > override the corresponding ones from above when present:
> > > > > > > > > > > > authorizer.zookeeper.client.secure
> > > > > > > > > > > > authorizer.zookeeper.clientCnxnSocket
> > > > > > > > > > > > authorizer.zookeeper.ssl.keyStore.location
> > > > > > > > > > > > authorizer.zookeeper.ssl.keyStore.password
> > > > > > > > > > > > authorizer.zookeeper.ssl.trustStore.location
> > > > > > > > > > > > authorizer.zookeeper.ssl.trustStore.password
> > > > > > > > > > > >
> > > > > > > > > > > > 3) The three CLI tools mentioned above will support a new
> > > > > > > > > > > --zk-tls-config-file
> > > > > > > > > > > > <String: Zookeeper TLS configuration file path>"
> > > option. The
> > > > > > > > > following
> > > > > > > > > > > > properties will be recognized in that file, and
> > > unrecognized
> > > > > > > > > properties
> > > > > > > > > > > > will be ignored to allow the possibility of pointing
> > > > > > > > > zk-tls-config-file
> > > > > > > > > > > at
> > > > > > > > > > > > the broker's config file.
> > > > > > > > > > > > zookeeper.client.secure (default value = false)
> > > > > > > > > > > > zookeeper.clientCnxnSocket
> > > > > > > > > > > > zookeeper.ssl.keyStore.location
> > > > > > > > > > > > zookeeper.ssl.keyStore.password
> > > > > > > > > > > > zookeeper.ssl.trustStore.location
> > > > > > > > > > > > zookeeper.ssl.trustStore.password
> > > > > > > > > > > > It will be an error for any of the last 5 values to be
> > > left
> > > > > > > > > unspecified
> > > > > > > > > > > if
> > > > > > > > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > > > > > > >
> > > > > > > > > > > Do we really need a --zk-tls-config-file flag? It seems
> > > like this
> > > > > > > > > could
> > > > > > > > > > > just appear in a configuration file, which all of these
> > > tools already
> > > > > > > > > > > accept (I think).
> > > > > > > > > > >
> > > > > > > > > > > best,
> > > > > > > > > > > Colin
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Ron
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <
> > > rndgstn@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi everyone. Let's get this discussion going again
> > > now that Kafka
> > > > > > > > > 2.4
> > > > > > > > > > > > > with Zookeeper 3.5.6 is out.
> > > > > > > > > > > > >
> > > > > > > > > > > > > First, regarding the KIP number, the other KIP that
> > > was using this
> > > > > > > > > > > number
> > > > > > > > > > > > > moved to KIP 534, so KIP 515 remains the correct
> > > number for this
> > > > > > > > > > > > > discussion. I've updated the Kafka Improvement
> > > Proposal page to
> > > > > > > > > list
> > > > > > > > > > > this
> > > > > > > > > > > > > KIP in the 515 slot, so we're all set there.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regarding the actual issues under discussion, I think
> > > there are
> > > > > > > > > some
> > > > > > > > > > > > > things we should clarify.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1) It is possible to use TLS connectivity to Zookeeper
> > > from Apache
> > > > > > > > > > > Kafka
> > > > > > > > > > > > > 2.4 -- the problem is that configuration information
> > > has to be
> > > > > > > > > passed
> > > > > > > > > > > via
> > > > > > > > > > > > > system properties as "-D" command line options on the
> > > java
> > > > > > > > > invocation
> > > > > > > > > > > of
> > > > > > > > > > > > > the broker, and those are not secure (anyone with
> > > access to the box
> > > > > > > > > > > can see
> > > > > > > > > > > > > the command line used to invoke the broker); the
> > > configuration
> > > > > > > > > includes
> > > > > > > > > > > > > sensitive information (e.g. a keystore password), so
> > > we need a
> > > > > > > > > secure
> > > > > > > > > > > > > mechanism for passing the configuration values. I
> > > believe the real
> > > > > > > > > > > > > motivation for this KIP is to harden the configuration
> > > mechanism
> > > > > > > > > for
> > > > > > > > > > > > > Zookeeper TLS connectivity.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2) I believe the list of CLI tools that continue to
> > > use direct
> > > > > > > > > > > Zookeeper
> > > > > > > > > > > > > connectivity in a non-deprecated fashion is:
> > > > > > > > > > > > > a)
> > > > > > > > > zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > > > > > > > > > > b)
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > > > > > > > > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3) I believe Kafka.admin.ConfigCommand presents a
> > > conundrum
> > > > > > > > > because it
> > > > > > > > > > > > > explicitly states in a comment that a supported use
> > > case is
> > > > > > > > > > > bootstrapping a
> > > > > > > > > > > > > Kafka cluster with encrypted passwords in Zookeeper
> > > (see
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > > > > > > > > > > ).
> > > > > > > > > > > > > This means it will be especially difficult to
> > > deprecate this
> > > > > > > > > particular
> > > > > > > > > > > > > direct Zookeeper connectivity without a different
> > > storage
> > > > > > > > > mechanism for
> > > > > > > > > > > > > dynamic configuration values being available (i.e. the
> > > self-managed
> > > > > > > > > > > quorum
> > > > > > > > > > > > > referred to in KIP-500).
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think it would be easier and simpler to harden the
> > > Zookeeper TLS
> > > > > > > > > > > > > configuration in both Kafka and in CLI tools compared
> > > to trying to
> > > > > > > > > > > > > deprecate the direct Zookeeper connectivity in the
> > > above 3 CLI
> > > > > > > > > tools --
> > > > > > > > > > > > > especially when ConfigCommand has no obvious
> > > short-term path to
> > > > > > > > > > > deprecation.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regarding how to harden the TLS configuration, adding
> > > the
> > > > > > > > > appropriate
> > > > > > > > > > > > > configs to Kafka is an obvious choice for the
> > > brokers. Not that
> > > > > > > > > these
> > > > > > > > > > > > > values cannot be dynamically reconfigurable because
> > > the values
> > > > > > > > > would be
> > > > > > > > > > > > > required to connect to Zookeeper via TLS in the first
> > > place.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regarding how to harden the TLS configuration for the
> > > 3 remaining
> > > > > > > > > CLI
> > > > > > > > > > > > > tools, it would be relatively straightforward to add
> > > support for a
> > > > > > > > > > > > > "--zk-tls-config-file <String: Zookeeper TLS
> > > configuration system
> > > > > > > > > > > > > properties file path>" option to each one.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thoughts?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Ron
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <
> > > > > > > > > pere.urbon@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> True, the number is duplicated.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> do you know how can we get the problem fix? I was
> > > not aware, I
> > > > > > > > > > > missed,
> > > > > > > > > > > > >> sorry the need to add the KIP to the table.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> -- Pere
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Missatge de Jorge Esteban Quilcate Otoya <
> > > > > > > > > quilcate.jorge@gmail.com>
> > > > > > > > > > > del
> > > > > > > > > > > > >> dia
> > > > > > > > > > > > >> dt., 3 de set. 2019 a les 13:43:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> > Hi Pere,
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Have you add your KIP to the list here
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > > > > > > > > >> ?
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > I found the KIP number assigned to another.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> > > > > > > > > > > pere.urbon@gmail.com>
> > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >> Thanks for your time Harsha,
> > > > > > > > > > > > >> >> anyone else with comments? looking forward to
> > > hearing from
> > > > > > > > > you.
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> Stupid question: when do you move from discussion
> > > to vote?
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io>
> > > del dia
> > > > > > > > > dv., 30
> > > > > > > > > > > > >> d’ag.
> > > > > > > > > > > > >> >> 2019 a les 21:59:
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> > Thanks Pere. KIP looks good to me.
> > > > > > > > > > > > >> >> > -Harsha
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón
> > > Bayes <
> > > > > > > > > > > > >> >> pere.urbon@gmail.com>
> > > > > > > > > > > > >> >> > wrote:
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> >> Not really,
> > > > > > > > > > > > >> >> >> my idea is to keep the JAAS parameter, so
> > > people don't see
> > > > > > > > > > > major
> > > > > > > > > > > > >> >> >> changes. But if you pass a properties file,
> > > then this takes
> > > > > > > > > > > > >> precedence
> > > > > > > > > > > > >> >> over
> > > > > > > > > > > > >> >> >> the other, with the idea that you can do sasl
> > > as well with
> > > > > > > > > the
> > > > > > > > > > > > >> >> properties
> > > > > > > > > > > > >> >> >> files.
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> Makes sense?
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> -- Pere
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> Missatge de Harsha Chintalapani <
> > > kafka@harsha.io> del dia
> > > > > > > > > dv.,
> > > > > > > > > > > 30
> > > > > > > > > > > > >> >> d’ag.
> > > > > > > > > > > > >> >> >> 2019 a les 19:00:
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >>> Hi Pere,
> > > > > > > > > > > > >> >> >>> Thanks for the KIP. Enabling SSL for
> > > zookeeper
> > > > > > > > > for
> > > > > > > > > > > Kafka
> > > > > > > > > > > > >> >> makes
> > > > > > > > > > > > >> >> >>> sense.
> > > > > > > > > > > > >> >> >>> "The changes are planned to be introduced in a
> > > compatible
> > > > > > > > > way,
> > > > > > > > > > > by
> > > > > > > > > > > > >> >> >>> keeping the current JAAS variable precedence."
> > > > > > > > > > > > >> >> >>> Can you elaborate a bit here. If the user
> > > configures a JAAS
> > > > > > > > > > > file
> > > > > > > > > > > > >> with
> > > > > > > > > > > > >> >> >>> Client section it will take precedence over
> > > zookeeper SSL
> > > > > > > > > > > configs?
> > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > >> >> >>> Thanks,
> > > > > > > > > > > > >> >> >>> Harsha
> > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón
> > > Bayes <
> > > > > > > > > > > > >> >> pere.urbon@gmail.com>
> > > > > > > > > > > > >> >> >>> wrote:
> > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > >> >> >>>> Hi,
> > > > > > > > > > > > >> >> >>>> quick question, I saw in another mail that
> > > 2.4 release is
> > > > > > > > > > > planned
> > > > > > > > > > > > >> for
> > > > > > > > > > > > >> >> >>>> September. I think it would be really awesome
> > > to have
> > > > > > > > > this for
> > > > > > > > > > > > >> this
> > > > > > > > > > > > >> >> >>>> release, do you think we can make it?
> > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > >> >> >>>> -- Pere
> > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > >> >> >>>> Missatge de Pere Urbón Bayes <
> > > pere.urbon@gmail.com> del
> > > > > > > > > dia
> > > > > > > > > > > dj.,
> > > > > > > > > > > > >> 29
> > > > > > > > > > > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > >> >> >>>> Hi,
> > > > > > > > > > > > >> >> >>>> this is my first KIP for a change in Apache
> > > Kafka, so I'm
> > > > > > > > > > > really
> > > > > > > > > > > > >> need
> > > > > > > > > > > > >> >> >>>> to the process. Looking forward to hearing
> > > from you and
> > > > > > > > > learn
> > > > > > > > > > > the
> > > > > > > > > > > > >> >> best
> > > > > > > > > > > > >> >> >>>> ropes here.
> > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > >> >> >>>> I would like to propose this KIP-515 to
> > > enable the
> > > > > > > > > > > > >> ZookeeperClients
> > > > > > > > > > > > >> >> to
> > > > > > > > > > > > >> >> >>>> take full advantage of the TLS communication
> > > in the new
> > > > > > > > > > > Zookeeper
> > > > > > > > > > > > >> >> 3.5.5.
> > > > > > > > > > > > >> >> >>>> Specially interesting it the Zookeeper
> > > Security Migration,
> > > > > > > > > > > that
> > > > > > > > > > > > >> >> without
> > > > > > > > > > > > >> >> >>>> this change will not work with TLS, disabling
> > > users to use
> > > > > > > > > > > ACLs
> > > > > > > > > > > > >> when
> > > > > > > > > > > > >> >> the
> > > > > > > > > > > > >> >> >>>> Zookeeper cluster use TLS.
> > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > >> >> >>>> link:
> > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > >> >> >>>> Looking forward to hearing from you on this,
> > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > >> >> >>>> /cheers
> > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > >> >> >>>> --
> > > > > > > > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > > > > > > > >> >> >>>> Software Architect
> > > > > > > > > > > > >> >> >>>> http://www.purbon.com
> > > > > > > > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > >> >> >>>> --
> > > > > > > > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > > > > > > > >> >> >>>> Software Architect
> > > > > > > > > > > > >> >> >>>> http://www.purbon.com
> > > > > > > > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > >> >> >>>
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> --
> > > > > > > > > > > > >> >> >> Pere Urbon-Bayes
> > > > > > > > > > > > >> >> >> Software Architect
> > > > > > > > > > > > >> >> >> http://www.purbon.com
> > > > > > > > > > > > >> >> >> https://twitter.com/purbon
> > > > > > > > > > > > >> >> >> https://www.linkedin.com/in/purbon/
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> --
> > > > > > > > > > > > >> >> Pere Urbon-Bayes
> > > > > > > > > > > > >> >> Software Architect
> > > > > > > > > > > > >> >> http://www.purbon.com
> > > > > > > > > > > > >> >> https://twitter.com/purbon
> > > > > > > > > > > > >> >> https://www.linkedin.com/in/purbon/
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> --
> > > > > > > > > > > > >> Pere Urbon-Bayes
> > > > > > > > > > > > >> Software Architect
> > > > > > > > > > > > >> http://www.purbon.com
> > > > > > > > > > > > >> https://twitter.com/purbon
> > > > > > > > > > > > >> https://www.linkedin.com/in/purbon/
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Colin McCabe <cm...@apache.org>.
On Thu, Jan 16, 2020, at 02:58, Rajini Sivaram wrote:
> Hi Ron/Colin,
>
> Thanks for the responses. I think we need to consider these cases:
>
> 1) The recommended approach: This is the one we document. Configs
> consistent with Kafka make sense here. It seems a lot more confusing to
> configure `ssl.cipher.suites`, `ssl.enabled.protocols`, `
> zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols` with
> multiple naming inconsistencies in server.properties. For users who have
> never used ZK system properties, this is clearly simpler. Another config to
> consider is `ssl.endpoint.identification.algorithm` which does the same
> thing as `zookeeper.ssl.hostnameVerification` - if we are staying
> consistent, we should use the Kafka format for this too. I would also
> consider using something like `zookeeper.client.ssl.enabled` instead of `
> zookeeper.client.secure` since it is difficult to tell what `secure` means
> when you have options to configure Kerberos, ACLs and/or SSL.
>
> 2) Inheritance using prefixed configs as we do for other prefixed configs
> in Kafka: I think it may be useful for configs like `ssl.protocol` and `
> ssl.cipher.suites`.
>
> 3) Default values: Do we want to use Kafka defaults? The inconsistency we
> have is `ssl.protocol/ssl.enabled.protocols` since we still have insecure
> protocols enabled for Kafka. We have KIP-553 (
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956)
> open to disable these. It is currently blocked since it also talks about
> enabling TLSv1.3 by default and we haven't done sufficient testing for
> that. Since it will be good to disable the older insecure protocols anyway,
> perhaps we could do that without enabling TLSv1.3 by default in 2.5?
Hi Rajini,
Inheriting the SSL settings from Kafka would have the effect of changing the defaults for almost everyone using SSL from non-SSL ZK to SSL-based ZK, right? This seems to expand the scope of the KIP greatly, since it moves it from being opt-in to opt-out. Are there any potential negative implications of this?
best,
Colin
>
> 4) Backward compatibility for deployments which are using system
> properties. We shouldn't override system properties with Kafka defaults or
> inherited values. But we do want to override if user configures properties
> explicitly. In the current approach, this was simpler since we just had to
> set the configured values. But if we decide to inherit Kafka configs, then
> we will need to check system properties and update accordingly. But that
> should be doable?
>
> On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe <cm...@apache.org> wrote:
>
> > On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote:
> > > Hi Colin. Two things come to mind with respect to ZooKeeper camelCase
> > > style vs Kafka-style config names for ZooKeeper. First, I think it
> > > would be desirable for the client configs and broker configs to be
> > > interoperable. For example, it feels like it would be convenient to
> > > be able to pass the broker's config file to the ZK Security Migrator
> > > tool. I think whatever we use (ZooKeeper camelCase style or Kafka
> > > style), I think we should use the same for both.
> > >
> > > The second thing to think about is the fact that ZooKeeper
> > > configuration inherently uses Java system properties as a first pass.
> > > So while we might switch to Kafka-style configs in the config file
> > > (e.g. zookeeper.ssl.trust.store.location), the
> > > org.apache.zookeeper.client.ZKClientConfig class (which is how TLS
> > > configs are set) is going to look for the camelCase
> > > zookeeper.ssl.trustStore.location system property and use any value
> > > there. I think this could be very confusing for people. Granted, we
> > > are doing this so that people don't have to use system properties, but
> > > there is no way to turn off the use of system properties, so I think
> > > there would be pretty substantial potential for confusion.
> >
> > I don't think we document or recommend that anyone use system properties
> > to configure Zookeeper usage within Kafka. And I think the reason is
> > exactly the issue you pointed out -- it wouldn't be consistent with our
> > other configurations.
> >
> > best,
> > Colin
> >
> > >
> > > One idea is to migrate the system properties -- i.e. look to see if
> > > zookeeper.ssl.trustStore.location is set and if it is clear out the
> > > value there and move it under the zookeeper.ssl.trust.store.location
> > > system property. (I said it was an idea -- not necessarily a good one
> > > :-)
> > >
> > > Ron
> > >
> > > On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe <cm...@apache.org> wrote:
> > > >
> > > > On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
> > > > > Thanks Colin and Rajini.
> > > > >
> > > > > I've updated the KIP to say that KIP-421 functionality is available
> > to
> > > > > encrypt sensitive configs like the ZK key store and trust store
> > > > > passwords. (I've also made it clear that the configs are not
> > > > > dynamically reconfigurable since dynamic values are stored in ZK and
> > > > > the data is needed to get to ZK in the first place). Colin, let me
> > > > > know if you think anything else needs to be said/done about it -- I
> > > > > wasn't sure if your comment above implies that there are additional
> > > > > direct actions that we need to take aside from this.
> > > > >
> > > > > I agree that making the brokers' ZooKeeper clients dynamic with
> > > > > respect to key and trust stores (e.g. via
> > > > > zookeeper.ssl.context.supplier.class) may increase the risk that this
> > > > > KIP may not land in time for 2.5.
> > > > >
> > > > > Regarding the config names for ZK being different than the ZooKeeper
> > > > > configs (e.g. camel-case keyStore/trustStore instead of
> > > > > keystore/truststore, ciphersuites instead of cipher.suites,
> > > > > enabledProtocols instead of enabled.protocols), I agree it is an
> > > > > issue. I tried to mitigate it by putting warning text in the config
> > > > > docs for these. Regarding configuration inheritance, I think what
> > you
> > > > > are saying is that for any configs where the broker has a clear
> > > > > semantic equivalent, the broker could fall back to the broker value
> > if
> > > > > the ZK value is not given. The potential list is:
> > > > >
> > > > > zookeeper.ssl.keyStore.location => ssl.keystore.location
> > > > > zookeeper.ssl.keyStore.password => ssl.keystore.password
> > > > > zookeeper.ssl.keyStore.type => ssl.keystore.type
> > > > > zookeeper.ssl.trustStore.location => ssl.truststore.location
> > > > > zookeeper.ssl.trustStore.password => ssl.truststore.password
> > > > > zookeeper.ssl.trustStore.type => ssl.truststore.type
> > > > > zookeeper.ssl.ciphersuites => ssl.cipher.suites
> > > > >
> > > > > Note that there are two configs that look the same based on their key
> > > > > names but whose allowable values differ:
> > > > >
> > > > > zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default:
> > TLS)
> > > > > zookeeper.ssl.enabledProtocols(Default: value of protocol property) =
> > > > > ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
> > > > >
> > > > > Not sure what the right answer is, but hopefully this detail will
> > help
> > > > > get us there.
> > > > >
> > > > > Ron
> > > >
> > > > I think, on balance, I agree with Rajini that it would be nice to make
> > the configs look more like Kafka configs. We don't have camel-case in
> > configuration keys, so we should avoid it here.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > >
> > > > > On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <cm...@apache.org>
> > wrote:
> > > > > >
> > > > > > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> > > > > > > Hi Ron,
> > > > > > >
> > > > > > > Thanks for the detailed explanation, sounds good to me.
> > > > > > >
> > > > > > > A few more questions:
> > > > > > >
> > > > > > > 1) At the moment, all sensitive broker configs including
> > > > > > > keystore/truststore passwords can be stored encrypted in
> > ZooKeeper prior to
> > > > > > > starting up brokers. We are now adding SSL keystore/truststore
> > passwords
> > > > > > > for ZooKeeper client that cannot be stored in ZooKeeper since
> > you need
> > > > > > > these to connect to ZK. We should perhaps document that these
> > configs can
> > > > > > > be encrypted using KIP-421.
> > > > > >
> > > > > > That's a good point. Previously, we didn't have ZK on-the-wire
> > security support. However, now that we do, sending sensitive keystore and
> > truststore passwords to ZK in cleartext should be deprecated in favor of
> > using KIP-421.
> > > > > >
> > > > > > >
> > > > > > > 2) We can dynamically update keystores and trust stores used by
> > brokers
> > > > > > > without restarting the broker. Can we support this easily for ZK
> > clients
> > > > > > > created by the broker, for example by adding our own
> > > > > > > `zookeeper.ssl.context.supplier.class`?
> > > > > > >
> > > > > >
> > > > > > Hmm. That might be better to consider in a follow-up, since the
> > deadline for 2.5 KIPs is coming up?
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > > > 3) Looks like we are using config names that directly map to ZK
> > configs.
> > > > > > > Have we considered using equivalent Kafka config names with
> > prefixes,
> > > > > > > perhaps with inheritance from the non-prefixed names? Not sure
> > if this is a
> > > > > > > good idea, but perhaps worth documenting in Rejected
> > Alternatives at least.
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <cm...@apache.org>
> > wrote:
> > > > > > >
> > > > > > > > Hi Ron,
> > > > > > > >
> > > > > > > > Thanks for the explanation. I guess thinking about it a
> > little bit more,
> > > > > > > > we should just add --zk-tls-config-file to all of these
> > commands.
> > > > > > > >
> > > > > > > > We will be removing this option (plus ZK in general) from
> > these commands
> > > > > > > > in the next major release, but ZK is still supported in 2.5,
> > so we should
> > > > > > > > just do the logical thing. (The exception is
> > ZkSecurityMigrator, which
> > > > > > > > will stay).
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > > > > > > > Hi Colin.
> > > > > > > > >
> > > > > > > > > <<< It seems like this [--zk-tls-config-file information]
> > could just
> > > > > > > > appear
> > > > > > > > > in a configuration file, which all of these tools already
> > accept (I
> > > > > > > > think)
> > > > > > > > >
> > > > > > > > > ZkSecurityMigrator has no such property file facility;
> > adding a
> > > > > > > > > "--zk-tls-config-file" parameter is exactly for this
> > purpose. If we add
> > > > > > > > > that to ZkSecurityMigrator then it is trivial to add it to
> > other commands
> > > > > > > > > (the same code is simply reused; it ends up being just a few
> > extra
> > > > > > > > lines).
> > > > > > > > > I do not see any parameter in the other two commands to
> > adjust the ZK
> > > > > > > > > connection; ConfigCommand accepts a "--command-config" flag,
> > but
> > > > > > > > according
> > > > > > > > > to the code "This is used only with --bootstrap-server
> > option for
> > > > > > > > > describing and altering broker configs."
> > > > > > > > >
> > > > > > > > > I do agree there would be no need to add
> > "--zk-tls-config-file" to
> > > > > > > > > ReassignPartitionsCommand if its direct ZK connectivity is
> > replaced in
> > > > > > > > time
> > > > > > > > > for the next release.
> > > > > > > > >
> > > > > > > > > ConfigCommand supports the "--bootstrap-server" option and
> > will have its
> > > > > > > > > direct ZooKeeper access formally deprecated as per KIP-555,
> > but the
> > > > > > > > special
> > > > > > > > > use case of bootstrapping a ZooKeeper ensemble with
> > encrypted credentials
> > > > > > > > > prior to starting Kafka will still exist, so it feels like
> > while
> > > > > > > > > "--zk-tls-config-file" would never be used except for this
> > use case it
> > > > > > > > > could probably still be added for this particular situation.
> > > > > > > > >
> > > > > > > > > Ron
> > > > > > > > >
> > > > > > > > > P.S. I responded on 1/6 but I just discovered that e, ail
> > (and 3 more I
> > > > > > > > > sent) did not go through. I am trying to get emails through
> > now to move
> > > > > > > > > this discussion forward.
> > > > > > > > >
> > > > > > > > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <
> > cmccabe@apache.org> wrote:
> > > > > > > > >
> > > > > > > > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > > > > > > > > Hi everyone. I would like to make the following changes
> > to the KIP.
> > > > > > > > > > >
> > > > > > > > > > > MOTIVATION:
> > > > > > > > > > > Include a statement that it will be difficult in the
> > short term to
> > > > > > > > > > > deprecate direct Zookeeper communication in
> > kafka-configs.{sh, bat}
> > > > > > > > > > (which
> > > > > > > > > > > invoke kafka.admin.ConfigCommand) because bootstrapping
> > a Kafka
> > > > > > > > cluster
> > > > > > > > > > > with encrypted passwords in Zookeeper is an
> > explicitly-supported use
> > > > > > > > > > case;
> > > > > > > > > > > therefore it is in scope to be able to securely
> > configure the CLI
> > > > > > > > tools
> > > > > > > > > > > that still leverage non-deprecated direct Zookeeper
> > communication
> > > > > > > > for TLS
> > > > > > > > > > > (the other 2 tools are kafka-reassign-partitions.{sh,
> > bat} and
> > > > > > > > > > > zookeeper-security-migration.sh).
> > > > > > > > > >
> > > > > > > > > > Hi Ron,
> > > > > > > > > >
> > > > > > > > > > Thanks for the KIP.
> > > > > > > > > >
> > > > > > > > > > About deprecations:
> > > > > > > > > >
> > > > > > > > > > * zookeeper-security-migration: clearly, deprecating ZK
> > access in this
> > > > > > > > one
> > > > > > > > > > would not make sense, since it would defeat the whole
> > point of the
> > > > > > > > tool :)
> > > > > > > > > >
> > > > > > > > > > * kafka-reassign-partitions: ZK access should be
> > deprecated here.
> > > > > > > > KIP-455
> > > > > > > > > > implementation has dragged a bit, but this should get done
> > soon.
> > > > > > > > Certainly
> > > > > > > > > > before the next release.
> > > > > > > > > >
> > > > > > > > > > * kafka-configs: I think ZK access should be deprecated
> > here as well.
> > > > > > > > I
> > > > > > > > > > agree there is a super-special bootstrapping case here,
> > but that should
> > > > > > > > > > have its own tool, not use kafka-configs.
> > > > > > > > > >
> > > > > > > > > > I will post a separate KIP for this, though.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > GOALS:
> > > > > > > > > > > Support the secure configuration of TLS-encrypted
> > communication
> > > > > > > > between
> > > > > > > > > > > Zookeeper and:
> > > > > > > > > > > a) Kafka brokers
> > > > > > > > > > > b) The three CLI tools mentioned above that still
> > support direct,
> > > > > > > > > > > non-deprecated communication to Zookeeper
> > > > > > > > > > > It is explicitly out-of-scope to deprecate any direct
> > Zookeeper
> > > > > > > > > > > communication in CLI tools as part of this KIP; such
> > work will occur
> > > > > > > > in
> > > > > > > > > > > future KIPs instead.
> > > > > > > > > > >
> > > > > > > > > > > PUBLIC INTERFACES:
> > > > > > > > > > > 1) The following new broker configurations will be
> > recognized.
> > > > > > > > > > > zookeeper.client.secure (default value = false, for
> > backwards
> > > > > > > > > > > compatibility)
> > > > > > > > > > > zookeeper.clientCnxnSocket
> > > > > > > > > > > zookeeper.ssl.keyStore.location
> > > > > > > > > > > zookeeper.ssl.keyStore.password
> > > > > > > > > > > zookeeper.ssl.trustStore.location
> > > > > > > > > > > zookeeper.ssl.trustStore.password
> > > > > > > > > > > It will be an error for any of the last 5 values to be
> > left
> > > > > > > > unspecified
> > > > > > > > > > if
> > > > > > > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > > > > > > >
> > > > > > > > > > > 2) In addition, the
> > kafka.security.authorizer.AclAuthorizer class
> > > > > > > > > > supports
> > > > > > > > > > > the ability to connect to a different Zookeeper instance
> > than the
> > > > > > > > one the
> > > > > > > > > > > brokers use. We therefore also add the following
> > optional configs,
> > > > > > > > which
> > > > > > > > > > > override the corresponding ones from above when present:
> > > > > > > > > > > authorizer.zookeeper.client.secure
> > > > > > > > > > > authorizer.zookeeper.clientCnxnSocket
> > > > > > > > > > > authorizer.zookeeper.ssl.keyStore.location
> > > > > > > > > > > authorizer.zookeeper.ssl.keyStore.password
> > > > > > > > > > > authorizer.zookeeper.ssl.trustStore.location
> > > > > > > > > > > authorizer.zookeeper.ssl.trustStore.password
> > > > > > > > > > >
> > > > > > > > > > > 3) The three CLI tools mentioned above will support a new
> > > > > > > > > > --zk-tls-config-file
> > > > > > > > > > > <String: Zookeeper TLS configuration file path>"
> > option. The
> > > > > > > > following
> > > > > > > > > > > properties will be recognized in that file, and
> > unrecognized
> > > > > > > > properties
> > > > > > > > > > > will be ignored to allow the possibility of pointing
> > > > > > > > zk-tls-config-file
> > > > > > > > > > at
> > > > > > > > > > > the broker's config file.
> > > > > > > > > > > zookeeper.client.secure (default value = false)
> > > > > > > > > > > zookeeper.clientCnxnSocket
> > > > > > > > > > > zookeeper.ssl.keyStore.location
> > > > > > > > > > > zookeeper.ssl.keyStore.password
> > > > > > > > > > > zookeeper.ssl.trustStore.location
> > > > > > > > > > > zookeeper.ssl.trustStore.password
> > > > > > > > > > > It will be an error for any of the last 5 values to be
> > left
> > > > > > > > unspecified
> > > > > > > > > > if
> > > > > > > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > > > > > >
> > > > > > > > > > Do we really need a --zk-tls-config-file flag? It seems
> > like this
> > > > > > > > could
> > > > > > > > > > just appear in a configuration file, which all of these
> > tools already
> > > > > > > > > > accept (I think).
> > > > > > > > > >
> > > > > > > > > > best,
> > > > > > > > > > Colin
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Ron
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <
> > rndgstn@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi everyone. Let's get this discussion going again
> > now that Kafka
> > > > > > > > 2.4
> > > > > > > > > > > > with Zookeeper 3.5.6 is out.
> > > > > > > > > > > >
> > > > > > > > > > > > First, regarding the KIP number, the other KIP that
> > was using this
> > > > > > > > > > number
> > > > > > > > > > > > moved to KIP 534, so KIP 515 remains the correct
> > number for this
> > > > > > > > > > > > discussion. I've updated the Kafka Improvement
> > Proposal page to
> > > > > > > > list
> > > > > > > > > > this
> > > > > > > > > > > > KIP in the 515 slot, so we're all set there.
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding the actual issues under discussion, I think
> > there are
> > > > > > > > some
> > > > > > > > > > > > things we should clarify.
> > > > > > > > > > > >
> > > > > > > > > > > > 1) It is possible to use TLS connectivity to Zookeeper
> > from Apache
> > > > > > > > > > Kafka
> > > > > > > > > > > > 2.4 -- the problem is that configuration information
> > has to be
> > > > > > > > passed
> > > > > > > > > > via
> > > > > > > > > > > > system properties as "-D" command line options on the
> > java
> > > > > > > > invocation
> > > > > > > > > > of
> > > > > > > > > > > > the broker, and those are not secure (anyone with
> > access to the box
> > > > > > > > > > can see
> > > > > > > > > > > > the command line used to invoke the broker); the
> > configuration
> > > > > > > > includes
> > > > > > > > > > > > sensitive information (e.g. a keystore password), so
> > we need a
> > > > > > > > secure
> > > > > > > > > > > > mechanism for passing the configuration values. I
> > believe the real
> > > > > > > > > > > > motivation for this KIP is to harden the configuration
> > mechanism
> > > > > > > > for
> > > > > > > > > > > > Zookeeper TLS connectivity.
> > > > > > > > > > > >
> > > > > > > > > > > > 2) I believe the list of CLI tools that continue to
> > use direct
> > > > > > > > > > Zookeeper
> > > > > > > > > > > > connectivity in a non-deprecated fashion is:
> > > > > > > > > > > > a)
> > > > > > > > zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > > > > > > > > > b)
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > > > > > > > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > > > > > > > > > > >
> > > > > > > > > > > > 3) I believe Kafka.admin.ConfigCommand presents a
> > conundrum
> > > > > > > > because it
> > > > > > > > > > > > explicitly states in a comment that a supported use
> > case is
> > > > > > > > > > bootstrapping a
> > > > > > > > > > > > Kafka cluster with encrypted passwords in Zookeeper
> > (see
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > > > > > > > > > ).
> > > > > > > > > > > > This means it will be especially difficult to
> > deprecate this
> > > > > > > > particular
> > > > > > > > > > > > direct Zookeeper connectivity without a different
> > storage
> > > > > > > > mechanism for
> > > > > > > > > > > > dynamic configuration values being available (i.e. the
> > self-managed
> > > > > > > > > > quorum
> > > > > > > > > > > > referred to in KIP-500).
> > > > > > > > > > > >
> > > > > > > > > > > > I think it would be easier and simpler to harden the
> > Zookeeper TLS
> > > > > > > > > > > > configuration in both Kafka and in CLI tools compared
> > to trying to
> > > > > > > > > > > > deprecate the direct Zookeeper connectivity in the
> > above 3 CLI
> > > > > > > > tools --
> > > > > > > > > > > > especially when ConfigCommand has no obvious
> > short-term path to
> > > > > > > > > > deprecation.
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding how to harden the TLS configuration, adding
> > the
> > > > > > > > appropriate
> > > > > > > > > > > > configs to Kafka is an obvious choice for the
> > brokers. Not that
> > > > > > > > these
> > > > > > > > > > > > values cannot be dynamically reconfigurable because
> > the values
> > > > > > > > would be
> > > > > > > > > > > > required to connect to Zookeeper via TLS in the first
> > place.
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding how to harden the TLS configuration for the
> > 3 remaining
> > > > > > > > CLI
> > > > > > > > > > > > tools, it would be relatively straightforward to add
> > support for a
> > > > > > > > > > > > "--zk-tls-config-file <String: Zookeeper TLS
> > configuration system
> > > > > > > > > > > > properties file path>" option to each one.
> > > > > > > > > > > >
> > > > > > > > > > > > Thoughts?
> > > > > > > > > > > >
> > > > > > > > > > > > Ron
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <
> > > > > > > > pere.urbon@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> True, the number is duplicated.
> > > > > > > > > > > >>
> > > > > > > > > > > >> do you know how can we get the problem fix? I was
> > not aware, I
> > > > > > > > > > missed,
> > > > > > > > > > > >> sorry the need to add the KIP to the table.
> > > > > > > > > > > >>
> > > > > > > > > > > >> -- Pere
> > > > > > > > > > > >>
> > > > > > > > > > > >> Missatge de Jorge Esteban Quilcate Otoya <
> > > > > > > > quilcate.jorge@gmail.com>
> > > > > > > > > > del
> > > > > > > > > > > >> dia
> > > > > > > > > > > >> dt., 3 de set. 2019 a les 13:43:
> > > > > > > > > > > >>
> > > > > > > > > > > >> > Hi Pere,
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Have you add your KIP to the list here
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > > > > > > > >> ?
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > I found the KIP number assigned to another.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> > > > > > > > > > pere.urbon@gmail.com>
> > > > > > > > > > > >> > wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >> Thanks for your time Harsha,
> > > > > > > > > > > >> >> anyone else with comments? looking forward to
> > hearing from
> > > > > > > > you.
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> Stupid question: when do you move from discussion
> > to vote?
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io>
> > del dia
> > > > > > > > dv., 30
> > > > > > > > > > > >> d’ag.
> > > > > > > > > > > >> >> 2019 a les 21:59:
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> > Thanks Pere. KIP looks good to me.
> > > > > > > > > > > >> >> > -Harsha
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón
> > Bayes <
> > > > > > > > > > > >> >> pere.urbon@gmail.com>
> > > > > > > > > > > >> >> > wrote:
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> >> Not really,
> > > > > > > > > > > >> >> >> my idea is to keep the JAAS parameter, so
> > people don't see
> > > > > > > > > > major
> > > > > > > > > > > >> >> >> changes. But if you pass a properties file,
> > then this takes
> > > > > > > > > > > >> precedence
> > > > > > > > > > > >> >> over
> > > > > > > > > > > >> >> >> the other, with the idea that you can do sasl
> > as well with
> > > > > > > > the
> > > > > > > > > > > >> >> properties
> > > > > > > > > > > >> >> >> files.
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> Makes sense?
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> -- Pere
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> Missatge de Harsha Chintalapani <
> > kafka@harsha.io> del dia
> > > > > > > > dv.,
> > > > > > > > > > 30
> > > > > > > > > > > >> >> d’ag.
> > > > > > > > > > > >> >> >> 2019 a les 19:00:
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >>> Hi Pere,
> > > > > > > > > > > >> >> >>> Thanks for the KIP. Enabling SSL for
> > zookeeper
> > > > > > > > for
> > > > > > > > > > Kafka
> > > > > > > > > > > >> >> makes
> > > > > > > > > > > >> >> >>> sense.
> > > > > > > > > > > >> >> >>> "The changes are planned to be introduced in a
> > compatible
> > > > > > > > way,
> > > > > > > > > > by
> > > > > > > > > > > >> >> >>> keeping the current JAAS variable precedence."
> > > > > > > > > > > >> >> >>> Can you elaborate a bit here. If the user
> > configures a JAAS
> > > > > > > > > > file
> > > > > > > > > > > >> with
> > > > > > > > > > > >> >> >>> Client section it will take precedence over
> > zookeeper SSL
> > > > > > > > > > configs?
> > > > > > > > > > > >> >> >>>
> > > > > > > > > > > >> >> >>> Thanks,
> > > > > > > > > > > >> >> >>> Harsha
> > > > > > > > > > > >> >> >>>
> > > > > > > > > > > >> >> >>>
> > > > > > > > > > > >> >> >>>
> > > > > > > > > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón
> > Bayes <
> > > > > > > > > > > >> >> pere.urbon@gmail.com>
> > > > > > > > > > > >> >> >>> wrote:
> > > > > > > > > > > >> >> >>>
> > > > > > > > > > > >> >> >>>> Hi,
> > > > > > > > > > > >> >> >>>> quick question, I saw in another mail that
> > 2.4 release is
> > > > > > > > > > planned
> > > > > > > > > > > >> for
> > > > > > > > > > > >> >> >>>> September. I think it would be really awesome
> > to have
> > > > > > > > this for
> > > > > > > > > > > >> this
> > > > > > > > > > > >> >> >>>> release, do you think we can make it?
> > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > >> >> >>>> -- Pere
> > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > >> >> >>>> Missatge de Pere Urbón Bayes <
> > pere.urbon@gmail.com> del
> > > > > > > > dia
> > > > > > > > > > dj.,
> > > > > > > > > > > >> 29
> > > > > > > > > > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > >> >> >>>> Hi,
> > > > > > > > > > > >> >> >>>> this is my first KIP for a change in Apache
> > Kafka, so I'm
> > > > > > > > > > really
> > > > > > > > > > > >> need
> > > > > > > > > > > >> >> >>>> to the process. Looking forward to hearing
> > from you and
> > > > > > > > learn
> > > > > > > > > > the
> > > > > > > > > > > >> >> best
> > > > > > > > > > > >> >> >>>> ropes here.
> > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > >> >> >>>> I would like to propose this KIP-515 to
> > enable the
> > > > > > > > > > > >> ZookeeperClients
> > > > > > > > > > > >> >> to
> > > > > > > > > > > >> >> >>>> take full advantage of the TLS communication
> > in the new
> > > > > > > > > > Zookeeper
> > > > > > > > > > > >> >> 3.5.5.
> > > > > > > > > > > >> >> >>>> Specially interesting it the Zookeeper
> > Security Migration,
> > > > > > > > > > that
> > > > > > > > > > > >> >> without
> > > > > > > > > > > >> >> >>>> this change will not work with TLS, disabling
> > users to use
> > > > > > > > > > ACLs
> > > > > > > > > > > >> when
> > > > > > > > > > > >> >> the
> > > > > > > > > > > >> >> >>>> Zookeeper cluster use TLS.
> > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > >> >> >>>> link:
> > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > >> >>
> > > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > >> >> >>>> Looking forward to hearing from you on this,
> > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > >> >> >>>> /cheers
> > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > >> >> >>>> --
> > > > > > > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > > > > > > >> >> >>>> Software Architect
> > > > > > > > > > > >> >> >>>> http://www.purbon.com
> > > > > > > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > >> >> >>>> --
> > > > > > > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > > > > > > >> >> >>>> Software Architect
> > > > > > > > > > > >> >> >>>> http://www.purbon.com
> > > > > > > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > > > > > > >> >> >>>>
> > > > > > > > > > > >> >> >>>
> > > > > > > > > > > >> >> >>>
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >> --
> > > > > > > > > > > >> >> >> Pere Urbon-Bayes
> > > > > > > > > > > >> >> >> Software Architect
> > > > > > > > > > > >> >> >> http://www.purbon.com
> > > > > > > > > > > >> >> >> https://twitter.com/purbon
> > > > > > > > > > > >> >> >> https://www.linkedin.com/in/purbon/
> > > > > > > > > > > >> >> >>
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> --
> > > > > > > > > > > >> >> Pere Urbon-Bayes
> > > > > > > > > > > >> >> Software Architect
> > > > > > > > > > > >> >> http://www.purbon.com
> > > > > > > > > > > >> >> https://twitter.com/purbon
> > > > > > > > > > > >> >> https://www.linkedin.com/in/purbon/
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > > >> --
> > > > > > > > > > > >> Pere Urbon-Bayes
> > > > > > > > > > > >> Software Architect
> > > > > > > > > > > >> http://www.purbon.com
> > > > > > > > > > > >> https://twitter.com/purbon
> > > > > > > > > > > >> https://www.linkedin.com/in/purbon/
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> >
>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Ron/Colin,
Thanks for the responses. I think we need to consider these cases:
1) The recommended approach: This is the one we document. Configs
consistent with Kafka make sense here. It seems a lot more confusing to
configure `ssl.cipher.suites`, `ssl.enabled.protocols`, `
zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols` with
multiple naming inconsistencies in server.properties. For users who have
never used ZK system properties, this is clearly simpler. Another config to
consider is `ssl.endpoint.identification.algorithm` which does the same
thing as `zookeeper.ssl.hostnameVerification` - if we are staying
consistent, we should use the Kafka format for this too. I would also
consider using something like `zookeeper.client.ssl.enabled` instead of `
zookeeper.client.secure` since it is difficult to tell what `secure` means
when you have options to configure Kerberos, ACLs and/or SSL.
2) Inheritance using prefixed configs as we do for other prefixed configs
in Kafka: I think it may be useful for configs like `ssl.protocol` and `
ssl.cipher.suites`.
3) Default values: Do we want to use Kafka defaults? The inconsistency we
have is `ssl.protocol/ssl.enabled.protocols` since we still have insecure
protocols enabled for Kafka. We have KIP-553 (
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956)
open to disable these. It is currently blocked since it also talks about
enabling TLSv1.3 by default and we haven't done sufficient testing for
that. Since it will be good to disable the older insecure protocols anyway,
perhaps we could do that without enabling TLSv1.3 by default in 2.5?
4) Backward compatibility for deployments which are using system
properties. We shouldn't override system properties with Kafka defaults or
inherited values. But we do want to override if user configures properties
explicitly. In the current approach, this was simpler since we just had to
set the configured values. But if we decide to inherit Kafka configs, then
we will need to check system properties and update accordingly. But that
should be doable?
On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe <cm...@apache.org> wrote:
> On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote:
> > Hi Colin. Two things come to mind with respect to ZooKeeper camelCase
> > style vs Kafka-style config names for ZooKeeper. First, I think it
> > would be desirable for the client configs and broker configs to be
> > interoperable. For example, it feels like it would be convenient to
> > be able to pass the broker's config file to the ZK Security Migrator
> > tool. I think whatever we use (ZooKeeper camelCase style or Kafka
> > style), I think we should use the same for both.
> >
> > The second thing to think about is the fact that ZooKeeper
> > configuration inherently uses Java system properties as a first pass.
> > So while we might switch to Kafka-style configs in the config file
> > (e.g. zookeeper.ssl.trust.store.location), the
> > org.apache.zookeeper.client.ZKClientConfig class (which is how TLS
> > configs are set) is going to look for the camelCase
> > zookeeper.ssl.trustStore.location system property and use any value
> > there. I think this could be very confusing for people. Granted, we
> > are doing this so that people don't have to use system properties, but
> > there is no way to turn off the use of system properties, so I think
> > there would be pretty substantial potential for confusion.
>
> I don't think we document or recommend that anyone use system properties
> to configure Zookeeper usage within Kafka. And I think the reason is
> exactly the issue you pointed out -- it wouldn't be consistent with our
> other configurations.
>
> best,
> Colin
>
> >
> > One idea is to migrate the system properties -- i.e. look to see if
> > zookeeper.ssl.trustStore.location is set and if it is clear out the
> > value there and move it under the zookeeper.ssl.trust.store.location
> > system property. (I said it was an idea -- not necessarily a good one
> > :-)
> >
> > Ron
> >
> > On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
> > > > Thanks Colin and Rajini.
> > > >
> > > > I've updated the KIP to say that KIP-421 functionality is available
> to
> > > > encrypt sensitive configs like the ZK key store and trust store
> > > > passwords. (I've also made it clear that the configs are not
> > > > dynamically reconfigurable since dynamic values are stored in ZK and
> > > > the data is needed to get to ZK in the first place). Colin, let me
> > > > know if you think anything else needs to be said/done about it -- I
> > > > wasn't sure if your comment above implies that there are additional
> > > > direct actions that we need to take aside from this.
> > > >
> > > > I agree that making the brokers' ZooKeeper clients dynamic with
> > > > respect to key and trust stores (e.g. via
> > > > zookeeper.ssl.context.supplier.class) may increase the risk that this
> > > > KIP may not land in time for 2.5.
> > > >
> > > > Regarding the config names for ZK being different than the ZooKeeper
> > > > configs (e.g. camel-case keyStore/trustStore instead of
> > > > keystore/truststore, ciphersuites instead of cipher.suites,
> > > > enabledProtocols instead of enabled.protocols), I agree it is an
> > > > issue. I tried to mitigate it by putting warning text in the config
> > > > docs for these. Regarding configuration inheritance, I think what
> you
> > > > are saying is that for any configs where the broker has a clear
> > > > semantic equivalent, the broker could fall back to the broker value
> if
> > > > the ZK value is not given. The potential list is:
> > > >
> > > > zookeeper.ssl.keyStore.location => ssl.keystore.location
> > > > zookeeper.ssl.keyStore.password => ssl.keystore.password
> > > > zookeeper.ssl.keyStore.type => ssl.keystore.type
> > > > zookeeper.ssl.trustStore.location => ssl.truststore.location
> > > > zookeeper.ssl.trustStore.password => ssl.truststore.password
> > > > zookeeper.ssl.trustStore.type => ssl.truststore.type
> > > > zookeeper.ssl.ciphersuites => ssl.cipher.suites
> > > >
> > > > Note that there are two configs that look the same based on their key
> > > > names but whose allowable values differ:
> > > >
> > > > zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default:
> TLS)
> > > > zookeeper.ssl.enabledProtocols(Default: value of protocol property) =
> > > > ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
> > > >
> > > > Not sure what the right answer is, but hopefully this detail will
> help
> > > > get us there.
> > > >
> > > > Ron
> > >
> > > I think, on balance, I agree with Rajini that it would be nice to make
> the configs look more like Kafka configs. We don't have camel-case in
> configuration keys, so we should avoid it here.
> > >
> > > best,
> > > Colin
> > >
> > > >
> > > > On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <cm...@apache.org>
> wrote:
> > > > >
> > > > > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> > > > > > Hi Ron,
> > > > > >
> > > > > > Thanks for the detailed explanation, sounds good to me.
> > > > > >
> > > > > > A few more questions:
> > > > > >
> > > > > > 1) At the moment, all sensitive broker configs including
> > > > > > keystore/truststore passwords can be stored encrypted in
> ZooKeeper prior to
> > > > > > starting up brokers. We are now adding SSL keystore/truststore
> passwords
> > > > > > for ZooKeeper client that cannot be stored in ZooKeeper since
> you need
> > > > > > these to connect to ZK. We should perhaps document that these
> configs can
> > > > > > be encrypted using KIP-421.
> > > > >
> > > > > That's a good point. Previously, we didn't have ZK on-the-wire
> security support. However, now that we do, sending sensitive keystore and
> truststore passwords to ZK in cleartext should be deprecated in favor of
> using KIP-421.
> > > > >
> > > > > >
> > > > > > 2) We can dynamically update keystores and trust stores used by
> brokers
> > > > > > without restarting the broker. Can we support this easily for ZK
> clients
> > > > > > created by the broker, for example by adding our own
> > > > > > `zookeeper.ssl.context.supplier.class`?
> > > > > >
> > > > >
> > > > > Hmm. That might be better to consider in a follow-up, since the
> deadline for 2.5 KIPs is coming up?
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > > 3) Looks like we are using config names that directly map to ZK
> configs.
> > > > > > Have we considered using equivalent Kafka config names with
> prefixes,
> > > > > > perhaps with inheritance from the non-prefixed names? Not sure
> if this is a
> > > > > > good idea, but perhaps worth documenting in Rejected
> Alternatives at least.
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <cm...@apache.org>
> wrote:
> > > > > >
> > > > > > > Hi Ron,
> > > > > > >
> > > > > > > Thanks for the explanation. I guess thinking about it a
> little bit more,
> > > > > > > we should just add --zk-tls-config-file to all of these
> commands.
> > > > > > >
> > > > > > > We will be removing this option (plus ZK in general) from
> these commands
> > > > > > > in the next major release, but ZK is still supported in 2.5,
> so we should
> > > > > > > just do the logical thing. (The exception is
> ZkSecurityMigrator, which
> > > > > > > will stay).
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > > > > > > Hi Colin.
> > > > > > > >
> > > > > > > > <<< It seems like this [--zk-tls-config-file information]
> could just
> > > > > > > appear
> > > > > > > > in a configuration file, which all of these tools already
> accept (I
> > > > > > > think)
> > > > > > > >
> > > > > > > > ZkSecurityMigrator has no such property file facility;
> adding a
> > > > > > > > "--zk-tls-config-file" parameter is exactly for this
> purpose. If we add
> > > > > > > > that to ZkSecurityMigrator then it is trivial to add it to
> other commands
> > > > > > > > (the same code is simply reused; it ends up being just a few
> extra
> > > > > > > lines).
> > > > > > > > I do not see any parameter in the other two commands to
> adjust the ZK
> > > > > > > > connection; ConfigCommand accepts a "--command-config" flag,
> but
> > > > > > > according
> > > > > > > > to the code "This is used only with --bootstrap-server
> option for
> > > > > > > > describing and altering broker configs."
> > > > > > > >
> > > > > > > > I do agree there would be no need to add
> "--zk-tls-config-file" to
> > > > > > > > ReassignPartitionsCommand if its direct ZK connectivity is
> replaced in
> > > > > > > time
> > > > > > > > for the next release.
> > > > > > > >
> > > > > > > > ConfigCommand supports the "--bootstrap-server" option and
> will have its
> > > > > > > > direct ZooKeeper access formally deprecated as per KIP-555,
> but the
> > > > > > > special
> > > > > > > > use case of bootstrapping a ZooKeeper ensemble with
> encrypted credentials
> > > > > > > > prior to starting Kafka will still exist, so it feels like
> while
> > > > > > > > "--zk-tls-config-file" would never be used except for this
> use case it
> > > > > > > > could probably still be added for this particular situation.
> > > > > > > >
> > > > > > > > Ron
> > > > > > > >
> > > > > > > > P.S. I responded on 1/6 but I just discovered that e, ail
> (and 3 more I
> > > > > > > > sent) did not go through. I am trying to get emails through
> now to move
> > > > > > > > this discussion forward.
> > > > > > > >
> > > > > > > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <
> cmccabe@apache.org> wrote:
> > > > > > > >
> > > > > > > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > > > > > > > Hi everyone. I would like to make the following changes
> to the KIP.
> > > > > > > > > >
> > > > > > > > > > MOTIVATION:
> > > > > > > > > > Include a statement that it will be difficult in the
> short term to
> > > > > > > > > > deprecate direct Zookeeper communication in
> kafka-configs.{sh, bat}
> > > > > > > > > (which
> > > > > > > > > > invoke kafka.admin.ConfigCommand) because bootstrapping
> a Kafka
> > > > > > > cluster
> > > > > > > > > > with encrypted passwords in Zookeeper is an
> explicitly-supported use
> > > > > > > > > case;
> > > > > > > > > > therefore it is in scope to be able to securely
> configure the CLI
> > > > > > > tools
> > > > > > > > > > that still leverage non-deprecated direct Zookeeper
> communication
> > > > > > > for TLS
> > > > > > > > > > (the other 2 tools are kafka-reassign-partitions.{sh,
> bat} and
> > > > > > > > > > zookeeper-security-migration.sh).
> > > > > > > > >
> > > > > > > > > Hi Ron,
> > > > > > > > >
> > > > > > > > > Thanks for the KIP.
> > > > > > > > >
> > > > > > > > > About deprecations:
> > > > > > > > >
> > > > > > > > > * zookeeper-security-migration: clearly, deprecating ZK
> access in this
> > > > > > > one
> > > > > > > > > would not make sense, since it would defeat the whole
> point of the
> > > > > > > tool :)
> > > > > > > > >
> > > > > > > > > * kafka-reassign-partitions: ZK access should be
> deprecated here.
> > > > > > > KIP-455
> > > > > > > > > implementation has dragged a bit, but this should get done
> soon.
> > > > > > > Certainly
> > > > > > > > > before the next release.
> > > > > > > > >
> > > > > > > > > * kafka-configs: I think ZK access should be deprecated
> here as well.
> > > > > > > I
> > > > > > > > > agree there is a super-special bootstrapping case here,
> but that should
> > > > > > > > > have its own tool, not use kafka-configs.
> > > > > > > > >
> > > > > > > > > I will post a separate KIP for this, though.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > GOALS:
> > > > > > > > > > Support the secure configuration of TLS-encrypted
> communication
> > > > > > > between
> > > > > > > > > > Zookeeper and:
> > > > > > > > > > a) Kafka brokers
> > > > > > > > > > b) The three CLI tools mentioned above that still
> support direct,
> > > > > > > > > > non-deprecated communication to Zookeeper
> > > > > > > > > > It is explicitly out-of-scope to deprecate any direct
> Zookeeper
> > > > > > > > > > communication in CLI tools as part of this KIP; such
> work will occur
> > > > > > > in
> > > > > > > > > > future KIPs instead.
> > > > > > > > > >
> > > > > > > > > > PUBLIC INTERFACES:
> > > > > > > > > > 1) The following new broker configurations will be
> recognized.
> > > > > > > > > > zookeeper.client.secure (default value = false, for
> backwards
> > > > > > > > > > compatibility)
> > > > > > > > > > zookeeper.clientCnxnSocket
> > > > > > > > > > zookeeper.ssl.keyStore.location
> > > > > > > > > > zookeeper.ssl.keyStore.password
> > > > > > > > > > zookeeper.ssl.trustStore.location
> > > > > > > > > > zookeeper.ssl.trustStore.password
> > > > > > > > > > It will be an error for any of the last 5 values to be
> left
> > > > > > > unspecified
> > > > > > > > > if
> > > > > > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > > > > > >
> > > > > > > > > > 2) In addition, the
> kafka.security.authorizer.AclAuthorizer class
> > > > > > > > > supports
> > > > > > > > > > the ability to connect to a different Zookeeper instance
> than the
> > > > > > > one the
> > > > > > > > > > brokers use. We therefore also add the following
> optional configs,
> > > > > > > which
> > > > > > > > > > override the corresponding ones from above when present:
> > > > > > > > > > authorizer.zookeeper.client.secure
> > > > > > > > > > authorizer.zookeeper.clientCnxnSocket
> > > > > > > > > > authorizer.zookeeper.ssl.keyStore.location
> > > > > > > > > > authorizer.zookeeper.ssl.keyStore.password
> > > > > > > > > > authorizer.zookeeper.ssl.trustStore.location
> > > > > > > > > > authorizer.zookeeper.ssl.trustStore.password
> > > > > > > > > >
> > > > > > > > > > 3) The three CLI tools mentioned above will support a new
> > > > > > > > > --zk-tls-config-file
> > > > > > > > > > <String: Zookeeper TLS configuration file path>"
> option. The
> > > > > > > following
> > > > > > > > > > properties will be recognized in that file, and
> unrecognized
> > > > > > > properties
> > > > > > > > > > will be ignored to allow the possibility of pointing
> > > > > > > zk-tls-config-file
> > > > > > > > > at
> > > > > > > > > > the broker's config file.
> > > > > > > > > > zookeeper.client.secure (default value = false)
> > > > > > > > > > zookeeper.clientCnxnSocket
> > > > > > > > > > zookeeper.ssl.keyStore.location
> > > > > > > > > > zookeeper.ssl.keyStore.password
> > > > > > > > > > zookeeper.ssl.trustStore.location
> > > > > > > > > > zookeeper.ssl.trustStore.password
> > > > > > > > > > It will be an error for any of the last 5 values to be
> left
> > > > > > > unspecified
> > > > > > > > > if
> > > > > > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > > > > >
> > > > > > > > > Do we really need a --zk-tls-config-file flag? It seems
> like this
> > > > > > > could
> > > > > > > > > just appear in a configuration file, which all of these
> tools already
> > > > > > > > > accept (I think).
> > > > > > > > >
> > > > > > > > > best,
> > > > > > > > > Colin
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Ron
> > > > > > > > > >
> > > > > > > > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <
> rndgstn@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi everyone. Let's get this discussion going again
> now that Kafka
> > > > > > > 2.4
> > > > > > > > > > > with Zookeeper 3.5.6 is out.
> > > > > > > > > > >
> > > > > > > > > > > First, regarding the KIP number, the other KIP that
> was using this
> > > > > > > > > number
> > > > > > > > > > > moved to KIP 534, so KIP 515 remains the correct
> number for this
> > > > > > > > > > > discussion. I've updated the Kafka Improvement
> Proposal page to
> > > > > > > list
> > > > > > > > > this
> > > > > > > > > > > KIP in the 515 slot, so we're all set there.
> > > > > > > > > > >
> > > > > > > > > > > Regarding the actual issues under discussion, I think
> there are
> > > > > > > some
> > > > > > > > > > > things we should clarify.
> > > > > > > > > > >
> > > > > > > > > > > 1) It is possible to use TLS connectivity to Zookeeper
> from Apache
> > > > > > > > > Kafka
> > > > > > > > > > > 2.4 -- the problem is that configuration information
> has to be
> > > > > > > passed
> > > > > > > > > via
> > > > > > > > > > > system properties as "-D" command line options on the
> java
> > > > > > > invocation
> > > > > > > > > of
> > > > > > > > > > > the broker, and those are not secure (anyone with
> access to the box
> > > > > > > > > can see
> > > > > > > > > > > the command line used to invoke the broker); the
> configuration
> > > > > > > includes
> > > > > > > > > > > sensitive information (e.g. a keystore password), so
> we need a
> > > > > > > secure
> > > > > > > > > > > mechanism for passing the configuration values. I
> believe the real
> > > > > > > > > > > motivation for this KIP is to harden the configuration
> mechanism
> > > > > > > for
> > > > > > > > > > > Zookeeper TLS connectivity.
> > > > > > > > > > >
> > > > > > > > > > > 2) I believe the list of CLI tools that continue to
> use direct
> > > > > > > > > Zookeeper
> > > > > > > > > > > connectivity in a non-deprecated fashion is:
> > > > > > > > > > > a)
> > > > > > > zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > > > > > > > > b)
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > > > > > > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > > > > > > > > > >
> > > > > > > > > > > 3) I believe Kafka.admin.ConfigCommand presents a
> conundrum
> > > > > > > because it
> > > > > > > > > > > explicitly states in a comment that a supported use
> case is
> > > > > > > > > bootstrapping a
> > > > > > > > > > > Kafka cluster with encrypted passwords in Zookeeper
> (see
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > > > > > > > > ).
> > > > > > > > > > > This means it will be especially difficult to
> deprecate this
> > > > > > > particular
> > > > > > > > > > > direct Zookeeper connectivity without a different
> storage
> > > > > > > mechanism for
> > > > > > > > > > > dynamic configuration values being available (i.e. the
> self-managed
> > > > > > > > > quorum
> > > > > > > > > > > referred to in KIP-500).
> > > > > > > > > > >
> > > > > > > > > > > I think it would be easier and simpler to harden the
> Zookeeper TLS
> > > > > > > > > > > configuration in both Kafka and in CLI tools compared
> to trying to
> > > > > > > > > > > deprecate the direct Zookeeper connectivity in the
> above 3 CLI
> > > > > > > tools --
> > > > > > > > > > > especially when ConfigCommand has no obvious
> short-term path to
> > > > > > > > > deprecation.
> > > > > > > > > > >
> > > > > > > > > > > Regarding how to harden the TLS configuration, adding
> the
> > > > > > > appropriate
> > > > > > > > > > > configs to Kafka is an obvious choice for the
> brokers. Not that
> > > > > > > these
> > > > > > > > > > > values cannot be dynamically reconfigurable because
> the values
> > > > > > > would be
> > > > > > > > > > > required to connect to Zookeeper via TLS in the first
> place.
> > > > > > > > > > >
> > > > > > > > > > > Regarding how to harden the TLS configuration for the
> 3 remaining
> > > > > > > CLI
> > > > > > > > > > > tools, it would be relatively straightforward to add
> support for a
> > > > > > > > > > > "--zk-tls-config-file <String: Zookeeper TLS
> configuration system
> > > > > > > > > > > properties file path>" option to each one.
> > > > > > > > > > >
> > > > > > > > > > > Thoughts?
> > > > > > > > > > >
> > > > > > > > > > > Ron
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <
> > > > > > > pere.urbon@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> True, the number is duplicated.
> > > > > > > > > > >>
> > > > > > > > > > >> do you know how can we get the problem fix? I was
> not aware, I
> > > > > > > > > missed,
> > > > > > > > > > >> sorry the need to add the KIP to the table.
> > > > > > > > > > >>
> > > > > > > > > > >> -- Pere
> > > > > > > > > > >>
> > > > > > > > > > >> Missatge de Jorge Esteban Quilcate Otoya <
> > > > > > > quilcate.jorge@gmail.com>
> > > > > > > > > del
> > > > > > > > > > >> dia
> > > > > > > > > > >> dt., 3 de set. 2019 a les 13:43:
> > > > > > > > > > >>
> > > > > > > > > > >> > Hi Pere,
> > > > > > > > > > >> >
> > > > > > > > > > >> > Have you add your KIP to the list here
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > >
> > > > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > > > > > > >> ?
> > > > > > > > > > >> >
> > > > > > > > > > >> > I found the KIP number assigned to another.
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> > > > > > > > > pere.urbon@gmail.com>
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> >> Thanks for your time Harsha,
> > > > > > > > > > >> >> anyone else with comments? looking forward to
> hearing from
> > > > > > > you.
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> Stupid question: when do you move from discussion
> to vote?
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io>
> del dia
> > > > > > > dv., 30
> > > > > > > > > > >> d’ag.
> > > > > > > > > > >> >> 2019 a les 21:59:
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> > Thanks Pere. KIP looks good to me.
> > > > > > > > > > >> >> > -Harsha
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón
> Bayes <
> > > > > > > > > > >> >> pere.urbon@gmail.com>
> > > > > > > > > > >> >> > wrote:
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >> Not really,
> > > > > > > > > > >> >> >> my idea is to keep the JAAS parameter, so
> people don't see
> > > > > > > > > major
> > > > > > > > > > >> >> >> changes. But if you pass a properties file,
> then this takes
> > > > > > > > > > >> precedence
> > > > > > > > > > >> >> over
> > > > > > > > > > >> >> >> the other, with the idea that you can do sasl
> as well with
> > > > > > > the
> > > > > > > > > > >> >> properties
> > > > > > > > > > >> >> >> files.
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> Makes sense?
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> -- Pere
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> Missatge de Harsha Chintalapani <
> kafka@harsha.io> del dia
> > > > > > > dv.,
> > > > > > > > > 30
> > > > > > > > > > >> >> d’ag.
> > > > > > > > > > >> >> >> 2019 a les 19:00:
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >>> Hi Pere,
> > > > > > > > > > >> >> >>> Thanks for the KIP. Enabling SSL for
> zookeeper
> > > > > > > for
> > > > > > > > > Kafka
> > > > > > > > > > >> >> makes
> > > > > > > > > > >> >> >>> sense.
> > > > > > > > > > >> >> >>> "The changes are planned to be introduced in a
> compatible
> > > > > > > way,
> > > > > > > > > by
> > > > > > > > > > >> >> >>> keeping the current JAAS variable precedence."
> > > > > > > > > > >> >> >>> Can you elaborate a bit here. If the user
> configures a JAAS
> > > > > > > > > file
> > > > > > > > > > >> with
> > > > > > > > > > >> >> >>> Client section it will take precedence over
> zookeeper SSL
> > > > > > > > > configs?
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>> Thanks,
> > > > > > > > > > >> >> >>> Harsha
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón
> Bayes <
> > > > > > > > > > >> >> pere.urbon@gmail.com>
> > > > > > > > > > >> >> >>> wrote:
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>>> Hi,
> > > > > > > > > > >> >> >>>> quick question, I saw in another mail that
> 2.4 release is
> > > > > > > > > planned
> > > > > > > > > > >> for
> > > > > > > > > > >> >> >>>> September. I think it would be really awesome
> to have
> > > > > > > this for
> > > > > > > > > > >> this
> > > > > > > > > > >> >> >>>> release, do you think we can make it?
> > > > > > > > > > >> >> >>>>
> > > > > > > > > > >> >> >>>> -- Pere
> > > > > > > > > > >> >> >>>>
> > > > > > > > > > >> >> >>>> Missatge de Pere Urbón Bayes <
> pere.urbon@gmail.com> del
> > > > > > > dia
> > > > > > > > > dj.,
> > > > > > > > > > >> 29
> > > > > > > > > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > > > > > > > > >> >> >>>>
> > > > > > > > > > >> >> >>>> Hi,
> > > > > > > > > > >> >> >>>> this is my first KIP for a change in Apache
> Kafka, so I'm
> > > > > > > > > really
> > > > > > > > > > >> need
> > > > > > > > > > >> >> >>>> to the process. Looking forward to hearing
> from you and
> > > > > > > learn
> > > > > > > > > the
> > > > > > > > > > >> >> best
> > > > > > > > > > >> >> >>>> ropes here.
> > > > > > > > > > >> >> >>>>
> > > > > > > > > > >> >> >>>> I would like to propose this KIP-515 to
> enable the
> > > > > > > > > > >> ZookeeperClients
> > > > > > > > > > >> >> to
> > > > > > > > > > >> >> >>>> take full advantage of the TLS communication
> in the new
> > > > > > > > > Zookeeper
> > > > > > > > > > >> >> 3.5.5.
> > > > > > > > > > >> >> >>>> Specially interesting it the Zookeeper
> Security Migration,
> > > > > > > > > that
> > > > > > > > > > >> >> without
> > > > > > > > > > >> >> >>>> this change will not work with TLS, disabling
> users to use
> > > > > > > > > ACLs
> > > > > > > > > > >> when
> > > > > > > > > > >> >> the
> > > > > > > > > > >> >> >>>> Zookeeper cluster use TLS.
> > > > > > > > > > >> >> >>>>
> > > > > > > > > > >> >> >>>> link:
> > > > > > > > > > >> >> >>>>
> > > > > > > > > > >> >> >>>>
> > > > > > > > > > >> >>
> > > > > > > > > > >>
> > > > > > > > >
> > > > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > > > > > > > > >> >> >>>>
> > > > > > > > > > >> >> >>>> Looking forward to hearing from you on this,
> > > > > > > > > > >> >> >>>>
> > > > > > > > > > >> >> >>>> /cheers
> > > > > > > > > > >> >> >>>>
> > > > > > > > > > >> >> >>>> --
> > > > > > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > > > > > >> >> >>>> Software Architect
> > > > > > > > > > >> >> >>>> http://www.purbon.com
> > > > > > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > > > > > >> >> >>>>
> > > > > > > > > > >> >> >>>> --
> > > > > > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > > > > > >> >> >>>> Software Architect
> > > > > > > > > > >> >> >>>> http://www.purbon.com
> > > > > > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > > > > > >> >> >>>>
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> --
> > > > > > > > > > >> >> >> Pere Urbon-Bayes
> > > > > > > > > > >> >> >> Software Architect
> > > > > > > > > > >> >> >> http://www.purbon.com
> > > > > > > > > > >> >> >> https://twitter.com/purbon
> > > > > > > > > > >> >> >> https://www.linkedin.com/in/purbon/
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> --
> > > > > > > > > > >> >> Pere Urbon-Bayes
> > > > > > > > > > >> >> Software Architect
> > > > > > > > > > >> >> http://www.purbon.com
> > > > > > > > > > >> >> https://twitter.com/purbon
> > > > > > > > > > >> >> https://www.linkedin.com/in/purbon/
> > > > > > > > > > >> >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >> --
> > > > > > > > > > >> Pere Urbon-Bayes
> > > > > > > > > > >> Software Architect
> > > > > > > > > > >> http://www.purbon.com
> > > > > > > > > > >> https://twitter.com/purbon
> > > > > > > > > > >> https://www.linkedin.com/in/purbon/
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Colin McCabe <cm...@apache.org>.
On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote:
> Hi Colin. Two things come to mind with respect to ZooKeeper camelCase
> style vs Kafka-style config names for ZooKeeper. First, I think it
> would be desirable for the client configs and broker configs to be
> interoperable. For example, it feels like it would be convenient to
> be able to pass the broker's config file to the ZK Security Migrator
> tool. I think whatever we use (ZooKeeper camelCase style or Kafka
> style), I think we should use the same for both.
>
> The second thing to think about is the fact that ZooKeeper
> configuration inherently uses Java system properties as a first pass.
> So while we might switch to Kafka-style configs in the config file
> (e.g. zookeeper.ssl.trust.store.location), the
> org.apache.zookeeper.client.ZKClientConfig class (which is how TLS
> configs are set) is going to look for the camelCase
> zookeeper.ssl.trustStore.location system property and use any value
> there. I think this could be very confusing for people. Granted, we
> are doing this so that people don't have to use system properties, but
> there is no way to turn off the use of system properties, so I think
> there would be pretty substantial potential for confusion.
I don't think we document or recommend that anyone use system properties to configure Zookeeper usage within Kafka. And I think the reason is exactly the issue you pointed out -- it wouldn't be consistent with our other configurations.
best,
Colin
>
> One idea is to migrate the system properties -- i.e. look to see if
> zookeeper.ssl.trustStore.location is set and if it is clear out the
> value there and move it under the zookeeper.ssl.trust.store.location
> system property. (I said it was an idea -- not necessarily a good one
> :-)
>
> Ron
>
> On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
> > > Thanks Colin and Rajini.
> > >
> > > I've updated the KIP to say that KIP-421 functionality is available to
> > > encrypt sensitive configs like the ZK key store and trust store
> > > passwords. (I've also made it clear that the configs are not
> > > dynamically reconfigurable since dynamic values are stored in ZK and
> > > the data is needed to get to ZK in the first place). Colin, let me
> > > know if you think anything else needs to be said/done about it -- I
> > > wasn't sure if your comment above implies that there are additional
> > > direct actions that we need to take aside from this.
> > >
> > > I agree that making the brokers' ZooKeeper clients dynamic with
> > > respect to key and trust stores (e.g. via
> > > zookeeper.ssl.context.supplier.class) may increase the risk that this
> > > KIP may not land in time for 2.5.
> > >
> > > Regarding the config names for ZK being different than the ZooKeeper
> > > configs (e.g. camel-case keyStore/trustStore instead of
> > > keystore/truststore, ciphersuites instead of cipher.suites,
> > > enabledProtocols instead of enabled.protocols), I agree it is an
> > > issue. I tried to mitigate it by putting warning text in the config
> > > docs for these. Regarding configuration inheritance, I think what you
> > > are saying is that for any configs where the broker has a clear
> > > semantic equivalent, the broker could fall back to the broker value if
> > > the ZK value is not given. The potential list is:
> > >
> > > zookeeper.ssl.keyStore.location => ssl.keystore.location
> > > zookeeper.ssl.keyStore.password => ssl.keystore.password
> > > zookeeper.ssl.keyStore.type => ssl.keystore.type
> > > zookeeper.ssl.trustStore.location => ssl.truststore.location
> > > zookeeper.ssl.trustStore.password => ssl.truststore.password
> > > zookeeper.ssl.trustStore.type => ssl.truststore.type
> > > zookeeper.ssl.ciphersuites => ssl.cipher.suites
> > >
> > > Note that there are two configs that look the same based on their key
> > > names but whose allowable values differ:
> > >
> > > zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default: TLS)
> > > zookeeper.ssl.enabledProtocols(Default: value of protocol property) =
> > > ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
> > >
> > > Not sure what the right answer is, but hopefully this detail will help
> > > get us there.
> > >
> > > Ron
> >
> > I think, on balance, I agree with Rajini that it would be nice to make the configs look more like Kafka configs. We don't have camel-case in configuration keys, so we should avoid it here.
> >
> > best,
> > Colin
> >
> > >
> > > On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <cm...@apache.org> wrote:
> > > >
> > > > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> > > > > Hi Ron,
> > > > >
> > > > > Thanks for the detailed explanation, sounds good to me.
> > > > >
> > > > > A few more questions:
> > > > >
> > > > > 1) At the moment, all sensitive broker configs including
> > > > > keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
> > > > > starting up brokers. We are now adding SSL keystore/truststore passwords
> > > > > for ZooKeeper client that cannot be stored in ZooKeeper since you need
> > > > > these to connect to ZK. We should perhaps document that these configs can
> > > > > be encrypted using KIP-421.
> > > >
> > > > That's a good point. Previously, we didn't have ZK on-the-wire security support. However, now that we do, sending sensitive keystore and truststore passwords to ZK in cleartext should be deprecated in favor of using KIP-421.
> > > >
> > > > >
> > > > > 2) We can dynamically update keystores and trust stores used by brokers
> > > > > without restarting the broker. Can we support this easily for ZK clients
> > > > > created by the broker, for example by adding our own
> > > > > `zookeeper.ssl.context.supplier.class`?
> > > > >
> > > >
> > > > Hmm. That might be better to consider in a follow-up, since the deadline for 2.5 KIPs is coming up?
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > > 3) Looks like we are using config names that directly map to ZK configs.
> > > > > Have we considered using equivalent Kafka config names with prefixes,
> > > > > perhaps with inheritance from the non-prefixed names? Not sure if this is a
> > > > > good idea, but perhaps worth documenting in Rejected Alternatives at least.
> > > > >
> > > > >
> > > > > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <cm...@apache.org> wrote:
> > > > >
> > > > > > Hi Ron,
> > > > > >
> > > > > > Thanks for the explanation. I guess thinking about it a little bit more,
> > > > > > we should just add --zk-tls-config-file to all of these commands.
> > > > > >
> > > > > > We will be removing this option (plus ZK in general) from these commands
> > > > > > in the next major release, but ZK is still supported in 2.5, so we should
> > > > > > just do the logical thing. (The exception is ZkSecurityMigrator, which
> > > > > > will stay).
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > > > > > Hi Colin.
> > > > > > >
> > > > > > > <<< It seems like this [--zk-tls-config-file information] could just
> > > > > > appear
> > > > > > > in a configuration file, which all of these tools already accept (I
> > > > > > think)
> > > > > > >
> > > > > > > ZkSecurityMigrator has no such property file facility; adding a
> > > > > > > "--zk-tls-config-file" parameter is exactly for this purpose. If we add
> > > > > > > that to ZkSecurityMigrator then it is trivial to add it to other commands
> > > > > > > (the same code is simply reused; it ends up being just a few extra
> > > > > > lines).
> > > > > > > I do not see any parameter in the other two commands to adjust the ZK
> > > > > > > connection; ConfigCommand accepts a "--command-config" flag, but
> > > > > > according
> > > > > > > to the code "This is used only with --bootstrap-server option for
> > > > > > > describing and altering broker configs."
> > > > > > >
> > > > > > > I do agree there would be no need to add "--zk-tls-config-file" to
> > > > > > > ReassignPartitionsCommand if its direct ZK connectivity is replaced in
> > > > > > time
> > > > > > > for the next release.
> > > > > > >
> > > > > > > ConfigCommand supports the "--bootstrap-server" option and will have its
> > > > > > > direct ZooKeeper access formally deprecated as per KIP-555, but the
> > > > > > special
> > > > > > > use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> > > > > > > prior to starting Kafka will still exist, so it feels like while
> > > > > > > "--zk-tls-config-file" would never be used except for this use case it
> > > > > > > could probably still be added for this particular situation.
> > > > > > >
> > > > > > > Ron
> > > > > > >
> > > > > > > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> > > > > > > sent) did not go through. I am trying to get emails through now to move
> > > > > > > this discussion forward.
> > > > > > >
> > > > > > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cm...@apache.org> wrote:
> > > > > > >
> > > > > > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > > > > > > Hi everyone. I would like to make the following changes to the KIP.
> > > > > > > > >
> > > > > > > > > MOTIVATION:
> > > > > > > > > Include a statement that it will be difficult in the short term to
> > > > > > > > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > > > > > > > (which
> > > > > > > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
> > > > > > cluster
> > > > > > > > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > > > > > > > case;
> > > > > > > > > therefore it is in scope to be able to securely configure the CLI
> > > > > > tools
> > > > > > > > > that still leverage non-deprecated direct Zookeeper communication
> > > > > > for TLS
> > > > > > > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > > > > > > > zookeeper-security-migration.sh).
> > > > > > > >
> > > > > > > > Hi Ron,
> > > > > > > >
> > > > > > > > Thanks for the KIP.
> > > > > > > >
> > > > > > > > About deprecations:
> > > > > > > >
> > > > > > > > * zookeeper-security-migration: clearly, deprecating ZK access in this
> > > > > > one
> > > > > > > > would not make sense, since it would defeat the whole point of the
> > > > > > tool :)
> > > > > > > >
> > > > > > > > * kafka-reassign-partitions: ZK access should be deprecated here.
> > > > > > KIP-455
> > > > > > > > implementation has dragged a bit, but this should get done soon.
> > > > > > Certainly
> > > > > > > > before the next release.
> > > > > > > >
> > > > > > > > * kafka-configs: I think ZK access should be deprecated here as well.
> > > > > > I
> > > > > > > > agree there is a super-special bootstrapping case here, but that should
> > > > > > > > have its own tool, not use kafka-configs.
> > > > > > > >
> > > > > > > > I will post a separate KIP for this, though.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > GOALS:
> > > > > > > > > Support the secure configuration of TLS-encrypted communication
> > > > > > between
> > > > > > > > > Zookeeper and:
> > > > > > > > > a) Kafka brokers
> > > > > > > > > b) The three CLI tools mentioned above that still support direct,
> > > > > > > > > non-deprecated communication to Zookeeper
> > > > > > > > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > > > > > > > communication in CLI tools as part of this KIP; such work will occur
> > > > > > in
> > > > > > > > > future KIPs instead.
> > > > > > > > >
> > > > > > > > > PUBLIC INTERFACES:
> > > > > > > > > 1) The following new broker configurations will be recognized.
> > > > > > > > > zookeeper.client.secure (default value = false, for backwards
> > > > > > > > > compatibility)
> > > > > > > > > zookeeper.clientCnxnSocket
> > > > > > > > > zookeeper.ssl.keyStore.location
> > > > > > > > > zookeeper.ssl.keyStore.password
> > > > > > > > > zookeeper.ssl.trustStore.location
> > > > > > > > > zookeeper.ssl.trustStore.password
> > > > > > > > > It will be an error for any of the last 5 values to be left
> > > > > > unspecified
> > > > > > > > if
> > > > > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > > > > >
> > > > > > > > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> > > > > > > > supports
> > > > > > > > > the ability to connect to a different Zookeeper instance than the
> > > > > > one the
> > > > > > > > > brokers use. We therefore also add the following optional configs,
> > > > > > which
> > > > > > > > > override the corresponding ones from above when present:
> > > > > > > > > authorizer.zookeeper.client.secure
> > > > > > > > > authorizer.zookeeper.clientCnxnSocket
> > > > > > > > > authorizer.zookeeper.ssl.keyStore.location
> > > > > > > > > authorizer.zookeeper.ssl.keyStore.password
> > > > > > > > > authorizer.zookeeper.ssl.trustStore.location
> > > > > > > > > authorizer.zookeeper.ssl.trustStore.password
> > > > > > > > >
> > > > > > > > > 3) The three CLI tools mentioned above will support a new
> > > > > > > > --zk-tls-config-file
> > > > > > > > > <String: Zookeeper TLS configuration file path>" option. The
> > > > > > following
> > > > > > > > > properties will be recognized in that file, and unrecognized
> > > > > > properties
> > > > > > > > > will be ignored to allow the possibility of pointing
> > > > > > zk-tls-config-file
> > > > > > > > at
> > > > > > > > > the broker's config file.
> > > > > > > > > zookeeper.client.secure (default value = false)
> > > > > > > > > zookeeper.clientCnxnSocket
> > > > > > > > > zookeeper.ssl.keyStore.location
> > > > > > > > > zookeeper.ssl.keyStore.password
> > > > > > > > > zookeeper.ssl.trustStore.location
> > > > > > > > > zookeeper.ssl.trustStore.password
> > > > > > > > > It will be an error for any of the last 5 values to be left
> > > > > > unspecified
> > > > > > > > if
> > > > > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > > > >
> > > > > > > > Do we really need a --zk-tls-config-file flag? It seems like this
> > > > > > could
> > > > > > > > just appear in a configuration file, which all of these tools already
> > > > > > > > accept (I think).
> > > > > > > >
> > > > > > > > best,
> > > > > > > > Colin
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Ron
> > > > > > > > >
> > > > > > > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rn...@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi everyone. Let's get this discussion going again now that Kafka
> > > > > > 2.4
> > > > > > > > > > with Zookeeper 3.5.6 is out.
> > > > > > > > > >
> > > > > > > > > > First, regarding the KIP number, the other KIP that was using this
> > > > > > > > number
> > > > > > > > > > moved to KIP 534, so KIP 515 remains the correct number for this
> > > > > > > > > > discussion. I've updated the Kafka Improvement Proposal page to
> > > > > > list
> > > > > > > > this
> > > > > > > > > > KIP in the 515 slot, so we're all set there.
> > > > > > > > > >
> > > > > > > > > > Regarding the actual issues under discussion, I think there are
> > > > > > some
> > > > > > > > > > things we should clarify.
> > > > > > > > > >
> > > > > > > > > > 1) It is possible to use TLS connectivity to Zookeeper from Apache
> > > > > > > > Kafka
> > > > > > > > > > 2.4 -- the problem is that configuration information has to be
> > > > > > passed
> > > > > > > > via
> > > > > > > > > > system properties as "-D" command line options on the java
> > > > > > invocation
> > > > > > > > of
> > > > > > > > > > the broker, and those are not secure (anyone with access to the box
> > > > > > > > can see
> > > > > > > > > > the command line used to invoke the broker); the configuration
> > > > > > includes
> > > > > > > > > > sensitive information (e.g. a keystore password), so we need a
> > > > > > secure
> > > > > > > > > > mechanism for passing the configuration values. I believe the real
> > > > > > > > > > motivation for this KIP is to harden the configuration mechanism
> > > > > > for
> > > > > > > > > > Zookeeper TLS connectivity.
> > > > > > > > > >
> > > > > > > > > > 2) I believe the list of CLI tools that continue to use direct
> > > > > > > > Zookeeper
> > > > > > > > > > connectivity in a non-deprecated fashion is:
> > > > > > > > > > a)
> > > > > > zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > > > > > > > b)
> > > > > > > > > >
> > > > > > > >
> > > > > > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > > > > > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > > > > > > > > >
> > > > > > > > > > 3) I believe Kafka.admin.ConfigCommand presents a conundrum
> > > > > > because it
> > > > > > > > > > explicitly states in a comment that a supported use case is
> > > > > > > > bootstrapping a
> > > > > > > > > > Kafka cluster with encrypted passwords in Zookeeper (see
> > > > > > > > > >
> > > > > > > >
> > > > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > > > > > > > ).
> > > > > > > > > > This means it will be especially difficult to deprecate this
> > > > > > particular
> > > > > > > > > > direct Zookeeper connectivity without a different storage
> > > > > > mechanism for
> > > > > > > > > > dynamic configuration values being available (i.e. the self-managed
> > > > > > > > quorum
> > > > > > > > > > referred to in KIP-500).
> > > > > > > > > >
> > > > > > > > > > I think it would be easier and simpler to harden the Zookeeper TLS
> > > > > > > > > > configuration in both Kafka and in CLI tools compared to trying to
> > > > > > > > > > deprecate the direct Zookeeper connectivity in the above 3 CLI
> > > > > > tools --
> > > > > > > > > > especially when ConfigCommand has no obvious short-term path to
> > > > > > > > deprecation.
> > > > > > > > > >
> > > > > > > > > > Regarding how to harden the TLS configuration, adding the
> > > > > > appropriate
> > > > > > > > > > configs to Kafka is an obvious choice for the brokers. Not that
> > > > > > these
> > > > > > > > > > values cannot be dynamically reconfigurable because the values
> > > > > > would be
> > > > > > > > > > required to connect to Zookeeper via TLS in the first place.
> > > > > > > > > >
> > > > > > > > > > Regarding how to harden the TLS configuration for the 3 remaining
> > > > > > CLI
> > > > > > > > > > tools, it would be relatively straightforward to add support for a
> > > > > > > > > > "--zk-tls-config-file <String: Zookeeper TLS configuration system
> > > > > > > > > > properties file path>" option to each one.
> > > > > > > > > >
> > > > > > > > > > Thoughts?
> > > > > > > > > >
> > > > > > > > > > Ron
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <
> > > > > > pere.urbon@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> True, the number is duplicated.
> > > > > > > > > >>
> > > > > > > > > >> do you know how can we get the problem fix? I was not aware, I
> > > > > > > > missed,
> > > > > > > > > >> sorry the need to add the KIP to the table.
> > > > > > > > > >>
> > > > > > > > > >> -- Pere
> > > > > > > > > >>
> > > > > > > > > >> Missatge de Jorge Esteban Quilcate Otoya <
> > > > > > quilcate.jorge@gmail.com>
> > > > > > > > del
> > > > > > > > > >> dia
> > > > > > > > > >> dt., 3 de set. 2019 a les 13:43:
> > > > > > > > > >>
> > > > > > > > > >> > Hi Pere,
> > > > > > > > > >> >
> > > > > > > > > >> > Have you add your KIP to the list here
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > > > > > >> ?
> > > > > > > > > >> >
> > > > > > > > > >> > I found the KIP number assigned to another.
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> > > > > > > > pere.urbon@gmail.com>
> > > > > > > > > >> > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> >> Thanks for your time Harsha,
> > > > > > > > > >> >> anyone else with comments? looking forward to hearing from
> > > > > > you.
> > > > > > > > > >> >>
> > > > > > > > > >> >> Stupid question: when do you move from discussion to vote?
> > > > > > > > > >> >>
> > > > > > > > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> > > > > > dv., 30
> > > > > > > > > >> d’ag.
> > > > > > > > > >> >> 2019 a les 21:59:
> > > > > > > > > >> >>
> > > > > > > > > >> >> > Thanks Pere. KIP looks good to me.
> > > > > > > > > >> >> > -Harsha
> > > > > > > > > >> >> >
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> > > > > > > > > >> >> pere.urbon@gmail.com>
> > > > > > > > > >> >> > wrote:
> > > > > > > > > >> >> >
> > > > > > > > > >> >> >> Not really,
> > > > > > > > > >> >> >> my idea is to keep the JAAS parameter, so people don't see
> > > > > > > > major
> > > > > > > > > >> >> >> changes. But if you pass a properties file, then this takes
> > > > > > > > > >> precedence
> > > > > > > > > >> >> over
> > > > > > > > > >> >> >> the other, with the idea that you can do sasl as well with
> > > > > > the
> > > > > > > > > >> >> properties
> > > > > > > > > >> >> >> files.
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> Makes sense?
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> -- Pere
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> > > > > > dv.,
> > > > > > > > 30
> > > > > > > > > >> >> d’ag.
> > > > > > > > > >> >> >> 2019 a les 19:00:
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >>> Hi Pere,
> > > > > > > > > >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper
> > > > > > for
> > > > > > > > Kafka
> > > > > > > > > >> >> makes
> > > > > > > > > >> >> >>> sense.
> > > > > > > > > >> >> >>> "The changes are planned to be introduced in a compatible
> > > > > > way,
> > > > > > > > by
> > > > > > > > > >> >> >>> keeping the current JAAS variable precedence."
> > > > > > > > > >> >> >>> Can you elaborate a bit here. If the user configures a JAAS
> > > > > > > > file
> > > > > > > > > >> with
> > > > > > > > > >> >> >>> Client section it will take precedence over zookeeper SSL
> > > > > > > > configs?
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>> Thanks,
> > > > > > > > > >> >> >>> Harsha
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> > > > > > > > > >> >> pere.urbon@gmail.com>
> > > > > > > > > >> >> >>> wrote:
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>>> Hi,
> > > > > > > > > >> >> >>>> quick question, I saw in another mail that 2.4 release is
> > > > > > > > planned
> > > > > > > > > >> for
> > > > > > > > > >> >> >>>> September. I think it would be really awesome to have
> > > > > > this for
> > > > > > > > > >> this
> > > > > > > > > >> >> >>>> release, do you think we can make it?
> > > > > > > > > >> >> >>>>
> > > > > > > > > >> >> >>>> -- Pere
> > > > > > > > > >> >> >>>>
> > > > > > > > > >> >> >>>> Missatge de Pere Urbón Bayes <pe...@gmail.com> del
> > > > > > dia
> > > > > > > > dj.,
> > > > > > > > > >> 29
> > > > > > > > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > > > > > > > >> >> >>>>
> > > > > > > > > >> >> >>>> Hi,
> > > > > > > > > >> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm
> > > > > > > > really
> > > > > > > > > >> need
> > > > > > > > > >> >> >>>> to the process. Looking forward to hearing from you and
> > > > > > learn
> > > > > > > > the
> > > > > > > > > >> >> best
> > > > > > > > > >> >> >>>> ropes here.
> > > > > > > > > >> >> >>>>
> > > > > > > > > >> >> >>>> I would like to propose this KIP-515 to enable the
> > > > > > > > > >> ZookeeperClients
> > > > > > > > > >> >> to
> > > > > > > > > >> >> >>>> take full advantage of the TLS communication in the new
> > > > > > > > Zookeeper
> > > > > > > > > >> >> 3.5.5.
> > > > > > > > > >> >> >>>> Specially interesting it the Zookeeper Security Migration,
> > > > > > > > that
> > > > > > > > > >> >> without
> > > > > > > > > >> >> >>>> this change will not work with TLS, disabling users to use
> > > > > > > > ACLs
> > > > > > > > > >> when
> > > > > > > > > >> >> the
> > > > > > > > > >> >> >>>> Zookeeper cluster use TLS.
> > > > > > > > > >> >> >>>>
> > > > > > > > > >> >> >>>> link:
> > > > > > > > > >> >> >>>>
> > > > > > > > > >> >> >>>>
> > > > > > > > > >> >>
> > > > > > > > > >>
> > > > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > > > > > > > >> >> >>>>
> > > > > > > > > >> >> >>>> Looking forward to hearing from you on this,
> > > > > > > > > >> >> >>>>
> > > > > > > > > >> >> >>>> /cheers
> > > > > > > > > >> >> >>>>
> > > > > > > > > >> >> >>>> --
> > > > > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > > > > >> >> >>>> Software Architect
> > > > > > > > > >> >> >>>> http://www.purbon.com
> > > > > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > > > > >> >> >>>>
> > > > > > > > > >> >> >>>> --
> > > > > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > > > > >> >> >>>> Software Architect
> > > > > > > > > >> >> >>>> http://www.purbon.com
> > > > > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > > > > >> >> >>>>
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> --
> > > > > > > > > >> >> >> Pere Urbon-Bayes
> > > > > > > > > >> >> >> Software Architect
> > > > > > > > > >> >> >> http://www.purbon.com
> > > > > > > > > >> >> >> https://twitter.com/purbon
> > > > > > > > > >> >> >> https://www.linkedin.com/in/purbon/
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >
> > > > > > > > > >> >> >
> > > > > > > > > >> >>
> > > > > > > > > >> >> --
> > > > > > > > > >> >> Pere Urbon-Bayes
> > > > > > > > > >> >> Software Architect
> > > > > > > > > >> >> http://www.purbon.com
> > > > > > > > > >> >> https://twitter.com/purbon
> > > > > > > > > >> >> https://www.linkedin.com/in/purbon/
> > > > > > > > > >> >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >> --
> > > > > > > > > >> Pere Urbon-Bayes
> > > > > > > > > >> Software Architect
> > > > > > > > > >> http://www.purbon.com
> > > > > > > > > >> https://twitter.com/purbon
> > > > > > > > > >> https://www.linkedin.com/in/purbon/
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Ron Dagostino <rn...@gmail.com>.
Hi Colin. Two things come to mind with respect to ZooKeeper camelCase
style vs Kafka-style config names for ZooKeeper. First, I think it
would be desirable for the client configs and broker configs to be
interoperable. For example, it feels like it would be convenient to
be able to pass the broker's config file to the ZK Security Migrator
tool. I think whatever we use (ZooKeeper camelCase style or Kafka
style), I think we should use the same for both.
The second thing to think about is the fact that ZooKeeper
configuration inherently uses Java system properties as a first pass.
So while we might switch to Kafka-style configs in the config file
(e.g. zookeeper.ssl.trust.store.location), the
org.apache.zookeeper.client.ZKClientConfig class (which is how TLS
configs are set) is going to look for the camelCase
zookeeper.ssl.trustStore.location system property and use any value
there. I think this could be very confusing for people. Granted, we
are doing this so that people don't have to use system properties, but
there is no way to turn off the use of system properties, so I think
there would be pretty substantial potential for confusion.
One idea is to migrate the system properties -- i.e. look to see if
zookeeper.ssl.trustStore.location is set and if it is clear out the
value there and move it under the zookeeper.ssl.trust.store.location
system property. (I said it was an idea -- not necessarily a good one
:-)
Ron
On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe <cm...@apache.org> wrote:
>
> On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
> > Thanks Colin and Rajini.
> >
> > I've updated the KIP to say that KIP-421 functionality is available to
> > encrypt sensitive configs like the ZK key store and trust store
> > passwords. (I've also made it clear that the configs are not
> > dynamically reconfigurable since dynamic values are stored in ZK and
> > the data is needed to get to ZK in the first place). Colin, let me
> > know if you think anything else needs to be said/done about it -- I
> > wasn't sure if your comment above implies that there are additional
> > direct actions that we need to take aside from this.
> >
> > I agree that making the brokers' ZooKeeper clients dynamic with
> > respect to key and trust stores (e.g. via
> > zookeeper.ssl.context.supplier.class) may increase the risk that this
> > KIP may not land in time for 2.5.
> >
> > Regarding the config names for ZK being different than the ZooKeeper
> > configs (e.g. camel-case keyStore/trustStore instead of
> > keystore/truststore, ciphersuites instead of cipher.suites,
> > enabledProtocols instead of enabled.protocols), I agree it is an
> > issue. I tried to mitigate it by putting warning text in the config
> > docs for these. Regarding configuration inheritance, I think what you
> > are saying is that for any configs where the broker has a clear
> > semantic equivalent, the broker could fall back to the broker value if
> > the ZK value is not given. The potential list is:
> >
> > zookeeper.ssl.keyStore.location => ssl.keystore.location
> > zookeeper.ssl.keyStore.password => ssl.keystore.password
> > zookeeper.ssl.keyStore.type => ssl.keystore.type
> > zookeeper.ssl.trustStore.location => ssl.truststore.location
> > zookeeper.ssl.trustStore.password => ssl.truststore.password
> > zookeeper.ssl.trustStore.type => ssl.truststore.type
> > zookeeper.ssl.ciphersuites => ssl.cipher.suites
> >
> > Note that there are two configs that look the same based on their key
> > names but whose allowable values differ:
> >
> > zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default: TLS)
> > zookeeper.ssl.enabledProtocols(Default: value of protocol property) =
> > ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
> >
> > Not sure what the right answer is, but hopefully this detail will help
> > get us there.
> >
> > Ron
>
> I think, on balance, I agree with Rajini that it would be nice to make the configs look more like Kafka configs. We don't have camel-case in configuration keys, so we should avoid it here.
>
> best,
> Colin
>
> >
> > On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> > > > Hi Ron,
> > > >
> > > > Thanks for the detailed explanation, sounds good to me.
> > > >
> > > > A few more questions:
> > > >
> > > > 1) At the moment, all sensitive broker configs including
> > > > keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
> > > > starting up brokers. We are now adding SSL keystore/truststore passwords
> > > > for ZooKeeper client that cannot be stored in ZooKeeper since you need
> > > > these to connect to ZK. We should perhaps document that these configs can
> > > > be encrypted using KIP-421.
> > >
> > > That's a good point. Previously, we didn't have ZK on-the-wire security support. However, now that we do, sending sensitive keystore and truststore passwords to ZK in cleartext should be deprecated in favor of using KIP-421.
> > >
> > > >
> > > > 2) We can dynamically update keystores and trust stores used by brokers
> > > > without restarting the broker. Can we support this easily for ZK clients
> > > > created by the broker, for example by adding our own
> > > > `zookeeper.ssl.context.supplier.class`?
> > > >
> > >
> > > Hmm. That might be better to consider in a follow-up, since the deadline for 2.5 KIPs is coming up?
> > >
> > > best,
> > > Colin
> > >
> > > > 3) Looks like we are using config names that directly map to ZK configs.
> > > > Have we considered using equivalent Kafka config names with prefixes,
> > > > perhaps with inheritance from the non-prefixed names? Not sure if this is a
> > > > good idea, but perhaps worth documenting in Rejected Alternatives at least.
> > > >
> > > >
> > > > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <cm...@apache.org> wrote:
> > > >
> > > > > Hi Ron,
> > > > >
> > > > > Thanks for the explanation. I guess thinking about it a little bit more,
> > > > > we should just add --zk-tls-config-file to all of these commands.
> > > > >
> > > > > We will be removing this option (plus ZK in general) from these commands
> > > > > in the next major release, but ZK is still supported in 2.5, so we should
> > > > > just do the logical thing. (The exception is ZkSecurityMigrator, which
> > > > > will stay).
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > > > > Hi Colin.
> > > > > >
> > > > > > <<< It seems like this [--zk-tls-config-file information] could just
> > > > > appear
> > > > > > in a configuration file, which all of these tools already accept (I
> > > > > think)
> > > > > >
> > > > > > ZkSecurityMigrator has no such property file facility; adding a
> > > > > > "--zk-tls-config-file" parameter is exactly for this purpose. If we add
> > > > > > that to ZkSecurityMigrator then it is trivial to add it to other commands
> > > > > > (the same code is simply reused; it ends up being just a few extra
> > > > > lines).
> > > > > > I do not see any parameter in the other two commands to adjust the ZK
> > > > > > connection; ConfigCommand accepts a "--command-config" flag, but
> > > > > according
> > > > > > to the code "This is used only with --bootstrap-server option for
> > > > > > describing and altering broker configs."
> > > > > >
> > > > > > I do agree there would be no need to add "--zk-tls-config-file" to
> > > > > > ReassignPartitionsCommand if its direct ZK connectivity is replaced in
> > > > > time
> > > > > > for the next release.
> > > > > >
> > > > > > ConfigCommand supports the "--bootstrap-server" option and will have its
> > > > > > direct ZooKeeper access formally deprecated as per KIP-555, but the
> > > > > special
> > > > > > use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> > > > > > prior to starting Kafka will still exist, so it feels like while
> > > > > > "--zk-tls-config-file" would never be used except for this use case it
> > > > > > could probably still be added for this particular situation.
> > > > > >
> > > > > > Ron
> > > > > >
> > > > > > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> > > > > > sent) did not go through. I am trying to get emails through now to move
> > > > > > this discussion forward.
> > > > > >
> > > > > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cm...@apache.org> wrote:
> > > > > >
> > > > > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > > > > > Hi everyone. I would like to make the following changes to the KIP.
> > > > > > > >
> > > > > > > > MOTIVATION:
> > > > > > > > Include a statement that it will be difficult in the short term to
> > > > > > > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > > > > > > (which
> > > > > > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
> > > > > cluster
> > > > > > > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > > > > > > case;
> > > > > > > > therefore it is in scope to be able to securely configure the CLI
> > > > > tools
> > > > > > > > that still leverage non-deprecated direct Zookeeper communication
> > > > > for TLS
> > > > > > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > > > > > > zookeeper-security-migration.sh).
> > > > > > >
> > > > > > > Hi Ron,
> > > > > > >
> > > > > > > Thanks for the KIP.
> > > > > > >
> > > > > > > About deprecations:
> > > > > > >
> > > > > > > * zookeeper-security-migration: clearly, deprecating ZK access in this
> > > > > one
> > > > > > > would not make sense, since it would defeat the whole point of the
> > > > > tool :)
> > > > > > >
> > > > > > > * kafka-reassign-partitions: ZK access should be deprecated here.
> > > > > KIP-455
> > > > > > > implementation has dragged a bit, but this should get done soon.
> > > > > Certainly
> > > > > > > before the next release.
> > > > > > >
> > > > > > > * kafka-configs: I think ZK access should be deprecated here as well.
> > > > > I
> > > > > > > agree there is a super-special bootstrapping case here, but that should
> > > > > > > have its own tool, not use kafka-configs.
> > > > > > >
> > > > > > > I will post a separate KIP for this, though.
> > > > > > >
> > > > > > > >
> > > > > > > > GOALS:
> > > > > > > > Support the secure configuration of TLS-encrypted communication
> > > > > between
> > > > > > > > Zookeeper and:
> > > > > > > > a) Kafka brokers
> > > > > > > > b) The three CLI tools mentioned above that still support direct,
> > > > > > > > non-deprecated communication to Zookeeper
> > > > > > > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > > > > > > communication in CLI tools as part of this KIP; such work will occur
> > > > > in
> > > > > > > > future KIPs instead.
> > > > > > > >
> > > > > > > > PUBLIC INTERFACES:
> > > > > > > > 1) The following new broker configurations will be recognized.
> > > > > > > > zookeeper.client.secure (default value = false, for backwards
> > > > > > > > compatibility)
> > > > > > > > zookeeper.clientCnxnSocket
> > > > > > > > zookeeper.ssl.keyStore.location
> > > > > > > > zookeeper.ssl.keyStore.password
> > > > > > > > zookeeper.ssl.trustStore.location
> > > > > > > > zookeeper.ssl.trustStore.password
> > > > > > > > It will be an error for any of the last 5 values to be left
> > > > > unspecified
> > > > > > > if
> > > > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > > > >
> > > > > > > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> > > > > > > supports
> > > > > > > > the ability to connect to a different Zookeeper instance than the
> > > > > one the
> > > > > > > > brokers use. We therefore also add the following optional configs,
> > > > > which
> > > > > > > > override the corresponding ones from above when present:
> > > > > > > > authorizer.zookeeper.client.secure
> > > > > > > > authorizer.zookeeper.clientCnxnSocket
> > > > > > > > authorizer.zookeeper.ssl.keyStore.location
> > > > > > > > authorizer.zookeeper.ssl.keyStore.password
> > > > > > > > authorizer.zookeeper.ssl.trustStore.location
> > > > > > > > authorizer.zookeeper.ssl.trustStore.password
> > > > > > > >
> > > > > > > > 3) The three CLI tools mentioned above will support a new
> > > > > > > --zk-tls-config-file
> > > > > > > > <String: Zookeeper TLS configuration file path>" option. The
> > > > > following
> > > > > > > > properties will be recognized in that file, and unrecognized
> > > > > properties
> > > > > > > > will be ignored to allow the possibility of pointing
> > > > > zk-tls-config-file
> > > > > > > at
> > > > > > > > the broker's config file.
> > > > > > > > zookeeper.client.secure (default value = false)
> > > > > > > > zookeeper.clientCnxnSocket
> > > > > > > > zookeeper.ssl.keyStore.location
> > > > > > > > zookeeper.ssl.keyStore.password
> > > > > > > > zookeeper.ssl.trustStore.location
> > > > > > > > zookeeper.ssl.trustStore.password
> > > > > > > > It will be an error for any of the last 5 values to be left
> > > > > unspecified
> > > > > > > if
> > > > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > > >
> > > > > > > Do we really need a --zk-tls-config-file flag? It seems like this
> > > > > could
> > > > > > > just appear in a configuration file, which all of these tools already
> > > > > > > accept (I think).
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > Ron
> > > > > > > >
> > > > > > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rn...@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi everyone. Let's get this discussion going again now that Kafka
> > > > > 2.4
> > > > > > > > > with Zookeeper 3.5.6 is out.
> > > > > > > > >
> > > > > > > > > First, regarding the KIP number, the other KIP that was using this
> > > > > > > number
> > > > > > > > > moved to KIP 534, so KIP 515 remains the correct number for this
> > > > > > > > > discussion. I've updated the Kafka Improvement Proposal page to
> > > > > list
> > > > > > > this
> > > > > > > > > KIP in the 515 slot, so we're all set there.
> > > > > > > > >
> > > > > > > > > Regarding the actual issues under discussion, I think there are
> > > > > some
> > > > > > > > > things we should clarify.
> > > > > > > > >
> > > > > > > > > 1) It is possible to use TLS connectivity to Zookeeper from Apache
> > > > > > > Kafka
> > > > > > > > > 2.4 -- the problem is that configuration information has to be
> > > > > passed
> > > > > > > via
> > > > > > > > > system properties as "-D" command line options on the java
> > > > > invocation
> > > > > > > of
> > > > > > > > > the broker, and those are not secure (anyone with access to the box
> > > > > > > can see
> > > > > > > > > the command line used to invoke the broker); the configuration
> > > > > includes
> > > > > > > > > sensitive information (e.g. a keystore password), so we need a
> > > > > secure
> > > > > > > > > mechanism for passing the configuration values. I believe the real
> > > > > > > > > motivation for this KIP is to harden the configuration mechanism
> > > > > for
> > > > > > > > > Zookeeper TLS connectivity.
> > > > > > > > >
> > > > > > > > > 2) I believe the list of CLI tools that continue to use direct
> > > > > > > Zookeeper
> > > > > > > > > connectivity in a non-deprecated fashion is:
> > > > > > > > > a)
> > > > > zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > > > > > > b)
> > > > > > > > >
> > > > > > >
> > > > > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > > > > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > > > > > > > >
> > > > > > > > > 3) I believe Kafka.admin.ConfigCommand presents a conundrum
> > > > > because it
> > > > > > > > > explicitly states in a comment that a supported use case is
> > > > > > > bootstrapping a
> > > > > > > > > Kafka cluster with encrypted passwords in Zookeeper (see
> > > > > > > > >
> > > > > > >
> > > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > > > > > > ).
> > > > > > > > > This means it will be especially difficult to deprecate this
> > > > > particular
> > > > > > > > > direct Zookeeper connectivity without a different storage
> > > > > mechanism for
> > > > > > > > > dynamic configuration values being available (i.e. the self-managed
> > > > > > > quorum
> > > > > > > > > referred to in KIP-500).
> > > > > > > > >
> > > > > > > > > I think it would be easier and simpler to harden the Zookeeper TLS
> > > > > > > > > configuration in both Kafka and in CLI tools compared to trying to
> > > > > > > > > deprecate the direct Zookeeper connectivity in the above 3 CLI
> > > > > tools --
> > > > > > > > > especially when ConfigCommand has no obvious short-term path to
> > > > > > > deprecation.
> > > > > > > > >
> > > > > > > > > Regarding how to harden the TLS configuration, adding the
> > > > > appropriate
> > > > > > > > > configs to Kafka is an obvious choice for the brokers. Not that
> > > > > these
> > > > > > > > > values cannot be dynamically reconfigurable because the values
> > > > > would be
> > > > > > > > > required to connect to Zookeeper via TLS in the first place.
> > > > > > > > >
> > > > > > > > > Regarding how to harden the TLS configuration for the 3 remaining
> > > > > CLI
> > > > > > > > > tools, it would be relatively straightforward to add support for a
> > > > > > > > > "--zk-tls-config-file <String: Zookeeper TLS configuration system
> > > > > > > > > properties file path>" option to each one.
> > > > > > > > >
> > > > > > > > > Thoughts?
> > > > > > > > >
> > > > > > > > > Ron
> > > > > > > > >
> > > > > > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <
> > > > > pere.urbon@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> True, the number is duplicated.
> > > > > > > > >>
> > > > > > > > >> do you know how can we get the problem fix? I was not aware, I
> > > > > > > missed,
> > > > > > > > >> sorry the need to add the KIP to the table.
> > > > > > > > >>
> > > > > > > > >> -- Pere
> > > > > > > > >>
> > > > > > > > >> Missatge de Jorge Esteban Quilcate Otoya <
> > > > > quilcate.jorge@gmail.com>
> > > > > > > del
> > > > > > > > >> dia
> > > > > > > > >> dt., 3 de set. 2019 a les 13:43:
> > > > > > > > >>
> > > > > > > > >> > Hi Pere,
> > > > > > > > >> >
> > > > > > > > >> > Have you add your KIP to the list here
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > > > > >> ?
> > > > > > > > >> >
> > > > > > > > >> > I found the KIP number assigned to another.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> > > > > > > pere.urbon@gmail.com>
> > > > > > > > >> > wrote:
> > > > > > > > >> >
> > > > > > > > >> >> Thanks for your time Harsha,
> > > > > > > > >> >> anyone else with comments? looking forward to hearing from
> > > > > you.
> > > > > > > > >> >>
> > > > > > > > >> >> Stupid question: when do you move from discussion to vote?
> > > > > > > > >> >>
> > > > > > > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> > > > > dv., 30
> > > > > > > > >> d’ag.
> > > > > > > > >> >> 2019 a les 21:59:
> > > > > > > > >> >>
> > > > > > > > >> >> > Thanks Pere. KIP looks good to me.
> > > > > > > > >> >> > -Harsha
> > > > > > > > >> >> >
> > > > > > > > >> >> >
> > > > > > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> > > > > > > > >> >> pere.urbon@gmail.com>
> > > > > > > > >> >> > wrote:
> > > > > > > > >> >> >
> > > > > > > > >> >> >> Not really,
> > > > > > > > >> >> >> my idea is to keep the JAAS parameter, so people don't see
> > > > > > > major
> > > > > > > > >> >> >> changes. But if you pass a properties file, then this takes
> > > > > > > > >> precedence
> > > > > > > > >> >> over
> > > > > > > > >> >> >> the other, with the idea that you can do sasl as well with
> > > > > the
> > > > > > > > >> >> properties
> > > > > > > > >> >> >> files.
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> Makes sense?
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> -- Pere
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> > > > > dv.,
> > > > > > > 30
> > > > > > > > >> >> d’ag.
> > > > > > > > >> >> >> 2019 a les 19:00:
> > > > > > > > >> >> >>
> > > > > > > > >> >> >>> Hi Pere,
> > > > > > > > >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper
> > > > > for
> > > > > > > Kafka
> > > > > > > > >> >> makes
> > > > > > > > >> >> >>> sense.
> > > > > > > > >> >> >>> "The changes are planned to be introduced in a compatible
> > > > > way,
> > > > > > > by
> > > > > > > > >> >> >>> keeping the current JAAS variable precedence."
> > > > > > > > >> >> >>> Can you elaborate a bit here. If the user configures a JAAS
> > > > > > > file
> > > > > > > > >> with
> > > > > > > > >> >> >>> Client section it will take precedence over zookeeper SSL
> > > > > > > configs?
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>> Thanks,
> > > > > > > > >> >> >>> Harsha
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> > > > > > > > >> >> pere.urbon@gmail.com>
> > > > > > > > >> >> >>> wrote:
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>>> Hi,
> > > > > > > > >> >> >>>> quick question, I saw in another mail that 2.4 release is
> > > > > > > planned
> > > > > > > > >> for
> > > > > > > > >> >> >>>> September. I think it would be really awesome to have
> > > > > this for
> > > > > > > > >> this
> > > > > > > > >> >> >>>> release, do you think we can make it?
> > > > > > > > >> >> >>>>
> > > > > > > > >> >> >>>> -- Pere
> > > > > > > > >> >> >>>>
> > > > > > > > >> >> >>>> Missatge de Pere Urbón Bayes <pe...@gmail.com> del
> > > > > dia
> > > > > > > dj.,
> > > > > > > > >> 29
> > > > > > > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > > > > > > >> >> >>>>
> > > > > > > > >> >> >>>> Hi,
> > > > > > > > >> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm
> > > > > > > really
> > > > > > > > >> need
> > > > > > > > >> >> >>>> to the process. Looking forward to hearing from you and
> > > > > learn
> > > > > > > the
> > > > > > > > >> >> best
> > > > > > > > >> >> >>>> ropes here.
> > > > > > > > >> >> >>>>
> > > > > > > > >> >> >>>> I would like to propose this KIP-515 to enable the
> > > > > > > > >> ZookeeperClients
> > > > > > > > >> >> to
> > > > > > > > >> >> >>>> take full advantage of the TLS communication in the new
> > > > > > > Zookeeper
> > > > > > > > >> >> 3.5.5.
> > > > > > > > >> >> >>>> Specially interesting it the Zookeeper Security Migration,
> > > > > > > that
> > > > > > > > >> >> without
> > > > > > > > >> >> >>>> this change will not work with TLS, disabling users to use
> > > > > > > ACLs
> > > > > > > > >> when
> > > > > > > > >> >> the
> > > > > > > > >> >> >>>> Zookeeper cluster use TLS.
> > > > > > > > >> >> >>>>
> > > > > > > > >> >> >>>> link:
> > > > > > > > >> >> >>>>
> > > > > > > > >> >> >>>>
> > > > > > > > >> >>
> > > > > > > > >>
> > > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > > > > > > >> >> >>>>
> > > > > > > > >> >> >>>> Looking forward to hearing from you on this,
> > > > > > > > >> >> >>>>
> > > > > > > > >> >> >>>> /cheers
> > > > > > > > >> >> >>>>
> > > > > > > > >> >> >>>> --
> > > > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > > > >> >> >>>> Software Architect
> > > > > > > > >> >> >>>> http://www.purbon.com
> > > > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > > > >> >> >>>>
> > > > > > > > >> >> >>>> --
> > > > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > > > >> >> >>>> Software Architect
> > > > > > > > >> >> >>>> http://www.purbon.com
> > > > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > > > >> >> >>>>
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> --
> > > > > > > > >> >> >> Pere Urbon-Bayes
> > > > > > > > >> >> >> Software Architect
> > > > > > > > >> >> >> http://www.purbon.com
> > > > > > > > >> >> >> https://twitter.com/purbon
> > > > > > > > >> >> >> https://www.linkedin.com/in/purbon/
> > > > > > > > >> >> >>
> > > > > > > > >> >> >
> > > > > > > > >> >> >
> > > > > > > > >> >>
> > > > > > > > >> >> --
> > > > > > > > >> >> Pere Urbon-Bayes
> > > > > > > > >> >> Software Architect
> > > > > > > > >> >> http://www.purbon.com
> > > > > > > > >> >> https://twitter.com/purbon
> > > > > > > > >> >> https://www.linkedin.com/in/purbon/
> > > > > > > > >> >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >> --
> > > > > > > > >> Pere Urbon-Bayes
> > > > > > > > >> Software Architect
> > > > > > > > >> http://www.purbon.com
> > > > > > > > >> https://twitter.com/purbon
> > > > > > > > >> https://www.linkedin.com/in/purbon/
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Colin McCabe <cm...@apache.org>.
On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
> Thanks Colin and Rajini.
>
> I've updated the KIP to say that KIP-421 functionality is available to
> encrypt sensitive configs like the ZK key store and trust store
> passwords. (I've also made it clear that the configs are not
> dynamically reconfigurable since dynamic values are stored in ZK and
> the data is needed to get to ZK in the first place). Colin, let me
> know if you think anything else needs to be said/done about it -- I
> wasn't sure if your comment above implies that there are additional
> direct actions that we need to take aside from this.
>
> I agree that making the brokers' ZooKeeper clients dynamic with
> respect to key and trust stores (e.g. via
> zookeeper.ssl.context.supplier.class) may increase the risk that this
> KIP may not land in time for 2.5.
>
> Regarding the config names for ZK being different than the ZooKeeper
> configs (e.g. camel-case keyStore/trustStore instead of
> keystore/truststore, ciphersuites instead of cipher.suites,
> enabledProtocols instead of enabled.protocols), I agree it is an
> issue. I tried to mitigate it by putting warning text in the config
> docs for these. Regarding configuration inheritance, I think what you
> are saying is that for any configs where the broker has a clear
> semantic equivalent, the broker could fall back to the broker value if
> the ZK value is not given. The potential list is:
>
> zookeeper.ssl.keyStore.location => ssl.keystore.location
> zookeeper.ssl.keyStore.password => ssl.keystore.password
> zookeeper.ssl.keyStore.type => ssl.keystore.type
> zookeeper.ssl.trustStore.location => ssl.truststore.location
> zookeeper.ssl.trustStore.password => ssl.truststore.password
> zookeeper.ssl.trustStore.type => ssl.truststore.type
> zookeeper.ssl.ciphersuites => ssl.cipher.suites
>
> Note that there are two configs that look the same based on their key
> names but whose allowable values differ:
>
> zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default: TLS)
> zookeeper.ssl.enabledProtocols(Default: value of protocol property) =
> ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
>
> Not sure what the right answer is, but hopefully this detail will help
> get us there.
>
> Ron
I think, on balance, I agree with Rajini that it would be nice to make the configs look more like Kafka configs. We don't have camel-case in configuration keys, so we should avoid it here.
best,
Colin
>
> On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> > > Hi Ron,
> > >
> > > Thanks for the detailed explanation, sounds good to me.
> > >
> > > A few more questions:
> > >
> > > 1) At the moment, all sensitive broker configs including
> > > keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
> > > starting up brokers. We are now adding SSL keystore/truststore passwords
> > > for ZooKeeper client that cannot be stored in ZooKeeper since you need
> > > these to connect to ZK. We should perhaps document that these configs can
> > > be encrypted using KIP-421.
> >
> > That's a good point. Previously, we didn't have ZK on-the-wire security support. However, now that we do, sending sensitive keystore and truststore passwords to ZK in cleartext should be deprecated in favor of using KIP-421.
> >
> > >
> > > 2) We can dynamically update keystores and trust stores used by brokers
> > > without restarting the broker. Can we support this easily for ZK clients
> > > created by the broker, for example by adding our own
> > > `zookeeper.ssl.context.supplier.class`?
> > >
> >
> > Hmm. That might be better to consider in a follow-up, since the deadline for 2.5 KIPs is coming up?
> >
> > best,
> > Colin
> >
> > > 3) Looks like we are using config names that directly map to ZK configs.
> > > Have we considered using equivalent Kafka config names with prefixes,
> > > perhaps with inheritance from the non-prefixed names? Not sure if this is a
> > > good idea, but perhaps worth documenting in Rejected Alternatives at least.
> > >
> > >
> > > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > Hi Ron,
> > > >
> > > > Thanks for the explanation. I guess thinking about it a little bit more,
> > > > we should just add --zk-tls-config-file to all of these commands.
> > > >
> > > > We will be removing this option (plus ZK in general) from these commands
> > > > in the next major release, but ZK is still supported in 2.5, so we should
> > > > just do the logical thing. (The exception is ZkSecurityMigrator, which
> > > > will stay).
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > > > Hi Colin.
> > > > >
> > > > > <<< It seems like this [--zk-tls-config-file information] could just
> > > > appear
> > > > > in a configuration file, which all of these tools already accept (I
> > > > think)
> > > > >
> > > > > ZkSecurityMigrator has no such property file facility; adding a
> > > > > "--zk-tls-config-file" parameter is exactly for this purpose. If we add
> > > > > that to ZkSecurityMigrator then it is trivial to add it to other commands
> > > > > (the same code is simply reused; it ends up being just a few extra
> > > > lines).
> > > > > I do not see any parameter in the other two commands to adjust the ZK
> > > > > connection; ConfigCommand accepts a "--command-config" flag, but
> > > > according
> > > > > to the code "This is used only with --bootstrap-server option for
> > > > > describing and altering broker configs."
> > > > >
> > > > > I do agree there would be no need to add "--zk-tls-config-file" to
> > > > > ReassignPartitionsCommand if its direct ZK connectivity is replaced in
> > > > time
> > > > > for the next release.
> > > > >
> > > > > ConfigCommand supports the "--bootstrap-server" option and will have its
> > > > > direct ZooKeeper access formally deprecated as per KIP-555, but the
> > > > special
> > > > > use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> > > > > prior to starting Kafka will still exist, so it feels like while
> > > > > "--zk-tls-config-file" would never be used except for this use case it
> > > > > could probably still be added for this particular situation.
> > > > >
> > > > > Ron
> > > > >
> > > > > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> > > > > sent) did not go through. I am trying to get emails through now to move
> > > > > this discussion forward.
> > > > >
> > > > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cm...@apache.org> wrote:
> > > > >
> > > > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > > > > Hi everyone. I would like to make the following changes to the KIP.
> > > > > > >
> > > > > > > MOTIVATION:
> > > > > > > Include a statement that it will be difficult in the short term to
> > > > > > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > > > > > (which
> > > > > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
> > > > cluster
> > > > > > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > > > > > case;
> > > > > > > therefore it is in scope to be able to securely configure the CLI
> > > > tools
> > > > > > > that still leverage non-deprecated direct Zookeeper communication
> > > > for TLS
> > > > > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > > > > > zookeeper-security-migration.sh).
> > > > > >
> > > > > > Hi Ron,
> > > > > >
> > > > > > Thanks for the KIP.
> > > > > >
> > > > > > About deprecations:
> > > > > >
> > > > > > * zookeeper-security-migration: clearly, deprecating ZK access in this
> > > > one
> > > > > > would not make sense, since it would defeat the whole point of the
> > > > tool :)
> > > > > >
> > > > > > * kafka-reassign-partitions: ZK access should be deprecated here.
> > > > KIP-455
> > > > > > implementation has dragged a bit, but this should get done soon.
> > > > Certainly
> > > > > > before the next release.
> > > > > >
> > > > > > * kafka-configs: I think ZK access should be deprecated here as well.
> > > > I
> > > > > > agree there is a super-special bootstrapping case here, but that should
> > > > > > have its own tool, not use kafka-configs.
> > > > > >
> > > > > > I will post a separate KIP for this, though.
> > > > > >
> > > > > > >
> > > > > > > GOALS:
> > > > > > > Support the secure configuration of TLS-encrypted communication
> > > > between
> > > > > > > Zookeeper and:
> > > > > > > a) Kafka brokers
> > > > > > > b) The three CLI tools mentioned above that still support direct,
> > > > > > > non-deprecated communication to Zookeeper
> > > > > > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > > > > > communication in CLI tools as part of this KIP; such work will occur
> > > > in
> > > > > > > future KIPs instead.
> > > > > > >
> > > > > > > PUBLIC INTERFACES:
> > > > > > > 1) The following new broker configurations will be recognized.
> > > > > > > zookeeper.client.secure (default value = false, for backwards
> > > > > > > compatibility)
> > > > > > > zookeeper.clientCnxnSocket
> > > > > > > zookeeper.ssl.keyStore.location
> > > > > > > zookeeper.ssl.keyStore.password
> > > > > > > zookeeper.ssl.trustStore.location
> > > > > > > zookeeper.ssl.trustStore.password
> > > > > > > It will be an error for any of the last 5 values to be left
> > > > unspecified
> > > > > > if
> > > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > > >
> > > > > > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> > > > > > supports
> > > > > > > the ability to connect to a different Zookeeper instance than the
> > > > one the
> > > > > > > brokers use. We therefore also add the following optional configs,
> > > > which
> > > > > > > override the corresponding ones from above when present:
> > > > > > > authorizer.zookeeper.client.secure
> > > > > > > authorizer.zookeeper.clientCnxnSocket
> > > > > > > authorizer.zookeeper.ssl.keyStore.location
> > > > > > > authorizer.zookeeper.ssl.keyStore.password
> > > > > > > authorizer.zookeeper.ssl.trustStore.location
> > > > > > > authorizer.zookeeper.ssl.trustStore.password
> > > > > > >
> > > > > > > 3) The three CLI tools mentioned above will support a new
> > > > > > --zk-tls-config-file
> > > > > > > <String: Zookeeper TLS configuration file path>" option. The
> > > > following
> > > > > > > properties will be recognized in that file, and unrecognized
> > > > properties
> > > > > > > will be ignored to allow the possibility of pointing
> > > > zk-tls-config-file
> > > > > > at
> > > > > > > the broker's config file.
> > > > > > > zookeeper.client.secure (default value = false)
> > > > > > > zookeeper.clientCnxnSocket
> > > > > > > zookeeper.ssl.keyStore.location
> > > > > > > zookeeper.ssl.keyStore.password
> > > > > > > zookeeper.ssl.trustStore.location
> > > > > > > zookeeper.ssl.trustStore.password
> > > > > > > It will be an error for any of the last 5 values to be left
> > > > unspecified
> > > > > > if
> > > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > >
> > > > > > Do we really need a --zk-tls-config-file flag? It seems like this
> > > > could
> > > > > > just appear in a configuration file, which all of these tools already
> > > > > > accept (I think).
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Ron
> > > > > > >
> > > > > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rn...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone. Let's get this discussion going again now that Kafka
> > > > 2.4
> > > > > > > > with Zookeeper 3.5.6 is out.
> > > > > > > >
> > > > > > > > First, regarding the KIP number, the other KIP that was using this
> > > > > > number
> > > > > > > > moved to KIP 534, so KIP 515 remains the correct number for this
> > > > > > > > discussion. I've updated the Kafka Improvement Proposal page to
> > > > list
> > > > > > this
> > > > > > > > KIP in the 515 slot, so we're all set there.
> > > > > > > >
> > > > > > > > Regarding the actual issues under discussion, I think there are
> > > > some
> > > > > > > > things we should clarify.
> > > > > > > >
> > > > > > > > 1) It is possible to use TLS connectivity to Zookeeper from Apache
> > > > > > Kafka
> > > > > > > > 2.4 -- the problem is that configuration information has to be
> > > > passed
> > > > > > via
> > > > > > > > system properties as "-D" command line options on the java
> > > > invocation
> > > > > > of
> > > > > > > > the broker, and those are not secure (anyone with access to the box
> > > > > > can see
> > > > > > > > the command line used to invoke the broker); the configuration
> > > > includes
> > > > > > > > sensitive information (e.g. a keystore password), so we need a
> > > > secure
> > > > > > > > mechanism for passing the configuration values. I believe the real
> > > > > > > > motivation for this KIP is to harden the configuration mechanism
> > > > for
> > > > > > > > Zookeeper TLS connectivity.
> > > > > > > >
> > > > > > > > 2) I believe the list of CLI tools that continue to use direct
> > > > > > Zookeeper
> > > > > > > > connectivity in a non-deprecated fashion is:
> > > > > > > > a)
> > > > zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > > > > > b)
> > > > > > > >
> > > > > >
> > > > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > > > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > > > > > > >
> > > > > > > > 3) I believe Kafka.admin.ConfigCommand presents a conundrum
> > > > because it
> > > > > > > > explicitly states in a comment that a supported use case is
> > > > > > bootstrapping a
> > > > > > > > Kafka cluster with encrypted passwords in Zookeeper (see
> > > > > > > >
> > > > > >
> > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > > > > > ).
> > > > > > > > This means it will be especially difficult to deprecate this
> > > > particular
> > > > > > > > direct Zookeeper connectivity without a different storage
> > > > mechanism for
> > > > > > > > dynamic configuration values being available (i.e. the self-managed
> > > > > > quorum
> > > > > > > > referred to in KIP-500).
> > > > > > > >
> > > > > > > > I think it would be easier and simpler to harden the Zookeeper TLS
> > > > > > > > configuration in both Kafka and in CLI tools compared to trying to
> > > > > > > > deprecate the direct Zookeeper connectivity in the above 3 CLI
> > > > tools --
> > > > > > > > especially when ConfigCommand has no obvious short-term path to
> > > > > > deprecation.
> > > > > > > >
> > > > > > > > Regarding how to harden the TLS configuration, adding the
> > > > appropriate
> > > > > > > > configs to Kafka is an obvious choice for the brokers. Not that
> > > > these
> > > > > > > > values cannot be dynamically reconfigurable because the values
> > > > would be
> > > > > > > > required to connect to Zookeeper via TLS in the first place.
> > > > > > > >
> > > > > > > > Regarding how to harden the TLS configuration for the 3 remaining
> > > > CLI
> > > > > > > > tools, it would be relatively straightforward to add support for a
> > > > > > > > "--zk-tls-config-file <String: Zookeeper TLS configuration system
> > > > > > > > properties file path>" option to each one.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > Ron
> > > > > > > >
> > > > > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <
> > > > pere.urbon@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> True, the number is duplicated.
> > > > > > > >>
> > > > > > > >> do you know how can we get the problem fix? I was not aware, I
> > > > > > missed,
> > > > > > > >> sorry the need to add the KIP to the table.
> > > > > > > >>
> > > > > > > >> -- Pere
> > > > > > > >>
> > > > > > > >> Missatge de Jorge Esteban Quilcate Otoya <
> > > > quilcate.jorge@gmail.com>
> > > > > > del
> > > > > > > >> dia
> > > > > > > >> dt., 3 de set. 2019 a les 13:43:
> > > > > > > >>
> > > > > > > >> > Hi Pere,
> > > > > > > >> >
> > > > > > > >> > Have you add your KIP to the list here
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > > > >> ?
> > > > > > > >> >
> > > > > > > >> > I found the KIP number assigned to another.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> > > > > > pere.urbon@gmail.com>
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> >> Thanks for your time Harsha,
> > > > > > > >> >> anyone else with comments? looking forward to hearing from
> > > > you.
> > > > > > > >> >>
> > > > > > > >> >> Stupid question: when do you move from discussion to vote?
> > > > > > > >> >>
> > > > > > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> > > > dv., 30
> > > > > > > >> d’ag.
> > > > > > > >> >> 2019 a les 21:59:
> > > > > > > >> >>
> > > > > > > >> >> > Thanks Pere. KIP looks good to me.
> > > > > > > >> >> > -Harsha
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> > > > > > > >> >> pere.urbon@gmail.com>
> > > > > > > >> >> > wrote:
> > > > > > > >> >> >
> > > > > > > >> >> >> Not really,
> > > > > > > >> >> >> my idea is to keep the JAAS parameter, so people don't see
> > > > > > major
> > > > > > > >> >> >> changes. But if you pass a properties file, then this takes
> > > > > > > >> precedence
> > > > > > > >> >> over
> > > > > > > >> >> >> the other, with the idea that you can do sasl as well with
> > > > the
> > > > > > > >> >> properties
> > > > > > > >> >> >> files.
> > > > > > > >> >> >>
> > > > > > > >> >> >> Makes sense?
> > > > > > > >> >> >>
> > > > > > > >> >> >> -- Pere
> > > > > > > >> >> >>
> > > > > > > >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> > > > dv.,
> > > > > > 30
> > > > > > > >> >> d’ag.
> > > > > > > >> >> >> 2019 a les 19:00:
> > > > > > > >> >> >>
> > > > > > > >> >> >>> Hi Pere,
> > > > > > > >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper
> > > > for
> > > > > > Kafka
> > > > > > > >> >> makes
> > > > > > > >> >> >>> sense.
> > > > > > > >> >> >>> "The changes are planned to be introduced in a compatible
> > > > way,
> > > > > > by
> > > > > > > >> >> >>> keeping the current JAAS variable precedence."
> > > > > > > >> >> >>> Can you elaborate a bit here. If the user configures a JAAS
> > > > > > file
> > > > > > > >> with
> > > > > > > >> >> >>> Client section it will take precedence over zookeeper SSL
> > > > > > configs?
> > > > > > > >> >> >>>
> > > > > > > >> >> >>> Thanks,
> > > > > > > >> >> >>> Harsha
> > > > > > > >> >> >>>
> > > > > > > >> >> >>>
> > > > > > > >> >> >>>
> > > > > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> > > > > > > >> >> pere.urbon@gmail.com>
> > > > > > > >> >> >>> wrote:
> > > > > > > >> >> >>>
> > > > > > > >> >> >>>> Hi,
> > > > > > > >> >> >>>> quick question, I saw in another mail that 2.4 release is
> > > > > > planned
> > > > > > > >> for
> > > > > > > >> >> >>>> September. I think it would be really awesome to have
> > > > this for
> > > > > > > >> this
> > > > > > > >> >> >>>> release, do you think we can make it?
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> -- Pere
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> Missatge de Pere Urbón Bayes <pe...@gmail.com> del
> > > > dia
> > > > > > dj.,
> > > > > > > >> 29
> > > > > > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> Hi,
> > > > > > > >> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm
> > > > > > really
> > > > > > > >> need
> > > > > > > >> >> >>>> to the process. Looking forward to hearing from you and
> > > > learn
> > > > > > the
> > > > > > > >> >> best
> > > > > > > >> >> >>>> ropes here.
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> I would like to propose this KIP-515 to enable the
> > > > > > > >> ZookeeperClients
> > > > > > > >> >> to
> > > > > > > >> >> >>>> take full advantage of the TLS communication in the new
> > > > > > Zookeeper
> > > > > > > >> >> 3.5.5.
> > > > > > > >> >> >>>> Specially interesting it the Zookeeper Security Migration,
> > > > > > that
> > > > > > > >> >> without
> > > > > > > >> >> >>>> this change will not work with TLS, disabling users to use
> > > > > > ACLs
> > > > > > > >> when
> > > > > > > >> >> the
> > > > > > > >> >> >>>> Zookeeper cluster use TLS.
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> link:
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>>
> > > > > > > >> >>
> > > > > > > >>
> > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> Looking forward to hearing from you on this,
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> /cheers
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> --
> > > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > > >> >> >>>> Software Architect
> > > > > > > >> >> >>>> http://www.purbon.com
> > > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> --
> > > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > > >> >> >>>> Software Architect
> > > > > > > >> >> >>>> http://www.purbon.com
> > > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>
> > > > > > > >> >> >>>
> > > > > > > >> >> >>
> > > > > > > >> >> >> --
> > > > > > > >> >> >> Pere Urbon-Bayes
> > > > > > > >> >> >> Software Architect
> > > > > > > >> >> >> http://www.purbon.com
> > > > > > > >> >> >> https://twitter.com/purbon
> > > > > > > >> >> >> https://www.linkedin.com/in/purbon/
> > > > > > > >> >> >>
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >>
> > > > > > > >> >> --
> > > > > > > >> >> Pere Urbon-Bayes
> > > > > > > >> >> Software Architect
> > > > > > > >> >> http://www.purbon.com
> > > > > > > >> >> https://twitter.com/purbon
> > > > > > > >> >> https://www.linkedin.com/in/purbon/
> > > > > > > >> >>
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> Pere Urbon-Bayes
> > > > > > > >> Software Architect
> > > > > > > >> http://www.purbon.com
> > > > > > > >> https://twitter.com/purbon
> > > > > > > >> https://www.linkedin.com/in/purbon/
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Ron Dagostino <rn...@gmail.com>.
Thanks Colin and Rajini.
I've updated the KIP to say that KIP-421 functionality is available to
encrypt sensitive configs like the ZK key store and trust store
passwords. (I've also made it clear that the configs are not
dynamically reconfigurable since dynamic values are stored in ZK and
the data is needed to get to ZK in the first place). Colin, let me
know if you think anything else needs to be said/done about it -- I
wasn't sure if your comment above implies that there are additional
direct actions that we need to take aside from this.
I agree that making the brokers' ZooKeeper clients dynamic with
respect to key and trust stores (e.g. via
zookeeper.ssl.context.supplier.class) may increase the risk that this
KIP may not land in time for 2.5.
Regarding the config names for ZK being different than the ZooKeeper
configs (e.g. camel-case keyStore/trustStore instead of
keystore/truststore, ciphersuites instead of cipher.suites,
enabledProtocols instead of enabled.protocols), I agree it is an
issue. I tried to mitigate it by putting warning text in the config
docs for these. Regarding configuration inheritance, I think what you
are saying is that for any configs where the broker has a clear
semantic equivalent, the broker could fall back to the broker value if
the ZK value is not given. The potential list is:
zookeeper.ssl.keyStore.location => ssl.keystore.location
zookeeper.ssl.keyStore.password => ssl.keystore.password
zookeeper.ssl.keyStore.type => ssl.keystore.type
zookeeper.ssl.trustStore.location => ssl.truststore.location
zookeeper.ssl.trustStore.password => ssl.truststore.password
zookeeper.ssl.trustStore.type => ssl.truststore.type
zookeeper.ssl.ciphersuites => ssl.cipher.suites
Note that there are two configs that look the same based on their key
names but whose allowable values differ:
zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default: TLS)
zookeeper.ssl.enabledProtocols(Default: value of protocol property) =
ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
Not sure what the right answer is, but hopefully this detail will help
get us there.
Ron
On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <cm...@apache.org> wrote:
>
> On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> > Hi Ron,
> >
> > Thanks for the detailed explanation, sounds good to me.
> >
> > A few more questions:
> >
> > 1) At the moment, all sensitive broker configs including
> > keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
> > starting up brokers. We are now adding SSL keystore/truststore passwords
> > for ZooKeeper client that cannot be stored in ZooKeeper since you need
> > these to connect to ZK. We should perhaps document that these configs can
> > be encrypted using KIP-421.
>
> That's a good point. Previously, we didn't have ZK on-the-wire security support. However, now that we do, sending sensitive keystore and truststore passwords to ZK in cleartext should be deprecated in favor of using KIP-421.
>
> >
> > 2) We can dynamically update keystores and trust stores used by brokers
> > without restarting the broker. Can we support this easily for ZK clients
> > created by the broker, for example by adding our own
> > `zookeeper.ssl.context.supplier.class`?
> >
>
> Hmm. That might be better to consider in a follow-up, since the deadline for 2.5 KIPs is coming up?
>
> best,
> Colin
>
> > 3) Looks like we are using config names that directly map to ZK configs.
> > Have we considered using equivalent Kafka config names with prefixes,
> > perhaps with inheritance from the non-prefixed names? Not sure if this is a
> > good idea, but perhaps worth documenting in Rejected Alternatives at least.
> >
> >
> > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > Hi Ron,
> > >
> > > Thanks for the explanation. I guess thinking about it a little bit more,
> > > we should just add --zk-tls-config-file to all of these commands.
> > >
> > > We will be removing this option (plus ZK in general) from these commands
> > > in the next major release, but ZK is still supported in 2.5, so we should
> > > just do the logical thing. (The exception is ZkSecurityMigrator, which
> > > will stay).
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > > Hi Colin.
> > > >
> > > > <<< It seems like this [--zk-tls-config-file information] could just
> > > appear
> > > > in a configuration file, which all of these tools already accept (I
> > > think)
> > > >
> > > > ZkSecurityMigrator has no such property file facility; adding a
> > > > "--zk-tls-config-file" parameter is exactly for this purpose. If we add
> > > > that to ZkSecurityMigrator then it is trivial to add it to other commands
> > > > (the same code is simply reused; it ends up being just a few extra
> > > lines).
> > > > I do not see any parameter in the other two commands to adjust the ZK
> > > > connection; ConfigCommand accepts a "--command-config" flag, but
> > > according
> > > > to the code "This is used only with --bootstrap-server option for
> > > > describing and altering broker configs."
> > > >
> > > > I do agree there would be no need to add "--zk-tls-config-file" to
> > > > ReassignPartitionsCommand if its direct ZK connectivity is replaced in
> > > time
> > > > for the next release.
> > > >
> > > > ConfigCommand supports the "--bootstrap-server" option and will have its
> > > > direct ZooKeeper access formally deprecated as per KIP-555, but the
> > > special
> > > > use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> > > > prior to starting Kafka will still exist, so it feels like while
> > > > "--zk-tls-config-file" would never be used except for this use case it
> > > > could probably still be added for this particular situation.
> > > >
> > > > Ron
> > > >
> > > > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> > > > sent) did not go through. I am trying to get emails through now to move
> > > > this discussion forward.
> > > >
> > > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cm...@apache.org> wrote:
> > > >
> > > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > > > Hi everyone. I would like to make the following changes to the KIP.
> > > > > >
> > > > > > MOTIVATION:
> > > > > > Include a statement that it will be difficult in the short term to
> > > > > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > > > > (which
> > > > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
> > > cluster
> > > > > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > > > > case;
> > > > > > therefore it is in scope to be able to securely configure the CLI
> > > tools
> > > > > > that still leverage non-deprecated direct Zookeeper communication
> > > for TLS
> > > > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > > > > zookeeper-security-migration.sh).
> > > > >
> > > > > Hi Ron,
> > > > >
> > > > > Thanks for the KIP.
> > > > >
> > > > > About deprecations:
> > > > >
> > > > > * zookeeper-security-migration: clearly, deprecating ZK access in this
> > > one
> > > > > would not make sense, since it would defeat the whole point of the
> > > tool :)
> > > > >
> > > > > * kafka-reassign-partitions: ZK access should be deprecated here.
> > > KIP-455
> > > > > implementation has dragged a bit, but this should get done soon.
> > > Certainly
> > > > > before the next release.
> > > > >
> > > > > * kafka-configs: I think ZK access should be deprecated here as well.
> > > I
> > > > > agree there is a super-special bootstrapping case here, but that should
> > > > > have its own tool, not use kafka-configs.
> > > > >
> > > > > I will post a separate KIP for this, though.
> > > > >
> > > > > >
> > > > > > GOALS:
> > > > > > Support the secure configuration of TLS-encrypted communication
> > > between
> > > > > > Zookeeper and:
> > > > > > a) Kafka brokers
> > > > > > b) The three CLI tools mentioned above that still support direct,
> > > > > > non-deprecated communication to Zookeeper
> > > > > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > > > > communication in CLI tools as part of this KIP; such work will occur
> > > in
> > > > > > future KIPs instead.
> > > > > >
> > > > > > PUBLIC INTERFACES:
> > > > > > 1) The following new broker configurations will be recognized.
> > > > > > zookeeper.client.secure (default value = false, for backwards
> > > > > > compatibility)
> > > > > > zookeeper.clientCnxnSocket
> > > > > > zookeeper.ssl.keyStore.location
> > > > > > zookeeper.ssl.keyStore.password
> > > > > > zookeeper.ssl.trustStore.location
> > > > > > zookeeper.ssl.trustStore.password
> > > > > > It will be an error for any of the last 5 values to be left
> > > unspecified
> > > > > if
> > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > >
> > > > > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> > > > > supports
> > > > > > the ability to connect to a different Zookeeper instance than the
> > > one the
> > > > > > brokers use. We therefore also add the following optional configs,
> > > which
> > > > > > override the corresponding ones from above when present:
> > > > > > authorizer.zookeeper.client.secure
> > > > > > authorizer.zookeeper.clientCnxnSocket
> > > > > > authorizer.zookeeper.ssl.keyStore.location
> > > > > > authorizer.zookeeper.ssl.keyStore.password
> > > > > > authorizer.zookeeper.ssl.trustStore.location
> > > > > > authorizer.zookeeper.ssl.trustStore.password
> > > > > >
> > > > > > 3) The three CLI tools mentioned above will support a new
> > > > > --zk-tls-config-file
> > > > > > <String: Zookeeper TLS configuration file path>" option. The
> > > following
> > > > > > properties will be recognized in that file, and unrecognized
> > > properties
> > > > > > will be ignored to allow the possibility of pointing
> > > zk-tls-config-file
> > > > > at
> > > > > > the broker's config file.
> > > > > > zookeeper.client.secure (default value = false)
> > > > > > zookeeper.clientCnxnSocket
> > > > > > zookeeper.ssl.keyStore.location
> > > > > > zookeeper.ssl.keyStore.password
> > > > > > zookeeper.ssl.trustStore.location
> > > > > > zookeeper.ssl.trustStore.password
> > > > > > It will be an error for any of the last 5 values to be left
> > > unspecified
> > > > > if
> > > > > > zookeeper.client.secure is explicitly set to true.
> > > > >
> > > > > Do we really need a --zk-tls-config-file flag? It seems like this
> > > could
> > > > > just appear in a configuration file, which all of these tools already
> > > > > accept (I think).
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > >
> > > > > > Ron
> > > > > >
> > > > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rn...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Hi everyone. Let's get this discussion going again now that Kafka
> > > 2.4
> > > > > > > with Zookeeper 3.5.6 is out.
> > > > > > >
> > > > > > > First, regarding the KIP number, the other KIP that was using this
> > > > > number
> > > > > > > moved to KIP 534, so KIP 515 remains the correct number for this
> > > > > > > discussion. I've updated the Kafka Improvement Proposal page to
> > > list
> > > > > this
> > > > > > > KIP in the 515 slot, so we're all set there.
> > > > > > >
> > > > > > > Regarding the actual issues under discussion, I think there are
> > > some
> > > > > > > things we should clarify.
> > > > > > >
> > > > > > > 1) It is possible to use TLS connectivity to Zookeeper from Apache
> > > > > Kafka
> > > > > > > 2.4 -- the problem is that configuration information has to be
> > > passed
> > > > > via
> > > > > > > system properties as "-D" command line options on the java
> > > invocation
> > > > > of
> > > > > > > the broker, and those are not secure (anyone with access to the box
> > > > > can see
> > > > > > > the command line used to invoke the broker); the configuration
> > > includes
> > > > > > > sensitive information (e.g. a keystore password), so we need a
> > > secure
> > > > > > > mechanism for passing the configuration values. I believe the real
> > > > > > > motivation for this KIP is to harden the configuration mechanism
> > > for
> > > > > > > Zookeeper TLS connectivity.
> > > > > > >
> > > > > > > 2) I believe the list of CLI tools that continue to use direct
> > > > > Zookeeper
> > > > > > > connectivity in a non-deprecated fashion is:
> > > > > > > a)
> > > zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > > > > b)
> > > > > > >
> > > > >
> > > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > > > > > >
> > > > > > > 3) I believe Kafka.admin.ConfigCommand presents a conundrum
> > > because it
> > > > > > > explicitly states in a comment that a supported use case is
> > > > > bootstrapping a
> > > > > > > Kafka cluster with encrypted passwords in Zookeeper (see
> > > > > > >
> > > > >
> > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > > > > ).
> > > > > > > This means it will be especially difficult to deprecate this
> > > particular
> > > > > > > direct Zookeeper connectivity without a different storage
> > > mechanism for
> > > > > > > dynamic configuration values being available (i.e. the self-managed
> > > > > quorum
> > > > > > > referred to in KIP-500).
> > > > > > >
> > > > > > > I think it would be easier and simpler to harden the Zookeeper TLS
> > > > > > > configuration in both Kafka and in CLI tools compared to trying to
> > > > > > > deprecate the direct Zookeeper connectivity in the above 3 CLI
> > > tools --
> > > > > > > especially when ConfigCommand has no obvious short-term path to
> > > > > deprecation.
> > > > > > >
> > > > > > > Regarding how to harden the TLS configuration, adding the
> > > appropriate
> > > > > > > configs to Kafka is an obvious choice for the brokers. Not that
> > > these
> > > > > > > values cannot be dynamically reconfigurable because the values
> > > would be
> > > > > > > required to connect to Zookeeper via TLS in the first place.
> > > > > > >
> > > > > > > Regarding how to harden the TLS configuration for the 3 remaining
> > > CLI
> > > > > > > tools, it would be relatively straightforward to add support for a
> > > > > > > "--zk-tls-config-file <String: Zookeeper TLS configuration system
> > > > > > > properties file path>" option to each one.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > Ron
> > > > > > >
> > > > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <
> > > pere.urbon@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> True, the number is duplicated.
> > > > > > >>
> > > > > > >> do you know how can we get the problem fix? I was not aware, I
> > > > > missed,
> > > > > > >> sorry the need to add the KIP to the table.
> > > > > > >>
> > > > > > >> -- Pere
> > > > > > >>
> > > > > > >> Missatge de Jorge Esteban Quilcate Otoya <
> > > quilcate.jorge@gmail.com>
> > > > > del
> > > > > > >> dia
> > > > > > >> dt., 3 de set. 2019 a les 13:43:
> > > > > > >>
> > > > > > >> > Hi Pere,
> > > > > > >> >
> > > > > > >> > Have you add your KIP to the list here
> > > > > > >> >
> > > > > > >>
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > > >> ?
> > > > > > >> >
> > > > > > >> > I found the KIP number assigned to another.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> > > > > pere.urbon@gmail.com>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> >> Thanks for your time Harsha,
> > > > > > >> >> anyone else with comments? looking forward to hearing from
> > > you.
> > > > > > >> >>
> > > > > > >> >> Stupid question: when do you move from discussion to vote?
> > > > > > >> >>
> > > > > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> > > dv., 30
> > > > > > >> d’ag.
> > > > > > >> >> 2019 a les 21:59:
> > > > > > >> >>
> > > > > > >> >> > Thanks Pere. KIP looks good to me.
> > > > > > >> >> > -Harsha
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> > > > > > >> >> pere.urbon@gmail.com>
> > > > > > >> >> > wrote:
> > > > > > >> >> >
> > > > > > >> >> >> Not really,
> > > > > > >> >> >> my idea is to keep the JAAS parameter, so people don't see
> > > > > major
> > > > > > >> >> >> changes. But if you pass a properties file, then this takes
> > > > > > >> precedence
> > > > > > >> >> over
> > > > > > >> >> >> the other, with the idea that you can do sasl as well with
> > > the
> > > > > > >> >> properties
> > > > > > >> >> >> files.
> > > > > > >> >> >>
> > > > > > >> >> >> Makes sense?
> > > > > > >> >> >>
> > > > > > >> >> >> -- Pere
> > > > > > >> >> >>
> > > > > > >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> > > dv.,
> > > > > 30
> > > > > > >> >> d’ag.
> > > > > > >> >> >> 2019 a les 19:00:
> > > > > > >> >> >>
> > > > > > >> >> >>> Hi Pere,
> > > > > > >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper
> > > for
> > > > > Kafka
> > > > > > >> >> makes
> > > > > > >> >> >>> sense.
> > > > > > >> >> >>> "The changes are planned to be introduced in a compatible
> > > way,
> > > > > by
> > > > > > >> >> >>> keeping the current JAAS variable precedence."
> > > > > > >> >> >>> Can you elaborate a bit here. If the user configures a JAAS
> > > > > file
> > > > > > >> with
> > > > > > >> >> >>> Client section it will take precedence over zookeeper SSL
> > > > > configs?
> > > > > > >> >> >>>
> > > > > > >> >> >>> Thanks,
> > > > > > >> >> >>> Harsha
> > > > > > >> >> >>>
> > > > > > >> >> >>>
> > > > > > >> >> >>>
> > > > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> > > > > > >> >> pere.urbon@gmail.com>
> > > > > > >> >> >>> wrote:
> > > > > > >> >> >>>
> > > > > > >> >> >>>> Hi,
> > > > > > >> >> >>>> quick question, I saw in another mail that 2.4 release is
> > > > > planned
> > > > > > >> for
> > > > > > >> >> >>>> September. I think it would be really awesome to have
> > > this for
> > > > > > >> this
> > > > > > >> >> >>>> release, do you think we can make it?
> > > > > > >> >> >>>>
> > > > > > >> >> >>>> -- Pere
> > > > > > >> >> >>>>
> > > > > > >> >> >>>> Missatge de Pere Urbón Bayes <pe...@gmail.com> del
> > > dia
> > > > > dj.,
> > > > > > >> 29
> > > > > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > > > > >> >> >>>>
> > > > > > >> >> >>>> Hi,
> > > > > > >> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm
> > > > > really
> > > > > > >> need
> > > > > > >> >> >>>> to the process. Looking forward to hearing from you and
> > > learn
> > > > > the
> > > > > > >> >> best
> > > > > > >> >> >>>> ropes here.
> > > > > > >> >> >>>>
> > > > > > >> >> >>>> I would like to propose this KIP-515 to enable the
> > > > > > >> ZookeeperClients
> > > > > > >> >> to
> > > > > > >> >> >>>> take full advantage of the TLS communication in the new
> > > > > Zookeeper
> > > > > > >> >> 3.5.5.
> > > > > > >> >> >>>> Specially interesting it the Zookeeper Security Migration,
> > > > > that
> > > > > > >> >> without
> > > > > > >> >> >>>> this change will not work with TLS, disabling users to use
> > > > > ACLs
> > > > > > >> when
> > > > > > >> >> the
> > > > > > >> >> >>>> Zookeeper cluster use TLS.
> > > > > > >> >> >>>>
> > > > > > >> >> >>>> link:
> > > > > > >> >> >>>>
> > > > > > >> >> >>>>
> > > > > > >> >>
> > > > > > >>
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > > > > >> >> >>>>
> > > > > > >> >> >>>> Looking forward to hearing from you on this,
> > > > > > >> >> >>>>
> > > > > > >> >> >>>> /cheers
> > > > > > >> >> >>>>
> > > > > > >> >> >>>> --
> > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > >> >> >>>> Software Architect
> > > > > > >> >> >>>> http://www.purbon.com
> > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > >> >> >>>>
> > > > > > >> >> >>>> --
> > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > >> >> >>>> Software Architect
> > > > > > >> >> >>>> http://www.purbon.com
> > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > >> >> >>>>
> > > > > > >> >> >>>
> > > > > > >> >> >>>
> > > > > > >> >> >>
> > > > > > >> >> >> --
> > > > > > >> >> >> Pere Urbon-Bayes
> > > > > > >> >> >> Software Architect
> > > > > > >> >> >> http://www.purbon.com
> > > > > > >> >> >> https://twitter.com/purbon
> > > > > > >> >> >> https://www.linkedin.com/in/purbon/
> > > > > > >> >> >>
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >>
> > > > > > >> >> --
> > > > > > >> >> Pere Urbon-Bayes
> > > > > > >> >> Software Architect
> > > > > > >> >> http://www.purbon.com
> > > > > > >> >> https://twitter.com/purbon
> > > > > > >> >> https://www.linkedin.com/in/purbon/
> > > > > > >> >>
> > > > > > >> >
> > > > > > >>
> > > > > > >> --
> > > > > > >> Pere Urbon-Bayes
> > > > > > >> Software Architect
> > > > > > >> http://www.purbon.com
> > > > > > >> https://twitter.com/purbon
> > > > > > >> https://www.linkedin.com/in/purbon/
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Colin McCabe <cm...@apache.org>.
On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> Hi Ron,
>
> Thanks for the detailed explanation, sounds good to me.
>
> A few more questions:
>
> 1) At the moment, all sensitive broker configs including
> keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
> starting up brokers. We are now adding SSL keystore/truststore passwords
> for ZooKeeper client that cannot be stored in ZooKeeper since you need
> these to connect to ZK. We should perhaps document that these configs can
> be encrypted using KIP-421.
That's a good point. Previously, we didn't have ZK on-the-wire security support. However, now that we do, sending sensitive keystore and truststore passwords to ZK in cleartext should be deprecated in favor of using KIP-421.
>
> 2) We can dynamically update keystores and trust stores used by brokers
> without restarting the broker. Can we support this easily for ZK clients
> created by the broker, for example by adding our own
> `zookeeper.ssl.context.supplier.class`?
>
Hmm. That might be better to consider in a follow-up, since the deadline for 2.5 KIPs is coming up?
best,
Colin
> 3) Looks like we are using config names that directly map to ZK configs.
> Have we considered using equivalent Kafka config names with prefixes,
> perhaps with inheritance from the non-prefixed names? Not sure if this is a
> good idea, but perhaps worth documenting in Rejected Alternatives at least.
>
>
> On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <cm...@apache.org> wrote:
>
> > Hi Ron,
> >
> > Thanks for the explanation. I guess thinking about it a little bit more,
> > we should just add --zk-tls-config-file to all of these commands.
> >
> > We will be removing this option (plus ZK in general) from these commands
> > in the next major release, but ZK is still supported in 2.5, so we should
> > just do the logical thing. (The exception is ZkSecurityMigrator, which
> > will stay).
> >
> > best,
> > Colin
> >
> >
> > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > Hi Colin.
> > >
> > > <<< It seems like this [--zk-tls-config-file information] could just
> > appear
> > > in a configuration file, which all of these tools already accept (I
> > think)
> > >
> > > ZkSecurityMigrator has no such property file facility; adding a
> > > "--zk-tls-config-file" parameter is exactly for this purpose. If we add
> > > that to ZkSecurityMigrator then it is trivial to add it to other commands
> > > (the same code is simply reused; it ends up being just a few extra
> > lines).
> > > I do not see any parameter in the other two commands to adjust the ZK
> > > connection; ConfigCommand accepts a "--command-config" flag, but
> > according
> > > to the code "This is used only with --bootstrap-server option for
> > > describing and altering broker configs."
> > >
> > > I do agree there would be no need to add "--zk-tls-config-file" to
> > > ReassignPartitionsCommand if its direct ZK connectivity is replaced in
> > time
> > > for the next release.
> > >
> > > ConfigCommand supports the "--bootstrap-server" option and will have its
> > > direct ZooKeeper access formally deprecated as per KIP-555, but the
> > special
> > > use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> > > prior to starting Kafka will still exist, so it feels like while
> > > "--zk-tls-config-file" would never be used except for this use case it
> > > could probably still be added for this particular situation.
> > >
> > > Ron
> > >
> > > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> > > sent) did not go through. I am trying to get emails through now to move
> > > this discussion forward.
> > >
> > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cm...@apache.org> wrote:
> > >
> > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > > Hi everyone. I would like to make the following changes to the KIP.
> > > > >
> > > > > MOTIVATION:
> > > > > Include a statement that it will be difficult in the short term to
> > > > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > > > (which
> > > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
> > cluster
> > > > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > > > case;
> > > > > therefore it is in scope to be able to securely configure the CLI
> > tools
> > > > > that still leverage non-deprecated direct Zookeeper communication
> > for TLS
> > > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > > > zookeeper-security-migration.sh).
> > > >
> > > > Hi Ron,
> > > >
> > > > Thanks for the KIP.
> > > >
> > > > About deprecations:
> > > >
> > > > * zookeeper-security-migration: clearly, deprecating ZK access in this
> > one
> > > > would not make sense, since it would defeat the whole point of the
> > tool :)
> > > >
> > > > * kafka-reassign-partitions: ZK access should be deprecated here.
> > KIP-455
> > > > implementation has dragged a bit, but this should get done soon.
> > Certainly
> > > > before the next release.
> > > >
> > > > * kafka-configs: I think ZK access should be deprecated here as well.
> > I
> > > > agree there is a super-special bootstrapping case here, but that should
> > > > have its own tool, not use kafka-configs.
> > > >
> > > > I will post a separate KIP for this, though.
> > > >
> > > > >
> > > > > GOALS:
> > > > > Support the secure configuration of TLS-encrypted communication
> > between
> > > > > Zookeeper and:
> > > > > a) Kafka brokers
> > > > > b) The three CLI tools mentioned above that still support direct,
> > > > > non-deprecated communication to Zookeeper
> > > > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > > > communication in CLI tools as part of this KIP; such work will occur
> > in
> > > > > future KIPs instead.
> > > > >
> > > > > PUBLIC INTERFACES:
> > > > > 1) The following new broker configurations will be recognized.
> > > > > zookeeper.client.secure (default value = false, for backwards
> > > > > compatibility)
> > > > > zookeeper.clientCnxnSocket
> > > > > zookeeper.ssl.keyStore.location
> > > > > zookeeper.ssl.keyStore.password
> > > > > zookeeper.ssl.trustStore.location
> > > > > zookeeper.ssl.trustStore.password
> > > > > It will be an error for any of the last 5 values to be left
> > unspecified
> > > > if
> > > > > zookeeper.client.secure is explicitly set to true.
> > > > >
> > > > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> > > > supports
> > > > > the ability to connect to a different Zookeeper instance than the
> > one the
> > > > > brokers use. We therefore also add the following optional configs,
> > which
> > > > > override the corresponding ones from above when present:
> > > > > authorizer.zookeeper.client.secure
> > > > > authorizer.zookeeper.clientCnxnSocket
> > > > > authorizer.zookeeper.ssl.keyStore.location
> > > > > authorizer.zookeeper.ssl.keyStore.password
> > > > > authorizer.zookeeper.ssl.trustStore.location
> > > > > authorizer.zookeeper.ssl.trustStore.password
> > > > >
> > > > > 3) The three CLI tools mentioned above will support a new
> > > > --zk-tls-config-file
> > > > > <String: Zookeeper TLS configuration file path>" option. The
> > following
> > > > > properties will be recognized in that file, and unrecognized
> > properties
> > > > > will be ignored to allow the possibility of pointing
> > zk-tls-config-file
> > > > at
> > > > > the broker's config file.
> > > > > zookeeper.client.secure (default value = false)
> > > > > zookeeper.clientCnxnSocket
> > > > > zookeeper.ssl.keyStore.location
> > > > > zookeeper.ssl.keyStore.password
> > > > > zookeeper.ssl.trustStore.location
> > > > > zookeeper.ssl.trustStore.password
> > > > > It will be an error for any of the last 5 values to be left
> > unspecified
> > > > if
> > > > > zookeeper.client.secure is explicitly set to true.
> > > >
> > > > Do we really need a --zk-tls-config-file flag? It seems like this
> > could
> > > > just appear in a configuration file, which all of these tools already
> > > > accept (I think).
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > >
> > > > > Ron
> > > > >
> > > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rn...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hi everyone. Let's get this discussion going again now that Kafka
> > 2.4
> > > > > > with Zookeeper 3.5.6 is out.
> > > > > >
> > > > > > First, regarding the KIP number, the other KIP that was using this
> > > > number
> > > > > > moved to KIP 534, so KIP 515 remains the correct number for this
> > > > > > discussion. I've updated the Kafka Improvement Proposal page to
> > list
> > > > this
> > > > > > KIP in the 515 slot, so we're all set there.
> > > > > >
> > > > > > Regarding the actual issues under discussion, I think there are
> > some
> > > > > > things we should clarify.
> > > > > >
> > > > > > 1) It is possible to use TLS connectivity to Zookeeper from Apache
> > > > Kafka
> > > > > > 2.4 -- the problem is that configuration information has to be
> > passed
> > > > via
> > > > > > system properties as "-D" command line options on the java
> > invocation
> > > > of
> > > > > > the broker, and those are not secure (anyone with access to the box
> > > > can see
> > > > > > the command line used to invoke the broker); the configuration
> > includes
> > > > > > sensitive information (e.g. a keystore password), so we need a
> > secure
> > > > > > mechanism for passing the configuration values. I believe the real
> > > > > > motivation for this KIP is to harden the configuration mechanism
> > for
> > > > > > Zookeeper TLS connectivity.
> > > > > >
> > > > > > 2) I believe the list of CLI tools that continue to use direct
> > > > Zookeeper
> > > > > > connectivity in a non-deprecated fashion is:
> > > > > > a)
> > zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > > > b)
> > > > > >
> > > >
> > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > > > > >
> > > > > > 3) I believe Kafka.admin.ConfigCommand presents a conundrum
> > because it
> > > > > > explicitly states in a comment that a supported use case is
> > > > bootstrapping a
> > > > > > Kafka cluster with encrypted passwords in Zookeeper (see
> > > > > >
> > > >
> > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > > > ).
> > > > > > This means it will be especially difficult to deprecate this
> > particular
> > > > > > direct Zookeeper connectivity without a different storage
> > mechanism for
> > > > > > dynamic configuration values being available (i.e. the self-managed
> > > > quorum
> > > > > > referred to in KIP-500).
> > > > > >
> > > > > > I think it would be easier and simpler to harden the Zookeeper TLS
> > > > > > configuration in both Kafka and in CLI tools compared to trying to
> > > > > > deprecate the direct Zookeeper connectivity in the above 3 CLI
> > tools --
> > > > > > especially when ConfigCommand has no obvious short-term path to
> > > > deprecation.
> > > > > >
> > > > > > Regarding how to harden the TLS configuration, adding the
> > appropriate
> > > > > > configs to Kafka is an obvious choice for the brokers. Not that
> > these
> > > > > > values cannot be dynamically reconfigurable because the values
> > would be
> > > > > > required to connect to Zookeeper via TLS in the first place.
> > > > > >
> > > > > > Regarding how to harden the TLS configuration for the 3 remaining
> > CLI
> > > > > > tools, it would be relatively straightforward to add support for a
> > > > > > "--zk-tls-config-file <String: Zookeeper TLS configuration system
> > > > > > properties file path>" option to each one.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > Ron
> > > > > >
> > > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <
> > pere.urbon@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> True, the number is duplicated.
> > > > > >>
> > > > > >> do you know how can we get the problem fix? I was not aware, I
> > > > missed,
> > > > > >> sorry the need to add the KIP to the table.
> > > > > >>
> > > > > >> -- Pere
> > > > > >>
> > > > > >> Missatge de Jorge Esteban Quilcate Otoya <
> > quilcate.jorge@gmail.com>
> > > > del
> > > > > >> dia
> > > > > >> dt., 3 de set. 2019 a les 13:43:
> > > > > >>
> > > > > >> > Hi Pere,
> > > > > >> >
> > > > > >> > Have you add your KIP to the list here
> > > > > >> >
> > > > > >>
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > >> ?
> > > > > >> >
> > > > > >> > I found the KIP number assigned to another.
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> > > > pere.urbon@gmail.com>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> >> Thanks for your time Harsha,
> > > > > >> >> anyone else with comments? looking forward to hearing from
> > you.
> > > > > >> >>
> > > > > >> >> Stupid question: when do you move from discussion to vote?
> > > > > >> >>
> > > > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> > dv., 30
> > > > > >> d’ag.
> > > > > >> >> 2019 a les 21:59:
> > > > > >> >>
> > > > > >> >> > Thanks Pere. KIP looks good to me.
> > > > > >> >> > -Harsha
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> > > > > >> >> pere.urbon@gmail.com>
> > > > > >> >> > wrote:
> > > > > >> >> >
> > > > > >> >> >> Not really,
> > > > > >> >> >> my idea is to keep the JAAS parameter, so people don't see
> > > > major
> > > > > >> >> >> changes. But if you pass a properties file, then this takes
> > > > > >> precedence
> > > > > >> >> over
> > > > > >> >> >> the other, with the idea that you can do sasl as well with
> > the
> > > > > >> >> properties
> > > > > >> >> >> files.
> > > > > >> >> >>
> > > > > >> >> >> Makes sense?
> > > > > >> >> >>
> > > > > >> >> >> -- Pere
> > > > > >> >> >>
> > > > > >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> > dv.,
> > > > 30
> > > > > >> >> d’ag.
> > > > > >> >> >> 2019 a les 19:00:
> > > > > >> >> >>
> > > > > >> >> >>> Hi Pere,
> > > > > >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper
> > for
> > > > Kafka
> > > > > >> >> makes
> > > > > >> >> >>> sense.
> > > > > >> >> >>> "The changes are planned to be introduced in a compatible
> > way,
> > > > by
> > > > > >> >> >>> keeping the current JAAS variable precedence."
> > > > > >> >> >>> Can you elaborate a bit here. If the user configures a JAAS
> > > > file
> > > > > >> with
> > > > > >> >> >>> Client section it will take precedence over zookeeper SSL
> > > > configs?
> > > > > >> >> >>>
> > > > > >> >> >>> Thanks,
> > > > > >> >> >>> Harsha
> > > > > >> >> >>>
> > > > > >> >> >>>
> > > > > >> >> >>>
> > > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> > > > > >> >> pere.urbon@gmail.com>
> > > > > >> >> >>> wrote:
> > > > > >> >> >>>
> > > > > >> >> >>>> Hi,
> > > > > >> >> >>>> quick question, I saw in another mail that 2.4 release is
> > > > planned
> > > > > >> for
> > > > > >> >> >>>> September. I think it would be really awesome to have
> > this for
> > > > > >> this
> > > > > >> >> >>>> release, do you think we can make it?
> > > > > >> >> >>>>
> > > > > >> >> >>>> -- Pere
> > > > > >> >> >>>>
> > > > > >> >> >>>> Missatge de Pere Urbón Bayes <pe...@gmail.com> del
> > dia
> > > > dj.,
> > > > > >> 29
> > > > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > > > >> >> >>>>
> > > > > >> >> >>>> Hi,
> > > > > >> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm
> > > > really
> > > > > >> need
> > > > > >> >> >>>> to the process. Looking forward to hearing from you and
> > learn
> > > > the
> > > > > >> >> best
> > > > > >> >> >>>> ropes here.
> > > > > >> >> >>>>
> > > > > >> >> >>>> I would like to propose this KIP-515 to enable the
> > > > > >> ZookeeperClients
> > > > > >> >> to
> > > > > >> >> >>>> take full advantage of the TLS communication in the new
> > > > Zookeeper
> > > > > >> >> 3.5.5.
> > > > > >> >> >>>> Specially interesting it the Zookeeper Security Migration,
> > > > that
> > > > > >> >> without
> > > > > >> >> >>>> this change will not work with TLS, disabling users to use
> > > > ACLs
> > > > > >> when
> > > > > >> >> the
> > > > > >> >> >>>> Zookeeper cluster use TLS.
> > > > > >> >> >>>>
> > > > > >> >> >>>> link:
> > > > > >> >> >>>>
> > > > > >> >> >>>>
> > > > > >> >>
> > > > > >>
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > > > >> >> >>>>
> > > > > >> >> >>>> Looking forward to hearing from you on this,
> > > > > >> >> >>>>
> > > > > >> >> >>>> /cheers
> > > > > >> >> >>>>
> > > > > >> >> >>>> --
> > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > >> >> >>>> Software Architect
> > > > > >> >> >>>> http://www.purbon.com
> > > > > >> >> >>>> https://twitter.com/purbon
> > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > >> >> >>>>
> > > > > >> >> >>>> --
> > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > >> >> >>>> Software Architect
> > > > > >> >> >>>> http://www.purbon.com
> > > > > >> >> >>>> https://twitter.com/purbon
> > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > >> >> >>>>
> > > > > >> >> >>>
> > > > > >> >> >>>
> > > > > >> >> >>
> > > > > >> >> >> --
> > > > > >> >> >> Pere Urbon-Bayes
> > > > > >> >> >> Software Architect
> > > > > >> >> >> http://www.purbon.com
> > > > > >> >> >> https://twitter.com/purbon
> > > > > >> >> >> https://www.linkedin.com/in/purbon/
> > > > > >> >> >>
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >>
> > > > > >> >> --
> > > > > >> >> Pere Urbon-Bayes
> > > > > >> >> Software Architect
> > > > > >> >> http://www.purbon.com
> > > > > >> >> https://twitter.com/purbon
> > > > > >> >> https://www.linkedin.com/in/purbon/
> > > > > >> >>
> > > > > >> >
> > > > > >>
> > > > > >> --
> > > > > >> Pere Urbon-Bayes
> > > > > >> Software Architect
> > > > > >> http://www.purbon.com
> > > > > >> https://twitter.com/purbon
> > > > > >> https://www.linkedin.com/in/purbon/
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Ron,
Thanks for the detailed explanation, sounds good to me.
A few more questions:
1) At the moment, all sensitive broker configs including
keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
starting up brokers. We are now adding SSL keystore/truststore passwords
for ZooKeeper client that cannot be stored in ZooKeeper since you need
these to connect to ZK. We should perhaps document that these configs can
be encrypted using KIP-421.
2) We can dynamically update keystores and trust stores used by brokers
without restarting the broker. Can we support this easily for ZK clients
created by the broker, for example by adding our own
`zookeeper.ssl.context.supplier.class`?
3) Looks like we are using config names that directly map to ZK configs.
Have we considered using equivalent Kafka config names with prefixes,
perhaps with inheritance from the non-prefixed names? Not sure if this is a
good idea, but perhaps worth documenting in Rejected Alternatives at least.
On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <cm...@apache.org> wrote:
> Hi Ron,
>
> Thanks for the explanation. I guess thinking about it a little bit more,
> we should just add --zk-tls-config-file to all of these commands.
>
> We will be removing this option (plus ZK in general) from these commands
> in the next major release, but ZK is still supported in 2.5, so we should
> just do the logical thing. (The exception is ZkSecurityMigrator, which
> will stay).
>
> best,
> Colin
>
>
> On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > Hi Colin.
> >
> > <<< It seems like this [--zk-tls-config-file information] could just
> appear
> > in a configuration file, which all of these tools already accept (I
> think)
> >
> > ZkSecurityMigrator has no such property file facility; adding a
> > "--zk-tls-config-file" parameter is exactly for this purpose. If we add
> > that to ZkSecurityMigrator then it is trivial to add it to other commands
> > (the same code is simply reused; it ends up being just a few extra
> lines).
> > I do not see any parameter in the other two commands to adjust the ZK
> > connection; ConfigCommand accepts a "--command-config" flag, but
> according
> > to the code "This is used only with --bootstrap-server option for
> > describing and altering broker configs."
> >
> > I do agree there would be no need to add "--zk-tls-config-file" to
> > ReassignPartitionsCommand if its direct ZK connectivity is replaced in
> time
> > for the next release.
> >
> > ConfigCommand supports the "--bootstrap-server" option and will have its
> > direct ZooKeeper access formally deprecated as per KIP-555, but the
> special
> > use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> > prior to starting Kafka will still exist, so it feels like while
> > "--zk-tls-config-file" would never be used except for this use case it
> > could probably still be added for this particular situation.
> >
> > Ron
> >
> > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> > sent) did not go through. I am trying to get emails through now to move
> > this discussion forward.
> >
> > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > Hi everyone. I would like to make the following changes to the KIP.
> > > >
> > > > MOTIVATION:
> > > > Include a statement that it will be difficult in the short term to
> > > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > > (which
> > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
> cluster
> > > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > > case;
> > > > therefore it is in scope to be able to securely configure the CLI
> tools
> > > > that still leverage non-deprecated direct Zookeeper communication
> for TLS
> > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > > zookeeper-security-migration.sh).
> > >
> > > Hi Ron,
> > >
> > > Thanks for the KIP.
> > >
> > > About deprecations:
> > >
> > > * zookeeper-security-migration: clearly, deprecating ZK access in this
> one
> > > would not make sense, since it would defeat the whole point of the
> tool :)
> > >
> > > * kafka-reassign-partitions: ZK access should be deprecated here.
> KIP-455
> > > implementation has dragged a bit, but this should get done soon.
> Certainly
> > > before the next release.
> > >
> > > * kafka-configs: I think ZK access should be deprecated here as well.
> I
> > > agree there is a super-special bootstrapping case here, but that should
> > > have its own tool, not use kafka-configs.
> > >
> > > I will post a separate KIP for this, though.
> > >
> > > >
> > > > GOALS:
> > > > Support the secure configuration of TLS-encrypted communication
> between
> > > > Zookeeper and:
> > > > a) Kafka brokers
> > > > b) The three CLI tools mentioned above that still support direct,
> > > > non-deprecated communication to Zookeeper
> > > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > > communication in CLI tools as part of this KIP; such work will occur
> in
> > > > future KIPs instead.
> > > >
> > > > PUBLIC INTERFACES:
> > > > 1) The following new broker configurations will be recognized.
> > > > zookeeper.client.secure (default value = false, for backwards
> > > > compatibility)
> > > > zookeeper.clientCnxnSocket
> > > > zookeeper.ssl.keyStore.location
> > > > zookeeper.ssl.keyStore.password
> > > > zookeeper.ssl.trustStore.location
> > > > zookeeper.ssl.trustStore.password
> > > > It will be an error for any of the last 5 values to be left
> unspecified
> > > if
> > > > zookeeper.client.secure is explicitly set to true.
> > > >
> > > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> > > supports
> > > > the ability to connect to a different Zookeeper instance than the
> one the
> > > > brokers use. We therefore also add the following optional configs,
> which
> > > > override the corresponding ones from above when present:
> > > > authorizer.zookeeper.client.secure
> > > > authorizer.zookeeper.clientCnxnSocket
> > > > authorizer.zookeeper.ssl.keyStore.location
> > > > authorizer.zookeeper.ssl.keyStore.password
> > > > authorizer.zookeeper.ssl.trustStore.location
> > > > authorizer.zookeeper.ssl.trustStore.password
> > > >
> > > > 3) The three CLI tools mentioned above will support a new
> > > --zk-tls-config-file
> > > > <String: Zookeeper TLS configuration file path>" option. The
> following
> > > > properties will be recognized in that file, and unrecognized
> properties
> > > > will be ignored to allow the possibility of pointing
> zk-tls-config-file
> > > at
> > > > the broker's config file.
> > > > zookeeper.client.secure (default value = false)
> > > > zookeeper.clientCnxnSocket
> > > > zookeeper.ssl.keyStore.location
> > > > zookeeper.ssl.keyStore.password
> > > > zookeeper.ssl.trustStore.location
> > > > zookeeper.ssl.trustStore.password
> > > > It will be an error for any of the last 5 values to be left
> unspecified
> > > if
> > > > zookeeper.client.secure is explicitly set to true.
> > >
> > > Do we really need a --zk-tls-config-file flag? It seems like this
> could
> > > just appear in a configuration file, which all of these tools already
> > > accept (I think).
> > >
> > > best,
> > > Colin
> > >
> > >
> > > >
> > > > Ron
> > > >
> > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rn...@gmail.com>
> wrote:
> > > >
> > > > > Hi everyone. Let's get this discussion going again now that Kafka
> 2.4
> > > > > with Zookeeper 3.5.6 is out.
> > > > >
> > > > > First, regarding the KIP number, the other KIP that was using this
> > > number
> > > > > moved to KIP 534, so KIP 515 remains the correct number for this
> > > > > discussion. I've updated the Kafka Improvement Proposal page to
> list
> > > this
> > > > > KIP in the 515 slot, so we're all set there.
> > > > >
> > > > > Regarding the actual issues under discussion, I think there are
> some
> > > > > things we should clarify.
> > > > >
> > > > > 1) It is possible to use TLS connectivity to Zookeeper from Apache
> > > Kafka
> > > > > 2.4 -- the problem is that configuration information has to be
> passed
> > > via
> > > > > system properties as "-D" command line options on the java
> invocation
> > > of
> > > > > the broker, and those are not secure (anyone with access to the box
> > > can see
> > > > > the command line used to invoke the broker); the configuration
> includes
> > > > > sensitive information (e.g. a keystore password), so we need a
> secure
> > > > > mechanism for passing the configuration values. I believe the real
> > > > > motivation for this KIP is to harden the configuration mechanism
> for
> > > > > Zookeeper TLS connectivity.
> > > > >
> > > > > 2) I believe the list of CLI tools that continue to use direct
> > > Zookeeper
> > > > > connectivity in a non-deprecated fashion is:
> > > > > a)
> zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > > b)
> > > > >
> > >
> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > > > >
> > > > > 3) I believe Kafka.admin.ConfigCommand presents a conundrum
> because it
> > > > > explicitly states in a comment that a supported use case is
> > > bootstrapping a
> > > > > Kafka cluster with encrypted passwords in Zookeeper (see
> > > > >
> > >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > > ).
> > > > > This means it will be especially difficult to deprecate this
> particular
> > > > > direct Zookeeper connectivity without a different storage
> mechanism for
> > > > > dynamic configuration values being available (i.e. the self-managed
> > > quorum
> > > > > referred to in KIP-500).
> > > > >
> > > > > I think it would be easier and simpler to harden the Zookeeper TLS
> > > > > configuration in both Kafka and in CLI tools compared to trying to
> > > > > deprecate the direct Zookeeper connectivity in the above 3 CLI
> tools --
> > > > > especially when ConfigCommand has no obvious short-term path to
> > > deprecation.
> > > > >
> > > > > Regarding how to harden the TLS configuration, adding the
> appropriate
> > > > > configs to Kafka is an obvious choice for the brokers. Not that
> these
> > > > > values cannot be dynamically reconfigurable because the values
> would be
> > > > > required to connect to Zookeeper via TLS in the first place.
> > > > >
> > > > > Regarding how to harden the TLS configuration for the 3 remaining
> CLI
> > > > > tools, it would be relatively straightforward to add support for a
> > > > > "--zk-tls-config-file <String: Zookeeper TLS configuration system
> > > > > properties file path>" option to each one.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Ron
> > > > >
> > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <
> pere.urbon@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > >> True, the number is duplicated.
> > > > >>
> > > > >> do you know how can we get the problem fix? I was not aware, I
> > > missed,
> > > > >> sorry the need to add the KIP to the table.
> > > > >>
> > > > >> -- Pere
> > > > >>
> > > > >> Missatge de Jorge Esteban Quilcate Otoya <
> quilcate.jorge@gmail.com>
> > > del
> > > > >> dia
> > > > >> dt., 3 de set. 2019 a les 13:43:
> > > > >>
> > > > >> > Hi Pere,
> > > > >> >
> > > > >> > Have you add your KIP to the list here
> > > > >> >
> > > > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > >> ?
> > > > >> >
> > > > >> > I found the KIP number assigned to another.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> > > pere.urbon@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> >> Thanks for your time Harsha,
> > > > >> >> anyone else with comments? looking forward to hearing from
> you.
> > > > >> >>
> > > > >> >> Stupid question: when do you move from discussion to vote?
> > > > >> >>
> > > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> dv., 30
> > > > >> d’ag.
> > > > >> >> 2019 a les 21:59:
> > > > >> >>
> > > > >> >> > Thanks Pere. KIP looks good to me.
> > > > >> >> > -Harsha
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> > > > >> >> pere.urbon@gmail.com>
> > > > >> >> > wrote:
> > > > >> >> >
> > > > >> >> >> Not really,
> > > > >> >> >> my idea is to keep the JAAS parameter, so people don't see
> > > major
> > > > >> >> >> changes. But if you pass a properties file, then this takes
> > > > >> precedence
> > > > >> >> over
> > > > >> >> >> the other, with the idea that you can do sasl as well with
> the
> > > > >> >> properties
> > > > >> >> >> files.
> > > > >> >> >>
> > > > >> >> >> Makes sense?
> > > > >> >> >>
> > > > >> >> >> -- Pere
> > > > >> >> >>
> > > > >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> dv.,
> > > 30
> > > > >> >> d’ag.
> > > > >> >> >> 2019 a les 19:00:
> > > > >> >> >>
> > > > >> >> >>> Hi Pere,
> > > > >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper
> for
> > > Kafka
> > > > >> >> makes
> > > > >> >> >>> sense.
> > > > >> >> >>> "The changes are planned to be introduced in a compatible
> way,
> > > by
> > > > >> >> >>> keeping the current JAAS variable precedence."
> > > > >> >> >>> Can you elaborate a bit here. If the user configures a JAAS
> > > file
> > > > >> with
> > > > >> >> >>> Client section it will take precedence over zookeeper SSL
> > > configs?
> > > > >> >> >>>
> > > > >> >> >>> Thanks,
> > > > >> >> >>> Harsha
> > > > >> >> >>>
> > > > >> >> >>>
> > > > >> >> >>>
> > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> > > > >> >> pere.urbon@gmail.com>
> > > > >> >> >>> wrote:
> > > > >> >> >>>
> > > > >> >> >>>> Hi,
> > > > >> >> >>>> quick question, I saw in another mail that 2.4 release is
> > > planned
> > > > >> for
> > > > >> >> >>>> September. I think it would be really awesome to have
> this for
> > > > >> this
> > > > >> >> >>>> release, do you think we can make it?
> > > > >> >> >>>>
> > > > >> >> >>>> -- Pere
> > > > >> >> >>>>
> > > > >> >> >>>> Missatge de Pere Urbón Bayes <pe...@gmail.com> del
> dia
> > > dj.,
> > > > >> 29
> > > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > > >> >> >>>>
> > > > >> >> >>>> Hi,
> > > > >> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm
> > > really
> > > > >> need
> > > > >> >> >>>> to the process. Looking forward to hearing from you and
> learn
> > > the
> > > > >> >> best
> > > > >> >> >>>> ropes here.
> > > > >> >> >>>>
> > > > >> >> >>>> I would like to propose this KIP-515 to enable the
> > > > >> ZookeeperClients
> > > > >> >> to
> > > > >> >> >>>> take full advantage of the TLS communication in the new
> > > Zookeeper
> > > > >> >> 3.5.5.
> > > > >> >> >>>> Specially interesting it the Zookeeper Security Migration,
> > > that
> > > > >> >> without
> > > > >> >> >>>> this change will not work with TLS, disabling users to use
> > > ACLs
> > > > >> when
> > > > >> >> the
> > > > >> >> >>>> Zookeeper cluster use TLS.
> > > > >> >> >>>>
> > > > >> >> >>>> link:
> > > > >> >> >>>>
> > > > >> >> >>>>
> > > > >> >>
> > > > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > > >> >> >>>>
> > > > >> >> >>>> Looking forward to hearing from you on this,
> > > > >> >> >>>>
> > > > >> >> >>>> /cheers
> > > > >> >> >>>>
> > > > >> >> >>>> --
> > > > >> >> >>>> Pere Urbon-Bayes
> > > > >> >> >>>> Software Architect
> > > > >> >> >>>> http://www.purbon.com
> > > > >> >> >>>> https://twitter.com/purbon
> > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > >> >> >>>>
> > > > >> >> >>>> --
> > > > >> >> >>>> Pere Urbon-Bayes
> > > > >> >> >>>> Software Architect
> > > > >> >> >>>> http://www.purbon.com
> > > > >> >> >>>> https://twitter.com/purbon
> > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > >> >> >>>>
> > > > >> >> >>>
> > > > >> >> >>>
> > > > >> >> >>
> > > > >> >> >> --
> > > > >> >> >> Pere Urbon-Bayes
> > > > >> >> >> Software Architect
> > > > >> >> >> http://www.purbon.com
> > > > >> >> >> https://twitter.com/purbon
> > > > >> >> >> https://www.linkedin.com/in/purbon/
> > > > >> >> >>
> > > > >> >> >
> > > > >> >> >
> > > > >> >>
> > > > >> >> --
> > > > >> >> Pere Urbon-Bayes
> > > > >> >> Software Architect
> > > > >> >> http://www.purbon.com
> > > > >> >> https://twitter.com/purbon
> > > > >> >> https://www.linkedin.com/in/purbon/
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >> --
> > > > >> Pere Urbon-Bayes
> > > > >> Software Architect
> > > > >> http://www.purbon.com
> > > > >> https://twitter.com/purbon
> > > > >> https://www.linkedin.com/in/purbon/
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Colin McCabe <cm...@apache.org>.
Hi Ron,
Thanks for the explanation. I guess thinking about it a little bit more, we should just add --zk-tls-config-file to all of these commands.
We will be removing this option (plus ZK in general) from these commands in the next major release, but ZK is still supported in 2.5, so we should just do the logical thing. (The exception is ZkSecurityMigrator, which will stay).
best,
Colin
On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> Hi Colin.
>
> <<< It seems like this [--zk-tls-config-file information] could just appear
> in a configuration file, which all of these tools already accept (I think)
>
> ZkSecurityMigrator has no such property file facility; adding a
> "--zk-tls-config-file" parameter is exactly for this purpose. If we add
> that to ZkSecurityMigrator then it is trivial to add it to other commands
> (the same code is simply reused; it ends up being just a few extra lines).
> I do not see any parameter in the other two commands to adjust the ZK
> connection; ConfigCommand accepts a "--command-config" flag, but according
> to the code "This is used only with --bootstrap-server option for
> describing and altering broker configs."
>
> I do agree there would be no need to add "--zk-tls-config-file" to
> ReassignPartitionsCommand if its direct ZK connectivity is replaced in time
> for the next release.
>
> ConfigCommand supports the "--bootstrap-server" option and will have its
> direct ZooKeeper access formally deprecated as per KIP-555, but the special
> use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> prior to starting Kafka will still exist, so it feels like while
> "--zk-tls-config-file" would never be used except for this use case it
> could probably still be added for this particular situation.
>
> Ron
>
> P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> sent) did not go through. I am trying to get emails through now to move
> this discussion forward.
>
> On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cm...@apache.org> wrote:
>
> > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > Hi everyone. I would like to make the following changes to the KIP.
> > >
> > > MOTIVATION:
> > > Include a statement that it will be difficult in the short term to
> > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > (which
> > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka cluster
> > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > case;
> > > therefore it is in scope to be able to securely configure the CLI tools
> > > that still leverage non-deprecated direct Zookeeper communication for TLS
> > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > zookeeper-security-migration.sh).
> >
> > Hi Ron,
> >
> > Thanks for the KIP.
> >
> > About deprecations:
> >
> > * zookeeper-security-migration: clearly, deprecating ZK access in this one
> > would not make sense, since it would defeat the whole point of the tool :)
> >
> > * kafka-reassign-partitions: ZK access should be deprecated here. KIP-455
> > implementation has dragged a bit, but this should get done soon. Certainly
> > before the next release.
> >
> > * kafka-configs: I think ZK access should be deprecated here as well. I
> > agree there is a super-special bootstrapping case here, but that should
> > have its own tool, not use kafka-configs.
> >
> > I will post a separate KIP for this, though.
> >
> > >
> > > GOALS:
> > > Support the secure configuration of TLS-encrypted communication between
> > > Zookeeper and:
> > > a) Kafka brokers
> > > b) The three CLI tools mentioned above that still support direct,
> > > non-deprecated communication to Zookeeper
> > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > communication in CLI tools as part of this KIP; such work will occur in
> > > future KIPs instead.
> > >
> > > PUBLIC INTERFACES:
> > > 1) The following new broker configurations will be recognized.
> > > zookeeper.client.secure (default value = false, for backwards
> > > compatibility)
> > > zookeeper.clientCnxnSocket
> > > zookeeper.ssl.keyStore.location
> > > zookeeper.ssl.keyStore.password
> > > zookeeper.ssl.trustStore.location
> > > zookeeper.ssl.trustStore.password
> > > It will be an error for any of the last 5 values to be left unspecified
> > if
> > > zookeeper.client.secure is explicitly set to true.
> > >
> > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> > supports
> > > the ability to connect to a different Zookeeper instance than the one the
> > > brokers use. We therefore also add the following optional configs, which
> > > override the corresponding ones from above when present:
> > > authorizer.zookeeper.client.secure
> > > authorizer.zookeeper.clientCnxnSocket
> > > authorizer.zookeeper.ssl.keyStore.location
> > > authorizer.zookeeper.ssl.keyStore.password
> > > authorizer.zookeeper.ssl.trustStore.location
> > > authorizer.zookeeper.ssl.trustStore.password
> > >
> > > 3) The three CLI tools mentioned above will support a new
> > --zk-tls-config-file
> > > <String: Zookeeper TLS configuration file path>" option. The following
> > > properties will be recognized in that file, and unrecognized properties
> > > will be ignored to allow the possibility of pointing zk-tls-config-file
> > at
> > > the broker's config file.
> > > zookeeper.client.secure (default value = false)
> > > zookeeper.clientCnxnSocket
> > > zookeeper.ssl.keyStore.location
> > > zookeeper.ssl.keyStore.password
> > > zookeeper.ssl.trustStore.location
> > > zookeeper.ssl.trustStore.password
> > > It will be an error for any of the last 5 values to be left unspecified
> > if
> > > zookeeper.client.secure is explicitly set to true.
> >
> > Do we really need a --zk-tls-config-file flag? It seems like this could
> > just appear in a configuration file, which all of these tools already
> > accept (I think).
> >
> > best,
> > Colin
> >
> >
> > >
> > > Ron
> > >
> > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rn...@gmail.com> wrote:
> > >
> > > > Hi everyone. Let's get this discussion going again now that Kafka 2.4
> > > > with Zookeeper 3.5.6 is out.
> > > >
> > > > First, regarding the KIP number, the other KIP that was using this
> > number
> > > > moved to KIP 534, so KIP 515 remains the correct number for this
> > > > discussion. I've updated the Kafka Improvement Proposal page to list
> > this
> > > > KIP in the 515 slot, so we're all set there.
> > > >
> > > > Regarding the actual issues under discussion, I think there are some
> > > > things we should clarify.
> > > >
> > > > 1) It is possible to use TLS connectivity to Zookeeper from Apache
> > Kafka
> > > > 2.4 -- the problem is that configuration information has to be passed
> > via
> > > > system properties as "-D" command line options on the java invocation
> > of
> > > > the broker, and those are not secure (anyone with access to the box
> > can see
> > > > the command line used to invoke the broker); the configuration includes
> > > > sensitive information (e.g. a keystore password), so we need a secure
> > > > mechanism for passing the configuration values. I believe the real
> > > > motivation for this KIP is to harden the configuration mechanism for
> > > > Zookeeper TLS connectivity.
> > > >
> > > > 2) I believe the list of CLI tools that continue to use direct
> > Zookeeper
> > > > connectivity in a non-deprecated fashion is:
> > > > a) zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > b)
> > > >
> > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > > >
> > > > 3) I believe Kafka.admin.ConfigCommand presents a conundrum because it
> > > > explicitly states in a comment that a supported use case is
> > bootstrapping a
> > > > Kafka cluster with encrypted passwords in Zookeeper (see
> > > >
> > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > ).
> > > > This means it will be especially difficult to deprecate this particular
> > > > direct Zookeeper connectivity without a different storage mechanism for
> > > > dynamic configuration values being available (i.e. the self-managed
> > quorum
> > > > referred to in KIP-500).
> > > >
> > > > I think it would be easier and simpler to harden the Zookeeper TLS
> > > > configuration in both Kafka and in CLI tools compared to trying to
> > > > deprecate the direct Zookeeper connectivity in the above 3 CLI tools --
> > > > especially when ConfigCommand has no obvious short-term path to
> > deprecation.
> > > >
> > > > Regarding how to harden the TLS configuration, adding the appropriate
> > > > configs to Kafka is an obvious choice for the brokers. Not that these
> > > > values cannot be dynamically reconfigurable because the values would be
> > > > required to connect to Zookeeper via TLS in the first place.
> > > >
> > > > Regarding how to harden the TLS configuration for the 3 remaining CLI
> > > > tools, it would be relatively straightforward to add support for a
> > > > "--zk-tls-config-file <String: Zookeeper TLS configuration system
> > > > properties file path>" option to each one.
> > > >
> > > > Thoughts?
> > > >
> > > > Ron
> > > >
> > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <pere.urbon@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> True, the number is duplicated.
> > > >>
> > > >> do you know how can we get the problem fix? I was not aware, I
> > missed,
> > > >> sorry the need to add the KIP to the table.
> > > >>
> > > >> -- Pere
> > > >>
> > > >> Missatge de Jorge Esteban Quilcate Otoya <qu...@gmail.com>
> > del
> > > >> dia
> > > >> dt., 3 de set. 2019 a les 13:43:
> > > >>
> > > >> > Hi Pere,
> > > >> >
> > > >> > Have you add your KIP to the list here
> > > >> >
> > > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > >> ?
> > > >> >
> > > >> > I found the KIP number assigned to another.
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> > pere.urbon@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> >> Thanks for your time Harsha,
> > > >> >> anyone else with comments? looking forward to hearing from you.
> > > >> >>
> > > >> >> Stupid question: when do you move from discussion to vote?
> > > >> >>
> > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv., 30
> > > >> d’ag.
> > > >> >> 2019 a les 21:59:
> > > >> >>
> > > >> >> > Thanks Pere. KIP looks good to me.
> > > >> >> > -Harsha
> > > >> >> >
> > > >> >> >
> > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> > > >> >> pere.urbon@gmail.com>
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> >> Not really,
> > > >> >> >> my idea is to keep the JAAS parameter, so people don't see
> > major
> > > >> >> >> changes. But if you pass a properties file, then this takes
> > > >> precedence
> > > >> >> over
> > > >> >> >> the other, with the idea that you can do sasl as well with the
> > > >> >> properties
> > > >> >> >> files.
> > > >> >> >>
> > > >> >> >> Makes sense?
> > > >> >> >>
> > > >> >> >> -- Pere
> > > >> >> >>
> > > >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv.,
> > 30
> > > >> >> d’ag.
> > > >> >> >> 2019 a les 19:00:
> > > >> >> >>
> > > >> >> >>> Hi Pere,
> > > >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper for
> > Kafka
> > > >> >> makes
> > > >> >> >>> sense.
> > > >> >> >>> "The changes are planned to be introduced in a compatible way,
> > by
> > > >> >> >>> keeping the current JAAS variable precedence."
> > > >> >> >>> Can you elaborate a bit here. If the user configures a JAAS
> > file
> > > >> with
> > > >> >> >>> Client section it will take precedence over zookeeper SSL
> > configs?
> > > >> >> >>>
> > > >> >> >>> Thanks,
> > > >> >> >>> Harsha
> > > >> >> >>>
> > > >> >> >>>
> > > >> >> >>>
> > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> > > >> >> pere.urbon@gmail.com>
> > > >> >> >>> wrote:
> > > >> >> >>>
> > > >> >> >>>> Hi,
> > > >> >> >>>> quick question, I saw in another mail that 2.4 release is
> > planned
> > > >> for
> > > >> >> >>>> September. I think it would be really awesome to have this for
> > > >> this
> > > >> >> >>>> release, do you think we can make it?
> > > >> >> >>>>
> > > >> >> >>>> -- Pere
> > > >> >> >>>>
> > > >> >> >>>> Missatge de Pere Urbón Bayes <pe...@gmail.com> del dia
> > dj.,
> > > >> 29
> > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > >> >> >>>>
> > > >> >> >>>> Hi,
> > > >> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm
> > really
> > > >> need
> > > >> >> >>>> to the process. Looking forward to hearing from you and learn
> > the
> > > >> >> best
> > > >> >> >>>> ropes here.
> > > >> >> >>>>
> > > >> >> >>>> I would like to propose this KIP-515 to enable the
> > > >> ZookeeperClients
> > > >> >> to
> > > >> >> >>>> take full advantage of the TLS communication in the new
> > Zookeeper
> > > >> >> 3.5.5.
> > > >> >> >>>> Specially interesting it the Zookeeper Security Migration,
> > that
> > > >> >> without
> > > >> >> >>>> this change will not work with TLS, disabling users to use
> > ACLs
> > > >> when
> > > >> >> the
> > > >> >> >>>> Zookeeper cluster use TLS.
> > > >> >> >>>>
> > > >> >> >>>> link:
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >>
> > > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > >> >> >>>>
> > > >> >> >>>> Looking forward to hearing from you on this,
> > > >> >> >>>>
> > > >> >> >>>> /cheers
> > > >> >> >>>>
> > > >> >> >>>> --
> > > >> >> >>>> Pere Urbon-Bayes
> > > >> >> >>>> Software Architect
> > > >> >> >>>> http://www.purbon.com
> > > >> >> >>>> https://twitter.com/purbon
> > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > >> >> >>>>
> > > >> >> >>>> --
> > > >> >> >>>> Pere Urbon-Bayes
> > > >> >> >>>> Software Architect
> > > >> >> >>>> http://www.purbon.com
> > > >> >> >>>> https://twitter.com/purbon
> > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > >> >> >>>>
> > > >> >> >>>
> > > >> >> >>>
> > > >> >> >>
> > > >> >> >> --
> > > >> >> >> Pere Urbon-Bayes
> > > >> >> >> Software Architect
> > > >> >> >> http://www.purbon.com
> > > >> >> >> https://twitter.com/purbon
> > > >> >> >> https://www.linkedin.com/in/purbon/
> > > >> >> >>
> > > >> >> >
> > > >> >> >
> > > >> >>
> > > >> >> --
> > > >> >> Pere Urbon-Bayes
> > > >> >> Software Architect
> > > >> >> http://www.purbon.com
> > > >> >> https://twitter.com/purbon
> > > >> >> https://www.linkedin.com/in/purbon/
> > > >> >>
> > > >> >
> > > >>
> > > >> --
> > > >> Pere Urbon-Bayes
> > > >> Software Architect
> > > >> http://www.purbon.com
> > > >> https://twitter.com/purbon
> > > >> https://www.linkedin.com/in/purbon/
> > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Ron Dagostino <rn...@gmail.com>.
Hi Colin.
<<< It seems like this [--zk-tls-config-file information] could just appear
in a configuration file, which all of these tools already accept (I think)
ZkSecurityMigrator has no such property file facility; adding a
"--zk-tls-config-file" parameter is exactly for this purpose. If we add
that to ZkSecurityMigrator then it is trivial to add it to other commands
(the same code is simply reused; it ends up being just a few extra lines).
I do not see any parameter in the other two commands to adjust the ZK
connection; ConfigCommand accepts a "--command-config" flag, but according
to the code "This is used only with --bootstrap-server option for
describing and altering broker configs."
I do agree there would be no need to add "--zk-tls-config-file" to
ReassignPartitionsCommand if its direct ZK connectivity is replaced in time
for the next release.
ConfigCommand supports the "--bootstrap-server" option and will have its
direct ZooKeeper access formally deprecated as per KIP-555, but the special
use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
prior to starting Kafka will still exist, so it feels like while
"--zk-tls-config-file" would never be used except for this use case it
could probably still be added for this particular situation.
Ron
P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
sent) did not go through. I am trying to get emails through now to move
this discussion forward.
On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cm...@apache.org> wrote:
> On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > Hi everyone. I would like to make the following changes to the KIP.
> >
> > MOTIVATION:
> > Include a statement that it will be difficult in the short term to
> > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> (which
> > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka cluster
> > with encrypted passwords in Zookeeper is an explicitly-supported use
> case;
> > therefore it is in scope to be able to securely configure the CLI tools
> > that still leverage non-deprecated direct Zookeeper communication for TLS
> > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > zookeeper-security-migration.sh).
>
> Hi Ron,
>
> Thanks for the KIP.
>
> About deprecations:
>
> * zookeeper-security-migration: clearly, deprecating ZK access in this one
> would not make sense, since it would defeat the whole point of the tool :)
>
> * kafka-reassign-partitions: ZK access should be deprecated here. KIP-455
> implementation has dragged a bit, but this should get done soon. Certainly
> before the next release.
>
> * kafka-configs: I think ZK access should be deprecated here as well. I
> agree there is a super-special bootstrapping case here, but that should
> have its own tool, not use kafka-configs.
>
> I will post a separate KIP for this, though.
>
> >
> > GOALS:
> > Support the secure configuration of TLS-encrypted communication between
> > Zookeeper and:
> > a) Kafka brokers
> > b) The three CLI tools mentioned above that still support direct,
> > non-deprecated communication to Zookeeper
> > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > communication in CLI tools as part of this KIP; such work will occur in
> > future KIPs instead.
> >
> > PUBLIC INTERFACES:
> > 1) The following new broker configurations will be recognized.
> > zookeeper.client.secure (default value = false, for backwards
> > compatibility)
> > zookeeper.clientCnxnSocket
> > zookeeper.ssl.keyStore.location
> > zookeeper.ssl.keyStore.password
> > zookeeper.ssl.trustStore.location
> > zookeeper.ssl.trustStore.password
> > It will be an error for any of the last 5 values to be left unspecified
> if
> > zookeeper.client.secure is explicitly set to true.
> >
> > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> supports
> > the ability to connect to a different Zookeeper instance than the one the
> > brokers use. We therefore also add the following optional configs, which
> > override the corresponding ones from above when present:
> > authorizer.zookeeper.client.secure
> > authorizer.zookeeper.clientCnxnSocket
> > authorizer.zookeeper.ssl.keyStore.location
> > authorizer.zookeeper.ssl.keyStore.password
> > authorizer.zookeeper.ssl.trustStore.location
> > authorizer.zookeeper.ssl.trustStore.password
> >
> > 3) The three CLI tools mentioned above will support a new
> --zk-tls-config-file
> > <String: Zookeeper TLS configuration file path>" option. The following
> > properties will be recognized in that file, and unrecognized properties
> > will be ignored to allow the possibility of pointing zk-tls-config-file
> at
> > the broker's config file.
> > zookeeper.client.secure (default value = false)
> > zookeeper.clientCnxnSocket
> > zookeeper.ssl.keyStore.location
> > zookeeper.ssl.keyStore.password
> > zookeeper.ssl.trustStore.location
> > zookeeper.ssl.trustStore.password
> > It will be an error for any of the last 5 values to be left unspecified
> if
> > zookeeper.client.secure is explicitly set to true.
>
> Do we really need a --zk-tls-config-file flag? It seems like this could
> just appear in a configuration file, which all of these tools already
> accept (I think).
>
> best,
> Colin
>
>
> >
> > Ron
> >
> > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rn...@gmail.com> wrote:
> >
> > > Hi everyone. Let's get this discussion going again now that Kafka 2.4
> > > with Zookeeper 3.5.6 is out.
> > >
> > > First, regarding the KIP number, the other KIP that was using this
> number
> > > moved to KIP 534, so KIP 515 remains the correct number for this
> > > discussion. I've updated the Kafka Improvement Proposal page to list
> this
> > > KIP in the 515 slot, so we're all set there.
> > >
> > > Regarding the actual issues under discussion, I think there are some
> > > things we should clarify.
> > >
> > > 1) It is possible to use TLS connectivity to Zookeeper from Apache
> Kafka
> > > 2.4 -- the problem is that configuration information has to be passed
> via
> > > system properties as "-D" command line options on the java invocation
> of
> > > the broker, and those are not secure (anyone with access to the box
> can see
> > > the command line used to invoke the broker); the configuration includes
> > > sensitive information (e.g. a keystore password), so we need a secure
> > > mechanism for passing the configuration values. I believe the real
> > > motivation for this KIP is to harden the configuration mechanism for
> > > Zookeeper TLS connectivity.
> > >
> > > 2) I believe the list of CLI tools that continue to use direct
> Zookeeper
> > > connectivity in a non-deprecated fashion is:
> > > a) zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > b)
> > >
> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > >
> > > 3) I believe Kafka.admin.ConfigCommand presents a conundrum because it
> > > explicitly states in a comment that a supported use case is
> bootstrapping a
> > > Kafka cluster with encrypted passwords in Zookeeper (see
> > >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> ).
> > > This means it will be especially difficult to deprecate this particular
> > > direct Zookeeper connectivity without a different storage mechanism for
> > > dynamic configuration values being available (i.e. the self-managed
> quorum
> > > referred to in KIP-500).
> > >
> > > I think it would be easier and simpler to harden the Zookeeper TLS
> > > configuration in both Kafka and in CLI tools compared to trying to
> > > deprecate the direct Zookeeper connectivity in the above 3 CLI tools --
> > > especially when ConfigCommand has no obvious short-term path to
> deprecation.
> > >
> > > Regarding how to harden the TLS configuration, adding the appropriate
> > > configs to Kafka is an obvious choice for the brokers. Not that these
> > > values cannot be dynamically reconfigurable because the values would be
> > > required to connect to Zookeeper via TLS in the first place.
> > >
> > > Regarding how to harden the TLS configuration for the 3 remaining CLI
> > > tools, it would be relatively straightforward to add support for a
> > > "--zk-tls-config-file <String: Zookeeper TLS configuration system
> > > properties file path>" option to each one.
> > >
> > > Thoughts?
> > >
> > > Ron
> > >
> > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <pere.urbon@gmail.com
> >
> > > wrote:
> > >
> > >> True, the number is duplicated.
> > >>
> > >> do you know how can we get the problem fix? I was not aware, I
> missed,
> > >> sorry the need to add the KIP to the table.
> > >>
> > >> -- Pere
> > >>
> > >> Missatge de Jorge Esteban Quilcate Otoya <qu...@gmail.com>
> del
> > >> dia
> > >> dt., 3 de set. 2019 a les 13:43:
> > >>
> > >> > Hi Pere,
> > >> >
> > >> > Have you add your KIP to the list here
> > >> >
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > >> ?
> > >> >
> > >> > I found the KIP number assigned to another.
> > >> >
> > >> >
> > >> >
> > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> pere.urbon@gmail.com>
> > >> > wrote:
> > >> >
> > >> >> Thanks for your time Harsha,
> > >> >> anyone else with comments? looking forward to hearing from you.
> > >> >>
> > >> >> Stupid question: when do you move from discussion to vote?
> > >> >>
> > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv., 30
> > >> d’ag.
> > >> >> 2019 a les 21:59:
> > >> >>
> > >> >> > Thanks Pere. KIP looks good to me.
> > >> >> > -Harsha
> > >> >> >
> > >> >> >
> > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> > >> >> pere.urbon@gmail.com>
> > >> >> > wrote:
> > >> >> >
> > >> >> >> Not really,
> > >> >> >> my idea is to keep the JAAS parameter, so people don't see
> major
> > >> >> >> changes. But if you pass a properties file, then this takes
> > >> precedence
> > >> >> over
> > >> >> >> the other, with the idea that you can do sasl as well with the
> > >> >> properties
> > >> >> >> files.
> > >> >> >>
> > >> >> >> Makes sense?
> > >> >> >>
> > >> >> >> -- Pere
> > >> >> >>
> > >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv.,
> 30
> > >> >> d’ag.
> > >> >> >> 2019 a les 19:00:
> > >> >> >>
> > >> >> >>> Hi Pere,
> > >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper for
> Kafka
> > >> >> makes
> > >> >> >>> sense.
> > >> >> >>> "The changes are planned to be introduced in a compatible way,
> by
> > >> >> >>> keeping the current JAAS variable precedence."
> > >> >> >>> Can you elaborate a bit here. If the user configures a JAAS
> file
> > >> with
> > >> >> >>> Client section it will take precedence over zookeeper SSL
> configs?
> > >> >> >>>
> > >> >> >>> Thanks,
> > >> >> >>> Harsha
> > >> >> >>>
> > >> >> >>>
> > >> >> >>>
> > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> > >> >> pere.urbon@gmail.com>
> > >> >> >>> wrote:
> > >> >> >>>
> > >> >> >>>> Hi,
> > >> >> >>>> quick question, I saw in another mail that 2.4 release is
> planned
> > >> for
> > >> >> >>>> September. I think it would be really awesome to have this for
> > >> this
> > >> >> >>>> release, do you think we can make it?
> > >> >> >>>>
> > >> >> >>>> -- Pere
> > >> >> >>>>
> > >> >> >>>> Missatge de Pere Urbón Bayes <pe...@gmail.com> del dia
> dj.,
> > >> 29
> > >> >> >>>> d’ag. 2019 a les 20:10:
> > >> >> >>>>
> > >> >> >>>> Hi,
> > >> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm
> really
> > >> need
> > >> >> >>>> to the process. Looking forward to hearing from you and learn
> the
> > >> >> best
> > >> >> >>>> ropes here.
> > >> >> >>>>
> > >> >> >>>> I would like to propose this KIP-515 to enable the
> > >> ZookeeperClients
> > >> >> to
> > >> >> >>>> take full advantage of the TLS communication in the new
> Zookeeper
> > >> >> 3.5.5.
> > >> >> >>>> Specially interesting it the Zookeeper Security Migration,
> that
> > >> >> without
> > >> >> >>>> this change will not work with TLS, disabling users to use
> ACLs
> > >> when
> > >> >> the
> > >> >> >>>> Zookeeper cluster use TLS.
> > >> >> >>>>
> > >> >> >>>> link:
> > >> >> >>>>
> > >> >> >>>>
> > >> >>
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > >> >> >>>>
> > >> >> >>>> Looking forward to hearing from you on this,
> > >> >> >>>>
> > >> >> >>>> /cheers
> > >> >> >>>>
> > >> >> >>>> --
> > >> >> >>>> Pere Urbon-Bayes
> > >> >> >>>> Software Architect
> > >> >> >>>> http://www.purbon.com
> > >> >> >>>> https://twitter.com/purbon
> > >> >> >>>> https://www.linkedin.com/in/purbon/
> > >> >> >>>>
> > >> >> >>>> --
> > >> >> >>>> Pere Urbon-Bayes
> > >> >> >>>> Software Architect
> > >> >> >>>> http://www.purbon.com
> > >> >> >>>> https://twitter.com/purbon
> > >> >> >>>> https://www.linkedin.com/in/purbon/
> > >> >> >>>>
> > >> >> >>>
> > >> >> >>>
> > >> >> >>
> > >> >> >> --
> > >> >> >> Pere Urbon-Bayes
> > >> >> >> Software Architect
> > >> >> >> http://www.purbon.com
> > >> >> >> https://twitter.com/purbon
> > >> >> >> https://www.linkedin.com/in/purbon/
> > >> >> >>
> > >> >> >
> > >> >> >
> > >> >>
> > >> >> --
> > >> >> Pere Urbon-Bayes
> > >> >> Software Architect
> > >> >> http://www.purbon.com
> > >> >> https://twitter.com/purbon
> > >> >> https://www.linkedin.com/in/purbon/
> > >> >>
> > >> >
> > >>
> > >> --
> > >> Pere Urbon-Bayes
> > >> Software Architect
> > >> http://www.purbon.com
> > >> https://twitter.com/purbon
> > >> https://www.linkedin.com/in/purbon/
> > >>
> > >
> >
>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Colin McCabe <cm...@apache.org>.
On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> Hi everyone. I would like to make the following changes to the KIP.
>
> MOTIVATION:
> Include a statement that it will be difficult in the short term to
> deprecate direct Zookeeper communication in kafka-configs.{sh, bat} (which
> invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka cluster
> with encrypted passwords in Zookeeper is an explicitly-supported use case;
> therefore it is in scope to be able to securely configure the CLI tools
> that still leverage non-deprecated direct Zookeeper communication for TLS
> (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> zookeeper-security-migration.sh).
Hi Ron,
Thanks for the KIP.
About deprecations:
* zookeeper-security-migration: clearly, deprecating ZK access in this one would not make sense, since it would defeat the whole point of the tool :)
* kafka-reassign-partitions: ZK access should be deprecated here. KIP-455 implementation has dragged a bit, but this should get done soon. Certainly before the next release.
* kafka-configs: I think ZK access should be deprecated here as well. I agree there is a super-special bootstrapping case here, but that should have its own tool, not use kafka-configs.
I will post a separate KIP for this, though.
>
> GOALS:
> Support the secure configuration of TLS-encrypted communication between
> Zookeeper and:
> a) Kafka brokers
> b) The three CLI tools mentioned above that still support direct,
> non-deprecated communication to Zookeeper
> It is explicitly out-of-scope to deprecate any direct Zookeeper
> communication in CLI tools as part of this KIP; such work will occur in
> future KIPs instead.
>
> PUBLIC INTERFACES:
> 1) The following new broker configurations will be recognized.
> zookeeper.client.secure (default value = false, for backwards
> compatibility)
> zookeeper.clientCnxnSocket
> zookeeper.ssl.keyStore.location
> zookeeper.ssl.keyStore.password
> zookeeper.ssl.trustStore.location
> zookeeper.ssl.trustStore.password
> It will be an error for any of the last 5 values to be left unspecified if
> zookeeper.client.secure is explicitly set to true.
>
> 2) In addition, the kafka.security.authorizer.AclAuthorizer class supports
> the ability to connect to a different Zookeeper instance than the one the
> brokers use. We therefore also add the following optional configs, which
> override the corresponding ones from above when present:
> authorizer.zookeeper.client.secure
> authorizer.zookeeper.clientCnxnSocket
> authorizer.zookeeper.ssl.keyStore.location
> authorizer.zookeeper.ssl.keyStore.password
> authorizer.zookeeper.ssl.trustStore.location
> authorizer.zookeeper.ssl.trustStore.password
>
> 3) The three CLI tools mentioned above will support a new --zk-tls-config-file
> <String: Zookeeper TLS configuration file path>" option. The following
> properties will be recognized in that file, and unrecognized properties
> will be ignored to allow the possibility of pointing zk-tls-config-file at
> the broker's config file.
> zookeeper.client.secure (default value = false)
> zookeeper.clientCnxnSocket
> zookeeper.ssl.keyStore.location
> zookeeper.ssl.keyStore.password
> zookeeper.ssl.trustStore.location
> zookeeper.ssl.trustStore.password
> It will be an error for any of the last 5 values to be left unspecified if
> zookeeper.client.secure is explicitly set to true.
Do we really need a --zk-tls-config-file flag? It seems like this could just appear in a configuration file, which all of these tools already accept (I think).
best,
Colin
>
> Ron
>
> On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rn...@gmail.com> wrote:
>
> > Hi everyone. Let's get this discussion going again now that Kafka 2.4
> > with Zookeeper 3.5.6 is out.
> >
> > First, regarding the KIP number, the other KIP that was using this number
> > moved to KIP 534, so KIP 515 remains the correct number for this
> > discussion. I've updated the Kafka Improvement Proposal page to list this
> > KIP in the 515 slot, so we're all set there.
> >
> > Regarding the actual issues under discussion, I think there are some
> > things we should clarify.
> >
> > 1) It is possible to use TLS connectivity to Zookeeper from Apache Kafka
> > 2.4 -- the problem is that configuration information has to be passed via
> > system properties as "-D" command line options on the java invocation of
> > the broker, and those are not secure (anyone with access to the box can see
> > the command line used to invoke the broker); the configuration includes
> > sensitive information (e.g. a keystore password), so we need a secure
> > mechanism for passing the configuration values. I believe the real
> > motivation for this KIP is to harden the configuration mechanism for
> > Zookeeper TLS connectivity.
> >
> > 2) I believe the list of CLI tools that continue to use direct Zookeeper
> > connectivity in a non-deprecated fashion is:
> > a) zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > b)
> > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> >
> > 3) I believe Kafka.admin.ConfigCommand presents a conundrum because it
> > explicitly states in a comment that a supported use case is bootstrapping a
> > Kafka cluster with encrypted passwords in Zookeeper (see
> > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65).
> > This means it will be especially difficult to deprecate this particular
> > direct Zookeeper connectivity without a different storage mechanism for
> > dynamic configuration values being available (i.e. the self-managed quorum
> > referred to in KIP-500).
> >
> > I think it would be easier and simpler to harden the Zookeeper TLS
> > configuration in both Kafka and in CLI tools compared to trying to
> > deprecate the direct Zookeeper connectivity in the above 3 CLI tools --
> > especially when ConfigCommand has no obvious short-term path to deprecation.
> >
> > Regarding how to harden the TLS configuration, adding the appropriate
> > configs to Kafka is an obvious choice for the brokers. Not that these
> > values cannot be dynamically reconfigurable because the values would be
> > required to connect to Zookeeper via TLS in the first place.
> >
> > Regarding how to harden the TLS configuration for the 3 remaining CLI
> > tools, it would be relatively straightforward to add support for a
> > "--zk-tls-config-file <String: Zookeeper TLS configuration system
> > properties file path>" option to each one.
> >
> > Thoughts?
> >
> > Ron
> >
> > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <pe...@gmail.com>
> > wrote:
> >
> >> True, the number is duplicated.
> >>
> >> do you know how can we get the problem fix? I was not aware, I missed,
> >> sorry the need to add the KIP to the table.
> >>
> >> -- Pere
> >>
> >> Missatge de Jorge Esteban Quilcate Otoya <qu...@gmail.com> del
> >> dia
> >> dt., 3 de set. 2019 a les 13:43:
> >>
> >> > Hi Pere,
> >> >
> >> > Have you add your KIP to the list here
> >> >
> >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >> ?
> >> >
> >> > I found the KIP number assigned to another.
> >> >
> >> >
> >> >
> >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <pe...@gmail.com>
> >> > wrote:
> >> >
> >> >> Thanks for your time Harsha,
> >> >> anyone else with comments? looking forward to hearing from you.
> >> >>
> >> >> Stupid question: when do you move from discussion to vote?
> >> >>
> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv., 30
> >> d’ag.
> >> >> 2019 a les 21:59:
> >> >>
> >> >> > Thanks Pere. KIP looks good to me.
> >> >> > -Harsha
> >> >> >
> >> >> >
> >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> >> >> pere.urbon@gmail.com>
> >> >> > wrote:
> >> >> >
> >> >> >> Not really,
> >> >> >> my idea is to keep the JAAS parameter, so people don't see major
> >> >> >> changes. But if you pass a properties file, then this takes
> >> precedence
> >> >> over
> >> >> >> the other, with the idea that you can do sasl as well with the
> >> >> properties
> >> >> >> files.
> >> >> >>
> >> >> >> Makes sense?
> >> >> >>
> >> >> >> -- Pere
> >> >> >>
> >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv., 30
> >> >> d’ag.
> >> >> >> 2019 a les 19:00:
> >> >> >>
> >> >> >>> Hi Pere,
> >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper for Kafka
> >> >> makes
> >> >> >>> sense.
> >> >> >>> "The changes are planned to be introduced in a compatible way, by
> >> >> >>> keeping the current JAAS variable precedence."
> >> >> >>> Can you elaborate a bit here. If the user configures a JAAS file
> >> with
> >> >> >>> Client section it will take precedence over zookeeper SSL configs?
> >> >> >>>
> >> >> >>> Thanks,
> >> >> >>> Harsha
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> >> >> pere.urbon@gmail.com>
> >> >> >>> wrote:
> >> >> >>>
> >> >> >>>> Hi,
> >> >> >>>> quick question, I saw in another mail that 2.4 release is planned
> >> for
> >> >> >>>> September. I think it would be really awesome to have this for
> >> this
> >> >> >>>> release, do you think we can make it?
> >> >> >>>>
> >> >> >>>> -- Pere
> >> >> >>>>
> >> >> >>>> Missatge de Pere Urbón Bayes <pe...@gmail.com> del dia dj.,
> >> 29
> >> >> >>>> d’ag. 2019 a les 20:10:
> >> >> >>>>
> >> >> >>>> Hi,
> >> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm really
> >> need
> >> >> >>>> to the process. Looking forward to hearing from you and learn the
> >> >> best
> >> >> >>>> ropes here.
> >> >> >>>>
> >> >> >>>> I would like to propose this KIP-515 to enable the
> >> ZookeeperClients
> >> >> to
> >> >> >>>> take full advantage of the TLS communication in the new Zookeeper
> >> >> 3.5.5.
> >> >> >>>> Specially interesting it the Zookeeper Security Migration, that
> >> >> without
> >> >> >>>> this change will not work with TLS, disabling users to use ACLs
> >> when
> >> >> the
> >> >> >>>> Zookeeper cluster use TLS.
> >> >> >>>>
> >> >> >>>> link:
> >> >> >>>>
> >> >> >>>>
> >> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> >> >> >>>>
> >> >> >>>> Looking forward to hearing from you on this,
> >> >> >>>>
> >> >> >>>> /cheers
> >> >> >>>>
> >> >> >>>> --
> >> >> >>>> Pere Urbon-Bayes
> >> >> >>>> Software Architect
> >> >> >>>> http://www.purbon.com
> >> >> >>>> https://twitter.com/purbon
> >> >> >>>> https://www.linkedin.com/in/purbon/
> >> >> >>>>
> >> >> >>>> --
> >> >> >>>> Pere Urbon-Bayes
> >> >> >>>> Software Architect
> >> >> >>>> http://www.purbon.com
> >> >> >>>> https://twitter.com/purbon
> >> >> >>>> https://www.linkedin.com/in/purbon/
> >> >> >>>>
> >> >> >>>
> >> >> >>>
> >> >> >>
> >> >> >> --
> >> >> >> Pere Urbon-Bayes
> >> >> >> Software Architect
> >> >> >> http://www.purbon.com
> >> >> >> https://twitter.com/purbon
> >> >> >> https://www.linkedin.com/in/purbon/
> >> >> >>
> >> >> >
> >> >> >
> >> >>
> >> >> --
> >> >> Pere Urbon-Bayes
> >> >> Software Architect
> >> >> http://www.purbon.com
> >> >> https://twitter.com/purbon
> >> >> https://www.linkedin.com/in/purbon/
> >> >>
> >> >
> >>
> >> --
> >> Pere Urbon-Bayes
> >> Software Architect
> >> http://www.purbon.com
> >> https://twitter.com/purbon
> >> https://www.linkedin.com/in/purbon/
> >>
> >
>
Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Posted by Ron Dagostino <rn...@gmail.com>.
Hi everyone. I would like to make the following changes to the KIP.
MOTIVATION:
Include a statement that it will be difficult in the short term to
deprecate direct Zookeeper communication in kafka-configs.{sh, bat} (which
invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka cluster
with encrypted passwords in Zookeeper is an explicitly-supported use case;
therefore it is in scope to be able to securely configure the CLI tools
that still leverage non-deprecated direct Zookeeper communication for TLS
(the other 2 tools are kafka-reassign-partitions.{sh, bat} and
zookeeper-security-migration.sh).
GOALS:
Support the secure configuration of TLS-encrypted communication between
Zookeeper and:
a) Kafka brokers
b) The three CLI tools mentioned above that still support direct,
non-deprecated communication to Zookeeper
It is explicitly out-of-scope to deprecate any direct Zookeeper
communication in CLI tools as part of this KIP; such work will occur in
future KIPs instead.
PUBLIC INTERFACES:
1) The following new broker configurations will be recognized.
zookeeper.client.secure (default value = false, for backwards
compatibility)
zookeeper.clientCnxnSocket
zookeeper.ssl.keyStore.location
zookeeper.ssl.keyStore.password
zookeeper.ssl.trustStore.location
zookeeper.ssl.trustStore.password
It will be an error for any of the last 5 values to be left unspecified if
zookeeper.client.secure is explicitly set to true.
2) In addition, the kafka.security.authorizer.AclAuthorizer class supports
the ability to connect to a different Zookeeper instance than the one the
brokers use. We therefore also add the following optional configs, which
override the corresponding ones from above when present:
authorizer.zookeeper.client.secure
authorizer.zookeeper.clientCnxnSocket
authorizer.zookeeper.ssl.keyStore.location
authorizer.zookeeper.ssl.keyStore.password
authorizer.zookeeper.ssl.trustStore.location
authorizer.zookeeper.ssl.trustStore.password
3) The three CLI tools mentioned above will support a new --zk-tls-config-file
<String: Zookeeper TLS configuration file path>" option. The following
properties will be recognized in that file, and unrecognized properties
will be ignored to allow the possibility of pointing zk-tls-config-file at
the broker's config file.
zookeeper.client.secure (default value = false)
zookeeper.clientCnxnSocket
zookeeper.ssl.keyStore.location
zookeeper.ssl.keyStore.password
zookeeper.ssl.trustStore.location
zookeeper.ssl.trustStore.password
It will be an error for any of the last 5 values to be left unspecified if
zookeeper.client.secure is explicitly set to true.
Ron
On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rn...@gmail.com> wrote:
> Hi everyone. Let's get this discussion going again now that Kafka 2.4
> with Zookeeper 3.5.6 is out.
>
> First, regarding the KIP number, the other KIP that was using this number
> moved to KIP 534, so KIP 515 remains the correct number for this
> discussion. I've updated the Kafka Improvement Proposal page to list this
> KIP in the 515 slot, so we're all set there.
>
> Regarding the actual issues under discussion, I think there are some
> things we should clarify.
>
> 1) It is possible to use TLS connectivity to Zookeeper from Apache Kafka
> 2.4 -- the problem is that configuration information has to be passed via
> system properties as "-D" command line options on the java invocation of
> the broker, and those are not secure (anyone with access to the box can see
> the command line used to invoke the broker); the configuration includes
> sensitive information (e.g. a keystore password), so we need a secure
> mechanism for passing the configuration values. I believe the real
> motivation for this KIP is to harden the configuration mechanism for
> Zookeeper TLS connectivity.
>
> 2) I believe the list of CLI tools that continue to use direct Zookeeper
> connectivity in a non-deprecated fashion is:
> a) zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> b)
> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
>
> 3) I believe Kafka.admin.ConfigCommand presents a conundrum because it
> explicitly states in a comment that a supported use case is bootstrapping a
> Kafka cluster with encrypted passwords in Zookeeper (see
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65).
> This means it will be especially difficult to deprecate this particular
> direct Zookeeper connectivity without a different storage mechanism for
> dynamic configuration values being available (i.e. the self-managed quorum
> referred to in KIP-500).
>
> I think it would be easier and simpler to harden the Zookeeper TLS
> configuration in both Kafka and in CLI tools compared to trying to
> deprecate the direct Zookeeper connectivity in the above 3 CLI tools --
> especially when ConfigCommand has no obvious short-term path to deprecation.
>
> Regarding how to harden the TLS configuration, adding the appropriate
> configs to Kafka is an obvious choice for the brokers. Not that these
> values cannot be dynamically reconfigurable because the values would be
> required to connect to Zookeeper via TLS in the first place.
>
> Regarding how to harden the TLS configuration for the 3 remaining CLI
> tools, it would be relatively straightforward to add support for a
> "--zk-tls-config-file <String: Zookeeper TLS configuration system
> properties file path>" option to each one.
>
> Thoughts?
>
> Ron
>
> On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <pe...@gmail.com>
> wrote:
>
>> True, the number is duplicated.
>>
>> do you know how can we get the problem fix? I was not aware, I missed,
>> sorry the need to add the KIP to the table.
>>
>> -- Pere
>>
>> Missatge de Jorge Esteban Quilcate Otoya <qu...@gmail.com> del
>> dia
>> dt., 3 de set. 2019 a les 13:43:
>>
>> > Hi Pere,
>> >
>> > Have you add your KIP to the list here
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>> ?
>> >
>> > I found the KIP number assigned to another.
>> >
>> >
>> >
>> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <pe...@gmail.com>
>> > wrote:
>> >
>> >> Thanks for your time Harsha,
>> >> anyone else with comments? looking forward to hearing from you.
>> >>
>> >> Stupid question: when do you move from discussion to vote?
>> >>
>> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv., 30
>> d’ag.
>> >> 2019 a les 21:59:
>> >>
>> >> > Thanks Pere. KIP looks good to me.
>> >> > -Harsha
>> >> >
>> >> >
>> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
>> >> pere.urbon@gmail.com>
>> >> > wrote:
>> >> >
>> >> >> Not really,
>> >> >> my idea is to keep the JAAS parameter, so people don't see major
>> >> >> changes. But if you pass a properties file, then this takes
>> precedence
>> >> over
>> >> >> the other, with the idea that you can do sasl as well with the
>> >> properties
>> >> >> files.
>> >> >>
>> >> >> Makes sense?
>> >> >>
>> >> >> -- Pere
>> >> >>
>> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv., 30
>> >> d’ag.
>> >> >> 2019 a les 19:00:
>> >> >>
>> >> >>> Hi Pere,
>> >> >>> Thanks for the KIP. Enabling SSL for zookeeper for Kafka
>> >> makes
>> >> >>> sense.
>> >> >>> "The changes are planned to be introduced in a compatible way, by
>> >> >>> keeping the current JAAS variable precedence."
>> >> >>> Can you elaborate a bit here. If the user configures a JAAS file
>> with
>> >> >>> Client section it will take precedence over zookeeper SSL configs?
>> >> >>>
>> >> >>> Thanks,
>> >> >>> Harsha
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
>> >> pere.urbon@gmail.com>
>> >> >>> wrote:
>> >> >>>
>> >> >>>> Hi,
>> >> >>>> quick question, I saw in another mail that 2.4 release is planned
>> for
>> >> >>>> September. I think it would be really awesome to have this for
>> this
>> >> >>>> release, do you think we can make it?
>> >> >>>>
>> >> >>>> -- Pere
>> >> >>>>
>> >> >>>> Missatge de Pere Urbón Bayes <pe...@gmail.com> del dia dj.,
>> 29
>> >> >>>> d’ag. 2019 a les 20:10:
>> >> >>>>
>> >> >>>> Hi,
>> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm really
>> need
>> >> >>>> to the process. Looking forward to hearing from you and learn the
>> >> best
>> >> >>>> ropes here.
>> >> >>>>
>> >> >>>> I would like to propose this KIP-515 to enable the
>> ZookeeperClients
>> >> to
>> >> >>>> take full advantage of the TLS communication in the new Zookeeper
>> >> 3.5.5.
>> >> >>>> Specially interesting it the Zookeeper Security Migration, that
>> >> without
>> >> >>>> this change will not work with TLS, disabling users to use ACLs
>> when
>> >> the
>> >> >>>> Zookeeper cluster use TLS.
>> >> >>>>
>> >> >>>> link:
>> >> >>>>
>> >> >>>>
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
>> >> >>>>
>> >> >>>> Looking forward to hearing from you on this,
>> >> >>>>
>> >> >>>> /cheers
>> >> >>>>
>> >> >>>> --
>> >> >>>> Pere Urbon-Bayes
>> >> >>>> Software Architect
>> >> >>>> http://www.purbon.com
>> >> >>>> https://twitter.com/purbon
>> >> >>>> https://www.linkedin.com/in/purbon/
>> >> >>>>
>> >> >>>> --
>> >> >>>> Pere Urbon-Bayes
>> >> >>>> Software Architect
>> >> >>>> http://www.purbon.com
>> >> >>>> https://twitter.com/purbon
>> >> >>>> https://www.linkedin.com/in/purbon/
>> >> >>>>
>> >> >>>
>> >> >>>
>> >> >>
>> >> >> --
>> >> >> Pere Urbon-Bayes
>> >> >> Software Architect
>> >> >> http://www.purbon.com
>> >> >> https://twitter.com/purbon
>> >> >> https://www.linkedin.com/in/purbon/
>> >> >>
>> >> >
>> >> >
>> >>
>> >> --
>> >> Pere Urbon-Bayes
>> >> Software Architect
>> >> http://www.purbon.com
>> >> https://twitter.com/purbon
>> >> https://www.linkedin.com/in/purbon/
>> >>
>> >
>>
>> --
>> Pere Urbon-Bayes
>> Software Architect
>> http://www.purbon.com
>> https://twitter.com/purbon
>> https://www.linkedin.com/in/purbon/
>>
>