You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Dávid Paksy <pa...@apache.org> on 2026/09/01 13:50:25 UTC
Re: [DISCUSS] Single purpose TLS extended key usages (EKU) support
Hi David,
Many thanks for bringing this up, this is indeed a valid point that Istio
service-mesh can manage TLS and in this case the application does not have
to care about it.
However as far as I understand Istio cannot be used without a Kubernetes
cluster and if we don't have Kubernetes (e.g. use VM-s or bare metal
servers) this is not an option for us.
So probably it is beneficial to have a solution for this case as well.
Best Regards,
Dávid
David Smiley <ds...@apache.org> ezt írta (időpont: 2026. aug. 19., Sze,
15:08):
> Are any of you working in an organization that has embraced
> managed-mesh/Istio [1]? It's a game changer for the better; I have seen
> the future with that. It pushes mTLS down to an infrastructure concern;
> the TCP/IP services above communicate in cleartext unaware of the lower
> level encryption. No service needs to care about TLS, let alone mTLS, let
> alone FIPS. It removed a considerable burden across all the services in
> the organization, including complexity / quirky matters on a service by
> service basis. Only the team managing the Istio infrastructure needed to
> know how that stuff worked. More importantly, it significantly raised the
> security posture across the board.
>
> [1] https://istio.io/latest/about/service-mesh/
>
> On Wed, Aug 19, 2026 at 4:36 AM Dávid Paksy <pa...@apache.org> wrote:
>
> > Hi,
> >
> > I now changed the implementation of option #1 to use separate SSLContext
> > instances in SSLContextAndOptions for client and server role. Each
> > SSLContext is initialized with standard PKIX managers from its own
> keystore
> > / truststore. There is no custom key- or trust-manager. As far as I
> > understand, this should work in FIPS mode.
> > There is no change in config properties, and it is fully backward
> > compatible.
> >
> > What are your thoughts?
> >
> > Thanks,
> > Dávid
> >
> > Dávid Paksy <pa...@gmail.com> ezt írta (időpont: 2026. aug. 17., H,
> > 11:31):
> >
> > > Hi Andor,
> > >
> > > Sorry, I was too quick to send my response. Probably you didn't meant
> the
> > > FIPS mode property in ZooKeeper but FIPS mode in Java.
> > >
> > > Enabling FIPS mode in Java imposes strict constraints on which
> KeyManager
> > > and TrustManager implementations can be instantiated and initialized.
> > > I looked into this and in FIPS mode, SSLContext.init() will throw an
> > > exception if custom manager instances are passed. Something like:
> > > java.security.KeyManagementException: FIPS mode: only SunJSSE
> > > TrustManagers may be used.
> > >
> > > I agree that this can be an issue for the custom KeyManager and
> > > TrustManager implementation as both custom classes get passed directly
> to
> > > SSLContext.init(). So this approach will not work in FIPS mode.
> > >
> > > I'll look into this more and come up with another proposal.
> > >
> > > Best Regards,
> > > Dávid
> > >
> > >
> > > Dávid Paksy <pa...@apache.org> ezt írta (időpont: 2026. aug. 17., H,
> > > 10:20):
> > >
> > >> Hi Andor,
> > >>
> > >> Many thanks for your feedback.
> > >>
> > >> Hmm, very good question. As far as I understand, in FIPS mode, the
> > >> split-truststore routing should still function.
> > >>
> > >> The certificate routing in ClientServerX509TrustManager is still
> active
> > >> in FIPS mode.
> > >>
> > >> This class "sits above" the FIPS branching. It routes:
> > >> - checkServerTrusted -> client trust manager (validates remote server
> > >> certs)
> > >> - checkClientTrusted -> server trust manager (validates remote client
> > >> certs)
> > >>
> > >> Each inner trust manager is whatever createTrustManager() returns - in
> > >> FIPS mode that's the raw JDK PKIX trust manager instead of
> > ZKTrustManager.
> > >> The routing itself is unaffected.
> > >>
> > >> Regarding EKU enforcement, there's no explicit EKU checking in
> > >> ZKTrustManager or ClientServerX509TrustManager.The JDK's built-in PKIX
> > >> validator enforces EKU constraints. So in FIPS mode, EKU validation
> > should
> > >> work identically.
> > >>
> > >> Or do you think I'm missing something?
> > >>
> > >> Best Regards,
> > >> Dávid
> > >>
> > >> Andor Molnár <an...@apache.org> ezt írta (időpont: 2026. aug. 13.,
> Cs,
> > >> 23:00):
> > >>
> > >>> Hi David,
> > >>>
> > >>> Option #1
> > >>>
> > >>> You mentioned a custom ZooKeeper trustmanager implementation which
> > picks
> > >>> the right certificate for client/server validation. How would it work
> > in
> > >>> FIPS mode when we don’t use custom trustmanagers, but rely on JDK
> > >>> implementation?
> > >>>
> > >>> Apart from this I tend to support this option. Sounds more secure and
> > it
> > >>> could include a fallback mechanism to Option #2 if only one keystore
> is
> > >>> specified in the config.
> > >>>
> > >>> Andor
> > >>>
> > >>>
> > >>>
> > >>> > On Aug 12, 2026, at 10:44, Dávid Paksy <pa...@apache.org> wrote:
> > >>> >
> > >>> > Hi All,
> > >>> >
> > >>> > I’d like to start a discussion to improve public Certificate
> > >>> Authorities
> > >>> > (CA) compatibility. Industry standards and public CA-s (like
> > DigiCert)
> > >>> are
> > >>> > sunsetting multi-use certificates, making the current requirement
> for
> > >>> dual
> > >>> > serverAuth and clientAuth Extended Key Usages (EKUs) difficult to
> > >>> manage.
> > >>> >
> > >>> > ZooKeeper currently only supports dual-purpose certificates (or no
> > EKU
> > >>> > extension at all which means unconstrained). These certificates
> carry
> > >>> both
> > >>> > the serverAuth and clientAuth EKUs, meaning the same key and
> > >>> certificate is
> > >>> > used whether the service running on the host is acting as a TLS
> > server
> > >>> or
> > >>> > as a client in a mutual-TLS (mTLS) handshake.
> > >>> > In a ZooKeeper quorum cluster, every node acts as both TLS client
> and
> > >>> > server to its peers, making the single-EKU question especially
> > >>> relevant for
> > >>> > inter-node communication.
> > >>> >
> > >>> > In order to support single EKU certs I see the following two
> possible
> > >>> > options.
> > >>> >
> > >>> > 1. **Separate keystore / truststore files for client and server TLS
> > >>> roles**
> > >>> >
> > >>> > This requires new config properties (under "ssl." and "ssl.quorum."
> > >>> > prefixes):
> > >>> > - client.keyStore.{location,password,passwordPath,type}
> > >>> > - server.trustStore.{location,password,passwordPath,type}
> > >>> >
> > >>> > For code changes we could introduce new X509ExtendedKeyManager and
> > >>> > X509ExtendedTrustManager implementations:
> > >>> > - ClientServerX509KeyManager - routes chooseServerAlias() to the
> > >>> server
> > >>> > keystore and chooseClientAlias() to a dedicated client keystore
> > >>> > - ClientServerX509TrustManager - routes checkClientTrusted()
> > >>> (validating
> > >>> > incoming client certs when acting as server) to a dedicated
> > server-role
> > >>> > truststore, and checkServerTrusted() (validating remote server
> certs
> > >>> when
> > >>> > acting as client) to the default truststore
> > >>> >
> > >>> > If the new properties are absent, we fall back to the existing
> shared
> > >>> > keystore / truststore (to be fully backward compatible).
> > >>> >
> > >>> > This approach would give independent trust chains and better
> > >>> > least-privilege in hardened environments - file permissions can
> > >>> restrict
> > >>> > which process/role accesses which key, however this means more
> files
> > >>> per
> > >>> > node and more config properties to set.
> > >>> >
> > >>> > I filed ZOOKEEPER-5070 for this and have a draft PR there.
> > >>> >
> > >>> > ---
> > >>> >
> > >>> > 2. **Single keystore with different alias for the client and server
> > >>> roles**
> > >>> >
> > >>> > We use the PKIX KeyManagerFactory which performs EKU-aware alias
> > >>> selection.
> > >>> > Here we might not need code change. But this way we are relying on
> > JDK
> > >>> > implementation behavior that isn't formally specified in the
> > >>> > KeyManagerFactory API contract; may differ across JDK vendors (IBM
> > >>> Semeru,
> > >>> > Azul, GraalVM).
> > >>> >
> > >>> > However Trust side has no equivalent auto-routing — the
> TrustManager
> > >>> uses
> > >>> > one truststore for both checkClientTrusted() and
> > checkServerTrusted().
> > >>> We
> > >>> > cannot use different CAs for client-facing vs quorum communication.
> > >>> >
> > >>> > This approach would mean fewer files, simpler deployment but
> > principle
> > >>> of
> > >>> > least privilege is weaker - a compromised process always has both
> > keys.
> > >>> >
> > >>> > ---
> > >>> >
> > >>> > This single EKU topic is also being discussed on the Apache Kafka
> > >>> developer
> > >>> > mailing list and they tend to favor option #2 (Single keystore with
> > >>> > different aliases):
> > >>> > https://lists.apache.org/list?dev@kafka.apache.org:lte=1M:eku
> > >>> >
> > >>> >
> > >>> > What are your thoughts on these?
> > >>> > Any experience using single EKU certificates with ZooKeeper?
> > >>> > Do you see more possible options?
> > >>> >
> > >>> > Best Regards,
> > >>> > Dávid
> > >>>
> > >>>
> >
>
Re: [DISCUSS] Single purpose TLS extended key usages (EKU) support
Posted by Andor Molnár <an...@apache.org>.
Sorry for the delay, I’m still validating the patch in my local dev environment.
For some reason I’m unable to get mTLS working with self-signed certificates, but I’m still looking into it.
> On Sep 1, 2026, at 08:50, Dávid Paksy <pa...@apache.org> wrote:
>
> Hi David,
>
> Many thanks for bringing this up, this is indeed a valid point that Istio
> service-mesh can manage TLS and in this case the application does not have
> to care about it.
> However as far as I understand Istio cannot be used without a Kubernetes
> cluster and if we don't have Kubernetes (e.g. use VM-s or bare metal
> servers) this is not an option for us.
> So probably it is beneficial to have a solution for this case as well.
>
> Best Regards,
> Dávid
>
> David Smiley <ds...@apache.org> ezt írta (időpont: 2026. aug. 19., Sze,
> 15:08):
>
>> Are any of you working in an organization that has embraced
>> managed-mesh/Istio [1]? It's a game changer for the better; I have seen
>> the future with that. It pushes mTLS down to an infrastructure concern;
>> the TCP/IP services above communicate in cleartext unaware of the lower
>> level encryption. No service needs to care about TLS, let alone mTLS, let
>> alone FIPS. It removed a considerable burden across all the services in
>> the organization, including complexity / quirky matters on a service by
>> service basis. Only the team managing the Istio infrastructure needed to
>> know how that stuff worked. More importantly, it significantly raised the
>> security posture across the board.
>>
>> [1] https://istio.io/latest/about/service-mesh/
>>
>> On Wed, Aug 19, 2026 at 4:36 AM Dávid Paksy <pa...@apache.org> wrote:
>>
>>> Hi,
>>>
>>> I now changed the implementation of option #1 to use separate SSLContext
>>> instances in SSLContextAndOptions for client and server role. Each
>>> SSLContext is initialized with standard PKIX managers from its own
>> keystore
>>> / truststore. There is no custom key- or trust-manager. As far as I
>>> understand, this should work in FIPS mode.
>>> There is no change in config properties, and it is fully backward
>>> compatible.
>>>
>>> What are your thoughts?
>>>
>>> Thanks,
>>> Dávid
>>>
>>> Dávid Paksy <pa...@gmail.com> ezt írta (időpont: 2026. aug. 17., H,
>>> 11:31):
>>>
>>>> Hi Andor,
>>>>
>>>> Sorry, I was too quick to send my response. Probably you didn't meant
>> the
>>>> FIPS mode property in ZooKeeper but FIPS mode in Java.
>>>>
>>>> Enabling FIPS mode in Java imposes strict constraints on which
>> KeyManager
>>>> and TrustManager implementations can be instantiated and initialized.
>>>> I looked into this and in FIPS mode, SSLContext.init() will throw an
>>>> exception if custom manager instances are passed. Something like:
>>>> java.security.KeyManagementException: FIPS mode: only SunJSSE
>>>> TrustManagers may be used.
>>>>
>>>> I agree that this can be an issue for the custom KeyManager and
>>>> TrustManager implementation as both custom classes get passed directly
>> to
>>>> SSLContext.init(). So this approach will not work in FIPS mode.
>>>>
>>>> I'll look into this more and come up with another proposal.
>>>>
>>>> Best Regards,
>>>> Dávid
>>>>
>>>>
>>>> Dávid Paksy <pa...@apache.org> ezt írta (időpont: 2026. aug. 17., H,
>>>> 10:20):
>>>>
>>>>> Hi Andor,
>>>>>
>>>>> Many thanks for your feedback.
>>>>>
>>>>> Hmm, very good question. As far as I understand, in FIPS mode, the
>>>>> split-truststore routing should still function.
>>>>>
>>>>> The certificate routing in ClientServerX509TrustManager is still
>> active
>>>>> in FIPS mode.
>>>>>
>>>>> This class "sits above" the FIPS branching. It routes:
>>>>> - checkServerTrusted -> client trust manager (validates remote server
>>>>> certs)
>>>>> - checkClientTrusted -> server trust manager (validates remote client
>>>>> certs)
>>>>>
>>>>> Each inner trust manager is whatever createTrustManager() returns - in
>>>>> FIPS mode that's the raw JDK PKIX trust manager instead of
>>> ZKTrustManager.
>>>>> The routing itself is unaffected.
>>>>>
>>>>> Regarding EKU enforcement, there's no explicit EKU checking in
>>>>> ZKTrustManager or ClientServerX509TrustManager.The JDK's built-in PKIX
>>>>> validator enforces EKU constraints. So in FIPS mode, EKU validation
>>> should
>>>>> work identically.
>>>>>
>>>>> Or do you think I'm missing something?
>>>>>
>>>>> Best Regards,
>>>>> Dávid
>>>>>
>>>>> Andor Molnár <an...@apache.org> ezt írta (időpont: 2026. aug. 13.,
>> Cs,
>>>>> 23:00):
>>>>>
>>>>>> Hi David,
>>>>>>
>>>>>> Option #1
>>>>>>
>>>>>> You mentioned a custom ZooKeeper trustmanager implementation which
>>> picks
>>>>>> the right certificate for client/server validation. How would it work
>>> in
>>>>>> FIPS mode when we don’t use custom trustmanagers, but rely on JDK
>>>>>> implementation?
>>>>>>
>>>>>> Apart from this I tend to support this option. Sounds more secure and
>>> it
>>>>>> could include a fallback mechanism to Option #2 if only one keystore
>> is
>>>>>> specified in the config.
>>>>>>
>>>>>> Andor
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Aug 12, 2026, at 10:44, Dávid Paksy <pa...@apache.org> wrote:
>>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I’d like to start a discussion to improve public Certificate
>>>>>> Authorities
>>>>>>> (CA) compatibility. Industry standards and public CA-s (like
>>> DigiCert)
>>>>>> are
>>>>>>> sunsetting multi-use certificates, making the current requirement
>> for
>>>>>> dual
>>>>>>> serverAuth and clientAuth Extended Key Usages (EKUs) difficult to
>>>>>> manage.
>>>>>>>
>>>>>>> ZooKeeper currently only supports dual-purpose certificates (or no
>>> EKU
>>>>>>> extension at all which means unconstrained). These certificates
>> carry
>>>>>> both
>>>>>>> the serverAuth and clientAuth EKUs, meaning the same key and
>>>>>> certificate is
>>>>>>> used whether the service running on the host is acting as a TLS
>>> server
>>>>>> or
>>>>>>> as a client in a mutual-TLS (mTLS) handshake.
>>>>>>> In a ZooKeeper quorum cluster, every node acts as both TLS client
>> and
>>>>>>> server to its peers, making the single-EKU question especially
>>>>>> relevant for
>>>>>>> inter-node communication.
>>>>>>>
>>>>>>> In order to support single EKU certs I see the following two
>> possible
>>>>>>> options.
>>>>>>>
>>>>>>> 1. **Separate keystore / truststore files for client and server TLS
>>>>>> roles**
>>>>>>>
>>>>>>> This requires new config properties (under "ssl." and "ssl.quorum."
>>>>>>> prefixes):
>>>>>>> - client.keyStore.{location,password,passwordPath,type}
>>>>>>> - server.trustStore.{location,password,passwordPath,type}
>>>>>>>
>>>>>>> For code changes we could introduce new X509ExtendedKeyManager and
>>>>>>> X509ExtendedTrustManager implementations:
>>>>>>> - ClientServerX509KeyManager - routes chooseServerAlias() to the
>>>>>> server
>>>>>>> keystore and chooseClientAlias() to a dedicated client keystore
>>>>>>> - ClientServerX509TrustManager - routes checkClientTrusted()
>>>>>> (validating
>>>>>>> incoming client certs when acting as server) to a dedicated
>>> server-role
>>>>>>> truststore, and checkServerTrusted() (validating remote server
>> certs
>>>>>> when
>>>>>>> acting as client) to the default truststore
>>>>>>>
>>>>>>> If the new properties are absent, we fall back to the existing
>> shared
>>>>>>> keystore / truststore (to be fully backward compatible).
>>>>>>>
>>>>>>> This approach would give independent trust chains and better
>>>>>>> least-privilege in hardened environments - file permissions can
>>>>>> restrict
>>>>>>> which process/role accesses which key, however this means more
>> files
>>>>>> per
>>>>>>> node and more config properties to set.
>>>>>>>
>>>>>>> I filed ZOOKEEPER-5070 for this and have a draft PR there.
>>>>>>>
>>>>>>> ---
>>>>>>>
>>>>>>> 2. **Single keystore with different alias for the client and server
>>>>>> roles**
>>>>>>>
>>>>>>> We use the PKIX KeyManagerFactory which performs EKU-aware alias
>>>>>> selection.
>>>>>>> Here we might not need code change. But this way we are relying on
>>> JDK
>>>>>>> implementation behavior that isn't formally specified in the
>>>>>>> KeyManagerFactory API contract; may differ across JDK vendors (IBM
>>>>>> Semeru,
>>>>>>> Azul, GraalVM).
>>>>>>>
>>>>>>> However Trust side has no equivalent auto-routing — the
>> TrustManager
>>>>>> uses
>>>>>>> one truststore for both checkClientTrusted() and
>>> checkServerTrusted().
>>>>>> We
>>>>>>> cannot use different CAs for client-facing vs quorum communication.
>>>>>>>
>>>>>>> This approach would mean fewer files, simpler deployment but
>>> principle
>>>>>> of
>>>>>>> least privilege is weaker - a compromised process always has both
>>> keys.
>>>>>>>
>>>>>>> ---
>>>>>>>
>>>>>>> This single EKU topic is also being discussed on the Apache Kafka
>>>>>> developer
>>>>>>> mailing list and they tend to favor option #2 (Single keystore with
>>>>>>> different aliases):
>>>>>>> https://lists.apache.org/list?dev@kafka.apache.org:lte=1M:eku
>>>>>>>
>>>>>>>
>>>>>>> What are your thoughts on these?
>>>>>>> Any experience using single EKU certificates with ZooKeeper?
>>>>>>> Do you see more possible options?
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Dávid
>>>>>>
>>>>>>
>>>
>>