You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "d80tb7 (via GitHub)" <gi...@apache.org> on 2023/04/25 09:01:34 UTC
[GitHub] [pulsar-helm-chart] d80tb7 opened a new issue, #359: Zookeeper SSL Failures When Certificate Is Rolled
d80tb7 opened a new issue, #359:
URL: https://github.com/apache/pulsar-helm-chart/issues/359
**Describe the bug**
Zookeeper doesn't handle SSL certificate rolling gracefully. Specifically, if a certificate is rolled Zookeeper will continue to use the old, expired cert until it is restarted, which can lead to an outage as other components will be unable to communicate with it.
I'm not sure if this is an issue with the Pulsar Helm chart, or with Pulsar itself. If the latter, please let me know and I'll raise the issue there.
**To Reproduce**
Steps to reproduce the behavior:
The following is valid for Pulsar 2.92 using Helm chart 2.92
1. Deploy Pulsar into a K8s cluster using the Helm chart with tls enabled for zookeeper and certs managed by certmanager
2. Wait for the certificate to be rolled
3. See connections to zookeeper fail with "io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed...Caused by java.security.cert.CertificateExpiredException "
4. Restart Pulsar Pods and see that errors go away
**Expected behavior**
Pulsar should continue to operate normally when a certificate is rolled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Zookeeper SSL Failures When Certificate Is Rolled [pulsar-helm-chart]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari closed issue #359: Zookeeper SSL Failures When Certificate Is Rolled
URL: https://github.com/apache/pulsar-helm-chart/issues/359
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Zookeeper SSL Failures When Certificate Is Rolled [pulsar-helm-chart]
Posted by "bhavyaravilla (via GitHub)" <gi...@apache.org>.
bhavyaravilla commented on issue #359:
URL: https://github.com/apache/pulsar-helm-chart/issues/359#issuecomment-1884695225
I have the same issue. And zookeeper keeps failing with the below errors
2024-01-10T11:30:44,178+0000 [epollEventLoopGroup-7-1] ERROR org.apache.zookeeper.server.NettyServerCnxnFactory - Unsuccessful handshake with session 0x02024-01-10T11:30:44,178+0000 [epollEventLoopGroup-7-1] WARN org.apache.zookeeper.server.NettyServerCnxnFactory - Exception caughtio.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[io.netty-netty-codec-4.1.93.Final.jar:4.1.93.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[io.netty-netty-codec-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.netty-netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.
93.Final.jar:4.1.93.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[io.netty-netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) ~[io.netty-netty-transport-classes-epoll-4.1
.93.Final.jar:4.1.93.Final] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499) ~[io.netty-netty-transport-classes-epoll-4.1.93.Final.jar:4.1.93.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) ~[io.netty-netty-transport-classes-epoll-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty-netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.93.Final.jar:4.1.93.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.93.Final.jar:4.1.93.Final] at java.lang.Thread.run(Thread.java:833) ~[?:?]Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException
(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:365) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:297) ~[io.netty-netty-handler-4.1.93.Final.jar:4.1.93.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1353) ~[io.netty-netty-handler-4.1.93.Final.jar:4.1.93.Final]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Zookeeper SSL Failures When Certificate Is Rolled [pulsar-helm-chart]
Posted by "HaimKortovich (via GitHub)" <gi...@apache.org>.
HaimKortovich commented on issue #359:
URL: https://github.com/apache/pulsar-helm-chart/issues/359#issuecomment-2403516331
Restarting zookeeper did not fix the error:
` Caused by: java.security.cert.CertificateExpiredException: NotAfter: Tue Oct 01 17:19:15 UTC 2024
at sun.security.x509.CertificateValidity.valid(CertificateValidity.java:277) ~[?:?]
at sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:621) ~[?:?]
at sun.security.provider.certpath.BasicChecker.verifyValidity(BasicChecker.java:190) ~[?:?]
at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:144) ~[?:?]
at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125) ~[?:?]
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:224) ~[?:?]
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:144) ~[?:?]
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83) ~[?:?]
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:364) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:138) ~[?:?]
at io.netty.handler.ssl.EnhancingX509ExtendedTrustManager.checkClientTrusted(EnhancingX509ExtendedTrustManager.java:62) ~[io.netty-netty-handler-4.1.108.Final.jar:4.1.108.Final]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1273) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1198) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1651) ~[io.netty-netty-handler-4.1.108.Final.jar:4.1.108.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1497) ~[io.netty-netty-handler-4.1.108.Final.jar:4.1.108.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) ~[io.netty-netty-handler-4.1.108.Final.jar:4.1.108.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[io.netty-netty-handler-4.1.108.Final.jar:4.1.108.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[io.netty-netty-codec-4.1.108.Final.jar:4.1.108.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[io.netty-netty-codec-4.1.108.Final.jar:4.1.108.Final]
... 15 more`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Zookeeper SSL Failures When Certificate Is Rolled [pulsar-helm-chart]
Posted by "lenglet-k (via GitHub)" <gi...@apache.org>.
lenglet-k commented on issue #359:
URL: https://github.com/apache/pulsar-helm-chart/issues/359#issuecomment-2915167205
> > Hello,
> > We have the same problem, in fact, the zookeeper pod obtains this certificate with certificateManager, this certificate has a duration of 2160h and this pod executes this command:
> > /pulsar/keytool/keytool.sh
> > This script does not update the jks in the /tmp/directory directory.
> > So during the certificate renewal process, zookeeper does not update this jks file, so the ssl certificate is not up to date and the broker/bookie and the other components crash.
>
> Yes, this is a problem. The keytool.sh script is pretty much useless since Java has direct support for pkcs12 keystore format. The script probably predates pkcs12 support in Java when you needed to convert to JKS format. The pkcs12 support became the default in Java 9 with JEP 229, https://openjdk.org/jeps/229.
>
> [@lenglet-k](https://github.com/lenglet-k) would you be interested in contributing a solution where the keytool.sh script would be eliminated completely? I'd assume that there would be a better chance in getting the certificate update to work if there wouldn't be a need to convert the certificate files once they get updated.
>
> The defaults in Pulsar are to reload the certs for ZK server: https://github.com/apache/pulsar/blob/397b0211cf1c9069d80daa691ca530e09bc37999/conf/zookeeper.conf#L66-L69
>
> I guess `-Dzookeeper.client.certReload=true` would be needed on Zookeeper clients in addition to the current TLS options. (docs: https://zookeeper.apache.org/doc/r3.9.3/zookeeperAdmin.html)
>
> Zookeeper [has direct support for pkcs12 keystore format besides JKS, PEM and BCFKS](https://zookeeper.apache.org/doc/r3.9.3/zookeeperAdmin.html#:~:text=ssl.trustStore.type%20and%20ssl.quorum.trustStore.type) too:
>
> > ssl.trustStore.type and ssl.quorum.trustStore.type : (Java system properties: zookeeper.ssl.trustStore.type and zookeeper.ssl.quorum.trustStore.type) New in 3.5.5: Specifies the file format of client and quorum trustStores. Values: JKS, PEM, PKCS12 or null (detect by filename). Default: null. New in 3.5.10, 3.6.3, 3.7.0: The format BCFKS was added.
>
> It seems that cert-manager creates PEM format files by default. I thought that PKCS12 was configured to be used in pulsar-helm-chart since the keytool.sh script references pkcs12.
Hello,
For me there is 2 solutions:
1. Keep keystore and truststore but use certmanager for generate keystore file directly: https://cert-manager.io/docs/releases/release-notes/release-notes-0.15/#general-availability-of-jks-and-pkcs12-keystores, then mount secret in zookeeper.
2. Use PEM ssl type in `ssl.quorum.trustStore.type` and `ssl.trustStore.type`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Zookeeper SSL Failures When Certificate Is Rolled [pulsar-helm-chart]
Posted by "Loahrs (via GitHub)" <gi...@apache.org>.
Loahrs commented on issue #359:
URL: https://github.com/apache/pulsar-helm-chart/issues/359#issuecomment-2082587454
I am encountering the same issue with version **3.3.0** of the helm chart. The Pulsar Pods threw SSL-Exception( "notAfter: 15.04.2024").
Restarting the pods solved the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Zookeeper SSL Failures When Certificate Is Rolled [pulsar-helm-chart]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #359:
URL: https://github.com/apache/pulsar-helm-chart/issues/359#issuecomment-2447462849
Another issue report: #524
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Zookeeper SSL Failures When Certificate Is Rolled [pulsar-helm-chart]
Posted by "d80tb7 (via GitHub)" <gi...@apache.org>.
d80tb7 commented on issue #359:
URL: https://github.com/apache/pulsar-helm-chart/issues/359#issuecomment-2081182686
I think the issue here is that although the Pulsar Helm Chart sets the `zookeeper.client.certReload` property, this isn't enough. All that property does is to get Zookeeper to update the certs when the truststore or keystore files change. When cert-manager updates the certs, this will cause the cert failes in `pulsar/certs/zookeeper/` to update but nothing is going to update the keystore.
The other Pulsar components (e.g. the bookie) solve this by having code inside them that watches the files under `/pulsar/certs/` and then updates the keystore accordingly. Zookeeper doesn't have such code and therefore it seems to me that the certs will never be refreshed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Zookeeper SSL Failures When Certificate Is Rolled [pulsar-helm-chart]
Posted by "beelis (via GitHub)" <gi...@apache.org>.
beelis commented on issue #359:
URL: https://github.com/apache/pulsar-helm-chart/issues/359#issuecomment-2582120929
Same issue with version 3.7.0 (pulsar 4.0.0). After a restart of all pulsar pods, the zookeeper could recover.
After that, the pulsar functions were not able to connect to the broker, they also seem to have the old cert cached:
`2025/01/10 08:31:36.353 asm_amd64.s:1700: [info] Connecting to broker remote_addr=pulsar+ssl://pulsar-broker:6651
2025/01/10 08:31:36.368 asm_amd64.s:1700: [warning] Failed to connect to broker. error=tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-01-10T08:31:36Z is after 2025-01-05T18:12:06Z remote_addr=pulsar+ssl://pulsar-broker:6651`
Restarting the pulsar function pod did not help, only a redeploy of the function solved the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Zookeeper SSL Failures When Certificate Is Rolled [pulsar-helm-chart]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #359:
URL: https://github.com/apache/pulsar-helm-chart/issues/359#issuecomment-2913204154
> Hello,
>
> We have the same problem, in fact, the zookeeper pod obtains this certificate with certificateManager, this certificate has a duration of 2160h and this pod executes this command:
>
> /pulsar/keytool/keytool.sh
>
> This script does not update the jks in the /tmp/directory directory.
>
> So during the certificate renewal process, zookeeper does not update this jks file, so the ssl certificate is not up to date and the broker/bookie and the other components crash.
Yes, this is a problem. The keytool.sh script is pretty much useless since Java has direct support for pkcs12 keystore format. The script probably predates pkcs12 support in Java when you needed to convert to JKS format. The pkcs12 support became the default in Java 9 with JEP 229, https://openjdk.org/jeps/229.
@lenglet-k would you be interested in contributing a solution where the keytool.sh script would be eliminated completely?
I'd assume that there would be a better chance in getting the certificate update to work if there wouldn't be a need to convert the certificate files once they get updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Zookeeper SSL Failures When Certificate Is Rolled [pulsar-helm-chart]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #359:
URL: https://github.com/apache/pulsar-helm-chart/issues/359#issuecomment-2916591995
> 2\. Use PEM ssl type in `ssl.quorum.trustStore.type` and `ssl.trustStore.type`
@lenglet-k this is what I'm working on in #613
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Zookeeper SSL Failures When Certificate Is Rolled [pulsar-helm-chart]
Posted by "lenglet-k (via GitHub)" <gi...@apache.org>.
lenglet-k commented on issue #359:
URL: https://github.com/apache/pulsar-helm-chart/issues/359#issuecomment-2913071339
Hello,
We have the same problem, in fact, the zookeeper pod obtains this certificate with certificateManager, this certificate has a duration of 2160h and this pod executes this command:
/pulsar/keytool/keytool.sh
This script does not update the jks in the /tmp/directory directory.
So during the certificate renewal process, zookeeper does not update this jks file, so the ssl certificate is not up to date and the broker/bookie and the other components crash.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org