You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/08/11 12:53:34 UTC
[GitHub] [pulsar-helm-chart] danny-krueger opened a new issue, #285: Automatic certificate renewal does not work
danny-krueger opened a new issue, #285:
URL: https://github.com/apache/pulsar-helm-chart/issues/285
**Describe the bug**
The reissued certificates from the Cert Manager will not be used by Pulsar until the pods are rebooted.
**To Reproduce**
Steps to reproduce the behavior:
1. Activate Cert Manager with the internal issuer.
2. Wait until the certificate has expired.
3. The certificates have been updated, but the Java Services have not checked this and SSL errors occur in the Zookeeper.
`ERROR org.apache.zookeeper.server.NettyServerCnxnFactory - Unsuccessful handshake with session 0x0`
`2022-08-03T14:53:45,862+0000 [epollEventLoopGroup-7-2] WARN org.apache.zookeeper.server.NettyServerCnxnFactory - Exception caught
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[io.netty-netty-codec-4.1.74.Final.jar:4.1.74.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[io.netty-netty-codec-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) [io.netty-netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [io.netty-netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [io.netty-netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]`
**Expected behavior**
When the Cert Manager issues new certificates, Pulsar should also be aware of this and use the new certificates.
**Additional context**
The certificates have been correctly reissued by the Cert Manager. They were also correct in the Config Maps and in the Secrets. Also where we were in the pods per shell, the new correct SSL certificates were there. But since Pulsar itself was already running, it did not re-read them. We think that Java caches the certificates.
**Quick fix**
After all pods were restarted everything worked again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] danny-krueger commented on issue #285: Automatic certificate renewal does not work
Posted by GitBox <gi...@apache.org>.
danny-krueger commented on issue #285:
URL: https://github.com/apache/pulsar-helm-chart/issues/285#issuecomment-1285667038
Thanks for fixing it. We will try it out, if we update to the next version.
Currently we have helped ourselves by restarting all pods one after the other when the certificate is updated. We have solved this with [Stakater Reloader](https://github.com/stakater/Reloader). This worked great for us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Automatic certificate renewal does not work [pulsar-helm-chart]
Posted by "d80tb7 (via GitHub)" <gi...@apache.org>.
d80tb7 commented on issue #285:
URL: https://github.com/apache/pulsar-helm-chart/issues/285#issuecomment-2081183119
Just to check- has anyone got cert refresh working with zookeeper? I'm of the opinion that this still won't work. Copying the explanation here from [my comment ](https://github.com/apache/pulsar-helm-chart/issues/359#issuecomment-2081182686)on #359 .
```I think the issue here is that although the Pulsar Helm Chart sets the zookeeper.client.certReload property, this isn't enough. All that property does is to get Zookeeper to update the certs when the truststore or keystore files change. When cert-manager updates the certs, this will cause the cert failes in pulsar/certs/zookeeper/ to update but nothing is going to update the keystore.
The other Pulsar components (e.g. the bookie) solve this by having code inside them that watches the files under /pulsar/certs/ and then updates the keystore accordingly. Zookeeper doesn't have such code and therefore it seems to me that the certs will never be refreshed.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] michaeljmarshall commented on issue #285: Automatic certificate renewal does not work
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #285:
URL: https://github.com/apache/pulsar-helm-chart/issues/285#issuecomment-1282951696
The right place to fix this is in our default zk config. I've opened a PR here: https://github.com/apache/pulsar/pull/18097.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] michaeljmarshall commented on issue #285: Automatic certificate renewal does not work
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #285:
URL: https://github.com/apache/pulsar-helm-chart/issues/285#issuecomment-1282922120
Looks like my fork was out of date and I was looking at old information. This issue has already been resolved in Apache Zookeeper just earlier this year: https://issues.apache.org/jira/browse/ZOOKEEPER-3806. Here is the relevant [documentation](https://github.com/apache/zookeeper/blob/master/zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md):
> client.certReload : (Java system property: zookeeper.client.certReload) New in 3.7.2, 3.8.1, 3.9.0: Allows client SSL keyStore and trustStore reloading when the certificates on the filesystem change without having to restart the ZK process. Default: false
We can keep this issue open for now. Then, we we upgrade to a version of zookeeper that supports the feature, we'll add it in as a default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] michaeljmarshall commented on issue #285: Automatic certificate renewal does not work
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #285:
URL: https://github.com/apache/pulsar-helm-chart/issues/285#issuecomment-1282912631
@danny-krueger - it's possible that zookeeper does not support reloading certificates for the client facing server. I can see that quorum server allows for reloading cert files, but I don't see the same setting for the client facing server:
https://github.com/apache/zookeeper/blob/b4f9aab099880ba8ef08eaff697debe6cdeae057/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/QuorumPeerConfig.java#L78
@eolivelli - would you happen to know?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] michaeljmarshall commented on issue #285: Automatic certificate renewal does not work
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #285:
URL: https://github.com/apache/pulsar-helm-chart/issues/285#issuecomment-1285770897
@danny-krueger - thanks for highlighting a work around. Apache Zookeeper hasn't released 3.8.1 or 3.9.0, so the complete feature won't be available until ZK gets released and Pulsar upgrades to use that ZK version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] nodece closed issue #285: Automatic certificate renewal does not work
Posted by GitBox <gi...@apache.org>.
nodece closed issue #285: Automatic certificate renewal does not work
URL: https://github.com/apache/pulsar-helm-chart/issues/285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org