You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "sagb (via GitHub)" <gi...@apache.org> on 2024/07/19 07:36:31 UTC
[I] Kubernetes certificates not automatically renewing [cloudstack]
sagb opened a new issue, #9418:
URL: https://github.com/apache/cloudstack/issues/9418
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
Kubernetes
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main branch.
-->
4.19.0.2
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network, advanced networking, etc. N/A otherwise
-->
Kubernetes 1.27.3, two control nodes.
##### SUMMARY
<!-- Explain the problem/feature briefly -->
Our K8s certificates expired. Seems like Cloudstack didn't automatically renew them.
I tried to renew them manually on both control nodes using:
```
kubeadm certs renew all
systemctl restart kubelet
```
This updated the certificates separately for each control node, and both are recognized by Kubernetes when using /etc/kubernetes/admin.conf as kubeconfig. However, CloudStack's "Kubernetes access" page only showed the old, expired certificate.
In attempt to trigger an automatic renewal, I've restored nodes from snapshot, stopped the k8s cluster from Cloudstack's web UI, and started it.
It doesn't start, and there is an exception in /var/log/cloudstack/management/management-server.log:
```
2024-07-19 10:19:43,690 WARN [c.c.k.c.u.KubernetesClusterUtil] (API-Job-Executor-19:ctx-b594d6ec job-4889 ctx-21a09500) (logid:767ef7b5) API endpoint for Kubernetes cluster : mediatech not available
javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1701)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1519)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427)
at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:580)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:201)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1614)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1542)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
at com.cloud.kubernetes.cluster.utils.KubernetesClusterUtil.isKubernetesClusterServerRunning(KubernetesClusterUtil.java:239)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterStartWorker.startStoppedKubernetesCluster(KubernetesClusterStartWorker.java:590)
at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.startKubernetesCluster(KubernetesClusterManagerImpl.java:1324)
at org.apache.cloudstack.api.command.user.kubernetes.cluster.StartKubernetesClusterCmd.execute(StartKubernetesClusterCmd.java:113)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
... 28 more
```
The abscence of automatic renewal is probably a bug.
Also, I would be grateful for a hint how to recover from the current situation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kubernetes certificates not automatically renewing [cloudstack]
Posted by "kiranchavala (via GitHub)" <gi...@apache.org>.
kiranchavala commented on issue #9418:
URL: https://github.com/apache/cloudstack/issues/9418#issuecomment-2250135189
@sagb
Cloudstack doesn't provide a way to automatic renewal k8s component certificates.
Your request can be an improvement request.
When you launch a cks cluster , internally CKS uses kubeadm to setup the kubernetes cluster
Client certificates generated by [kubeadm](https://kubernetes.io/docs/reference/setup-tools/kubeadm/) expire after 1 year.
```
root@test-control-190e8277e14:~# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jul 25, 2025 04:32 UTC 364d ca no
apiserver Jul 25, 2025 04:32 UTC 364d ca no
apiserver-etcd-client Jul 25, 2025 04:32 UTC 364d etcd-ca no
apiserver-kubelet-client Jul 25, 2025 04:32 UTC 364d ca no
controller-manager.conf Jul 25, 2025 04:32 UTC 364d ca no
etcd-healthcheck-client Jul 25, 2025 04:32 UTC 364d etcd-ca no
etcd-peer Jul 25, 2025 04:32 UTC 364d etcd-ca no
etcd-server Jul 25, 2025 04:32 UTC 364d etcd-ca no
front-proxy-client Jul 25, 2025 04:32 UTC 364d front-proxy-ca no
scheduler.conf Jul 25, 2025 04:32 UTC 364d ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jul 23, 2034 04:32 UTC 9y no
etcd-ca Jul 23, 2034 04:32 UTC 9y no
front-proxy-ca Jul 23, 2034 04:32 UTC 9y no
```
Its up the admin user to login to control node and renew the client certificates
As a workaround
Login to the control node and try to delete the following pods after executing "kubeadm certs renew all".
```
root@primary1-node:~# kubectl delete pod -n kube-system -l component=kube-apiserver
root@primary1-node:~# kubectl delete pod -n kube-system -l component=kube-scheduler
root@primary1-node:~# kubectl delete pod -n kube-system -l component=kube-controller-manager
root@primary1-node:~# kubectl delete pod -n kube-system -l component=etcd
```
Another workaround is to upgrade the kubernetes version
kubeadm renews all the certificates during control plane [upgrade](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/).
Register 1.28.4 cks iso and upgrade the cks cluster which should renew the certificates
https://download.cloudstack.org/cks/
ref:
https://www.juniper.net/documentation/us/en/software/paragon-automation23.2/paragon-automation-troubleshooting-guide/topics/task/tg-manual-renew-kubeadm-cert.html
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kubernetes certificates not automatically renewing [cloudstack]
Posted by "sagb (via GitHub)" <gi...@apache.org>.
sagb commented on issue #9418:
URL: https://github.com/apache/cloudstack/issues/9418#issuecomment-2238602307
The certificates on k8s control nodes are really expired:
```
control-node-1:~# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jul 18, 2024 08:23 UTC <invalid> ca no
apiserver Jul 18, 2024 08:22 UTC <invalid> ca no
apiserver-etcd-client Jul 18, 2024 08:22 UTC <invalid> etcd-ca no
apiserver-kubelet-client Jul 18, 2024 08:22 UTC <invalid> ca no
controller-manager.conf Jul 18, 2024 08:22 UTC <invalid> ca no
etcd-healthcheck-client May 26, 2024 08:19 UTC <invalid> etcd-ca no
etcd-peer May 26, 2024 08:19 UTC <invalid> etcd-ca no
etcd-server May 26, 2024 08:19 UTC <invalid> etcd-ca no
front-proxy-client Jul 18, 2024 08:22 UTC <invalid> front-proxy-ca no
scheduler.conf Jul 18, 2024 08:23 UTC <invalid> ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca May 24, 2033 08:19 UTC 8y no
etcd-ca May 24, 2033 08:19 UTC 8y no
front-proxy-ca May 24, 2033 08:19 UTC 8y no
```
Since Cloudstack shows the expired certificate in "kubernetes control" web ui page, it has some control over them. How can I trigger the renewal?
It doesn't seem that Cloudstack will be able to connect to the control nodes later, as it has already been trying for several hours.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kubernetes certificates not automatically renewing [cloudstack]
Posted by "sagb (via GitHub)" <gi...@apache.org>.
sagb commented on issue #9418:
URL: https://github.com/apache/cloudstack/issues/9418#issuecomment-2254205637
Thank you for the detailed reply.
My issue was complicated by a split-brain situation with etcd on the control nodes, likely caused by restoring control nodes from snapshots. We manually upgraded the certificates, then removed one of the nodes from the cluster and bootstrapped it again. This seems to have fixed everything.
The new kubectl appeared in the CloudStack GUI soon after this, though I still don't understand where it comes from.
To summarize your reply and the third-party documentation, the options for upgrading Kubernetes certificates in CloudStack are clear: either upgrade the certificates manually or upgrade the Kubernetes version.
I think the issue can be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kubernetes certificates not automatically renewing [cloudstack]
Posted by "sagb (via GitHub)" <gi...@apache.org>.
sagb closed issue #9418: Kubernetes certificates not automatically renewing
URL: https://github.com/apache/cloudstack/issues/9418
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kubernetes certificates not automatically renewing [cloudstack]
Posted by "shwstppr (via GitHub)" <gi...@apache.org>.
shwstppr commented on issue #9418:
URL: https://github.com/apache/cloudstack/issues/9418#issuecomment-2238574881
@sagb I don't think current code has the auto-renew functionality.
https://github.com/apache/cloudstack/blob/main/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterStartWorker.java#L147-L149
Also, I can see certificates from ACS are issued for 10 years so it is interesting why it expired in your case.
The logs you shared are WARN level. They are logged initially when ACS MS tries to connect to the control node. The operation was tried repeatedly several times. It may be possible that the management server was able to connect to the control node later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Kubernetes certificates not automatically renewing [cloudstack]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #9418:
URL: https://github.com/apache/cloudstack/issues/9418#issuecomment-2238559966
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org