You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "sagb (via GitHub)" <gi...@apache.org> on 2024/07/19 07:36:31 UTC

[I] Kubernetes certificates not automatically renewing [cloudstack]

sagb opened a new issue, #9418:
URL: https://github.com/apache/cloudstack/issues/9418

   <!--
   Verify first that your issue/request is not already reported on GitHub.
   Also test if the latest release and main branch are affected too.
   Always add information AFTER of these HTML comments, but no need to delete the comments.
   -->
   
   ##### ISSUE TYPE
   <!-- Pick one below and delete the rest -->
   Bug Report
   
   ##### COMPONENT NAME
   <!--
   Categorize the issue, e.g. API, VR, VPN, UI, etc.
   -->
   Kubernetes
   
   ##### CLOUDSTACK VERSION
   <!--
   New line separated list of affected versions, commit ID for issues on main branch.
   -->
   4.19.0.2
   
   ##### CONFIGURATION
   <!--
   Information about the configuration if relevant, e.g. basic network, advanced networking, etc.  N/A otherwise
   -->
   Kubernetes 1.27.3, two control nodes.
   
   
   ##### SUMMARY
   <!-- Explain the problem/feature briefly -->
   Our K8s certificates expired. Seems like Cloudstack didn't automatically renew them.  
   I tried to renew them manually on both control nodes using:
   ```
   kubeadm certs renew all
   systemctl restart kubelet
   ```
   This updated the certificates separately for each control node, and both are recognized by Kubernetes when using /etc/kubernetes/admin.conf as kubeconfig. However, CloudStack's "Kubernetes access" page only showed the old, expired certificate.
   
   In attempt to trigger an automatic renewal, I've restored nodes from snapshot, stopped the k8s cluster from Cloudstack's web UI, and started it.
   It doesn't start, and there is an exception in /var/log/cloudstack/management/management-server.log:
   
   ```
   2024-07-19 10:19:43,690 WARN  [c.c.k.c.u.KubernetesClusterUtil] (API-Job-Executor-19:ctx-b594d6ec job-4889 ctx-21a09500) (logid:767ef7b5) API endpoint for Kubernetes cluster : mediatech not available
   javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
   	at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1701)
   	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1519)
   	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421)
   	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
   	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427)
   	at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:580)
   	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:201)
   	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1614)
   	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1542)
   	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
   	at com.cloud.kubernetes.cluster.utils.KubernetesClusterUtil.isKubernetesClusterServerRunning(KubernetesClusterUtil.java:239)
   	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterStartWorker.startStoppedKubernetesCluster(KubernetesClusterStartWorker.java:590)
   	at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.startKubernetesCluster(KubernetesClusterManagerImpl.java:1324)
   	at org.apache.cloudstack.api.command.user.kubernetes.cluster.StartKubernetesClusterCmd.execute(StartKubernetesClusterCmd.java:113)
   	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
   	at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
   	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
   	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
   	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
   	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
   	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
   	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
   	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
   	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
   	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   	at java.base/java.lang.Thread.run(Thread.java:829)
   Caused by: java.io.EOFException: SSL peer shut down incorrectly
   	at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
   	at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
   	at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
   	... 28 more
   ```
   
   The abscence of automatic renewal is probably a bug.
   
   Also, I would be grateful for a hint how to recover from the current situation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Kubernetes certificates not automatically renewing [cloudstack]

Posted by "kiranchavala (via GitHub)" <gi...@apache.org>.
kiranchavala commented on issue #9418:
URL: https://github.com/apache/cloudstack/issues/9418#issuecomment-2250135189

   @sagb 
   
   Cloudstack doesn't provide a way to automatic renewal k8s  component certificates. 
   
   Your request can be an improvement request.
   
   
   
   When  you launch a cks cluster , internally CKS uses kubeadm to setup the kubernetes cluster
   
   Client certificates generated by [kubeadm](https://kubernetes.io/docs/reference/setup-tools/kubeadm/) expire after 1 year.
   
   
   ```
   root@test-control-190e8277e14:~# kubeadm certs check-expiration
   [check-expiration] Reading configuration from the cluster...
   [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
   
   CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
   admin.conf                 Jul 25, 2025 04:32 UTC   364d            ca                      no
   apiserver                  Jul 25, 2025 04:32 UTC   364d            ca                      no
   apiserver-etcd-client      Jul 25, 2025 04:32 UTC   364d            etcd-ca                 no
   apiserver-kubelet-client   Jul 25, 2025 04:32 UTC   364d            ca                      no
   controller-manager.conf    Jul 25, 2025 04:32 UTC   364d            ca                      no
   etcd-healthcheck-client    Jul 25, 2025 04:32 UTC   364d            etcd-ca                 no
   etcd-peer                  Jul 25, 2025 04:32 UTC   364d            etcd-ca                 no
   etcd-server                Jul 25, 2025 04:32 UTC   364d            etcd-ca                 no
   front-proxy-client         Jul 25, 2025 04:32 UTC   364d            front-proxy-ca          no
   scheduler.conf             Jul 25, 2025 04:32 UTC   364d            ca                      no
   
   CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
   ca                      Jul 23, 2034 04:32 UTC   9y              no
   etcd-ca                 Jul 23, 2034 04:32 UTC   9y              no
   front-proxy-ca          Jul 23, 2034 04:32 UTC   9y              no
   ```
   
   
   Its up the admin user to login to control node and renew the client certificates 
   
   
   
   
   As a workaround
   
   Login to the control node  and try to delete the following pods after executing "kubeadm certs renew all".
   
   ```
   
           root@primary1-node:~# kubectl delete pod -n kube-system -l component=kube-apiserver
           root@primary1-node:~# kubectl delete pod -n kube-system -l component=kube-scheduler
           root@primary1-node:~# kubectl delete pod -n kube-system -l component=kube-controller-manager
           root@primary1-node:~# kubectl delete pod -n kube-system -l component=etcd
           
   ```
   
   Another workaround is to upgrade the kubernetes version 
   
   kubeadm renews all the certificates during control plane [upgrade](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/).
   
   Register 1.28.4 cks iso and upgrade the cks cluster which should renew the certificates 
   
    https://download.cloudstack.org/cks/
    
    ref:
   
   https://www.juniper.net/documentation/us/en/software/paragon-automation23.2/paragon-automation-troubleshooting-guide/topics/task/tg-manual-renew-kubeadm-cert.html
   
   https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Kubernetes certificates not automatically renewing [cloudstack]

Posted by "sagb (via GitHub)" <gi...@apache.org>.
sagb commented on issue #9418:
URL: https://github.com/apache/cloudstack/issues/9418#issuecomment-2238602307

   The certificates on k8s control nodes are really expired:
   ```
   control-node-1:~# kubeadm certs check-expiration
   [check-expiration] Reading configuration from the cluster...
   [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
   [check-expiration] Error reading configuration from the Cluster. Falling back to default configuration
   
   CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
   admin.conf                 Jul 18, 2024 08:23 UTC   <invalid>       ca                      no
   apiserver                  Jul 18, 2024 08:22 UTC   <invalid>       ca                      no
   apiserver-etcd-client      Jul 18, 2024 08:22 UTC   <invalid>       etcd-ca                 no
   apiserver-kubelet-client   Jul 18, 2024 08:22 UTC   <invalid>       ca                      no
   controller-manager.conf    Jul 18, 2024 08:22 UTC   <invalid>       ca                      no
   etcd-healthcheck-client    May 26, 2024 08:19 UTC   <invalid>       etcd-ca                 no
   etcd-peer                  May 26, 2024 08:19 UTC   <invalid>       etcd-ca                 no
   etcd-server                May 26, 2024 08:19 UTC   <invalid>       etcd-ca                 no
   front-proxy-client         Jul 18, 2024 08:22 UTC   <invalid>       front-proxy-ca          no
   scheduler.conf             Jul 18, 2024 08:23 UTC   <invalid>       ca                      no
   
   CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
   ca                      May 24, 2033 08:19 UTC   8y              no
   etcd-ca                 May 24, 2033 08:19 UTC   8y              no
   front-proxy-ca          May 24, 2033 08:19 UTC   8y              no
   ```
   Since Cloudstack shows the expired certificate in "kubernetes control" web ui page, it has some control over them. How can I trigger the renewal?
   
   It doesn't seem that Cloudstack will be able to connect to the control nodes later, as it has already been trying for several hours.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Kubernetes certificates not automatically renewing [cloudstack]

Posted by "sagb (via GitHub)" <gi...@apache.org>.
sagb commented on issue #9418:
URL: https://github.com/apache/cloudstack/issues/9418#issuecomment-2254205637

   Thank you for the detailed reply.
   
   My issue was complicated by a split-brain situation with etcd on the control nodes, likely caused by restoring control nodes from snapshots. We manually upgraded the certificates, then removed one of the nodes from the cluster and bootstrapped it again. This seems to have fixed everything.
   
   The new kubectl appeared in the CloudStack GUI soon after this, though I still don't understand where it comes from.
   
   To summarize your reply and the third-party documentation, the options for upgrading Kubernetes certificates in CloudStack are clear: either upgrade the certificates manually or upgrade the Kubernetes version.
   
   I think the issue can be closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Kubernetes certificates not automatically renewing [cloudstack]

Posted by "sagb (via GitHub)" <gi...@apache.org>.
sagb closed issue #9418: Kubernetes certificates not automatically renewing
URL: https://github.com/apache/cloudstack/issues/9418


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Kubernetes certificates not automatically renewing [cloudstack]

Posted by "shwstppr (via GitHub)" <gi...@apache.org>.
shwstppr commented on issue #9418:
URL: https://github.com/apache/cloudstack/issues/9418#issuecomment-2238574881

   @sagb I don't think current code has the auto-renew functionality.
   https://github.com/apache/cloudstack/blob/main/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterStartWorker.java#L147-L149
   
   Also, I can see certificates from ACS are issued for 10 years so it is interesting why it expired in your case.
   
   The logs you shared are WARN level. They are logged initially when ACS MS tries to connect to the control node. The operation was tried repeatedly several times. It may be possible that the management server was able to connect to the control node later


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Kubernetes certificates not automatically renewing [cloudstack]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #9418:
URL: https://github.com/apache/cloudstack/issues/9418#issuecomment-2238559966

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org