You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "havengit (via GitHub)" <gi...@apache.org> on 2024/09/23 07:01:33 UTC
[I] VM console cannot access after VNC certificate expired [cloudstack]
havengit opened a new issue, #9718:
URL: https://github.com/apache/cloudstack/issues/9718
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
CPVM
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main branch.
-->
~~~
4.18
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network, advanced networking, etc. N/A otherwise
-->
advanced networking
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
Hypervisro: KVM virtualization with ubuntu 22.04
##### SUMMARY
<!-- Explain the problem/feature briefly -->
VM console cannot access after VNC certificate expired。
Default certificate valid for 1 year ,then automatically renews certificates when they expire by cloudstack。
I made sure the certificate was updated on the host,but Th running VM instances need stop and start to apply the new certificate or migraton to other host. For production environments, this is hard to do
I don't know how to get a new certificate to take effect without rebooting the vm
https://github.com/apache/cloudstack/pull/7015
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
~~~
~~~
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
Make a new certificate to take effect without rebooting the vm
Provide a setting to turn off tls for vnc
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
~~~
~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] VM console cannot access after VNC certificate expired [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-5068479935
@tatay188 , can you have a look at https://github.com/apache/cloudstack/pull/12911 and evaluate if that meats your requirements?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] VM console cannot access after VNC certificate expired [cloudstack]
Posted by "rohityadavcloud (via GitHub)" <gi...@apache.org>.
rohityadavcloud commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2478454546
Hi all, by default I think this should work. The ca.framework.cert.automatic.renewal needs to be enabled (true), and there's also ca.framework.cert.expiry.alert.period and ca.framework.background.task.delay. For agents that are expired certs but are connected it's not an issue, but such agents risk failing to join when restarted - for them an explicit API can be called:
```
(homecloud) 🐵 > provision certificate hostid= -h
provisionCertificate: Issues and propagates client certificate on a connected host/agent using configured CA plugin
This API is asynchronous.
Required params: hostid,
API Params Type Description
========== ==== ===========
hostid uuid The host/agent uuid to which the certific
ate has to be provisioned (issued and pr
opagated)
provider string Name of the CA service provider, otherwis
e the default configured provider plugin
will be used
reconnect boolean Whether to attempt reconnection with host
/agent after successful deployment of ce
rtificate. When option is not provided,
configured global setting is used
```
However, VNC console to users browser uses admin uploaded certificate - when they expire, admin needs to upload new end-user TLS/SSL certs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] VM console cannot access after VNC certificate expired [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2367473163
there are some global settings ("ca.framework.*"), including
- ca.framework.background.task.delay
- ca.framework.cert.automatic.renewal
- ca.framework.cert.expiry.alert.period
if auto renewal is enabled, the agent cert should be auto-renewed.
otherwise you can do it manually . refer to https://github.com/apache/cloudstack/issues/9562#issuecomment-2302208986
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] VM console cannot access after VNC certificate expired [cloudstack]
Posted by "tatay188 (via GitHub)" <gi...@apache.org>.
tatay188 commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-5045356427
@DaanHoogland @weizhouapache is there any movement on the certificate management, is really a problem when the certificates expire for a critical environment have to restart or migrate the VMs - Making a certificate update on a major issue - mostly because CA authorities are not providing certificates for a period longer than a year, and it does not matter if you have the certificate to auto-renew.
rebooting the Systemproxy VM i think that is perfectly sustainable, however have to restart VMs and on routed VPCs we have had to restart the VPC itself then the vrouters and then the VMs, even when whe have HA VPC - obviously the backup vrouter took over, but have to restart the whole VPC as restarting just the vrouters did not work.
Certificate rules are changing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] VM console cannot access after VNC certificate expired [cloudstack]
Posted by "tatay188 (via GitHub)" <gi...@apache.org>.
tatay188 commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-5070313309
@DaanHoogland I have to add something interesting, after updating the CA (not selfsigned certificate) and after rebooting the System proxy, windows machines with virtio drivers installed reconnect to the vnc console, however linux fails have to restart the linux VMs including the VRs. Just a note: we do everything using the UI only not CMK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] VM console cannot access after VNC certificate expired [cloudstack]
Posted by "havengit (via GitHub)" <gi...@apache.org>.
havengit commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2370610757
Thanks ,weizhou and community , I have change **Ca framework cert validity period** to a very long time . I won't run into this problem in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] VM console cannot access after VNC certificate expired [cloudstack]
Posted by "vishesh92 (via GitHub)" <gi...@apache.org>.
vishesh92 commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-5619483322
It should be possible to do this without restart using `virsh domdisplay-reload <domain> --type vnc`.
This command is available from libvirt 10.2.0
https://www.libvirt.org/manpages/virsh.html#domdisplay-reload
For older versions, `virsh qemu-monitor-command i-2-4-VM '{"execute":"display-reload","arguments":{"type":"vnc","tls-certs":true}}'` should work. Please note that `"tls-certs": true` needs to be added, otherwise it doesn't work.
A small script for doing it for all VMs on a host.
```bash
for vm in $(virsh list --name); do
echo "Reloading VNC certs for: $vm"
virsh qemu-monitor-command "$vm" '{"execute":"display-reload","arguments":{"type":"vnc","tls-certs":true}}''
done
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] VM console cannot access after VNC certificate expired [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2367637128
> Yes , I have restarted cpvm and libvirt on host. I confirm that the certificate has been successfully updated on the host. May be same bug in qemu. After stop vm and start vm console works fine .
@havengit
is it necessary to stop/start vm if you have restarted libvirtd and cloudstack-agent ?
if so, it has big impact
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] VM console cannot access after VNC certificate expired [cloudstack]
Posted by "havengit (via GitHub)" <gi...@apache.org>.
havengit commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2370303318
@weizhouapache Restart libvirtd and cloudstack-agent has no effect ,VM must be rebooted or live migration to other host .
I don't know if anyone else has encountered this, but this feature should have been added in 4.18, so maybe not many people are using it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] VM console cannot access after VNC certificate expired [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-5070783091
ok, this does not ring a bell at all. Please start a new issue or discussion/qa item?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] VM console cannot access after VNC certificate expired [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2370433665
this may be related to #7015
can you check if
- `vnc_tls` is set to 1 in /etc/libvirt/qemu.conf ?
- console works after live migration to another host ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] VM console cannot access after VNC certificate expired [cloudstack]
Posted by "havengit (via GitHub)" <gi...@apache.org>.
havengit closed issue #9718: VM console cannot access after VNC certificate expired
URL: https://github.com/apache/cloudstack/issues/9718
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] VM console cannot access after VNC certificate expired [cloudstack]
Posted by "havengit (via GitHub)" <gi...@apache.org>.
havengit commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2367598767
Yes , I have restarted cpvm and libvirt on host. I confirm that the certificate has been successfully updated on the host. May be same bug in qemu. After stop vm and start vm console works fine .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] VM console cannot access after VNC certificate expired [cloudstack]
Posted by "havengit (via GitHub)" <gi...@apache.org>.
havengit commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2370338502
Using virt-viewer also fails to connect, so it shouldn't be an ACS issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] VM console cannot access after VNC certificate expired [cloudstack]
Posted by "havengit (via GitHub)" <gi...@apache.org>.
havengit commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2370459380
Yes , the vnc tls were enabled in all host. It seems that the qemu process that is running, does not recognize the certificate change and still uses the old certificate. Stop the startup or migrate and the new qemu process will use the new certificate.
vnc_tls=1
vnc_tls_x509_verify=1
vnc_tls_x509_cert_dir="/etc/pki/libvirt-vnc"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] VM console cannot access after VNC certificate expired [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2370584807
> Yes , the vnc tls were enabled in all host. It seems that the qemu process that is running, does not recognize the certificate change and still uses the old certificate. Stop the startup or migrate and the new qemu process will use the new certificate. vnc_tls=1 vnc_tls_x509_verify=1 vnc_tls_x509_cert_dir="/etc/pki/libvirt-vnc"
thanks @havengit
good to know that migration fixes the issue. stopping/starting all vms is not possible for large production environments.
To summarize,
- vm console becomes unavailable if cloudstack-agent certificates expires and is renewed.
- it is a qemu issue. ACS cannot do anything to fix it.
- users have to migrate or stop/start the VMs so that new qemu process will use the new certificates
cc @DaanHoogland @rohityadavcloud @nvazquez
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] VM console cannot access after VNC certificate expired [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2367428376
@havengit , I think this is more a lack of feature than a bug, which admittedly comes down to the same to you. So I will mark it as an improvement. Dynamically setting the certificate will not be easy. Did you try rebooting just the console proxy? I would guess that is the actually needed action.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] VM console cannot access after VNC certificate expired [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-5630487444
I have reopened this ticket and added to 4.22.2 milestone
we can apply the fix on all running vms automatically when the certificate is renewed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org