You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "havengit (via GitHub)" <gi...@apache.org> on 2024/09/23 07:01:33 UTC

[I] VM console cannot access after VNC certificate expired [cloudstack]

havengit opened a new issue, #9718:
URL: https://github.com/apache/cloudstack/issues/9718

   <!--
   Verify first that your issue/request is not already reported on GitHub.
   Also test if the latest release and main branch are affected too.
   Always add information AFTER of these HTML comments, but no need to delete the comments.
   -->
   
   ##### ISSUE TYPE
   <!-- Pick one below and delete the rest -->
    * Bug Report
   
   
   ##### COMPONENT NAME
   <!--
   Categorize the issue, e.g. API, VR, VPN, UI, etc.
   -->
   ~~~
   CPVM
   ~~~
   
   ##### CLOUDSTACK VERSION
   <!--
   New line separated list of affected versions, commit ID for issues on main branch.
   -->
   
   ~~~
   4.18 
   ~~~
   
   ##### CONFIGURATION
   <!--
   Information about the configuration if relevant, e.g. basic network, advanced networking, etc.  N/A otherwise
   -->
    advanced networking
   
   ##### OS / ENVIRONMENT
   <!--
   Information about the environment if relevant, N/A otherwise
   -->
   Hypervisro: KVM virtualization  with ubuntu 22.04
   
   
   
   ##### SUMMARY
   <!-- Explain the problem/feature briefly -->
   
   VM console cannot access  after VNC certificate expired。
   Default certificate valid for 1 year ,then automatically renews certificates when they expire by cloudstack。
   I made sure the certificate was updated on the host,but Th running  VM instances  need stop and start  to apply the new certificate or migraton to other host.  For production environments, this is  hard to do
   
   I don't know how to get a new certificate to take effect without rebooting the vm
   
   
   
   
   
   https://github.com/apache/cloudstack/pull/7015
   
   ##### STEPS TO REPRODUCE
   <!--
   For bugs, show exactly how to reproduce the problem, using a minimal test-case. Use Screenshots if accurate.
   
   For new features, show how the feature would be used.
   -->
   
   <!-- Paste example playbooks or commands between quotes below -->
   ~~~
   
   ~~~
   
   <!-- You can also paste gist.github.com links for larger files -->
   
   ##### EXPECTED RESULTS
   <!-- What did you expect to happen when running the steps above? -->
   
   ~~~
   Make a new certificate to take effect without rebooting the vm
   Provide a setting to turn off tls for vnc
   ~~~
   
   ##### ACTUAL RESULTS
   <!-- What actually happened? -->
   
   <!-- Paste verbatim command output between quotes below -->
   ~~~
   
   ~~~
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] VM console cannot access after VNC certificate expired [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-5068479935

   @tatay188 , can you have a look at https://github.com/apache/cloudstack/pull/12911 and evaluate if that meats your requirements?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] VM console cannot access after VNC certificate expired [cloudstack]

Posted by "rohityadavcloud (via GitHub)" <gi...@apache.org>.
rohityadavcloud commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2478454546

   Hi all, by default I think this should work. The ca.framework.cert.automatic.renewal needs to be enabled (true), and there's also ca.framework.cert.expiry.alert.period and ca.framework.background.task.delay. For agents that are expired certs but are connected it's not an issue, but such agents risk failing to join when restarted - for them an explicit API can be called:
   ```
   (homecloud) 🐵 > provision certificate hostid= -h
   provisionCertificate: Issues and propagates client certificate on a connected host/agent using configured CA plugin
   This API is asynchronous.
   Required params: hostid,
   API Params               Type     Description
   ==========               ====     ===========
   hostid                   uuid     The host/agent uuid to which the certific
                                     ate has to be provisioned (issued and pr
                                     opagated)
   provider                 string   Name of the CA service provider, otherwis
                                     e the default configured provider plugin
                                      will be used
   reconnect                boolean  Whether to attempt reconnection with host
                                     /agent after successful deployment of ce
                                     rtificate. When option is not provided,
                                     configured global setting is used
   ```
   
   However, VNC console to users browser uses admin uploaded certificate - when they expire, admin needs to upload new end-user TLS/SSL certs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] VM console cannot access after VNC certificate expired [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2367473163

   there are some global settings ("ca.framework.*"), including
   - ca.framework.background.task.delay
   - ca.framework.cert.automatic.renewal
   - ca.framework.cert.expiry.alert.period
   
   if auto renewal is enabled, the agent cert should be auto-renewed. 
   
   otherwise you can do it manually . refer to https://github.com/apache/cloudstack/issues/9562#issuecomment-2302208986


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] VM console cannot access after VNC certificate expired [cloudstack]

Posted by "tatay188 (via GitHub)" <gi...@apache.org>.
tatay188 commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-5045356427

   @DaanHoogland @weizhouapache is there any movement on the certificate management, is really a problem when the certificates expire for a critical environment have to restart or migrate the VMs - Making a certificate update on a major issue - mostly because CA authorities are not providing certificates for a period longer than a year, and it does not matter if you have the certificate to auto-renew.
   
   rebooting the Systemproxy VM i think that is perfectly sustainable, however have to restart VMs and on routed VPCs we have had to restart the VPC itself then the vrouters and then the VMs, even when whe have HA VPC - obviously the backup vrouter took over, but have to restart the whole VPC as restarting just the vrouters did not work.
   
   Certificate rules are changing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] VM console cannot access after VNC certificate expired [cloudstack]

Posted by "tatay188 (via GitHub)" <gi...@apache.org>.
tatay188 commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-5070313309

   @DaanHoogland  I have to add something interesting, after updating the CA (not selfsigned certificate) and after rebooting the System proxy, windows machines with virtio drivers installed reconnect to the vnc console, however linux fails have to restart the linux VMs including the VRs. Just a note: we do everything using the UI only not CMK.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] VM console cannot access after VNC certificate expired [cloudstack]

Posted by "havengit (via GitHub)" <gi...@apache.org>.
havengit commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2370610757

   Thanks ,weizhou and community , I have  change **Ca framework cert validity period** to a very long time . I won't run into this problem in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] VM console cannot access after VNC certificate expired [cloudstack]

Posted by "vishesh92 (via GitHub)" <gi...@apache.org>.
vishesh92 commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-5619483322

   It should be possible to do this without restart using `virsh domdisplay-reload <domain> --type vnc`.
   This command is available from libvirt 10.2.0
   https://www.libvirt.org/manpages/virsh.html#domdisplay-reload
   For older versions, `virsh qemu-monitor-command i-2-4-VM '{"execute":"display-reload","arguments":{"type":"vnc","tls-certs":true}}'` should work. Please note that `"tls-certs": true` needs to be added, otherwise it doesn't work.
   
   A small script for doing it for all VMs on a host.
   ```bash
   for vm in $(virsh list --name); do
       echo "Reloading VNC certs for: $vm"
       virsh qemu-monitor-command "$vm" '{"execute":"display-reload","arguments":{"type":"vnc","tls-certs":true}}''
   done
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] VM console cannot access after VNC certificate expired [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2367637128

   > Yes , I have restarted cpvm and libvirt on host. I confirm that the certificate has been successfully updated on the host. May be same bug in qemu. After stop vm and start vm console works fine .
   
   @havengit 
   is it necessary to stop/start vm if you have restarted libvirtd and cloudstack-agent ?
   if so, it has big impact
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] VM console cannot access after VNC certificate expired [cloudstack]

Posted by "havengit (via GitHub)" <gi...@apache.org>.
havengit commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2370303318

   @weizhouapache  Restart  libvirtd and cloudstack-agent  has no effect ,VM must be rebooted or live migration to other host .
   I don't know if anyone else has encountered this, but this feature should have been added in 4.18, so maybe not many people are using it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] VM console cannot access after VNC certificate expired [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-5070783091

   ok, this does not ring a bell at all. Please start a new issue or discussion/qa item?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] VM console cannot access after VNC certificate expired [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2370433665

   this may be related to #7015 
   
   can you check if 
   - `vnc_tls` is set to 1 in /etc/libvirt/qemu.conf ?
   - console works after live migration to another host ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] VM console cannot access after VNC certificate expired [cloudstack]

Posted by "havengit (via GitHub)" <gi...@apache.org>.
havengit closed issue #9718: VM console cannot access  after VNC certificate expired 
URL: https://github.com/apache/cloudstack/issues/9718


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] VM console cannot access after VNC certificate expired [cloudstack]

Posted by "havengit (via GitHub)" <gi...@apache.org>.
havengit commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2367598767

   Yes , I have restarted cpvm and  libvirt on host.  I confirm that the certificate has been successfully updated on the host.  May be same bug in qemu. After stop vm and start vm  console  works  fine .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] VM console cannot access after VNC certificate expired [cloudstack]

Posted by "havengit (via GitHub)" <gi...@apache.org>.
havengit commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2370338502

   Using virt-viewer also fails to connect, so it shouldn't be an ACS issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] VM console cannot access after VNC certificate expired [cloudstack]

Posted by "havengit (via GitHub)" <gi...@apache.org>.
havengit commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2370459380

   Yes , the vnc tls were enabled in all host. It seems that the qemu process that is running, does not recognize the certificate change and still uses the old certificate. Stop the startup or migrate and the new qemu process will use the new certificate.
   vnc_tls=1
   vnc_tls_x509_verify=1
   vnc_tls_x509_cert_dir="/etc/pki/libvirt-vnc" 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] VM console cannot access after VNC certificate expired [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2370584807

   > Yes , the vnc tls were enabled in all host. It seems that the qemu process that is running, does not recognize the certificate change and still uses the old certificate. Stop the startup or migrate and the new qemu process will use the new certificate. vnc_tls=1 vnc_tls_x509_verify=1 vnc_tls_x509_cert_dir="/etc/pki/libvirt-vnc"
   
   thanks @havengit 
   good to know that migration fixes the issue. stopping/starting all vms is not possible for large production environments.
   
   To summarize, 
   - vm console becomes unavailable if cloudstack-agent certificates expires and is renewed.
   - it is a qemu issue. ACS cannot do anything to fix it.
   - users have to migrate or stop/start the VMs so that new qemu process will use the new certificates
   
   cc @DaanHoogland @rohityadavcloud @nvazquez 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] VM console cannot access after VNC certificate expired [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-2367428376

   @havengit , I think this is more a lack of feature than a bug, which admittedly comes down to the same to you. So I will mark it as an improvement. Dynamically setting the certificate will not be easy. Did you try rebooting just the console proxy? I would guess that is the actually needed action.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] VM console cannot access after VNC certificate expired [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #9718:
URL: https://github.com/apache/cloudstack/issues/9718#issuecomment-5630487444

   I have reopened this ticket and added to 4.22.2 milestone
   
   we can apply the fix on all running vms automatically when the certificate is renewed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org