You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "wverleger (via GitHub)" <gi...@apache.org> on 2025/04/24 11:39:16 UTC
[D] Cloudstack 4.19.2.0 - KVM Cloudstack Agent Stuck [cloudstack]
GitHub user wverleger created a discussion: Cloudstack 4.19.2.0 - KVM Cloudstack Agent Stuck
Hi,
We are running ACS 4.19.2.0 with KVM hypervisors (Ubuntu 24.04 LTS), using linstor as hyperconverged storage and nfs as secondary storage.
We have the phenomen, that one of our hosts seems to have a cloudstack agent problem.
In the UI the "State" is "Up" and the "Resource state" is "Enabled" of the host but for some reason the agent seems to be unresponsive.
We have noticed this because the agent.log stopped working (no new entries and no logrotate) and volume snapshots started to fail and ended in error state.
What we also can see is, that the communication to (redundant) virtual routers on this host is disrupted, health checks fail, and the requested host is not repsonding.
After failed health checks the router is getting replaced and the old one should be expunged but the router is still in expunging status.
I have attached a management-server log entry regarding virtual router "r-749-VM" not able to perform health checks.
The cloudstack-agent log on the host is inconspicuous.
[cloudstack_management_log.txt](https://github.com/user-attachments/files/19890401/cloudstack_management_log.txt)
After restarting the cloudstack-agent the logging starts to work again. But the router state is still "expunging".
Has anyone had any experience with it?
Does anyone know a good way to troubleshoot?
Thanks!
BR
Wilken
GitHub link: https://github.com/apache/cloudstack/discussions/10771
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Cloudstack 4.19.2.0 - KVM Cloudstack Agent Stuck [cloudstack]
Posted by "wverleger (via GitHub)" <gi...@apache.org>.
GitHub user wverleger closed a discussion: Cloudstack 4.19.2.0 - KVM Cloudstack Agent Stuck
Hi,
We are running ACS 4.19.2.0 with KVM hypervisors (Ubuntu 24.04 LTS), using linstor as hyperconverged storage and nfs as secondary storage.
We have the phenomen, that one of our hosts seems to have a cloudstack agent problem.
In the UI the "State" is "Up" and the "Resource state" is "Enabled" of the host but for some reason the agent seems to be unresponsive.
We have noticed this because the agent.log stopped working (no new entries and no logrotate) and volume snapshots started to fail and ended in error state.
What we also can see is, that the communication to (redundant) virtual routers on this host is disrupted, health checks fail, and the requested host is not repsonding.
After failed health checks the router is getting replaced and the old one should be expunged but the router is still in expunging status.
I have attached a management-server log entry regarding virtual router "r-749-VM" not able to perform health checks.
The cloudstack-agent log on the host is inconspicuous.
[cloudstack_management_log.txt](https://github.com/user-attachments/files/19890401/cloudstack_management_log.txt)
After restarting the cloudstack-agent the logging starts to work again. But the router state is still "expunging".
Has anyone had any experience with it?
Does anyone know a good way to troubleshoot?
Thanks!
BR
Wilken
GitHub link: https://github.com/apache/cloudstack/discussions/10771
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Cloudstack 4.19.2.0 - KVM Cloudstack Agent Stuck [cloudstack]
Posted by "wverleger (via GitHub)" <gi...@apache.org>.
GitHub user wverleger added a comment to the discussion: Cloudstack 4.19.2.0 - KVM Cloudstack Agent Stuck
Hi,
As an additional info we receive the following log entries from cloudstack-agent log now:
[cloudstack-agent_log.txt](https://github.com/user-attachments/files/19891120/cloudstack-agent_log.txt)
While creating a volume snapshot:
2025-04-24 11:55:00,183 INFO [resource.wrapper.LinstorBackupSnapshotCommandWrapper] (agentRequest-Handler-5:null) (logid:23fbdeeb) Src: /dev/mapper/vg_nvme-cs--53eced61--c5b3--4cb0--85c3--b7013e1d6ca6_00000_cs--a2db7f5c--b44c--48b4--bbdf--c39ad8e49318 | wilken-test
2025-04-24 11:55:00,186 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:23fbdeeb) Attempting to create storage pool 638de611-0552-36a1-91ca-6df1c493dc8f (NetworkFilesystem) in libvirt
I can not send the command: virsh pool-list --all
Seems to load forever. On other hosts it works instantly.
GitHub link: https://github.com/apache/cloudstack/discussions/10771#discussioncomment-12934724
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Cloudstack 4.19.2.0 - KVM Cloudstack Agent Stuck [cloudstack]
Posted by "wverleger (via GitHub)" <gi...@apache.org>.
GitHub user wverleger added a comment to the discussion: Cloudstack 4.19.2.0 - KVM Cloudstack Agent Stuck
Hi Daan,
Thank you!
I've restarted qemu/libvirtd and cloudstack-agent and it works as intended for now.
I will have a close look and double check next week.
GitHub link: https://github.com/apache/cloudstack/discussions/10771#discussioncomment-12945970
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Cloudstack 4.19.2.0 - KVM Cloudstack Agent Stuck [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: Cloudstack 4.19.2.0 - KVM Cloudstack Agent Stuck
these “ERROR”s you can ignore. The agent tries several strategies to retrieve the cpu-speed and the last one you see succeeding:
```
2025-04-24 10:14:18,062 INFO [utils.linux.KVMHostInfo] (Agent-Handler-1:null) (logid:) Fetching CPU speed from "host capabilities"
2025-04-24 10:14:18,065 INFO [utils.linux.KVMHostInfo] (Agent-Handler-1:null) (logid:) Retrieved value [2800000000] from "host capabilities". This corresponds to a CPU speed of [2800] MHz.
```
GitHub link: https://github.com/apache/cloudstack/discussions/10771#discussioncomment-12935362
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Cloudstack 4.19.2.0 - KVM Cloudstack Agent Stuck [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: Cloudstack 4.19.2.0 - KVM Cloudstack Agent Stuck
@wverleger , I think you should maybe also restart qemu/libvirtd to resolve the issues. If there are no vital VMs on the machine , even reboot it, though this should not be necessary.
- Can you migrate VMs away? and
- What happens if you put the host in maintenance?
GitHub link: https://github.com/apache/cloudstack/discussions/10771#discussioncomment-12935401
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org