You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "yadvr (via GitHub)" <gi...@apache.org> on 2026/01/12 18:36:59 UTC
[I] Powersync doesn't sync VM states after power outtage [cloudstack]
yadvr opened a new issue, #12411:
URL: https://github.com/apache/cloudstack/issues/12411
### problem
After a power outtage, the host comes online and ssvm/cpvm on it start but not VRs, and the HA-enabled (via compute offering) user VMs are powered on in ACS but not running (off) on the KVM host. I'm not sure if some config is required, I expected this to work out of the box to sync VM states (i.e. set vm to stopped when it's not really running on the hypervisor host), or try to get the VM to run as it's on VM-HA enabled offering.
### versions
ACS 4.22, on both x86 mini pc and arm64/raspberrypi5 KVM hosts; in adv zone using HA-enabled compute offering and uses NFS & local storage only.
### The steps to reproduce the bug
1.
2.
3.
...
### What to do about it?
Ideally the power state of the VM should sync, and shutdown VMs on a rebooted/started KVM host after power outtage should try to bring up the user VMs (which are HA enabled).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Powersync doesn't sync VM states after power outtage [cloudstack]
Posted by "yadvr (via GitHub)" <gi...@apache.org>.
yadvr commented on issue #12411:
URL: https://github.com/apache/cloudstack/issues/12411#issuecomment-3756234286
requesting for a higher priority issue as this is related to VM HA being broken cc @DaanHoogland
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Powersync doesn't sync VM states after power outtage [cloudstack]
Posted by "kiranchavala (via GitHub)" <gi...@apache.org>.
kiranchavala commented on issue #12411:
URL: https://github.com/apache/cloudstack/issues/12411#issuecomment-3743070850
@yadvr Not able to reproduce the issue
1. Launched a 2 vm's one with HA enabled and another not ha enable on a KVM host 1
2. Made sure the global setting 'commands.timeout' value to a suitable value like "CheckHealthCommand=5,CheckOnHostCommand=5"
3. Shutdown the kvm host 1
4. The router and HA enabled vm restarted on another kvm host
5. The non-ha enabled vm went into stopped state
logs of ha enabled vm
```
root@acs420:~# cat /var/log/cloudstack/management/management-server.log |grep -i "logid:48d451aa"
2026-01-13 08:38:08,653 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Processing work HAWork[577-HA-155-Running-Investigating]
2026-01-13 08:38:08,654 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) RESTART with HAWORK
2026-01-13 08:38:08,656 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) SimpleInvestigator unable to determine the state of the host. Moving on.
2026-01-13 08:38:08,656 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) XenServerInvestigator unable to determine the state of the host. Moving on.
2026-01-13 08:38:08,657 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Can not send command com.cloud.agent.api.CheckOnHostCommand due to Host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} not being up
2026-01-13 08:38:08,658 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Investigating host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} via neighbouring host:Host {"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
2026-01-13 08:38:08,658 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Wait time setting on com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,659 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 21-5292855462067175506: Routed from 206863097656310
2026-01-13 08:38:08,659 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 7-5292855462067175506: Sending { Cmd , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,674 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 7-5292855462067175506: Received: { Ans: , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,674 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Neighbouring host:Host {"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"} returned status:Down for the investigated host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,675 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Investigating host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} via neighbouring host:Host {"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"}
2026-01-13 08:38:08,675 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Wait time setting on com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,676 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 21-5194902170171867174: Routed from 206863097656310
2026-01-13 08:38:08,676 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 13-5194902170171867174: Sending { Cmd , MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,723 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 13-5194902170171867174: Received: { Ans: , MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,723 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Neighbouring host:Host {"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"} returned status:Down for the investigated host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,723 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) HA: HOST is ineligible legacy state Down for host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,723 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) KVMInvestigator was able to determine host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} is in Down
2026-01-13 08:38:08,723 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) HA on VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,725 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Wait time setting on com.cloud.agent.api.CheckVirtualMachineCommand is 20 seconds
2026-01-13 08:38:08,725 DEBUG [c.c.h.CheckOnAgentInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Unable to reach the agent for VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}: Resource [Host:6] is unreachable: Host 6: Host with specified id is not in the right state: Down
2026-01-13 08:38:08,725 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) SimpleInvestigator could not find VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,725 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) XenServerInvestigator could not find VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,727 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Can not send command com.cloud.agent.api.CheckOnHostCommand due to Host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} not being up
2026-01-13 08:38:08,727 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Investigating host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} via neighbouring host:Host {"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
2026-01-13 08:38:08,728 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Wait time setting on com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,728 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 21-5292855462067175509: Routed from 206863097656310
2026-01-13 08:38:08,728 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 7-5292855462067175509: Sending { Cmd , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,771 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 7-5292855462067175509: Received: { Ans: , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,771 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Neighbouring host:Host {"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"} returned status:Down for the investigated host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,771 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Investigating host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} via neighbouring host:Host {"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"}
2026-01-13 08:38:08,771 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Wait time setting on com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,772 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 21-5194902170171867176: Routed from 206863097656310
2026-01-13 08:38:08,772 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 13-5194902170171867176: Sending { Cmd , MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,780 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 13-5194902170171867176: Received: { Ans: , MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,780 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Neighbouring host:Host {"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"} returned status:Down for the investigated host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,780 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) HA: HOST is ineligible legacy state Down for host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,780 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) HA: HOST is ineligible legacy state Down for host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,780 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) KVMInvestigator could not find VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,780 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) HypervInvestigator could not find VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,781 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) VMwareInvestigator could not find VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,781 DEBUG [c.c.h.UserVmDomRInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) testing if VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"} is alive
2026-01-13 08:38:08,783 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Can not send command com.cloud.agent.api.PingTestCommand due to Host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} not being up
2026-01-13 08:38:08,783 DEBUG [c.c.h.UserVmDomRInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"} could not be pinged, returning that it is unknown
2026-01-13 08:38:08,783 DEBUG [c.c.h.UserVmDomRInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Returning null since we're unable to determine state of VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,783 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) PingInvestigator could not find VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,783 DEBUG [c.c.h.ManagementIPSystemVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Not a System Vm, unable to determine state of VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"} returning null
2026-01-13 08:38:08,784 DEBUG [c.c.h.ManagementIPSystemVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Testing if VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"} is alive
2026-01-13 08:38:08,784 DEBUG [c.c.h.ManagementIPSystemVMInvestigator] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Unable to find a management nic, cannot ping this system VM, unable to determine state of VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"} returning null
2026-01-13 08:38:08,784 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) ManagementIPSysVMInvestigator could not find VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"}
2026-01-13 08:38:08,784 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Fencing off VM that we don't know the state of
2026-01-13 08:38:08,785 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Wait time setting on com.cloud.agent.api.FenceCommand is 1800 seconds
2026-01-13 08:38:08,785 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 21-5292855462067175512: Routed from 206863097656310
2026-01-13 08:38:08,786 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 7-5292855462067175512: Sending { Cmd , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.FenceCommand":{"vmName":"i-2-155-VM","hostGuid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","hostIp":"192.168.55.164","inSeq":"false","host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,833 DEBUG [c.c.a.t.Request] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Seq 7-5292855462067175512: Received: { Ans: , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 10, { FenceAnswer } }
2026-01-13 08:38:08,833 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Fencer KVMFenceBuilder returned true
2026-01-13 08:38:08,833 WARN [o.a.c.f.j.AsyncJobExecutionContext] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Job is executed without a context, setup psudo job for the executing thread
2026-01-13 08:38:08,840 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Sync job-2541 execution on object VmWorkJobQueue.155
2026-01-13 08:38:10,544 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577, ctx-99fd3c33]) (logid:48d451aa) start parameter value of enterHardwareSetup == null during processing of queued job
2026-01-13 08:38:10,548 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577, ctx-99fd3c33]) (logid:48d451aa) Sync job-2543 execution on object VmWorkJobQueue.155
2026-01-13 08:40:14,151 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) HA is now restarting VM instance {"id":155,"instanceName":"i-2-155-VM","state":"Running","type":"User","uuid":"0597fa15-2dc4-4a92-9c0d-4025bd5568c6"} on Host {"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
2026-01-13 08:40:14,152 WARN [c.c.a.AlertManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) alertType=[8] dataCenterId=[1] podId=[1] clusterId=[null] message=[HA starting VM: vm2-haenabled (i-2-155-VM)].
2026-01-13 08:40:14,155 WARN [c.c.a.AlertManagerImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [HA starting VM: vm2-haenabled (i-2-155-VM)] and content [HA starting VM: vm2-haenabled (i-2-155-VM)].
2026-01-13 08:40:14,155 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-0:[ctx-0d486e07, work-577]) (logid:48d451aa) Completed work HAWork[577-HA-155-Running-Scheduled]. Took 1/5 attempts.
```
Logs for Non ha enabled vm
```
root@acs420:~# cat /var/log/cloudstack/management/management-server.log |grep -i "logid:62b9b072"
2026-01-13 08:38:08,648 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Processing work HAWork[576-HA-153-Running-Investigating]
2026-01-13 08:38:08,649 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) RESTART with HAWORK
2026-01-13 08:38:08,651 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) SimpleInvestigator unable to determine the state of the host. Moving on.
2026-01-13 08:38:08,651 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) XenServerInvestigator unable to determine the state of the host. Moving on.
2026-01-13 08:38:08,652 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Can not send command com.cloud.agent.api.CheckOnHostCommand due to Host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} not being up
2026-01-13 08:38:08,653 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Investigating host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} via neighbouring host:Host {"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
2026-01-13 08:38:08,654 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Wait time setting on com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,654 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 21-5292855462067175505: Routed from 206863097656310
2026-01-13 08:38:08,654 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 7-5292855462067175505: Sending { Cmd , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,664 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 7-5292855462067175505: Received: { Ans: , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,664 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Neighbouring host:Host {"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"} returned status:Down for the investigated host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,664 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Investigating host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} via neighbouring host:Host {"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"}
2026-01-13 08:38:08,665 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Wait time setting on com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,665 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 21-5194902170171867173: Routed from 206863097656310
2026-01-13 08:38:08,665 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 13-5194902170171867173: Sending { Cmd , MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,676 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 13-5194902170171867173: Received: { Ans: , MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,676 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Neighbouring host:Host {"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"} returned status:Down for the investigated host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,677 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) HA: HOST is ineligible legacy state Down for host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,677 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) KVMInvestigator was able to determine host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} is in Down
2026-01-13 08:38:08,677 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) HA on VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,678 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Wait time setting on com.cloud.agent.api.CheckVirtualMachineCommand is 20 seconds
2026-01-13 08:38:08,679 DEBUG [c.c.h.CheckOnAgentInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Unable to reach the agent for VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}: Resource [Host:6] is unreachable: Host 6: Host with specified id is not in the right state: Down
2026-01-13 08:38:08,679 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) SimpleInvestigator could not find VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,679 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) XenServerInvestigator could not find VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,681 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Can not send command com.cloud.agent.api.CheckOnHostCommand due to Host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} not being up
2026-01-13 08:38:08,682 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Investigating host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} via neighbouring host:Host {"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"}
2026-01-13 08:38:08,682 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Wait time setting on com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,683 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 21-5292855462067175508: Routed from 206863097656310
2026-01-13 08:38:08,683 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 7-5292855462067175508: Sending { Cmd , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,771 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 7-5292855462067175508: Received: { Ans: , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,771 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Neighbouring host:Host {"id":7,"name":"kvmhost2","type":"Routing","uuid":"c30b8e0d-0a48-40fa-bb57-c2de13513ca5"} returned status:Down for the investigated host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,771 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Investigating host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} via neighbouring host:Host {"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"}
2026-01-13 08:38:08,771 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Wait time setting on com.cloud.agent.api.CheckOnHostCommand is 5 seconds
2026-01-13 08:38:08,772 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 21-5194902170171867177: Routed from 206863097656310
2026-01-13 08:38:08,772 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 13-5194902170171867177: Sending { Cmd , MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"5","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,780 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 13-5194902170171867177: Received: { Ans: , MgmtId: 206863097656310, via: 13(kvmhost3), Ver: v1, Flags: 10, { Answer } }
2026-01-13 08:38:08,780 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Neighbouring host:Host {"id":13,"name":"kvmhost3","type":"Routing","uuid":"3cfc4b1f-1aa9-40e5-a549-484ab4fcc921"} returned status:Down for the investigated host:Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,780 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) HA: HOST is ineligible legacy state Down for host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,780 DEBUG [c.c.h.KVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) HA: HOST is ineligible legacy state Down for host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"}
2026-01-13 08:38:08,780 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) KVMInvestigator could not find VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,781 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) HypervInvestigator could not find VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,781 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) VMwareInvestigator could not find VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,781 DEBUG [c.c.h.UserVmDomRInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) testing if VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"} is alive
2026-01-13 08:38:08,783 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Can not send command com.cloud.agent.api.PingTestCommand due to Host Host {"id":6,"name":"kvmhost1","type":"Routing","uuid":"650ff5c6-4cf3-4d26-99a8-ce8c0b9b153f"} not being up
2026-01-13 08:38:08,783 DEBUG [c.c.h.UserVmDomRInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"} could not be pinged, returning that it is unknown
2026-01-13 08:38:08,783 DEBUG [c.c.h.UserVmDomRInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Returning null since we're unable to determine state of VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,783 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) PingInvestigator could not find VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,783 DEBUG [c.c.h.ManagementIPSystemVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Not a System Vm, unable to determine state of VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"} returning null
2026-01-13 08:38:08,784 DEBUG [c.c.h.ManagementIPSystemVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Testing if VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"} is alive
2026-01-13 08:38:08,784 DEBUG [c.c.h.ManagementIPSystemVMInvestigator] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Unable to find a management nic, cannot ping this system VM, unable to determine state of VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"} returning null
2026-01-13 08:38:08,784 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) ManagementIPSysVMInvestigator could not find VM instance {"id":153,"instanceName":"i-2-153-VM","state":"Running","type":"User","uuid":"669a5d32-0b9a-4e84-8e9f-16597722cf98"}
2026-01-13 08:38:08,784 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Fencing off VM that we don't know the state of
2026-01-13 08:38:08,785 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Wait time setting on com.cloud.agent.api.FenceCommand is 1800 seconds
2026-01-13 08:38:08,785 DEBUG [c.c.a.m.ClusteredAgentAttache] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 21-5292855462067175511: Routed from 206863097656310
2026-01-13 08:38:08,785 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 7-5292855462067175511: Sending { Cmd , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.FenceCommand":{"vmName":"i-2-153-VM","hostGuid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","hostIp":"192.168.55.164","inSeq":"false","host":{"guid":"0218d19d-3cbe-3827-bb03-7fd636ad202b-LibvirtComputingResource","privateNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"publicNetwork":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"},"storageNetwork1":{"ip":"192.168.55.164","netmask":"255.255.255.0","mac":"9e:8f:77:52:26:21","securityGroupEnabled":"false"}},"reportCheckFailureIfOneStorageIsDown":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2026-01-13 08:38:08,793 DEBUG [c.c.a.t.Request] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Seq 7-5292855462067175511: Received: { Ans: , MgmtId: 206863097656310, via: 7(kvmhost2), Ver: v1, Flags: 10, { FenceAnswer } }
2026-01-13 08:38:08,793 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Fencer KVMFenceBuilder returned true
2026-01-13 08:38:08,793 WARN [o.a.c.f.j.AsyncJobExecutionContext] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Job is executed without a context, setup psudo job for the executing thread
2026-01-13 08:38:08,807 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Sync job-2539 execution on object VmWorkJobQueue.153
2026-01-13 08:38:10,517 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) VM is not HA enabled so we're done.
2026-01-13 08:38:10,517 INFO [c.c.h.HighAvailabilityManagerExtImpl] (HA-Worker-4:[ctx-9060fa2b, work-576]) (logid:62b9b072) Completed work HAWork[576-HA-153-Running-Scheduled]. Took 1/5 attempts.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Powersync doesn't sync VM states after power outtage [cloudstack]
Posted by "kiranchavala (via GitHub)" <gi...@apache.org>.
kiranchavala closed issue #12411: Powersync doesn't sync VM states after power outtage
URL: https://github.com/apache/cloudstack/issues/12411
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Powersync doesn't sync VM states after power outtage [cloudstack]
Posted by "kiranchavala (via GitHub)" <gi...@apache.org>.
kiranchavala commented on issue #12411:
URL: https://github.com/apache/cloudstack/issues/12411#issuecomment-3744245621
Thanks @yadvr
Able to reproduce the issue if both control plane (mgmt server) and KVM host are down at the same time.
When the mgmt server and kvm host comes up later, the ha enabled vm and router says its running on kvm host but there is no vm running on the host ( virsh list --all)
Agent log
```
2026-01-13 13:04:35,239 WARN [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:f3f12b6e) Can not find a connection for Instance r-154-VM. Assuming the default connection.
2026-01-13 13:04:35,239 WARN [resource.wrapper.LibvirtGetVmStatsCommandWrapper] (AgentRequest-Handler-2:[]) (logid:f3f12b6e) Can't get vm stats: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'r-154-VM', continue
2026-01-13 13:04:35,240 INFO [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:f3f12b6e) No existing libvirtd connection found. Opening a new one
2026-01-13 13:04:35,240 WARN [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:f3f12b6e) Can not find a connection for Instance r-104-VM. Assuming the default connection.
2026-01-13 13:04:35,240 WARN [resource.wrapper.LibvirtGetVmStatsCommandWrapper] (AgentRequest-Handler-2:[]) (logid:f3f12b6e) Can't get vm stats: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'r-104-VM', continue
2026-01-13 13:04:35,241 INFO [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:f3f12b6e) No existing libvirtd connection found. Opening a new one
2026-01-13 13:04:35,241 WARN [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:f3f12b6e) Can not find a connection for Instance i-2-155-VM. Assuming the default connection.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Powersync doesn't sync VM states after power outtage [cloudstack]
Posted by "kiranchavala (via GitHub)" <gi...@apache.org>.
kiranchavala commented on issue #12411:
URL: https://github.com/apache/cloudstack/issues/12411#issuecomment-3847089750
Resolved with #12463
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Powersync doesn't sync VM states after power outtage [cloudstack]
Posted by "yadvr (via GitHub)" <gi...@apache.org>.
yadvr commented on issue #12411:
URL: https://github.com/apache/cloudstack/issues/12411#issuecomment-3743137569
@kiranchavala try to poweroff both control plane and KVM host at the same time and try ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org