You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@cloudstack.apache.org by Granwille Strauss <gr...@namhost.com.INVALID> on 2023/10/25 06:21:00 UTC

VM Randomly Freezing

Hi Guys

Any help will be appreciated, my VMs on my KVM hyp (Almalinux 8)
randomly freezes. It does not happen with all of them at once but at
least once a week one VM randomly freezes. It does not respond to
console nor can I check /var/log/libvirt/qemu/i-2-118-VM.log in the KVM
there is nothing. I also check cloudstack and agent logs and I am not
seeing anything obvious. I then investigate the VM too. There too I see
nothing journalctl only has the latest boot information once the VM is
forced reboot via Cloudstack UI, which is the only way I can bring it up
again.

What's also weird, I have Ha enabled for all my VMs it works when I for
example shutdown the VM inside the VM then Ha bring it up again. But in
this case, Ha doesn't seem to detect the VM is in a "frozen" state to
reboot it.

If anyone has had a similar experience before, I will appreciate any
advice to help troubleshoot this, please.

--
Regards / Groete

<https://www.namhost.com> Granwille Strauss // Senior Systems Admin

*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by
mistake, please inform us by email reply and then delete the message. It
is forbidden to copy, forward, or in any way reveal the contents of this
message to anyone without our explicit consent. The integrity and
security of this email cannot be guaranteed over the Internet.
Therefore, the sender will not be held liable for any damage caused by
the message. For our full privacy policy and disclaimers, please go to
https://www.namhost.com/privacy-policy

Re: VM Randomly Freezing

Posted by Ricardo Pertuz <ri...@kuasar.co.INVALID>.

Check your primary storage, it happened to me and it was at 100%


Atte,

Ricardo Pertuz


On 25 Oct 2023 at 1:22 AM -0500, Granwille Strauss <gr...@namhost.com.INVALID>, wrote:
> Hi Guys
> Any help will be appreciated, my VMs on my KVM hyp (Almalinux 8) randomly freezes. It does not happen with all of them at once but at least once a week one VM randomly freezes. It does not respond to console nor can I check /var/log/libvirt/qemu/i-2-118-VM.log in the KVM there is nothing. I also check cloudstack and agent logs and I am not seeing anything obvious. I then investigate the VM too. There too I see nothing journalctl only has the latest boot information once the VM is forced reboot via Cloudstack UI, which is the only way I can bring it up again.
>
> What's also weird, I have Ha enabled for all my VMs it works when I for example shutdown the VM inside the VM then Ha bring it up again. But in this case, Ha doesn't seem to detect the VM is in a "frozen" state to reboot it.
> If anyone has had a similar experience before, I will appreciate any advice to help troubleshoot this, please.
> --
> Regards / Groete
>
> Granwille Strauss  //  Senior Systems Admin
>
> e: granwille@namhost.com
> m: +264 81 323 1260
> w: www.namhost.com
>
>
>
>
> Namhost Internet Services (Pty) Ltd,
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy
>

Re: VM Randomly Freezing

Posted by Granwille Strauss <gr...@namhost.com.INVALID>.

Thank you for the feedback. The issue is very weird, the KVM host is 
under no stress whatsoever while the VM was down. The monitoring tools 
we have in place also show zero spikes on the VM load, RAM or network 
prior to it "freezing". Virsh in KVM at the time still saw the VM 
instance as running, and the fact that I could not find anything in 
/var/log/libvirt/qemu/i-2-118-VM.log prob means there's no issue with 
libvirt itself. Or was there possible other log locations that I may not 
be aware of?

On 10/25/23 11:04, Boris Stoyanov wrote:
> The way HA works in Cloudstack is it reads the hypervisor report, so if your kvm hosts says it’s working, Cloudstack wouldn’t do anything, despite being frozen.
>
> Cloudstack manages the provisioning of the VMs at the host. How the VM behaves is dependent on the hardware and hypervisor (libvirt), unfortunately I cannot relate this issue to Cloudstack, especially no errors/exceptions hound in logs. Sounds like a libvirt issue that you’re hitting or some hardware related problem..
>
> Thanks,
> Bobby.
>
> From: Granwille Strauss<gr...@namhost.com.INVALID>
> Date: Wednesday, 25 October 2023, 9:21
> To:users@cloudstack.apache.org  <us...@cloudstack.apache.org>
> Subject: VM Randomly Freezing
>
> Hi Guys
>
> Any help will be appreciated, my VMs on my KVM hyp (Almalinux 8) randomly freezes. It does not happen with all of them at once but at least once a week one VM randomly freezes. It does not respond to console nor can I check /var/log/libvirt/qemu/i-2-118-VM.log in the KVM there is nothing. I also check cloudstack and agent logs and I am not seeing anything obvious. I then investigate the VM too. There too I see nothing journalctl only has the latest boot information once the VM is forced reboot via Cloudstack UI, which is the only way I can bring it up again.
>
> What's also weird, I have Ha enabled for all my VMs it works when I for example shutdown the VM inside the VM then Ha bring it up again. But in this case, Ha doesn't seem to detect the VM is in a "frozen" state to reboot it.
>
> If anyone has had a similar experience before, I will appreciate any advice to help troubleshoot this, please.
> --
>
> Regards / Groete
> [Image removed by sender.]<https://www.namhost.com/>
> Granwille Strauss  //  Senior Systems Admin
>
> e:granwille@namhost.com<ma...@namhost.com>
> m: +264 81 323 1260<tel:+264813231260>
> w:www.namhost.com<https://www.namhost.com/>
>
> [Image removed by sender.]<https://www.facebook.com/namhost>[Image removed by sender.]<https://twitter.com/namhost>[Image removed by sender.]<https://www.instagram.com/namhostinternetservices/>[Image removed by sender.]<https://www.linkedin.com/company/namhos>[Image removed by sender.]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>
>
> [Image removed by sender.]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
> Namhost Internet Services (Pty) Ltd,
>
> 24 Black Eagle Rd, Hermanus, 7210, RSA
>
> The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go tohttps://www.namhost.com/privacy-policy
>
> [Image removed by sender. Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>
>
>   
>
>
-- 
Regards / Groete

<https://www.namhost.com> 	Granwille Strauss  // Senior Systems Admin

*e:* granwille@namhost.com
*m:* +264 81 323 1260 <tel:+264813231260>
*w:* www.namhost.com <https://www.namhost.com/>

<https://www.facebook.com/namhost><https://twitter.com/namhost><https://www.instagram.com/namhostinternetservices/><https://www.linkedin.com/company/namhos><https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>

Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA



The content of this message is confidential. If you have received it by 
mistake, please inform us by email reply and then delete the message. It 
is forbidden to copy, forward, or in any way reveal the contents of this 
message to anyone without our explicit consent. The integrity and 
security of this email cannot be guaranteed over the Internet. 
Therefore, the sender will not be held liable for any damage caused by 
the message. For our full privacy policy and disclaimers, please go to 
https://www.namhost.com/privacy-policy

Powered by AdSigner 
<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>

Re: VM Randomly Freezing

Posted by Boris Stoyanov <bo...@shapeblue.com>.

The way HA works in Cloudstack is it reads the hypervisor report, so if your kvm hosts says it’s working, Cloudstack wouldn’t do anything, despite being frozen.

Cloudstack manages the provisioning of the VMs at the host. How the VM behaves is dependent on the hardware and hypervisor (libvirt), unfortunately I cannot relate this issue to Cloudstack, especially no errors/exceptions hound in logs. Sounds like a libvirt issue that you’re hitting or some hardware related problem..

Thanks,
Bobby.

From: Granwille Strauss <gr...@namhost.com.INVALID>
Date: Wednesday, 25 October 2023, 9:21
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: VM Randomly Freezing

Hi Guys

Any help will be appreciated, my VMs on my KVM hyp (Almalinux 8) randomly freezes. It does not happen with all of them at once but at least once a week one VM randomly freezes. It does not respond to console nor can I check /var/log/libvirt/qemu/i-2-118-VM.log in the KVM there is nothing. I also check cloudstack and agent logs and I am not seeing anything obvious. I then investigate the VM too. There too I see nothing journalctl only has the latest boot information once the VM is forced reboot via Cloudstack UI, which is the only way I can bring it up again.

What's also weird, I have Ha enabled for all my VMs it works when I for example shutdown the VM inside the VM then Ha bring it up again. But in this case, Ha doesn't seem to detect the VM is in a "frozen" state to reboot it.

If anyone has had a similar experience before, I will appreciate any advice to help troubleshoot this, please.
--

Regards / Groete
[Image removed by sender.]<https://www.namhost.com/>
Granwille Strauss  //  Senior Systems Admin

e: granwille@namhost.com<ma...@namhost.com>
m: +264 81 323 1260<tel:+264813231260>
w: www.namhost.com<https://www.namhost.com/>

[Image removed by sender.]<https://www.facebook.com/namhost>[Image removed by sender.]<https://twitter.com/namhost>[Image removed by sender.]<https://www.instagram.com/namhostinternetservices/>[Image removed by sender.]<https://www.linkedin.com/company/namhos>[Image removed by sender.]<https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA>

[Image removed by sender.]<https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner>
Namhost Internet Services (Pty) Ltd,

24 Black Eagle Rd, Hermanus, 7210, RSA

The content of this message is confidential. If you have received it by mistake, please inform us by email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone without our explicit consent. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message. For our full privacy policy and disclaimers, please go to https://www.namhost.com/privacy-policy

[Image removed by sender. Powered by AdSigner]<https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818>