You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Fariborz Navidan <md...@gmail.com> on 2026/06/16 09:24:59 UTC
URGENT: KVM Host freezes even in emergency mode
Hi Everbody
I have a very odd issue on the KVm host where both agent and management is
installed.
Recently I have tried to mirror outbound smtp traffic from a test vm to IFB
device to analyze it using zeek for spam filtering. I did the mirroring
using tc filters. Everything was working before I added a custom libvirt
hook inside */etc/libvirt/hooks/qemu.d* and restarted libvirtd service to
add tc filter rule to any new VM being created.
Once I restarted libvirtd host freezed and afterwards everytime host boots
it freezes not accepting any input even in emergency mode. When I enter
emergency mode, it only allows me a few seconds to issue a command and then
freezes. This happens even if I mask NetworkManager, libvirtd and
cloudstack-agent during boot to disable any service that cause system freeze
I have reverted back everything but still kernel freezes
As I needed to analyze existing smtp traffic I needed to do on the
production server but everything is down.
Please help!
Best Regards
Re: URGENT: KVM Host freezes even in emergency mode
Posted by Fariborz Navidan <md...@gmail.com>.
Hi
I know running a single node setup has single point of failure. However, we
do not have resources at this time.
By the way, the problem was not related to CloudStack and was a hardware
failure which has been resolved.
Best Regards
On Wed, Jun 17, 2026 at 9:22 AM Boris Stoyanov <bo...@shapeblue.com>
wrote:
> I’m assuming running all on one node (management, DB, Usage and KVM host),
> this is not ideal and not recommended at all, I’d suggest add another node
> and migrate management, db and usage to it, then you can reinstall your
> current compute node (also assuming you have shared storage, this will not
> work for local storage).
>
> Regards,
> Boris.
>
> <https://www.cloudstackcollab.org>
> *From: *Fariborz Navidan <md...@gmail.com>
> *Date: *Tuesday, 16 June 2026 at 1:57 pm
> *To: *users@cloudstack.apache.org <us...@cloudstack.apache.org>
> *Subject: *Re: URGENT: KVM Host freezes even in emergency mode
>
> Hi
>
> This is a single node cloudstack setup. If I take a backup of all VM
> volumes and cloudstack database, can I get them running on a fresh
> cloudstack installation?
>
>
> Regards
>
>
> On Tue, Jun 16, 2026 at 12:23 PM Boris Stoyanov <
> boris.stoyanov@shapeblue.com> wrote:
>
> > Hi Fariborz,
> >
> > This does not look like a Cloudstack issue, assuming this is a compute
> > host and storage is somewhere else, why don’t you reinstall it to a stock
> > version of the linux distro you are running and try again?
> >
> > Thanks,
> > Boris.
> >
> >
> > <https://www.cloudstackcollab.org>
> > *From: *Fariborz Navidan <md...@gmail.com>
> > *Date: *Tuesday, 16 June 2026 at 12:25 pm
> > *To: *users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > *Subject: *URGENT: KVM Host freezes even in emergency mode
> >
> > Hi Everbody
> >
> > I have a very odd issue on the KVm host where both agent and management
> is
> > installed.
> >
> > Recently I have tried to mirror outbound smtp traffic from a test vm to
> IFB
> > device to analyze it using zeek for spam filtering. I did the mirroring
> > using tc filters. Everything was working before I added a custom libvirt
> > hook inside */etc/libvirt/hooks/qemu.d* and restarted libvirtd service to
> > add tc filter rule to any new VM being created.
> >
> > Once I restarted libvirtd host freezed and afterwards everytime host
> boots
> > it freezes not accepting any input even in emergency mode. When I enter
> > emergency mode, it only allows me a few seconds to issue a command and
> then
> > freezes. This happens even if I mask NetworkManager, libvirtd and
> > cloudstack-agent during boot to disable any service that cause system
> > freeze
> >
> > I have reverted back everything but still kernel freezes
> >
> > As I needed to analyze existing smtp traffic I needed to do on the
> > production server but everything is down.
> >
> > Please help!
> >
> > Best Regards
> >
>
Re: URGENT: KVM Host freezes even in emergency mode
Posted by Boris Stoyanov <bo...@shapeblue.com>.
I’m assuming running all on one node (management, DB, Usage and KVM host), this is not ideal and not recommended at all, I’d suggest add another node and migrate management, db and usage to it, then you can reinstall your current compute node (also assuming you have shared storage, this will not work for local storage).
Regards,
Boris.
From: Fariborz Navidan <md...@gmail.com>
Date: Tuesday, 16 June 2026 at 1:57 pm
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: URGENT: KVM Host freezes even in emergency mode
Hi
This is a single node cloudstack setup. If I take a backup of all VM
volumes and cloudstack database, can I get them running on a fresh
cloudstack installation?
Regards
On Tue, Jun 16, 2026 at 12:23 PM Boris Stoyanov <
boris.stoyanov@shapeblue.com> wrote:
> Hi Fariborz,
>
> This does not look like a Cloudstack issue, assuming this is a compute
> host and storage is somewhere else, why don’t you reinstall it to a stock
> version of the linux distro you are running and try again?
>
> Thanks,
> Boris.
>
>
> <https://www.cloudstackcollab.org>
> *From: *Fariborz Navidan <md...@gmail.com>
> *Date: *Tuesday, 16 June 2026 at 12:25 pm
> *To: *users@cloudstack.apache.org <us...@cloudstack.apache.org>
> *Subject: *URGENT: KVM Host freezes even in emergency mode
>
> Hi Everbody
>
> I have a very odd issue on the KVm host where both agent and management is
> installed.
>
> Recently I have tried to mirror outbound smtp traffic from a test vm to IFB
> device to analyze it using zeek for spam filtering. I did the mirroring
> using tc filters. Everything was working before I added a custom libvirt
> hook inside */etc/libvirt/hooks/qemu.d* and restarted libvirtd service to
> add tc filter rule to any new VM being created.
>
> Once I restarted libvirtd host freezed and afterwards everytime host boots
> it freezes not accepting any input even in emergency mode. When I enter
> emergency mode, it only allows me a few seconds to issue a command and then
> freezes. This happens even if I mask NetworkManager, libvirtd and
> cloudstack-agent during boot to disable any service that cause system
> freeze
>
> I have reverted back everything but still kernel freezes
>
> As I needed to analyze existing smtp traffic I needed to do on the
> production server but everything is down.
>
> Please help!
>
> Best Regards
>
Re: URGENT: KVM Host freezes even in emergency mode
Posted by Fariborz Navidan <md...@gmail.com>.
Hi
This is a single node cloudstack setup. If I take a backup of all VM
volumes and cloudstack database, can I get them running on a fresh
cloudstack installation?
Regards
On Tue, Jun 16, 2026 at 12:23 PM Boris Stoyanov <
boris.stoyanov@shapeblue.com> wrote:
> Hi Fariborz,
>
> This does not look like a Cloudstack issue, assuming this is a compute
> host and storage is somewhere else, why don’t you reinstall it to a stock
> version of the linux distro you are running and try again?
>
> Thanks,
> Boris.
>
>
> <https://www.cloudstackcollab.org>
> *From: *Fariborz Navidan <md...@gmail.com>
> *Date: *Tuesday, 16 June 2026 at 12:25 pm
> *To: *users@cloudstack.apache.org <us...@cloudstack.apache.org>
> *Subject: *URGENT: KVM Host freezes even in emergency mode
>
> Hi Everbody
>
> I have a very odd issue on the KVm host where both agent and management is
> installed.
>
> Recently I have tried to mirror outbound smtp traffic from a test vm to IFB
> device to analyze it using zeek for spam filtering. I did the mirroring
> using tc filters. Everything was working before I added a custom libvirt
> hook inside */etc/libvirt/hooks/qemu.d* and restarted libvirtd service to
> add tc filter rule to any new VM being created.
>
> Once I restarted libvirtd host freezed and afterwards everytime host boots
> it freezes not accepting any input even in emergency mode. When I enter
> emergency mode, it only allows me a few seconds to issue a command and then
> freezes. This happens even if I mask NetworkManager, libvirtd and
> cloudstack-agent during boot to disable any service that cause system
> freeze
>
> I have reverted back everything but still kernel freezes
>
> As I needed to analyze existing smtp traffic I needed to do on the
> production server but everything is down.
>
> Please help!
>
> Best Regards
>
Re: URGENT: KVM Host freezes even in emergency mode
Posted by Boris Stoyanov <bo...@shapeblue.com>.
Hi Fariborz,
This does not look like a Cloudstack issue, assuming this is a compute host and storage is somewhere else, why don’t you reinstall it to a stock version of the linux distro you are running and try again?
Thanks,
Boris.
From: Fariborz Navidan <md...@gmail.com>
Date: Tuesday, 16 June 2026 at 12:25 pm
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: URGENT: KVM Host freezes even in emergency mode
Hi Everbody
I have a very odd issue on the KVm host where both agent and management is
installed.
Recently I have tried to mirror outbound smtp traffic from a test vm to IFB
device to analyze it using zeek for spam filtering. I did the mirroring
using tc filters. Everything was working before I added a custom libvirt
hook inside */etc/libvirt/hooks/qemu.d* and restarted libvirtd service to
add tc filter rule to any new VM being created.
Once I restarted libvirtd host freezed and afterwards everytime host boots
it freezes not accepting any input even in emergency mode. When I enter
emergency mode, it only allows me a few seconds to issue a command and then
freezes. This happens even if I mask NetworkManager, libvirtd and
cloudstack-agent during boot to disable any service that cause system freeze
I have reverted back everything but still kernel freezes
As I needed to analyze existing smtp traffic I needed to do on the
production server but everything is down.
Please help!
Best Regards