You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Bryan Tiang <br...@hotmail.com> on 2024/01/24 08:40:52 UTC

Cloudstack takes 5 Minutes to Realise Hypervisor is down.

Hi Community,

We are performing failover test to see how quickly a VM will failover to another compute node in the event of a compute node hardware failure.

We forcefully power off the compute node where the VM is residing and monitor how long it takes to failover, and after a few tests the result is averagely around 5 mins for the compute node to be declared down and for the VM to failover to another node.

Referring to https://cwiki.apache.org/confluence/display/cloudstack/host+ha, we've made a few changes in global settings hoping we can shorten the total time needed to failover but it doesn’t seem to work.

We're wondering which settings that will effectively achieve our objective, we're looking at around 2 mins total for failover to complete.

We are using Cloudstack 4.18.1 + Linstor + KVM Ubuntu

Regards,
Bryan

Re: Cloudstack takes 5 Minutes to Realise Hypervisor is down.

Posted by Bryan Tiang <br...@hotmail.com>.
Hi Nux and Wei,

We got the 5 minutes by:

(1) Killing the Server Power <- Time Start
(2) When Cloudstack shows the Host is Offline <- Time Stop

So this is what makes us think that 5 minutes for cloudstack to even detect that the host is down takes a long time.

However, we notice as well that the moment Cloudstack Detects the host is down, all the VMs have already moved to other hosts.

So in this case, if 2 minutes is not enough, what is the general rule of thumb?

Regards,
Bryan
On 24 Jan 2024 at 5:16 PM +0800, Nux <nu...@li.nux.ro>, wrote:
> When dealing with these timeouts, try not to be too stingy with the
> values.
> 2 minutes may not be enough for other timeouts in your infra to complete
> so that the whole process happens gracefully.
> Whoever thought of 5 minutes as default must have had serious
> considerations.
>
> On 2024-01-24 09:02, Wei ZHOU wrote:
> > Hi,
> >
> > You can change the global setting "ping.interval" to 10 (seconds),
> > restart
> > the management service, and retest.
> >
> > -Wei
> >
> > On Wed, 24 Jan 2024 at 09:41, Bryan Tiang <br...@hotmail.com>
> > wrote:
> >
> > > Hi Community,
> > >
> > > We are performing failover test to see how quickly a VM will failover
> > > to
> > > another compute node in the event of a compute node hardware failure.
> > >
> > > We forcefully power off the compute node where the VM is residing and
> > > monitor how long it takes to failover, and after a few tests the
> > > result is
> > > averagely around 5 mins for the compute node to be declared down and
> > > for
> > > the VM to failover to another node.
> > >
> > > Referring to
> > > https://cwiki.apache.org/confluence/display/cloudstack/host+ha, we've
> > > made a few changes in global settings hoping we can shorten the total
> > > time
> > > needed to failover but it doesn’t seem to work.
> > >
> > > We're wondering which settings that will effectively achieve our
> > > objective, we're looking at around 2 mins total for failover to
> > > complete.
> > >
> > > We are using Cloudstack 4.18.1 + Linstor + KVM Ubuntu
> > >
> > > Regards,
> > > Bryan
> > >

Re: Cloudstack takes 5 Minutes to Realise Hypervisor is down.

Posted by Nux <nu...@li.nux.ro>.
When dealing with these timeouts, try not to be too stingy with the 
values.
2 minutes may not be enough for other timeouts in your infra to complete 
so that the whole process happens gracefully.
Whoever thought of 5 minutes as default must have had serious 
considerations.

On 2024-01-24 09:02, Wei ZHOU wrote:
> Hi,
> 
> You can change the global setting "ping.interval" to 10 (seconds), 
> restart
> the management service, and retest.
> 
> -Wei
> 
> On Wed, 24 Jan 2024 at 09:41, Bryan Tiang <br...@hotmail.com> 
> wrote:
> 
>> Hi Community,
>> 
>> We are performing failover test to see how quickly a VM will failover 
>> to
>> another compute node in the event of a compute node hardware failure.
>> 
>> We forcefully power off the compute node where the VM is residing and
>> monitor how long it takes to failover, and after a few tests the 
>> result is
>> averagely around 5 mins for the compute node to be declared down and 
>> for
>> the VM to failover to another node.
>> 
>> Referring to
>> https://cwiki.apache.org/confluence/display/cloudstack/host+ha, we've
>> made a few changes in global settings hoping we can shorten the total 
>> time
>> needed to failover but it doesn’t seem to work.
>> 
>> We're wondering which settings that will effectively achieve our
>> objective, we're looking at around 2 mins total for failover to 
>> complete.
>> 
>> We are using Cloudstack 4.18.1 + Linstor + KVM Ubuntu
>> 
>> Regards,
>> Bryan
>> 

Re: Cloudstack takes 5 Minutes to Realise Hypervisor is down.

Posted by Wei ZHOU <us...@gmail.com>.
Hi,

You can change the global setting "ping.interval" to 10 (seconds), restart
the management service, and retest.

-Wei

On Wed, 24 Jan 2024 at 09:41, Bryan Tiang <br...@hotmail.com> wrote:

> Hi Community,
>
> We are performing failover test to see how quickly a VM will failover to
> another compute node in the event of a compute node hardware failure.
>
> We forcefully power off the compute node where the VM is residing and
> monitor how long it takes to failover, and after a few tests the result is
> averagely around 5 mins for the compute node to be declared down and for
> the VM to failover to another node.
>
> Referring to
> https://cwiki.apache.org/confluence/display/cloudstack/host+ha, we've
> made a few changes in global settings hoping we can shorten the total time
> needed to failover but it doesn’t seem to work.
>
> We're wondering which settings that will effectively achieve our
> objective, we're looking at around 2 mins total for failover to complete.
>
> We are using Cloudstack 4.18.1 + Linstor + KVM Ubuntu
>
> Regards,
> Bryan
>