You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Jafar Aghabalayev <Ja...@pasha-technology.com> on 2023/02/10 18:32:21 UTC
Cluster HA - KVM Hosts
Hello Community,
I have configured cluster with 2 KVM Hosts.
HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage.
I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same.
Can anyone help me with this issue?
RE: Cluster HA - KVM Hosts
Posted by Jafar Aghabalayev <Ja...@pasha-technology.com>.
Hello,
I changed timers for HA settings to minimize the number of attempts to recover the hosts and maximize the SLA for VMs.
For now if I run the echo c > /proc/sysrq-trigger the host failed and the HA state show Degraded and after a while changed to recovered state, however the VMs is not accessible.
How can I change settings to disable the number of attempts for recover or set it to 1 time attempt for recover and if host not recovered (OOBM ON and server loading failed) trigger HA event and restart VMs on another host (like VMware VCenter perform the HA event)
Appreciate your support
Regards,
Jafar Aghabalayev
-----Original Message-----
From: Boris Stoyanov <bo...@shapeblue.com>
Sent: Saturday, February 11, 2023 2:37 PM
To: users@cloudstack.apache.org; Wei ZHOU <us...@gmail.com>
Subject: Re: Cluster HA - KVM Hosts
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to soc@pasha-technology.com.
How did you power off the host? If you shut it down gracefully HA will not kick in, hence considered an intended action. The goal of this feature is to guard against unexpected events, like for example host has crashed.
If you're looking to test it out you can try to provoke a kernel panic in the kvm host, something like: echo c > /proc/sysrq-trigger
Then soon HA investigator will determin the host as unhealthy, (pings failing), soon after that there will be an activity checks on the VMs disks running on the host. It will check if there's any changes and disk activity, if that also fails it will determine the host for recovery. It will issue a restart with the IPMI tool, if number of restarts (depending on configuration) also fails it will fence the host and will start the VMs on another host in the same cluster.
I hope this is insightful for you. Let me know how it goes at your end.
Bobby.
From: Jafar Aghabalayev <Ja...@pasha-technology.com>
Date: Saturday, 11 February 2023, 11:21
To: Wei ZHOU <us...@gmail.com>, users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Cluster HA - KVM Hosts
Hello,
I created new offering, with HA enabled and used it
Sent from Outlook for Android<https://aka.ms/AAb9ysg> ________________________________
From: Wei ZHOU <us...@gmail.com>
Sent: Saturday, February 11, 2023 12:44:45 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Cc: Jafar Aghabalayev <Ja...@pasha-technology.com>
Subject: Re: Cluster HA - KVM Hosts
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to soc@pasha-technology.com.
Hi,
Is HA enabled on the VM?
You can check the vm details. HA is not enabled for built-in service offerings (Small/Medium).
-Wei
On Saturday, 11 February 2023, pspal83@hotmail.com<ma...@hotmail.com> <ps...@hotmail.com>> wrote:
Hi Jafar,
FYI.
VM HA does not work when server is stopped gracefully * Issue #4211 * apache/cloudstack
|
|
|
| | |
|
|
|
| |
VM HA does not work when server is stopped gracefully * Issue #4211 * apache/cloudstack
When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI...
| |
|
|
Thanks Pradeep
Sent from Yahoo Mail on Android
On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayev<Ja...@pasha-technology.com>> wrote: Hello Community,
I have configured cluster with 2 KVM Hosts.
HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage.
I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same.
Can anyone help me with this issue?
RE: Cluster HA - KVM Hosts
Posted by Jafar Aghabalayev <Ja...@pasha-technology.com>.
Hello,
In case if i used local storage for system VMs (using local storage for system vms enabled) and for guest VMs shared NFS storage in use, can it cause problem for guest VMs HA trigger?
Regards,
Jafar Aghabalayev
-----Original Message-----
From: Jafar Aghabalayev
Sent: Tuesday, February 21, 2023 4:58 PM
To: users@cloudstack.apache.org
Cc: Wei ZHOU <us...@gmail.com>; boris.stoyanov@shapeblue.com
Subject: RE: Cluster HA - KVM Hosts
Hello,
Thank you for your response.
One primary storage in use and it is NFS and VM volumes located on this NFS storage.
After unplugged the power cable State show alert and power state unknown.
Regards,
Jafar Aghabalayev
-----Original Message-----
From: Vivek Kumar <vi...@indiqus.com.INVALID>
Sent: Tuesday, February 21, 2023 4:10 PM
To: users@cloudstack.apache.org
Cc: Wei ZHOU <us...@gmail.com>; boris.stoyanov@shapeblue.com
Subject: Re: Cluster HA - KVM Hosts
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to soc@pasha-technology.com.
Hello Jafar,
This is the way to test the HA. I did same before putting into the production.
When you unplugged the cable of physical hosts, what is the current state of host ? Is it showing “Disconnected”, “Alert” or “Down”, When I tested the HA it should have atleast one NFS as primary storage, it worked in my case with NFS as a primary storage or atleast one of the primary storage on NFS . So without any NFS storage, whenever I disconnected the host, it was either showing in “Disconnected” or “alert” and then HA will not work.
Vivek Kumar
Sr. Manager - Cloud & DevOps
TechOps | Indiqus Technologies
vivek.kumar@indiqus.com <ma...@indiqus.com>
www.indiqus.com <https://www.indiqus.com/>
> On 21-Feb-2023, at 5:29 PM, Jafar Aghabalayev <Ja...@pasha-technology.com> wrote:
>
> Hello Community,
> My further steps are below:
>
> 1. I fixed the error related to Java and HA state display correct
> state (Available for both hosts) 2. I shutdown the host by unplugging
> power cable 3. The HA state show fencing state 4. No VMs migrated to
> the second available host (all VMs used offering with HA enabled)
>
> Is there are any other options to test HA?
>
>
>
>
> Regards,
> Jafar Aghabalayev
>
>
> -----Original Message-----
> From: Jafar Aghabalayev <Ja...@pasha-technology.com>
> Sent: Monday, February 13, 2023 9:13 PM
> To: users@cloudstack.apache.org; Wei ZHOU <us...@gmail.com>;
> boris.stoyanov@shapeblue.com
> Subject: RE: Cluster HA - KVM Hosts
>
> CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to soc@pasha-technology.com.
>
> Hello,
> I run the echo c > /proc/sysrq-trigger command and host went to
> restart (is the HA provider initialize OOBM restart or not ?), however
> after host up cloudstack agent not in running state
>
> Process: 4107 ExecStart=/usr/bin/java $JAVA_OPTS $JAVA_DEBUG -cp
> $CLASSPATH $JAVA_CLASS (code=exited, status=0/SUCCESS)
>
> The HA state stucking in Suspect state for a while and then host went to shutdown. Cloudstack show as Down state, resource state - Maintance and Power state Off. However instances not rebooted on healty host within the same cluster. Is any recommendation about that?
>
> Thank you for your time and efforts.
>
>
> Regards,
>
>
>
> Jafar Aghabalayev | Senior IT Infrastructure Engineer
> 157 Azadliq Avenue, AZ1106, Baku, Azerbaijan
> mob: +994 55 900 19 34
> email: jafar.aghabalayev@pasha-technology.com
>
> This communication contains information issued by "PASHA Technology" LLC. This e-mail message and all attachments transmitted with it are intended solely for the use of the addressee and may contain legally privileged and confidential information. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, the reader is hereby notified that any dissemination, distribution, copying, or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to this message and please delete it from your computer. Within the bounds of law "PASHA Technology" LLC may monitor electronic transmissions through its internal and external networks to ensure compliance with internal policies and for legitimate business purposes.
>
> -----Original Message-----
> From: Boris Stoyanov <bo...@shapeblue.com>
> Sent: Saturday, February 11, 2023 2:37 PM
> To: users@cloudstack.apache.org; Wei ZHOU <us...@gmail.com>
> Subject: Re: Cluster HA - KVM Hosts
>
> How did you power off the host? If you shut it down gracefully HA will not kick in, hence considered an intended action. The goal of this feature is to guard against unexpected events, like for example host has crashed.
>
> If you're looking to test it out you can try to provoke a kernel panic
> in the kvm host, something like: echo c > /proc/sysrq-trigger
>
> Then soon HA investigator will determin the host as unhealthy, (pings failing), soon after that there will be an activity checks on the VMs disks running on the host. It will check if there's any changes and disk activity, if that also fails it will determine the host for recovery. It will issue a restart with the IPMI tool, if number of restarts (depending on configuration) also fails it will fence the host and will start the VMs on another host in the same cluster.
>
> I hope this is insightful for you. Let me know how it goes at your end.
>
> Bobby.
>
> From: Jafar Aghabalayev <Ja...@pasha-technology.com>
> Date: Saturday, 11 February 2023, 11:21
> To: Wei ZHOU <us...@gmail.com>, users@cloudstack.apache.org
> <us...@cloudstack.apache.org>
> Subject: Re: Cluster HA - KVM Hosts
> Hello,
> I created new offering, with HA enabled and used it
>
> Sent from Outlook for Android<https://aka.ms/AAb9ysg>
> ________________________________
> From: Wei ZHOU <us...@gmail.com>
> Sent: Saturday, February 11, 2023 12:44:45 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Cc: Jafar Aghabalayev <Ja...@pasha-technology.com>
> Subject: Re: Cluster HA - KVM Hosts
>
>
> CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to soc@pasha-technology.com.
>
> Hi,
>
> Is HA enabled on the VM?
>
> You can check the vm details. HA is not enabled for built-in service offerings (Small/Medium).
>
> -Wei
>
>
>
>
>
> On Saturday, 11 February 2023, pspal83@hotmail.com<ma...@hotmail.com> <ps...@hotmail.com>> wrote:
> Hi Jafar,
> FYI.
> VM HA does not work when server is stopped gracefully * Issue #4211 *
> apache/cloudstack
> |
> |
> |
> | | |
>
> |
>
> |
> |
> | |
> VM HA does not work when server is stopped gracefully * Issue #4211 *
> apache/cloudstack
>
> When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI...
> | |
>
> |
>
> |
>
>
>
> Thanks Pradeep
>
> Sent from Yahoo Mail on Android
>
> On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayev<Ja...@pasha-technology.com>> wrote: Hello Community,
>
> I have configured cluster with 2 KVM Hosts.
> HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage.
> I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same.
>
> Can anyone help me with this issue?
>
>
--
This message is intended only for the use of the individual or entity to which it is addressed and may contain confidential and/or privileged information. If you are not the intended recipient, please delete the original message and any copy of it from your computer system. You are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited unless proper authorization has been obtained for such action. If you have received this communication in error, please notify the sender immediately. Although IndiQus attempts to sweep e-mail and attachments for viruses, it does not guarantee that both are virus-free and accepts no liability for any damage sustained as a result of viruses.
RE: Cluster HA - KVM Hosts
Posted by Jafar Aghabalayev <Ja...@pasha-technology.com>.
Hello,
Thank you for your response.
One primary storage in use and it is NFS and VM volumes located on this NFS storage.
After unplugged the power cable State show alert and power state unknown.
Regards,
Jafar Aghabalayev
-----Original Message-----
From: Vivek Kumar <vi...@indiqus.com.INVALID>
Sent: Tuesday, February 21, 2023 4:10 PM
To: users@cloudstack.apache.org
Cc: Wei ZHOU <us...@gmail.com>; boris.stoyanov@shapeblue.com
Subject: Re: Cluster HA - KVM Hosts
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to soc@pasha-technology.com.
Hello Jafar,
This is the way to test the HA. I did same before putting into the production.
When you unplugged the cable of physical hosts, what is the current state of host ? Is it showing “Disconnected”, “Alert” or “Down”, When I tested the HA it should have atleast one NFS as primary storage, it worked in my case with NFS as a primary storage or atleast one of the primary storage on NFS . So without any NFS storage, whenever I disconnected the host, it was either showing in “Disconnected” or “alert” and then HA will not work.
Vivek Kumar
Sr. Manager - Cloud & DevOps
TechOps | Indiqus Technologies
vivek.kumar@indiqus.com <ma...@indiqus.com>
www.indiqus.com <https://www.indiqus.com/>
> On 21-Feb-2023, at 5:29 PM, Jafar Aghabalayev <Ja...@pasha-technology.com> wrote:
>
> Hello Community,
> My further steps are below:
>
> 1. I fixed the error related to Java and HA state display correct
> state (Available for both hosts) 2. I shutdown the host by unplugging
> power cable 3. The HA state show fencing state 4. No VMs migrated to
> the second available host (all VMs used offering with HA enabled)
>
> Is there are any other options to test HA?
>
>
>
>
> Regards,
> Jafar Aghabalayev
>
>
> -----Original Message-----
> From: Jafar Aghabalayev <Ja...@pasha-technology.com>
> Sent: Monday, February 13, 2023 9:13 PM
> To: users@cloudstack.apache.org; Wei ZHOU <us...@gmail.com>;
> boris.stoyanov@shapeblue.com
> Subject: RE: Cluster HA - KVM Hosts
>
> CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to soc@pasha-technology.com.
>
> Hello,
> I run the echo c > /proc/sysrq-trigger command and host went to
> restart (is the HA provider initialize OOBM restart or not ?), however
> after host up cloudstack agent not in running state
>
> Process: 4107 ExecStart=/usr/bin/java $JAVA_OPTS $JAVA_DEBUG -cp
> $CLASSPATH $JAVA_CLASS (code=exited, status=0/SUCCESS)
>
> The HA state stucking in Suspect state for a while and then host went to shutdown. Cloudstack show as Down state, resource state - Maintance and Power state Off. However instances not rebooted on healty host within the same cluster. Is any recommendation about that?
>
> Thank you for your time and efforts.
>
>
> Regards,
>
>
>
> Jafar Aghabalayev | Senior IT Infrastructure Engineer
> 157 Azadliq Avenue, AZ1106, Baku, Azerbaijan
> mob: +994 55 900 19 34
> email: jafar.aghabalayev@pasha-technology.com
>
> This communication contains information issued by "PASHA Technology" LLC. This e-mail message and all attachments transmitted with it are intended solely for the use of the addressee and may contain legally privileged and confidential information. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, the reader is hereby notified that any dissemination, distribution, copying, or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to this message and please delete it from your computer. Within the bounds of law "PASHA Technology" LLC may monitor electronic transmissions through its internal and external networks to ensure compliance with internal policies and for legitimate business purposes.
>
> -----Original Message-----
> From: Boris Stoyanov <bo...@shapeblue.com>
> Sent: Saturday, February 11, 2023 2:37 PM
> To: users@cloudstack.apache.org; Wei ZHOU <us...@gmail.com>
> Subject: Re: Cluster HA - KVM Hosts
>
> How did you power off the host? If you shut it down gracefully HA will not kick in, hence considered an intended action. The goal of this feature is to guard against unexpected events, like for example host has crashed.
>
> If you're looking to test it out you can try to provoke a kernel panic
> in the kvm host, something like: echo c > /proc/sysrq-trigger
>
> Then soon HA investigator will determin the host as unhealthy, (pings failing), soon after that there will be an activity checks on the VMs disks running on the host. It will check if there's any changes and disk activity, if that also fails it will determine the host for recovery. It will issue a restart with the IPMI tool, if number of restarts (depending on configuration) also fails it will fence the host and will start the VMs on another host in the same cluster.
>
> I hope this is insightful for you. Let me know how it goes at your end.
>
> Bobby.
>
> From: Jafar Aghabalayev <Ja...@pasha-technology.com>
> Date: Saturday, 11 February 2023, 11:21
> To: Wei ZHOU <us...@gmail.com>, users@cloudstack.apache.org
> <us...@cloudstack.apache.org>
> Subject: Re: Cluster HA - KVM Hosts
> Hello,
> I created new offering, with HA enabled and used it
>
> Sent from Outlook for Android<https://aka.ms/AAb9ysg>
> ________________________________
> From: Wei ZHOU <us...@gmail.com>
> Sent: Saturday, February 11, 2023 12:44:45 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Cc: Jafar Aghabalayev <Ja...@pasha-technology.com>
> Subject: Re: Cluster HA - KVM Hosts
>
>
> CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to soc@pasha-technology.com.
>
> Hi,
>
> Is HA enabled on the VM?
>
> You can check the vm details. HA is not enabled for built-in service offerings (Small/Medium).
>
> -Wei
>
>
>
>
>
> On Saturday, 11 February 2023, pspal83@hotmail.com<ma...@hotmail.com> <ps...@hotmail.com>> wrote:
> Hi Jafar,
> FYI.
> VM HA does not work when server is stopped gracefully * Issue #4211 *
> apache/cloudstack
> |
> |
> |
> | | |
>
> |
>
> |
> |
> | |
> VM HA does not work when server is stopped gracefully * Issue #4211 *
> apache/cloudstack
>
> When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI...
> | |
>
> |
>
> |
>
>
>
> Thanks Pradeep
>
> Sent from Yahoo Mail on Android
>
> On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayev<Ja...@pasha-technology.com>> wrote: Hello Community,
>
> I have configured cluster with 2 KVM Hosts.
> HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage.
> I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same.
>
> Can anyone help me with this issue?
>
>
--
This message is intended only for the use of the individual or entity to which it is addressed and may contain confidential and/or privileged information. If you are not the intended recipient, please delete the original message and any copy of it from your computer system. You are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited unless proper authorization has been obtained for such action. If you have received this communication in error, please notify the sender immediately. Although IndiQus attempts to sweep e-mail and attachments for viruses, it does not guarantee that both are virus-free and accepts no liability for any damage sustained as a result of viruses.
Re: Cluster HA - KVM Hosts
Posted by Vivek Kumar <vi...@indiqus.com.INVALID>.
Hello Jafar,
This is the way to test the HA. I did same before putting into the production.
When you unplugged the cable of physical hosts, what is the current state of host ? Is it showing “Disconnected”, “Alert” or “Down”, When I tested the HA it should have atleast one NFS as primary storage, it worked in my case with NFS as a primary storage or atleast one of the primary storage on NFS . So without any NFS storage, whenever I disconnected the host, it was either showing in “Disconnected” or “alert” and then HA will not work.
Vivek Kumar
Sr. Manager - Cloud & DevOps
TechOps | Indiqus Technologies
vivek.kumar@indiqus.com <ma...@indiqus.com>
www.indiqus.com <https://www.indiqus.com/>
> On 21-Feb-2023, at 5:29 PM, Jafar Aghabalayev <Ja...@pasha-technology.com> wrote:
>
> Hello Community,
> My further steps are below:
>
> 1. I fixed the error related to Java and HA state display correct state (Available for both hosts)
> 2. I shutdown the host by unplugging power cable
> 3. The HA state show fencing state
> 4. No VMs migrated to the second available host (all VMs used offering with HA enabled)
>
> Is there are any other options to test HA?
>
>
>
>
> Regards,
> Jafar Aghabalayev
>
>
> -----Original Message-----
> From: Jafar Aghabalayev <Ja...@pasha-technology.com>
> Sent: Monday, February 13, 2023 9:13 PM
> To: users@cloudstack.apache.org; Wei ZHOU <us...@gmail.com>; boris.stoyanov@shapeblue.com
> Subject: RE: Cluster HA - KVM Hosts
>
> CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to soc@pasha-technology.com.
>
> Hello,
> I run the echo c > /proc/sysrq-trigger command and host went to restart (is the HA provider initialize OOBM restart or not ?), however after host up cloudstack agent not in running state
>
> Process: 4107 ExecStart=/usr/bin/java $JAVA_OPTS $JAVA_DEBUG -cp $CLASSPATH $JAVA_CLASS (code=exited, status=0/SUCCESS)
>
> The HA state stucking in Suspect state for a while and then host went to shutdown. Cloudstack show as Down state, resource state - Maintance and Power state Off. However instances not rebooted on healty host within the same cluster. Is any recommendation about that?
>
> Thank you for your time and efforts.
>
>
> Regards,
>
>
>
> Jafar Aghabalayev | Senior IT Infrastructure Engineer
> 157 Azadliq Avenue, AZ1106, Baku, Azerbaijan
> mob: +994 55 900 19 34
> email: jafar.aghabalayev@pasha-technology.com
>
> This communication contains information issued by "PASHA Technology" LLC. This e-mail message and all attachments transmitted with it are intended solely for the use of the addressee and may contain legally privileged and confidential information. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, the reader is hereby notified that any dissemination, distribution, copying, or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to this message and please delete it from your computer. Within the bounds of law "PASHA Technology" LLC may monitor electronic transmissions through its internal and external networks to ensure compliance with internal policies and for legitimate business purposes.
>
> -----Original Message-----
> From: Boris Stoyanov <bo...@shapeblue.com>
> Sent: Saturday, February 11, 2023 2:37 PM
> To: users@cloudstack.apache.org; Wei ZHOU <us...@gmail.com>
> Subject: Re: Cluster HA - KVM Hosts
>
> How did you power off the host? If you shut it down gracefully HA will not kick in, hence considered an intended action. The goal of this feature is to guard against unexpected events, like for example host has crashed.
>
> If you're looking to test it out you can try to provoke a kernel panic in the kvm host, something like: echo c > /proc/sysrq-trigger
>
> Then soon HA investigator will determin the host as unhealthy, (pings failing), soon after that there will be an activity checks on the VMs disks running on the host. It will check if there's any changes and disk activity, if that also fails it will determine the host for recovery. It will issue a restart with the IPMI tool, if number of restarts (depending on configuration) also fails it will fence the host and will start the VMs on another host in the same cluster.
>
> I hope this is insightful for you. Let me know how it goes at your end.
>
> Bobby.
>
> From: Jafar Aghabalayev <Ja...@pasha-technology.com>
> Date: Saturday, 11 February 2023, 11:21
> To: Wei ZHOU <us...@gmail.com>, users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Cluster HA - KVM Hosts
> Hello,
> I created new offering, with HA enabled and used it
>
> Sent from Outlook for Android<https://aka.ms/AAb9ysg> ________________________________
> From: Wei ZHOU <us...@gmail.com>
> Sent: Saturday, February 11, 2023 12:44:45 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Cc: Jafar Aghabalayev <Ja...@pasha-technology.com>
> Subject: Re: Cluster HA - KVM Hosts
>
>
> CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to soc@pasha-technology.com.
>
> Hi,
>
> Is HA enabled on the VM?
>
> You can check the vm details. HA is not enabled for built-in service offerings (Small/Medium).
>
> -Wei
>
>
>
>
>
> On Saturday, 11 February 2023, pspal83@hotmail.com<ma...@hotmail.com> <ps...@hotmail.com>> wrote:
> Hi Jafar,
> FYI.
> VM HA does not work when server is stopped gracefully * Issue #4211 * apache/cloudstack
> |
> |
> |
> | | |
>
> |
>
> |
> |
> | |
> VM HA does not work when server is stopped gracefully * Issue #4211 * apache/cloudstack
>
> When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI...
> | |
>
> |
>
> |
>
>
>
> Thanks Pradeep
>
> Sent from Yahoo Mail on Android
>
> On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayev<Ja...@pasha-technology.com>> wrote: Hello Community,
>
> I have configured cluster with 2 KVM Hosts.
> HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage.
> I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same.
>
> Can anyone help me with this issue?
>
>
--
This message is intended only for the use of the individual or entity to
which it is addressed and may contain confidential and/or privileged
information. If you are not the intended recipient, please delete the
original message and any copy of it from your computer system. You are
hereby notified that any dissemination, distribution or copying of this
communication is strictly prohibited unless proper authorization has been
obtained for such action. If you have received this communication in error,
please notify the sender immediately. Although IndiQus attempts to sweep
e-mail and attachments for viruses, it does not guarantee that both are
virus-free and accepts no liability for any damage sustained as a result of
viruses.
RE: Cluster HA - KVM Hosts
Posted by Jafar Aghabalayev <Ja...@pasha-technology.com>.
Hello Community,
My further steps are below:
1. I fixed the error related to Java and HA state display correct state (Available for both hosts)
2. I shutdown the host by unplugging power cable
3. The HA state show fencing state
4. No VMs migrated to the second available host (all VMs used offering with HA enabled)
Is there are any other options to test HA?
Regards,
Jafar Aghabalayev
-----Original Message-----
From: Jafar Aghabalayev <Ja...@pasha-technology.com>
Sent: Monday, February 13, 2023 9:13 PM
To: users@cloudstack.apache.org; Wei ZHOU <us...@gmail.com>; boris.stoyanov@shapeblue.com
Subject: RE: Cluster HA - KVM Hosts
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to soc@pasha-technology.com.
Hello,
I run the echo c > /proc/sysrq-trigger command and host went to restart (is the HA provider initialize OOBM restart or not ?), however after host up cloudstack agent not in running state
Process: 4107 ExecStart=/usr/bin/java $JAVA_OPTS $JAVA_DEBUG -cp $CLASSPATH $JAVA_CLASS (code=exited, status=0/SUCCESS)
The HA state stucking in Suspect state for a while and then host went to shutdown. Cloudstack show as Down state, resource state - Maintance and Power state Off. However instances not rebooted on healty host within the same cluster. Is any recommendation about that?
Thank you for your time and efforts.
Regards,
Jafar Aghabalayev | Senior IT Infrastructure Engineer
157 Azadliq Avenue, AZ1106, Baku, Azerbaijan
mob: +994 55 900 19 34
email: jafar.aghabalayev@pasha-technology.com
This communication contains information issued by "PASHA Technology" LLC. This e-mail message and all attachments transmitted with it are intended solely for the use of the addressee and may contain legally privileged and confidential information. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, the reader is hereby notified that any dissemination, distribution, copying, or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to this message and please delete it from your computer. Within the bounds of law "PASHA Technology" LLC may monitor electronic transmissions through its internal and external networks to ensure compliance with internal policies and for legitimate business purposes.
-----Original Message-----
From: Boris Stoyanov <bo...@shapeblue.com>
Sent: Saturday, February 11, 2023 2:37 PM
To: users@cloudstack.apache.org; Wei ZHOU <us...@gmail.com>
Subject: Re: Cluster HA - KVM Hosts
How did you power off the host? If you shut it down gracefully HA will not kick in, hence considered an intended action. The goal of this feature is to guard against unexpected events, like for example host has crashed.
If you're looking to test it out you can try to provoke a kernel panic in the kvm host, something like: echo c > /proc/sysrq-trigger
Then soon HA investigator will determin the host as unhealthy, (pings failing), soon after that there will be an activity checks on the VMs disks running on the host. It will check if there's any changes and disk activity, if that also fails it will determine the host for recovery. It will issue a restart with the IPMI tool, if number of restarts (depending on configuration) also fails it will fence the host and will start the VMs on another host in the same cluster.
I hope this is insightful for you. Let me know how it goes at your end.
Bobby.
From: Jafar Aghabalayev <Ja...@pasha-technology.com>
Date: Saturday, 11 February 2023, 11:21
To: Wei ZHOU <us...@gmail.com>, users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Cluster HA - KVM Hosts
Hello,
I created new offering, with HA enabled and used it
Sent from Outlook for Android<https://aka.ms/AAb9ysg> ________________________________
From: Wei ZHOU <us...@gmail.com>
Sent: Saturday, February 11, 2023 12:44:45 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Cc: Jafar Aghabalayev <Ja...@pasha-technology.com>
Subject: Re: Cluster HA - KVM Hosts
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to soc@pasha-technology.com.
Hi,
Is HA enabled on the VM?
You can check the vm details. HA is not enabled for built-in service offerings (Small/Medium).
-Wei
On Saturday, 11 February 2023, pspal83@hotmail.com<ma...@hotmail.com> <ps...@hotmail.com>> wrote:
Hi Jafar,
FYI.
VM HA does not work when server is stopped gracefully * Issue #4211 * apache/cloudstack
|
|
|
| | |
|
|
|
| |
VM HA does not work when server is stopped gracefully * Issue #4211 * apache/cloudstack
When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI...
| |
|
|
Thanks Pradeep
Sent from Yahoo Mail on Android
On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayev<Ja...@pasha-technology.com>> wrote: Hello Community,
I have configured cluster with 2 KVM Hosts.
HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage.
I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same.
Can anyone help me with this issue?
RE: Cluster HA - KVM Hosts
Posted by Jafar Aghabalayev <Ja...@pasha-technology.com>.
Hello,
I run the echo c > /proc/sysrq-trigger command and host went to restart (is the HA provider initialize OOBM restart or not ?), however after host up cloudstack agent not in running state
Process: 4107 ExecStart=/usr/bin/java $JAVA_OPTS $JAVA_DEBUG -cp $CLASSPATH $JAVA_CLASS (code=exited, status=0/SUCCESS)
The HA state stucking in Suspect state for a while and then host went to shutdown. Cloudstack show as Down state, resource state - Maintance and Power state Off. However instances not rebooted on healty host within the same cluster. Is any recommendation about that?
Thank you for your time and efforts.
Regards,
Jafar Aghabalayev | Senior IT Infrastructure Engineer
157 Azadliq Avenue, AZ1106, Baku, Azerbaijan
mob: +994 55 900 19 34
email: jafar.aghabalayev@pasha-technology.com
This communication contains information issued by "PASHA Technology" LLC. This e-mail message and all attachments transmitted with it are intended solely for the use of the addressee and may contain legally privileged and confidential information. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, the reader is hereby notified that any dissemination, distribution, copying, or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to this message and please delete it from your computer. Within the bounds of law "PASHA Technology" LLC may monitor electronic transmissions through its internal and external networks to ensure compliance with internal policies and for legitimate business purposes.
-----Original Message-----
From: Boris Stoyanov <bo...@shapeblue.com>
Sent: Saturday, February 11, 2023 2:37 PM
To: users@cloudstack.apache.org; Wei ZHOU <us...@gmail.com>
Subject: Re: Cluster HA - KVM Hosts
How did you power off the host? If you shut it down gracefully HA will not kick in, hence considered an intended action. The goal of this feature is to guard against unexpected events, like for example host has crashed.
If you're looking to test it out you can try to provoke a kernel panic in the kvm host, something like: echo c > /proc/sysrq-trigger
Then soon HA investigator will determin the host as unhealthy, (pings failing), soon after that there will be an activity checks on the VMs disks running on the host. It will check if there's any changes and disk activity, if that also fails it will determine the host for recovery. It will issue a restart with the IPMI tool, if number of restarts (depending on configuration) also fails it will fence the host and will start the VMs on another host in the same cluster.
I hope this is insightful for you. Let me know how it goes at your end.
Bobby.
From: Jafar Aghabalayev <Ja...@pasha-technology.com>
Date: Saturday, 11 February 2023, 11:21
To: Wei ZHOU <us...@gmail.com>, users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Cluster HA - KVM Hosts
Hello,
I created new offering, with HA enabled and used it
Sent from Outlook for Android<https://aka.ms/AAb9ysg> ________________________________
From: Wei ZHOU <us...@gmail.com>
Sent: Saturday, February 11, 2023 12:44:45 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Cc: Jafar Aghabalayev <Ja...@pasha-technology.com>
Subject: Re: Cluster HA - KVM Hosts
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to soc@pasha-technology.com.
Hi,
Is HA enabled on the VM?
You can check the vm details. HA is not enabled for built-in service offerings (Small/Medium).
-Wei
On Saturday, 11 February 2023, pspal83@hotmail.com<ma...@hotmail.com> <ps...@hotmail.com>> wrote:
Hi Jafar,
FYI.
VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack
|
|
|
| | |
|
|
|
| |
VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack
When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI...
| |
|
|
Thanks Pradeep
Sent from Yahoo Mail on Android
On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayev<Ja...@pasha-technology.com>> wrote: Hello Community,
I have configured cluster with 2 KVM Hosts.
HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage.
I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same.
Can anyone help me with this issue?
Re: Cluster HA - KVM Hosts
Posted by Boris Stoyanov <bo...@shapeblue.com>.
How did you power off the host? If you shut it down gracefully HA will not kick in, hence considered an intended action. The goal of this feature is to guard against unexpected events, like for example host has crashed.
If you’re looking to test it out you can try to provoke a kernel panic in the kvm host, something like: echo c > /proc/sysrq-trigger
Then soon HA investigator will determin the host as unhealthy, (pings failing), soon after that there will be an activity checks on the VMs disks running on the host. It will check if there’s any changes and disk activity, if that also fails it will determine the host for recovery. It will issue a restart with the IPMI tool, if number of restarts (depending on configuration) also fails it will fence the host and will start the VMs on another host in the same cluster.
I hope this is insightful for you. Let me know how it goes at your end.
Bobby.
From: Jafar Aghabalayev <Ja...@pasha-technology.com>
Date: Saturday, 11 February 2023, 11:21
To: Wei ZHOU <us...@gmail.com>, users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Cluster HA - KVM Hosts
Hello,
I created new offering, with HA enabled and used it
Sent from Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Wei ZHOU <us...@gmail.com>
Sent: Saturday, February 11, 2023 12:44:45 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Cc: Jafar Aghabalayev <Ja...@pasha-technology.com>
Subject: Re: Cluster HA - KVM Hosts
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to soc@pasha-technology.com.
Hi,
Is HA enabled on the VM?
You can check the vm details. HA is not enabled for built-in service offerings (Small/Medium).
-Wei
On Saturday, 11 February 2023, pspal83@hotmail.com<ma...@hotmail.com> <ps...@hotmail.com>> wrote:
Hi Jafar,
FYI.
VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack
|
|
|
| | |
|
|
|
| |
VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack
When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI...
| |
|
|
Thanks Pradeep
Sent from Yahoo Mail on Android
On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayev<Ja...@pasha-technology.com>> wrote: Hello Community,
I have configured cluster with 2 KVM Hosts.
HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage.
I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same.
Can anyone help me with this issue?
Re: Cluster HA - KVM Hosts
Posted by Jafar Aghabalayev <Ja...@pasha-technology.com>.
Hello,
I created new offering, with HA enabled and used it
Sent from Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Wei ZHOU <us...@gmail.com>
Sent: Saturday, February 11, 2023 12:44:45 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Cc: Jafar Aghabalayev <Ja...@pasha-technology.com>
Subject: Re: Cluster HA - KVM Hosts
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to soc@pasha-technology.com.
Hi,
Is HA enabled on the VM?
You can check the vm details. HA is not enabled for built-in service offerings (Small/Medium).
-Wei
On Saturday, 11 February 2023, pspal83@hotmail.com<ma...@hotmail.com> <ps...@hotmail.com>> wrote:
Hi Jafar,
FYI.
VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack
|
|
|
| | |
|
|
|
| |
VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack
When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI...
| |
|
|
Thanks Pradeep
Sent from Yahoo Mail on Android
On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayev<Ja...@pasha-technology.com>> wrote: Hello Community,
I have configured cluster with 2 KVM Hosts.
HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage.
I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same.
Can anyone help me with this issue?
Re: Cluster HA - KVM Hosts
Posted by Wei ZHOU <us...@gmail.com>.
Hi,
Is HA enabled on the VM?
You can check the vm details. HA is not enabled for built-in service
offerings (Small/Medium).
-Wei
On Saturday, 11 February 2023, pspal83@hotmail.com <ps...@hotmail.com>
wrote:
> Hi Jafar,
> FYI.
> VM HA does not work when server is stopped gracefully · Issue #4211 ·
> apache/cloudstack
> |
> |
> |
> | | |
>
> |
>
> |
> |
> | |
> VM HA does not work when server is stopped gracefully · Issue #4211 ·
> apache/cloudstack
>
> When we stop a server (hypervisor) gracefully, vms on it will NOT be
> started on other servers even HA is enabled. ISSUE TYPE Bug Report
> COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI...
> | |
>
> |
>
> |
>
>
>
> Thanks Pradeep
>
> Sent from Yahoo Mail on Android
>
> On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayev<Jafar.Aghabalayev@
> pasha-technology.com> wrote: Hello Community,
>
> I have configured cluster with 2 KVM Hosts.
> HA enabled for hosts, ha.tag attribute on global configuration is the same
> as for ha.tag attribute for hosts. VMs are running with HA enabled
> offering. NFS used as primary storage.
> I tried to shutdown one of the hosts, however HA not performed. State and
> power state indicated as down, but resource state show as up and VM located
> at the failed host show in running state (in real it is unaccessible). I
> tried to stop libvirtd service and the result same.
>
> Can anyone help me with this issue?
>
>
>
>
Re: Cluster HA - KVM Hosts
Posted by Boris Stoyanov <bo...@shapeblue.com>.
Hi Jafar, this issue has been fixed since 4.14 as far as I read it, so it should be in 4.17.
As for the failures you have with agent not starting you should check the management and agent logs, you can trace what did restarted the KVM host (likely OOBM action) and why is the agent not starting.
From: Jafar Aghabalayev <Ja...@pasha-technology.com>
Date: Monday, 13 February 2023, 19:14
To: pspal83@hotmail.com <ps...@hotmail.com>, users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: RE: Cluster HA - KVM Hosts
Hello,
Do you have any information about fixing this bug on cloudstack ver 4.17 ?
Thank you
Regards,
[cid:image001.png@01D93FF0.1A70EED0]
Jafar Aghabalayev | Senior IT Infrastructure Engineer
157 Azadliq Avenue, AZ1106, Baku, Azerbaijan
mob: +994 55 900 19 34
email: jafar.aghabalayev@pasha-technology.com<ma...@pasha-technology.com>
This communication contains information issued by "PASHA Technology" LLC. This e-mail message and all attachments transmitted with it are intended solely for the use of the addressee and may contain legally privileged and confidential information. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, the reader is hereby notified that any dissemination, distribution, copying, or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to this message and please delete it from your computer. Within the bounds of law "PASHA Technology" LLC may monitor electronic transmissions through its internal and external networks to ensure compliance with internal policies and for legitimate business purposes.
From: pspal83@hotmail.com <ps...@hotmail.com>
Sent: Saturday, February 11, 2023 9:02 AM
To: users@cloudstack.apache.org; Jafar Aghabalayev <Ja...@pasha-technology.com>
Subject: Re: Cluster HA - KVM Hosts
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to soc@pasha-technology.com<ma...@pasha-technology.com>.
Hi Jafar,
FYI.
VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack<https://github.com/apache/cloudstack/issues/4211>
VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack
When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI...
Thanks
Pradeep
Sent from Yahoo Mail on Android<https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayev
<Ja...@pasha-technology.com>> wrote:
Hello Community,
I have configured cluster with 2 KVM Hosts.
HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage.
I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same.
Can anyone help me with this issue?
RE: Cluster HA - KVM Hosts
Posted by Jafar Aghabalayev <Ja...@pasha-technology.com>.
Hello,
Do you have any information about fixing this bug on cloudstack ver 4.17 ?
Thank you
Regards,
[cid:image001.png@01D93FF0.1A70EED0]
Jafar Aghabalayev | Senior IT Infrastructure Engineer
157 Azadliq Avenue, AZ1106, Baku, Azerbaijan
mob: +994 55 900 19 34
email: jafar.aghabalayev@pasha-technology.com<ma...@pasha-technology.com>
This communication contains information issued by "PASHA Technology" LLC. This e-mail message and all attachments transmitted with it are intended solely for the use of the addressee and may contain legally privileged and confidential information. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, the reader is hereby notified that any dissemination, distribution, copying, or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately by replying to this message and please delete it from your computer. Within the bounds of law "PASHA Technology" LLC may monitor electronic transmissions through its internal and external networks to ensure compliance with internal policies and for legitimate business purposes.
From: pspal83@hotmail.com <ps...@hotmail.com>
Sent: Saturday, February 11, 2023 9:02 AM
To: users@cloudstack.apache.org; Jafar Aghabalayev <Ja...@pasha-technology.com>
Subject: Re: Cluster HA - KVM Hosts
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to soc@pasha-technology.com<ma...@pasha-technology.com>.
Hi Jafar,
FYI.
VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack<https://github.com/apache/cloudstack/issues/4211>
VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack
When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI...
Thanks
Pradeep
Sent from Yahoo Mail on Android<https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayev
<Ja...@pasha-technology.com>> wrote:
Hello Community,
I have configured cluster with 2 KVM Hosts.
HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage.
I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same.
Can anyone help me with this issue?
Re: Cluster HA - KVM Hosts
Posted by "pspal83@hotmail.com" <ps...@hotmail.com>.
Hi Jafar,
FYI.
VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack
|
|
|
| | |
|
|
|
| |
VM HA does not work when server is stopped gracefully · Issue #4211 · apache/cloudstack
When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled. ISSUE TYPE Bug Report COMPONENT NAME VM HA CLOUDSTACK VERSION 4.14/4.15 CONFIGURATI...
| |
|
|
Thanks Pradeep
Sent from Yahoo Mail on Android
On Sat, 11 Feb 2023 at 12:02 am, Jafar Aghabalayev<Ja...@pasha-technology.com> wrote: Hello Community,
I have configured cluster with 2 KVM Hosts.
HA enabled for hosts, ha.tag attribute on global configuration is the same as for ha.tag attribute for hosts. VMs are running with HA enabled offering. NFS used as primary storage.
I tried to shutdown one of the hosts, however HA not performed. State and power state indicated as down, but resource state show as up and VM located at the failed host show in running state (in real it is unaccessible). I tried to stop libvirtd service and the result same.
Can anyone help me with this issue?
Re: Cluster HA - KVM Hosts
Posted by Jafar Aghabalayev <Ja...@pasha-technology.com>.
Hello,
OOBM also configured using IPMI and when i shut the host, power state show correct state and im able to control (power off, power on) host using OOBM on cloudstack
Sent from Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Simon Weller <si...@gmail.com>
Sent: Friday, February 10, 2023 11:07:15 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Cluster HA - KVM Hosts
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Please report all suspicious emails to soc@pasha-technology.com.
Hey Jafar,
For HA to work on KVM, you need to configure each host with out-of-band
management using ipmitool or Redfish. This is so Cloudstack can fence your
nodes.
This covers the concept -
https://docs.cloudstack.apache.org/en/latest/adminguide/reliability.html?highlight=fencing#ha-enabled-hosts
Out-of-band documentation is here -
https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html#out-of-band-management
For details on how the feature works, check out the original design
document - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA
-Si
On Fri, Feb 10, 2023 at 12:32 PM Jafar Aghabalayev <
Jafar.Aghabalayev@pasha-technology.com> wrote:
> Hello Community,
>
> I have configured cluster with 2 KVM Hosts.
> HA enabled for hosts, ha.tag attribute on global configuration is the same
> as for ha.tag attribute for hosts. VMs are running with HA enabled
> offering. NFS used as primary storage.
> I tried to shutdown one of the hosts, however HA not performed. State and
> power state indicated as down, but resource state show as up and VM located
> at the failed host show in running state (in real it is unaccessible). I
> tried to stop libvirtd service and the result same.
>
> Can anyone help me with this issue?
>
>
>
Re: Cluster HA - KVM Hosts
Posted by Simon Weller <si...@gmail.com>.
Hey Jafar,
For HA to work on KVM, you need to configure each host with out-of-band
management using ipmitool or Redfish. This is so Cloudstack can fence your
nodes.
This covers the concept -
https://docs.cloudstack.apache.org/en/latest/adminguide/reliability.html?highlight=fencing#ha-enabled-hosts
Out-of-band documentation is here -
https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html#out-of-band-management
For details on how the feature works, check out the original design
document - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA
-Si
On Fri, Feb 10, 2023 at 12:32 PM Jafar Aghabalayev <
Jafar.Aghabalayev@pasha-technology.com> wrote:
> Hello Community,
>
> I have configured cluster with 2 KVM Hosts.
> HA enabled for hosts, ha.tag attribute on global configuration is the same
> as for ha.tag attribute for hosts. VMs are running with HA enabled
> offering. NFS used as primary storage.
> I tried to shutdown one of the hosts, however HA not performed. State and
> power state indicated as down, but resource state show as up and VM located
> at the failed host show in running state (in real it is unaccessible). I
> tried to stop libvirtd service and the result same.
>
> Can anyone help me with this issue?
>
>
>