You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "rahultolearn59 (via GitHub)" <gi...@apache.org> on 2024/04/15 18:48:44 UTC
[I] OOBM Sends Resthe et Command to DRAC if Hypervisor is in powerOff state [cloudstack]
rahultolearn59 opened a new issue, #8918:
URL: https://github.com/apache/cloudstack/issues/8918
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
KVMHA
-->
~~~
~~~
##### CLOUDSTACK VERSION
<!--
4.19
-->
~~~
~~~
##### CONFIGURATION
<!--
OOBM Command Implementation
-->
##### OS / ENVIRONMENT
<!--
KVM (RHEL8.X)
-->
##### SUMMARY
<!-- Explain the problem/feature briefly -->
When a hypervisor with RHEL8/9 running on a Dell PowerEdge Server gets into the power-off state, the ACS management system cannot fence it to recover the VMs on another hypervisor as it keeps sending DRAC a system reset request. Still, Drac has Poweron as the only option while its server is in a power-off state.
I waited for 1-2 hours to conclude.
As soon as I power on the server manually, VMs are restored on different hypervisors immediately.
I think the server's power status should be checked first, and a power-on/reset command should be sent accordingly.
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
~~~
Power off the hypervisor.
~~~
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
VMs should be recovered on active hypervisor.
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
~~~
VMs were not able to recover
~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OOBM Sends the Reset Command to DRAC if Hypervisor is in PowerOff State [cloudstack]
Posted by "tanganellilore (via GitHub)" <gi...@apache.org>.
tanganellilore commented on issue #8918:
URL: https://github.com/apache/cloudstack/issues/8918#issuecomment-2304416773
Hi team,
same issue on my test case.
If I simulate a disruption with powering off the server, idrac command fail.
If i simulate a disruption like power off machine and idrac not reachable, idrac command fail.
In both case host remain in "fancing" ad libitum until i restart the server or idrac will be reachable.
All vm on this host remain on this host in all tests, until i degraded node manually from UI.
OOBM command in a powerd off host, return an exit code 1 (like reset for recovery state).
I read some of your code and i think that error can be on these two pices:
https://github.com/apache/cloudstack/blob/b215abc30a22d6b11f016b8f402981445140f577/server/src/main/java/org/apache/cloudstack/ha/HAManagerImpl.java#L523-L529
and
https://github.com/apache/cloudstack/blob/b215abc30a22d6b11f016b8f402981445140f577/server/src/main/java/org/apache/cloudstack/ha/task/FenceTask.java#L48-L53
because function return always true on `fancing` state and due to the fact that OOBM go in exception, result will be `false` and we not going out to the "loop" of fancing.
One workaround could be introduce how may attempts we can wait until host change state to `inegilible` or `disabled`, like for recovery, or manage reset and powerOff OOBM error to avoid exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OOBM Sends the Reset Command to DRAC if Hypervisor is in PowerOff State [cloudstack]
Posted by "rahultolearn59 (via GitHub)" <gi...@apache.org>.
rahultolearn59 commented on issue #8918:
URL: https://github.com/apache/cloudstack/issues/8918#issuecomment-2085757523
Thanks again for looking into this issue, @rohityadavcloud !
Please find the setup information below:
- ACS 4.19 on RHEL9
- 4 KVM (RHEL8) hosts (Dell PowerEdge R740)
- All KVM hosts have OOBM enabled
- All the 3 VMs sitting in the cluster are HA-Enabled
- Using NFS as primary storage pool
To reproduce:
- Power off the KVM host (hosting one or more VM) by either init 0 or from DRAC console
- The expectation is that VMs sitting on that host should be powered on onto a different Host in the cluster
Observation:
- ACS tries resetting the KVM host via DRAC command (redfish api) before fencing it. But since the host is powered off already, DRAC has PowerON as the only option available
- ACS gets into some sort of loop here it seems
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OOBM Sends the Reset Command to DRAC if Hypervisor is in PowerOff State [cloudstack]
Posted by "rohityadavcloud (via GitHub)" <gi...@apache.org>.
rohityadavcloud commented on issue #8918:
URL: https://github.com/apache/cloudstack/issues/8918#issuecomment-2085353448
Were the VMs using a HA-enabled service offering @rahultolearn59 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OOBM Sends the Reset Command to DRAC if Hypervisor is in PowerOff State [cloudstack]
Posted by "rohityadavcloud (via GitHub)" <gi...@apache.org>.
rohityadavcloud commented on issue #8918:
URL: https://github.com/apache/cloudstack/issues/8918#issuecomment-2085585083
Is the hypervisor host enabled with OOBM and Host-HA feature @rahultolearn59 ? It's possible the configuration puts Host HA before VM HA. In case your env isn't configure for host HA, disable that and see if VM HA still works?
Could you also share steps of reproducing this? On a high level, how have you configured the env, what is enabled/configured to reproduce this behaviour ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OOBM Sends the Reset Command to DRAC if Hypervisor is in PowerOff State [cloudstack]
Posted by "rohityadavcloud (via GitHub)" <gi...@apache.org>.
rohityadavcloud commented on issue #8918:
URL: https://github.com/apache/cloudstack/issues/8918#issuecomment-2811632119
@rahultolearn59 @tanganellilore I think you're both mixing Host-HA with VM-HA, VM HA is still achievable without configuring Host-HA feature at all. For Host-HA to work, you'll need to ensure you're using the right oobm plugin & it's configured properly. On some Linux env such as EL8/9, you may need to install a specific version of ipmitool to get the oobm working with ipmi (this is due to the CLI/tool issue). This is mentioned here: https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html#out-of-band-management:
_
> IPMI based out-of-band management and Host HA may not work on Centos 8 using the default ipmitool version - Installing ipmitool-1.8.18-12.el8_1.x86_64.rpm may solve the problem. Make sure to test the ipmitool on your physical equipment before using the IPMI-based out-of-band management and Host HA features.
_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OOBM Sends the Reset Command to DRAC if Hypervisor is in PowerOff State [cloudstack]
Posted by "rahultolearn59 (via GitHub)" <gi...@apache.org>.
rahultolearn59 commented on issue #8918:
URL: https://github.com/apache/cloudstack/issues/8918#issuecomment-2060033217
Thanks, @DaanHoogland! Just for FYI; this is the exception message I see in the logs.
2024-04-16 22:43:30,146 WARN [o.a.c.h.t.BaseHATask] (pool-4-thread-1:null) (logid:3e5b7727) Exception occurred while running RecoveryTask on a resource: org.apache.cloudstack.ha.provider.HARecoveryException: OOBM service is not configured or enabled for this host <drac-domain>
org.apache.cloudstack.ha.provider.HARecoveryException: OOBM service is not configured or enabled for this host <drac-domain>
at org.apache.cloudstack.kvm.ha.KVMHAProvider.recover(KVMHAProvider.java:83)
at org.apache.cloudstack.kvm.ha.KVMHAProvider.recover(KVMHAProvider.java:42)
at org.apache.cloudstack.ha.task.RecoveryTask.performAction(RecoveryTask.java:43)
at org.apache.cloudstack.ha.task.BaseHATask$1.call(BaseHATask.java:86)
at org.apache.cloudstack.ha.task.BaseHATask$1.call(BaseHATask.java:83)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.cloudstack.utils.redfish.RedfishException: Failed to get System power state for host 'GET' with request 'https://<drac-domain>/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset: <drac-domain>'. The expected HTTP status code is '2XX' but it got '409'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OOBM Sends Resthe et Command to DRAC if Hypervisor is in powerOff state [cloudstack]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #8918:
URL: https://github.com/apache/cloudstack/issues/8918#issuecomment-2057585549
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OOBM Sends the Reset Command to DRAC if Hypervisor is in PowerOff State [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #8918:
URL: https://github.com/apache/cloudstack/issues/8918#issuecomment-2058585600
@rahultolearn59 , I think it pays to send a mail to users@cloudstack.apache.org about this. I have no experience with DRAC but I am sure there are plenty of users that do. Not sure if what you say is a matter of configuration and if others have a work around for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OOBM Sends the Reset Command to DRAC if Hypervisor is in PowerOff State [cloudstack]
Posted by "rahultolearn59 (via GitHub)" <gi...@apache.org>.
rahultolearn59 commented on issue #8918:
URL: https://github.com/apache/cloudstack/issues/8918#issuecomment-2085374263
Yes, @rohityadavcloud; as soon as we manually power on the host, VMs are moved to another KVM host.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] OOBM Sends the Reset Command to DRAC if Hypervisor is in PowerOff State [cloudstack]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8918:
URL: https://github.com/apache/cloudstack/issues/8918#issuecomment-5673403564
This issue is stale because it has been open for 120 days with no activity. It may be removed by administrators of this project at any time. Remove the stale label or comment to request for removal of it to prevent this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org