You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by marcaurele <gi...@git.apache.org> on 2016/08/16 06:13:38 UTC
[GitHub] cloudstack pull request #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stop...
GitHub user marcaurele opened a pull request:
https://github.com/apache/cloudstack/pull/1640
CLOUDSTACK-9458: Fix HA bug when VMs are stopped on agent disconnect
VM are being restarted even if they don't have HA enabled. If the agent
gets back online before the command expires some VMs are being stopped
and not even restarted.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/exoscale/cloudstack bug/CLOUDSTACK-9458
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/1640.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1640
----
commit c8f57541006536c4719044405b7691fad244fbf7
Author: Marc-Aur�le Brothier <m...@brothier.org>
Date: 2016-08-16T06:10:51Z
CLOUDSTACK-9458: Fix HA bug when VMs are stopped on agent disconnect
VM are being restarted even if they don't have HA enabled. If the agent
gets back online before the command expires some VMs are being stopped
and not even restarted.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@abhinandanprateek @koushik-das @marcaurele have you been able to come to agreement about the correct functionality for this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stop...
Posted by marcaurele <gi...@git.apache.org>.
Github user marcaurele commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1640#discussion_r75252604
--- Diff: server/src/com/cloud/ha/HighAvailabilityManagerImpl.java ---
@@ -497,7 +498,7 @@ protected Long restart(final HaWorkVO work) {
boolean fenced = false;
if (alive == null) {
- s_logger.debug("Fencing off VM that we don't know the state of");
+ s_logger.debug("Fencing off VM " + vm + " that we don't know the state of");
--- End diff --
Turning all of those log debug lines without a `if` statement to lambda expressions when we require Java8 would be easier if there're not wrapped within a `if`:
```
s_logger.debug(() -> "Fencing off VM " + vm + " that we don't know the state of");
```
with
```
default void debug(Supplier<String> stringSupplier) {
if (isDebugEnabled()) {
debug(stringSupplier.get());
}
}
```
The concatenation will be only done when debug is enabled.
You're right, I turn it to `warn`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by abhinandanprateek <gi...@git.apache.org>.
Github user abhinandanprateek commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@marcaurele @koushik-das When the MS thinks that the VM is down, it issues a stop command. This is done to clear up the resources on management server db tied up for that VM. Now it is seen several times that this actually kills a healthy VM. I have seen this issue in MS cluster with agent.lb turned on.
The issue is that we do need a state cleanup when a running VM is found to be stopped on the host. But this probably should not induce a shutdown on the host ? really, but again this is a tricky boundary condition.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stop...
Posted by marcaurele <gi...@git.apache.org>.
Github user marcaurele commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1640#discussion_r75091738
--- Diff: server/src/com/cloud/ha/HighAvailabilityManagerImpl.java ---
@@ -282,7 +282,9 @@ public void scheduleRestartForVmsOnHost(final HostVO host, boolean investigate)
+ hostId + " VM HA is done");
continue;
}
- scheduleRestart(vm, investigate);
+ if (vm.isHaEnabled()) {
--- End diff --
Sure, I'll update
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by marcaurele <gi...@git.apache.org>.
Github user marcaurele commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@koushik-das which is IMHO wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by blueorangutan <gi...@git.apache.org>.
Github user blueorangutan commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@blueorangutan package
LGTM, @abhinandanprateek can you review this as well?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by koushik-das <gi...@git.apache.org>.
Github user koushik-das commented on the issue:
https://github.com/apache/cloudstack/pull/1640
I had already mentioned in a previous comment that there is no need for this PR in 4.9/master. So that means a -1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stop...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1640#discussion_r75758946
--- Diff: server/src/com/cloud/ha/HighAvailabilityManagerImpl.java ---
@@ -497,7 +498,7 @@ protected Long restart(final HaWorkVO work) {
boolean fenced = false;
if (alive == null) {
- s_logger.debug("Fencing off VM that we don't know the state of");
+ s_logger.debug("Fencing off VM " + vm + " that we don't know the state of");
--- End diff --
In my view, the ``Supplier`` approach is just as complex, but in another way. In the long term, we will be adopting SLF4J which supports string templates -- allowing constructs as such as:
```
LOG.debug("I am having a problem VM %s", vm.getId());
```
This approach defers string concatenation in a more straight forward manner.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by abhinandanprateek <gi...@git.apache.org>.
Github user abhinandanprateek commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@jburwell @koushik-das @marcaurele When MS is unable to determine the state of the VM, Or it thinks VM requires a HA operation then it issues a stop command as part of fence operation.
The affect of this is to clean up the resources on the MS and keep the resource book keeping on MS in tack. This has the potential to kill a healthy VM in some boundary cases. We need to fix these boundary cases.
In case this cleanup/fence operation does not happen on MS then the resource allocation on MS will be not in sync with the actual capacity causes further complications and issues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by koushik-das <gi...@git.apache.org>.
Github user koushik-das commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@abhinandanprateek In latest master the sequence of event described above only happens when the host has been determined as 'Down'. Refer to the below code. So the bug described won't happen. Earlier even when host state was 'Alert' the same sequence used to get triggered which possibly killed healthy VMs.
> if (host != null && host.getStatus() == Status.Down) {
> _haMgr.scheduleRestartForVmsOnHost(host, true);
> }
In case there is still a possibility of healthy VMs getting killed then the scenario needs to be clearly identified. If we need to fix anything, the first thing would be look at improving the VM investigators rather than changing the existing fencing logic.
If we go ahead with the above fix then I can think of the following scenario that is broken. In case of a genuine host down scenario non-HA VMs continue to remain in 'Running' state and no operations can be done on it. Currently non-HA VMs are marked as 'Stopped' after fencing is successful and they can be manually started on another host.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by koushik-das <gi...@git.apache.org>.
Github user koushik-das commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@marcaurele Based on the initial few lines of the logs the agent went to Alert state.
srv02 2016-08-08 11:56:03,895 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-16:ctx-8b5b6956) The next status of agent 44692is Alert, current status is Up
srv02 2016-08-08 11:56:03,896 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-16:ctx-8b5b6956) Deregistering link for 44692 with state Alert
As per the latest ACS code (4.9/master) restart of VMs on a host are scheduled only if the state of host is determined as Down. In case of Alert nothing is done.
On what version of CS are you seeing this issue?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by marcaurele <gi...@git.apache.org>.
Github user marcaurele commented on the issue:
https://github.com/apache/cloudstack/pull/1640
I understand your point of the release, but we're not in an ideal world where everyone runs the latest version. I try to do my best to look at the current code in CS to find possible fixes of any bug/problem we encounter or changes we want to do in our version. I want us to get back to the master version but that's not the topic here, neither going to happen in the next weeks.
The point 2 does not make sense to me. If the management server cannot determine the state of the VM, it could mark them as stopped (*even though I don't think it should*). But it should not create a StopVM job, because that might trigger a proper stop of the VM if the agent is reconnecting while the job is picked by async job workers.
If the VM is really down because the host has crashed, then the command is pointless, and in a customer point of view it would not make a difference. If the host is still up and fine, but we have a network glitch, then requesting a stop of the VM is really bad in a customer point of view. By not doing anything, not requesting a stop, we would end up in a better situation.
Going back to which state should be set on the VM when the management server cannot determine it, taking the assumption that the VM is stopped because the management server cannot reach the agent is as much incorrect as leaving it as it is (running, migrating, creating...). I'd rather create a new state `UNKNOWN` for such special case, when the management server does really not know. In a management point of view it will be also easier to know there are *ghost* VMs somewhere for which the management server cannot determine the exact state and proper investigation (*manual*) should be done if the state stays like this, regarding the billing part too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on the issue:
https://github.com/apache/cloudstack/pull/1640
Are there Marvin test cases to verify this behavior?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by marcaurele <gi...@git.apache.org>.
Github user marcaurele commented on the issue:
https://github.com/apache/cloudstack/pull/1640
To get back to your previous comment @koushik-das on the broken scenario: what happen if the host is not reachable and the VMs are using a remote storage. With the fencing operation marking the VM as stopped, does it mean that the same remote disk volume is used if the VM is spawned on another host (while the other one still running on the first host)?
@abhinandanprateek if the reason to fence off the VM is to clean up resources, IMO this should be the job of the VM sync, on the ping command/startup command. In case a host is lost, the capacity of the cluster should reflect the lose of that host and the stat capacity should calculate its value based on the hosts that are Up only. When a host comes back (possibly with some VMs still running), the startup command should sync the VM states and the capacity of the cluster/zone should be updated.
In short, cleaning up resources that are not "reachable" anymore should not be needed and should not be taken into account when calculating the actual capacity of the cluster/zone.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by marcaurele <gi...@git.apache.org>.
Github user marcaurele commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@koushik-das We are running a fork based on 4.4.2 with lots of cherry-pickings.
But even if the host is down, why would you want to schedule a restart if the VM are not HA enabled?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stop...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1640#discussion_r75089504
--- Diff: server/src/com/cloud/ha/HighAvailabilityManagerImpl.java ---
@@ -282,7 +282,9 @@ public void scheduleRestartForVmsOnHost(final HostVO host, boolean investigate)
+ hostId + " VM HA is done");
continue;
}
- scheduleRestart(vm, investigate);
+ if (vm.isHaEnabled()) {
--- End diff --
Why not skip the loop with a `continue` around line 268 when vm is not HA enabled?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@marcaurele Would you mind adding these notes to the bug ticket? It seems like valuable information that people searching JIRA for issues would find very useful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@marcaurele per @koushik-das, what is the issue with re-pointing this PR to the 4.9 release branch? When the PR is merged, it will be forward merged to master. Therefore, the concern you expressed about the change getting to master is not an issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by abhinandanprateek <gi...@git.apache.org>.
Github user abhinandanprateek commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@marcaurele for a host that is found to be down we go ahead and schedule a restart for HA enabled VM, this is good.
For the VMs that are not HA enabled they will continue to show as running. This works in the scenario where the host finally comes around. What if host is gone for long or forever, then the VMs will continue to show as running. The user will have to guess that he has to stop and then start the VM. Can you check if VMs will be eventually marked down by VM sync ? If that is the case, I think this fix should be good then
Another suggestion: In the specific case where host drops and then come back in certain interval. Can we make the parameter that times out to mark a host down as configurable. In your case you can increase it to several hours and it will not start HA during that time and host can still connect back ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by marcaurele <gi...@git.apache.org>.
Github user marcaurele commented on the issue:
https://github.com/apache/cloudstack/pull/1640
I'll do the long explanation regarding the bug in production we had.
Twice we had the VPN going down between 2 of our zones which resulted in the lost of the communication between the management server and all the agents of the other zone. After a few minutes the network went back and as the agents were reconnecting we started to see VMs being shutdown, even though we don't have HA enabled. We have 2 management servers in front of haproxy to balance agents & requests. When the network went back, the agent reconnected to the other management server 01. The server 02 had already started to issue some commands to shutdown the VM.
The important lines are here, I kept the line showing one VM being shutdown `111750`:
```
srv02 2016-08-08 11:56:03,854 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-16:ctx-8b5b6956) PingInvestigator was able to determine host 44692 is in Disconnected
srv02 2016-08-08 11:56:03,855 WARN [agent.manager.AgentManagerImpl] (AgentTaskPool-16:ctx-8b5b6956) Agent is disconnected but the host is still up: 44692-virt-hv041
srv02 2016-08-08 11:56:03,857 WARN [apache.cloudstack.alerts] (AgentTaskPool-16:ctx-8b5b6956) alertType:: 7 // dataCenterId:: 2 // podId:: 2 // clusterId:: null // message:: Host disconnected, name: virt-hv041 (id:44692), availability zone: ch-dk-2, pod: DK2-AZ1-POD01
srv02 2016-08-08 11:56:03,884 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-16:ctx-8b5b6956) Host 44692 is disconnecting with event AgentDisconnected
srv02 2016-08-08 11:56:03,895 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-16:ctx-8b5b6956) The next status of agent 44692is Alert, current status is Up
srv02 2016-08-08 11:56:03,896 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-16:ctx-8b5b6956) Deregistering link for 44692 with state Alert
srv02 2016-08-08 11:56:03,896 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-16:ctx-8b5b6956) Remove Agent : 44692
srv02 2016-08-08 11:56:03,897 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-16:ctx-8b5b6956) Seq 44692-4932286016900831481: Sending disconnect to class com.cloud.network.security.SecurityGroupListener
srv02 2016-08-08 11:56:03,897 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-16:ctx-8b5b6956) Seq 44692-4932286016900834802: Sending disconnect to class com.cloud.network.security.SecurityGroupListener
srv02 2016-08-08 11:56:03,897 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-16:ctx-8b5b6956) Seq 44692-4932286016900886805: Sending disconnect to class com.cloud.network.security.SecurityGroupListener
...
srv02 2016-08-08 11:56:03,979 DEBUG [cloud.network.NetworkUsageManagerImpl] (AgentTaskPool-16:ctx-8b5b6956) Disconnected called on 44692 with status Alert
srv02 2016-08-08 11:56:03,987 DEBUG [cloud.host.Status] (AgentTaskPool-16:ctx-8b5b6956) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 44692, name = virt-hv041]
srv02 2016-08-08 11:56:03,998 DEBUG [cloud.cluster.ClusterManagerImpl] (AgentTaskPool-16:ctx-8b5b6956) Forwarding [{"com.cloud.agent.api.ChangeAgentCommand":{"agentId":44692,"event":"AgentDisconnected","contextMap":{},"wait":0}}] to 345049010805
srv02 2016-08-08 11:56:03,998 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Worker-2:ctx-e087e71f) Cluster PDU 90520739220960 -> 345049010805. agent: 44692, pdu seq: 15, pdu ack seq: 0, json: [{"com.cloud.agent.api.ChangeAgentCommand":{"agentId":44692,"event":"AgentDisconnected","contextMap":{},"wait":0}}]
srv01 2016-08-08 11:56:04,002 DEBUG [agent.manager.ClusteredAgentManagerImpl] (Cluster-Worker-9:ctx-9101a6d4) Dispatch ->44692, json: [{"com.cloud.agent.api.ChangeAgentCommand":{"agentId":44692,"event":"AgentDisconnected","contextMap":{},"wait":0}}]
srv01 2016-08-08 11:56:04,002 DEBUG [agent.manager.ClusteredAgentManagerImpl] (Cluster-Worker-9:ctx-9101a6d4) Intercepting command for agent change: agent 44692 event: AgentDisconnected
srv01 2016-08-08 11:56:04,002 DEBUG [agent.manager.ClusteredAgentManagerImpl] (Cluster-Worker-9:ctx-9101a6d4) Received agent disconnect event for host 44692
srv02 2016-08-08 11:56:04,002 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Worker-2:ctx-e087e71f) Cluster PDU 90520739220960 -> 345049010805 completed. time: 3ms. agent: 44692, pdu seq: 15, pdu ack seq: 0, json: [{"com.cloud.agent.api.ChangeAgentCommand":{"agentId":44692,"event":"AgentDisconnected","contextMap":{},"wait":0}}]
srv01 2016-08-08 11:56:04,004 DEBUG [agent.manager.ClusteredAgentManagerImpl] (Cluster-Worker-9:ctx-9101a6d4) Not processing AgentDisconnected event for the host id=44692 as the host is directly connected to the current management server 345049010805
srv02 2016-08-08 11:56:04,004 WARN [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-16:ctx-8b5b6956) Scheduling restart for VMs on host 44692-virt-hv041
...
srv02 2016-08-08 11:56:04,286 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-16:ctx-8b5b6956) Notifying HA Mgr of to restart vm 111750-i-8114-111750-VM
srv02 2016-08-08 11:56:04,296 INFO [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-16:ctx-8b5b6956) Schedule vm for HA: VM[User|i-8114-111750-VM]
...
srv01 2016-08-08 11:56:17,394 DEBUG [agent.manager.AgentManagerImpl] (StatsCollector-4:ctx-dcd63254) Can not send command com.cloud.agent.api.GetVmStatsCommand due to Host 44692 is not up
srv02 2016-08-08 11:56:17,395 DEBUG [agent.manager.AgentManagerImpl] (StatsCollector-3:ctx-8cb3d7fb) Can not send command com.cloud.agent.api.GetVmStatsCommand due to Host 44692 is not up
srv01 2016-08-08 11:56:37,825 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-13:null) Ping from 44692
srv01 2016-08-08 11:56:37,826 DEBUG [cloud.vm.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:null) Process host VM state report from ping process. host: 44692
srv01 2016-08-08 11:56:37,886 DEBUG [cloud.vm.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:null) Process VM state report. host: 44692, number of records in report: 26
srv01 2016-08-08 11:56:37,891 DEBUG [cloud.vm.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:null) VM state report is updated. host: 44692, vm id: 81688, power state: PowerOn
...
srv01 2016-08-08 11:57:38,581 DEBUG [cloud.vm.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:null) VM state report is updated. host: 44692, vm id: 111750, power state: PowerOn
...
srv01 2016-08-08 12:00:50,544 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:ctx-f9c5c881 work-858) Processing HAWork[858-HA-111750-Running-Investigating]
srv01 2016-08-08 12:00:50,555 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:ctx-f9c5c881 work-858) HA on VM[User|i-8114-111750-VM]
srv01 2016-08-08 12:00:50,565 DEBUG [agent.transport.Request] (HA-Worker-2:ctx-f9c5c881 work-858) Seq 44692-6380193297100439580: Sending { Cmd , MgmtId: 345049010805, via: 44692(virt-hv041), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckVirtualMachineCommand":{"vmName":"i-8114-111750-VM","wait":20}}] }
...
srv01 2016-08-08 12:00:50,658 DEBUG [agent.transport.Request] (HA-Worker-2:ctx-f9c5c881 work-858) Seq 44692-6380193297100439580: Received: { Ans: , MgmtId: 345049010805, via: 44692, Ver: v1, Flags: 10, { CheckVirtualMachineAnswer } }
srv01 2016-08-08 12:00:50,658 DEBUG [cloud.ha.CheckOnAgentInvestigator] (HA-Worker-2:ctx-f9c5c881 work-858) Agent responded with state Running
srv01 2016-08-08 12:00:50,658 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:ctx-f9c5c881 work-858) Rescheduling because the host is not up but the vm is alive
srv01 2016-08-08 12:00:50,658 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:ctx-f9c5c881 work-858) VM i-8114-111750-VM is found to be alive by SimpleInvestigator
srv01 2016-08-08 12:00:50,658 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:ctx-f9c5c881 work-858) Rescheduling HAWork[858-HA-111750-Running-Investigating] to try again at Mon Aug 08 12:01:51 CEST 2016
srv01 2016-08-08 12:00:50,658 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:ctx-f9c5c881 work-858) SimpleInvestigator found VM[User|i-8114-111750-VM]to be alive? true
...
srv01 2016-08-08 12:02:35,374 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:ctx-ee61de22 work-858) Processing HAWork[858-HA-111750-Running-Investigating]
srv01 2016-08-08 12:02:35,381 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:ctx-ee61de22 work-858) HA on VM[User|i-8114-111750-VM]
...
srv01 2016-08-08 12:02:35,478 DEBUG [cloud.ha.CheckOnAgentInvestigator] (HA-Worker-4:ctx-ee61de22 work-858) Agent responded with state Running
srv01 2016-08-08 12:02:35,478 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:ctx-ee61de22 work-858) Rescheduling because the host is not up but the vm is alive
srv01 2016-08-08 12:02:35,478 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:ctx-ee61de22 work-858) VM i-8114-111750-VM is found to be alive by SimpleInvestigator
srv01 2016-08-08 12:02:35,478 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:ctx-ee61de22 work-858) Rescheduling HAWork[858-HA-111750-Running-Investigating] to try again at Mon Aug 08 12:03:36 CEST 2016
srv01 2016-08-08 12:02:35,478 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:ctx-ee61de22 work-858) SimpleInvestigator found VM[User|i-8114-111750-VM]to be alive? true
...
srv02 2016-08-08 12:03:46,230 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:ctx-11e5dfc9 work-858) Processing HAWork[858-HA-111750-Running-Investigating]
srv02 2016-08-08 12:03:46,237 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:ctx-11e5dfc9 work-858) HA on VM[User|i-8114-111750-VM]
srv02 2016-08-08 12:03:46,246 DEBUG [cloud.ha.CheckOnAgentInvestigator] (HA-Worker-2:ctx-11e5dfc9 work-858) Unable to reach the agent for VM[User|i-8114-111750-VM]: Resource [Host:44692] is unreachable: Host 44692: Host with specified id is not in the right state: Alert
srv02 2016-08-08 12:03:46,246 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-2:ctx-11e5dfc9 work-858) testing if VM[User|i-8114-111750-VM] is alive
srv02 2016-08-08 12:03:46,246 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:ctx-11e5dfc9 work-858) SimpleInvestigator found VM[User|i-8114-111750-VM]to be alive? null
srv02 2016-08-08 12:03:46,246 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:ctx-11e5dfc9 work-858) XenServerInvestigator found VM[User|i-8114-111750-VM]to be alive? null
srv02 2016-08-08 12:03:46,259 DEBUG [agent.manager.AgentManagerImpl] (HA-Worker-2:ctx-11e5dfc9 work-858) Host with id null doesn't exist
srv02 2016-08-08 12:03:46,259 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-2:ctx-11e5dfc9 work-858) Not a System Vm, unable to determine state of VM[User|i-8114-111750-VM] returning null
srv02 2016-08-08 12:03:46,259 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-2:ctx-11e5dfc9 work-858) Testing if VM[User|i-8114-111750-VM] is alive
srv02 2016-08-08 12:03:46,259 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-2:ctx-11e5dfc9 work-858) Returning null since we're unable to determine state of VM[User|i-8114-111750-VM]
srv02 2016-08-08 12:03:46,259 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-2:ctx-11e5dfc9 work-858) VM[User|i-8114-111750-VM] could not be pinged, returning that it is unknown
srv02 2016-08-08 12:03:46,259 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:ctx-11e5dfc9 work-858) PingInvestigator found VM[User|i-8114-111750-VM]to be alive? null
srv01 2016-08-08 12:03:46,261 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:ctx-e1a90562 work-859) HA on VM[User|i-10355-116755-VM]
srv02 2016-08-08 12:03:46,264 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-2:ctx-11e5dfc9 work-858) Unable to find a management nic, cannot ping this system VM, unable to determine state of VM[User|i-8114-111750-VM] returning null
srv02 2016-08-08 12:03:46,265 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:ctx-11e5dfc9 work-858) ManagementIPSysVMInvestigator found VM[User|i-8114-111750-VM]to be alive? null
srv01 2016-08-08 12:03:46,266 DEBUG [agent.transport.Request] (HA-Worker-3:ctx-e1a90562 work-859) Seq 44680-8591460714139353168: Sending { Cmd , MgmtId: 345049010805, via: 44680(virt-hv029), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckVirtualMachineCommand":{"vmName":"i-10355-116755-VM","wait":20}}] }
srv02 2016-08-08 12:03:46,272 DEBUG [agent.transport.Request] (HA-Worker-2:ctx-11e5dfc9 work-858) Seq 44672-2138365398070947852: Sending { Cmd , MgmtId: 90520739220960, via: 44672(virt-hv019), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"eea983e8-6156-4ddb-b5c5-e4c4efa9ff4b-LibvirtComputingResource","privateNetwork":{"ip":"....","netmask":"....","mac":"....","isSecurityGroupEnabled":false},"storageNetwork1":{"ip":"....","netmask":"....","mac":"....","isSecurityGroupEnabled":false}},"wait":20}}] }
srv02 2016-08-08 12:03:46,327 DEBUG [agent.transport.Request] (AgentManager-Handler-3:null) Seq 44672-2138365398070947852: Processing: { Ans: , MgmtId: 90520739220960, via: 44672, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
srv02 2016-08-08 12:03:46,328 DEBUG [agent.transport.Request] (HA-Worker-2:ctx-11e5dfc9 work-858) Seq 44672-2138365398070947852: Received: { Ans: , MgmtId: 90520739220960, via: 44672, Ver: v1, Flags: 10, { Answer } }
srv02 2016-08-08 12:03:46,328 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:ctx-11e5dfc9 work-858) Fencing off VM that we don't know the state of
srv02 2016-08-08 12:03:46,328 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:ctx-11e5dfc9 work-858) KVMInvestigator found VM[User|i-8114-111750-VM]to be alive? null
srv01 2016-08-08 12:03:46,333 DEBUG [cloud.capacity.CapacityManagerImpl] (CapacityChecker:ctx-bc9b35f1) No need to calibrate cpu capacity, host:44687 usedCpu: 0 reservedCpu: 83524
srv01 2016-08-08 12:03:46,333 DEBUG [cloud.capacity.CapacityManagerImpl] (CapacityChecker:ctx-bc9b35f1) No need to calibrate memory capacity, host:44687 usedMem: 0 reservedMem: 44023414784
srv02 2016-08-08 12:03:46,333 DEBUG [agent.transport.Request] (HA-Worker-2:ctx-11e5dfc9 work-858) Seq 44672-2138365398070947853: Sending { Cmd , MgmtId: 90520739220960, via: 44672(virt-hv019), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.FenceCommand":{"vmName":"i-8114-111750-VM","hostGuid":"eea983e8-6156-4ddb-b5c5-e4c4efa9ff4b-LibvirtComputingResource","hostIp":"....","inSeq":false,"wait":0}}] }
```
The way I reproduced it in our lab is my dropping the traffic from one hypervisor to/from the management servers:
```
sudo iptables -I OUTPUT -d 10.16.13.83 -j DROP
sudo iptables -I INPUT -s 10.16.13.83 -j DROP
```
When you see the host changing its state to Alert, restart the agent so that it will send quickly a `startupCommand` and re-enable the traffic if you want to see the VMs being shutdown.
```
sudo service cloudstack-agent restart
sudo iptables -D OUTPUT -d 10.16.13.83 -j DROP
sudo iptables -D INPUT -s 10.16.13.83 -j DROP
```
During that time window (right after the host/agent turned in Alert) you'll start to see AsyncJobs coming in the database to stop VMs, which is the thing I'm requesting a change for. If the agent is disconnected for enough time the commands will vanished since the host cannot be reached. But if the timing is just *bad*, the agent is back online while the jobs are being created, and it will process some of those commands. Playing with the `ping.timeout` value would only move the window of the bug in another direction.
It's not too easy to make it happen but feasible.
The problem could be handle differently, but I found this way a good solution without adding too much complexity in the HA worker tasks, like checking between investigation the state of the host. IMO if the VM are not HA, then nothing should be done: **the fencing should not be started** and let the powerState report change the VMs state when it could reach back the agent on the host. Currently the fencing is being done(and the restart is skipped since a check for HA is done before).
We are running a custom version based on 4.4.2 with many merges, but as I could see the bug still present on master, I pushed this PR.
Hope this helps you better understand the problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by marcaurele <gi...@git.apache.org>.
Github user marcaurele commented on the issue:
https://github.com/apache/cloudstack/pull/1640
> Would you mind adding these notes to the bug ticket?
@jburwell All PR comments are going automatically into the jira ticket comment thanks to the ID matching (I think).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stop...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1640#discussion_r75147713
--- Diff: server/src/com/cloud/ha/HighAvailabilityManagerImpl.java ---
@@ -497,7 +498,7 @@ protected Long restart(final HaWorkVO work) {
boolean fenced = false;
if (alive == null) {
- s_logger.debug("Fencing off VM that we don't know the state of");
+ s_logger.debug("Fencing off VM " + vm + " that we don't know the state of");
--- End diff --
Please wrap in a isDebugEnabled check to avoid expensive string concatenation when DEBUG is off. Also, should this message be a WARN instead of a DEBUG?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by marcaurele <gi...@git.apache.org>.
Github user marcaurele commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@koushik-das
> If the MS is not able to determine the state of the VM, it tries fencing off the VM (using the various fencers available). If VM cannot be fenced off successfully, the state of the VM is left unchanged.
Apparently I found a way where the VMs are successfully fenced off even though they should not.
What is the reason to try fencing off VMs when the MS is not able to determine its state? I cannot see a good reason so far but you seem to think there is at least one. Can you explain it?
@jburwell It does not cover my case exactly as it's a timing issue. I'll keep a note to find a way to create a scenario.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by koushik-das <gi...@git.apache.org>.
Github user koushik-das commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@marcaurele
> What is the reason to try fencing off VMs when the MS is not able to determine its state? I cannot see a good reason so far but you seem to think there is at least one. Can you explain it?
If MS is able to determine VM state as up or down why do you need fencing? Fencing is tried only when the state cannot be determined.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by koushik-das <gi...@git.apache.org>.
Github user koushik-das commented on the issue:
https://github.com/apache/cloudstack/pull/1640
Also since the automated test coverage of this area is less, changes should be made after taking into account all possible scenarios. Otherwise there might be regressions in some valid scenarios.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by koushik-das <gi...@git.apache.org>.
Github user koushik-das commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@marcaurele Can you share the MS logs for this issue? We need to understand the exact cause for restart of VM? When an agent/host is detected as 'Down', CS tries to check if VMs on it are alive or not, if found alive nothing is done on the VM.
Also if you think that the host got disconnected intermittently, then there are ways to adjust the timeout in CS after which it will start investigating the host status. Try adjusting the ping.timeout configuration parameter to see if the issue is resolved.
The investigation to check if VM is alive or not is done for all VMs irrespective of HA enabled or not. If a host is really down then it makes sense to mark the VMs as stopped. Additionally for HA enabled VMs, after they are successfully fenced off, attempt is made to restart them on other hosts in the cluster.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by koushik-das <gi...@git.apache.org>.
Github user koushik-das commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@jburwell The issue that has been reported is on a custom branch, probably @marcaurele needs to cherry-pick some additional commits from ACS. Master/4.9 doesn't have this issue, so in that way the PR is not needed.
@marcaurele Please read my last comment again and go through the restart() method logic in HA manager code.
>>> If the management server cannot determine the state of the VM, it could mark them as stopped (even though I don't think it should). But it should not create a StopVM job, because that might trigger a proper stop of the VM if the agent is reconnecting while the job is picked by async job workers.
The above is not correct. If the MS is not able to determine the state of the VM, it tries fencing off the VM (using the various fencers available). If VM cannot be fenced off successfully, the state of the VM is left unchanged. Also if any of the investigators is able to determine the VM state as Down then only the VM is marked as stopped. Hope that clarifies things.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by koushik-das <gi...@git.apache.org>.
Github user koushik-das commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@marcaurele Thats correct. In case of shared/remote storage the same disk is used to spawn a VM on another host once the VM is successfully fenced. If the fencer has successfully fenced off a VM, it is assumed that the original VM is correctly stopped. Now if you are saying that the original VM continues to run then that means that the specific fencer has bugs and needs fixing. Note that there are different types of fencer available in cloudstack based on hypervisor types.
@abhinandanprateek In the scenario you mentioned vmsync won't be able to mark VM as stopped as the ping command is no longer running as the host is in alert/down state.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by marcaurele <gi...@git.apache.org>.
Github user marcaurele commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@jburwell I changed the commit and PR to point to 4.9
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@marcaurele @koushik-das @jburwell @abhinandanprateek can we have a conclusion on this PR, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by jburwell <gi...@git.apache.org>.
Github user jburwell commented on the issue:
https://github.com/apache/cloudstack/pull/1640
@marcaurele I agree that an ``UNKNOWN`` state is proper way to handle a network partition. As an example, out-of-band management uses this approach which the management server loses connectivity with the IPMI interface.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by blueorangutan <gi...@git.apache.org>.
Github user blueorangutan commented on the issue:
https://github.com/apache/cloudstack/pull/1640
Packaging result: \u2714centos6 \u2714centos7 \u2714debian repo: http://packages.shapeblue.com/cloudstack/pr/1640
Job ID-89
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1640: CLOUDSTACK-9458: Fix HA bug when VMs are stopped on ...
Posted by koushik-das <gi...@git.apache.org>.
Github user koushik-das commented on the issue:
https://github.com/apache/cloudstack/pull/1640
Please use a proper ACS release for reporting bugs. In your case you may have to do some additional cherry-picks.
"Schedule restart" does multiple tasks. There is a method by this name in code, the name may not be the most appropriate but it does the following. So don't get confused with the name
1. Tries to find out if the VM is alive or not
2. If it is not able to determine conclusively if VM is alive, then it tries to fence off VM
3. After successful fencing, HA enabled VMs are restarted on another host, non-HA VMs are marked as Stopped
So as you see non-HA VMs are simply stopped when the host is determined as down and not restarted. It makes sense to mark them as stopped so that subsequent operations can be performed on the VMs, for e.g. selective VMs may be explicitly started on another host. If a host is down then power sync won't happen for that host and VM states on that host won't get updated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---