You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by DaanHoogland <gi...@git.apache.org> on 2015/09/15 12:50:48 UTC
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
GitHub user DaanHoogland opened a pull request:
https://github.com/apache/cloudstack/pull/829
CLOUDSTACK-8848: extra state to handle; no power report from host
there no no real handling code now so this can be regarded as a DISCUSS item. It may well be an improvement over existing code as is, however.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/DaanHoogland/cloudstack CLOUDSTACK-8848
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/829.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #829
----
commit c4d6f9b7a87d8549bcc7efc2c8e529636d0a1d86
Author: Daan Hoogland <da...@onecht.net>
Date: 2015-09-15T10:48:56Z
CLOUDSTACK-8848: extra state to handle; no power report from host
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by resmo <gi...@git.apache.org>.
Github user resmo commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-142912573
Cloud you follow my explanations? If this would help, I must set the ping interval to a year or somewhat identical. Are you REALLY followed my explanations? What did I miss?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-144949845
This change will have to go in a more solid and complete refactor, probably in smaller steps. The fix it is intended for will go in #885.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-141482058
@koushik-das @anshul1886 the grace period is not going to work. updat_time is not getting updated. We really need to fix the statemachine
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by remibergsma <gi...@git.apache.org>.
Github user remibergsma commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-140983520
@resmo You're right and I wasn't clear enough I guess. Because I think the 'PowerReportMissing' comes from the VM on the hypervisor that you are migrating _to_, as in the beginning it is in a non-running state (probably 'paused', 'stopped' or similar). This happens in order to get all memory over and promote it to 'running' later on, in which case we have one VM left again.
The confusing part is that two hypervisors report about the same VM during live migrate of which one is running and one non-running.
Just wanted to share.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by resmo <gi...@git.apache.org>.
Github user resmo commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-140879864
@remibergsma does not look related to me, the keyword seems to be PowerReportMissing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-141421211
@koushik-das about the bug probably not being in 4.6 i agree.
@resmo I understood that the router was rebooted while you set the reboot on out of band flag to false. Is that correct?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-140791892
@koushik-das could you comment, please. Regarding power off and no power state equal in some case might make sense but in general it certainly doesn't as we have seen an occasion were a missing report on a running VM was delivered (see René's logs in CLOUDSTACK-8848). As per Anshul's comments most or at least a lot of the times the handlers should act the same. This is considered a blocker for 4.6 so help is appreciated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by resmo <gi...@git.apache.org>.
Github user resmo commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-142882541
@anshul1886 @koushik-das
@DaanHoogland and I had a debug session last friday, and since he is off for the next couple of days I can give you more details about we analysed.
The powerReportMissing is not the problem, it is only the trigger. The graceful period is the problem. The calculation of this period is relaying (see https://github.com/apache/cloudstack/blob/4.5.2/engine/orchestration/src/com/cloud/vm/VirtualMachinePowerStateSyncImpl.java#L114) on the field `update_time` in table `vm_instance`. But if I look at the value it seems it doesn't get updated. So the grace period has most likely always passed.
I tried to do a workaround doing the following, I ran an update sql for every 5 seconds which updated the `update_time` for my router r-342 which I was migrating around esx cluster nodes:
~~~
mysql -e 'update cloud.vm_instance set update_time=NOW() where id=342;'
~~~
And the router didn't get rebooted:
~~~
2015-09-24 11:47:07,685 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (DirectAgentCronJob-218:ctx-5849bd19) VM state report. host: 25, vm id: 342, power state: PowerOn
2015-09-24 11:47:07,696 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (DirectAgentCronJob-218:ctx-5849bd19) VM state report is updated. host: 25, vm id: 342, power state: PowerOn
2015-09-24 11:48:06,462 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (DirectAgentCronJob-55:ctx-84cd4323) VM state report. host: 19, vm id: 342, power state: PowerOn
2015-09-24 11:48:06,471 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (DirectAgentCronJob-55:ctx-84cd4323) VM state report is updated. host: 19, vm id: 342, power state: PowerOn
2015-09-24 11:48:06,493 WARN [o.a.c.alerts] (DirectAgentCronJob-55:ctx-84cd4323) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Router has been migrated out of band: r-342-VM
2015-09-24 11:49:06,539 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (DirectAgentCronJob-29:ctx-2a57d676) Detected missing VM. host: 19, vm id: 342, power state: PowerReportMissing, last state update: 1443095344000
2015-09-24 11:49:06,539 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (DirectAgentCronJob-29:ctx-2a57d676) vm id: 342 - time since last state update(-7197461ms) has not passed graceful period yet
2015-09-24 11:49:07,719 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (DirectAgentCronJob-444:ctx-fdd4c055) VM state report. host: 20, vm id: 342, power state: PowerOn
~~~
Which means this patch is not fix the root cause. To me the root cause is that `update_time` is not updated or the gracePeriod calculation is wrong.
Any thoughts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by resmo <gi...@git.apache.org>.
Github user resmo commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-142940617
So I made a new PR #885 as a replacement for new discussion how this issue can be solved.
IMHO this PR can be closed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by anshul1886 <gi...@git.apache.org>.
Github user anshul1886 commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-142920071
I have followed your explanation but what I am trying to say is that it may get updated due to side effect of some other code as from my experience this was working as expected.
If you want then we can safely change it to
Date vmStateUpdateTime = instance.getPowerStateUpdateTime();
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by resmo <gi...@git.apache.org>.
Github user resmo commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-142901959
Increasing the grace period for a few minutes won't help. If you look at these update_time values I queried from 10 running routers, you see these are days, weeks behind.
~~~
mysql> select instance_name, type, update_time from vm_instance where instance_name like 'r-%' and removed is NULL and state = 'Running' limit 10\G;
*************************** 1. row ***************************
instance_name: r-2978-VM
type: DomainRouter
update_time: 2015-08-24 21:54:42
*************************** 2. row ***************************
instance_name: r-3650-VM
type: DomainRouter
update_time: 2015-08-24 22:23:53
*************************** 3. row ***************************
instance_name: r-3651-VM
type: DomainRouter
update_time: 2015-08-26 00:57:40
*************************** 4. row ***************************
instance_name: r-3652-VM
type: DomainRouter
update_time: 2015-09-17 06:44:14
*************************** 5. row ***************************
instance_name: r-3653-VM
type: DomainRouter
update_time: 2015-08-24 22:18:43
*************************** 6. row ***************************
instance_name: r-3654-VM
type: DomainRouter
update_time: 2015-09-17 06:44:11
*************************** 7. row ***************************
instance_name: r-3655-VM
type: DomainRouter
update_time: 2015-09-15 05:34:30
*************************** 8. row ***************************
instance_name: r-3656-VM
type: DomainRouter
update_time: 2015-08-24 21:53:57
*************************** 9. row ***************************
instance_name: r-3657-VM
type: DomainRouter
update_time: 2015-08-24 22:33:40
*************************** 10. row ***************************
instance_name: r-3659-VM
type: DomainRouter
update_time: 2015-08-26 00:33:54
10 rows in set (0.01 sec)
~~~
But not updating the powerState (which would also update the update_date I assume) results in the gracePeriod uses a update_time weeks, months back.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by resmo <gi...@git.apache.org>.
Github user resmo commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-141407138
@koushik-das @DaanHoogland unless we have a "better" solution I would like to go with @anshul1886 suggested fix. It seems the best way not breaking anything else.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by anshul1886 <gi...@git.apache.org>.
Github user anshul1886 commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-142903425
This simply means that there power_state has not changed for that period of time. As I pointed out in my previous comment that if state is consecutively updated with same power state thrice then we don't update state. If its state gets changed then only we update it. So I am assuming for these routers power_state_update_count is 3.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by koushik-das <gi...@git.apache.org>.
Github user koushik-das commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-141418450
@DaanHoogland As per logs attached in bug VR reboot happened due to out-of-band movement from one host to another and not due to power report missing.
Host 19 reports VM is missing.
2015-09-15 09:37:06,508 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (DirectAgentCronJob-253:ctx-c4f59216) Detected missing VM. host: 19, vm id: 342, power state: PowerReportMissing, last state update: 1442302506000
Host 20 reports that VM is running. Now since VM moved out-of-band from host 19 to 20, VR got rebooted as per the previous code.
2015-09-15 09:37:07,809 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (DirectAgentCronJob-254:ctx-757ec6a9) VM state report. host: 20, vm id: 342, power state: PowerOn
2015-09-15 09:37:07,845 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (DirectAgentCronJob-254:ctx-757ec6a9) Schedule a router reboot task as router 342 is powered-on out-of-band. we need to reboot to refresh network rules
There was a commit to remove the VR reboot. So now in this scenario VR should continue to run. So the bug should no longer be there.
@resmo Before we talk about any fix, can you verify if the issue is still happening on latest master?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by anshul1886 <gi...@git.apache.org>.
Github user anshul1886 commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-142909650
Yes, increasing time should help.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by remibergsma <gi...@git.apache.org>.
Github user remibergsma commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-140867296
@DaanHoogland @resmo Not sure if this is related, but in 4.4 (and running XenServer/KVM probably any hypervisor) I see sometimes alerts when a router is live-migrated even from CloudStack. This happens because during live migrate two instances of the VM co-exist (one read/only the other the running) due to the nature of how live migrate works. Here CloudStack gets confused as two hypervisors report the power state now. When the power report is during the live migrate you see this alert, otherwise you will not see it. Could be the same issue.
Is this also relevant to 4.6? We kicked the code out recently (@koushik-das helped me out).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by koushik-das <gi...@git.apache.org>.
Github user koushik-das commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-141377920
-1 to this PR as I am not able to follow how the change is related to the bug. Also this is going to impact any HV having VM changes outside of CS like Vmware and Hyper-V.
CS updates the power state of a VM to "missing" when HV doesn't report that VM for 2 consecutive ping intervals (ping.interval). Now if due to any operation in the HV the VM is not available for more than 2 ping interval then as per current behaviour CS tries to do the same thing as "power off" state.
The bug reported is that VR is getting rebooted. Earlier there was a logic to reboot VR if it has moved out-of-band from one host to another. That code is no longer there. So it needs to be seen if the bug is still happening.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by bhaisaab <gi...@git.apache.org>.
Github user bhaisaab commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-140641376
@DaanHoogland this is for just master, or also applies for 4.5; if so please send a PR for 4.5 as well. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-141050957
If @remibergsma is right then the design is really broken. two events come in during migration:
# PowerOn (or Off or what ever is actual)
# PowerReportMissing (interpreted as whatever, in our present case PowerOff)
The second one should definitely not be interpreted as some other event but merits its treatment (handler code) of its own. It should investigate if a migration is busy and ignore if it is so. In this case it should have been ignored. So the question, @anshul1886, @koushik-das is what other possible states yielding a PowerReportMissing event are.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by anshul1886 <gi...@git.apache.org>.
Github user anshul1886 commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-142899514
From code it seems to be getting updated and DB also suggests that. It will not be updated if there is no power change for MAX_CONSECUTIVE_SAME_STATE_UPDATE_COUNT. But that is to reduce DB transactions and will not create issues as it is updated if there is change in power state.
Grace period can be increased by increasing ping interval. By default it is 60 seconds. You can increase it to 90 or 120 seconds to increase the grace period.
Code which is doing update
for (Map.Entry<Long, VirtualMachine.PowerState> entry : translatedInfo.entrySet()) {
if (s_logger.isDebugEnabled())
s_logger.debug("VM state report. host: " + hostId + ", vm id: " + entry.getKey() + ", power state: " + entry.getValue());
if (_instanceDao.updatePowerState(entry.getKey(), hostId, entry.getValue())) {
if (s_logger.isDebugEnabled())
s_logger.debug("VM state report is updated. host: " + hostId + ", vm id: " + entry.getKey() + ", power state: " + entry.getValue());
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland closed the pull request at:
https://github.com/apache/cloudstack/pull/829
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-140643483
@bhaisaab it should be for 4.5 as well but It is not complete yet, unless we decide that we don't take any action on a missing report. I will make a PR for 4.5 once we decide this is done like this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by anshul1886 <gi...@git.apache.org>.
Github user anshul1886 commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-141613687
@DaanHoogland what update_time are you talking about?
I think as a work around increasing ping interval to some higher value like 90 sec or 120 seconds in global settings will also do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by resmo <gi...@git.apache.org>.
Github user resmo commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-142906721
This is perfectly clear to me. The problem is we are relaying on the `update_time` (not to mix up with `power_state_update_time`).
~~~
*************************** 1. row ***************************
instance_name: r-2978-VM
type: DomainRouter
update_time: 2015-08-24 21:54:42
power_state_update_time: 2015-09-09 08:30:42
power_state_update_count: 3
1 row in set (0.00 sec)
~~~
here
~~~java
// 2 times of sync-update interval for graceful period
long milliSecondsGracefullPeriod = PingInterval.value() * 2000L;
for (VMInstanceVO instance : vmsThatAreMissingReport) {
Date vmStateUpdateTime = instance.getUpdateTime();
//...
long milliSecondsSinceLastStateUpdate = currentTime.getTime() - vmStateUpdateTime.getTime();
if (milliSecondsSinceLastStateUpdate > milliSecondsGracefullPeriod) {
~~~
so this calculation would do this:
~~~
milliSecondsSinceLastStateUpdate = currentTime.getTime() - vmStateUpdateTime.getTime()
2015-09-24 13:46:48 - 2015-08-24 21:54:42 ~= 1 month
if (1 month > 2 minutes grace period)
~~~
still the opinion, increasing grace period would help?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by anshul1886 <gi...@git.apache.org>.
Github user anshul1886 commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-140665483
Currently we report only PowerOn VMs and do not report PowerOff VMs that's why we consider Missing and PowerOff as same And that's how whole code is written and each Hypervisor resource has same understanding. This will effect HA and many more unknown places. So please do not even consider to merge this change.
So Now coming to bug we can fix that by changing global setting pingInterval to appropriate value according to hypervisor settings which takes care of these transitional period of missing report here or can be handled by introducing gracePeriod global setting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-8848: extra state to handle; n...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/829#issuecomment-141403050
@koushik-das I am perfectly fine with not putting this code in but I don't understand that you don't follow how it is related to the bug reported. The logs supplied shows how a running router is marked as stopped while it is running because no reports are received. These missing report events are treated as power-off events. It seems very clear to e that this is a bug in the state machine. The result is that the router is rebooted causing downtime for the users. If @resmo is satisfied with the solution @anshul1886 suggests we will settle for this bug remaining in?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---