You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by da...@apache.org on 2015/04/10 22:15:03 UTC
[1/3] git commit: updated refs/heads/4.5 to 43a9eb4
Repository: cloudstack
Updated Branches:
refs/heads/4.5 bf3a0d608 -> 43a9eb40b
reboot much faster in case of storage failure
When storage cannot be reached, it does not make sense to reboot as it will try to flush buffers, umount NFS mounts, etc. This will not work and thus cause a long delay. With this change, the box will reboot immediately (like pressing the reset button).
Project: http://git-wip-us.apache.org/repos/asf/cloudstack/repo
Commit: http://git-wip-us.apache.org/repos/asf/cloudstack/commit/a92315f2
Tree: http://git-wip-us.apache.org/repos/asf/cloudstack/tree/a92315f2
Diff: http://git-wip-us.apache.org/repos/asf/cloudstack/diff/a92315f2
Branch: refs/heads/4.5
Commit: a92315f28c75df02b05368ca8cc87ad0832a4f97
Parents: bf3a0d6
Author: Remi Bergsma <gi...@remi.nl>
Authored: Wed Apr 1 19:45:16 2015 +0200
Committer: Daan Hoogland <dh...@schubergphilis.com>
Committed: Fri Apr 10 15:13:27 2015 -0500
----------------------------------------------------------------------
scripts/vm/hypervisor/kvm/kvmheartbeat.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/cloudstack/blob/a92315f2/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
----------------------------------------------------------------------
diff --git a/scripts/vm/hypervisor/kvm/kvmheartbeat.sh b/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
index 7293b14..0c07ffa 100755
--- a/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
+++ b/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
@@ -155,7 +155,7 @@ then
exit 0
elif [ "$cflag" == "1" ]
then
- reboot
+ echo b > /proc/sysrq-trigger
exit $?
else
write_hbLog
[2/3] git commit: updated refs/heads/4.5 to 43a9eb4
Posted by da...@apache.org.
write logfile just before rebooting the host
As discussed with @wido @pyr and @nuxro added an extra log line.
Tested it and it logs fine (tested to local disk) when syncing first:
Apr 3 15:31:23 mcctest2 heartbeat: kvmheartbeat.sh system because it was unable to write the heartbeat to the storage
By the way, it did also log to the agent.log but this extra log has the benefit of ending up in the system log so you'll probably find it easier there. Existing logs:
2015-04-03 15:27:23,943 WARN [kvm.resource.KVMHAMonitor] (Thread-24:null) write heartbeat failed: timeout, retry: 0
2015-04-03 15:28:23,944 WARN [kvm.resource.KVMHAMonitor] (Thread-24:null) write heartbeat failed: timeout, retry: 1
2015-04-03 15:29:23,946 WARN [kvm.resource.KVMHAMonitor] (Thread-24:null) write heartbeat failed: timeout, retry: 2
2015-04-03 15:30:23,948 WARN [kvm.resource.KVMHAMonitor] (Thread-24:null) write heartbeat failed: timeout, retry: 3
2015-04-03 15:31:23,950 WARN [kvm.resource.KVMHAMonitor] (Thread-24:null) write heartbeat failed: timeout, retry: 4
2015-04-03 15:31:23,950 WARN [kvm.resource.KVMHAMonitor] (Thread-24:null) write heartbeat failed: timeout; reboot the host
This closes #145
Signed-off-by: Rohit Yadav <ro...@shapeblue.com>
Project: http://git-wip-us.apache.org/repos/asf/cloudstack/repo
Commit: http://git-wip-us.apache.org/repos/asf/cloudstack/commit/b2366193
Tree: http://git-wip-us.apache.org/repos/asf/cloudstack/tree/b2366193
Diff: http://git-wip-us.apache.org/repos/asf/cloudstack/diff/b2366193
Branch: refs/heads/4.5
Commit: b23661931a6810986fc39f3f59c0a205979e11ab
Parents: a92315f
Author: Remi Bergsma <gi...@remi.nl>
Authored: Fri Apr 3 21:46:02 2015 +0200
Committer: Daan Hoogland <dh...@schubergphilis.com>
Committed: Fri Apr 10 15:13:37 2015 -0500
----------------------------------------------------------------------
scripts/vm/hypervisor/kvm/kvmheartbeat.sh | 2 ++
1 file changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/cloudstack/blob/b2366193/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
----------------------------------------------------------------------
diff --git a/scripts/vm/hypervisor/kvm/kvmheartbeat.sh b/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
index 0c07ffa..ff6fd0a 100755
--- a/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
+++ b/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
@@ -155,6 +155,8 @@ then
exit 0
elif [ "$cflag" == "1" ]
then
+ /usr/bin/logger -t heartbeat "kvmheartbeat.sh rebooted system because it was unable to write the heartbeat to the storage."
+ sync
echo b > /proc/sysrq-trigger
exit $?
else
[3/3] git commit: updated refs/heads/4.5 to 43a9eb4
Posted by da...@apache.org.
make sure sync cannot block reboot
The recent discussed improvement has the risk that if 'sync' hangs, the reboot may be delayed in the same way as the 'reboot' command would do. To work around, we're adding a 5 second timeout. If it cannot sync in 5 seconds, it will not succeed anyway and we should proceed the reset.
@snuf: Could we use your OVM3 heartbeat script for other hypervisors as well? One way to do it seems like a nice idea :-)
Project: http://git-wip-us.apache.org/repos/asf/cloudstack/repo
Commit: http://git-wip-us.apache.org/repos/asf/cloudstack/commit/43a9eb40
Tree: http://git-wip-us.apache.org/repos/asf/cloudstack/tree/43a9eb40
Diff: http://git-wip-us.apache.org/repos/asf/cloudstack/diff/43a9eb40
Branch: refs/heads/4.5
Commit: 43a9eb40b87adf470b9cbfbd75a12b48bd1bcb30
Parents: b236619
Author: Remi Bergsma <gi...@remi.nl>
Authored: Thu Apr 9 12:18:21 2015 +0200
Committer: Daan Hoogland <dh...@schubergphilis.com>
Committed: Fri Apr 10 15:14:08 2015 -0500
----------------------------------------------------------------------
scripts/vm/hypervisor/kvm/kvmheartbeat.sh | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/cloudstack/blob/43a9eb40/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
----------------------------------------------------------------------
diff --git a/scripts/vm/hypervisor/kvm/kvmheartbeat.sh b/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
index ff6fd0a..7c8ee67 100755
--- a/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
+++ b/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
@@ -156,7 +156,8 @@ then
elif [ "$cflag" == "1" ]
then
/usr/bin/logger -t heartbeat "kvmheartbeat.sh rebooted system because it was unable to write the heartbeat to the storage."
- sync
+ sync &
+ sleep 5
echo b > /proc/sysrq-trigger
exit $?
else