Re: [DISCUSS] Host HA in 4.11

[Rohit Yadav <ro...@shapeblue.com>]

All,


Over last couple of weeks, we've improved the host-ha implementation drastically with following changes:


- Reduced number of background polling tasks from 4 to 1

- Added additional tests around degrading, fencing, and recovering FSM states with real KVM hypervisor, these tests run/work with Trillian

- Refactored the implementation to be state/event driven than poll/loop driven

- Fixed VM HA for recovered host


The host-ha PR is ready for merging now, it has passed tests on Travis, Trillian and BVT:

https://github.com/apache/cloudstack/pull/1960


I'll keep the PR open for sometime this week to seek any last minute feedback and/or comment and merge by eod tomorrow, thanks.


- Rohit

________________________________
From: Rohit Yadav <ro...@shapeblue.com>
Sent: Thursday, July 13, 2017 12:18:58 PM
To: dev@cloudstack.apache.org
Cc: Nathan Johnson; David Mabry; Aaron Hurt
Subject: Re: [DISCUSS] Host HA in 4.11

Thanks for the interest Simon, the feature has two HA provider (plugins, one for simulator and another for KVM+NFS) that can be used as reference implementations.


- Rohit

________________________________
From: Simon Weller <sw...@ena.com.INVALID>
Sent: 12 July 2017 20:59:38
To: dev@cloudstack.apache.org
Cc: Nathan Johnson; David Mabry; Aaron Hurt
Subject: Re: [DISCUSS] Host HA in 4.11

We are very excited about this feature set, as it adds some really important features for KVM.

We don't use NFS, so I think our goal will be seeing what we can contribute to include Ceph on the supported storage list.

- Si
________________________________
From: Rohit Yadav <ro...@shapeblue.com>
Sent: Wednesday, July 12, 2017 5:43 AM
To: dev@cloudstack.apache.org
Subject: [DISCUSS] Host HA in 4.11

All,


Few months ago I had started discussion on Host HA for CloudStack and given 4.10 is voted and to be announced with master branch cut I would like to re-kick discussion around reviewing and acceptance of the feature that is pending since Feb 2017.


To briefly share some key points:

- This feature is disabled by default and provides zone/cluster/host level kill switches

- This brings in a reliable way to fence (power off) and recover (reboot) a host

- Allows implementation of HA provider plugin specific to a hypervisor and storage stack, by default we've implemented a plugin for hosts that have KVM+NFS

- For more details please read the FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA
Host HA - Apache Cloudstack - Apache Software Foundation<https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA>
cwiki.apache.org
CLOUDSTACK-9782. Branch. Yet to start, share the PR. Introduction. CloudStack lacks a way to reliably fence a host, the idea of the host-ha feature is to provide a ...





I had also given a talk about this feature during CCCNA17:

Reliable host fencing - http://rohit.yadav.xyz/files/talks/cccna17-reliable-host-fencing.pdf


Pull request: https://github.com/apache/cloudstack/pull/1960 (as soon as the 4.10->4.11 db upgrade paths are fixed, I can rebase and fix the branch)
[https://avatars1.githubusercontent.com/u/95203?v=3&s=400]<https://github.com/apache/cloudstack/pull/1960>

[4.11/Future] CLOUDSTACK-9782: Host HA and KVM HA provider by rhtyd · Pull Request #1960 · apache/cloudstack<https://github.com/apache/cloudstack/pull/1960>
github.com
Host-HA offers investigation, fencing and recovery mechanisms for host that for any reason are malfunctioning. It uses Activity and Health checks to determine current host state based on which it m...





- Rohit

rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<http://www.shapeblue.com/>

Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
www.shapeblue.com<http://www.shapeblue.com>
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a framework developed by ShapeBlue to deliver the rapid deployment of a standardised ...



53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




rohit.yadav@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue