You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Edison Su <Ed...@citrix.com> on 2013/08/07 22:28:35 UTC
[DISCUSS] KVM HA
There is long time issue related to KVM HA, see bug: CLOUDSTACK-3535. Basically, HA won't be triggered, if KVM agent is stopped either normally nor abnormally, HA only be triggered if the network between mgt server and kvm host is disconnected and the network between KVM hosts in the same cluster is disconnected.
Here is how the KVM HA works after the fix for CLOUDSTACK-3535:
1. If agent is stopped, agent will send a shutdown request to mgt server, mgt server will mark the host as disconnected, while still maintain the host in pingmap. Code is in AgentManagerImpl->AgentHandler- >ProcessRequest-> disconnectWithoutInvestigation
2. After ping.interval, mgt server will find the host is ping timeout, then start HA investigation for the host. Code is in AgentMonitor->run-> disconnectWithInvestigation
3. Mgt server will call all the available Investigators to investigate the status of host.
The current investigators will be called for KVM host:
UserVmDomRInvestigator->isAgentAlive, will send PingTestCommand to the host's neighbor. PingTestCommand will ping host's private ip address, if ping is reachable, means host is up, otherwise, host's state is unknown. So this investigator can only detect host is in up state.
KVMInvestigator, which is newly added, will send a CheckOnHostCommand to host's neighbor. CheckOnHostCommand will check the heartbeat of host(heartbeat is stored on shared primary storage). Ideally, it will detect host is down or up.
Combined with UserVmDomRInvestigator and KVMInvestigator, mgt server should find out the status of host. But there is case, these two investigators can report wrong status of host:
Host is in a network partition, while the KVM agent is down(thus heartbeat is stopped)
4. After investigator reports status of host, if host is down, then start HA for VMs created on the host.
Improvement:
Per suggestion from Lennert den Teuling, we'd better use IPMI to detect host status, which is more reliable than ping and heartbeat, as IPMI has its own network, less likely has network partition.
RE: [DISCUSS] KVM HA
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> Sent: Wednesday, August 07, 2013 1:42 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [DISCUSS] KVM HA
>
> Does KVMInvestigator work on all shared primary storage, or just NFS?
Right now, the agent only writes heartbeat into NFS primary storage, so the current implementation only works with NFS primary storage.
But the heartbeat file is just only one implementation, if using IPMI, it should be able to detect host status also.
> I'm only familiar with the NFS KVMHA directories.
>
> From this it seems like a clean stop of the KVM agent still shouldn't trigger
> any issues/HA, correct?
Yes, a clean stop wont' trigger HA. "reboot -f" or plug off power supply will trigger HA.
>
> On Wed, Aug 7, 2013 at 2:28 PM, Edison Su <Ed...@citrix.com> wrote:
> > There is long time issue related to KVM HA, see bug: CLOUDSTACK-3535.
> Basically, HA won't be triggered, if KVM agent is stopped either normally nor
> abnormally, HA only be triggered if the network between mgt server and
> kvm host is disconnected and the network between KVM hosts in the same
> cluster is disconnected.
> > Here is how the KVM HA works after the fix for CLOUDSTACK-3535:
> > 1. If agent is stopped, agent will send a shutdown request to mgt server,
> mgt server will mark the host as disconnected, while still maintain the host in
> pingmap. Code is in AgentManagerImpl->AgentHandler- >ProcessRequest->
> disconnectWithoutInvestigation
> > 2. After ping.interval, mgt server will find the host is ping timeout,
> > then start HA investigation for the host. Code is in AgentMonitor->run->
> disconnectWithInvestigation 3. Mgt server will call all the available
> Investigators to investigate the status of host.
> > The current investigators will be called for KVM host:
> > UserVmDomRInvestigator->isAgentAlive, will send PingTestCommand
> to the host's neighbor. PingTestCommand will ping host's private ip address,
> if ping is reachable, means host is up, otherwise, host's state is unknown. So
> this investigator can only detect host is in up state.
> > KVMInvestigator, which is newly added, will send a
> CheckOnHostCommand to host's neighbor. CheckOnHostCommand will
> check the heartbeat of host(heartbeat is stored on shared primary storage).
> Ideally, it will detect host is down or up.
> >
> > Combined with UserVmDomRInvestigator and KVMInvestigator, mgt
> server should find out the status of host. But there is case, these two
> investigators can report wrong status of host:
> > Host is in a network partition, while the KVM agent is
> > down(thus heartbeat is stopped) 4. After investigator reports status of
> host, if host is down, then start HA for VMs created on the host.
> >
> >
> > Improvement:
> > Per suggestion from Lennert den Teuling, we'd better use IPMI to
> detect host status, which is more reliable than ping and heartbeat, as IPMI
> has its own network, less likely has network partition.
> >
> >
RE: [DISCUSS] KVM HA
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> Sent: Wednesday, August 07, 2013 1:46 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [DISCUSS] KVM HA
>
> I'm not sure we can rely on IPMI to tell us much about the host status itself.
> It's easy to use it for checking on basic poweron/poweroff, temperature, etc,
> but not so easy to tell if something is wrong with the OS, config, or at the
> software level.
Check the host status is enough, the investigator only needs to know, the host is power on or off.
If host is power off, then cloudstack can start VM on another host, without worried about the split brain issue.
>
> However, I did mention support in that thread early on for sending an IPMI
> poweroff for hosts that cloudstack has determined are down and starts
> migrating vms for, as a safety precaution.
>
> On Wed, Aug 7, 2013 at 2:41 PM, Marcus Sorensen <sh...@gmail.com>
> wrote:
> > Does KVMInvestigator work on all shared primary storage, or just NFS?
> > I'm only familiar with the NFS KVMHA directories.
> >
> > From this it seems like a clean stop of the KVM agent still shouldn't
> > trigger any issues/HA, correct?
> >
> > On Wed, Aug 7, 2013 at 2:28 PM, Edison Su <Ed...@citrix.com> wrote:
> >> There is long time issue related to KVM HA, see bug: CLOUDSTACK-3535.
> Basically, HA won't be triggered, if KVM agent is stopped either normally nor
> abnormally, HA only be triggered if the network between mgt server and
> kvm host is disconnected and the network between KVM hosts in the same
> cluster is disconnected.
> >> Here is how the KVM HA works after the fix for CLOUDSTACK-3535:
> >> 1. If agent is stopped, agent will send a shutdown request to mgt server,
> mgt server will mark the host as disconnected, while still maintain the host in
> pingmap. Code is in AgentManagerImpl->AgentHandler- >ProcessRequest->
> disconnectWithoutInvestigation
> >> 2. After ping.interval, mgt server will find the host is ping
> >> timeout, then start HA investigation for the host. Code is in
> AgentMonitor->run-> disconnectWithInvestigation 3. Mgt server will call all
> the available Investigators to investigate the status of host.
> >> The current investigators will be called for KVM host:
> >> UserVmDomRInvestigator->isAgentAlive, will send PingTestCommand
> to the host's neighbor. PingTestCommand will ping host's private ip address,
> if ping is reachable, means host is up, otherwise, host's state is unknown. So
> this investigator can only detect host is in up state.
> >> KVMInvestigator, which is newly added, will send a
> CheckOnHostCommand to host's neighbor. CheckOnHostCommand will
> check the heartbeat of host(heartbeat is stored on shared primary storage).
> Ideally, it will detect host is down or up.
> >>
> >> Combined with UserVmDomRInvestigator and KVMInvestigator, mgt
> server should find out the status of host. But there is case, these two
> investigators can report wrong status of host:
> >> Host is in a network partition, while the KVM agent is
> >> down(thus heartbeat is stopped) 4. After investigator reports status of
> host, if host is down, then start HA for VMs created on the host.
> >>
> >>
> >> Improvement:
> >> Per suggestion from Lennert den Teuling, we'd better use IPMI to
> detect host status, which is more reliable than ping and heartbeat, as IPMI
> has its own network, less likely has network partition.
> >>
> >>
Re: [DISCUSS] KVM HA
Posted by Marcus Sorensen <sh...@gmail.com>.
I'm not sure we can rely on IPMI to tell us much about the host status
itself. It's easy to use it for checking on basic poweron/poweroff,
temperature, etc, but not so easy to tell if something is wrong with
the OS, config, or at the software level.
However, I did mention support in that thread early on for sending an
IPMI poweroff for hosts that cloudstack has determined are down and
starts migrating vms for, as a safety precaution.
On Wed, Aug 7, 2013 at 2:41 PM, Marcus Sorensen <sh...@gmail.com> wrote:
> Does KVMInvestigator work on all shared primary storage, or just NFS?
> I'm only familiar with the NFS KVMHA directories.
>
> From this it seems like a clean stop of the KVM agent still shouldn't
> trigger any issues/HA, correct?
>
> On Wed, Aug 7, 2013 at 2:28 PM, Edison Su <Ed...@citrix.com> wrote:
>> There is long time issue related to KVM HA, see bug: CLOUDSTACK-3535. Basically, HA won't be triggered, if KVM agent is stopped either normally nor abnormally, HA only be triggered if the network between mgt server and kvm host is disconnected and the network between KVM hosts in the same cluster is disconnected.
>> Here is how the KVM HA works after the fix for CLOUDSTACK-3535:
>> 1. If agent is stopped, agent will send a shutdown request to mgt server, mgt server will mark the host as disconnected, while still maintain the host in pingmap. Code is in AgentManagerImpl->AgentHandler- >ProcessRequest-> disconnectWithoutInvestigation
>> 2. After ping.interval, mgt server will find the host is ping timeout, then start HA investigation for the host. Code is in AgentMonitor->run-> disconnectWithInvestigation
>> 3. Mgt server will call all the available Investigators to investigate the status of host.
>> The current investigators will be called for KVM host:
>> UserVmDomRInvestigator->isAgentAlive, will send PingTestCommand to the host's neighbor. PingTestCommand will ping host's private ip address, if ping is reachable, means host is up, otherwise, host's state is unknown. So this investigator can only detect host is in up state.
>> KVMInvestigator, which is newly added, will send a CheckOnHostCommand to host's neighbor. CheckOnHostCommand will check the heartbeat of host(heartbeat is stored on shared primary storage). Ideally, it will detect host is down or up.
>>
>> Combined with UserVmDomRInvestigator and KVMInvestigator, mgt server should find out the status of host. But there is case, these two investigators can report wrong status of host:
>> Host is in a network partition, while the KVM agent is down(thus heartbeat is stopped)
>> 4. After investigator reports status of host, if host is down, then start HA for VMs created on the host.
>>
>>
>> Improvement:
>> Per suggestion from Lennert den Teuling, we'd better use IPMI to detect host status, which is more reliable than ping and heartbeat, as IPMI has its own network, less likely has network partition.
>>
>>
Re: [DISCUSS] KVM HA
Posted by Marcus Sorensen <sh...@gmail.com>.
Does KVMInvestigator work on all shared primary storage, or just NFS?
I'm only familiar with the NFS KVMHA directories.
>From this it seems like a clean stop of the KVM agent still shouldn't
trigger any issues/HA, correct?
On Wed, Aug 7, 2013 at 2:28 PM, Edison Su <Ed...@citrix.com> wrote:
> There is long time issue related to KVM HA, see bug: CLOUDSTACK-3535. Basically, HA won't be triggered, if KVM agent is stopped either normally nor abnormally, HA only be triggered if the network between mgt server and kvm host is disconnected and the network between KVM hosts in the same cluster is disconnected.
> Here is how the KVM HA works after the fix for CLOUDSTACK-3535:
> 1. If agent is stopped, agent will send a shutdown request to mgt server, mgt server will mark the host as disconnected, while still maintain the host in pingmap. Code is in AgentManagerImpl->AgentHandler- >ProcessRequest-> disconnectWithoutInvestigation
> 2. After ping.interval, mgt server will find the host is ping timeout, then start HA investigation for the host. Code is in AgentMonitor->run-> disconnectWithInvestigation
> 3. Mgt server will call all the available Investigators to investigate the status of host.
> The current investigators will be called for KVM host:
> UserVmDomRInvestigator->isAgentAlive, will send PingTestCommand to the host's neighbor. PingTestCommand will ping host's private ip address, if ping is reachable, means host is up, otherwise, host's state is unknown. So this investigator can only detect host is in up state.
> KVMInvestigator, which is newly added, will send a CheckOnHostCommand to host's neighbor. CheckOnHostCommand will check the heartbeat of host(heartbeat is stored on shared primary storage). Ideally, it will detect host is down or up.
>
> Combined with UserVmDomRInvestigator and KVMInvestigator, mgt server should find out the status of host. But there is case, these two investigators can report wrong status of host:
> Host is in a network partition, while the KVM agent is down(thus heartbeat is stopped)
> 4. After investigator reports status of host, if host is down, then start HA for VMs created on the host.
>
>
> Improvement:
> Per suggestion from Lennert den Teuling, we'd better use IPMI to detect host status, which is more reliable than ping and heartbeat, as IPMI has its own network, less likely has network partition.
>
>