You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Jonathan Gowar <jo...@whiteheat.org.uk> on 2014/04/08 20:31:35 UTC
Waiting some more to make sure there's no activity on VM
Hi,
I wanted to test the cluster, so created a Debian VM, saved it as a
template and then created 4 identical hosts from that template.
They appeared to create okay, as there are instances for each guest I
created. But, they will not start, are are stuck in a Stopped state.
The logs are being flooded with these messages:-
Waiting some more to make sure there's no activity on
VM[User|<hostname>]
Any ideas to fix this?
Thanks,
Jon
Re: Waiting some more to make sure there's no activity on VM
Posted by Jonathan Gowar <jo...@whiteheat.org.uk>.
On Wed, 2014-04-09 at 17:46 +0100, Jonathan Gowar wrote:
> * all guests have stopped
> * ha guest is STILL trying to start
> * hv-1 is attempting to connect
> * hv-2 alerting
> * virtual router running
> * ssvm + proxy running
It was the HA guest holding the cluster to ransom. The only way to deal
with it was to update the DB vm_instance state to Stopped. The cluster
then slowly came back to life. Is there a was to avoid this behaviour
from crippling the cluster? Perhaps a quorum of 3 hypervisors would be
better.
Thanks,
Jon
Re: Waiting some more to make sure there's no activity on VM
Posted by Jonathan Gowar <jo...@whiteheat.org.uk>.
On Wed, 2014-04-09 at 16:19 +0100, Jonathan Gowar wrote:
> The HA instance has not yet come back online, I'll wait to see what
> happens with this.
I left this for another hour, still nothing. I had left 1 of 2
hypervisor machines switched off during the testing, and then the
remaining host went to alert state.
I then powered the off host up, and CS registered it up. The other
machine stayed in an alerted state. I restarted cs agent, it made no
difference, so I rebooted it. It's come back, still in an alerted
state, with these messages "Lost connection to the server. Dealing with
the remaining commands...".
Next I restart CS management, and now the both hosts are offline, 1
still in it's alert state the other disconnected. All non-HA VMs have
stopped, and the 1 HA VM that's been trying to restart all this time, is
still trying to (bless!).
The System VMs have now entered Starting states, so they're not happy
either.
So, to recap, concisely. This has been result of 1 (invoked) powerfail
on 1 hypervisor.
Events
* lost power to hv-1
* ssv + proxy migrate and start
* virt router + ha guest fails to start
* .... waits
* after an hour virt router starts
* .... waits longer
* after another ha guest fails to start
* power on hv-1 in running state
* hv-2 changes to alert state
* restart agent on hv-2
* restart management on management
* reboot hv-2
* hv-2 returns with alert state
* hv-1 disconnects
Current state
* all guests have stopped
* ha guest is STILL trying to start
* hv-1 is attempting to connect
* hv-2 alerting
* virtual router running
* ssvm + proxy running
Seems like quite a horrific fallout in the event of loosing 1
hypervisor.
Jon
Re: Waiting some more to make sure there's no activity on VM
Posted by Jonathan Gowar <jo...@whiteheat.org.uk>.
On Wed, 2014-04-09 at 14:15 +0100, Jonathan Gowar wrote:
> My 2nd test took the same 3 minutes to register. All non HA instances
> went down, and the HA enabled instance attempted to migrate. The SSVM
> had to migrate first, and then the virtual router, but it hasn't and
> again more flooding the logs with these:-
There is seemingly no change to the cluster, and the virtual router
decided it was time to work:-
14:30:11 r-4-VM Current State = Starting, Previous State = Stopped
15:34:33 r-4-VM Current State = Running, Previous State = Starting
An hour to decide that. What are the usual causes for this sort of
behavour? How can I create redundant virtual routers?
The HA instance has not yet come back online, I'll wait to see what
happens with this.
Regards,
Jon
Re: Waiting some more to make sure there's no activity on VM
Posted by Jonathan Gowar <jo...@whiteheat.org.uk>.
I'm now testing DR by pulling the power on hypervisors.
My first test took 3 minutes for CS to register, and for the UI to
update. None of the VMs on that host had HA enabled, so also went down.
When I powered the host up, CS registered the change, but the VMs
remained down, I can imagine this is a desired default behavour. Is it
possible to tag specific instances to power on when host returns?
I created an HA instance.
My 2nd test took the same 3 minutes to register. All non HA instances
went down, and the HA enabled instance attempted to migrate. The SSVM
had to migrate first, and then the virtual router, but it hasn't and
again more flooding the logs with these:-
Waiting some more to make sure there's no activity on VM[DomainRouter|
r-4-VM]
It appears the HA instance will not start until the domain router does,
I assume this would therefore make all HA instances redundant until it's
back online.. is there a way to avoid this single point of failure?
Current cluster: 2x hypervisor, 1x NFS, 1x Management, 3x Ceph cluster
Regards,
Jon
On Wed, 2014-04-09 at 12:07 +0100, Jonathan Gowar wrote:
> On Tue, 2014-04-08 at 20:22 +0000, Geoff Higginbottom wrote:
> > If these were new VMs on a new guest network it's possible that the VR took a long time to start (poor secondary storage performance) so the VMs timed out.
>
> They were new VMs, but not on a new guest network, there was 1 VM
> already on it, that I used to template the others. There is a 1G
> private network.
>
> > It looks like one of your VMs had started, was this by chance the last VM you tried to start?
>
> The only one that was started, was the one that had always been online,
> the first VM I created.
>
> > Once the templates for VR and Guest VMs had copied from sec to pri storage new VMs should start very quickly.
>
> I'll do another test then.
>
> > Is this a single zone or part of a multi zone setup?
>
> Single.
>
> Thanks,
> Jon
>
Re: Waiting some more to make sure there's no activity on VM
Posted by Jonathan Gowar <jo...@whiteheat.org.uk>.
On Tue, 2014-04-08 at 20:22 +0000, Geoff Higginbottom wrote:
> If these were new VMs on a new guest network it's possible that the VR took a long time to start (poor secondary storage performance) so the VMs timed out.
They were new VMs, but not on a new guest network, there was 1 VM
already on it, that I used to template the others. There is a 1G
private network.
> It looks like one of your VMs had started, was this by chance the last VM you tried to start?
The only one that was started, was the one that had always been online,
the first VM I created.
> Once the templates for VR and Guest VMs had copied from sec to pri storage new VMs should start very quickly.
I'll do another test then.
> Is this a single zone or part of a multi zone setup?
Single.
Thanks,
Jon
Re: Waiting some more to make sure there's no activity on VM
Posted by Geoff Higginbottom <ge...@shapeblue.com>.
I've ran scripts which attempt to start 100 VMs at once, so no starting 5 should not be an issue.
If these were new VMs on a new guest network it's possible that the VR took a long time to start (poor secondary storage performance) so the VMs timed out.
It looks like one of your VMs had started, was this by chance the last VM you tried to start?
Once the templates for VR and Guest VMs had copied from sec to pri storage new VMs should start very quickly.
Is this a single zone or part of a multi zone setup?
Regards
Geoff Higginbottom
CTO / Cloud Architect
D: +44 20 3603 0542<tel:+442036030542> | S: +44 20 3603 0540<tel:+442036030540> | M: +447968161581<tel:+447968161581>
geoff.higginbottom@shapeblue.com<ma...@shapeblue.com> | www.shapeblue.com<htp://www.shapeblue.com/>
ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N 4HS<x-apple-data-detectors://37>
On 8 Apr 2014, at 21:11, "Jonathan Gowar" <jo...@whiteheat.org.uk>> wrote:
On Tue, 2014-04-08 at 19:14 +0000, Geoff Higginbottom wrote:
Assuming the VMs are not actually running on hosts, change the state field in vm_instance to stopped, this will give you back control. Suggest you then start one at a time.
Thanks, Geoff.
As they (4 of 5 VMs) were in a stopped state, and back within control,
I started just 1 and left it. I came back some time later, and they'd
all started!?
As far as stressing a system goes, would creating 5 VMs simultaneously
be a problem. Are there statistics to review such limits? or even
general rule of thumb.
Regards,
Jon
Need Enterprise Grade Support for Apache CloudStack?
Our CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> offers the best 24/7 SLA for CloudStack Environments.
Apache CloudStack Bootcamp training courses
**NEW!** CloudStack 4.2.1 training<http://shapeblue.com/cloudstack-training/>
28th-29th May 2014, Bangalore. Classromm<http://shapeblue.com/cloudstack-training/>
16th-20th June 2014, Region A. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
23rd-27th June 2014, Region B. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
15th-20th September 2014, Region A. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
22nd-27th September 2014, Region B. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
1st-6th December 2014, Region A. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
8th-12th December 2014, Region B. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Re: Waiting some more to make sure there's no activity on VM
Posted by Jonathan Gowar <jo...@whiteheat.org.uk>.
On Tue, 2014-04-08 at 19:14 +0000, Geoff Higginbottom wrote:
> Assuming the VMs are not actually running on hosts, change the state field in vm_instance to stopped, this will give you back control. Suggest you then start one at a time.
Thanks, Geoff.
As they (4 of 5 VMs) were in a stopped state, and back within control,
I started just 1 and left it. I came back some time later, and they'd
all started!?
As far as stressing a system goes, would creating 5 VMs simultaneously
be a problem. Are there statistics to review such limits? or even
general rule of thumb.
Regards,
Jon
Re: Waiting some more to make sure there's no activity on VM
Posted by Geoff Higginbottom <ge...@shapeblue.com>.
Assuming the VMs are not actually running on hosts, change the state field in vm_instance to stopped, this will give you back control. Suggest you then start one at a time.
Regards
Geoff Higginbottom
CTO / Cloud Architect
D: +44 20 3603 0542<tel:+442036030542> | S: +44 20 3603 0540<tel:+442036030540> | M: +447968161581<tel:+447968161581>
geoff.higginbottom@shapeblue.com<ma...@shapeblue.com> | www.shapeblue.com<htp://www.shapeblue.com/> | Twitter:@cloudstackguru<https://twitter.com/#!/cloudstackguru>
ShapeBlue Ltd, 53 Chandos Place, Covent Garden, London, WC2N 4HS<x-apple-data-detectors://5>
On 8 Apr 2014, at 19:32, "Jonathan Gowar" <jo...@whiteheat.org.uk>> wrote:
Hi,
I wanted to test the cluster, so created a Debian VM, saved it as a
template and then created 4 identical hosts from that template.
They appeared to create okay, as there are instances for each guest I
created. But, they will not start, are are stuck in a Stopped state.
The logs are being flooded with these messages:-
Waiting some more to make sure there's no activity on
VM[User|<hostname>]
Any ideas to fix this?
Thanks,
Jon
Need Enterprise Grade Support for Apache CloudStack?
Our CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> offers the best 24/7 SLA for CloudStack Environments.
Apache CloudStack Bootcamp training courses
**NEW!** CloudStack 4.2.1 training<http://shapeblue.com/cloudstack-training/>
28th-29th May 2014, Bangalore. Classromm<http://shapeblue.com/cloudstack-training/>
16th-20th June 2014, Region A. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
23rd-27th June 2014, Region B. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
15th-20th September 2014, Region A. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
22nd-27th September 2014, Region B. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
1st-6th December 2014, Region A. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
8th-12th December 2014, Region B. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a registered trademark.