You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Ian Service <is...@ts2.ca> on 2015/05/11 16:09:39 UTC

Recover from outage

My management server became disconnected from one of the clusters it
manages yesterday evening.  I stopped the cloudstack-management process
because in the past this has wreaked havoc on that cluster, restarting
networks, bringing them up on different VLANs, causing unnecessary reboots
on instances, etc.

The database now shows most of the vm_instances as Stopped, even though
they are still very much running in XenCenter.

To get things back up and running, what's the recommended procedure to
avoid the unnecessary outages that CloudStack has caused me in the past?

If I simply start up cloudstack-management will it assess the currently
running machines or try to start them all again because the database says
they're stopped?

If I update the database to set each machine that's actually running as
Running and update host_id=last_host_id (as it appears to be null'd) would
that make CloudStack just check that they're still running and move on?

Any assistance/suggestions you can provide would be fantastic.

Thanks,

- Ian

Re: Recover from outage

Posted by Remi Bergsma <re...@remi.nl>.
Hi Ian,

From experience I know that as soon as CloudStack talks to a hypervisor, it will get a report of all running VMs and update its database accordingly. Even if you migrate a VM from outside of CloudStack, it will just pick this up. Starting CloudStack should be safe I’d say. Monitor the HA worker table for any scheduled HA work. Then you’ll have an idea of what it will do. You can then always decide to stop it again.

Good luck!

Regards,
Remi


> On 11 May 2015, at 20:18 , Vadim Kimlaychuk <Va...@Elion.ee> wrote:
> 
> Hello Ian,
> 
>              Cloudstack should get VM state from cloudstack agent automatically.  Check running SSVM and proxy VM that agent status is "running".  It just takes some time. Whether CS will automatically start VM depends on compute offering. If you have HA enabled than CS will try to start VM in a case it "thinks" it is stopped by itself.  But this is normal operational flow.  If you've had some trouble than it may be harder to fix. I wouldn't dare to make chages to database directly anyway. Will try to clarify that everything else works as it suppose to.  If you would like to experiment on DB -- make a backup first. 
> 
> Vadim.
> 
> ________________________________________
> From: Ian Service <is...@ts2.ca>
> Sent: Monday, May 11, 2015 17:09
> To: users@cloudstack.apache.org
> Subject: Recover from outage
> 
> My management server became disconnected from one of the clusters it
> manages yesterday evening.  I stopped the cloudstack-management process
> because in the past this has wreaked havoc on that cluster, restarting
> networks, bringing them up on different VLANs, causing unnecessary reboots
> on instances, etc.
> 
> The database now shows most of the vm_instances as Stopped, even though
> they are still very much running in XenCenter.
> 
> To get things back up and running, what's the recommended procedure to
> avoid the unnecessary outages that CloudStack has caused me in the past?
> 
> If I simply start up cloudstack-management will it assess the currently
> running machines or try to start them all again because the database says
> they're stopped?
> 
> If I update the database to set each machine that's actually running as
> Running and update host_id=last_host_id (as it appears to be null'd) would
> that make CloudStack just check that they're still running and move on?
> 
> Any assistance/suggestions you can provide would be fantastic.
> 
> Thanks,
> 
> - Ian


Re: Recover from outage

Posted by Vadim Kimlaychuk <Va...@Elion.ee>.
Hello Ian,

              Cloudstack should get VM state from cloudstack agent automatically.  Check running SSVM and proxy VM that agent status is "running".  It just takes some time. Whether CS will automatically start VM depends on compute offering. If you have HA enabled than CS will try to start VM in a case it "thinks" it is stopped by itself.  But this is normal operational flow.  If you've had some trouble than it may be harder to fix. I wouldn't dare to make chages to database directly anyway. Will try to clarify that everything else works as it suppose to.  If you would like to experiment on DB -- make a backup first. 

Vadim.
   
________________________________________
From: Ian Service <is...@ts2.ca>
Sent: Monday, May 11, 2015 17:09
To: users@cloudstack.apache.org
Subject: Recover from outage

My management server became disconnected from one of the clusters it
manages yesterday evening.  I stopped the cloudstack-management process
because in the past this has wreaked havoc on that cluster, restarting
networks, bringing them up on different VLANs, causing unnecessary reboots
on instances, etc.

The database now shows most of the vm_instances as Stopped, even though
they are still very much running in XenCenter.

To get things back up and running, what's the recommended procedure to
avoid the unnecessary outages that CloudStack has caused me in the past?

If I simply start up cloudstack-management will it assess the currently
running machines or try to start them all again because the database says
they're stopped?

If I update the database to set each machine that's actually running as
Running and update host_id=last_host_id (as it appears to be null'd) would
that make CloudStack just check that they're still running and move on?

Any assistance/suggestions you can provide would be fantastic.

Thanks,

- Ian