You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Fraysse Guillaume <gf...@gmail.com> on 2013/07/25 19:57:34 UTC
Proofreading of our procedure to cleanly re-create vrouters and
restart CloudStack on a running VMware cluster after an interruption
Hello everyone,
We had an unexpected problem on our infrastructure while we were
upgrading CloudStack from 2.2.13 to 4.1 which led us to stop
CloudStack before being able to finalize the update.We still have the
vrouters to update but have a production platform that is a bit messy.
We're trying to find out how to finish he upgrade of the vrouters and
restart CloudStack when the VMware cluster has "lived his life" in the
meantime and some VM are in different states that when CLoudStack was
stopped.
Here is the procedure we are testing in a preproduction environment,
any advice/comment would be greatly appreciated as this is a risky
operation to perform on our production environment:
* Removing all vRouters from vCenter
* Updating vRouters states to « Stopped » in CloudStack database :
mysql> update vm_instance set state = 'Stopped' where state <>
'Expunging' and type = 'DomainRouter';
* Get the list of all vms :
mysql> select id, instance_name, state from vm_instance where state <>
'Expunging' and type = 'User';
* Updating table vm_instance to set the correct states and hosts of
vms. Example of SQL query :
mysql> update vm_instance set state = 'Running', host_id = 17,
last_host_id = 17 where id = 43;
* Restarting CloudStack
* Restarting all networks with « cleanup » option via CloudStack API.
Example of HTTP request :
$ curl http://cloudstack:8096/?command=restartNetwork\&cleanup=true\&response=json\&id=227
This procedure seems to be working except for a little detail : we see
that we use more Management IP adresses than expected, like they are
not de-allocated, could it be a side effect of the procedure ?
If you see any problem in this procedure or any problem it might lead
us to, your help would be appreciated.
Best regards,
Guillaume
Re: Proofreading of our procedure to cleanly re-create vrouters and
restart CloudStack on a running VMware cluster after an interruption
Posted by Ahmad Emneina <ae...@gmail.com>.
Hey Guillaume, not sure why you have to manually edit the db. After an
upgrade. Assuming you followed the steps in the release notes, you could
make the api call to rebootrouter on all the routers and that should do the
trick of recreating them properly.
On Jul 25, 2013 12:13 PM, "Sebastien Goasguen" <ru...@gmail.com> wrote:
> Ahmad, Alena, you might have some thoughts on Guillaume's email below ?
>
> -sebastien
>
> On Jul 25, 2013, at 1:57 PM, Fraysse Guillaume <gf...@gmail.com>
> wrote:
>
> > Hello everyone,
> >
> > We had an unexpected problem on our infrastructure while we were
> > upgrading CloudStack from 2.2.13 to 4.1 which led us to stop
> > CloudStack before being able to finalize the update.We still have the
> > vrouters to update but have a production platform that is a bit messy.
> > We're trying to find out how to finish he upgrade of the vrouters and
> > restart CloudStack when the VMware cluster has "lived his life" in the
> > meantime and some VM are in different states that when CLoudStack was
> > stopped.
> >
> > Here is the procedure we are testing in a preproduction environment,
> > any advice/comment would be greatly appreciated as this is a risky
> > operation to perform on our production environment:
> >
> > * Removing all vRouters from vCenter
> >
> > * Updating vRouters states to « Stopped » in CloudStack database :
> > mysql> update vm_instance set state = 'Stopped' where state <>
> > 'Expunging' and type = 'DomainRouter';
> >
> > * Get the list of all vms :
> > mysql> select id, instance_name, state from vm_instance where state <>
> > 'Expunging' and type = 'User';
> >
> > * Updating table vm_instance to set the correct states and hosts of
> > vms. Example of SQL query :
> > mysql> update vm_instance set state = 'Running', host_id = 17,
> > last_host_id = 17 where id = 43;
> >
> > * Restarting CloudStack
> >
> > * Restarting all networks with « cleanup » option via CloudStack API.
> > Example of HTTP request :
> > $ curl
> http://cloudstack:8096/?command=restartNetwork\&cleanup=true\&response=json\&id=227
> >
> > This procedure seems to be working except for a little detail : we see
> > that we use more Management IP adresses than expected, like they are
> > not de-allocated, could it be a side effect of the procedure ?
> >
> > If you see any problem in this procedure or any problem it might lead
> > us to, your help would be appreciated.
> >
> > Best regards,
> > Guillaume
>
>
Re: Proofreading of our procedure to cleanly re-create vrouters and
restart CloudStack on a running VMware cluster after an interruption
Posted by ni...@orange.com.
Hi,
Thanks for your help. We made another test using the API (stopRouter
with force=true) and restart network. All resources have been released.
We need to update the CloudStack's database in the same state as the
VMWare's database (like : VM host, VM state, .... )
We would like to make this request :
mysql> update vm_instance set state = 'Running', host_id = 17,
last_host_id = 17 where id = 43;
to update the hostID and the lastHostID of each virtual machine.
Could there be any edge effects ?
Regards.
Le 26/07/2013 04:56, Alex Huang a écrit :
> Instead of doing that, always call stop with the forced flag = true. That causes CloudStack to do a best effort to stop the vm but
> even if it cannot be stopped, CloudStack will cleanup all resources.
>
> --Alex
>
>> -----Original Message-----
>> From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
>> Sent: Thursday, July 25, 2013 1:27 PM
>> To: Sebastien Goasguen; dev@cloudstack.apache.org; Ahmad Emneina
>> Subject: Re: Proofreading of our procedure to cleanly re-create vrouters and
>> restart CloudStack on a running VMware cluster after an interruption
>>
>> Its incorrect that system vms are being 'stopped' by modifying the DB. In this
>> case CS doesn't release the resources - that's why you see private ip address
>> still being allocated. So the correct way to do things would be:
>>
>>
>> 1. Don't stop anything
>> 2. Upgrade
>> 3. Run restartNetwork&clenaup=true for all the networks. It will re-create
>> the routers. Old ones will get destroyed (and their resources will be released
>> - ip address, cpu, ram, etc - in the CS DB); the new routers will get re-
>> created.
>>
>> -Alena.
>>
>> From: Sebastien Goasguen <ru...@gmail.com>>
>> Date: Thursday, July 25, 2013 12:13 PM
>> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>"
>> <de...@cloudstack.apache.org>>, Ahmad
>> Emneina
>> <Ah...@citrix.com>>, Alena
>> Prokharchyk
>> <al...@citrix.com>>
>> Subject: Re: Proofreading of our procedure to cleanly re-create vrouters and
>> restart CloudStack on a running VMware cluster after an interruption
>>
>> Ahmad, Alena, you might have some thoughts on Guillaume's email below ?
>>
>> -sebastien
>>
>> On Jul 25, 2013, at 1:57 PM, Fraysse Guillaume
>> <gf...@gmail.com>> wrote:
>>
>> Hello everyone,
>> We had an unexpected problem on our infrastructure while we were
>> upgrading CloudStack from 2.2.13 to 4.1 which led us to stop CloudStack
>> before being able to finalize the update.We still have the vrouters to update
>> but have a production platform that is a bit messy.
>> We're trying to find out how to finish he upgrade of the vrouters and restart
>> CloudStack when the VMware cluster has "lived his life" in the meantime and
>> some VM are in different states that when CLoudStack was stopped.
>> Here is the procedure we are testing in a preproduction environment, any
>> advice/comment would be greatly appreciated as this is a risky operation to
>> perform on our production environment:
>> * Removing all vRouters from vCenter
>> * Updating vRouters states to < Stopped > in CloudStack database :
>> mysql> update vm_instance set state = 'Stopped' where state <>
>> 'Expunging' and type = 'DomainRouter';
>> * Get the list of all vms :
>> mysql> select id, instance_name, state from vm_instance where state <>
>> 'Expunging' and type = 'User';
>> * Updating table vm_instance to set the correct states and hosts of vms.
>> Example of SQL query :
>> mysql> update vm_instance set state = 'Running', host_id = 17,
>> last_host_id = 17 where id = 43;
>> * Restarting CloudStack
>> * Restarting all networks with < cleanup > option via CloudStack API.
>> Example of HTTP request :
>> $ curl
>> http://cloudstack:8096/?command=restartNetwork\&cleanup=true\&respo
>> nse=json\&id=227
>> This procedure seems to be working except for a little detail : we see that we
>> use more Management IP adresses than expected, like they are not de-
>> allocated, could it be a side effect of the procedure ?
>> If you see any problem in this procedure or any problem it might lead us to,
>> your help would be appreciated.
>> Best regards,
>> Guillaume
>>
>
>
>
--
Nicolas Lamirault
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
RE: Proofreading of our procedure to cleanly re-create vrouters and
restart CloudStack on a running VMware cluster after an interruption
Posted by Alex Huang <Al...@citrix.com>.
Instead of doing that, always call stop with the forced flag = true. That causes CloudStack to do a best effort to stop the vm but even if it cannot be stopped, CloudStack will cleanup all resources.
--Alex
> -----Original Message-----
> From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
> Sent: Thursday, July 25, 2013 1:27 PM
> To: Sebastien Goasguen; dev@cloudstack.apache.org; Ahmad Emneina
> Subject: Re: Proofreading of our procedure to cleanly re-create vrouters and
> restart CloudStack on a running VMware cluster after an interruption
>
> Its incorrect that system vms are being 'stopped' by modifying the DB. In this
> case CS doesn't release the resources - that's why you see private ip address
> still being allocated. So the correct way to do things would be:
>
>
> 1. Don't stop anything
> 2. Upgrade
> 3. Run restartNetwork&clenaup=true for all the networks. It will re-create
> the routers. Old ones will get destroyed (and their resources will be released
> - ip address, cpu, ram, etc - in the CS DB); the new routers will get re-
> created.
>
> -Alena.
>
> From: Sebastien Goasguen <ru...@gmail.com>>
> Date: Thursday, July 25, 2013 12:13 PM
> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>"
> <de...@cloudstack.apache.org>>, Ahmad
> Emneina
> <Ah...@citrix.com>>, Alena
> Prokharchyk
> <al...@citrix.com>>
> Subject: Re: Proofreading of our procedure to cleanly re-create vrouters and
> restart CloudStack on a running VMware cluster after an interruption
>
> Ahmad, Alena, you might have some thoughts on Guillaume's email below ?
>
> -sebastien
>
> On Jul 25, 2013, at 1:57 PM, Fraysse Guillaume
> <gf...@gmail.com>> wrote:
>
> Hello everyone,
> We had an unexpected problem on our infrastructure while we were
> upgrading CloudStack from 2.2.13 to 4.1 which led us to stop CloudStack
> before being able to finalize the update.We still have the vrouters to update
> but have a production platform that is a bit messy.
> We're trying to find out how to finish he upgrade of the vrouters and restart
> CloudStack when the VMware cluster has "lived his life" in the meantime and
> some VM are in different states that when CLoudStack was stopped.
> Here is the procedure we are testing in a preproduction environment, any
> advice/comment would be greatly appreciated as this is a risky operation to
> perform on our production environment:
> * Removing all vRouters from vCenter
> * Updating vRouters states to < Stopped > in CloudStack database :
> mysql> update vm_instance set state = 'Stopped' where state <>
> 'Expunging' and type = 'DomainRouter';
> * Get the list of all vms :
> mysql> select id, instance_name, state from vm_instance where state <>
> 'Expunging' and type = 'User';
> * Updating table vm_instance to set the correct states and hosts of vms.
> Example of SQL query :
> mysql> update vm_instance set state = 'Running', host_id = 17,
> last_host_id = 17 where id = 43;
> * Restarting CloudStack
> * Restarting all networks with < cleanup > option via CloudStack API.
> Example of HTTP request :
> $ curl
> http://cloudstack:8096/?command=restartNetwork\&cleanup=true\&respo
> nse=json\&id=227
> This procedure seems to be working except for a little detail : we see that we
> use more Management IP adresses than expected, like they are not de-
> allocated, could it be a side effect of the procedure ?
> If you see any problem in this procedure or any problem it might lead us to,
> your help would be appreciated.
> Best regards,
> Guillaume
>
Re: Proofreading of our procedure to cleanly re-create vrouters and
restart CloudStack on a running VMware cluster after an interruption
Posted by Alena Prokharchyk <Al...@citrix.com>.
Its incorrect that system vms are being 'stopped' by modifying the DB. In this case CS doesn't release the resources – that's why you see private ip address still being allocated. So the correct way to do things would be:
1. Don't stop anything
2. Upgrade
3. Run restartNetwork&clenaup=true for all the networks. It will re-create the routers. Old ones will get destroyed (and their resources will be released – ip address, cpu, ram, etc – in the CS DB); the new routers will get re-created.
-Alena.
From: Sebastien Goasguen <ru...@gmail.com>>
Date: Thursday, July 25, 2013 12:13 PM
To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>, Ahmad Emneina <Ah...@citrix.com>>, Alena Prokharchyk <al...@citrix.com>>
Subject: Re: Proofreading of our procedure to cleanly re-create vrouters and restart CloudStack on a running VMware cluster after an interruption
Ahmad, Alena, you might have some thoughts on Guillaume's email below ?
-sebastien
On Jul 25, 2013, at 1:57 PM, Fraysse Guillaume <gf...@gmail.com>> wrote:
Hello everyone,
We had an unexpected problem on our infrastructure while we were
upgrading CloudStack from 2.2.13 to 4.1 which led us to stop
CloudStack before being able to finalize the update.We still have the
vrouters to update but have a production platform that is a bit messy.
We're trying to find out how to finish he upgrade of the vrouters and
restart CloudStack when the VMware cluster has "lived his life" in the
meantime and some VM are in different states that when CLoudStack was
stopped.
Here is the procedure we are testing in a preproduction environment,
any advice/comment would be greatly appreciated as this is a risky
operation to perform on our production environment:
* Removing all vRouters from vCenter
* Updating vRouters states to « Stopped » in CloudStack database :
mysql> update vm_instance set state = 'Stopped' where state <>
'Expunging' and type = 'DomainRouter';
* Get the list of all vms :
mysql> select id, instance_name, state from vm_instance where state <>
'Expunging' and type = 'User';
* Updating table vm_instance to set the correct states and hosts of
vms. Example of SQL query :
mysql> update vm_instance set state = 'Running', host_id = 17,
last_host_id = 17 where id = 43;
* Restarting CloudStack
* Restarting all networks with « cleanup » option via CloudStack API.
Example of HTTP request :
$ curl http://cloudstack:8096/?command=restartNetwork\&cleanup=true\&response=json\&id=227
This procedure seems to be working except for a little detail : we see
that we use more Management IP adresses than expected, like they are
not de-allocated, could it be a side effect of the procedure ?
If you see any problem in this procedure or any problem it might lead
us to, your help would be appreciated.
Best regards,
Guillaume
Re: Proofreading of our procedure to cleanly re-create vrouters and restart CloudStack on a running VMware cluster after an interruption
Posted by Sebastien Goasguen <ru...@gmail.com>.
Ahmad, Alena, you might have some thoughts on Guillaume's email below ?
-sebastien
On Jul 25, 2013, at 1:57 PM, Fraysse Guillaume <gf...@gmail.com> wrote:
> Hello everyone,
>
> We had an unexpected problem on our infrastructure while we were
> upgrading CloudStack from 2.2.13 to 4.1 which led us to stop
> CloudStack before being able to finalize the update.We still have the
> vrouters to update but have a production platform that is a bit messy.
> We're trying to find out how to finish he upgrade of the vrouters and
> restart CloudStack when the VMware cluster has "lived his life" in the
> meantime and some VM are in different states that when CLoudStack was
> stopped.
>
> Here is the procedure we are testing in a preproduction environment,
> any advice/comment would be greatly appreciated as this is a risky
> operation to perform on our production environment:
>
> * Removing all vRouters from vCenter
>
> * Updating vRouters states to « Stopped » in CloudStack database :
> mysql> update vm_instance set state = 'Stopped' where state <>
> 'Expunging' and type = 'DomainRouter';
>
> * Get the list of all vms :
> mysql> select id, instance_name, state from vm_instance where state <>
> 'Expunging' and type = 'User';
>
> * Updating table vm_instance to set the correct states and hosts of
> vms. Example of SQL query :
> mysql> update vm_instance set state = 'Running', host_id = 17,
> last_host_id = 17 where id = 43;
>
> * Restarting CloudStack
>
> * Restarting all networks with « cleanup » option via CloudStack API.
> Example of HTTP request :
> $ curl http://cloudstack:8096/?command=restartNetwork\&cleanup=true\&response=json\&id=227
>
> This procedure seems to be working except for a little detail : we see
> that we use more Management IP adresses than expected, like they are
> not de-allocated, could it be a side effect of the procedure ?
>
> If you see any problem in this procedure or any problem it might lead
> us to, your help would be appreciated.
>
> Best regards,
> Guillaume