You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Makrand <ma...@gmail.com> on 2016/05/13 06:58:28 UTC
Upgrade of Primary storage.
Guys,
Need little help. So in my second week on new job, I am suppose to upgrade
Nexeta (Primary storage and secondary storage for cloud) OS.
Overview of setup:-
1Zone>>1POD>>1 Cluster>>8 Hosts
Hypervisor:- Citrix Xenserer 6.5 (Free edition)
Storage Primary (Cluster Level):- On nexenta
Secondary Storage:- On same nexenta BOX (which caters primary)
Management Node is VM (Ubuntu 12004 LTS) whos root disk resides again on
primary storage coming from same nexenta BOX. Seems LUN is different. This
VM is running on Management cluster (2 XEN 6.5 hosts).
Well not sure why and how they kept single storage BOX for all primary and
secondary. Pretty wiered. It was done by a Vendor or someone long back.
Thats whole different story.
Any-who, coming to point of this email. I've chalked out plan like below.
------------------------------------------------------
A) Shutdown all VMs in following order:- First user VMs, then VRs & then
System VMs.
B) Put all XEN hosts in MM (Maintenance mode) from cloudstack.
1) Verify that all hosts are down (Shutdown them one by one) from XEN
Center
C) ssh into Management Server VM. Backup cloudstack DB save it on JumpBOX
(Use WinSCP)
D) Shutdown Cloudstack Management Server VM and other VMs on management
cluster. This needs to be done by issuing 'shutdown' command from OS level.
E) Shutdown Management cluster XEN hosts.
F) Upgrade the storage. (Reboot is needed at end.......hence so many
shutdowns)
G) Once upgrade is done start the Machines (Physical and Virtual) in
reveres order than that of shutdown. (Step E to A above) Something like
below
1) Management clsuter XEN hosts
2) Cloudstack management server VM and other management cluster VMs
3) XEN Hypervisor Nodes under cloud.
4) Take XEN hosts out of MM from cloudstack admin GUI.
5) Start SSVM and console proxy VM (I guess this time cloudstack will
recreate CPVM and SSVM if we start first user VM.....not sure if this
step will be needed)
6) Customer VMs and corresponding VRs
H) Verify from and try to deploy test VM.
------------------------------------------------------
All this downtime is ok. Do you think I am missing anything. Any comments
on improvising? Should I expect some more glitches? What was your previous
experience with primary storage upgrade.
Note:-
Last time same upgrade happened for one of similar zone and there was issue
of mapping primary storage LUNs with XEN hosts. Restart of hosts did trick
that time and hence my manager wants to shutdown all physical hosts this
time (Since all VMs have their disks on this storage box...which will
reboot once or twice during upgrade)
BTW
1) Is it necessary to put XEN hosts in MM from cloudstack? (If I am
shutting down all VMs).
2) After I bring UP Management server in step G-1, will it still have host
in MM? or What would it try to do after its up? (Assuming all hosts in
cluster are down at that time.
Thanks for reading.
--
Best,
Makrand
Re: Upgrade of Primary storage.
Posted by Prashant Mishra <pr...@accelerite.com>.
Here you go https://support.accelerite.com/hc/en-us/articles/208896246 ,
you may need to login to access it .
Thanks
Prashant
On 5/13/16, 6:38 PM, "Makrand" <ma...@gmail.com> wrote:
>Hi Adrian,
>
>Thanks for reply and hints.Will keep that in mind. But as of now I am more
>keen on upgrading storage BOX OS and absolutely ok with downtime. (aka
>management is not ready to spend anything on additional hardware/infra :D)
>
>Makrand
>
>
>
>--
>Best,
>Makrand
>
>
>On Fri, May 13, 2016 at 6:26 PM, Adrian Sender <as...@testlabs.com.au>
>wrote:
>
>> Hi Makrand,
>>
>> There are many different ways to achieve storage migration; I would
>> probably
>> add the new LUN and within cloudstack perform a storage migration on the
>> instances (there would potentially be no outage).
>>
>> For the NFS you could rsync preserving the directory structure, update
>>the
>> database and rebuild SSVM.
>>
>> There was an article for secondary storage migration but it seems to
>>have
>> vanished from the Citrix website; maybe someone from Accelerite knows
>> where it
>> is now - http://support.citrix.com/article/CTX135229
>>
>> - Adrian Sender
>>
>> ---------- Original Message -----------
>> From: Makrand <ma...@gmail.com>
>> To: users@cloudstack.apache.org
>> Sent: Fri, 13 May 2016 12:28:28 +0530
>> Subject: Upgrade of Primary storage.
>>
>> > Guys,
>> >
>> > Need little help. So in my second week on new job, I am suppose to
>> upgrade
>> > Nexeta (Primary storage and secondary storage for cloud) OS.
>> >
>> > Overview of setup:-
>> >
>> > 1Zone>>1POD>>1 Cluster>>8 Hosts
>> >
>> > Hypervisor:- Citrix Xenserer 6.5 (Free edition)
>> >
>> > Storage Primary (Cluster Level):- On nexenta
>> >
>> > Secondary Storage:- On same nexenta BOX (which caters primary)
>> >
>> > Management Node is VM (Ubuntu 12004 LTS) whos root disk resides
>> > again on primary storage coming from same nexenta BOX. Seems LUN is
>> > different. This VM is running on Management cluster (2 XEN 6.5 hosts).
>> >
>> > Well not sure why and how they kept single storage BOX for all
>> > primary and secondary. Pretty wiered. It was done by a Vendor or
>> > someone long back. Thats whole different story.
>> >
>> > Any-who, coming to point of this email. I've chalked out plan like
>>below.
>> > ------------------------------------------------------
>> > A) Shutdown all VMs in following order:- First user VMs, then VRs &
>>then
>> > System VMs.
>> >
>> > B) Put all XEN hosts in MM (Maintenance mode) from cloudstack.
>> >
>> > 1) Verify that all hosts are down (Shutdown them one by one) from
>> > XEN Center
>> >
>> > C) ssh into Management Server VM. Backup cloudstack DB save it on
>> JumpBOX
>> > (Use WinSCP)
>> >
>> > D) Shutdown Cloudstack Management Server VM and other VMs on
>>management
>> > cluster. This needs to be done by issuing 'shutdown' command from OS
>> > level.
>> >
>> > E) Shutdown Management cluster XEN hosts.
>> >
>> > F) Upgrade the storage. (Reboot is needed at end.......hence so many
>> > shutdowns)
>> >
>> > G) Once upgrade is done start the Machines (Physical and Virtual) in
>> > reveres order than that of shutdown. (Step E to A above) Something
>>like
>> > below
>> >
>> > 1) Management clsuter XEN hosts
>> > 2) Cloudstack management server VM and other management cluster VMs
>> > 3) XEN Hypervisor Nodes under cloud.
>> > 4) Take XEN hosts out of MM from cloudstack admin GUI.
>> > 5) Start SSVM and console proxy VM (I guess this time cloudstack
>> > will recreate CPVM and SSVM if we start first user VM.....not
>> > sure if this step will be needed) 6) Customer VMs and corresponding
>>VRs
>> >
>> > H) Verify from and try to deploy test VM.
>> > ------------------------------------------------------
>> >
>> > All this downtime is ok. Do you think I am missing anything. Any
>>comments
>> > on improvising? Should I expect some more glitches? What was your
>> previous
>> > experience with primary storage upgrade.
>> >
>> > Note:-
>> > Last time same upgrade happened for one of similar zone and there
>> > was issue of mapping primary storage LUNs with XEN hosts. Restart of
>> > hosts did trick that time and hence my manager wants to shutdown all
>> > physical hosts this time (Since all VMs have their disks on this
>> > storage box...which will reboot once or twice during upgrade)
>> >
>> > BTW
>> >
>> > 1) Is it necessary to put XEN hosts in MM from cloudstack? (If I am
>> > shutting down all VMs).
>> >
>> > 2) After I bring UP Management server in step G-1, will it still
>> > have host in MM? or What would it try to do after its up? (Assuming
>> > all hosts in cluster are down at that time.
>> >
>> > Thanks for reading.
>> >
>> > --
>> > Best,
>> > Makrand
>> ------- End of Original Message -------
>>
>>
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
Re: Upgrade of Primary storage.
Posted by Makrand <ma...@gmail.com>.
Hi Adrian,
Thanks for reply and hints.Will keep that in mind. But as of now I am more
keen on upgrading storage BOX OS and absolutely ok with downtime. (aka
management is not ready to spend anything on additional hardware/infra :D)
Makrand
--
Best,
Makrand
On Fri, May 13, 2016 at 6:26 PM, Adrian Sender <as...@testlabs.com.au>
wrote:
> Hi Makrand,
>
> There are many different ways to achieve storage migration; I would
> probably
> add the new LUN and within cloudstack perform a storage migration on the
> instances (there would potentially be no outage).
>
> For the NFS you could rsync preserving the directory structure, update the
> database and rebuild SSVM.
>
> There was an article for secondary storage migration but it seems to have
> vanished from the Citrix website; maybe someone from Accelerite knows
> where it
> is now - http://support.citrix.com/article/CTX135229
>
> - Adrian Sender
>
> ---------- Original Message -----------
> From: Makrand <ma...@gmail.com>
> To: users@cloudstack.apache.org
> Sent: Fri, 13 May 2016 12:28:28 +0530
> Subject: Upgrade of Primary storage.
>
> > Guys,
> >
> > Need little help. So in my second week on new job, I am suppose to
> upgrade
> > Nexeta (Primary storage and secondary storage for cloud) OS.
> >
> > Overview of setup:-
> >
> > 1Zone>>1POD>>1 Cluster>>8 Hosts
> >
> > Hypervisor:- Citrix Xenserer 6.5 (Free edition)
> >
> > Storage Primary (Cluster Level):- On nexenta
> >
> > Secondary Storage:- On same nexenta BOX (which caters primary)
> >
> > Management Node is VM (Ubuntu 12004 LTS) whos root disk resides
> > again on primary storage coming from same nexenta BOX. Seems LUN is
> > different. This VM is running on Management cluster (2 XEN 6.5 hosts).
> >
> > Well not sure why and how they kept single storage BOX for all
> > primary and secondary. Pretty wiered. It was done by a Vendor or
> > someone long back. Thats whole different story.
> >
> > Any-who, coming to point of this email. I've chalked out plan like below.
> > ------------------------------------------------------
> > A) Shutdown all VMs in following order:- First user VMs, then VRs & then
> > System VMs.
> >
> > B) Put all XEN hosts in MM (Maintenance mode) from cloudstack.
> >
> > 1) Verify that all hosts are down (Shutdown them one by one) from
> > XEN Center
> >
> > C) ssh into Management Server VM. Backup cloudstack DB save it on
> JumpBOX
> > (Use WinSCP)
> >
> > D) Shutdown Cloudstack Management Server VM and other VMs on management
> > cluster. This needs to be done by issuing 'shutdown' command from OS
> > level.
> >
> > E) Shutdown Management cluster XEN hosts.
> >
> > F) Upgrade the storage. (Reboot is needed at end.......hence so many
> > shutdowns)
> >
> > G) Once upgrade is done start the Machines (Physical and Virtual) in
> > reveres order than that of shutdown. (Step E to A above) Something like
> > below
> >
> > 1) Management clsuter XEN hosts
> > 2) Cloudstack management server VM and other management cluster VMs
> > 3) XEN Hypervisor Nodes under cloud.
> > 4) Take XEN hosts out of MM from cloudstack admin GUI.
> > 5) Start SSVM and console proxy VM (I guess this time cloudstack
> > will recreate CPVM and SSVM if we start first user VM.....not
> > sure if this step will be needed) 6) Customer VMs and corresponding VRs
> >
> > H) Verify from and try to deploy test VM.
> > ------------------------------------------------------
> >
> > All this downtime is ok. Do you think I am missing anything. Any comments
> > on improvising? Should I expect some more glitches? What was your
> previous
> > experience with primary storage upgrade.
> >
> > Note:-
> > Last time same upgrade happened for one of similar zone and there
> > was issue of mapping primary storage LUNs with XEN hosts. Restart of
> > hosts did trick that time and hence my manager wants to shutdown all
> > physical hosts this time (Since all VMs have their disks on this
> > storage box...which will reboot once or twice during upgrade)
> >
> > BTW
> >
> > 1) Is it necessary to put XEN hosts in MM from cloudstack? (If I am
> > shutting down all VMs).
> >
> > 2) After I bring UP Management server in step G-1, will it still
> > have host in MM? or What would it try to do after its up? (Assuming
> > all hosts in cluster are down at that time.
> >
> > Thanks for reading.
> >
> > --
> > Best,
> > Makrand
> ------- End of Original Message -------
>
>
Re: Upgrade of Primary storage.
Posted by Adrian Sender <as...@testlabs.com.au>.
Hi Makrand,
There are many different ways to achieve storage migration; I would probably
add the new LUN and within cloudstack perform a storage migration on the
instances (there would potentially be no outage).
For the NFS you could rsync preserving the directory structure, update the
database and rebuild SSVM.
There was an article for secondary storage migration but it seems to have
vanished from the Citrix website; maybe someone from Accelerite knows where it
is now - http://support.citrix.com/article/CTX135229
- Adrian Sender
---------- Original Message -----------
From: Makrand <ma...@gmail.com>
To: users@cloudstack.apache.org
Sent: Fri, 13 May 2016 12:28:28 +0530
Subject: Upgrade of Primary storage.
> Guys,
>
> Need little help. So in my second week on new job, I am suppose to upgrade
> Nexeta (Primary storage and secondary storage for cloud) OS.
>
> Overview of setup:-
>
> 1Zone>>1POD>>1 Cluster>>8 Hosts
>
> Hypervisor:- Citrix Xenserer 6.5 (Free edition)
>
> Storage Primary (Cluster Level):- On nexenta
>
> Secondary Storage:- On same nexenta BOX (which caters primary)
>
> Management Node is VM (Ubuntu 12004 LTS) whos root disk resides
> again on primary storage coming from same nexenta BOX. Seems LUN is
> different. This VM is running on Management cluster (2 XEN 6.5 hosts).
>
> Well not sure why and how they kept single storage BOX for all
> primary and secondary. Pretty wiered. It was done by a Vendor or
> someone long back. Thats whole different story.
>
> Any-who, coming to point of this email. I've chalked out plan like below.
> ------------------------------------------------------
> A) Shutdown all VMs in following order:- First user VMs, then VRs & then
> System VMs.
>
> B) Put all XEN hosts in MM (Maintenance mode) from cloudstack.
>
> 1) Verify that all hosts are down (Shutdown them one by one) from
> XEN Center
>
> C) ssh into Management Server VM. Backup cloudstack DB save it on JumpBOX
> (Use WinSCP)
>
> D) Shutdown Cloudstack Management Server VM and other VMs on management
> cluster. This needs to be done by issuing 'shutdown' command from OS
> level.
>
> E) Shutdown Management cluster XEN hosts.
>
> F) Upgrade the storage. (Reboot is needed at end.......hence so many
> shutdowns)
>
> G) Once upgrade is done start the Machines (Physical and Virtual) in
> reveres order than that of shutdown. (Step E to A above) Something like
> below
>
> 1) Management clsuter XEN hosts
> 2) Cloudstack management server VM and other management cluster VMs
> 3) XEN Hypervisor Nodes under cloud.
> 4) Take XEN hosts out of MM from cloudstack admin GUI.
> 5) Start SSVM and console proxy VM (I guess this time cloudstack
> will recreate CPVM and SSVM if we start first user VM.....not
> sure if this step will be needed) 6) Customer VMs and corresponding VRs
>
> H) Verify from and try to deploy test VM.
> ------------------------------------------------------
>
> All this downtime is ok. Do you think I am missing anything. Any comments
> on improvising? Should I expect some more glitches? What was your previous
> experience with primary storage upgrade.
>
> Note:-
> Last time same upgrade happened for one of similar zone and there
> was issue of mapping primary storage LUNs with XEN hosts. Restart of
> hosts did trick that time and hence my manager wants to shutdown all
> physical hosts this time (Since all VMs have their disks on this
> storage box...which will reboot once or twice during upgrade)
>
> BTW
>
> 1) Is it necessary to put XEN hosts in MM from cloudstack? (If I am
> shutting down all VMs).
>
> 2) After I bring UP Management server in step G-1, will it still
> have host in MM? or What would it try to do after its up? (Assuming
> all hosts in cluster are down at that time.
>
> Thanks for reading.
>
> --
> Best,
> Makrand
------- End of Original Message -------
Re: Upgrade of Primary storage.
Posted by Makrand <ma...@gmail.com>.
Thanks a ton Dag.....this is what I love about cloudstack email
lists....explain ur situation thoroughly and someone will improvise it.
BTW all primary storage is under iSCSI. Secondary under NFS.
Will iSCSI add any other step to plan?
This ain't a big zone (less than 100 VMS) and customer access to cloud is
via another GUI portal.....which is completely blocked during process. I
will give mm of primary storage a shot though.
--
Makrand
Sent from smartphone, using (over) smart autocorrect. Thus typos!
On May 13, 2016 3:53 PM, "Dag Sonstebo" <Da...@shapeblue.com> wrote:
> Hi Makrand,
>
> first of all - using the same storage appliance for all purposes
> (management VMs, primary and secondary storage) is fairly common, and fully
> viable as long as you run separate LUNs / NFS shares and tie down access
> accordingly.
>
> With regards to your shutdown / startup steps you're pretty much spot on.
> You haven't mentioned if you use NFS or iSCSI but the steps should be more
> or less the same.
> A couple of suggestions inline below:
>
>
> A) Shutdown all VMs in following order:- First user VMs, then VRs & then
> System VMs.
> >>> You probably want to disable the zone first of all to stop any further
> user access happening during any shutdown and startup procedures.
> >>> With regards to shutting down VMs you can achieve this by putting each
> primary storage pool into maintenance - this will shut down all running VMs
> and system VMs.
>
> B) Put all XEN hosts in MM (Maintenance mode) from cloudstack.
> >>> Once they are in maintenance you will still have to shut them down
> from XenCenter or command line.
>
> 1) Verify that all hosts are down (Shutdown them one by one) from XEN
> Center
>
> C) ssh into Management Server VM. Backup cloudstack DB save it on JumpBOX
> (Use WinSCP)
> >>> good practice, make sure you dump both the cloud and cloud_usage
> databases if you have both of these present
>
> D) Shutdown Cloudstack Management Server VM and other VMs on management
> cluster. This needs to be done by issuing 'shutdown' command from OS level.
>
> E) Shutdown Management cluster XEN hosts.
>
> F) Upgrade the storage. (Reboot is needed at end.......hence so many
> shutdowns)
>
> G) Once upgrade is done start the Machines (Physical and Virtual) in
> reveres order than that of shutdown. (Step E to A above) Something like
> below
>
> 1) Management clsuter XEN hosts
> 2) Cloudstack management server VM and other management cluster VMs
> 3) XEN Hypervisor Nodes under cloud.
> 4) Take XEN hosts out of MM from cloudstack admin GUI.
> >>> Take primary storage out of maintenance mode if you followed the
> suggestion above. Also re-enable the zone.
>
> 5) Start SSVM and console proxy VM (I guess this time cloudstack will
> recreate CPVM and SSVM if we start first user VM.....not sure if this
> step will be needed)
> >>> Correct, as long as everything is healthy these will be started
> automatically.
>
> 6) Customer VMs and corresponding VRs
> H) Verify from and try to deploy test VM.
>
> --
>
> Last time same upgrade happened for one of similar zone and there was issue
> of mapping primary storage LUNs with XEN hosts. Restart of hosts did trick
> that time and hence my manager wants to shutdown all physical hosts this
> time (Since all VMs have their disks on this storage box...which will
> reboot once or twice during upgrade)
> >>> Yes this seems to be a logical precaution, it ensures storage is
> correctly rescanned on startup.
>
> --
>
> 1) Is it necessary to put XEN hosts in MM from cloudstack? (If I am
> shutting down all VMs).
> >>> yes, in general always make sure CloudStack is aware of what is going
> on the underlying hosts, it keeps everything in synch.
>
> 2) After I bring UP Management server in step G-1, will it still have host
> in MM? or What would it try to do after its up? (Assuming all hosts in
> cluster are down at that time.
> >>> Yes they will still be in maintenance.
>
>
>
>
> Hope this helps, good luck with your upgrade.
>
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
>
>
>
>
>
> On 13/05/2016, 07:58, "Makrand" <ma...@gmail.com> wrote:
>
> >Guys,
> >
> >Need little help. So in my second week on new job, I am suppose to upgrade
> >Nexeta (Primary storage and secondary storage for cloud) OS.
> >
> >Overview of setup:-
> >
> >1Zone>>1POD>>1 Cluster>>8 Hosts
> >
> >Hypervisor:- Citrix Xenserer 6.5 (Free edition)
> >
> >Storage Primary (Cluster Level):- On nexenta
> >
> >Secondary Storage:- On same nexenta BOX (which caters primary)
> >
> >Management Node is VM (Ubuntu 12004 LTS) whos root disk resides again on
> >primary storage coming from same nexenta BOX. Seems LUN is different. This
> >VM is running on Management cluster (2 XEN 6.5 hosts).
> >
> >
> >Well not sure why and how they kept single storage BOX for all primary and
> >secondary. Pretty wiered. It was done by a Vendor or someone long back.
> >Thats whole different story.
> >
> >Any-who, coming to point of this email. I've chalked out plan like below.
> >------------------------------------------------------
> >A) Shutdown all VMs in following order:- First user VMs, then VRs & then
> >System VMs.
> >
> >B) Put all XEN hosts in MM (Maintenance mode) from cloudstack.
> >
> > 1) Verify that all hosts are down (Shutdown them one by one) from XEN
> >Center
> >
> >C) ssh into Management Server VM. Backup cloudstack DB save it on JumpBOX
> >(Use WinSCP)
> >
> >D) Shutdown Cloudstack Management Server VM and other VMs on management
> >cluster. This needs to be done by issuing 'shutdown' command from OS
> level.
> >
> >E) Shutdown Management cluster XEN hosts.
> >
> >F) Upgrade the storage. (Reboot is needed at end.......hence so many
> >shutdowns)
> >
> >G) Once upgrade is done start the Machines (Physical and Virtual) in
> >reveres order than that of shutdown. (Step E to A above) Something like
> >below
> >
> > 1) Management clsuter XEN hosts
> > 2) Cloudstack management server VM and other management cluster VMs
> > 3) XEN Hypervisor Nodes under cloud.
> > 4) Take XEN hosts out of MM from cloudstack admin GUI.
> > 5) Start SSVM and console proxy VM (I guess this time cloudstack will
> >recreate CPVM and SSVM if we start first user VM.....not sure if this
> >step will be needed)
> > 6) Customer VMs and corresponding VRs
> >
> >H) Verify from and try to deploy test VM.
> >------------------------------------------------------
> >
> >All this downtime is ok. Do you think I am missing anything. Any comments
> >on improvising? Should I expect some more glitches? What was your previous
> >experience with primary storage upgrade.
> >
> >Note:-
> >Last time same upgrade happened for one of similar zone and there was
> issue
> >of mapping primary storage LUNs with XEN hosts. Restart of hosts did trick
> >that time and hence my manager wants to shutdown all physical hosts this
> >time (Since all VMs have their disks on this storage box...which will
> >reboot once or twice during upgrade)
> >
> >BTW
> >
> >1) Is it necessary to put XEN hosts in MM from cloudstack? (If I am
> >shutting down all VMs).
> >
> >2) After I bring UP Management server in step G-1, will it still have host
> >in MM? or What would it try to do after its up? (Assuming all hosts in
> >cluster are down at that time.
> >
> >Thanks for reading.
> >
> >--
> >Best,
> >Makrand
>
> Regards,
>
> Dag Sonstebo
>
> Dag.Sonstebo@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
Re: Upgrade of Primary storage.
Posted by Dag Sonstebo <Da...@shapeblue.com>.
Hi Makrand,
first of all - using the same storage appliance for all purposes (management VMs, primary and secondary storage) is fairly common, and fully viable as long as you run separate LUNs / NFS shares and tie down access accordingly.
With regards to your shutdown / startup steps you're pretty much spot on. You haven't mentioned if you use NFS or iSCSI but the steps should be more or less the same.
A couple of suggestions inline below:
A) Shutdown all VMs in following order:- First user VMs, then VRs & then
System VMs.
>>> You probably want to disable the zone first of all to stop any further user access happening during any shutdown and startup procedures.
>>> With regards to shutting down VMs you can achieve this by putting each primary storage pool into maintenance - this will shut down all running VMs and system VMs.
B) Put all XEN hosts in MM (Maintenance mode) from cloudstack.
>>> Once they are in maintenance you will still have to shut them down from XenCenter or command line.
1) Verify that all hosts are down (Shutdown them one by one) from XEN
Center
C) ssh into Management Server VM. Backup cloudstack DB save it on JumpBOX
(Use WinSCP)
>>> good practice, make sure you dump both the cloud and cloud_usage databases if you have both of these present
D) Shutdown Cloudstack Management Server VM and other VMs on management
cluster. This needs to be done by issuing 'shutdown' command from OS level.
E) Shutdown Management cluster XEN hosts.
F) Upgrade the storage. (Reboot is needed at end.......hence so many
shutdowns)
G) Once upgrade is done start the Machines (Physical and Virtual) in
reveres order than that of shutdown. (Step E to A above) Something like
below
1) Management clsuter XEN hosts
2) Cloudstack management server VM and other management cluster VMs
3) XEN Hypervisor Nodes under cloud.
4) Take XEN hosts out of MM from cloudstack admin GUI.
>>> Take primary storage out of maintenance mode if you followed the suggestion above. Also re-enable the zone.
5) Start SSVM and console proxy VM (I guess this time cloudstack will
recreate CPVM and SSVM if we start first user VM.....not sure if this
step will be needed)
>>> Correct, as long as everything is healthy these will be started automatically.
6) Customer VMs and corresponding VRs
H) Verify from and try to deploy test VM.
--
Last time same upgrade happened for one of similar zone and there was issue
of mapping primary storage LUNs with XEN hosts. Restart of hosts did trick
that time and hence my manager wants to shutdown all physical hosts this
time (Since all VMs have their disks on this storage box...which will
reboot once or twice during upgrade)
>>> Yes this seems to be a logical precaution, it ensures storage is correctly rescanned on startup.
--
1) Is it necessary to put XEN hosts in MM from cloudstack? (If I am
shutting down all VMs).
>>> yes, in general always make sure CloudStack is aware of what is going on the underlying hosts, it keeps everything in synch.
2) After I bring UP Management server in step G-1, will it still have host
in MM? or What would it try to do after its up? (Assuming all hosts in
cluster are down at that time.
>>> Yes they will still be in maintenance.
Hope this helps, good luck with your upgrade.
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
On 13/05/2016, 07:58, "Makrand" <ma...@gmail.com> wrote:
>Guys,
>
>Need little help. So in my second week on new job, I am suppose to upgrade
>Nexeta (Primary storage and secondary storage for cloud) OS.
>
>Overview of setup:-
>
>1Zone>>1POD>>1 Cluster>>8 Hosts
>
>Hypervisor:- Citrix Xenserer 6.5 (Free edition)
>
>Storage Primary (Cluster Level):- On nexenta
>
>Secondary Storage:- On same nexenta BOX (which caters primary)
>
>Management Node is VM (Ubuntu 12004 LTS) whos root disk resides again on
>primary storage coming from same nexenta BOX. Seems LUN is different. This
>VM is running on Management cluster (2 XEN 6.5 hosts).
>
>
>Well not sure why and how they kept single storage BOX for all primary and
>secondary. Pretty wiered. It was done by a Vendor or someone long back.
>Thats whole different story.
>
>Any-who, coming to point of this email. I've chalked out plan like below.
>------------------------------------------------------
>A) Shutdown all VMs in following order:- First user VMs, then VRs & then
>System VMs.
>
>B) Put all XEN hosts in MM (Maintenance mode) from cloudstack.
>
> 1) Verify that all hosts are down (Shutdown them one by one) from XEN
>Center
>
>C) ssh into Management Server VM. Backup cloudstack DB save it on JumpBOX
>(Use WinSCP)
>
>D) Shutdown Cloudstack Management Server VM and other VMs on management
>cluster. This needs to be done by issuing 'shutdown' command from OS level.
>
>E) Shutdown Management cluster XEN hosts.
>
>F) Upgrade the storage. (Reboot is needed at end.......hence so many
>shutdowns)
>
>G) Once upgrade is done start the Machines (Physical and Virtual) in
>reveres order than that of shutdown. (Step E to A above) Something like
>below
>
> 1) Management clsuter XEN hosts
> 2) Cloudstack management server VM and other management cluster VMs
> 3) XEN Hypervisor Nodes under cloud.
> 4) Take XEN hosts out of MM from cloudstack admin GUI.
> 5) Start SSVM and console proxy VM (I guess this time cloudstack will
>recreate CPVM and SSVM if we start first user VM.....not sure if this
>step will be needed)
> 6) Customer VMs and corresponding VRs
>
>H) Verify from and try to deploy test VM.
>------------------------------------------------------
>
>All this downtime is ok. Do you think I am missing anything. Any comments
>on improvising? Should I expect some more glitches? What was your previous
>experience with primary storage upgrade.
>
>Note:-
>Last time same upgrade happened for one of similar zone and there was issue
>of mapping primary storage LUNs with XEN hosts. Restart of hosts did trick
>that time and hence my manager wants to shutdown all physical hosts this
>time (Since all VMs have their disks on this storage box...which will
>reboot once or twice during upgrade)
>
>BTW
>
>1) Is it necessary to put XEN hosts in MM from cloudstack? (If I am
>shutting down all VMs).
>
>2) After I bring UP Management server in step G-1, will it still have host
>in MM? or What would it try to do after its up? (Assuming all hosts in
>cluster are down at that time.
>
>Thanks for reading.
>
>--
>Best,
>Makrand
Regards,
Dag Sonstebo
Dag.Sonstebo@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue