You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by David Merrill <da...@otelco.com> on 2020/06/19 21:23:53 UTC
Upgrading XenServer Clusters managed by ACS...
Hi All,
I have a production deployment of ACS managing three XenServer Clusters (XenServer pools of 6 hosts each) in two different Zones. I now find myself in the position of needing to do a major version upgrade of those hosts. Happily I have a ACS lab managing a XenServer cluster running the same (old) version of XenServer that I can practice on.
I have plenty of practice operating ACS to “quiesce things” for XenServer patches (start with the pool master, move guest VMs off, put that host into maintenance mode, unmanage the cluster, patch the host, then reverse & move onto the next host with the same steps except we don’t bother w/un-managing/re-managing the cluster), but as I understand a XenServer version upgrade backs up the whole original XenServer installation to another partition and makes a clean installation of XenServer on the original partition (the problem there being that when the upgraded XenServer boots up all the ACS provided/copied scripts are not there & ACS can’t manage the host).
So not much of an ask here (OK maybe at the end – have I missed something obvious or doing anything foolish?), I wanted to share a bit research & lay out a set of steps that I think will work to get a pool of XenServers in a cluster upgraded and end up in a place where ACS is happy with them.
Bear with me it’s a little long,
1. In XenCenter – if HA is enabled for the XenServer pool, disable it
2. Stop ACS management/usage services
3. Do MySQL database backups
4. Start ACS management/usage services
5. Start with the pool master.
6. In ACS – Migrate all guest VMs to other hosts in the cluster
7. In ACS – Prepare to put the pool master into maintenance mode (so no new guest VMs)
* A caveat here related to this item I found when researching – https://www.shapeblue.com/how-to-upgrade-an-apache-cloudstack-citrix-xenserver-cluster/
i. A recommendation is made here to edit /etc/cloudstack/management/environment.properties
ii. And set manage.xenserver.pool.master=false
iii. And restart CloudStack management services
iv. BECAUSE if one didn’t I understand CloudStack WOULD force an election for another host to become the pool master (which is “bad” as ASCs is configured to speak to the currently configured pool master)
* HOWEVER THIS MAY NOT BE NECESSARY
i. Found a thread titled “A Story of a failed XenServer Upgrade” here – http://mail-archives.apache.org/mod_mbox/cloudstack-users/201601.mbox/browser
ii. At the end of the thread Paul Angus states that Geoff’s ShapeBlue blog article was written in the ACS 4.3 era and that ACS’ behavior “used to be that putting a host that was the pool master into maintenance would cause CloudStack to force an election for another host to become pool master - stopping you from then upgrading the active pool master first. There wasn't an 'unmanage' button at the time.”
iii. The implication, (in my humble estimation) is that, today, one no longer needs to make these edits to /etc/cloudstack/management/environment.properties
1. In ACS – Put the pool master into maintenance mode (so no new guest VMs)
2. In ACS – Un-manage the cluster
3. NOW – Upgrade the XenServer pool master to the latest release
* Exercise left to the reader…(I’ll share what I end up doing)
4. In XenServer – Do the following (found this nugget in a thread last November about XenServer upgrades & how to re-setup the host – thanks Richard Lawley!)
* Remove this host tag: xe host-param-remove uuid=HOSTUUID param-name=tags param-key=vmops-version-com.cloud.hypervisor.xenserver.resource.XenServer650R
* This makes management server set the host back up, presumably since ACS has credentials to the host in the database it can copy all the scripts back
5. In ACS – Re-manage the cluster
6. In ACS – Exit maintenance-mode for the newly upgraded host
7. In ACS – Observe that the newly upgraded host is “Enabled” and “Up” in the UI (Infrastructure --> Hosts)
8. In ACS – Testing (e.g. move an existing router/VM to the upgraded host, create new networks/VMs on the upgraded host)
9. Rinse & repeat with the remaining XenServer pool members in the ACS cluster.
* WITH THIS EXCEPTION: No need to manipulate management of the cluster in ACS
10. In XenCenter – if HA is required re-enable it
Now all of this completely bypasses steps that are spelled out here (which I *suspect* might now be “old”?):
* http://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/xenserver.html#upgrading-xenserver-versions
which asks one to run a lot of scripts (cleaning VLANs, preparing for upgrade etc.) in addition to copying files from the management server to the XenServer host (to set it back up again for ACS). I’m really hoping that step 11 saves me from that.
So (asks after all):
1. Is this a reasonable/viable plan?
2. Is my take on step 7 correct?
Thanks for taking a look,
David
David Merrill
Senior Systems Engineer,
Managed and Private/Hybrid Cloud Services
OTELCO
92 Oak Street, Portland ME 04101
office 207.772.5678<callto:207.772.5678>
http://www.otelco.com/cloud-and-managed-services
Confidentiality Message
The information contained in this e-mail transmission may be confidential and legally privileged. If you are not the intended recipient, you are notified that any dissemination, distribution, copying or other use of this information, including attachments, is prohibited. If you received this message in error, please call me at 207.772.5678<callto:207.772.5678> so this error can be corrected.
Re: Upgrading XenServer Clusters managed by ACS...
Posted by Andrija Panic <an...@gmail.com>.
What David said ^^^ above.
IMO, don't even bother without having a LAB first - you are making HUGE
jumps with both hypervisor and ACS version, and that needs to be tested
"the hell out of it" to put it that way - 4.11 sounds like a feasible
mid-version. XCP-ng needs to be 7.4 or 7.6 at most (please confirm in the
Release notes - and also please test upgrading XS to XCP-ng, as this is
again an exercise on it's own)
And please note 8.2 will be only supported in 4.15.1 which is not yet out -
8.1 is the latest one you can use in 4.15.0.0
Read the 4.11 Release notes to see if 6.2.0 is still supported (not being
officially supported, doesn't mean there is any piece of code really
removed from ACS - so if 6.5 is supported, 6.2 should also work 99.99% -
again, a thing to confirm in lab)
Best,
On Wed, 31 Mar 2021 at 19:33, David Merrill <da...@otelco.com>
wrote:
> I suspect the answer is a qualified "maybe/yes"? A couple (humble)
> thoughts as it sounds like you've got the two upgrades in mind:
>
> - focus first on the upgrade path to the latest ACS 4.*.* that still
> supports XS 6.2.0
> - determine the upgrade path from XS 6.2.0 to XCP 8.2 (not sure - just
> something work considering - maybe you'd need/want to upgrade to XS 6.5/7.*
> first?)
> - (sorry to be pedantic), lab it all up & test before doing anything in
> production!
>
> I'd defer to Andrija (& the rest of the list!) for any other advice.
>
> David Merrill
> Senior Systems Engineer,
> Managed and Private/Hybrid Cloud Services
> OTELCO
> 92 Oak Street, Portland ME 04101
> office 207.772.5678 <callto:207.772.5678>
> http://www.otelco.com/cloud-and-managed-services
>
> Confidentiality Message
> The information contained in this e-mail transmission may be confidential
> and legally privileged. If you are not the intended recipient, you are
> notified that any dissemination, distribution, copying or other use of this
> information, including attachments, is prohibited. If you received this
> message in error, please call me at 207.772.5678 <callto:207.772.5678> so
> this error can be corrected.
>
>
> On 3/31/21, 6:21 AM, "benoit lair" <ku...@gmail.com> wrote:
>
> Hello David,
>
> We have an ACS 4.3 install with somes XS 6.2.0 clusters.
> Would you think we could perform an upgrade from these XS 6.2.0 to ACS
> 4.11
> with the same way ?
>
> The finality would be to move our ACS 4.3 to ACS 4.15 in order to
> convert
> in fine our XS 6.2 to XCP 8.2
>
> @Andrija, would you think this could be achieved ?*
>
> Thanks a lot
>
> Le mar. 14 juil. 2020 à 04:34, David Merrill <da...@otelco.com>
> a
> écrit :
>
> > Reporting in on this - turns out was fairly painless (I never ended
> up
> > goofing around with host tags at all).
> >
> >
> >
> > Here's an updated to-do list (with some observations):
> >
> >
> >
> > 1. In XenCenter – if HA is enabled for the XenServer pool,
> disable it
> > 2. Stop ACS management/usage services
> > 3. Do MySQL database backups
> > 4. Start ACS management/usage services
> > 5. Start with the pool master
> > * In ACS – Put the pool master into maintenance mode
> (this
> > migrate all guest VMs to other hosts in the cluster)
> > * In ACS – Un-manage the cluster (this keeps any
> activity
> > from happening in the pool)
> > * NOW – Upgrade the XenServer pool master to the
> latest
> > release
> >
> > i. Do this by picking up the
> “correct”
> > ISO from Citrix, burning it to a CD/DVD/USB-stick, booting the host
> with it
> > & performing a manual upgrade to the version of XenServer you’re
> going to
> >
> > ii. Before upgrading the installer
> should
> > make a backup of the existing installation
> >
> > iii. When the upgrade is complete
> you’ll
> > be prompted to reboot the host
> >
> > iv. OBSERVATION (after booting):
> >
> > * The XenServer console (the ncurses interface)
> > still said XenServer 6.5 in the upper-left-hand corner (which led me
> to
> > believe that the upgrade hadn’t worked)
> > * However when I reconnected with XenCenter it
> > reported XenServer 7.1 CU2 was installed
> > * So…OK, fine?
> > * There were no host tags for that newly
> upgraded
> > host, so my to-do to remove the tag wasn’t necessary
> > * In ACS – Re-manage the cluster
> > * In ACS – Exit maintenance-mode for the newly
> upgraded
> > host
> > * In ACS – Observe that the newly upgraded host is
> > “Enabled” and “Up” in the UI (Infrastructure > Hosts)
> > * OBSERVATION:
> >
> > i. After finishing the 2 steps
> above on
> > checking in XenCenter the host now had the host tag:
> >
> vmops-version-com.cloud.hypervisor.xenerver.resource.XenServer650Resource-4.11.3.0
> >
> > ii. Scripts in /opt/cloud/bin have a
> > timestamp that coincides with the cluster being re-managed and the
> pool
> > master coming out of maintenance mode
> >
> > 1. In ACS – Testing (e.g. move an existing router/VM to the
> upgraded
> > host, create new networks/VMs on the upgraded host)
> >
> > i. OBSERVATION:
> >
> > * Moving existing router/VMs to the upgraded
> host
> > worked
> > * I was not able to create new VMs until all
> pool
> > members were at the same level of XenServer
> > 1. Rinse & repeat with the remaining XenServer pool members in
> the ACS
> > cluster
> > * Follow the same steps as the pool master EXCEPT do
> not
> > un-manage/re-manage the cluster in ACS (no need to do so really
> although
> > from the perspective of operators new VM creation is clearly not
> possible
> > until were done and who knows maybe you don’t really want folks
> trying to
> > take actions while you’re in the middle of all this?)
> > * OBSERVATION (unexpected):
> >
> > i. I noticed that even before I
> had
> > brought a newly upgraded pool member out of maintenance in ACS that
> the
> > following host tag
> >
> vmops-version-com.cloud.hypervisor.xenerver.resource.XenServer650Resource-4.11.3.0
> > was already there
> >
> > ii. AND that the Scripts in
> > /opt/cloud/bin had a timestamp that coincides with the pool member’s
> recent
> > reboot
> >
> > 1. In XenCenter – if HA was enabled at the start, re-enable it
> >
> >
> >
> > So my lab pool is up and running upgraded from XenServer 6.5 to
> XenServer
> > 7.1.2 CU2 LTSR and so far, CloudStack 4.11.3 seems to be happy with
> it.
> >
> >
> >
> > Next steps are to apply the latest XenServer hotfixes (following the
> same
> > recipe above) and re-test activities in ACS.
> >
> >
> >
> > Thanks,
> >
> > David
> >
> >
> >
> > David Merrill
> >
> > Senior Systems Engineer,
> >
> > Managed and Private/Hybrid Cloud Services
> >
> > OTELCO
> >
> > 92 Oak Street, Portland ME 04101
> >
> > office 207.772.5678 <callto:207.772.5678>
> >
> > http://www.otelco.com/cloud-and-managed-services
> >
> > Confidentiality Message
> >
> > The information contained in this e-mail transmission may be
> confidential
> > and legally privileged. If you are not the intended recipient, you
> are
> > notified that any dissemination, distribution, copying or other use
> of this
> > information, including attachments, is prohibited. If you received
> this
> > message in error, please call me at 207.772.5678
> <callto:207.772.5678> so
> > this error can be corrected.
> >
> >
> >
> >
> >
> >
> >
>
>
>
--
Andrija Panić
Re: Upgrading XenServer Clusters managed by ACS...
Posted by David Merrill <da...@otelco.com>.
I suspect the answer is a qualified "maybe/yes"? A couple (humble) thoughts as it sounds like you've got the two upgrades in mind:
- focus first on the upgrade path to the latest ACS 4.*.* that still supports XS 6.2.0
- determine the upgrade path from XS 6.2.0 to XCP 8.2 (not sure - just something work considering - maybe you'd need/want to upgrade to XS 6.5/7.* first?)
- (sorry to be pedantic), lab it all up & test before doing anything in production!
I'd defer to Andrija (& the rest of the list!) for any other advice.
David Merrill
Senior Systems Engineer,
Managed and Private/Hybrid Cloud Services
OTELCO
92 Oak Street, Portland ME 04101
office 207.772.5678 <callto:207.772.5678>
http://www.otelco.com/cloud-and-managed-services
Confidentiality Message
The information contained in this e-mail transmission may be confidential and legally privileged. If you are not the intended recipient, you are notified that any dissemination, distribution, copying or other use of this information, including attachments, is prohibited. If you received this message in error, please call me at 207.772.5678 <callto:207.772.5678> so this error can be corrected.
On 3/31/21, 6:21 AM, "benoit lair" <ku...@gmail.com> wrote:
Hello David,
We have an ACS 4.3 install with somes XS 6.2.0 clusters.
Would you think we could perform an upgrade from these XS 6.2.0 to ACS 4.11
with the same way ?
The finality would be to move our ACS 4.3 to ACS 4.15 in order to convert
in fine our XS 6.2 to XCP 8.2
@Andrija, would you think this could be achieved ?*
Thanks a lot
Le mar. 14 juil. 2020 à 04:34, David Merrill <da...@otelco.com> a
écrit :
> Reporting in on this - turns out was fairly painless (I never ended up
> goofing around with host tags at all).
>
>
>
> Here's an updated to-do list (with some observations):
>
>
>
> 1. In XenCenter – if HA is enabled for the XenServer pool, disable it
> 2. Stop ACS management/usage services
> 3. Do MySQL database backups
> 4. Start ACS management/usage services
> 5. Start with the pool master
> * In ACS – Put the pool master into maintenance mode (this
> migrate all guest VMs to other hosts in the cluster)
> * In ACS – Un-manage the cluster (this keeps any activity
> from happening in the pool)
> * NOW – Upgrade the XenServer pool master to the latest
> release
>
> i. Do this by picking up the “correct”
> ISO from Citrix, burning it to a CD/DVD/USB-stick, booting the host with it
> & performing a manual upgrade to the version of XenServer you’re going to
>
> ii. Before upgrading the installer should
> make a backup of the existing installation
>
> iii. When the upgrade is complete you’ll
> be prompted to reboot the host
>
> iv. OBSERVATION (after booting):
>
> * The XenServer console (the ncurses interface)
> still said XenServer 6.5 in the upper-left-hand corner (which led me to
> believe that the upgrade hadn’t worked)
> * However when I reconnected with XenCenter it
> reported XenServer 7.1 CU2 was installed
> * So…OK, fine?
> * There were no host tags for that newly upgraded
> host, so my to-do to remove the tag wasn’t necessary
> * In ACS – Re-manage the cluster
> * In ACS – Exit maintenance-mode for the newly upgraded
> host
> * In ACS – Observe that the newly upgraded host is
> “Enabled” and “Up” in the UI (Infrastructure > Hosts)
> * OBSERVATION:
>
> i. After finishing the 2 steps above on
> checking in XenCenter the host now had the host tag:
> vmops-version-com.cloud.hypervisor.xenerver.resource.XenServer650Resource-4.11.3.0
>
> ii. Scripts in /opt/cloud/bin have a
> timestamp that coincides with the cluster being re-managed and the pool
> master coming out of maintenance mode
>
> 1. In ACS – Testing (e.g. move an existing router/VM to the upgraded
> host, create new networks/VMs on the upgraded host)
>
> i. OBSERVATION:
>
> * Moving existing router/VMs to the upgraded host
> worked
> * I was not able to create new VMs until all pool
> members were at the same level of XenServer
> 1. Rinse & repeat with the remaining XenServer pool members in the ACS
> cluster
> * Follow the same steps as the pool master EXCEPT do not
> un-manage/re-manage the cluster in ACS (no need to do so really although
> from the perspective of operators new VM creation is clearly not possible
> until were done and who knows maybe you don’t really want folks trying to
> take actions while you’re in the middle of all this?)
> * OBSERVATION (unexpected):
>
> i. I noticed that even before I had
> brought a newly upgraded pool member out of maintenance in ACS that the
> following host tag
> vmops-version-com.cloud.hypervisor.xenerver.resource.XenServer650Resource-4.11.3.0
> was already there
>
> ii. AND that the Scripts in
> /opt/cloud/bin had a timestamp that coincides with the pool member’s recent
> reboot
>
> 1. In XenCenter – if HA was enabled at the start, re-enable it
>
>
>
> So my lab pool is up and running upgraded from XenServer 6.5 to XenServer
> 7.1.2 CU2 LTSR and so far, CloudStack 4.11.3 seems to be happy with it.
>
>
>
> Next steps are to apply the latest XenServer hotfixes (following the same
> recipe above) and re-test activities in ACS.
>
>
>
> Thanks,
>
> David
>
>
>
> David Merrill
>
> Senior Systems Engineer,
>
> Managed and Private/Hybrid Cloud Services
>
> OTELCO
>
> 92 Oak Street, Portland ME 04101
>
> office 207.772.5678 <callto:207.772.5678>
>
> http://www.otelco.com/cloud-and-managed-services
>
> Confidentiality Message
>
> The information contained in this e-mail transmission may be confidential
> and legally privileged. If you are not the intended recipient, you are
> notified that any dissemination, distribution, copying or other use of this
> information, including attachments, is prohibited. If you received this
> message in error, please call me at 207.772.5678 <callto:207.772.5678> so
> this error can be corrected.
>
>
>
>
>
>
>
Re: Upgrading XenServer Clusters managed by ACS...
Posted by benoit lair <ku...@gmail.com>.
Hello David,
We have an ACS 4.3 install with somes XS 6.2.0 clusters.
Would you think we could perform an upgrade from these XS 6.2.0 to ACS 4.11
with the same way ?
The finality would be to move our ACS 4.3 to ACS 4.15 in order to convert
in fine our XS 6.2 to XCP 8.2
@Andrija, would you think this could be achieved ?*
Thanks a lot
Le mar. 14 juil. 2020 à 04:34, David Merrill <da...@otelco.com> a
écrit :
> Reporting in on this - turns out was fairly painless (I never ended up
> goofing around with host tags at all).
>
>
>
> Here's an updated to-do list (with some observations):
>
>
>
> 1. In XenCenter – if HA is enabled for the XenServer pool, disable it
> 2. Stop ACS management/usage services
> 3. Do MySQL database backups
> 4. Start ACS management/usage services
> 5. Start with the pool master
> * In ACS – Put the pool master into maintenance mode (this
> migrate all guest VMs to other hosts in the cluster)
> * In ACS – Un-manage the cluster (this keeps any activity
> from happening in the pool)
> * NOW – Upgrade the XenServer pool master to the latest
> release
>
> i. Do this by picking up the “correct”
> ISO from Citrix, burning it to a CD/DVD/USB-stick, booting the host with it
> & performing a manual upgrade to the version of XenServer you’re going to
>
> ii. Before upgrading the installer should
> make a backup of the existing installation
>
> iii. When the upgrade is complete you’ll
> be prompted to reboot the host
>
> iv. OBSERVATION (after booting):
>
> * The XenServer console (the ncurses interface)
> still said XenServer 6.5 in the upper-left-hand corner (which led me to
> believe that the upgrade hadn’t worked)
> * However when I reconnected with XenCenter it
> reported XenServer 7.1 CU2 was installed
> * So…OK, fine?
> * There were no host tags for that newly upgraded
> host, so my to-do to remove the tag wasn’t necessary
> * In ACS – Re-manage the cluster
> * In ACS – Exit maintenance-mode for the newly upgraded
> host
> * In ACS – Observe that the newly upgraded host is
> “Enabled” and “Up” in the UI (Infrastructure > Hosts)
> * OBSERVATION:
>
> i. After finishing the 2 steps above on
> checking in XenCenter the host now had the host tag:
> vmops-version-com.cloud.hypervisor.xenerver.resource.XenServer650Resource-4.11.3.0
>
> ii. Scripts in /opt/cloud/bin have a
> timestamp that coincides with the cluster being re-managed and the pool
> master coming out of maintenance mode
>
> 1. In ACS – Testing (e.g. move an existing router/VM to the upgraded
> host, create new networks/VMs on the upgraded host)
>
> i. OBSERVATION:
>
> * Moving existing router/VMs to the upgraded host
> worked
> * I was not able to create new VMs until all pool
> members were at the same level of XenServer
> 1. Rinse & repeat with the remaining XenServer pool members in the ACS
> cluster
> * Follow the same steps as the pool master EXCEPT do not
> un-manage/re-manage the cluster in ACS (no need to do so really although
> from the perspective of operators new VM creation is clearly not possible
> until were done and who knows maybe you don’t really want folks trying to
> take actions while you’re in the middle of all this?)
> * OBSERVATION (unexpected):
>
> i. I noticed that even before I had
> brought a newly upgraded pool member out of maintenance in ACS that the
> following host tag
> vmops-version-com.cloud.hypervisor.xenerver.resource.XenServer650Resource-4.11.3.0
> was already there
>
> ii. AND that the Scripts in
> /opt/cloud/bin had a timestamp that coincides with the pool member’s recent
> reboot
>
> 1. In XenCenter – if HA was enabled at the start, re-enable it
>
>
>
> So my lab pool is up and running upgraded from XenServer 6.5 to XenServer
> 7.1.2 CU2 LTSR and so far, CloudStack 4.11.3 seems to be happy with it.
>
>
>
> Next steps are to apply the latest XenServer hotfixes (following the same
> recipe above) and re-test activities in ACS.
>
>
>
> Thanks,
>
> David
>
>
>
> David Merrill
>
> Senior Systems Engineer,
>
> Managed and Private/Hybrid Cloud Services
>
> OTELCO
>
> 92 Oak Street, Portland ME 04101
>
> office 207.772.5678 <callto:207.772.5678>
>
> http://www.otelco.com/cloud-and-managed-services
>
> Confidentiality Message
>
> The information contained in this e-mail transmission may be confidential
> and legally privileged. If you are not the intended recipient, you are
> notified that any dissemination, distribution, copying or other use of this
> information, including attachments, is prohibited. If you received this
> message in error, please call me at 207.772.5678 <callto:207.772.5678> so
> this error can be corrected.
>
>
>
>
>
>
>
Re: Upgrading XenServer Clusters managed by ACS...
Posted by David Merrill <da...@otelco.com>.
Reporting in on this - turns out was fairly painless (I never ended up goofing around with host tags at all).
Here's an updated to-do list (with some observations):
1. In XenCenter – if HA is enabled for the XenServer pool, disable it
2. Stop ACS management/usage services
3. Do MySQL database backups
4. Start ACS management/usage services
5. Start with the pool master
* In ACS – Put the pool master into maintenance mode (this migrate all guest VMs to other hosts in the cluster)
* In ACS – Un-manage the cluster (this keeps any activity from happening in the pool)
* NOW – Upgrade the XenServer pool master to the latest release
i. Do this by picking up the “correct” ISO from Citrix, burning it to a CD/DVD/USB-stick, booting the host with it & performing a manual upgrade to the version of XenServer you’re going to
ii. Before upgrading the installer should make a backup of the existing installation
iii. When the upgrade is complete you’ll be prompted to reboot the host
iv. OBSERVATION (after booting):
* The XenServer console (the ncurses interface) still said XenServer 6.5 in the upper-left-hand corner (which led me to believe that the upgrade hadn’t worked)
* However when I reconnected with XenCenter it reported XenServer 7.1 CU2 was installed
* So…OK, fine?
* There were no host tags for that newly upgraded host, so my to-do to remove the tag wasn’t necessary
* In ACS – Re-manage the cluster
* In ACS – Exit maintenance-mode for the newly upgraded host
* In ACS – Observe that the newly upgraded host is “Enabled” and “Up” in the UI (Infrastructure > Hosts)
* OBSERVATION:
i. After finishing the 2 steps above on checking in XenCenter the host now had the host tag: vmops-version-com.cloud.hypervisor.xenerver.resource.XenServer650Resource-4.11.3.0
ii. Scripts in /opt/cloud/bin have a timestamp that coincides with the cluster being re-managed and the pool master coming out of maintenance mode
1. In ACS – Testing (e.g. move an existing router/VM to the upgraded host, create new networks/VMs on the upgraded host)
i. OBSERVATION:
* Moving existing router/VMs to the upgraded host worked
* I was not able to create new VMs until all pool members were at the same level of XenServer
1. Rinse & repeat with the remaining XenServer pool members in the ACS cluster
* Follow the same steps as the pool master EXCEPT do not un-manage/re-manage the cluster in ACS (no need to do so really although from the perspective of operators new VM creation is clearly not possible until were done and who knows maybe you don’t really want folks trying to take actions while you’re in the middle of all this?)
* OBSERVATION (unexpected):
i. I noticed that even before I had brought a newly upgraded pool member out of maintenance in ACS that the following host tag vmops-version-com.cloud.hypervisor.xenerver.resource.XenServer650Resource-4.11.3.0 was already there
ii. AND that the Scripts in /opt/cloud/bin had a timestamp that coincides with the pool member’s recent reboot
1. In XenCenter – if HA was enabled at the start, re-enable it
So my lab pool is up and running upgraded from XenServer 6.5 to XenServer 7.1.2 CU2 LTSR and so far, CloudStack 4.11.3 seems to be happy with it.
Next steps are to apply the latest XenServer hotfixes (following the same recipe above) and re-test activities in ACS.
Thanks,
David
David Merrill
Senior Systems Engineer,
Managed and Private/Hybrid Cloud Services
OTELCO
92 Oak Street, Portland ME 04101
office 207.772.5678 <callto:207.772.5678>
http://www.otelco.com/cloud-and-managed-services
Confidentiality Message
The information contained in this e-mail transmission may be confidential and legally privileged. If you are not the intended recipient, you are notified that any dissemination, distribution, copying or other use of this information, including attachments, is prohibited. If you received this message in error, please call me at 207.772.5678 <callto:207.772.5678> so this error can be corrected.
Re: Upgrading XenServer Clusters managed by ACS...
Posted by David Merrill <da...@otelco.com>.
Andrija,
I appreciate you sharing your thoughts on this. I'm getting my ducks in a row on the XenServer side (for the upgrade) & I'll give the plan a go.
I'll report in afterwards, wish me luck!
Thanks,
David
David Merrill
Senior Systems Engineer,
Managed and Private/Hybrid Cloud Services
OTELCO
92 Oak Street, Portland ME 04101
office 207.772.5678 <callto:207.772.5678>
http://www.otelco.com/cloud-and-managed-services
Confidentiality Message
The information contained in this e-mail transmission may be confidential and legally privileged. If you are not the intended recipient, you are notified that any dissemination, distribution, copying or other use of this information, including attachments, is prohibited. If you received this message in error, please call me at 207.772.5678 <callto:207.772.5678> so this error can be corrected.
On 6/30/20, 1:03 PM, "Andrija Panic" <an...@gmail.com> wrote:
Hi David,
with a bit of delay...
those steps need to be tested - I would skip the whole
"environment.properties" file and see how it behaves today - the reason
being that, though the article on shapeblue.com was old, it does mention
both "Unmanage" cluster and Maintenace mode - so I'm not quite sure what is
the difference today vs. how it behaved back in time of ACS 4.3 / XS 6.2 -
the explanation that you've found on the mailing thread may not make sense,
I mean specifically the "There wasn't an 'unmanage' button at the time",
since the article clearly mentions it).
There is also no need to manually migrate VM's away from a host (i.e. pool
master) - simply put it into the Maintenance mode and it will move VMs away
to other hosts.
Assuming that putting the pool-master into Maintenance mode in ACS will,
these days, NOT trigger a new host to become a master, your steps look fine.
For the records, I've updated&&tested the official documentation, for
XenServer 6.5+ : https://github.com/apache/cloudstack-documentation/pull/140
/
https://acs-www.shapeblue.com/docs/WIP-PROOFING/pr140/installguide/hypervisor/xenserver.html#upgrading-xenserver-versions
-
i.e. removed unneeded steps and explained what each script is doing. This
guide assumes a much longer management plane downtime, as the cluster is
unmanaged while you update all the hosts in the pool. Works fine also for
XCP-ng upgrades, etc.
Either way, I prefer doing it based on the plan you laid out here.
Regards,
Andrija
On Fri, 19 Jun 2020 at 23:24, David Merrill <da...@otelco.com>
wrote:
> Hi All,
>
> I have a production deployment of ACS managing three XenServer Clusters
> (XenServer pools of 6 hosts each) in two different Zones. I now find myself
> in the position of needing to do a major version upgrade of those hosts.
> Happily I have a ACS lab managing a XenServer cluster running the same
> (old) version of XenServer that I can practice on.
>
> I have plenty of practice operating ACS to “quiesce things” for XenServer
> patches (start with the pool master, move guest VMs off, put that host into
> maintenance mode, unmanage the cluster, patch the host, then reverse & move
> onto the next host with the same steps except we don’t bother
> w/un-managing/re-managing the cluster), but as I understand a XenServer
> version upgrade backs up the whole original XenServer installation to
> another partition and makes a clean installation of XenServer on the
> original partition (the problem there being that when the upgraded
> XenServer boots up all the ACS provided/copied scripts are not there & ACS
> can’t manage the host).
>
> So not much of an ask here (OK maybe at the end – have I missed something
> obvious or doing anything foolish?), I wanted to share a bit research & lay
> out a set of steps that I think will work to get a pool of XenServers in a
> cluster upgraded and end up in a place where ACS is happy with them.
>
> Bear with me it’s a little long,
>
>
> 1. In XenCenter – if HA is enabled for the XenServer pool, disable it
> 2. Stop ACS management/usage services
> 3. Do MySQL database backups
> 4. Start ACS management/usage services
> 5. Start with the pool master.
> 6. In ACS – Migrate all guest VMs to other hosts in the cluster
> 7. In ACS – Prepare to put the pool master into maintenance mode (so no
> new guest VMs)
> * A caveat here related to this item I found when researching –
> https://www.shapeblue.com/how-to-upgrade-an-apache-cloudstack-citrix-xenserver-cluster/
>
> i. A
> recommendation is made here to edit
> /etc/cloudstack/management/environment.properties
>
> ii. And
> set manage.xenserver.pool.master=false
>
> iii. And
> restart CloudStack management services
>
> iv.
> BECAUSE if one didn’t I understand CloudStack WOULD force an election for
> another host to become the pool master (which is “bad” as ASCs is
> configured to speak to the currently configured pool master)
>
> * HOWEVER THIS MAY NOT BE NECESSARY
>
> i.
> Found a thread titled “A Story of a failed XenServer Upgrade” here –
> http://mail-archives.apache.org/mod_mbox/cloudstack-users/201601.mbox/browser
>
> ii. At
> the end of the thread Paul Angus states that Geoff’s ShapeBlue blog article
> was written in the ACS 4.3 era and that ACS’ behavior “used to be that
> putting a host that was the pool master into maintenance would cause
> CloudStack to force an election for another host to become pool master -
> stopping you from then upgrading the active pool master first. There wasn't
> an 'unmanage' button at the time.”
>
> iii. The
> implication, (in my humble estimation) is that, today, one no longer needs
> to make these edits to /etc/cloudstack/management/environment.properties
>
> 1. In ACS – Put the pool master into maintenance mode (so no new guest
> VMs)
> 2. In ACS – Un-manage the cluster
> 3. NOW – Upgrade the XenServer pool master to the latest release
> * Exercise left to the reader…(I’ll share what I end up doing)
> 4. In XenServer – Do the following (found this nugget in a thread last
> November about XenServer upgrades & how to re-setup the host – thanks
> Richard Lawley!)
> * Remove this host tag: xe host-param-remove uuid=HOSTUUID
> param-name=tags
> param-key=vmops-version-com.cloud.hypervisor.xenserver.resource.XenServer650R
> * This makes management server set the host back up, presumably
> since ACS has credentials to the host in the database it can copy all the
> scripts back
> 5. In ACS – Re-manage the cluster
> 6. In ACS – Exit maintenance-mode for the newly upgraded host
> 7. In ACS – Observe that the newly upgraded host is “Enabled” and “Up”
> in the UI (Infrastructure --> Hosts)
> 8. In ACS – Testing (e.g. move an existing router/VM to the upgraded
> host, create new networks/VMs on the upgraded host)
> 9. Rinse & repeat with the remaining XenServer pool members in the ACS
> cluster.
> * WITH THIS EXCEPTION: No need to manipulate management of the
> cluster in ACS
> 10. In XenCenter – if HA is required re-enable it
>
> Now all of this completely bypasses steps that are spelled out here (which
> I *suspect* might now be “old”?):
>
>
> *
> http://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/xenserver.html#upgrading-xenserver-versions
>
> which asks one to run a lot of scripts (cleaning VLANs, preparing for
> upgrade etc.) in addition to copying files from the management server to
> the XenServer host (to set it back up again for ACS). I’m really hoping
> that step 11 saves me from that.
>
> So (asks after all):
>
>
> 1. Is this a reasonable/viable plan?
> 2. Is my take on step 7 correct?
>
> Thanks for taking a look,
> David
>
> David Merrill
> Senior Systems Engineer,
> Managed and Private/Hybrid Cloud Services
> OTELCO
> 92 Oak Street, Portland ME 04101
> office 207.772.5678<callto:207.772.5678>
> http://www.otelco.com/cloud-and-managed-services
>
> Confidentiality Message
> The information contained in this e-mail transmission may be confidential
> and legally privileged. If you are not the intended recipient, you are
> notified that any dissemination, distribution, copying or other use of this
> information, including attachments, is prohibited. If you received this
> message in error, please call me at 207.772.5678<callto:207.772.5678> so
> this error can be corrected.
>
>
--
Andrija Panić
Re: Upgrading XenServer Clusters managed by ACS...
Posted by Andrija Panic <an...@gmail.com>.
Hi David,
with a bit of delay...
those steps need to be tested - I would skip the whole
"environment.properties" file and see how it behaves today - the reason
being that, though the article on shapeblue.com was old, it does mention
both "Unmanage" cluster and Maintenace mode - so I'm not quite sure what is
the difference today vs. how it behaved back in time of ACS 4.3 / XS 6.2 -
the explanation that you've found on the mailing thread may not make sense,
I mean specifically the "There wasn't an 'unmanage' button at the time",
since the article clearly mentions it).
There is also no need to manually migrate VM's away from a host (i.e. pool
master) - simply put it into the Maintenance mode and it will move VMs away
to other hosts.
Assuming that putting the pool-master into Maintenance mode in ACS will,
these days, NOT trigger a new host to become a master, your steps look fine.
For the records, I've updated&&tested the official documentation, for
XenServer 6.5+ : https://github.com/apache/cloudstack-documentation/pull/140
/
https://acs-www.shapeblue.com/docs/WIP-PROOFING/pr140/installguide/hypervisor/xenserver.html#upgrading-xenserver-versions
-
i.e. removed unneeded steps and explained what each script is doing. This
guide assumes a much longer management plane downtime, as the cluster is
unmanaged while you update all the hosts in the pool. Works fine also for
XCP-ng upgrades, etc.
Either way, I prefer doing it based on the plan you laid out here.
Regards,
Andrija
On Fri, 19 Jun 2020 at 23:24, David Merrill <da...@otelco.com>
wrote:
> Hi All,
>
> I have a production deployment of ACS managing three XenServer Clusters
> (XenServer pools of 6 hosts each) in two different Zones. I now find myself
> in the position of needing to do a major version upgrade of those hosts.
> Happily I have a ACS lab managing a XenServer cluster running the same
> (old) version of XenServer that I can practice on.
>
> I have plenty of practice operating ACS to “quiesce things” for XenServer
> patches (start with the pool master, move guest VMs off, put that host into
> maintenance mode, unmanage the cluster, patch the host, then reverse & move
> onto the next host with the same steps except we don’t bother
> w/un-managing/re-managing the cluster), but as I understand a XenServer
> version upgrade backs up the whole original XenServer installation to
> another partition and makes a clean installation of XenServer on the
> original partition (the problem there being that when the upgraded
> XenServer boots up all the ACS provided/copied scripts are not there & ACS
> can’t manage the host).
>
> So not much of an ask here (OK maybe at the end – have I missed something
> obvious or doing anything foolish?), I wanted to share a bit research & lay
> out a set of steps that I think will work to get a pool of XenServers in a
> cluster upgraded and end up in a place where ACS is happy with them.
>
> Bear with me it’s a little long,
>
>
> 1. In XenCenter – if HA is enabled for the XenServer pool, disable it
> 2. Stop ACS management/usage services
> 3. Do MySQL database backups
> 4. Start ACS management/usage services
> 5. Start with the pool master.
> 6. In ACS – Migrate all guest VMs to other hosts in the cluster
> 7. In ACS – Prepare to put the pool master into maintenance mode (so no
> new guest VMs)
> * A caveat here related to this item I found when researching –
> https://www.shapeblue.com/how-to-upgrade-an-apache-cloudstack-citrix-xenserver-cluster/
>
> i. A
> recommendation is made here to edit
> /etc/cloudstack/management/environment.properties
>
> ii. And
> set manage.xenserver.pool.master=false
>
> iii. And
> restart CloudStack management services
>
> iv.
> BECAUSE if one didn’t I understand CloudStack WOULD force an election for
> another host to become the pool master (which is “bad” as ASCs is
> configured to speak to the currently configured pool master)
>
> * HOWEVER THIS MAY NOT BE NECESSARY
>
> i.
> Found a thread titled “A Story of a failed XenServer Upgrade” here –
> http://mail-archives.apache.org/mod_mbox/cloudstack-users/201601.mbox/browser
>
> ii. At
> the end of the thread Paul Angus states that Geoff’s ShapeBlue blog article
> was written in the ACS 4.3 era and that ACS’ behavior “used to be that
> putting a host that was the pool master into maintenance would cause
> CloudStack to force an election for another host to become pool master -
> stopping you from then upgrading the active pool master first. There wasn't
> an 'unmanage' button at the time.”
>
> iii. The
> implication, (in my humble estimation) is that, today, one no longer needs
> to make these edits to /etc/cloudstack/management/environment.properties
>
> 1. In ACS – Put the pool master into maintenance mode (so no new guest
> VMs)
> 2. In ACS – Un-manage the cluster
> 3. NOW – Upgrade the XenServer pool master to the latest release
> * Exercise left to the reader…(I’ll share what I end up doing)
> 4. In XenServer – Do the following (found this nugget in a thread last
> November about XenServer upgrades & how to re-setup the host – thanks
> Richard Lawley!)
> * Remove this host tag: xe host-param-remove uuid=HOSTUUID
> param-name=tags
> param-key=vmops-version-com.cloud.hypervisor.xenserver.resource.XenServer650R
> * This makes management server set the host back up, presumably
> since ACS has credentials to the host in the database it can copy all the
> scripts back
> 5. In ACS – Re-manage the cluster
> 6. In ACS – Exit maintenance-mode for the newly upgraded host
> 7. In ACS – Observe that the newly upgraded host is “Enabled” and “Up”
> in the UI (Infrastructure --> Hosts)
> 8. In ACS – Testing (e.g. move an existing router/VM to the upgraded
> host, create new networks/VMs on the upgraded host)
> 9. Rinse & repeat with the remaining XenServer pool members in the ACS
> cluster.
> * WITH THIS EXCEPTION: No need to manipulate management of the
> cluster in ACS
> 10. In XenCenter – if HA is required re-enable it
>
> Now all of this completely bypasses steps that are spelled out here (which
> I *suspect* might now be “old”?):
>
>
> *
> http://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/xenserver.html#upgrading-xenserver-versions
>
> which asks one to run a lot of scripts (cleaning VLANs, preparing for
> upgrade etc.) in addition to copying files from the management server to
> the XenServer host (to set it back up again for ACS). I’m really hoping
> that step 11 saves me from that.
>
> So (asks after all):
>
>
> 1. Is this a reasonable/viable plan?
> 2. Is my take on step 7 correct?
>
> Thanks for taking a look,
> David
>
> David Merrill
> Senior Systems Engineer,
> Managed and Private/Hybrid Cloud Services
> OTELCO
> 92 Oak Street, Portland ME 04101
> office 207.772.5678<callto:207.772.5678>
> http://www.otelco.com/cloud-and-managed-services
>
> Confidentiality Message
> The information contained in this e-mail transmission may be confidential
> and legally privileged. If you are not the intended recipient, you are
> notified that any dissemination, distribution, copying or other use of this
> information, including attachments, is prohibited. If you received this
> message in error, please call me at 207.772.5678<callto:207.772.5678> so
> this error can be corrected.
>
>
--
Andrija Panić