Help with migrating primary storage (NFS) from Zone to Cluster scope

[Rohan T <ro...@gmail.com>]

Hi all,


We're attempting to make a production change to a CloudStack deployment
(currently installed at 4.4.1), where we want to change the scope of the
attached primary storage from being attached at ZONE scope to CLUSTER scope
(this deployment currently has exactly 1 zone, 1 pod and 1 cluster in use).

The purpose of the change is to allow us to deploy a second primary storage
device to a new cluster, providing no single points of failure within the
system.

(Note: adding 2 primary storage devices to the same zone has had the
opposite effect, effectively halving the reliability of the system since
each node considers failure of _any_ NFS primary storage to be sufficient
grounds to initiate a HA reboot event).

All VM's are KVM, all used storage is NFS shared storage.  There are dozens
of deployed VM's running and 20 hosts deployed.

The expectation is that this change would be made during a system
maintenance event (and shutdown), but the hope is that we can effect this
change without complete removal of the storage and associated redeployment
of all VMs.

I note that the design notes for 4.2 (where storage scope was introduced),
indicate that the major design change that needs to be accounted for was
the that a column, called scope, was added into storage_pool table.
(see:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Zone-wide+primary+storage+target
).

This suggests that the change could be as trivial as:
0. Shutdown of cloudstack management server.
1.  Identifying the primary storage device in cloud.storage_pool.
2.  Changing the 'scope' field from 'ZONE' to 'CLUSTER'.
3.  Populating the 'pod_id' and 'cluster_id' fields to reflect the
appropriate pod & cluster.
4.  Restart cloudstack.

Questions:

1. Has anyone attempted a similar migration?
2. Is this the way it's designed to work (in 4.4.1)?  Are there any other
values to be accounted for?
3. Is there another way to effect this through an API/UI? (We would
consider an upgrade if this is supported in a later release)?



Thanks in advance,

Rohan

Re: Help with migrating primary storage (NFS) from Zone to Cluster scope

[Rohan T <ro...@gmail.com>]

Hi All,


Responding to myself.  We've now completed a bunch of lab-testing,
including following Ilya's previous suggestions.

The following process is expected to work, (on a 'simple' deployment
1-zone, 1-pod, 1-cluster), and should be safe to perform live.

1.  Identifying the primary storage device in cloud.storage_pool.
2.  Populating the 'pod_id' and 'cluster_id' fields to reflect the
 appropriate pod & cluster for that device id.
3.  Changing the 'scope' field from 'ZONE' to 'CLUSTER'.
4. Identify the same host_id in op_host_capacity, and set pod_id and
cluster_id in that table as well.

i.e.  for a storage_pool with id=99, the following should be all the SQL
required:

update cloud.storage_pool set pod_id=1,cluster_id=1 where id=99;
update cloud.storage_pool set scope='CLUSTER' where id=99;
update cloud.op_host_capacity set pod_id=1,cluster_id=1 where host_id=99;

In the case that you have more than 1 cluster in your zone, you will also
need to pay attention to the cloud.storage_pool_host_ref table and adjust
records for hosts that are not in the cluster the primary storage device is
being re-scoped to serve.

I agree with Wayne, that this would be a reasonable feature to include in
the API/UI for future releases, as there are good reasons why this design
may be changed post-implementation.


Cheers,
Rohan


On Fri, Feb 12, 2016 at 10:21 AM, ilya <il...@gmail.com> wrote:

> Rohan
>
> This is the first time i'm seeing this question.
>
> Here is how i would try to migrate this over, ideally - i'd spend few
> hours setting up a demo lab environment to see how things work out.
>
> With that said, i believe it wont be too complicated, but i'd not do it
> in production without testing the lab first.
>
> In the lab, do the same setup as you have in prod, you can even do the
> whole lab setup on 1 VM, though few VMs would be ideal.
>
> Limit any changes to ACS
> Enable MYSQL Query logging, add cluster level storage
> Inspect any inserts and updates that a relevant
> Migrate global storage to cluster level scope storage in your lab
> Run basic tests to confirm it works
>
> Regards
> ilya
>
> On 2/10/16 7:39 PM, Rohan T wrote:
> > Hi all,
> >
> >
> > We're attempting to make a production change to a CloudStack deployment
> > (currently installed at 4.4.1), where we want to change the scope of the
> > attached primary storage from being attached at ZONE scope to CLUSTER
> scope
> > (this deployment currently has exactly 1 zone, 1 pod and 1 cluster in
> use).
> >
> > The purpose of the change is to allow us to deploy a second primary
> storage
> > device to a new cluster, providing no single points of failure within the
> > system.
> >
> > (Note: adding 2 primary storage devices to the same zone has had the
> > opposite effect, effectively halving the reliability of the system since
> > each node considers failure of _any_ NFS primary storage to be sufficient
> > grounds to initiate a HA reboot event).
> >
> > All VM's are KVM, all used storage is NFS shared storage.  There are
> dozens
> > of deployed VM's running and 20 hosts deployed.
> >
> > The expectation is that this change would be made during a system
> > maintenance event (and shutdown), but the hope is that we can effect this
> > change without complete removal of the storage and associated
> redeployment
> > of all VMs.
> >
> > I note that the design notes for 4.2 (where storage scope was
> introduced),
> > indicate that the major design change that needs to be accounted for was
> > the that a column, called scope, was added into storage_pool table.
> > (see:
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Zone-wide+primary+storage+target
> > ).
> >
> > This suggests that the change could be as trivial as:
> > 0. Shutdown of cloudstack management server.
> > 1.  Identifying the primary storage device in cloud.storage_pool.
> > 2.  Changing the 'scope' field from 'ZONE' to 'CLUSTER'.
> > 3.  Populating the 'pod_id' and 'cluster_id' fields to reflect the
> > appropriate pod & cluster.
> > 4.  Restart cloudstack.
> >
> > Questions:
> >
> > 1. Has anyone attempted a similar migration?
> > 2. Is this the way it's designed to work (in 4.4.1)?  Are there any other
> > values to be accounted for?
> > 3. Is there another way to effect this through an API/UI? (We would
> > consider an upgrade if this is supported in a later release)?
> >
> >
> >
> > Thanks in advance,
> >
> > Rohan
> >
>

Re: Help with migrating primary storage (NFS) from Zone to Cluster scope

[ilya <il...@gmail.com>]

Rohan

This is the first time i'm seeing this question.

Here is how i would try to migrate this over, ideally - i'd spend few
hours setting up a demo lab environment to see how things work out.

With that said, i believe it wont be too complicated, but i'd not do it
in production without testing the lab first.

In the lab, do the same setup as you have in prod, you can even do the
whole lab setup on 1 VM, though few VMs would be ideal.

Limit any changes to ACS
Enable MYSQL Query logging, add cluster level storage
Inspect any inserts and updates that a relevant
Migrate global storage to cluster level scope storage in your lab
Run basic tests to confirm it works

Regards
ilya

On 2/10/16 7:39 PM, Rohan T wrote:
> Hi all,
> 
> 
> We're attempting to make a production change to a CloudStack deployment
> (currently installed at 4.4.1), where we want to change the scope of the
> attached primary storage from being attached at ZONE scope to CLUSTER scope
> (this deployment currently has exactly 1 zone, 1 pod and 1 cluster in use).
> 
> The purpose of the change is to allow us to deploy a second primary storage
> device to a new cluster, providing no single points of failure within the
> system.
> 
> (Note: adding 2 primary storage devices to the same zone has had the
> opposite effect, effectively halving the reliability of the system since
> each node considers failure of _any_ NFS primary storage to be sufficient
> grounds to initiate a HA reboot event).
> 
> All VM's are KVM, all used storage is NFS shared storage.  There are dozens
> of deployed VM's running and 20 hosts deployed.
> 
> The expectation is that this change would be made during a system
> maintenance event (and shutdown), but the hope is that we can effect this
> change without complete removal of the storage and associated redeployment
> of all VMs.
> 
> I note that the design notes for 4.2 (where storage scope was introduced),
> indicate that the major design change that needs to be accounted for was
> the that a column, called scope, was added into storage_pool table.
> (see:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Zone-wide+primary+storage+target
> ).
> 
> This suggests that the change could be as trivial as:
> 0. Shutdown of cloudstack management server.
> 1.  Identifying the primary storage device in cloud.storage_pool.
> 2.  Changing the 'scope' field from 'ZONE' to 'CLUSTER'.
> 3.  Populating the 'pod_id' and 'cluster_id' fields to reflect the
> appropriate pod & cluster.
> 4.  Restart cloudstack.
> 
> Questions:
> 
> 1. Has anyone attempted a similar migration?
> 2. Is this the way it's designed to work (in 4.4.1)?  Are there any other
> values to be accounted for?
> 3. Is there another way to effect this through an API/UI? (We would
> consider an upgrade if this is supported in a later release)?
> 
> 
> 
> Thanks in advance,
> 
> Rohan
> 

Re: Help with migrating primary storage (NFS) from Zone to Cluster scope

[Dustin Wright <dw...@untangledtechnology.com>]

I have a similar need and would like to know how this change-over goes.

This seems like a common thing to do as you grow your cloud.

On Wed, Feb 10, 2016 at 10:39 PM, Rohan T <ro...@gmail.com> wrote:

> Hi all,
>
>
> We're attempting to make a production change to a CloudStack deployment
> (currently installed at 4.4.1), where we want to change the scope of the
> attached primary storage from being attached at ZONE scope to CLUSTER scope
> (this deployment currently has exactly 1 zone, 1 pod and 1 cluster in use).
>
> The purpose of the change is to allow us to deploy a second primary storage
> device to a new cluster, providing no single points of failure within the
> system.
>
> (Note: adding 2 primary storage devices to the same zone has had the
> opposite effect, effectively halving the reliability of the system since
> each node considers failure of _any_ NFS primary storage to be sufficient
> grounds to initiate a HA reboot event).
>
> All VM's are KVM, all used storage is NFS shared storage.  There are dozens
> of deployed VM's running and 20 hosts deployed.
>
> The expectation is that this change would be made during a system
> maintenance event (and shutdown), but the hope is that we can effect this
> change without complete removal of the storage and associated redeployment
> of all VMs.
>
> I note that the design notes for 4.2 (where storage scope was introduced),
> indicate that the major design change that needs to be accounted for was
> the that a column, called scope, was added into storage_pool table.
> (see:
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Zone-wide+primary+storage+target
> ).
>
> This suggests that the change could be as trivial as:
> 0. Shutdown of cloudstack management server.
> 1.  Identifying the primary storage device in cloud.storage_pool.
> 2.  Changing the 'scope' field from 'ZONE' to 'CLUSTER'.
> 3.  Populating the 'pod_id' and 'cluster_id' fields to reflect the
> appropriate pod & cluster.
> 4.  Restart cloudstack.
>
> Questions:
>
> 1. Has anyone attempted a similar migration?
> 2. Is this the way it's designed to work (in 4.4.1)?  Are there any other
> values to be accounted for?
> 3. Is there another way to effect this through an API/UI? (We would
> consider an upgrade if this is supported in a later release)?
>
>
>
> Thanks in advance,
>
> Rohan
>