Re: Help with migrating primary storage (NFS) from Zone to Cluster scope

[Rohan T <ro...@gmail.com>]

Hi All,


Responding to myself.  We've now completed a bunch of lab-testing,
including following Ilya's previous suggestions.

The following process is expected to work, (on a 'simple' deployment
1-zone, 1-pod, 1-cluster), and should be safe to perform live.

1.  Identifying the primary storage device in cloud.storage_pool.
2.  Populating the 'pod_id' and 'cluster_id' fields to reflect the
 appropriate pod & cluster for that device id.
3.  Changing the 'scope' field from 'ZONE' to 'CLUSTER'.
4. Identify the same host_id in op_host_capacity, and set pod_id and
cluster_id in that table as well.

i.e.  for a storage_pool with id=99, the following should be all the SQL
required:

update cloud.storage_pool set pod_id=1,cluster_id=1 where id=99;
update cloud.storage_pool set scope='CLUSTER' where id=99;
update cloud.op_host_capacity set pod_id=1,cluster_id=1 where host_id=99;

In the case that you have more than 1 cluster in your zone, you will also
need to pay attention to the cloud.storage_pool_host_ref table and adjust
records for hosts that are not in the cluster the primary storage device is
being re-scoped to serve.

I agree with Wayne, that this would be a reasonable feature to include in
the API/UI for future releases, as there are good reasons why this design
may be changed post-implementation.


Cheers,
Rohan


On Fri, Feb 12, 2016 at 10:21 AM, ilya <il...@gmail.com> wrote:

> Rohan
>
> This is the first time i'm seeing this question.
>
> Here is how i would try to migrate this over, ideally - i'd spend few
> hours setting up a demo lab environment to see how things work out.
>
> With that said, i believe it wont be too complicated, but i'd not do it
> in production without testing the lab first.
>
> In the lab, do the same setup as you have in prod, you can even do the
> whole lab setup on 1 VM, though few VMs would be ideal.
>
> Limit any changes to ACS
> Enable MYSQL Query logging, add cluster level storage
> Inspect any inserts and updates that a relevant
> Migrate global storage to cluster level scope storage in your lab
> Run basic tests to confirm it works
>
> Regards
> ilya
>
> On 2/10/16 7:39 PM, Rohan T wrote:
> > Hi all,
> >
> >
> > We're attempting to make a production change to a CloudStack deployment
> > (currently installed at 4.4.1), where we want to change the scope of the
> > attached primary storage from being attached at ZONE scope to CLUSTER
> scope
> > (this deployment currently has exactly 1 zone, 1 pod and 1 cluster in
> use).
> >
> > The purpose of the change is to allow us to deploy a second primary
> storage
> > device to a new cluster, providing no single points of failure within the
> > system.
> >
> > (Note: adding 2 primary storage devices to the same zone has had the
> > opposite effect, effectively halving the reliability of the system since
> > each node considers failure of _any_ NFS primary storage to be sufficient
> > grounds to initiate a HA reboot event).
> >
> > All VM's are KVM, all used storage is NFS shared storage.  There are
> dozens
> > of deployed VM's running and 20 hosts deployed.
> >
> > The expectation is that this change would be made during a system
> > maintenance event (and shutdown), but the hope is that we can effect this
> > change without complete removal of the storage and associated
> redeployment
> > of all VMs.
> >
> > I note that the design notes for 4.2 (where storage scope was
> introduced),
> > indicate that the major design change that needs to be accounted for was
> > the that a column, called scope, was added into storage_pool table.
> > (see:
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Zone-wide+primary+storage+target
> > ).
> >
> > This suggests that the change could be as trivial as:
> > 0. Shutdown of cloudstack management server.
> > 1.  Identifying the primary storage device in cloud.storage_pool.
> > 2.  Changing the 'scope' field from 'ZONE' to 'CLUSTER'.
> > 3.  Populating the 'pod_id' and 'cluster_id' fields to reflect the
> > appropriate pod & cluster.
> > 4.  Restart cloudstack.
> >
> > Questions:
> >
> > 1. Has anyone attempted a similar migration?
> > 2. Is this the way it's designed to work (in 4.4.1)?  Are there any other
> > values to be accounted for?
> > 3. Is there another way to effect this through an API/UI? (We would
> > consider an upgrade if this is supported in a later release)?
> >
> >
> >
> > Thanks in advance,
> >
> > Rohan
> >
>