[osuosl-openpower] NVMe failure affecting some VMs

[Lance Albertson <la...@osuosl.org>]

All,

One of our nodes which uses local storage with NVMe has had a failure and
all the VMs on that node are offline. Due to the way we had storage
configured on this node for performance, we're unlikely to save any of the
ephemeral disks. However if you had any volumes attached, those will still
be intact since they are on our Ceph cluster.

I'm going to be working on getting the NVMe replaced, in the meantime, if
you need us to rebuild any of your VMs, please let us know.

Thanks and sorry again for the issue.

-- 
Lance Albertson
Director
Oregon State University | Open Source Lab

Re: [osuosl-openpower] NVMe failure affecting some VMs

[Lance Albertson <la...@osuosl.org>]

All,

The system is back online and you should be able to recreate the VMs now. I
unfortunately needed to remove all current VMs to get the hypervisor back
in a sane state so using "rebuild" will not work right now.

I did change the storage so that the NVMe's are in a RAID1. I have a plan
to do the same with our other node that has NVMe, however I will be
migrating VMs off of it before making that change to a temporary node.

Again, if you need any help with us recreating the VMs, please send an
email to powerdev-request@osuosl.org and let us know what OS/specs you want.

Apologies again for this situation!

Thanks-

On Tue, Jan 14, 2025 at 11:03 AM Lance Albertson <la...@osuosl.org> wrote:

> All,
>
> One of our nodes which uses local storage with NVMe has had a failure and
> all the VMs on that node are offline. Due to the way we had storage
> configured on this node for performance, we're unlikely to save any of the
> ephemeral disks. However if you had any volumes attached, those will still
> be intact since they are on our Ceph cluster.
>
> I'm going to be working on getting the NVMe replaced, in the meantime, if
> you need us to rebuild any of your VMs, please let us know.
>
> Thanks and sorry again for the issue.
>
> --
> Lance Albertson
> Director
> Oregon State University | Open Source Lab
>


-- 
Lance Albertson
Director
Oregon State University | Open Source Lab