[osuosl-openpower] Backend storage issues

[Lance Albertson <la...@osuosl.org>]

All,

It appears we had an issue with our backend storage (Ceph) overnight that
has caused some of the VMs to be offline. I'm currently working on
resolving the issue right now and will send an update once I have
everything back online.

I'll reply to any of your related tickets directly once I have an update as
well.

Thanks for your patience!

-- 
Lance Albertson
Director
Oregon State University | Open Source Lab

Re: [osuosl-openpower] Backend storage issues

[Lance Albertson <la...@osuosl.org>]

It appears the VMs that were offline may or may not have been related to
the backend storage issues but all seems to be working now as far as I can
tell. We had several VMs reboot around the same time the Ceph storage issue
happened however the hypervisors were all OK. Nothing in the logs provided
details on why those were rebooted.

We had an SSD fail on one of the Ceph nodes which is used for caching
metadata which caused issues to some of the other OSD's on the same host.
The replacement SSD was supposed to be delivered on Friday but got delayed
and wasn't going to arrive until tomorrow. Unfortunately the clock ran out
on the SSD and decided to fail overnight.

I'm going in today to replace the failing SSD with a temporary drive until
we can get the proper replacement in tomorrow.

If any of you are still having issues with your VMs, please don't hesitate
to email powerdev-request@osuosl.org to open a ticket.

Thanks for your patience!

On Sun, Dec 19, 2021 at 10:21 AM Lance Albertson <la...@osuosl.org> wrote:

> All,
>
> It appears we had an issue with our backend storage (Ceph) overnight that
> has caused some of the VMs to be offline. I'm currently working on
> resolving the issue right now and will send an update once I have
> everything back online.
>
> I'll reply to any of your related tickets directly once I have an update
> as well.
>
> Thanks for your patience!
>
> --
> Lance Albertson
> Director
> Oregon State University | Open Source Lab
>


-- 
Lance Albertson
Director
Oregon State University | Open Source Lab