Ceph for Secondary Storage and Primary Storage

[Mevludin Blazevic <mb...@uni-koblenz.de>]

Hi all,

is there someone who is using a ceph cluster both for secondary storage 
(NFS service) and primary storage (rbd pool)? In my test environment I 
have experienced some issues starting VMs after updating cloudstack not 
able to convert images to RBD.

Regards,

Mevludin

Re: Ceph for Secondary Storage and Primary Storage

[Mevludin Blazevic <mb...@uni-koblenz.de>]

Hi,

thanks for your information and sharing of your experience! One time we 
had a powerloss for 1 hour. After that the cloud as well as the Ceph 
cluster where up with absolutely no issues. I think a corrupted Ceph 
cluster might be result of hardware failure, e.g. using Erasure Coding 
with to many broken hosts/OSDs. Some other Ceph admins told me they use 
Object Storage, RBD and CephFS on the same cluster with nearly no effect 
on performace in their environment.

We also used simple linux machines as NFS servers for Cloudstack. Since 
we need them to shut down, I am going to setup Ceph FS and NFS on our 
existing cluster.

Regards,

Mevludin

Am 04.11.2022 um 13:58 schrieb Wido den Hollander:
>
>
> Op 03-11-2022 om 12:00 schreef Mauro Ferraro - G2K Hosting:
>> It is a very promising technology, but in our experience, ceph made 
>> our life a hell. We had an energy event in our Datacenter and ceph 
>> couldnt support it, we can understand that these events may not 
>> happend, but shits happends.
>>
>
> That's a totally different experience! We are using multiple Petabytes 
> of Ceph clusters behind our CloudStack environments and this works 
> flawlessly.
>
> A powerloss which causes Ceph to go corrupt is probably a hardware 
> related issue and not Ceph.
>
>> Our traditional infrastructure (servers with hardware raid and nfs) 
>> supported this event with no major problems. With ceph we had a lot 
>> of data loss, we were searching data in the "garbage" (blocks) with 
>> rbd tools, months of work, stress of the team and lots of furious 
>> clients.
>>
>> Ceph can support some contingency but if the problem is big it will 
>> take a lot of time to recover and in this time you can't put your 
>> clusters and VMs to work.
>>
>> If you will use ceph make sure that you have a mirror working in 
>> another site that can run when have a big problem.
>>
>>
>> El 3/11/2022 a las 06:58, Mevludin Blazevic escribió:
>>> Hi all,
>>>
>>> is there someone who is using a ceph cluster both for secondary 
>>> storage (NFS service) and primary storage (rbd pool)? In my test 
>>> environment I have experienced some issues starting VMs after 
>>> updating cloudstack not able to convert images to RBD.
>>>
>
> Yes, as said, we have.
>
> Our Primary Storage is all RBD with KVM behind CloudStack.
>
> Secondary Storage comes from simple Linux machines just exposing NFS 
> towards CloudStack.
>
> Wido
>
>>> Regards,
>>>
>>> Mevludin
>>>
-- 
Mevludin Blazevic, M.Sc.

University of Koblenz-Landau
Computing Centre (GHRKO)
Universitaetsstrasse 1
D-56070 Koblenz, Germany
Room A023
Tel: +49 261/287-1326

Re: Ceph for Secondary Storage and Primary Storage

[Wido den Hollander <wi...@widodh.nl>]

Op 03-11-2022 om 12:00 schreef Mauro Ferraro - G2K Hosting:
> It is a very promising technology, but in our experience, ceph made our 
> life a hell. We had an energy event in our Datacenter and ceph couldnt 
> support it, we can understand that these events may not happend, but 
> shits happends.
> 

That's a totally different experience! We are using multiple Petabytes 
of Ceph clusters behind our CloudStack environments and this works 
flawlessly.

A powerloss which causes Ceph to go corrupt is probably a hardware 
related issue and not Ceph.

> Our traditional infrastructure (servers with hardware raid and nfs) 
> supported this event with no major problems. With ceph we had a lot of 
> data loss, we were searching data in the "garbage" (blocks) with rbd 
> tools, months of work, stress of the team and lots of furious clients.
> 
> Ceph can support some contingency but if the problem is big it will take 
> a lot of time to recover and in this time you can't put your clusters 
> and VMs to work.
> 
> If you will use ceph make sure that you have a mirror working in another 
> site that can run when have a big problem.
> 
> 
> El 3/11/2022 a las 06:58, Mevludin Blazevic escribió:
>> Hi all,
>>
>> is there someone who is using a ceph cluster both for secondary 
>> storage (NFS service) and primary storage (rbd pool)? In my test 
>> environment I have experienced some issues starting VMs after updating 
>> cloudstack not able to convert images to RBD.
>>

Yes, as said, we have.

Our Primary Storage is all RBD with KVM behind CloudStack.

Secondary Storage comes from simple Linux machines just exposing NFS 
towards CloudStack.

Wido

>> Regards,
>>
>> Mevludin
>>

Re: Ceph for Secondary Storage and Primary Storage

[Mauro Ferraro - G2K Hosting <mf...@g2khosting.com>]

It is a very promising technology, but in our experience, ceph made our 
life a hell. We had an energy event in our Datacenter and ceph couldnt 
support it, we can understand that these events may not happend, but 
shits happends.

Our traditional infrastructure (servers with hardware raid and nfs) 
supported this event with no major problems. With ceph we had a lot of 
data loss, we were searching data in the "garbage" (blocks) with rbd 
tools, months of work, stress of the team and lots of furious clients.

Ceph can support some contingency but if the problem is big it will take 
a lot of time to recover and in this time you can't put your clusters 
and VMs to work.

If you will use ceph make sure that you have a mirror working in another 
site that can run when have a big problem.


El 3/11/2022 a las 06:58, Mevludin Blazevic escribió:
> Hi all,
>
> is there someone who is using a ceph cluster both for secondary 
> storage (NFS service) and primary storage (rbd pool)? In my test 
> environment I have experienced some issues starting VMs after updating 
> cloudstack not able to convert images to RBD.
>
> Regards,
>
> Mevludin
>