You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Logan Barfield <lb...@tqhosting.com> on 2015/02/18 21:00:34 UTC
Libvirt & RBD caching
Our current deployment is KVM with Ceph RBD primary storage. We have
rbd_cache enabled, and use "cache=none" in Qemu by default.
I've been running some tests to try to figure out why our write speeds
with FreeBSD are significantly lower than Linux. I was testing both
RBD and local SSD storage, with various cache configurations. Out of
all of them the only one that performed close to our standard Linux
images was local SSD, Qemu cache=writeback, FreeBSD gpt journal
enabled.
I've been reading on various lists the reasons and risks for
cache=none vs cache=writeback:
- cache=none: Safer for live migration
- cache=writeback: Ceph RBD docs claim that this is required for data
integrity when using rbd_cache
>From what I can tell performance is generally the same with both,
except in the case of FreeBSD.
What is the current line of thinking on this? Should be using 'none'
or 'writeback' with RBD by default? Is 'writeback' considered safe
for live migration?
Re: Libvirt & RBD caching
Posted by Logan Barfield <lb...@tqhosting.com>.
Hi Ilya,
I would like to document or blog about a great many things.
Unfortunately my time for that is limited right now.
As for what tests were run: I was basically just doing a standard
'dd' test, which isn't a great indicator of real world usage, but
works well enough for comparison testing.
We were running write and read tests with 'bs=1M count=1024' and
'oflag/iflag=direct' in Linux. The tests are all relative though, so
any options should work.
On Linux backed by RBD we were seeing roughly 150MB/s writes and
175MB/s+ reads. On FreeBSD on RBD (with VirtIO) we were seeing
roughly 30MB/s writes and 100MB/s reads.
For FreeBSD we tested the following configurations (write speeds
indicated, reads were always reasonable @ ~100-200MB/s), we ran each
test a few times to get an average:
- RBD - Qemu 'cache=none': 30MB/s
- RBD - Qemu 'cache=writeback': 50MB/s
- Local SSD - Qemu 'cache=none': 50MB/s
- Local SSD - Qemu 'cache=writeback': ~125MB/s
As I mentioned before the 'Local SSD: writeback' configuration was the
only one that got us even close to Linux speeds.
I got a response on the Ceph list that I've yet to fully look over,
but it seems to indicate that there may be an issue with the FreeBSD
I/O sizes. I'm also thinking it may just be a problem with the
FreeBSD VirtIO drivers, as they are still fairly new.
We'd like to be able to offer FreeBSD services on RBD, but right now
it's looking like we'd have to go with local storage to make it
viable.
Thank You,
Logan Barfield
Tranquil Hosting
On Thu, Feb 19, 2015 at 8:23 AM, Andrei Mikhailovsky <an...@arhont.com> wrote:
> Ilya, i have followed the instructions on ceph website and it worked perfectly well. The only addition is to enable rbd caching in the ceph.conf
>
> Andrei
>
> ----- Original Message -----
>
>> From: "ilya musayev" <il...@gmail.com>
>> To: dev@cloudstack.apache.org
>> Sent: Thursday, 19 February, 2015 10:21:22 AM
>> Subject: Re: Libvirt & RBD caching
>
>> Logan
>
>> Side note: it would help great if you can post your notes/guide on
>> setting up Ceph as primary with CloudStack.
>> There arent any docs out there.
>
>> Thanks
>> ilya
>> On 2/18/15 12:00 PM, Logan Barfield wrote:
>> > Our current deployment is KVM with Ceph RBD primary storage. We
>> > have
>> > rbd_cache enabled, and use "cache=none" in Qemu by default.
>> >
>> > I've been running some tests to try to figure out why our write
>> > speeds
>> > with FreeBSD are significantly lower than Linux. I was testing both
>> > RBD and local SSD storage, with various cache configurations. Out
>> > of
>> > all of them the only one that performed close to our standard Linux
>> > images was local SSD, Qemu cache=writeback, FreeBSD gpt journal
>> > enabled.
>> >
>> > I've been reading on various lists the reasons and risks for
>> > cache=none vs cache=writeback:
>> > - cache=none: Safer for live migration
>> > - cache=writeback: Ceph RBD docs claim that this is required for
>> > data
>> > integrity when using rbd_cache
>> >
>> > From what I can tell performance is generally the same with both,
>> > except in the case of FreeBSD.
>> >
>> > What is the current line of thinking on this? Should be using
>> > 'none'
>> > or 'writeback' with RBD by default? Is 'writeback' considered safe
>> > for live migration?
Re: Libvirt & RBD caching
Posted by Andrei Mikhailovsky <an...@arhont.com>.
Ilya, i have followed the instructions on ceph website and it worked perfectly well. The only addition is to enable rbd caching in the ceph.conf
Andrei
----- Original Message -----
> From: "ilya musayev" <il...@gmail.com>
> To: dev@cloudstack.apache.org
> Sent: Thursday, 19 February, 2015 10:21:22 AM
> Subject: Re: Libvirt & RBD caching
> Logan
> Side note: it would help great if you can post your notes/guide on
> setting up Ceph as primary with CloudStack.
> There arent any docs out there.
> Thanks
> ilya
> On 2/18/15 12:00 PM, Logan Barfield wrote:
> > Our current deployment is KVM with Ceph RBD primary storage. We
> > have
> > rbd_cache enabled, and use "cache=none" in Qemu by default.
> >
> > I've been running some tests to try to figure out why our write
> > speeds
> > with FreeBSD are significantly lower than Linux. I was testing both
> > RBD and local SSD storage, with various cache configurations. Out
> > of
> > all of them the only one that performed close to our standard Linux
> > images was local SSD, Qemu cache=writeback, FreeBSD gpt journal
> > enabled.
> >
> > I've been reading on various lists the reasons and risks for
> > cache=none vs cache=writeback:
> > - cache=none: Safer for live migration
> > - cache=writeback: Ceph RBD docs claim that this is required for
> > data
> > integrity when using rbd_cache
> >
> > From what I can tell performance is generally the same with both,
> > except in the case of FreeBSD.
> >
> > What is the current line of thinking on this? Should be using
> > 'none'
> > or 'writeback' with RBD by default? Is 'writeback' considered safe
> > for live migration?
Re: Libvirt & RBD caching
Posted by ilya musayev <il...@gmail.com>.
Logan
Side note: it would help great if you can post your notes/guide on
setting up Ceph as primary with CloudStack.
There arent any docs out there.
Thanks
ilya
On 2/18/15 12:00 PM, Logan Barfield wrote:
> Our current deployment is KVM with Ceph RBD primary storage. We have
> rbd_cache enabled, and use "cache=none" in Qemu by default.
>
> I've been running some tests to try to figure out why our write speeds
> with FreeBSD are significantly lower than Linux. I was testing both
> RBD and local SSD storage, with various cache configurations. Out of
> all of them the only one that performed close to our standard Linux
> images was local SSD, Qemu cache=writeback, FreeBSD gpt journal
> enabled.
>
> I've been reading on various lists the reasons and risks for
> cache=none vs cache=writeback:
> - cache=none: Safer for live migration
> - cache=writeback: Ceph RBD docs claim that this is required for data
> integrity when using rbd_cache
>
> From what I can tell performance is generally the same with both,
> except in the case of FreeBSD.
>
> What is the current line of thinking on this? Should be using 'none'
> or 'writeback' with RBD by default? Is 'writeback' considered safe
> for live migration?
Re: Libvirt & RBD caching
Posted by Nux! <nu...@li.nux.ro>.
What tests have you done exactly?
I'll try to replicate them in my setup.
FreeBSD also seemed slower to me, though I was more concerned with the network throughput, it struggles to saturate 1Gbps whereas linux ..just flies.
I would have loved to be able to use pfSense more seriously, but performance sucked in a VM.
Lucian
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Logan Barfield" <lb...@tqhosting.com>
> To: dev@cloudstack.apache.org
> Sent: Wednesday, 18 February, 2015 20:35:12
> Subject: Re: Libvirt & RBD caching
> We're running FreeBSD with VirtIO. Disk is VirtIO in the libvirt
> config, and shows up as a 'vtbd' device in /dev. I'm kind of stumped
> on where the performance discrepancy could be coming from.
>
> Thank You,
>
> Logan Barfield
> Tranquil Hosting
>
>
> On Wed, Feb 18, 2015 at 3:23 PM, Wido den Hollander <wi...@widodh.nl> wrote:
>>
>>
>> On 18-02-15 21:00, Logan Barfield wrote:
>>> Our current deployment is KVM with Ceph RBD primary storage. We have
>>> rbd_cache enabled, and use "cache=none" in Qemu by default.
>>>
>>> I've been running some tests to try to figure out why our write speeds
>>> with FreeBSD are significantly lower than Linux. I was testing both
>>> RBD and local SSD storage, with various cache configurations. Out of
>>> all of them the only one that performed close to our standard Linux
>>> images was local SSD, Qemu cache=writeback, FreeBSD gpt journal
>>> enabled.
>>>
>>
>> I think the main problem is that FreeBSD uses ide/scsi and Linux uses
>> VirtIO.
>>
>> Have you tried running FreeBSD 10 with the OS type set to Ubuntu for
>> example?
>>
>>> I've been reading on various lists the reasons and risks for
>>> cache=none vs cache=writeback:
>>> - cache=none: Safer for live migration
>>> - cache=writeback: Ceph RBD docs claim that this is required for data
>>> integrity when using rbd_cache
>>>
>>> From what I can tell performance is generally the same with both,
>>> except in the case of FreeBSD.
>>>
>>> What is the current line of thinking on this? Should be using 'none'
>>> or 'writeback' with RBD by default? Is 'writeback' considered safe
>>> for live migration?
>>>
>>
>> None should still be ok, since librbd does the caching. It's not Qemu
>> who does the caching.
>>
> > Wido
Re: Libvirt & RBD caching
Posted by Logan Barfield <lb...@tqhosting.com>.
We're running FreeBSD with VirtIO. Disk is VirtIO in the libvirt
config, and shows up as a 'vtbd' device in /dev. I'm kind of stumped
on where the performance discrepancy could be coming from.
Thank You,
Logan Barfield
Tranquil Hosting
On Wed, Feb 18, 2015 at 3:23 PM, Wido den Hollander <wi...@widodh.nl> wrote:
>
>
> On 18-02-15 21:00, Logan Barfield wrote:
>> Our current deployment is KVM with Ceph RBD primary storage. We have
>> rbd_cache enabled, and use "cache=none" in Qemu by default.
>>
>> I've been running some tests to try to figure out why our write speeds
>> with FreeBSD are significantly lower than Linux. I was testing both
>> RBD and local SSD storage, with various cache configurations. Out of
>> all of them the only one that performed close to our standard Linux
>> images was local SSD, Qemu cache=writeback, FreeBSD gpt journal
>> enabled.
>>
>
> I think the main problem is that FreeBSD uses ide/scsi and Linux uses
> VirtIO.
>
> Have you tried running FreeBSD 10 with the OS type set to Ubuntu for
> example?
>
>> I've been reading on various lists the reasons and risks for
>> cache=none vs cache=writeback:
>> - cache=none: Safer for live migration
>> - cache=writeback: Ceph RBD docs claim that this is required for data
>> integrity when using rbd_cache
>>
>> From what I can tell performance is generally the same with both,
>> except in the case of FreeBSD.
>>
>> What is the current line of thinking on this? Should be using 'none'
>> or 'writeback' with RBD by default? Is 'writeback' considered safe
>> for live migration?
>>
>
> None should still be ok, since librbd does the caching. It's not Qemu
> who does the caching.
>
> Wido
Re: Libvirt & RBD caching
Posted by Wido den Hollander <wi...@widodh.nl>.
On 18-02-15 21:00, Logan Barfield wrote:
> Our current deployment is KVM with Ceph RBD primary storage. We have
> rbd_cache enabled, and use "cache=none" in Qemu by default.
>
> I've been running some tests to try to figure out why our write speeds
> with FreeBSD are significantly lower than Linux. I was testing both
> RBD and local SSD storage, with various cache configurations. Out of
> all of them the only one that performed close to our standard Linux
> images was local SSD, Qemu cache=writeback, FreeBSD gpt journal
> enabled.
>
I think the main problem is that FreeBSD uses ide/scsi and Linux uses
VirtIO.
Have you tried running FreeBSD 10 with the OS type set to Ubuntu for
example?
> I've been reading on various lists the reasons and risks for
> cache=none vs cache=writeback:
> - cache=none: Safer for live migration
> - cache=writeback: Ceph RBD docs claim that this is required for data
> integrity when using rbd_cache
>
> From what I can tell performance is generally the same with both,
> except in the case of FreeBSD.
>
> What is the current line of thinking on this? Should be using 'none'
> or 'writeback' with RBD by default? Is 'writeback' considered safe
> for live migration?
>
None should still be ok, since librbd does the caching. It's not Qemu
who does the caching.
Wido