You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Ivan Kudryavtsev <ku...@bw-sw.com> on 2019/05/17 13:04:50 UTC
Re: Poor NVMe Performance with KVM
Well, just FYI, I changed cache_mode from NULL (none), to writethrough
directly in DB and the performance boosted greatly. It may be an important
feature for NVME drives.
Currently, on 4.11, the user can set cache-mode for disk offerings, but
cannot for service offerings, which are translated to cache=none
corresponding disk offerings.
The only way is to use SQL to change that for root disk disk offerings.
CreateServiceOffering API doesn't support cache mode. It can be a serious
limitation for NVME users, because by default they could meet poor
read/write performance.
пт, 17 мая 2019 г., 19:30 Ivan Kudryavtsev <ku...@bw-sw.com>:
> Darius, thanks for your participation,
>
> first, I used 4.14 kernel which is the default one for my cluster. Next,
> switched to 4.15 with dist-upgrade.
>
> Do you have an idea how to turn on amount of queues for virtio-scsi with
> Cloudstack?
>
> пт, 17 мая 2019 г., 19:26 Darius Kasparavičius <da...@gmail.com>:
>
>> Hi,
>>
>> I can see a few issues with your xml file. You can try using "queues"
>> inside your disk definitions. This should help a little, not sure by
>> how much for your case, but for my specific it went up by almost the
>> number of queues. Also try cache directsync or writethrough. You
>> should switch kernel if bugs are still there with 4.15 kernel.
>>
>> On Fri, May 17, 2019 at 12:14 PM Ivan Kudryavtsev
>> <ku...@bw-sw.com> wrote:
>> >
>> > Hello, colleagues.
>> >
>> > Hope, someone could help me. I just deployed a new VM host with Intel
>> P4500
>> > local storage NVMe drive.
>> >
>> > From Hypervisor host I can get expected performance, 200K RIOPS, 3GBs
>> with
>> > FIO, write performance is also high as expected.
>> >
>> > I've created a new KVM VM Service offering with virtio-scsi controller
>> > (tried virtio as well) and VM is deployed. Now I try to benchmark it
>> with
>> > FIO. Results are very strange:
>> >
>> > 1. Read/Write with large blocks (1M) shows expected performance (my
>> limits
>> > are R=1000/W=500 MBs).
>> >
>> > 2. Write with direct=0 leads to expected 50K IOPS, while write with
>> > direct=1 leads to very moderate 2-3K IOPS.
>> >
>> > 3. Read with direct=0, direct=1 both lead to 3000 IOPS.
>> >
>> > During the benchmark I see VM IOWAIT=20%, while host IOWAIT is 0% which
>> is
>> > strange.
>> >
>> > So, basically, from inside VM my NVMe works very slow when small IOPS
>> are
>> > executed. From the host, it works great.
>> >
>> > I tried to mount the volume with NBD to /dev/nbd0 and benchmark. Read
>> > performance is nice. Maybe someone managed to use NVME with KVM with
>> small
>> > IOPS?
>> >
>> > The filesystem is XFS, previously tried with EXT4 - results are the
>> same.
>> >
>> > This is the part of VM XML definition generated by CloudStack:
>> >
>> > <devices>
>> > <emulator>/usr/bin/kvm-spice</emulator>
>> > <disk type='file' device='disk'>
>> > <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
>> > <source
>> > file='/var/lib/libvirt/images/6809dbd0-4a15-4014-9322-fe9010695934'/>
>> > <backingStore type='file' index='1'>
>> > <format type='raw'/>
>> > <source
>> > file='/var/lib/libvirt/images/ac43742c-3991-4be1-bff1-7617bf4fc6ef'/>
>> > <backingStore/>
>> > </backingStore>
>> > <target dev='sda' bus='scsi'/>
>> > <iotune>
>> > <read_bytes_sec>1048576000</read_bytes_sec>
>> > <write_bytes_sec>524288000</write_bytes_sec>
>> > <read_iops_sec>100000</read_iops_sec>
>> > <write_iops_sec>50000</write_iops_sec>
>> > </iotune>
>> > <serial>6809dbd04a1540149322</serial>
>> > <alias name='scsi0-0-0-0'/>
>> > <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>> > </disk>
>> > <disk type='file' device='cdrom'>
>> > <driver name='qemu' type='raw'/>
>> > <backingStore/>
>> > <target dev='hdc' bus='ide'/>
>> > <readonly/>
>> > <alias name='ide0-1-0'/>
>> > <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>> > </disk>
>> > <controller type='scsi' index='0' model='virtio-scsi'>
>> > <alias name='scsi0'/>
>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>> > function='0x0'/>
>> > </controller>
>> >
>> > So, what I see now, is that it works slower than couple of two Samsung
>> 960
>> > PRO which is extremely strange.
>> >
>> > Thanks in advance.
>> >
>> >
>> > --
>> > With best regards, Ivan Kudryavtsev
>> > Bitworks LLC
>> > Cell RU: +7-923-414-1515
>> > Cell USA: +1-201-257-1512
>> > WWW: http://bitworks.software/ <http://bw-sw.com/>
>>
>
Re: Poor NVMe Performance with KVM
Posted by Wido den Hollander <wi...@widodh.nl>.
You also might want to set the allocation mode to something else then
shared. This causes the qcow2 metadata to be pre-allocated and that will
improve performance.
Wido
On 5/17/19 3:04 PM, Ivan Kudryavtsev wrote:
> Well, just FYI, I changed cache_mode from NULL (none), to writethrough
> directly in DB and the performance boosted greatly. It may be an important
> feature for NVME drives.
>
> Currently, on 4.11, the user can set cache-mode for disk offerings, but
> cannot for service offerings, which are translated to cache=none
> corresponding disk offerings.
>
> The only way is to use SQL to change that for root disk disk offerings.
> CreateServiceOffering API doesn't support cache mode. It can be a serious
> limitation for NVME users, because by default they could meet poor
> read/write performance.
>
> пт, 17 мая 2019 г., 19:30 Ivan Kudryavtsev <ku...@bw-sw.com>:
>
>> Darius, thanks for your participation,
>>
>> first, I used 4.14 kernel which is the default one for my cluster. Next,
>> switched to 4.15 with dist-upgrade.
>>
>> Do you have an idea how to turn on amount of queues for virtio-scsi with
>> Cloudstack?
>>
>> пт, 17 мая 2019 г., 19:26 Darius Kasparavičius <da...@gmail.com>:
>>
>>> Hi,
>>>
>>> I can see a few issues with your xml file. You can try using "queues"
>>> inside your disk definitions. This should help a little, not sure by
>>> how much for your case, but for my specific it went up by almost the
>>> number of queues. Also try cache directsync or writethrough. You
>>> should switch kernel if bugs are still there with 4.15 kernel.
>>>
>>> On Fri, May 17, 2019 at 12:14 PM Ivan Kudryavtsev
>>> <ku...@bw-sw.com> wrote:
>>>>
>>>> Hello, colleagues.
>>>>
>>>> Hope, someone could help me. I just deployed a new VM host with Intel
>>> P4500
>>>> local storage NVMe drive.
>>>>
>>>> From Hypervisor host I can get expected performance, 200K RIOPS, 3GBs
>>> with
>>>> FIO, write performance is also high as expected.
>>>>
>>>> I've created a new KVM VM Service offering with virtio-scsi controller
>>>> (tried virtio as well) and VM is deployed. Now I try to benchmark it
>>> with
>>>> FIO. Results are very strange:
>>>>
>>>> 1. Read/Write with large blocks (1M) shows expected performance (my
>>> limits
>>>> are R=1000/W=500 MBs).
>>>>
>>>> 2. Write with direct=0 leads to expected 50K IOPS, while write with
>>>> direct=1 leads to very moderate 2-3K IOPS.
>>>>
>>>> 3. Read with direct=0, direct=1 both lead to 3000 IOPS.
>>>>
>>>> During the benchmark I see VM IOWAIT=20%, while host IOWAIT is 0% which
>>> is
>>>> strange.
>>>>
>>>> So, basically, from inside VM my NVMe works very slow when small IOPS
>>> are
>>>> executed. From the host, it works great.
>>>>
>>>> I tried to mount the volume with NBD to /dev/nbd0 and benchmark. Read
>>>> performance is nice. Maybe someone managed to use NVME with KVM with
>>> small
>>>> IOPS?
>>>>
>>>> The filesystem is XFS, previously tried with EXT4 - results are the
>>> same.
>>>>
>>>> This is the part of VM XML definition generated by CloudStack:
>>>>
>>>> <devices>
>>>> <emulator>/usr/bin/kvm-spice</emulator>
>>>> <disk type='file' device='disk'>
>>>> <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
>>>> <source
>>>> file='/var/lib/libvirt/images/6809dbd0-4a15-4014-9322-fe9010695934'/>
>>>> <backingStore type='file' index='1'>
>>>> <format type='raw'/>
>>>> <source
>>>> file='/var/lib/libvirt/images/ac43742c-3991-4be1-bff1-7617bf4fc6ef'/>
>>>> <backingStore/>
>>>> </backingStore>
>>>> <target dev='sda' bus='scsi'/>
>>>> <iotune>
>>>> <read_bytes_sec>1048576000</read_bytes_sec>
>>>> <write_bytes_sec>524288000</write_bytes_sec>
>>>> <read_iops_sec>100000</read_iops_sec>
>>>> <write_iops_sec>50000</write_iops_sec>
>>>> </iotune>
>>>> <serial>6809dbd04a1540149322</serial>
>>>> <alias name='scsi0-0-0-0'/>
>>>> <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>>> </disk>
>>>> <disk type='file' device='cdrom'>
>>>> <driver name='qemu' type='raw'/>
>>>> <backingStore/>
>>>> <target dev='hdc' bus='ide'/>
>>>> <readonly/>
>>>> <alias name='ide0-1-0'/>
>>>> <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>>>> </disk>
>>>> <controller type='scsi' index='0' model='virtio-scsi'>
>>>> <alias name='scsi0'/>
>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>>>> function='0x0'/>
>>>> </controller>
>>>>
>>>> So, what I see now, is that it works slower than couple of two Samsung
>>> 960
>>>> PRO which is extremely strange.
>>>>
>>>> Thanks in advance.
>>>>
>>>>
>>>> --
>>>> With best regards, Ivan Kudryavtsev
>>>> Bitworks LLC
>>>> Cell RU: +7-923-414-1515
>>>> Cell USA: +1-201-257-1512
>>>> WWW: http://bitworks.software/ <http://bw-sw.com/>
>>>
>>
>
Re: Poor NVMe Performance with KVM
Posted by Ivan Kudryavtsev <ku...@bw-sw.com>.
Nux,
there is no way to set it for NVME as it only has [none] option in
/sys/block/nvme0n1/queue/scheduler
Setting any scheduler for VM volume doesn't improve a thing.
пт, 17 мая 2019 г., 20:21 Nux! <nu...@li.nux.ro>:
> What happens when you set deadline scheduler in both HV and guest?
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Ivan Kudryavtsev" <ku...@bw-sw.com>
> > To: "users" <us...@cloudstack.apache.org>, "dev" <
> dev@cloudstack.apache.org>
> > Sent: Friday, 17 May, 2019 14:16:31
> > Subject: Re: Poor NVMe Performance with KVM
>
> > BTW, You may think that the improvement is achieved by caching, but I
> clear
> > the cache with
> > sync & echo 3 > /proc/sys/vm/drop_caches
> >
> > So, can't claim for sure, need other opinion, but looks like for NVMe,
> > writethrough must be used if you want high IO rate. At least with Intel
> > p4500.
> >
> >
> > пт, 17 мая 2019 г., 20:04 Ivan Kudryavtsev <ku...@bw-sw.com>:
> >
> >> Well, just FYI, I changed cache_mode from NULL (none), to writethrough
> >> directly in DB and the performance boosted greatly. It may be an
> important
> >> feature for NVME drives.
> >>
> >> Currently, on 4.11, the user can set cache-mode for disk offerings, but
> >> cannot for service offerings, which are translated to cache=none
> >> corresponding disk offerings.
> >>
> >> The only way is to use SQL to change that for root disk disk offerings.
> >> CreateServiceOffering API doesn't support cache mode. It can be a
> serious
> >> limitation for NVME users, because by default they could meet poor
> >> read/write performance.
> >>
> >> пт, 17 мая 2019 г., 19:30 Ivan Kudryavtsev <ku...@bw-sw.com>:
> >>
> >>> Darius, thanks for your participation,
> >>>
> >>> first, I used 4.14 kernel which is the default one for my cluster.
> Next,
> >>> switched to 4.15 with dist-upgrade.
> >>>
> >>> Do you have an idea how to turn on amount of queues for virtio-scsi
> with
> >>> Cloudstack?
> >>>
> >>> пт, 17 мая 2019 г., 19:26 Darius Kasparavičius <da...@gmail.com>:
> >>>
> >>>> Hi,
> >>>>
> >>>> I can see a few issues with your xml file. You can try using "queues"
> >>>> inside your disk definitions. This should help a little, not sure by
> >>>> how much for your case, but for my specific it went up by almost the
> >>>> number of queues. Also try cache directsync or writethrough. You
> >>>> should switch kernel if bugs are still there with 4.15 kernel.
> >>>>
> >>>> On Fri, May 17, 2019 at 12:14 PM Ivan Kudryavtsev
> >>>> <ku...@bw-sw.com> wrote:
> >>>> >
> >>>> > Hello, colleagues.
> >>>> >
> >>>> > Hope, someone could help me. I just deployed a new VM host with
> Intel
> >>>> P4500
> >>>> > local storage NVMe drive.
> >>>> >
> >>>> > From Hypervisor host I can get expected performance, 200K RIOPS,
> 3GBs
> >>>> with
> >>>> > FIO, write performance is also high as expected.
> >>>> >
> >>>> > I've created a new KVM VM Service offering with virtio-scsi
> controller
> >>>> > (tried virtio as well) and VM is deployed. Now I try to benchmark it
> >>>> with
> >>>> > FIO. Results are very strange:
> >>>> >
> >>>> > 1. Read/Write with large blocks (1M) shows expected performance (my
> >>>> limits
> >>>> > are R=1000/W=500 MBs).
> >>>> >
> >>>> > 2. Write with direct=0 leads to expected 50K IOPS, while write with
> >>>> > direct=1 leads to very moderate 2-3K IOPS.
> >>>> >
> >>>> > 3. Read with direct=0, direct=1 both lead to 3000 IOPS.
> >>>> >
> >>>> > During the benchmark I see VM IOWAIT=20%, while host IOWAIT is 0%
> >>>> which is
> >>>> > strange.
> >>>> >
> >>>> > So, basically, from inside VM my NVMe works very slow when small
> IOPS
> >>>> are
> >>>> > executed. From the host, it works great.
> >>>> >
> >>>> > I tried to mount the volume with NBD to /dev/nbd0 and benchmark.
> Read
> >>>> > performance is nice. Maybe someone managed to use NVME with KVM with
> >>>> small
> >>>> > IOPS?
> >>>> >
> >>>> > The filesystem is XFS, previously tried with EXT4 - results are the
> >>>> same.
> >>>> >
> >>>> > This is the part of VM XML definition generated by CloudStack:
> >>>> >
> >>>> > <devices>
> >>>> > <emulator>/usr/bin/kvm-spice</emulator>
> >>>> > <disk type='file' device='disk'>
> >>>> > <driver name='qemu' type='qcow2' cache='none'
> discard='unmap'/>
> >>>> > <source
> >>>> >
> file='/var/lib/libvirt/images/6809dbd0-4a15-4014-9322-fe9010695934'/>
> >>>> > <backingStore type='file' index='1'>
> >>>> > <format type='raw'/>
> >>>> > <source
> >>>> >
> file='/var/lib/libvirt/images/ac43742c-3991-4be1-bff1-7617bf4fc6ef'/>
> >>>> > <backingStore/>
> >>>> > </backingStore>
> >>>> > <target dev='sda' bus='scsi'/>
> >>>> > <iotune>
> >>>> > <read_bytes_sec>1048576000</read_bytes_sec>
> >>>> > <write_bytes_sec>524288000</write_bytes_sec>
> >>>> > <read_iops_sec>100000</read_iops_sec>
> >>>> > <write_iops_sec>50000</write_iops_sec>
> >>>> > </iotune>
> >>>> > <serial>6809dbd04a1540149322</serial>
> >>>> > <alias name='scsi0-0-0-0'/>
> >>>> > <address type='drive' controller='0' bus='0' target='0'
> >>>> unit='0'/>
> >>>> > </disk>
> >>>> > <disk type='file' device='cdrom'>
> >>>> > <driver name='qemu' type='raw'/>
> >>>> > <backingStore/>
> >>>> > <target dev='hdc' bus='ide'/>
> >>>> > <readonly/>
> >>>> > <alias name='ide0-1-0'/>
> >>>> > <address type='drive' controller='0' bus='1' target='0'
> >>>> unit='0'/>
> >>>> > </disk>
> >>>> > <controller type='scsi' index='0' model='virtio-scsi'>
> >>>> > <alias name='scsi0'/>
> >>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
> >>>> > function='0x0'/>
> >>>> > </controller>
> >>>> >
> >>>> > So, what I see now, is that it works slower than couple of two
> Samsung
> >>>> 960
> >>>> > PRO which is extremely strange.
> >>>> >
> >>>> > Thanks in advance.
> >>>> >
> >>>> >
> >>>> > --
> >>>> > With best regards, Ivan Kudryavtsev
> >>>> > Bitworks LLC
> >>>> > Cell RU: +7-923-414-1515
> >>>> > Cell USA: +1-201-257-1512
> >>>> > WWW: http://bitworks.software/ <http://bw-sw.com/>
> >>>>
>
Re: Poor NVMe Performance with KVM
Posted by Ivan Kudryavtsev <ku...@bw-sw.com>.
Nux,
there is no way to set it for NVME as it only has [none] option in
/sys/block/nvme0n1/queue/scheduler
Setting any scheduler for VM volume doesn't improve a thing.
пт, 17 мая 2019 г., 20:21 Nux! <nu...@li.nux.ro>:
> What happens when you set deadline scheduler in both HV and guest?
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Ivan Kudryavtsev" <ku...@bw-sw.com>
> > To: "users" <us...@cloudstack.apache.org>, "dev" <
> dev@cloudstack.apache.org>
> > Sent: Friday, 17 May, 2019 14:16:31
> > Subject: Re: Poor NVMe Performance with KVM
>
> > BTW, You may think that the improvement is achieved by caching, but I
> clear
> > the cache with
> > sync & echo 3 > /proc/sys/vm/drop_caches
> >
> > So, can't claim for sure, need other opinion, but looks like for NVMe,
> > writethrough must be used if you want high IO rate. At least with Intel
> > p4500.
> >
> >
> > пт, 17 мая 2019 г., 20:04 Ivan Kudryavtsev <ku...@bw-sw.com>:
> >
> >> Well, just FYI, I changed cache_mode from NULL (none), to writethrough
> >> directly in DB and the performance boosted greatly. It may be an
> important
> >> feature for NVME drives.
> >>
> >> Currently, on 4.11, the user can set cache-mode for disk offerings, but
> >> cannot for service offerings, which are translated to cache=none
> >> corresponding disk offerings.
> >>
> >> The only way is to use SQL to change that for root disk disk offerings.
> >> CreateServiceOffering API doesn't support cache mode. It can be a
> serious
> >> limitation for NVME users, because by default they could meet poor
> >> read/write performance.
> >>
> >> пт, 17 мая 2019 г., 19:30 Ivan Kudryavtsev <ku...@bw-sw.com>:
> >>
> >>> Darius, thanks for your participation,
> >>>
> >>> first, I used 4.14 kernel which is the default one for my cluster.
> Next,
> >>> switched to 4.15 with dist-upgrade.
> >>>
> >>> Do you have an idea how to turn on amount of queues for virtio-scsi
> with
> >>> Cloudstack?
> >>>
> >>> пт, 17 мая 2019 г., 19:26 Darius Kasparavičius <da...@gmail.com>:
> >>>
> >>>> Hi,
> >>>>
> >>>> I can see a few issues with your xml file. You can try using "queues"
> >>>> inside your disk definitions. This should help a little, not sure by
> >>>> how much for your case, but for my specific it went up by almost the
> >>>> number of queues. Also try cache directsync or writethrough. You
> >>>> should switch kernel if bugs are still there with 4.15 kernel.
> >>>>
> >>>> On Fri, May 17, 2019 at 12:14 PM Ivan Kudryavtsev
> >>>> <ku...@bw-sw.com> wrote:
> >>>> >
> >>>> > Hello, colleagues.
> >>>> >
> >>>> > Hope, someone could help me. I just deployed a new VM host with
> Intel
> >>>> P4500
> >>>> > local storage NVMe drive.
> >>>> >
> >>>> > From Hypervisor host I can get expected performance, 200K RIOPS,
> 3GBs
> >>>> with
> >>>> > FIO, write performance is also high as expected.
> >>>> >
> >>>> > I've created a new KVM VM Service offering with virtio-scsi
> controller
> >>>> > (tried virtio as well) and VM is deployed. Now I try to benchmark it
> >>>> with
> >>>> > FIO. Results are very strange:
> >>>> >
> >>>> > 1. Read/Write with large blocks (1M) shows expected performance (my
> >>>> limits
> >>>> > are R=1000/W=500 MBs).
> >>>> >
> >>>> > 2. Write with direct=0 leads to expected 50K IOPS, while write with
> >>>> > direct=1 leads to very moderate 2-3K IOPS.
> >>>> >
> >>>> > 3. Read with direct=0, direct=1 both lead to 3000 IOPS.
> >>>> >
> >>>> > During the benchmark I see VM IOWAIT=20%, while host IOWAIT is 0%
> >>>> which is
> >>>> > strange.
> >>>> >
> >>>> > So, basically, from inside VM my NVMe works very slow when small
> IOPS
> >>>> are
> >>>> > executed. From the host, it works great.
> >>>> >
> >>>> > I tried to mount the volume with NBD to /dev/nbd0 and benchmark.
> Read
> >>>> > performance is nice. Maybe someone managed to use NVME with KVM with
> >>>> small
> >>>> > IOPS?
> >>>> >
> >>>> > The filesystem is XFS, previously tried with EXT4 - results are the
> >>>> same.
> >>>> >
> >>>> > This is the part of VM XML definition generated by CloudStack:
> >>>> >
> >>>> > <devices>
> >>>> > <emulator>/usr/bin/kvm-spice</emulator>
> >>>> > <disk type='file' device='disk'>
> >>>> > <driver name='qemu' type='qcow2' cache='none'
> discard='unmap'/>
> >>>> > <source
> >>>> >
> file='/var/lib/libvirt/images/6809dbd0-4a15-4014-9322-fe9010695934'/>
> >>>> > <backingStore type='file' index='1'>
> >>>> > <format type='raw'/>
> >>>> > <source
> >>>> >
> file='/var/lib/libvirt/images/ac43742c-3991-4be1-bff1-7617bf4fc6ef'/>
> >>>> > <backingStore/>
> >>>> > </backingStore>
> >>>> > <target dev='sda' bus='scsi'/>
> >>>> > <iotune>
> >>>> > <read_bytes_sec>1048576000</read_bytes_sec>
> >>>> > <write_bytes_sec>524288000</write_bytes_sec>
> >>>> > <read_iops_sec>100000</read_iops_sec>
> >>>> > <write_iops_sec>50000</write_iops_sec>
> >>>> > </iotune>
> >>>> > <serial>6809dbd04a1540149322</serial>
> >>>> > <alias name='scsi0-0-0-0'/>
> >>>> > <address type='drive' controller='0' bus='0' target='0'
> >>>> unit='0'/>
> >>>> > </disk>
> >>>> > <disk type='file' device='cdrom'>
> >>>> > <driver name='qemu' type='raw'/>
> >>>> > <backingStore/>
> >>>> > <target dev='hdc' bus='ide'/>
> >>>> > <readonly/>
> >>>> > <alias name='ide0-1-0'/>
> >>>> > <address type='drive' controller='0' bus='1' target='0'
> >>>> unit='0'/>
> >>>> > </disk>
> >>>> > <controller type='scsi' index='0' model='virtio-scsi'>
> >>>> > <alias name='scsi0'/>
> >>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
> >>>> > function='0x0'/>
> >>>> > </controller>
> >>>> >
> >>>> > So, what I see now, is that it works slower than couple of two
> Samsung
> >>>> 960
> >>>> > PRO which is extremely strange.
> >>>> >
> >>>> > Thanks in advance.
> >>>> >
> >>>> >
> >>>> > --
> >>>> > With best regards, Ivan Kudryavtsev
> >>>> > Bitworks LLC
> >>>> > Cell RU: +7-923-414-1515
> >>>> > Cell USA: +1-201-257-1512
> >>>> > WWW: http://bitworks.software/ <http://bw-sw.com/>
> >>>>
>
Re: Poor NVMe Performance with KVM
Posted by Darius Kasparavičius <da...@gmail.com>.
Back when I worked in a company that used cloudstack. We had it
modified to add queue option. It was not available on default
cloudstack.
As for cache. You can set it in disk offering option.
On Fri, May 17, 2019 at 4:21 PM Nux! <nu...@li.nux.ro> wrote:
>
> What happens when you set deadline scheduler in both HV and guest?
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Ivan Kudryavtsev" <ku...@bw-sw.com>
> > To: "users" <us...@cloudstack.apache.org>, "dev" <de...@cloudstack.apache.org>
> > Sent: Friday, 17 May, 2019 14:16:31
> > Subject: Re: Poor NVMe Performance with KVM
>
> > BTW, You may think that the improvement is achieved by caching, but I clear
> > the cache with
> > sync & echo 3 > /proc/sys/vm/drop_caches
> >
> > So, can't claim for sure, need other opinion, but looks like for NVMe,
> > writethrough must be used if you want high IO rate. At least with Intel
> > p4500.
> >
> >
> > пт, 17 мая 2019 г., 20:04 Ivan Kudryavtsev <ku...@bw-sw.com>:
> >
> >> Well, just FYI, I changed cache_mode from NULL (none), to writethrough
> >> directly in DB and the performance boosted greatly. It may be an important
> >> feature for NVME drives.
> >>
> >> Currently, on 4.11, the user can set cache-mode for disk offerings, but
> >> cannot for service offerings, which are translated to cache=none
> >> corresponding disk offerings.
> >>
> >> The only way is to use SQL to change that for root disk disk offerings.
> >> CreateServiceOffering API doesn't support cache mode. It can be a serious
> >> limitation for NVME users, because by default they could meet poor
> >> read/write performance.
> >>
> >> пт, 17 мая 2019 г., 19:30 Ivan Kudryavtsev <ku...@bw-sw.com>:
> >>
> >>> Darius, thanks for your participation,
> >>>
> >>> first, I used 4.14 kernel which is the default one for my cluster. Next,
> >>> switched to 4.15 with dist-upgrade.
> >>>
> >>> Do you have an idea how to turn on amount of queues for virtio-scsi with
> >>> Cloudstack?
> >>>
> >>> пт, 17 мая 2019 г., 19:26 Darius Kasparavičius <da...@gmail.com>:
> >>>
> >>>> Hi,
> >>>>
> >>>> I can see a few issues with your xml file. You can try using "queues"
> >>>> inside your disk definitions. This should help a little, not sure by
> >>>> how much for your case, but for my specific it went up by almost the
> >>>> number of queues. Also try cache directsync or writethrough. You
> >>>> should switch kernel if bugs are still there with 4.15 kernel.
> >>>>
> >>>> On Fri, May 17, 2019 at 12:14 PM Ivan Kudryavtsev
> >>>> <ku...@bw-sw.com> wrote:
> >>>> >
> >>>> > Hello, colleagues.
> >>>> >
> >>>> > Hope, someone could help me. I just deployed a new VM host with Intel
> >>>> P4500
> >>>> > local storage NVMe drive.
> >>>> >
> >>>> > From Hypervisor host I can get expected performance, 200K RIOPS, 3GBs
> >>>> with
> >>>> > FIO, write performance is also high as expected.
> >>>> >
> >>>> > I've created a new KVM VM Service offering with virtio-scsi controller
> >>>> > (tried virtio as well) and VM is deployed. Now I try to benchmark it
> >>>> with
> >>>> > FIO. Results are very strange:
> >>>> >
> >>>> > 1. Read/Write with large blocks (1M) shows expected performance (my
> >>>> limits
> >>>> > are R=1000/W=500 MBs).
> >>>> >
> >>>> > 2. Write with direct=0 leads to expected 50K IOPS, while write with
> >>>> > direct=1 leads to very moderate 2-3K IOPS.
> >>>> >
> >>>> > 3. Read with direct=0, direct=1 both lead to 3000 IOPS.
> >>>> >
> >>>> > During the benchmark I see VM IOWAIT=20%, while host IOWAIT is 0%
> >>>> which is
> >>>> > strange.
> >>>> >
> >>>> > So, basically, from inside VM my NVMe works very slow when small IOPS
> >>>> are
> >>>> > executed. From the host, it works great.
> >>>> >
> >>>> > I tried to mount the volume with NBD to /dev/nbd0 and benchmark. Read
> >>>> > performance is nice. Maybe someone managed to use NVME with KVM with
> >>>> small
> >>>> > IOPS?
> >>>> >
> >>>> > The filesystem is XFS, previously tried with EXT4 - results are the
> >>>> same.
> >>>> >
> >>>> > This is the part of VM XML definition generated by CloudStack:
> >>>> >
> >>>> > <devices>
> >>>> > <emulator>/usr/bin/kvm-spice</emulator>
> >>>> > <disk type='file' device='disk'>
> >>>> > <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
> >>>> > <source
> >>>> > file='/var/lib/libvirt/images/6809dbd0-4a15-4014-9322-fe9010695934'/>
> >>>> > <backingStore type='file' index='1'>
> >>>> > <format type='raw'/>
> >>>> > <source
> >>>> > file='/var/lib/libvirt/images/ac43742c-3991-4be1-bff1-7617bf4fc6ef'/>
> >>>> > <backingStore/>
> >>>> > </backingStore>
> >>>> > <target dev='sda' bus='scsi'/>
> >>>> > <iotune>
> >>>> > <read_bytes_sec>1048576000</read_bytes_sec>
> >>>> > <write_bytes_sec>524288000</write_bytes_sec>
> >>>> > <read_iops_sec>100000</read_iops_sec>
> >>>> > <write_iops_sec>50000</write_iops_sec>
> >>>> > </iotune>
> >>>> > <serial>6809dbd04a1540149322</serial>
> >>>> > <alias name='scsi0-0-0-0'/>
> >>>> > <address type='drive' controller='0' bus='0' target='0'
> >>>> unit='0'/>
> >>>> > </disk>
> >>>> > <disk type='file' device='cdrom'>
> >>>> > <driver name='qemu' type='raw'/>
> >>>> > <backingStore/>
> >>>> > <target dev='hdc' bus='ide'/>
> >>>> > <readonly/>
> >>>> > <alias name='ide0-1-0'/>
> >>>> > <address type='drive' controller='0' bus='1' target='0'
> >>>> unit='0'/>
> >>>> > </disk>
> >>>> > <controller type='scsi' index='0' model='virtio-scsi'>
> >>>> > <alias name='scsi0'/>
> >>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
> >>>> > function='0x0'/>
> >>>> > </controller>
> >>>> >
> >>>> > So, what I see now, is that it works slower than couple of two Samsung
> >>>> 960
> >>>> > PRO which is extremely strange.
> >>>> >
> >>>> > Thanks in advance.
> >>>> >
> >>>> >
> >>>> > --
> >>>> > With best regards, Ivan Kudryavtsev
> >>>> > Bitworks LLC
> >>>> > Cell RU: +7-923-414-1515
> >>>> > Cell USA: +1-201-257-1512
> >>>> > WWW: http://bitworks.software/ <http://bw-sw.com/>
> >>>>
Re: Poor NVMe Performance with KVM
Posted by Nux! <nu...@li.nux.ro>.
What happens when you set deadline scheduler in both HV and guest?
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Ivan Kudryavtsev" <ku...@bw-sw.com>
> To: "users" <us...@cloudstack.apache.org>, "dev" <de...@cloudstack.apache.org>
> Sent: Friday, 17 May, 2019 14:16:31
> Subject: Re: Poor NVMe Performance with KVM
> BTW, You may think that the improvement is achieved by caching, but I clear
> the cache with
> sync & echo 3 > /proc/sys/vm/drop_caches
>
> So, can't claim for sure, need other opinion, but looks like for NVMe,
> writethrough must be used if you want high IO rate. At least with Intel
> p4500.
>
>
> пт, 17 мая 2019 г., 20:04 Ivan Kudryavtsev <ku...@bw-sw.com>:
>
>> Well, just FYI, I changed cache_mode from NULL (none), to writethrough
>> directly in DB and the performance boosted greatly. It may be an important
>> feature for NVME drives.
>>
>> Currently, on 4.11, the user can set cache-mode for disk offerings, but
>> cannot for service offerings, which are translated to cache=none
>> corresponding disk offerings.
>>
>> The only way is to use SQL to change that for root disk disk offerings.
>> CreateServiceOffering API doesn't support cache mode. It can be a serious
>> limitation for NVME users, because by default they could meet poor
>> read/write performance.
>>
>> пт, 17 мая 2019 г., 19:30 Ivan Kudryavtsev <ku...@bw-sw.com>:
>>
>>> Darius, thanks for your participation,
>>>
>>> first, I used 4.14 kernel which is the default one for my cluster. Next,
>>> switched to 4.15 with dist-upgrade.
>>>
>>> Do you have an idea how to turn on amount of queues for virtio-scsi with
>>> Cloudstack?
>>>
>>> пт, 17 мая 2019 г., 19:26 Darius Kasparavičius <da...@gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> I can see a few issues with your xml file. You can try using "queues"
>>>> inside your disk definitions. This should help a little, not sure by
>>>> how much for your case, but for my specific it went up by almost the
>>>> number of queues. Also try cache directsync or writethrough. You
>>>> should switch kernel if bugs are still there with 4.15 kernel.
>>>>
>>>> On Fri, May 17, 2019 at 12:14 PM Ivan Kudryavtsev
>>>> <ku...@bw-sw.com> wrote:
>>>> >
>>>> > Hello, colleagues.
>>>> >
>>>> > Hope, someone could help me. I just deployed a new VM host with Intel
>>>> P4500
>>>> > local storage NVMe drive.
>>>> >
>>>> > From Hypervisor host I can get expected performance, 200K RIOPS, 3GBs
>>>> with
>>>> > FIO, write performance is also high as expected.
>>>> >
>>>> > I've created a new KVM VM Service offering with virtio-scsi controller
>>>> > (tried virtio as well) and VM is deployed. Now I try to benchmark it
>>>> with
>>>> > FIO. Results are very strange:
>>>> >
>>>> > 1. Read/Write with large blocks (1M) shows expected performance (my
>>>> limits
>>>> > are R=1000/W=500 MBs).
>>>> >
>>>> > 2. Write with direct=0 leads to expected 50K IOPS, while write with
>>>> > direct=1 leads to very moderate 2-3K IOPS.
>>>> >
>>>> > 3. Read with direct=0, direct=1 both lead to 3000 IOPS.
>>>> >
>>>> > During the benchmark I see VM IOWAIT=20%, while host IOWAIT is 0%
>>>> which is
>>>> > strange.
>>>> >
>>>> > So, basically, from inside VM my NVMe works very slow when small IOPS
>>>> are
>>>> > executed. From the host, it works great.
>>>> >
>>>> > I tried to mount the volume with NBD to /dev/nbd0 and benchmark. Read
>>>> > performance is nice. Maybe someone managed to use NVME with KVM with
>>>> small
>>>> > IOPS?
>>>> >
>>>> > The filesystem is XFS, previously tried with EXT4 - results are the
>>>> same.
>>>> >
>>>> > This is the part of VM XML definition generated by CloudStack:
>>>> >
>>>> > <devices>
>>>> > <emulator>/usr/bin/kvm-spice</emulator>
>>>> > <disk type='file' device='disk'>
>>>> > <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
>>>> > <source
>>>> > file='/var/lib/libvirt/images/6809dbd0-4a15-4014-9322-fe9010695934'/>
>>>> > <backingStore type='file' index='1'>
>>>> > <format type='raw'/>
>>>> > <source
>>>> > file='/var/lib/libvirt/images/ac43742c-3991-4be1-bff1-7617bf4fc6ef'/>
>>>> > <backingStore/>
>>>> > </backingStore>
>>>> > <target dev='sda' bus='scsi'/>
>>>> > <iotune>
>>>> > <read_bytes_sec>1048576000</read_bytes_sec>
>>>> > <write_bytes_sec>524288000</write_bytes_sec>
>>>> > <read_iops_sec>100000</read_iops_sec>
>>>> > <write_iops_sec>50000</write_iops_sec>
>>>> > </iotune>
>>>> > <serial>6809dbd04a1540149322</serial>
>>>> > <alias name='scsi0-0-0-0'/>
>>>> > <address type='drive' controller='0' bus='0' target='0'
>>>> unit='0'/>
>>>> > </disk>
>>>> > <disk type='file' device='cdrom'>
>>>> > <driver name='qemu' type='raw'/>
>>>> > <backingStore/>
>>>> > <target dev='hdc' bus='ide'/>
>>>> > <readonly/>
>>>> > <alias name='ide0-1-0'/>
>>>> > <address type='drive' controller='0' bus='1' target='0'
>>>> unit='0'/>
>>>> > </disk>
>>>> > <controller type='scsi' index='0' model='virtio-scsi'>
>>>> > <alias name='scsi0'/>
>>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>>>> > function='0x0'/>
>>>> > </controller>
>>>> >
>>>> > So, what I see now, is that it works slower than couple of two Samsung
>>>> 960
>>>> > PRO which is extremely strange.
>>>> >
>>>> > Thanks in advance.
>>>> >
>>>> >
>>>> > --
>>>> > With best regards, Ivan Kudryavtsev
>>>> > Bitworks LLC
>>>> > Cell RU: +7-923-414-1515
>>>> > Cell USA: +1-201-257-1512
>>>> > WWW: http://bitworks.software/ <http://bw-sw.com/>
>>>>
Re: Poor NVMe Performance with KVM
Posted by Nux! <nu...@li.nux.ro>.
What happens when you set deadline scheduler in both HV and guest?
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Ivan Kudryavtsev" <ku...@bw-sw.com>
> To: "users" <us...@cloudstack.apache.org>, "dev" <de...@cloudstack.apache.org>
> Sent: Friday, 17 May, 2019 14:16:31
> Subject: Re: Poor NVMe Performance with KVM
> BTW, You may think that the improvement is achieved by caching, but I clear
> the cache with
> sync & echo 3 > /proc/sys/vm/drop_caches
>
> So, can't claim for sure, need other opinion, but looks like for NVMe,
> writethrough must be used if you want high IO rate. At least with Intel
> p4500.
>
>
> пт, 17 мая 2019 г., 20:04 Ivan Kudryavtsev <ku...@bw-sw.com>:
>
>> Well, just FYI, I changed cache_mode from NULL (none), to writethrough
>> directly in DB and the performance boosted greatly. It may be an important
>> feature for NVME drives.
>>
>> Currently, on 4.11, the user can set cache-mode for disk offerings, but
>> cannot for service offerings, which are translated to cache=none
>> corresponding disk offerings.
>>
>> The only way is to use SQL to change that for root disk disk offerings.
>> CreateServiceOffering API doesn't support cache mode. It can be a serious
>> limitation for NVME users, because by default they could meet poor
>> read/write performance.
>>
>> пт, 17 мая 2019 г., 19:30 Ivan Kudryavtsev <ku...@bw-sw.com>:
>>
>>> Darius, thanks for your participation,
>>>
>>> first, I used 4.14 kernel which is the default one for my cluster. Next,
>>> switched to 4.15 with dist-upgrade.
>>>
>>> Do you have an idea how to turn on amount of queues for virtio-scsi with
>>> Cloudstack?
>>>
>>> пт, 17 мая 2019 г., 19:26 Darius Kasparavičius <da...@gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> I can see a few issues with your xml file. You can try using "queues"
>>>> inside your disk definitions. This should help a little, not sure by
>>>> how much for your case, but for my specific it went up by almost the
>>>> number of queues. Also try cache directsync or writethrough. You
>>>> should switch kernel if bugs are still there with 4.15 kernel.
>>>>
>>>> On Fri, May 17, 2019 at 12:14 PM Ivan Kudryavtsev
>>>> <ku...@bw-sw.com> wrote:
>>>> >
>>>> > Hello, colleagues.
>>>> >
>>>> > Hope, someone could help me. I just deployed a new VM host with Intel
>>>> P4500
>>>> > local storage NVMe drive.
>>>> >
>>>> > From Hypervisor host I can get expected performance, 200K RIOPS, 3GBs
>>>> with
>>>> > FIO, write performance is also high as expected.
>>>> >
>>>> > I've created a new KVM VM Service offering with virtio-scsi controller
>>>> > (tried virtio as well) and VM is deployed. Now I try to benchmark it
>>>> with
>>>> > FIO. Results are very strange:
>>>> >
>>>> > 1. Read/Write with large blocks (1M) shows expected performance (my
>>>> limits
>>>> > are R=1000/W=500 MBs).
>>>> >
>>>> > 2. Write with direct=0 leads to expected 50K IOPS, while write with
>>>> > direct=1 leads to very moderate 2-3K IOPS.
>>>> >
>>>> > 3. Read with direct=0, direct=1 both lead to 3000 IOPS.
>>>> >
>>>> > During the benchmark I see VM IOWAIT=20%, while host IOWAIT is 0%
>>>> which is
>>>> > strange.
>>>> >
>>>> > So, basically, from inside VM my NVMe works very slow when small IOPS
>>>> are
>>>> > executed. From the host, it works great.
>>>> >
>>>> > I tried to mount the volume with NBD to /dev/nbd0 and benchmark. Read
>>>> > performance is nice. Maybe someone managed to use NVME with KVM with
>>>> small
>>>> > IOPS?
>>>> >
>>>> > The filesystem is XFS, previously tried with EXT4 - results are the
>>>> same.
>>>> >
>>>> > This is the part of VM XML definition generated by CloudStack:
>>>> >
>>>> > <devices>
>>>> > <emulator>/usr/bin/kvm-spice</emulator>
>>>> > <disk type='file' device='disk'>
>>>> > <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
>>>> > <source
>>>> > file='/var/lib/libvirt/images/6809dbd0-4a15-4014-9322-fe9010695934'/>
>>>> > <backingStore type='file' index='1'>
>>>> > <format type='raw'/>
>>>> > <source
>>>> > file='/var/lib/libvirt/images/ac43742c-3991-4be1-bff1-7617bf4fc6ef'/>
>>>> > <backingStore/>
>>>> > </backingStore>
>>>> > <target dev='sda' bus='scsi'/>
>>>> > <iotune>
>>>> > <read_bytes_sec>1048576000</read_bytes_sec>
>>>> > <write_bytes_sec>524288000</write_bytes_sec>
>>>> > <read_iops_sec>100000</read_iops_sec>
>>>> > <write_iops_sec>50000</write_iops_sec>
>>>> > </iotune>
>>>> > <serial>6809dbd04a1540149322</serial>
>>>> > <alias name='scsi0-0-0-0'/>
>>>> > <address type='drive' controller='0' bus='0' target='0'
>>>> unit='0'/>
>>>> > </disk>
>>>> > <disk type='file' device='cdrom'>
>>>> > <driver name='qemu' type='raw'/>
>>>> > <backingStore/>
>>>> > <target dev='hdc' bus='ide'/>
>>>> > <readonly/>
>>>> > <alias name='ide0-1-0'/>
>>>> > <address type='drive' controller='0' bus='1' target='0'
>>>> unit='0'/>
>>>> > </disk>
>>>> > <controller type='scsi' index='0' model='virtio-scsi'>
>>>> > <alias name='scsi0'/>
>>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>>>> > function='0x0'/>
>>>> > </controller>
>>>> >
>>>> > So, what I see now, is that it works slower than couple of two Samsung
>>>> 960
>>>> > PRO which is extremely strange.
>>>> >
>>>> > Thanks in advance.
>>>> >
>>>> >
>>>> > --
>>>> > With best regards, Ivan Kudryavtsev
>>>> > Bitworks LLC
>>>> > Cell RU: +7-923-414-1515
>>>> > Cell USA: +1-201-257-1512
>>>> > WWW: http://bitworks.software/ <http://bw-sw.com/>
>>>>
Re: Poor NVMe Performance with KVM
Posted by Ivan Kudryavtsev <ku...@bw-sw.com>.
BTW, You may think that the improvement is achieved by caching, but I clear
the cache with
sync & echo 3 > /proc/sys/vm/drop_caches
So, can't claim for sure, need other opinion, but looks like for NVMe,
writethrough must be used if you want high IO rate. At least with Intel
p4500.
пт, 17 мая 2019 г., 20:04 Ivan Kudryavtsev <ku...@bw-sw.com>:
> Well, just FYI, I changed cache_mode from NULL (none), to writethrough
> directly in DB and the performance boosted greatly. It may be an important
> feature for NVME drives.
>
> Currently, on 4.11, the user can set cache-mode for disk offerings, but
> cannot for service offerings, which are translated to cache=none
> corresponding disk offerings.
>
> The only way is to use SQL to change that for root disk disk offerings.
> CreateServiceOffering API doesn't support cache mode. It can be a serious
> limitation for NVME users, because by default they could meet poor
> read/write performance.
>
> пт, 17 мая 2019 г., 19:30 Ivan Kudryavtsev <ku...@bw-sw.com>:
>
>> Darius, thanks for your participation,
>>
>> first, I used 4.14 kernel which is the default one for my cluster. Next,
>> switched to 4.15 with dist-upgrade.
>>
>> Do you have an idea how to turn on amount of queues for virtio-scsi with
>> Cloudstack?
>>
>> пт, 17 мая 2019 г., 19:26 Darius Kasparavičius <da...@gmail.com>:
>>
>>> Hi,
>>>
>>> I can see a few issues with your xml file. You can try using "queues"
>>> inside your disk definitions. This should help a little, not sure by
>>> how much for your case, but for my specific it went up by almost the
>>> number of queues. Also try cache directsync or writethrough. You
>>> should switch kernel if bugs are still there with 4.15 kernel.
>>>
>>> On Fri, May 17, 2019 at 12:14 PM Ivan Kudryavtsev
>>> <ku...@bw-sw.com> wrote:
>>> >
>>> > Hello, colleagues.
>>> >
>>> > Hope, someone could help me. I just deployed a new VM host with Intel
>>> P4500
>>> > local storage NVMe drive.
>>> >
>>> > From Hypervisor host I can get expected performance, 200K RIOPS, 3GBs
>>> with
>>> > FIO, write performance is also high as expected.
>>> >
>>> > I've created a new KVM VM Service offering with virtio-scsi controller
>>> > (tried virtio as well) and VM is deployed. Now I try to benchmark it
>>> with
>>> > FIO. Results are very strange:
>>> >
>>> > 1. Read/Write with large blocks (1M) shows expected performance (my
>>> limits
>>> > are R=1000/W=500 MBs).
>>> >
>>> > 2. Write with direct=0 leads to expected 50K IOPS, while write with
>>> > direct=1 leads to very moderate 2-3K IOPS.
>>> >
>>> > 3. Read with direct=0, direct=1 both lead to 3000 IOPS.
>>> >
>>> > During the benchmark I see VM IOWAIT=20%, while host IOWAIT is 0%
>>> which is
>>> > strange.
>>> >
>>> > So, basically, from inside VM my NVMe works very slow when small IOPS
>>> are
>>> > executed. From the host, it works great.
>>> >
>>> > I tried to mount the volume with NBD to /dev/nbd0 and benchmark. Read
>>> > performance is nice. Maybe someone managed to use NVME with KVM with
>>> small
>>> > IOPS?
>>> >
>>> > The filesystem is XFS, previously tried with EXT4 - results are the
>>> same.
>>> >
>>> > This is the part of VM XML definition generated by CloudStack:
>>> >
>>> > <devices>
>>> > <emulator>/usr/bin/kvm-spice</emulator>
>>> > <disk type='file' device='disk'>
>>> > <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
>>> > <source
>>> > file='/var/lib/libvirt/images/6809dbd0-4a15-4014-9322-fe9010695934'/>
>>> > <backingStore type='file' index='1'>
>>> > <format type='raw'/>
>>> > <source
>>> > file='/var/lib/libvirt/images/ac43742c-3991-4be1-bff1-7617bf4fc6ef'/>
>>> > <backingStore/>
>>> > </backingStore>
>>> > <target dev='sda' bus='scsi'/>
>>> > <iotune>
>>> > <read_bytes_sec>1048576000</read_bytes_sec>
>>> > <write_bytes_sec>524288000</write_bytes_sec>
>>> > <read_iops_sec>100000</read_iops_sec>
>>> > <write_iops_sec>50000</write_iops_sec>
>>> > </iotune>
>>> > <serial>6809dbd04a1540149322</serial>
>>> > <alias name='scsi0-0-0-0'/>
>>> > <address type='drive' controller='0' bus='0' target='0'
>>> unit='0'/>
>>> > </disk>
>>> > <disk type='file' device='cdrom'>
>>> > <driver name='qemu' type='raw'/>
>>> > <backingStore/>
>>> > <target dev='hdc' bus='ide'/>
>>> > <readonly/>
>>> > <alias name='ide0-1-0'/>
>>> > <address type='drive' controller='0' bus='1' target='0'
>>> unit='0'/>
>>> > </disk>
>>> > <controller type='scsi' index='0' model='virtio-scsi'>
>>> > <alias name='scsi0'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>>> > function='0x0'/>
>>> > </controller>
>>> >
>>> > So, what I see now, is that it works slower than couple of two Samsung
>>> 960
>>> > PRO which is extremely strange.
>>> >
>>> > Thanks in advance.
>>> >
>>> >
>>> > --
>>> > With best regards, Ivan Kudryavtsev
>>> > Bitworks LLC
>>> > Cell RU: +7-923-414-1515
>>> > Cell USA: +1-201-257-1512
>>> > WWW: http://bitworks.software/ <http://bw-sw.com/>
>>>
>>
Re: Poor NVMe Performance with KVM
Posted by Ivan Kudryavtsev <ku...@bw-sw.com>.
BTW, You may think that the improvement is achieved by caching, but I clear
the cache with
sync & echo 3 > /proc/sys/vm/drop_caches
So, can't claim for sure, need other opinion, but looks like for NVMe,
writethrough must be used if you want high IO rate. At least with Intel
p4500.
пт, 17 мая 2019 г., 20:04 Ivan Kudryavtsev <ku...@bw-sw.com>:
> Well, just FYI, I changed cache_mode from NULL (none), to writethrough
> directly in DB and the performance boosted greatly. It may be an important
> feature for NVME drives.
>
> Currently, on 4.11, the user can set cache-mode for disk offerings, but
> cannot for service offerings, which are translated to cache=none
> corresponding disk offerings.
>
> The only way is to use SQL to change that for root disk disk offerings.
> CreateServiceOffering API doesn't support cache mode. It can be a serious
> limitation for NVME users, because by default they could meet poor
> read/write performance.
>
> пт, 17 мая 2019 г., 19:30 Ivan Kudryavtsev <ku...@bw-sw.com>:
>
>> Darius, thanks for your participation,
>>
>> first, I used 4.14 kernel which is the default one for my cluster. Next,
>> switched to 4.15 with dist-upgrade.
>>
>> Do you have an idea how to turn on amount of queues for virtio-scsi with
>> Cloudstack?
>>
>> пт, 17 мая 2019 г., 19:26 Darius Kasparavičius <da...@gmail.com>:
>>
>>> Hi,
>>>
>>> I can see a few issues with your xml file. You can try using "queues"
>>> inside your disk definitions. This should help a little, not sure by
>>> how much for your case, but for my specific it went up by almost the
>>> number of queues. Also try cache directsync or writethrough. You
>>> should switch kernel if bugs are still there with 4.15 kernel.
>>>
>>> On Fri, May 17, 2019 at 12:14 PM Ivan Kudryavtsev
>>> <ku...@bw-sw.com> wrote:
>>> >
>>> > Hello, colleagues.
>>> >
>>> > Hope, someone could help me. I just deployed a new VM host with Intel
>>> P4500
>>> > local storage NVMe drive.
>>> >
>>> > From Hypervisor host I can get expected performance, 200K RIOPS, 3GBs
>>> with
>>> > FIO, write performance is also high as expected.
>>> >
>>> > I've created a new KVM VM Service offering with virtio-scsi controller
>>> > (tried virtio as well) and VM is deployed. Now I try to benchmark it
>>> with
>>> > FIO. Results are very strange:
>>> >
>>> > 1. Read/Write with large blocks (1M) shows expected performance (my
>>> limits
>>> > are R=1000/W=500 MBs).
>>> >
>>> > 2. Write with direct=0 leads to expected 50K IOPS, while write with
>>> > direct=1 leads to very moderate 2-3K IOPS.
>>> >
>>> > 3. Read with direct=0, direct=1 both lead to 3000 IOPS.
>>> >
>>> > During the benchmark I see VM IOWAIT=20%, while host IOWAIT is 0%
>>> which is
>>> > strange.
>>> >
>>> > So, basically, from inside VM my NVMe works very slow when small IOPS
>>> are
>>> > executed. From the host, it works great.
>>> >
>>> > I tried to mount the volume with NBD to /dev/nbd0 and benchmark. Read
>>> > performance is nice. Maybe someone managed to use NVME with KVM with
>>> small
>>> > IOPS?
>>> >
>>> > The filesystem is XFS, previously tried with EXT4 - results are the
>>> same.
>>> >
>>> > This is the part of VM XML definition generated by CloudStack:
>>> >
>>> > <devices>
>>> > <emulator>/usr/bin/kvm-spice</emulator>
>>> > <disk type='file' device='disk'>
>>> > <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
>>> > <source
>>> > file='/var/lib/libvirt/images/6809dbd0-4a15-4014-9322-fe9010695934'/>
>>> > <backingStore type='file' index='1'>
>>> > <format type='raw'/>
>>> > <source
>>> > file='/var/lib/libvirt/images/ac43742c-3991-4be1-bff1-7617bf4fc6ef'/>
>>> > <backingStore/>
>>> > </backingStore>
>>> > <target dev='sda' bus='scsi'/>
>>> > <iotune>
>>> > <read_bytes_sec>1048576000</read_bytes_sec>
>>> > <write_bytes_sec>524288000</write_bytes_sec>
>>> > <read_iops_sec>100000</read_iops_sec>
>>> > <write_iops_sec>50000</write_iops_sec>
>>> > </iotune>
>>> > <serial>6809dbd04a1540149322</serial>
>>> > <alias name='scsi0-0-0-0'/>
>>> > <address type='drive' controller='0' bus='0' target='0'
>>> unit='0'/>
>>> > </disk>
>>> > <disk type='file' device='cdrom'>
>>> > <driver name='qemu' type='raw'/>
>>> > <backingStore/>
>>> > <target dev='hdc' bus='ide'/>
>>> > <readonly/>
>>> > <alias name='ide0-1-0'/>
>>> > <address type='drive' controller='0' bus='1' target='0'
>>> unit='0'/>
>>> > </disk>
>>> > <controller type='scsi' index='0' model='virtio-scsi'>
>>> > <alias name='scsi0'/>
>>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>>> > function='0x0'/>
>>> > </controller>
>>> >
>>> > So, what I see now, is that it works slower than couple of two Samsung
>>> 960
>>> > PRO which is extremely strange.
>>> >
>>> > Thanks in advance.
>>> >
>>> >
>>> > --
>>> > With best regards, Ivan Kudryavtsev
>>> > Bitworks LLC
>>> > Cell RU: +7-923-414-1515
>>> > Cell USA: +1-201-257-1512
>>> > WWW: http://bitworks.software/ <http://bw-sw.com/>
>>>
>>
Re: Poor NVMe Performance with KVM
Posted by Wido den Hollander <wi...@widodh.nl>.
You also might want to set the allocation mode to something else then
shared. This causes the qcow2 metadata to be pre-allocated and that will
improve performance.
Wido
On 5/17/19 3:04 PM, Ivan Kudryavtsev wrote:
> Well, just FYI, I changed cache_mode from NULL (none), to writethrough
> directly in DB and the performance boosted greatly. It may be an important
> feature for NVME drives.
>
> Currently, on 4.11, the user can set cache-mode for disk offerings, but
> cannot for service offerings, which are translated to cache=none
> corresponding disk offerings.
>
> The only way is to use SQL to change that for root disk disk offerings.
> CreateServiceOffering API doesn't support cache mode. It can be a serious
> limitation for NVME users, because by default they could meet poor
> read/write performance.
>
> пт, 17 мая 2019 г., 19:30 Ivan Kudryavtsev <ku...@bw-sw.com>:
>
>> Darius, thanks for your participation,
>>
>> first, I used 4.14 kernel which is the default one for my cluster. Next,
>> switched to 4.15 with dist-upgrade.
>>
>> Do you have an idea how to turn on amount of queues for virtio-scsi with
>> Cloudstack?
>>
>> пт, 17 мая 2019 г., 19:26 Darius Kasparavičius <da...@gmail.com>:
>>
>>> Hi,
>>>
>>> I can see a few issues with your xml file. You can try using "queues"
>>> inside your disk definitions. This should help a little, not sure by
>>> how much for your case, but for my specific it went up by almost the
>>> number of queues. Also try cache directsync or writethrough. You
>>> should switch kernel if bugs are still there with 4.15 kernel.
>>>
>>> On Fri, May 17, 2019 at 12:14 PM Ivan Kudryavtsev
>>> <ku...@bw-sw.com> wrote:
>>>>
>>>> Hello, colleagues.
>>>>
>>>> Hope, someone could help me. I just deployed a new VM host with Intel
>>> P4500
>>>> local storage NVMe drive.
>>>>
>>>> From Hypervisor host I can get expected performance, 200K RIOPS, 3GBs
>>> with
>>>> FIO, write performance is also high as expected.
>>>>
>>>> I've created a new KVM VM Service offering with virtio-scsi controller
>>>> (tried virtio as well) and VM is deployed. Now I try to benchmark it
>>> with
>>>> FIO. Results are very strange:
>>>>
>>>> 1. Read/Write with large blocks (1M) shows expected performance (my
>>> limits
>>>> are R=1000/W=500 MBs).
>>>>
>>>> 2. Write with direct=0 leads to expected 50K IOPS, while write with
>>>> direct=1 leads to very moderate 2-3K IOPS.
>>>>
>>>> 3. Read with direct=0, direct=1 both lead to 3000 IOPS.
>>>>
>>>> During the benchmark I see VM IOWAIT=20%, while host IOWAIT is 0% which
>>> is
>>>> strange.
>>>>
>>>> So, basically, from inside VM my NVMe works very slow when small IOPS
>>> are
>>>> executed. From the host, it works great.
>>>>
>>>> I tried to mount the volume with NBD to /dev/nbd0 and benchmark. Read
>>>> performance is nice. Maybe someone managed to use NVME with KVM with
>>> small
>>>> IOPS?
>>>>
>>>> The filesystem is XFS, previously tried with EXT4 - results are the
>>> same.
>>>>
>>>> This is the part of VM XML definition generated by CloudStack:
>>>>
>>>> <devices>
>>>> <emulator>/usr/bin/kvm-spice</emulator>
>>>> <disk type='file' device='disk'>
>>>> <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
>>>> <source
>>>> file='/var/lib/libvirt/images/6809dbd0-4a15-4014-9322-fe9010695934'/>
>>>> <backingStore type='file' index='1'>
>>>> <format type='raw'/>
>>>> <source
>>>> file='/var/lib/libvirt/images/ac43742c-3991-4be1-bff1-7617bf4fc6ef'/>
>>>> <backingStore/>
>>>> </backingStore>
>>>> <target dev='sda' bus='scsi'/>
>>>> <iotune>
>>>> <read_bytes_sec>1048576000</read_bytes_sec>
>>>> <write_bytes_sec>524288000</write_bytes_sec>
>>>> <read_iops_sec>100000</read_iops_sec>
>>>> <write_iops_sec>50000</write_iops_sec>
>>>> </iotune>
>>>> <serial>6809dbd04a1540149322</serial>
>>>> <alias name='scsi0-0-0-0'/>
>>>> <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>>> </disk>
>>>> <disk type='file' device='cdrom'>
>>>> <driver name='qemu' type='raw'/>
>>>> <backingStore/>
>>>> <target dev='hdc' bus='ide'/>
>>>> <readonly/>
>>>> <alias name='ide0-1-0'/>
>>>> <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>>>> </disk>
>>>> <controller type='scsi' index='0' model='virtio-scsi'>
>>>> <alias name='scsi0'/>
>>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>>>> function='0x0'/>
>>>> </controller>
>>>>
>>>> So, what I see now, is that it works slower than couple of two Samsung
>>> 960
>>>> PRO which is extremely strange.
>>>>
>>>> Thanks in advance.
>>>>
>>>>
>>>> --
>>>> With best regards, Ivan Kudryavtsev
>>>> Bitworks LLC
>>>> Cell RU: +7-923-414-1515
>>>> Cell USA: +1-201-257-1512
>>>> WWW: http://bitworks.software/ <http://bw-sw.com/>
>>>
>>
>