You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Rohit Yadav <ro...@shapeblue.com> on 2016/12/21 11:00:09 UTC
[DISCUSS][KVM][BUG] Detaching of volume fails on KVM
All,
Based on results from recent Trillian test runs [1], I've discovered that on KVM (CentOS7) based detaching a volume fails to update the virt/domain xml and fails to remove the xml. So, while the agent and cloudstack-mgmt server succeeds, the entry in the xml is not removed. When the volume is attached again, we can an error like:
Failed to attach volume xxx to VM VM-yyyy; org.libvirt.LibvirtException: XML error: target 'vdb' duplicated for disk sources '/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-a452-43de-8c6b-948dc44aae21' and '/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-a452-43de-8c6b-948dc44aae21'This is seen in agent logs:
Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: DEBUG [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-2:) (logid:0648ae70) Detaching device: <disk device='disk' type='file'>
Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: <driver name='qemu' type='qcow2' cache='none' />
Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: <source file='/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-a452-43de-8c6b-948dc44aae21'/>
Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: <target dev='vdb' bus='virtio'/>
Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: </disk>
While, after above completes. This is still seen in the VM's dumped xml:
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-a452-43de-8c6b-948dc44aae21'/>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<serial>af85ff7ea45243de8c6b</serial>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
Steps to reproduce:
1. Deploy a VM, create a data volume disk and attach to the VM.
2. Detach the volume.
3. Attach the volume to the same VM again, exception is caught.Thoughts, comments?[1] https://github.com/apache/cloudstack/pull/1837
Regards.
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: [DISCUSS][KVM][BUG] Detaching of volume fails on KVM
Posted by Wido den Hollander <wi...@widodh.nl>.
> Op 21 december 2016 om 12:00 schreef Rohit Yadav <ro...@shapeblue.com>:
>
>
> All,
>
>
> Based on results from recent Trillian test runs [1], I've discovered that on KVM (CentOS7) based detaching a volume fails to update the virt/domain xml and fails to remove the xml. So, while the agent and cloudstack-mgmt server succeeds, the entry in the xml is not removed. When the volume is attached again, we can an error like:
>
>
> Failed to attach volume xxx to VM VM-yyyy; org.libvirt.LibvirtException: XML error: target 'vdb' duplicated for disk sources '/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-a452-43de-8c6b-948dc44aae21' and '/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-a452-43de-8c6b-948dc44aae21'This is seen in agent logs:
>
> Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: DEBUG [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-2:) (logid:0648ae70) Detaching device: <disk device='disk' type='file'>
> Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: <driver name='qemu' type='qcow2' cache='none' />
> Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: <source file='/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-a452-43de-8c6b-948dc44aae21'/>
> Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: <target dev='vdb' bus='virtio'/>
> Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: </disk>
>
> While, after above completes. This is still seen in the VM's dumped xml:
> <disk type='file' device='disk'>
> <driver name='qemu' type='qcow2' cache='none'/>
> <source file='/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-a452-43de-8c6b-948dc44aae21'/>
> <backingStore/>
> <target dev='vdb' bus='virtio'/>
> <serial>af85ff7ea45243de8c6b</serial>
> <alias name='virtio-disk1'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
> </disk>
> Steps to reproduce:
> 1. Deploy a VM, create a data volume disk and attach to the VM.
> 2. Detach the volume.
> 3. Attach the volume to the same VM again, exception is caught.Thoughts, comments?[1] https://github.com/apache/cloudstack/pull/1837
> Regards.
>
Isn't this a Qemu bug? That it thinks it detached it, but it doesn't. Or a libvirt thing?
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
Re: [DISCUSS][KVM][BUG] Detaching of volume fails on KVM
Posted by Rohit Yadav <ro...@shapeblue.com>.
All,
I've tested this against CentOS6 and able to reproduce this issue as well.
Regards.
________________________________
From: Rohit Yadav <ro...@shapeblue.com>
Sent: 21 December 2016 16:30:09
To: dev@cloudstack.apache.org
Subject: [DISCUSS][KVM][BUG] Detaching of volume fails on KVM
All,
Based on results from recent Trillian test runs [1], I've discovered that on KVM (CentOS7) based detaching a volume fails to update the virt/domain xml and fails to remove the xml. So, while the agent and cloudstack-mgmt server succeeds, the entry in the xml is not removed. When the volume is attached again, we can an error like:
Failed to attach volume xxx to VM VM-yyyy; org.libvirt.LibvirtException: XML error: target 'vdb' duplicated for disk sources '/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-a452-43de-8c6b-948dc44aae21' and '/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-a452-43de-8c6b-948dc44aae21'This is seen in agent logs:
Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: DEBUG [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-2:) (logid:0648ae70) Detaching device: <disk device='disk' type='file'>
Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: <driver name='qemu' type='qcow2' cache='none' />
Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: <source file='/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-a452-43de-8c6b-948dc44aae21'/>
Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: <target dev='vdb' bus='virtio'/>
Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: </disk>
While, after above completes. This is still seen in the VM's dumped xml:
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-a452-43de-8c6b-948dc44aae21'/>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<serial>af85ff7ea45243de8c6b</serial>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
Steps to reproduce:
1. Deploy a VM, create a data volume disk and attach to the VM.
2. Detach the volume.
3. Attach the volume to the same VM again, exception is caught.Thoughts, comments?[1] https://github.com/apache/cloudstack/pull/1837
Regards.
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: [DISCUSS][KVM][BUG] Detaching of volume fails on KVM
Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi Wei,
Thanks for sharing.
I discovered the issue due to failing test_volumes tests which were previously passing on Trillian's CentOS7 based kvm environment:
https://github.com/apache/cloudstack/pull/1837
<https://github.com/apache/cloudstack/pull/1837>Trillian deployed environment are all automated and unless libvirt/qemu packages were changes/updated from upstream repos, there was no change added to Trillian/CentOS7. I'll have a look at actual test as well.
One thing we can do is that after detaching the disk, CloudStack can check the domain's XML to see if the disk was actually detached? This would reflect and notify admin/user whether the operation actually was successful?
Thanks Wido, I think it could be related to libvirt/OS as well.
Regards.
________________________________
From: Wei ZHOU <us...@gmail.com>
Sent: 21 December 2016 17:26:19
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS][KVM][BUG] Detaching of volume fails on KVM
Hi Rohit,
I donot think it is an issue in cloudstack.
We have this issue from long time ago, it still exist now.
I have a testing just now, by virsh command, not cloudstack.
================this is the working one===================
root@KVM015:~# virsh domblklist 39
Target Source
------------------------------------------------
vda
/mnt/1dcbc42c-99bc-3276-9d86-4ad81ef1ad8e/75d35578-ed6d-4019-8239-c2d3ff87af25
hdc -
root@KVM015:~# virsh attach-disk 39
/mnt/f773b66d-fd8c-3576-aa37-e3f0e685b183/ba11047c-ce69-4982-892a-38b343b4f841
vdb
Disk attached successfully
root@KVM015:~#
root@KVM015:~# virsh domblklist 39
Target Source
------------------------------------------------
vda
/mnt/1dcbc42c-99bc-3276-9d86-4ad81ef1ad8e/75d35578-ed6d-4019-8239-c2d3ff87af25
vdb
/mnt/f773b66d-fd8c-3576-aa37-e3f0e685b183/ba11047c-ce69-4982-892a-38b343b4f841
hdc -
root@KVM015:~# virsh detach-disk 39 vdb
Disk detached successfully
root@KVM015:~# virsh domblklist 39
Target Source
------------------------------------------------
vda
/mnt/1dcbc42c-99bc-3276-9d86-4ad81ef1ad8e/75d35578-ed6d-4019-8239-c2d3ff87af25
hdc -
============this is not working================
root@KVM015:~# virsh domblklist 26
Target Source
------------------------------------------------
vda
/mnt/f773b66d-fd8c-3576-aa37-e3f0e685b183/2311416f-b778-4490-8365-cfbad2214842
vdb
/mnt/f773b66d-fd8c-3576-aa37-e3f0e685b183/ba11047c-ce69-4982-892a-38b343b4f841
hdc -
root@KVM015:~# virsh detach-disk i-2-7585-VM vdb
Disk detached successfully
root@KVM015:~# virsh domblklist 26
Target Source
------------------------------------------------
vda
/mnt/f773b66d-fd8c-3576-aa37-e3f0e685b183/2311416f-b778-4490-8365-cfbad2214842
vdb
/mnt/f773b66d-fd8c-3576-aa37-e3f0e685b183/ba11047c-ce69-4982-892a-38b343b4f841
hdc -
root@KVM015:~# virsh detach-disk i-2-7585-VM vdb
Disk detached successfully
root@KVM015:~# virsh domblklist 26
Target Source
------------------------------------------------
vda
/mnt/f773b66d-fd8c-3576-aa37-e3f0e685b183/2311416f-b778-4490-8365-cfbad2214842
vdb
/mnt/f773b66d-fd8c-3576-aa37-e3f0e685b183/ba11047c-ce69-4982-892a-38b343b4f841
hdc -
root@KVM015:~# virsh attach-disk 26
/mnt/f773b66d-fd8c-3576-aa37-e3f0e685b183/ba11047c-ce69-4982-892a-38b343b4f841
vdb
error: Failed to attach disk
error: operation failed: target vdb already exists
================end==================
I believe this is highly related to the OS and configuration in VM, not
hypervisor, or cloudstack.
IN my testing, I use Ubuntu 12.04 as hypervisor, it works if vm OS is
CentOS7/CentOS6/Ubuntu 16.04, but not working if it is Ubuntu 12.04 .
-Wei
2016-12-21 12:00 GMT+01:00 Rohit Yadav <ro...@shapeblue.com>:
> All,
>
>
> Based on results from recent Trillian test runs [1], I've discovered that
> on KVM (CentOS7) based detaching a volume fails to update the virt/domain
> xml and fails to remove the xml. So, while the agent and cloudstack-mgmt
> server succeeds, the entry in the xml is not removed. When the volume is
> attached again, we can an error like:
>
>
> Failed to attach volume xxx to VM VM-yyyy; org.libvirt.LibvirtException:
> XML error: target 'vdb' duplicated for disk sources
> '/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-a452-43de-8c6b-948dc44aae21'
> and '/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-a452-43de-8c6b-948dc44aae21'This
> is seen in agent logs:
>
> Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: DEBUG [kvm.storage.KVMStorageProcessor]
> (agentRequest-Handler-2:) (logid:0648ae70) Detaching device: <disk
> device='disk' type='file'>
> Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: <driver
> name='qemu' type='qcow2' cache='none' />
> Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: <source
> file='/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-
> a452-43de-8c6b-948dc44aae21'/>
> Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: <target dev='vdb'
> bus='virtio'/>
> Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: </disk>
>
> While, after above completes. This is still seen in the VM's dumped xml:
> <disk type='file' device='disk'>
> <driver name='qemu' type='qcow2' cache='none'/>
> <source file='/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-
> a452-43de-8c6b-948dc44aae21'/>
> <backingStore/>
> <target dev='vdb' bus='virtio'/>
> <serial>af85ff7ea45243de8c6b</serial>
> <alias name='virtio-disk1'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x0'/>
> </disk>
> Steps to reproduce:
> 1. Deploy a VM, create a data volume disk and attach to the VM.
> 2. Detach the volume.
> 3. Attach the volume to the same VM again, exception is caught.Thoughts,
> comments?[1] https://github.com/apache/cloudstack/pull/1837
> Regards.
>
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: [DISCUSS][KVM][BUG] Detaching of volume fails on KVM
Posted by Wei ZHOU <us...@gmail.com>.
Hi Rohit,
I donot think it is an issue in cloudstack.
We have this issue from long time ago, it still exist now.
I have a testing just now, by virsh command, not cloudstack.
================this is the working one===================
root@KVM015:~# virsh domblklist 39
Target Source
------------------------------------------------
vda
/mnt/1dcbc42c-99bc-3276-9d86-4ad81ef1ad8e/75d35578-ed6d-4019-8239-c2d3ff87af25
hdc -
root@KVM015:~# virsh attach-disk 39
/mnt/f773b66d-fd8c-3576-aa37-e3f0e685b183/ba11047c-ce69-4982-892a-38b343b4f841
vdb
Disk attached successfully
root@KVM015:~#
root@KVM015:~# virsh domblklist 39
Target Source
------------------------------------------------
vda
/mnt/1dcbc42c-99bc-3276-9d86-4ad81ef1ad8e/75d35578-ed6d-4019-8239-c2d3ff87af25
vdb
/mnt/f773b66d-fd8c-3576-aa37-e3f0e685b183/ba11047c-ce69-4982-892a-38b343b4f841
hdc -
root@KVM015:~# virsh detach-disk 39 vdb
Disk detached successfully
root@KVM015:~# virsh domblklist 39
Target Source
------------------------------------------------
vda
/mnt/1dcbc42c-99bc-3276-9d86-4ad81ef1ad8e/75d35578-ed6d-4019-8239-c2d3ff87af25
hdc -
============this is not working================
root@KVM015:~# virsh domblklist 26
Target Source
------------------------------------------------
vda
/mnt/f773b66d-fd8c-3576-aa37-e3f0e685b183/2311416f-b778-4490-8365-cfbad2214842
vdb
/mnt/f773b66d-fd8c-3576-aa37-e3f0e685b183/ba11047c-ce69-4982-892a-38b343b4f841
hdc -
root@KVM015:~# virsh detach-disk i-2-7585-VM vdb
Disk detached successfully
root@KVM015:~# virsh domblklist 26
Target Source
------------------------------------------------
vda
/mnt/f773b66d-fd8c-3576-aa37-e3f0e685b183/2311416f-b778-4490-8365-cfbad2214842
vdb
/mnt/f773b66d-fd8c-3576-aa37-e3f0e685b183/ba11047c-ce69-4982-892a-38b343b4f841
hdc -
root@KVM015:~# virsh detach-disk i-2-7585-VM vdb
Disk detached successfully
root@KVM015:~# virsh domblklist 26
Target Source
------------------------------------------------
vda
/mnt/f773b66d-fd8c-3576-aa37-e3f0e685b183/2311416f-b778-4490-8365-cfbad2214842
vdb
/mnt/f773b66d-fd8c-3576-aa37-e3f0e685b183/ba11047c-ce69-4982-892a-38b343b4f841
hdc -
root@KVM015:~# virsh attach-disk 26
/mnt/f773b66d-fd8c-3576-aa37-e3f0e685b183/ba11047c-ce69-4982-892a-38b343b4f841
vdb
error: Failed to attach disk
error: operation failed: target vdb already exists
================end==================
I believe this is highly related to the OS and configuration in VM, not
hypervisor, or cloudstack.
IN my testing, I use Ubuntu 12.04 as hypervisor, it works if vm OS is
CentOS7/CentOS6/Ubuntu 16.04, but not working if it is Ubuntu 12.04 .
-Wei
2016-12-21 12:00 GMT+01:00 Rohit Yadav <ro...@shapeblue.com>:
> All,
>
>
> Based on results from recent Trillian test runs [1], I've discovered that
> on KVM (CentOS7) based detaching a volume fails to update the virt/domain
> xml and fails to remove the xml. So, while the agent and cloudstack-mgmt
> server succeeds, the entry in the xml is not removed. When the volume is
> attached again, we can an error like:
>
>
> Failed to attach volume xxx to VM VM-yyyy; org.libvirt.LibvirtException:
> XML error: target 'vdb' duplicated for disk sources
> '/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-a452-43de-8c6b-948dc44aae21'
> and '/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-a452-43de-8c6b-948dc44aae21'This
> is seen in agent logs:
>
> Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: DEBUG [kvm.storage.KVMStorageProcessor]
> (agentRequest-Handler-2:) (logid:0648ae70) Detaching device: <disk
> device='disk' type='file'>
> Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: <driver
> name='qemu' type='qcow2' cache='none' />
> Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: <source
> file='/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-
> a452-43de-8c6b-948dc44aae21'/>
> Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: <target dev='vdb'
> bus='virtio'/>
> Dec 21 10:46:35 pr1837-t692-kvm-centos7-kvm2 sh[27400]: </disk>
>
> While, after above completes. This is still seen in the VM's dumped xml:
> <disk type='file' device='disk'>
> <driver name='qemu' type='qcow2' cache='none'/>
> <source file='/mnt/8a70be4e-4c3c-38e5-aea2-4b38fef83fd5/af85ff7e-
> a452-43de-8c6b-948dc44aae21'/>
> <backingStore/>
> <target dev='vdb' bus='virtio'/>
> <serial>af85ff7ea45243de8c6b</serial>
> <alias name='virtio-disk1'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x0'/>
> </disk>
> Steps to reproduce:
> 1. Deploy a VM, create a data volume disk and attach to the VM.
> 2. Detach the volume.
> 3. Attach the volume to the same VM again, exception is caught.Thoughts,
> comments?[1] https://github.com/apache/cloudstack/pull/1837
> Regards.
>
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>