You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Andrija Panic <an...@gmail.com> on 2015/06/10 10:55:46 UTC
[BUG?] ACS 4.5.1 can't create VM from template on CEPH, bigger than 20GB
Hi guys,
we just experienced a very strange problem:
ACS 4.5.1 vanila
Ubuntu 14.04.2 - latest Qemu* binaries (2.0.0+dfsg-2ubuntu1.11)
CEPH 0.94.1
We are unable to create VM from template that is bigger than 20GB - on CEPH.
Works with 10,14,18GB, but not with templates 20GB or bigger...
Problem with 20GB or more: mgmt server / agent tries 2 times to move/create
CEPH/RAW volume from NFS/QCOW2, although the first execution is successfull.
(some timeout problem?)
So, first time it detects it needs to copy from Qcow2/NFS to RAW/CEPH, it
issues qemu-img convert and gets success on execution.
Than again it tries to do EXACT same thing, so of course CEPH gives error
because destinatin image already exists and VM provisioning fails.
We dont see this behaviour with 10GB,14GB,18GB, but only with 20GB and
bigger.
Please find logs from a host, I marked critical points with "!!!!!!!":
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/204/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2","origUrl":"
http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2
","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
10.23.2.1/data/tank/secondary
","_role":"Image"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2
","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8457c284-cf5d-3979-b82e-32ea5efeb97b","id":1,"poolType":"RBD","host":"mon.swiss2.local","path":"cold-storage","port":6789,"url":"RBD://mon.swiss2.local/cold-storage/?ROLE=Primary&STOREUUID=8457c284-cf5d-3979-b82e-32ea5efeb97b"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"executeInSequence":true,"options":{},"wait":21600}}]
2015-06-10 01:42:36,661 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) Processing command:
org.apache.cloudstack.storage.command.CopyCommand
2015-06-10 01:42:36,661 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Attempting to create storage pool
b6d6c679-8475-31f6-b28f-a2e56746c7b9 (NetworkFilesystem) in libvirt
2015-06-10 01:42:36,669 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) <pool type='netfs'>
<name>b6d6c679-8475-31f6-b28f-a2e56746c7b9</name>
<uuid>b6d6c679-8475-31f6-b28f-a2e56746c7b9</uuid>
<source>
<host name='10.23.2.1'/>
<dir path='/data/tank/secondary/template/tmpl/2/204'/>
</source>
<target>
<path>/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9</path>
</target>
</pool>
2015-06-10 01:42:37,764 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Trying to fetch storage pool
b6d6c679-8475-31f6-b28f-a2e56746c7b9 from libvirt
2015-06-10 01:42:37,775 DEBUG [kvm.storage.KVMStorageProcessor]
(agentRequest-Handler-4:null) Copying template to primary storage, template
format is qcow2
2015-06-10 01:42:37,775 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Trying to fetch storage pool
8457c284-cf5d-3979-b82e-32ea5efeb97b from libvirt
2015-06-10 01:42:37,781 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) copyPhysicalDisk: disk size:1203378688,
virtualsize:25769803776 format:qcow2
2015-06-10 01:42:37,781 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) The source image is not RBD, but the
destination is. We will convert into RBD format 2
2015-06-10 01:42:37,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Starting copy from source image
/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
to RBD image cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2
(!!!!!!!!!!)
2015-06-10 01:42:37,782 DEBUG [utils.script.Script]
(agentRequest-Handler-4:null) Executing:
qemu-img convert -O raw
/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
rbd:cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2:mon_host=mon.swiss2.local:auth_supported=cephx:id=admin:key=AQA9E2pVzqotGRAALYrGr1zI+eO0i5F5ghU5/g==:rbd_default_format=2:client_mount_timeout=30
2015-06-10 01:47:12,430 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Succesfully converted source image
/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
to RBD image cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2
2015-06-10 01:47:12,549 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Succesfully connected to Ceph cluster at
mon.swiss2.local:6789
2015-06-10 01:47:12,576 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Attempting to remove storage pool
b6d6c679-8475-31f6-b28f-a2e56746c7b9 from libvirt
2015-06-10 01:47:16,683 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) Request:Seq 13-5696772053646966888: { Cmd ,
MgmtId: 90520740254323, via: 13, Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/204/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2","origUrl":"
http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2
","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
10.23.2.1/data/tank/secondary
","_role":"Image"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8457c284-cf5d-3979-b82e-32ea5efeb97b","id":1,"poolType":"RBD","host":"mon.swiss2.local","path":"cold-storage","port":6789,"url":"RBD://mon.swiss2.local/cold-storage/?ROLE=Primary&STOREUUID=8457c284-cf5d-3979-b82e-32ea5efeb97b"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"executeInSequence":true,"options":{},"wait":21600}}]
}
2015-06-10 01:47:16,686 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) createStoragePool didn't find existing
running pool: org.libvirt.LibvirtException: Storage pool not found: no
storage pool with matching uuid ���y�u1�����gFǹ, need to create it
2015-06-10 01:47:16,692 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) Attempting to create storage pool
b6d6c679-8475-31f6-b28f-a2e56746c7b9
2015-06-10 01:47:16,693 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) <pool type='netfs'>
<name>b6d6c679-8475-31f6-b28f-a2e56746c7b9</name>
<uuid>b6d6c679-8475-31f6-b28f-a2e56746c7b9</uuid>
<source>
<host name='10.23.2.1'/>
<dir path='/data/tank/secondary/template/tmpl/2/204'/>
</source>
<target>
<path>/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9</path>
</target>
</pool>
2015-06-10 01:47:16,775 DEBUG [kvm.storage.KVMStorageProcessor]
(agentRequest-Handler-5:null) Copying template to primary storage, template
format is qcow2
2015-06-10 01:47:16,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) copyPhysicalDisk: disk size:1203378688,
virtualsize:25769803776 format:qcow2
2015-06-10 01:47:16,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) The source image is not RBD, but the
destination is. We will convert into RBD format 2
2015-06-10 01:47:16,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) Starting copy from source image
/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
to RBD image cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2
(!!!!!!!!!!)
2015-06-10 01:47:16,782 DEBUG [utils.script.Script]
(agentRequest-Handler-5:null) Executing: qemu-img convert -O raw
/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
rbd:cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2:mon_host=mon.swiss2.local:auth_supported=cephx:id=admin:key=AQA9E2pVzqotGRAALYrGr1zI+eO0i5F5ghU5/g==:rbd_default_format=2:client_mount_timeout=30
2015-06-10 01:47:16,844 DEBUG [utils.script.Script]
(agentRequest-Handler-5:null) Exit value is 1
2015-06-10 01:47:16,845 DEBUG [utils.script.Script]
(agentRequest-Handler-5:null) libust[3503/3503]: Warning: HOME environment
variable not set. Disabling LTTng-UST per-user tracing. (in
setup_local_apps() at lttng-ust-comm.c:305)qemu-img:
rbd:cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2:mon_host=mon.swiss2.local:auth_supported=cephx:id=admin:key=AQA9E2pVzqotGRAALYrGr1zI+eO0i5F5ghU5/g==:rbd_default_format=2:client_mount_timeout=30:
error while converting raw: Could not create image: File exists
--
Andrija Panić
Re: [BUG?] ACS 4.5.1 can't create VM from template on CEPH, bigger
than 20GB
Posted by Andrija Panic <an...@gmail.com>.
Actually,
my collegue made a patch for
the plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java,
where we first check if the image is already present on CEPH, and if not,
only then it tries to copy it to the CEPH.
But this is not permanent solution, since in the first place, ACS should
not try to copy image 2 times, but only 1...
Anybody possible to test creating VM from bigger template on CEPH ?
Thanks,
On 10 June 2015 at 10:55, Andrija Panic <an...@gmail.com> wrote:
> Hi guys,
>
> we just experienced a very strange problem:
>
> ACS 4.5.1 vanila
> Ubuntu 14.04.2 - latest Qemu* binaries (2.0.0+dfsg-2ubuntu1.11)
> CEPH 0.94.1
>
> We are unable to create VM from template that is bigger than 20GB - on
> CEPH.
> Works with 10,14,18GB, but not with templates 20GB or bigger...
>
> Problem with 20GB or more: mgmt server / agent tries 2 times to
> move/create CEPH/RAW volume from NFS/QCOW2, although the first execution is
> successfull.
> (some timeout problem?)
>
> So, first time it detects it needs to copy from Qcow2/NFS to RAW/CEPH, it
> issues qemu-img convert and gets success on execution.
> Than again it tries to do EXACT same thing, so of course CEPH gives error
> because destinatin image already exists and VM provisioning fails.
>
> We dont see this behaviour with 10GB,14GB,18GB, but only with 20GB and
> bigger.
>
>
> Please find logs from a host, I marked critical points with "!!!!!!!":
>
>
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/204/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2","origUrl":"
> http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2
> ","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
> 10.23.2.1/data/tank/secondary
> ","_role":"Image"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2
> ","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8457c284-cf5d-3979-b82e-32ea5efeb97b","id":1,"poolType":"RBD","host":"mon.swiss2.local","path":"cold-storage","port":6789,"url":"RBD://mon.swiss2.local/cold-storage/?ROLE=Primary&STOREUUID=8457c284-cf5d-3979-b82e-32ea5efeb97b"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"executeInSequence":true,"options":{},"wait":21600}}]
> 2015-06-10 01:42:36,661 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Processing command:
> org.apache.cloudstack.storage.command.CopyCommand
> 2015-06-10 01:42:36,661 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Attempting to create storage pool
> b6d6c679-8475-31f6-b28f-a2e56746c7b9 (NetworkFilesystem) in libvirt
> 2015-06-10 01:42:36,669 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) <pool type='netfs'>
> <name>b6d6c679-8475-31f6-b28f-a2e56746c7b9</name>
> <uuid>b6d6c679-8475-31f6-b28f-a2e56746c7b9</uuid>
> <source>
> <host name='10.23.2.1'/>
> <dir path='/data/tank/secondary/template/tmpl/2/204'/>
> </source>
> <target>
> <path>/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9</path>
> </target>
> </pool>
>
> 2015-06-10 01:42:37,764 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Trying to fetch storage pool
> b6d6c679-8475-31f6-b28f-a2e56746c7b9 from libvirt
> 2015-06-10 01:42:37,775 DEBUG [kvm.storage.KVMStorageProcessor]
> (agentRequest-Handler-4:null) Copying template to primary storage, template
> format is qcow2
> 2015-06-10 01:42:37,775 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Trying to fetch storage pool
> 8457c284-cf5d-3979-b82e-32ea5efeb97b from libvirt
> 2015-06-10 01:42:37,781 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) copyPhysicalDisk: disk size:1203378688,
> virtualsize:25769803776 format:qcow2
> 2015-06-10 01:42:37,781 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) The source image is not RBD, but the
> destination is. We will convert into RBD format 2
>
> 2015-06-10 01:42:37,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Starting copy from source image
> /mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
> to RBD image cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2
>
> (!!!!!!!!!!)
> 2015-06-10 01:42:37,782 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Executing:
> qemu-img convert -O raw
> /mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
> rbd:cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2:mon_host=mon.swiss2.local:auth_supported=cephx:id=admin:key=AQA9E2pVzqotGRAALYrGr1zI+eO0i5F5ghU5/g==:rbd_default_format=2:client_mount_timeout=30
>
> 2015-06-10 01:47:12,430 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Succesfully converted source image
> /mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
> to RBD image cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2
> 2015-06-10 01:47:12,549 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Succesfully connected to Ceph cluster at
> mon.swiss2.local:6789
> 2015-06-10 01:47:12,576 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Attempting to remove storage pool
> b6d6c679-8475-31f6-b28f-a2e56746c7b9 from libvirt
>
> 2015-06-10 01:47:16,683 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) Request:Seq 13-5696772053646966888: { Cmd ,
> MgmtId: 90520740254323, via: 13, Ver: v1, Flags: 100111,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/204/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2","origUrl":"
> http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2
> ","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
> 10.23.2.1/data/tank/secondary
> ","_role":"Image"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8457c284-cf5d-3979-b82e-32ea5efeb97b","id":1,"poolType":"RBD","host":"mon.swiss2.local","path":"cold-storage","port":6789,"url":"RBD://mon.swiss2.local/cold-storage/?ROLE=Primary&STOREUUID=8457c284-cf5d-3979-b82e-32ea5efeb97b"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"executeInSequence":true,"options":{},"wait":21600}}]
> }
> 2015-06-10 01:47:16,686 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) createStoragePool didn't find existing
> running pool: org.libvirt.LibvirtException: Storage pool not found: no
> storage pool with matching uuid ���y�u1�����gFǹ, need to create it
> 2015-06-10 01:47:16,692 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) Attempting to create storage pool
> b6d6c679-8475-31f6-b28f-a2e56746c7b9
> 2015-06-10 01:47:16,693 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) <pool type='netfs'>
> <name>b6d6c679-8475-31f6-b28f-a2e56746c7b9</name>
> <uuid>b6d6c679-8475-31f6-b28f-a2e56746c7b9</uuid>
> <source>
> <host name='10.23.2.1'/>
> <dir path='/data/tank/secondary/template/tmpl/2/204'/>
> </source>
> <target>
> <path>/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9</path>
> </target>
> </pool>
>
> 2015-06-10 01:47:16,775 DEBUG [kvm.storage.KVMStorageProcessor]
> (agentRequest-Handler-5:null) Copying template to primary storage, template
> format is qcow2
> 2015-06-10 01:47:16,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) copyPhysicalDisk: disk size:1203378688,
> virtualsize:25769803776 format:qcow2
> 2015-06-10 01:47:16,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) The source image is not RBD, but the
> destination is. We will convert into RBD format 2
> 2015-06-10 01:47:16,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) Starting copy from source image
> /mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
> to RBD image cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2
>
> (!!!!!!!!!!)
> 2015-06-10 01:47:16,782 DEBUG [utils.script.Script]
> (agentRequest-Handler-5:null) Executing: qemu-img convert -O raw
> /mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
> rbd:cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2:mon_host=mon.swiss2.local:auth_supported=cephx:id=admin:key=AQA9E2pVzqotGRAALYrGr1zI+eO0i5F5ghU5/g==:rbd_default_format=2:client_mount_timeout=30
> 2015-06-10 01:47:16,844 DEBUG [utils.script.Script]
> (agentRequest-Handler-5:null) Exit value is 1
>
> 2015-06-10 01:47:16,845 DEBUG [utils.script.Script]
> (agentRequest-Handler-5:null) libust[3503/3503]: Warning: HOME environment
> variable not set. Disabling LTTng-UST per-user tracing. (in
> setup_local_apps() at lttng-ust-comm.c:305)qemu-img:
> rbd:cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2:mon_host=mon.swiss2.local:auth_supported=cephx:id=admin:key=AQA9E2pVzqotGRAALYrGr1zI+eO0i5F5ghU5/g==:rbd_default_format=2:client_mount_timeout=30:
> error while converting raw: Could not create image: File exists
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: [BUG?] ACS 4.5.1 can't create VM from template on CEPH, bigger
than 20GB
Posted by Andrija Panic <an...@gmail.com>.
Actually,
my collegue made a patch for
the plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java,
where we first check if the image is already present on CEPH, and if not,
only then it tries to copy it to the CEPH.
But this is not permanent solution, since in the first place, ACS should
not try to copy image 2 times, but only 1...
Anybody possible to test creating VM from bigger template on CEPH ?
Thanks,
On 10 June 2015 at 10:55, Andrija Panic <an...@gmail.com> wrote:
> Hi guys,
>
> we just experienced a very strange problem:
>
> ACS 4.5.1 vanila
> Ubuntu 14.04.2 - latest Qemu* binaries (2.0.0+dfsg-2ubuntu1.11)
> CEPH 0.94.1
>
> We are unable to create VM from template that is bigger than 20GB - on
> CEPH.
> Works with 10,14,18GB, but not with templates 20GB or bigger...
>
> Problem with 20GB or more: mgmt server / agent tries 2 times to
> move/create CEPH/RAW volume from NFS/QCOW2, although the first execution is
> successfull.
> (some timeout problem?)
>
> So, first time it detects it needs to copy from Qcow2/NFS to RAW/CEPH, it
> issues qemu-img convert and gets success on execution.
> Than again it tries to do EXACT same thing, so of course CEPH gives error
> because destinatin image already exists and VM provisioning fails.
>
> We dont see this behaviour with 10GB,14GB,18GB, but only with 20GB and
> bigger.
>
>
> Please find logs from a host, I marked critical points with "!!!!!!!":
>
>
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/204/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2","origUrl":"
> http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2
> ","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
> 10.23.2.1/data/tank/secondary
> ","_role":"Image"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2
> ","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8457c284-cf5d-3979-b82e-32ea5efeb97b","id":1,"poolType":"RBD","host":"mon.swiss2.local","path":"cold-storage","port":6789,"url":"RBD://mon.swiss2.local/cold-storage/?ROLE=Primary&STOREUUID=8457c284-cf5d-3979-b82e-32ea5efeb97b"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"executeInSequence":true,"options":{},"wait":21600}}]
> 2015-06-10 01:42:36,661 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Processing command:
> org.apache.cloudstack.storage.command.CopyCommand
> 2015-06-10 01:42:36,661 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Attempting to create storage pool
> b6d6c679-8475-31f6-b28f-a2e56746c7b9 (NetworkFilesystem) in libvirt
> 2015-06-10 01:42:36,669 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) <pool type='netfs'>
> <name>b6d6c679-8475-31f6-b28f-a2e56746c7b9</name>
> <uuid>b6d6c679-8475-31f6-b28f-a2e56746c7b9</uuid>
> <source>
> <host name='10.23.2.1'/>
> <dir path='/data/tank/secondary/template/tmpl/2/204'/>
> </source>
> <target>
> <path>/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9</path>
> </target>
> </pool>
>
> 2015-06-10 01:42:37,764 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Trying to fetch storage pool
> b6d6c679-8475-31f6-b28f-a2e56746c7b9 from libvirt
> 2015-06-10 01:42:37,775 DEBUG [kvm.storage.KVMStorageProcessor]
> (agentRequest-Handler-4:null) Copying template to primary storage, template
> format is qcow2
> 2015-06-10 01:42:37,775 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Trying to fetch storage pool
> 8457c284-cf5d-3979-b82e-32ea5efeb97b from libvirt
> 2015-06-10 01:42:37,781 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) copyPhysicalDisk: disk size:1203378688,
> virtualsize:25769803776 format:qcow2
> 2015-06-10 01:42:37,781 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) The source image is not RBD, but the
> destination is. We will convert into RBD format 2
>
> 2015-06-10 01:42:37,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Starting copy from source image
> /mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
> to RBD image cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2
>
> (!!!!!!!!!!)
> 2015-06-10 01:42:37,782 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Executing:
> qemu-img convert -O raw
> /mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
> rbd:cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2:mon_host=mon.swiss2.local:auth_supported=cephx:id=admin:key=AQA9E2pVzqotGRAALYrGr1zI+eO0i5F5ghU5/g==:rbd_default_format=2:client_mount_timeout=30
>
> 2015-06-10 01:47:12,430 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Succesfully converted source image
> /mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
> to RBD image cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2
> 2015-06-10 01:47:12,549 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Succesfully connected to Ceph cluster at
> mon.swiss2.local:6789
> 2015-06-10 01:47:12,576 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Attempting to remove storage pool
> b6d6c679-8475-31f6-b28f-a2e56746c7b9 from libvirt
>
> 2015-06-10 01:47:16,683 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) Request:Seq 13-5696772053646966888: { Cmd ,
> MgmtId: 90520740254323, via: 13, Ver: v1, Flags: 100111,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/204/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2","origUrl":"
> http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2
> ","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
> 10.23.2.1/data/tank/secondary
> ","_role":"Image"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8457c284-cf5d-3979-b82e-32ea5efeb97b","id":1,"poolType":"RBD","host":"mon.swiss2.local","path":"cold-storage","port":6789,"url":"RBD://mon.swiss2.local/cold-storage/?ROLE=Primary&STOREUUID=8457c284-cf5d-3979-b82e-32ea5efeb97b"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"executeInSequence":true,"options":{},"wait":21600}}]
> }
> 2015-06-10 01:47:16,686 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) createStoragePool didn't find existing
> running pool: org.libvirt.LibvirtException: Storage pool not found: no
> storage pool with matching uuid ���y�u1�����gFǹ, need to create it
> 2015-06-10 01:47:16,692 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) Attempting to create storage pool
> b6d6c679-8475-31f6-b28f-a2e56746c7b9
> 2015-06-10 01:47:16,693 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) <pool type='netfs'>
> <name>b6d6c679-8475-31f6-b28f-a2e56746c7b9</name>
> <uuid>b6d6c679-8475-31f6-b28f-a2e56746c7b9</uuid>
> <source>
> <host name='10.23.2.1'/>
> <dir path='/data/tank/secondary/template/tmpl/2/204'/>
> </source>
> <target>
> <path>/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9</path>
> </target>
> </pool>
>
> 2015-06-10 01:47:16,775 DEBUG [kvm.storage.KVMStorageProcessor]
> (agentRequest-Handler-5:null) Copying template to primary storage, template
> format is qcow2
> 2015-06-10 01:47:16,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) copyPhysicalDisk: disk size:1203378688,
> virtualsize:25769803776 format:qcow2
> 2015-06-10 01:47:16,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) The source image is not RBD, but the
> destination is. We will convert into RBD format 2
> 2015-06-10 01:47:16,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) Starting copy from source image
> /mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
> to RBD image cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2
>
> (!!!!!!!!!!)
> 2015-06-10 01:47:16,782 DEBUG [utils.script.Script]
> (agentRequest-Handler-5:null) Executing: qemu-img convert -O raw
> /mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
> rbd:cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2:mon_host=mon.swiss2.local:auth_supported=cephx:id=admin:key=AQA9E2pVzqotGRAALYrGr1zI+eO0i5F5ghU5/g==:rbd_default_format=2:client_mount_timeout=30
> 2015-06-10 01:47:16,844 DEBUG [utils.script.Script]
> (agentRequest-Handler-5:null) Exit value is 1
>
> 2015-06-10 01:47:16,845 DEBUG [utils.script.Script]
> (agentRequest-Handler-5:null) libust[3503/3503]: Warning: HOME environment
> variable not set. Disabling LTTng-UST per-user tracing. (in
> setup_local_apps() at lttng-ust-comm.c:305)qemu-img:
> rbd:cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2:mon_host=mon.swiss2.local:auth_supported=cephx:id=admin:key=AQA9E2pVzqotGRAALYrGr1zI+eO0i5F5ghU5/g==:rbd_default_format=2:client_mount_timeout=30:
> error while converting raw: Could not create image: File exists
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić