You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2013/12/19 22:17:08 UTC
[jira] [Commented] (CLOUDSTACK-4939) Failed to create snapshot
(KVM, Multiple hosts, Sharedstorage)
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853333#comment-13853333 ]
ASF subversion and git services commented on CLOUDSTACK-4939:
-------------------------------------------------------------
Commit 88a5202eea403636146f881b83a245e0d6fae4a9 in branch refs/heads/4.3 from [~edison]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=88a5202 ]
CLOUDSTACK-4939 - Failed to create snapshot (KVM, Multiple hosts, Sharedstorage)
Conflicts:
engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/AncientDataMotionStrategy.java
engine/storage/src/org/apache/cloudstack/storage/endpoint/DefaultEndPointSelector.java
plugins/storage/volume/default/src/org/apache/cloudstack/storage/datastore/driver/CloudStackPrimaryDataStoreDriverImpl.java
> Failed to create snapshot (KVM, Multiple hosts, Sharedstorage)
> --------------------------------------------------------------
>
> Key: CLOUDSTACK-4939
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: KVM, Snapshot
> Affects Versions: 4.2.0, 4.2.1
> Environment: CentOS 6.4, KVM, Shared mount point primary storage, GFS2, iSCSI
> Reporter: Ivan Kozlov
> Assignee: edison su
> Priority: Blocker
> Labels: kvm, sharedstorage, snapshot
> Fix For: 4.2.1
>
> Attachments: 4939.doc, 513agent.doc, 514agent.doc, management-server(1).log.gz
>
>
> With one host snapshots are created ok. After adding second host some snapshots fail (Failed to create snapshot due to an internal error creating snapshot for volume 14) stucking with state "CreatedOnPrimary". Even when all VMs are running on the same host.
> debug libvirt log shows:
> 2013-10-23 17:31:21.634+0000: 20007: debug : virStorageFileGetMetadataInternal:673 : path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2
> 2013-10-23 17:32:57.189+0000: 20015: debug : qemuSnapObjFromName:233 : Domain snapshot not found: no domain snapshot with matching name '909848a0-b3ec-4657-a53a-c449dc24365b'
> 2013-10-23 17:32:57.474+0000: 20009: debug : virStorageFileGetMetadataInternal:673 : path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2
> 2013-10-23 17:34:28.264+0000: 20008: debug : qemuSnapObjFromName:233 : Domain snapshot not found: no domain snapshot with matching name 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca'
> management server log:
> 2013-10-23 20:29:50,561 INFO [user.snapshot.CreateSnapshotCmd] (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) VOLSS: createSnapshotCmd starts:1382549390561
> 2013-10-23 20:29:52,053 DEBUG [agent.transport.Request] (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Seq 6-1170407437: Waiting for Seq 1170407434 Scheduling: { Cmd , MgmtId: 161342718518, via: 6, Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/f4e51b11-ac79-4a6a-b887-8926ffbd5cca","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/14","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"snapshots/2/14/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.10.31/export/secondary","_role":"Image"}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"executeInSequence":true,"wait":21600}}] }
> 2013-10-23 20:31:21,560 DEBUG [agent.transport.Request] (AgentManager-Handler-8:null) Seq 6-1170407434: Processing: { Ans: , MgmtId: 161342718518, via: 6, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot with matching name '65113136-dfb5-4cea-8e65-1065462ca2fe'","wait":0}}] }
> 2013-10-23 20:31:21,832 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-49:job-91 = [ e2bf2454-4273-4a89-bc38-35add8297eb1 ]) Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot with matching name '65113136-dfb5-4cea-8e65-1065462ca2fe'
> at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:264)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013)
> at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
> 2013-10-23 20:31:21,999 DEBUG [storage.volume.VolumeServiceImpl] (Job-Executor-49:job-91 = [ e2bf2454-4273-4a89-bc38-35add8297eb1 ]) Take snapshot: 18 failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1040)
> at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot with matching name '65113136-dfb5-4cea-8e65-1065462ca2fe'
> at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:264)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013)
> 2013-10-23 20:31:22,167 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-49:job-91 = [ e2bf2454-4273-4a89-bc38-35add8297eb1 ]) Complete async job-91 = [ e2bf2454-4273-4a89-bc38-35add8297eb1 ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to create snapshot due to an internal error creating snapshot for volume 18
> 2013-10-23 20:31:24,709 DEBUG [agent.transport.Request] (AgentManager-Handler-13:null) Seq 9-1437990929: Processing: { Ans: , MgmtId: 161342718518, via: 9, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/16/157016cb-5e57-428f-b747-5d9b628d2864","id":0}},"result":true,"wait":0}}] }
> 2013-10-23 20:31:25,760 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-51:job-93 = [ 25e157c0-f966-401e-9263-c42dac56e0c1 ]) Done executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-93 = [ 25e157c0-f966-401e-9263-c42dac56e0c1 ]
> 2013-10-23 20:32:57,416 DEBUG [agent.transport.Request] (AgentManager-Handler-8:null) Seq 6-1170407435: Processing: { Ans: , MgmtId: 161342718518, via: 6, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot with matching name '909848a0-b3ec-4657-a53a-c449dc24365b'","wait":0}}] }
> 2013-10-23 20:32:57,680 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-50:job-92 = [ b8bbb5be-54ba-43df-b429-5b5fb61416ad ]) Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot with matching name '909848a0-b3ec-4657-a53a-c449dc24365b'
> at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:264)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013)
> at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
> 2013-10-23 20:32:57,763 DEBUG [storage.volume.VolumeServiceImpl] (Job-Executor-50:job-92 = [ b8bbb5be-54ba-43df-b429-5b5fb61416ad ]) Take snapshot: 17 failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1040)
> at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot with matching name '909848a0-b3ec-4657-a53a-c449dc24365b'
> at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:264)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013)
> 2013-10-23 20:32:57,849 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-50:job-92 = [ b8bbb5be-54ba-43df-b429-5b5fb61416ad ]) Complete async job-92 = [ b8bbb5be-54ba-43df-b429-5b5fb61416ad ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to create snapshot due to an internal error creating snapshot for volume 17
> 2013-10-23 20:33:50,627 DEBUG [storage.snapshot.SnapshotSchedulerImpl] (SnapshotPollTask:null) Snapshot scheduler.poll is being called at 2013-10-23 17:33:50 GMT
> 2013-10-23 20:33:50,627 DEBUG [storage.snapshot.SnapshotSchedulerImpl] (SnapshotPollTask:null) Got 0 snapshots to be executed at 2013-10-23 17:33:50 GMT
> 2013-10-23 20:34:28,514 DEBUG [agent.transport.Request] (AgentManager-Handler-3:null) Seq 6-1170407437: Processing: { Ans: , MgmtId: 161342718518, via: 6, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot with matching name 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca'","wait":0}}] }
> 2013-10-23 20:34:28,779 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot with matching name 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca'
> at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:264)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013)
> at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
> 2013-10-23 20:34:28,870 DEBUG [storage.volume.VolumeServiceImpl] (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Take snapshot: 14 failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1040)
> at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot with matching name 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca'
> at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:264)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013)
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)