You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Thomas O'Dowd <tp...@cloudian.com> on 2014/03/27 03:55:04 UTC
Snapshot exceptions on 4.2.1 with NFS and S3
I remember this used to work on Cloudstack 4.2 when I was testing it
with devcloud but with CloudPlatform 4.2.1 and KVM I can't get snapshots
to work.
I've setup a single node using KVM and management server, db and agent.
I can start an instance using a template stored on NFS secondary
storage. If I stop the instance and goto storage, I can see it's disk
and then I can select "take snapshot". This always seems to fail for me
and I'm not sure why.
The error seems to be:
Failed to create snapshot
com.cloud.utils.exception.CloudRuntimeException: Failed to
backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for
disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3
/mnt/8a218... is mounted fine if I look at df.
/mnt/e5c41... is not mounted.
If I browse to /mnt, the e5c41... directory (used for mounting) exists.
If I manually mount it, I can see the
directory /mnt/e5c41.../snapshots/2/3 so it exists. If I leave it
mounted and try to take a snapshot again, it fails again with the same
error and again when I check the /mnt/e5c41... directory is not mounted.
As I mentioned above, the NFS server I'm using is actually the same
node. I'm using KVM but I don't think that is causing the issue? I'm
using CentOS 6.4.
If I setup S3 as secondary storage and use NFS as the staging server, I
get the same error. Any hints on what I am doing wrong is greatly
appreciated.
Here is more of the exception log:
2014-03-24 16:12:33,450 DEBUG [cloud.api.ApiServlet] (catalina-exec-10:null) ===START=== 10.181.164.198 -- GET command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401
2014-03-24 16:12:33,819 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-10:null) submit async job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ], details: AsyncJobVO {id:10, userId: 2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 1, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdOriginator: null, cmdInfo: {"id":"1","response":"json","sessionkey":"TnWYMJmMIZRly5DfGPNX+pZeT+I\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1395645153401","volumeid":"96f59602-8a84-4731-91a5-218d4f45ee70","ctxAccountId":"2","ctxStartEventId":"45"}, cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0, result: null, initMsid: 143618408992, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2014-03-24 16:12:33,820 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]
2014-03-24 16:12:33,821 DEBUG [cloud.api.ApiServlet] (catalina-exec-10:null) ===END=== 10.181.164.198 -- GET command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401
2014-03-24 16:12:33,847 INFO [user.snapshot.CreateSnapshotCmd] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) VOLSS: createSnapshotCmd starts:1395645153847
2014-03-24 16:12:34,179 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 1-249561236: Sending { Cmd , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"wait":0}}] }
2014-03-24 16:12:34,374 DEBUG [agent.transport.Request] (AgentManager-Handler-8:null) Seq 1-249561236: Processing: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9","id":0}},"result":true,"wait":0}}] }
2014-03-24 16:12:34,374 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 1-249561236: Received: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 10, { CreateObjectAnswer } }
2014-03-24 16:12:34,784 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) copyAsync inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT
2014-03-24 16:12:34,802 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) needCacheStorage true, dest at snapshots/2/3 dest role Image/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9 src role Primary
2014-03-24 16:12:34,938 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 1-249561237: Sending { Cmd , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/3","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.S3TO":{"id":2,"uuid":"c18a1785-f578-4102-bb94-0298289786e0","endPoint":"s3.cloudian.com:18080","bucketName":"cloudstack","httpsFlag":false,"created":"Mar 24, 2014 3:39:20 PM","enableRRS":false,"maxSingleUploadSizeInBytes":5368709120}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"cacheTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/3","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.181.168.230/home/staging","_role":"ImageCache"}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"executeInSequence":true,"wait":21600}}] }
2014-03-24 16:12:35,090 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 1-249561237: Processing: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Failed to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3","wait":0}}] }
2014-03-24 16:12:35,090 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-7:null) Seq 1-249561237: No more commands found
2014-03-24 16:12:35,090 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 1-249561237: Processing: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Failed to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3","wait":0}}] }
2014-03-24 16:12:35,090 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-7:null) Seq 1-249561237: No more commands found
2014-03-24 16:12:35,090 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 1-249561237: Received: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 110, { CopyCmdAnswer } }
2014-03-24 16:12:35,098 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 3-1564409959: Sending { Cmd , MgmtId: 143618408992, via: 3, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/3","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.181.168.230/home/staging","_role":"ImageCache"}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"wait":0}}] }
2014-03-24 16:12:35,260 DEBUG [agent.transport.Request] (AgentManager-Handler-9:null) Seq 3-1564409959: Processing: { Ans: , MgmtId: 143618408992, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2014-03-24 16:12:35,260 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 3-1564409959: Received: { Ans: , MgmtId: 143618408992, via: 3, Ver: v1, Flags: 10, { Answer } }
2014-03-24 16:12:35,468 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Failed to create snapshot
com.cloud.utils.exception.CloudRuntimeException: Failed to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1001)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2779)
at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2014-03-24 16:12:35,635 DEBUG [storage.volume.VolumeServiceImpl] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Take snapshot: 3 failed
com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1028)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2779)
at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1001)
... 16 more
2014-03-24 16:12:35,635 DEBUG [user.snapshot.CreateSnapshotCmd] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Failed to create snapshot
org.apache.cloudstack.api.ServerApiException
at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:176)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2014-03-24 16:12:35,637 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Complete async job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to create snapshot due to an internal error creating snapshot for volume 3
Thanks,
Tom.
--
Cloudian KK - http://www.cloudian.com/get-started.html
Fancy 10TB of full featured S3 Storage?
Checkout the Cloudian® Community Edition!
Re: Snapshot exceptions on 4.2.1 with NFS and S3
Posted by Thomas O'Dowd <tp...@cloudian.com>.
Thanks for the feedback Andrei,
In my case the problem seems unrelated to S3 as when I configure NFS as
my secondary storage I get the same problem. My problem seems to be that
the system tries to write the snapshot to directory path
like /mnt/e5c41.../snapshots/2/3/ but as for some reason it finds
that /mnt/e5c41.../snapshots/2/3/ doesn't exist as the /mnt/e5c41... is
not actually mounting secondary storage at that time. If I mount it
manually, the path is there but the system is not doing it automatically
for me.
Tom.
On Tue, 2014-04-01 at 22:41 +0100, Andrei Mikhailovsky wrote:
> From what I remember, there was an issue with S3 and snapshots.
>
> I might be mixing version numbers, but it could be related to the size of the image. In the past, there was no support for multi-part upload I was not able to snapshot anything larger than 5gb. Not sure if this is the same problem that you are experiencing.
>
> I would suggest trying 4.3 and letting us know how it works for you.
>
> I am planning to upgrade to 4.3 and test snapshotting with nfs / s3 / ceph. Will update on the progress.
>
> Andrei
>
>
> ----- Original Message -----
> From: "Thomas O'Dowd" <tp...@cloudian.com>
> To: users@cloudstack.apache.org
> Sent: Tuesday, 1 April, 2014 6:44:28 AM
> Subject: Re: Snapshot exceptions on 4.2.1 with NFS and S3
>
> Anyone any hints on this? I've installed it a few times and always have
> the same issue. Wondering if it's some configuration problem or a known
> bug.
>
> Thanks,
>
> Tom.
>
> On Thu, 2014-03-27 at 11:55 +0900, Thomas O'Dowd wrote:
> > I remember this used to work on Cloudstack 4.2 when I was testing it
> > with devcloud but with CloudPlatform 4.2.1 and KVM I can't get snapshots
> > to work.
> >
> > I've setup a single node using KVM and management server, db and agent.
> >
> > I can start an instance using a template stored on NFS secondary
> > storage. If I stop the instance and goto storage, I can see it's disk
> > and then I can select "take snapshot". This always seems to fail for me
> > and I'm not sure why.
> >
> > The error seems to be:
> > Failed to create snapshot
> > com.cloud.utils.exception.CloudRuntimeException: Failed to
> > backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for
> > disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3
> >
> > /mnt/8a218... is mounted fine if I look at df.
> > /mnt/e5c41... is not mounted.
> >
> > If I browse to /mnt, the e5c41... directory (used for mounting) exists.
> > If I manually mount it, I can see the
> > directory /mnt/e5c41.../snapshots/2/3 so it exists. If I leave it
> > mounted and try to take a snapshot again, it fails again with the same
> > error and again when I check the /mnt/e5c41... directory is not mounted.
> >
> > As I mentioned above, the NFS server I'm using is actually the same
> > node. I'm using KVM but I don't think that is causing the issue? I'm
> > using CentOS 6.4.
> >
> > If I setup S3 as secondary storage and use NFS as the staging server, I
> > get the same error. Any hints on what I am doing wrong is greatly
> > appreciated.
> >
> > Here is more of the exception log:
> > 2014-03-24 16:12:33,450 DEBUG [cloud.api.ApiServlet] (catalina-exec-10:null) ===START=== 10.181.164.198 -- GET command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401
> > 2014-03-24 16:12:33,819 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-10:null) submit async job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ], details: AsyncJobVO {id:10, userId: 2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 1, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdOriginator: null, cmdInfo: {"id":"1","response":"json","sessionkey":"TnWYMJmMIZRly5DfGPNX+pZeT+I\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1395645153401","volumeid":"96f59602-8a84-4731-91a5-218d4f45ee70","ctxAccountId":"2","ctxStartEventId":"45"}, cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0, result: null, initMsid: 143618408992, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> > 2014-03-24 16:12:33,820 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]
> > 2014-03-24 16:12:33,821 DEBUG [cloud.api.ApiServlet] (catalina-exec-10:null) ===END=== 10.181.164.198 -- GET command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401
> > 2014-03-24 16:12:33,847 INFO [user.snapshot.CreateSnapshotCmd] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) VOLSS: createSnapshotCmd starts:1395645153847
> > 2014-03-24 16:12:34,179 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 1-249561236: Sending { Cmd , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"wait":0}}] }
> > 2014-03-24 16:12:34,374 DEBUG [agent.transport.Request] (AgentManager-Handler-8:null) Seq 1-249561236: Processing: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9","id":0}},"result":true,"wait":0}}] }
> > 2014-03-24 16:12:34,374 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 1-249561236: Received: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 10, { CreateObjectAnswer } }
> > 2014-03-24 16:12:34,784 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) copyAsync inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT
> > 2014-03-24 16:12:34,802 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) needCacheStorage true, dest at snapshots/2/3 dest role Image/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9 src role Primary
> > 2014-03-24 16:12:34,938 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 1-249561237: Sending { Cmd , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/3","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.S3TO":{"id":2,"uuid":"c18a1785-f578-4102-bb94-0298289786e0","endPoint":"s3.cloudian.com:18080","bucketName":"cloudstack","httpsFlag":false,"created":"Mar 24, 2014 3:39:20 PM","enableRRS":false,"maxSingleUploadSizeInBytes":5368709120}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"cacheTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/3","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.181.168.230/home/staging","_role":"ImageCache"}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"executeInSequence":true,"wait":21600}}] }
> > 2014-03-24 16:12:35,090 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 1-249561237: Processing: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Failed to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3","wait":0}}] }
> > 2014-03-24 16:12:35,090 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-7:null) Seq 1-249561237: No more commands found
> > 2014-03-24 16:12:35,090 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 1-249561237: Processing: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Failed to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3","wait":0}}] }
> > 2014-03-24 16:12:35,090 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-7:null) Seq 1-249561237: No more commands found
> > 2014-03-24 16:12:35,090 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 1-249561237: Received: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 110, { CopyCmdAnswer } }
> > 2014-03-24 16:12:35,098 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 3-1564409959: Sending { Cmd , MgmtId: 143618408992, via: 3, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/3","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.181.168.230/home/staging","_role":"ImageCache"}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"wait":0}}] }
> > 2014-03-24 16:12:35,260 DEBUG [agent.transport.Request] (AgentManager-Handler-9:null) Seq 3-1564409959: Processing: { Ans: , MgmtId: 143618408992, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
> > 2014-03-24 16:12:35,260 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 3-1564409959: Received: { Ans: , MgmtId: 143618408992, via: 3, Ver: v1, Flags: 10, { Answer } }
> > 2014-03-24 16:12:35,468 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Failed to create snapshot
> > com.cloud.utils.exception.CloudRuntimeException: Failed to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3
> > at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
> > at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
> > at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
> > at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1001)
> > at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> > at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
> > at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2779)
> > at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
> > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> > at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> > at java.lang.Thread.run(Thread.java:722)
> > 2014-03-24 16:12:35,635 DEBUG [storage.volume.VolumeServiceImpl] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Take snapshot: 3 failed
> > com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
> > at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1028)
> > at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> > at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
> > at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2779)
> > at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
> > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> > at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> > at java.lang.Thread.run(Thread.java:722)
> > Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3
> > at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
> > at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
> > at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
> > at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1001)
> > ... 16 more
> > 2014-03-24 16:12:35,635 DEBUG [user.snapshot.CreateSnapshotCmd] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Failed to create snapshot
> > org.apache.cloudstack.api.ServerApiException
> > at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:176)
> > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> > at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> > at java.lang.Thread.run(Thread.java:722)
> > 2014-03-24 16:12:35,637 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Complete async job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to create snapshot due to an internal error creating snapshot for volume 3
> >
> >
> > Thanks,
> >
> > Tom.
>
--
Cloudian KK - http://www.cloudian.com/get-started.html
Fancy 10TB of full featured S3 Storage?
Checkout the Cloudian® Community Edition!
Re: Snapshot exceptions on 4.2.1 with NFS and S3
Posted by Andrei Mikhailovsky <an...@arhont.com>.
>From what I remember, there was an issue with S3 and snapshots.
I might be mixing version numbers, but it could be related to the size of the image. In the past, there was no support for multi-part upload I was not able to snapshot anything larger than 5gb. Not sure if this is the same problem that you are experiencing.
I would suggest trying 4.3 and letting us know how it works for you.
I am planning to upgrade to 4.3 and test snapshotting with nfs / s3 / ceph. Will update on the progress.
Andrei
----- Original Message -----
From: "Thomas O'Dowd" <tp...@cloudian.com>
To: users@cloudstack.apache.org
Sent: Tuesday, 1 April, 2014 6:44:28 AM
Subject: Re: Snapshot exceptions on 4.2.1 with NFS and S3
Anyone any hints on this? I've installed it a few times and always have
the same issue. Wondering if it's some configuration problem or a known
bug.
Thanks,
Tom.
On Thu, 2014-03-27 at 11:55 +0900, Thomas O'Dowd wrote:
> I remember this used to work on Cloudstack 4.2 when I was testing it
> with devcloud but with CloudPlatform 4.2.1 and KVM I can't get snapshots
> to work.
>
> I've setup a single node using KVM and management server, db and agent.
>
> I can start an instance using a template stored on NFS secondary
> storage. If I stop the instance and goto storage, I can see it's disk
> and then I can select "take snapshot". This always seems to fail for me
> and I'm not sure why.
>
> The error seems to be:
> Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException: Failed to
> backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for
> disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3
>
> /mnt/8a218... is mounted fine if I look at df.
> /mnt/e5c41... is not mounted.
>
> If I browse to /mnt, the e5c41... directory (used for mounting) exists.
> If I manually mount it, I can see the
> directory /mnt/e5c41.../snapshots/2/3 so it exists. If I leave it
> mounted and try to take a snapshot again, it fails again with the same
> error and again when I check the /mnt/e5c41... directory is not mounted.
>
> As I mentioned above, the NFS server I'm using is actually the same
> node. I'm using KVM but I don't think that is causing the issue? I'm
> using CentOS 6.4.
>
> If I setup S3 as secondary storage and use NFS as the staging server, I
> get the same error. Any hints on what I am doing wrong is greatly
> appreciated.
>
> Here is more of the exception log:
> 2014-03-24 16:12:33,450 DEBUG [cloud.api.ApiServlet] (catalina-exec-10:null) ===START=== 10.181.164.198 -- GET command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401
> 2014-03-24 16:12:33,819 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-10:null) submit async job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ], details: AsyncJobVO {id:10, userId: 2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 1, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdOriginator: null, cmdInfo: {"id":"1","response":"json","sessionkey":"TnWYMJmMIZRly5DfGPNX+pZeT+I\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1395645153401","volumeid":"96f59602-8a84-4731-91a5-218d4f45ee70","ctxAccountId":"2","ctxStartEventId":"45"}, cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0, result: null, initMsid: 143618408992, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2014-03-24 16:12:33,820 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]
> 2014-03-24 16:12:33,821 DEBUG [cloud.api.ApiServlet] (catalina-exec-10:null) ===END=== 10.181.164.198 -- GET command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401
> 2014-03-24 16:12:33,847 INFO [user.snapshot.CreateSnapshotCmd] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) VOLSS: createSnapshotCmd starts:1395645153847
> 2014-03-24 16:12:34,179 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 1-249561236: Sending { Cmd , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"wait":0}}] }
> 2014-03-24 16:12:34,374 DEBUG [agent.transport.Request] (AgentManager-Handler-8:null) Seq 1-249561236: Processing: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9","id":0}},"result":true,"wait":0}}] }
> 2014-03-24 16:12:34,374 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 1-249561236: Received: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 10, { CreateObjectAnswer } }
> 2014-03-24 16:12:34,784 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) copyAsync inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT
> 2014-03-24 16:12:34,802 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) needCacheStorage true, dest at snapshots/2/3 dest role Image/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9 src role Primary
> 2014-03-24 16:12:34,938 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 1-249561237: Sending { Cmd , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/3","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.S3TO":{"id":2,"uuid":"c18a1785-f578-4102-bb94-0298289786e0","endPoint":"s3.cloudian.com:18080","bucketName":"cloudstack","httpsFlag":false,"created":"Mar 24, 2014 3:39:20 PM","enableRRS":false,"maxSingleUploadSizeInBytes":5368709120}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"cacheTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/3","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.181.168.230/home/staging","_role":"ImageCache"}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"executeInSequence":true,"wait":21600}}] }
> 2014-03-24 16:12:35,090 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 1-249561237: Processing: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Failed to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3","wait":0}}] }
> 2014-03-24 16:12:35,090 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-7:null) Seq 1-249561237: No more commands found
> 2014-03-24 16:12:35,090 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 1-249561237: Processing: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Failed to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3","wait":0}}] }
> 2014-03-24 16:12:35,090 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-7:null) Seq 1-249561237: No more commands found
> 2014-03-24 16:12:35,090 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 1-249561237: Received: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 110, { CopyCmdAnswer } }
> 2014-03-24 16:12:35,098 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 3-1564409959: Sending { Cmd , MgmtId: 143618408992, via: 3, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/3","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.181.168.230/home/staging","_role":"ImageCache"}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"wait":0}}] }
> 2014-03-24 16:12:35,260 DEBUG [agent.transport.Request] (AgentManager-Handler-9:null) Seq 3-1564409959: Processing: { Ans: , MgmtId: 143618408992, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
> 2014-03-24 16:12:35,260 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 3-1564409959: Received: { Ans: , MgmtId: 143618408992, via: 3, Ver: v1, Flags: 10, { Answer } }
> 2014-03-24 16:12:35,468 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException: Failed to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3
> at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1001)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
> at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2779)
> at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2014-03-24 16:12:35,635 DEBUG [storage.volume.VolumeServiceImpl] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Take snapshot: 3 failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1028)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
> at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2779)
> at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3
> at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1001)
> ... 16 more
> 2014-03-24 16:12:35,635 DEBUG [user.snapshot.CreateSnapshotCmd] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Failed to create snapshot
> org.apache.cloudstack.api.ServerApiException
> at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:176)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2014-03-24 16:12:35,637 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Complete async job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to create snapshot due to an internal error creating snapshot for volume 3
>
>
> Thanks,
>
> Tom.
--
Cloudian KK - http://www.cloudian.com/get-started.html
Fancy 10TB of full featured S3 Storage?
Checkout the Cloudian® Community Edition!
Re: Snapshot exceptions on 4.2.1 with NFS and S3
Posted by Thomas O'Dowd <tp...@cloudian.com>.
Anyone any hints on this? I've installed it a few times and always have
the same issue. Wondering if it's some configuration problem or a known
bug.
Thanks,
Tom.
On Thu, 2014-03-27 at 11:55 +0900, Thomas O'Dowd wrote:
> I remember this used to work on Cloudstack 4.2 when I was testing it
> with devcloud but with CloudPlatform 4.2.1 and KVM I can't get snapshots
> to work.
>
> I've setup a single node using KVM and management server, db and agent.
>
> I can start an instance using a template stored on NFS secondary
> storage. If I stop the instance and goto storage, I can see it's disk
> and then I can select "take snapshot". This always seems to fail for me
> and I'm not sure why.
>
> The error seems to be:
> Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException: Failed to
> backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for
> disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3
>
> /mnt/8a218... is mounted fine if I look at df.
> /mnt/e5c41... is not mounted.
>
> If I browse to /mnt, the e5c41... directory (used for mounting) exists.
> If I manually mount it, I can see the
> directory /mnt/e5c41.../snapshots/2/3 so it exists. If I leave it
> mounted and try to take a snapshot again, it fails again with the same
> error and again when I check the /mnt/e5c41... directory is not mounted.
>
> As I mentioned above, the NFS server I'm using is actually the same
> node. I'm using KVM but I don't think that is causing the issue? I'm
> using CentOS 6.4.
>
> If I setup S3 as secondary storage and use NFS as the staging server, I
> get the same error. Any hints on what I am doing wrong is greatly
> appreciated.
>
> Here is more of the exception log:
> 2014-03-24 16:12:33,450 DEBUG [cloud.api.ApiServlet] (catalina-exec-10:null) ===START=== 10.181.164.198 -- GET command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401
> 2014-03-24 16:12:33,819 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-10:null) submit async job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ], details: AsyncJobVO {id:10, userId: 2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 1, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdOriginator: null, cmdInfo: {"id":"1","response":"json","sessionkey":"TnWYMJmMIZRly5DfGPNX+pZeT+I\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1395645153401","volumeid":"96f59602-8a84-4731-91a5-218d4f45ee70","ctxAccountId":"2","ctxStartEventId":"45"}, cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0, result: null, initMsid: 143618408992, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2014-03-24 16:12:33,820 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]
> 2014-03-24 16:12:33,821 DEBUG [cloud.api.ApiServlet] (catalina-exec-10:null) ===END=== 10.181.164.198 -- GET command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401
> 2014-03-24 16:12:33,847 INFO [user.snapshot.CreateSnapshotCmd] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) VOLSS: createSnapshotCmd starts:1395645153847
> 2014-03-24 16:12:34,179 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 1-249561236: Sending { Cmd , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"wait":0}}] }
> 2014-03-24 16:12:34,374 DEBUG [agent.transport.Request] (AgentManager-Handler-8:null) Seq 1-249561236: Processing: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9","id":0}},"result":true,"wait":0}}] }
> 2014-03-24 16:12:34,374 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 1-249561236: Received: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 10, { CreateObjectAnswer } }
> 2014-03-24 16:12:34,784 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) copyAsync inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT
> 2014-03-24 16:12:34,802 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) needCacheStorage true, dest at snapshots/2/3 dest role Image/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9 src role Primary
> 2014-03-24 16:12:34,938 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 1-249561237: Sending { Cmd , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/3","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.S3TO":{"id":2,"uuid":"c18a1785-f578-4102-bb94-0298289786e0","endPoint":"s3.cloudian.com:18080","bucketName":"cloudstack","httpsFlag":false,"created":"Mar 24, 2014 3:39:20 PM","enableRRS":false,"maxSingleUploadSizeInBytes":5368709120}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"cacheTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/3","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.181.168.230/home/staging","_role":"ImageCache"}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"executeInSequence":true,"wait":21600}}] }
> 2014-03-24 16:12:35,090 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 1-249561237: Processing: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Failed to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3","wait":0}}] }
> 2014-03-24 16:12:35,090 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-7:null) Seq 1-249561237: No more commands found
> 2014-03-24 16:12:35,090 DEBUG [agent.transport.Request] (AgentManager-Handler-7:null) Seq 1-249561237: Processing: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Failed to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3","wait":0}}] }
> 2014-03-24 16:12:35,090 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-7:null) Seq 1-249561237: No more commands found
> 2014-03-24 16:12:35,090 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 1-249561237: Received: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 110, { CopyCmdAnswer } }
> 2014-03-24 16:12:35,098 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 3-1564409959: Sending { Cmd , MgmtId: 143618408992, via: 3, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/3","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.181.168.230/home/staging","_role":"ImageCache"}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"wait":0}}] }
> 2014-03-24 16:12:35,260 DEBUG [agent.transport.Request] (AgentManager-Handler-9:null) Seq 3-1564409959: Processing: { Ans: , MgmtId: 143618408992, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
> 2014-03-24 16:12:35,260 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 3-1564409959: Received: { Ans: , MgmtId: 143618408992, via: 3, Ver: v1, Flags: 10, { Answer } }
> 2014-03-24 16:12:35,468 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException: Failed to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3
> at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1001)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
> at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2779)
> at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2014-03-24 16:12:35,635 DEBUG [storage.volume.VolumeServiceImpl] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Take snapshot: 3 failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1028)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
> at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2779)
> at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3
> at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1001)
> ... 16 more
> 2014-03-24 16:12:35,635 DEBUG [user.snapshot.CreateSnapshotCmd] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Failed to create snapshot
> org.apache.cloudstack.api.ServerApiException
> at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:176)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2014-03-24 16:12:35,637 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Complete async job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to create snapshot due to an internal error creating snapshot for volume 3
>
>
> Thanks,
>
> Tom.
--
Cloudian KK - http://www.cloudian.com/get-started.html
Fancy 10TB of full featured S3 Storage?
Checkout the Cloudian® Community Edition!
Re: Snapshot exceptions on 4.2.1 with NFS and S3
Posted by Nux! <nu...@li.nux.ro>.
On 02.04.2014 08:38, Thomas O'Dowd wrote:
> Anyway - decided to
> uninstall my package and install the 355 version that you were using.
> After installing I rebooted. I really didn't expect anything to change
> but Volume Snapshots now work! YEAH!
Glad it works.
I'd still recommend my approach, though, otherwise you're left running
an old and probably insecure KVM.
HTH
Lucian
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
Re: Snapshot exceptions on 4.2.1 with NFS and S3
Posted by Thomas O'Dowd <tp...@cloudian.com>.
Nux! Thanks for this link. I found this earlier through the list also.
I initially discounted it as it seemed to be talking about VM snapshots
on CentOS 6.5 and I'm having trouble with Volume snapshots on COS 6.4.
However, I checked what version of RPM I had installed:
Installed on my COS 6.4 box:
- qemu-img-0.12.1.2-2.415.el6_5.6.x86_64
Your linked page shows this version:
- qemu-img-0.12.1.2-2.355.el6_4_4.1.x86_64.rpm
Hmmm. Different. My RPM looks like a newer version. I had a look at yum
info qemu-img and my package comes from 6.4 Updates. Anyway - decided to
uninstall my package and install the 355 version that you were using.
After installing I rebooted. I really didn't expect anything to change
but Volume Snapshots now work! YEAH!
Snapshot also goes all the way to S3 so that part is working fine too.
Happy days and thanks for your help!
Tom.
On Wed, 2014-04-02 at 02:08 +0100, Nux! wrote:
> On 27.03.2014 02:55, Thomas O'Dowd wrote:
> > I remember this used to work on Cloudstack 4.2 when I was testing it
> > with devcloud but with CloudPlatform 4.2.1 and KVM I can't get
> > snapshots
> > to work.
>
> Might be this
> http://www.nux.ro/archive/2014/01/Taking_KVM_volume_snapshots_with_Cloudstack_4_2_on_CentOS_6_5.html
>
> And update your OS. :)
>
--
Cloudian KK - http://www.cloudian.com/get-started.html
Fancy 10TB of full featured S3 Storage?
Checkout the Cloudian® Community Edition!
Re: Snapshot exceptions on 4.2.1 with NFS and S3
Posted by Nux! <nu...@li.nux.ro>.
On 27.03.2014 02:55, Thomas O'Dowd wrote:
> I remember this used to work on Cloudstack 4.2 when I was testing it
> with devcloud but with CloudPlatform 4.2.1 and KVM I can't get
> snapshots
> to work.
Might be this
http://www.nux.ro/archive/2014/01/Taking_KVM_volume_snapshots_with_Cloudstack_4_2_on_CentOS_6_5.html
And update your OS. :)
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro