You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "edison su (JIRA)" <ji...@apache.org> on 2013/12/30 20:33:51 UTC
[jira] [Commented] (CLOUDSTACK-5262) Few of the snapshot creation
from ROOT volume fails when there are concurrent snapshots in progress.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859015#comment-13859015 ]
edison su commented on CLOUDSTACK-5262:
---------------------------------------
Anthony, could you help to take a look at this issue, any idea how it could happen?
> Few of the snapshot creation from ROOT volume fails when there are concurrent snapshots in progress.
> -----------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-5262
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5262
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: Management Server
> Affects Versions: 4.3.0
> Environment: Build from 4.3
> Reporter: Sangeetha Hariharan
> Assignee: Anthony Xu
> Priority: Critical
> Fix For: 4.3.0
>
> Attachments: management-server.rar, xen1-2.rar, xen1.rar, xen2.rar
>
>
> Steps to reproduce the problem:
> Set up - Advanced zone with 2 Xenserver 6.2 hosts
> Deploy 10 Vms in each of the hosts , so we start with 20 Vms.
> We are constantly writing to the ROOT volume ( print timestamp every 1 minute).
> Start concurrent snapshots for ROOT volumes for all the Vms.
> Few of the snapshot jobs ( 4 of them) are failing due to the following exception:
> 2013-11-25 08:16:14,239 DEBUG [c.c.a.t.Request] (Job-Executor-165:ctx-bae63a1a ctx-4f637cc6) Seq 2-756819795: Sending { Cmd , MgmtId: 112516401760401, via: 2(Rack3Host23.lab.vmops.com), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"ee56cec0-632d-469f-9326-d683e9fc1425","volume":{"uuid":"aee2805e-6394-47a6-9360-65829ce61929","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"name":"ROOT-73","size":21474836480,"path":"d1988b67-9f8f-4685-9bfd-a171238e8abd","volumeId":73,"vmName":"i-9-73-MyTestVM","accountId":9,"format":"VHD","id":73,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"d3975b00-d9c3-418a-9a39-f09139754a32","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"vmName":"i-9-73-MyTestVM","name":"TestVM-tiny-host-0ps-0-3_ROOT-73_20131125131535","hypervisorType":"XenServer","id":296,"quiescevm":false}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/9/73","volume":{"uuid":"aee2805e-6394-47a6-9360-65829ce61929","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"name":"ROOT-73","size":21474836480,"path":"d1988b67-9f8f-4685-9bfd-a171238e8abd","volumeId":73,"vmName":"i-9-73-MyTestVM","accountId":9,"format":"VHD","id":73,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/9/73/20ebe402-5c2f-4a61-a0c4-525d97ebfa71","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.223.110.233:/export/home/sangeetha/felton/xen/secondary","_role":"Image"}},"vmName":"i-9-73-MyTestVM","name":"TestVM-tiny-host-0ps-0-3_ROOT-73_20131125131535","hypervisorType":"XenServer","id":296,"quiescevm":false}},"executeInSequence":false,"wait":21600}}] }
> 2013-11-25 08:16:14,239 DEBUG [c.c.a.t.Request] (Job-Executor-165:ctx-bae63a1a ctx-4f637cc6) Seq 2-756819795: Executing: { Cmd , MgmtId: 112516401760401, via: 2(Rack3Host23.lab.vmops.com), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"ee56cec0-632d-469f-9326-d683e9fc1425","volume":{"uuid":"aee2805e-6394-47a6-9360-65829ce61929","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"name":"ROOT-73","size":21474836480,"path":"d1988b67-9f8f-4685-9bfd-a171238e8abd","volumeId":73,"vmName":"i-9-73-MyTestVM","accountId":9,"format":"VHD","id":73,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"d3975b00-d9c3-418a-9a39-f09139754a32","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"vmName":"i-9-73-MyTestVM","name":"TestVM-tiny-host-0ps-0-3_ROOT-73_20131125131535","hypervisorType":"XenServer","id":296,"quiescevm":false}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/9/73","volume":{"uuid":"aee2805e-6394-47a6-9360-65829ce61929","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"name":"ROOT-73","size":21474836480,"path":"d1988b67-9f8f-4685-9bfd-a171238e8abd","volumeId":73,"vmName":"i-9-73-MyTestVM","accountId":9,"format":"VHD","id":73,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/9/73/20ebe402-5c2f-4a61-a0c4-525d97ebfa71","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.223.110.233:/export/home/sangeetha/felton/xen/secondary","_role":"Image"}},"vmName":"i-9-73-MyTestVM","name":"TestVM-tiny-host-0ps-0-3_ROOT-73_20131125131535","hypervisorType":"XenServer","id":296,"quiescevm":false}},"executeInSequence":false,"wait":21600}}] }
> 2013-11-25 08:16:14,239 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-364:ctx-a502b0ee) Seq 2-756819795: Executing request
> 2013-11-25 08:16:16,738 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-a502b0ee) callHostPlugin failed for cmd: backupSnapshot with args secondaryStorageMountPath: 10.223.110.233:/export/home/sangeetha/felton/xen/secondary, localMountPoint: /var/run/cloud_mount/e8eabfa9-3012-36f4-adf5-895bbf085947, backupUuid: 07f55f61-e1a5-4880-a9a0-37e063e88e39, path: snapshots/9/73, prevBackupUuid: snapshots/9/73/20ebe402-5c2f-4a61-a0c4-525d97ebfa71, snapshotUuid: ee56cec0-632d-469f-9326-d683e9fc1425, isISCSI: false, primaryStorageSRUuid: 69ec4a37-f19c-8951-8b52-ebe3437f4b5e, due to HandleInvalid clazz:task, handle:OpaqueRef:2d1621a0-7c9e-ea80-48ef-c7114ce3a63d
> 2013-11-25 08:16:16,740 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-364:ctx-a502b0ee) unable to destroy task(com.xensource.xenapi.Task@4b81265b) on host(b8c15cde-f075-487b-86bc-9ef564900887) due to
> You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given.
> at com.xensource.xenapi.Types.checkResponse(Types.java:209)
> at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
> at com.xensource.xenapi.Task.destroy(Task.java:616)
> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPluginAsync(CitrixResourceBase.java:4067)
> at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1131)
> at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1320)
> at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:90)
> at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)
> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:612)
> at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
> at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2013-11-25 08:16:17,097 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-2:null) SeqA 3-37675: Processing Seq 3-37675: { Cmd , MgmtId: -1, via: 3, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] }
> 2013-11-25 08:16:17,100 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-2:null) SeqA 3-37675: Sending Seq 3-37675: { Ans: , MgmtId: 112516401760401, via: 3, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 2013-11-25 08:16:17,755 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-364:ctx-a502b0ee) Could not copy backupUuid: null from primary storage 69ec4a37-f19c-8951-8b52-ebe3437f4b5e to secondary storage 10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null
> 2013-11-25 08:16:17,755 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-364:ctx-a502b0ee) BackupSnapshot Failed due to Could not copy backupUuid: null from primary storage 69ec4a37-f19c-8951-8b52-ebe3437f4b5e to secondary storage 10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null
> com.cloud.utils.exception.CloudRuntimeException: Could not copy backupUuid: null from primary storage 69ec4a37-f19c-8951-8b52-ebe3437f4b5e to secondary storage 10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null
> at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1160)
> at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1320)
> at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:90)
> at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)
> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:612)
> at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
> at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2013-11-25 08:16:17,756 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-364:ctx-a502b0ee) Seq 2-756819795: Response Received:
> 2013-11-25 08:16:17,756 DEBUG [c.c.a.t.Request] (DirectAgent-364:ctx-a502b0ee) Seq 2-756819795: Processing: { Ans: , MgmtId: 112516401760401, via: 2, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"BackupSnapshot Failed due to Could not copy backupUuid: null from primary storage 69ec4a37-f19c-8951-8b52-ebe3437f4b5e to secondary storage 10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null","wait":0}}] }
> 2013-11-25 08:16:17,756 DEBUG [c.c.a.t.Request] (Job-Executor-165:ctx-bae63a1a ctx-4f637cc6) Seq 2-756819795: Received: { Ans: , MgmtId: 112516401760401, via: 2, Ver: v1, Flags: 10, { CopyCmdAnswer } }
> 2013-11-25 08:16:17,768 DEBUG [c.c.s.s.SnapshotManagerImpl] (Job-Executor-165:ctx-bae63a1a ctx-4f637cc6) Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Could not copy backupUuid: null from primary storage 69ec4a37-f19c-8951-8b52-ebe3437f4b5e to secondary storage 10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null
> at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:289)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
> at sun.reflect.GeneratedMethodAccessor444.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy160.takeSnapshot(Unknown Source)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1342)
> at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1402)
> at sun.reflect.GeneratedMethodAccessor443.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy232.takeSnapshot(Unknown Source)
> at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
> at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
> at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2013-11-25 08:16:17,775 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Job-Executor-165:ctx-bae63a1a ctx-4f637cc6) Take snapshot: 73 failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:972)
> at sun.reflect.GeneratedMethodAccessor444.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy160.takeSnapshot(Unknown Source)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1342)
> at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1402)
> at sun.reflect.GeneratedMethodAccessor443.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy232.takeSnapshot(Unknown Source)
> at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
> at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
> at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Could not copy backupUuid: null from primary storage 69ec4a37-f19c-8951-8b52-ebe3437f4b5e to secondary storage 10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null
> at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
> at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:289)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
> ... 42 more
> 2013-11-25 08:16:17,776 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-165:ctx-bae63a1a) Complete async job-631, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 73"}
> mysql> select job_cmd , account_id , job_result, job_result_code, job_status from async_job where job_status=2 and job_cmd="org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd";
> +-------------------------------------------------------------------+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------+
> | job_cmd | account_id | job_result | job_result_code | job_status |
> +-------------------------------------------------------------------+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------+
> | org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd | 9 | org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 73"} | 530 | 2 |
> | org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd | 9 | org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 80"} | 530 | 2 |
> | org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd | 9 | org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 83"} | 530 | 2 |
> | org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd | 9 | org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 85"} | 530 | 2 |
> +-------------------------------------------------------------------+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------+
> 4 rows in set (0.01 sec)
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)