[jira] [Created] (CLOUDSTACK-5469) Snapshot creation fails with following exception - "Failed to backup snapshot: qemu-img: Could not delete snapshot '89eced14-9121-44a7-bb97-26b567795726': -2 (No such file or directory)"

["Sangeetha Hariharan (JIRA)" <ji...@apache.org>]

Sangeetha Hariharan created CLOUDSTACK-5469:
-----------------------------------------------

             Summary: Snapshot creation fails with following exception - "Failed to backup snapshot: qemu-img: Could not delete snapshot '89eced14-9121-44a7-bb97-26b567795726': -2 (No such file or directory)"
                 Key: CLOUDSTACK-5469
                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5469
             Project: CloudStack
          Issue Type: Bug
      Security Level: Public (Anyone can view this level - this is the default.)
          Components: Management Server
    Affects Versions: 4.3.0
         Environment: Build from 4.3
            Reporter: Sangeetha Hariharan
            Priority: Critical
             Fix For: 4.3.0


Set up: 
Advanced Zone with 2 KVM (RHEL 6.3) hosts.
 2 NFS secondary stores set up. 
Steps to reproduce the problem:
 1. Deploy 5 Vms in each of the hosts with 10 GB ROOT volume size , so we start with 10 Vms. 
2. Start concurrent snapshots for ROOT volumes of all the Vms. 

1 of the secondary store -ss1- had the nfs server down for 1 and 1/2 hours. 
The other secondary store -ss2 - was always reachable. 
Snapshot tasks that went to the ss1 , succeeded after the nfs server was brought up (It temporarily halted when the the nfs server was down and resumed when the nsf server was made available). 

First set of snapshot tasks that went to the ss2 all succeeded.
 But the next hourly snapshot tasks, few of them failed with following exception: 2013-12-11 16:33:22,427 DEBUG [c.c.s.s.SnapshotManagerImpl] (Job-Executor-64:ctx-9c70ad77 ctx-3d959fa6) Failed t o create snapshot com.cloud.utils.exception.CloudRuntimeException: Failed to backup snapshot: qemu-img: Could not delete snapshot '89eced14-9121-44a7-bb97-26b567795726': -2 (No such file or directory)Failed to delete snapshot 89eced14-9121-44 a7-bb97-26b567795726 for path /mnt/c20ea198-e8ca-33c3-9f11-e361ec9b5532/71a5dce2-da7c-4692-8f25-ba37e5296886 at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:27 5) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStra tegy.java:135) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrate gy.java:294) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocati on.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java: 150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.ja va:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java: 172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at $Proxy161.takeSnapshot(Unknown Source) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341) at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1461) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)

 Copy to the secondary has succeed. Failure happens after this. 

[root@Rack3Host5 118]# ls -ltr
total 10002852
-rw-r--r--. 1 root root 3637903360 Dec 11 20:33 89eced14-9121-44a7-bb97-26b567795726
-rw-r--r--. 1 root root 3638755328 Dec 11 21:37 b38d93db-4c14-45a7-9274-639ad95a3f29
-rw-r--r--. 1 root root 2956619776 Dec 11 22:24 452c8841-2025-41da-b6ec-49cea2a49da8
[root@Rack3Host5 118]#


Following are the volumes which are in "CreatedOnPrimary" state for which the failure occured. 
| 113 | BackedUp | 2013-12-11 20:57:03 | 
| 112 | BackedUp | 2013-12-11 20:57:03 |
 | 110 | BackedUp | 2013-12-11 20:57:03 |
 | 121 | CreatedOnPrimary | 2013-12-11 20:57:04 | 
| 118 | CreatedOnPrimary | 2013-12-11 20:57:04 |
 | 117 | BackedUp | 2013-12-11 20:57:04 |
 | 116 | CreatedOnPrimary | 2013-12-11 20:57:04 | 




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)