You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Joris van Lieshout (JIRA)" <ji...@apache.org> on 2013/09/17 15:55:51 UTC
[jira] [Updated] (CLOUDSTACK-692) The CleanupSnapshotBackup process
on SSVM deletes snapshots that are still in the process of being created.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joris van Lieshout updated CLOUDSTACK-692:
------------------------------------------
Summary: The CleanupSnapshotBackup process on SSVM deletes snapshots that are still in the process of being created. (was: The StorageManager-Scavenger deletes snapshots that are still in the process of being created.)
> The CleanupSnapshotBackup process on SSVM deletes snapshots that are still in the process of being created.
> -----------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-692
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-692
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: Snapshot
> Reporter: Joris van Lieshout
> Priority: Minor
>
> Hi there,
> I think we ran into a bug due to a concurrence of circumstances regarding snapshotting and the cleanup of snapshots.
> The CleanupSnapshotBackup process on the SSVM deletes vhd files that are not known in the database but when, especially long running snapshot, are being copied to secondary storeage there is a gap between the start and finish of the VDI-copy, where the uuid of the destination vhd is not registered in the database. If the CleanupSnapshotBackup deletes the destinaion vhd during this window it results in hanging sparse_dd process on the XenServer hypervisor pointing to a tapdisk2 process with no file behind it.
> ===Secondary storage vm (2 hour time difference due to time zone). second to last line you see the vhd being deleted.
> 2013-09-04 03:14:45,721 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) Request:Seq 261-1870805144: { Cmd , MgmtId: 345052370018, via: 261, Ver: v1, Flags: 100011, [{"CleanupSnapshotBackupCommand":{"secondaryStoragePoolURL":"nfs://mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1","dcId":1,"accountId":45,"volumeId":5863,"validBackupUUIDs":["1a56760b-d1c0-4620-8cf7-271951500d70","b6157bc9-085b-4ed6-95c2-4341f31c64bf","1ff967e3-3606-4112-9155-b1145b2ef576","12fbe4e3-1fdd-4c35-a961-0fce07cff584","278e9915-4f94-40c8-bef4-9c6bc82d4653","6fba1dd7-4736-47b3-9eed-148304c0e192","b9d8c9d8-6445-463b-b4e1-ab3b3f3a67a2","40ba5d72-c69a-46c2-973b-0570c1cabeac","774f2b0e-cdaf-4594-a9f9-4f872dcaad6e","8269f50b-6bec-427c-8186-540df6a75dbf","7b0c6e75-40cf-4dd7-826a-09b39f3da7b5","df7eac9c-137a-4655-9d21-d781916351f1","11ec2db1-a2fc-4221-ae1a-c1ab2bd59509","dfc348e1-af50-4d77-b4a0-6e86fc954e1c","98f64c0f-7498-4c70-8b70-beaefd723b45","c42f9dd5-079d-4b77-86dc-c19b7fbed817"],"wait":0}}] }
> 2013-09-04 03:14:45,722 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) Processing command: com.cloud.agent.api.CleanupSnapshotBackupCommand
> 2013-09-04 03:14:45,723 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Executing: mount
> 2013-09-04 03:14:45,732 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) Execution is successful.
> 2013-09-04 03:14:45,772 WARN [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:) snapshot 8ca9fea4-8a98-4cc3-bba7-cc1dcf32bb24.vhd is not recorded in DB, remove it
> 2013-09-04 03:14:45,772 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) Seq 261-1870805144: { Ans: , MgmtId: 345052370018, via: 261, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
> ==== management-server.log. here you see the snapshot being created, the copyToSecStorage process starting, eventually timing out due to the hanging vdi-copy, failing on retrying because vdi in use (although not existing any more the vdi is still know on xen), retrying some more on another HV and eventuall giving up because it tries to create a duplicate SR.
> 2013-09-04 04:27:10,931 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-69:job-95137) Executing com.cloud.api.commands.CreateSnapshotCmd for job-95137
> 2013-09-04 04:27:10,971 DEBUG [agent.transport.Request] (Job-Executor-69:job-95137) Seq 91-780303147: Sending { Cmd , MgmtId: 345052370017, via: 91, Ver: v1, Flags: 100011, [{"ManageSnapshotCommand":{"_commandSwitch":"-c","_volumePath":"9cb7af90-ca88-4b34-aa6f-bc21c3d4a3aa","_pool":{"id":208,"uuid":"b290385b-466d-3243-a939-3d242164e034","host":"mccpnas3-4-vip1.mccp.mcinfra.net","path":"/volumes/pool0/MCCP-S-SBP1-1_MCCP-XEN-1","port":2049,"type":"NetworkFilesystem"},"_snapshotName":"vlstws3_ROOT-2736_20130904022710","_snapshotId":71889,"_vmName":"i-45-2736-VM","wait":0}}] }
> 2013-09-04 04:27:10,971 DEBUG [agent.transport.Request] (Job-Executor-69:job-95137) Seq 91-780303147: Executing: { Cmd , MgmtId: 345052370017, via: 91, Ver: v1, Flags: 100011, [{"ManageSnapshotCommand":{"_commandSwitch":"-c","_volumePath":"9cb7af90-ca88-4b34-aa6f-bc21c3d4a3aa","_pool":{"id":208,"uuid":"b290385b-466d-3243-a939-3d242164e034","host":"mccpnas3-4-vip1.mccp.mcinfra.net","path":"/volumes/pool0/MCCP-S-SBP1-1_MCCP-XEN-1","port":2049,"type":"NetworkFilesystem"},"_snapshotName":"vlstws3_ROOT-2736_20130904022710","_snapshotId":71889,"_vmName":"i-45-2736-VM","wait":0}}] }
> 2013-09-04 04:27:12,949 DEBUG [agent.transport.Request] (Job-Executor-69:job-95137) Seq 91-780303147: Received: { Ans: , MgmtId: 345052370017, via: 91, Ver: v1, Flags: 10, { ManageSnapshotAnswer } }
> 2013-09-04 04:27:12,991 DEBUG [agent.transport.Request] (Job-Executor-69:job-95137) Seq 91-780303148: Sending { Cmd , MgmtId: 345052370017, via: 91, Ver: v1, Flags: 100011, [{"BackupSnapshotCommand":{"isVolumeInactive":false,"vmName":"i-45-2736-VM","snapshotId":71889,"pool":{"id":208,"uuid":"b290385b-466d-3243-a939-3d242164e034","host":"mccpnas3-4-vip1.mccp.mcinfra.net","path":"/volumes/pool0/MCCP-S-SBP1-1_MCCP-XEN-1","port":2049,"type":"NetworkFilesystem"},"primaryStoragePoolNameLabel":"b290385b-466d-3243-a939-3d242164e034","snapshotUuid":"6396ec81-4128-44ca-a121-d9f1bf789d6c","snapshotName":"vlstws3_ROOT-2736_20130904022710","secondaryStorageUrl":"nfs://mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1","dcId":1,"accountId":45,"volumeId":5863,"volumePath":"9cb7af90-ca88-4b34-aa6f-bc21c3d4a3aa","wait":21600}}] }
> 2013-09-04 04:27:12,991 DEBUG [agent.transport.Request] (Job-Executor-69:job-95137) Seq 91-780303148: Executing: { Cmd , MgmtId: 345052370017, via: 91, Ver: v1, Flags: 100011, [{"BackupSnapshotCommand":{"isVolumeInactive":false,"vmName":"i-45-2736-VM","snapshotId":71889,"pool":{"id":208,"uuid":"b290385b-466d-3243-a939-3d242164e034","host":"mccpnas3-4-vip1.mccp.mcinfra.net","path":"/volumes/pool0/MCCP-S-SBP1-1_MCCP-XEN-1","port":2049,"type":"NetworkFilesystem"},"primaryStoragePoolNameLabel":"b290385b-466d-3243-a939-3d242164e034","snapshotUuid":"6396ec81-4128-44ca-a121-d9f1bf789d6c","snapshotName":"vlstws3_ROOT-2736_20130904022710","secondaryStorageUrl":"nfs://mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1","dcId":1,"accountId":45,"volumeId":5863,"volumePath":"9cb7af90-ca88-4b34-aa6f-bc21c3d4a3aa","wait":21600}}] }
> 2013-09-04 04:27:12,992 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-441:null) Seq 91-780303148: Executing request
> 2013-09-04 04:27:13,561 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-441:null) Creating a not shared SR for nfs://mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1/snapshots/45/5863
> 2013-09-04 04:27:14,499 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-441:null) Checking 00e04f60-4c80-37e4-9abe-855b16fbc9e1 or SR a4ca3b8c-6e6d-5f07-cb71-f6c9b7f950cb on XS[d868bf32-d747-4ec0-b1c2-fba3df1c85bd-192.168.52.15]
> 2013-09-04 04:27:14,512 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-441:null) Host 192.168.52.15 OpaqueRef:5ee9850b-b15c-b39b-dd27-ff226a1b3126: Created a SR; UUID is a4ca3b8c-6e6d-5f07-cb71-f6c9b7f950cb device config is {serverpath=/volumes/pool0/MCCP-SHARED-1-1/snapshots/45/5863, server=mccpnas7.storage.mccp.mcinfra.net}
> 2013-09-04 10:27:12,992 DEBUG [agent.manager.AgentAttache] (Job-Executor-69:job-95137) Seq 91-780303148: Waiting some more time because this is the current command
> 2013-09-04 10:27:34,262 WARN [xen.resource.CitrixResourceBase] (DirectAgent-441:null) Async 21600 seconds timeout for task com.xensource.xenapi.Task@ec1aac0e
> 2013-09-04 10:27:34,585 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-441:null) Host 192.168.52.15 OpaqueRef:5ee9850b-b15c-b39b-dd27-ff226a1b3126: Removing SR
> 2013-09-04 10:27:34,624 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-441:null) Host 192.168.52.15 OpaqueRef:5ee9850b-b15c-b39b-dd27-ff226a1b3126: Catch XenAPIException: This operation cannot be performed because this VDI is in use by some other operation
> 2013-09-04 10:27:34,650 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-441:null) Host 192.168.52.15 OpaqueRef:5ee9850b-b15c-b39b-dd27-ff226a1b3126: Catch XenAPIException: This operation cannot be performed because this VDI is in use by some other operation
> 2013-09-04 10:27:34,650 WARN [xen.resource.CitrixResourceBase] (DirectAgent-441:null) Host 192.168.52.15 OpaqueRef:5ee9850b-b15c-b39b-dd27-ff226a1b3126: Unable to remove SR
> 2013-09-04 10:27:34,650 WARN [xen.resource.CitrixResourceBase] (DirectAgent-441:null) BackupSnapshot Failed due to Async 21600 seconds timeout for task com.xensource.xenapi.Task@ec1aac0e
> 2013-09-04 10:27:34,651 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-441:null) Seq 91-780303148: Response Received:
> 2013-09-04 10:27:34,651 DEBUG [agent.transport.Request] (DirectAgent-441:null) Seq 91-780303148: Processing: { Ans: , MgmtId: 345052370017, via: 91, Ver: v1, Flags: 10, [{"BackupSnapshotAnswer":{"full":true,"result":false,"details":"BackupSnapshot Failed due to Async 21600 seconds timeout for task com.xensource.xenapi.Task@ec1aac0e","wait":0}}] }
> 2013-09-04 10:27:34,651 DEBUG [agent.transport.Request] (Job-Executor-69:job-95137) Seq 91-780303148: Received: { Ans: , MgmtId: 345052370017, via: 91, Ver: v1, Flags: 10, { BackupSnapshotAnswer } }
> 2013-09-04 10:27:34,651 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-69:job-95137) The result for com.cloud.agent.api.BackupSnapshotCommand is BackupSnapshot Failed due to Async 21600 seconds timeout for task com.xensource.xenapi.Task@ec1aac0e through 91
> 2013-09-04 10:29:34,652 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-69:job-95137) Retrying...
> 2013-09-04 10:29:34,659 DEBUG [agent.manager.ClusteredAgentAttache] (Job-Executor-69:job-95137) Seq 89-1251159458: Forwarding Seq 89-1251159458: { Cmd , MgmtId: 345052370017, via: 89, Ver: v1, Flags: 100011, [{"BackupSnapshotCommand":{"isVolumeInactive":false,"vmName":"i-45-2736-VM","snapshotId":71889,"pool":{"id":208,"uuid":"b290385b-466d-3243-a939-3d242164e034","host":"mccpnas3-4-vip1.mccp.mcinfra.net","path":"/volumes/pool0/MCCP-S-SBP1-1_MCCP-XEN-1","port":2049,"type":"NetworkFilesystem"},"primaryStoragePoolNameLabel":"b290385b-466d-3243-a939-3d242164e034","snapshotUuid":"6396ec81-4128-44ca-a121-d9f1bf789d6c","snapshotName":"vlstws3_ROOT-2736_20130904022710","secondaryStorageUrl":"nfs://mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1","dcId":1,"accountId":45,"volumeId":5863,"volumePath":"9cb7af90-ca88-4b34-aa6f-bc21c3d4a3aa","wait":21600}}] } to 345052370018
> 2013-09-04 10:29:43,085 DEBUG [agent.transport.Request] (Job-Executor-69:job-95137) Seq 89-1251159458: Received: { Ans: , MgmtId: 345052370017, via: 89, Ver: v1, Flags: 10, { BackupSnapshotAnswer } }
> 2013-09-04 10:29:43,086 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-69:job-95137) The result for com.cloud.agent.api.BackupSnapshotCommand is BackupSnapshot Failed due to Can not create second storage SR mountpoint: mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1/snapshots/45/5863 due to SR_BACKEND_FAILURE_40The SR scan failed [opterr=['INTERNAL_ERROR', 'Db_exn.Uniqueness_constraint_violation("VDI", "uuid", "df7eac9c-137a-4655-9d21-d781916351f1")']] through 89
> 2013-09-04 10:30:17,489 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-441:null) Ping from 109
> 2013-09-04 10:31:18,345 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-441:null) Seq 95-1001324546: Executing request
> 2013-09-04 10:31:18,642 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-441:null) Seq 95-1001324546: Response Received:
> 2013-09-04 10:31:18,642 DEBUG [agent.transport.Request] (DirectAgent-441:null) Seq 95-1001324546: Processing: { Ans: , MgmtId: 345052370017, via: 95, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
> 2013-09-04 10:31:19,423 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-441:null) Seq 41-1598291970: Executing request
> 2013-09-04 10:31:19,663 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-441:null) Seq 41-1598291970: Response Received:
> 2013-09-04 10:31:19,663 DEBUG [agent.transport.Request] (DirectAgent-441:null) Seq 41-1598291970: Processing: { Ans: , MgmtId: 345052370017, via: 41, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] }
> 2013-09-04 10:31:43,086 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-69:job-95137) Retrying...
> 2013-09-04 10:31:43,091 DEBUG [agent.transport.Request] (Job-Executor-69:job-95137) Seq 93-156960768: Sending { Cmd , MgmtId: 345052370017, via: 93, Ver: v1, Flags: 100011, [{"BackupSnapshotCommand":{"isVolumeInactive":false,"vmName":"i-45-2736-VM","snapshotId":71889,"pool":{"id":208,"uuid":"b290385b-466d-3243-a939-3d242164e034","host":"mccpnas3-4-vip1.mccp.mcinfra.net","path":"/volumes/pool0/MCCP-S-SBP1-1_MCCP-XEN-1","port":2049,"type":"NetworkFilesystem"},"primaryStoragePoolNameLabel":"b290385b-466d-3243-a939-3d242164e034","snapshotUuid":"6396ec81-4128-44ca-a121-d9f1bf789d6c","snapshotName":"vlstws3_ROOT-2736_20130904022710","secondaryStorageUrl":"nfs://mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1","dcId":1,"accountId":45,"volumeId":5863,"volumePath":"9cb7af90-ca88-4b34-aa6f-bc21c3d4a3aa","wait":21600}}] }
> 2013-09-04 10:31:43,091 DEBUG [agent.transport.Request] (Job-Executor-69:job-95137) Seq 93-156960768: Executing: { Cmd , MgmtId: 345052370017, via: 93, Ver: v1, Flags: 100011, [{"BackupSnapshotCommand":{"isVolumeInactive":false,"vmName":"i-45-2736-VM","snapshotId":71889,"pool":{"id":208,"uuid":"b290385b-466d-3243-a939-3d242164e034","host":"mccpnas3-4-vip1.mccp.mcinfra.net","path":"/volumes/pool0/MCCP-S-SBP1-1_MCCP-XEN-1","port":2049,"type":"NetworkFilesystem"},"primaryStoragePoolNameLabel":"b290385b-466d-3243-a939-3d242164e034","snapshotUuid":"6396ec81-4128-44ca-a121-d9f1bf789d6c","snapshotName":"vlstws3_ROOT-2736_20130904022710","secondaryStorageUrl":"nfs://mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1","dcId":1,"accountId":45,"volumeId":5863,"volumePath":"9cb7af90-ca88-4b34-aa6f-bc21c3d4a3aa","wait":21600}}] }
> 2013-09-04 10:31:56,375 DEBUG [agent.transport.Request] (Job-Executor-69:job-95137) Seq 93-156960768: Received: { Ans: , MgmtId: 345052370017, via: 93, Ver: v1, Flags: 10, { BackupSnapshotAnswer } }
> 2013-09-04 10:31:56,376 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-69:job-95137) The result for com.cloud.agent.api.BackupSnapshotCommand is BackupSnapshot Failed due to Can not create second storage SR mountpoint: mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1/snapshots/45/5863 due to SR_BACKEND_FAILURE_40The SR scan failed [opterr=['INTERNAL_ERROR', 'Db_exn.Uniqueness_constraint_violation("VDI", "uuid", "df7eac9c-137a-4655-9d21-d781916351f1")']] through 93
> 2013-09-04 10:33:56,376 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-69:job-95137) Retrying...
> 2013-09-04 10:33:56,381 DEBUG [agent.transport.Request] (Job-Executor-69:job-95137) Seq 95-1001356346: Sending { Cmd , MgmtId: 345052370017, via: 95, Ver: v1, Flags: 100011, [{"BackupSnapshotCommand":{"isVolumeInactive":false,"vmName":"i-45-2736-VM","snapshotId":71889,"pool":{"id":208,"uuid":"b290385b-466d-3243-a939-3d242164e034","host":"mccpnas3-4-vip1.mccp.mcinfra.net","path":"/volumes/pool0/MCCP-S-SBP1-1_MCCP-XEN-1","port":2049,"type":"NetworkFilesystem"},"primaryStoragePoolNameLabel":"b290385b-466d-3243-a939-3d242164e034","snapshotUuid":"6396ec81-4128-44ca-a121-d9f1bf789d6c","snapshotName":"vlstws3_ROOT-2736_20130904022710","secondaryStorageUrl":"nfs://mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1","dcId":1,"accountId":45,"volumeId":5863,"volumePath":"9cb7af90-ca88-4b34-aa6f-bc21c3d4a3aa","wait":21600}}] }
> 2013-09-04 10:33:56,381 DEBUG [agent.transport.Request] (Job-Executor-69:job-95137) Seq 95-1001356346: Executing: { Cmd , MgmtId: 345052370017, via: 95, Ver: v1, Flags: 100011, [{"BackupSnapshotCommand":{"isVolumeInactive":false,"vmName":"i-45-2736-VM","snapshotId":71889,"pool":{"id":208,"uuid":"b290385b-466d-3243-a939-3d242164e034","host":"mccpnas3-4-vip1.mccp.mcinfra.net","path":"/volumes/pool0/MCCP-S-SBP1-1_MCCP-XEN-1","port":2049,"type":"NetworkFilesystem"},"primaryStoragePoolNameLabel":"b290385b-466d-3243-a939-3d242164e034","snapshotUuid":"6396ec81-4128-44ca-a121-d9f1bf789d6c","snapshotName":"vlstws3_ROOT-2736_20130904022710","secondaryStorageUrl":"nfs://mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1","dcId":1,"accountId":45,"volumeId":5863,"volumePath":"9cb7af90-ca88-4b34-aa6f-bc21c3d4a3aa","wait":21600}}] }
> 2013-09-04 10:34:13,682 DEBUG [agent.transport.Request] (Job-Executor-69:job-95137) Seq 95-1001356346: Received: { Ans: , MgmtId: 345052370017, via: 95, Ver: v1, Flags: 10, { BackupSnapshotAnswer } }
> 2013-09-04 10:34:13,683 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-69:job-95137) The result for com.cloud.agent.api.BackupSnapshotCommand is BackupSnapshot Failed due to Can not create second storage SR mountpoint: mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1/snapshots/45/5863 due to SR_BACKEND_FAILURE_40The SR scan failed [opterr=['INTERNAL_ERROR', 'Db_exn.Uniqueness_constraint_violation("VDI", "uuid", "df7eac9c-137a-4655-9d21-d781916351f1")']] through 95
> 2013-09-04 10:36:13,683 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-69:job-95137) Retrying...
> 2013-09-04 10:36:13,685 WARN [storage.snapshot.SnapshotManagerImpl] (Job-Executor-69:job-95137) Storage unavailable
> 2013-09-04 10:36:13,685 WARN [storage.snapshot.SnapshotManagerImpl] (Job-Executor-69:job-95137) Failed to back up snapshot on secondary storage, deleting the record from the DB
> 2013-09-04 10:36:13,702 ERROR [cloud.api.ApiDispatcher] (Job-Executor-69:job-95137) Exception while executing CreateSnapshotCmd:
> 2013-09-04 10:36:13,702 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-69:job-95137) Complete async job-95137, jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Created snapshot: com.cloud.storage.SnapshotVO$$EnhancerByCGLIB$$3173e384@40abfe02 on primary but failed to backup on secondary
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira