You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Kishan Kavala (JIRA)" <ji...@apache.org> on 2013/07/25 12:59:49 UTC

[jira] [Resolved] (CLOUDSTACK-3769) [Object_Store_Refactor] Create snapshot failed when tried for more than once

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kishan Kavala resolved CLOUDSTACK-3769.
---------------------------------------

    Resolution: Fixed
    
> [Object_Store_Refactor] Create snapshot failed when tried for more than once
> ----------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-3769
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3769
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Snapshot, Storage Controller, XenServer
>    Affects Versions: 4.2.0
>         Environment: Latest build from ACS 4.2 branch
> Storage: NFS for both primary and secondary
>            Reporter: Sanjeev N
>            Assignee: Kishan Kavala
>            Priority: Critical
>             Fix For: 4.2.0
>
>         Attachments: management-server.rar
>
>
> [Object_Store_Refactor] Create snapshot failed when tried for more than once
> Steps to reproduce the problem:
> ==========================
> Deploy a Vm with root and data disk.
> Take 1 snapshot of root volume.
> Do some changes to the root volume like create files.
> Take another snapshot of root volume. 
> Observations:
> ===========
> Takings 2nd snapshot on the root volume failed with following exceptions:
> 2013-07-24 03:25:21,442 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-1:null) submit async job-42 = [ df65e67e-7912-4e69-b397-ea7f6bbeeb86 ], details: AsyncJobVO {id:42, userId: 2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 6, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdOriginator: null, cmdInfo: {"id":"6","response":"json","sessionkey":"gxyIuffvqQNym83RCq1S1RkHA1Q\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1374650721318","volumeid":"76802bbc-07a7-4afb-8425-178fe4b5f699","ctxAccountId":"2","ctxStartEventId":"169"}, cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0, result: null, initMsid: 6615759585382, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2013-07-24 03:25:21,445 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null) ===END===  10.146.0.20 -- GET  command=createSnapshot&volumeid=76802bbc-07a7-4afb-8425-178fe4b5f699&response=json&sessionkey=gxyIuffvqQNym83RCq1S1RkHA1Q%3D&_=1374650721318
> 2013-07-24 03:25:21,448 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-6:job-42 = [ df65e67e-7912-4e69-b397-ea7f6bbeeb86 ]) Executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-42 = [ df65e67e-7912-4e69-b397-ea7f6bbeeb86 ]
> 2013-07-24 03:25:21,509 INFO  [user.snapshot.CreateSnapshotCmd] (Job-Executor-6:job-42 = [ df65e67e-7912-4e69-b397-ea7f6bbeeb86 ]) VOLSS: createSnapshotCmd starts:1374650721509
> 2013-07-24 03:25:21,616 DEBUG [agent.transport.Request] (Job-Executor-6:job-42 = [ df65e67e-7912-4e69-b397-ea7f6bbeeb86 ]) Seq 1-1326448839: Sending  { Cmd , MgmtId: 6615759585382, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"76802bbc-07a7-4afb-8425-178fe4b5f699","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"05820c30-2a0e-33f7-b358-cc9c08864e10","id":2,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os2","port":2049}},"name":"ROOT-7","size":21474836480,"path":"d91e634e-4eae-4306-8d46-2e587f1f4961","volumeId":14,"vmName":"i-2-7-VM","accountId":2,"format":"VHD","id":14},"parentSnapshotPath":"8ebcb2c7-c728-40be-adbd-5adb3ff26ae8","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"05820c30-2a0e-33f7-b358-cc9c08864e10","id":2,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os2","port":2049}},"vmName":"i-2-7-VM","name":"Delte-snaps_ROOT-7_20130724072521","hypervisorType":"XenServer","id":6}},"wait":0}}] }
> 2013-07-24 03:25:21,617 DEBUG [agent.transport.Request] (Job-Executor-6:job-42 = [ df65e67e-7912-4e69-b397-ea7f6bbeeb86 ]) Seq 1-1326448839: Executing:  { Cmd , MgmtId: 6615759585382, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"76802bbc-07a7-4afb-8425-178fe4b5f699","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"05820c30-2a0e-33f7-b358-cc9c08864e10","id":2,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os2","port":2049}},"name":"ROOT-7","size":21474836480,"path":"d91e634e-4eae-4306-8d46-2e587f1f4961","volumeId":14,"vmName":"i-2-7-VM","accountId":2,"format":"VHD","id":14},"parentSnapshotPath":"8ebcb2c7-c728-40be-adbd-5adb3ff26ae8","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"05820c30-2a0e-33f7-b358-cc9c08864e10","id":2,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os2","port":2049}},"vmName":"i-2-7-VM","name":"Delte-snaps_ROOT-7_20130724072521","hypervisorType":"XenServer","id":6}},"wait":0}}] }
> 2013-07-24 03:25:21,617 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-119:null) Seq 1-1326448839: Executing request
> 2013-07-24 03:25:23,203 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-119:null) Seq 1-1326448839: Response Received:
> 2013-07-24 03:25:23,204 DEBUG [agent.transport.Request] (DirectAgent-119:null) Seq 1-1326448839: Processing:  { Ans: , MgmtId: 6615759585382, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"27f8a618-d6c7-447f-8a38-930d087757bf","id":0}},"result":true,"wait":0}}] }
> 2013-07-24 03:25:23,204 DEBUG [agent.transport.Request] (Job-Executor-6:job-42 = [ df65e67e-7912-4e69-b397-ea7f6bbeeb86 ]) Seq 1-1326448839: Received:  { Ans: , MgmtId: 6615759585382, via: 1, Ver: v1, Flags: 10, { CreateObjectAnswer } }
> 2013-07-24 03:25:23,319 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-6:job-42 = [ df65e67e-7912-4e69-b397-ea7f6bbeeb86 ]) copyAsync inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT
> 2013-07-24 03:25:23,428 DEBUG [agent.transport.Request] (Job-Executor-6:job-42 = [ df65e67e-7912-4e69-b397-ea7f6bbeeb86 ]) Seq 1-1326448840: Sending  { Cmd , MgmtId: 6615759585382, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"27f8a618-d6c7-447f-8a38-930d087757bf","volume":{"uuid":"76802bbc-07a7-4afb-8425-178fe4b5f699","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"05820c30-2a0e-33f7-b358-cc9c08864e10","id":2,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os2","port":2049}},"name":"ROOT-7","size":21474836480,"path":"d91e634e-4eae-4306-8d46-2e587f1f4961","volumeId":14,"vmName":"i-2-7-VM","accountId":2,"format":"VHD","id":14},"parentSnapshotPath":"8ebcb2c7-c728-40be-adbd-5adb3ff26ae8","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"05820c30-2a0e-33f7-b358-cc9c08864e10","id":2,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os2","port":2049}},"vmName":"i-2-7-VM","name":"Delte-snaps_ROOT-7_20130724072521","hypervisorType":"XenServer","id":6}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/14","volume":{"uuid":"76802bbc-07a7-4afb-8425-178fe4b5f699","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"05820c30-2a0e-33f7-b358-cc9c08864e10","id":2,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os2","port":2049}},"name":"ROOT-7","size":21474836480,"path":"d91e634e-4eae-4306-8d46-2e587f1f4961","volumeId":14,"vmName":"i-2-7-VM","accountId":2,"format":"VHD","id":14},"parentSnapshotPath":"snapshots/2/14/bb46b02e-22ad-4642-b1cb-7c4fd0bd0aef","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_xen_os","_role":"Image"}},"vmName":"i-2-7-VM","name":"Delte-snaps_ROOT-7_20130724072521","hypervisorType":"XenServer","id":6}},"executeInSequence":false,"wait":21600}}] }
> 2013-07-24 03:25:23,430 DEBUG [agent.transport.Request] (Job-Executor-6:job-42 = [ df65e67e-7912-4e69-b397-ea7f6bbeeb86 ]) Seq 1-1326448840: Executing:  { Cmd , MgmtId: 6615759585382, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"27f8a618-d6c7-447f-8a38-930d087757bf","volume":{"uuid":"76802bbc-07a7-4afb-8425-178fe4b5f699","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"05820c30-2a0e-33f7-b358-cc9c08864e10","id":2,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os2","port":2049}},"name":"ROOT-7","size":21474836480,"path":"d91e634e-4eae-4306-8d46-2e587f1f4961","volumeId":14,"vmName":"i-2-7-VM","accountId":2,"format":"VHD","id":14},"parentSnapshotPath":"8ebcb2c7-c728-40be-adbd-5adb3ff26ae8","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"05820c30-2a0e-33f7-b358-cc9c08864e10","id":2,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os2","port":2049}},"vmName":"i-2-7-VM","name":"Delte-snaps_ROOT-7_20130724072521","hypervisorType":"XenServer","id":6}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/14","volume":{"uuid":"76802bbc-07a7-4afb-8425-178fe4b5f699","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"05820c30-2a0e-33f7-b358-cc9c08864e10","id":2,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os2","port":2049}},"name":"ROOT-7","size":21474836480,"path":"d91e634e-4eae-4306-8d46-2e587f1f4961","volumeId":14,"vmName":"i-2-7-VM","accountId":2,"format":"VHD","id":14},"parentSnapshotPath":"snapshots/2/14/bb46b02e-22ad-4642-b1cb-7c4fd0bd0aef","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_xen_os","_role":"Image"}},"vmName":"i-2-7-VM","name":"Delte-snaps_ROOT-7_20130724072521","hypervisorType":"XenServer","id":6}},"executeInSequence":false,"wait":21600}}] }
> 2013-07-24 03:25:23,430 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-49:null) Seq 1-1326448840: Executing request
> 2013-07-24 03:25:23,982 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1 is ready to launch secondary storage VM
> 2013-07-24 03:25:24,245 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is ready to launch console proxy
> 2013-07-24 03:25:24,544 DEBUG [cloud.api.ApiServlet] (catalina-exec-16:null) ===START===  10.146.0.20 -- GET  command=queryAsyncJobResult&jobId=df65e67e-7912-4e69-b397-ea7f6bbeeb86&response=json&sessionkey=gxyIuffvqQNym83RCq1S1RkHA1Q%3D&_=1374650724624
> 2013-07-24 03:25:24,611 DEBUG [cloud.api.ApiServlet] (catalina-exec-16:null) ===END===  10.146.0.20 -- GET  command=queryAsyncJobResult&jobId=df65e67e-7912-4e69-b397-ea7f6bbeeb86&response=json&sessionkey=gxyIuffvqQNym83RCq1S1RkHA1Q%3D&_=1374650724624
> 2013-07-24 03:25:24,653 DEBUG [cloud.network.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:null) External devices stats collector is running...
> 2013-07-24 03:25:24,763 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:null) Found 0 running routers.
> 2013-07-24 03:25:24,765 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 routers to update status.
> 2013-07-24 03:25:24,768 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> 2013-07-24 03:25:24,848 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:null) Found 0 running routers.
> 2013-07-24 03:25:24,859 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 routers to update status.
> 2013-07-24 03:25:24,861 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> 2013-07-24 03:25:24,966 WARN  [xen.resource.CitrixResourceBase] (DirectAgent-49:null) Task failed! Task record:                 uuid: fe2ffb5c-69a2-d8ee-4da7-cce570510c86
>            nameLabel: Async.host.call_plugin
>      nameDescription:
>    allowedOperations: []
>    currentOperations: {}
>              created: Wed Jul 24 03:25:22 EDT 2013
>             finished: Wed Jul 24 03:25:22 EDT 2013
>               status: failure
>           residentOn: com.xensource.xenapi.Host@2d122f30
>             progress: 1.0
>                 type: <none/>
>               result:
>            errorInfo: [XENAPI_PLUGIN_FAILURE, backupSnapshot, NameError, global name 'localMountPointPath' is not defined]
>          otherConfig: {}
>            subtaskOf: com.xensource.xenapi.Task@aaf13f6f
>             subtasks: []
> 2013-07-24 03:25:24,975 WARN  [xen.resource.CitrixResourceBase] (DirectAgent-49:null) callHostPlugin failed for cmd: backupSnapshot with args secondaryStorageMountPath: 10.147.28.7:/export/home/sanjeev/sec_xen_os, localMountPoint: /var/run/cloud_mount/c6789378-d1dc-3c93-808d-8c27375483c0, backupUuid: 2b36ca41-14ba-493c-bb9c-fa9bedb59015, path: snapshots/2/14, prevBackupUuid: snapshots/2/14/bb46b02e-22ad-4642-b1cb-7c4fd0bd0aef, snapshotUuid: 27f8a618-d6c7-447f-8a38-930d087757bf, isISCSI: false, primaryStorageSRUuid: 667cf98d-5b59-77c5-3cd5-da4b04513888,  due to Task failed! Task record:                 uuid: fe2ffb5c-69a2-d8ee-4da7-cce570510c86
>            nameLabel: Async.host.call_plugin
>      nameDescription:
>    allowedOperations: []
>    currentOperations: {}
>              created: Wed Jul 24 03:25:22 EDT 2013
>             finished: Wed Jul 24 03:25:22 EDT 2013
>               status: failure
>           residentOn: com.xensource.xenapi.Host@2d122f30
>             progress: 1.0
>                 type: <none/>
>               result:
>            errorInfo: [XENAPI_PLUGIN_FAILURE, backupSnapshot, NameError, global name 'localMountPointPath' is not defined]
>          otherConfig: {}
>            subtaskOf: com.xensource.xenapi.Task@aaf13f6f
>             subtasks: []
> Task failed! Task record:                 uuid: fe2ffb5c-69a2-d8ee-4da7-cce570510c86
>            nameLabel: Async.host.call_plugin
>      nameDescription:
>    allowedOperations: []
>    currentOperations: {}
>              created: Wed Jul 24 03:25:22 EDT 2013
>             finished: Wed Jul 24 03:25:22 EDT 2013
>               status: failure
>           residentOn: com.xensource.xenapi.Host@2d122f30
>             progress: 1.0
>                 type: <none/>
>               result:
>            errorInfo: [XENAPI_PLUGIN_FAILURE, backupSnapshot, NameError, global name 'localMountPointPath' is not defined]
>          otherConfig: {}
>            subtaskOf: com.xensource.xenapi.Task@aaf13f6f
>             subtasks: []
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:3644)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPluginAsync(CitrixResourceBase.java:3958)
>         at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1161)
>         at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1340)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:84)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:623)
>         at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
>         at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:104)
>         at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> But UI shows the snapshot creation successful even for 2nd snapshot.
> DB entries are as follows:
> ====================
> mysql> select * from snapshots where volume_id=14;
> +----+----------------+------------+-----------+-----------+------------------+----------+------+-----------------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
> | id | data_center_id | account_id | domain_id | volume_id | disk_offering_id | status   | path | name                              | uuid                                 | snapshot_type | type_description | size        | created             | removed | backup_snap_id | swift_id | sechost_id | prev_snap_id | hypervisor_type | version | s3_id |
> +----+----------------+------------+-----------+-----------+------------------+----------+------+-----------------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
> |  5 |              1 |          2 |         1 |        14 |                1 | BackedUp | NULL | Delte-snaps_ROOT-7_20130724071710 | 10c21e2e-0aaf-4d65-b002-ded1b9afa64b |             0 | MANUAL           | 21474836480 | 2013-07-24 07:17:10 | NULL    | NULL           |     NULL |       NULL |         NULL | XenServer       | 2.2     |  NULL |
> |  6 |              1 |          2 |         1 |        14 |                1 | BackedUp | NULL | Delte-snaps_ROOT-7_20130724072521 | 5cdf7485-ac85-4565-8c4e-efb304e25eef |             0 | MANUAL           | 21474836480 | 2013-07-24 07:25:21 | NULL    | NULL           |     NULL |       NULL |         NULL | XenServer       | 2.2     |  NULL |
> +----+----------------+------------+-----------+-----------+------------------+----------+------+-----------------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
> 2 rows in set (0.00 sec)
> Here volume id 14 is the Root disk of the vm on which parent snapshot was created.
> mysql> select * from snapshot_store_ref where snapshot_id in (5,6)\G;
> *************************** 1. row ***************************
>                 id: 10
>           store_id: 1
>        snapshot_id: 5
>            created: 2013-07-24 07:17:11
>       last_updated: NULL
>             job_id: NULL
>         store_role: Image
>               size: 0
>      physical_size: 0
> parent_snapshot_id: 0
>       install_path: snapshots/2/14/bb46b02e-22ad-4642-b1cb-7c4fd0bd0aef
>              state: Ready
>       update_count: 2
>            ref_cnt: 0
>            updated: 2013-07-24 07:18:41
>          volume_id: 14
> *************************** 2. row ***************************
>                 id: 11
>           store_id: 2
>        snapshot_id: 6
>            created: 2013-07-24 07:25:21
>       last_updated: NULL
>             job_id: NULL
>         store_role: Primary
>               size: 0
>      physical_size: 0
> parent_snapshot_id: 0
>       install_path: 27f8a618-d6c7-447f-8a38-930d087757bf
>              state: Ready
>       update_count: 2
>            ref_cnt: 0
>            updated: 2013-07-24 07:25:23
>          volume_id: 14
> *************************** 3. row ***************************
>                 id: 12
>           store_id: 1
>        snapshot_id: 6
>            created: 2013-07-24 07:25:23
>       last_updated: NULL
>             job_id: NULL
>         store_role: Image
>               size: 0
>      physical_size: 0
> parent_snapshot_id: 5
>       install_path: snapshots/2/14/null
>              state: Ready
>       update_count: 2
>            ref_cnt: 0
>            updated: 2013-07-24 07:25:26
>          volume_id: 14
> 3 rows in set (0.00 sec)
> ERROR:
> No query specified
> For the 2nd snapshot we see the Install_pass as "snapshots/2/14/null" . Also did not find delta snapshot in the secondary storage in the install_path.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira