You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Sangeetha Hariharan (JIRA)" <ji...@apache.org> on 2013/07/16 02:12:48 UTC
[jira] [Reopened] (CLOUDSTACK-2497) Object_Store_Refactor -
Snapshots - Not able to delete snapshots.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sangeetha Hariharan reopened CLOUDSTACK-2497:
---------------------------------------------
Tested with 4.2 build.
Deletion of snapshots from root/data volume succeeds but they are not actually cleaned up from the secondary store.
[root@nfs2 3]# ls -ltr 87
total 1168148
-rw-r--r-- 1 root root 1767191040 Jul 15 16:46 d575dde9-8dc1-4021-bd5c-7ea933f13d2c.vhd
[root@nfs2 3]# ls -ltr 88
total 4448
-rw-r--r-- 1 root root 157606400 Jul 15 16:47 79fe9f80-2d10-4061-9771-743bd9274bbc.vhd
[root@nfs2 3]#
mysql> select * from snapshots;
+----+----------------+------------+-----------+-----------+------------------+------------------+------+---------------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
| id | data_center_id | account_id | domain_id | volume_id | disk_offering_id | status | path | name | uuid | snapshot_type | type_description | size | created | removed | backup_snap_id | swift_id | sechost_id | prev_snap_id | hypervisor_type | version | s3_id |
+----+----------------+------------+-----------+-----------+------------------+------------------+------+---------------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
| 1 | 1 | 2 | 1 | 76 | 12 | BackedUp | NULL | pass123_ROOT-76_20130712180734 | 1b6c063c-d349-4516-ba00-e43685f8a4f6 | 0 | MANUAL | 21474836480 | 2013-07-12 18:07:34 | NULL | NULL | NULL | NULL | NULL | XenServer | 2.2 | NULL |
| 2 | 1 | 3 | 1 | 83 | 3 | BackedUp | NULL | test-99_DATA-79_20130715204931 | ad315ebf-4ee7-42cc-8f79-57b4d126b8f6 | 0 | MANUAL | 5368709120 | 2013-07-15 20:49:31 | NULL | NULL | NULL | NULL | NULL | XenServer | 2.2 | NULL |
| 3 | 1 | 3 | 1 | 82 | 12 | BackedUp | NULL | test-99_ROOT-79_20130715205139 | 6efca827-beb9-4c80-adee-7ca69683fe93 | 0 | MANUAL | 21474836480 | 2013-07-15 20:51:39 | NULL | NULL | NULL | NULL | NULL | XenServer | 2.2 | NULL |
| 4 | 1 | 3 | 1 | 84 | 3 | BackedUp | NULL | test-99_hello_20130715214043 | 3f8f3247-8ea6-4ab5-9f15-d7cc7ba05b47 | 0 | MANUAL | 5368709120 | 2013-07-15 21:40:43 | NULL | NULL | NULL | NULL | NULL | XenServer | 2.2 | NULL |
| 5 | 1 | 3 | 1 | 84 | 3 | BackedUp | NULL | detached_hello_20130715215457 | 90f9ad34-dac4-4555-8aec-55071aaf4506 | 0 | MANUAL | 5368709120 | 2013-07-15 21:54:57 | NULL | NULL | NULL | NULL | NULL | XenServer | 2.2 | NULL |
| 6 | 1 | 3 | 1 | 82 | 12 | CreatedOnPrimary | NULL | test-99_ROOT-79_20130715220708 | d28c7b33-0be6-4b55-8b5e-4b29601a1cb7 | 3 | HOURLY | 21474836480 | 2013-07-15 22:07:08 | NULL | NULL | NULL | NULL | NULL | XenServer | 2.2 | NULL |
| 7 | 1 | 3 | 1 | 82 | 12 | CreatedOnPrimary | NULL | test-99_ROOT-79_20130715230708 | d4352eef-b88b-4ff5-84ce-5f0f1ad7a428 | 3 | HOURLY | 21474836480 | 2013-07-15 23:07:08 | NULL | NULL | NULL | NULL | NULL | XenServer | 2.2 | NULL |
| 8 | 1 | 3 | 1 | 85 | 12 | BackedUp | NULL | yello_ROOT-80_20130715231208 | 80841ce7-bfdf-4be8-9761-a2649e7cfd9b | 3 | HOURLY | 21474836480 | 2013-07-15 23:12:08 | NULL | NULL | NULL | NULL | NULL | XenServer | 2.2 | NULL |
| 9 | 1 | 3 | 1 | 86 | 12 | BackedUp | NULL | yepyep_ROOT-82_20130715231730 | acc10122-e5f4-4a9a-a5e7-42b934d34761 | 0 | MANUAL | 21474836480 | 2013-07-15 23:17:30 | NULL | NULL | NULL | NULL | NULL | XenServer | 2.2 | NULL |
| 10 | 1 | 3 | 1 | 86 | 12 | BackedUp | NULL | yepyep_ROOT-82_20130715232708 | 67df3a27-9f69-4f8a-a3ca-9f271887b8e7 | 3 | HOURLY | 21474836480 | 2013-07-15 23:27:08 | NULL | NULL | NULL | NULL | NULL | XenServer | 2.2 | NULL |
| 11 | 1 | 3 | 1 | 87 | 12 | Destroyed | NULL | trial_ROOT-83_20130715234118 | cb7b7768-981a-491c-9da5-38389acd53d4 | 0 | MANUAL | 21474836480 | 2013-07-15 23:41:18 | NULL | NULL | NULL | NULL | NULL | XenServer | 2.2 | NULL |
| 12 | 1 | 3 | 1 | 88 | 3 | Destroyed | NULL | trial_trial-data_20130715234122 | 3e9f270b-79e4-4d57-8e9b-c256761d4a42 | 0 | MANUAL | 5368709120 | 2013-07-15 23:41:22 | NULL | NULL | NULL | NULL | NULL | XenServer | 2.2 | NULL |
+----+----------------+------------+-----------+-----------+------------------+------------------+------+---------------------------------+--------------------------------------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
12 rows in set (0.00 sec)
mysql> select * from snapshot_store_ref;
+----+----------+-------------+---------------------+--------------+--------+------------+------+---------------+--------------------+-----------------------------------------------------+-----------+--------------+---------+---------------------+
| id | store_id | snapshot_id | created | last_updated | job_id | store_role | size | physical_size | parent_snapshot_id | install_path | state | update_count | ref_cnt | updated |
+----+----------+-------------+---------------------+--------------+--------+------------+------+---------------+--------------------+-----------------------------------------------------+-----------+--------------+---------+---------------------+
| 1 | 1 | 1 | 2013-07-12 18:07:35 | NULL | NULL | Primary | 0 | 0 | 0 | 494a95dc-6a58-4cd5-a908-09c18d309100 | Ready | 2 | NULL | 2013-07-12 18:07:36 |
| 2 | 1 | 1 | 2013-07-12 18:07:36 | NULL | NULL | Image | 0 | 0 | 0 | snapshots/2/76/0e2761d7-5ed1-441a-adfc-9a5c5c288f90 | Ready | 4 | NULL | 2013-07-12 18:32:37 |
| 3 | 1 | 2 | 2013-07-15 20:49:32 | NULL | NULL | Primary | 0 | 0 | 0 | 2aee46f6-8fab-494d-82c5-2e0e6259afc4 | Ready | 2 | NULL | 2013-07-15 20:49:33 |
| 4 | 1 | 2 | 2013-07-15 20:49:33 | NULL | NULL | Image | 0 | 0 | 0 | snapshots/3/83/f5eaf901-a10d-4a74-a5bd-dcabc4f19a40 | Ready | 2 | NULL | 2013-07-15 20:49:59 |
| 5 | 1 | 3 | 2013-07-15 20:51:39 | NULL | NULL | Primary | 0 | 0 | 0 | 0f177e5f-18d6-462f-bef1-e144b40c8597 | Ready | 2 | NULL | 2013-07-15 20:51:41 |
| 6 | 1 | 3 | 2013-07-15 20:51:41 | NULL | NULL | Image | 0 | 0 | 0 | snapshots/3/82/5b4f22ab-1c3a-4547-8f7f-88ff7e9a56e1 | Ready | 2 | NULL | 2013-07-15 20:53:46 |
| 7 | 1 | 4 | 2013-07-15 21:40:43 | NULL | NULL | Primary | 0 | 0 | 0 | d996ce44-64fd-4be4-8703-1420ba248966 | Ready | 2 | NULL | 2013-07-15 21:40:49 |
| 8 | 1 | 4 | 2013-07-15 21:40:49 | NULL | NULL | Image | 0 | 0 | 0 | snapshots/3/84/9c090b7b-681c-4d6c-8c3a-afa9a43201b6 | Ready | 2 | NULL | 2013-07-15 21:41:15 |
| 9 | 1 | 5 | 2013-07-15 21:54:57 | NULL | NULL | Primary | 0 | 0 | 0 | 11fecb31-c5ab-4fd7-a4a3-348b0e9a34cc | Ready | 2 | NULL | 2013-07-15 21:54:58 |
| 10 | 1 | 5 | 2013-07-15 21:54:58 | NULL | NULL | Image | 0 | 0 | 0 | snapshots/3/84/0b403276-ac4a-4b94-ab32-57e8b7a86084 | Ready | 2 | NULL | 2013-07-15 21:55:19 |
| 11 | 1 | 6 | 2013-07-15 22:07:08 | NULL | NULL | Primary | 0 | 0 | 0 | 451ed3c5-9f1a-4bde-8750-38d7a9811b16 | Ready | 2 | NULL | 2013-07-15 22:07:11 |
| 13 | 1 | 7 | 2013-07-15 23:07:08 | NULL | NULL | Primary | 0 | 0 | 0 | 12ae750d-9b5b-4e62-b0a3-7afd7f144ec2 | Ready | 2 | NULL | 2013-07-15 23:07:09 |
| 15 | 1 | 8 | 2013-07-15 23:12:08 | NULL | NULL | Primary | 0 | 0 | 0 | 114f00f5-fbba-4cdf-b4e4-7fb307b37c88 | Ready | 2 | NULL | 2013-07-15 23:12:12 |
| 16 | 1 | 8 | 2013-07-15 23:12:12 | NULL | NULL | Image | 0 | 0 | 0 | snapshots/3/85/35477e1e-9d51-4a4d-be56-087ace6f54eb | Ready | 2 | NULL | 2013-07-15 23:13:27 |
| 17 | 1 | 9 | 2013-07-15 23:17:30 | NULL | NULL | Primary | 0 | 0 | 0 | bfa091df-d46e-43f6-a5f8-6148e25d4450 | Ready | 2 | NULL | 2013-07-15 23:17:31 |
| 18 | 1 | 9 | 2013-07-15 23:17:31 | NULL | NULL | Image | 0 | 0 | 0 | snapshots/3/86/d5eb2119-e9ad-44c1-91e7-10dfcc5408e4 | Ready | 2 | NULL | 2013-07-15 23:18:38 |
| 19 | 1 | 10 | 2013-07-15 23:27:08 | NULL | NULL | Primary | 0 | 0 | 0 | 773e53da-430f-4489-a9a8-96e3650e351c | Ready | 2 | NULL | 2013-07-15 23:27:11 |
| 20 | 1 | 10 | 2013-07-15 23:27:11 | NULL | NULL | Image | 0 | 0 | 0 | snapshots/3/86/b0ba0c64-5806-4c40-8876-83e08f98d89f | Ready | 2 | NULL | 2013-07-15 23:28:12 |
| 21 | 1 | 11 | 2013-07-15 23:41:18 | NULL | NULL | Primary | 0 | 0 | 0 | f1679ce3-7a86-4a21-9dcd-c1c9afddf92e | Ready | 2 | NULL | 2013-07-15 23:41:19 |
| 22 | 1 | 11 | 2013-07-15 23:41:19 | NULL | NULL | Image | 0 | 0 | 0 | snapshots/3/87/d575dde9-8dc1-4021-bd5c-7ea933f13d2c | Destroyed | 4 | NULL | 2013-07-15 23:58:44 |
| 23 | 1 | 12 | 2013-07-15 23:41:22 | NULL | NULL | Primary | 0 | 0 | 0 | 450c5092-b6c5-46a9-8bce-fa1fe8a053ba | Ready | 2 | NULL | 2013-07-15 23:41:23 |
| 24 | 1 | 12 | 2013-07-15 23:41:24 | NULL | NULL | Image | 0 | 0 | 0 | snapshots/3/88/79fe9f80-2d10-4061-9771-743bd9274bbc | Destroyed | 4 | NULL | 2013-07-15 23:49:05 |
+----+----------+-------------+---------------------+--------------+--------+------------+------+---------------+--------------------+-----------------------------------------------------+-----------+--------------+---------+---------------------+
Management server logs:
013-07-15 16:58:43,806 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-15:null) submit async job-292, details: AsyncJobVO {id:292, userId: 3, accountId: 3, sessionKey: null, instanceType: Snapshot, instanceId: 11, cmd: org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd, cmdOriginator: null, cmdInfo: {"response":"json","id":"cb7b7768-981a-491c-9da5-38389acd53d4","sessionkey":"88qIXdzD/QnH8qzcf9Dr8F5gHPw\u003d","cmdEventType":"SNAPSHOT.DELETE","ctxUserId":"3","httpmethod":"GET","_":"1373932724721","ctxAccountId":"3","ctxStartEventId":"1125"}, cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0, result: null, initMsid: 7200344900649, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2013-07-15 16:58:43,808 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null) ===END=== 10.216.50.187 -- GET command=deleteSnapshot&id=cb7b7768-981a-491c-9da5-38389acd53d4&response=json&sessionkey=88qIXdzD%2FQnH8qzcf9Dr8F5gHPw%3D&_=1373932724721
2013-07-15 16:58:43,892 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-11:job-292) Executing org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd for job-292
2013-07-15 16:58:43,982 DEBUG [cloud.user.AccountManagerImpl] (Job-Executor-11:job-292) Access to com.cloud.storage.SnapshotVO$$EnhancerByCGLIB$$95a4dc0f@48ee9639 granted to Acct[3-sangee] by DomainChecker_EnhancerByCloudStack_99f2622d
2013-07-15 16:58:43,984 DEBUG [storage.snapshot.XenserverSnapshotStrategy] (Job-Executor-11:job-292) Calling deleteSnapshot for snapshotId: 11
2013-07-15 16:58:44,026 DEBUG [agent.transport.Request] (Job-Executor-11:job-292) Seq 3-558891095: Sending { Cmd , MgmtId: 7200344900649, via: 3, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/3/87/d575dde9-8dc1-4021-bd5c-7ea933f13d2c","volume":{"uuid":"892ab7cc-94c6-480a-b4a2-93efda70699a","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"be66004e-e6d4-3312-8f15-283793aa26d2","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.232","path":"/export/home/sangeetha/xen/primary","port":2049}},"name":"ROOT-83","size":21474836480,"path":"773f49ab-f41e-4e07-be99-da2f67d3efb0","volumeId":87,"vmName":"i-3-83-VM","accountId":3,"format":"VHD","id":87},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.223.110.232/export/home/sangeetha/xen/secondary/","_role":"Image"}},"vmName":"i-3-83-VM","name":"trial_ROOT-83_20130715234118","hypervisorType":"XenServer","id":11}},"wait":0}}] }
2013-07-15 16:58:44,092 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 3-558891095: Processing: { Ans: , MgmtId: 7200344900649, via: 3, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot file d575dde9-8dc1-4021-bd5c-7ea933f13d2c doesn't exist","wait":0}}] }
2013-07-15 16:58:44,093 DEBUG [agent.transport.Request] (Job-Executor-11:job-292) Seq 3-558891095: Received: { Ans: , MgmtId: 7200344900649, via: 3, Ver: v1, Flags: 10, { Answer } }
2013-07-15 16:58:44,212 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-11:job-292) Complete async job-292, jobStatus: 1, resultCode: 0, result: org.apache.cloudstack.api.response.SuccessResponse@79910c16
2013-07-15 16:58:44,350 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-11:job-292) Done executing org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd for job-292
> Object_Store_Refactor - Snapshots - Not able to delete snapshots.
> -----------------------------------------------------------------
>
> Key: CLOUDSTACK-2497
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2497
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: Management Server
> Affects Versions: 4.2.0
> Environment: Build from object_store
> Reporter: Sangeetha Hariharan
> Priority: Critical
> Fix For: 4.2.0
>
>
> Steps to reproduce the problem:
> Deploy a Vm.
> Take a snapshot of the root volume.
> Delete the snapshot.
> Snapshot deletion job reports success , but the snapshot itself is left in the DB in "Error" state.
> Also the snapshots are not cleaned up from the secondary storage.
> mysql> select * from snapshots;
> +----+----------------+------------+-----------+-----------+------------------+----------+------+-----------------------------------+---------------------------- ----------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+------------- ----+---------+-------+
> | id | data_center_id | account_id | domain_id | volume_id | disk_offering_id | status | path | name | uuid | snapshot_type | type_description | size | created | removed | backup_snap_id | swift_id | sechost_id | prev_snap_id | hypervisor_t ype | version | s3_id |
> +----+----------------+------------+-----------+-----------+------------------+----------+------+-----------------------------------+---------------------------- ----------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+------------- ----+---------+-------+
> | 1 | 1 | 2 | 1 | 5 | 1 | Error | NULL | test-2_ROOT-5_20130514161859 | d44ae73f-3cbd-49d3-8964-6c3 325c24286 | 0 | MANUAL | 8589934592 | 2013-05-14 16:18:59 | NULL | NULL | NULL | NULL | NULL | XenServer | 2.2 | NULL |
> | 2 | 1 | 2 | 1 | 5 | 1 | Error | NULL | test-2_ROOT-5_20130514162437 | c7bc851c-a7aa-4125-8d8e-f3d 70b807e2c | 0 | MANUAL | 8589934592 | 2013-05-14 16:24:37 | NULL | NULL | NULL | NULL | NULL | XenServer | 2.2 | NULL |
> | 3 | 1 | 2 | 1 | 11 | 2 | BackedUp | NULL | test-new-vm_ROOT-9_20130514234030 | 0824997f-8127-463b-b62e-cb4 1714ae0b8 | 0 | MANUAL | 21474836480 | 2013-05-14 23:40:30 | NULL | NULL | NULL | NULL | NULL | XenServer | 2.2 | NULL |
> +----+----------------+------------+-----------+-----------+------------------+----------+------+-----------------------------------+---------------------------- ----------+---------------+------------------+-------------+---------------------+---------+----------------+----------+------------+--------------+------------- ----+---------+-------+
> 3 rows in set (0.00 sec)
> mysql> select * from snapshot_store_ref;
> +----+----------+-------------+---------------------+--------------+--------+------------+------+---------------+--------------------+--------------------------- --------------------------+-------+---------+--------------+---------------------+
> | id | store_id | snapshot_id | created | last_updated | job_id | store_role | size | physical_size | parent_snapshot_id | install_path | state | removed | update_count | updated |
> +----+----------+-------------+---------------------+--------------+--------+------------+------+---------------+--------------------+--------------------------- --------------------------+-------+---------+--------------+---------------------+
> | 1 | 1 | 1 | 2013-05-14 16:18:59 | NULL | NULL | Primary | 0 | 0 | 0 | ae65ebf9-1c85-4ccc-bb91-3f b3b9c5316b | Ready | NULL | 2 | 2013-05-14 16:19:03 |
> | 2 | 1 | 1 | 2013-05-14 16:19:03 | NULL | NULL | Image | 0 | 0 | 0 | snapshots/2/1/29d8dd25-c22 1-4bbc-bc8b-b89d37e362de | Ready | NULL | 2 | 2013-05-14 16:20:45 |
> | 3 | 1 | 2 | 2013-05-14 16:24:37 | NULL | NULL | Primary | 0 | 0 | 0 | c58584dc-a294-4a80-a835-09 0a2504f474 | Ready | NULL | 2 | 2013-05-14 16:24:41 |
> | 4 | 1 | 2 | 2013-05-14 16:24:41 | NULL | NULL | Image | 0 | 0 | 0 | snapshots/2/2/76979672-c72 5-4b20-a517-eb8410205ef6 | Ready | NULL | 4 | 2013-05-14 16:55:53 |
> | 5 | 1 | 3 | 2013-05-14 23:40:30 | NULL | NULL | Primary | 0 | 0 | 0 | 80a3f476-c3ce-421f-9502-a0 d87b87d0d1 | Ready | NULL | 2 | 2013-05-14 23:40:32 |
> | 6 | 1 | 3 | 2013-05-14 23:40:32 | NULL | NULL | Image | 0 | 0 | 0 | snapshots/2/11/e3386b08-b2 a2-4611-9500-0d7fabc32246 | Ready | NULL | 2 | 2013-05-14 23:41:53 |
> +----+----------+-------------+---------------------+--------------+--------+------------+------+---------------+--------------------+--------------------------- --------------------------+-------+---------+--------------+---------------------+
> 6 rows in set (0.00 sec)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira