You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Sanjeev N (JIRA)" <ji...@apache.org> on 2014/10/28 06:05:33 UTC
[jira] [Created] (CLOUDSTACK-7799) [Critical][Accounts] Account
deletion failed to remove all the entities related to it in case of failure
in deleting one entity
Sanjeev N created CLOUDSTACK-7799:
-------------------------------------
Summary: [Critical][Accounts] Account deletion failed to remove all the entities related to it in case of failure in deleting one entity
Key: CLOUDSTACK-7799
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7799
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.5.0
Environment: Latest build from 4.5 with commit [root@BPKxDmS ~]# cloudstack-sccs
385c4f673dfbd1fd326e539625e2c06db4cdc27d
Reporter: Sanjeev N
Priority: Critical
Fix For: 4.5.0
[Critical][Accounts] Account deletion failed to remove all the entities related to it in case of failure in deleting one entity
Steps to Reproduce:
================
1.Bring up CS in advanced zone with one vmware cluster
2.Create a user account
3.With the user account deploy few (4-5) vms
4.Take snapshots on all of the vms root disks
5.Deploy another vm and simulate snapshot failure operation(In my case I tried snapshot operation with quiesce option set to true) so that snapshot will be in "Allocated" state
6.Delete this user account
Expected Behavior:
==============
Whatever may the state of the snapshot account deletion shall clean all the objects related that account
Actual Behavior:
=============
Account deletion started deleting snapshots. Since one of the snapshots is in Allocated state it failed to delete that snapshot and the Account clean up job ended there. So vms and networks related to the accounts are still there but the account was deleted.
Here is the log snippet:
=================
2014-10-28 15:16:38,822 DEBUG [c.c.a.ApiServlet] (catalina-exec-5:ctx-1c16bc72) ===START=== 10.252.193.8 -- GET command=deleteAccount&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&id=83f26103-9d83-4e4e-8bcc-c282c2acb498&_=1414470667367
2014-10-28 15:16:38,974 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina-exec-5:ctx-1c16bc72 ctx-51140d28) submit async job-53, details: AsyncJobVO {id:53, userId: 2, accountId: 2, instanceType: Account, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd, cmdInfo: {"response":"json","id":"83f26103-9d83-4e4e-8bcc-c282c2acb498","sessionkey":"EpI1EuP5ZFF0VKispQyNb9AxWF4\u003d","ctxDetails":"{\"com.cloud.user.Account\":\"83f26103-9d83-4e4e-8bcc-c282c2acb498\"}","cmdEventType":"ACCOUNT.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1414470667367","uuid":"83f26103-9d83-4e4e-8bcc-c282c2acb498","ctxAccountId":"2","ctxStartEventId":"118"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 6637838401571, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2014-10-28 15:16:38,978 DEBUG [c.c.a.ApiServlet] (catalina-exec-5:ctx-1c16bc72 ctx-51140d28) ===END=== 10.252.193.8 -- GET command=deleteAccount&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&id=83f26103-9d83-4e4e-8bcc-c282c2acb498&_=1414470667367
2014-10-28 15:16:38,996 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-30:ctx-0b9151e2 job-53) Add job-53 into job monitoring
2014-10-28 15:16:38,996 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-30:ctx-0b9151e2 job-53) Executing AsyncJobVO {id:53, userId: 2, accountId: 2, instanceType: Account, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd, cmdInfo: {"response":"json","id":"83f26103-9d83-4e4e-8bcc-c282c2acb498","sessionkey":"EpI1EuP5ZFF0VKispQyNb9AxWF4\u003d","ctxDetails":"{\"com.cloud.user.Account\":\"83f26103-9d83-4e4e-8bcc-c282c2acb498\"}","cmdEventType":"ACCOUNT.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1414470667367","uuid":"83f26103-9d83-4e4e-8bcc-c282c2acb498","ctxAccountId":"2","ctxStartEventId":"118"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 6637838401571, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2014-10-28 15:16:39,153 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Removed account 4
2014-10-28 15:16:39,287 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292948: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/5","wait":0}}] }
2014-10-28 15:16:40,556 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) Seq 4-4574812796478292948: Processing: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2014-10-28 15:16:40,556 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292948: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, { Answer } }
2014-10-28 15:16:40,557 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots for volume: 5 under account: 4
2014-10-28 15:16:40,561 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-12:null) Seq 4-4574812796478292948: No more commands found
2014-10-28 15:16:40,628 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain for snapshot: 1
2014-10-28 15:16:40,637 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 1 doesn't have children, so it's ok to delete it and its parents
2014-10-28 15:16:40,713 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292949: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/5/a2b6fc7f-c086-4494-aa89-d271f0b96d36/a2b6fc7f-c086-4494-aa89-d271f0b96d36","volume":{"uuid":"280c4f37-87ff-4b62-b1f5-5c610e90531d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-5","size":2147483648,"path":"ROOT-5","volumeId":5,"vmName":"i-4-5-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9] i-4-5-QA/ROOT-5.vmdk\",\"[adc6929588b23db89e55fe7db3847af9] 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":5,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-5-QA","name":"v1_ROOT-5_20141027213127","hypervisorType":"VMware","id":1,"quiescevm":false,"physicalSize":0}},"wait":0}}] }
2014-10-28 15:16:40,768 DEBUG [c.c.a.t.Request] (AgentManager-Handler-17:null) Seq 4-4574812796478292949: Processing: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory a2b6fc7f-c086-4494-aa89-d271f0b96d36 doesn't exist","wait":0}}] }
2014-10-28 15:16:40,768 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292949: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, { Answer } }
2014-10-28 15:16:40,874 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292950: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/7","wait":0}}] }
2014-10-28 15:16:42,048 DEBUG [c.c.a.ApiServlet] (catalina-exec-6:ctx-452a9eca) ===START=== 10.252.193.8 -- GET command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470670595
2014-10-28 15:16:42,100 DEBUG [c.c.a.ApiServlet] (catalina-exec-6:ctx-452a9eca ctx-9e36e958) ===END=== 10.252.193.8 -- GET command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470670595
2014-10-28 15:16:42,267 DEBUG [c.c.a.t.Request] (AgentManager-Handler-15:null) Seq 4-4574812796478292950: Processing: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2014-10-28 15:16:42,268 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292950: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, { Answer } }
2014-10-28 15:16:42,268 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots for volume: 7 under account: 4
2014-10-28 15:16:42,271 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-15:null) Seq 4-4574812796478292950: No more commands found
2014-10-28 15:16:42,311 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain for snapshot: 2
2014-10-28 15:16:42,314 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 2 doesn't have children, so it's ok to delete it and its parents
2014-10-28 15:16:42,347 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292951: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/7/ecb858aa-1656-4594-996a-869f1695bce2/ecb858aa-1656-4594-996a-869f1695bce2","volume":{"uuid":"5b639b64-a539-4651-ac4e-f06b69f37d97","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-7","size":2147483648,"path":"ROOT-7","volumeId":7,"vmName":"i-4-7-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9] i-4-7-QA/ROOT-7.vmdk\",\"[adc6929588b23db89e55fe7db3847af9] 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":7,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-7-QA","name":"v2_ROOT-7_20141027213212","hypervisorType":"VMware","id":2,"quiescevm":false,"physicalSize":0}},"wait":0}}] }
2014-10-28 15:16:42,359 DEBUG [c.c.a.t.Request] (AgentManager-Handler-16:null) Seq 4-4574812796478292951: Processing: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory ecb858aa-1656-4594-996a-869f1695bce2 doesn't exist","wait":0}}] }
2014-10-28 15:16:42,360 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292951: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, { Answer } }
2014-10-28 15:16:42,434 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292952: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/8","wait":0}}] }
2014-10-28 15:16:43,338 DEBUG [c.c.a.t.Request] (AgentManager-Handler-19:null) Seq 4-4574812796478292952: Processing: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2014-10-28 15:16:43,339 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292952: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, { Answer } }
2014-10-28 15:16:43,339 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots for volume: 8 under account: 4
2014-10-28 15:16:43,342 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-19:null) Seq 4-4574812796478292952: No more commands found
2014-10-28 15:16:43,354 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain for snapshot: 3
2014-10-28 15:16:43,356 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 3 doesn't have children, so it's ok to delete it and its parents
2014-10-28 15:16:43,389 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292953: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/8/657a763c-4fc6-49e0-8e7b-cb1ad2471a41/657a763c-4fc6-49e0-8e7b-cb1ad2471a41","volume":{"uuid":"1aebff64-f4c6-45bb-add1-249d54eca270","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-8","size":2147483648,"path":"ROOT-8","volumeId":8,"vmName":"i-4-8-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9] i-4-8-QA/ROOT-8.vmdk\",\"[adc6929588b23db89e55fe7db3847af9] 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":8,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-8-QA","name":"v3_ROOT-8_20141027213237","hypervisorType":"VMware","id":3,"quiescevm":false,"physicalSize":0}},"wait":0}}] }
2014-10-28 15:16:43,400 DEBUG [c.c.a.t.Request] (AgentManager-Handler-20:null) Seq 4-4574812796478292953: Processing: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory 657a763c-4fc6-49e0-8e7b-cb1ad2471a41 doesn't exist","wait":0}}] }
2014-10-28 15:16:43,400 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292953: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, { Answer } }
2014-10-28 15:16:43,483 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292954: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/9","wait":0}}] }
2014-10-28 15:16:43,758 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-6ecc0a29) AutoScaling Monitor is running...
2014-10-28 15:16:44,555 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:null) Seq 4-4574812796478292954: Processing: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2014-10-28 15:16:44,556 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292954: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, { Answer } }
2014-10-28 15:16:44,556 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots for volume: 9 under account: 4
2014-10-28 15:16:44,559 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-2:null) Seq 4-4574812796478292954: No more commands found
2014-10-28 15:16:44,579 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain for snapshot: 4
2014-10-28 15:16:44,582 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 4 doesn't have children, so it's ok to delete it and its parents
2014-10-28 15:16:44,616 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292955: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/9/e7d15b83-06be-4596-9572-09478415e776/e7d15b83-06be-4596-9572-09478415e776","volume":{"uuid":"e88d9c15-e81b-44fd-99be-4d14f3d19c06","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-9","size":2147483648,"path":"ROOT-9","volumeId":9,"vmName":"i-4-9-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9] i-4-9-QA/ROOT-9.vmdk\",\"[adc6929588b23db89e55fe7db3847af9] 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":9,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-9-QA","name":"v4_ROOT-9_20141027213330","hypervisorType":"VMware","id":4,"quiescevm":false,"physicalSize":0}},"wait":0}}] }
2014-10-28 15:16:44,628 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:null) Seq 4-4574812796478292955: Processing: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory e7d15b83-06be-4596-9572-09478415e776 doesn't exist","wait":0}}] }
2014-10-28 15:16:44,628 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292955: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, { Answer } }
2014-10-28 15:16:44,692 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292956: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/10","wait":0}}] }
2014-10-28 15:16:44,805 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:null) Seq 4-4574812796478292956: Processing: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2014-10-28 15:16:44,805 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-3:null) Seq 4-4574812796478292956: No more commands found
2014-10-28 15:16:44,805 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292956: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, { Answer } }
2014-10-28 15:16:44,805 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots for volume: 10 under account: 4
2014-10-28 15:16:44,820 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain for snapshot: 5
2014-10-28 15:16:44,822 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 5 doesn't have children, so it's ok to delete it and its parents
2014-10-28 15:16:44,849 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292957: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/10/1c4b5c4a-b14e-4e42-9a0e-698215db758a/1c4b5c4a-b14e-4e42-9a0e-698215db758a","volume":{"uuid":"cb40dae3-6c47-40b1-8f07-f7ef6d095995","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"Data-10","size":5368709120,"path":"9bd9c800f78545db87fc805e9f3fec1d","volumeId":10,"accountId":4,"format":"OVA","provisioningType":"THIN","id":10,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"name":"detached_Data-10_20141027214535","hypervisorType":"VMware","id":5,"quiescevm":false,"physicalSize":0}},"wait":0}}] }
2014-10-28 15:16:44,860 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:null) Seq 4-4574812796478292957: Processing: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory 1c4b5c4a-b14e-4e42-9a0e-698215db758a doesn't exist","wait":0}}] }
2014-10-28 15:16:44,860 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292957: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10, { Answer } }
2014-10-28 15:16:44,928 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292958: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/11","wait":0}}] }
2014-10-28 15:16:44,975 DEBUG [c.c.a.t.Request] (AgentManager-Handler-4:null) Seq 4-4574812796478292958: Processing: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory 11 doesn't exist","wait":0}}] }
2014-10-28 15:16:44,975 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-4:null) Seq 4-4574812796478292958: No more commands found
2014-10-28 15:16:44,975 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq 4-4574812796478292958: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110, { Answer } }
2014-10-28 15:16:44,975 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots for volume: 11 under account: 4
2014-10-28 15:16:44,985 WARN [c.c.u.AccountManagerImpl] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Failed to cleanup account Acct[83f26103-9d83-4e4e-8bcc-c282c2acb498-acc] due to
com.cloud.exception.InvalidParameterValueException: Can't delete snapshotshot 6 due to it is in Allocated Status
at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.deleteSnapshot(XenserverSnapshotStrategy.java:213)
at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshotDirsForAccount(SnapshotManagerImpl.java:623)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy180.deleteSnapshotDirsForAccount(Unknown Source)
at com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManagerImpl.java:706)
at com.cloud.user.AccountManagerImpl.deleteAccount(AccountManagerImpl.java:665)
at com.cloud.user.AccountManagerImpl.deleteUserAccount(AccountManagerImpl.java:1444)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy103.deleteUserAccount(Unknown Source)
at org.apache.cloudstack.region.RegionManagerImpl.deleteUserAccount(RegionManagerImpl.java:187)
at org.apache.cloudstack.region.RegionServiceImpl.deleteUserAccount(RegionServiceImpl.java:121)
at org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd.execute(DeleteAccountCmd.java:104)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:513)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:470)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2014-10-28 15:16:45,008 INFO [c.c.u.AccountManagerImpl] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Cleanup for account 4 is needed.
2014-10-28 15:16:45,047 DEBUG [c.c.a.ApiServlet] (catalina-exec-24:ctx-bcb44ca9) ===START=== 10.252.193.8 -- GET command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470673593
2014-10-28 15:16:45,082 DEBUG [c.c.a.ApiServlet] (catalina-exec-24:ctx-bcb44ca9 ctx-82b3efc5) ===END=== 10.252.193.8 -- GET command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470673593
2014-10-28 15:16:45,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Complete async job-53, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.SuccessResponse/null/{"success":true}
2014-10-28 15:16:45,232 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Publish async job-53 complete on message bus
2014-10-28 15:16:45,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Wake up jobs related to job- 53
2014-10-28 15:16:45,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Update db status for job- 53
2014-10-28 15:16:45,236 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Wake up jobs joined with job- 53 and disjoin all subjobs created from job- 53
2014-10-28 15:16:45,241 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-30:ctx-0b9151e2 job-53) Done executing org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd for job-53
2014-10-28 15:16:45,248 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-30:ctx-0b9151e2 job-53) Remove job-53 from job monitoring
Even the account cleanup thread could not clean any of the remained artifacts like vms,networks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)