You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@cloudstack.apache.org by "midhunpjos (via GitHub)" <gi...@apache.org> on 2025/05/23 07:45:50 UTC

[D] Unexpected Deletion of Valid Disks by CloudStack + XCP-ng 8.2.1 [cloudstack]

GitHub user midhunpjos created a discussion: Unexpected Deletion of Valid Disks by CloudStack + XCP-ng 8.2.1

We recently encountered an unusual issue with CloudStack, where some valid disks assigned to stopped instances were unexpectedly deleted.

Upon investigation, logs from the XCP-ng Center indicated that the delete requests for the VDIs originated from CloudStack. Interestingly, this coincided with the execution of a volume cleanup thread within CloudStack.

However, during this cleanup process, CloudStack also issued a delete command for a disk that was actually in a "Ready" state. we could not find any corresponding delete request for that valid disk in the CloudStack management server logs.

This raises concerns about a potential issue in the volume cleanup logic or logging mechanisms within CloudStack.

Has anyone else experienced a similar scenario, or is aware of what might cause such behavior?


audit logs from xen getting requests from cloudstack

`May 20 06:39:12 dhcld-az1c2-xcph01 xapi: [20250520T00:54:12.925Z|audit||178683230 HTTPS 10.140.1.13->:::80|VDI.destroy R:c81eccd0efa4|audit] ('trackid=47508e295ef7610e1fc86ae9a02090b2' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:VDI_IN_USE: [ OpaqueRef:03e5cebb-c885-4ed9-a7db-27a40bcbee6d; destroy ]' 'API' 'VDI.destroy' (('self' 'ROOT-546' 'e9253423-48eb-471d-be43-4e0f46d44593' 'OpaqueRef:03e5cebb-c885-4ed9-a7db-27a40bcbee6d')))
May 20 06:42:18 dhcld-az1c2-xcph01 xapi: [20250520T00:57:18.072Z|audit||178683230 HTTPS 10.140.1.13->:::80|VDI.destroy R:a5223d94b8f5|audit] ('trackid=47508e295ef7610e1fc86ae9a02090b2' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'VDI.destroy' (('self' 'ROOT-12558' '4d7f2a7d-637e-446c-aa5f-57c867d7848a' 'OpaqueRef:fe55984f-df0f-4a92-ab57-2f381f67c669')))
May 20 06:42:18 dhcld-az1c2-xcph01 xapi: [20250520T00:57:18.107Z|audit||178683230 HTTPS 10.140.1.13->:::80|VDI.destroy R:70523a4fd400|audit] ('trackid=47508e295ef7610e1fc86ae9a02090b2' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:VDI_IN_USE: [ OpaqueRef:fcb01b3f-98a5-4d13-a1ac-fd5337fe2ec0; destroy ]' 'API' 'VDI.destroy' (('self' 'ROOT-12776' '6b03ea00-657b-4567-8063-5b8370e6bfda' 'OpaqueRef:fcb01b3f-98a5-4d13-a1ac-fd5337fe2ec0')))
May 20 06:42:19 dhcld-az1c2-xcph01 xapi: [20250520T00:57:19.883Z|audit||178683230 HTTPS 10.140.1.13->:::80|VDI.destroy R:ef1a9d18f7d6|audit] ('trackid=47508e295ef7610e1fc86ae9a02090b2' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:VDI_IN_USE: [ OpaqueRef:aa7e8124-a669-4edc-b8b8-1610b395b7cd; destroy ]' 'API' 'VDI.destroy' (('self' 'ROOT-13929' 'a0b4f644-bc40-4b82-9f2a-dafcaf6db0ae' 'OpaqueRef:aa7e8124-a669-4edc-b8b8-1610b395b7cd')))
May 20 11:41:28 dhcld-az1c2-xcph01 xapi: [20250520T05:56:28.476Z|audit||178964351 HTTPS 10.140.1.13->:::80|VDI.destroy R:92b87bb5e694|audit] ('trackid=47508e295ef7610e1fc86ae9a02090b2' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'VDI.destroy' (('self' 'ROOT-14400' 'e3bad622-578e-4d29-924e-47b66a6439d6' 'OpaqueRef:e8db1761-e15d-45c3-891e-d7766a867cee')))
May 20 12:05:42 dhcld-az1c2-xcph01 xapi: [20250520T06:20:42.998Z|audit||178983321 HTTPS 10.140.1.13->:::80|VDI.destroy R:ef9634e5414d|audit] ('trackid=47508e295ef7610e1fc86ae9a02090b2' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'VDI.destroy' (('self' 'ROOT-12820' '0f707aa3-d858-4231-b6dc-e92dfe910ba3' 'OpaqueRef:00b6bd67-21a5-43d9-ab86-2442ad13516f')`


Cleanup log reference at the time of incident:
`2025-05-20 06:39:12,873 DEBUG [c.c.a.t.Request] (StorageManager-Scavenger-1:ctx-6ed39762) (logid:69e084ff) Seq 32-4469822630165282172: Sending  { Cmd , MgmtId: 108462341209969, via: 32(DHCLD-AZ1C2-XCPH04), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"649ab11c-ff50-4a81-8e8b-7fe1110a8608","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"PUREC-DHCLD-AZ1C2-VOL1","name":"PUREC-DHCLD-AZ1C2-VOL1","id":"5","poolType":"PreSetup","host":"localhost","path":"/PUREC-DHCLD-AZ1C2-VOL1","port":"0","url":"PreSetup://localhost/PUREC-DHCLD-AZ1C2-VOL1/?ROLE=Primary&STOREUUID=PUREC-DHCLD-AZ1C2-VOL1","isManaged":"false"}},"name":"ROOT-546","size":"(4.89 GB) 5253365760","path":"e9253423-48eb-471d-be43-4e0f46d44593","volumeId":"2164","accountId":"74","format":"VHD","provisioningType":"THIN","poolId":"5","id":"2164","deviceId":"0","hypervisorType":"XenServer"
 ,"directDownload":"false","deployAsIs":"false","followRedirects":"true"}},"wait":"0","bypassHostMaintenance":"false"}}] }
2025-05-20 06:39:12,875 DEBUG [c.c.a.t.Request] (StorageManager-Scavenger-1:ctx-6ed39762) (logid:69e084ff) Seq 32-4469822630165282172: Executing:  { Cmd , MgmtId: 108462341209969, via: 32(DHCLD-AZ1C2-XCPH04), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"649ab11c-ff50-4a81-8e8b-7fe1110a8608","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"PUREC-DHCLD-AZ1C2-VOL1","name":"PUREC-DHCLD-AZ1C2-VOL1","id":"5","poolType":"PreSetup","host":"localhost","path":"/PUREC-DHCLD-AZ1C2-VOL1","port":"0","url":"PreSetup://localhost/PUREC-DHCLD-AZ1C2-VOL1/?ROLE=Primary&STOREUUID=PUREC-DHCLD-AZ1C2-VOL1","isManaged":"false"}},"name":"ROOT-546","size":"(4.89 GB) 5253365760","path":"e9253423-48eb-471d-be43-4e0f46d44593","volumeId":"2164","accountId":"74","format":"VHD","provisioningType":"THIN","poolId":"5","id":"2164","deviceId":"0","hypervisorType":"XenServe
 r","directDownload":"false","deployAsIs":"false","followRedirects":"true"}},"wait":"0","bypassHostMaintenance":"false"}}] }`

GitHub link: https://github.com/apache/cloudstack/discussions/10911

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org

Re: [D] Unexpected Deletion of Valid Disks by CloudStack + XCP-ng 8.2.1 [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.

GitHub user weizhouapache added a comment to the discussion: Unexpected Deletion of Valid Disks by CloudStack + XCP-ng 8.2.1

@midhunpjos 
it might be because vm is destroyed on xenserver when ACS stops it.

the behaviour will be changed in 4.19.3 and 4.20.1
https://github.com/apache/cloudstack/pull/9175



GitHub link: https://github.com/apache/cloudstack/discussions/10911#discussioncomment-13246349

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org