You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "tverkade (via GitHub)" <gi...@apache.org> on 2026/03/06 21:27:59 UTC
[I] Unable to Expunge Mirrored RBD Image from Cloudstack [cloudstack]
tverkade opened a new issue, #12761:
URL: https://github.com/apache/cloudstack/issues/12761
### problem
I'm running into an issue in Cloudstack where when using ceph RBD as primary storage, and having snapshot-based mirroring in Ceph, I cannot delete and expunge the image without manually disabling mirroring:
```
Request failed. (530)
Failed to expunge the volume Volume {"id":1751,"instanceId":null,"name":"TV-REP-02","uuid":"6ad0914e-9172-45d3-87a0-1f34661b1d3f","volumeType":"DATADISK"} in Primary data store : com.cloud.utils.exception.CloudRuntimeException: com.ceph.rbd.RbdException: Failed to find out if snapshot .mirror.primary.625a7cf1-1946-48a3-8fc3-5a4de98bec29.b8365252-c64b-4862-850b-cb29e333ceb7 is protected - No such file or directory
```
### versions
ACS: 4.22.0.0
KVM: Ubuntu 22.04
Ceph: version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable)
### The steps to reproduce the bug
1. Create a disk image in Cloudstack on the RBD storage pool
2. Enable mirroring on the image in ceph
3. Either define a schedule for snapshots at the pool level, define a snapshot schedule for the image itself, or manually initiate a snapshot of the image in ceph
4. Attempt to delete the image in Cloudstack with the expunge flag enabled (or without, it will just fail to delete it later)
5. Get the error
`Failed to find out if snapshot <Image and snapshot IDs> is protected - No such file or directory`
### What to do about it?
It would be excellent if when trying to delete an RBD image from Cloudstack if the command could be passed through first determine whether or not mirroring is enabled, and if so, disable it prior to attempting to delete it:
```
rbd info <pool>/<image ID>
rbd mirror image disable <pool>/<image ID>
```
And then run the actual delete command.
So for example, if the image ID is `df1e2a8b-00ff-4854-b2d4-f8a2d685fa1e'`:
```
root@ceph:/# rbd info cloudstack/df1e2a8b-00ff-4854-b2d4-f8a2d685fa1e
rbd image 'df1e2a8b-00ff-4854-b2d4-f8a2d685fa1e':
size 25 GiB in 6400 objects
order 22 (4 MiB objects)
snapshot_count: 2
id: 3ad0e7be094cd5
block_name_prefix: rbd_data.3ad0e7be094cd5
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling
op_features:
flags:
create_timestamp: Fri Mar 6 20:14:52 2026
access_timestamp: Fri Mar 6 20:14:52 2026
modify_timestamp: Fri Mar 6 20:14:52 2026
journal: 3ad0e7be094cd5
mirroring state: enabled
mirroring mode: snapshot
mirroring global id: a29acb3e-0d54-4de9-ac62-c98e435462ca
mirroring primary: true
root@ceph:/# rbd mirror image disable cloudstack/df1e2a8b-00ff-4854-b2d4-f8a2d685fa1e
Mirroring disabled
```
It would also be excellent to include an option to enable mirroring on an image when it's created in Cloudstack, but that's a little bit outside of the scope of this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Expunge Mirrored RBD Image from Cloudstack [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #12761:
URL: https://github.com/apache/cloudstack/issues/12761#issuecomment-4023742366
@tverkade , I am right in understanding that you are basically asking for defensive code agains an out-of-bounds change in cheph, am I?
As you say/imply we could also natively support snapshot mirroring, which would be a good feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Expunge Mirrored RBD Image from Cloudstack [cloudstack]
Posted by "tverkade (via GitHub)" <gi...@apache.org>.
tverkade commented on issue #12761:
URL: https://github.com/apache/cloudstack/issues/12761#issuecomment-4025033986
> [@tverkade](https://github.com/tverkade) , I am right in understanding that you are basically asking for defensive code agains an out-of-bounds change in cheph, am I? As you say/imply we could also natively support snapshot mirroring, which would be a good feature.
@DaanHoogland yes, even if the scheduling is not managed by Cloudstack (it would be the default schedule applied to the pool), if there is a way that creating an RBD image in Cloudstack in a specific primary storage target and enable the snapshot mirroring feature for that RBD, that would be excellent.
So, for the example In my initial issue, creating an RBD image in Cloudstack that results in the image ID `df1e2a8b-00ff-4854-b2d4-f8a2d685fa1e` with pool `cloudstack` would require snapshot mirroring enabled with:
`image enable cloudstack/df1e2a8b-00ff-4854-b2d4-f8a2d685fa1e snapshot`
That would solve the biggest issue I'm dealing with right now, however that also only solves the issue of enabling mirroring. The other issue is that if you simply enable the snapshot mirroring, it will default to the pool's snapshot schedule. This is a problem as not staggering the schedule (which results in all VMs on this storage are taking snapshots and replicating changes at exactly the same time), would cause potentially noticeable performance issues. The other piece of this is that in order to maintain crash-consistency, we also need to make sure that all disks on a particular VM have a snapshot taken at the exact same time, or the changes on the remote side will be inconsistent with the data on the source.
I'm currently using a series of custom systemd units that trigger scripts to:
1) Query the Cloudstack API for all VMs and their disks that are configured the desired RBD pools
2) Query the images in ceph to see which already have a snapshot schedule and which do not
3) If a new disk is created on a VM that already has other disks with a snapshot schedule, the scripts will apply a schedule that matches the other disk(s).
4) If this is a new VM, it will generate a random offset between 1 and 55 minutes and use that for an hourly snapshot schedule
5) Query the Cloudstack API for any images marked for deletion that reside on RBD pools with mirroring, and disable mirroring so the disks can be expunged.
While this works, these scripts run every 5 minutes, and as I had previously mentioned it prevents expunging directly in the UI until after mirroring is disabled.
I hope this helps explain a bit more of what I'm running into
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Expunge Mirrored RBD Image from Cloudstack [cloudstack]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12761:
URL: https://github.com/apache/cloudstack/issues/12761#issuecomment-5294165061
## ๐ฏ Triage report
CloudStack fails to expunge an RBD volume when Ceph snapshot-based mirroring is enabled on the underlying image, because the delete path doesn't disable mirroring before removing snapshots/image. The reporter proposes automatically disabling mirroring prior to deletion, and separately requests native mirroring support (out of scope for this bug).
### ๐ Assessment
| Dimension | Value | Reasoning |
|---|---|---|
| **Type** | type:bug | Expunge operation fails due to unhandled externally-managed Ceph state (mirroring), not itself a new feature |
| **Component** | component:ceph, component:primary-storage | RBD/Ceph primary storage volume deletion path |
| **Severity** | Severity:Minor | Reporter has a manual workaround (disable mirroring via `rbd` CLI before delete); has a couple of +1 reactions indicating some interest |
| **Labels** | type:bug, component:ceph, component:primary-storage, Severity:Minor | |
| **Coding agent** | Needs more info | The requested defensive fix (check + disable mirroring before delete) is well-scoped and self-contained, but native "enable mirroring from CloudStack" is a separate feature request that would need maintainer design input |
### ๐ Similar issues
No duplicate or closely related open issues found.
<details><summary>๐ก Notes and suggestions</summary>
- A maintainer (DaanHoogland) already engaged, framing this as needing "defensive code against an out-of-bounds change in Ceph" (i.e., mirroring enabled outside CloudStack's knowledge).
- Suggested minimal fix: before `rbd rm`/expunge in the KVM RBD storage adaptor, run `rbd info <pool>/<image>` and if `mirroring state: enabled`, call `rbd mirror image disable <pool>/<image>` first.
- The broader ask (native mirroring management from CloudStack UI/API) should likely be tracked as a separate `type:new-feature` issue if pursued.
</details>
> Generated by [Daily Issue Triage](https://github.com/apache/cloudstack/actions/runs/31806898936) ยท sonnet50 357.5K ยท [โท](https://github.com/search?q=repo%3Aapache%2Fcloudstack+%22gh-aw-workflow-call-id%3A+apache%2Fcloudstack%2Fdaily-issue-triage%22&type=issues)
>
<details>
<summary>Add this agentic workflows to your repo</summary>
To install this agentic workflow, run
```
gh aw add githubnext/agentics/workflows/daily-issue-triage.md@d7c1dc4b72b00607a67caaffdcc216cb64379cf9
```
</details>
<!-- gh-aw-agentic-workflow: Daily Issue Triage, engine: copilot, version: 1.0.52, model: claude-sonnet-5, id: 31806898936, workflow_id: daily-issue-triage, run: https://github.com/apache/cloudstack/actions/runs/31806898936 -->
<!-- gh-aw-workflow-call-id: apache/cloudstack/daily-issue-triage -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org