You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "zidz (via GitHub)" <gi...@apache.org> on 2026/10/09 11:31:15 UTC
[D] RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype [cloudstack]
GitHub user zidz created a discussion: RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype
## TL;DR
I built an **incremental RBD backup provider** for CloudStack (port to **4.23.0.0**).
It uses Ceph RBD snapshots + `export-diff`/`import-diff` so an incremental backup
is ~32x smaller than the full, and a restore replays the full→incremental chain.
Full + incremental + restore are verified end-to-end through the management API
against a real Ceph/KVM host. This is a **prototype for design feedback**, not a
finished PR — I'd like input on two open questions below.
## Motivation
The existing NAS backup provider copies whole disk images every time. On Ceph
RBD primary storage we can do far better: take an RBD snapshot per backup and
export only the diff since the previous snapshot. Backups become small and fast,
and a restore is `import` (base) + `import-diff` (chain).
## Design
- New plugin `plugins/backup/rbd` implementing `BackupProvider`.
- **Full** = `rbd snap create` + `protect` + `export` → `<disk>.<vol>.rbd`.
- **Incremental** = `export-diff --from-snap <parent>` → `<disk>.<vol>.<ts>.rbdiff`.
- **Restore** = `import` base + `import-diff` each link, **truncated at the
selected backup** (restoring an earlier backup lands on that point-in-time).
- Chain metadata (chain id, position, parent snap) stored as backup details.
- Host-side engine: `scripts/vm/hypervisor/kvm/rbdbackup.sh` (called by the KVM
agent wrappers, mirroring the NAS path).
## What changed vs 4.23.0.0
- **New (additive):** `plugins/backup/rbd/**` (provider + tests + spring module),
`scripts/vm/hypervisor/kvm/rbdbackup.sh`.
- **Existing code (minimal, null-guarded):**
- `core/.../backup/{Take,Delete,Restore}BackupCommand.java` — added RBD fields
(parent snap, base snap, rbd uri/snap, disk types). New `Command` fields only;
rolling-upgrade safe (old agents ignore unknown fields).
- `kvm/.../Libvirt{Take,Delete,Restore}BackupCommandWrapper.java` — route
**pure-RBD** VMs to `rbdbackup.sh`; NAS path unchanged (guarded branch).
- `plugins/pom.xml` + `debian/rules` — build & package the new module.
## Enabling it today
`createBackupOffering` is currently **hard-gated to the KBOSS provider**, and
there's no `createBackupRepository` API, so the offering + repository must be
created via DB for now:
- set `backup.framework.enabled=true`, `backup.framework.provider.plugin=rbd`,
`rbd.backup.incremental.enabled=true`
- insert a `backup_repository` (NFS) and a `backup_offering` (provider=`rbd`),
linked by `backup_offering.external_id = backup_repository.uuid`
Full step-by-step is in `plugins/backup/rbd/README.md`.
## Verification (real Ceph/KVM host)
| Scenario | Result |
| --- | --- |
| Full backup | `.rbd` object, ~5 GB |
| Incremental (no disk writes) | `.rbdiff`, ~132 B |
| Incremental (512 MB written) | `.rbdiff`, ~512 MB |
| Restore #1 / #2 / #3 of a 3-backup chain | lands on the correct point-in-time state each time |
| Restore-to-existing-VM | purges dest image, replays chain, data intact |
| `restore-as-template` | rebuilds chain into a new, freely-named image |
## Limitations / known issues
- **Restore is bound to the original VM** (framework limitation — `restoreBackup`
resolves the VM from `backup.getVmId()`). For a new, custom-named VM use
`restore-as-template` → register as template → deploy.
- Offering/repository creation is manual (KBOSS-gated API) — see open question 1.
- The new module is always built/loaded (`plugins/pom.xml`, `debian/rules`), so a
defect there has global blast radius — see open question 2.
## Open questions for the community
1. **Offering API:** should `createBackupOffering` be generalized beyond KBOSS so
RBD (and other providers) can create offerings via API/UI, removing the manual
DB step? Or is there a preferred existing mechanism I missed?
2. **Packaging:** is always-on module registration acceptable, or should the RBD
provider be opt-in (profile/flag) to limit build/startup coupling?
## Try it
Branch `feature/rbd-incremental-backup-4.23` (6 commits on top of `4.23.0.0`): [compare link](https://github.com/apache/cloudstack/compare/4.23.0.0...zidz:cloudstack:feature/rbd-incremental-backup-4.23?expand=1)
Docs: `plugins/backup/rbd/README.md`.
Feel free to get inspiration from, use full or parts of this implementation if anyone have use for it in any way.
GitHub link: https://github.com/apache/cloudstack/discussions/14374
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype
@zidz
I created PR [#14386 – NAS backup: support incremental backups of RBD volumes](https://github.com/apache/cloudstack/pull/14386). This is for the NAS backup provider, not KBOSS.
I think much of the code could be reused to implement the same functionality in KBOSS.
GitHub link: https://github.com/apache/cloudstack/discussions/14374#discussioncomment-18877818
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype
> The backup will be stored in the secondary storage. If you are working on it, please go ahead. My fork is more of a working proof of concept to put the idea and examples out there.
@zidz
IMHO, all changes should be put in plugins/backup/kboss/. no extra jar is needed.
@JoaoJandre @winterhazel can give some advise
GitHub link: https://github.com/apache/cloudstack/discussions/14374#discussioncomment-18834908
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype [cloudstack]
Posted by "JoaoJandre (via GitHub)" <gi...@apache.org>.
GitHub user JoaoJandre added a comment to the discussion: RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype
Also, @zidz, If you decide to extend KBOSS and have any doubts, feel free to reach out to me.
GitHub link: https://github.com/apache/cloudstack/discussions/14374#discussioncomment-18836191
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype [cloudstack]
Posted by "mathwest (via GitHub)" <gi...@apache.org>.
GitHub user mathwest added a comment to the discussion: RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype
This is an incredibly important feature for us as well as it is essentially the only feature missing for us to have 100% parity with what we offered our customers in XO and then some with all the extra that cloudstack offers, Cant state enough how exited i am to see that some work is done on this front and i cannot thank you enough for your time.
GitHub link: https://github.com/apache/cloudstack/discussions/14374#discussioncomment-18842179
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype [cloudstack]
Posted by "StepBee (via GitHub)" <gi...@apache.org>.
GitHub user StepBee added a comment to the discussion: RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype
@zidz @weizhouapache Thats awesome work from both of you, exactly what i've been looking for.
At the moment i'm solving the requirement for incremental backups based on Ceph snapshot-diffs using external scripts - which unfortunately have no integration with Cloudstack so far.
So i am very excited about the work of both of you, thanks a lot!
GitHub link: https://github.com/apache/cloudstack/discussions/14374#discussioncomment-18835317
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype [cloudstack]
Posted by "JoaoJandre (via GitHub)" <gi...@apache.org>.
GitHub user JoaoJandre added a comment to the discussion: RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype
Hello, @zidz
Regarding your first two questions:
1. The `createBackupOffering` API can be generalized for other providers. I created it for KBOSS because it was needed. Currently, no other providers have any use for it so it is gated for KBOSS only. If you want to implement a new provider, I think it is a good idea to extend the `createBackupOffering` API.
2. If you are going to create a new provider, and it does not depend on any non-OSS libs, I would prefer it is always registered. No need for a new flag to build it.
Regarding extending KBOSS with support for RBD incremental backups. I think it would be a great idea, especially if you could make it support all the current workflows that KBOSS supports (backup creation, many types of restore, backup compression (if it makes sense for RBD), backup validation, backup extraction #13901). But even if you make it support only the basics (backup create, restore and delete) it would be great. The other workflows could be extended in the future.
However, if you are going to extend KBOSS, all code should be Java, as I do not believe that maintaining a bash script for a critical feature such as backups is a good idea, so you would have to change a lot of your current PR.
Regarding your process idea, I have one issue: From my testing, when exporting the base images, the whole image seems to be exported, even if most of it is empty. If I have a 100 GB disk with only 10 GB written, this process creates a 100 GB image in the secondary storage; Is there a way to avoid creating thick images?
GitHub link: https://github.com/apache/cloudstack/discussions/14374#discussioncomment-18836179
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype
@zidz
thanks for the sharing.
If I understand correctly, the same ceph RBD pool will be used as the backup repository, right ?
for your information, I am working on incremental backup of RBD volumes to NAS backup right now.
the code has some conflicts with #14194 and #14203, I will maybe wait until the two PRs to be merged.
GitHub link: https://github.com/apache/cloudstack/discussions/14374#discussioncomment-18834570
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype [cloudstack]
Posted by "zidz (via GitHub)" <gi...@apache.org>.
GitHub user zidz added a comment to the discussion: RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype
Note that the rbd export is sparse/thin, it seems to be written as a whole when written to NFS. I guess something is happening when it's written to the secondary storage. This is a big issue actually and breaks the whole idea of the incremental backups... Will look into this.
I've noted your other comments and I will look into a potential refactor of this proof of concept and see if it materialize in something usable.
I will continue to follow up on comments and improve this over time.
GitHub link: https://github.com/apache/cloudstack/discussions/14374#discussioncomment-18839749
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype [cloudstack]
Posted by "zidz (via GitHub)" <gi...@apache.org>.
GitHub user zidz added a comment to the discussion: RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype
The backup will be stored in the secondary storage.
If you are working on it, please go ahead. My fork is more of a working proof of concept to put the idea and examples out there.
GitHub link: https://github.com/apache/cloudstack/discussions/14374#discussioncomment-18834832
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org