You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "nxsbi (via GitHub)" <gi...@apache.org> on 2026/01/27 07:12:30 UTC
[I] After upgrade to 4.22.0.0 snapshot.delta.max not respected [cloudstack]
nxsbi opened a new issue, #12524:
URL: https://github.com/apache/cloudstack/issues/12524
### problem
I upgraded to 4.22.0.0 on Nov 11. I have many volumes that are set to 3 snapshots max stored by my users.
I noticed that all snapshots since Nov 11 (all existing and new ones) seem to create a massive chain of snapshots.
I had 60+ days snapshot files on the disk for every volume with daily snapshots, while the snapshot.delta.max is only set to 7. And snapshot policy is 3 snapshots only.
It seems the full snapshot before Nov 11 in my case was the last full snapshot. After that there is no more full snapshots, only incremental snapshots.
What I see in database:
in snapshots table:
<img width="1350" height="348" alt="Image" src="https://github.com/user-attachments/assets/46be9729-75d2-4898-9afb-b7e2af1d86c7" />
And in snapshot_store_ref (have hidden rows in between - but they all show as "Hidden"
<img width="1341" height="480" alt="Image" src="https://github.com/user-attachments/assets/9d6c7cca-fc07-4a29-9003-03814c71e55c" />
Please help! I cannot even clear my storage which is now hit 95% used capacity and about to blow up!! Its tens of extra terabytes of data!! with no way to clear it other than deleting all snapshots which is a huge problem for users...
### versions
ACS 4.22.0
XCP-NG 8.2
### The steps to reproduce the bug
1.Upgraded from 4.19 .1 to 4.22
2.
3.
...
### What to do about it?
Identify what is causing the snapshot.delta.max to not be respected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] After upgrade to 4.22.0.0 snapshot.delta.max not respected [cloudstack]
Posted by "nvazquez (via GitHub)" <gi...@apache.org>.
nvazquez commented on issue #12524:
URL: https://github.com/apache/cloudstack/issues/12524#issuecomment-3853475294
Thanks @JoaoJandre for your investigation and the fix PR - will review it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] After upgrade to 4.22.0.0 snapshot.delta.max not respected [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti closed issue #12524: After upgrade to 4.22.0.0 snapshot.delta.max not respected
URL: https://github.com/apache/cloudstack/issues/12524
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] After upgrade to 4.22.0.0 snapshot.delta.max not respected [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #12524:
URL: https://github.com/apache/cloudstack/issues/12524#issuecomment-3811231000
@nvazquez , do we need to check this in 20.2 as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] After upgrade to 4.22.0.0 snapshot.delta.max not respected [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on issue #12524:
URL: https://github.com/apache/cloudstack/issues/12524#issuecomment-4221051781
Fixed in #12597
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] After upgrade to 4.22.0.0 snapshot.delta.max not respected [cloudstack]
Posted by "nxsbi (via GitHub)" <gi...@apache.org>.
nxsbi commented on issue #12524:
URL: https://github.com/apache/cloudstack/issues/12524#issuecomment-3807150384
Hi @nvazquez
Before upgrade the data reflected only two values - Destroyed or BackedUp (or Ready)
I did not do any extra steps around this.
Related to this - I had originally reported a related issue - https://github.com/apache/cloudstack/issues/9501
This one is same issue, except that now there are no Full snapshots every 7 days at all. I used to be able to determine which is a Full snapshot based on Parent ID = 0 (in snapshot_store_ref) and now I no longer see that.
I also reviewed the actual snapshot sizes and there is no large files except the one from before upgrade.
Not sure if any other xcp-ng + cloudstack users can comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] After upgrade to 4.22.0.0 snapshot.delta.max not respected [cloudstack]
Posted by "JoaoJandre (via GitHub)" <gi...@apache.org>.
JoaoJandre commented on issue #12524:
URL: https://github.com/apache/cloudstack/issues/12524#issuecomment-3849420368
Hello, @nxsbi
I was able to reproduce the problem. This happens when the number of snapshots configured to be kept for the policies is lower than the `snapshot.delta.max` configuration, what happens is that the chain is never ended and thus the older snapshots are never removed in order to keep the newer ones valid. I will create a PR to fix this in 4.22.1.0.
The workaround that I would suggest is setting the maximum number of snapshots for the user snapshot policies to be the same as the value of `snapshot.delta.max`. This will ensure that a full snapshot chain is created and finished each cycle and the older snapshot chains will be deleted. Could you test and check if this workaround works for you?
@nvazquez This affects only XCP-ng, the snapshot creation logic is different for KVM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] After upgrade to 4.22.0.0 snapshot.delta.max not respected [cloudstack]
Posted by "JoaoJandre (via GitHub)" <gi...@apache.org>.
JoaoJandre commented on issue #12524:
URL: https://github.com/apache/cloudstack/issues/12524#issuecomment-3847305105
> Hi [@nxsbi](https://github.com/nxsbi) thanks for reporting the issue. An additional question, before the upgrade did you see these 'Hidden' entries on database, or were them removed?
>
> On 4.21 the KVM incremental snapshot feature has been introduced ([#8907](https://github.com/apache/cloudstack/issues/8907)) which seems to do a similar approach leaving Hidden entries on database but not removing them from storage, is it possible that its also the default behavior now for Xen/XCP? [@sureshanaparti](https://github.com/sureshanaparti) [@JoaoJandre](https://github.com/JoaoJandre) [@DaanHoogland](https://github.com/DaanHoogland)
@nvazquez As far as I remember the logic for snapshot creation and removal is the same for KVM and Xen. I'll try to reproduce the issue on 4.22 using XCP-ng.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] After upgrade to 4.22.0.0 snapshot.delta.max not respected [cloudstack]
Posted by "nvazquez (via GitHub)" <gi...@apache.org>.
nvazquez commented on issue #12524:
URL: https://github.com/apache/cloudstack/issues/12524#issuecomment-3806619899
Hi @nxsbi thanks for reporting the issue. An additional question, before the upgrade did you see these 'Hidden' entries on database, or were them removed?
On 4.21 the KVM incremental snapshot feature has been introduced (https://github.com/apache/cloudstack/issues/8907) which seems to do a similar approach leaving Hidden entries on database but not removing them from storage, is it possible that its also the default behavior now for Xen/XCP? @sureshanaparti @JoaoJandre @DaanHoogland
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org