You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "draggeta (via GitHub)" <gi...@apache.org> on 2026/08/15 10:33:16 UTC
[I] NAS backup of large VMs fails [cloudstack]
draggeta opened a new issue, #13883:
URL: https://github.com/apache/cloudstack/issues/13883
### problem
When backing up up larger VMs (the ones we've tested have 2+ TB of storage), scheduled and manual backups eventually fail to NFS storage. The files are written until some sort of timeout seems to happen.
The files are available on the NFS storage but every backup they stop after a certain period of time. A side effect of this is that cloudstack doesn't apply retention to the backups in this case so they keep filling up storage.
Smaller servers (tested until 600 GB backup correctly. There is no firewall in the path. The hypervisors and NFS storage are in the same subnet.
```
2026-08-15 06:04:06,169 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-f0772149]) (logid:34dc33a0) Scheduling 1 backup for VM (VM instance {"id":1083,"instanceName":"i-10-1083-VM","state":"Running","type":"User","uuid":"cac7bcdc-2ab2-43b4-81ce-64a5e5d0bf59"}) for backup schedule (BackupSchedule {"id":6,"schedule":"00:06:7","scheduleType":2,"vmId":1083}) at [2026-08-15T06:00:00+0000].
2026-08-15 06:04:06,607 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:[ctx-98b8e7e5, job-5367, ctx-8cb83fa7]) (logid:b96ba19b) Seq 10-4578471971175552676: Sending { Cmd , MgmtId: 90520740805077, via: 10(nc-lab03), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.backup.TakeBackupCommand":{"vmName":"i-10-1083-VM","backupPath":"i-10-1083-VM/2026.08.15.06.04.06","backupRepoType":"nfs","backupRepoAddress":"[fd33:9999:1::21]:/mnt/POOL01/Backup/cs-backup","wait":"0","bypassHostMaintenance":"false"}}] }
2026-08-15 06:08:39,558 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-6:[ctx-9f798002]) (logid:1d647320) Trying to sync backups of VM [VM instance {"id":1083,"instanceName":"i-10-1083-VM","state":"Running","type":"User","uuid":"cac7bcdc-2ab2-43b4-81ce-64a5e5d0bf59"}] using backup provider [nas].
2026-08-15 06:13:40,517 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-5:[ctx-55244db9]) (logid:55fc3c21) Trying to sync backups of VM [VM instance {"id":1083,"instanceName":"i-10-1083-VM","state":"Running","type":"User","uuid":"cac7bcdc-2ab2-43b4-81ce-64a5e5d0bf59"}] using backup provider [nas].
2026-08-15 07:04:06,609 WARN [c.c.a.m.ClusteredAgentAttache] (API-Job-Executor-30:[ctx-98b8e7e5, job-5367, ctx-8cb83fa7]) (logid:b96ba19b) Seq 10-4578471971175552676: Timed out on Seq 10-4578471971175552676: { Cmd , MgmtId: 90520740805077, via: 10(nc-lab03), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.backup.TakeBackupCommand":{"vmName":"i-10-1083-VM","backupPath":"i-10-1083-VM/2026.08.15.06.04.06","backupRepoType":"nfs","backupRepoAddress":"[fd33:9999:1::21]:/mnt/POOL01/Backup/cs-backup","wait":"0","bypassHostMaintenance":"false"}}] }
2026-08-15 07:04:06,609 ERROR [o.a.c.b.NASBackupProvider] (API-Job-Executor-30:[ctx-98b8e7e5, job-5367, ctx-8cb83fa7]) (logid:b96ba19b) Operation to initiate backup timed out for VM i-10-1083-VM
```
### versions
Ubuntu 24.04.4
10G networking over IPv6
Truenas NFS storage
Primary storage is Ceph
Other backup solutions for other hypervisors to the same storage have no issues with long running backups.
### The steps to reproduce the bug
1. Schedule or run backup
2. Runs for a while
3. Fails, no backups show in UI, data files seen on NFS storage
### What to do about it?
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] NAS backup of large VMs fails [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland closed issue #13883: NAS backup of large VMs fails
URL: https://github.com/apache/cloudstack/issues/13883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] NAS backup of large VMs fails [cloudstack]
Posted by "draggeta (via GitHub)" <gi...@apache.org>.
draggeta commented on issue #13883:
URL: https://github.com/apache/cloudstack/issues/13883#issuecomment-5301879255
Now that I looked at it more, it does seem related to maybe the setting:
```
Job cancel threshold minutes (job.cancel.threshold.minutes)
Time (in minutes) for async-jobs to be forcely cancelled if it has been in process for long
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] NAS backup of large VMs fails [cloudstack]
Posted by "draggeta (via GitHub)" <gi...@apache.org>.
draggeta commented on issue #13883:
URL: https://github.com/apache/cloudstack/issues/13883#issuecomment-5309017365
Wow, didn't even see that option documented somewhere. I'll try it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] NAS backup of large VMs fails [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #13883:
URL: https://github.com/apache/cloudstack/issues/13883#issuecomment-5317913083
we welcome any improvements [to the docs](https://github.com/apache/cloudstack-documentation), @draggeta .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] NAS backup of large VMs fails [cloudstack]
Posted by "TadiosAbebe (via GitHub)" <gi...@apache.org>.
TadiosAbebe commented on issue #13883:
URL: https://github.com/apache/cloudstack/issues/13883#issuecomment-5305335879
@draggeta You can add a comma separated key value pairs containing timeouts for different command for the global setting ``commands.timeout`.
For example `TakeBackupCommand=7200,RestoreBackupCommand=7200` and restart your management servers. Then you will be able to see that parameter logged on the management server logs when taking or restoring a backup
```
2026-08-16 05:09:27.883
2026-08-16 05:09:27,645 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-1:[ctx-7ceb121d, job-116486, ctx-16299080]) (logid:f5b8d17e) Wait time setting on org.apache.cloudstack.backup.TakeBackupCommand is 7200 seconds
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] NAS backup of large VMs fails [cloudstack]
Posted by "draggeta (via GitHub)" <gi...@apache.org>.
draggeta commented on issue #13883:
URL: https://github.com/apache/cloudstack/issues/13883#issuecomment-5320034916
That seems to have worked. Thank you guys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org