You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "abh1sar (via GitHub)" <gi...@apache.org> on 2026/08/29 10:20:20 UTC
[PR] Add missing timeout handling in LibvirtRestoreBackupCommandWrapper [cloudstack]
abh1sar opened a new pull request, #14006:
URL: https://github.com/apache/cloudstack/pull/14006
### Description
Script.executeCommand returns null when the command fails, it does not throw, so the try/catch around the mount and umount of the backup repository could never fire and the return value was discarded. A repository that fails to mount was therefore treated as mounted, and the restore carried on against an empty directory until it failed later with a misleading "backup file not found". A failed umount was ignored the same way, leaking the mount. Both now go through executeCommandForExitValue and check the exit value.
The same refactor also dropped the timeouts. mountTimeout was still passed into mountBackupDirectory but never used, and the rsync of the volume lost the command timeout, so both fell back to the one hour default in Script instead of the configured values. An unresponsive repository could hold a restore up for an hour rather than failing after nas.backup.restore.mount.timeout seconds.
This is a regression from https://github.com/apache/cloudstack/commit/56ad044865bc539231a27fcb459326674b0d1fa5
<!--- Describe your changes in DETAIL - And how has behaviour functionally changed. -->
<!-- For new features, provide link to FS, dev ML discussion etc. -->
<!-- In case of bug fix, the expected and actual behaviours, steps to reproduce. -->
<!-- When "Fixes: #<id>" is specified, the issue/PR will automatically be closed when this PR gets merged -->
<!-- For addressing multiple issues/PRs, use multiple "Fixes: #<id>" -->
<!-- Fixes: # -->
<!--- ******************************************************************************* -->
<!--- NOTE: AUTOMATION USES THE DESCRIPTIONS TO SET LABELS AND PRODUCE DOCUMENTATION. -->
<!--- PLEASE PUT AN 'X' in only **ONE** box -->
<!--- ******************************************************************************* -->
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Feature/Enhancement Scale
- [ ] Major
- [ ] Minor
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [ ] Major
- [x] Minor
- [ ] Trivial
### Screenshots (if appropriate):
### How Has This Been Tested?
Verified that restore fails after the specified nas.backup.restore.mount.timeout with a dead address.
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
#### How did you try to break this feature and the system with this change?
<!-- see how your change affects other areas of the code, etc. -->
<!-- Please read the [CONTRIBUTING](https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md) document -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Nas backup: Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006#issuecomment-5664407237
@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Nas backup: Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "shwstppr (via GitHub)" <gi...@apache.org>.
shwstppr commented on PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006#issuecomment-5600544002
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006#discussion_r3887067573
##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtRestoreBackupCommandWrapper.java:
##########
@@ -221,23 +222,33 @@ private String mountBackupDirectory(String backupRepoAddress, String backupRepoT
mountCmd.add("-o");
mountCmd.add(mountOptions);
}
- Script.executeCommand(mountCmd.toArray(new String[0]));
+ exitValue = Script.executeCommandForExitValue(mountTimeout, mountCmd.toArray(new String[0]));
} catch (Exception e) {
logger.error("Failed to mount repository {} of type {} to the directory {}", backupRepoAddress, backupRepoType, mountDirectory, e);
throw new CloudRuntimeException("Failed to mount the backup repository on the KVM host");
}
+ if (exitValue != 0) {
+ logger.error("Failed to mount repository {} of type {} to the directory {}, mount exited with {}", backupRepoAddress,
+ backupRepoType, mountDirectory, exitValue);
+ throw new CloudRuntimeException("Failed to mount the backup repository on the KVM host");
+ }
return mountDirectory;
}
private void unmountBackupDirectory(String backupDirectory) {
+ int exitValue;
try {
String umountPath = Script.getExecutableAbsolutePath("umount");
String[] umountCmd = new String[] { "sudo", umountPath, backupDirectory };
- Script.executeCommand(umountCmd);
+ exitValue = Script.executeCommandForExitValue(umountCmd);
} catch (Exception e) {
Review Comment:
unmountBackupDirectory now uses Script.executeCommandForExitValue(String...) without passing an explicit timeout. That overload delegates to executeCommandForExitValue(0, ...) which falls back to Script's 1-hour default timeout; if umount blocks (common with unreachable NFS/CIFS), restores can still hang in the finally cleanup path far longer than the configured mount timeout.
##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtRestoreBackupCommandWrapper.java:
##########
@@ -221,23 +222,33 @@ private String mountBackupDirectory(String backupRepoAddress, String backupRepoT
mountCmd.add("-o");
mountCmd.add(mountOptions);
}
- Script.executeCommand(mountCmd.toArray(new String[0]));
+ exitValue = Script.executeCommandForExitValue(mountTimeout, mountCmd.toArray(new String[0]));
} catch (Exception e) {
logger.error("Failed to mount repository {} of type {} to the directory {}", backupRepoAddress, backupRepoType, mountDirectory, e);
throw new CloudRuntimeException("Failed to mount the backup repository on the KVM host");
}
+ if (exitValue != 0) {
+ logger.error("Failed to mount repository {} of type {} to the directory {}, mount exited with {}", backupRepoAddress,
+ backupRepoType, mountDirectory, exitValue);
+ throw new CloudRuntimeException("Failed to mount the backup repository on the KVM host");
+ }
Review Comment:
When the mount command exits non-zero, mountBackupDirectory throws but leaves the temporary mount directory created by Files.createTempDirectory() behind. Since the mount failure short-circuits before the restore methods' finally blocks run, this can accumulate stale /tmp/csbackup.* directories on repeated failures.
Consider cleaning up the temporary directory on mount failure before throwing (best-effort, log on failure).
##########
plugins/hypervisors/kvm/src/test/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtRestoreBackupCommandWrapperTest.java:
##########
@@ -404,7 +404,7 @@ public void testExecuteWithRsyncFailure() throws Exception {
.thenAnswer(invocation -> invocation.getArgument(0));
scriptMock.when(() -> Script.executeCommand(any(String[].class)))
.thenReturn(null);
- scriptMock.when(() -> Script.executeCommandForExitValue(any(String[].class)))
+ scriptMock.when(() -> Script.executeCommandForExitValue(anyLong(), any(String[].class)))
.thenAnswer(invocation -> {
if (Arrays.stream(invocation.getArguments()).map(String::valueOf).anyMatch("rsync"::equals)) {
return 1; // Rsync failure
Review Comment:
The Mockito stub meant to simulate an rsync failure never matches because invocation.getArguments() contains (timeout, String[] cmd). Converting the String[] to String via String.valueOf(...) yields an array identity (e.g., "[Ljava.lang.String;@...") rather than the command contents, so the "rsync" check is always false and the test doesn't actually exercise the failure path.
This issue also appears on line 603 of the same file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Nas backup: Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006#issuecomment-5599869736
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Nas backup: Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006#issuecomment-5665407129
@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Nas backup: Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006#issuecomment-5601293225
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19173
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add missing timeout handling in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "abh1sar (via GitHub)" <gi...@apache.org>.
abh1sar commented on PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006#issuecomment-5461962211
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Nas backup: Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "abh1sar (via GitHub)" <gi...@apache.org>.
abh1sar commented on PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006#issuecomment-5664971008
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add missing timeout handling in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006#issuecomment-5461967375
@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Nas backup: Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache merged PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Nas backup: Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "abh1sar (via GitHub)" <gi...@apache.org>.
abh1sar commented on PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006#issuecomment-5585541364
Hi @shwstppr can you please take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Nas backup: Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006#issuecomment-5665374799
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Nas backup: Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006#issuecomment-5665196724
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19239
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add missing timeout handling in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006#issuecomment-5461804057
## [Codecov](https://app.codecov.io/gh/apache/cloudstack/pull/14006?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 3.69%. Comparing base ([`7ea1dca`](https://app.codecov.io/gh/apache/cloudstack/commit/7ea1dca5167ba46c3ef812826b7bfb4083bf408f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`ad63ada`](https://app.codecov.io/gh/apache/cloudstack/commit/ad63ada649c6d2cd3dda8910b42ac0ec22863e1d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (7ea1dca) and HEAD (ad63ada). Click for more details.
>
> <details><summary>HEAD has 1 upload less than BASE</summary>
>
>| Flag | BASE (7ea1dca) | HEAD (ad63ada) |
>|------|------|------|
>|unittests|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## 4.22 #14006 +/- ##
=============================================
- Coverage 17.79% 3.69% -14.11%
=============================================
Files 5928 449 -5479
Lines 534306 38177 -496129
Branches 65383 7072 -58311
=============================================
- Hits 95069 1409 -93660
+ Misses 428467 36581 -391886
+ Partials 10770 187 -10583
```
| [Flag](https://app.codecov.io/gh/apache/cloudstack/pull/14006/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [uitests](https://app.codecov.io/gh/apache/cloudstack/pull/14006/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `3.69% <ø> (ø)` | |
| [unittests](https://app.codecov.io/gh/apache/cloudstack/pull/14006/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/cloudstack/pull/14006?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Nas backup: Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006#issuecomment-5600561469
@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006#issuecomment-5462293994
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19015
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Nas backup: Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006#issuecomment-5664404874
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Nas backup: Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006#issuecomment-5675538079
merging based on approvals and Abhishek's manual verification
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Nas backup: Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #14006:
URL: https://github.com/apache/cloudstack/pull/14006#issuecomment-5674910256
<b>[SF] Trillian test result (tid-16986)</b>
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 49160 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr14006-t16986-kvm-ol8.zip
Smoke tests completed. 149 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org