You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "hsato03 (via GitHub)" <gi...@apache.org> on 2025/11/17 20:18:52 UTC
[PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
hsato03 opened a new pull request, #12086:
URL: https://github.com/apache/cloudstack/pull/12086
### Description
ACS allows data to be migrated from one secondary storage to another; however, this process disregards incremental volume snapshots created using the KVM hypervisor.
Since then, the migration process has been extended to also allow the migration of these resources.
<!--- Describe your changes in DETAIL - And how has behaviour functionally changed. -->
<!-- For new features, provide link to FS, dev ML discussion etc. -->
<!-- In case of bug fix, the expected and actual behaviours, steps to reproduce. -->
<!-- When "Fixes: #<id>" is specified, the issue/PR will automatically be closed when this PR gets merged -->
<!-- For addressing multiple issues/PRs, use multiple "Fixes: #<id>" -->
<!-- Fixes: # -->
<!--- ******************************************************************************* -->
<!--- NOTE: AUTOMATION USES THE DESCRIPTIONS TO SET LABELS AND PRODUCE DOCUMENTATION. -->
<!--- PLEASE PUT AN 'X' in only **ONE** box -->
<!--- ******************************************************************************* -->
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [X] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Feature/Enhancement Scale
- [ ] Major
- [ ] Minor
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [ ] Major
- [X] Minor
- [ ] Trivial
### Screenshots (if appropriate):
### How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
| N° | Test | API | Result | Expected result (Y/N) |
| ------ | ------ | ------ | ------ | :------: |
| 1 | Migrate a single incremental volume snapshot | `migrateResourceToAnotherSecondaryStorage` | The snapshot, along with its checkpoint file, was migrated to the destination storage, and its child snapshot, which remained in the old storage, was rebased onto the snapshot in the new storage | Y |
| 2 | Migrate the entire chain of incremental volume snapshots to another secondary storage | `migrateSecondaryStorageData` | All snapshots were migrated and the chain was correctly updated with the files from the new storage | Y |
| 3 | Migrate a chain of incremental volume snapshots split between 2 secondary storages | `migrateSecondaryStorageData` | The chain was updated based on existing snapshots in the destination storage and the new file from the migrated snapshots | Y |
#### How did you try to break this feature and the system with this change?
<!-- see how your change affects other areas of the code, etc. -->
<!-- Please read the [CONTRIBUTING](https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md) document -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3544000865
Packaging result [SF]: ✔️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 15785
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4010460717
<b>[SF] Trillian Build Failed (tid-15574)<b/>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4008931230
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17018
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "hsato03 (via GitHub)" <gi...@apache.org>.
hsato03 commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4008667032
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "JoaoJandre (via GitHub)" <gi...@apache.org>.
JoaoJandre commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4327477150
Manually tested:
### Using Xen server
- Took 1 snapshot, migrated secondary storage and checked manually on secondary that the snapshot was migrated
- Took 2 snapshots, migrated secondary storage and checked manually on secondary that the snapshots were migrated
### Using KVM without incremental snapshot
- Took 1 snapshot, migrated secondary storage and checked manually on secondary that the snapshot was migrated
- Took 2 snapshots, migrated secondary storage and checked manually on secondary that the snapshots were migrated
### Using KVM with incremental snapshot
- Took 1 snapshot, migrated secondary storage and checked manually on secondary that the snapshot was migrated
- Created 3 different snapshot chains, migrated, restored a few snapshots, and checked manually on secondary that the chains were consistent.
All operations on all tests were successful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4329107264
@hsato03 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3957726464
@hsato03 , Do you want thios in 23 or an earlier release? (seems like 22.1 would make sense to me)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-5937049445
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19373
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "hsato03 (via GitHub)" <gi...@apache.org>.
hsato03 commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3543728708
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3570404504
<b>[SF] Trillian Build Failed (tid-14853)<b/>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3562233102
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 15805
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4341559748
@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4312992081
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4028777150
<b>[SF] Trillian test result (tid-15580)</b>
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 52325 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12086-t15580-kvm-ol8.zip
Smoke tests completed. 146 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
ContextSuite context=TestListIdsParams>:teardown | `Error` | 1.12 | test_list_ids_parameter.py
test_01_snapshot_root_disk | `Error` | 4.93 | test_snapshots.py
test_02_list_snapshots_with_removed_data_store | `Error` | 49.18 | test_snapshots.py
test_02_list_snapshots_with_removed_data_store | `Error` | 49.19 | test_snapshots.py
ContextSuite context=TestSnapshotStandaloneBackup>:teardown | `Error` | 27.51 | test_snapshots.py
test_01_snapshot_usage | `Error` | 28.92 | test_usage.py
test_01_vpn_usage | `Error` | 1.09 | test_usage.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "hsato03 (via GitHub)" <gi...@apache.org>.
hsato03 commented on code in PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#discussion_r3149059912
##########
engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java:
##########
@@ -346,20 +370,102 @@ protected Map<Long, Pair<Long, Long>> migrateAway(
executor.setCorePoolSize((int) (totalJobs));
}
- MigrateDataTask task = new MigrateDataTask(chosenFileForMigration, srcDatastore, dataStoreManager.getDataStore(destDatastoreId, DataStoreRole.Image));
- if (chosenFileForMigration instanceof SnapshotInfo ) {
+ DataStore destDataStore = dataStoreManager.getDataStore(destDatastoreId, DataStoreRole.Image);
+
+ boolean isKvmIncrementalSnapshot = chosenFileForMigration instanceof SnapshotInfo && ((SnapshotInfo) chosenFileForMigration).isKvmIncrementalSnapshot() && snapshotChains.containsKey(chosenFileForMigration);
+
+ if (isKvmIncrementalSnapshot) {
+ MigrateKvmIncrementalSnapshotTask task = new MigrateKvmIncrementalSnapshotTask(chosenFileForMigration, snapshotChains, srcDatastore, destDataStore, snapshotIdsToMigrate);
+ futures.add(submitKvmIncrementalMigration(srcDatastore.getScope().getScopeId(), task));
+ logger.debug("Incremental snapshot migration {} submitted to incremental pool.", chosenFileForMigration.getUuid());
+ } else {
+ createMigrateDataTask(chosenFileForMigration, snapshotChains, templateChains, srcDatastore, destDataStore, executor, futures);
+ }
+
+ return storageCapacities;
+ }
+
+ private AsyncCallFuture<DataObjectResult> migrateKvmIncrementalSnapshotChain(DataObject chosenFileForMigration, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains, DataStore srcDatastore, DataStore destDataStore, Set<Long> snapshotIdsToMigrate) {
+ return Transaction.execute((TransactionCallback<AsyncCallFuture<DataObjectResult>>) status -> {
+ MigrateBetweenSecondaryStoragesCommandAnswer answer = null;
+ AsyncCallFuture<DataObjectResult> future = new AsyncCallFuture<>();
+ DataObjectResult result = new DataObjectResult(chosenFileForMigration);
+
+ try {
+ List<SnapshotInfo> snapshotChain = snapshotChains.get(chosenFileForMigration).first();
+ MigrateSnapshotsBetweenSecondaryStoragesCommand migrateBetweenSecondaryStoragesCmd = new MigrateSnapshotsBetweenSecondaryStoragesCommand(snapshotChain.stream().map(DataObject::getTO).collect(Collectors.toList()), srcDatastore.getTO(), destDataStore.getTO(), snapshotIdsToMigrate);
+
+ HostVO host = getAvailableHost(((SnapshotInfo) chosenFileForMigration).getDataCenterId());
+ if (host == null) {
+ throw new CloudRuntimeException("No suitable hosts found to send migrate command.");
+ }
+
+ migrateBetweenSecondaryStoragesCmd.setWait(StorageManager.AgentMaxDataMigrationWaitTime.valueIn(host.getClusterId()));
+ answer = (MigrateBetweenSecondaryStoragesCommandAnswer) agentManager.send(host.getId(), migrateBetweenSecondaryStoragesCmd);
+ if (answer == null || !answer.getResult()) {
+ logger.warn("Unable to migrate snapshots [{}].", snapshotChain);
+ throw new CloudRuntimeException("Unable to migrate KVM incremental snapshots to another secondary storage");
+ }
+ } catch (final OperationTimedoutException | AgentUnavailableException e) {
+ throw new CloudRuntimeException("Error while migrating KVM incremental snapshot chain. Check the logs for more information.", e);
Review Comment:
Yes. https://github.com/apache/cloudstack/pull/12086/changes#diff-3165894c0b4560ccad18e0e756b025d6ab17148f095cb4422e03092952b6d0d2R108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3588536732
@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3562560198
@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3543711306
## [Codecov](https://app.codecov.io/gh/apache/cloudstack/pull/12086?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 3.58%. Comparing base ([`6dc259c`](https://app.codecov.io/gh/apache/cloudstack/commit/6dc259c7dab7bee0c4ff731271c61e2f2ee97532?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f185688`](https://app.codecov.io/gh/apache/cloudstack/commit/f1856883bf09fa9021a3327487c04e20892c70ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (6dc259c) and HEAD (f185688). Click for more details.
>
> <details><summary>HEAD has 1 upload less than BASE</summary>
>
>| Flag | BASE (6dc259c) | HEAD (f185688) |
>|------|------|------|
>|unittests|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #12086 +/- ##
=============================================
- Coverage 17.55% 3.58% -13.98%
=============================================
Files 5911 445 -5466
Lines 529359 37536 -491823
Branches 64655 6901 -57754
=============================================
- Hits 92949 1347 -91602
+ Misses 425952 36025 -389927
+ Partials 10458 164 -10294
```
| [Flag](https://app.codecov.io/gh/apache/cloudstack/pull/12086/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [uitests](https://app.codecov.io/gh/apache/cloudstack/pull/12086/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `3.58% <ø> (ø)` | |
| [unittests](https://app.codecov.io/gh/apache/cloudstack/pull/12086/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/cloudstack/pull/12086?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4336456613
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17632
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-5950721937
@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4011176050
<b>[LL] Trillian Build Failed (tid-7174)<b/>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "hsato03 (via GitHub)" <gi...@apache.org>.
hsato03 commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3960622385
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "hsato03 (via GitHub)" <gi...@apache.org>.
hsato03 commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4031852651
> > @JoaoJandre The errors are unrelated to the changes in this PR and will be fixed by #12597.
>
> @hsato03 #12597 is dealing with Xen, do you mean to say they solve generic issues?
@DaanHoogland The PR #12597 is also addressing a regression introduced in commit d700e2db643c6d1adbda82957f21a6a8b696f704, which makes the snapshot deletion impossible and is causing these test failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "hsato03 (via GitHub)" <gi...@apache.org>.
hsato03 commented on code in PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#discussion_r2884297594
##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtMigrateSnapshotsBetweenSecondaryStoragesCommandWrapper.java:
##########
@@ -0,0 +1,190 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package com.cloud.hypervisor.kvm.resource.wrapper;
+
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.MigrateSnapshotsBetweenSecondaryStoragesCommand;
+import com.cloud.agent.api.MigrateBetweenSecondaryStoragesCommandAnswer;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.agent.api.to.DataTO;
+import com.cloud.hypervisor.kvm.resource.LibvirtComputingResource;
+import com.cloud.hypervisor.kvm.storage.KVMStoragePool;
+import com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager;
+import com.cloud.resource.CommandWrapper;
+import com.cloud.resource.ResourceWrapper;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.cloudstack.utils.qemu.QemuImageOptions;
+import org.apache.cloudstack.utils.qemu.QemuImg;
+import org.apache.cloudstack.utils.qemu.QemuImgException;
+import org.apache.cloudstack.utils.qemu.QemuImgFile;
+import org.libvirt.LibvirtException;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@ResourceWrapper(handles = MigrateSnapshotsBetweenSecondaryStoragesCommand.class)
+public class LibvirtMigrateSnapshotsBetweenSecondaryStoragesCommandWrapper extends CommandWrapper<MigrateSnapshotsBetweenSecondaryStoragesCommand, Answer, LibvirtComputingResource> {
+
+ protected Set<String> filesToRemove;
+ protected List<Pair<Long, String>> resourcesToUpdate;
+ protected String resourceType;
+ protected int wait;
+
+ @Override
+ public Answer execute(MigrateSnapshotsBetweenSecondaryStoragesCommand command, LibvirtComputingResource serverResource) {
+ filesToRemove = new HashSet<>();
+ resourcesToUpdate = new ArrayList<>();
+ wait = command.getWait() * 1000;
+
+ DataStoreTO srcDataStore = command.getSrcDataStore();
+ DataStoreTO destDataStore = command.getDestDataStore();
+ KVMStoragePoolManager storagePoolManager = serverResource.getStoragePoolMgr();
+
+ Set<KVMStoragePool> imagePools = new HashSet<>();
+ KVMStoragePool destImagePool = storagePoolManager.getStoragePoolByURI(destDataStore.getUrl());
+ imagePools.add(destImagePool);
+
+ String imagePoolUrl;
+ KVMStoragePool imagePool = null;
+
+ String parentSnapshotPath = null;
+ boolean parentSnapshotWasMigrated = false;
+ Set<Long> snapshotsIdToMigrate = command.getSnapshotsIdToMigrate();
+
+ DataTO currentSnapshot = null;
+
+ try {
+ for (DataTO snapshot : command.getSnapshotChain()) {
+ currentSnapshot = snapshot;
+ imagePoolUrl = snapshot.getDataStore().getUrl();
+ imagePool = storagePoolManager.getStoragePoolByURI(imagePoolUrl);
+ imagePools.add(imagePool);
+
+ String resourceCurrentPath = imagePool.getLocalPathFor(snapshot.getPath());
+
+ if (imagePoolUrl.equals(srcDataStore.getUrl()) && snapshotsIdToMigrate.contains(snapshot.getId())) {
+ logger.debug("Migrating snapshot [{}] to destination storage pool [{}].", snapshot.getId(), destImagePool.getUuid());
+ parentSnapshotPath = copyResourceToDestDataStore(snapshot, resourceCurrentPath, destImagePool, parentSnapshotPath);
+ parentSnapshotWasMigrated = true;
+ logger.debug("Snapshot [{}] migrated successfully. New parent path: [{}]", snapshot.getId(), parentSnapshotPath);
+ } else {
+ if (parentSnapshotWasMigrated) {
+ logger.debug("Rebasing snapshot [{}] to new parent path: [{}]", snapshot.getId(), parentSnapshotPath);
+ parentSnapshotPath = rebaseResourceToNewParentPath(resourceCurrentPath, parentSnapshotPath);
+ } else {
+ logger.debug("Skipping migration/rebase for snapshot [{}]. Keeping current path.", snapshot.getId());
+ parentSnapshotPath = resourceCurrentPath;
+ }
+ parentSnapshotWasMigrated = false;
+ }
+ }
+ logger.info("Successfully completed migrating snapshot chain [{}].", command.getSnapshotChain());
+ } catch (LibvirtException | QemuImgException e) {
+ logger.error("Exception while migrating snapshots [{}] to secondary storage [{}] due to: [{}].", command.getSnapshotChain(), imagePool, e.getMessage(), e);
+ return new MigrateBetweenSecondaryStoragesCommandAnswer(command, false, String.format("Unable to migrate snapshot with ID [%s].", currentSnapshot.getId()), resourcesToUpdate);
Review Comment:
Considering the workflow executed on the Management Server, it is not possible for the snapshot chain to be empty; therefore, the variable `currentSnapshot` cannot be null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "hsato03 (via GitHub)" <gi...@apache.org>.
hsato03 commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4329101013
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3587364633
@hsato03 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4010244351
@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "hsato03 (via GitHub)" <gi...@apache.org>.
hsato03 commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3953438454
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "hsato03 (via GitHub)" <gi...@apache.org>.
hsato03 commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3769843082
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "JoaoJandre (via GitHub)" <gi...@apache.org>.
JoaoJandre commented on code in PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#discussion_r3154004852
##########
engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/DataMigrationUtility.java:
##########
@@ -297,16 +311,30 @@ protected List<DataObject> getAllReadySnapshotsAndChains(DataStore srcDataStore,
logger.debug("Not migrating snapshot [{}] because its hypervisor type is simulator.", snapshot);
continue;
}
- if (snapshot.getParentSnapshotId() != 0) {
+ boolean isKvmIncrementalSnapshot = snapshot.getKvmCheckpointPath() != null;
+ if (snapshot.getParentSnapshotId() != 0 && !isKvmIncrementalSnapshot) {
continue; // The child snapshot will be migrated in the for loop below.
}
- SnapshotInfo snap = snapshotFactory.getSnapshot(snapshotVO.getSnapshotId(), snapshot.getDataStoreId(), snapshot.getRole());
- if (snap == null) {
- logger.debug("Not migrating snapshot [{}] because we could not get its information.", snapshot);
+ if (snapshotIdsAlreadyInChain.contains(snapshotVO.getId())) {
+ logger.debug("Skipping snapshot {} since it is already in snapshot chain.", snapshot);
continue;
}
- files.add(snap);
- idsForMigration.add(snapshotId);
+
+ if (isKvmIncrementalSnapshot) {
+ List<SnapshotInfo> kvmIncrementalSnapshotChain = createKvmIncrementalSnapshotChain(snapshot);
+ SnapshotInfo parent = kvmIncrementalSnapshotChain.get(0);
+ snapshotIdsAlreadyInChain.addAll(kvmIncrementalSnapshotChain.stream().map(DataObject::getId).collect(Collectors.toSet()));
+ snapshotChains.put(parent, new Pair<>(kvmIncrementalSnapshotChain, getTotalChainSize(kvmIncrementalSnapshotChain.stream().filter(snapInfo -> snapshotIdsToMigrate.contains(snapInfo.getId())).collect(Collectors.toList()))));
+ files.add(parent);
+ } else {
Review Comment:
I reread the code and we actually add the first snapshot in the chain. It may or may not be the parent, it may be the snapshot itself, if the chain only contains one snapshot.
I would rename the variable to `firstSnapshotInTheChain` or something like that, just for clarity. Functionally its the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4335875864
@hsato03 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on code in PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#discussion_r3170373731
##########
engine/components-api/src/main/java/com/cloud/storage/StorageManager.java:
##########
@@ -233,6 +233,9 @@ public interface StorageManager extends StorageService {
"while adding a new Secondary Storage. If the copy operation fails, the system falls back to downloading the template from the source URL.",
true, ConfigKey.Scope.Zone, null);
+ ConfigKey<Integer> AgentMaxDataMigrationWaitTime = new ConfigKey<>("Advanced", Integer.class, "agent.max.data.migration.wait.time", "3600",
Review Comment:
It would be nice to have a single configuration for secondary storage template, snapshot and volume migration. Templates, volumes and non-incremental snapshots will use `secstorage.cmd.execution.time.max`, while incremental snapshots will use `agent.max.data.migration.wait.time` with this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4329448408
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17620
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3960633322
@hsato03 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4022511146
<b>[LL] Trillian Build Failed (tid-7175)<b/>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "hsato03 (via GitHub)" <gi...@apache.org>.
hsato03 commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4335855818
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4312996319
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3681486711
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3769849448
@hsato03 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "hsato03 (via GitHub)" <gi...@apache.org>.
hsato03 commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-5936137999
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-5950672350
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3953740509
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16929
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "hsato03 (via GitHub)" <gi...@apache.org>.
hsato03 commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3958882653
> @hsato03 , Do you want thios in 23 or an earlier release? (seems like 22.1 would make sense to me)
@DaanHoogland I agree. Since it is a bug fix, I think it could be included in 4.22.1. I will rebase and change the target branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3770032695
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16432
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "hsato03 (via GitHub)" <gi...@apache.org>.
hsato03 commented on code in PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#discussion_r2884086981
##########
engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java:
##########
@@ -352,26 +374,111 @@ protected Pair<String, Boolean> migrateCompleted(Long destDatastoreId, DataStore
protected Map<Long, Pair<Long, Long>> migrateAway(
DataObject chosenFileForMigration,
Map<Long, Pair<Long, Long>> storageCapacities,
- Map<DataObject,
- Pair<List<SnapshotInfo>, Long>> snapshotChains,
+ Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains,
Map<DataObject, Pair<List<TemplateInfo>, Long>> templateChains,
+ Set<Long> snapshotIdsToMigrate,
DataStore srcDatastore,
Long destDatastoreId,
List<Future<DataObjectResult>> futures) {
Long fileSize = migrationHelper.getFileSize(chosenFileForMigration, snapshotChains, templateChains);
storageCapacities = assumeMigrate(storageCapacities, srcDatastore.getId(), destDatastoreId, fileSize);
- MigrateDataTask task = new MigrateDataTask(chosenFileForMigration, srcDatastore, dataStoreManager.getDataStore(destDatastoreId, DataStoreRole.Image));
- if (chosenFileForMigration instanceof SnapshotInfo ) {
+ DataStore destDataStore = dataStoreManager.getDataStore(destDatastoreId, DataStoreRole.Image);
+
+ boolean isKvmIncrementalSnapshot = chosenFileForMigration instanceof SnapshotInfo && ((SnapshotInfo) chosenFileForMigration).isKvmIncrementalSnapshot() && snapshotChains.containsKey(chosenFileForMigration);
+
+ if (isKvmIncrementalSnapshot) {
+ MigrateKvmIncrementalSnapshotTask task = new MigrateKvmIncrementalSnapshotTask(chosenFileForMigration, snapshotChains, srcDatastore, destDataStore, snapshotIdsToMigrate);
+ futures.add(submitKvmIncrementalMigration(srcDatastore.getScope().getScopeId(), task));
+ logger.debug("Incremental snapshot migration {} submitted to incremental pool.", chosenFileForMigration.getUuid());
+ } else {
+ createMigrateDataTask(chosenFileForMigration, snapshotChains, templateChains, srcDatastore, destDataStore, futures);
+ }
+
+ return storageCapacities;
+ }
+
+ private AsyncCallFuture<DataObjectResult> migrateKvmIncrementalSnapshotChain(DataObject chosenFileForMigration, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains, DataStore srcDatastore, DataStore destDataStore, Set<Long> snapshotIdsToMigrate) {
+ return Transaction.execute((TransactionCallback<AsyncCallFuture<DataObjectResult>>) status -> {
+ MigrateBetweenSecondaryStoragesCommandAnswer answer = null;
+ AsyncCallFuture<DataObjectResult> future = new AsyncCallFuture<>();
+ DataObjectResult result = new DataObjectResult(chosenFileForMigration);
+
+ try {
+ List<SnapshotInfo> snapshotChain = snapshotChains.get(chosenFileForMigration).first();
+ MigrateSnapshotsBetweenSecondaryStoragesCommand migrateBetweenSecondaryStoragesCmd = new MigrateSnapshotsBetweenSecondaryStoragesCommand(snapshotChain.stream().map(DataObject::getTO).collect(Collectors.toList()), srcDatastore.getTO(), destDataStore.getTO(), snapshotIdsToMigrate);
+
+ HostVO host = getAvailableHost(((SnapshotInfo) chosenFileForMigration).getDataCenterId());
+ if (host == null) {
+ throw new CloudRuntimeException("No suitable hosts found to send migrate command.");
+ }
+
+ migrateBetweenSecondaryStoragesCmd.setWait(StorageManager.AgentMaxDataMigrationWaitTime.valueIn(host.getClusterId()));
+ answer = (MigrateBetweenSecondaryStoragesCommandAnswer) agentManager.send(host.getId(), migrateBetweenSecondaryStoragesCmd);
+ if (answer == null || !answer.getResult()) {
+ if (answer != null) {
+ logger.warn(answer.getDetails());
+ }
+ throw new CloudRuntimeException("Unable to migrate KVM incremental snapshots to another secondary storage.");
+ }
+ } catch (final OperationTimedoutException | AgentUnavailableException e) {
+ throw new CloudRuntimeException("Error while migrating KVM incremental snapshot chain. Check the logs for more information.", e);
+ } finally {
+ if (answer != null) {
+ updateSnapshotsReference(destDataStore, answer);
+ }
+ }
+ result.setSuccess(true);
+ future.complete(result);
+ return future;
+ });
+ }
+
+ private void updateSnapshotsReference(DataStore destDataStore, MigrateBetweenSecondaryStoragesCommandAnswer answer) {
+ for (Pair<Long, String> snapshotIdAndUpdatedCheckpointPath : answer.getMigratedResources()) {
+ Long snapshotId = snapshotIdAndUpdatedCheckpointPath.first();
+ String newCheckpointPath = snapshotIdAndUpdatedCheckpointPath.second();
+
+ SnapshotDataStoreVO snapshotDataStore = snapshotDataStoreDao.findOneBySnapshotAndDatastoreRole(snapshotId, DataStoreRole.Image);
+
+ if (snapshotDataStore == null) {
+ logger.warn("Snapshot [{}] not found.", snapshotId);
+ continue;
+ }
+
+ snapshotDataStore.setDataStoreId(destDataStore.getId());
+ snapshotDataStore.setKvmCheckpointPath(newCheckpointPath);
+ snapshotDataStoreDao.update(snapshotDataStore.getId(), snapshotDataStore);
+ }
+ }
+
+ protected <T> Future<T> submitKvmIncrementalMigration(Long zoneId, Callable<T> task) {
+ if (!zoneKvmIncrementalExecutorMap.containsKey(zoneId)) {
+ zoneKvmIncrementalExecutorMap.put(zoneId, Executors.newSingleThreadExecutor());
+ }
+ return zoneKvmIncrementalExecutorMap.get(zoneId).submit(task);
+ }
+
+ private HostVO getAvailableHost(long zoneId) throws AgentUnavailableException, OperationTimedoutException {
+ List<HostVO> hosts = hostDao.listByDataCenterIdAndHypervisorType(zoneId, Hypervisor.HypervisorType.KVM);
+ if (CollectionUtils.isNotEmpty(hosts)) {
+ return hosts.get(new Random().nextInt(hosts.size()));
Review Comment:
The method `hostDao.listByDataCenterIdAndHypervisorType` already performs this filtering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "hsato03 (via GitHub)" <gi...@apache.org>.
hsato03 commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4030897916
@JoaoJandre The errors are unrelated to the changes in this PR and will be fixed by #12597.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "JoaoJandre (via GitHub)" <gi...@apache.org>.
JoaoJandre commented on code in PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#discussion_r3139567572
##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtMigrateSnapshotsBetweenSecondaryStoragesCommandWrapper.java:
##########
@@ -0,0 +1,193 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package com.cloud.hypervisor.kvm.resource.wrapper;
+
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.MigrateSnapshotsBetweenSecondaryStoragesCommand;
+import com.cloud.agent.api.MigrateBetweenSecondaryStoragesCommandAnswer;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.agent.api.to.DataTO;
+import com.cloud.hypervisor.kvm.resource.LibvirtComputingResource;
+import com.cloud.hypervisor.kvm.storage.KVMStoragePool;
+import com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager;
+import com.cloud.resource.CommandWrapper;
+import com.cloud.resource.ResourceWrapper;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.cloudstack.utils.qemu.QemuImageOptions;
+import org.apache.cloudstack.utils.qemu.QemuImg;
+import org.apache.cloudstack.utils.qemu.QemuImgException;
+import org.apache.cloudstack.utils.qemu.QemuImgFile;
+import org.libvirt.LibvirtException;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@ResourceWrapper(handles = MigrateSnapshotsBetweenSecondaryStoragesCommand.class)
+public class LibvirtMigrateSnapshotsBetweenSecondaryStoragesCommandWrapper extends CommandWrapper<MigrateSnapshotsBetweenSecondaryStoragesCommand, Answer, LibvirtComputingResource> {
+
+ protected Set<String> filesToRemove;
+ protected List<Pair<Long, String>> resourcesToUpdate;
+ protected int wait;
+
+ @Override
+ public Answer execute(MigrateSnapshotsBetweenSecondaryStoragesCommand command, LibvirtComputingResource serverResource) {
+ filesToRemove = new HashSet<>();
+ resourcesToUpdate = new ArrayList<>();
+ wait = command.getWait() * 1000;
+
+ DataStoreTO srcDataStore = command.getSrcDataStore();
+ DataStoreTO destDataStore = command.getDestDataStore();
+ KVMStoragePoolManager storagePoolManager = serverResource.getStoragePoolMgr();
+
+ Set<KVMStoragePool> imagePools = new HashSet<>();
+ KVMStoragePool destImagePool = storagePoolManager.getStoragePoolByURI(destDataStore.getUrl());
+ imagePools.add(destImagePool);
+
+ String imagePoolUrl;
+ KVMStoragePool imagePool = null;
+
+ String parentSnapshotPath = null;
+ boolean parentSnapshotWasMigrated = false;
+ Set<Long> snapshotIdsToMigrate = command.getSnapshotIdsToMigrate();
+
+ DataTO currentSnapshot = null;
+
+ try {
+ for (DataTO snapshot : command.getSnapshotChain()) {
+ currentSnapshot = snapshot;
+ imagePoolUrl = snapshot.getDataStore().getUrl();
+ imagePool = storagePoolManager.getStoragePoolByURI(imagePoolUrl);
+ imagePools.add(imagePool);
+
+ String resourceCurrentPath = imagePool.getLocalPathFor(snapshot.getPath());
+
+ if (imagePoolUrl.equals(srcDataStore.getUrl()) && snapshotIdsToMigrate.contains(snapshot.getId())) {
+ logger.debug("Migrating snapshot [{}] to destination storage pool [{}].", snapshot.getId(), destImagePool.getUuid());
+ parentSnapshotPath = copyResourceToDestDataStore(snapshot, resourceCurrentPath, destImagePool, parentSnapshotPath);
+ parentSnapshotWasMigrated = !parentSnapshotPath.equals(resourceCurrentPath);
+ logger.debug("Snapshot [{}] migrated successfully. New parent path: [{}].", snapshot.getId(), parentSnapshotPath);
Review Comment:
```suggestion
logger.debug("Snapshot [{}] migrated successfully. New path: [{}].", snapshot.getId(), parentSnapshotPath);
```
This is the path of the current snapshot that was migrated, correct?
##########
engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java:
##########
@@ -346,20 +370,102 @@ protected Map<Long, Pair<Long, Long>> migrateAway(
executor.setCorePoolSize((int) (totalJobs));
}
- MigrateDataTask task = new MigrateDataTask(chosenFileForMigration, srcDatastore, dataStoreManager.getDataStore(destDatastoreId, DataStoreRole.Image));
- if (chosenFileForMigration instanceof SnapshotInfo ) {
+ DataStore destDataStore = dataStoreManager.getDataStore(destDatastoreId, DataStoreRole.Image);
+
+ boolean isKvmIncrementalSnapshot = chosenFileForMigration instanceof SnapshotInfo && ((SnapshotInfo) chosenFileForMigration).isKvmIncrementalSnapshot() && snapshotChains.containsKey(chosenFileForMigration);
+
+ if (isKvmIncrementalSnapshot) {
+ MigrateKvmIncrementalSnapshotTask task = new MigrateKvmIncrementalSnapshotTask(chosenFileForMigration, snapshotChains, srcDatastore, destDataStore, snapshotIdsToMigrate);
+ futures.add(submitKvmIncrementalMigration(srcDatastore.getScope().getScopeId(), task));
+ logger.debug("Incremental snapshot migration {} submitted to incremental pool.", chosenFileForMigration.getUuid());
+ } else {
+ createMigrateDataTask(chosenFileForMigration, snapshotChains, templateChains, srcDatastore, destDataStore, executor, futures);
+ }
+
+ return storageCapacities;
+ }
+
+ private AsyncCallFuture<DataObjectResult> migrateKvmIncrementalSnapshotChain(DataObject chosenFileForMigration, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains, DataStore srcDatastore, DataStore destDataStore, Set<Long> snapshotIdsToMigrate) {
+ return Transaction.execute((TransactionCallback<AsyncCallFuture<DataObjectResult>>) status -> {
+ MigrateBetweenSecondaryStoragesCommandAnswer answer = null;
+ AsyncCallFuture<DataObjectResult> future = new AsyncCallFuture<>();
+ DataObjectResult result = new DataObjectResult(chosenFileForMigration);
+
+ try {
+ List<SnapshotInfo> snapshotChain = snapshotChains.get(chosenFileForMigration).first();
+ MigrateSnapshotsBetweenSecondaryStoragesCommand migrateBetweenSecondaryStoragesCmd = new MigrateSnapshotsBetweenSecondaryStoragesCommand(snapshotChain.stream().map(DataObject::getTO).collect(Collectors.toList()), srcDatastore.getTO(), destDataStore.getTO(), snapshotIdsToMigrate);
+
+ HostVO host = getAvailableHost(((SnapshotInfo) chosenFileForMigration).getDataCenterId());
+ if (host == null) {
+ throw new CloudRuntimeException("No suitable hosts found to send migrate command.");
+ }
+
+ migrateBetweenSecondaryStoragesCmd.setWait(StorageManager.AgentMaxDataMigrationWaitTime.valueIn(host.getClusterId()));
+ answer = (MigrateBetweenSecondaryStoragesCommandAnswer) agentManager.send(host.getId(), migrateBetweenSecondaryStoragesCmd);
+ if (answer == null || !answer.getResult()) {
+ logger.warn("Unable to migrate snapshots [{}].", snapshotChain);
+ throw new CloudRuntimeException("Unable to migrate KVM incremental snapshots to another secondary storage");
+ }
+ } catch (final OperationTimedoutException | AgentUnavailableException e) {
+ throw new CloudRuntimeException("Error while migrating KVM incremental snapshot chain. Check the logs for more information.", e);
Review Comment:
Was this addressed?
##########
engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/DataMigrationUtility.java:
##########
@@ -297,16 +311,30 @@ protected List<DataObject> getAllReadySnapshotsAndChains(DataStore srcDataStore,
logger.debug("Not migrating snapshot [{}] because its hypervisor type is simulator.", snapshot);
continue;
}
- if (snapshot.getParentSnapshotId() != 0) {
+ boolean isKvmIncrementalSnapshot = snapshot.getKvmCheckpointPath() != null;
+ if (snapshot.getParentSnapshotId() != 0 && !isKvmIncrementalSnapshot) {
continue; // The child snapshot will be migrated in the for loop below.
}
- SnapshotInfo snap = snapshotFactory.getSnapshot(snapshotVO.getSnapshotId(), snapshot.getDataStoreId(), snapshot.getRole());
- if (snap == null) {
- logger.debug("Not migrating snapshot [{}] because we could not get its information.", snapshot);
+ if (snapshotIdsAlreadyInChain.contains(snapshotVO.getId())) {
+ logger.debug("Skipping snapshot {} since it is already in snapshot chain.", snapshot);
continue;
}
- files.add(snap);
- idsForMigration.add(snapshotId);
+
+ if (isKvmIncrementalSnapshot) {
+ List<SnapshotInfo> kvmIncrementalSnapshotChain = createKvmIncrementalSnapshotChain(snapshot);
+ SnapshotInfo parent = kvmIncrementalSnapshotChain.get(0);
+ snapshotIdsAlreadyInChain.addAll(kvmIncrementalSnapshotChain.stream().map(DataObject::getId).collect(Collectors.toSet()));
+ snapshotChains.put(parent, new Pair<>(kvmIncrementalSnapshotChain, getTotalChainSize(kvmIncrementalSnapshotChain.stream().filter(snapInfo -> snapshotIdsToMigrate.contains(snapInfo.getId())).collect(Collectors.toList()))));
+ files.add(parent);
+ } else {
Review Comment:
Why do we add the parent snapshot and not the snapshot itself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4685328620
Ping @hsato03, could you have a look at the pending comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "JoaoJandre (via GitHub)" <gi...@apache.org>.
JoaoJandre commented on code in PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#discussion_r2571547124
##########
engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java:
##########
@@ -346,20 +370,102 @@ protected Map<Long, Pair<Long, Long>> migrateAway(
executor.setCorePoolSize((int) (totalJobs));
}
- MigrateDataTask task = new MigrateDataTask(chosenFileForMigration, srcDatastore, dataStoreManager.getDataStore(destDatastoreId, DataStoreRole.Image));
- if (chosenFileForMigration instanceof SnapshotInfo ) {
+ DataStore destDataStore = dataStoreManager.getDataStore(destDatastoreId, DataStoreRole.Image);
+
+ boolean isKvmIncrementalSnapshot = chosenFileForMigration instanceof SnapshotInfo && ((SnapshotInfo) chosenFileForMigration).isKvmIncrementalSnapshot() && snapshotChains.containsKey(chosenFileForMigration);
+
+ if (isKvmIncrementalSnapshot) {
+ MigrateKvmIncrementalSnapshotTask task = new MigrateKvmIncrementalSnapshotTask(chosenFileForMigration, snapshotChains, srcDatastore, destDataStore, snapshotIdsToMigrate);
+ futures.add(submitKvmIncrementalMigration(srcDatastore.getScope().getScopeId(), task));
+ logger.debug("Incremental snapshot migration {} submitted to incremental pool.", chosenFileForMigration.getUuid());
+ } else {
+ createMigrateDataTask(chosenFileForMigration, snapshotChains, templateChains, srcDatastore, destDataStore, executor, futures);
+ }
+
+ return storageCapacities;
+ }
+
+ private AsyncCallFuture<DataObjectResult> migrateKvmIncrementalSnapshotChain(DataObject chosenFileForMigration, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains, DataStore srcDatastore, DataStore destDataStore, Set<Long> snapshotIdsToMigrate) {
+ return Transaction.execute((TransactionCallback<AsyncCallFuture<DataObjectResult>>) status -> {
+ MigrateBetweenSecondaryStoragesCommandAnswer answer = null;
+ AsyncCallFuture<DataObjectResult> future = new AsyncCallFuture<>();
+ DataObjectResult result = new DataObjectResult(chosenFileForMigration);
+
+ try {
+ List<SnapshotInfo> snapshotChain = snapshotChains.get(chosenFileForMigration).first();
+ MigrateSnapshotsBetweenSecondaryStoragesCommand migrateBetweenSecondaryStoragesCmd = new MigrateSnapshotsBetweenSecondaryStoragesCommand(snapshotChain.stream().map(DataObject::getTO).collect(Collectors.toList()), srcDatastore.getTO(), destDataStore.getTO(), snapshotIdsToMigrate);
+
+ HostVO host = getAvailableHost(((SnapshotInfo) chosenFileForMigration).getDataCenterId());
+ if (host == null) {
+ throw new CloudRuntimeException("No suitable hosts found to send migrate command.");
+ }
+
+ migrateBetweenSecondaryStoragesCmd.setWait(StorageManager.AgentMaxDataMigrationWaitTime.valueIn(host.getClusterId()));
+ answer = (MigrateBetweenSecondaryStoragesCommandAnswer) agentManager.send(host.getId(), migrateBetweenSecondaryStoragesCmd);
+ if (answer == null || !answer.getResult()) {
+ logger.warn("Unable to migrate snapshots [{}].", snapshotChain);
+ throw new CloudRuntimeException("Unable to migrate KVM incremental snapshots to another secondary storage");
+ }
+ } catch (final OperationTimedoutException | AgentUnavailableException e) {
+ throw new CloudRuntimeException("Error while migrating KVM incremental snapshot chain. Check the logs for more information.", e);
Review Comment:
We could inform which snapshot failed.
##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtMigrateSnapshotsBetweenSecondaryStoragesCommandWrapper.java:
##########
@@ -0,0 +1,182 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package com.cloud.hypervisor.kvm.resource.wrapper;
+
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.MigrateSnapshotsBetweenSecondaryStoragesCommand;
+import com.cloud.agent.api.MigrateBetweenSecondaryStoragesCommandAnswer;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.agent.api.to.DataTO;
+import com.cloud.hypervisor.kvm.resource.LibvirtComputingResource;
+import com.cloud.hypervisor.kvm.storage.KVMStoragePool;
+import com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager;
+import com.cloud.resource.CommandWrapper;
+import com.cloud.resource.ResourceWrapper;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.cloudstack.utils.qemu.QemuImageOptions;
+import org.apache.cloudstack.utils.qemu.QemuImg;
+import org.apache.cloudstack.utils.qemu.QemuImgException;
+import org.apache.cloudstack.utils.qemu.QemuImgFile;
+import org.libvirt.LibvirtException;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@ResourceWrapper(handles = MigrateSnapshotsBetweenSecondaryStoragesCommand.class)
+public class LibvirtMigrateSnapshotsBetweenSecondaryStoragesCommandWrapper extends CommandWrapper<MigrateSnapshotsBetweenSecondaryStoragesCommand, Answer, LibvirtComputingResource> {
+
+ protected Set<String> filesToRemove;
+ protected List<Pair<Long, String>> resourcesToUpdate;
+ protected String resourceType;
+ protected int wait;
+
+ @Override
+ public Answer execute(MigrateSnapshotsBetweenSecondaryStoragesCommand command, LibvirtComputingResource serverResource) {
+ filesToRemove = new HashSet<>();
+ resourcesToUpdate = new ArrayList<>();
+ wait = command.getWait() * 1000;
+
+ DataStoreTO srcDataStore = command.getSrcDataStore();
+ DataStoreTO destDataStore = command.getDestDataStore();
+ KVMStoragePoolManager storagePoolManager = serverResource.getStoragePoolMgr();
+
+ Set<KVMStoragePool> imagePools = new HashSet<>();
+ KVMStoragePool destImagePool = storagePoolManager.getStoragePoolByURI(destDataStore.getUrl());
+ imagePools.add(destImagePool);
+
+ String imagePoolUrl;
+ KVMStoragePool imagePool = null;
+
+ String parentSnapshotPath = null;
+ boolean parentSnapshotWasMigrated = false;
+ Set<Long> snapshotsIdToMigrate = command.getSnapshotsIdToMigrate();
+
+ try {
+ for (DataTO snapshot : command.getSnapshotChain()) {
+ imagePoolUrl = snapshot.getDataStore().getUrl();
+ imagePool = storagePoolManager.getStoragePoolByURI(imagePoolUrl);
+ imagePools.add(imagePool);
+
+ String resourceCurrentPath = imagePool.getLocalPathFor(snapshot.getPath());
+
+ if (imagePoolUrl.equals(srcDataStore.getUrl()) && snapshotsIdToMigrate.contains(snapshot.getId())) {
+ parentSnapshotPath = copyResourceToDestDataStore(snapshot, resourceCurrentPath, destImagePool, parentSnapshotPath);
+ parentSnapshotWasMigrated = true;
+ } else {
+ if (parentSnapshotWasMigrated) {
+ parentSnapshotPath = rebaseResourceToNewParentPath(resourceCurrentPath, parentSnapshotPath);
+ } else {
+ parentSnapshotPath = resourceCurrentPath;
+ }
+ parentSnapshotWasMigrated = false;
Review Comment:
we could add some logging for these decisions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "hsato03 (via GitHub)" <gi...@apache.org>.
hsato03 commented on code in PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#discussion_r4160805647
##########
engine/components-api/src/main/java/com/cloud/storage/StorageManager.java:
##########
@@ -233,6 +233,9 @@ public interface StorageManager extends StorageService {
"while adding a new Secondary Storage. If the copy operation fails, the system falls back to downloading the template from the source URL.",
true, ConfigKey.Scope.Zone, null);
+ ConfigKey<Integer> AgentMaxDataMigrationWaitTime = new ConfigKey<>("Advanced", Integer.class, "agent.max.data.migration.wait.time", "3600",
Review Comment:
Since secondary storage migration uses a generic copy function shared with other features, it is better to handle this in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-5964210143
<b>[SF] Trillian test result (tid-17066)</b>
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 49867 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12086-t17066-kvm-ol8.zip
Smoke tests completed. 148 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
ContextSuite context=TestClusterDRS>:setup | `Error` | 0.00 | test_cluster_drs.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3578024375
<b>[SF] Trillian test result (tid-14864)</b>
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 50816 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12086-t14864-kvm-ol8.zip
Smoke tests completed. 149 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers | `Failure` | 471.99 | test_vpc_redundant.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "hsato03 (via GitHub)" <gi...@apache.org>.
hsato03 commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3587363411
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3543734364
@hsato03 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3565345591
<b>[SF] Trillian test result (tid-14849)</b>
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 51711 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12086-t14849-kvm-ol8.zip
Smoke tests completed. 149 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_01_redundant_vpc_site2site_vpn | `Failure` | 381.24 | test_vpc_vpn.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4347801700
<b>[SF] Trillian test result (tid-15977)</b>
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 50334 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12086-t15977-kvm-ol8.zip
Smoke tests completed. 148 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_isolate_network_password_server | `Failure` | 47.16 | test_password_server.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on code in PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#discussion_r3170373731
##########
engine/components-api/src/main/java/com/cloud/storage/StorageManager.java:
##########
@@ -233,6 +233,9 @@ public interface StorageManager extends StorageService {
"while adding a new Secondary Storage. If the copy operation fails, the system falls back to downloading the template from the source URL.",
true, ConfigKey.Scope.Zone, null);
+ ConfigKey<Integer> AgentMaxDataMigrationWaitTime = new ConfigKey<>("Advanced", Integer.class, "agent.max.data.migration.wait.time", "3600",
Review Comment:
It would be nice to open an issue to use this timeout instead of `secstorage.cmd.execution.time.max` for template and volume migration as well in the future
##########
engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java:
##########
@@ -421,6 +526,28 @@ protected synchronized void tryCleaningUpExecutor(Long zoneId) {
executor.shutdown();
}
+ protected void tryCleaningUpKvmIncrementalExecutor(Long zoneId) {
Review Comment:
We may need to synchronize this with `submitKvmIncrementalMigration` in a logic similar to `zoneExecutorMap`/`zonePendingWorkCountMap` in order to avoid the MS attempting to submit new tasks after an executon has been shutdown.
##########
engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java:
##########
@@ -128,6 +146,7 @@ public class StorageOrchestrator extends ManagerBase implements StorageOrchestra
private final Map<Long, ThreadPoolExecutor> zoneExecutorMap = new HashMap<>();
private final Map<Long, Integer> zonePendingWorkCountMap = new HashMap<>();
+ private final Map<Long, ThreadPoolExecutor> zoneKvmIncrementalResourcesExecutorMap = new ConcurrentHashMap<>();
Review Comment:
Could you rename these 3 variables in order to make it clearer that the first 2 ones are for SSVM commands, and the last for Agent commands?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4031710985
> @JoaoJandre The errors are unrelated to the changes in this PR and will be fixed by #12597.
@hsato03 #12597 is dealing with Xen, do you mean to say they solve generic issues?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3961033791
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16947
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3953447104
@hsato03 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4008670867
@hsato03 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "hsato03 (via GitHub)" <gi...@apache.org>.
hsato03 commented on code in PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#discussion_r3149035875
##########
engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/DataMigrationUtility.java:
##########
@@ -297,16 +311,30 @@ protected List<DataObject> getAllReadySnapshotsAndChains(DataStore srcDataStore,
logger.debug("Not migrating snapshot [{}] because its hypervisor type is simulator.", snapshot);
continue;
}
- if (snapshot.getParentSnapshotId() != 0) {
+ boolean isKvmIncrementalSnapshot = snapshot.getKvmCheckpointPath() != null;
+ if (snapshot.getParentSnapshotId() != 0 && !isKvmIncrementalSnapshot) {
continue; // The child snapshot will be migrated in the for loop below.
}
- SnapshotInfo snap = snapshotFactory.getSnapshot(snapshotVO.getSnapshotId(), snapshot.getDataStoreId(), snapshot.getRole());
- if (snap == null) {
- logger.debug("Not migrating snapshot [{}] because we could not get its information.", snapshot);
+ if (snapshotIdsAlreadyInChain.contains(snapshotVO.getId())) {
+ logger.debug("Skipping snapshot {} since it is already in snapshot chain.", snapshot);
continue;
}
- files.add(snap);
- idsForMigration.add(snapshotId);
+
+ if (isKvmIncrementalSnapshot) {
+ List<SnapshotInfo> kvmIncrementalSnapshotChain = createKvmIncrementalSnapshotChain(snapshot);
+ SnapshotInfo parent = kvmIncrementalSnapshotChain.get(0);
+ snapshotIdsAlreadyInChain.addAll(kvmIncrementalSnapshotChain.stream().map(DataObject::getId).collect(Collectors.toSet()));
+ snapshotChains.put(parent, new Pair<>(kvmIncrementalSnapshotChain, getTotalChainSize(kvmIncrementalSnapshotChain.stream().filter(snapInfo -> snapshotIdsToMigrate.contains(snapInfo.getId())).collect(Collectors.toList()))));
+ files.add(parent);
+ } else {
Review Comment:
Since the snapshot chain is created with a single iteration, and multiple snapshots from the same chain can go through this `for` loop, I thought this way would be more organized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "JoaoJandre (via GitHub)" <gi...@apache.org>.
JoaoJandre commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4030765619
> [SF] Trillian test result (tid-15580) Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8 Total time taken: 52325 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12086-t15580-kvm-ol8.zip Smoke tests completed. 146 look OK, 3 have errors, 0 did not run Only failed and skipped tests results shown below:
> Test Result Time (s) Test File
> ContextSuite context=TestListIdsParams>:teardown `Error` 1.12 test_list_ids_parameter.py
> test_01_snapshot_root_disk `Error` 4.93 test_snapshots.py
> test_02_list_snapshots_with_removed_data_store `Error` 49.18 test_snapshots.py
> test_02_list_snapshots_with_removed_data_store `Error` 49.19 test_snapshots.py
> ContextSuite context=TestSnapshotStandaloneBackup>:teardown `Error` 27.51 test_snapshots.py
> test_01_snapshot_usage `Error` 28.92 test_usage.py
> test_01_vpn_usage `Error` 1.09 test_usage.py
This is not the first time I see these errors in a recent PR. They might be broken.
However, since this PR deals with snapshot, it might be a good idea to check it anyway @hsato03
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3816796605
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4010238801
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#discussion_r2563966707
##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java:
##########
@@ -1619,7 +1619,7 @@ to support snapshots(backuped) as qcow2 files. */
destFile = new QemuImgFile(destPath, destFormat);
try {
boolean isQCOW2 = PhysicalDiskFormat.QCOW2.equals(sourceFormat);
- qemu.convert(srcFile, destFile, null, null, new QemuImageOptions(srcFile.getFormat(), srcFile.getFileName(), null),
+ qemu.convert(srcFile, destFile, null, null, null, new QemuImageOptions(srcFile.getFormat(), srcFile.getFileName(), null),
Review Comment:
[nitpick] Extra whitespace between 'null,' parameters. Should have consistent single-space separation between parameters.
```suggestion
qemu.convert(srcFile, destFile, null, null, null, new QemuImageOptions(srcFile.getFormat(), srcFile.getFileName(), null),
```
##########
engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java:
##########
@@ -272,7 +296,7 @@ public MigrationResponse migrateResources(Long srcImgStoreId, Long destImgStoreI
}
if (storageCapacityBelowThreshold(storageCapacities, destImgStoreId)) {
- storageCapacities = migrateAway(chosenFileForMigration, storageCapacities, snapshotChains, childTemplates, srcDatastore, destImgStoreId, executor, futures);
+ storageCapacities = migrateAway(chosenFileForMigration, storageCapacities, snapshotChains, childTemplates, snapshotIdsToMigrate, srcDatastore, destImgStoreId, executor, futures);
Review Comment:
[nitpick] Extra whitespace before 'snapshotIdsToMigrate' parameter. Should have consistent single-space separation between parameters.
```suggestion
storageCapacities = migrateAway(chosenFileForMigration, storageCapacities, snapshotChains, childTemplates, snapshotIdsToMigrate, srcDatastore, destImgStoreId, executor, futures);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3587473268
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15849
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3588622608
<b>[SF] Trillian Build Failed (tid-14880)<b/>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3588533730
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3590926255
<b>[SF] Trillian test result (tid-14882)</b>
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 53722 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12086-t14882-kvm-ol8.zip
Smoke tests completed. 149 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_02_list_cpvm_vm | `Failure` | 0.04 | test_ssvm.py
test_04_cpvm_internals | `Failure` | 0.05 | test_ssvm.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-3562554264
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4341555597
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-4313411351
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17602
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "Damans227 (via GitHub)" <gi...@apache.org>.
Damans227 commented on code in PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#discussion_r4175385076
##########
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtMigrateSnapshotsBetweenSecondaryStoragesCommandWrapper.java:
##########
@@ -0,0 +1,193 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package com.cloud.hypervisor.kvm.resource.wrapper;
+
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.MigrateSnapshotsBetweenSecondaryStoragesCommand;
+import com.cloud.agent.api.MigrateBetweenSecondaryStoragesCommandAnswer;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.agent.api.to.DataTO;
+import com.cloud.hypervisor.kvm.resource.LibvirtComputingResource;
+import com.cloud.hypervisor.kvm.storage.KVMStoragePool;
+import com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager;
+import com.cloud.resource.CommandWrapper;
+import com.cloud.resource.ResourceWrapper;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.cloudstack.utils.qemu.QemuImageOptions;
+import org.apache.cloudstack.utils.qemu.QemuImg;
+import org.apache.cloudstack.utils.qemu.QemuImgException;
+import org.apache.cloudstack.utils.qemu.QemuImgFile;
+import org.libvirt.LibvirtException;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@ResourceWrapper(handles = MigrateSnapshotsBetweenSecondaryStoragesCommand.class)
+public class LibvirtMigrateSnapshotsBetweenSecondaryStoragesCommandWrapper extends CommandWrapper<MigrateSnapshotsBetweenSecondaryStoragesCommand, Answer, LibvirtComputingResource> {
+
+ protected Set<String> filesToRemove;
+ protected List<Pair<Long, String>> resourcesToUpdate;
+ protected int wait;
+
+ @Override
+ public Answer execute(MigrateSnapshotsBetweenSecondaryStoragesCommand command, LibvirtComputingResource serverResource) {
+ filesToRemove = new HashSet<>();
+ resourcesToUpdate = new ArrayList<>();
+ wait = command.getWait() * 1000;
+
+ DataStoreTO srcDataStore = command.getSrcDataStore();
+ DataStoreTO destDataStore = command.getDestDataStore();
+ KVMStoragePoolManager storagePoolManager = serverResource.getStoragePoolMgr();
+
+ Set<KVMStoragePool> imagePools = new HashSet<>();
+ KVMStoragePool destImagePool = storagePoolManager.getStoragePoolByURI(destDataStore.getUrl());
+ imagePools.add(destImagePool);
+
+ String imagePoolUrl;
+ KVMStoragePool imagePool = null;
+
+ String parentSnapshotPath = null;
+ boolean parentSnapshotWasMigrated = false;
+ Set<Long> snapshotIdsToMigrate = command.getSnapshotIdsToMigrate();
+
+ DataTO currentSnapshot = null;
+
+ try {
+ for (DataTO snapshot : command.getSnapshotChain()) {
+ currentSnapshot = snapshot;
+ imagePoolUrl = snapshot.getDataStore().getUrl();
+ imagePool = storagePoolManager.getStoragePoolByURI(imagePoolUrl);
+ imagePools.add(imagePool);
+
+ String resourceCurrentPath = imagePool.getLocalPathFor(snapshot.getPath());
+
+ if (imagePoolUrl.equals(srcDataStore.getUrl()) && snapshotIdsToMigrate.contains(snapshot.getId())) {
+ logger.debug("Migrating snapshot [{}] to destination storage pool [{}].", snapshot.getId(), destImagePool.getUuid());
+ parentSnapshotPath = copyResourceToDestDataStore(snapshot, resourceCurrentPath, destImagePool, parentSnapshotPath);
+ parentSnapshotWasMigrated = !parentSnapshotPath.equals(resourceCurrentPath);
+ logger.debug("Snapshot [{}] migrated successfully. New path: [{}].", snapshot.getId(), parentSnapshotPath);
+ } else {
+ if (parentSnapshotWasMigrated) {
+ logger.debug("Rebasing snapshot [{}] to new parent path: [{}].", snapshot.getId(), parentSnapshotPath);
+ parentSnapshotPath = rebaseResourceToNewParentPath(resourceCurrentPath, parentSnapshotPath);
+ } else {
+ logger.debug("Skipping migration/rebase for snapshot [{}]. Keeping current path.", snapshot.getId());
+ parentSnapshotPath = resourceCurrentPath;
+ }
+ parentSnapshotWasMigrated = false;
+ }
+ }
+ logger.info("Successfully completed migrating snapshot chain [{}].", command.getSnapshotChain());
+ } catch (LibvirtException | QemuImgException e) {
+ logger.error("Exception while migrating snapshots [{}] to secondary storage [{}] due to: [{}].", command.getSnapshotChain(), imagePool, e.getMessage(), e);
+ return new MigrateBetweenSecondaryStoragesCommandAnswer(command, false, String.format("Unable to migrate snapshot with ID [%s].", currentSnapshot.getId()), resourcesToUpdate);
+ } finally {
+ for (String file : filesToRemove) {
Review Comment:
if a rebase fails partway through, does this still delete the old copy of the parent? the child that wasnt rebased yet would then point at a file thats gone
##########
engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java:
##########
@@ -352,39 +374,130 @@ protected Pair<String, Boolean> migrateCompleted(Long destDatastoreId, DataStore
protected Map<Long, Pair<Long, Long>> migrateAway(
DataObject chosenFileForMigration,
Map<Long, Pair<Long, Long>> storageCapacities,
- Map<DataObject,
- Pair<List<SnapshotInfo>, Long>> snapshotChains,
+ Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains,
Map<DataObject, Pair<List<TemplateInfo>, Long>> templateChains,
+ Set<Long> snapshotIdsToMigrate,
DataStore srcDatastore,
Long destDatastoreId,
List<Future<DataObjectResult>> futures) {
Long fileSize = migrationHelper.getFileSize(chosenFileForMigration, snapshotChains, templateChains);
storageCapacities = assumeMigrate(storageCapacities, srcDatastore.getId(), destDatastoreId, fileSize);
- MigrateDataTask task = new MigrateDataTask(chosenFileForMigration, srcDatastore, dataStoreManager.getDataStore(destDatastoreId, DataStoreRole.Image));
- if (chosenFileForMigration instanceof SnapshotInfo ) {
+ DataStore destDataStore = dataStoreManager.getDataStore(destDatastoreId, DataStoreRole.Image);
+
+ boolean isKvmIncrementalSnapshot = chosenFileForMigration instanceof SnapshotInfo && ((SnapshotInfo) chosenFileForMigration).isKvmIncrementalSnapshot() && snapshotChains.containsKey(chosenFileForMigration);
+
+ if (isKvmIncrementalSnapshot) {
+ MigrateKvmIncrementalSnapshotTask task = new MigrateKvmIncrementalSnapshotTask(chosenFileForMigration, snapshotChains, srcDatastore, destDataStore, snapshotIdsToMigrate);
+ futures.add(submitMigrationOnKvmAgent(srcDatastore.getScope().getScopeId(), task));
+ logger.debug("Incremental snapshot migration {} submitted to KVM Agent pool.", chosenFileForMigration.getUuid());
+ } else {
+ createMigrateDataTask(chosenFileForMigration, snapshotChains, templateChains, srcDatastore, destDataStore, futures);
+ }
+
+ return storageCapacities;
+ }
+
+ private void migrateKvmIncrementalSnapshotChain(DataObject chosenFileForMigration, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains, DataStore srcDatastore, DataStore destDataStore, Set<Long> snapshotIdsToMigrate) {
+ Transaction.execute((TransactionCallback<AsyncCallFuture<DataObjectResult>>) status -> {
+ MigrateBetweenSecondaryStoragesCommandAnswer answer = null;
+ AsyncCallFuture<DataObjectResult> future = new AsyncCallFuture<>();
+ DataObjectResult result = new DataObjectResult(chosenFileForMigration);
+
+ try {
+ List<SnapshotInfo> snapshotChain = snapshotChains.get(chosenFileForMigration).first();
+ MigrateSnapshotsBetweenSecondaryStoragesCommand migrateBetweenSecondaryStoragesCmd = new MigrateSnapshotsBetweenSecondaryStoragesCommand(snapshotChain.stream().map(DataObject::getTO).collect(Collectors.toList()), srcDatastore.getTO(), destDataStore.getTO(), snapshotIdsToMigrate);
+
+ HostVO host = getAvailableHost(((SnapshotInfo) chosenFileForMigration).getDataCenterId());
+ if (host == null) {
+ throw new CloudRuntimeException("No suitable hosts found to send migrate command.");
+ }
+
+ migrateBetweenSecondaryStoragesCmd.setWait(StorageManager.AgentMaxDataMigrationWaitTime.valueIn(host.getClusterId()));
+ answer = (MigrateBetweenSecondaryStoragesCommandAnswer) agentManager.send(host.getId(), migrateBetweenSecondaryStoragesCmd);
+ if (answer == null || !answer.getResult()) {
+ if (answer != null) {
+ logger.warn(answer.getDetails());
+ }
+ throw new CloudRuntimeException("Unable to migrate KVM incremental snapshots to another secondary storage.");
+ }
+ } catch (final OperationTimedoutException | AgentUnavailableException e) {
+ throw new CloudRuntimeException("Error while migrating KVM incremental snapshot chain. Check the logs for more information.", e);
+ } finally {
+ if (answer != null) {
+ updateSnapshotsReference(destDataStore, answer);
+ }
+ }
+ result.setSuccess(true);
+ future.complete(result);
+ return future;
+ });
+ }
+
+ private void updateSnapshotsReference(DataStore destDataStore, MigrateBetweenSecondaryStoragesCommandAnswer answer) {
+ for (Pair<Long, String> snapshotIdAndUpdatedCheckpointPath : answer.getMigratedResources()) {
+ Long snapshotId = snapshotIdAndUpdatedCheckpointPath.first();
+ String newCheckpointPath = snapshotIdAndUpdatedCheckpointPath.second();
+
+ SnapshotDataStoreVO snapshotDataStore = snapshotDataStoreDao.findOneBySnapshotAndDatastoreRole(snapshotId, DataStoreRole.Image);
Review Comment:
what happens if the snapshot is on more than one secondary storage, like after a copy to another zone? this lookup doesnt filter by the source store so it could move the wrong record
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM incremental volume snapshot migration between secondary storages [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12086:
URL: https://github.com/apache/cloudstack/pull/12086#issuecomment-5936170942
@hsato03 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org