You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "vanquyen020920 (via GitHub)" <gi...@apache.org> on 2026/06/02 14:50:35 UTC
[PR] Fix KVM live volume migration command payload [cloudstack]
vanquyen020920 opened a new pull request, #13319:
URL: https://github.com/apache/cloudstack/pull/13319
### Description
This PR...
<!--- Describe your changes in DETAIL - And how has behaviour functionally changed. -->
<!-- For new features, provide link to FS, dev ML discussion etc. -->
<!-- In case of bug fix, the expected and actual behaviours, steps to reproduce. -->
<!-- When "Fixes: #<id>" is specified, the issue/PR will automatically be closed when this PR gets merged -->
<!-- For addressing multiple issues/PRs, use multiple "Fixes: #<id>" -->
<!-- Fixes: # -->
<!--- ******************************************************************************* -->
<!--- NOTE: AUTOMATION USES THE DESCRIPTIONS TO SET LABELS AND PRODUCE DOCUMENTATION. -->
<!--- PLEASE PUT AN 'X' in only **ONE** box -->
<!--- ******************************************************************************* -->
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Feature/Enhancement Scale
- [ ] Major
- [ ] Minor
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [ ] Major
- [ ] Minor
- [ ] Trivial
### Screenshots (if appropriate):
### How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
#### How did you try to break this feature and the system with this change?
<!-- see how your change affects other areas of the code, etc. -->
<!-- Please read the [CONTRIBUTING](https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md) document -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM live volume migration command payload [cloudstack]
Posted by "wido (via GitHub)" <gi...@apache.org>.
wido closed pull request #13319: Fix KVM live volume migration command payload
URL: https://github.com/apache/cloudstack/pull/13319
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM live volume migration command payload [cloudstack]
Posted by "wido (via GitHub)" <gi...@apache.org>.
wido commented on PR #13319:
URL: https://github.com/apache/cloudstack/pull/13319#issuecomment-6221234886
@Damans227 Got it. With that Iām closing this PR. The author is free to re-open it if that seems like the best thing. But as this one has been silent, closing it seems like the best option for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM live volume migration command payload [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on code in PR #13319:
URL: https://github.com/apache/cloudstack/pull/13319#discussion_r3343617293
##########
engine/storage/datamotion/src/main/java/org/apache/cloudstack/storage/motion/AncientDataMotionStrategy.java:
##########
@@ -545,7 +545,7 @@ protected Answer migrateVolumeToPool(DataObject srcData, DataObject destData) {
VolumeInfo volume = (VolumeInfo)srcData;
StoragePool srcPool = (StoragePool)dataStoreMgr.getDataStore(srcData.getDataStore().getId(), DataStoreRole.Primary);
StoragePool destPool = (StoragePool)dataStoreMgr.getDataStore(destData.getDataStore().getId(), DataStoreRole.Primary);
- MigrateVolumeCommand command = new MigrateVolumeCommand(volume.getId(), volume.getPath(), destPool, volume.getAttachedVmName(), volume.getVolumeType(), waitInterval, volume.getChainInfo());
+ MigrateVolumeCommand command = new MigrateVolumeCommand(srcData.getTO(), destData.getTO(), null, null, waitInterval);
Review Comment:
this will have huge impact
##########
server/src/main/java/com/cloud/storage/VolumeApiServiceImpl.java:
##########
@@ -3531,9 +3531,8 @@ public Volume migrateVolume(MigrateVolumeCmd cmd) {
StoragePoolVO destinationStoragePoolVo = _storagePoolDao.findById(storagePoolId);
if (isSourceOrDestNotOnStorPool(srcStoragePoolVO, destinationStoragePoolVo)) {
- throw new InvalidParameterValueException("KVM does not support volume live migration due to the limited possibility to refresh VM XML domain. " +
- "Therefore, to live migrate a volume between storage pools, one must migrate the VM to a different host as well to force the VM XML domain update. " +
- "Use 'migrateVirtualMachineWithVolumes' instead.");
+ logger.debug("Allowing KVM live volume migration between different storage pools. VM [{}], volume [{}], source pool [{}], destination pool [{}].",
Review Comment:
this will have huge impact too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM live volume migration command payload [cloudstack]
Posted by "wido (via GitHub)" <gi...@apache.org>.
wido commented on PR #13319:
URL: https://github.com/apache/cloudstack/pull/13319#issuecomment-6219338549
How are we with this one? Looks like a great PR, but it has been silent for a while
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM live volume migration command payload [cloudstack]
Posted by "vanquyen020920 (via GitHub)" <gi...@apache.org>.
vanquyen020920 commented on PR #13319:
URL: https://github.com/apache/cloudstack/pull/13319#issuecomment-4610312674
Thanks @bernardodemarco for testing and for the detailed feedback.
You were right. My initial patch was incomplete: it allowed the API/data-motion flow to continue and fixed the `srcData == null` issue, but it did not update the active libvirt domain disk source. As a result, CloudStack volume metadata could be updated while the running VM was still using the old source disk.
I reworked the KVM agent implementation locally to handle the regular live migration path using libvirt block copy + pivot instead of only relying on `copyPhysicalDisk()`.
The updated flow I tested is:
1. Detect regular live migration for an attached/running KVM volume.
2. Prepare the destination disk on the destination storage pool.
3. Create the destination physical disk if it does not already exist.
4. Generate the destination libvirt disk XML.
5. Run libvirt `blockCopy`.
6. Wait for the block job to complete.
7. Pivot the running disk to the destination.
8. Verify that the active and inactive libvirt domain XML point to the destination disk before returning success.
Test environment:
* Apache CloudStack 4.20.0.0
* Ubuntu 22.04
* KVM
* Source primary storage: NFS / file-based primary storage
* Destination primary storage: Ceph RBD
### Test 1: system VM / virtual router root disk migration
Before migration, the running VM was using the NFS/file-based source:
```text
file disk vda /mnt/0d464e3f-5176-3ba8-8c5f-01b8f4da5a2d/cbe98e70-0ec8-4dfa-8888-42b38a763672
```
During migration, the agent created the destination RBD volume and completed the live block copy:
```text
Destination disk [cbe98e70-0ec8-4dfa-8888-42b38a763672] does not exist on pool [a15210c6-c858-3174-a390-183f4ed25096]. Creating it before live block copy.
Attempting to create volume cbe98e70-0ec8-4dfa-8888-42b38a763672 (RBD) in pool a15210c6-c858-3174-a390-183f4ed25096 with size (4.88 GB) 5242880000
Block copy has started for regular volume vda : cbe98e70-0ec8-4dfa-8888-42b38a763672
Block copy completed for the volume vda : cbe98e70-0ec8-4dfa-8888-42b38a763672
```
After migration, `virsh domblklist` shows that the active disk source was pivoted to Ceph RBD:
```text
network disk vda CMC-CLOUDSTACK/cbe98e70-0ec8-4dfa-8888-42b38a763672
```
I also verified both active and inactive libvirt XML, and both now point to the RBD destination:
```xml
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source protocol='rbd' name='CMC-CLOUDSTACK/cbe98e70-0ec8-4dfa-8888-42b38a763672'>
<host name='10.14.5.55'/>
<host name='10.14.5.56'/>
<host name='10.14.5.57'/>
<auth username='admin'>
<secret type='ceph' uuid='a15210c6-c858-3174-a390-183f4ed25096'/>
</auth>
</source>
<backingStore/>
<target dev='vda' bus='virtio'/>
</disk>
```
<!-- Attach screenshot: active/inactive XML and block copy completed log -->
### Test 2: user VM root disk migration
I also tested with a normal user VM.
Before migration, the VM root disk was on NFS/file-based storage:
```text
file disk vda /mnt/0d464e3f-5176-3ba8-8c5f-01b8f4da5a2d/65ca9af4-ccb8-43a2-96f1-c91202cd192f
```
<img width="1107" height="850" alt="image" src="https://github.com/user-attachments/assets/4bff61a1-8032-496d-bd3e-bc2e8b9a5f42" />
<img width="1280" height="700" alt="image" src="https://github.com/user-attachments/assets/58707519-4d21-4601-ae5b-6b71066b2f9b" />
<img width="1280" height="680" alt="image" src="https://github.com/user-attachments/assets/3c9fc1ed-ad9b-4372-99b7-a2a6e68ccd9d" />
<img width="1280" height="640" alt="image" src="https://github.com/user-attachments/assets/7dc3f750-ead0-4daa-9c96-286fb3b1d901" />
The guest had a test file before migration:
```text
cat test.txt
Test Migrate Disk
```
After live migration to Ceph RBD, CloudStack UI shows the root volume on `CPM-CEPH`.
The agent log shows the destination RBD volume was created and block copy completed:
```text
Using live block copy path for regular volume migration. VM [i-2-213-VM], source path [65ca9af4-ccb8-43a2-96f1-c91202cd192f], destination path [65ca9af4-ccb8-43a2-96f1-c91202cd192f], destination pool [a15210c6-c858-3174-a390-183f4ed25096].
Preparing destination disk for regular live volume migration. Destination path [65ca9af4-ccb8-43a2-96f1-c91202cd192f], destination pool [a15210c6-c858-3174-a390-183f4ed25096].
Destination disk [65ca9af4-ccb8-43a2-96f1-c91202cd192f] was not found on pool [a15210c6-c858-3174-a390-183f4ed25096]. It will be created before live block copy.
Destination disk [65ca9af4-ccb8-43a2-96f1-c91202cd192f] does not exist on pool [a15210c6-c858-3174-a390-183f4ed25096]. Creating it before live block copy.
Attempting to create volume 65ca9af4-ccb8-43a2-96f1-c91202cd192f (RBD) in pool a15210c6-c858-3174-a390-183f4ed25096 with size (10.00 GB) 10737418240
Block copy has started for regular volume vda : 65ca9af4-ccb8-43a2-96f1-c91202cd192f
Block copy completed for the volume vda : 65ca9af4-ccb8-43a2-96f1-c91202cd192f
```
After migration, I verified the guest remained running and the test file was still readable:
```text
cat test.txt
Test Migrate Disk
Final
```
<img width="1280" height="705" alt="image" src="https://github.com/user-attachments/assets/a447a84f-1680-46d4-a893-8020834676f4" />
<img width="1280" height="668" alt="image" src="https://github.com/user-attachments/assets/368da1c3-7dd5-4651-a3fc-bec42e02442d" />
<img width="1280" height="799" alt="image" src="https://github.com/user-attachments/assets/497be0ed-f512-4b07-8f47-07c8338c18fb" />
<img width="1280" height="225" alt="image" src="https://github.com/user-attachments/assets/c4718366-a5b0-40df-ba96-694776f0cb38" />
<img width="1201" height="828" alt="image" src="https://github.com/user-attachments/assets/b735e7ee-cd5c-49a6-a685-bb450646135d" />
This addresses the specific issue you found where only CloudStack metadata changed while the VM domain XML remained unchanged. With the updated implementation, the running libvirt domain is pivoted to the destination storage.
I agree that the broader feature still needs careful validation for additional edge cases, including:
* backing chains with multiple deltas;
* incremental volume snapshots;
* VM snapshots;
* file-based to file-based migrations;
* RBD to file-based migrations;
* multi-disk VMs;
* reboot validation after migration.
I will push the updated implementation and include this validation evidence so it can be reviewed and tested further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM live volume migration command payload [cloudstack]
Posted by "GutoVeronezi (via GitHub)" <gi...@apache.org>.
GutoVeronezi commented on PR #13319:
URL: https://github.com/apache/cloudstack/pull/13319#issuecomment-4605120984
@vanquyen020920, did you check the final VM XML?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM live volume migration command payload [cloudstack]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #13319:
URL: https://github.com/apache/cloudstack/pull/13319#issuecomment-4603623183
Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
Here are some useful points:
- In case of a new feature add useful documentation (raise doc PR at https://github.com/apache/cloudstack-documentation)
- Be patient and persistent. It might take some time to get a review or get the final approval from the committers.
- Pay attention to the quality of your code, ensure tests are passing and your PR doesn't have conflicts.
- Please follow [ASF Code of Conduct](https://github.com/apache/.github/blob/main/.github/CODE_OF_CONDUCT.md) for all communication including (but not limited to) comments on Pull Requests, Issues, Mailing list and Slack.
- Be sure to read the [CloudStack Coding Conventions](https://cwiki.apache.org/confluence/display/CLOUDSTACK/Coding+conventions).
Apache CloudStack is a community-driven project and together we are making it better š.
In case of doubts contact the developers at:
Mailing List: dev@cloudstack.apache.org (https://cloudstack.apache.org/mailing-lists.html)
Slack: https://apachecloudstack.slack.com/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix KVM live volume migration command payload [cloudstack]
Posted by "Damans227 (via GitHub)" <gi...@apache.org>.
Damans227 commented on PR #13319:
URL: https://github.com/apache/cloudstack/pull/13319#issuecomment-6219921889
> How are we with this one? Looks like a great PR, but it has been silent for a while
@wido the code here only changes the database, the running vm keeps the old disk (see bernardo's test above). the rework the author mentioned on jun 3 was never pushed. #14426 from @weizhouapache does live volume moves between ceph, nfs, shared mount point and local pools, so should we close this in favour of it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org