You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "sureshanaparti (via GitHub)" <gi...@apache.org> on 2025/12/24 06:42:38 UTC
[PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
sureshanaparti opened a new pull request, #12335:
URL: https://github.com/apache/cloudstack/pull/12335
### Description
This PR addresses Clone VM issue with VMware 80u3, retries cloneVM task when any file access issue while cloning from volume or template and should succeed after few attempts.
This is another issue, recently identified with VMware 80u3 (other related issues addressed here: https://github.com/apache/cloudstack/pull/10586)
Fixes #12328
<!--- Describe your changes in DETAIL - And how has behaviour functionally changed. -->
<!-- For new features, provide link to FS, dev ML discussion etc. -->
<!-- In case of bug fix, the expected and actual behaviours, steps to reproduce. -->
<!-- When "Fixes: #<id>" is specified, the issue/PR will automatically be closed when this PR gets merged -->
<!-- For addressing multiple issues/PRs, use multiple "Fixes: #<id>" -->
<!-- Fixes: # -->
<!--- ******************************************************************************* -->
<!--- NOTE: AUTOMATION USES THE DESCRIPTIONS TO SET LABELS AND PRODUCE DOCUMENTATION. -->
<!--- PLEASE PUT AN 'X' in only **ONE** box -->
<!--- ******************************************************************************* -->
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Feature/Enhancement Scale
- [ ] Major
- [ ] Minor
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [x] Major
- [ ] Minor
- [ ] Trivial
### Screenshots (if appropriate):
### How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
#### How did you try to break this feature and the system with this change?
<!-- see how your change affects other areas of the code, etc. -->
<!-- Please read the [CONTRIBUTING](https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md) document -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#issuecomment-3688864530
## [Codecov](https://app.codecov.io/gh/apache/cloudstack/pull/12335?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 4.00%. Comparing base ([`b394b5b`](https://app.codecov.io/gh/apache/cloudstack/commit/b394b5ba74e01b6b82580fc0152e676e5e4c16d0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`c72cce8`](https://app.codecov.io/gh/apache/cloudstack/commit/c72cce8bdd2226dfbcf6557ba27562a6ae6b5f5f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (b394b5b) and HEAD (c72cce8). Click for more details.
>
> <details><summary>HEAD has 1 upload less than BASE</summary>
>
>| Flag | BASE (b394b5b) | HEAD (c72cce8) |
>|------|------|------|
>|unittests|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## 4.20 #12335 +/- ##
=============================================
- Coverage 16.23% 4.00% -12.24%
=============================================
Files 5657 402 -5255
Lines 498865 32686 -466179
Branches 60545 5817 -54728
=============================================
- Hits 80991 1309 -79682
+ Misses 408843 31222 -377621
+ Partials 9031 155 -8876
```
| [Flag](https://app.codecov.io/gh/apache/cloudstack/pull/12335/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [uitests](https://app.codecov.io/gh/apache/cloudstack/pull/12335/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `4.00% <ø> (ø)` | |
| [unittests](https://app.codecov.io/gh/apache/cloudstack/pull/12335/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/cloudstack/pull/12335?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#issuecomment-3689119093
@kiranchavala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#issuecomment-3693482612
<b>[SF] Trillian test result (tid-15067)</b>
Environment: xcpng82 (x2), zone: Advanced Networking with Mgmt server ol9
Total time taken: 53169 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12335-t15067-xcpng82.zip
Smoke tests completed. 135 look OK, 6 have errors, 0 did not run
Only failed and skipped tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
ContextSuite context=TestClusterDRS>:setup | `Error` | 0.00 | test_cluster_drs.py
test_01_add_primary_storage_disabled_host | `Error` | 15.34 | test_primary_storage.py
test_01_primary_storage_iscsi | `Error` | 0.15 | test_primary_storage.py
test_01_primary_storage_nfs | `Error` | 0.14 | test_primary_storage.py
ContextSuite context=TestStorageTags>:setup | `Error` | 0.23 | test_primary_storage.py
test_02_list_snapshots_with_removed_data_store | `Error` | 16.59 | test_snapshots.py
test_02_list_snapshots_with_removed_data_store | `Error` | 16.59 | test_snapshots.py
test_01_deploy_vm_on_specific_host | `Error` | 0.10 | test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster | `Error` | 0.09 | test_vm_deployment_planner.py
test_08_migrate_vm | `Error` | 0.05 | test_vm_life_cycle.py
test_01_migrate_vm_strict_tags_success | `Error` | 0.19 | test_vm_strict_host_tags.py
test_02_migrate_vm_strict_tags_failure | `Error` | 0.19 | test_vm_strict_host_tags.py
test_01_restore_vm_strict_tags_success | `Error` | 0.17 | test_vm_strict_host_tags.py
test_02_restore_vm_strict_tags_failure | `Error` | 0.17 | test_vm_strict_host_tags.py
test_01_scale_vm_strict_tags_success | `Error` | 0.19 | test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failure | `Error` | 0.20 | test_vm_strict_host_tags.py
test_01_deploy_vm_on_specific_host_without_strict_tags | `Error` | 0.17 | test_vm_strict_host_tags.py
test_02_deploy_vm_on_any_host_without_strict_tags | `Error` | 5.47 | test_vm_strict_host_tags.py
test_03_deploy_vm_on_specific_host_with_strict_tags_success | `Error` | 0.16 | test_vm_strict_host_tags.py
test_04_deploy_vm_on_any_host_with_strict_tags_success | `Error` | 5.51 | test_vm_strict_host_tags.py
test_05_deploy_vm_on_specific_host_with_strict_tags_failure | `Failure` | 0.19 | test_vm_strict_host_tags.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#issuecomment-3700546873
> > > LGTM, Tested with multicluster env
> > > Was able to deploy a vm from a ova file
> > > ```
> > > 2025-12-26 07:48:12,899 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} with id 202 is already in store:ImageStore {"id":1,"name":"NFS:\/\/10.1.32.4\/acs\/secondary\/ref-trl-6142-v-Mu24-kiran-chavala\/ref-trl-6142-v-Mu24-kiran-chavala-sec1","uuid":"59a429d6-9367-45ed-b06f-af5d18201119"}, type: Image
> > > 2025-12-26 07:48:12,903 DEBUG [o.a.c.s.d.PrimaryDataStoreImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Not found (templateId:202poolId:1conf:null) in template_spool_ref, persisting it
> > > 2025-12-26 07:48:12,910 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} with id 202 is already in store:StoragePool {"id":1,"name":"ref-trl-6142-v-Mu24-kiran-chavala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"f5a29843-7c46-3fb1-a0ea-844860f848f1"}, type: Primary
> > > 2025-12-26 07:48:12,911 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Found template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} in storage pool StoragePool {"id":1,"name":"ref-trl-6142-v-Mu24-kiran-chavala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"f5a29843-7c46-3fb1-a0ea-844860f848f1"} with VMTemplateStoragePool: TmplPool[3-202-1-null]
> > > 2025-12-26 07:48:12,912 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Acquire lock on VMTemplateStoragePool 3 with timeout 3600 seconds
> > > 2025-12-26 07:48:12,917 INFO [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) lock is acquired for VMTemplateStoragePool 3
> > >
> > > 2025-12-26 07:49:50,874 DEBUG [c.c.a.t.Request] (DirectAgent-482:[ctx-16173aa5]) (logid:aae472c7) Seq 2-8532350969030120521: Processing: { Ans: , MgmtId: 32989492808056, via: 2(10.1.32.115), Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":"(3.50 GB) 3758096384","path":"ROOT-3","accountId":"0","id":"0","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
> > > 2025-12-26 07:49:50,874 DEBUG [c.c.a.t.Request] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Seq 2-8532350969030120521: Received: { Ans: , MgmtId: 32989492808056, via: 2(10.1.32.115), Ver: v1, Flags: 10, { CopyCmdAnswer } }
> > > 2025-12-26 07:49:50,887 DEBUG [o.a.c.s.v.VolumeObject] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Updated {"name":"ROOT-3","uuid":"085a9cfe-ae78-4cee-9dd1-2f4697b2c643"} from {"encryptFormat":null,"format":"OVA","path":null,"poolId":1,"size":3758096384} to {"encryptFormat":null,"format":"OVA","path":"ROOT-3","poolId":1,"size":3758096384}
> > >
> > >
> > > 2025-12-26 07:50:17,958 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-29:[ctx-7c16c813, job-35, ctx-a9d8fb99]) (logid:aae472c7) Complete async job-35, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.UserVmResponse/virtualmachine/{"id":"18646751-7e92-45e6-b858-4ed896eaf58d","name":"VM-18646751
> > > ```
> >
> >
> > as I remember, the issue appears very randomly. it would better to register several tempates and deploy vms to see if the issue can be reproduced and the retry fixes the issue
>
> Tested with multiple templates and the issue is not reproducible
does this remove your doubts @weizhouapache ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "kiranchavala (via GitHub)" <gi...@apache.org>.
kiranchavala commented on PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#issuecomment-3692449722
@blueorangutan test matrix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on code in PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#discussion_r2652173117
##########
vmware-base/src/main/java/com/cloud/hypervisor/vmware/mo/VirtualMachineMO.java:
##########
@@ -852,16 +848,42 @@ public boolean createFullClone(String cloneName, ManagedObjectReference morFolde
setDiskProvisioningType(relocSpec, morDs, diskProvisioningType);
- ManagedObjectReference morTask = _context.getService().cloneVMTask(_mor, morFolder, cloneName, cloneSpec);
+ return cloneVM(cloneName, morFolder, cloneSpec);
+ }
+ private boolean cloneVMTask(String cloneName, ManagedObjectReference morFolder, VirtualMachineCloneSpec cloneSpec) throws Exception {
+ ManagedObjectReference morTask = _context.getService().cloneVMTask(_mor, morFolder, cloneName, cloneSpec);
boolean result = _context.getVimClient().waitForTask(morTask);
if (result) {
_context.waitForTaskProgressDone(morTask);
return true;
- } else {
- logger.error("VMware cloneVM_Task failed due to " + TaskMO.getTaskFailureInfo(_context, morTask));
}
+ logger.error("VMware cloneVM_Task failed due to {}", TaskMO.getTaskFailureInfo(_context, morTask));
+ return false;
+ }
+
+ private boolean cloneVM(final String cloneName, final ManagedObjectReference morFolder, final VirtualMachineCloneSpec cloneSpec) throws Exception {
+ final int retry = 20;
+ int retryAttempt = 0;
+ while (++retryAttempt <= retry) {
+ try {
+ logger.debug("Cloning VM {}, attempt #{}", cloneName, retryAttempt);
+ return cloneVMTask(cloneName, morFolder, cloneSpec);
+ } catch (Exception e) {
+ logger.info("Got exception while cloning VM {}", cloneName, e);
+ if (e.getMessage() != null && e.getMessage().contains("Unable to access file")) {
+ logger.debug("Failed to clone VM {}. Retrying", cloneName);
+ try {
+ Thread.sleep(1000);
+ } catch (InterruptedException ie) {
+ logger.debug("Waiting to clone VM {} been interrupted: ", cloneName);
Review Comment:
that's for sleep call @abh1sar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#issuecomment-3688853704
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on code in PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#discussion_r2652173117
##########
vmware-base/src/main/java/com/cloud/hypervisor/vmware/mo/VirtualMachineMO.java:
##########
@@ -852,16 +848,42 @@ public boolean createFullClone(String cloneName, ManagedObjectReference morFolde
setDiskProvisioningType(relocSpec, morDs, diskProvisioningType);
- ManagedObjectReference morTask = _context.getService().cloneVMTask(_mor, morFolder, cloneName, cloneSpec);
+ return cloneVM(cloneName, morFolder, cloneSpec);
+ }
+ private boolean cloneVMTask(String cloneName, ManagedObjectReference morFolder, VirtualMachineCloneSpec cloneSpec) throws Exception {
+ ManagedObjectReference morTask = _context.getService().cloneVMTask(_mor, morFolder, cloneName, cloneSpec);
boolean result = _context.getVimClient().waitForTask(morTask);
if (result) {
_context.waitForTaskProgressDone(morTask);
return true;
- } else {
- logger.error("VMware cloneVM_Task failed due to " + TaskMO.getTaskFailureInfo(_context, morTask));
}
+ logger.error("VMware cloneVM_Task failed due to {}", TaskMO.getTaskFailureInfo(_context, morTask));
+ return false;
+ }
+
+ private boolean cloneVM(final String cloneName, final ManagedObjectReference morFolder, final VirtualMachineCloneSpec cloneSpec) throws Exception {
+ final int retry = 20;
+ int retryAttempt = 0;
+ while (++retryAttempt <= retry) {
+ try {
+ logger.debug("Cloning VM {}, attempt #{}", cloneName, retryAttempt);
+ return cloneVMTask(cloneName, morFolder, cloneSpec);
+ } catch (Exception e) {
+ logger.info("Got exception while cloning VM {}", cloneName, e);
+ if (e.getMessage() != null && e.getMessage().contains("Unable to access file")) {
+ logger.debug("Failed to clone VM {}. Retrying", cloneName);
+ try {
+ Thread.sleep(1000);
+ } catch (InterruptedException ie) {
+ logger.debug("Waiting to clone VM {} been interrupted: ", cloneName);
Review Comment:
that's for only sleep call @abh1sar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#issuecomment-3689355012
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16156
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#issuecomment-3693462663
<b>[SF] Trillian test result (tid-15064)</b>
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 50783 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12335-t15064-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#issuecomment-3693649961
<b>[SF] Trillian test result (tid-15066)</b>
Environment: vmware-70u3 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 69048 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12335-t15066-vmware-70u3.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_01_redundant_vpc_site2site_vpn | `Error` | 872.37 | test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options | `Error` | 741.51 | test_vpc_vpn.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#issuecomment-3688852045
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#issuecomment-3692451071
@kiranchavala a [SL] Trillian-Jenkins matrix job (EL8 mgmt + EL8 KVM, Ubuntu22 mgmt + Ubuntu22 KVM, EL8 mgmt + VMware 7.0u3, EL9 mgmt + XCP-ng 8.2 ) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#issuecomment-3692549665
> LGTM, Tested with multicluster env
>
> Was able to deploy a vm from a ova file
>
> ```
> 2025-12-26 07:48:12,899 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} with id 202 is already in store:ImageStore {"id":1,"name":"NFS:\/\/10.1.32.4\/acs\/secondary\/ref-trl-6142-v-Mu24-kiran-chavala\/ref-trl-6142-v-Mu24-kiran-chavala-sec1","uuid":"59a429d6-9367-45ed-b06f-af5d18201119"}, type: Image
> 2025-12-26 07:48:12,903 DEBUG [o.a.c.s.d.PrimaryDataStoreImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Not found (templateId:202poolId:1conf:null) in template_spool_ref, persisting it
> 2025-12-26 07:48:12,910 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} with id 202 is already in store:StoragePool {"id":1,"name":"ref-trl-6142-v-Mu24-kiran-chavala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"f5a29843-7c46-3fb1-a0ea-844860f848f1"}, type: Primary
> 2025-12-26 07:48:12,911 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Found template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} in storage pool StoragePool {"id":1,"name":"ref-trl-6142-v-Mu24-kiran-chavala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"f5a29843-7c46-3fb1-a0ea-844860f848f1"} with VMTemplateStoragePool: TmplPool[3-202-1-null]
> 2025-12-26 07:48:12,912 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Acquire lock on VMTemplateStoragePool 3 with timeout 3600 seconds
> 2025-12-26 07:48:12,917 INFO [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) lock is acquired for VMTemplateStoragePool 3
>
> 2025-12-26 07:49:50,874 DEBUG [c.c.a.t.Request] (DirectAgent-482:[ctx-16173aa5]) (logid:aae472c7) Seq 2-8532350969030120521: Processing: { Ans: , MgmtId: 32989492808056, via: 2(10.1.32.115), Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":"(3.50 GB) 3758096384","path":"ROOT-3","accountId":"0","id":"0","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
> 2025-12-26 07:49:50,874 DEBUG [c.c.a.t.Request] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Seq 2-8532350969030120521: Received: { Ans: , MgmtId: 32989492808056, via: 2(10.1.32.115), Ver: v1, Flags: 10, { CopyCmdAnswer } }
> 2025-12-26 07:49:50,887 DEBUG [o.a.c.s.v.VolumeObject] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Updated {"name":"ROOT-3","uuid":"085a9cfe-ae78-4cee-9dd1-2f4697b2c643"} from {"encryptFormat":null,"format":"OVA","path":null,"poolId":1,"size":3758096384} to {"encryptFormat":null,"format":"OVA","path":"ROOT-3","poolId":1,"size":3758096384}
>
>
> 2025-12-26 07:50:17,958 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-29:[ctx-7c16c813, job-35, ctx-a9d8fb99]) (logid:aae472c7) Complete async job-35, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.UserVmResponse/virtualmachine/{"id":"18646751-7e92-45e6-b858-4ed896eaf58d","name":"VM-18646751
> ```
as I remember, the issue appears very randomly.
it would better to register several tempates and deploy vms to see if the issue can be reproduced and the retry fixes the issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#issuecomment-3693579092
<b>[SF] Trillian test result (tid-15065)</b>
Environment: kvm-ubuntu22 (x2), zone: Advanced Networking with Mgmt server u22
Total time taken: 62159 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12335-t15065-kvm-ubuntu22.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false | `Failure` | 241.82 | test_routers_network_ops.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "kiranchavala (via GitHub)" <gi...@apache.org>.
kiranchavala commented on PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#issuecomment-3698949225
> > LGTM, Tested with multicluster env
> > Was able to deploy a vm from a ova file
> > ```
> > 2025-12-26 07:48:12,899 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} with id 202 is already in store:ImageStore {"id":1,"name":"NFS:\/\/10.1.32.4\/acs\/secondary\/ref-trl-6142-v-Mu24-kiran-chavala\/ref-trl-6142-v-Mu24-kiran-chavala-sec1","uuid":"59a429d6-9367-45ed-b06f-af5d18201119"}, type: Image
> > 2025-12-26 07:48:12,903 DEBUG [o.a.c.s.d.PrimaryDataStoreImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Not found (templateId:202poolId:1conf:null) in template_spool_ref, persisting it
> > 2025-12-26 07:48:12,910 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} with id 202 is already in store:StoragePool {"id":1,"name":"ref-trl-6142-v-Mu24-kiran-chavala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"f5a29843-7c46-3fb1-a0ea-844860f848f1"}, type: Primary
> > 2025-12-26 07:48:12,911 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Found template Template {"format":"OVA","id":202,"name":"ubuntu24","uniqueName":"202-2-979c3d6c-906b-3186-9fef-d05872b1fc3a","uuid":"1d8b5254-c5f4-4b1f-8997-7029bd904531"} in storage pool StoragePool {"id":1,"name":"ref-trl-6142-v-Mu24-kiran-chavala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"f5a29843-7c46-3fb1-a0ea-844860f848f1"} with VMTemplateStoragePool: TmplPool[3-202-1-null]
> > 2025-12-26 07:48:12,912 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Acquire lock on VMTemplateStoragePool 3 with timeout 3600 seconds
> > 2025-12-26 07:48:12,917 INFO [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) lock is acquired for VMTemplateStoragePool 3
> >
> > 2025-12-26 07:49:50,874 DEBUG [c.c.a.t.Request] (DirectAgent-482:[ctx-16173aa5]) (logid:aae472c7) Seq 2-8532350969030120521: Processing: { Ans: , MgmtId: 32989492808056, via: 2(10.1.32.115), Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":"(3.50 GB) 3758096384","path":"ROOT-3","accountId":"0","id":"0","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
> > 2025-12-26 07:49:50,874 DEBUG [c.c.a.t.Request] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Seq 2-8532350969030120521: Received: { Ans: , MgmtId: 32989492808056, via: 2(10.1.32.115), Ver: v1, Flags: 10, { CopyCmdAnswer } }
> > 2025-12-26 07:49:50,887 DEBUG [o.a.c.s.v.VolumeObject] (Work-Job-Executor-3:[ctx-880d0346, job-35/job-36, ctx-31015cc3]) (logid:aae472c7) Updated {"name":"ROOT-3","uuid":"085a9cfe-ae78-4cee-9dd1-2f4697b2c643"} from {"encryptFormat":null,"format":"OVA","path":null,"poolId":1,"size":3758096384} to {"encryptFormat":null,"format":"OVA","path":"ROOT-3","poolId":1,"size":3758096384}
> >
> >
> > 2025-12-26 07:50:17,958 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-29:[ctx-7c16c813, job-35, ctx-a9d8fb99]) (logid:aae472c7) Complete async job-35, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.UserVmResponse/virtualmachine/{"id":"18646751-7e92-45e6-b858-4ed896eaf58d","name":"VM-18646751
> > ```
>
> as I remember, the issue appears very randomly. it would better to register several tempates and deploy vms to see if the issue can be reproduced and the retry fixes the issue
Tested with multiple templates and the issue is not reproducible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland merged PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "abh1sar (via GitHub)" <gi...@apache.org>.
abh1sar commented on code in PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#discussion_r2650507719
##########
vmware-base/src/main/java/com/cloud/hypervisor/vmware/mo/VirtualMachineMO.java:
##########
@@ -852,16 +848,42 @@ public boolean createFullClone(String cloneName, ManagedObjectReference morFolde
setDiskProvisioningType(relocSpec, morDs, diskProvisioningType);
- ManagedObjectReference morTask = _context.getService().cloneVMTask(_mor, morFolder, cloneName, cloneSpec);
+ return cloneVM(cloneName, morFolder, cloneSpec);
+ }
+ private boolean cloneVMTask(String cloneName, ManagedObjectReference morFolder, VirtualMachineCloneSpec cloneSpec) throws Exception {
+ ManagedObjectReference morTask = _context.getService().cloneVMTask(_mor, morFolder, cloneName, cloneSpec);
boolean result = _context.getVimClient().waitForTask(morTask);
if (result) {
_context.waitForTaskProgressDone(morTask);
return true;
- } else {
- logger.error("VMware cloneVM_Task failed due to " + TaskMO.getTaskFailureInfo(_context, morTask));
}
+ logger.error("VMware cloneVM_Task failed due to {}", TaskMO.getTaskFailureInfo(_context, morTask));
+ return false;
+ }
+
+ private boolean cloneVM(final String cloneName, final ManagedObjectReference morFolder, final VirtualMachineCloneSpec cloneSpec) throws Exception {
+ final int retry = 20;
+ int retryAttempt = 0;
+ while (++retryAttempt <= retry) {
+ try {
+ logger.debug("Cloning VM {}, attempt #{}", cloneName, retryAttempt);
+ return cloneVMTask(cloneName, morFolder, cloneSpec);
+ } catch (Exception e) {
+ logger.info("Got exception while cloning VM {}", cloneName, e);
+ if (e.getMessage() != null && e.getMessage().contains("Unable to access file")) {
+ logger.debug("Failed to clone VM {}. Retrying", cloneName);
+ try {
+ Thread.sleep(1000);
+ } catch (InterruptedException ie) {
+ logger.debug("Waiting to clone VM {} been interrupted: ", cloneName);
Review Comment:
Do we really need to ignore interrupts for vm cloning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#issuecomment-3689019922
Packaging result [SF]: ✖️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 16151
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix for Clone VM issue (Unable to access file...since it is locked) with VMware 80u3 [cloudstack]
Posted by "kiranchavala (via GitHub)" <gi...@apache.org>.
kiranchavala commented on PR #12335:
URL: https://github.com/apache/cloudstack/pull/12335#issuecomment-3689115149
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org