You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "tdtmusic2 (via GitHub)" <gi...@apache.org> on 2025/03/17 20:39:21 UTC
[D] VM on Xen - weird ISO issue [cloudstack]
GitHub user tdtmusic2 created a discussion: VM on Xen - weird ISO issue
Hi everyone. Got a strange error today and it doesn't make sense as it's not related to the actual action that I want to do, or at least I don't think so, maybe someone else has an idea. The setup is simple, a xen host added in ACS, with a GPU in passtrough mode, a Windows VM deployed with a W11 iso attached and Windows PV as OS version. Setup goes all the way, although I noticed it's MUCH slower than deploying the same way but with Windows 11 as OS version instead of PV (and manually adding the virtio drivers for the disk controller in the setup). After the setup I wanted to detach the ISO to attach another one and I got this error. The only way to detach it was to stop the VM and do it. I tested with other iso's and it's the same behavior. The logs bellow don't seem to reveal much about the issue either. Any thoughts?
```
2025-03-17 16:13:19,472 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:8eec4d00) Add job-337 into job monitoring
2025-03-17 16:13:19,474 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Executing AsyncJobVO: {id:337, usene, instanceId: 49, cmd: org.apache.cloudstack.api.command.admin.iso.DetachIsoCmdByAdmin, cmdInfo: {"virtualmachineid":"066a0d43-a8e1-47a8-a302-d84a51df8d84",uZWE79gkLJARV7XWYGQoTHFX7Cg","httpmethod":"GET","ctxStartEventId":"623","ctxDetails":"{\"interface com.cloud.vm.VirtualMachine\":\"066a0d43-a8e1-47a8-a302-d84"ISO.DETACH"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 52242796961, completeMsid: null, lastUpdated: null}
2025-03-17 16:13:19,474 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp146370526-15:ctx-0015a751 ctx-63f74fd1) (logid:c3b82fb4) submit async job-337, details: AsanceType: VirtualMachine, instanceId: 49, cmd: org.apache.cloudstack.api.command.admin.iso.DetachIsoCmdByAdmin, cmdInfo: {"virtualmachineid":"066a0d43-a8e1-47Id":"2","sessionkey":"uZWE79gkLJARV7XWYGQoTHFX7Cg","httpmethod":"GET","ctxStartEventId":"623","ctxDetails":"{\"interface com.cloud.vm.VirtualMachine\":\"066a0d":"2","cmdEventType":"ISO.DETACH"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 52242796961, completeMsid: nd: null, removed: null}
2025-03-17 16:13:19,474 DEBUG [c.c.a.ApiServlet] (qtp146370526-15:ctx-0015a751 ctx-63f74fd1) (logid:c3b82fb4) ===END=== 10.84.84.1 -- GET virtualmachineid=0chIso&response=json&sessionkey=uZWE79gkLJARV7XWYGQoTHFX7Cg
2025-03-17 16:13:19,487 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (API-Job-Executor-12:ctx-5af21047 job-337 ctx-d5fbc440) (logid:e034732e) template 204 is alr
2025-03-17 16:13:19,492 ERROR [c.c.r.ResourceManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337 ctx-d5fbc440) (logid:e034732e) Host 11 does not have requioup of Nvidia Corporation TU104GL [Tesla T4] GPUs, vGPU Type: passthrough
2025-03-17 16:13:19,495 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Unexpected exception while executing ohIsoCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Host 11 does not have required GPU device or out of capacity. GPU group: Group of Nvidia Corporation TU104GL
at com.cloud.resource.ResourceManagerImpl.getGPUDevice(ResourceManagerImpl.java:3469)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy200.getGPUDevice(Unknown Source)
at com.cloud.hypervisor.HypervisorGuruBase.toVirtualMachineTO(HypervisorGuruBase.java:285)
at com.cloud.hypervisor.XenServerGuru.implement(XenServerGuru.java:93)
at com.cloud.template.TemplateManagerImpl.attachISOToVM(TemplateManagerImpl.java:1296)
at com.cloud.template.TemplateManagerImpl.attachISOToVM(TemplateManagerImpl.java:1314)
at com.cloud.template.TemplateManagerImpl.detachIso(TemplateManagerImpl.java:1175)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy236.detachIso(Unknown Source)
at org.apache.cloudstack.api.command.user.iso.DetachIsoCmd.execute(DetachIsoCmd.java:109)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2025-03-17 16:13:19,497 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Complete async job-337, jobStatus:udstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Host 11 does not have required GPU device or out of capacity. GPU grT4] GPUs, vGPU Type: passthrough"}
2025-03-17 16:13:19,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Publish async job-337 complete on
2025-03-17 16:13:19,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Wake up jobs related to job-337
2025-03-17 16:13:19,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Update db status for job-337
2025-03-17 16:13:19,499 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Wake up jobs joined with job-337 a
2025-03-17 16:13:19,500 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Done executing org.apache.cloudsta job-337
2025-03-17 16:13:19,500 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Remove job-337 from job monitoring
```
GitHub link: https://github.com/apache/cloudstack/discussions/10581
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] VM on Xen - weird ISO issue [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: VM on Xen - weird ISO issue
@tdtmusic2 , can you check the contents of your `host_gpu_groups` table? The code is expecting to find content in it and doesn't.
In the logs above I see `...does not have requioup of...` , the word requioup does not exist and does not occur anywhere in the code.
```
2025-03-17 16:13:19,492 ERROR [c.c.r.ResourceManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337 ctx-d5fbc440) (logid:e034732e) Host 11 does not have requioup of Nvidia Corporation TU104GL [Tesla T4] GPUs, vGPU Type: passthrough
```
the line should read something like:
```
"Host " + hostId + " does not have required GPU device or out of capacity. GPU group: " + groupName + ", vGPU Type: " + vgpuType;
```
You might also want to try restarting the management servers.
GitHub link: https://github.com/apache/cloudstack/discussions/10581#discussioncomment-12638915
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] VM on Xen - weird ISO issue [cloudstack]
Posted by "tdtmusic2 (via GitHub)" <gi...@apache.org>.
GitHub user tdtmusic2 edited a discussion: VM on Xen - weird ISO issue
Hi everyone. Got a strange error today and it doesn't make sense as it's not related to the actual action that I want to do, or at least I don't think so, maybe someone else has an idea. The setup is simple, a xen host added in ACS 4.19.2, with a GPU in passtrough mode, a Windows VM deployed with a W11 iso attached and Windows PV as OS version. Setup goes all the way, although I noticed it's MUCH slower than deploying the same way but with Windows 11 as OS version instead of PV (and manually adding the virtio drivers for the disk controller in the setup). After the setup I wanted to detach the ISO to attach another one and I got this error. The only way to detach it was to stop the VM and do it. I tested with other iso's and it's the same behavior. The logs bellow don't seem to reveal much about the issue either. Any thoughts?
```
2025-03-17 16:13:19,472 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:8eec4d00) Add job-337 into job monitoring
2025-03-17 16:13:19,474 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Executing AsyncJobVO: {id:337, usene, instanceId: 49, cmd: org.apache.cloudstack.api.command.admin.iso.DetachIsoCmdByAdmin, cmdInfo: {"virtualmachineid":"066a0d43-a8e1-47a8-a302-d84a51df8d84",uZWE79gkLJARV7XWYGQoTHFX7Cg","httpmethod":"GET","ctxStartEventId":"623","ctxDetails":"{\"interface com.cloud.vm.VirtualMachine\":\"066a0d43-a8e1-47a8-a302-d84"ISO.DETACH"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 52242796961, completeMsid: null, lastUpdated: null}
2025-03-17 16:13:19,474 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp146370526-15:ctx-0015a751 ctx-63f74fd1) (logid:c3b82fb4) submit async job-337, details: AsanceType: VirtualMachine, instanceId: 49, cmd: org.apache.cloudstack.api.command.admin.iso.DetachIsoCmdByAdmin, cmdInfo: {"virtualmachineid":"066a0d43-a8e1-47Id":"2","sessionkey":"uZWE79gkLJARV7XWYGQoTHFX7Cg","httpmethod":"GET","ctxStartEventId":"623","ctxDetails":"{\"interface com.cloud.vm.VirtualMachine\":\"066a0d":"2","cmdEventType":"ISO.DETACH"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 52242796961, completeMsid: nd: null, removed: null}
2025-03-17 16:13:19,474 DEBUG [c.c.a.ApiServlet] (qtp146370526-15:ctx-0015a751 ctx-63f74fd1) (logid:c3b82fb4) ===END=== 10.84.84.1 -- GET virtualmachineid=0chIso&response=json&sessionkey=uZWE79gkLJARV7XWYGQoTHFX7Cg
2025-03-17 16:13:19,487 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (API-Job-Executor-12:ctx-5af21047 job-337 ctx-d5fbc440) (logid:e034732e) template 204 is alr
2025-03-17 16:13:19,492 ERROR [c.c.r.ResourceManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337 ctx-d5fbc440) (logid:e034732e) Host 11 does not have requioup of Nvidia Corporation TU104GL [Tesla T4] GPUs, vGPU Type: passthrough
2025-03-17 16:13:19,495 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Unexpected exception while executing ohIsoCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Host 11 does not have required GPU device or out of capacity. GPU group: Group of Nvidia Corporation TU104GL
at com.cloud.resource.ResourceManagerImpl.getGPUDevice(ResourceManagerImpl.java:3469)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy200.getGPUDevice(Unknown Source)
at com.cloud.hypervisor.HypervisorGuruBase.toVirtualMachineTO(HypervisorGuruBase.java:285)
at com.cloud.hypervisor.XenServerGuru.implement(XenServerGuru.java:93)
at com.cloud.template.TemplateManagerImpl.attachISOToVM(TemplateManagerImpl.java:1296)
at com.cloud.template.TemplateManagerImpl.attachISOToVM(TemplateManagerImpl.java:1314)
at com.cloud.template.TemplateManagerImpl.detachIso(TemplateManagerImpl.java:1175)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy236.detachIso(Unknown Source)
at org.apache.cloudstack.api.command.user.iso.DetachIsoCmd.execute(DetachIsoCmd.java:109)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2025-03-17 16:13:19,497 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Complete async job-337, jobStatus:udstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Host 11 does not have required GPU device or out of capacity. GPU grT4] GPUs, vGPU Type: passthrough"}
2025-03-17 16:13:19,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Publish async job-337 complete on
2025-03-17 16:13:19,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Wake up jobs related to job-337
2025-03-17 16:13:19,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Update db status for job-337
2025-03-17 16:13:19,499 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Wake up jobs joined with job-337 a
2025-03-17 16:13:19,500 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Done executing org.apache.cloudsta job-337
2025-03-17 16:13:19,500 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Remove job-337 from job monitoring
```
GitHub link: https://github.com/apache/cloudstack/discussions/10581
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] VM on Xen - weird ISO issue [cloudstack]
Posted by "tdtmusic2 (via GitHub)" <gi...@apache.org>.
GitHub user tdtmusic2 added a comment to the discussion: VM on Xen - weird ISO issue
This is what's in the sql table, I guess it's as it should be. Just tried to deploy another instance now, with gpu passthrough, and while powered on I tried to detach the iso, got the same error. I restarted the management service before this.
GitHub link: https://github.com/apache/cloudstack/discussions/10581#discussioncomment-12639063
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] VM on Xen - weird ISO issue [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: VM on Xen - weird ISO issue
So there is no host 11 in that table and this is what the system is looking for. I am not sure (yet) if this is a configuration error or whether the system should handle this.
GitHub link: https://github.com/apache/cloudstack/discussions/10581#discussioncomment-12639570
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] VM on Xen - weird ISO issue [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland closed the discussion with a comment: VM on Xen - weird ISO issue
assuming this is no longer an issue @tdtmusic2 . please comment if you think it is.
GitHub link: https://github.com/apache/cloudstack/discussions/10581#discussioncomment-13559813
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] VM on Xen - weird ISO issue [cloudstack]
Posted by "tdtmusic2 (via GitHub)" <gi...@apache.org>.
GitHub user tdtmusic2 added a comment to the discussion: VM on Xen - weird ISO issue
> So there is no host 11 in that table and this is what the system is looking for. I am not sure (yet) if this is a configuration error or whether the system should handle this.
Host 11 was removed since then, now it's 12 and it's the same error message about it:
```
2025-03-27 12:22:14,670 ERROR [c.c.r.ResourceManagerImpl] (API-Job-Executor-3:ctx-77c18f69 job-415 ctx-8f3c75db) (logid:50f8449f) Host 12 does not have required GPU device or out of capacity. GPU group: Group of Nvidia Corporation TU104GL [Tesla T4] GPUs, vGPU Type: passthrough
2025-03-27 12:22:14,672 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-3:ctx-77c18f69 job-415) (logid:50f8449f) Unexpected exception while executing org.apache.cloudstack.api.command.admin.iso.DetachIsoCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Host 12 does not have required GPU device or out of capacity. GPU group: Group of Nvidia Corporation TU104GL [Tesla T4] GPUs, vGPU Type: passthrough
at com.cloud.resource.ResourceManagerImpl.getGPUDevice(ResourceManagerImpl.java:3469)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy200.getGPUDevice(Unknown Source)
at com.cloud.hypervisor.HypervisorGuruBase.toVirtualMachineTO(HypervisorGuruBase.java:285)
at com.cloud.hypervisor.XenServerGuru.implement(XenServerGuru.java:93)
at com.cloud.template.TemplateManagerImpl.attachISOToVM(TemplateManagerImpl.java:1296)
at com.cloud.template.TemplateManagerImpl.attachISOToVM(TemplateManagerImpl.java:1314)
at com.cloud.template.TemplateManagerImpl.detachIso(TemplateManagerImpl.java:1175)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy236.detachIso(Unknown Source)
at org.apache.cloudstack.api.command.user.iso.DetachIsoCmd.execute(DetachIsoCmd.java:109)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2025-03-27 12:22:14,674 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-3:ctx-77c18f69 job-415) (logid:50f8449f) Complete async job-415, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Host 12 does not have required GPU device or out of capacity. GPU group: Group of Nvidia Corporation TU104GL [Tesla T4] GPUs, vGPU Type: passthrough"}
```
GitHub link: https://github.com/apache/cloudstack/discussions/10581#discussioncomment-12639610
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] VM on Xen - weird ISO issue [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: VM on Xen - weird ISO issue
@tdtmusic2 , i cannot give you a direct answer. Can you tell me the exact version which that stacktrace came from? I'll have a look.
GitHub link: https://github.com/apache/cloudstack/discussions/10581#discussioncomment-12626596
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] VM on Xen - weird ISO issue [cloudstack]
Posted by "tdtmusic2 (via GitHub)" <gi...@apache.org>.
GitHub user tdtmusic2 added a comment to the discussion: VM on Xen - weird ISO issue
> @tdtmusic2 , i cannot give you a direct answer. Can you tell me the exact version which that stacktrace came from? I'll have a look.
Hi Daan. You mean CS version? It's 4.19.2.0.
GitHub link: https://github.com/apache/cloudstack/discussions/10581#discussioncomment-12638591
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] VM on Xen - weird ISO issue [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: VM on Xen - weird ISO issue
@tdtmusic2 , are you still looking at this?
GitHub link: https://github.com/apache/cloudstack/discussions/10581#discussioncomment-13002586
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] VM on Xen - weird ISO issue [cloudstack]
Posted by "harikrishna-patnala (via GitHub)" <gi...@apache.org>.
GitHub user harikrishna-patnala added a comment to the discussion: VM on Xen - weird ISO issue
@tdtmusic2 what I can see from the error in the logs is, CS is calling the below sql query, the output of it seems to be empty. Please check if that can give you any hint
SELECT hgg.*
FROM host_gpu_groups hgg
INNER JOIN vgpu_types vt ON hgg.id = vt.gpu_group_id
WHERE hgg.host_id = 12
AND hgg.group_name = 'Group of Nvidia Corporation TU104GL [Tesla T4] GPUs'
AND vt.vgpu_type = 'passthrough'
AND vt.remaining_capacity > 0
ORDER BY vt.remaining_capacity DESC, hgg.id;
GitHub link: https://github.com/apache/cloudstack/discussions/10581#discussioncomment-12648656
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] VM on Xen - weird ISO issue [cloudstack]
Posted by "tdtmusic2 (via GitHub)" <gi...@apache.org>.
GitHub user tdtmusic2 edited a comment on the discussion: VM on Xen - weird ISO issue
This is what's in the sql table, I guess it's as it should be. Just tried to deploy another instance now, with gpu passthrough, and while powered on I tried to detach the iso, got the same error. I restarted the management service before this.
What I don't undestand is what's the connection between available gpus (in my case there is only one, so I guess it IS out of capacity after being attached to the new vm) and ejecting a iso image.
GitHub link: https://github.com/apache/cloudstack/discussions/10581#discussioncomment-12639063
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] VM on Xen - weird ISO issue [cloudstack]
Posted by "tdtmusic2 (via GitHub)" <gi...@apache.org>.
GitHub user tdtmusic2 closed a discussion: VM on Xen - weird ISO issue
Hi everyone. Got a strange error today and it doesn't make sense as it's not related to the actual action that I want to do, or at least I don't think so, maybe someone else has an idea. The setup is simple, a xen host added in ACS 4.19.2, with a GPU in passtrough mode, a Windows VM deployed with a W11 iso attached and Windows PV as OS version. Setup goes all the way, although I noticed it's MUCH slower than deploying the same way but with Windows 11 as OS version instead of PV (and manually adding the virtio drivers for the disk controller in the setup). After the setup I wanted to detach the ISO to attach another one and I got this error. The only way to detach it was to stop the VM and do it. I tested with other iso's and it's the same behavior. The logs bellow don't seem to reveal much about the issue either. Any thoughts?
```
2025-03-17 16:13:19,472 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:8eec4d00) Add job-337 into job monitoring
2025-03-17 16:13:19,474 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Executing AsyncJobVO: {id:337, usene, instanceId: 49, cmd: org.apache.cloudstack.api.command.admin.iso.DetachIsoCmdByAdmin, cmdInfo: {"virtualmachineid":"066a0d43-a8e1-47a8-a302-d84a51df8d84",uZWE79gkLJARV7XWYGQoTHFX7Cg","httpmethod":"GET","ctxStartEventId":"623","ctxDetails":"{\"interface com.cloud.vm.VirtualMachine\":\"066a0d43-a8e1-47a8-a302-d84"ISO.DETACH"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 52242796961, completeMsid: null, lastUpdated: null}
2025-03-17 16:13:19,474 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp146370526-15:ctx-0015a751 ctx-63f74fd1) (logid:c3b82fb4) submit async job-337, details: AsanceType: VirtualMachine, instanceId: 49, cmd: org.apache.cloudstack.api.command.admin.iso.DetachIsoCmdByAdmin, cmdInfo: {"virtualmachineid":"066a0d43-a8e1-47Id":"2","sessionkey":"uZWE79gkLJARV7XWYGQoTHFX7Cg","httpmethod":"GET","ctxStartEventId":"623","ctxDetails":"{\"interface com.cloud.vm.VirtualMachine\":\"066a0d":"2","cmdEventType":"ISO.DETACH"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 52242796961, completeMsid: nd: null, removed: null}
2025-03-17 16:13:19,474 DEBUG [c.c.a.ApiServlet] (qtp146370526-15:ctx-0015a751 ctx-63f74fd1) (logid:c3b82fb4) ===END=== 10.84.84.1 -- GET virtualmachineid=0chIso&response=json&sessionkey=uZWE79gkLJARV7XWYGQoTHFX7Cg
2025-03-17 16:13:19,487 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (API-Job-Executor-12:ctx-5af21047 job-337 ctx-d5fbc440) (logid:e034732e) template 204 is alr
2025-03-17 16:13:19,492 ERROR [c.c.r.ResourceManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337 ctx-d5fbc440) (logid:e034732e) Host 11 does not have requioup of Nvidia Corporation TU104GL [Tesla T4] GPUs, vGPU Type: passthrough
2025-03-17 16:13:19,495 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Unexpected exception while executing ohIsoCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Host 11 does not have required GPU device or out of capacity. GPU group: Group of Nvidia Corporation TU104GL
at com.cloud.resource.ResourceManagerImpl.getGPUDevice(ResourceManagerImpl.java:3469)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy200.getGPUDevice(Unknown Source)
at com.cloud.hypervisor.HypervisorGuruBase.toVirtualMachineTO(HypervisorGuruBase.java:285)
at com.cloud.hypervisor.XenServerGuru.implement(XenServerGuru.java:93)
at com.cloud.template.TemplateManagerImpl.attachISOToVM(TemplateManagerImpl.java:1296)
at com.cloud.template.TemplateManagerImpl.attachISOToVM(TemplateManagerImpl.java:1314)
at com.cloud.template.TemplateManagerImpl.detachIso(TemplateManagerImpl.java:1175)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy236.detachIso(Unknown Source)
at org.apache.cloudstack.api.command.user.iso.DetachIsoCmd.execute(DetachIsoCmd.java:109)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2025-03-17 16:13:19,497 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Complete async job-337, jobStatus:udstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Host 11 does not have required GPU device or out of capacity. GPU grT4] GPUs, vGPU Type: passthrough"}
2025-03-17 16:13:19,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Publish async job-337 complete on
2025-03-17 16:13:19,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Wake up jobs related to job-337
2025-03-17 16:13:19,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Update db status for job-337
2025-03-17 16:13:19,499 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Wake up jobs joined with job-337 a
2025-03-17 16:13:19,500 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Done executing org.apache.cloudsta job-337
2025-03-17 16:13:19,500 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-12:ctx-5af21047 job-337) (logid:e034732e) Remove job-337 from job monitoring
```
GitHub link: https://github.com/apache/cloudstack/discussions/10581
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org