You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "daviftorres (via GitHub)" <gi...@apache.org> on 2025/12/01 21:08:59 UTC
[I] Instance does not start with message "Command failed due to Internal Server Error." [cloudstack]
daviftorres opened a new issue, #12171:
URL: https://github.com/apache/cloudstack/issues/12171
### The required feature described as a wish
Thank you, @Pearl1594, for helping confirm that the root cause of the failure was:
The attached ISO was no longer available after the storage was decommissioned.
The system didn’t provide proper error handling for this case (`"templateVO" was null`), resulting in a generic and unclear message.
```
2025-11-30 04:02:51,769 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-102:[ctx-33237947, job-67184/job-67186, ctx-8874b4da]) (logid:22521ecd) Invocation exception, caused by: java.lang.NullPointerException: Cannot invoke "com.cloud.storage.VMTemplateVO.isDirectDownload()" because "templateVO" is null
2025-11-30 04:02:51,769 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-102:[ctx-33237947, job-67184/job-67186, ctx-8874b4da]) (logid:22521ecd) Rethrow exception java.lang.NullPointerException: Cannot invoke "com.cloud.storage.VMTemplateVO.isDirectDownload()" because "templateVO" is null
2025-11-30 04:02:51,769 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-102:[ctx-33237947, job-67184/job-67186]) (logid:22521ecd) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 6682, job origin: 67184
2025-11-30 04:02:51,769 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-102:[ctx-33237947, job-67184/job-67186]) (logid:22521ecd) Unable to complete AsyncJob {"accountId":298,"cmd":"com.cloud.vm.VmWorkStart","cmdInfo":"rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAEqAAAAAAAAATcAAAAAAAAaGnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwc3IAEWphdmEudXRpbC5IYXNoTWFwBQfawcMWYNEDAAJGAApsb2FkRmFjdG9ySQAJdGhyZXNob2xkeHA_QAAAAAAADHcIAAAAEAAAAAV0ABBDb25zaWRlckxhc3RIb3N0dAAQck8wQUJYUUFCV1poYkhObHQAClZ
tUGFzc3dvcmR0ABxyTzBBQlhRQURuTmhkbVZrWDNCaGMzTjNiM0prdAAIQm9vdE1vZGV0ABJyTzBBQlhRQUJreEZSMEZEV1F0AAhCb290VHlwZXQAD3JPMEFCWFFBQkZWRlJra3QACFVlZmlGbGFndAAOck8wQUJYUUFBMWxsY3d4cA","cmdVersion":0,"completeMsid":null,"created":"Sun Nov 30 04:02:43 UTC 2025","id":67186,"initMsid":90520736259046,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":311,"uuid":"baca3bee-acb5-452a-809e-a8e201dadcb0"}, job origin: 67184 java.lang.NullPointerException: Cannot invoke "com.cloud.storage.VMTemplateVO.isDirectDownload()" because "templateVO" is null
at org.apache.cloudstack.storage.image.TemplateDataFactoryImpl.isTemplateMarkedForDirectDownload(TemplateDataFactoryImpl.java:299)
at com.cloud.template.TemplateManagerImpl.prepareIso(TemplateManagerImpl.java:1264)
at com.cloud.template.TemplateManagerImpl.prepareIsoForVmProfile(TemplateManagerImpl.java:636)
at jdk.internal.reflect.GeneratedMethodAccessor325.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
```
Would it be possible to improve the error message for this scenario? A more descriptive message would help regular users understand the issue without needing admin assistance.
Additionally, administrators could benefit from seeing more detailed information when the failure isn’t directly related to a missing ISO.
Thank you and regards,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Instance does not start with message "Command failed due to Internal Server Error." [cloudstack]
Posted by "vladimirpetrov (via GitHub)" <gi...@apache.org>.
vladimirpetrov commented on issue #12171:
URL: https://github.com/apache/cloudstack/issues/12171#issuecomment-3750478446
Can you provide more detailed info how to reproduce this, @daviftorres? I tried two scenarios:
1. Manually marking the ISO image in the DB as 'Destroyed'.
2. Manually deleted the iso from the secondary storage.
In both cases, when tried to start the VM, it failed with the following error message:
<img width="390" height="155" alt="Image" src="https://github.com/user-attachments/assets/24f8f458-590e-4bdc-bc14-0f45b57b3248" />
And there were no NPEs in the log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Instance does not start with message "Command failed due to Internal Server Error." [cloudstack]
Posted by "daviftorres (via GitHub)" <gi...@apache.org>.
daviftorres commented on issue #12171:
URL: https://github.com/apache/cloudstack/issues/12171#issuecomment-3755261120
Hey @vladimirpetrov, thanks for looking into this.
When we initially deployed the Zones, we were using one storage appliance (I’ll call it “old”). Later, after acquiring larger storage appliances dedicated to CloudStack (I’ll call them “new”), we had to replace them in CS Management.
Migrating Primary Storage is straightforward: we add the "new" storage to the zone, live-migrate the instances using “Migrate with storage”, and once nothing is left, CloudStack allows us to remove the "old" storage from the zone via the Web UI.
However, this is not what happens with Secondary Storage. Even after removing all assets, CloudStack still blocks its removal, mainly because of System VM templates.
For our regular templates, if we set the "old" storage to Read-Only and make sure it’s not in use, we can delete the template reference from the Zone and then copy it back from another Zone. This forces CloudStack to place it on the "new" storage (since there’s no way to choose the target Secondary Storage).
What we’ve noticed is that CloudStack seems to prioritize Secondary Storage based on the order in which they were added, always preferring the first one. This makes the first Secondary Storage effectively impossible to remove, forcing us to delete it directly from the database.
As you can imagine, this leaves orphaned assets that we later have to locate and clean up manually (see https://github.com/apache/cloudstack/issues/11641#issuecomment-3746236056
for details about the internal tool we built to handle this).
So, after that background, here’s the direct answer to your question (as best as I can remember):
We discovered that some instances had ISOs attached and were left that way by mistake since deployment. When we later powered them down and tried to start them again, the ISO could not be found. On the UI, the message `Command failed due to Internal Server Error`.
In that situation, there was no clear indication that the VM failed to start because the ISO file was missing. We would like either a more descriptive error message, or for CloudStack to automatically detach the missing ISO and continue starting the VM.
Another case where auto-detach would make sense is for Compute Offerings with HA enabled, as this would improve the reliability of the HA feature.
Please feel free to ask more questions, and I’ll try to respond faster this time.
Best regards,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Instance does not start with message "Command failed due to Internal Server Error." [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland closed issue #12171: Instance does not start with message "Command failed due to Internal Server Error."
URL: https://github.com/apache/cloudstack/issues/12171
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Instance does not start with message "Command failed due to Internal Server Error." [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #12171:
URL: https://github.com/apache/cloudstack/issues/12171#issuecomment-3799073186
fixed in #12406
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org