You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Eric Green <er...@gmail.com> on 2025/01/09 22:35:32 UTC

A possible bug in instance start for end users when a host is deleted

This appears to me to be a bug. Is it a known bug? I tried searching the Github bug list but Github's search function is awful.

Cloudstack 4.19.1.3, Ubuntu 20.04 management server, Ubuntu 22.04 compute hosts, KVM hypervisor.

Scenario: There are four physical hosts, A, B, C, D.  Let's say they're host ID 62,63,64,65 in the hosts table.  Each host has at least one virtual machine running on it.

Shut down the virtual machines on host A (without starting them elsewhere) which sets the last_host_id to 62, and delete host A in the Infrastructure tab.

As an end user (not an admin), try to start one of the virtual machines that were formerly on host A.

End result: The deployment planner returns the last_host_id of host A as the host to start the VM on. Since that host is marked deleted, the deployment executor removes it from the list of available hosts to start the VM on, and then throws an exception. The exception then appears on the GUI as a red box that basically gives the end user zero useful information.

Expected result:  The same as for admin users — the deployment planner notices that host A is deleted and cannot be used to deploy the VM, and instead picks a host from [B,C,D].

2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0) Adding pods to avoid lists for non-explicit VM deployment: []
2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0) Adding clusters to avoid lists for non-explicit VM deployment: []
2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0) Adding hosts to avoid lists for non-explicit VM deployment: []
2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0) DeploymentPlanner allocation algorithm: null
2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0) Trying to allocate a host and storage pools from dc:1, pod:null,cluster:null, requested cpu: 4000, requested ram: (4.00 GB) 4294967296
2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0) Is ROOT volume READY (pool already allocated)?: Yes
2025-01-09 13:28:35,466 DEBUG [c.c.a.ApiServer] (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) CIDRs from which account 'Account [{"accountName":"eric.green","id":8,"uuid":"e8baeda0-90a5-4c97-aa51-0adbfa3c0e4c"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-01-09 13:28:35,466 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0) Deploy avoids pods: [], clusters: [], hosts: []
2025-01-09 13:28:35,466 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0) Deploy hosts with priorities {} , hosts have NORMAL priority by default
2025-01-09 13:28:35,467 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0) This VM has last host_id specified, trying to choose the same host: 62
2025-01-09 13:28:35,471 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-01-09 13:28:35,471 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) API rate limiting is disabled. We will not use ApiRateLimitService.
2025-01-09 13:28:35,473 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-81:ctx-4b9a2500 job-418252) (logid:086f0ba0) Unexpected exception while executing org.apache.cloudstack.api.command.user.vm.StartVMCmd
java.lang.NullPointerException
        at com.cloud.host.dao.HostDaoImpl.loadHostTags(HostDaoImpl.java:879)
        at jdk.internal.reflect.GeneratedMethodAccessor336.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy96.loadHostTags(Unknown Source)
        at com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:451)
        at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveVirtualMachine(VMEntityManagerImpl.java:206)
        at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.reserve(VirtualMachineEntityImpl.java:202)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5445)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5299)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3224)
        at jdk.internal.reflect.GeneratedMethodAccessor787.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy186.startVirtualMachine(Unknown Source)
        at org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:181)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)


Re: A possible bug in instance start for end users when a host is deleted

Posted by Nux <nu...@li.nux.ro>.
Good call, Daan,

I remember this bug getting fixed a while ago, seems to show up like a 
bad penny.

On 2025-01-10 12:38, Daan wrote:
> Eric,
> Do you have the means to test a PR? if so, please have a look at 
> [10175](https://github.com/apache/cloudstack/pull/10175/)
> 
> On 2025/01/10 08:54:30 Daan Hoogland wrote:
>> Makes sense Eric, I'll reapply this old PR on 4.19
>> 
>> On Fri, Jan 10, 2025 at 9:31 AM Eric Green <er...@gmail.com> 
>> wrote:
>> 
>> > I don’t know that it’s acceptable as a known bug for 4.19 unless the
>> > workaround that I did for my cluster (whack last_host_id in the database if
>> > it points at removed hosts) is documented.
>> >
>> > But honestly, any bug that requires manually whacking things in the
>> > database as a workaround probably needs to be fixed. Especially if it is in
>> > the n-1 release. I, and many other people using Cloudstack in production,
>> > don’t update to the very latest and greatest release, we stick back on the
>> > n-1 release, which has had more testing and bug fixing. Usually.
>> >
>> > > On Jan 10, 2025, at 12:22 AM, Daan Hoogland <da...@gmail.com>
>> > wrote:
>> > >
>> > > Wei, Eric,
>> > >
>> > > This sounds like we must reapply for 4.19, and make sure it doesn't get
>> > > pulled forward. The 4.20 code for this class has been heavily
>> > refactorred.
>> > > We could also accept this as a known bug for 4.19?
>> > >
>> > > On Fri, Jan 10, 2025 at 8:47 AM Eric Green <er...@gmail.com>
>> > wrote:
>> > >
>> > >> Thanks, Wei.  The log definitely stated that ‘considerlasthost’ was
>> > >> getting defaulted to true (and honestly that should be configurable
>> > because
>> > >> I really want the planner to start up the vm for users on whatever host
>> > has
>> > >> the most excess capacity at the moment), and that the so I was confused
>> > >> when Kiran claimed it was not. That’s not what the log said! (“This VM
>> > has
>> > >> last host_id specified, trying to choose the same host: 62”).
>> > >>
>> > >> So it appears that this is a bug that was fixed in 4.18.x, was
>> > supposedly
>> > >> pulled into 4.19.x, but somehow did not actually get pulled into 4.19.x.
>> > >> Since 4.20.x does not appear to have the bug, the solution appear to be
>> > to
>> > >> upgrade to 4.20.x before I delete another host 😊. (I do not plan to
>> > delete
>> > >> any hosts anytime soon, I just got all the old hosts retired).
>> > >>
>> > >> I resolved the issue in my own Cloudstack deployment by whacking
>> > >> last_host_id in the database to a current host if it pointed at a
>> > retired
>> > >> host. That allowed my users to start virtual machines last started on
>> > the
>> > >> retired hosts without involving an administrator. But it does appear
>> > that I
>> > >> happened upon a bug here, from what you say. I presume that since you
>> > know
>> > >> the exact cause, you will file an actual bug in Github that this fix has
>> > >> not been integrated to 4.19.x branch?
>> > >>
>> > >> From: Wei ZHOU <us...@gmail.com>
>> > >> Date: Thursday, January 9, 2025 at 11:27 PM
>> > >> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>, Daan
>> > >> Hoogland <da...@shapeblue.com>, Suresh.Anaparti@shapeblue.com <
>> > >> Suresh.Anaparti@shapeblue.com>
>> > >> Subject: Re: A possible bug in instance start for end users when a host
>> > is
>> > >> deleted
>> > >> Hi Kiran,
>> > >>
>> > >> For backward compatibility, the `considerlasthost` is default to true
>> > if it
>> > >> is not set.
>> > >>
>> > >> The issue seems to be fixed in 4.19.1.0 by
>> > >> https://github.com/apache/cloudstack/pull/9037
>> > >>
>> > >> However, I cannot find the changes in the code base of 4.19 branch.
>> > >>
>> > >>
>> > https://github.com/apache/cloudstack/blob/4.19/server/src/main/java/com/cloud/deploy/DeploymentPlanningManagerImpl.java#L449-L453
>> > >>
>> > >> It looks like something is wrong during the merge of the 4.18 to 4.19
>> > >> branch.
>> > >> cc @Daan Hoogland <da...@shapeblue.com>
>> > >> @Suresh.Anaparti@shapeblue.com <Su...@shapeblue.com>
>> > >>
>> > >>
>> > >> -Wei
>> > >>
>> > >> On Fri, Jan 10, 2025 at 7:58 AM Kiran Chavala <
>> > kiran.chavala@shapeblue.com
>> > >>>
>> > >> wrote:
>> > >>
>> > >>> Hi Eric
>> > >>>
>> > >>> I am not hitting the issue on the latest 4.20 release of cloudstack
>> > >>>
>> > >>> For a normal end user, it  doesn’t consider the last host to start the
>> > >>> virtual machine
>> > >>>
>> > >>> the startVirtualMachine Api call has the parameter “considerlasthost”
>> > >> for
>> > >>> the  admin user
>> > >>>
>> > >>>
>> > >>>
>> > >>
>> > https://cloudstack.apache.org/api/apidocs-4.20/apis/startVirtualMachine.html
>> > >>>
>> > >>> For a normal user there is no parameter “ considerlasthost” , the
>> > >>> deployment planner picks up the available host
>> > >>>
>> > >>> Normal user
>> > >>>
>> > >>> (localcloud2) 🐱 > start virtualmachine
>> > >>> bootintosetup= clusterid=     filter=        hostid=        id=
>> > >>> podid=
>> > >>>
>> > >>>
>> > >>> Regards
>> > >>> Kiran
>> > >>>
>> > >>> From: Eric Green <er...@gmail.com>
>> > >>> Date: Friday, 10 January 2025 at 4:05 AM
>> > >>> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> > >>> Subject: A possible bug in instance start for end users when a host is
>> > >>> deleted
>> > >>> This appears to me to be a bug. Is it a known bug? I tried searching
>> > the
>> > >>> Github bug list but Github's search function is awful.
>> > >>>
>> > >>> Cloudstack 4.19.1.3, Ubuntu 20.04 management server, Ubuntu 22.04
>> > compute
>> > >>> hosts, KVM hypervisor.
>> > >>>
>> > >>> Scenario: There are four physical hosts, A, B, C, D.  Let's say they're
>> > >>> host ID 62,63,64,65 in the hosts table.  Each host has at least one
>> > >> virtual
>> > >>> machine running on it.
>> > >>>
>> > >>> Shut down the virtual machines on host A (without starting them
>> > >> elsewhere)
>> > >>> which sets the last_host_id to 62, and delete host A in the
>> > >> Infrastructure
>> > >>> tab.
>> > >>>
>> > >>> As an end user (not an admin), try to start one of the virtual machines
>> > >>> that were formerly on host A.
>> > >>>
>> > >>> End result: The deployment planner returns the last_host_id of host A
>> > as
>> > >>> the host to start the VM on. Since that host is marked deleted, the
>> > >>> deployment executor removes it from the list of available hosts to
>> > start
>> > >>> the VM on, and then throws an exception. The exception then appears on
>> > >> the
>> > >>> GUI as a red box that basically gives the end user zero useful
>> > >> information.
>> > >>>
>> > >>> Expected result:  The same as for admin users — the deployment planner
>> > >>> notices that host A is deleted and cannot be used to deploy the VM, and
>> > >>> instead picks a host from [B,C,D].
>> > >>>
>> > >>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> > >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
>> > >> (logid:086f0ba0)
>> > >>> Adding pods to avoid lists for non-explicit VM deployment: []
>> > >>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> > >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
>> > >> (logid:086f0ba0)
>> > >>> Adding clusters to avoid lists for non-explicit VM deployment: []
>> > >>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> > >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
>> > >> (logid:086f0ba0)
>> > >>> Adding hosts to avoid lists for non-explicit VM deployment: []
>> > >>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> > >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
>> > >> (logid:086f0ba0)
>> > >>> DeploymentPlanner allocation algorithm: null
>> > >>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> > >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
>> > >> (logid:086f0ba0)
>> > >>> Trying to allocate a host and storage pools from dc:1,
>> > >>> pod:null,cluster:null, requested cpu: 4000, requested ram: (4.00 GB)
>> > >>> 4294967296
>> > >>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> > >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
>> > >> (logid:086f0ba0)
>> > >>> Is ROOT volume READY (pool already allocated)?: Yes
>> > >>> 2025-01-09 13:28:35,466 DEBUG [c.c.a.ApiServer]
>> > >>> (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) CIDRs
>> > from
>> > >>> which account 'Account
>> > >>>
>> > >>
>> > [{"accountName":"eric.green","id":8,"uuid":"e8baeda0-90a5-4c97-aa51-0adbfa3c0e4c"}]'
>> > >>> is allowed to perform API calls: 0.0.0.0/0,::/0
>> > >>> 2025-01-09 <http://0.0.0.0/0,::/02025-01-09> 13:28:35,466 DEBUG
>> > >>> [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500
>> > >>> job-418252 ctx-34040330) (logid:086f0ba0) Deploy avoids pods: [],
>> > >> clusters:
>> > >>> [], hosts: []
>> > >>> 2025-01-09 13:28:35,466 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> > >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
>> > >> (logid:086f0ba0)
>> > >>> Deploy hosts with priorities {} , hosts have NORMAL priority by default
>> > >>> 2025-01-09 13:28:35,467 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> > >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
>> > >> (logid:086f0ba0)
>> > >>> This VM has last host_id specified, trying to choose the same host: 62
>> > >>> 2025-01-09 13:28:35,471 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker]
>> > >>> (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491)
>> > >> RoleService
>> > >>> is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
>> > >>> 2025-01-09 13:28:35,471 DEBUG [o.a.c.r.ApiRateLimitServiceImpl]
>> > >>> (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) API rate
>> > >>> limiting is disabled. We will not use ApiRateLimitService.
>> > >>> 2025-01-09 13:28:35,473 ERROR [c.c.a.ApiAsyncJobDispatcher]
>> > >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252) (logid:086f0ba0)
>> > Unexpected
>> > >>> exception while executing
>> > >>> org.apache.cloudstack.api.command.user.vm.StartVMCmd
>> > >>> java.lang.NullPointerException
>> > >>>        at
>> > >>> com.cloud.host.dao.HostDaoImpl.loadHostTags(HostDaoImpl.java:879)
>> > >>>        at
>> > jdk.internal.reflect.GeneratedMethodAccessor336.invoke(Unknown
>> > >>> Source)
>> > >>>        at
>> > >>>
>> > >>
>> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > >>>        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> > >>>        at
>> > >>>
>> > >>
>> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>> > >>>        at
>> > >>>
>> > >>
>> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>> > >>>        at
>> > >>>
>> > >>
>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>> > >>>        at
>> > >>>
>> > >>
>> > com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
>> > >>>        at
>> > >>>
>> > >>
>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
>> > >>>        at
>> > >>>
>> > >>
>> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
>> > >>>        at
>> > >>>
>> > >>
>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>> > >>>        at
>> > >>>
>> > >>
>> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
>> > >>>        at com.sun.proxy.$Proxy96.loadHostTags(Unknown Source)
>> > >>>        at
>> > >>>
>> > >>
>> > com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:451)
>> > >>>        at
>> > >>>
>> > >>
>> > org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveVirtualMachine(VMEntityManagerImpl.java:206)
>> > >>>        at
>> > >>>
>> > >>
>> > org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.reserve(VirtualMachineEntityImpl.java:202)
>> > >>>        at
>> > >>>
>> > >>
>> > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5445)
>> > >>>        at
>> > >>>
>> > >>
>> > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5299)
>> > >>>        at
>> > >>>
>> > >>
>> > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3224)
>> > >>>        at
>> > jdk.internal.reflect.GeneratedMethodAccessor787.invoke(Unknown
>> > >>> Source)
>> > >>>        at
>> > >>>
>> > >>
>> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > >>>        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> > >>>        at
>> > >>>
>> > >>
>> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>> > >>>        at
>> > >>>
>> > >>
>> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>> > >>>        at
>> > >>>
>> > >>
>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>> > >>>        at
>> > >>>
>> > >>
>> > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>> > >>>        at
>> > >>>
>> > >>
>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
>> > >>>        at
>> > >>>
>> > >>
>> > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
>> > >>>        at
>> > >>>
>> > >>
>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
>> > >>>        at
>> > >>>
>> > >>
>> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
>> > >>>        at
>> > >>>
>> > >>
>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>> > >>>        at
>> > >>>
>> > >>
>> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
>> > >>>        at com.sun.proxy.$Proxy186.startVirtualMachine(Unknown Source)
>> > >>>        at
>> > >>>
>> > >>
>> > org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:181)
>> > >>>        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
>> > >>>        at
>> > >>>
>> > >>
>> > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
>> > >>>        at
>> > >>>
>> > >>
>> > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
>> > >>>        at
>> > >>>
>> > >>
>> > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>> > >>>        at
>> > >>>
>> > >>
>> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>> > >>>        at
>> > >>>
>> > >>
>> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>> > >>>        at
>> > >>>
>> > >>
>> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>> > >>>        at
>> > >>>
>> > >>
>> > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>> > >>>        at
>> > >>>
>> > >>
>> > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
>> > >>>        at
>> > >>>
>> > >>
>> > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> > >>>        at
>> > >>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>> > >>>        at
>> > >>>
>> > >>
>> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> > >>>        at
>> > >>>
>> > >>
>> > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> > >>>        at java.base/java.lang.Thread.run(Thread.java:829)
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>
>> > >
>> > >
>> > > --
>> > > Daan
>> >
>> >
>> 
>> --
>> Daan
>> 

Re: A possible bug in instance start for end users when a host is deleted

Posted by Daan <da...@apache.org>.
Eric,
Do you have the means to test a PR? if so, please have a look at [10175](https://github.com/apache/cloudstack/pull/10175/)

On 2025/01/10 08:54:30 Daan Hoogland wrote:
> Makes sense Eric, I'll reapply this old PR on 4.19
> 
> On Fri, Jan 10, 2025 at 9:31 AM Eric Green <er...@gmail.com> wrote:
> 
> > I don’t know that it’s acceptable as a known bug for 4.19 unless the
> > workaround that I did for my cluster (whack last_host_id in the database if
> > it points at removed hosts) is documented.
> >
> > But honestly, any bug that requires manually whacking things in the
> > database as a workaround probably needs to be fixed. Especially if it is in
> > the n-1 release. I, and many other people using Cloudstack in production,
> > don’t update to the very latest and greatest release, we stick back on the
> > n-1 release, which has had more testing and bug fixing. Usually.
> >
> > > On Jan 10, 2025, at 12:22 AM, Daan Hoogland <da...@gmail.com>
> > wrote:
> > >
> > > Wei, Eric,
> > >
> > > This sounds like we must reapply for 4.19, and make sure it doesn't get
> > > pulled forward. The 4.20 code for this class has been heavily
> > refactorred.
> > > We could also accept this as a known bug for 4.19?
> > >
> > > On Fri, Jan 10, 2025 at 8:47 AM Eric Green <er...@gmail.com>
> > wrote:
> > >
> > >> Thanks, Wei.  The log definitely stated that ‘considerlasthost’ was
> > >> getting defaulted to true (and honestly that should be configurable
> > because
> > >> I really want the planner to start up the vm for users on whatever host
> > has
> > >> the most excess capacity at the moment), and that the so I was confused
> > >> when Kiran claimed it was not. That’s not what the log said! (“This VM
> > has
> > >> last host_id specified, trying to choose the same host: 62”).
> > >>
> > >> So it appears that this is a bug that was fixed in 4.18.x, was
> > supposedly
> > >> pulled into 4.19.x, but somehow did not actually get pulled into 4.19.x.
> > >> Since 4.20.x does not appear to have the bug, the solution appear to be
> > to
> > >> upgrade to 4.20.x before I delete another host 😊. (I do not plan to
> > delete
> > >> any hosts anytime soon, I just got all the old hosts retired).
> > >>
> > >> I resolved the issue in my own Cloudstack deployment by whacking
> > >> last_host_id in the database to a current host if it pointed at a
> > retired
> > >> host. That allowed my users to start virtual machines last started on
> > the
> > >> retired hosts without involving an administrator. But it does appear
> > that I
> > >> happened upon a bug here, from what you say. I presume that since you
> > know
> > >> the exact cause, you will file an actual bug in Github that this fix has
> > >> not been integrated to 4.19.x branch?
> > >>
> > >> From: Wei ZHOU <us...@gmail.com>
> > >> Date: Thursday, January 9, 2025 at 11:27 PM
> > >> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>, Daan
> > >> Hoogland <da...@shapeblue.com>, Suresh.Anaparti@shapeblue.com <
> > >> Suresh.Anaparti@shapeblue.com>
> > >> Subject: Re: A possible bug in instance start for end users when a host
> > is
> > >> deleted
> > >> Hi Kiran,
> > >>
> > >> For backward compatibility, the `considerlasthost` is default to true
> > if it
> > >> is not set.
> > >>
> > >> The issue seems to be fixed in 4.19.1.0 by
> > >> https://github.com/apache/cloudstack/pull/9037
> > >>
> > >> However, I cannot find the changes in the code base of 4.19 branch.
> > >>
> > >>
> > https://github.com/apache/cloudstack/blob/4.19/server/src/main/java/com/cloud/deploy/DeploymentPlanningManagerImpl.java#L449-L453
> > >>
> > >> It looks like something is wrong during the merge of the 4.18 to 4.19
> > >> branch.
> > >> cc @Daan Hoogland <da...@shapeblue.com>
> > >> @Suresh.Anaparti@shapeblue.com <Su...@shapeblue.com>
> > >>
> > >>
> > >> -Wei
> > >>
> > >> On Fri, Jan 10, 2025 at 7:58 AM Kiran Chavala <
> > kiran.chavala@shapeblue.com
> > >>>
> > >> wrote:
> > >>
> > >>> Hi Eric
> > >>>
> > >>> I am not hitting the issue on the latest 4.20 release of cloudstack
> > >>>
> > >>> For a normal end user, it  doesn’t consider the last host to start the
> > >>> virtual machine
> > >>>
> > >>> the startVirtualMachine Api call has the parameter “considerlasthost”
> > >> for
> > >>> the  admin user
> > >>>
> > >>>
> > >>>
> > >>
> > https://cloudstack.apache.org/api/apidocs-4.20/apis/startVirtualMachine.html
> > >>>
> > >>> For a normal user there is no parameter “ considerlasthost” , the
> > >>> deployment planner picks up the available host
> > >>>
> > >>> Normal user
> > >>>
> > >>> (localcloud2) 🐱 > start virtualmachine
> > >>> bootintosetup= clusterid=     filter=        hostid=        id=
> > >>> podid=
> > >>>
> > >>>
> > >>> Regards
> > >>> Kiran
> > >>>
> > >>> From: Eric Green <er...@gmail.com>
> > >>> Date: Friday, 10 January 2025 at 4:05 AM
> > >>> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > >>> Subject: A possible bug in instance start for end users when a host is
> > >>> deleted
> > >>> This appears to me to be a bug. Is it a known bug? I tried searching
> > the
> > >>> Github bug list but Github's search function is awful.
> > >>>
> > >>> Cloudstack 4.19.1.3, Ubuntu 20.04 management server, Ubuntu 22.04
> > compute
> > >>> hosts, KVM hypervisor.
> > >>>
> > >>> Scenario: There are four physical hosts, A, B, C, D.  Let's say they're
> > >>> host ID 62,63,64,65 in the hosts table.  Each host has at least one
> > >> virtual
> > >>> machine running on it.
> > >>>
> > >>> Shut down the virtual machines on host A (without starting them
> > >> elsewhere)
> > >>> which sets the last_host_id to 62, and delete host A in the
> > >> Infrastructure
> > >>> tab.
> > >>>
> > >>> As an end user (not an admin), try to start one of the virtual machines
> > >>> that were formerly on host A.
> > >>>
> > >>> End result: The deployment planner returns the last_host_id of host A
> > as
> > >>> the host to start the VM on. Since that host is marked deleted, the
> > >>> deployment executor removes it from the list of available hosts to
> > start
> > >>> the VM on, and then throws an exception. The exception then appears on
> > >> the
> > >>> GUI as a red box that basically gives the end user zero useful
> > >> information.
> > >>>
> > >>> Expected result:  The same as for admin users — the deployment planner
> > >>> notices that host A is deleted and cannot be used to deploy the VM, and
> > >>> instead picks a host from [B,C,D].
> > >>>
> > >>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> > >> (logid:086f0ba0)
> > >>> Adding pods to avoid lists for non-explicit VM deployment: []
> > >>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> > >> (logid:086f0ba0)
> > >>> Adding clusters to avoid lists for non-explicit VM deployment: []
> > >>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> > >> (logid:086f0ba0)
> > >>> Adding hosts to avoid lists for non-explicit VM deployment: []
> > >>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> > >> (logid:086f0ba0)
> > >>> DeploymentPlanner allocation algorithm: null
> > >>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> > >> (logid:086f0ba0)
> > >>> Trying to allocate a host and storage pools from dc:1,
> > >>> pod:null,cluster:null, requested cpu: 4000, requested ram: (4.00 GB)
> > >>> 4294967296
> > >>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> > >> (logid:086f0ba0)
> > >>> Is ROOT volume READY (pool already allocated)?: Yes
> > >>> 2025-01-09 13:28:35,466 DEBUG [c.c.a.ApiServer]
> > >>> (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) CIDRs
> > from
> > >>> which account 'Account
> > >>>
> > >>
> > [{"accountName":"eric.green","id":8,"uuid":"e8baeda0-90a5-4c97-aa51-0adbfa3c0e4c"}]'
> > >>> is allowed to perform API calls: 0.0.0.0/0,::/0
> > >>> 2025-01-09 <http://0.0.0.0/0,::/02025-01-09> 13:28:35,466 DEBUG
> > >>> [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500
> > >>> job-418252 ctx-34040330) (logid:086f0ba0) Deploy avoids pods: [],
> > >> clusters:
> > >>> [], hosts: []
> > >>> 2025-01-09 13:28:35,466 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> > >> (logid:086f0ba0)
> > >>> Deploy hosts with priorities {} , hosts have NORMAL priority by default
> > >>> 2025-01-09 13:28:35,467 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> > >> (logid:086f0ba0)
> > >>> This VM has last host_id specified, trying to choose the same host: 62
> > >>> 2025-01-09 13:28:35,471 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker]
> > >>> (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491)
> > >> RoleService
> > >>> is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
> > >>> 2025-01-09 13:28:35,471 DEBUG [o.a.c.r.ApiRateLimitServiceImpl]
> > >>> (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) API rate
> > >>> limiting is disabled. We will not use ApiRateLimitService.
> > >>> 2025-01-09 13:28:35,473 ERROR [c.c.a.ApiAsyncJobDispatcher]
> > >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252) (logid:086f0ba0)
> > Unexpected
> > >>> exception while executing
> > >>> org.apache.cloudstack.api.command.user.vm.StartVMCmd
> > >>> java.lang.NullPointerException
> > >>>        at
> > >>> com.cloud.host.dao.HostDaoImpl.loadHostTags(HostDaoImpl.java:879)
> > >>>        at
> > jdk.internal.reflect.GeneratedMethodAccessor336.invoke(Unknown
> > >>> Source)
> > >>>        at
> > >>>
> > >>
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >>>        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > >>>        at
> > >>>
> > >>
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> > >>>        at
> > >>>
> > >>
> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> > >>>        at
> > >>>
> > >>
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > >>>        at
> > >>>
> > >>
> > com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
> > >>>        at
> > >>>
> > >>
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
> > >>>        at
> > >>>
> > >>
> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> > >>>        at
> > >>>
> > >>
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> > >>>        at
> > >>>
> > >>
> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> > >>>        at com.sun.proxy.$Proxy96.loadHostTags(Unknown Source)
> > >>>        at
> > >>>
> > >>
> > com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:451)
> > >>>        at
> > >>>
> > >>
> > org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveVirtualMachine(VMEntityManagerImpl.java:206)
> > >>>        at
> > >>>
> > >>
> > org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.reserve(VirtualMachineEntityImpl.java:202)
> > >>>        at
> > >>>
> > >>
> > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5445)
> > >>>        at
> > >>>
> > >>
> > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5299)
> > >>>        at
> > >>>
> > >>
> > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3224)
> > >>>        at
> > jdk.internal.reflect.GeneratedMethodAccessor787.invoke(Unknown
> > >>> Source)
> > >>>        at
> > >>>
> > >>
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >>>        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > >>>        at
> > >>>
> > >>
> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> > >>>        at
> > >>>
> > >>
> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> > >>>        at
> > >>>
> > >>
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > >>>        at
> > >>>
> > >>
> > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> > >>>        at
> > >>>
> > >>
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
> > >>>        at
> > >>>
> > >>
> > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
> > >>>        at
> > >>>
> > >>
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
> > >>>        at
> > >>>
> > >>
> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> > >>>        at
> > >>>
> > >>
> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> > >>>        at
> > >>>
> > >>
> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> > >>>        at com.sun.proxy.$Proxy186.startVirtualMachine(Unknown Source)
> > >>>        at
> > >>>
> > >>
> > org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:181)
> > >>>        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
> > >>>        at
> > >>>
> > >>
> > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
> > >>>        at
> > >>>
> > >>
> > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
> > >>>        at
> > >>>
> > >>
> > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > >>>        at
> > >>>
> > >>
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > >>>        at
> > >>>
> > >>
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > >>>        at
> > >>>
> > >>
> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > >>>        at
> > >>>
> > >>
> > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > >>>        at
> > >>>
> > >>
> > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
> > >>>        at
> > >>>
> > >>
> > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > >>>        at
> > >>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > >>>        at
> > >>>
> > >>
> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > >>>        at
> > >>>
> > >>
> > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > >>>        at java.base/java.lang.Thread.run(Thread.java:829)
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> > >
> > >
> > > --
> > > Daan
> >
> >
> 
> -- 
> Daan
> 

Re: A possible bug in instance start for end users when a host is deleted

Posted by Daan Hoogland <da...@gmail.com>.
Makes sense Eric, I'll reapply this old PR on 4.19

On Fri, Jan 10, 2025 at 9:31 AM Eric Green <er...@gmail.com> wrote:

> I don’t know that it’s acceptable as a known bug for 4.19 unless the
> workaround that I did for my cluster (whack last_host_id in the database if
> it points at removed hosts) is documented.
>
> But honestly, any bug that requires manually whacking things in the
> database as a workaround probably needs to be fixed. Especially if it is in
> the n-1 release. I, and many other people using Cloudstack in production,
> don’t update to the very latest and greatest release, we stick back on the
> n-1 release, which has had more testing and bug fixing. Usually.
>
> > On Jan 10, 2025, at 12:22 AM, Daan Hoogland <da...@gmail.com>
> wrote:
> >
> > Wei, Eric,
> >
> > This sounds like we must reapply for 4.19, and make sure it doesn't get
> > pulled forward. The 4.20 code for this class has been heavily
> refactorred.
> > We could also accept this as a known bug for 4.19?
> >
> > On Fri, Jan 10, 2025 at 8:47 AM Eric Green <er...@gmail.com>
> wrote:
> >
> >> Thanks, Wei.  The log definitely stated that ‘considerlasthost’ was
> >> getting defaulted to true (and honestly that should be configurable
> because
> >> I really want the planner to start up the vm for users on whatever host
> has
> >> the most excess capacity at the moment), and that the so I was confused
> >> when Kiran claimed it was not. That’s not what the log said! (“This VM
> has
> >> last host_id specified, trying to choose the same host: 62”).
> >>
> >> So it appears that this is a bug that was fixed in 4.18.x, was
> supposedly
> >> pulled into 4.19.x, but somehow did not actually get pulled into 4.19.x.
> >> Since 4.20.x does not appear to have the bug, the solution appear to be
> to
> >> upgrade to 4.20.x before I delete another host 😊. (I do not plan to
> delete
> >> any hosts anytime soon, I just got all the old hosts retired).
> >>
> >> I resolved the issue in my own Cloudstack deployment by whacking
> >> last_host_id in the database to a current host if it pointed at a
> retired
> >> host. That allowed my users to start virtual machines last started on
> the
> >> retired hosts without involving an administrator. But it does appear
> that I
> >> happened upon a bug here, from what you say. I presume that since you
> know
> >> the exact cause, you will file an actual bug in Github that this fix has
> >> not been integrated to 4.19.x branch?
> >>
> >> From: Wei ZHOU <us...@gmail.com>
> >> Date: Thursday, January 9, 2025 at 11:27 PM
> >> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>, Daan
> >> Hoogland <da...@shapeblue.com>, Suresh.Anaparti@shapeblue.com <
> >> Suresh.Anaparti@shapeblue.com>
> >> Subject: Re: A possible bug in instance start for end users when a host
> is
> >> deleted
> >> Hi Kiran,
> >>
> >> For backward compatibility, the `considerlasthost` is default to true
> if it
> >> is not set.
> >>
> >> The issue seems to be fixed in 4.19.1.0 by
> >> https://github.com/apache/cloudstack/pull/9037
> >>
> >> However, I cannot find the changes in the code base of 4.19 branch.
> >>
> >>
> https://github.com/apache/cloudstack/blob/4.19/server/src/main/java/com/cloud/deploy/DeploymentPlanningManagerImpl.java#L449-L453
> >>
> >> It looks like something is wrong during the merge of the 4.18 to 4.19
> >> branch.
> >> cc @Daan Hoogland <da...@shapeblue.com>
> >> @Suresh.Anaparti@shapeblue.com <Su...@shapeblue.com>
> >>
> >>
> >> -Wei
> >>
> >> On Fri, Jan 10, 2025 at 7:58 AM Kiran Chavala <
> kiran.chavala@shapeblue.com
> >>>
> >> wrote:
> >>
> >>> Hi Eric
> >>>
> >>> I am not hitting the issue on the latest 4.20 release of cloudstack
> >>>
> >>> For a normal end user, it  doesn’t consider the last host to start the
> >>> virtual machine
> >>>
> >>> the startVirtualMachine Api call has the parameter “considerlasthost”
> >> for
> >>> the  admin user
> >>>
> >>>
> >>>
> >>
> https://cloudstack.apache.org/api/apidocs-4.20/apis/startVirtualMachine.html
> >>>
> >>> For a normal user there is no parameter “ considerlasthost” , the
> >>> deployment planner picks up the available host
> >>>
> >>> Normal user
> >>>
> >>> (localcloud2) 🐱 > start virtualmachine
> >>> bootintosetup= clusterid=     filter=        hostid=        id=
> >>> podid=
> >>>
> >>>
> >>> Regards
> >>> Kiran
> >>>
> >>> From: Eric Green <er...@gmail.com>
> >>> Date: Friday, 10 January 2025 at 4:05 AM
> >>> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> >>> Subject: A possible bug in instance start for end users when a host is
> >>> deleted
> >>> This appears to me to be a bug. Is it a known bug? I tried searching
> the
> >>> Github bug list but Github's search function is awful.
> >>>
> >>> Cloudstack 4.19.1.3, Ubuntu 20.04 management server, Ubuntu 22.04
> compute
> >>> hosts, KVM hypervisor.
> >>>
> >>> Scenario: There are four physical hosts, A, B, C, D.  Let's say they're
> >>> host ID 62,63,64,65 in the hosts table.  Each host has at least one
> >> virtual
> >>> machine running on it.
> >>>
> >>> Shut down the virtual machines on host A (without starting them
> >> elsewhere)
> >>> which sets the last_host_id to 62, and delete host A in the
> >> Infrastructure
> >>> tab.
> >>>
> >>> As an end user (not an admin), try to start one of the virtual machines
> >>> that were formerly on host A.
> >>>
> >>> End result: The deployment planner returns the last_host_id of host A
> as
> >>> the host to start the VM on. Since that host is marked deleted, the
> >>> deployment executor removes it from the list of available hosts to
> start
> >>> the VM on, and then throws an exception. The exception then appears on
> >> the
> >>> GUI as a red box that basically gives the end user zero useful
> >> information.
> >>>
> >>> Expected result:  The same as for admin users — the deployment planner
> >>> notices that host A is deleted and cannot be used to deploy the VM, and
> >>> instead picks a host from [B,C,D].
> >>>
> >>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> >> (logid:086f0ba0)
> >>> Adding pods to avoid lists for non-explicit VM deployment: []
> >>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> >> (logid:086f0ba0)
> >>> Adding clusters to avoid lists for non-explicit VM deployment: []
> >>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> >> (logid:086f0ba0)
> >>> Adding hosts to avoid lists for non-explicit VM deployment: []
> >>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> >> (logid:086f0ba0)
> >>> DeploymentPlanner allocation algorithm: null
> >>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> >> (logid:086f0ba0)
> >>> Trying to allocate a host and storage pools from dc:1,
> >>> pod:null,cluster:null, requested cpu: 4000, requested ram: (4.00 GB)
> >>> 4294967296
> >>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> >> (logid:086f0ba0)
> >>> Is ROOT volume READY (pool already allocated)?: Yes
> >>> 2025-01-09 13:28:35,466 DEBUG [c.c.a.ApiServer]
> >>> (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) CIDRs
> from
> >>> which account 'Account
> >>>
> >>
> [{"accountName":"eric.green","id":8,"uuid":"e8baeda0-90a5-4c97-aa51-0adbfa3c0e4c"}]'
> >>> is allowed to perform API calls: 0.0.0.0/0,::/0
> >>> 2025-01-09 <http://0.0.0.0/0,::/02025-01-09> 13:28:35,466 DEBUG
> >>> [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500
> >>> job-418252 ctx-34040330) (logid:086f0ba0) Deploy avoids pods: [],
> >> clusters:
> >>> [], hosts: []
> >>> 2025-01-09 13:28:35,466 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> >> (logid:086f0ba0)
> >>> Deploy hosts with priorities {} , hosts have NORMAL priority by default
> >>> 2025-01-09 13:28:35,467 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> >> (logid:086f0ba0)
> >>> This VM has last host_id specified, trying to choose the same host: 62
> >>> 2025-01-09 13:28:35,471 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker]
> >>> (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491)
> >> RoleService
> >>> is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
> >>> 2025-01-09 13:28:35,471 DEBUG [o.a.c.r.ApiRateLimitServiceImpl]
> >>> (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) API rate
> >>> limiting is disabled. We will not use ApiRateLimitService.
> >>> 2025-01-09 13:28:35,473 ERROR [c.c.a.ApiAsyncJobDispatcher]
> >>> (API-Job-Executor-81:ctx-4b9a2500 job-418252) (logid:086f0ba0)
> Unexpected
> >>> exception while executing
> >>> org.apache.cloudstack.api.command.user.vm.StartVMCmd
> >>> java.lang.NullPointerException
> >>>        at
> >>> com.cloud.host.dao.HostDaoImpl.loadHostTags(HostDaoImpl.java:879)
> >>>        at
> jdk.internal.reflect.GeneratedMethodAccessor336.invoke(Unknown
> >>> Source)
> >>>        at
> >>>
> >>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >>>        at
> >>>
> >>
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> >>>        at
> >>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> >>>        at
> >>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >>>        at
> >>>
> >>
> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
> >>>        at
> >>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
> >>>        at
> >>>
> >>
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> >>>        at
> >>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> >>>        at
> >>>
> >>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> >>>        at com.sun.proxy.$Proxy96.loadHostTags(Unknown Source)
> >>>        at
> >>>
> >>
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:451)
> >>>        at
> >>>
> >>
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveVirtualMachine(VMEntityManagerImpl.java:206)
> >>>        at
> >>>
> >>
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.reserve(VirtualMachineEntityImpl.java:202)
> >>>        at
> >>>
> >>
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5445)
> >>>        at
> >>>
> >>
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5299)
> >>>        at
> >>>
> >>
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3224)
> >>>        at
> jdk.internal.reflect.GeneratedMethodAccessor787.invoke(Unknown
> >>> Source)
> >>>        at
> >>>
> >>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >>>        at
> >>>
> >>
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> >>>        at
> >>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> >>>        at
> >>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >>>        at
> >>>
> >>
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> >>>        at
> >>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
> >>>        at
> >>>
> >>
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
> >>>        at
> >>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
> >>>        at
> >>>
> >>
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> >>>        at
> >>>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> >>>        at
> >>>
> >>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> >>>        at com.sun.proxy.$Proxy186.startVirtualMachine(Unknown Source)
> >>>        at
> >>>
> >>
> org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:181)
> >>>        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
> >>>        at
> >>>
> >>
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
> >>>        at
> >>>
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
> >>>        at
> >>>
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >>>        at
> >>>
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >>>        at
> >>>
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >>>        at
> >>>
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >>>        at
> >>>
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >>>        at
> >>>
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
> >>>        at
> >>>
> >>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >>>        at
> >>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >>>        at
> >>>
> >>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >>>        at
> >>>
> >>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >>>        at java.base/java.lang.Thread.run(Thread.java:829)
> >>>
> >>>
> >>>
> >>>
> >>
> >
> >
> > --
> > Daan
>
>

-- 
Daan

Re: A possible bug in instance start for end users when a host is deleted

Posted by Eric Green <er...@gmail.com>.
I don’t know that it’s acceptable as a known bug for 4.19 unless the workaround that I did for my cluster (whack last_host_id in the database if it points at removed hosts) is documented. 

But honestly, any bug that requires manually whacking things in the database as a workaround probably needs to be fixed. Especially if it is in the n-1 release. I, and many other people using Cloudstack in production, don’t update to the very latest and greatest release, we stick back on the n-1 release, which has had more testing and bug fixing. Usually. 

> On Jan 10, 2025, at 12:22 AM, Daan Hoogland <da...@gmail.com> wrote:
> 
> Wei, Eric,
> 
> This sounds like we must reapply for 4.19, and make sure it doesn't get
> pulled forward. The 4.20 code for this class has been heavily refactorred.
> We could also accept this as a known bug for 4.19?
> 
> On Fri, Jan 10, 2025 at 8:47 AM Eric Green <er...@gmail.com> wrote:
> 
>> Thanks, Wei.  The log definitely stated that ‘considerlasthost’ was
>> getting defaulted to true (and honestly that should be configurable because
>> I really want the planner to start up the vm for users on whatever host has
>> the most excess capacity at the moment), and that the so I was confused
>> when Kiran claimed it was not. That’s not what the log said! (“This VM has
>> last host_id specified, trying to choose the same host: 62”).
>> 
>> So it appears that this is a bug that was fixed in 4.18.x, was supposedly
>> pulled into 4.19.x, but somehow did not actually get pulled into 4.19.x.
>> Since 4.20.x does not appear to have the bug, the solution appear to be to
>> upgrade to 4.20.x before I delete another host 😊. (I do not plan to delete
>> any hosts anytime soon, I just got all the old hosts retired).
>> 
>> I resolved the issue in my own Cloudstack deployment by whacking
>> last_host_id in the database to a current host if it pointed at a retired
>> host. That allowed my users to start virtual machines last started on the
>> retired hosts without involving an administrator. But it does appear that I
>> happened upon a bug here, from what you say. I presume that since you know
>> the exact cause, you will file an actual bug in Github that this fix has
>> not been integrated to 4.19.x branch?
>> 
>> From: Wei ZHOU <us...@gmail.com>
>> Date: Thursday, January 9, 2025 at 11:27 PM
>> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>, Daan
>> Hoogland <da...@shapeblue.com>, Suresh.Anaparti@shapeblue.com <
>> Suresh.Anaparti@shapeblue.com>
>> Subject: Re: A possible bug in instance start for end users when a host is
>> deleted
>> Hi Kiran,
>> 
>> For backward compatibility, the `considerlasthost` is default to true if it
>> is not set.
>> 
>> The issue seems to be fixed in 4.19.1.0 by
>> https://github.com/apache/cloudstack/pull/9037
>> 
>> However, I cannot find the changes in the code base of 4.19 branch.
>> 
>> https://github.com/apache/cloudstack/blob/4.19/server/src/main/java/com/cloud/deploy/DeploymentPlanningManagerImpl.java#L449-L453
>> 
>> It looks like something is wrong during the merge of the 4.18 to 4.19
>> branch.
>> cc @Daan Hoogland <da...@shapeblue.com>
>> @Suresh.Anaparti@shapeblue.com <Su...@shapeblue.com>
>> 
>> 
>> -Wei
>> 
>> On Fri, Jan 10, 2025 at 7:58 AM Kiran Chavala <kiran.chavala@shapeblue.com
>>> 
>> wrote:
>> 
>>> Hi Eric
>>> 
>>> I am not hitting the issue on the latest 4.20 release of cloudstack
>>> 
>>> For a normal end user, it  doesn’t consider the last host to start the
>>> virtual machine
>>> 
>>> the startVirtualMachine Api call has the parameter “considerlasthost”
>> for
>>> the  admin user
>>> 
>>> 
>>> 
>> https://cloudstack.apache.org/api/apidocs-4.20/apis/startVirtualMachine.html
>>> 
>>> For a normal user there is no parameter “ considerlasthost” , the
>>> deployment planner picks up the available host
>>> 
>>> Normal user
>>> 
>>> (localcloud2) 🐱 > start virtualmachine
>>> bootintosetup= clusterid=     filter=        hostid=        id=
>>> podid=
>>> 
>>> 
>>> Regards
>>> Kiran
>>> 
>>> From: Eric Green <er...@gmail.com>
>>> Date: Friday, 10 January 2025 at 4:05 AM
>>> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>>> Subject: A possible bug in instance start for end users when a host is
>>> deleted
>>> This appears to me to be a bug. Is it a known bug? I tried searching the
>>> Github bug list but Github's search function is awful.
>>> 
>>> Cloudstack 4.19.1.3, Ubuntu 20.04 management server, Ubuntu 22.04 compute
>>> hosts, KVM hypervisor.
>>> 
>>> Scenario: There are four physical hosts, A, B, C, D.  Let's say they're
>>> host ID 62,63,64,65 in the hosts table.  Each host has at least one
>> virtual
>>> machine running on it.
>>> 
>>> Shut down the virtual machines on host A (without starting them
>> elsewhere)
>>> which sets the last_host_id to 62, and delete host A in the
>> Infrastructure
>>> tab.
>>> 
>>> As an end user (not an admin), try to start one of the virtual machines
>>> that were formerly on host A.
>>> 
>>> End result: The deployment planner returns the last_host_id of host A as
>>> the host to start the VM on. Since that host is marked deleted, the
>>> deployment executor removes it from the list of available hosts to start
>>> the VM on, and then throws an exception. The exception then appears on
>> the
>>> GUI as a red box that basically gives the end user zero useful
>> information.
>>> 
>>> Expected result:  The same as for admin users — the deployment planner
>>> notices that host A is deleted and cannot be used to deploy the VM, and
>>> instead picks a host from [B,C,D].
>>> 
>>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
>> (logid:086f0ba0)
>>> Adding pods to avoid lists for non-explicit VM deployment: []
>>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
>> (logid:086f0ba0)
>>> Adding clusters to avoid lists for non-explicit VM deployment: []
>>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
>> (logid:086f0ba0)
>>> Adding hosts to avoid lists for non-explicit VM deployment: []
>>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
>> (logid:086f0ba0)
>>> DeploymentPlanner allocation algorithm: null
>>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
>> (logid:086f0ba0)
>>> Trying to allocate a host and storage pools from dc:1,
>>> pod:null,cluster:null, requested cpu: 4000, requested ram: (4.00 GB)
>>> 4294967296
>>> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
>> (logid:086f0ba0)
>>> Is ROOT volume READY (pool already allocated)?: Yes
>>> 2025-01-09 13:28:35,466 DEBUG [c.c.a.ApiServer]
>>> (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) CIDRs from
>>> which account 'Account
>>> 
>> [{"accountName":"eric.green","id":8,"uuid":"e8baeda0-90a5-4c97-aa51-0adbfa3c0e4c"}]'
>>> is allowed to perform API calls: 0.0.0.0/0,::/0
>>> 2025-01-09 <http://0.0.0.0/0,::/02025-01-09> 13:28:35,466 DEBUG
>>> [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500
>>> job-418252 ctx-34040330) (logid:086f0ba0) Deploy avoids pods: [],
>> clusters:
>>> [], hosts: []
>>> 2025-01-09 13:28:35,466 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
>> (logid:086f0ba0)
>>> Deploy hosts with priorities {} , hosts have NORMAL priority by default
>>> 2025-01-09 13:28:35,467 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
>> (logid:086f0ba0)
>>> This VM has last host_id specified, trying to choose the same host: 62
>>> 2025-01-09 13:28:35,471 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker]
>>> (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491)
>> RoleService
>>> is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
>>> 2025-01-09 13:28:35,471 DEBUG [o.a.c.r.ApiRateLimitServiceImpl]
>>> (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) API rate
>>> limiting is disabled. We will not use ApiRateLimitService.
>>> 2025-01-09 13:28:35,473 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>> (API-Job-Executor-81:ctx-4b9a2500 job-418252) (logid:086f0ba0) Unexpected
>>> exception while executing
>>> org.apache.cloudstack.api.command.user.vm.StartVMCmd
>>> java.lang.NullPointerException
>>>        at
>>> com.cloud.host.dao.HostDaoImpl.loadHostTags(HostDaoImpl.java:879)
>>>        at jdk.internal.reflect.GeneratedMethodAccessor336.invoke(Unknown
>>> Source)
>>>        at
>>> 
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>        at
>>> 
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>>>        at
>>> 
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>>>        at
>>> 
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>>>        at
>>> 
>> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
>>>        at
>>> 
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
>>>        at
>>> 
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
>>>        at
>>> 
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>>>        at
>>> 
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
>>>        at com.sun.proxy.$Proxy96.loadHostTags(Unknown Source)
>>>        at
>>> 
>> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:451)
>>>        at
>>> 
>> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveVirtualMachine(VMEntityManagerImpl.java:206)
>>>        at
>>> 
>> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.reserve(VirtualMachineEntityImpl.java:202)
>>>        at
>>> 
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5445)
>>>        at
>>> 
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5299)
>>>        at
>>> 
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3224)
>>>        at jdk.internal.reflect.GeneratedMethodAccessor787.invoke(Unknown
>>> Source)
>>>        at
>>> 
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>        at
>>> 
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>>>        at
>>> 
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>>>        at
>>> 
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>>>        at
>>> 
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>>>        at
>>> 
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
>>>        at
>>> 
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
>>>        at
>>> 
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
>>>        at
>>> 
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
>>>        at
>>> 
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>>>        at
>>> 
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
>>>        at com.sun.proxy.$Proxy186.startVirtualMachine(Unknown Source)
>>>        at
>>> 
>> org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:181)
>>>        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
>>>        at
>>> 
>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
>>>        at
>>> 
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
>>>        at
>>> 
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>>        at
>>> 
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>>        at
>>> 
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>>        at
>>> 
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>>        at
>>> 
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>>        at
>>> 
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
>>>        at
>>> 
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>>        at
>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>>        at
>>> 
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>        at
>>> 
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>        at java.base/java.lang.Thread.run(Thread.java:829)
>>> 
>>> 
>>> 
>>> 
>> 
> 
> 
> -- 
> Daan


Re: A possible bug in instance start for end users when a host is deleted

Posted by Daan Hoogland <da...@gmail.com>.
Wei, Eric,

This sounds like we must reapply for 4.19, and make sure it doesn't get
pulled forward. The 4.20 code for this class has been heavily refactorred.
We could also accept this as a known bug for 4.19?

On Fri, Jan 10, 2025 at 8:47 AM Eric Green <er...@gmail.com> wrote:

> Thanks, Wei.  The log definitely stated that ‘considerlasthost’ was
> getting defaulted to true (and honestly that should be configurable because
> I really want the planner to start up the vm for users on whatever host has
> the most excess capacity at the moment), and that the so I was confused
> when Kiran claimed it was not. That’s not what the log said! (“This VM has
> last host_id specified, trying to choose the same host: 62”).
>
> So it appears that this is a bug that was fixed in 4.18.x, was supposedly
> pulled into 4.19.x, but somehow did not actually get pulled into 4.19.x.
> Since 4.20.x does not appear to have the bug, the solution appear to be to
> upgrade to 4.20.x before I delete another host 😊. (I do not plan to delete
> any hosts anytime soon, I just got all the old hosts retired).
>
> I resolved the issue in my own Cloudstack deployment by whacking
> last_host_id in the database to a current host if it pointed at a retired
> host. That allowed my users to start virtual machines last started on the
> retired hosts without involving an administrator. But it does appear that I
> happened upon a bug here, from what you say. I presume that since you know
> the exact cause, you will file an actual bug in Github that this fix has
> not been integrated to 4.19.x branch?
>
> From: Wei ZHOU <us...@gmail.com>
> Date: Thursday, January 9, 2025 at 11:27 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>, Daan
> Hoogland <da...@shapeblue.com>, Suresh.Anaparti@shapeblue.com <
> Suresh.Anaparti@shapeblue.com>
> Subject: Re: A possible bug in instance start for end users when a host is
> deleted
> Hi Kiran,
>
> For backward compatibility, the `considerlasthost` is default to true if it
> is not set.
>
> The issue seems to be fixed in 4.19.1.0 by
> https://github.com/apache/cloudstack/pull/9037
>
> However, I cannot find the changes in the code base of 4.19 branch.
>
> https://github.com/apache/cloudstack/blob/4.19/server/src/main/java/com/cloud/deploy/DeploymentPlanningManagerImpl.java#L449-L453
>
> It looks like something is wrong during the merge of the 4.18 to 4.19
> branch.
> cc @Daan Hoogland <da...@shapeblue.com>
> @Suresh.Anaparti@shapeblue.com <Su...@shapeblue.com>
>
>
> -Wei
>
> On Fri, Jan 10, 2025 at 7:58 AM Kiran Chavala <kiran.chavala@shapeblue.com
> >
> wrote:
>
> > Hi Eric
> >
> > I am not hitting the issue on the latest 4.20 release of cloudstack
> >
> > For a normal end user, it  doesn’t consider the last host to start the
> > virtual machine
> >
> > the startVirtualMachine Api call has the parameter “considerlasthost”
> for
> > the  admin user
> >
> >
> >
> https://cloudstack.apache.org/api/apidocs-4.20/apis/startVirtualMachine.html
> >
> > For a normal user there is no parameter “ considerlasthost” , the
> > deployment planner picks up the available host
> >
> > Normal user
> >
> > (localcloud2) 🐱 > start virtualmachine
> > bootintosetup= clusterid=     filter=        hostid=        id=
> > podid=
> >
> >
> > Regards
> > Kiran
> >
> > From: Eric Green <er...@gmail.com>
> > Date: Friday, 10 January 2025 at 4:05 AM
> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > Subject: A possible bug in instance start for end users when a host is
> > deleted
> > This appears to me to be a bug. Is it a known bug? I tried searching the
> > Github bug list but Github's search function is awful.
> >
> > Cloudstack 4.19.1.3, Ubuntu 20.04 management server, Ubuntu 22.04 compute
> > hosts, KVM hypervisor.
> >
> > Scenario: There are four physical hosts, A, B, C, D.  Let's say they're
> > host ID 62,63,64,65 in the hosts table.  Each host has at least one
> virtual
> > machine running on it.
> >
> > Shut down the virtual machines on host A (without starting them
> elsewhere)
> > which sets the last_host_id to 62, and delete host A in the
> Infrastructure
> > tab.
> >
> > As an end user (not an admin), try to start one of the virtual machines
> > that were formerly on host A.
> >
> > End result: The deployment planner returns the last_host_id of host A as
> > the host to start the VM on. Since that host is marked deleted, the
> > deployment executor removes it from the list of available hosts to start
> > the VM on, and then throws an exception. The exception then appears on
> the
> > GUI as a red box that basically gives the end user zero useful
> information.
> >
> > Expected result:  The same as for admin users — the deployment planner
> > notices that host A is deleted and cannot be used to deploy the VM, and
> > instead picks a host from [B,C,D].
> >
> > 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> (logid:086f0ba0)
> > Adding pods to avoid lists for non-explicit VM deployment: []
> > 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> (logid:086f0ba0)
> > Adding clusters to avoid lists for non-explicit VM deployment: []
> > 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> (logid:086f0ba0)
> > Adding hosts to avoid lists for non-explicit VM deployment: []
> > 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> (logid:086f0ba0)
> > DeploymentPlanner allocation algorithm: null
> > 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> (logid:086f0ba0)
> > Trying to allocate a host and storage pools from dc:1,
> > pod:null,cluster:null, requested cpu: 4000, requested ram: (4.00 GB)
> > 4294967296
> > 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> (logid:086f0ba0)
> > Is ROOT volume READY (pool already allocated)?: Yes
> > 2025-01-09 13:28:35,466 DEBUG [c.c.a.ApiServer]
> > (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) CIDRs from
> > which account 'Account
> >
> [{"accountName":"eric.green","id":8,"uuid":"e8baeda0-90a5-4c97-aa51-0adbfa3c0e4c"}]'
> > is allowed to perform API calls: 0.0.0.0/0,::/0
> > 2025-01-09 <http://0.0.0.0/0,::/02025-01-09> 13:28:35,466 DEBUG
> > [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500
> > job-418252 ctx-34040330) (logid:086f0ba0) Deploy avoids pods: [],
> clusters:
> > [], hosts: []
> > 2025-01-09 13:28:35,466 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> (logid:086f0ba0)
> > Deploy hosts with priorities {} , hosts have NORMAL priority by default
> > 2025-01-09 13:28:35,467 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330)
> (logid:086f0ba0)
> > This VM has last host_id specified, trying to choose the same host: 62
> > 2025-01-09 13:28:35,471 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker]
> > (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491)
> RoleService
> > is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
> > 2025-01-09 13:28:35,471 DEBUG [o.a.c.r.ApiRateLimitServiceImpl]
> > (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) API rate
> > limiting is disabled. We will not use ApiRateLimitService.
> > 2025-01-09 13:28:35,473 ERROR [c.c.a.ApiAsyncJobDispatcher]
> > (API-Job-Executor-81:ctx-4b9a2500 job-418252) (logid:086f0ba0) Unexpected
> > exception while executing
> > org.apache.cloudstack.api.command.user.vm.StartVMCmd
> > java.lang.NullPointerException
> >         at
> > com.cloud.host.dao.HostDaoImpl.loadHostTags(HostDaoImpl.java:879)
> >         at jdk.internal.reflect.GeneratedMethodAccessor336.invoke(Unknown
> > Source)
> >         at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >         at
> >
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> >         at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> >         at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >         at
> >
> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
> >         at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
> >         at
> >
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> >         at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> >         at
> >
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> >         at com.sun.proxy.$Proxy96.loadHostTags(Unknown Source)
> >         at
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:451)
> >         at
> >
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveVirtualMachine(VMEntityManagerImpl.java:206)
> >         at
> >
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.reserve(VirtualMachineEntityImpl.java:202)
> >         at
> >
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5445)
> >         at
> >
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5299)
> >         at
> >
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3224)
> >         at jdk.internal.reflect.GeneratedMethodAccessor787.invoke(Unknown
> > Source)
> >         at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >         at
> >
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> >         at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> >         at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >         at
> >
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> >         at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
> >         at
> >
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
> >         at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
> >         at
> >
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> >         at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> >         at
> >
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> >         at com.sun.proxy.$Proxy186.startVirtualMachine(Unknown Source)
> >         at
> >
> org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:181)
> >         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
> >         at
> >
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
> >         at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
> >         at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >         at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >         at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >         at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >         at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >         at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
> >         at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >         at
> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >         at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >         at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >         at java.base/java.lang.Thread.run(Thread.java:829)
> >
> >
> >
> >
>


-- 
Daan

Re: A possible bug in instance start for end users when a host is deleted

Posted by Eric Green <er...@gmail.com>.
Thanks, Wei.  The log definitely stated that ‘considerlasthost’ was getting defaulted to true (and honestly that should be configurable because I really want the planner to start up the vm for users on whatever host has the most excess capacity at the moment), and that the so I was confused when Kiran claimed it was not. That’s not what the log said! (“This VM has last host_id specified, trying to choose the same host: 62”).

So it appears that this is a bug that was fixed in 4.18.x, was supposedly pulled into 4.19.x, but somehow did not actually get pulled into 4.19.x. Since 4.20.x does not appear to have the bug, the solution appear to be to upgrade to 4.20.x before I delete another host 😊. (I do not plan to delete any hosts anytime soon, I just got all the old hosts retired).

I resolved the issue in my own Cloudstack deployment by whacking last_host_id in the database to a current host if it pointed at a retired host. That allowed my users to start virtual machines last started on the retired hosts without involving an administrator. But it does appear that I happened upon a bug here, from what you say. I presume that since you know the exact cause, you will file an actual bug in Github that this fix has not been integrated to 4.19.x branch?

From: Wei ZHOU <us...@gmail.com>
Date: Thursday, January 9, 2025 at 11:27 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>, Daan Hoogland <da...@shapeblue.com>, Suresh.Anaparti@shapeblue.com <Su...@shapeblue.com>
Subject: Re: A possible bug in instance start for end users when a host is deleted
Hi Kiran,

For backward compatibility, the `considerlasthost` is default to true if it
is not set.

The issue seems to be fixed in 4.19.1.0 by
https://github.com/apache/cloudstack/pull/9037

However, I cannot find the changes in the code base of 4.19 branch.
https://github.com/apache/cloudstack/blob/4.19/server/src/main/java/com/cloud/deploy/DeploymentPlanningManagerImpl.java#L449-L453

It looks like something is wrong during the merge of the 4.18 to 4.19
branch.
cc @Daan Hoogland <da...@shapeblue.com>
@Suresh.Anaparti@shapeblue.com <Su...@shapeblue.com>


-Wei

On Fri, Jan 10, 2025 at 7:58 AM Kiran Chavala <ki...@shapeblue.com>
wrote:

> Hi Eric
>
> I am not hitting the issue on the latest 4.20 release of cloudstack
>
> For a normal end user, it  doesn’t consider the last host to start the
> virtual machine
>
> the startVirtualMachine Api call has the parameter “considerlasthost”  for
> the  admin user
>
>
> https://cloudstack.apache.org/api/apidocs-4.20/apis/startVirtualMachine.html
>
> For a normal user there is no parameter “ considerlasthost” , the
> deployment planner picks up the available host
>
> Normal user
>
> (localcloud2) 🐱 > start virtualmachine
> bootintosetup= clusterid=     filter=        hostid=        id=
> podid=
>
>
> Regards
> Kiran
>
> From: Eric Green <er...@gmail.com>
> Date: Friday, 10 January 2025 at 4:05 AM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: A possible bug in instance start for end users when a host is
> deleted
> This appears to me to be a bug. Is it a known bug? I tried searching the
> Github bug list but Github's search function is awful.
>
> Cloudstack 4.19.1.3, Ubuntu 20.04 management server, Ubuntu 22.04 compute
> hosts, KVM hypervisor.
>
> Scenario: There are four physical hosts, A, B, C, D.  Let's say they're
> host ID 62,63,64,65 in the hosts table.  Each host has at least one virtual
> machine running on it.
>
> Shut down the virtual machines on host A (without starting them elsewhere)
> which sets the last_host_id to 62, and delete host A in the Infrastructure
> tab.
>
> As an end user (not an admin), try to start one of the virtual machines
> that were formerly on host A.
>
> End result: The deployment planner returns the last_host_id of host A as
> the host to start the VM on. Since that host is marked deleted, the
> deployment executor removes it from the list of available hosts to start
> the VM on, and then throws an exception. The exception then appears on the
> GUI as a red box that basically gives the end user zero useful information.
>
> Expected result:  The same as for admin users — the deployment planner
> notices that host A is deleted and cannot be used to deploy the VM, and
> instead picks a host from [B,C,D].
>
> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0)
> Adding pods to avoid lists for non-explicit VM deployment: []
> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0)
> Adding clusters to avoid lists for non-explicit VM deployment: []
> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0)
> Adding hosts to avoid lists for non-explicit VM deployment: []
> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0)
> DeploymentPlanner allocation algorithm: null
> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0)
> Trying to allocate a host and storage pools from dc:1,
> pod:null,cluster:null, requested cpu: 4000, requested ram: (4.00 GB)
> 4294967296
> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0)
> Is ROOT volume READY (pool already allocated)?: Yes
> 2025-01-09 13:28:35,466 DEBUG [c.c.a.ApiServer]
> (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) CIDRs from
> which account 'Account
> [{"accountName":"eric.green","id":8,"uuid":"e8baeda0-90a5-4c97-aa51-0adbfa3c0e4c"}]'
> is allowed to perform API calls: 0.0.0.0/0,::/0
> 2025-01-09 <http://0.0.0.0/0,::/02025-01-09> 13:28:35,466 DEBUG
> [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500
> job-418252 ctx-34040330) (logid:086f0ba0) Deploy avoids pods: [], clusters:
> [], hosts: []
> 2025-01-09 13:28:35,466 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0)
> Deploy hosts with priorities {} , hosts have NORMAL priority by default
> 2025-01-09 13:28:35,467 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0)
> This VM has last host_id specified, trying to choose the same host: 62
> 2025-01-09 13:28:35,471 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker]
> (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) RoleService
> is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
> 2025-01-09 13:28:35,471 DEBUG [o.a.c.r.ApiRateLimitServiceImpl]
> (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) API rate
> limiting is disabled. We will not use ApiRateLimitService.
> 2025-01-09 13:28:35,473 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-81:ctx-4b9a2500 job-418252) (logid:086f0ba0) Unexpected
> exception while executing
> org.apache.cloudstack.api.command.user.vm.StartVMCmd
> java.lang.NullPointerException
>         at
> com.cloud.host.dao.HostDaoImpl.loadHostTags(HostDaoImpl.java:879)
>         at jdk.internal.reflect.GeneratedMethodAccessor336.invoke(Unknown
> Source)
>         at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>         at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>         at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>         at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>         at
> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
>         at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
>         at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
>         at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>         at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
>         at com.sun.proxy.$Proxy96.loadHostTags(Unknown Source)
>         at
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:451)
>         at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveVirtualMachine(VMEntityManagerImpl.java:206)
>         at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.reserve(VirtualMachineEntityImpl.java:202)
>         at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5445)
>         at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5299)
>         at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3224)
>         at jdk.internal.reflect.GeneratedMethodAccessor787.invoke(Unknown
> Source)
>         at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>         at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>         at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>         at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>         at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>         at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
>         at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
>         at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
>         at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
>         at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>         at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
>         at com.sun.proxy.$Proxy186.startVirtualMachine(Unknown Source)
>         at
> org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:181)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
>         at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
>         at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
>         at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>         at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>         at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>         at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>         at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>         at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
>         at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>         at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>         at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>         at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>         at java.base/java.lang.Thread.run(Thread.java:829)
>
>
>
>

Re: A possible bug in instance start for end users when a host is deleted

Posted by Wei ZHOU <us...@gmail.com>.
Hi Kiran,

For backward compatibility, the `considerlasthost` is default to true if it
is not set.

The issue seems to be fixed in 4.19.1.0 by
https://github.com/apache/cloudstack/pull/9037

However, I cannot find the changes in the code base of 4.19 branch.
https://github.com/apache/cloudstack/blob/4.19/server/src/main/java/com/cloud/deploy/DeploymentPlanningManagerImpl.java#L449-L453

It looks like something is wrong during the merge of the 4.18 to 4.19
branch.
cc @Daan Hoogland <da...@shapeblue.com>
@Suresh.Anaparti@shapeblue.com <Su...@shapeblue.com>


-Wei

On Fri, Jan 10, 2025 at 7:58 AM Kiran Chavala <ki...@shapeblue.com>
wrote:

> Hi Eric
>
> I am not hitting the issue on the latest 4.20 release of cloudstack
>
> For a normal end user, it  doesn’t consider the last host to start the
> virtual machine
>
> the startVirtualMachine Api call has the parameter “considerlasthost”  for
> the  admin user
>
>
> https://cloudstack.apache.org/api/apidocs-4.20/apis/startVirtualMachine.html
>
> For a normal user there is no parameter “ considerlasthost” , the
> deployment planner picks up the available host
>
> Normal user
>
> (localcloud2) 🐱 > start virtualmachine
> bootintosetup= clusterid=     filter=        hostid=        id=
> podid=
>
>
> Regards
> Kiran
>
> From: Eric Green <er...@gmail.com>
> Date: Friday, 10 January 2025 at 4:05 AM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: A possible bug in instance start for end users when a host is
> deleted
> This appears to me to be a bug. Is it a known bug? I tried searching the
> Github bug list but Github's search function is awful.
>
> Cloudstack 4.19.1.3, Ubuntu 20.04 management server, Ubuntu 22.04 compute
> hosts, KVM hypervisor.
>
> Scenario: There are four physical hosts, A, B, C, D.  Let's say they're
> host ID 62,63,64,65 in the hosts table.  Each host has at least one virtual
> machine running on it.
>
> Shut down the virtual machines on host A (without starting them elsewhere)
> which sets the last_host_id to 62, and delete host A in the Infrastructure
> tab.
>
> As an end user (not an admin), try to start one of the virtual machines
> that were formerly on host A.
>
> End result: The deployment planner returns the last_host_id of host A as
> the host to start the VM on. Since that host is marked deleted, the
> deployment executor removes it from the list of available hosts to start
> the VM on, and then throws an exception. The exception then appears on the
> GUI as a red box that basically gives the end user zero useful information.
>
> Expected result:  The same as for admin users — the deployment planner
> notices that host A is deleted and cannot be used to deploy the VM, and
> instead picks a host from [B,C,D].
>
> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0)
> Adding pods to avoid lists for non-explicit VM deployment: []
> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0)
> Adding clusters to avoid lists for non-explicit VM deployment: []
> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0)
> Adding hosts to avoid lists for non-explicit VM deployment: []
> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0)
> DeploymentPlanner allocation algorithm: null
> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0)
> Trying to allocate a host and storage pools from dc:1,
> pod:null,cluster:null, requested cpu: 4000, requested ram: (4.00 GB)
> 4294967296
> 2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0)
> Is ROOT volume READY (pool already allocated)?: Yes
> 2025-01-09 13:28:35,466 DEBUG [c.c.a.ApiServer]
> (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) CIDRs from
> which account 'Account
> [{"accountName":"eric.green","id":8,"uuid":"e8baeda0-90a5-4c97-aa51-0adbfa3c0e4c"}]'
> is allowed to perform API calls: 0.0.0.0/0,::/0
> 2025-01-09 <http://0.0.0.0/0,::/02025-01-09> 13:28:35,466 DEBUG
> [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500
> job-418252 ctx-34040330) (logid:086f0ba0) Deploy avoids pods: [], clusters:
> [], hosts: []
> 2025-01-09 13:28:35,466 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0)
> Deploy hosts with priorities {} , hosts have NORMAL priority by default
> 2025-01-09 13:28:35,467 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0)
> This VM has last host_id specified, trying to choose the same host: 62
> 2025-01-09 13:28:35,471 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker]
> (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) RoleService
> is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
> 2025-01-09 13:28:35,471 DEBUG [o.a.c.r.ApiRateLimitServiceImpl]
> (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) API rate
> limiting is disabled. We will not use ApiRateLimitService.
> 2025-01-09 13:28:35,473 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-81:ctx-4b9a2500 job-418252) (logid:086f0ba0) Unexpected
> exception while executing
> org.apache.cloudstack.api.command.user.vm.StartVMCmd
> java.lang.NullPointerException
>         at
> com.cloud.host.dao.HostDaoImpl.loadHostTags(HostDaoImpl.java:879)
>         at jdk.internal.reflect.GeneratedMethodAccessor336.invoke(Unknown
> Source)
>         at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>         at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>         at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>         at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>         at
> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
>         at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
>         at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
>         at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>         at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
>         at com.sun.proxy.$Proxy96.loadHostTags(Unknown Source)
>         at
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:451)
>         at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveVirtualMachine(VMEntityManagerImpl.java:206)
>         at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.reserve(VirtualMachineEntityImpl.java:202)
>         at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5445)
>         at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5299)
>         at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3224)
>         at jdk.internal.reflect.GeneratedMethodAccessor787.invoke(Unknown
> Source)
>         at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>         at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>         at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>         at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>         at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>         at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
>         at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
>         at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
>         at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
>         at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>         at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
>         at com.sun.proxy.$Proxy186.startVirtualMachine(Unknown Source)
>         at
> org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:181)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
>         at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
>         at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
>         at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>         at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>         at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>         at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>         at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>         at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
>         at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>         at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>         at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>         at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>         at java.base/java.lang.Thread.run(Thread.java:829)
>
>
>
>

Re: A possible bug in instance start for end users when a host is deleted

Posted by Kiran Chavala <ki...@shapeblue.com>.
Hi Eric

I am not hitting the issue on the latest 4.20 release of cloudstack

For a normal end user, it  doesn’t consider the last host to start the virtual machine

the startVirtualMachine Api call has the parameter “considerlasthost”  for the  admin user

https://cloudstack.apache.org/api/apidocs-4.20/apis/startVirtualMachine.html

For a normal user there is no parameter “ considerlasthost” , the deployment planner picks up the available host

Normal user

(localcloud2) 🐱 > start virtualmachine
bootintosetup= clusterid=     filter=        hostid=        id=            podid=


Regards
Kiran

From: Eric Green <er...@gmail.com>
Date: Friday, 10 January 2025 at 4:05 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: A possible bug in instance start for end users when a host is deleted
This appears to me to be a bug. Is it a known bug? I tried searching the Github bug list but Github's search function is awful.

Cloudstack 4.19.1.3, Ubuntu 20.04 management server, Ubuntu 22.04 compute hosts, KVM hypervisor.

Scenario: There are four physical hosts, A, B, C, D.  Let's say they're host ID 62,63,64,65 in the hosts table.  Each host has at least one virtual machine running on it.

Shut down the virtual machines on host A (without starting them elsewhere) which sets the last_host_id to 62, and delete host A in the Infrastructure tab.

As an end user (not an admin), try to start one of the virtual machines that were formerly on host A.

End result: The deployment planner returns the last_host_id of host A as the host to start the VM on. Since that host is marked deleted, the deployment executor removes it from the list of available hosts to start the VM on, and then throws an exception. The exception then appears on the GUI as a red box that basically gives the end user zero useful information.

Expected result:  The same as for admin users — the deployment planner notices that host A is deleted and cannot be used to deploy the VM, and instead picks a host from [B,C,D].

2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0) Adding pods to avoid lists for non-explicit VM deployment: []
2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0) Adding clusters to avoid lists for non-explicit VM deployment: []
2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0) Adding hosts to avoid lists for non-explicit VM deployment: []
2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0) DeploymentPlanner allocation algorithm: null
2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0) Trying to allocate a host and storage pools from dc:1, pod:null,cluster:null, requested cpu: 4000, requested ram: (4.00 GB) 4294967296
2025-01-09 13:28:35,463 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0) Is ROOT volume READY (pool already allocated)?: Yes
2025-01-09 13:28:35,466 DEBUG [c.c.a.ApiServer] (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) CIDRs from which account 'Account [{"accountName":"eric.green","id":8,"uuid":"e8baeda0-90a5-4c97-aa51-0adbfa3c0e4c"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-01-09 13:28:35,466 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0) Deploy avoids pods: [], clusters: [], hosts: []
2025-01-09 13:28:35,466 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0) Deploy hosts with priorities {} , hosts have NORMAL priority by default
2025-01-09 13:28:35,467 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-81:ctx-4b9a2500 job-418252 ctx-34040330) (logid:086f0ba0) This VM has last host_id specified, trying to choose the same host: 62
2025-01-09 13:28:35,471 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-01-09 13:28:35,471 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp364604394-5358:ctx-c649c5e4 ctx-618420eb) (logid:14ea3491) API rate limiting is disabled. We will not use ApiRateLimitService.
2025-01-09 13:28:35,473 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-81:ctx-4b9a2500 job-418252) (logid:086f0ba0) Unexpected exception while executing org.apache.cloudstack.api.command.user.vm.StartVMCmd
java.lang.NullPointerException
        at com.cloud.host.dao.HostDaoImpl.loadHostTags(HostDaoImpl.java:879)
        at jdk.internal.reflect.GeneratedMethodAccessor336.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy96.loadHostTags(Unknown Source)
        at com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:451)
        at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveVirtualMachine(VMEntityManagerImpl.java:206)
        at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.reserve(VirtualMachineEntityImpl.java:202)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5445)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5299)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3224)
        at jdk.internal.reflect.GeneratedMethodAccessor787.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy186.startVirtualMachine(Unknown Source)
        at org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:181)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)