You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "yashi4engg (via GitHub)" <gi...@apache.org> on 2024/02/08 16:17:28 UTC
[I] DRS failed for cluster if have another cluster in zone [cloudstack]
yashi4engg opened a new issue, #8629:
URL: https://github.com/apache/cloudstack/issues/8629
<!--
DRS failed for cluster if have another cluster in zone
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
cloudstack 4.19 , KVM hypervisors with OEL9 ,
-->
~~~
~~~
##### CLOUDSTACK VERSION
<!--
cloudstack 4.19
-->
~~~
~~~
##### CONFIGURATION
<!--
Configured 2 zones with 2 clusters in each zone.
-->
##### OS / ENVIRONMENT
<!--
OEL9
-->
##### SUMMARY
We have zone1 where we have two clusters ex- cluster01 and cluster02 ,
Where cluster01 have 3 hypervisors with same h/w model and cluster02 has one hypervisor with dfifferent h/w model.
I have enabled drs setiings in globals setting and then disbaled it for cluster02 in cluster setting "drs.automatic.enable -- false" but enabled for cluster01 .
In the above scenario drs plan failed with below logs -
2024-02-08 10:26:33,532 DEBUG [c.c.s.ManagementServerImpl] (VMSchedulerPollTask:ctx-0bba1590) (logid:f0f7966b) Hosts having capacity and suitable for migration: [Host {"id":25,"name":"node-cluster01","type":"Routing","uuid":"5d145861-e4ad-4f94-a805-266711321d59"}, Host {"id":40,"name":"node-cluster02","type":"Routing","uuid":"66b47d2a-b047-452c-b5d3-65c160666b50"}, Host {"id":48,"name":"node-cluster01","type":"Routing","uuid":"da1be373-cd08-4ae1-948f-7592daabb3fc"}]
2024-02-08 10:26:33,535 ERROR [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-0bba1590) (logid:f0f7966b) Unable to generate DRS plans for cluster Cluster-Z01 [id=5366c5fb-0ed0-4caf-b2c7-93ebea15a717]
If i disable host from cluster02 our drs works as expected and migrated VMs based on load . But when we have both clusters and all nodes in cluster enabled its failed to generate plan.
Below are cluster level settings for reference from DB.
621 | 1 | drs.automatic.enable | true |
| 622 | 1 | drs.automatic.interval | 10 |
| 623 | 1 | drs.imbalance | 0.4 |
| 624 | 16 | drs.automatic.enable | false
Even tried by keep it disabled in global settings and then enable it for just one cluster in cluster settings.
As our use case is where we have multiple clusters in same zone with different type of h/w.
##### STEPS TO REPRODUCE
<!--
Can try be creating 2 clusters with differnet type of h/w.
-->
##### EXPECTED RESULTS
<!-- it should work for cluster if enabled for cluster and should not tried to include other cluster node in drs plan -->
~~~
~~~
~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DRS failed for cluster if have another cluster in zone [cloudstack]
Posted by "vishesh92 (via GitHub)" <gi...@apache.org>.
vishesh92 commented on issue #8629:
URL: https://github.com/apache/cloudstack/issues/8629#issuecomment-1939391012
@yashi4engg The error log you have shared seems to be truncated. Can you share the complete error log?
Also, a cloudstack cluster supports only one hypervisor. It's possible the error is happening because of your current setup. The complete error logs might help in identifying the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DRS failed for cluster if have another cluster in zone [cloudstack]
Posted by "rohityadavcloud (via GitHub)" <gi...@apache.org>.
rohityadavcloud commented on issue #8629:
URL: https://github.com/apache/cloudstack/issues/8629#issuecomment-1934635551
cc @vishesh92
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DRS failed for cluster if have another cluster in zone [cloudstack]
Posted by "yashi4engg (via GitHub)" <gi...@apache.org>.
yashi4engg commented on issue #8629:
URL: https://github.com/apache/cloudstack/issues/8629#issuecomment-1943911770
We were able to fix above errors and good to close this.
Now DRS is working as expected for us . It was due to zone level storage pool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DRS failed for cluster if have another cluster in zone [cloudstack]
Posted by "yashi4engg (via GitHub)" <gi...@apache.org>.
yashi4engg commented on issue #8629:
URL: https://github.com/apache/cloudstack/issues/8629#issuecomment-2059466006
@vishesh92 -- it works fine where we have primary storage as NFS ..But its not working where we are using OCFS2 FS as primary storage pool in cluster.Even while generating DRS plan manually i am not seeing error it shows success but not getting any plan .
below are logs -
[root@test01 ~]# cat /var/log/cloudstack/management/management-server.log |grep -i edcf30ae
2024-04-16 10:43:57,568 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-cf16f762) (logid:edcf30ae) ClusterDRS.poll is being called at 2024-04-16 14:44:00 GMT
2024-04-16 10:43:57,579 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-cf16f762) (logid:edcf30ae) Removed 0 old drs migration plans
[root@test01 ~]# cat /var/log/cloudstack/management/management-server.log |grep -i edcf30ae
2024-04-16 10:43:57,568 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-cf16f762) (logid:edcf30ae) ClusterDRS.poll is being called at 2024-04-16 14:44:00 GMT
2024-04-16 10:43:57,579 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-cf16f762) (logid:edcf30ae) Removed 0 old drs migration plans
where as in my cluster VM count is like, i set drs.imbalance to 1.0 as well -
node1 - 150VM
node2 - 150 VM
node3 - 1 VM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DRS failed for cluster if have another cluster in zone [cloudstack]
Posted by "yashi4engg (via GitHub)" <gi...@apache.org>.
yashi4engg commented on issue #8629:
URL: https://github.com/apache/cloudstack/issues/8629#issuecomment-1934946155
@vishesh92 -- While trying to reproduce above error I am in another issue where when trying to migrate VMs back to one node getting below error - '
2024-02-08 16:12:14,043 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-4:ctx-d7001bd1 job-297243/job-297244) (logid:71937cca) Remove job-297244 from job monitoring
2024-02-08 16:12:14,081 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-4:ctx-e6ce6a44 job-297243) (logid:71937cca) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd
com.cloud.utils.exception.CloudRuntimeException: API failed due to: {"message":"The license has expired","httpStatusCode":500,"errorCode":159}
at org.apache.cloudstack.storage.datastore.driver.ScaleIOPrimaryDataStoreDriver.grantAccess(ScaleIOPrimaryDataStoreDriver.java:206)
at org.apache.cloudstack.storage.volume.VolumeServiceImpl.grantAccess(VolumeServiceImpl.java:249)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepareForMigration(VolumeOrchestrator.java:1512)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy227.prepareForMigration(Unknown Source)
at com.cloud.vm.VirtualMachineManagerImpl.migrate(VirtualMachineManagerImpl.java:2731)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrate(VirtualMachineManagerImpl.java:2680)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrate(VirtualMachineManagerImpl.java:5484)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5581)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.cloudstack.api.ServerApiException: API failed due to: {"message":"The license has expired","httpStatusCode":500,"errorCode":159}
at org.apache.cloudstack.storage.datastore.client.ScaleIOGatewayClientImpl.checkResponseOK(ScaleIOGatewayClientImpl.java:263)
at org.apache.cloudstack.storage.datastore.client.ScaleIOGatewayClientImpl.processResponse(ScaleIOGatewayClientImpl.java:359)
at org.apache.cloudstack.storage.datastore.client.ScaleIOGatewayClientImpl.post(ScaleIOGatewayClientImpl.java:342)
... 39 more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DRS failed for cluster if have another cluster in zone [cloudstack]
Posted by "yashi4engg (via GitHub)" <gi...@apache.org>.
yashi4engg closed issue #8629: DRS failed for cluster if have another cluster in zone
URL: https://github.com/apache/cloudstack/issues/8629
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DRS failed for cluster if have another cluster in zone [cloudstack]
Posted by "vishesh92 (via GitHub)" <gi...@apache.org>.
vishesh92 commented on issue #8629:
URL: https://github.com/apache/cloudstack/issues/8629#issuecomment-1934887965
@yashi4engg I am not able to reproduce the issue. DRS runs independently for each cluster and this shouldn't be happening.
Can you share the complete logs including the error as well as more details about your clusters?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org