You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by "Musayev, Ilya" <im...@webmd.net> on 2013/04/08 19:24:17 UTC
RE: [jira] [Created] (CLOUDSTACK-1967) Maintenance mode not fully
working with VMWare
++Kelven Yang,
Francois,
I would ask if this behavior is by design.
Especially if the environment is shared with non CS VMs. CS maintenance mode does not exactly imply VMWARE maintenance mode.
If we can ask for enhancement where there is an explicit rule that says - this hypervisor is exclusively managed by CloudStack, in that case, we can invoke vmware sdk API call for maintenance mode - which will be much faster than what CS does at present moment - in sequence - one at a time.
Regards
ilya
> -----Original Message-----
> From: Francois Gaudreault (JIRA) [mailto:jira@apache.org]
> Sent: Monday, April 08, 2013 1:07 PM
> To: cloudstack-issues@incubator.apache.org
> Subject: [jira] [Created] (CLOUDSTACK-1967) Maintenance mode not fully
> working with VMWare
>
> Francois Gaudreault created CLOUDSTACK-1967:
> -----------------------------------------------
>
> Summary: Maintenance mode not fully working with VMWare
> Key: CLOUDSTACK-1967
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-1967
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public (Anyone can view this level - this is the default.)
> Components: VMware
> Affects Versions: 4.1.0
> Reporter: Francois Gaudreault
> Attachments: managementlog_maint.log
>
> When we turn on maintenance mode on one node of the cluster, the VMs
> are migrating fine, but the host is never fully sent to maintenance mode on
> the VMWare side. We do see bunch of stack traces related to the
> management server loosing connectivity, but machine will be available to
> ICMP for the entire process. The host will appear as Disconnected. This is not
> a critical issue (major tho) since the VMs are migrated properly, but the error
> should be investigated/reproduced.
>
> See the attachments for logs.
>
> Catalina.out StackTrace:
> INFO [vmware.resource.VmwareResource] (ClusteredAgentManager
> Timer:) VmwareResource network configuration info. private vSwitch:
> vSwitch0, public vSwitch: vSwitch0, guest network: vSwitch0 INFO
> [vmware.resource.VmwareResource] (AgentTaskPool-6:) Startup report on
> host hardware info.
> {"memoryBytes":68708646912,"cpuCount":8,"cpuSpeed":2800}
> INFO [vmware.resource.VmwareResource] (AgentTaskPool-6:) Startup
> report on host network info.
> {"hostIp":"10.0.22.65","hostNetmask":"255.255.255.0","hostMacAddress":"0
> 0:26:55:2b:c0:8a"}
> INFO [vmware.resource.VmwareResource] (AgentTaskPool-6:) Add local
> storage startup command: {"totalSize":0,"poolInfo":{"uuid":"4e656219-c3f9-
> 4a7d-93e3-
> 5c4d09e8c59c","host":"yul01vi13.ops.dot","localPath":"","hostPath":"datasto
> re-
> 9206","poolType":"LVM","capacityBytes":67914170368,"availableBytes":6689
> 6003072},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage
> ","dataCenter":"3","pod":"3","cluster":"12","guid":"4e656219-c3f9-4a7d-
> 93e3-5c4d09e8c59c","name":"4e656219-c3f9-4a7d-93e3-
> 5c4d09e8c59c","wait":0}
> INFO [vmware.resource.VmwareResource] (AgentTaskPool-6:) Add local
> storage startup command: {"totalSize":0,"poolInfo":{"uuid":"ddd756cb-8592-
> 4473-9482-
> 5a4b68085a05","host":"yul01vi13.ops.dot","localPath":"","hostPath":"datast
> ore-
> 9221","poolType":"LVM","capacityBytes":733634101248,"availableBytes":732
> 612788224},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Stora
> ge","dataCenter":"3","pod":"3","cluster":"12","guid":"ddd756cb-8592-4473-
> 9482-5a4b68085a05","name":"ddd756cb-8592-4473-9482-
> 5a4b68085a05","wait":0}
> INFO [agent.manager.DirectAgentAttache] (AgentTaskPool-6:)
> StartupAnswer received 4 Interval = 60 INFO
> [vmware.resource.VmwareResource] (DirectAgent-494:yul01vi13.ops.dot)
> Executing resource CheckNetworkCommand
> {"networkInfoList":[{"physicalNetworkId":206,"privateNetworkName":"vSwi
> tch0"},{"physicalNetworkId":207,"storageNetworkName":"vSwitch1"},{"phys
> icalNetworkId":208,"publicNetworkName":"vSwitch2","guestNetworkName"
> :"vSwitch2"}],"wait":0}
> INFO [network.security.SecurityGroupListener] (AgentTaskPool-6:)
> Received a host startup notification INFO
> [network.security.SecurityGroupListener] (AgentTaskPool-6:) Scheduled
> network rules cleanup, interval=1961 INFO
> [network.security.SecurityGroupListener] (AgentTaskPool-6:) Received a
> host startup notification INFO [network.security.SecurityGroupListener]
> (AgentTaskPool-6:) Received a host startup notification WARN
> [agent.manager.AgentManagerImpl] (AgentTaskPool-6:) Resource [Host:4] is
> unreachable: Host 4: Unable to send class
> com.cloud.agent.api.ModifyStoragePoolCommand because agent is in
> maintenance mode WARN [storage.listener.StoragePoolMonitor]
> (AgentTaskPool-6:) Unable to connect host 4 to pool Pool[205|VMFS] due to
> com.cloud.exception.StorageUnavailableException: Resource
> [StoragePool:205] is unreachable: Unable to get an answer to the modify
> storage pool command
> com.cloud.exception.StorageUnavailableException: Resource
> [StoragePool:205] is unreachable: Unable to get an answer to the modify
> storage pool command
> at
> com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(Storage
> ManagerImpl.java:1615)
> at
> com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoo
> lMonitor.java:94)
> at
> com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(
> AgentManagerImpl.java:611)
> at
> com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(A
> gentManagerImpl.java:1504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(
> AopUtils.java:319)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi
> npoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
> ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.pr
> oceed(MethodInvocationProceedingJoinPoint.java:80)
> at
> com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(Transactio
> nContextBuilder.java:45)
> at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMeth
> odWithGivenArgs(AbstractAspectJAdvice.java:621)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMeth
> od(AbstractAspectJAdvice.java:610)
> at
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJArou
> ndAdvice.java:65)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
> ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(E
> xposeInvocationInterceptor.java:90)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
> ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDyna
> micAopProxy.java:202)
> at sun.proxy.$Proxy109.handleDirectConnectAgent(Unknown Source)
> at
> com.cloud.resource.ResourceManagerImpl.createHostAndAgent(Resource
> ManagerImpl.java:1937)
> at
> com.cloud.resource.ResourceManagerImpl.createHostAndAgent(Resource
> ManagerImpl.java:2074)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(
> AopUtils.java:319)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi
> npoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
> ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.pr
> oceed(MethodInvocationProceedingJoinPoint.java:80)
> at
> com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(Transactio
> nContextBuilder.java:45)
> at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMeth
> odWithGivenArgs(AbstractAspectJAdvice.java:621)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMeth
> od(AbstractAspectJAdvice.java:610)
> at
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJArou
> ndAdvice.java:65)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
> ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(E
> xposeInvocationInterceptor.java:90)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
> ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDyna
> micAopProxy.java:202)
> at sun.proxy.$Proxy119.createHostAndAgent(Unknown Source)
> at
> com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.run(Agent
> ManagerImpl.java:1156)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:615)
> at java.lang.Thread.run(Thread.java:679)
> WARN [agent.manager.AgentManagerImpl] (AgentTaskPool-6:) Resource
> [Host:4] is unreachable: Host 4: Unable to send class
> com.cloud.agent.api.ModifyStoragePoolCommand because agent is in
> maintenance mode WARN [storage.listener.StoragePoolMonitor]
> (AgentTaskPool-6:) Unable to connect host 4 to pool Pool[206|VMFS] due to
> com.cloud.exception.StorageUnavailableException: Resource
> [StoragePool:206] is unreachable: Unable to get an answer to the modify
> storage pool command
> com.cloud.exception.StorageUnavailableException: Resource
> [StoragePool:206] is unreachable: Unable to get an answer to the modify
> storage pool command
> at
> com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(Storage
> ManagerImpl.java:1615)
> at
> com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoo
> lMonitor.java:94)
> at
> com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(
> AgentManagerImpl.java:611)
> at
> com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(A
> gentManagerImpl.java:1504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(
> AopUtils.java:319)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi
> npoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
> ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.pr
> oceed(MethodInvocationProceedingJoinPoint.java:80)
> at
> com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(Transactio
> nContextBuilder.java:45)
> at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMeth
> odWithGivenArgs(AbstractAspectJAdvice.java:621)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMeth
> od(AbstractAspectJAdvice.java:610)
> at
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJArou
> ndAdvice.java:65)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
> ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(E
> xposeInvocationInterceptor.java:90)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
> ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDyna
> micAopProxy.java:202)
> at sun.proxy.$Proxy109.handleDirectConnectAgent(Unknown Source)
> at
> com.cloud.resource.ResourceManagerImpl.createHostAndAgent(Resource
> ManagerImpl.java:1937)
> at
> com.cloud.resource.ResourceManagerImpl.createHostAndAgent(Resource
> ManagerImpl.java:2074)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(
> AopUtils.java:319)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi
> npoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
> ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.pr
> oceed(MethodInvocationProceedingJoinPoint.java:80)
> at
> com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(Transactio
> nContextBuilder.java:45)
> at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMeth
> odWithGivenArgs(AbstractAspectJAdvice.java:621)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMeth
> od(AbstractAspectJAdvice.java:610)
> at
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJArou
> ndAdvice.java:65)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
> ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(E
> xposeInvocationInterceptor.java:90)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
> ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDyna
> micAopProxy.java:202)
> at sun.proxy.$Proxy119.createHostAndAgent(Unknown Source)
> at
> com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.run(Agent
> ManagerImpl.java:1156)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> a:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> va:615)
> at java.lang.Thread.run(Thread.java:679)
> INFO [vmware.resource.VmwareResource] (DirectAgent-
> 403:yul01vi13.ops.dot) Executing resource ReadyCommand:
> {"dcId":3,"hostId":4,"wait":0} INFO [agent.manager.AgentMonitor] (Thread-
> 5:) Found the following agents behind on ping: [8, 4] INFO
> [agent.manager.AgentManagerImpl] (AgentTaskPool-7:) Host 4 is
> disconnecting with event ShutdownRequested INFO
> [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) Investigating why
> host 8 has disconnected with event PingTimeout INFO
> [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) The state
> determined is Up INFO [agent.manager.AgentManagerImpl]
> (AgentTaskPool-10:) Agent is determined to be up and running
>
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA administrators
> For more information on JIRA, see: http://www.atlassian.com/software/jira