You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Chandan Purushothama (JIRA)" <ji...@apache.org> on 2014/10/15 08:35:33 UTC
[jira] [Updated] (CLOUDSTACK-7726) [Automation][XenServer] Failed
to Remove LoadBalancer Rule due to callHostPlugin failure
[ https://issues.apache.org/jira/browse/CLOUDSTACK-7726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chandan Purushothama updated CLOUDSTACK-7726:
---------------------------------------------
Attachment: management-server.zip
> [Automation][XenServer] Failed to Remove LoadBalancer Rule due to callHostPlugin failure
> ----------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-7726
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7726
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: Automation
> Affects Versions: 4.5.0
> Reporter: Chandan Purushothama
> Priority: Critical
> Fix For: 4.5.0
>
> Attachments: management-server.zip
>
>
> =======================================================
> CallHost Plugin Failure while trying to remove the Load Balancing Rule:
> =======================================================
> 2014-10-13 19:20:06,408 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-45:ctx-fb2667f4) Seq 1-122723089845846136: Executing request
> 2014-10-13 19:20:06,410 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-89:ctx-0a5fbdf2) callHostPlugin failed for cmd: createFileInDomr with args filepath: /etc/haproxy/haproxy.cfg.new.1413228006238, domrip: 169.254.2.250, filecontents: global
> log 127.0.0.1:3914 local0 warning
> maxconn 4096
> maxpipes 1024
> chroot /var/lib/haproxy
> user haproxy
> group haproxy
> daemon
>
> defaults
> log global
> mode tcp
> option dontlognull
> retries 3
> option redispatch
> option forwardfor
> option forceclose
> timeout connect 5000
> timeout client 50000
> timeout server 50000
> listen stats_on_public 10.220.166.19:8081
> mode http
> option httpclose
> stats enable
> stats uri /admin?stats
> stats realm Haproxy\ Statistics
> stats auth admin1:AdMiN123
>
> listen 10_220_166_28-2222 10.220.166.28:2222
> balance roundrobin
>
>
> , due to There was a failure communicating with the plugin.
> 2014-10-13 19:20:06,410 WARN [c.c.a.m.DirectAgentAttache] (DirectAgent-89:ctx-0a5fbdf2) Seq 1-122723089845846135: Throwable caught while executing command
> com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for cmd: createFileInDomr with args filepath: /etc/haproxy/haproxy.cfg.new.1413228006238, domrip: 169.254.2.250, filecontents: global
> log 127.0.0.1:3914 local0 warning
> maxconn 4096
> maxpipes 1024
> chroot /var/lib/haproxy
> user haproxy
> group haproxy
> daemon
>
> defaults
> log global
> mode tcp
> option dontlognull
> retries 3
> option redispatch
> option forwardfor
> option forceclose
> timeout connect 5000
> timeout client 50000
> timeout server 50000
> listen stats_on_public 10.220.166.19:8081
> mode http
> option httpclose
> stats enable
> stats uri /admin?stats
> stats realm Haproxy\ Statistics
> stats auth admin1:AdMiN123
>
> listen 10_220_166_28-2222 10.220.166.28:2222
> balance roundrobin
>
>
> , due to There was a failure communicating with the plugin.
> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:3670)
> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.createFileInVR(CitrixResourceBase.java:575)
> at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfigToVR(VirtualRoutingResource.java:161)
> at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfigToVR(VirtualRoutingResource.java:155)
> at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfig(VirtualRoutingResource.java:179)
> at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:125)
> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:430)
> at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:64)
> at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
> at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:304)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2014-10-13 19:20:06,411 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-89:ctx-0a5fbdf2) Seq 1-122723089845846135: Response Received:
> 2014-10-13 19:20:06,412 DEBUG [c.c.a.t.Request] (DirectAgent-89:ctx-0a5fbdf2) Seq 1-122723089845846135: Processing: { Ans: , MgmtId: 59408290029589, via: 1, Ver: v1, Flags: 0, [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for cmd: createFileInDomr with args filepath: /etc/haproxy/haproxy.cfg.new.1413228006238, domrip: 169.254.2.250, filecontents: global\n\tlog 127.0.0.1:3914 local0 warning\n\tmaxconn 4096\n\tmaxpipes 1024\n\tchroot /var/lib/haproxy\n\tuser haproxy\n\tgroup haproxy\n\tdaemon\n\t \ndefaults\n\tlog global\n\tmode tcp\n\toption dontlognull\n\tretries 3\n\toption redispatch\n\toption forwardfor\n\toption forceclose\n\ttimeout connect 5000\n\ttimeout client 50000\n\ttimeout server 50000\n\nlisten stats_on_public 10.220.166.19:8081\n\tmode http\n\toption httpclose\n\tstats enable\n\tstats uri /admin?stats\n\tstats realm Haproxy\\ Statistics\n\tstats auth admin1:AdMiN123\n\n\t \nlisten 10_220_166_28-2222 10.220.166.28:2222\n\tbalance roundrobin\n\t \n\t \n, due to There was a failure communicating with the plugin.","wait":0}}] }
> 2014-10-13 19:20:06,412 DEBUG [c.c.a.t.Request] (API-Job-Executor-41:ctx-9ee60aa8 job-84 ctx-803aafba) Seq 1-122723089845846135: Received: { Ans: , MgmtId: 59408290029589, via: 1, Ver: v1, Flags: 0, { Answer } }
> 2014-10-13 19:20:06,424 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-41:ctx-9ee60aa8 job-84 ctx-803aafba) LB Rollback rule id: 11,while removing vmId 4
> 2014-10-13 19:20:06,427 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-41:ctx-9ee60aa8 job-84 ctx-803aafba) LB Rollback rule id: 11 while removing vm instances
> 2014-10-13 19:20:06,427 WARN [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-41:ctx-9ee60aa8 job-84 ctx-803aafba) Unable to apply the load balancer config because resource is unavaliable.
> com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Unable to apply loadbalancing rules on router
> at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:4050)
> at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyLoadBalancingRules(VirtualNetworkApplianceManagerImpl.java:3901)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy190.applyLoadBalancingRules(Unknown Source)
> at com.cloud.network.element.VirtualRouterElement.applyLBRules(VirtualRouterElement.java:384)
> at com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLbRules(LoadBalancingRulesManagerImpl.java:1845)
> at com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLbRules(LoadBalancingRulesManagerImpl.java:2433)
> at com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLoadBalancerRules(LoadBalancingRulesManagerImpl.java:1881)
> at com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLoadBalancerConfig(LoadBalancingRulesManagerImpl.java:1805)
> at com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancerInternal(LoadBalancingRulesManagerImpl.java:1377)
> at com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancer(LoadBalancingRulesManagerImpl.java:1163)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy138.removeFromLoadBalancer(Unknown Source)
> at org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd.execute(RemoveFromLoadBalancerRuleCmd.java:162)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
> at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:513)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:470)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2014-10-13 19:20:06,444 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-23:ctx-238092a2) Created a vif 825c5404-4bc8-3a35-8961-db3d95bde69d on 1
> 2014-10-13 19:20:06,452 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-41:ctx-9ee60aa8 job-84) Unexpected exception while executing org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd
> com.cloud.utils.exception.CloudRuntimeException: Failed to remove specified load balancer rule id for vms [4]
> at com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancerInternal(LoadBalancingRulesManagerImpl.java:1413)
> at com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancer(LoadBalancingRulesManagerImpl.java:1163)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy138.removeFromLoadBalancer(Unknown Source)
> at org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd.execute(RemoveFromLoadBalancerRuleCmd.java:162)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
> at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:513)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:470)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2014-10-13 19:20:06,455 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-41:ctx-9ee60aa8 job-84) Complete async job-84, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to remove specified load balancer rule id for vms [4]"}
> 2014-10-13 19:20:06,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-41:ctx-9ee60aa8 job-84) Publish async job-84 complete on message bus
> 2014-10-13 19:20:06,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-41:ctx-9ee60aa8 job-84) Wake up jobs related to job- 84
> 2014-10-13 19:20:06,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-41:ctx-9ee60aa8 job-84) Update db status for job- 84
> 2014-10-13 19:20:06,458 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-41:ctx-9ee60aa8 job-84) Wake up jobs joined with job- 84 and disjoin all subjobs created from job- 84
> 2014-10-13 19:20:06,463 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-41:ctx-9ee60aa8 job-84) Done executing org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd for job-84
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)