You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Chandan Purushothama (JIRA)" <ji...@apache.org> on 2013/04/22 02:35:15 UTC
[jira] [Created] (CLOUDSTACK-2127) Used Master Branch System VM
Template for VMWare: Failed to Create Load Balancing Rule
Chandan Purushothama created CLOUDSTACK-2127:
------------------------------------------------
Summary: Used Master Branch System VM Template for VMWare: Failed to Create Load Balancing Rule
Key: CLOUDSTACK-2127
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2127
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.2.0
Reporter: Chandan Purushothama
Priority: Critical
Fix For: 4.2.0
===============
Steps to Reproduce:
===============
1. Create a Load Balancing Rule where the Virtual Router is deployed using the new Master System VM Template for VMWare
============
Observations:
============
------------------------------------------------
Create Load Balancing Rule Job:
------------------------------------------------
[root@asfmgmt ~]# grep -i "job-24" /var/log/cloudstack/management/management-server.log
2013-04-21 17:22:03,866 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-3:null) submit async job-24, details: AsyncJobVO {id:24, userId: 3, accountId: 3, sessionKey: null, instanceType: null, instanceId: null, cmd: org.apache.cloudstack.api.command.user.loadbalancer.AssignToLoadBalancerRuleCmd, cmdOriginator: null, cmdInfo: {"response":"json","id":"0d569a49-ee02-4f32-93b5-31f1bc9aaa99","sessionkey":"LmU2BWQx2jc+w2FawY1qyY9foFk\u003d","virtualmachineids":"d68f7c60-d963-4a32-bcd4-1b5f37aaa992,3a5f350d-0740-424a-9172-7fb893effdeb","ctxUserId":"3","_":"1366590152786","ctxAccountId":"3","ctxStartEventId":"115"}, cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0, result: null, initMsid: 7200344900649, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2013-04-21 17:22:03,870 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-30:job-24) Executing org.apache.cloudstack.api.command.user.loadbalancer.AssignToLoadBalancerRuleCmd for job-24
2013-04-21 17:22:03,882 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-30:job-24) Sync job-24 execution on object network.204
2013-04-21 17:22:03,898 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-30:job-24) job org.apache.cloudstack.api.command.user.loadbalancer.AssignToLoadBalancerRuleCmd for job-24 was queued, processing the queue.
2013-04-21 17:22:03,904 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-30:job-24) Executing sync queue item: SyncQueueItemVO {id:7, queueId: 1, contentType: AsyncJob, contentId: 24, lastProcessMsid: 7200344900649, lastprocessNumber: 7, lastProcessTime: Sun Apr 21 17:22:03 PDT 2013, created: Sun Apr 21 17:22:03 PDT 2013}
2013-04-21 17:22:03,905 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-30:job-24) Schedule queued job-24
2013-04-21 17:22:03,909 DEBUG [cloud.async.SyncQueueManagerImpl] (Job-Executor-30:job-24) There is a pending process in sync queue(id: 1)
2013-04-21 17:22:03,911 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-31:job-24) Executing org.apache.cloudstack.api.command.user.loadbalancer.AssignToLoadBalancerRuleCmd for job-24
2013-04-21 17:22:03,944 DEBUG [cloud.user.AccountManagerImpl] (Job-Executor-31:job-24) Access to Rule[4-LoadBalancing-Add] granted to Acct[3-atoms] by DomainChecker_EnhancerByCloudStack_64236710
2013-04-21 17:22:03,945 DEBUG [cloud.user.AccountManagerImpl] (Job-Executor-31:job-24) Access to VM[User|Atoms-VM-2] granted to Acct[3-atoms] by DomainChecker_EnhancerByCloudStack_64236710
2013-04-21 17:22:03,947 DEBUG [network.lb.LoadBalancingRulesManagerImpl] (Job-Executor-31:job-24) Adding VM[User|Atoms-VM-2] to the load balancer pool
2013-04-21 17:22:03,954 DEBUG [cloud.user.AccountManagerImpl] (Job-Executor-31:job-24) Access to Rule[4-LoadBalancing-Add] granted to Acct[3-atoms] by DomainChecker_EnhancerByCloudStack_64236710
2013-04-21 17:22:03,955 DEBUG [cloud.user.AccountManagerImpl] (Job-Executor-31:job-24) Access to VM[User|Atoms-VM-1] granted to Acct[3-atoms] by DomainChecker_EnhancerByCloudStack_64236710
2013-04-21 17:22:03,957 DEBUG [network.lb.LoadBalancingRulesManagerImpl] (Job-Executor-31:job-24) Adding VM[User|Atoms-VM-1] to the load balancer pool
2013-04-21 17:22:04,109 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-31:job-24) Applying ip association in network Ntwk[204|Guest|8]
2013-04-21 17:22:04,129 DEBUG [agent.transport.Request] (Job-Executor-31:job-24) Seq 1-1166875769: Sending { Cmd , MgmtId: 7200344900649, via: 1, Ver: v1, Flags: 100001, [{"routing.IpAssocCommand":{"ipAddresses":[{"accountId":3,"publicIp":"10.223.139.70","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"vlanId":"1391","vlanGateway":"10.223.139.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:c6:4a:00:00:0c","networkRate":200,"trafficType":"Public"},{"accountId":3,"publicIp":"10.223.139.74","sourceNat":false,"add":true,"oneToOneNat":false,"firstIP":false,"vlanId":"1391","vlanGateway":"10.223.139.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:ef:5a:00:00:10","networkRate":200,"trafficType":"Public"},{"accountId":3,"publicIp":"10.223.139.75","sourceNat":false,"add":true,"oneToOneNat":false,"firstIP":false,"vlanId":"1391","vlanGateway":"10.223.139.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:89:90:00:00:11","networkRate":200,"trafficType":"Public"},{"accountId":3,"publicIp":"10.223.139.73","sourceNat":false,"add":true,"oneToOneNat":true,"firstIP":false,"vlanId":"1391","vlanGateway":"10.223.139.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:81:ce:00:00:0f","networkRate":200,"trafficType":"Public"}],"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"10.223.59.80","router.name":"r-4-SYSVMTMPL"},"wait":0}}] }
2013-04-21 17:22:04,130 DEBUG [agent.transport.Request] (Job-Executor-31:job-24) Seq 1-1166875769: Executing: { Cmd , MgmtId: 7200344900649, via: 1, Ver: v1, Flags: 100001, [{"routing.IpAssocCommand":{"ipAddresses":[{"accountId":3,"publicIp":"10.223.139.70","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"vlanId":"1391","vlanGateway":"10.223.139.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:c6:4a:00:00:0c","networkRate":200,"trafficType":"Public"},{"accountId":3,"publicIp":"10.223.139.74","sourceNat":false,"add":true,"oneToOneNat":false,"firstIP":false,"vlanId":"1391","vlanGateway":"10.223.139.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:ef:5a:00:00:10","networkRate":200,"trafficType":"Public"},{"accountId":3,"publicIp":"10.223.139.75","sourceNat":false,"add":true,"oneToOneNat":false,"firstIP":false,"vlanId":"1391","vlanGateway":"10.223.139.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:89:90:00:00:11","networkRate":200,"trafficType":"Public"},{"accountId":3,"publicIp":"10.223.139.73","sourceNat":false,"add":true,"oneToOneNat":true,"firstIP":false,"vlanId":"1391","vlanGateway":"10.223.139.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:81:ce:00:00:0f","networkRate":200,"trafficType":"Public"}],"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"10.223.59.80","router.name":"r-4-SYSVMTMPL"},"wait":0}}] }
2013-04-21 17:22:22,917 DEBUG [agent.transport.Request] (Job-Executor-31:job-24) Seq 1-1166875769: Received: { Ans: , MgmtId: 7200344900649, via: 1, Ver: v1, Flags: 0, { IpAssocAnswer } }
2013-04-21 17:22:22,925 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-31:job-24) Applying firewall rules in network Ntwk[204|Guest|8]
2013-04-21 17:22:22,965 DEBUG [cloud.network.NetworkModelImpl] (Job-Executor-31:job-24) Service SecurityGroup is not supported in the network id=204
2013-04-21 17:22:23,002 DEBUG [agent.transport.Request] (Job-Executor-31:job-24) Seq 1-1166875771: Sending { Cmd , MgmtId: 7200344900649, via: 1, Ver: v1, Flags: 100001, [{"routing.LoadBalancerConfigCommand":{"loadBalancers":[{"uuid":"0d569a49-ee02-4f32-93b5-31f1bc9aaa99","srcIp":"10.223.139.75","srcPort":22,"protocol":"tcp","algorithm":"roundrobin","revoked":false,"alreadyAdded":false,"inline":false,"destinations":[{"destIp":"10.1.1.63","destPort":22,"revoked":false,"alreadyAdded":false},{"destIp":"10.1.1.146","destPort":22,"revoked":false,"alreadyAdded":false}]}],"lbStatsVisibility":"global","lbStatsPublicIP":"10.223.139.70","lbStatsPrivateIP":"10.223.59.80","lbStatsGuestIP":"10.1.1.1","lbStatsPort":"8081","lbStatsSrcCidrs":"0/0","lbStatsAuth":"admin1:AdMiN123","lbStatsUri":"/admin?stats","nic":{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"uuid":"6e104794-6262-4516-b886-6130eabc1e5f","ip":"10.1.1.1","netmask":"255.255.255.0","mac":"02:00:40:dd:00:02","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2586","isolationUri":"vlan://2586","isSecurityGroupEnabled":false},"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"10.223.59.80","router.name":"r-4-SYSVMTMPL"},"wait":0}}] }
2013-04-21 17:22:23,002 DEBUG [agent.transport.Request] (Job-Executor-31:job-24) Seq 1-1166875771: Executing: { Cmd , MgmtId: 7200344900649, via: 1, Ver: v1, Flags: 100001, [{"routing.LoadBalancerConfigCommand":{"loadBalancers":[{"uuid":"0d569a49-ee02-4f32-93b5-31f1bc9aaa99","srcIp":"10.223.139.75","srcPort":22,"protocol":"tcp","algorithm":"roundrobin","revoked":false,"alreadyAdded":false,"inline":false,"destinations":[{"destIp":"10.1.1.63","destPort":22,"revoked":false,"alreadyAdded":false},{"destIp":"10.1.1.146","destPort":22,"revoked":false,"alreadyAdded":false}]}],"lbStatsVisibility":"global","lbStatsPublicIP":"10.223.139.70","lbStatsPrivateIP":"10.223.59.80","lbStatsGuestIP":"10.1.1.1","lbStatsPort":"8081","lbStatsSrcCidrs":"0/0","lbStatsAuth":"admin1:AdMiN123","lbStatsUri":"/admin?stats","nic":{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"uuid":"6e104794-6262-4516-b886-6130eabc1e5f","ip":"10.1.1.1","netmask":"255.255.255.0","mac":"02:00:40:dd:00:02","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2586","isolationUri":"vlan://2586","isSecurityGroupEnabled":false},"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"10.223.59.80","router.name":"r-4-SYSVMTMPL"},"wait":0}}] }
2013-04-21 17:22:30,957 DEBUG [agent.transport.Request] (Job-Executor-31:job-24) Seq 1-1166875771: Received: { Ans: , MgmtId: 7200344900649, via: 1, Ver: v1, Flags: 0, { Answer } }
2013-04-21 17:22:30,961 WARN [network.lb.LoadBalancingRulesManagerImpl] (Job-Executor-31:job-24) Unable to apply the load balancer config because resource is unavaliable.
2013-04-21 17:22:30,963 INFO [cloud.utils.AnnotationHelper] (Job-Executor-31:job-24)
2013-04-21 17:22:30,984 ERROR [cloud.async.AsyncJobManagerImpl] (Job-Executor-31:job-24) Unexpected exception while executing org.apache.cloudstack.api.command.user.loadbalancer.AssignToLoadBalancerRuleCmd
2013-04-21 17:22:30,985 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-31:job-24) Complete async job-24, jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to add specified loadbalancerruleid for vms [6, 3]
2013-04-21 17:22:30,996 DEBUG [cloud.async.SyncQueueManagerImpl] (Job-Executor-31:job-24) Sync queue (1) is currently empty
2013-04-21 17:22:31,092 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-25:null) Async job-24 completed
================
ERROR:
================
2013-04-21 17:22:29,784 ERROR [utils.ssh.SshHelper] (DirectAgent-126:10.223.59.66) SSH execution of command /root/loadbalancer.sh -i 10.223.59.80 -f /tmp/10_223_59_80.cfg -a 10.223.139.75:22:, -s 10.223.139.70:8081:0/0:,, has an error status code in return. result output: mv: cannot stat `/var/run/haproxy.pid': No such file or directory
cat: /var/run/haproxy.pid.old: No such file or directory
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
/root/reconfigLB.sh: line 28: haproxy: command not found
cat: /var/run/haproxy.pid.old: No such file or directory
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
mv: cannot stat `/var/run/haproxy.pid.old': No such file or directory
2013-04-21 17:22:29,791 ERROR [vmware.resource.VmwareResource] (DirectAgent-126:10.223.59.66) LoadBalancerConfigCommand on domain router 10.223.59.80 failed. message: mv: cannot stat `/var/run/haproxy.pid': No such file or directory
cat: /var/run/haproxy.pid.old: No such file or directory
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
/root/reconfigLB.sh: line 28: haproxy: command not found
cat: /var/run/haproxy.pid.old: No such file or directory
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
mv: cannot stat `/var/run/haproxy.pid.old': No such file or directory
2013-04-21 17:22:30,957 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-126:null) Seq 1-1166875771: Response Received:
2013-04-21 17:22:30,957 DEBUG [agent.transport.Request] (DirectAgent-126:null) Seq 1-1166875771: Processing: { Ans: , MgmtId: 7200344900649, via: 1, Ver: v1, Flags: 0, [{"Answer":{"result":false,"details":"LoadBalancerConfigCommand on domain router 10.223.59.80 failed. message: mv: cannot stat `/var/run/haproxy.pid': No such file or directory\ncat: /var/run/haproxy.pid.old: No such file or directory\nkill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]\n/root/reconfigLB.sh: line 28: haproxy: command not found\ncat: /var/run/haproxy.pid.old: No such file or directory\nkill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]\nmv: cannot stat `/var/run/haproxy.pid.old': No such file or directory\n","wait":0}}] }
2013-04-21 17:22:30,957 DEBUG [agent.transport.Request] (Job-Executor-31:job-24) Seq 1-1166875771: Received: { Ans: , MgmtId: 7200344900649, via: 1, Ver: v1, Flags: 0, { Answer } }
2013-04-21 17:22:30,961 WARN [network.lb.LoadBalancingRulesManagerImpl] (Job-Executor-31:job-24) Unable to apply the load balancer config because resource is unavaliable.
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Unable to apply firewall rules on router
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3439)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyFirewallRules(VirtualNetworkApplianceManagerImpl.java:3294)
at com.cloud.network.element.VirtualRouterElement.applyLBRules(VirtualRouterElement.java:361)
at com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyRules(LoadBalancingRulesManagerImpl.java:1425)
at com.cloud.network.NetworkManagerImpl.applyRules(NetworkManagerImpl.java:2428)
at com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLoadBalancerRules(LoadBalancingRulesManagerImpl.java:1461)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLoadBalancerConfig(LoadBalancingRulesManagerImpl.java:1385)
at com.cloud.network.lb.LoadBalancingRulesManagerImpl.assignToLoadBalancer(LoadBalancingRulesManagerImpl.java:909)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.user.loadbalancer.AssignToLoadBalancerRuleCmd.execute(AssignToLoadBalancerRuleCmd.java:100)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:164)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-04-21 17:22:30,963 INFO [cloud.utils.AnnotationHelper] (Job-Executor-31:job-24)
class java.lang.Stringdoes not have a Table annotation
2013-04-21 17:22:30,984 ERROR [cloud.async.AsyncJobManagerImpl] (Job-Executor-31:job-24) Unexpected exception while executing org.apache.cloudstack.api.command.user.loadbalancer.AssignToLoadBalancerRuleCmd
com.cloud.utils.exception.CloudRuntimeException: Failed to add specified loadbalancerruleid for vms [6, 3]
at com.cloud.network.lb.LoadBalancingRulesManagerImpl.assignToLoadBalancer(LoadBalancingRulesManagerImpl.java:932)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.user.loadbalancer.AssignToLoadBalancerRuleCmd.execute(AssignToLoadBalancerRuleCmd.java:100)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:164)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-04-21 17:22:30,985 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-31:job-24) Complete async job-24, jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to add specified loadbalancerruleid for vms [6, 3]
2013-04-21 17:22:30,996 DEBUG [cloud.async.SyncQueueManagerImpl] (Job-Executor-31:job-24) Sync queue (1) is currently empty
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira