You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Andrija Panic <an...@gmail.com> on 2015/03/10 16:21:24 UTC
[HELP] VR sometimes fail to start
Hi,
we are having issues from time to time with starting VR - while creating
VPC.
Here is the few lines from MGMT log:
2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
(Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start vpc [VPC
[621-VPC-delete] due to
com.cloud.exception.AgentUnavailableException: Resource [Host:788] is
unreachable: Host 788: Unable to start instance due to Unable to start
VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not retrying
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1072)
The interesting part is on the Agent itself:
2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:) Timed out:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
r-2189-VM -p %vpccidr=
10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
. Output is:
Interesting stuff is - it's seems somewhat random, 3-4 out of 5 works.
Any suggestion are really appreciated, since this is live system :(
ACS 4.3.0, KVM, CEPH
Thanks,
--
Andrija Panić
Re: [HELP] VR sometimes fail to start
Posted by Marcus <sh...@gmail.com>.
People are mostly using upgraded 'virt' repos.
On Sat, Mar 14, 2015 at 8:58 AM, Marcus <sh...@gmail.com> wrote:
> Oh, this is an unsupported version. should be using libvirt 0.9.4+ and
> Qemu 1.0+ per the install docs.
>
> On Sat, Mar 14, 2015 at 8:56 AM, Andrija Panic <an...@gmail.com> wrote:
>> Tried the proposed solution - but it still don't work - BTW my qemu is
>> 0.12.x - from CentOS 6.4...
>>
>> When I manually start the script with all parameters, it simply hangs, no
>> output at all, until ctrl + c
>>
>> BTW this is with now upgraded ACS to 4.3.2...
>>
>> :(
>>
>> On 12 March 2015 at 09:24, Andrija Panic <an...@gmail.com> wrote:
>>
>>> Hi Marcus, will do - I'm just unconfortable changing live things, but that
>>> is my problem. Will let you know of the output.
>>>
>>> Thanks
>>>
>>> On 12 March 2015 at 01:34, ilya musayev <il...@gmail.com>
>>> wrote:
>>>
>>>> Andrija,
>>>>
>>>> Have you tried the solution Loic mentioned? Its seems like an easy one to
>>>> try..
>>>>
>>>> Thanks
>>>> ilya
>>>>
>>>>
>>>> On 3/11/15 3:54 AM, Andrija Panic wrote:
>>>>
>>>>> Oh well, I will then update to 4.3.2 anyway, and hope to see the problem
>>>>> go
>>>>> away...
>>>>>
>>>>> THanks guys
>>>>>
>>>>> On 10 March 2015 at 22:45, Loic Lambiel <lo...@exoscale.ch>
>>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>>
>>>>>> Maybe the same issue we faced here: https://github.com/exoscale/
>>>>>> cloudstack/commit/df3e7533a78c3594cdd39b8b18e6f4a3abcb8cfd
>>>>>>
>>>>>> The patch has not been yet submitted upstream. I'll do it once it is
>>>>>> running on our production.
>>>>>>
>>>>>> Anyway, you may easily test it by performing the small change directly
>>>>>> into the script.
>>>>>>
>>>>>> Hope that helps.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Loic Lambiel
>>>>>>
>>>>>>
>>>>>> On 10/03/15 19:40, Andrija Panic wrote:
>>>>>>
>>>>>> Hi guys,
>>>>>>>
>>>>>>> Kernel is Elrepo 3.10.67 (CentOS 6.6)
>>>>>>> Libvirt manually compiled 1.2.3 with RBD support.
>>>>>>> Qemu 0.12.1.2 - Inktank patched original RHEL 6.4 rpms with RBD
>>>>>>> support.
>>>>>>>
>>>>>>> And here is the output, with debug mode, from Agent.
>>>>>>>
>>>>>>> At the begining I see agent deleting routes for 169.254.x.x network...:
>>>>>>>
>>>>>>> Basically, It seems like the
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>> hviasocket.pl
>>>>>>> is
>>>>>>> failing, not being able to connect to socket...
>>>>>>> And I could not ping VR (169.254.2.99)...
>>>>>>>
>>>>>>> LOG:
>>>>>>>
>>>>>>> 2015-03-10 16:43:46,976{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-4:) Processing command:
>>>>>>> com.cloud.agent.api.GetVmStatsCommand
>>>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-1:) Request:Seq 788-1319632926: { Cmd , MgmtId:
>>>>>>> 161344838950, via: 788, Ver: v1, Flags: 100111,
>>>>>>> [{"com.cloud.agent.api.StartCommand":{"vm":{"id":
>>>>>>> 2191,"name":"r-2191-VM","type":"DomainRouter","cpus":1,"
>>>>>>> minSpeed":166,"maxSpeed":1000,"minRam":268435456,"maxRam":
>>>>>>> 268435456,"arch":"x86_64","os":"Debian
>>>>>>> GNU/Linux 7(64-bit)","bootArgs":" vpccidr=10.0.0.0/8
>>>>>>> domain=cs2cloud.internal dns1=8.8.8.8 dns2= template=domP
>>>>>>> name=r-2191-VM
>>>>>>> eth0ip=169.254.2.99 eth0mask=255.255.0.0 type=vpcrouter
>>>>>>> disable_rp_filter=true","rebootOnCrash":false,"
>>>>>>> enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":
>>>>>>> false,"vncPassword":"e9e24eb0b1ff8ec2","params":{"
>>>>>>> memoryOvercommitRatio":"1.0","cpuOvercommitRatio":"6.0"},"
>>>>>>> uuid":"485587f7-4a04-4d77-bc36-2367f2adf77e","disks":[{"
>>>>>>> data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
>>>>>>> uuid":"d042e003-5278-461d-8a35-3b1261164f93","
>>>>>>> volumeType":"ROOT","dataStore":{"org.apache.cloudstack.
>>>>>>> storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-
>>>>>>> 88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
>>>>>>> cephmon.domaina.net","path":"cloudstack","port":6789,"url":"RBD://
>>>>>>> cephmon.domaina.net/cloudstack/?ROLE=Primary&
>>>>>>> STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6
>>>>>>> "}},"name":"ROOT-2191","size":2621440000,"path":"d042e003-
>>>>>>> 5278-461d-8a35-3b1261164f93","volumeId":2857,"vmName":"r-
>>>>>>> 2191-VM","accountId":2,"format":"RAW","id":2857,"
>>>>>>> deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"
>>>>>>> d042e003-5278-461d-8a35-3b1261164f93","type":"ROOT","_
>>>>>>> details":{"managed":"false","storagePort":"6789","storageHost":"
>>>>>>> cephmon.domaina.net
>>>>>>> ","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"
>>>>>>> networkRateMbps":-1,"defaultNic":false,"uuid":"
>>>>>>> d27a7aad-2dcc-45c7-bf30-
>>>>>>> 3302d011da66","ip":"169.254.2.99","netmask":"255.255.0.0","
>>>>>>> gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:63","
>>>>>>> broadcastType":"LinkLocal","type":"Control","
>>>>>>> isSecurityGroupEnabled":false}]},"hostIp":"10.44.253.180","
>>>>>>> executeInSequence":false,"wait":0}},{"com.cloud.agent.
>>>>>>> api.check.CheckSshCommand":{"ip":"169.254.2.99","port":
>>>>>>> 3922,"interval":6,"retries":100,"name":"r-2191-VM","wait":
>>>>>>> 0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
>>>>>>> router.name
>>>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>>>>>>> com.cloud.agent.api.PlugNicCommand":{"nic":{"
>>>>>>> deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid"
>>>>>>> :"fe848dbe-cf4b-473e-8124-62289f86b144","ip":"5.226.146.
>>>>>>> 62","netmask":"255.255.255.128","gateway":"5.226.146.1","
>>>>>>> mac":"06:f0:d6:00:01:49","broadcastType":"Vlan","type":"
>>>>>>> Public","broadcastUri":"vlan://untagged","isolationUri":"
>>>>>>> vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"},"
>>>>>>> instanceName":"r-2191-VM","vmType":"DomainRouter","wait":
>>>>>>> 0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"
>>>>>>> ipAddresses":[{"accountId":2,"publicIp":"5.226.146.62","
>>>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>>>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>>>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>>>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>>>>>>> networkName":"breth1-500"}],"accessDetails":{"router.guest.
>>>>>>> ip":"5.226.146.62","zone.network.type":"Advanced","
>>>>>>> router.name
>>>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>>>>>>> com.cloud.agent.api.routing.SetSourceNatCommand":{"
>>>>>>> ipAddress":{"accountId":2,"publicIp":"5.226.146.62","
>>>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>>>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>>>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>>>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>>>>>>> networkName":"breth1-500"},"add":true,"accessDetails":{"
>>>>>>> zone.network.type":"Advanced","
>>>>>>> router.name":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{}] }
>>>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-1:) Processing command:
>>>>>>> com.cloud.agent.api.StartCommand
>>>>>>> 2015-03-10 16:43:58,257{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-5:) Processing command:
>>>>>>> com.cloud.agent.api.GetStorageStatsCommand
>>>>>>> 2015-03-10 16:44:01,747{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
>>>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
>>>>>>> GNU/Linux 7(64-bit)
>>>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-2:) Processing command:
>>>>>>> com.cloud.agent.api.NetworkUsageCommand
>>>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-2:) Executing:
>>>>>>> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
>>>>>>> vpc_netusage.sh 169.254.2.1 -l 46.232.180.55 -g
>>>>>>> 2015-03-10 16:44:06,439{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-2:) Execution is successful.
>>>>>>> 2015-03-10 16:44:06,441{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-2:) Seq 788-1319632928: { Ans: , MgmtId:
>>>>>>> 161344838950, via: 788, Ver: v1, Flags: 10,
>>>>>>> [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-1873-VM","
>>>>>>> bytesSent":198379,"bytesReceived":7380903,"result":true,"details":"","
>>>>>>> wait":0}}]
>>>>>>> }
>>>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [kvm.resource.BridgeVifDriver]
>>>>>>> (agentRequest-Handler-1:) nic=[Nic:Control-169.254.2.99-null]
>>>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [utils.script.Script]
>>>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route | grep
>>>>>>> 169.254.0.0/16
>>>>>>> 2015-03-10 16:44:11,060{GMT} DEBUG [utils.script.Script]
>>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>>> 2015-03-10 16:44:11,061{GMT} DEBUG [utils.script.Script]
>>>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route del
>>>>>>> 169.254.0.0/16
>>>>>>> 2015-03-10 16:44:11,064{GMT} DEBUG [utils.script.Script]
>>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
>>>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
>>>>>>> GNU/Linux 7(64-bit)
>>>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) starting r-2191-VM: <domain type='kvm'>
>>>>>>> <name>r-2191-VM</name>
>>>>>>> <uuid>485587f7-4a04-4d77-bc36-2367f2adf77e</uuid>
>>>>>>> <description>Debian GNU/Linux 7(64-bit)</description>
>>>>>>> <clock offset='utc'>
>>>>>>> </clock>
>>>>>>> <features>
>>>>>>> <pae/>
>>>>>>> <apic/>
>>>>>>> <acpi/>
>>>>>>> </features>
>>>>>>> <devices>
>>>>>>> <emulator>/usr/libexec/qemu-kvm</emulator>
>>>>>>> <interface type='bridge'>
>>>>>>> <source bridge='cloud0'/>
>>>>>>> <mac address='0e:00:a9:fe:02:63'/>
>>>>>>> <model type='virtio'/>
>>>>>>> </interface>
>>>>>>> <serial type='pty'>
>>>>>>> <target port='0'/>
>>>>>>> </serial>
>>>>>>> <graphics type='vnc' autoport='yes' listen='10.44.253.180'
>>>>>>> passwd='e9e24eb0b1ff8ec2'/>
>>>>>>> <disk device='disk' type='network'>
>>>>>>> <driver name='qemu' type='raw' cache='none' />
>>>>>>> <source protocol='rbd'
>>>>>>> name='cloudstack/d042e003-5278-461d-8a35-3b1261164f93'>
>>>>>>> <host name='cephmon.domain.internal' port='6789'/>
>>>>>>> </source>
>>>>>>> <auth username='cloudstack'>
>>>>>>> <secret type='ceph' uuid='5b93422e-1a66-353d-88a8-2203f79b1dc6'/>
>>>>>>> </auth>
>>>>>>> <target dev='vda' bus='virtio'/>
>>>>>>> </disk>
>>>>>>> <disk device='cdrom' type='file'>
>>>>>>> <driver name='qemu' type='raw' cache='none' />
>>>>>>> <source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
>>>>>>> <target dev='hdc' bus='ide'/>
>>>>>>> </disk>
>>>>>>> <console type='pty'>
>>>>>>> <target port='0'/>
>>>>>>> </console>
>>>>>>> <input type='tablet' bus='usb'/>
>>>>>>> <channel type='unix'>
>>>>>>> <source mode='bind' path='/var/lib/libvirt/qemu/r-2191-VM.agent'/>
>>>>>>> <target type='virtio' name='r-2191-VM.vport'/>
>>>>>>> <address type='virtio-serial'/>
>>>>>>> </channel>
>>>>>>> </devices>
>>>>>>> <memory>262144</memory>
>>>>>>> <devices>
>>>>>>> <memballoon model='none'/>
>>>>>>> </devices>
>>>>>>> <vcpu>1</vcpu>
>>>>>>> <os>
>>>>>>> <type arch='x86_64' machine='pc'>hvm</type>
>>>>>>> <boot dev='cdrom'/>
>>>>>>> <boot dev='hd'/>
>>>>>>> </os>
>>>>>>> <cputune>
>>>>>>> <shares>166</shares>
>>>>>>> </cputune>
>>>>>>> <cpu></cpu><on_reboot>restart</on_reboot>
>>>>>>> <on_poweroff>destroy</on_poweroff>
>>>>>>> <on_crash>destroy</on_crash>
>>>>>>> </domain>
>>>>>>>
>>>>>>> 2015-03-10 16:44:11,467{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Executing:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>> hviasocket.pl
>>>>>>> -n
>>>>>>> r-2191-VM -p %vpccidr=
>>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>>
>>>>>>> 2015-03-10 16:44:11,494{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>>> 2015-03-10 16:44:11,495{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:14,501{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>>>> (Thread-10:)
>>>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
>>>>>>> libvirt,
>>>>>>> continuing
>>>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>>>> (Thread-10:)
>>>>>>> Executing:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>>>>>>> -i
>>>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>>>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>>>>>>> 2015-03-10 16:44:17,019{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>>>> (Thread-10:)
>>>>>>> Execution is successful.
>>>>>>> 2015-03-10 16:44:17,953{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
>>>>>>> Sending ping: Seq 788-8: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags:
>>>>>>> 11,
>>>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>>>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>>>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>>>>>>> 18.domaina.net
>>>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>>>>>>> hostType":"Routing","hostId":788,"wait":0}}]
>>>>>>> }
>>>>>>> 2015-03-10 16:44:17,964{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (Agent-Handler-3:)
>>>>>>> Received response: Seq 788-8: { Ans: , MgmtId: 161344838950, via: 788,
>>>>>>> Ver: v1, Flags: 100010,
>>>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>>>>>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>>>>>>> }
>>>>>>> 2015-03-10 16:44:19,502{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:22,509{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:27,510{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Unable to logon to 169.254.2.99
>>>>>>> 2015-03-10 16:44:27,512{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Executing:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>> hviasocket.pl
>>>>>>> -n
>>>>>>> r-2191-VM -p %vpccidr=
>>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>>
>>>>>>> 2015-03-10 16:44:27,540{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>>> 2015-03-10 16:44:27,541{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:30,548{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:31,378{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-3:) Processing command:
>>>>>>> com.cloud.agent.api.GetHostStatsCommand
>>>>>>> 2015-03-10 16:44:31,379{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c idle=$(top -b -n
>>>>>>> 1|grep
>>>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>>>>>>> 2015-03-10 16:44:31,910{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-3:) Execution is successful.
>>>>>>> 2015-03-10 16:44:31,911{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c freeMem=$(free|grep
>>>>>>> cache:|awk '{print $4}');echo $freeMem
>>>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-3:) Execution is successful.
>>>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c free|grep Mem:|awk
>>>>>>> '{print $2}'
>>>>>>> 2015-03-10 16:44:31,920{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-3:) Execution is successful.
>>>>>>> 2015-03-10 16:44:35,549{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:38,556{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Unable to logon to 169.254.2.99
>>>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Executing:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>> hviasocket.pl
>>>>>>> -n
>>>>>>> r-2191-VM -p %vpccidr=
>>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>>
>>>>>>> 2015-03-10 16:44:43,587{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>>> 2015-03-10 16:44:43,588{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:46,594{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:49,449{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-4:) Processing command:
>>>>>>> com.cloud.agent.api.GetVmStatsCommand
>>>>>>> 2015-03-10 16:44:51,595{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:54,600{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:59,601{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Unable to logon to 169.254.2.99
>>>>>>> 2015-03-10 16:44:59,602{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Executing:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>> hviasocket.pl
>>>>>>> -n
>>>>>>> r-2191-VM -p %vpccidr=
>>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>>
>>>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:02,634{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:07,635{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:10,640{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:12,066{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-5:) Processing command:
>>>>>>> com.cloud.agent.api.GetStorageStatsCommand
>>>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Unable to logon to 169.254.2.99
>>>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Executing:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>> hviasocket.pl
>>>>>>> -n
>>>>>>> r-2191-VM -p %vpccidr=
>>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>>
>>>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:17,021{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>>>> (Thread-11:)
>>>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
>>>>>>> libvirt,
>>>>>>> continuing
>>>>>>> 2015-03-10 16:45:17,022{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>>>> (Thread-11:)
>>>>>>> Executing:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>>>>>>> -i
>>>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>>>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>>>>>>> 2015-03-10 16:45:17,036{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>>>> (Thread-11:)
>>>>>>> Execution is successful.
>>>>>>> 2015-03-10 16:45:17,951{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
>>>>>>> Sending ping: Seq 788-9: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags:
>>>>>>> 11,
>>>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>>>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>>>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>>>>>>> 18.domaina.net
>>>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>>>>>>> hostType":"Routing","hostId":788,"wait":0}}]
>>>>>>> }
>>>>>>> 2015-03-10 16:45:18,002{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (Agent-Handler-2:)
>>>>>>> Received response: Seq 788-9: { Ans: , MgmtId: 161344838950, via: 788,
>>>>>>> Ver: v1, Flags: 100010,
>>>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>>>>>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>>>>>>> }
>>>>>>> 2015-03-10 16:45:18,672{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:23,673{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:26,680{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:31,682{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Unable to logon to 169.254.2.99
>>>>>>> 2015-03-10 16:45:31,683{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Executing:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>> hviasocket.pl
>>>>>>> -n
>>>>>>> r-2191-VM -p %vpccidr=
>>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>>
>>>>>>> 2015-03-10 16:45:36,684{GMT} WARN [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (Script-5:) Interrupting script.
>>>>>>> 2015-03-10 16:45:36,686{GMT} WARN [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Timed out:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>> hviasocket.pl
>>>>>>> -n
>>>>>>> r-2191-VM -p %vpccidr=
>>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>> . Output is:
>>>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) passcmd failed:timeout
>>>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-2:) Processing command:
>>>>>>> com.cloud.agent.api.GetHostStatsCommand
>>>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c idle=$(top -b -n
>>>>>>> 1|grep
>>>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>>>>>>> 2015-03-10 16:45:37,967{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-2:) Execution is successful.
>>>>>>> 2015-03-10 16:45:37,968{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c freeMem=$(free|grep
>>>>>>> cache:|awk '{print $4}');echo $freeMem
>>>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-2:) Execution is successful.
>>>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c free|grep Mem:|awk
>>>>>>> '{print $2}'
>>>>>>> 2015-03-10 16:45:37,977{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-2:) Execution is successful.
>>>>>>> 2015-03-10 16:45:39,692{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:44,693{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:47,700{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:51,807{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-3:) Processing command:
>>>>>>> com.cloud.agent.api.GetVmStatsCommand
>>>>>>> 2015-03-10 16:45:52,701{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Unable to logon to 169.254.2.99
>>>>>>>
>>>>>>> On 10 March 2015 at 17:50, ilya musayev <il...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> What kernel are you running? Please include Libvirt and Qemu version.
>>>>>>>
>>>>>>>>
>>>>>>>> On 3/10/15 8:21 AM, Andrija Panic wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>> we are having issues from time to time with starting VR - while
>>>>>>>>> creating
>>>>>>>>> VPC.
>>>>>>>>>
>>>>>>>>> Here is the few lines from MGMT log:
>>>>>>>>>
>>>>>>>>> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
>>>>>>>>> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start vpc [VPC
>>>>>>>>> [621-VPC-delete] due to
>>>>>>>>> com.cloud.exception.AgentUnavailableException: Resource [Host:788]
>>>>>>>>> is
>>>>>>>>> unreachable: Host 788: Unable to start instance due to Unable to
>>>>>>>>> start
>>>>>>>>> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not
>>>>>>>>> retrying
>>>>>>>>> at
>>>>>>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
>>>>>>>>> VirtualMachineManagerImpl.java:1072)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The interesting part is on the Agent itself:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.
>>>>>>>>> LibvirtComputingResource]
>>>>>>>>> (agentRequest-Handler-2:) Timed out:
>>>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>>>> hviasocket.pl
>>>>>>>>> -n
>>>>>>>>> r-2189-VM -p %vpccidr=
>>>>>>>>> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>>>>> template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=
>>>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>>>> . Output is:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Interesting stuff is - it's seems somewhat random, 3-4 out of 5
>>>>>>>>> works.
>>>>>>>>>
>>>>>>>>> Any suggestion are really appreciated, since this is live system :(
>>>>>>>>> ACS 4.3.0, KVM, CEPH
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Andrija Panić
>>>
>>
>>
>>
>> --
>>
>> Andrija Panić
Re: [HELP] VR sometimes fail to start
Posted by Andrija Panic <an...@gmail.com>.
Args.... so this was the reason definitively: bash -c ip route - does
not work, and then CS agent deletes 169.254.0.0 route completely...
why - have no damn clue...
I made simple cron, creating route, sleep 30sec, then create route again...
- so at least VR creation will be prolonged for max of 30sec...but will be
created.
Just created bunch of VPCs with 100% sucess...
Any info, please = why is 169.254.0.0 route being deleted ?
On 14 March 2015 at 18:50, Andrija Panic <an...@gmail.com> wrote:
> I think I located the problem, sort of:
>
> it seems like ACS agent is deleting route to 168.254.0.0 - so after the
> patchviasocket - agent can not connect to private interface on VM
> (169.254.x.x)
>
> if I try to provision VPC - it waits, and waits, and waits... until I add
> route manually on the particualr host (ip route add 169.254.0.0/16 dev
> cloud0) - and then VPC creation (VR creation) is completeled.
>
> Any idea why is 169.254 route being deleted...: ???
>
>
> From log:
>
> 2015-03-14 18:41:32,187 DEBUG [utils.script.Script]
> (agentRequest-Handler-5:null) Executing: /bin/bash -c ip route | grep
> 169.254.0.0/16
> 2015-03-14 18:41:32,193 DEBUG [utils.script.Script]
> (agentRequest-Handler-5:null) Executing: /bin/bash -c ip route del
> 169.254.0.0/16
> 2015-03-14 18:41:32,648 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
> -n r-2236-VM -p %vpccidr=
> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2236-VM%eth0ip=169.254.2.56%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
> 2015-03-14 18:41:32,676 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-5:null) Trying to connect to 169.254.2.56
> 2015-03-14 18:41:35,682 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-5:null) Could not connect to 169.254.2.56
>
> and the first command /bin/bash -c ip route | grep 169.254.0.0/16
> fails in my case if I run it manually on centos
>
> ip route - works, but bash -c ip route does not work
> ACS 4.3.2 now...
>
>
> [root@cs18 ~]# /bin/bash -c ip route | grep 169.254.0.0/16
> Usage: ip [ OPTIONS ] OBJECT { COMMAND | help }
> ip [ -force ] -batch filename
> where OBJECT := { link | addr | addrlabel | route | rule | neigh | ntable
> |
> tunnel | maddr | mroute | mrule | monitor | xfrm |
> token }
> OPTIONS := { -V[ersion] | -s[tatistics] | -d[etails] | -r[esolve] |
> -f[amily] { inet | inet6 | ipx | dnet | link } |
> -o[neline] | -t[imestamp] | -b[atch] [filename] |
> -rc[vbuf] [size]}
>
> You can see this command fails on my CentOS 6.6...
>
> not sure what the hack is happening...
>
> On 14 March 2015 at 17:10, Andrija Panic <an...@gmail.com> wrote:
>
>> Yes I understadn that - more of a problem, qemu-img from CentOS 6.5
>> doesnt work snapshots (redhat removed -S flag, etc.), then we need RBD
>> support (for CEPH), which RH does not support in normal RHEL/CentOS, etc.
>>
>> Next cloud deployement - dump the the RHEL/CentOS, and go with Ubuntu
>> 14.04, which does support everythgin out of the box. I'm so frustrated with
>> trying anything new with CentOS/RHEL...simply hurts...
>>
>> On 14 March 2015 at 17:04, Marcus <sh...@gmail.com> wrote:
>>
>>> Yeah, I don't remember all of the discussion around the minimum
>>> version and why, but each libvirt and qemu has such different behavior
>>> (not all good), that we decided to target a minimum regardless of what
>>> was supplied by the distros. It's just too hard to support them all
>>> with their differing behavior. Open stack ran into the same issues.
>>>
>>> On Sat, Mar 14, 2015 at 9:02 AM, Andrija Panic <an...@gmail.com>
>>> wrote:
>>> > Marcus - QEMU is 0.12.x (CentOS 6.4)
>>> >
>>> > but libvirt is compiled 1.2.3
>>> >
>>> > On 14 March 2015 at 16:58, Marcus <sh...@gmail.com> wrote:
>>> >
>>> >> Oh, this is an unsupported version. should be using libvirt 0.9.4+ and
>>> >> Qemu 1.0+ per the install docs.
>>> >>
>>> >> On Sat, Mar 14, 2015 at 8:56 AM, Andrija Panic <
>>> andrija.panic@gmail.com>
>>> >> wrote:
>>> >> > Tried the proposed solution - but it still don't work - BTW my qemu
>>> is
>>> >> > 0.12.x - from CentOS 6.4...
>>> >> >
>>> >> > When I manually start the script with all parameters, it simply
>>> hangs, no
>>> >> > output at all, until ctrl + c
>>> >> >
>>> >> > BTW this is with now upgraded ACS to 4.3.2...
>>> >> >
>>> >> > :(
>>> >> >
>>> >> > On 12 March 2015 at 09:24, Andrija Panic <an...@gmail.com>
>>> >> wrote:
>>> >> >
>>> >> >> Hi Marcus, will do - I'm just unconfortable changing live things,
>>> but
>>> >> that
>>> >> >> is my problem. Will let you know of the output.
>>> >> >>
>>> >> >> Thanks
>>> >> >>
>>> >> >> On 12 March 2015 at 01:34, ilya musayev <
>>> ilya.mailing.lists@gmail.com>
>>> >> >> wrote:
>>> >> >>
>>> >> >>> Andrija,
>>> >> >>>
>>> >> >>> Have you tried the solution Loic mentioned? Its seems like an
>>> easy one
>>> >> to
>>> >> >>> try..
>>> >> >>>
>>> >> >>> Thanks
>>> >> >>> ilya
>>> >> >>>
>>> >> >>>
>>> >> >>> On 3/11/15 3:54 AM, Andrija Panic wrote:
>>> >> >>>
>>> >> >>>> Oh well, I will then update to 4.3.2 anyway, and hope to see the
>>> >> problem
>>> >> >>>> go
>>> >> >>>> away...
>>> >> >>>>
>>> >> >>>> THanks guys
>>> >> >>>>
>>> >> >>>> On 10 March 2015 at 22:45, Loic Lambiel <
>>> loic.lambiel@exoscale.ch>
>>> >> >>>> wrote:
>>> >> >>>>
>>> >> >>>> Hi,
>>> >> >>>>>
>>> >> >>>>> Maybe the same issue we faced here:
>>> https://github.com/exoscale/
>>> >> >>>>> cloudstack/commit/df3e7533a78c3594cdd39b8b18e6f4a3abcb8cfd
>>> >> >>>>>
>>> >> >>>>> The patch has not been yet submitted upstream. I'll do it once
>>> it is
>>> >> >>>>> running on our production.
>>> >> >>>>>
>>> >> >>>>> Anyway, you may easily test it by performing the small change
>>> >> directly
>>> >> >>>>> into the script.
>>> >> >>>>>
>>> >> >>>>> Hope that helps.
>>> >> >>>>>
>>> >> >>>>> Regards,
>>> >> >>>>>
>>> >> >>>>> Loic Lambiel
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> On 10/03/15 19:40, Andrija Panic wrote:
>>> >> >>>>>
>>> >> >>>>> Hi guys,
>>> >> >>>>>>
>>> >> >>>>>> Kernel is Elrepo 3.10.67 (CentOS 6.6)
>>> >> >>>>>> Libvirt manually compiled 1.2.3 with RBD support.
>>> >> >>>>>> Qemu 0.12.1.2 - Inktank patched original RHEL 6.4 rpms with RBD
>>> >> >>>>>> support.
>>> >> >>>>>>
>>> >> >>>>>> And here is the output, with debug mode, from Agent.
>>> >> >>>>>>
>>> >> >>>>>> At the begining I see agent deleting routes for 169.254.x.x
>>> >> network...:
>>> >> >>>>>>
>>> >> >>>>>> Basically, It seems like the
>>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>> >> >>>>>> hviasocket.pl
>>> >> >>>>>> is
>>> >> >>>>>> failing, not being able to connect to socket...
>>> >> >>>>>> And I could not ping VR (169.254.2.99)...
>>> >> >>>>>>
>>> >> >>>>>> LOG:
>>> >> >>>>>>
>>> >> >>>>>> 2015-03-10 16:43:46,976{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-4:) Processing command:
>>> >> >>>>>> com.cloud.agent.api.GetVmStatsCommand
>>> >> >>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-1:) Request:Seq 788-1319632926: { Cmd ,
>>> >> MgmtId:
>>> >> >>>>>> 161344838950, via: 788, Ver: v1, Flags: 100111,
>>> >> >>>>>> [{"com.cloud.agent.api.StartCommand":{"vm":{"id":
>>> >> >>>>>> 2191,"name":"r-2191-VM","type":"DomainRouter","cpus":1,"
>>> >> >>>>>> minSpeed":166,"maxSpeed":1000,"minRam":268435456,"maxRam":
>>> >> >>>>>> 268435456,"arch":"x86_64","os":"Debian
>>> >> >>>>>> GNU/Linux 7(64-bit)","bootArgs":" vpccidr=10.0.0.0/8
>>> >> >>>>>> domain=cs2cloud.internal dns1=8.8.8.8 dns2= template=domP
>>> >> >>>>>> name=r-2191-VM
>>> >> >>>>>> eth0ip=169.254.2.99 eth0mask=255.255.0.0 type=vpcrouter
>>> >> >>>>>> disable_rp_filter=true","rebootOnCrash":false,"
>>> >> >>>>>> enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":
>>> >> >>>>>> false,"vncPassword":"e9e24eb0b1ff8ec2","params":{"
>>> >> >>>>>> memoryOvercommitRatio":"1.0","cpuOvercommitRatio":"6.0"},"
>>> >> >>>>>> uuid":"485587f7-4a04-4d77-bc36-2367f2adf77e","disks":[{"
>>> >> >>>>>> data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
>>> >> >>>>>> uuid":"d042e003-5278-461d-8a35-3b1261164f93","
>>> >> >>>>>> volumeType":"ROOT","dataStore":{"org.apache.cloudstack.
>>> >> >>>>>> storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-
>>> >> >>>>>> 88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
>>> >> >>>>>> cephmon.domaina.net
>>> ","path":"cloudstack","port":6789,"url":"RBD://
>>> >> >>>>>> cephmon.domaina.net/cloudstack/?ROLE=Primary&
>>> >> >>>>>> STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6
>>> >> >>>>>> "}},"name":"ROOT-2191","size":2621440000,"path":"d042e003-
>>> >> >>>>>> 5278-461d-8a35-3b1261164f93","volumeId":2857,"vmName":"r-
>>> >> >>>>>> 2191-VM","accountId":2,"format":"RAW","id":2857,"
>>> >> >>>>>> deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"
>>> >> >>>>>> d042e003-5278-461d-8a35-3b1261164f93","type":"ROOT","_
>>> >> >>>>>>
>>> details":{"managed":"false","storagePort":"6789","storageHost":"
>>> >> >>>>>> cephmon.domaina.net
>>> >> >>>>>> ","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"
>>> >> >>>>>> networkRateMbps":-1,"defaultNic":false,"uuid":"
>>> >> >>>>>> d27a7aad-2dcc-45c7-bf30-
>>> >> >>>>>> 3302d011da66","ip":"169.254.2.99","netmask":"255.255.0.0","
>>> >> >>>>>> gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:63","
>>> >> >>>>>> broadcastType":"LinkLocal","type":"Control","
>>> >> >>>>>> isSecurityGroupEnabled":false}]},"hostIp":"10.44.253.180","
>>> >> >>>>>> executeInSequence":false,"wait":0}},{"com.cloud.agent.
>>> >> >>>>>> api.check.CheckSshCommand":{"ip":"169.254.2.99","port":
>>> >> >>>>>> 3922,"interval":6,"retries":100,"name":"r-2191-VM","wait":
>>> >> >>>>>>
>>> 0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
>>> >> >>>>>> router.name
>>> >> >>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>>> >> >>>>>> com.cloud.agent.api.PlugNicCommand":{"nic":{"
>>> >> >>>>>> deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid"
>>> >> >>>>>> :"fe848dbe-cf4b-473e-8124-62289f86b144","ip":"5.226.146.
>>> >> >>>>>> 62","netmask":"255.255.255.128","gateway":"5.226.146.1","
>>> >> >>>>>> mac":"06:f0:d6:00:01:49","broadcastType":"Vlan","type":"
>>> >> >>>>>> Public","broadcastUri":"vlan://untagged","isolationUri":"
>>> >> >>>>>>
>>> >> vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"},"
>>> >> >>>>>> instanceName":"r-2191-VM","vmType":"DomainRouter","wait":
>>> >> >>>>>> 0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"
>>> >> >>>>>> ipAddresses":[{"accountId":2,"publicIp":"5.226.146.62","
>>> >> >>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>>> >> >>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>>> >> >>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>>> >> >>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>>> >> >>>>>> networkName":"breth1-500"}],"accessDetails":{"router.guest.
>>> >> >>>>>> ip":"5.226.146.62","zone.network.type":"Advanced","
>>> >> >>>>>> router.name
>>> >> >>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>>> >> >>>>>> com.cloud.agent.api.routing.SetSourceNatCommand":{"
>>> >> >>>>>> ipAddress":{"accountId":2,"publicIp":"5.226.146.62","
>>> >> >>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>>> >> >>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>>> >> >>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>>> >> >>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>>> >> >>>>>> networkName":"breth1-500"},"add":true,"accessDetails":{"
>>> >> >>>>>> zone.network.type":"Advanced","
>>> >> >>>>>> router.name
>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{}]
>>> >> }
>>> >> >>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-1:) Processing command:
>>> >> >>>>>> com.cloud.agent.api.StartCommand
>>> >> >>>>>> 2015-03-10 16:43:58,257{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-5:) Processing command:
>>> >> >>>>>> com.cloud.agent.api.GetStorageStatsCommand
>>> >> >>>>>> 2015-03-10 16:44:01,747{GMT} DEBUG
>>> [kvm.resource.KVMGuestOsMapper]
>>> >> >>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os:
>>> Debian
>>> >> >>>>>> GNU/Linux 7(64-bit)
>>> >> >>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-2:) Processing command:
>>> >> >>>>>> com.cloud.agent.api.NetworkUsageCommand
>>> >> >>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-2:) Executing:
>>> >> >>>>>>
>>> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
>>> >> >>>>>> vpc_netusage.sh 169.254.2.1 -l 46.232.180.55 -g
>>> >> >>>>>> 2015-03-10 16:44:06,439{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-2:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:06,441{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-2:) Seq 788-1319632928: { Ans: , MgmtId:
>>> >> >>>>>> 161344838950, via: 788, Ver: v1, Flags: 10,
>>> >> >>>>>>
>>> >> [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-1873-VM","
>>> >> >>>>>>
>>> >> bytesSent":198379,"bytesReceived":7380903,"result":true,"details":"","
>>> >> >>>>>> wait":0}}]
>>> >> >>>>>> }
>>> >> >>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG
>>> [kvm.resource.BridgeVifDriver]
>>> >> >>>>>> (agentRequest-Handler-1:) nic=[Nic:Control-169.254.2.99-null]
>>> >> >>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [utils.script.Script]
>>> >> >>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route |
>>> grep
>>> >> >>>>>> 169.254.0.0/16
>>> >> >>>>>> 2015-03-10 16:44:11,060{GMT} DEBUG [utils.script.Script]
>>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:11,061{GMT} DEBUG [utils.script.Script]
>>> >> >>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route del
>>> >> >>>>>> 169.254.0.0/16
>>> >> >>>>>> 2015-03-10 16:44:11,064{GMT} DEBUG [utils.script.Script]
>>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG
>>> [kvm.resource.KVMGuestOsMapper]
>>> >> >>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os:
>>> Debian
>>> >> >>>>>> GNU/Linux 7(64-bit)
>>> >> >>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) starting r-2191-VM: <domain
>>> type='kvm'>
>>> >> >>>>>> <name>r-2191-VM</name>
>>> >> >>>>>> <uuid>485587f7-4a04-4d77-bc36-2367f2adf77e</uuid>
>>> >> >>>>>> <description>Debian GNU/Linux 7(64-bit)</description>
>>> >> >>>>>> <clock offset='utc'>
>>> >> >>>>>> </clock>
>>> >> >>>>>> <features>
>>> >> >>>>>> <pae/>
>>> >> >>>>>> <apic/>
>>> >> >>>>>> <acpi/>
>>> >> >>>>>> </features>
>>> >> >>>>>> <devices>
>>> >> >>>>>> <emulator>/usr/libexec/qemu-kvm</emulator>
>>> >> >>>>>> <interface type='bridge'>
>>> >> >>>>>> <source bridge='cloud0'/>
>>> >> >>>>>> <mac address='0e:00:a9:fe:02:63'/>
>>> >> >>>>>> <model type='virtio'/>
>>> >> >>>>>> </interface>
>>> >> >>>>>> <serial type='pty'>
>>> >> >>>>>> <target port='0'/>
>>> >> >>>>>> </serial>
>>> >> >>>>>> <graphics type='vnc' autoport='yes' listen='10.44.253.180'
>>> >> >>>>>> passwd='e9e24eb0b1ff8ec2'/>
>>> >> >>>>>> <disk device='disk' type='network'>
>>> >> >>>>>> <driver name='qemu' type='raw' cache='none' />
>>> >> >>>>>> <source protocol='rbd'
>>> >> >>>>>> name='cloudstack/d042e003-5278-461d-8a35-3b1261164f93'>
>>> >> >>>>>> <host name='cephmon.domain.internal' port='6789'/>
>>> >> >>>>>> </source>
>>> >> >>>>>> <auth username='cloudstack'>
>>> >> >>>>>> <secret type='ceph'
>>> uuid='5b93422e-1a66-353d-88a8-2203f79b1dc6'/>
>>> >> >>>>>> </auth>
>>> >> >>>>>> <target dev='vda' bus='virtio'/>
>>> >> >>>>>> </disk>
>>> >> >>>>>> <disk device='cdrom' type='file'>
>>> >> >>>>>> <driver name='qemu' type='raw' cache='none' />
>>> >> >>>>>> <source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
>>> >> >>>>>> <target dev='hdc' bus='ide'/>
>>> >> >>>>>> </disk>
>>> >> >>>>>> <console type='pty'>
>>> >> >>>>>> <target port='0'/>
>>> >> >>>>>> </console>
>>> >> >>>>>> <input type='tablet' bus='usb'/>
>>> >> >>>>>> <channel type='unix'>
>>> >> >>>>>> <source mode='bind'
>>> path='/var/lib/libvirt/qemu/r-2191-VM.agent'/>
>>> >> >>>>>> <target type='virtio' name='r-2191-VM.vport'/>
>>> >> >>>>>> <address type='virtio-serial'/>
>>> >> >>>>>> </channel>
>>> >> >>>>>> </devices>
>>> >> >>>>>> <memory>262144</memory>
>>> >> >>>>>> <devices>
>>> >> >>>>>> <memballoon model='none'/>
>>> >> >>>>>> </devices>
>>> >> >>>>>> <vcpu>1</vcpu>
>>> >> >>>>>> <os>
>>> >> >>>>>> <type arch='x86_64' machine='pc'>hvm</type>
>>> >> >>>>>> <boot dev='cdrom'/>
>>> >> >>>>>> <boot dev='hd'/>
>>> >> >>>>>> </os>
>>> >> >>>>>> <cputune>
>>> >> >>>>>> <shares>166</shares>
>>> >> >>>>>> </cputune>
>>> >> >>>>>> <cpu></cpu><on_reboot>restart</on_reboot>
>>> >> >>>>>> <on_poweroff>destroy</on_poweroff>
>>> >> >>>>>> <on_crash>destroy</on_crash>
>>> >> >>>>>> </domain>
>>> >> >>>>>>
>>> >> >>>>>> 2015-03-10 16:44:11,467{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Executing:
>>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>> >> >>>>>> hviasocket.pl
>>> >> >>>>>> -n
>>> >> >>>>>> r-2191-VM -p %vpccidr=
>>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> >> >>>>>>
>>> >> >>>>>> 2015-03-10 16:44:11,494{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:11,495{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:14,501{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>> >> >>>>>> (Thread-10:)
>>> >> >>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
>>> >> >>>>>> libvirt,
>>> >> >>>>>> continuing
>>> >> >>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>> >> >>>>>> (Thread-10:)
>>> >> >>>>>> Executing:
>>> >> >>>>>>
>>> >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>>> >> >>>>>> -i
>>> >> >>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>>> >> >>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>>> >> >>>>>> 2015-03-10 16:44:17,019{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>> >> >>>>>> (Thread-10:)
>>> >> >>>>>> Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:17,953{GMT} DEBUG [cloud.agent.Agent]
>>> >> (UgentTask-5:)
>>> >> >>>>>> Sending ping: Seq 788-8: { Cmd , MgmtId: -1, via: 788, Ver:
>>> v1,
>>> >> Flags:
>>> >> >>>>>> 11,
>>> >> >>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>>> >> >>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>>> >> >>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>>> >> >>>>>> 18.domaina.net
>>> >> >>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>>> >> >>>>>> hostType":"Routing","hostId":788,"wait":0}}]
>>> >> >>>>>> }
>>> >> >>>>>> 2015-03-10 16:44:17,964{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (Agent-Handler-3:)
>>> >> >>>>>> Received response: Seq 788-8: { Ans: , MgmtId: 161344838950,
>>> via:
>>> >> 788,
>>> >> >>>>>> Ver: v1, Flags: 100010,
>>> >> >>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>>> >> >>>>>>
>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>>> >> >>>>>> }
>>> >> >>>>>> 2015-03-10 16:44:19,502{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:22,509{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:27,510{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Unable to logon to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:27,512{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Executing:
>>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>> >> >>>>>> hviasocket.pl
>>> >> >>>>>> -n
>>> >> >>>>>> r-2191-VM -p %vpccidr=
>>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> >> >>>>>>
>>> >> >>>>>> 2015-03-10 16:44:27,540{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:27,541{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:30,548{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:31,378{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-3:) Processing command:
>>> >> >>>>>> com.cloud.agent.api.GetHostStatsCommand
>>> >> >>>>>> 2015-03-10 16:44:31,379{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c idle=$(top
>>> -b -n
>>> >> >>>>>> 1|grep
>>> >> >>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>>> >> >>>>>> 2015-03-10 16:44:31,910{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-3:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:31,911{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c
>>> >> freeMem=$(free|grep
>>> >> >>>>>> cache:|awk '{print $4}');echo $freeMem
>>> >> >>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-3:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c free|grep
>>> Mem:|awk
>>> >> >>>>>> '{print $2}'
>>> >> >>>>>> 2015-03-10 16:44:31,920{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-3:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:35,549{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:38,556{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Unable to logon to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Executing:
>>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>> >> >>>>>> hviasocket.pl
>>> >> >>>>>> -n
>>> >> >>>>>> r-2191-VM -p %vpccidr=
>>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> >> >>>>>>
>>> >> >>>>>> 2015-03-10 16:44:43,587{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:43,588{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:46,594{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:49,449{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-4:) Processing command:
>>> >> >>>>>> com.cloud.agent.api.GetVmStatsCommand
>>> >> >>>>>> 2015-03-10 16:44:51,595{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:54,600{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:59,601{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Unable to logon to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:59,602{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Executing:
>>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>> >> >>>>>> hviasocket.pl
>>> >> >>>>>> -n
>>> >> >>>>>> r-2191-VM -p %vpccidr=
>>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> >> >>>>>>
>>> >> >>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:02,634{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:07,635{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:10,640{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:12,066{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-5:) Processing command:
>>> >> >>>>>> com.cloud.agent.api.GetStorageStatsCommand
>>> >> >>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Unable to logon to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Executing:
>>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>> >> >>>>>> hviasocket.pl
>>> >> >>>>>> -n
>>> >> >>>>>> r-2191-VM -p %vpccidr=
>>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> >> >>>>>>
>>> >> >>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:17,021{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>> >> >>>>>> (Thread-11:)
>>> >> >>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
>>> >> >>>>>> libvirt,
>>> >> >>>>>> continuing
>>> >> >>>>>> 2015-03-10 16:45:17,022{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>> >> >>>>>> (Thread-11:)
>>> >> >>>>>> Executing:
>>> >> >>>>>>
>>> >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>>> >> >>>>>> -i
>>> >> >>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>>> >> >>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>>> >> >>>>>> 2015-03-10 16:45:17,036{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>> >> >>>>>> (Thread-11:)
>>> >> >>>>>> Execution is successful.
>>> >> >>>>>> 2015-03-10 16:45:17,951{GMT} DEBUG [cloud.agent.Agent]
>>> >> (UgentTask-5:)
>>> >> >>>>>> Sending ping: Seq 788-9: { Cmd , MgmtId: -1, via: 788, Ver:
>>> v1,
>>> >> Flags:
>>> >> >>>>>> 11,
>>> >> >>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>>> >> >>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>>> >> >>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>>> >> >>>>>> 18.domaina.net
>>> >> >>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>>> >> >>>>>> hostType":"Routing","hostId":788,"wait":0}}]
>>> >> >>>>>> }
>>> >> >>>>>> 2015-03-10 16:45:18,002{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (Agent-Handler-2:)
>>> >> >>>>>> Received response: Seq 788-9: { Ans: , MgmtId: 161344838950,
>>> via:
>>> >> 788,
>>> >> >>>>>> Ver: v1, Flags: 100010,
>>> >> >>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>>> >> >>>>>>
>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>>> >> >>>>>> }
>>> >> >>>>>> 2015-03-10 16:45:18,672{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:23,673{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:26,680{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:31,682{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Unable to logon to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:31,683{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Executing:
>>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>> >> >>>>>> hviasocket.pl
>>> >> >>>>>> -n
>>> >> >>>>>> r-2191-VM -p %vpccidr=
>>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> >> >>>>>>
>>> >> >>>>>> 2015-03-10 16:45:36,684{GMT} WARN [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (Script-5:) Interrupting script.
>>> >> >>>>>> 2015-03-10 16:45:36,686{GMT} WARN [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Timed out:
>>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>> >> >>>>>> hviasocket.pl
>>> >> >>>>>> -n
>>> >> >>>>>> r-2191-VM -p %vpccidr=
>>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> >> >>>>>> . Output is:
>>> >> >>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) passcmd failed:timeout
>>> >> >>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-2:) Processing command:
>>> >> >>>>>> com.cloud.agent.api.GetHostStatsCommand
>>> >> >>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c idle=$(top
>>> -b -n
>>> >> >>>>>> 1|grep
>>> >> >>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>>> >> >>>>>> 2015-03-10 16:45:37,967{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-2:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:45:37,968{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c
>>> >> freeMem=$(free|grep
>>> >> >>>>>> cache:|awk '{print $4}');echo $freeMem
>>> >> >>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-2:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c free|grep
>>> Mem:|awk
>>> >> >>>>>> '{print $2}'
>>> >> >>>>>> 2015-03-10 16:45:37,977{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-2:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:45:39,692{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:44,693{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:47,700{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:51,807{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-3:) Processing command:
>>> >> >>>>>> com.cloud.agent.api.GetVmStatsCommand
>>> >> >>>>>> 2015-03-10 16:45:52,701{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Unable to logon to 169.254.2.99
>>> >> >>>>>>
>>> >> >>>>>> On 10 March 2015 at 17:50, ilya musayev <
>>> >> ilya.mailing.lists@gmail.com>
>>> >> >>>>>> wrote:
>>> >> >>>>>>
>>> >> >>>>>> What kernel are you running? Please include Libvirt and Qemu
>>> >> version.
>>> >> >>>>>>
>>> >> >>>>>>>
>>> >> >>>>>>> On 3/10/15 8:21 AM, Andrija Panic wrote:
>>> >> >>>>>>>
>>> >> >>>>>>> Hi,
>>> >> >>>>>>>
>>> >> >>>>>>>> we are having issues from time to time with starting VR -
>>> while
>>> >> >>>>>>>> creating
>>> >> >>>>>>>> VPC.
>>> >> >>>>>>>>
>>> >> >>>>>>>> Here is the few lines from MGMT log:
>>> >> >>>>>>>>
>>> >> >>>>>>>> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
>>> >> >>>>>>>> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start
>>> vpc
>>> >> [VPC
>>> >> >>>>>>>> [621-VPC-delete] due to
>>> >> >>>>>>>> com.cloud.exception.AgentUnavailableException: Resource
>>> [Host:788]
>>> >> >>>>>>>> is
>>> >> >>>>>>>> unreachable: Host 788: Unable to start instance due to
>>> Unable to
>>> >> >>>>>>>> start
>>> >> >>>>>>>> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not
>>> >> >>>>>>>> retrying
>>> >> >>>>>>>> at
>>> >> >>>>>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
>>> >> >>>>>>>> VirtualMachineManagerImpl.java:1072)
>>> >> >>>>>>>>
>>> >> >>>>>>>>
>>> >> >>>>>>>> The interesting part is on the Agent itself:
>>> >> >>>>>>>>
>>> >> >>>>>>>>
>>> >> >>>>>>>> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.
>>> >> >>>>>>>> LibvirtComputingResource]
>>> >> >>>>>>>> (agentRequest-Handler-2:) Timed out:
>>> >> >>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>> >> >>>>>>>> hviasocket.pl
>>> >> >>>>>>>> -n
>>> >> >>>>>>>> r-2189-VM -p %vpccidr=
>>> >> >>>>>>>> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%
>>> >> >>>>>>>> template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=
>>> >> >>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> >> >>>>>>>> . Output is:
>>> >> >>>>>>>>
>>> >> >>>>>>>>
>>> >> >>>>>>>> Interesting stuff is - it's seems somewhat random, 3-4 out
>>> of 5
>>> >> >>>>>>>> works.
>>> >> >>>>>>>>
>>> >> >>>>>>>> Any suggestion are really appreciated, since this is live
>>> system
>>> >> :(
>>> >> >>>>>>>> ACS 4.3.0, KVM, CEPH
>>> >> >>>>>>>>
>>> >> >>>>>>>> Thanks,
>>> >> >>>>>>>>
>>> >> >>>>>>>>
>>> >> >>>>>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>
>>> >> >>>
>>> >> >>
>>> >> >>
>>> >> >> --
>>> >> >>
>>> >> >> Andrija Panić
>>> >> >>
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> >
>>> >> > Andrija Panić
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > Andrija Panić
>>>
>>
>>
>>
>> --
>>
>> Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: [HELP] VR sometimes fail to start
Posted by Andrija Panic <an...@gmail.com>.
I think I located the problem, sort of:
it seems like ACS agent is deleting route to 168.254.0.0 - so after the
patchviasocket - agent can not connect to private interface on VM
(169.254.x.x)
if I try to provision VPC - it waits, and waits, and waits... until I add
route manually on the particualr host (ip route add 169.254.0.0/16 dev
cloud0) - and then VPC creation (VR creation) is completeled.
Any idea why is 169.254 route being deleted...: ???
>From log:
2015-03-14 18:41:32,187 DEBUG [utils.script.Script]
(agentRequest-Handler-5:null) Executing: /bin/bash -c ip route | grep
169.254.0.0/16
2015-03-14 18:41:32,193 DEBUG [utils.script.Script]
(agentRequest-Handler-5:null) Executing: /bin/bash -c ip route del
169.254.0.0/16
2015-03-14 18:41:32,648 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) Executing:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
r-2236-VM -p %vpccidr=
10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2236-VM%eth0ip=169.254.2.56%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
2015-03-14 18:41:32,676 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) Trying to connect to 169.254.2.56
2015-03-14 18:41:35,682 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) Could not connect to 169.254.2.56
and the first command /bin/bash -c ip route | grep 169.254.0.0/16 fails
in my case if I run it manually on centos
ip route - works, but bash -c ip route does not work
ACS 4.3.2 now...
[root@cs18 ~]# /bin/bash -c ip route | grep 169.254.0.0/16
Usage: ip [ OPTIONS ] OBJECT { COMMAND | help }
ip [ -force ] -batch filename
where OBJECT := { link | addr | addrlabel | route | rule | neigh | ntable |
tunnel | maddr | mroute | mrule | monitor | xfrm | token
}
OPTIONS := { -V[ersion] | -s[tatistics] | -d[etails] | -r[esolve] |
-f[amily] { inet | inet6 | ipx | dnet | link } |
-o[neline] | -t[imestamp] | -b[atch] [filename] |
-rc[vbuf] [size]}
You can see this command fails on my CentOS 6.6...
not sure what the hack is happening...
On 14 March 2015 at 17:10, Andrija Panic <an...@gmail.com> wrote:
> Yes I understadn that - more of a problem, qemu-img from CentOS 6.5 doesnt
> work snapshots (redhat removed -S flag, etc.), then we need RBD support
> (for CEPH), which RH does not support in normal RHEL/CentOS, etc.
>
> Next cloud deployement - dump the the RHEL/CentOS, and go with Ubuntu
> 14.04, which does support everythgin out of the box. I'm so frustrated with
> trying anything new with CentOS/RHEL...simply hurts...
>
> On 14 March 2015 at 17:04, Marcus <sh...@gmail.com> wrote:
>
>> Yeah, I don't remember all of the discussion around the minimum
>> version and why, but each libvirt and qemu has such different behavior
>> (not all good), that we decided to target a minimum regardless of what
>> was supplied by the distros. It's just too hard to support them all
>> with their differing behavior. Open stack ran into the same issues.
>>
>> On Sat, Mar 14, 2015 at 9:02 AM, Andrija Panic <an...@gmail.com>
>> wrote:
>> > Marcus - QEMU is 0.12.x (CentOS 6.4)
>> >
>> > but libvirt is compiled 1.2.3
>> >
>> > On 14 March 2015 at 16:58, Marcus <sh...@gmail.com> wrote:
>> >
>> >> Oh, this is an unsupported version. should be using libvirt 0.9.4+ and
>> >> Qemu 1.0+ per the install docs.
>> >>
>> >> On Sat, Mar 14, 2015 at 8:56 AM, Andrija Panic <
>> andrija.panic@gmail.com>
>> >> wrote:
>> >> > Tried the proposed solution - but it still don't work - BTW my qemu
>> is
>> >> > 0.12.x - from CentOS 6.4...
>> >> >
>> >> > When I manually start the script with all parameters, it simply
>> hangs, no
>> >> > output at all, until ctrl + c
>> >> >
>> >> > BTW this is with now upgraded ACS to 4.3.2...
>> >> >
>> >> > :(
>> >> >
>> >> > On 12 March 2015 at 09:24, Andrija Panic <an...@gmail.com>
>> >> wrote:
>> >> >
>> >> >> Hi Marcus, will do - I'm just unconfortable changing live things,
>> but
>> >> that
>> >> >> is my problem. Will let you know of the output.
>> >> >>
>> >> >> Thanks
>> >> >>
>> >> >> On 12 March 2015 at 01:34, ilya musayev <
>> ilya.mailing.lists@gmail.com>
>> >> >> wrote:
>> >> >>
>> >> >>> Andrija,
>> >> >>>
>> >> >>> Have you tried the solution Loic mentioned? Its seems like an easy
>> one
>> >> to
>> >> >>> try..
>> >> >>>
>> >> >>> Thanks
>> >> >>> ilya
>> >> >>>
>> >> >>>
>> >> >>> On 3/11/15 3:54 AM, Andrija Panic wrote:
>> >> >>>
>> >> >>>> Oh well, I will then update to 4.3.2 anyway, and hope to see the
>> >> problem
>> >> >>>> go
>> >> >>>> away...
>> >> >>>>
>> >> >>>> THanks guys
>> >> >>>>
>> >> >>>> On 10 March 2015 at 22:45, Loic Lambiel <loic.lambiel@exoscale.ch
>> >
>> >> >>>> wrote:
>> >> >>>>
>> >> >>>> Hi,
>> >> >>>>>
>> >> >>>>> Maybe the same issue we faced here: https://github.com/exoscale/
>> >> >>>>> cloudstack/commit/df3e7533a78c3594cdd39b8b18e6f4a3abcb8cfd
>> >> >>>>>
>> >> >>>>> The patch has not been yet submitted upstream. I'll do it once
>> it is
>> >> >>>>> running on our production.
>> >> >>>>>
>> >> >>>>> Anyway, you may easily test it by performing the small change
>> >> directly
>> >> >>>>> into the script.
>> >> >>>>>
>> >> >>>>> Hope that helps.
>> >> >>>>>
>> >> >>>>> Regards,
>> >> >>>>>
>> >> >>>>> Loic Lambiel
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On 10/03/15 19:40, Andrija Panic wrote:
>> >> >>>>>
>> >> >>>>> Hi guys,
>> >> >>>>>>
>> >> >>>>>> Kernel is Elrepo 3.10.67 (CentOS 6.6)
>> >> >>>>>> Libvirt manually compiled 1.2.3 with RBD support.
>> >> >>>>>> Qemu 0.12.1.2 - Inktank patched original RHEL 6.4 rpms with RBD
>> >> >>>>>> support.
>> >> >>>>>>
>> >> >>>>>> And here is the output, with debug mode, from Agent.
>> >> >>>>>>
>> >> >>>>>> At the begining I see agent deleting routes for 169.254.x.x
>> >> network...:
>> >> >>>>>>
>> >> >>>>>> Basically, It seems like the
>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>> >> >>>>>> hviasocket.pl
>> >> >>>>>> is
>> >> >>>>>> failing, not being able to connect to socket...
>> >> >>>>>> And I could not ping VR (169.254.2.99)...
>> >> >>>>>>
>> >> >>>>>> LOG:
>> >> >>>>>>
>> >> >>>>>> 2015-03-10 16:43:46,976{GMT} DEBUG [cloud.agent.Agent]
>> >> >>>>>> (agentRequest-Handler-4:) Processing command:
>> >> >>>>>> com.cloud.agent.api.GetVmStatsCommand
>> >> >>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>> >> >>>>>> (agentRequest-Handler-1:) Request:Seq 788-1319632926: { Cmd ,
>> >> MgmtId:
>> >> >>>>>> 161344838950, via: 788, Ver: v1, Flags: 100111,
>> >> >>>>>> [{"com.cloud.agent.api.StartCommand":{"vm":{"id":
>> >> >>>>>> 2191,"name":"r-2191-VM","type":"DomainRouter","cpus":1,"
>> >> >>>>>> minSpeed":166,"maxSpeed":1000,"minRam":268435456,"maxRam":
>> >> >>>>>> 268435456,"arch":"x86_64","os":"Debian
>> >> >>>>>> GNU/Linux 7(64-bit)","bootArgs":" vpccidr=10.0.0.0/8
>> >> >>>>>> domain=cs2cloud.internal dns1=8.8.8.8 dns2= template=domP
>> >> >>>>>> name=r-2191-VM
>> >> >>>>>> eth0ip=169.254.2.99 eth0mask=255.255.0.0 type=vpcrouter
>> >> >>>>>> disable_rp_filter=true","rebootOnCrash":false,"
>> >> >>>>>> enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":
>> >> >>>>>> false,"vncPassword":"e9e24eb0b1ff8ec2","params":{"
>> >> >>>>>> memoryOvercommitRatio":"1.0","cpuOvercommitRatio":"6.0"},"
>> >> >>>>>> uuid":"485587f7-4a04-4d77-bc36-2367f2adf77e","disks":[{"
>> >> >>>>>> data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
>> >> >>>>>> uuid":"d042e003-5278-461d-8a35-3b1261164f93","
>> >> >>>>>> volumeType":"ROOT","dataStore":{"org.apache.cloudstack.
>> >> >>>>>> storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-
>> >> >>>>>> 88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
>> >> >>>>>> cephmon.domaina.net
>> ","path":"cloudstack","port":6789,"url":"RBD://
>> >> >>>>>> cephmon.domaina.net/cloudstack/?ROLE=Primary&
>> >> >>>>>> STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6
>> >> >>>>>> "}},"name":"ROOT-2191","size":2621440000,"path":"d042e003-
>> >> >>>>>> 5278-461d-8a35-3b1261164f93","volumeId":2857,"vmName":"r-
>> >> >>>>>> 2191-VM","accountId":2,"format":"RAW","id":2857,"
>> >> >>>>>> deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"
>> >> >>>>>> d042e003-5278-461d-8a35-3b1261164f93","type":"ROOT","_
>> >> >>>>>> details":{"managed":"false","storagePort":"6789","storageHost":"
>> >> >>>>>> cephmon.domaina.net
>> >> >>>>>> ","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"
>> >> >>>>>> networkRateMbps":-1,"defaultNic":false,"uuid":"
>> >> >>>>>> d27a7aad-2dcc-45c7-bf30-
>> >> >>>>>> 3302d011da66","ip":"169.254.2.99","netmask":"255.255.0.0","
>> >> >>>>>> gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:63","
>> >> >>>>>> broadcastType":"LinkLocal","type":"Control","
>> >> >>>>>> isSecurityGroupEnabled":false}]},"hostIp":"10.44.253.180","
>> >> >>>>>> executeInSequence":false,"wait":0}},{"com.cloud.agent.
>> >> >>>>>> api.check.CheckSshCommand":{"ip":"169.254.2.99","port":
>> >> >>>>>> 3922,"interval":6,"retries":100,"name":"r-2191-VM","wait":
>> >> >>>>>> 0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
>> >> >>>>>> router.name
>> >> >>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>> >> >>>>>> com.cloud.agent.api.PlugNicCommand":{"nic":{"
>> >> >>>>>> deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid"
>> >> >>>>>> :"fe848dbe-cf4b-473e-8124-62289f86b144","ip":"5.226.146.
>> >> >>>>>> 62","netmask":"255.255.255.128","gateway":"5.226.146.1","
>> >> >>>>>> mac":"06:f0:d6:00:01:49","broadcastType":"Vlan","type":"
>> >> >>>>>> Public","broadcastUri":"vlan://untagged","isolationUri":"
>> >> >>>>>>
>> >> vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"},"
>> >> >>>>>> instanceName":"r-2191-VM","vmType":"DomainRouter","wait":
>> >> >>>>>> 0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"
>> >> >>>>>> ipAddresses":[{"accountId":2,"publicIp":"5.226.146.62","
>> >> >>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>> >> >>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>> >> >>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>> >> >>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>> >> >>>>>> networkName":"breth1-500"}],"accessDetails":{"router.guest.
>> >> >>>>>> ip":"5.226.146.62","zone.network.type":"Advanced","
>> >> >>>>>> router.name
>> >> >>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>> >> >>>>>> com.cloud.agent.api.routing.SetSourceNatCommand":{"
>> >> >>>>>> ipAddress":{"accountId":2,"publicIp":"5.226.146.62","
>> >> >>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>> >> >>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>> >> >>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>> >> >>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>> >> >>>>>> networkName":"breth1-500"},"add":true,"accessDetails":{"
>> >> >>>>>> zone.network.type":"Advanced","
>> >> >>>>>> router.name
>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{}]
>> >> }
>> >> >>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>> >> >>>>>> (agentRequest-Handler-1:) Processing command:
>> >> >>>>>> com.cloud.agent.api.StartCommand
>> >> >>>>>> 2015-03-10 16:43:58,257{GMT} DEBUG [cloud.agent.Agent]
>> >> >>>>>> (agentRequest-Handler-5:) Processing command:
>> >> >>>>>> com.cloud.agent.api.GetStorageStatsCommand
>> >> >>>>>> 2015-03-10 16:44:01,747{GMT} DEBUG
>> [kvm.resource.KVMGuestOsMapper]
>> >> >>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os:
>> Debian
>> >> >>>>>> GNU/Linux 7(64-bit)
>> >> >>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [cloud.agent.Agent]
>> >> >>>>>> (agentRequest-Handler-2:) Processing command:
>> >> >>>>>> com.cloud.agent.api.NetworkUsageCommand
>> >> >>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-2:) Executing:
>> >> >>>>>>
>> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
>> >> >>>>>> vpc_netusage.sh 169.254.2.1 -l 46.232.180.55 -g
>> >> >>>>>> 2015-03-10 16:44:06,439{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-2:) Execution is successful.
>> >> >>>>>> 2015-03-10 16:44:06,441{GMT} DEBUG [cloud.agent.Agent]
>> >> >>>>>> (agentRequest-Handler-2:) Seq 788-1319632928: { Ans: , MgmtId:
>> >> >>>>>> 161344838950, via: 788, Ver: v1, Flags: 10,
>> >> >>>>>>
>> >> [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-1873-VM","
>> >> >>>>>>
>> >> bytesSent":198379,"bytesReceived":7380903,"result":true,"details":"","
>> >> >>>>>> wait":0}}]
>> >> >>>>>> }
>> >> >>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG
>> [kvm.resource.BridgeVifDriver]
>> >> >>>>>> (agentRequest-Handler-1:) nic=[Nic:Control-169.254.2.99-null]
>> >> >>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [utils.script.Script]
>> >> >>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route |
>> grep
>> >> >>>>>> 169.254.0.0/16
>> >> >>>>>> 2015-03-10 16:44:11,060{GMT} DEBUG [utils.script.Script]
>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>> >> >>>>>> 2015-03-10 16:44:11,061{GMT} DEBUG [utils.script.Script]
>> >> >>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route del
>> >> >>>>>> 169.254.0.0/16
>> >> >>>>>> 2015-03-10 16:44:11,064{GMT} DEBUG [utils.script.Script]
>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>> >> >>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG
>> [kvm.resource.KVMGuestOsMapper]
>> >> >>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os:
>> Debian
>> >> >>>>>> GNU/Linux 7(64-bit)
>> >> >>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-1:) starting r-2191-VM: <domain
>> type='kvm'>
>> >> >>>>>> <name>r-2191-VM</name>
>> >> >>>>>> <uuid>485587f7-4a04-4d77-bc36-2367f2adf77e</uuid>
>> >> >>>>>> <description>Debian GNU/Linux 7(64-bit)</description>
>> >> >>>>>> <clock offset='utc'>
>> >> >>>>>> </clock>
>> >> >>>>>> <features>
>> >> >>>>>> <pae/>
>> >> >>>>>> <apic/>
>> >> >>>>>> <acpi/>
>> >> >>>>>> </features>
>> >> >>>>>> <devices>
>> >> >>>>>> <emulator>/usr/libexec/qemu-kvm</emulator>
>> >> >>>>>> <interface type='bridge'>
>> >> >>>>>> <source bridge='cloud0'/>
>> >> >>>>>> <mac address='0e:00:a9:fe:02:63'/>
>> >> >>>>>> <model type='virtio'/>
>> >> >>>>>> </interface>
>> >> >>>>>> <serial type='pty'>
>> >> >>>>>> <target port='0'/>
>> >> >>>>>> </serial>
>> >> >>>>>> <graphics type='vnc' autoport='yes' listen='10.44.253.180'
>> >> >>>>>> passwd='e9e24eb0b1ff8ec2'/>
>> >> >>>>>> <disk device='disk' type='network'>
>> >> >>>>>> <driver name='qemu' type='raw' cache='none' />
>> >> >>>>>> <source protocol='rbd'
>> >> >>>>>> name='cloudstack/d042e003-5278-461d-8a35-3b1261164f93'>
>> >> >>>>>> <host name='cephmon.domain.internal' port='6789'/>
>> >> >>>>>> </source>
>> >> >>>>>> <auth username='cloudstack'>
>> >> >>>>>> <secret type='ceph'
>> uuid='5b93422e-1a66-353d-88a8-2203f79b1dc6'/>
>> >> >>>>>> </auth>
>> >> >>>>>> <target dev='vda' bus='virtio'/>
>> >> >>>>>> </disk>
>> >> >>>>>> <disk device='cdrom' type='file'>
>> >> >>>>>> <driver name='qemu' type='raw' cache='none' />
>> >> >>>>>> <source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
>> >> >>>>>> <target dev='hdc' bus='ide'/>
>> >> >>>>>> </disk>
>> >> >>>>>> <console type='pty'>
>> >> >>>>>> <target port='0'/>
>> >> >>>>>> </console>
>> >> >>>>>> <input type='tablet' bus='usb'/>
>> >> >>>>>> <channel type='unix'>
>> >> >>>>>> <source mode='bind'
>> path='/var/lib/libvirt/qemu/r-2191-VM.agent'/>
>> >> >>>>>> <target type='virtio' name='r-2191-VM.vport'/>
>> >> >>>>>> <address type='virtio-serial'/>
>> >> >>>>>> </channel>
>> >> >>>>>> </devices>
>> >> >>>>>> <memory>262144</memory>
>> >> >>>>>> <devices>
>> >> >>>>>> <memballoon model='none'/>
>> >> >>>>>> </devices>
>> >> >>>>>> <vcpu>1</vcpu>
>> >> >>>>>> <os>
>> >> >>>>>> <type arch='x86_64' machine='pc'>hvm</type>
>> >> >>>>>> <boot dev='cdrom'/>
>> >> >>>>>> <boot dev='hd'/>
>> >> >>>>>> </os>
>> >> >>>>>> <cputune>
>> >> >>>>>> <shares>166</shares>
>> >> >>>>>> </cputune>
>> >> >>>>>> <cpu></cpu><on_reboot>restart</on_reboot>
>> >> >>>>>> <on_poweroff>destroy</on_poweroff>
>> >> >>>>>> <on_crash>destroy</on_crash>
>> >> >>>>>> </domain>
>> >> >>>>>>
>> >> >>>>>> 2015-03-10 16:44:11,467{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-1:) Executing:
>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>> >> >>>>>> hviasocket.pl
>> >> >>>>>> -n
>> >> >>>>>> r-2191-VM -p %vpccidr=
>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>> >> >>>>>>
>> >> >>>>>> 2015-03-10 16:44:11,494{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>> >> >>>>>> 2015-03-10 16:44:11,495{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Trying to connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:44:14,501{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Could not connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>> >> >>>>>> (Thread-10:)
>> >> >>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
>> >> >>>>>> libvirt,
>> >> >>>>>> continuing
>> >> >>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>> >> >>>>>> (Thread-10:)
>> >> >>>>>> Executing:
>> >> >>>>>>
>> >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>> >> >>>>>> -i
>> >> >>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>> >> >>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>> >> >>>>>> 2015-03-10 16:44:17,019{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>> >> >>>>>> (Thread-10:)
>> >> >>>>>> Execution is successful.
>> >> >>>>>> 2015-03-10 16:44:17,953{GMT} DEBUG [cloud.agent.Agent]
>> >> (UgentTask-5:)
>> >> >>>>>> Sending ping: Seq 788-8: { Cmd , MgmtId: -1, via: 788, Ver: v1,
>> >> Flags:
>> >> >>>>>> 11,
>> >> >>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>> >> >>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>> >> >>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>> >> >>>>>> 18.domaina.net
>> >> >>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>> >> >>>>>> hostType":"Routing","hostId":788,"wait":0}}]
>> >> >>>>>> }
>> >> >>>>>> 2015-03-10 16:44:17,964{GMT} DEBUG [cloud.agent.Agent]
>> >> >>>>>> (Agent-Handler-3:)
>> >> >>>>>> Received response: Seq 788-8: { Ans: , MgmtId: 161344838950,
>> via:
>> >> 788,
>> >> >>>>>> Ver: v1, Flags: 100010,
>> >> >>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>> >> >>>>>>
>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>> >> >>>>>> }
>> >> >>>>>> 2015-03-10 16:44:19,502{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Trying to connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:44:22,509{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Could not connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:44:27,510{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Unable to logon to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:44:27,512{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-1:) Executing:
>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>> >> >>>>>> hviasocket.pl
>> >> >>>>>> -n
>> >> >>>>>> r-2191-VM -p %vpccidr=
>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>> >> >>>>>>
>> >> >>>>>> 2015-03-10 16:44:27,540{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>> >> >>>>>> 2015-03-10 16:44:27,541{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Trying to connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:44:30,548{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Could not connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:44:31,378{GMT} DEBUG [cloud.agent.Agent]
>> >> >>>>>> (agentRequest-Handler-3:) Processing command:
>> >> >>>>>> com.cloud.agent.api.GetHostStatsCommand
>> >> >>>>>> 2015-03-10 16:44:31,379{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c idle=$(top -b
>> -n
>> >> >>>>>> 1|grep
>> >> >>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>> >> >>>>>> 2015-03-10 16:44:31,910{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-3:) Execution is successful.
>> >> >>>>>> 2015-03-10 16:44:31,911{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c
>> >> freeMem=$(free|grep
>> >> >>>>>> cache:|awk '{print $4}');echo $freeMem
>> >> >>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-3:) Execution is successful.
>> >> >>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c free|grep
>> Mem:|awk
>> >> >>>>>> '{print $2}'
>> >> >>>>>> 2015-03-10 16:44:31,920{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-3:) Execution is successful.
>> >> >>>>>> 2015-03-10 16:44:35,549{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Trying to connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:44:38,556{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Could not connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Unable to logon to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-1:) Executing:
>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>> >> >>>>>> hviasocket.pl
>> >> >>>>>> -n
>> >> >>>>>> r-2191-VM -p %vpccidr=
>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>> >> >>>>>>
>> >> >>>>>> 2015-03-10 16:44:43,587{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>> >> >>>>>> 2015-03-10 16:44:43,588{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Trying to connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:44:46,594{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Could not connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:44:49,449{GMT} DEBUG [cloud.agent.Agent]
>> >> >>>>>> (agentRequest-Handler-4:) Processing command:
>> >> >>>>>> com.cloud.agent.api.GetVmStatsCommand
>> >> >>>>>> 2015-03-10 16:44:51,595{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Trying to connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:44:54,600{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Could not connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:44:59,601{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Unable to logon to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:44:59,602{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-1:) Executing:
>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>> >> >>>>>> hviasocket.pl
>> >> >>>>>> -n
>> >> >>>>>> r-2191-VM -p %vpccidr=
>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>> >> >>>>>>
>> >> >>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>> >> >>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Trying to connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:45:02,634{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Could not connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:45:07,635{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Trying to connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:45:10,640{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Could not connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:45:12,066{GMT} DEBUG [cloud.agent.Agent]
>> >> >>>>>> (agentRequest-Handler-5:) Processing command:
>> >> >>>>>> com.cloud.agent.api.GetStorageStatsCommand
>> >> >>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Unable to logon to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-1:) Executing:
>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>> >> >>>>>> hviasocket.pl
>> >> >>>>>> -n
>> >> >>>>>> r-2191-VM -p %vpccidr=
>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>> >> >>>>>>
>> >> >>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>> >> >>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Trying to connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:45:17,021{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>> >> >>>>>> (Thread-11:)
>> >> >>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
>> >> >>>>>> libvirt,
>> >> >>>>>> continuing
>> >> >>>>>> 2015-03-10 16:45:17,022{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>> >> >>>>>> (Thread-11:)
>> >> >>>>>> Executing:
>> >> >>>>>>
>> >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>> >> >>>>>> -i
>> >> >>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>> >> >>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>> >> >>>>>> 2015-03-10 16:45:17,036{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>> >> >>>>>> (Thread-11:)
>> >> >>>>>> Execution is successful.
>> >> >>>>>> 2015-03-10 16:45:17,951{GMT} DEBUG [cloud.agent.Agent]
>> >> (UgentTask-5:)
>> >> >>>>>> Sending ping: Seq 788-9: { Cmd , MgmtId: -1, via: 788, Ver: v1,
>> >> Flags:
>> >> >>>>>> 11,
>> >> >>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>> >> >>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>> >> >>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>> >> >>>>>> 18.domaina.net
>> >> >>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >> >>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>> >> >>>>>> hostType":"Routing","hostId":788,"wait":0}}]
>> >> >>>>>> }
>> >> >>>>>> 2015-03-10 16:45:18,002{GMT} DEBUG [cloud.agent.Agent]
>> >> >>>>>> (Agent-Handler-2:)
>> >> >>>>>> Received response: Seq 788-9: { Ans: , MgmtId: 161344838950,
>> via:
>> >> 788,
>> >> >>>>>> Ver: v1, Flags: 100010,
>> >> >>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>> >> >>>>>>
>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>> >> >>>>>> }
>> >> >>>>>> 2015-03-10 16:45:18,672{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Could not connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:45:23,673{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Trying to connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:45:26,680{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Could not connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:45:31,682{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Unable to logon to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:45:31,683{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-1:) Executing:
>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>> >> >>>>>> hviasocket.pl
>> >> >>>>>> -n
>> >> >>>>>> r-2191-VM -p %vpccidr=
>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>> >> >>>>>>
>> >> >>>>>> 2015-03-10 16:45:36,684{GMT} WARN [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (Script-5:) Interrupting script.
>> >> >>>>>> 2015-03-10 16:45:36,686{GMT} WARN [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-1:) Timed out:
>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>> >> >>>>>> hviasocket.pl
>> >> >>>>>> -n
>> >> >>>>>> r-2191-VM -p %vpccidr=
>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>> >> >>>>>> . Output is:
>> >> >>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-1:) passcmd failed:timeout
>> >> >>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Trying to connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [cloud.agent.Agent]
>> >> >>>>>> (agentRequest-Handler-2:) Processing command:
>> >> >>>>>> com.cloud.agent.api.GetHostStatsCommand
>> >> >>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c idle=$(top -b
>> -n
>> >> >>>>>> 1|grep
>> >> >>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>> >> >>>>>> 2015-03-10 16:45:37,967{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-2:) Execution is successful.
>> >> >>>>>> 2015-03-10 16:45:37,968{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c
>> >> freeMem=$(free|grep
>> >> >>>>>> cache:|awk '{print $4}');echo $freeMem
>> >> >>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-2:) Execution is successful.
>> >> >>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c free|grep
>> Mem:|awk
>> >> >>>>>> '{print $2}'
>> >> >>>>>> 2015-03-10 16:45:37,977{GMT} DEBUG [kvm.resource.
>> >> >>>>>> LibvirtComputingResource]
>> >> >>>>>> (agentRequest-Handler-2:) Execution is successful.
>> >> >>>>>> 2015-03-10 16:45:39,692{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Could not connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:45:44,693{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Trying to connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:45:47,700{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Could not connect to 169.254.2.99
>> >> >>>>>> 2015-03-10 16:45:51,807{GMT} DEBUG [cloud.agent.Agent]
>> >> >>>>>> (agentRequest-Handler-3:) Processing command:
>> >> >>>>>> com.cloud.agent.api.GetVmStatsCommand
>> >> >>>>>> 2015-03-10 16:45:52,701{GMT} DEBUG
>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >> >>>>>> (agentRequest-Handler-1:)
>> >> >>>>>> Unable to logon to 169.254.2.99
>> >> >>>>>>
>> >> >>>>>> On 10 March 2015 at 17:50, ilya musayev <
>> >> ilya.mailing.lists@gmail.com>
>> >> >>>>>> wrote:
>> >> >>>>>>
>> >> >>>>>> What kernel are you running? Please include Libvirt and Qemu
>> >> version.
>> >> >>>>>>
>> >> >>>>>>>
>> >> >>>>>>> On 3/10/15 8:21 AM, Andrija Panic wrote:
>> >> >>>>>>>
>> >> >>>>>>> Hi,
>> >> >>>>>>>
>> >> >>>>>>>> we are having issues from time to time with starting VR -
>> while
>> >> >>>>>>>> creating
>> >> >>>>>>>> VPC.
>> >> >>>>>>>>
>> >> >>>>>>>> Here is the few lines from MGMT log:
>> >> >>>>>>>>
>> >> >>>>>>>> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
>> >> >>>>>>>> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start
>> vpc
>> >> [VPC
>> >> >>>>>>>> [621-VPC-delete] due to
>> >> >>>>>>>> com.cloud.exception.AgentUnavailableException: Resource
>> [Host:788]
>> >> >>>>>>>> is
>> >> >>>>>>>> unreachable: Host 788: Unable to start instance due to Unable
>> to
>> >> >>>>>>>> start
>> >> >>>>>>>> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not
>> >> >>>>>>>> retrying
>> >> >>>>>>>> at
>> >> >>>>>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
>> >> >>>>>>>> VirtualMachineManagerImpl.java:1072)
>> >> >>>>>>>>
>> >> >>>>>>>>
>> >> >>>>>>>> The interesting part is on the Agent itself:
>> >> >>>>>>>>
>> >> >>>>>>>>
>> >> >>>>>>>> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.
>> >> >>>>>>>> LibvirtComputingResource]
>> >> >>>>>>>> (agentRequest-Handler-2:) Timed out:
>> >> >>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>> >> >>>>>>>> hviasocket.pl
>> >> >>>>>>>> -n
>> >> >>>>>>>> r-2189-VM -p %vpccidr=
>> >> >>>>>>>> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%
>> >> >>>>>>>> template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=
>> >> >>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>> >> >>>>>>>> . Output is:
>> >> >>>>>>>>
>> >> >>>>>>>>
>> >> >>>>>>>> Interesting stuff is - it's seems somewhat random, 3-4 out
>> of 5
>> >> >>>>>>>> works.
>> >> >>>>>>>>
>> >> >>>>>>>> Any suggestion are really appreciated, since this is live
>> system
>> >> :(
>> >> >>>>>>>> ACS 4.3.0, KVM, CEPH
>> >> >>>>>>>>
>> >> >>>>>>>> Thanks,
>> >> >>>>>>>>
>> >> >>>>>>>>
>> >> >>>>>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>
>> >> >>>
>> >> >>
>> >> >>
>> >> >> --
>> >> >>
>> >> >> Andrija Panić
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> >
>> >> > Andrija Panić
>> >>
>> >
>> >
>> >
>> > --
>> >
>> > Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: [HELP] VR sometimes fail to start
Posted by Andrija Panic <an...@gmail.com>.
Yes I understadn that - more of a problem, qemu-img from CentOS 6.5 doesnt
work snapshots (redhat removed -S flag, etc.), then we need RBD support
(for CEPH), which RH does not support in normal RHEL/CentOS, etc.
Next cloud deployement - dump the the RHEL/CentOS, and go with Ubuntu
14.04, which does support everythgin out of the box. I'm so frustrated with
trying anything new with CentOS/RHEL...simply hurts...
On 14 March 2015 at 17:04, Marcus <sh...@gmail.com> wrote:
> Yeah, I don't remember all of the discussion around the minimum
> version and why, but each libvirt and qemu has such different behavior
> (not all good), that we decided to target a minimum regardless of what
> was supplied by the distros. It's just too hard to support them all
> with their differing behavior. Open stack ran into the same issues.
>
> On Sat, Mar 14, 2015 at 9:02 AM, Andrija Panic <an...@gmail.com>
> wrote:
> > Marcus - QEMU is 0.12.x (CentOS 6.4)
> >
> > but libvirt is compiled 1.2.3
> >
> > On 14 March 2015 at 16:58, Marcus <sh...@gmail.com> wrote:
> >
> >> Oh, this is an unsupported version. should be using libvirt 0.9.4+ and
> >> Qemu 1.0+ per the install docs.
> >>
> >> On Sat, Mar 14, 2015 at 8:56 AM, Andrija Panic <andrija.panic@gmail.com
> >
> >> wrote:
> >> > Tried the proposed solution - but it still don't work - BTW my qemu is
> >> > 0.12.x - from CentOS 6.4...
> >> >
> >> > When I manually start the script with all parameters, it simply
> hangs, no
> >> > output at all, until ctrl + c
> >> >
> >> > BTW this is with now upgraded ACS to 4.3.2...
> >> >
> >> > :(
> >> >
> >> > On 12 March 2015 at 09:24, Andrija Panic <an...@gmail.com>
> >> wrote:
> >> >
> >> >> Hi Marcus, will do - I'm just unconfortable changing live things, but
> >> that
> >> >> is my problem. Will let you know of the output.
> >> >>
> >> >> Thanks
> >> >>
> >> >> On 12 March 2015 at 01:34, ilya musayev <
> ilya.mailing.lists@gmail.com>
> >> >> wrote:
> >> >>
> >> >>> Andrija,
> >> >>>
> >> >>> Have you tried the solution Loic mentioned? Its seems like an easy
> one
> >> to
> >> >>> try..
> >> >>>
> >> >>> Thanks
> >> >>> ilya
> >> >>>
> >> >>>
> >> >>> On 3/11/15 3:54 AM, Andrija Panic wrote:
> >> >>>
> >> >>>> Oh well, I will then update to 4.3.2 anyway, and hope to see the
> >> problem
> >> >>>> go
> >> >>>> away...
> >> >>>>
> >> >>>> THanks guys
> >> >>>>
> >> >>>> On 10 March 2015 at 22:45, Loic Lambiel <lo...@exoscale.ch>
> >> >>>> wrote:
> >> >>>>
> >> >>>> Hi,
> >> >>>>>
> >> >>>>> Maybe the same issue we faced here: https://github.com/exoscale/
> >> >>>>> cloudstack/commit/df3e7533a78c3594cdd39b8b18e6f4a3abcb8cfd
> >> >>>>>
> >> >>>>> The patch has not been yet submitted upstream. I'll do it once it
> is
> >> >>>>> running on our production.
> >> >>>>>
> >> >>>>> Anyway, you may easily test it by performing the small change
> >> directly
> >> >>>>> into the script.
> >> >>>>>
> >> >>>>> Hope that helps.
> >> >>>>>
> >> >>>>> Regards,
> >> >>>>>
> >> >>>>> Loic Lambiel
> >> >>>>>
> >> >>>>>
> >> >>>>> On 10/03/15 19:40, Andrija Panic wrote:
> >> >>>>>
> >> >>>>> Hi guys,
> >> >>>>>>
> >> >>>>>> Kernel is Elrepo 3.10.67 (CentOS 6.6)
> >> >>>>>> Libvirt manually compiled 1.2.3 with RBD support.
> >> >>>>>> Qemu 0.12.1.2 - Inktank patched original RHEL 6.4 rpms with RBD
> >> >>>>>> support.
> >> >>>>>>
> >> >>>>>> And here is the output, with debug mode, from Agent.
> >> >>>>>>
> >> >>>>>> At the begining I see agent deleting routes for 169.254.x.x
> >> network...:
> >> >>>>>>
> >> >>>>>> Basically, It seems like the
> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
> >> >>>>>> hviasocket.pl
> >> >>>>>> is
> >> >>>>>> failing, not being able to connect to socket...
> >> >>>>>> And I could not ping VR (169.254.2.99)...
> >> >>>>>>
> >> >>>>>> LOG:
> >> >>>>>>
> >> >>>>>> 2015-03-10 16:43:46,976{GMT} DEBUG [cloud.agent.Agent]
> >> >>>>>> (agentRequest-Handler-4:) Processing command:
> >> >>>>>> com.cloud.agent.api.GetVmStatsCommand
> >> >>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
> >> >>>>>> (agentRequest-Handler-1:) Request:Seq 788-1319632926: { Cmd ,
> >> MgmtId:
> >> >>>>>> 161344838950, via: 788, Ver: v1, Flags: 100111,
> >> >>>>>> [{"com.cloud.agent.api.StartCommand":{"vm":{"id":
> >> >>>>>> 2191,"name":"r-2191-VM","type":"DomainRouter","cpus":1,"
> >> >>>>>> minSpeed":166,"maxSpeed":1000,"minRam":268435456,"maxRam":
> >> >>>>>> 268435456,"arch":"x86_64","os":"Debian
> >> >>>>>> GNU/Linux 7(64-bit)","bootArgs":" vpccidr=10.0.0.0/8
> >> >>>>>> domain=cs2cloud.internal dns1=8.8.8.8 dns2= template=domP
> >> >>>>>> name=r-2191-VM
> >> >>>>>> eth0ip=169.254.2.99 eth0mask=255.255.0.0 type=vpcrouter
> >> >>>>>> disable_rp_filter=true","rebootOnCrash":false,"
> >> >>>>>> enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":
> >> >>>>>> false,"vncPassword":"e9e24eb0b1ff8ec2","params":{"
> >> >>>>>> memoryOvercommitRatio":"1.0","cpuOvercommitRatio":"6.0"},"
> >> >>>>>> uuid":"485587f7-4a04-4d77-bc36-2367f2adf77e","disks":[{"
> >> >>>>>> data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
> >> >>>>>> uuid":"d042e003-5278-461d-8a35-3b1261164f93","
> >> >>>>>> volumeType":"ROOT","dataStore":{"org.apache.cloudstack.
> >> >>>>>> storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-
> >> >>>>>> 88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
> >> >>>>>> cephmon.domaina.net
> ","path":"cloudstack","port":6789,"url":"RBD://
> >> >>>>>> cephmon.domaina.net/cloudstack/?ROLE=Primary&
> >> >>>>>> STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6
> >> >>>>>> "}},"name":"ROOT-2191","size":2621440000,"path":"d042e003-
> >> >>>>>> 5278-461d-8a35-3b1261164f93","volumeId":2857,"vmName":"r-
> >> >>>>>> 2191-VM","accountId":2,"format":"RAW","id":2857,"
> >> >>>>>> deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"
> >> >>>>>> d042e003-5278-461d-8a35-3b1261164f93","type":"ROOT","_
> >> >>>>>> details":{"managed":"false","storagePort":"6789","storageHost":"
> >> >>>>>> cephmon.domaina.net
> >> >>>>>> ","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"
> >> >>>>>> networkRateMbps":-1,"defaultNic":false,"uuid":"
> >> >>>>>> d27a7aad-2dcc-45c7-bf30-
> >> >>>>>> 3302d011da66","ip":"169.254.2.99","netmask":"255.255.0.0","
> >> >>>>>> gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:63","
> >> >>>>>> broadcastType":"LinkLocal","type":"Control","
> >> >>>>>> isSecurityGroupEnabled":false}]},"hostIp":"10.44.253.180","
> >> >>>>>> executeInSequence":false,"wait":0}},{"com.cloud.agent.
> >> >>>>>> api.check.CheckSshCommand":{"ip":"169.254.2.99","port":
> >> >>>>>> 3922,"interval":6,"retries":100,"name":"r-2191-VM","wait":
> >> >>>>>> 0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
> >> >>>>>> router.name
> >> >>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
> >> >>>>>> com.cloud.agent.api.PlugNicCommand":{"nic":{"
> >> >>>>>> deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid"
> >> >>>>>> :"fe848dbe-cf4b-473e-8124-62289f86b144","ip":"5.226.146.
> >> >>>>>> 62","netmask":"255.255.255.128","gateway":"5.226.146.1","
> >> >>>>>> mac":"06:f0:d6:00:01:49","broadcastType":"Vlan","type":"
> >> >>>>>> Public","broadcastUri":"vlan://untagged","isolationUri":"
> >> >>>>>>
> >> vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"},"
> >> >>>>>> instanceName":"r-2191-VM","vmType":"DomainRouter","wait":
> >> >>>>>> 0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"
> >> >>>>>> ipAddresses":[{"accountId":2,"publicIp":"5.226.146.62","
> >> >>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
> >> >>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
> >> >>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
> >> >>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
> >> >>>>>> networkName":"breth1-500"}],"accessDetails":{"router.guest.
> >> >>>>>> ip":"5.226.146.62","zone.network.type":"Advanced","
> >> >>>>>> router.name
> >> >>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
> >> >>>>>> com.cloud.agent.api.routing.SetSourceNatCommand":{"
> >> >>>>>> ipAddress":{"accountId":2,"publicIp":"5.226.146.62","
> >> >>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
> >> >>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
> >> >>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
> >> >>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
> >> >>>>>> networkName":"breth1-500"},"add":true,"accessDetails":{"
> >> >>>>>> zone.network.type":"Advanced","
> >> >>>>>> router.name
> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{}]
> >> }
> >> >>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
> >> >>>>>> (agentRequest-Handler-1:) Processing command:
> >> >>>>>> com.cloud.agent.api.StartCommand
> >> >>>>>> 2015-03-10 16:43:58,257{GMT} DEBUG [cloud.agent.Agent]
> >> >>>>>> (agentRequest-Handler-5:) Processing command:
> >> >>>>>> com.cloud.agent.api.GetStorageStatsCommand
> >> >>>>>> 2015-03-10 16:44:01,747{GMT} DEBUG
> [kvm.resource.KVMGuestOsMapper]
> >> >>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os:
> Debian
> >> >>>>>> GNU/Linux 7(64-bit)
> >> >>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [cloud.agent.Agent]
> >> >>>>>> (agentRequest-Handler-2:) Processing command:
> >> >>>>>> com.cloud.agent.api.NetworkUsageCommand
> >> >>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-2:) Executing:
> >> >>>>>> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
> >> >>>>>> vpc_netusage.sh 169.254.2.1 -l 46.232.180.55 -g
> >> >>>>>> 2015-03-10 16:44:06,439{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-2:) Execution is successful.
> >> >>>>>> 2015-03-10 16:44:06,441{GMT} DEBUG [cloud.agent.Agent]
> >> >>>>>> (agentRequest-Handler-2:) Seq 788-1319632928: { Ans: , MgmtId:
> >> >>>>>> 161344838950, via: 788, Ver: v1, Flags: 10,
> >> >>>>>>
> >> [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-1873-VM","
> >> >>>>>>
> >> bytesSent":198379,"bytesReceived":7380903,"result":true,"details":"","
> >> >>>>>> wait":0}}]
> >> >>>>>> }
> >> >>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [kvm.resource.BridgeVifDriver]
> >> >>>>>> (agentRequest-Handler-1:) nic=[Nic:Control-169.254.2.99-null]
> >> >>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [utils.script.Script]
> >> >>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route | grep
> >> >>>>>> 169.254.0.0/16
> >> >>>>>> 2015-03-10 16:44:11,060{GMT} DEBUG [utils.script.Script]
> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
> >> >>>>>> 2015-03-10 16:44:11,061{GMT} DEBUG [utils.script.Script]
> >> >>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route del
> >> >>>>>> 169.254.0.0/16
> >> >>>>>> 2015-03-10 16:44:11,064{GMT} DEBUG [utils.script.Script]
> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
> >> >>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG
> [kvm.resource.KVMGuestOsMapper]
> >> >>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os:
> Debian
> >> >>>>>> GNU/Linux 7(64-bit)
> >> >>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-1:) starting r-2191-VM: <domain type='kvm'>
> >> >>>>>> <name>r-2191-VM</name>
> >> >>>>>> <uuid>485587f7-4a04-4d77-bc36-2367f2adf77e</uuid>
> >> >>>>>> <description>Debian GNU/Linux 7(64-bit)</description>
> >> >>>>>> <clock offset='utc'>
> >> >>>>>> </clock>
> >> >>>>>> <features>
> >> >>>>>> <pae/>
> >> >>>>>> <apic/>
> >> >>>>>> <acpi/>
> >> >>>>>> </features>
> >> >>>>>> <devices>
> >> >>>>>> <emulator>/usr/libexec/qemu-kvm</emulator>
> >> >>>>>> <interface type='bridge'>
> >> >>>>>> <source bridge='cloud0'/>
> >> >>>>>> <mac address='0e:00:a9:fe:02:63'/>
> >> >>>>>> <model type='virtio'/>
> >> >>>>>> </interface>
> >> >>>>>> <serial type='pty'>
> >> >>>>>> <target port='0'/>
> >> >>>>>> </serial>
> >> >>>>>> <graphics type='vnc' autoport='yes' listen='10.44.253.180'
> >> >>>>>> passwd='e9e24eb0b1ff8ec2'/>
> >> >>>>>> <disk device='disk' type='network'>
> >> >>>>>> <driver name='qemu' type='raw' cache='none' />
> >> >>>>>> <source protocol='rbd'
> >> >>>>>> name='cloudstack/d042e003-5278-461d-8a35-3b1261164f93'>
> >> >>>>>> <host name='cephmon.domain.internal' port='6789'/>
> >> >>>>>> </source>
> >> >>>>>> <auth username='cloudstack'>
> >> >>>>>> <secret type='ceph' uuid='5b93422e-1a66-353d-88a8-2203f79b1dc6'/>
> >> >>>>>> </auth>
> >> >>>>>> <target dev='vda' bus='virtio'/>
> >> >>>>>> </disk>
> >> >>>>>> <disk device='cdrom' type='file'>
> >> >>>>>> <driver name='qemu' type='raw' cache='none' />
> >> >>>>>> <source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
> >> >>>>>> <target dev='hdc' bus='ide'/>
> >> >>>>>> </disk>
> >> >>>>>> <console type='pty'>
> >> >>>>>> <target port='0'/>
> >> >>>>>> </console>
> >> >>>>>> <input type='tablet' bus='usb'/>
> >> >>>>>> <channel type='unix'>
> >> >>>>>> <source mode='bind'
> path='/var/lib/libvirt/qemu/r-2191-VM.agent'/>
> >> >>>>>> <target type='virtio' name='r-2191-VM.vport'/>
> >> >>>>>> <address type='virtio-serial'/>
> >> >>>>>> </channel>
> >> >>>>>> </devices>
> >> >>>>>> <memory>262144</memory>
> >> >>>>>> <devices>
> >> >>>>>> <memballoon model='none'/>
> >> >>>>>> </devices>
> >> >>>>>> <vcpu>1</vcpu>
> >> >>>>>> <os>
> >> >>>>>> <type arch='x86_64' machine='pc'>hvm</type>
> >> >>>>>> <boot dev='cdrom'/>
> >> >>>>>> <boot dev='hd'/>
> >> >>>>>> </os>
> >> >>>>>> <cputune>
> >> >>>>>> <shares>166</shares>
> >> >>>>>> </cputune>
> >> >>>>>> <cpu></cpu><on_reboot>restart</on_reboot>
> >> >>>>>> <on_poweroff>destroy</on_poweroff>
> >> >>>>>> <on_crash>destroy</on_crash>
> >> >>>>>> </domain>
> >> >>>>>>
> >> >>>>>> 2015-03-10 16:44:11,467{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-1:) Executing:
> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
> >> >>>>>> hviasocket.pl
> >> >>>>>> -n
> >> >>>>>> r-2191-VM -p %vpccidr=
> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
> >> >>>>>>
> >> >>>>>> 2015-03-10 16:44:11,494{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
> >> >>>>>> 2015-03-10 16:44:11,495{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Trying to connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:44:14,501{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Could not connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
> >> >>>>>> (Thread-10:)
> >> >>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
> >> >>>>>> libvirt,
> >> >>>>>> continuing
> >> >>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
> >> >>>>>> (Thread-10:)
> >> >>>>>> Executing:
> >> >>>>>>
> >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
> >> >>>>>> -i
> >> >>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
> >> >>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
> >> >>>>>> 2015-03-10 16:44:17,019{GMT} DEBUG [kvm.resource.KVMHAMonitor]
> >> >>>>>> (Thread-10:)
> >> >>>>>> Execution is successful.
> >> >>>>>> 2015-03-10 16:44:17,953{GMT} DEBUG [cloud.agent.Agent]
> >> (UgentTask-5:)
> >> >>>>>> Sending ping: Seq 788-8: { Cmd , MgmtId: -1, via: 788, Ver: v1,
> >> Flags:
> >> >>>>>> 11,
> >> >>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
> >> >>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
> >> >>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
> >> >>>>>> 18.domaina.net
> >> >>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
> >> >>>>>> hostType":"Routing","hostId":788,"wait":0}}]
> >> >>>>>> }
> >> >>>>>> 2015-03-10 16:44:17,964{GMT} DEBUG [cloud.agent.Agent]
> >> >>>>>> (Agent-Handler-3:)
> >> >>>>>> Received response: Seq 788-8: { Ans: , MgmtId: 161344838950,
> via:
> >> 788,
> >> >>>>>> Ver: v1, Flags: 100010,
> >> >>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
> >> >>>>>>
> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
> >> >>>>>> }
> >> >>>>>> 2015-03-10 16:44:19,502{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Trying to connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:44:22,509{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Could not connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:44:27,510{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Unable to logon to 169.254.2.99
> >> >>>>>> 2015-03-10 16:44:27,512{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-1:) Executing:
> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
> >> >>>>>> hviasocket.pl
> >> >>>>>> -n
> >> >>>>>> r-2191-VM -p %vpccidr=
> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
> >> >>>>>>
> >> >>>>>> 2015-03-10 16:44:27,540{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
> >> >>>>>> 2015-03-10 16:44:27,541{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Trying to connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:44:30,548{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Could not connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:44:31,378{GMT} DEBUG [cloud.agent.Agent]
> >> >>>>>> (agentRequest-Handler-3:) Processing command:
> >> >>>>>> com.cloud.agent.api.GetHostStatsCommand
> >> >>>>>> 2015-03-10 16:44:31,379{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c idle=$(top -b
> -n
> >> >>>>>> 1|grep
> >> >>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
> >> >>>>>> 2015-03-10 16:44:31,910{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-3:) Execution is successful.
> >> >>>>>> 2015-03-10 16:44:31,911{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c
> >> freeMem=$(free|grep
> >> >>>>>> cache:|awk '{print $4}');echo $freeMem
> >> >>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-3:) Execution is successful.
> >> >>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c free|grep
> Mem:|awk
> >> >>>>>> '{print $2}'
> >> >>>>>> 2015-03-10 16:44:31,920{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-3:) Execution is successful.
> >> >>>>>> 2015-03-10 16:44:35,549{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Trying to connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:44:38,556{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Could not connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Unable to logon to 169.254.2.99
> >> >>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-1:) Executing:
> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
> >> >>>>>> hviasocket.pl
> >> >>>>>> -n
> >> >>>>>> r-2191-VM -p %vpccidr=
> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
> >> >>>>>>
> >> >>>>>> 2015-03-10 16:44:43,587{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
> >> >>>>>> 2015-03-10 16:44:43,588{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Trying to connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:44:46,594{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Could not connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:44:49,449{GMT} DEBUG [cloud.agent.Agent]
> >> >>>>>> (agentRequest-Handler-4:) Processing command:
> >> >>>>>> com.cloud.agent.api.GetVmStatsCommand
> >> >>>>>> 2015-03-10 16:44:51,595{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Trying to connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:44:54,600{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Could not connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:44:59,601{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Unable to logon to 169.254.2.99
> >> >>>>>> 2015-03-10 16:44:59,602{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-1:) Executing:
> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
> >> >>>>>> hviasocket.pl
> >> >>>>>> -n
> >> >>>>>> r-2191-VM -p %vpccidr=
> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
> >> >>>>>>
> >> >>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
> >> >>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Trying to connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:45:02,634{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Could not connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:45:07,635{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Trying to connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:45:10,640{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Could not connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:45:12,066{GMT} DEBUG [cloud.agent.Agent]
> >> >>>>>> (agentRequest-Handler-5:) Processing command:
> >> >>>>>> com.cloud.agent.api.GetStorageStatsCommand
> >> >>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Unable to logon to 169.254.2.99
> >> >>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-1:) Executing:
> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
> >> >>>>>> hviasocket.pl
> >> >>>>>> -n
> >> >>>>>> r-2191-VM -p %vpccidr=
> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
> >> >>>>>>
> >> >>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
> >> >>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Trying to connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:45:17,021{GMT} DEBUG [kvm.resource.KVMHAMonitor]
> >> >>>>>> (Thread-11:)
> >> >>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
> >> >>>>>> libvirt,
> >> >>>>>> continuing
> >> >>>>>> 2015-03-10 16:45:17,022{GMT} DEBUG [kvm.resource.KVMHAMonitor]
> >> >>>>>> (Thread-11:)
> >> >>>>>> Executing:
> >> >>>>>>
> >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
> >> >>>>>> -i
> >> >>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
> >> >>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
> >> >>>>>> 2015-03-10 16:45:17,036{GMT} DEBUG [kvm.resource.KVMHAMonitor]
> >> >>>>>> (Thread-11:)
> >> >>>>>> Execution is successful.
> >> >>>>>> 2015-03-10 16:45:17,951{GMT} DEBUG [cloud.agent.Agent]
> >> (UgentTask-5:)
> >> >>>>>> Sending ping: Seq 788-9: { Cmd , MgmtId: -1, via: 788, Ver: v1,
> >> Flags:
> >> >>>>>> 11,
> >> >>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
> >> >>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
> >> >>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
> >> >>>>>> 18.domaina.net
> >> >>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >> >>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
> >> >>>>>> hostType":"Routing","hostId":788,"wait":0}}]
> >> >>>>>> }
> >> >>>>>> 2015-03-10 16:45:18,002{GMT} DEBUG [cloud.agent.Agent]
> >> >>>>>> (Agent-Handler-2:)
> >> >>>>>> Received response: Seq 788-9: { Ans: , MgmtId: 161344838950,
> via:
> >> 788,
> >> >>>>>> Ver: v1, Flags: 100010,
> >> >>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
> >> >>>>>>
> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
> >> >>>>>> }
> >> >>>>>> 2015-03-10 16:45:18,672{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Could not connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:45:23,673{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Trying to connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:45:26,680{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Could not connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:45:31,682{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Unable to logon to 169.254.2.99
> >> >>>>>> 2015-03-10 16:45:31,683{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-1:) Executing:
> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
> >> >>>>>> hviasocket.pl
> >> >>>>>> -n
> >> >>>>>> r-2191-VM -p %vpccidr=
> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
> >> >>>>>>
> >> >>>>>> 2015-03-10 16:45:36,684{GMT} WARN [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (Script-5:) Interrupting script.
> >> >>>>>> 2015-03-10 16:45:36,686{GMT} WARN [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-1:) Timed out:
> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
> >> >>>>>> hviasocket.pl
> >> >>>>>> -n
> >> >>>>>> r-2191-VM -p %vpccidr=
> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
> >> >>>>>> . Output is:
> >> >>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-1:) passcmd failed:timeout
> >> >>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Trying to connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [cloud.agent.Agent]
> >> >>>>>> (agentRequest-Handler-2:) Processing command:
> >> >>>>>> com.cloud.agent.api.GetHostStatsCommand
> >> >>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c idle=$(top -b
> -n
> >> >>>>>> 1|grep
> >> >>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
> >> >>>>>> 2015-03-10 16:45:37,967{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-2:) Execution is successful.
> >> >>>>>> 2015-03-10 16:45:37,968{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c
> >> freeMem=$(free|grep
> >> >>>>>> cache:|awk '{print $4}');echo $freeMem
> >> >>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-2:) Execution is successful.
> >> >>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c free|grep
> Mem:|awk
> >> >>>>>> '{print $2}'
> >> >>>>>> 2015-03-10 16:45:37,977{GMT} DEBUG [kvm.resource.
> >> >>>>>> LibvirtComputingResource]
> >> >>>>>> (agentRequest-Handler-2:) Execution is successful.
> >> >>>>>> 2015-03-10 16:45:39,692{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Could not connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:45:44,693{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Trying to connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:45:47,700{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Could not connect to 169.254.2.99
> >> >>>>>> 2015-03-10 16:45:51,807{GMT} DEBUG [cloud.agent.Agent]
> >> >>>>>> (agentRequest-Handler-3:) Processing command:
> >> >>>>>> com.cloud.agent.api.GetVmStatsCommand
> >> >>>>>> 2015-03-10 16:45:52,701{GMT} DEBUG
> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >> >>>>>> (agentRequest-Handler-1:)
> >> >>>>>> Unable to logon to 169.254.2.99
> >> >>>>>>
> >> >>>>>> On 10 March 2015 at 17:50, ilya musayev <
> >> ilya.mailing.lists@gmail.com>
> >> >>>>>> wrote:
> >> >>>>>>
> >> >>>>>> What kernel are you running? Please include Libvirt and Qemu
> >> version.
> >> >>>>>>
> >> >>>>>>>
> >> >>>>>>> On 3/10/15 8:21 AM, Andrija Panic wrote:
> >> >>>>>>>
> >> >>>>>>> Hi,
> >> >>>>>>>
> >> >>>>>>>> we are having issues from time to time with starting VR - while
> >> >>>>>>>> creating
> >> >>>>>>>> VPC.
> >> >>>>>>>>
> >> >>>>>>>> Here is the few lines from MGMT log:
> >> >>>>>>>>
> >> >>>>>>>> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
> >> >>>>>>>> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start vpc
> >> [VPC
> >> >>>>>>>> [621-VPC-delete] due to
> >> >>>>>>>> com.cloud.exception.AgentUnavailableException: Resource
> [Host:788]
> >> >>>>>>>> is
> >> >>>>>>>> unreachable: Host 788: Unable to start instance due to Unable
> to
> >> >>>>>>>> start
> >> >>>>>>>> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not
> >> >>>>>>>> retrying
> >> >>>>>>>> at
> >> >>>>>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
> >> >>>>>>>> VirtualMachineManagerImpl.java:1072)
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> The interesting part is on the Agent itself:
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.
> >> >>>>>>>> LibvirtComputingResource]
> >> >>>>>>>> (agentRequest-Handler-2:) Timed out:
> >> >>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
> >> >>>>>>>> hviasocket.pl
> >> >>>>>>>> -n
> >> >>>>>>>> r-2189-VM -p %vpccidr=
> >> >>>>>>>> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%
> >> >>>>>>>> template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=
> >> >>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
> >> >>>>>>>> . Output is:
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> Interesting stuff is - it's seems somewhat random, 3-4 out of
> 5
> >> >>>>>>>> works.
> >> >>>>>>>>
> >> >>>>>>>> Any suggestion are really appreciated, since this is live
> system
> >> :(
> >> >>>>>>>> ACS 4.3.0, KVM, CEPH
> >> >>>>>>>>
> >> >>>>>>>> Thanks,
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >> >>
> >> >> --
> >> >>
> >> >> Andrija Panić
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Andrija Panić
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
>
--
Andrija Panić
Re: [HELP] VR sometimes fail to start
Posted by Marcus <sh...@gmail.com>.
Yeah, I don't remember all of the discussion around the minimum
version and why, but each libvirt and qemu has such different behavior
(not all good), that we decided to target a minimum regardless of what
was supplied by the distros. It's just too hard to support them all
with their differing behavior. Open stack ran into the same issues.
On Sat, Mar 14, 2015 at 9:02 AM, Andrija Panic <an...@gmail.com> wrote:
> Marcus - QEMU is 0.12.x (CentOS 6.4)
>
> but libvirt is compiled 1.2.3
>
> On 14 March 2015 at 16:58, Marcus <sh...@gmail.com> wrote:
>
>> Oh, this is an unsupported version. should be using libvirt 0.9.4+ and
>> Qemu 1.0+ per the install docs.
>>
>> On Sat, Mar 14, 2015 at 8:56 AM, Andrija Panic <an...@gmail.com>
>> wrote:
>> > Tried the proposed solution - but it still don't work - BTW my qemu is
>> > 0.12.x - from CentOS 6.4...
>> >
>> > When I manually start the script with all parameters, it simply hangs, no
>> > output at all, until ctrl + c
>> >
>> > BTW this is with now upgraded ACS to 4.3.2...
>> >
>> > :(
>> >
>> > On 12 March 2015 at 09:24, Andrija Panic <an...@gmail.com>
>> wrote:
>> >
>> >> Hi Marcus, will do - I'm just unconfortable changing live things, but
>> that
>> >> is my problem. Will let you know of the output.
>> >>
>> >> Thanks
>> >>
>> >> On 12 March 2015 at 01:34, ilya musayev <il...@gmail.com>
>> >> wrote:
>> >>
>> >>> Andrija,
>> >>>
>> >>> Have you tried the solution Loic mentioned? Its seems like an easy one
>> to
>> >>> try..
>> >>>
>> >>> Thanks
>> >>> ilya
>> >>>
>> >>>
>> >>> On 3/11/15 3:54 AM, Andrija Panic wrote:
>> >>>
>> >>>> Oh well, I will then update to 4.3.2 anyway, and hope to see the
>> problem
>> >>>> go
>> >>>> away...
>> >>>>
>> >>>> THanks guys
>> >>>>
>> >>>> On 10 March 2015 at 22:45, Loic Lambiel <lo...@exoscale.ch>
>> >>>> wrote:
>> >>>>
>> >>>> Hi,
>> >>>>>
>> >>>>> Maybe the same issue we faced here: https://github.com/exoscale/
>> >>>>> cloudstack/commit/df3e7533a78c3594cdd39b8b18e6f4a3abcb8cfd
>> >>>>>
>> >>>>> The patch has not been yet submitted upstream. I'll do it once it is
>> >>>>> running on our production.
>> >>>>>
>> >>>>> Anyway, you may easily test it by performing the small change
>> directly
>> >>>>> into the script.
>> >>>>>
>> >>>>> Hope that helps.
>> >>>>>
>> >>>>> Regards,
>> >>>>>
>> >>>>> Loic Lambiel
>> >>>>>
>> >>>>>
>> >>>>> On 10/03/15 19:40, Andrija Panic wrote:
>> >>>>>
>> >>>>> Hi guys,
>> >>>>>>
>> >>>>>> Kernel is Elrepo 3.10.67 (CentOS 6.6)
>> >>>>>> Libvirt manually compiled 1.2.3 with RBD support.
>> >>>>>> Qemu 0.12.1.2 - Inktank patched original RHEL 6.4 rpms with RBD
>> >>>>>> support.
>> >>>>>>
>> >>>>>> And here is the output, with debug mode, from Agent.
>> >>>>>>
>> >>>>>> At the begining I see agent deleting routes for 169.254.x.x
>> network...:
>> >>>>>>
>> >>>>>> Basically, It seems like the
>> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>> >>>>>> hviasocket.pl
>> >>>>>> is
>> >>>>>> failing, not being able to connect to socket...
>> >>>>>> And I could not ping VR (169.254.2.99)...
>> >>>>>>
>> >>>>>> LOG:
>> >>>>>>
>> >>>>>> 2015-03-10 16:43:46,976{GMT} DEBUG [cloud.agent.Agent]
>> >>>>>> (agentRequest-Handler-4:) Processing command:
>> >>>>>> com.cloud.agent.api.GetVmStatsCommand
>> >>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>> >>>>>> (agentRequest-Handler-1:) Request:Seq 788-1319632926: { Cmd ,
>> MgmtId:
>> >>>>>> 161344838950, via: 788, Ver: v1, Flags: 100111,
>> >>>>>> [{"com.cloud.agent.api.StartCommand":{"vm":{"id":
>> >>>>>> 2191,"name":"r-2191-VM","type":"DomainRouter","cpus":1,"
>> >>>>>> minSpeed":166,"maxSpeed":1000,"minRam":268435456,"maxRam":
>> >>>>>> 268435456,"arch":"x86_64","os":"Debian
>> >>>>>> GNU/Linux 7(64-bit)","bootArgs":" vpccidr=10.0.0.0/8
>> >>>>>> domain=cs2cloud.internal dns1=8.8.8.8 dns2= template=domP
>> >>>>>> name=r-2191-VM
>> >>>>>> eth0ip=169.254.2.99 eth0mask=255.255.0.0 type=vpcrouter
>> >>>>>> disable_rp_filter=true","rebootOnCrash":false,"
>> >>>>>> enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":
>> >>>>>> false,"vncPassword":"e9e24eb0b1ff8ec2","params":{"
>> >>>>>> memoryOvercommitRatio":"1.0","cpuOvercommitRatio":"6.0"},"
>> >>>>>> uuid":"485587f7-4a04-4d77-bc36-2367f2adf77e","disks":[{"
>> >>>>>> data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
>> >>>>>> uuid":"d042e003-5278-461d-8a35-3b1261164f93","
>> >>>>>> volumeType":"ROOT","dataStore":{"org.apache.cloudstack.
>> >>>>>> storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-
>> >>>>>> 88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
>> >>>>>> cephmon.domaina.net","path":"cloudstack","port":6789,"url":"RBD://
>> >>>>>> cephmon.domaina.net/cloudstack/?ROLE=Primary&
>> >>>>>> STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6
>> >>>>>> "}},"name":"ROOT-2191","size":2621440000,"path":"d042e003-
>> >>>>>> 5278-461d-8a35-3b1261164f93","volumeId":2857,"vmName":"r-
>> >>>>>> 2191-VM","accountId":2,"format":"RAW","id":2857,"
>> >>>>>> deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"
>> >>>>>> d042e003-5278-461d-8a35-3b1261164f93","type":"ROOT","_
>> >>>>>> details":{"managed":"false","storagePort":"6789","storageHost":"
>> >>>>>> cephmon.domaina.net
>> >>>>>> ","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"
>> >>>>>> networkRateMbps":-1,"defaultNic":false,"uuid":"
>> >>>>>> d27a7aad-2dcc-45c7-bf30-
>> >>>>>> 3302d011da66","ip":"169.254.2.99","netmask":"255.255.0.0","
>> >>>>>> gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:63","
>> >>>>>> broadcastType":"LinkLocal","type":"Control","
>> >>>>>> isSecurityGroupEnabled":false}]},"hostIp":"10.44.253.180","
>> >>>>>> executeInSequence":false,"wait":0}},{"com.cloud.agent.
>> >>>>>> api.check.CheckSshCommand":{"ip":"169.254.2.99","port":
>> >>>>>> 3922,"interval":6,"retries":100,"name":"r-2191-VM","wait":
>> >>>>>> 0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
>> >>>>>> router.name
>> >>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>> >>>>>> com.cloud.agent.api.PlugNicCommand":{"nic":{"
>> >>>>>> deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid"
>> >>>>>> :"fe848dbe-cf4b-473e-8124-62289f86b144","ip":"5.226.146.
>> >>>>>> 62","netmask":"255.255.255.128","gateway":"5.226.146.1","
>> >>>>>> mac":"06:f0:d6:00:01:49","broadcastType":"Vlan","type":"
>> >>>>>> Public","broadcastUri":"vlan://untagged","isolationUri":"
>> >>>>>>
>> vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"},"
>> >>>>>> instanceName":"r-2191-VM","vmType":"DomainRouter","wait":
>> >>>>>> 0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"
>> >>>>>> ipAddresses":[{"accountId":2,"publicIp":"5.226.146.62","
>> >>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>> >>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>> >>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>> >>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>> >>>>>> networkName":"breth1-500"}],"accessDetails":{"router.guest.
>> >>>>>> ip":"5.226.146.62","zone.network.type":"Advanced","
>> >>>>>> router.name
>> >>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>> >>>>>> com.cloud.agent.api.routing.SetSourceNatCommand":{"
>> >>>>>> ipAddress":{"accountId":2,"publicIp":"5.226.146.62","
>> >>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>> >>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>> >>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>> >>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>> >>>>>> networkName":"breth1-500"},"add":true,"accessDetails":{"
>> >>>>>> zone.network.type":"Advanced","
>> >>>>>> router.name":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{}]
>> }
>> >>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>> >>>>>> (agentRequest-Handler-1:) Processing command:
>> >>>>>> com.cloud.agent.api.StartCommand
>> >>>>>> 2015-03-10 16:43:58,257{GMT} DEBUG [cloud.agent.Agent]
>> >>>>>> (agentRequest-Handler-5:) Processing command:
>> >>>>>> com.cloud.agent.api.GetStorageStatsCommand
>> >>>>>> 2015-03-10 16:44:01,747{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
>> >>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
>> >>>>>> GNU/Linux 7(64-bit)
>> >>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [cloud.agent.Agent]
>> >>>>>> (agentRequest-Handler-2:) Processing command:
>> >>>>>> com.cloud.agent.api.NetworkUsageCommand
>> >>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-2:) Executing:
>> >>>>>> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
>> >>>>>> vpc_netusage.sh 169.254.2.1 -l 46.232.180.55 -g
>> >>>>>> 2015-03-10 16:44:06,439{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-2:) Execution is successful.
>> >>>>>> 2015-03-10 16:44:06,441{GMT} DEBUG [cloud.agent.Agent]
>> >>>>>> (agentRequest-Handler-2:) Seq 788-1319632928: { Ans: , MgmtId:
>> >>>>>> 161344838950, via: 788, Ver: v1, Flags: 10,
>> >>>>>>
>> [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-1873-VM","
>> >>>>>>
>> bytesSent":198379,"bytesReceived":7380903,"result":true,"details":"","
>> >>>>>> wait":0}}]
>> >>>>>> }
>> >>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [kvm.resource.BridgeVifDriver]
>> >>>>>> (agentRequest-Handler-1:) nic=[Nic:Control-169.254.2.99-null]
>> >>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [utils.script.Script]
>> >>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route | grep
>> >>>>>> 169.254.0.0/16
>> >>>>>> 2015-03-10 16:44:11,060{GMT} DEBUG [utils.script.Script]
>> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>> >>>>>> 2015-03-10 16:44:11,061{GMT} DEBUG [utils.script.Script]
>> >>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route del
>> >>>>>> 169.254.0.0/16
>> >>>>>> 2015-03-10 16:44:11,064{GMT} DEBUG [utils.script.Script]
>> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>> >>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
>> >>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
>> >>>>>> GNU/Linux 7(64-bit)
>> >>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-1:) starting r-2191-VM: <domain type='kvm'>
>> >>>>>> <name>r-2191-VM</name>
>> >>>>>> <uuid>485587f7-4a04-4d77-bc36-2367f2adf77e</uuid>
>> >>>>>> <description>Debian GNU/Linux 7(64-bit)</description>
>> >>>>>> <clock offset='utc'>
>> >>>>>> </clock>
>> >>>>>> <features>
>> >>>>>> <pae/>
>> >>>>>> <apic/>
>> >>>>>> <acpi/>
>> >>>>>> </features>
>> >>>>>> <devices>
>> >>>>>> <emulator>/usr/libexec/qemu-kvm</emulator>
>> >>>>>> <interface type='bridge'>
>> >>>>>> <source bridge='cloud0'/>
>> >>>>>> <mac address='0e:00:a9:fe:02:63'/>
>> >>>>>> <model type='virtio'/>
>> >>>>>> </interface>
>> >>>>>> <serial type='pty'>
>> >>>>>> <target port='0'/>
>> >>>>>> </serial>
>> >>>>>> <graphics type='vnc' autoport='yes' listen='10.44.253.180'
>> >>>>>> passwd='e9e24eb0b1ff8ec2'/>
>> >>>>>> <disk device='disk' type='network'>
>> >>>>>> <driver name='qemu' type='raw' cache='none' />
>> >>>>>> <source protocol='rbd'
>> >>>>>> name='cloudstack/d042e003-5278-461d-8a35-3b1261164f93'>
>> >>>>>> <host name='cephmon.domain.internal' port='6789'/>
>> >>>>>> </source>
>> >>>>>> <auth username='cloudstack'>
>> >>>>>> <secret type='ceph' uuid='5b93422e-1a66-353d-88a8-2203f79b1dc6'/>
>> >>>>>> </auth>
>> >>>>>> <target dev='vda' bus='virtio'/>
>> >>>>>> </disk>
>> >>>>>> <disk device='cdrom' type='file'>
>> >>>>>> <driver name='qemu' type='raw' cache='none' />
>> >>>>>> <source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
>> >>>>>> <target dev='hdc' bus='ide'/>
>> >>>>>> </disk>
>> >>>>>> <console type='pty'>
>> >>>>>> <target port='0'/>
>> >>>>>> </console>
>> >>>>>> <input type='tablet' bus='usb'/>
>> >>>>>> <channel type='unix'>
>> >>>>>> <source mode='bind' path='/var/lib/libvirt/qemu/r-2191-VM.agent'/>
>> >>>>>> <target type='virtio' name='r-2191-VM.vport'/>
>> >>>>>> <address type='virtio-serial'/>
>> >>>>>> </channel>
>> >>>>>> </devices>
>> >>>>>> <memory>262144</memory>
>> >>>>>> <devices>
>> >>>>>> <memballoon model='none'/>
>> >>>>>> </devices>
>> >>>>>> <vcpu>1</vcpu>
>> >>>>>> <os>
>> >>>>>> <type arch='x86_64' machine='pc'>hvm</type>
>> >>>>>> <boot dev='cdrom'/>
>> >>>>>> <boot dev='hd'/>
>> >>>>>> </os>
>> >>>>>> <cputune>
>> >>>>>> <shares>166</shares>
>> >>>>>> </cputune>
>> >>>>>> <cpu></cpu><on_reboot>restart</on_reboot>
>> >>>>>> <on_poweroff>destroy</on_poweroff>
>> >>>>>> <on_crash>destroy</on_crash>
>> >>>>>> </domain>
>> >>>>>>
>> >>>>>> 2015-03-10 16:44:11,467{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-1:) Executing:
>> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>> >>>>>> hviasocket.pl
>> >>>>>> -n
>> >>>>>> r-2191-VM -p %vpccidr=
>> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>> >>>>>>
>> >>>>>> 2015-03-10 16:44:11,494{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>> >>>>>> 2015-03-10 16:44:11,495{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Trying to connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:44:14,501{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Could not connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>> >>>>>> (Thread-10:)
>> >>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
>> >>>>>> libvirt,
>> >>>>>> continuing
>> >>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>> >>>>>> (Thread-10:)
>> >>>>>> Executing:
>> >>>>>>
>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>> >>>>>> -i
>> >>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>> >>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>> >>>>>> 2015-03-10 16:44:17,019{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>> >>>>>> (Thread-10:)
>> >>>>>> Execution is successful.
>> >>>>>> 2015-03-10 16:44:17,953{GMT} DEBUG [cloud.agent.Agent]
>> (UgentTask-5:)
>> >>>>>> Sending ping: Seq 788-8: { Cmd , MgmtId: -1, via: 788, Ver: v1,
>> Flags:
>> >>>>>> 11,
>> >>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>> >>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>> >>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>> >>>>>> 18.domaina.net
>> >>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>> >>>>>> hostType":"Routing","hostId":788,"wait":0}}]
>> >>>>>> }
>> >>>>>> 2015-03-10 16:44:17,964{GMT} DEBUG [cloud.agent.Agent]
>> >>>>>> (Agent-Handler-3:)
>> >>>>>> Received response: Seq 788-8: { Ans: , MgmtId: 161344838950, via:
>> 788,
>> >>>>>> Ver: v1, Flags: 100010,
>> >>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>> >>>>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>> >>>>>> }
>> >>>>>> 2015-03-10 16:44:19,502{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Trying to connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:44:22,509{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Could not connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:44:27,510{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Unable to logon to 169.254.2.99
>> >>>>>> 2015-03-10 16:44:27,512{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-1:) Executing:
>> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>> >>>>>> hviasocket.pl
>> >>>>>> -n
>> >>>>>> r-2191-VM -p %vpccidr=
>> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>> >>>>>>
>> >>>>>> 2015-03-10 16:44:27,540{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>> >>>>>> 2015-03-10 16:44:27,541{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Trying to connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:44:30,548{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Could not connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:44:31,378{GMT} DEBUG [cloud.agent.Agent]
>> >>>>>> (agentRequest-Handler-3:) Processing command:
>> >>>>>> com.cloud.agent.api.GetHostStatsCommand
>> >>>>>> 2015-03-10 16:44:31,379{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c idle=$(top -b -n
>> >>>>>> 1|grep
>> >>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>> >>>>>> 2015-03-10 16:44:31,910{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-3:) Execution is successful.
>> >>>>>> 2015-03-10 16:44:31,911{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c
>> freeMem=$(free|grep
>> >>>>>> cache:|awk '{print $4}');echo $freeMem
>> >>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-3:) Execution is successful.
>> >>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c free|grep Mem:|awk
>> >>>>>> '{print $2}'
>> >>>>>> 2015-03-10 16:44:31,920{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-3:) Execution is successful.
>> >>>>>> 2015-03-10 16:44:35,549{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Trying to connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:44:38,556{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Could not connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Unable to logon to 169.254.2.99
>> >>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-1:) Executing:
>> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>> >>>>>> hviasocket.pl
>> >>>>>> -n
>> >>>>>> r-2191-VM -p %vpccidr=
>> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>> >>>>>>
>> >>>>>> 2015-03-10 16:44:43,587{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>> >>>>>> 2015-03-10 16:44:43,588{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Trying to connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:44:46,594{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Could not connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:44:49,449{GMT} DEBUG [cloud.agent.Agent]
>> >>>>>> (agentRequest-Handler-4:) Processing command:
>> >>>>>> com.cloud.agent.api.GetVmStatsCommand
>> >>>>>> 2015-03-10 16:44:51,595{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Trying to connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:44:54,600{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Could not connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:44:59,601{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Unable to logon to 169.254.2.99
>> >>>>>> 2015-03-10 16:44:59,602{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-1:) Executing:
>> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>> >>>>>> hviasocket.pl
>> >>>>>> -n
>> >>>>>> r-2191-VM -p %vpccidr=
>> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>> >>>>>>
>> >>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>> >>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Trying to connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:45:02,634{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Could not connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:45:07,635{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Trying to connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:45:10,640{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Could not connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:45:12,066{GMT} DEBUG [cloud.agent.Agent]
>> >>>>>> (agentRequest-Handler-5:) Processing command:
>> >>>>>> com.cloud.agent.api.GetStorageStatsCommand
>> >>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Unable to logon to 169.254.2.99
>> >>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-1:) Executing:
>> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>> >>>>>> hviasocket.pl
>> >>>>>> -n
>> >>>>>> r-2191-VM -p %vpccidr=
>> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>> >>>>>>
>> >>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>> >>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Trying to connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:45:17,021{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>> >>>>>> (Thread-11:)
>> >>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
>> >>>>>> libvirt,
>> >>>>>> continuing
>> >>>>>> 2015-03-10 16:45:17,022{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>> >>>>>> (Thread-11:)
>> >>>>>> Executing:
>> >>>>>>
>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>> >>>>>> -i
>> >>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>> >>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>> >>>>>> 2015-03-10 16:45:17,036{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>> >>>>>> (Thread-11:)
>> >>>>>> Execution is successful.
>> >>>>>> 2015-03-10 16:45:17,951{GMT} DEBUG [cloud.agent.Agent]
>> (UgentTask-5:)
>> >>>>>> Sending ping: Seq 788-9: { Cmd , MgmtId: -1, via: 788, Ver: v1,
>> Flags:
>> >>>>>> 11,
>> >>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>> >>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>> >>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>> >>>>>> 18.domaina.net
>> >>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> >>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>> >>>>>> hostType":"Routing","hostId":788,"wait":0}}]
>> >>>>>> }
>> >>>>>> 2015-03-10 16:45:18,002{GMT} DEBUG [cloud.agent.Agent]
>> >>>>>> (Agent-Handler-2:)
>> >>>>>> Received response: Seq 788-9: { Ans: , MgmtId: 161344838950, via:
>> 788,
>> >>>>>> Ver: v1, Flags: 100010,
>> >>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>> >>>>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>> >>>>>> }
>> >>>>>> 2015-03-10 16:45:18,672{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Could not connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:45:23,673{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Trying to connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:45:26,680{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Could not connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:45:31,682{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Unable to logon to 169.254.2.99
>> >>>>>> 2015-03-10 16:45:31,683{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-1:) Executing:
>> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>> >>>>>> hviasocket.pl
>> >>>>>> -n
>> >>>>>> r-2191-VM -p %vpccidr=
>> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>> >>>>>>
>> >>>>>> 2015-03-10 16:45:36,684{GMT} WARN [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (Script-5:) Interrupting script.
>> >>>>>> 2015-03-10 16:45:36,686{GMT} WARN [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-1:) Timed out:
>> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>> >>>>>> hviasocket.pl
>> >>>>>> -n
>> >>>>>> r-2191-VM -p %vpccidr=
>> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>> >>>>>> . Output is:
>> >>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-1:) passcmd failed:timeout
>> >>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Trying to connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [cloud.agent.Agent]
>> >>>>>> (agentRequest-Handler-2:) Processing command:
>> >>>>>> com.cloud.agent.api.GetHostStatsCommand
>> >>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c idle=$(top -b -n
>> >>>>>> 1|grep
>> >>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>> >>>>>> 2015-03-10 16:45:37,967{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-2:) Execution is successful.
>> >>>>>> 2015-03-10 16:45:37,968{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c
>> freeMem=$(free|grep
>> >>>>>> cache:|awk '{print $4}');echo $freeMem
>> >>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-2:) Execution is successful.
>> >>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c free|grep Mem:|awk
>> >>>>>> '{print $2}'
>> >>>>>> 2015-03-10 16:45:37,977{GMT} DEBUG [kvm.resource.
>> >>>>>> LibvirtComputingResource]
>> >>>>>> (agentRequest-Handler-2:) Execution is successful.
>> >>>>>> 2015-03-10 16:45:39,692{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Could not connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:45:44,693{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Trying to connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:45:47,700{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Could not connect to 169.254.2.99
>> >>>>>> 2015-03-10 16:45:51,807{GMT} DEBUG [cloud.agent.Agent]
>> >>>>>> (agentRequest-Handler-3:) Processing command:
>> >>>>>> com.cloud.agent.api.GetVmStatsCommand
>> >>>>>> 2015-03-10 16:45:52,701{GMT} DEBUG
>> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>> >>>>>> (agentRequest-Handler-1:)
>> >>>>>> Unable to logon to 169.254.2.99
>> >>>>>>
>> >>>>>> On 10 March 2015 at 17:50, ilya musayev <
>> ilya.mailing.lists@gmail.com>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>> What kernel are you running? Please include Libvirt and Qemu
>> version.
>> >>>>>>
>> >>>>>>>
>> >>>>>>> On 3/10/15 8:21 AM, Andrija Panic wrote:
>> >>>>>>>
>> >>>>>>> Hi,
>> >>>>>>>
>> >>>>>>>> we are having issues from time to time with starting VR - while
>> >>>>>>>> creating
>> >>>>>>>> VPC.
>> >>>>>>>>
>> >>>>>>>> Here is the few lines from MGMT log:
>> >>>>>>>>
>> >>>>>>>> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
>> >>>>>>>> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start vpc
>> [VPC
>> >>>>>>>> [621-VPC-delete] due to
>> >>>>>>>> com.cloud.exception.AgentUnavailableException: Resource [Host:788]
>> >>>>>>>> is
>> >>>>>>>> unreachable: Host 788: Unable to start instance due to Unable to
>> >>>>>>>> start
>> >>>>>>>> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not
>> >>>>>>>> retrying
>> >>>>>>>> at
>> >>>>>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
>> >>>>>>>> VirtualMachineManagerImpl.java:1072)
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> The interesting part is on the Agent itself:
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.
>> >>>>>>>> LibvirtComputingResource]
>> >>>>>>>> (agentRequest-Handler-2:) Timed out:
>> >>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>> >>>>>>>> hviasocket.pl
>> >>>>>>>> -n
>> >>>>>>>> r-2189-VM -p %vpccidr=
>> >>>>>>>> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%
>> >>>>>>>> template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=
>> >>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>> >>>>>>>> . Output is:
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Interesting stuff is - it's seems somewhat random, 3-4 out of 5
>> >>>>>>>> works.
>> >>>>>>>>
>> >>>>>>>> Any suggestion are really appreciated, since this is live system
>> :(
>> >>>>>>>> ACS 4.3.0, KVM, CEPH
>> >>>>>>>>
>> >>>>>>>> Thanks,
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >>
>> >> --
>> >>
>> >> Andrija Panić
>> >>
>> >
>> >
>> >
>> > --
>> >
>> > Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
Re: [HELP] VR sometimes fail to start
Posted by Andrija Panic <an...@gmail.com>.
Marcus - QEMU is 0.12.x (CentOS 6.4)
but libvirt is compiled 1.2.3
On 14 March 2015 at 16:58, Marcus <sh...@gmail.com> wrote:
> Oh, this is an unsupported version. should be using libvirt 0.9.4+ and
> Qemu 1.0+ per the install docs.
>
> On Sat, Mar 14, 2015 at 8:56 AM, Andrija Panic <an...@gmail.com>
> wrote:
> > Tried the proposed solution - but it still don't work - BTW my qemu is
> > 0.12.x - from CentOS 6.4...
> >
> > When I manually start the script with all parameters, it simply hangs, no
> > output at all, until ctrl + c
> >
> > BTW this is with now upgraded ACS to 4.3.2...
> >
> > :(
> >
> > On 12 March 2015 at 09:24, Andrija Panic <an...@gmail.com>
> wrote:
> >
> >> Hi Marcus, will do - I'm just unconfortable changing live things, but
> that
> >> is my problem. Will let you know of the output.
> >>
> >> Thanks
> >>
> >> On 12 March 2015 at 01:34, ilya musayev <il...@gmail.com>
> >> wrote:
> >>
> >>> Andrija,
> >>>
> >>> Have you tried the solution Loic mentioned? Its seems like an easy one
> to
> >>> try..
> >>>
> >>> Thanks
> >>> ilya
> >>>
> >>>
> >>> On 3/11/15 3:54 AM, Andrija Panic wrote:
> >>>
> >>>> Oh well, I will then update to 4.3.2 anyway, and hope to see the
> problem
> >>>> go
> >>>> away...
> >>>>
> >>>> THanks guys
> >>>>
> >>>> On 10 March 2015 at 22:45, Loic Lambiel <lo...@exoscale.ch>
> >>>> wrote:
> >>>>
> >>>> Hi,
> >>>>>
> >>>>> Maybe the same issue we faced here: https://github.com/exoscale/
> >>>>> cloudstack/commit/df3e7533a78c3594cdd39b8b18e6f4a3abcb8cfd
> >>>>>
> >>>>> The patch has not been yet submitted upstream. I'll do it once it is
> >>>>> running on our production.
> >>>>>
> >>>>> Anyway, you may easily test it by performing the small change
> directly
> >>>>> into the script.
> >>>>>
> >>>>> Hope that helps.
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>> Loic Lambiel
> >>>>>
> >>>>>
> >>>>> On 10/03/15 19:40, Andrija Panic wrote:
> >>>>>
> >>>>> Hi guys,
> >>>>>>
> >>>>>> Kernel is Elrepo 3.10.67 (CentOS 6.6)
> >>>>>> Libvirt manually compiled 1.2.3 with RBD support.
> >>>>>> Qemu 0.12.1.2 - Inktank patched original RHEL 6.4 rpms with RBD
> >>>>>> support.
> >>>>>>
> >>>>>> And here is the output, with debug mode, from Agent.
> >>>>>>
> >>>>>> At the begining I see agent deleting routes for 169.254.x.x
> network...:
> >>>>>>
> >>>>>> Basically, It seems like the
> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
> >>>>>> hviasocket.pl
> >>>>>> is
> >>>>>> failing, not being able to connect to socket...
> >>>>>> And I could not ping VR (169.254.2.99)...
> >>>>>>
> >>>>>> LOG:
> >>>>>>
> >>>>>> 2015-03-10 16:43:46,976{GMT} DEBUG [cloud.agent.Agent]
> >>>>>> (agentRequest-Handler-4:) Processing command:
> >>>>>> com.cloud.agent.api.GetVmStatsCommand
> >>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
> >>>>>> (agentRequest-Handler-1:) Request:Seq 788-1319632926: { Cmd ,
> MgmtId:
> >>>>>> 161344838950, via: 788, Ver: v1, Flags: 100111,
> >>>>>> [{"com.cloud.agent.api.StartCommand":{"vm":{"id":
> >>>>>> 2191,"name":"r-2191-VM","type":"DomainRouter","cpus":1,"
> >>>>>> minSpeed":166,"maxSpeed":1000,"minRam":268435456,"maxRam":
> >>>>>> 268435456,"arch":"x86_64","os":"Debian
> >>>>>> GNU/Linux 7(64-bit)","bootArgs":" vpccidr=10.0.0.0/8
> >>>>>> domain=cs2cloud.internal dns1=8.8.8.8 dns2= template=domP
> >>>>>> name=r-2191-VM
> >>>>>> eth0ip=169.254.2.99 eth0mask=255.255.0.0 type=vpcrouter
> >>>>>> disable_rp_filter=true","rebootOnCrash":false,"
> >>>>>> enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":
> >>>>>> false,"vncPassword":"e9e24eb0b1ff8ec2","params":{"
> >>>>>> memoryOvercommitRatio":"1.0","cpuOvercommitRatio":"6.0"},"
> >>>>>> uuid":"485587f7-4a04-4d77-bc36-2367f2adf77e","disks":[{"
> >>>>>> data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
> >>>>>> uuid":"d042e003-5278-461d-8a35-3b1261164f93","
> >>>>>> volumeType":"ROOT","dataStore":{"org.apache.cloudstack.
> >>>>>> storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-
> >>>>>> 88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
> >>>>>> cephmon.domaina.net","path":"cloudstack","port":6789,"url":"RBD://
> >>>>>> cephmon.domaina.net/cloudstack/?ROLE=Primary&
> >>>>>> STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6
> >>>>>> "}},"name":"ROOT-2191","size":2621440000,"path":"d042e003-
> >>>>>> 5278-461d-8a35-3b1261164f93","volumeId":2857,"vmName":"r-
> >>>>>> 2191-VM","accountId":2,"format":"RAW","id":2857,"
> >>>>>> deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"
> >>>>>> d042e003-5278-461d-8a35-3b1261164f93","type":"ROOT","_
> >>>>>> details":{"managed":"false","storagePort":"6789","storageHost":"
> >>>>>> cephmon.domaina.net
> >>>>>> ","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"
> >>>>>> networkRateMbps":-1,"defaultNic":false,"uuid":"
> >>>>>> d27a7aad-2dcc-45c7-bf30-
> >>>>>> 3302d011da66","ip":"169.254.2.99","netmask":"255.255.0.0","
> >>>>>> gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:63","
> >>>>>> broadcastType":"LinkLocal","type":"Control","
> >>>>>> isSecurityGroupEnabled":false}]},"hostIp":"10.44.253.180","
> >>>>>> executeInSequence":false,"wait":0}},{"com.cloud.agent.
> >>>>>> api.check.CheckSshCommand":{"ip":"169.254.2.99","port":
> >>>>>> 3922,"interval":6,"retries":100,"name":"r-2191-VM","wait":
> >>>>>> 0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
> >>>>>> router.name
> >>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
> >>>>>> com.cloud.agent.api.PlugNicCommand":{"nic":{"
> >>>>>> deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid"
> >>>>>> :"fe848dbe-cf4b-473e-8124-62289f86b144","ip":"5.226.146.
> >>>>>> 62","netmask":"255.255.255.128","gateway":"5.226.146.1","
> >>>>>> mac":"06:f0:d6:00:01:49","broadcastType":"Vlan","type":"
> >>>>>> Public","broadcastUri":"vlan://untagged","isolationUri":"
> >>>>>>
> vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"},"
> >>>>>> instanceName":"r-2191-VM","vmType":"DomainRouter","wait":
> >>>>>> 0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"
> >>>>>> ipAddresses":[{"accountId":2,"publicIp":"5.226.146.62","
> >>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
> >>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
> >>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
> >>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
> >>>>>> networkName":"breth1-500"}],"accessDetails":{"router.guest.
> >>>>>> ip":"5.226.146.62","zone.network.type":"Advanced","
> >>>>>> router.name
> >>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
> >>>>>> com.cloud.agent.api.routing.SetSourceNatCommand":{"
> >>>>>> ipAddress":{"accountId":2,"publicIp":"5.226.146.62","
> >>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
> >>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
> >>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
> >>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
> >>>>>> networkName":"breth1-500"},"add":true,"accessDetails":{"
> >>>>>> zone.network.type":"Advanced","
> >>>>>> router.name":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{}]
> }
> >>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
> >>>>>> (agentRequest-Handler-1:) Processing command:
> >>>>>> com.cloud.agent.api.StartCommand
> >>>>>> 2015-03-10 16:43:58,257{GMT} DEBUG [cloud.agent.Agent]
> >>>>>> (agentRequest-Handler-5:) Processing command:
> >>>>>> com.cloud.agent.api.GetStorageStatsCommand
> >>>>>> 2015-03-10 16:44:01,747{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
> >>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
> >>>>>> GNU/Linux 7(64-bit)
> >>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [cloud.agent.Agent]
> >>>>>> (agentRequest-Handler-2:) Processing command:
> >>>>>> com.cloud.agent.api.NetworkUsageCommand
> >>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-2:) Executing:
> >>>>>> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
> >>>>>> vpc_netusage.sh 169.254.2.1 -l 46.232.180.55 -g
> >>>>>> 2015-03-10 16:44:06,439{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-2:) Execution is successful.
> >>>>>> 2015-03-10 16:44:06,441{GMT} DEBUG [cloud.agent.Agent]
> >>>>>> (agentRequest-Handler-2:) Seq 788-1319632928: { Ans: , MgmtId:
> >>>>>> 161344838950, via: 788, Ver: v1, Flags: 10,
> >>>>>>
> [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-1873-VM","
> >>>>>>
> bytesSent":198379,"bytesReceived":7380903,"result":true,"details":"","
> >>>>>> wait":0}}]
> >>>>>> }
> >>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [kvm.resource.BridgeVifDriver]
> >>>>>> (agentRequest-Handler-1:) nic=[Nic:Control-169.254.2.99-null]
> >>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [utils.script.Script]
> >>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route | grep
> >>>>>> 169.254.0.0/16
> >>>>>> 2015-03-10 16:44:11,060{GMT} DEBUG [utils.script.Script]
> >>>>>> (agentRequest-Handler-1:) Execution is successful.
> >>>>>> 2015-03-10 16:44:11,061{GMT} DEBUG [utils.script.Script]
> >>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route del
> >>>>>> 169.254.0.0/16
> >>>>>> 2015-03-10 16:44:11,064{GMT} DEBUG [utils.script.Script]
> >>>>>> (agentRequest-Handler-1:) Execution is successful.
> >>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
> >>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
> >>>>>> GNU/Linux 7(64-bit)
> >>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-1:) starting r-2191-VM: <domain type='kvm'>
> >>>>>> <name>r-2191-VM</name>
> >>>>>> <uuid>485587f7-4a04-4d77-bc36-2367f2adf77e</uuid>
> >>>>>> <description>Debian GNU/Linux 7(64-bit)</description>
> >>>>>> <clock offset='utc'>
> >>>>>> </clock>
> >>>>>> <features>
> >>>>>> <pae/>
> >>>>>> <apic/>
> >>>>>> <acpi/>
> >>>>>> </features>
> >>>>>> <devices>
> >>>>>> <emulator>/usr/libexec/qemu-kvm</emulator>
> >>>>>> <interface type='bridge'>
> >>>>>> <source bridge='cloud0'/>
> >>>>>> <mac address='0e:00:a9:fe:02:63'/>
> >>>>>> <model type='virtio'/>
> >>>>>> </interface>
> >>>>>> <serial type='pty'>
> >>>>>> <target port='0'/>
> >>>>>> </serial>
> >>>>>> <graphics type='vnc' autoport='yes' listen='10.44.253.180'
> >>>>>> passwd='e9e24eb0b1ff8ec2'/>
> >>>>>> <disk device='disk' type='network'>
> >>>>>> <driver name='qemu' type='raw' cache='none' />
> >>>>>> <source protocol='rbd'
> >>>>>> name='cloudstack/d042e003-5278-461d-8a35-3b1261164f93'>
> >>>>>> <host name='cephmon.domain.internal' port='6789'/>
> >>>>>> </source>
> >>>>>> <auth username='cloudstack'>
> >>>>>> <secret type='ceph' uuid='5b93422e-1a66-353d-88a8-2203f79b1dc6'/>
> >>>>>> </auth>
> >>>>>> <target dev='vda' bus='virtio'/>
> >>>>>> </disk>
> >>>>>> <disk device='cdrom' type='file'>
> >>>>>> <driver name='qemu' type='raw' cache='none' />
> >>>>>> <source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
> >>>>>> <target dev='hdc' bus='ide'/>
> >>>>>> </disk>
> >>>>>> <console type='pty'>
> >>>>>> <target port='0'/>
> >>>>>> </console>
> >>>>>> <input type='tablet' bus='usb'/>
> >>>>>> <channel type='unix'>
> >>>>>> <source mode='bind' path='/var/lib/libvirt/qemu/r-2191-VM.agent'/>
> >>>>>> <target type='virtio' name='r-2191-VM.vport'/>
> >>>>>> <address type='virtio-serial'/>
> >>>>>> </channel>
> >>>>>> </devices>
> >>>>>> <memory>262144</memory>
> >>>>>> <devices>
> >>>>>> <memballoon model='none'/>
> >>>>>> </devices>
> >>>>>> <vcpu>1</vcpu>
> >>>>>> <os>
> >>>>>> <type arch='x86_64' machine='pc'>hvm</type>
> >>>>>> <boot dev='cdrom'/>
> >>>>>> <boot dev='hd'/>
> >>>>>> </os>
> >>>>>> <cputune>
> >>>>>> <shares>166</shares>
> >>>>>> </cputune>
> >>>>>> <cpu></cpu><on_reboot>restart</on_reboot>
> >>>>>> <on_poweroff>destroy</on_poweroff>
> >>>>>> <on_crash>destroy</on_crash>
> >>>>>> </domain>
> >>>>>>
> >>>>>> 2015-03-10 16:44:11,467{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-1:) Executing:
> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
> >>>>>> hviasocket.pl
> >>>>>> -n
> >>>>>> r-2191-VM -p %vpccidr=
> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
> >>>>>>
> >>>>>> 2015-03-10 16:44:11,494{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-1:) Execution is successful.
> >>>>>> 2015-03-10 16:44:11,495{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Trying to connect to 169.254.2.99
> >>>>>> 2015-03-10 16:44:14,501{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Could not connect to 169.254.2.99
> >>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
> >>>>>> (Thread-10:)
> >>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
> >>>>>> libvirt,
> >>>>>> continuing
> >>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
> >>>>>> (Thread-10:)
> >>>>>> Executing:
> >>>>>>
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
> >>>>>> -i
> >>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
> >>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
> >>>>>> 2015-03-10 16:44:17,019{GMT} DEBUG [kvm.resource.KVMHAMonitor]
> >>>>>> (Thread-10:)
> >>>>>> Execution is successful.
> >>>>>> 2015-03-10 16:44:17,953{GMT} DEBUG [cloud.agent.Agent]
> (UgentTask-5:)
> >>>>>> Sending ping: Seq 788-8: { Cmd , MgmtId: -1, via: 788, Ver: v1,
> Flags:
> >>>>>> 11,
> >>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
> >>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
> >>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
> >>>>>> 18.domaina.net
> >>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
> >>>>>> hostType":"Routing","hostId":788,"wait":0}}]
> >>>>>> }
> >>>>>> 2015-03-10 16:44:17,964{GMT} DEBUG [cloud.agent.Agent]
> >>>>>> (Agent-Handler-3:)
> >>>>>> Received response: Seq 788-8: { Ans: , MgmtId: 161344838950, via:
> 788,
> >>>>>> Ver: v1, Flags: 100010,
> >>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
> >>>>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
> >>>>>> }
> >>>>>> 2015-03-10 16:44:19,502{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Trying to connect to 169.254.2.99
> >>>>>> 2015-03-10 16:44:22,509{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Could not connect to 169.254.2.99
> >>>>>> 2015-03-10 16:44:27,510{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Unable to logon to 169.254.2.99
> >>>>>> 2015-03-10 16:44:27,512{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-1:) Executing:
> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
> >>>>>> hviasocket.pl
> >>>>>> -n
> >>>>>> r-2191-VM -p %vpccidr=
> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
> >>>>>>
> >>>>>> 2015-03-10 16:44:27,540{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-1:) Execution is successful.
> >>>>>> 2015-03-10 16:44:27,541{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Trying to connect to 169.254.2.99
> >>>>>> 2015-03-10 16:44:30,548{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Could not connect to 169.254.2.99
> >>>>>> 2015-03-10 16:44:31,378{GMT} DEBUG [cloud.agent.Agent]
> >>>>>> (agentRequest-Handler-3:) Processing command:
> >>>>>> com.cloud.agent.api.GetHostStatsCommand
> >>>>>> 2015-03-10 16:44:31,379{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c idle=$(top -b -n
> >>>>>> 1|grep
> >>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
> >>>>>> 2015-03-10 16:44:31,910{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-3:) Execution is successful.
> >>>>>> 2015-03-10 16:44:31,911{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c
> freeMem=$(free|grep
> >>>>>> cache:|awk '{print $4}');echo $freeMem
> >>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-3:) Execution is successful.
> >>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c free|grep Mem:|awk
> >>>>>> '{print $2}'
> >>>>>> 2015-03-10 16:44:31,920{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-3:) Execution is successful.
> >>>>>> 2015-03-10 16:44:35,549{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Trying to connect to 169.254.2.99
> >>>>>> 2015-03-10 16:44:38,556{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Could not connect to 169.254.2.99
> >>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Unable to logon to 169.254.2.99
> >>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-1:) Executing:
> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
> >>>>>> hviasocket.pl
> >>>>>> -n
> >>>>>> r-2191-VM -p %vpccidr=
> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
> >>>>>>
> >>>>>> 2015-03-10 16:44:43,587{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-1:) Execution is successful.
> >>>>>> 2015-03-10 16:44:43,588{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Trying to connect to 169.254.2.99
> >>>>>> 2015-03-10 16:44:46,594{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Could not connect to 169.254.2.99
> >>>>>> 2015-03-10 16:44:49,449{GMT} DEBUG [cloud.agent.Agent]
> >>>>>> (agentRequest-Handler-4:) Processing command:
> >>>>>> com.cloud.agent.api.GetVmStatsCommand
> >>>>>> 2015-03-10 16:44:51,595{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Trying to connect to 169.254.2.99
> >>>>>> 2015-03-10 16:44:54,600{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Could not connect to 169.254.2.99
> >>>>>> 2015-03-10 16:44:59,601{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Unable to logon to 169.254.2.99
> >>>>>> 2015-03-10 16:44:59,602{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-1:) Executing:
> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
> >>>>>> hviasocket.pl
> >>>>>> -n
> >>>>>> r-2191-VM -p %vpccidr=
> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
> >>>>>>
> >>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-1:) Execution is successful.
> >>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Trying to connect to 169.254.2.99
> >>>>>> 2015-03-10 16:45:02,634{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Could not connect to 169.254.2.99
> >>>>>> 2015-03-10 16:45:07,635{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Trying to connect to 169.254.2.99
> >>>>>> 2015-03-10 16:45:10,640{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Could not connect to 169.254.2.99
> >>>>>> 2015-03-10 16:45:12,066{GMT} DEBUG [cloud.agent.Agent]
> >>>>>> (agentRequest-Handler-5:) Processing command:
> >>>>>> com.cloud.agent.api.GetStorageStatsCommand
> >>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Unable to logon to 169.254.2.99
> >>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-1:) Executing:
> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
> >>>>>> hviasocket.pl
> >>>>>> -n
> >>>>>> r-2191-VM -p %vpccidr=
> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
> >>>>>>
> >>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-1:) Execution is successful.
> >>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Trying to connect to 169.254.2.99
> >>>>>> 2015-03-10 16:45:17,021{GMT} DEBUG [kvm.resource.KVMHAMonitor]
> >>>>>> (Thread-11:)
> >>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
> >>>>>> libvirt,
> >>>>>> continuing
> >>>>>> 2015-03-10 16:45:17,022{GMT} DEBUG [kvm.resource.KVMHAMonitor]
> >>>>>> (Thread-11:)
> >>>>>> Executing:
> >>>>>>
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
> >>>>>> -i
> >>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
> >>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
> >>>>>> 2015-03-10 16:45:17,036{GMT} DEBUG [kvm.resource.KVMHAMonitor]
> >>>>>> (Thread-11:)
> >>>>>> Execution is successful.
> >>>>>> 2015-03-10 16:45:17,951{GMT} DEBUG [cloud.agent.Agent]
> (UgentTask-5:)
> >>>>>> Sending ping: Seq 788-9: { Cmd , MgmtId: -1, via: 788, Ver: v1,
> Flags:
> >>>>>> 11,
> >>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
> >>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
> >>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
> >>>>>> 18.domaina.net
> >>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
> >>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
> >>>>>> hostType":"Routing","hostId":788,"wait":0}}]
> >>>>>> }
> >>>>>> 2015-03-10 16:45:18,002{GMT} DEBUG [cloud.agent.Agent]
> >>>>>> (Agent-Handler-2:)
> >>>>>> Received response: Seq 788-9: { Ans: , MgmtId: 161344838950, via:
> 788,
> >>>>>> Ver: v1, Flags: 100010,
> >>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
> >>>>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
> >>>>>> }
> >>>>>> 2015-03-10 16:45:18,672{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Could not connect to 169.254.2.99
> >>>>>> 2015-03-10 16:45:23,673{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Trying to connect to 169.254.2.99
> >>>>>> 2015-03-10 16:45:26,680{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Could not connect to 169.254.2.99
> >>>>>> 2015-03-10 16:45:31,682{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Unable to logon to 169.254.2.99
> >>>>>> 2015-03-10 16:45:31,683{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-1:) Executing:
> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
> >>>>>> hviasocket.pl
> >>>>>> -n
> >>>>>> r-2191-VM -p %vpccidr=
> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
> >>>>>>
> >>>>>> 2015-03-10 16:45:36,684{GMT} WARN [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (Script-5:) Interrupting script.
> >>>>>> 2015-03-10 16:45:36,686{GMT} WARN [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-1:) Timed out:
> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
> >>>>>> hviasocket.pl
> >>>>>> -n
> >>>>>> r-2191-VM -p %vpccidr=
> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
> >>>>>> . Output is:
> >>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-1:) passcmd failed:timeout
> >>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Trying to connect to 169.254.2.99
> >>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [cloud.agent.Agent]
> >>>>>> (agentRequest-Handler-2:) Processing command:
> >>>>>> com.cloud.agent.api.GetHostStatsCommand
> >>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c idle=$(top -b -n
> >>>>>> 1|grep
> >>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
> >>>>>> 2015-03-10 16:45:37,967{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-2:) Execution is successful.
> >>>>>> 2015-03-10 16:45:37,968{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c
> freeMem=$(free|grep
> >>>>>> cache:|awk '{print $4}');echo $freeMem
> >>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-2:) Execution is successful.
> >>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c free|grep Mem:|awk
> >>>>>> '{print $2}'
> >>>>>> 2015-03-10 16:45:37,977{GMT} DEBUG [kvm.resource.
> >>>>>> LibvirtComputingResource]
> >>>>>> (agentRequest-Handler-2:) Execution is successful.
> >>>>>> 2015-03-10 16:45:39,692{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Could not connect to 169.254.2.99
> >>>>>> 2015-03-10 16:45:44,693{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Trying to connect to 169.254.2.99
> >>>>>> 2015-03-10 16:45:47,700{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Could not connect to 169.254.2.99
> >>>>>> 2015-03-10 16:45:51,807{GMT} DEBUG [cloud.agent.Agent]
> >>>>>> (agentRequest-Handler-3:) Processing command:
> >>>>>> com.cloud.agent.api.GetVmStatsCommand
> >>>>>> 2015-03-10 16:45:52,701{GMT} DEBUG
> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
> >>>>>> (agentRequest-Handler-1:)
> >>>>>> Unable to logon to 169.254.2.99
> >>>>>>
> >>>>>> On 10 March 2015 at 17:50, ilya musayev <
> ilya.mailing.lists@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>> What kernel are you running? Please include Libvirt and Qemu
> version.
> >>>>>>
> >>>>>>>
> >>>>>>> On 3/10/15 8:21 AM, Andrija Panic wrote:
> >>>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>>> we are having issues from time to time with starting VR - while
> >>>>>>>> creating
> >>>>>>>> VPC.
> >>>>>>>>
> >>>>>>>> Here is the few lines from MGMT log:
> >>>>>>>>
> >>>>>>>> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
> >>>>>>>> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start vpc
> [VPC
> >>>>>>>> [621-VPC-delete] due to
> >>>>>>>> com.cloud.exception.AgentUnavailableException: Resource [Host:788]
> >>>>>>>> is
> >>>>>>>> unreachable: Host 788: Unable to start instance due to Unable to
> >>>>>>>> start
> >>>>>>>> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not
> >>>>>>>> retrying
> >>>>>>>> at
> >>>>>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
> >>>>>>>> VirtualMachineManagerImpl.java:1072)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> The interesting part is on the Agent itself:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.
> >>>>>>>> LibvirtComputingResource]
> >>>>>>>> (agentRequest-Handler-2:) Timed out:
> >>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
> >>>>>>>> hviasocket.pl
> >>>>>>>> -n
> >>>>>>>> r-2189-VM -p %vpccidr=
> >>>>>>>> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%
> >>>>>>>> template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=
> >>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
> >>>>>>>> . Output is:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Interesting stuff is - it's seems somewhat random, 3-4 out of 5
> >>>>>>>> works.
> >>>>>>>>
> >>>>>>>> Any suggestion are really appreciated, since this is live system
> :(
> >>>>>>>> ACS 4.3.0, KVM, CEPH
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >>
> >> Andrija Panić
> >>
> >
> >
> >
> > --
> >
> > Andrija Panić
>
--
Andrija Panić
Re: [HELP] VR sometimes fail to start
Posted by Marcus <sh...@gmail.com>.
Oh, this is an unsupported version. should be using libvirt 0.9.4+ and
Qemu 1.0+ per the install docs.
On Sat, Mar 14, 2015 at 8:56 AM, Andrija Panic <an...@gmail.com> wrote:
> Tried the proposed solution - but it still don't work - BTW my qemu is
> 0.12.x - from CentOS 6.4...
>
> When I manually start the script with all parameters, it simply hangs, no
> output at all, until ctrl + c
>
> BTW this is with now upgraded ACS to 4.3.2...
>
> :(
>
> On 12 March 2015 at 09:24, Andrija Panic <an...@gmail.com> wrote:
>
>> Hi Marcus, will do - I'm just unconfortable changing live things, but that
>> is my problem. Will let you know of the output.
>>
>> Thanks
>>
>> On 12 March 2015 at 01:34, ilya musayev <il...@gmail.com>
>> wrote:
>>
>>> Andrija,
>>>
>>> Have you tried the solution Loic mentioned? Its seems like an easy one to
>>> try..
>>>
>>> Thanks
>>> ilya
>>>
>>>
>>> On 3/11/15 3:54 AM, Andrija Panic wrote:
>>>
>>>> Oh well, I will then update to 4.3.2 anyway, and hope to see the problem
>>>> go
>>>> away...
>>>>
>>>> THanks guys
>>>>
>>>> On 10 March 2015 at 22:45, Loic Lambiel <lo...@exoscale.ch>
>>>> wrote:
>>>>
>>>> Hi,
>>>>>
>>>>> Maybe the same issue we faced here: https://github.com/exoscale/
>>>>> cloudstack/commit/df3e7533a78c3594cdd39b8b18e6f4a3abcb8cfd
>>>>>
>>>>> The patch has not been yet submitted upstream. I'll do it once it is
>>>>> running on our production.
>>>>>
>>>>> Anyway, you may easily test it by performing the small change directly
>>>>> into the script.
>>>>>
>>>>> Hope that helps.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Loic Lambiel
>>>>>
>>>>>
>>>>> On 10/03/15 19:40, Andrija Panic wrote:
>>>>>
>>>>> Hi guys,
>>>>>>
>>>>>> Kernel is Elrepo 3.10.67 (CentOS 6.6)
>>>>>> Libvirt manually compiled 1.2.3 with RBD support.
>>>>>> Qemu 0.12.1.2 - Inktank patched original RHEL 6.4 rpms with RBD
>>>>>> support.
>>>>>>
>>>>>> And here is the output, with debug mode, from Agent.
>>>>>>
>>>>>> At the begining I see agent deleting routes for 169.254.x.x network...:
>>>>>>
>>>>>> Basically, It seems like the
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>> hviasocket.pl
>>>>>> is
>>>>>> failing, not being able to connect to socket...
>>>>>> And I could not ping VR (169.254.2.99)...
>>>>>>
>>>>>> LOG:
>>>>>>
>>>>>> 2015-03-10 16:43:46,976{GMT} DEBUG [cloud.agent.Agent]
>>>>>> (agentRequest-Handler-4:) Processing command:
>>>>>> com.cloud.agent.api.GetVmStatsCommand
>>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>>>>>> (agentRequest-Handler-1:) Request:Seq 788-1319632926: { Cmd , MgmtId:
>>>>>> 161344838950, via: 788, Ver: v1, Flags: 100111,
>>>>>> [{"com.cloud.agent.api.StartCommand":{"vm":{"id":
>>>>>> 2191,"name":"r-2191-VM","type":"DomainRouter","cpus":1,"
>>>>>> minSpeed":166,"maxSpeed":1000,"minRam":268435456,"maxRam":
>>>>>> 268435456,"arch":"x86_64","os":"Debian
>>>>>> GNU/Linux 7(64-bit)","bootArgs":" vpccidr=10.0.0.0/8
>>>>>> domain=cs2cloud.internal dns1=8.8.8.8 dns2= template=domP
>>>>>> name=r-2191-VM
>>>>>> eth0ip=169.254.2.99 eth0mask=255.255.0.0 type=vpcrouter
>>>>>> disable_rp_filter=true","rebootOnCrash":false,"
>>>>>> enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":
>>>>>> false,"vncPassword":"e9e24eb0b1ff8ec2","params":{"
>>>>>> memoryOvercommitRatio":"1.0","cpuOvercommitRatio":"6.0"},"
>>>>>> uuid":"485587f7-4a04-4d77-bc36-2367f2adf77e","disks":[{"
>>>>>> data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
>>>>>> uuid":"d042e003-5278-461d-8a35-3b1261164f93","
>>>>>> volumeType":"ROOT","dataStore":{"org.apache.cloudstack.
>>>>>> storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-
>>>>>> 88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
>>>>>> cephmon.domaina.net","path":"cloudstack","port":6789,"url":"RBD://
>>>>>> cephmon.domaina.net/cloudstack/?ROLE=Primary&
>>>>>> STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6
>>>>>> "}},"name":"ROOT-2191","size":2621440000,"path":"d042e003-
>>>>>> 5278-461d-8a35-3b1261164f93","volumeId":2857,"vmName":"r-
>>>>>> 2191-VM","accountId":2,"format":"RAW","id":2857,"
>>>>>> deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"
>>>>>> d042e003-5278-461d-8a35-3b1261164f93","type":"ROOT","_
>>>>>> details":{"managed":"false","storagePort":"6789","storageHost":"
>>>>>> cephmon.domaina.net
>>>>>> ","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"
>>>>>> networkRateMbps":-1,"defaultNic":false,"uuid":"
>>>>>> d27a7aad-2dcc-45c7-bf30-
>>>>>> 3302d011da66","ip":"169.254.2.99","netmask":"255.255.0.0","
>>>>>> gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:63","
>>>>>> broadcastType":"LinkLocal","type":"Control","
>>>>>> isSecurityGroupEnabled":false}]},"hostIp":"10.44.253.180","
>>>>>> executeInSequence":false,"wait":0}},{"com.cloud.agent.
>>>>>> api.check.CheckSshCommand":{"ip":"169.254.2.99","port":
>>>>>> 3922,"interval":6,"retries":100,"name":"r-2191-VM","wait":
>>>>>> 0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
>>>>>> router.name
>>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>>>>>> com.cloud.agent.api.PlugNicCommand":{"nic":{"
>>>>>> deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid"
>>>>>> :"fe848dbe-cf4b-473e-8124-62289f86b144","ip":"5.226.146.
>>>>>> 62","netmask":"255.255.255.128","gateway":"5.226.146.1","
>>>>>> mac":"06:f0:d6:00:01:49","broadcastType":"Vlan","type":"
>>>>>> Public","broadcastUri":"vlan://untagged","isolationUri":"
>>>>>> vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"},"
>>>>>> instanceName":"r-2191-VM","vmType":"DomainRouter","wait":
>>>>>> 0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"
>>>>>> ipAddresses":[{"accountId":2,"publicIp":"5.226.146.62","
>>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>>>>>> networkName":"breth1-500"}],"accessDetails":{"router.guest.
>>>>>> ip":"5.226.146.62","zone.network.type":"Advanced","
>>>>>> router.name
>>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>>>>>> com.cloud.agent.api.routing.SetSourceNatCommand":{"
>>>>>> ipAddress":{"accountId":2,"publicIp":"5.226.146.62","
>>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>>>>>> networkName":"breth1-500"},"add":true,"accessDetails":{"
>>>>>> zone.network.type":"Advanced","
>>>>>> router.name":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{}] }
>>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>>>>>> (agentRequest-Handler-1:) Processing command:
>>>>>> com.cloud.agent.api.StartCommand
>>>>>> 2015-03-10 16:43:58,257{GMT} DEBUG [cloud.agent.Agent]
>>>>>> (agentRequest-Handler-5:) Processing command:
>>>>>> com.cloud.agent.api.GetStorageStatsCommand
>>>>>> 2015-03-10 16:44:01,747{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
>>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
>>>>>> GNU/Linux 7(64-bit)
>>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [cloud.agent.Agent]
>>>>>> (agentRequest-Handler-2:) Processing command:
>>>>>> com.cloud.agent.api.NetworkUsageCommand
>>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-2:) Executing:
>>>>>> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
>>>>>> vpc_netusage.sh 169.254.2.1 -l 46.232.180.55 -g
>>>>>> 2015-03-10 16:44:06,439{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-2:) Execution is successful.
>>>>>> 2015-03-10 16:44:06,441{GMT} DEBUG [cloud.agent.Agent]
>>>>>> (agentRequest-Handler-2:) Seq 788-1319632928: { Ans: , MgmtId:
>>>>>> 161344838950, via: 788, Ver: v1, Flags: 10,
>>>>>> [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-1873-VM","
>>>>>> bytesSent":198379,"bytesReceived":7380903,"result":true,"details":"","
>>>>>> wait":0}}]
>>>>>> }
>>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [kvm.resource.BridgeVifDriver]
>>>>>> (agentRequest-Handler-1:) nic=[Nic:Control-169.254.2.99-null]
>>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [utils.script.Script]
>>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route | grep
>>>>>> 169.254.0.0/16
>>>>>> 2015-03-10 16:44:11,060{GMT} DEBUG [utils.script.Script]
>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>> 2015-03-10 16:44:11,061{GMT} DEBUG [utils.script.Script]
>>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route del
>>>>>> 169.254.0.0/16
>>>>>> 2015-03-10 16:44:11,064{GMT} DEBUG [utils.script.Script]
>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
>>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
>>>>>> GNU/Linux 7(64-bit)
>>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-1:) starting r-2191-VM: <domain type='kvm'>
>>>>>> <name>r-2191-VM</name>
>>>>>> <uuid>485587f7-4a04-4d77-bc36-2367f2adf77e</uuid>
>>>>>> <description>Debian GNU/Linux 7(64-bit)</description>
>>>>>> <clock offset='utc'>
>>>>>> </clock>
>>>>>> <features>
>>>>>> <pae/>
>>>>>> <apic/>
>>>>>> <acpi/>
>>>>>> </features>
>>>>>> <devices>
>>>>>> <emulator>/usr/libexec/qemu-kvm</emulator>
>>>>>> <interface type='bridge'>
>>>>>> <source bridge='cloud0'/>
>>>>>> <mac address='0e:00:a9:fe:02:63'/>
>>>>>> <model type='virtio'/>
>>>>>> </interface>
>>>>>> <serial type='pty'>
>>>>>> <target port='0'/>
>>>>>> </serial>
>>>>>> <graphics type='vnc' autoport='yes' listen='10.44.253.180'
>>>>>> passwd='e9e24eb0b1ff8ec2'/>
>>>>>> <disk device='disk' type='network'>
>>>>>> <driver name='qemu' type='raw' cache='none' />
>>>>>> <source protocol='rbd'
>>>>>> name='cloudstack/d042e003-5278-461d-8a35-3b1261164f93'>
>>>>>> <host name='cephmon.domain.internal' port='6789'/>
>>>>>> </source>
>>>>>> <auth username='cloudstack'>
>>>>>> <secret type='ceph' uuid='5b93422e-1a66-353d-88a8-2203f79b1dc6'/>
>>>>>> </auth>
>>>>>> <target dev='vda' bus='virtio'/>
>>>>>> </disk>
>>>>>> <disk device='cdrom' type='file'>
>>>>>> <driver name='qemu' type='raw' cache='none' />
>>>>>> <source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
>>>>>> <target dev='hdc' bus='ide'/>
>>>>>> </disk>
>>>>>> <console type='pty'>
>>>>>> <target port='0'/>
>>>>>> </console>
>>>>>> <input type='tablet' bus='usb'/>
>>>>>> <channel type='unix'>
>>>>>> <source mode='bind' path='/var/lib/libvirt/qemu/r-2191-VM.agent'/>
>>>>>> <target type='virtio' name='r-2191-VM.vport'/>
>>>>>> <address type='virtio-serial'/>
>>>>>> </channel>
>>>>>> </devices>
>>>>>> <memory>262144</memory>
>>>>>> <devices>
>>>>>> <memballoon model='none'/>
>>>>>> </devices>
>>>>>> <vcpu>1</vcpu>
>>>>>> <os>
>>>>>> <type arch='x86_64' machine='pc'>hvm</type>
>>>>>> <boot dev='cdrom'/>
>>>>>> <boot dev='hd'/>
>>>>>> </os>
>>>>>> <cputune>
>>>>>> <shares>166</shares>
>>>>>> </cputune>
>>>>>> <cpu></cpu><on_reboot>restart</on_reboot>
>>>>>> <on_poweroff>destroy</on_poweroff>
>>>>>> <on_crash>destroy</on_crash>
>>>>>> </domain>
>>>>>>
>>>>>> 2015-03-10 16:44:11,467{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-1:) Executing:
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>> hviasocket.pl
>>>>>> -n
>>>>>> r-2191-VM -p %vpccidr=
>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>
>>>>>> 2015-03-10 16:44:11,494{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>> 2015-03-10 16:44:11,495{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Trying to connect to 169.254.2.99
>>>>>> 2015-03-10 16:44:14,501{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Could not connect to 169.254.2.99
>>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>>> (Thread-10:)
>>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
>>>>>> libvirt,
>>>>>> continuing
>>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>>> (Thread-10:)
>>>>>> Executing:
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>>>>>> -i
>>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>>>>>> 2015-03-10 16:44:17,019{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>>> (Thread-10:)
>>>>>> Execution is successful.
>>>>>> 2015-03-10 16:44:17,953{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
>>>>>> Sending ping: Seq 788-8: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags:
>>>>>> 11,
>>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>>>>>> 18.domaina.net
>>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>>>>>> hostType":"Routing","hostId":788,"wait":0}}]
>>>>>> }
>>>>>> 2015-03-10 16:44:17,964{GMT} DEBUG [cloud.agent.Agent]
>>>>>> (Agent-Handler-3:)
>>>>>> Received response: Seq 788-8: { Ans: , MgmtId: 161344838950, via: 788,
>>>>>> Ver: v1, Flags: 100010,
>>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>>>>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>>>>>> }
>>>>>> 2015-03-10 16:44:19,502{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Trying to connect to 169.254.2.99
>>>>>> 2015-03-10 16:44:22,509{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Could not connect to 169.254.2.99
>>>>>> 2015-03-10 16:44:27,510{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Unable to logon to 169.254.2.99
>>>>>> 2015-03-10 16:44:27,512{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-1:) Executing:
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>> hviasocket.pl
>>>>>> -n
>>>>>> r-2191-VM -p %vpccidr=
>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>
>>>>>> 2015-03-10 16:44:27,540{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>> 2015-03-10 16:44:27,541{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Trying to connect to 169.254.2.99
>>>>>> 2015-03-10 16:44:30,548{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Could not connect to 169.254.2.99
>>>>>> 2015-03-10 16:44:31,378{GMT} DEBUG [cloud.agent.Agent]
>>>>>> (agentRequest-Handler-3:) Processing command:
>>>>>> com.cloud.agent.api.GetHostStatsCommand
>>>>>> 2015-03-10 16:44:31,379{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c idle=$(top -b -n
>>>>>> 1|grep
>>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>>>>>> 2015-03-10 16:44:31,910{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-3:) Execution is successful.
>>>>>> 2015-03-10 16:44:31,911{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c freeMem=$(free|grep
>>>>>> cache:|awk '{print $4}');echo $freeMem
>>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-3:) Execution is successful.
>>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c free|grep Mem:|awk
>>>>>> '{print $2}'
>>>>>> 2015-03-10 16:44:31,920{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-3:) Execution is successful.
>>>>>> 2015-03-10 16:44:35,549{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Trying to connect to 169.254.2.99
>>>>>> 2015-03-10 16:44:38,556{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Could not connect to 169.254.2.99
>>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Unable to logon to 169.254.2.99
>>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-1:) Executing:
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>> hviasocket.pl
>>>>>> -n
>>>>>> r-2191-VM -p %vpccidr=
>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>
>>>>>> 2015-03-10 16:44:43,587{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>> 2015-03-10 16:44:43,588{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Trying to connect to 169.254.2.99
>>>>>> 2015-03-10 16:44:46,594{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Could not connect to 169.254.2.99
>>>>>> 2015-03-10 16:44:49,449{GMT} DEBUG [cloud.agent.Agent]
>>>>>> (agentRequest-Handler-4:) Processing command:
>>>>>> com.cloud.agent.api.GetVmStatsCommand
>>>>>> 2015-03-10 16:44:51,595{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Trying to connect to 169.254.2.99
>>>>>> 2015-03-10 16:44:54,600{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Could not connect to 169.254.2.99
>>>>>> 2015-03-10 16:44:59,601{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Unable to logon to 169.254.2.99
>>>>>> 2015-03-10 16:44:59,602{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-1:) Executing:
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>> hviasocket.pl
>>>>>> -n
>>>>>> r-2191-VM -p %vpccidr=
>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>
>>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Trying to connect to 169.254.2.99
>>>>>> 2015-03-10 16:45:02,634{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Could not connect to 169.254.2.99
>>>>>> 2015-03-10 16:45:07,635{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Trying to connect to 169.254.2.99
>>>>>> 2015-03-10 16:45:10,640{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Could not connect to 169.254.2.99
>>>>>> 2015-03-10 16:45:12,066{GMT} DEBUG [cloud.agent.Agent]
>>>>>> (agentRequest-Handler-5:) Processing command:
>>>>>> com.cloud.agent.api.GetStorageStatsCommand
>>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Unable to logon to 169.254.2.99
>>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-1:) Executing:
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>> hviasocket.pl
>>>>>> -n
>>>>>> r-2191-VM -p %vpccidr=
>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>
>>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Trying to connect to 169.254.2.99
>>>>>> 2015-03-10 16:45:17,021{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>>> (Thread-11:)
>>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
>>>>>> libvirt,
>>>>>> continuing
>>>>>> 2015-03-10 16:45:17,022{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>>> (Thread-11:)
>>>>>> Executing:
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>>>>>> -i
>>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>>>>>> 2015-03-10 16:45:17,036{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>>> (Thread-11:)
>>>>>> Execution is successful.
>>>>>> 2015-03-10 16:45:17,951{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
>>>>>> Sending ping: Seq 788-9: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags:
>>>>>> 11,
>>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>>>>>> 18.domaina.net
>>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>>>>>> hostType":"Routing","hostId":788,"wait":0}}]
>>>>>> }
>>>>>> 2015-03-10 16:45:18,002{GMT} DEBUG [cloud.agent.Agent]
>>>>>> (Agent-Handler-2:)
>>>>>> Received response: Seq 788-9: { Ans: , MgmtId: 161344838950, via: 788,
>>>>>> Ver: v1, Flags: 100010,
>>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>>>>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>>>>>> }
>>>>>> 2015-03-10 16:45:18,672{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Could not connect to 169.254.2.99
>>>>>> 2015-03-10 16:45:23,673{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Trying to connect to 169.254.2.99
>>>>>> 2015-03-10 16:45:26,680{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Could not connect to 169.254.2.99
>>>>>> 2015-03-10 16:45:31,682{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Unable to logon to 169.254.2.99
>>>>>> 2015-03-10 16:45:31,683{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-1:) Executing:
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>> hviasocket.pl
>>>>>> -n
>>>>>> r-2191-VM -p %vpccidr=
>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>
>>>>>> 2015-03-10 16:45:36,684{GMT} WARN [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (Script-5:) Interrupting script.
>>>>>> 2015-03-10 16:45:36,686{GMT} WARN [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-1:) Timed out:
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>> hviasocket.pl
>>>>>> -n
>>>>>> r-2191-VM -p %vpccidr=
>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>> . Output is:
>>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-1:) passcmd failed:timeout
>>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Trying to connect to 169.254.2.99
>>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [cloud.agent.Agent]
>>>>>> (agentRequest-Handler-2:) Processing command:
>>>>>> com.cloud.agent.api.GetHostStatsCommand
>>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c idle=$(top -b -n
>>>>>> 1|grep
>>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>>>>>> 2015-03-10 16:45:37,967{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-2:) Execution is successful.
>>>>>> 2015-03-10 16:45:37,968{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c freeMem=$(free|grep
>>>>>> cache:|awk '{print $4}');echo $freeMem
>>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-2:) Execution is successful.
>>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c free|grep Mem:|awk
>>>>>> '{print $2}'
>>>>>> 2015-03-10 16:45:37,977{GMT} DEBUG [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-2:) Execution is successful.
>>>>>> 2015-03-10 16:45:39,692{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Could not connect to 169.254.2.99
>>>>>> 2015-03-10 16:45:44,693{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Trying to connect to 169.254.2.99
>>>>>> 2015-03-10 16:45:47,700{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Could not connect to 169.254.2.99
>>>>>> 2015-03-10 16:45:51,807{GMT} DEBUG [cloud.agent.Agent]
>>>>>> (agentRequest-Handler-3:) Processing command:
>>>>>> com.cloud.agent.api.GetVmStatsCommand
>>>>>> 2015-03-10 16:45:52,701{GMT} DEBUG
>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> (agentRequest-Handler-1:)
>>>>>> Unable to logon to 169.254.2.99
>>>>>>
>>>>>> On 10 March 2015 at 17:50, ilya musayev <il...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> What kernel are you running? Please include Libvirt and Qemu version.
>>>>>>
>>>>>>>
>>>>>>> On 3/10/15 8:21 AM, Andrija Panic wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>> we are having issues from time to time with starting VR - while
>>>>>>>> creating
>>>>>>>> VPC.
>>>>>>>>
>>>>>>>> Here is the few lines from MGMT log:
>>>>>>>>
>>>>>>>> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
>>>>>>>> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start vpc [VPC
>>>>>>>> [621-VPC-delete] due to
>>>>>>>> com.cloud.exception.AgentUnavailableException: Resource [Host:788]
>>>>>>>> is
>>>>>>>> unreachable: Host 788: Unable to start instance due to Unable to
>>>>>>>> start
>>>>>>>> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not
>>>>>>>> retrying
>>>>>>>> at
>>>>>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
>>>>>>>> VirtualMachineManagerImpl.java:1072)
>>>>>>>>
>>>>>>>>
>>>>>>>> The interesting part is on the Agent itself:
>>>>>>>>
>>>>>>>>
>>>>>>>> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.
>>>>>>>> LibvirtComputingResource]
>>>>>>>> (agentRequest-Handler-2:) Timed out:
>>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>>> hviasocket.pl
>>>>>>>> -n
>>>>>>>> r-2189-VM -p %vpccidr=
>>>>>>>> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>>>> template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=
>>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>>> . Output is:
>>>>>>>>
>>>>>>>>
>>>>>>>> Interesting stuff is - it's seems somewhat random, 3-4 out of 5
>>>>>>>> works.
>>>>>>>>
>>>>>>>> Any suggestion are really appreciated, since this is live system :(
>>>>>>>> ACS 4.3.0, KVM, CEPH
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
Re: [HELP] VR sometimes fail to start
Posted by Andrija Panic <an...@gmail.com>.
Tried the proposed solution - but it still don't work - BTW my qemu is
0.12.x - from CentOS 6.4...
When I manually start the script with all parameters, it simply hangs, no
output at all, until ctrl + c
BTW this is with now upgraded ACS to 4.3.2...
:(
On 12 March 2015 at 09:24, Andrija Panic <an...@gmail.com> wrote:
> Hi Marcus, will do - I'm just unconfortable changing live things, but that
> is my problem. Will let you know of the output.
>
> Thanks
>
> On 12 March 2015 at 01:34, ilya musayev <il...@gmail.com>
> wrote:
>
>> Andrija,
>>
>> Have you tried the solution Loic mentioned? Its seems like an easy one to
>> try..
>>
>> Thanks
>> ilya
>>
>>
>> On 3/11/15 3:54 AM, Andrija Panic wrote:
>>
>>> Oh well, I will then update to 4.3.2 anyway, and hope to see the problem
>>> go
>>> away...
>>>
>>> THanks guys
>>>
>>> On 10 March 2015 at 22:45, Loic Lambiel <lo...@exoscale.ch>
>>> wrote:
>>>
>>> Hi,
>>>>
>>>> Maybe the same issue we faced here: https://github.com/exoscale/
>>>> cloudstack/commit/df3e7533a78c3594cdd39b8b18e6f4a3abcb8cfd
>>>>
>>>> The patch has not been yet submitted upstream. I'll do it once it is
>>>> running on our production.
>>>>
>>>> Anyway, you may easily test it by performing the small change directly
>>>> into the script.
>>>>
>>>> Hope that helps.
>>>>
>>>> Regards,
>>>>
>>>> Loic Lambiel
>>>>
>>>>
>>>> On 10/03/15 19:40, Andrija Panic wrote:
>>>>
>>>> Hi guys,
>>>>>
>>>>> Kernel is Elrepo 3.10.67 (CentOS 6.6)
>>>>> Libvirt manually compiled 1.2.3 with RBD support.
>>>>> Qemu 0.12.1.2 - Inktank patched original RHEL 6.4 rpms with RBD
>>>>> support.
>>>>>
>>>>> And here is the output, with debug mode, from Agent.
>>>>>
>>>>> At the begining I see agent deleting routes for 169.254.x.x network...:
>>>>>
>>>>> Basically, It seems like the
>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>> hviasocket.pl
>>>>> is
>>>>> failing, not being able to connect to socket...
>>>>> And I could not ping VR (169.254.2.99)...
>>>>>
>>>>> LOG:
>>>>>
>>>>> 2015-03-10 16:43:46,976{GMT} DEBUG [cloud.agent.Agent]
>>>>> (agentRequest-Handler-4:) Processing command:
>>>>> com.cloud.agent.api.GetVmStatsCommand
>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>>>>> (agentRequest-Handler-1:) Request:Seq 788-1319632926: { Cmd , MgmtId:
>>>>> 161344838950, via: 788, Ver: v1, Flags: 100111,
>>>>> [{"com.cloud.agent.api.StartCommand":{"vm":{"id":
>>>>> 2191,"name":"r-2191-VM","type":"DomainRouter","cpus":1,"
>>>>> minSpeed":166,"maxSpeed":1000,"minRam":268435456,"maxRam":
>>>>> 268435456,"arch":"x86_64","os":"Debian
>>>>> GNU/Linux 7(64-bit)","bootArgs":" vpccidr=10.0.0.0/8
>>>>> domain=cs2cloud.internal dns1=8.8.8.8 dns2= template=domP
>>>>> name=r-2191-VM
>>>>> eth0ip=169.254.2.99 eth0mask=255.255.0.0 type=vpcrouter
>>>>> disable_rp_filter=true","rebootOnCrash":false,"
>>>>> enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":
>>>>> false,"vncPassword":"e9e24eb0b1ff8ec2","params":{"
>>>>> memoryOvercommitRatio":"1.0","cpuOvercommitRatio":"6.0"},"
>>>>> uuid":"485587f7-4a04-4d77-bc36-2367f2adf77e","disks":[{"
>>>>> data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
>>>>> uuid":"d042e003-5278-461d-8a35-3b1261164f93","
>>>>> volumeType":"ROOT","dataStore":{"org.apache.cloudstack.
>>>>> storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-
>>>>> 88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
>>>>> cephmon.domaina.net","path":"cloudstack","port":6789,"url":"RBD://
>>>>> cephmon.domaina.net/cloudstack/?ROLE=Primary&
>>>>> STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6
>>>>> "}},"name":"ROOT-2191","size":2621440000,"path":"d042e003-
>>>>> 5278-461d-8a35-3b1261164f93","volumeId":2857,"vmName":"r-
>>>>> 2191-VM","accountId":2,"format":"RAW","id":2857,"
>>>>> deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"
>>>>> d042e003-5278-461d-8a35-3b1261164f93","type":"ROOT","_
>>>>> details":{"managed":"false","storagePort":"6789","storageHost":"
>>>>> cephmon.domaina.net
>>>>> ","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"
>>>>> networkRateMbps":-1,"defaultNic":false,"uuid":"
>>>>> d27a7aad-2dcc-45c7-bf30-
>>>>> 3302d011da66","ip":"169.254.2.99","netmask":"255.255.0.0","
>>>>> gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:63","
>>>>> broadcastType":"LinkLocal","type":"Control","
>>>>> isSecurityGroupEnabled":false}]},"hostIp":"10.44.253.180","
>>>>> executeInSequence":false,"wait":0}},{"com.cloud.agent.
>>>>> api.check.CheckSshCommand":{"ip":"169.254.2.99","port":
>>>>> 3922,"interval":6,"retries":100,"name":"r-2191-VM","wait":
>>>>> 0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
>>>>> router.name
>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>>>>> com.cloud.agent.api.PlugNicCommand":{"nic":{"
>>>>> deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid"
>>>>> :"fe848dbe-cf4b-473e-8124-62289f86b144","ip":"5.226.146.
>>>>> 62","netmask":"255.255.255.128","gateway":"5.226.146.1","
>>>>> mac":"06:f0:d6:00:01:49","broadcastType":"Vlan","type":"
>>>>> Public","broadcastUri":"vlan://untagged","isolationUri":"
>>>>> vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"},"
>>>>> instanceName":"r-2191-VM","vmType":"DomainRouter","wait":
>>>>> 0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"
>>>>> ipAddresses":[{"accountId":2,"publicIp":"5.226.146.62","
>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>>>>> networkName":"breth1-500"}],"accessDetails":{"router.guest.
>>>>> ip":"5.226.146.62","zone.network.type":"Advanced","
>>>>> router.name
>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>>>>> com.cloud.agent.api.routing.SetSourceNatCommand":{"
>>>>> ipAddress":{"accountId":2,"publicIp":"5.226.146.62","
>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>>>>> networkName":"breth1-500"},"add":true,"accessDetails":{"
>>>>> zone.network.type":"Advanced","
>>>>> router.name":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{}] }
>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>>>>> (agentRequest-Handler-1:) Processing command:
>>>>> com.cloud.agent.api.StartCommand
>>>>> 2015-03-10 16:43:58,257{GMT} DEBUG [cloud.agent.Agent]
>>>>> (agentRequest-Handler-5:) Processing command:
>>>>> com.cloud.agent.api.GetStorageStatsCommand
>>>>> 2015-03-10 16:44:01,747{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
>>>>> GNU/Linux 7(64-bit)
>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [cloud.agent.Agent]
>>>>> (agentRequest-Handler-2:) Processing command:
>>>>> com.cloud.agent.api.NetworkUsageCommand
>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-2:) Executing:
>>>>> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
>>>>> vpc_netusage.sh 169.254.2.1 -l 46.232.180.55 -g
>>>>> 2015-03-10 16:44:06,439{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-2:) Execution is successful.
>>>>> 2015-03-10 16:44:06,441{GMT} DEBUG [cloud.agent.Agent]
>>>>> (agentRequest-Handler-2:) Seq 788-1319632928: { Ans: , MgmtId:
>>>>> 161344838950, via: 788, Ver: v1, Flags: 10,
>>>>> [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-1873-VM","
>>>>> bytesSent":198379,"bytesReceived":7380903,"result":true,"details":"","
>>>>> wait":0}}]
>>>>> }
>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [kvm.resource.BridgeVifDriver]
>>>>> (agentRequest-Handler-1:) nic=[Nic:Control-169.254.2.99-null]
>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [utils.script.Script]
>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route | grep
>>>>> 169.254.0.0/16
>>>>> 2015-03-10 16:44:11,060{GMT} DEBUG [utils.script.Script]
>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>> 2015-03-10 16:44:11,061{GMT} DEBUG [utils.script.Script]
>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route del
>>>>> 169.254.0.0/16
>>>>> 2015-03-10 16:44:11,064{GMT} DEBUG [utils.script.Script]
>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
>>>>> GNU/Linux 7(64-bit)
>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-1:) starting r-2191-VM: <domain type='kvm'>
>>>>> <name>r-2191-VM</name>
>>>>> <uuid>485587f7-4a04-4d77-bc36-2367f2adf77e</uuid>
>>>>> <description>Debian GNU/Linux 7(64-bit)</description>
>>>>> <clock offset='utc'>
>>>>> </clock>
>>>>> <features>
>>>>> <pae/>
>>>>> <apic/>
>>>>> <acpi/>
>>>>> </features>
>>>>> <devices>
>>>>> <emulator>/usr/libexec/qemu-kvm</emulator>
>>>>> <interface type='bridge'>
>>>>> <source bridge='cloud0'/>
>>>>> <mac address='0e:00:a9:fe:02:63'/>
>>>>> <model type='virtio'/>
>>>>> </interface>
>>>>> <serial type='pty'>
>>>>> <target port='0'/>
>>>>> </serial>
>>>>> <graphics type='vnc' autoport='yes' listen='10.44.253.180'
>>>>> passwd='e9e24eb0b1ff8ec2'/>
>>>>> <disk device='disk' type='network'>
>>>>> <driver name='qemu' type='raw' cache='none' />
>>>>> <source protocol='rbd'
>>>>> name='cloudstack/d042e003-5278-461d-8a35-3b1261164f93'>
>>>>> <host name='cephmon.domain.internal' port='6789'/>
>>>>> </source>
>>>>> <auth username='cloudstack'>
>>>>> <secret type='ceph' uuid='5b93422e-1a66-353d-88a8-2203f79b1dc6'/>
>>>>> </auth>
>>>>> <target dev='vda' bus='virtio'/>
>>>>> </disk>
>>>>> <disk device='cdrom' type='file'>
>>>>> <driver name='qemu' type='raw' cache='none' />
>>>>> <source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
>>>>> <target dev='hdc' bus='ide'/>
>>>>> </disk>
>>>>> <console type='pty'>
>>>>> <target port='0'/>
>>>>> </console>
>>>>> <input type='tablet' bus='usb'/>
>>>>> <channel type='unix'>
>>>>> <source mode='bind' path='/var/lib/libvirt/qemu/r-2191-VM.agent'/>
>>>>> <target type='virtio' name='r-2191-VM.vport'/>
>>>>> <address type='virtio-serial'/>
>>>>> </channel>
>>>>> </devices>
>>>>> <memory>262144</memory>
>>>>> <devices>
>>>>> <memballoon model='none'/>
>>>>> </devices>
>>>>> <vcpu>1</vcpu>
>>>>> <os>
>>>>> <type arch='x86_64' machine='pc'>hvm</type>
>>>>> <boot dev='cdrom'/>
>>>>> <boot dev='hd'/>
>>>>> </os>
>>>>> <cputune>
>>>>> <shares>166</shares>
>>>>> </cputune>
>>>>> <cpu></cpu><on_reboot>restart</on_reboot>
>>>>> <on_poweroff>destroy</on_poweroff>
>>>>> <on_crash>destroy</on_crash>
>>>>> </domain>
>>>>>
>>>>> 2015-03-10 16:44:11,467{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-1:) Executing:
>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>> hviasocket.pl
>>>>> -n
>>>>> r-2191-VM -p %vpccidr=
>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>
>>>>> 2015-03-10 16:44:11,494{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>> 2015-03-10 16:44:11,495{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Trying to connect to 169.254.2.99
>>>>> 2015-03-10 16:44:14,501{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Could not connect to 169.254.2.99
>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>> (Thread-10:)
>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
>>>>> libvirt,
>>>>> continuing
>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>> (Thread-10:)
>>>>> Executing:
>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>>>>> -i
>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>>>>> 2015-03-10 16:44:17,019{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>> (Thread-10:)
>>>>> Execution is successful.
>>>>> 2015-03-10 16:44:17,953{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
>>>>> Sending ping: Seq 788-8: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags:
>>>>> 11,
>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>>>>> 18.domaina.net
>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>>>>> hostType":"Routing","hostId":788,"wait":0}}]
>>>>> }
>>>>> 2015-03-10 16:44:17,964{GMT} DEBUG [cloud.agent.Agent]
>>>>> (Agent-Handler-3:)
>>>>> Received response: Seq 788-8: { Ans: , MgmtId: 161344838950, via: 788,
>>>>> Ver: v1, Flags: 100010,
>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>>>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>>>>> }
>>>>> 2015-03-10 16:44:19,502{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Trying to connect to 169.254.2.99
>>>>> 2015-03-10 16:44:22,509{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Could not connect to 169.254.2.99
>>>>> 2015-03-10 16:44:27,510{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Unable to logon to 169.254.2.99
>>>>> 2015-03-10 16:44:27,512{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-1:) Executing:
>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>> hviasocket.pl
>>>>> -n
>>>>> r-2191-VM -p %vpccidr=
>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>
>>>>> 2015-03-10 16:44:27,540{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>> 2015-03-10 16:44:27,541{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Trying to connect to 169.254.2.99
>>>>> 2015-03-10 16:44:30,548{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Could not connect to 169.254.2.99
>>>>> 2015-03-10 16:44:31,378{GMT} DEBUG [cloud.agent.Agent]
>>>>> (agentRequest-Handler-3:) Processing command:
>>>>> com.cloud.agent.api.GetHostStatsCommand
>>>>> 2015-03-10 16:44:31,379{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c idle=$(top -b -n
>>>>> 1|grep
>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>>>>> 2015-03-10 16:44:31,910{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-3:) Execution is successful.
>>>>> 2015-03-10 16:44:31,911{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c freeMem=$(free|grep
>>>>> cache:|awk '{print $4}');echo $freeMem
>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-3:) Execution is successful.
>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c free|grep Mem:|awk
>>>>> '{print $2}'
>>>>> 2015-03-10 16:44:31,920{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-3:) Execution is successful.
>>>>> 2015-03-10 16:44:35,549{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Trying to connect to 169.254.2.99
>>>>> 2015-03-10 16:44:38,556{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Could not connect to 169.254.2.99
>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Unable to logon to 169.254.2.99
>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-1:) Executing:
>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>> hviasocket.pl
>>>>> -n
>>>>> r-2191-VM -p %vpccidr=
>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>
>>>>> 2015-03-10 16:44:43,587{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>> 2015-03-10 16:44:43,588{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Trying to connect to 169.254.2.99
>>>>> 2015-03-10 16:44:46,594{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Could not connect to 169.254.2.99
>>>>> 2015-03-10 16:44:49,449{GMT} DEBUG [cloud.agent.Agent]
>>>>> (agentRequest-Handler-4:) Processing command:
>>>>> com.cloud.agent.api.GetVmStatsCommand
>>>>> 2015-03-10 16:44:51,595{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Trying to connect to 169.254.2.99
>>>>> 2015-03-10 16:44:54,600{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Could not connect to 169.254.2.99
>>>>> 2015-03-10 16:44:59,601{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Unable to logon to 169.254.2.99
>>>>> 2015-03-10 16:44:59,602{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-1:) Executing:
>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>> hviasocket.pl
>>>>> -n
>>>>> r-2191-VM -p %vpccidr=
>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>
>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Trying to connect to 169.254.2.99
>>>>> 2015-03-10 16:45:02,634{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Could not connect to 169.254.2.99
>>>>> 2015-03-10 16:45:07,635{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Trying to connect to 169.254.2.99
>>>>> 2015-03-10 16:45:10,640{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Could not connect to 169.254.2.99
>>>>> 2015-03-10 16:45:12,066{GMT} DEBUG [cloud.agent.Agent]
>>>>> (agentRequest-Handler-5:) Processing command:
>>>>> com.cloud.agent.api.GetStorageStatsCommand
>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Unable to logon to 169.254.2.99
>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-1:) Executing:
>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>> hviasocket.pl
>>>>> -n
>>>>> r-2191-VM -p %vpccidr=
>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>
>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Trying to connect to 169.254.2.99
>>>>> 2015-03-10 16:45:17,021{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>> (Thread-11:)
>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
>>>>> libvirt,
>>>>> continuing
>>>>> 2015-03-10 16:45:17,022{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>> (Thread-11:)
>>>>> Executing:
>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>>>>> -i
>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>>>>> 2015-03-10 16:45:17,036{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>> (Thread-11:)
>>>>> Execution is successful.
>>>>> 2015-03-10 16:45:17,951{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
>>>>> Sending ping: Seq 788-9: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags:
>>>>> 11,
>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>>>>> 18.domaina.net
>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>>>>> hostType":"Routing","hostId":788,"wait":0}}]
>>>>> }
>>>>> 2015-03-10 16:45:18,002{GMT} DEBUG [cloud.agent.Agent]
>>>>> (Agent-Handler-2:)
>>>>> Received response: Seq 788-9: { Ans: , MgmtId: 161344838950, via: 788,
>>>>> Ver: v1, Flags: 100010,
>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>>>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>>>>> }
>>>>> 2015-03-10 16:45:18,672{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Could not connect to 169.254.2.99
>>>>> 2015-03-10 16:45:23,673{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Trying to connect to 169.254.2.99
>>>>> 2015-03-10 16:45:26,680{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Could not connect to 169.254.2.99
>>>>> 2015-03-10 16:45:31,682{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Unable to logon to 169.254.2.99
>>>>> 2015-03-10 16:45:31,683{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-1:) Executing:
>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>> hviasocket.pl
>>>>> -n
>>>>> r-2191-VM -p %vpccidr=
>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>
>>>>> 2015-03-10 16:45:36,684{GMT} WARN [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (Script-5:) Interrupting script.
>>>>> 2015-03-10 16:45:36,686{GMT} WARN [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-1:) Timed out:
>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>> hviasocket.pl
>>>>> -n
>>>>> r-2191-VM -p %vpccidr=
>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>> . Output is:
>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-1:) passcmd failed:timeout
>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Trying to connect to 169.254.2.99
>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [cloud.agent.Agent]
>>>>> (agentRequest-Handler-2:) Processing command:
>>>>> com.cloud.agent.api.GetHostStatsCommand
>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c idle=$(top -b -n
>>>>> 1|grep
>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>>>>> 2015-03-10 16:45:37,967{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-2:) Execution is successful.
>>>>> 2015-03-10 16:45:37,968{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c freeMem=$(free|grep
>>>>> cache:|awk '{print $4}');echo $freeMem
>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-2:) Execution is successful.
>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c free|grep Mem:|awk
>>>>> '{print $2}'
>>>>> 2015-03-10 16:45:37,977{GMT} DEBUG [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-2:) Execution is successful.
>>>>> 2015-03-10 16:45:39,692{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Could not connect to 169.254.2.99
>>>>> 2015-03-10 16:45:44,693{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Trying to connect to 169.254.2.99
>>>>> 2015-03-10 16:45:47,700{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Could not connect to 169.254.2.99
>>>>> 2015-03-10 16:45:51,807{GMT} DEBUG [cloud.agent.Agent]
>>>>> (agentRequest-Handler-3:) Processing command:
>>>>> com.cloud.agent.api.GetVmStatsCommand
>>>>> 2015-03-10 16:45:52,701{GMT} DEBUG
>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>> (agentRequest-Handler-1:)
>>>>> Unable to logon to 169.254.2.99
>>>>>
>>>>> On 10 March 2015 at 17:50, ilya musayev <il...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> What kernel are you running? Please include Libvirt and Qemu version.
>>>>>
>>>>>>
>>>>>> On 3/10/15 8:21 AM, Andrija Panic wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>> we are having issues from time to time with starting VR - while
>>>>>>> creating
>>>>>>> VPC.
>>>>>>>
>>>>>>> Here is the few lines from MGMT log:
>>>>>>>
>>>>>>> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
>>>>>>> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start vpc [VPC
>>>>>>> [621-VPC-delete] due to
>>>>>>> com.cloud.exception.AgentUnavailableException: Resource [Host:788]
>>>>>>> is
>>>>>>> unreachable: Host 788: Unable to start instance due to Unable to
>>>>>>> start
>>>>>>> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not
>>>>>>> retrying
>>>>>>> at
>>>>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
>>>>>>> VirtualMachineManagerImpl.java:1072)
>>>>>>>
>>>>>>>
>>>>>>> The interesting part is on the Agent itself:
>>>>>>>
>>>>>>>
>>>>>>> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-2:) Timed out:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>> hviasocket.pl
>>>>>>> -n
>>>>>>> r-2189-VM -p %vpccidr=
>>>>>>> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>>> template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=
>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>> . Output is:
>>>>>>>
>>>>>>>
>>>>>>> Interesting stuff is - it's seems somewhat random, 3-4 out of 5
>>>>>>> works.
>>>>>>>
>>>>>>> Any suggestion are really appreciated, since this is live system :(
>>>>>>> ACS 4.3.0, KVM, CEPH
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>>>
>>>
>>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: [HELP] VR sometimes fail to start
Posted by Andrija Panic <an...@gmail.com>.
Hi Marcus, will do - I'm just unconfortable changing live things, but that
is my problem. Will let you know of the output.
Thanks
On 12 March 2015 at 01:34, ilya musayev <il...@gmail.com>
wrote:
> Andrija,
>
> Have you tried the solution Loic mentioned? Its seems like an easy one to
> try..
>
> Thanks
> ilya
>
>
> On 3/11/15 3:54 AM, Andrija Panic wrote:
>
>> Oh well, I will then update to 4.3.2 anyway, and hope to see the problem
>> go
>> away...
>>
>> THanks guys
>>
>> On 10 March 2015 at 22:45, Loic Lambiel <lo...@exoscale.ch> wrote:
>>
>> Hi,
>>>
>>> Maybe the same issue we faced here: https://github.com/exoscale/
>>> cloudstack/commit/df3e7533a78c3594cdd39b8b18e6f4a3abcb8cfd
>>>
>>> The patch has not been yet submitted upstream. I'll do it once it is
>>> running on our production.
>>>
>>> Anyway, you may easily test it by performing the small change directly
>>> into the script.
>>>
>>> Hope that helps.
>>>
>>> Regards,
>>>
>>> Loic Lambiel
>>>
>>>
>>> On 10/03/15 19:40, Andrija Panic wrote:
>>>
>>> Hi guys,
>>>>
>>>> Kernel is Elrepo 3.10.67 (CentOS 6.6)
>>>> Libvirt manually compiled 1.2.3 with RBD support.
>>>> Qemu 0.12.1.2 - Inktank patched original RHEL 6.4 rpms with RBD support.
>>>>
>>>> And here is the output, with debug mode, from Agent.
>>>>
>>>> At the begining I see agent deleting routes for 169.254.x.x network...:
>>>>
>>>> Basically, It seems like the
>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>> hviasocket.pl
>>>> is
>>>> failing, not being able to connect to socket...
>>>> And I could not ping VR (169.254.2.99)...
>>>>
>>>> LOG:
>>>>
>>>> 2015-03-10 16:43:46,976{GMT} DEBUG [cloud.agent.Agent]
>>>> (agentRequest-Handler-4:) Processing command:
>>>> com.cloud.agent.api.GetVmStatsCommand
>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>>>> (agentRequest-Handler-1:) Request:Seq 788-1319632926: { Cmd , MgmtId:
>>>> 161344838950, via: 788, Ver: v1, Flags: 100111,
>>>> [{"com.cloud.agent.api.StartCommand":{"vm":{"id":
>>>> 2191,"name":"r-2191-VM","type":"DomainRouter","cpus":1,"
>>>> minSpeed":166,"maxSpeed":1000,"minRam":268435456,"maxRam":
>>>> 268435456,"arch":"x86_64","os":"Debian
>>>> GNU/Linux 7(64-bit)","bootArgs":" vpccidr=10.0.0.0/8
>>>> domain=cs2cloud.internal dns1=8.8.8.8 dns2= template=domP name=r-2191-VM
>>>> eth0ip=169.254.2.99 eth0mask=255.255.0.0 type=vpcrouter
>>>> disable_rp_filter=true","rebootOnCrash":false,"
>>>> enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":
>>>> false,"vncPassword":"e9e24eb0b1ff8ec2","params":{"
>>>> memoryOvercommitRatio":"1.0","cpuOvercommitRatio":"6.0"},"
>>>> uuid":"485587f7-4a04-4d77-bc36-2367f2adf77e","disks":[{"
>>>> data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
>>>> uuid":"d042e003-5278-461d-8a35-3b1261164f93","
>>>> volumeType":"ROOT","dataStore":{"org.apache.cloudstack.
>>>> storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-
>>>> 88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
>>>> cephmon.domaina.net","path":"cloudstack","port":6789,"url":"RBD://
>>>> cephmon.domaina.net/cloudstack/?ROLE=Primary&
>>>> STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6
>>>> "}},"name":"ROOT-2191","size":2621440000,"path":"d042e003-
>>>> 5278-461d-8a35-3b1261164f93","volumeId":2857,"vmName":"r-
>>>> 2191-VM","accountId":2,"format":"RAW","id":2857,"
>>>> deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"
>>>> d042e003-5278-461d-8a35-3b1261164f93","type":"ROOT","_
>>>> details":{"managed":"false","storagePort":"6789","storageHost":"
>>>> cephmon.domaina.net
>>>> ","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"
>>>> networkRateMbps":-1,"defaultNic":false,"uuid":"d27a7aad-2dcc-45c7-bf30-
>>>> 3302d011da66","ip":"169.254.2.99","netmask":"255.255.0.0","
>>>> gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:63","
>>>> broadcastType":"LinkLocal","type":"Control","
>>>> isSecurityGroupEnabled":false}]},"hostIp":"10.44.253.180","
>>>> executeInSequence":false,"wait":0}},{"com.cloud.agent.
>>>> api.check.CheckSshCommand":{"ip":"169.254.2.99","port":
>>>> 3922,"interval":6,"retries":100,"name":"r-2191-VM","wait":
>>>> 0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
>>>> router.name
>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>>>> com.cloud.agent.api.PlugNicCommand":{"nic":{"
>>>> deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid"
>>>> :"fe848dbe-cf4b-473e-8124-62289f86b144","ip":"5.226.146.
>>>> 62","netmask":"255.255.255.128","gateway":"5.226.146.1","
>>>> mac":"06:f0:d6:00:01:49","broadcastType":"Vlan","type":"
>>>> Public","broadcastUri":"vlan://untagged","isolationUri":"
>>>> vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"},"
>>>> instanceName":"r-2191-VM","vmType":"DomainRouter","wait":
>>>> 0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"
>>>> ipAddresses":[{"accountId":2,"publicIp":"5.226.146.62","
>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>>>> networkName":"breth1-500"}],"accessDetails":{"router.guest.
>>>> ip":"5.226.146.62","zone.network.type":"Advanced","
>>>> router.name
>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>>>> com.cloud.agent.api.routing.SetSourceNatCommand":{"
>>>> ipAddress":{"accountId":2,"publicIp":"5.226.146.62","
>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>>>> networkName":"breth1-500"},"add":true,"accessDetails":{"
>>>> zone.network.type":"Advanced","
>>>> router.name":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{}] }
>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>>>> (agentRequest-Handler-1:) Processing command:
>>>> com.cloud.agent.api.StartCommand
>>>> 2015-03-10 16:43:58,257{GMT} DEBUG [cloud.agent.Agent]
>>>> (agentRequest-Handler-5:) Processing command:
>>>> com.cloud.agent.api.GetStorageStatsCommand
>>>> 2015-03-10 16:44:01,747{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
>>>> GNU/Linux 7(64-bit)
>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [cloud.agent.Agent]
>>>> (agentRequest-Handler-2:) Processing command:
>>>> com.cloud.agent.api.NetworkUsageCommand
>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-2:) Executing:
>>>> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
>>>> vpc_netusage.sh 169.254.2.1 -l 46.232.180.55 -g
>>>> 2015-03-10 16:44:06,439{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-2:) Execution is successful.
>>>> 2015-03-10 16:44:06,441{GMT} DEBUG [cloud.agent.Agent]
>>>> (agentRequest-Handler-2:) Seq 788-1319632928: { Ans: , MgmtId:
>>>> 161344838950, via: 788, Ver: v1, Flags: 10,
>>>> [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-1873-VM","
>>>> bytesSent":198379,"bytesReceived":7380903,"result":true,"details":"","
>>>> wait":0}}]
>>>> }
>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [kvm.resource.BridgeVifDriver]
>>>> (agentRequest-Handler-1:) nic=[Nic:Control-169.254.2.99-null]
>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [utils.script.Script]
>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route | grep
>>>> 169.254.0.0/16
>>>> 2015-03-10 16:44:11,060{GMT} DEBUG [utils.script.Script]
>>>> (agentRequest-Handler-1:) Execution is successful.
>>>> 2015-03-10 16:44:11,061{GMT} DEBUG [utils.script.Script]
>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route del
>>>> 169.254.0.0/16
>>>> 2015-03-10 16:44:11,064{GMT} DEBUG [utils.script.Script]
>>>> (agentRequest-Handler-1:) Execution is successful.
>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
>>>> GNU/Linux 7(64-bit)
>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-1:) starting r-2191-VM: <domain type='kvm'>
>>>> <name>r-2191-VM</name>
>>>> <uuid>485587f7-4a04-4d77-bc36-2367f2adf77e</uuid>
>>>> <description>Debian GNU/Linux 7(64-bit)</description>
>>>> <clock offset='utc'>
>>>> </clock>
>>>> <features>
>>>> <pae/>
>>>> <apic/>
>>>> <acpi/>
>>>> </features>
>>>> <devices>
>>>> <emulator>/usr/libexec/qemu-kvm</emulator>
>>>> <interface type='bridge'>
>>>> <source bridge='cloud0'/>
>>>> <mac address='0e:00:a9:fe:02:63'/>
>>>> <model type='virtio'/>
>>>> </interface>
>>>> <serial type='pty'>
>>>> <target port='0'/>
>>>> </serial>
>>>> <graphics type='vnc' autoport='yes' listen='10.44.253.180'
>>>> passwd='e9e24eb0b1ff8ec2'/>
>>>> <disk device='disk' type='network'>
>>>> <driver name='qemu' type='raw' cache='none' />
>>>> <source protocol='rbd'
>>>> name='cloudstack/d042e003-5278-461d-8a35-3b1261164f93'>
>>>> <host name='cephmon.domain.internal' port='6789'/>
>>>> </source>
>>>> <auth username='cloudstack'>
>>>> <secret type='ceph' uuid='5b93422e-1a66-353d-88a8-2203f79b1dc6'/>
>>>> </auth>
>>>> <target dev='vda' bus='virtio'/>
>>>> </disk>
>>>> <disk device='cdrom' type='file'>
>>>> <driver name='qemu' type='raw' cache='none' />
>>>> <source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
>>>> <target dev='hdc' bus='ide'/>
>>>> </disk>
>>>> <console type='pty'>
>>>> <target port='0'/>
>>>> </console>
>>>> <input type='tablet' bus='usb'/>
>>>> <channel type='unix'>
>>>> <source mode='bind' path='/var/lib/libvirt/qemu/r-2191-VM.agent'/>
>>>> <target type='virtio' name='r-2191-VM.vport'/>
>>>> <address type='virtio-serial'/>
>>>> </channel>
>>>> </devices>
>>>> <memory>262144</memory>
>>>> <devices>
>>>> <memballoon model='none'/>
>>>> </devices>
>>>> <vcpu>1</vcpu>
>>>> <os>
>>>> <type arch='x86_64' machine='pc'>hvm</type>
>>>> <boot dev='cdrom'/>
>>>> <boot dev='hd'/>
>>>> </os>
>>>> <cputune>
>>>> <shares>166</shares>
>>>> </cputune>
>>>> <cpu></cpu><on_reboot>restart</on_reboot>
>>>> <on_poweroff>destroy</on_poweroff>
>>>> <on_crash>destroy</on_crash>
>>>> </domain>
>>>>
>>>> 2015-03-10 16:44:11,467{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-1:) Executing:
>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>> hviasocket.pl
>>>> -n
>>>> r-2191-VM -p %vpccidr=
>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>
>>>> 2015-03-10 16:44:11,494{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-1:) Execution is successful.
>>>> 2015-03-10 16:44:11,495{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Trying to connect to 169.254.2.99
>>>> 2015-03-10 16:44:14,501{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Could not connect to 169.254.2.99
>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>> (Thread-10:)
>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in libvirt,
>>>> continuing
>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>> (Thread-10:)
>>>> Executing:
>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>>>> -i
>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>>>> 2015-03-10 16:44:17,019{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>> (Thread-10:)
>>>> Execution is successful.
>>>> 2015-03-10 16:44:17,953{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
>>>> Sending ping: Seq 788-8: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags:
>>>> 11,
>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>>>> 18.domaina.net
>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>>>> hostType":"Routing","hostId":788,"wait":0}}]
>>>> }
>>>> 2015-03-10 16:44:17,964{GMT} DEBUG [cloud.agent.Agent]
>>>> (Agent-Handler-3:)
>>>> Received response: Seq 788-8: { Ans: , MgmtId: 161344838950, via: 788,
>>>> Ver: v1, Flags: 100010,
>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>>>> }
>>>> 2015-03-10 16:44:19,502{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Trying to connect to 169.254.2.99
>>>> 2015-03-10 16:44:22,509{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Could not connect to 169.254.2.99
>>>> 2015-03-10 16:44:27,510{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Unable to logon to 169.254.2.99
>>>> 2015-03-10 16:44:27,512{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-1:) Executing:
>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>> hviasocket.pl
>>>> -n
>>>> r-2191-VM -p %vpccidr=
>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>
>>>> 2015-03-10 16:44:27,540{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-1:) Execution is successful.
>>>> 2015-03-10 16:44:27,541{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Trying to connect to 169.254.2.99
>>>> 2015-03-10 16:44:30,548{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Could not connect to 169.254.2.99
>>>> 2015-03-10 16:44:31,378{GMT} DEBUG [cloud.agent.Agent]
>>>> (agentRequest-Handler-3:) Processing command:
>>>> com.cloud.agent.api.GetHostStatsCommand
>>>> 2015-03-10 16:44:31,379{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c idle=$(top -b -n
>>>> 1|grep
>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>>>> 2015-03-10 16:44:31,910{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-3:) Execution is successful.
>>>> 2015-03-10 16:44:31,911{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c freeMem=$(free|grep
>>>> cache:|awk '{print $4}');echo $freeMem
>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-3:) Execution is successful.
>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c free|grep Mem:|awk
>>>> '{print $2}'
>>>> 2015-03-10 16:44:31,920{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-3:) Execution is successful.
>>>> 2015-03-10 16:44:35,549{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Trying to connect to 169.254.2.99
>>>> 2015-03-10 16:44:38,556{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Could not connect to 169.254.2.99
>>>> 2015-03-10 16:44:43,557{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Unable to logon to 169.254.2.99
>>>> 2015-03-10 16:44:43,557{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-1:) Executing:
>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>> hviasocket.pl
>>>> -n
>>>> r-2191-VM -p %vpccidr=
>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>
>>>> 2015-03-10 16:44:43,587{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-1:) Execution is successful.
>>>> 2015-03-10 16:44:43,588{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Trying to connect to 169.254.2.99
>>>> 2015-03-10 16:44:46,594{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Could not connect to 169.254.2.99
>>>> 2015-03-10 16:44:49,449{GMT} DEBUG [cloud.agent.Agent]
>>>> (agentRequest-Handler-4:) Processing command:
>>>> com.cloud.agent.api.GetVmStatsCommand
>>>> 2015-03-10 16:44:51,595{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Trying to connect to 169.254.2.99
>>>> 2015-03-10 16:44:54,600{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Could not connect to 169.254.2.99
>>>> 2015-03-10 16:44:59,601{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Unable to logon to 169.254.2.99
>>>> 2015-03-10 16:44:59,602{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-1:) Executing:
>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>> hviasocket.pl
>>>> -n
>>>> r-2191-VM -p %vpccidr=
>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>
>>>> 2015-03-10 16:44:59,628{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-1:) Execution is successful.
>>>> 2015-03-10 16:44:59,628{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Trying to connect to 169.254.2.99
>>>> 2015-03-10 16:45:02,634{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Could not connect to 169.254.2.99
>>>> 2015-03-10 16:45:07,635{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Trying to connect to 169.254.2.99
>>>> 2015-03-10 16:45:10,640{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Could not connect to 169.254.2.99
>>>> 2015-03-10 16:45:12,066{GMT} DEBUG [cloud.agent.Agent]
>>>> (agentRequest-Handler-5:) Processing command:
>>>> com.cloud.agent.api.GetStorageStatsCommand
>>>> 2015-03-10 16:45:15,641{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Unable to logon to 169.254.2.99
>>>> 2015-03-10 16:45:15,641{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-1:) Executing:
>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>> hviasocket.pl
>>>> -n
>>>> r-2191-VM -p %vpccidr=
>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>
>>>> 2015-03-10 16:45:15,667{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-1:) Execution is successful.
>>>> 2015-03-10 16:45:15,667{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Trying to connect to 169.254.2.99
>>>> 2015-03-10 16:45:17,021{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>> (Thread-11:)
>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in libvirt,
>>>> continuing
>>>> 2015-03-10 16:45:17,022{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>> (Thread-11:)
>>>> Executing:
>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>>>> -i
>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>>>> 2015-03-10 16:45:17,036{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>> (Thread-11:)
>>>> Execution is successful.
>>>> 2015-03-10 16:45:17,951{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
>>>> Sending ping: Seq 788-9: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags:
>>>> 11,
>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>>>> 18.domaina.net
>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>>>> hostType":"Routing","hostId":788,"wait":0}}]
>>>> }
>>>> 2015-03-10 16:45:18,002{GMT} DEBUG [cloud.agent.Agent]
>>>> (Agent-Handler-2:)
>>>> Received response: Seq 788-9: { Ans: , MgmtId: 161344838950, via: 788,
>>>> Ver: v1, Flags: 100010,
>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>>>> }
>>>> 2015-03-10 16:45:18,672{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Could not connect to 169.254.2.99
>>>> 2015-03-10 16:45:23,673{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Trying to connect to 169.254.2.99
>>>> 2015-03-10 16:45:26,680{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Could not connect to 169.254.2.99
>>>> 2015-03-10 16:45:31,682{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Unable to logon to 169.254.2.99
>>>> 2015-03-10 16:45:31,683{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-1:) Executing:
>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>> hviasocket.pl
>>>> -n
>>>> r-2191-VM -p %vpccidr=
>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>
>>>> 2015-03-10 16:45:36,684{GMT} WARN [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (Script-5:) Interrupting script.
>>>> 2015-03-10 16:45:36,686{GMT} WARN [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-1:) Timed out:
>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>> hviasocket.pl
>>>> -n
>>>> r-2191-VM -p %vpccidr=
>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>> . Output is:
>>>> 2015-03-10 16:45:36,686{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-1:) passcmd failed:timeout
>>>> 2015-03-10 16:45:36,686{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Trying to connect to 169.254.2.99
>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [cloud.agent.Agent]
>>>> (agentRequest-Handler-2:) Processing command:
>>>> com.cloud.agent.api.GetHostStatsCommand
>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c idle=$(top -b -n
>>>> 1|grep
>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>>>> 2015-03-10 16:45:37,967{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-2:) Execution is successful.
>>>> 2015-03-10 16:45:37,968{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c freeMem=$(free|grep
>>>> cache:|awk '{print $4}');echo $freeMem
>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-2:) Execution is successful.
>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c free|grep Mem:|awk
>>>> '{print $2}'
>>>> 2015-03-10 16:45:37,977{GMT} DEBUG [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-2:) Execution is successful.
>>>> 2015-03-10 16:45:39,692{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Could not connect to 169.254.2.99
>>>> 2015-03-10 16:45:44,693{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Trying to connect to 169.254.2.99
>>>> 2015-03-10 16:45:47,700{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Could not connect to 169.254.2.99
>>>> 2015-03-10 16:45:51,807{GMT} DEBUG [cloud.agent.Agent]
>>>> (agentRequest-Handler-3:) Processing command:
>>>> com.cloud.agent.api.GetVmStatsCommand
>>>> 2015-03-10 16:45:52,701{GMT} DEBUG
>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>> (agentRequest-Handler-1:)
>>>> Unable to logon to 169.254.2.99
>>>>
>>>> On 10 March 2015 at 17:50, ilya musayev <il...@gmail.com>
>>>> wrote:
>>>>
>>>> What kernel are you running? Please include Libvirt and Qemu version.
>>>>
>>>>>
>>>>> On 3/10/15 8:21 AM, Andrija Panic wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>>> we are having issues from time to time with starting VR - while
>>>>>> creating
>>>>>> VPC.
>>>>>>
>>>>>> Here is the few lines from MGMT log:
>>>>>>
>>>>>> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
>>>>>> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start vpc [VPC
>>>>>> [621-VPC-delete] due to
>>>>>> com.cloud.exception.AgentUnavailableException: Resource [Host:788] is
>>>>>> unreachable: Host 788: Unable to start instance due to Unable to start
>>>>>> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not retrying
>>>>>> at
>>>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
>>>>>> VirtualMachineManagerImpl.java:1072)
>>>>>>
>>>>>>
>>>>>> The interesting part is on the Agent itself:
>>>>>>
>>>>>>
>>>>>> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.
>>>>>> LibvirtComputingResource]
>>>>>> (agentRequest-Handler-2:) Timed out:
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>> hviasocket.pl
>>>>>> -n
>>>>>> r-2189-VM -p %vpccidr=
>>>>>> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>> template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=
>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>> . Output is:
>>>>>>
>>>>>>
>>>>>> Interesting stuff is - it's seems somewhat random, 3-4 out of 5
>>>>>> works.
>>>>>>
>>>>>> Any suggestion are really appreciated, since this is live system :(
>>>>>> ACS 4.3.0, KVM, CEPH
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>
>>>>>>
>>>
>>>
>>
>
--
Andrija Panić
Re: [HELP] VR sometimes fail to start
Posted by ilya musayev <il...@gmail.com>.
Andrija,
Have you tried the solution Loic mentioned? Its seems like an easy one
to try..
Thanks
ilya
On 3/11/15 3:54 AM, Andrija Panic wrote:
> Oh well, I will then update to 4.3.2 anyway, and hope to see the problem go
> away...
>
> THanks guys
>
> On 10 March 2015 at 22:45, Loic Lambiel <lo...@exoscale.ch> wrote:
>
>> Hi,
>>
>> Maybe the same issue we faced here: https://github.com/exoscale/
>> cloudstack/commit/df3e7533a78c3594cdd39b8b18e6f4a3abcb8cfd
>>
>> The patch has not been yet submitted upstream. I'll do it once it is
>> running on our production.
>>
>> Anyway, you may easily test it by performing the small change directly
>> into the script.
>>
>> Hope that helps.
>>
>> Regards,
>>
>> Loic Lambiel
>>
>>
>> On 10/03/15 19:40, Andrija Panic wrote:
>>
>>> Hi guys,
>>>
>>> Kernel is Elrepo 3.10.67 (CentOS 6.6)
>>> Libvirt manually compiled 1.2.3 with RBD support.
>>> Qemu 0.12.1.2 - Inktank patched original RHEL 6.4 rpms with RBD support.
>>>
>>> And here is the output, with debug mode, from Agent.
>>>
>>> At the begining I see agent deleting routes for 169.254.x.x network...:
>>>
>>> Basically, It seems like the
>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>>> is
>>> failing, not being able to connect to socket...
>>> And I could not ping VR (169.254.2.99)...
>>>
>>> LOG:
>>>
>>> 2015-03-10 16:43:46,976{GMT} DEBUG [cloud.agent.Agent]
>>> (agentRequest-Handler-4:) Processing command:
>>> com.cloud.agent.api.GetVmStatsCommand
>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>>> (agentRequest-Handler-1:) Request:Seq 788-1319632926: { Cmd , MgmtId:
>>> 161344838950, via: 788, Ver: v1, Flags: 100111,
>>> [{"com.cloud.agent.api.StartCommand":{"vm":{"id":
>>> 2191,"name":"r-2191-VM","type":"DomainRouter","cpus":1,"
>>> minSpeed":166,"maxSpeed":1000,"minRam":268435456,"maxRam":
>>> 268435456,"arch":"x86_64","os":"Debian
>>> GNU/Linux 7(64-bit)","bootArgs":" vpccidr=10.0.0.0/8
>>> domain=cs2cloud.internal dns1=8.8.8.8 dns2= template=domP name=r-2191-VM
>>> eth0ip=169.254.2.99 eth0mask=255.255.0.0 type=vpcrouter
>>> disable_rp_filter=true","rebootOnCrash":false,"
>>> enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":
>>> false,"vncPassword":"e9e24eb0b1ff8ec2","params":{"
>>> memoryOvercommitRatio":"1.0","cpuOvercommitRatio":"6.0"},"
>>> uuid":"485587f7-4a04-4d77-bc36-2367f2adf77e","disks":[{"
>>> data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
>>> uuid":"d042e003-5278-461d-8a35-3b1261164f93","
>>> volumeType":"ROOT","dataStore":{"org.apache.cloudstack.
>>> storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-
>>> 88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
>>> cephmon.domaina.net","path":"cloudstack","port":6789,"url":"RBD://
>>> cephmon.domaina.net/cloudstack/?ROLE=Primary&
>>> STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6
>>> "}},"name":"ROOT-2191","size":2621440000,"path":"d042e003-
>>> 5278-461d-8a35-3b1261164f93","volumeId":2857,"vmName":"r-
>>> 2191-VM","accountId":2,"format":"RAW","id":2857,"
>>> deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"
>>> d042e003-5278-461d-8a35-3b1261164f93","type":"ROOT","_
>>> details":{"managed":"false","storagePort":"6789","storageHost":"
>>> cephmon.domaina.net
>>> ","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"
>>> networkRateMbps":-1,"defaultNic":false,"uuid":"d27a7aad-2dcc-45c7-bf30-
>>> 3302d011da66","ip":"169.254.2.99","netmask":"255.255.0.0","
>>> gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:63","
>>> broadcastType":"LinkLocal","type":"Control","
>>> isSecurityGroupEnabled":false}]},"hostIp":"10.44.253.180","
>>> executeInSequence":false,"wait":0}},{"com.cloud.agent.
>>> api.check.CheckSshCommand":{"ip":"169.254.2.99","port":
>>> 3922,"interval":6,"retries":100,"name":"r-2191-VM","wait":
>>> 0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
>>> router.name
>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>>> com.cloud.agent.api.PlugNicCommand":{"nic":{"
>>> deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid"
>>> :"fe848dbe-cf4b-473e-8124-62289f86b144","ip":"5.226.146.
>>> 62","netmask":"255.255.255.128","gateway":"5.226.146.1","
>>> mac":"06:f0:d6:00:01:49","broadcastType":"Vlan","type":"
>>> Public","broadcastUri":"vlan://untagged","isolationUri":"
>>> vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"},"
>>> instanceName":"r-2191-VM","vmType":"DomainRouter","wait":
>>> 0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"
>>> ipAddresses":[{"accountId":2,"publicIp":"5.226.146.62","
>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>>> networkName":"breth1-500"}],"accessDetails":{"router.guest.
>>> ip":"5.226.146.62","zone.network.type":"Advanced","
>>> router.name
>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>>> com.cloud.agent.api.routing.SetSourceNatCommand":{"
>>> ipAddress":{"accountId":2,"publicIp":"5.226.146.62","
>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>>> networkName":"breth1-500"},"add":true,"accessDetails":{"
>>> zone.network.type":"Advanced","
>>> router.name":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{}] }
>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>>> (agentRequest-Handler-1:) Processing command:
>>> com.cloud.agent.api.StartCommand
>>> 2015-03-10 16:43:58,257{GMT} DEBUG [cloud.agent.Agent]
>>> (agentRequest-Handler-5:) Processing command:
>>> com.cloud.agent.api.GetStorageStatsCommand
>>> 2015-03-10 16:44:01,747{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
>>> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
>>> GNU/Linux 7(64-bit)
>>> 2015-03-10 16:44:06,309{GMT} DEBUG [cloud.agent.Agent]
>>> (agentRequest-Handler-2:) Processing command:
>>> com.cloud.agent.api.NetworkUsageCommand
>>> 2015-03-10 16:44:06,309{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-2:) Executing:
>>> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
>>> vpc_netusage.sh 169.254.2.1 -l 46.232.180.55 -g
>>> 2015-03-10 16:44:06,439{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-2:) Execution is successful.
>>> 2015-03-10 16:44:06,441{GMT} DEBUG [cloud.agent.Agent]
>>> (agentRequest-Handler-2:) Seq 788-1319632928: { Ans: , MgmtId:
>>> 161344838950, via: 788, Ver: v1, Flags: 10,
>>> [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-1873-VM","
>>> bytesSent":198379,"bytesReceived":7380903,"result":true,"details":"","
>>> wait":0}}]
>>> }
>>> 2015-03-10 16:44:11,055{GMT} DEBUG [kvm.resource.BridgeVifDriver]
>>> (agentRequest-Handler-1:) nic=[Nic:Control-169.254.2.99-null]
>>> 2015-03-10 16:44:11,055{GMT} DEBUG [utils.script.Script]
>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route | grep
>>> 169.254.0.0/16
>>> 2015-03-10 16:44:11,060{GMT} DEBUG [utils.script.Script]
>>> (agentRequest-Handler-1:) Execution is successful.
>>> 2015-03-10 16:44:11,061{GMT} DEBUG [utils.script.Script]
>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route del
>>> 169.254.0.0/16
>>> 2015-03-10 16:44:11,064{GMT} DEBUG [utils.script.Script]
>>> (agentRequest-Handler-1:) Execution is successful.
>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
>>> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
>>> GNU/Linux 7(64-bit)
>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-1:) starting r-2191-VM: <domain type='kvm'>
>>> <name>r-2191-VM</name>
>>> <uuid>485587f7-4a04-4d77-bc36-2367f2adf77e</uuid>
>>> <description>Debian GNU/Linux 7(64-bit)</description>
>>> <clock offset='utc'>
>>> </clock>
>>> <features>
>>> <pae/>
>>> <apic/>
>>> <acpi/>
>>> </features>
>>> <devices>
>>> <emulator>/usr/libexec/qemu-kvm</emulator>
>>> <interface type='bridge'>
>>> <source bridge='cloud0'/>
>>> <mac address='0e:00:a9:fe:02:63'/>
>>> <model type='virtio'/>
>>> </interface>
>>> <serial type='pty'>
>>> <target port='0'/>
>>> </serial>
>>> <graphics type='vnc' autoport='yes' listen='10.44.253.180'
>>> passwd='e9e24eb0b1ff8ec2'/>
>>> <disk device='disk' type='network'>
>>> <driver name='qemu' type='raw' cache='none' />
>>> <source protocol='rbd'
>>> name='cloudstack/d042e003-5278-461d-8a35-3b1261164f93'>
>>> <host name='cephmon.domain.internal' port='6789'/>
>>> </source>
>>> <auth username='cloudstack'>
>>> <secret type='ceph' uuid='5b93422e-1a66-353d-88a8-2203f79b1dc6'/>
>>> </auth>
>>> <target dev='vda' bus='virtio'/>
>>> </disk>
>>> <disk device='cdrom' type='file'>
>>> <driver name='qemu' type='raw' cache='none' />
>>> <source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
>>> <target dev='hdc' bus='ide'/>
>>> </disk>
>>> <console type='pty'>
>>> <target port='0'/>
>>> </console>
>>> <input type='tablet' bus='usb'/>
>>> <channel type='unix'>
>>> <source mode='bind' path='/var/lib/libvirt/qemu/r-2191-VM.agent'/>
>>> <target type='virtio' name='r-2191-VM.vport'/>
>>> <address type='virtio-serial'/>
>>> </channel>
>>> </devices>
>>> <memory>262144</memory>
>>> <devices>
>>> <memballoon model='none'/>
>>> </devices>
>>> <vcpu>1</vcpu>
>>> <os>
>>> <type arch='x86_64' machine='pc'>hvm</type>
>>> <boot dev='cdrom'/>
>>> <boot dev='hd'/>
>>> </os>
>>> <cputune>
>>> <shares>166</shares>
>>> </cputune>
>>> <cpu></cpu><on_reboot>restart</on_reboot>
>>> <on_poweroff>destroy</on_poweroff>
>>> <on_crash>destroy</on_crash>
>>> </domain>
>>>
>>> 2015-03-10 16:44:11,467{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-1:) Executing:
>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>>> -n
>>> r-2191-VM -p %vpccidr=
>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>
>>> 2015-03-10 16:44:11,494{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-1:) Execution is successful.
>>> 2015-03-10 16:44:11,495{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Trying to connect to 169.254.2.99
>>> 2015-03-10 16:44:14,501{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Could not connect to 169.254.2.99
>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>> (Thread-10:)
>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in libvirt,
>>> continuing
>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>> (Thread-10:)
>>> Executing:
>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>>> 2015-03-10 16:44:17,019{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>> (Thread-10:)
>>> Execution is successful.
>>> 2015-03-10 16:44:17,953{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
>>> Sending ping: Seq 788-8: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags:
>>> 11,
>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>>> 18.domaina.net
>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>>> hostType":"Routing","hostId":788,"wait":0}}]
>>> }
>>> 2015-03-10 16:44:17,964{GMT} DEBUG [cloud.agent.Agent] (Agent-Handler-3:)
>>> Received response: Seq 788-8: { Ans: , MgmtId: 161344838950, via: 788,
>>> Ver: v1, Flags: 100010,
>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>>> }
>>> 2015-03-10 16:44:19,502{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Trying to connect to 169.254.2.99
>>> 2015-03-10 16:44:22,509{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Could not connect to 169.254.2.99
>>> 2015-03-10 16:44:27,510{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Unable to logon to 169.254.2.99
>>> 2015-03-10 16:44:27,512{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-1:) Executing:
>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>>> -n
>>> r-2191-VM -p %vpccidr=
>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>
>>> 2015-03-10 16:44:27,540{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-1:) Execution is successful.
>>> 2015-03-10 16:44:27,541{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Trying to connect to 169.254.2.99
>>> 2015-03-10 16:44:30,548{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Could not connect to 169.254.2.99
>>> 2015-03-10 16:44:31,378{GMT} DEBUG [cloud.agent.Agent]
>>> (agentRequest-Handler-3:) Processing command:
>>> com.cloud.agent.api.GetHostStatsCommand
>>> 2015-03-10 16:44:31,379{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-3:) Executing: /bin/bash -c idle=$(top -b -n 1|grep
>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>>> 2015-03-10 16:44:31,910{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-3:) Execution is successful.
>>> 2015-03-10 16:44:31,911{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-3:) Executing: /bin/bash -c freeMem=$(free|grep
>>> cache:|awk '{print $4}');echo $freeMem
>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-3:) Execution is successful.
>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-3:) Executing: /bin/bash -c free|grep Mem:|awk
>>> '{print $2}'
>>> 2015-03-10 16:44:31,920{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-3:) Execution is successful.
>>> 2015-03-10 16:44:35,549{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Trying to connect to 169.254.2.99
>>> 2015-03-10 16:44:38,556{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Could not connect to 169.254.2.99
>>> 2015-03-10 16:44:43,557{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Unable to logon to 169.254.2.99
>>> 2015-03-10 16:44:43,557{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-1:) Executing:
>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>>> -n
>>> r-2191-VM -p %vpccidr=
>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>
>>> 2015-03-10 16:44:43,587{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-1:) Execution is successful.
>>> 2015-03-10 16:44:43,588{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Trying to connect to 169.254.2.99
>>> 2015-03-10 16:44:46,594{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Could not connect to 169.254.2.99
>>> 2015-03-10 16:44:49,449{GMT} DEBUG [cloud.agent.Agent]
>>> (agentRequest-Handler-4:) Processing command:
>>> com.cloud.agent.api.GetVmStatsCommand
>>> 2015-03-10 16:44:51,595{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Trying to connect to 169.254.2.99
>>> 2015-03-10 16:44:54,600{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Could not connect to 169.254.2.99
>>> 2015-03-10 16:44:59,601{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Unable to logon to 169.254.2.99
>>> 2015-03-10 16:44:59,602{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-1:) Executing:
>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>>> -n
>>> r-2191-VM -p %vpccidr=
>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>
>>> 2015-03-10 16:44:59,628{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-1:) Execution is successful.
>>> 2015-03-10 16:44:59,628{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Trying to connect to 169.254.2.99
>>> 2015-03-10 16:45:02,634{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Could not connect to 169.254.2.99
>>> 2015-03-10 16:45:07,635{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Trying to connect to 169.254.2.99
>>> 2015-03-10 16:45:10,640{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Could not connect to 169.254.2.99
>>> 2015-03-10 16:45:12,066{GMT} DEBUG [cloud.agent.Agent]
>>> (agentRequest-Handler-5:) Processing command:
>>> com.cloud.agent.api.GetStorageStatsCommand
>>> 2015-03-10 16:45:15,641{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Unable to logon to 169.254.2.99
>>> 2015-03-10 16:45:15,641{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-1:) Executing:
>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>>> -n
>>> r-2191-VM -p %vpccidr=
>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>
>>> 2015-03-10 16:45:15,667{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-1:) Execution is successful.
>>> 2015-03-10 16:45:15,667{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Trying to connect to 169.254.2.99
>>> 2015-03-10 16:45:17,021{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>> (Thread-11:)
>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in libvirt,
>>> continuing
>>> 2015-03-10 16:45:17,022{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>> (Thread-11:)
>>> Executing:
>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>>> 2015-03-10 16:45:17,036{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>> (Thread-11:)
>>> Execution is successful.
>>> 2015-03-10 16:45:17,951{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
>>> Sending ping: Seq 788-9: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags:
>>> 11,
>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>>> 18.domaina.net
>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>>> hostType":"Routing","hostId":788,"wait":0}}]
>>> }
>>> 2015-03-10 16:45:18,002{GMT} DEBUG [cloud.agent.Agent] (Agent-Handler-2:)
>>> Received response: Seq 788-9: { Ans: , MgmtId: 161344838950, via: 788,
>>> Ver: v1, Flags: 100010,
>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>>> }
>>> 2015-03-10 16:45:18,672{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Could not connect to 169.254.2.99
>>> 2015-03-10 16:45:23,673{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Trying to connect to 169.254.2.99
>>> 2015-03-10 16:45:26,680{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Could not connect to 169.254.2.99
>>> 2015-03-10 16:45:31,682{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Unable to logon to 169.254.2.99
>>> 2015-03-10 16:45:31,683{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-1:) Executing:
>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>>> -n
>>> r-2191-VM -p %vpccidr=
>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>
>>> 2015-03-10 16:45:36,684{GMT} WARN [kvm.resource.
>>> LibvirtComputingResource]
>>> (Script-5:) Interrupting script.
>>> 2015-03-10 16:45:36,686{GMT} WARN [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-1:) Timed out:
>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>>> -n
>>> r-2191-VM -p %vpccidr=
>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> . Output is:
>>> 2015-03-10 16:45:36,686{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-1:) passcmd failed:timeout
>>> 2015-03-10 16:45:36,686{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Trying to connect to 169.254.2.99
>>> 2015-03-10 16:45:37,436{GMT} DEBUG [cloud.agent.Agent]
>>> (agentRequest-Handler-2:) Processing command:
>>> com.cloud.agent.api.GetHostStatsCommand
>>> 2015-03-10 16:45:37,436{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-2:) Executing: /bin/bash -c idle=$(top -b -n 1|grep
>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>>> 2015-03-10 16:45:37,967{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-2:) Execution is successful.
>>> 2015-03-10 16:45:37,968{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-2:) Executing: /bin/bash -c freeMem=$(free|grep
>>> cache:|awk '{print $4}');echo $freeMem
>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-2:) Execution is successful.
>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-2:) Executing: /bin/bash -c free|grep Mem:|awk
>>> '{print $2}'
>>> 2015-03-10 16:45:37,977{GMT} DEBUG [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-2:) Execution is successful.
>>> 2015-03-10 16:45:39,692{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Could not connect to 169.254.2.99
>>> 2015-03-10 16:45:44,693{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Trying to connect to 169.254.2.99
>>> 2015-03-10 16:45:47,700{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Could not connect to 169.254.2.99
>>> 2015-03-10 16:45:51,807{GMT} DEBUG [cloud.agent.Agent]
>>> (agentRequest-Handler-3:) Processing command:
>>> com.cloud.agent.api.GetVmStatsCommand
>>> 2015-03-10 16:45:52,701{GMT} DEBUG
>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> (agentRequest-Handler-1:)
>>> Unable to logon to 169.254.2.99
>>>
>>> On 10 March 2015 at 17:50, ilya musayev <il...@gmail.com>
>>> wrote:
>>>
>>> What kernel are you running? Please include Libvirt and Qemu version.
>>>>
>>>> On 3/10/15 8:21 AM, Andrija Panic wrote:
>>>>
>>>> Hi,
>>>>> we are having issues from time to time with starting VR - while creating
>>>>> VPC.
>>>>>
>>>>> Here is the few lines from MGMT log:
>>>>>
>>>>> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
>>>>> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start vpc [VPC
>>>>> [621-VPC-delete] due to
>>>>> com.cloud.exception.AgentUnavailableException: Resource [Host:788] is
>>>>> unreachable: Host 788: Unable to start instance due to Unable to start
>>>>> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not retrying
>>>>> at
>>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
>>>>> VirtualMachineManagerImpl.java:1072)
>>>>>
>>>>>
>>>>> The interesting part is on the Agent itself:
>>>>>
>>>>>
>>>>> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.
>>>>> LibvirtComputingResource]
>>>>> (agentRequest-Handler-2:) Timed out:
>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>> hviasocket.pl
>>>>> -n
>>>>> r-2189-VM -p %vpccidr=
>>>>> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%
>>>>> template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=
>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>> . Output is:
>>>>>
>>>>>
>>>>> Interesting stuff is - it's seems somewhat random, 3-4 out of 5 works.
>>>>>
>>>>> Any suggestion are really appreciated, since this is live system :(
>>>>> ACS 4.3.0, KVM, CEPH
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>
>>
>
Re: [HELP] VR sometimes fail to start
Posted by Andrija Panic <an...@gmail.com>.
Oh well, I will then update to 4.3.2 anyway, and hope to see the problem go
away...
THanks guys
On 10 March 2015 at 22:45, Loic Lambiel <lo...@exoscale.ch> wrote:
> Hi,
>
> Maybe the same issue we faced here: https://github.com/exoscale/
> cloudstack/commit/df3e7533a78c3594cdd39b8b18e6f4a3abcb8cfd
>
> The patch has not been yet submitted upstream. I'll do it once it is
> running on our production.
>
> Anyway, you may easily test it by performing the small change directly
> into the script.
>
> Hope that helps.
>
> Regards,
>
> Loic Lambiel
>
>
> On 10/03/15 19:40, Andrija Panic wrote:
>
>> Hi guys,
>>
>> Kernel is Elrepo 3.10.67 (CentOS 6.6)
>> Libvirt manually compiled 1.2.3 with RBD support.
>> Qemu 0.12.1.2 - Inktank patched original RHEL 6.4 rpms with RBD support.
>>
>> And here is the output, with debug mode, from Agent.
>>
>> At the begining I see agent deleting routes for 169.254.x.x network...:
>>
>> Basically, It seems like the
>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>> is
>> failing, not being able to connect to socket...
>> And I could not ping VR (169.254.2.99)...
>>
>> LOG:
>>
>> 2015-03-10 16:43:46,976{GMT} DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-4:) Processing command:
>> com.cloud.agent.api.GetVmStatsCommand
>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-1:) Request:Seq 788-1319632926: { Cmd , MgmtId:
>> 161344838950, via: 788, Ver: v1, Flags: 100111,
>> [{"com.cloud.agent.api.StartCommand":{"vm":{"id":
>> 2191,"name":"r-2191-VM","type":"DomainRouter","cpus":1,"
>> minSpeed":166,"maxSpeed":1000,"minRam":268435456,"maxRam":
>> 268435456,"arch":"x86_64","os":"Debian
>> GNU/Linux 7(64-bit)","bootArgs":" vpccidr=10.0.0.0/8
>> domain=cs2cloud.internal dns1=8.8.8.8 dns2= template=domP name=r-2191-VM
>> eth0ip=169.254.2.99 eth0mask=255.255.0.0 type=vpcrouter
>> disable_rp_filter=true","rebootOnCrash":false,"
>> enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":
>> false,"vncPassword":"e9e24eb0b1ff8ec2","params":{"
>> memoryOvercommitRatio":"1.0","cpuOvercommitRatio":"6.0"},"
>> uuid":"485587f7-4a04-4d77-bc36-2367f2adf77e","disks":[{"
>> data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
>> uuid":"d042e003-5278-461d-8a35-3b1261164f93","
>> volumeType":"ROOT","dataStore":{"org.apache.cloudstack.
>> storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-
>> 88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
>> cephmon.domaina.net","path":"cloudstack","port":6789,"url":"RBD://
>> cephmon.domaina.net/cloudstack/?ROLE=Primary&
>> STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6
>> "}},"name":"ROOT-2191","size":2621440000,"path":"d042e003-
>> 5278-461d-8a35-3b1261164f93","volumeId":2857,"vmName":"r-
>> 2191-VM","accountId":2,"format":"RAW","id":2857,"
>> deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"
>> d042e003-5278-461d-8a35-3b1261164f93","type":"ROOT","_
>> details":{"managed":"false","storagePort":"6789","storageHost":"
>> cephmon.domaina.net
>> ","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"
>> networkRateMbps":-1,"defaultNic":false,"uuid":"d27a7aad-2dcc-45c7-bf30-
>> 3302d011da66","ip":"169.254.2.99","netmask":"255.255.0.0","
>> gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:63","
>> broadcastType":"LinkLocal","type":"Control","
>> isSecurityGroupEnabled":false}]},"hostIp":"10.44.253.180","
>> executeInSequence":false,"wait":0}},{"com.cloud.agent.
>> api.check.CheckSshCommand":{"ip":"169.254.2.99","port":
>> 3922,"interval":6,"retries":100,"name":"r-2191-VM","wait":
>> 0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
>> router.name
>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>> com.cloud.agent.api.PlugNicCommand":{"nic":{"
>> deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid"
>> :"fe848dbe-cf4b-473e-8124-62289f86b144","ip":"5.226.146.
>> 62","netmask":"255.255.255.128","gateway":"5.226.146.1","
>> mac":"06:f0:d6:00:01:49","broadcastType":"Vlan","type":"
>> Public","broadcastUri":"vlan://untagged","isolationUri":"
>> vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"},"
>> instanceName":"r-2191-VM","vmType":"DomainRouter","wait":
>> 0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"
>> ipAddresses":[{"accountId":2,"publicIp":"5.226.146.62","
>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>> networkName":"breth1-500"}],"accessDetails":{"router.guest.
>> ip":"5.226.146.62","zone.network.type":"Advanced","
>> router.name
>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>> com.cloud.agent.api.routing.SetSourceNatCommand":{"
>> ipAddress":{"accountId":2,"publicIp":"5.226.146.62","
>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>> networkName":"breth1-500"},"add":true,"accessDetails":{"
>> zone.network.type":"Advanced","
>> router.name":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{}] }
>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-1:) Processing command:
>> com.cloud.agent.api.StartCommand
>> 2015-03-10 16:43:58,257{GMT} DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-5:) Processing command:
>> com.cloud.agent.api.GetStorageStatsCommand
>> 2015-03-10 16:44:01,747{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
>> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
>> GNU/Linux 7(64-bit)
>> 2015-03-10 16:44:06,309{GMT} DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-2:) Processing command:
>> com.cloud.agent.api.NetworkUsageCommand
>> 2015-03-10 16:44:06,309{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-2:) Executing:
>> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
>> vpc_netusage.sh 169.254.2.1 -l 46.232.180.55 -g
>> 2015-03-10 16:44:06,439{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-2:) Execution is successful.
>> 2015-03-10 16:44:06,441{GMT} DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-2:) Seq 788-1319632928: { Ans: , MgmtId:
>> 161344838950, via: 788, Ver: v1, Flags: 10,
>> [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-1873-VM","
>> bytesSent":198379,"bytesReceived":7380903,"result":true,"details":"","
>> wait":0}}]
>> }
>> 2015-03-10 16:44:11,055{GMT} DEBUG [kvm.resource.BridgeVifDriver]
>> (agentRequest-Handler-1:) nic=[Nic:Control-169.254.2.99-null]
>> 2015-03-10 16:44:11,055{GMT} DEBUG [utils.script.Script]
>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route | grep
>> 169.254.0.0/16
>> 2015-03-10 16:44:11,060{GMT} DEBUG [utils.script.Script]
>> (agentRequest-Handler-1:) Execution is successful.
>> 2015-03-10 16:44:11,061{GMT} DEBUG [utils.script.Script]
>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route del
>> 169.254.0.0/16
>> 2015-03-10 16:44:11,064{GMT} DEBUG [utils.script.Script]
>> (agentRequest-Handler-1:) Execution is successful.
>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
>> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
>> GNU/Linux 7(64-bit)
>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-1:) starting r-2191-VM: <domain type='kvm'>
>> <name>r-2191-VM</name>
>> <uuid>485587f7-4a04-4d77-bc36-2367f2adf77e</uuid>
>> <description>Debian GNU/Linux 7(64-bit)</description>
>> <clock offset='utc'>
>> </clock>
>> <features>
>> <pae/>
>> <apic/>
>> <acpi/>
>> </features>
>> <devices>
>> <emulator>/usr/libexec/qemu-kvm</emulator>
>> <interface type='bridge'>
>> <source bridge='cloud0'/>
>> <mac address='0e:00:a9:fe:02:63'/>
>> <model type='virtio'/>
>> </interface>
>> <serial type='pty'>
>> <target port='0'/>
>> </serial>
>> <graphics type='vnc' autoport='yes' listen='10.44.253.180'
>> passwd='e9e24eb0b1ff8ec2'/>
>> <disk device='disk' type='network'>
>> <driver name='qemu' type='raw' cache='none' />
>> <source protocol='rbd'
>> name='cloudstack/d042e003-5278-461d-8a35-3b1261164f93'>
>> <host name='cephmon.domain.internal' port='6789'/>
>> </source>
>> <auth username='cloudstack'>
>> <secret type='ceph' uuid='5b93422e-1a66-353d-88a8-2203f79b1dc6'/>
>> </auth>
>> <target dev='vda' bus='virtio'/>
>> </disk>
>> <disk device='cdrom' type='file'>
>> <driver name='qemu' type='raw' cache='none' />
>> <source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
>> <target dev='hdc' bus='ide'/>
>> </disk>
>> <console type='pty'>
>> <target port='0'/>
>> </console>
>> <input type='tablet' bus='usb'/>
>> <channel type='unix'>
>> <source mode='bind' path='/var/lib/libvirt/qemu/r-2191-VM.agent'/>
>> <target type='virtio' name='r-2191-VM.vport'/>
>> <address type='virtio-serial'/>
>> </channel>
>> </devices>
>> <memory>262144</memory>
>> <devices>
>> <memballoon model='none'/>
>> </devices>
>> <vcpu>1</vcpu>
>> <os>
>> <type arch='x86_64' machine='pc'>hvm</type>
>> <boot dev='cdrom'/>
>> <boot dev='hd'/>
>> </os>
>> <cputune>
>> <shares>166</shares>
>> </cputune>
>> <cpu></cpu><on_reboot>restart</on_reboot>
>> <on_poweroff>destroy</on_poweroff>
>> <on_crash>destroy</on_crash>
>> </domain>
>>
>> 2015-03-10 16:44:11,467{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-1:) Executing:
>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>> -n
>> r-2191-VM -p %vpccidr=
>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>
>> 2015-03-10 16:44:11,494{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-1:) Execution is successful.
>> 2015-03-10 16:44:11,495{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Trying to connect to 169.254.2.99
>> 2015-03-10 16:44:14,501{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Could not connect to 169.254.2.99
>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>> (Thread-10:)
>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in libvirt,
>> continuing
>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>> (Thread-10:)
>> Executing:
>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>> 2015-03-10 16:44:17,019{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>> (Thread-10:)
>> Execution is successful.
>> 2015-03-10 16:44:17,953{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
>> Sending ping: Seq 788-8: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags:
>> 11,
>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>> 18.domaina.net
>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>> hostType":"Routing","hostId":788,"wait":0}}]
>> }
>> 2015-03-10 16:44:17,964{GMT} DEBUG [cloud.agent.Agent] (Agent-Handler-3:)
>> Received response: Seq 788-8: { Ans: , MgmtId: 161344838950, via: 788,
>> Ver: v1, Flags: 100010,
>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>> }
>> 2015-03-10 16:44:19,502{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Trying to connect to 169.254.2.99
>> 2015-03-10 16:44:22,509{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Could not connect to 169.254.2.99
>> 2015-03-10 16:44:27,510{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Unable to logon to 169.254.2.99
>> 2015-03-10 16:44:27,512{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-1:) Executing:
>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>> -n
>> r-2191-VM -p %vpccidr=
>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>
>> 2015-03-10 16:44:27,540{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-1:) Execution is successful.
>> 2015-03-10 16:44:27,541{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Trying to connect to 169.254.2.99
>> 2015-03-10 16:44:30,548{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Could not connect to 169.254.2.99
>> 2015-03-10 16:44:31,378{GMT} DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-3:) Processing command:
>> com.cloud.agent.api.GetHostStatsCommand
>> 2015-03-10 16:44:31,379{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-3:) Executing: /bin/bash -c idle=$(top -b -n 1|grep
>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>> 2015-03-10 16:44:31,910{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-3:) Execution is successful.
>> 2015-03-10 16:44:31,911{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-3:) Executing: /bin/bash -c freeMem=$(free|grep
>> cache:|awk '{print $4}');echo $freeMem
>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-3:) Execution is successful.
>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-3:) Executing: /bin/bash -c free|grep Mem:|awk
>> '{print $2}'
>> 2015-03-10 16:44:31,920{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-3:) Execution is successful.
>> 2015-03-10 16:44:35,549{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Trying to connect to 169.254.2.99
>> 2015-03-10 16:44:38,556{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Could not connect to 169.254.2.99
>> 2015-03-10 16:44:43,557{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Unable to logon to 169.254.2.99
>> 2015-03-10 16:44:43,557{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-1:) Executing:
>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>> -n
>> r-2191-VM -p %vpccidr=
>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>
>> 2015-03-10 16:44:43,587{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-1:) Execution is successful.
>> 2015-03-10 16:44:43,588{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Trying to connect to 169.254.2.99
>> 2015-03-10 16:44:46,594{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Could not connect to 169.254.2.99
>> 2015-03-10 16:44:49,449{GMT} DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-4:) Processing command:
>> com.cloud.agent.api.GetVmStatsCommand
>> 2015-03-10 16:44:51,595{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Trying to connect to 169.254.2.99
>> 2015-03-10 16:44:54,600{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Could not connect to 169.254.2.99
>> 2015-03-10 16:44:59,601{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Unable to logon to 169.254.2.99
>> 2015-03-10 16:44:59,602{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-1:) Executing:
>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>> -n
>> r-2191-VM -p %vpccidr=
>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>
>> 2015-03-10 16:44:59,628{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-1:) Execution is successful.
>> 2015-03-10 16:44:59,628{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Trying to connect to 169.254.2.99
>> 2015-03-10 16:45:02,634{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Could not connect to 169.254.2.99
>> 2015-03-10 16:45:07,635{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Trying to connect to 169.254.2.99
>> 2015-03-10 16:45:10,640{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Could not connect to 169.254.2.99
>> 2015-03-10 16:45:12,066{GMT} DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-5:) Processing command:
>> com.cloud.agent.api.GetStorageStatsCommand
>> 2015-03-10 16:45:15,641{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Unable to logon to 169.254.2.99
>> 2015-03-10 16:45:15,641{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-1:) Executing:
>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>> -n
>> r-2191-VM -p %vpccidr=
>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>
>> 2015-03-10 16:45:15,667{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-1:) Execution is successful.
>> 2015-03-10 16:45:15,667{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Trying to connect to 169.254.2.99
>> 2015-03-10 16:45:17,021{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>> (Thread-11:)
>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in libvirt,
>> continuing
>> 2015-03-10 16:45:17,022{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>> (Thread-11:)
>> Executing:
>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>> 2015-03-10 16:45:17,036{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>> (Thread-11:)
>> Execution is successful.
>> 2015-03-10 16:45:17,951{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
>> Sending ping: Seq 788-9: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags:
>> 11,
>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>> 18.domaina.net
>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>> hostType":"Routing","hostId":788,"wait":0}}]
>> }
>> 2015-03-10 16:45:18,002{GMT} DEBUG [cloud.agent.Agent] (Agent-Handler-2:)
>> Received response: Seq 788-9: { Ans: , MgmtId: 161344838950, via: 788,
>> Ver: v1, Flags: 100010,
>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>> }
>> 2015-03-10 16:45:18,672{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Could not connect to 169.254.2.99
>> 2015-03-10 16:45:23,673{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Trying to connect to 169.254.2.99
>> 2015-03-10 16:45:26,680{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Could not connect to 169.254.2.99
>> 2015-03-10 16:45:31,682{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Unable to logon to 169.254.2.99
>> 2015-03-10 16:45:31,683{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-1:) Executing:
>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>> -n
>> r-2191-VM -p %vpccidr=
>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>
>> 2015-03-10 16:45:36,684{GMT} WARN [kvm.resource.
>> LibvirtComputingResource]
>> (Script-5:) Interrupting script.
>> 2015-03-10 16:45:36,686{GMT} WARN [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-1:) Timed out:
>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>> -n
>> r-2191-VM -p %vpccidr=
>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>> . Output is:
>> 2015-03-10 16:45:36,686{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-1:) passcmd failed:timeout
>> 2015-03-10 16:45:36,686{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Trying to connect to 169.254.2.99
>> 2015-03-10 16:45:37,436{GMT} DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-2:) Processing command:
>> com.cloud.agent.api.GetHostStatsCommand
>> 2015-03-10 16:45:37,436{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-2:) Executing: /bin/bash -c idle=$(top -b -n 1|grep
>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>> 2015-03-10 16:45:37,967{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-2:) Execution is successful.
>> 2015-03-10 16:45:37,968{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-2:) Executing: /bin/bash -c freeMem=$(free|grep
>> cache:|awk '{print $4}');echo $freeMem
>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-2:) Execution is successful.
>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-2:) Executing: /bin/bash -c free|grep Mem:|awk
>> '{print $2}'
>> 2015-03-10 16:45:37,977{GMT} DEBUG [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-2:) Execution is successful.
>> 2015-03-10 16:45:39,692{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Could not connect to 169.254.2.99
>> 2015-03-10 16:45:44,693{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Trying to connect to 169.254.2.99
>> 2015-03-10 16:45:47,700{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Could not connect to 169.254.2.99
>> 2015-03-10 16:45:51,807{GMT} DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-3:) Processing command:
>> com.cloud.agent.api.GetVmStatsCommand
>> 2015-03-10 16:45:52,701{GMT} DEBUG
>> [resource.virtualnetwork.VirtualRoutingResource]
>> (agentRequest-Handler-1:)
>> Unable to logon to 169.254.2.99
>>
>> On 10 March 2015 at 17:50, ilya musayev <il...@gmail.com>
>> wrote:
>>
>> What kernel are you running? Please include Libvirt and Qemu version.
>>>
>>>
>>> On 3/10/15 8:21 AM, Andrija Panic wrote:
>>>
>>> Hi,
>>>>
>>>> we are having issues from time to time with starting VR - while creating
>>>> VPC.
>>>>
>>>> Here is the few lines from MGMT log:
>>>>
>>>> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
>>>> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start vpc [VPC
>>>> [621-VPC-delete] due to
>>>> com.cloud.exception.AgentUnavailableException: Resource [Host:788] is
>>>> unreachable: Host 788: Unable to start instance due to Unable to start
>>>> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not retrying
>>>> at
>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
>>>> VirtualMachineManagerImpl.java:1072)
>>>>
>>>>
>>>> The interesting part is on the Agent itself:
>>>>
>>>>
>>>> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.
>>>> LibvirtComputingResource]
>>>> (agentRequest-Handler-2:) Timed out:
>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>> hviasocket.pl
>>>> -n
>>>> r-2189-VM -p %vpccidr=
>>>> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%
>>>> template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=
>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>> . Output is:
>>>>
>>>>
>>>> Interesting stuff is - it's seems somewhat random, 3-4 out of 5 works.
>>>>
>>>> Any suggestion are really appreciated, since this is live system :(
>>>> ACS 4.3.0, KVM, CEPH
>>>>
>>>> Thanks,
>>>>
>>>>
>>>
>>
>
>
>
--
Andrija Panić
Re: [HELP] VR sometimes fail to start
Posted by Loic Lambiel <lo...@exoscale.ch>.
Hi,
Maybe the same issue we faced here:
https://github.com/exoscale/cloudstack/commit/df3e7533a78c3594cdd39b8b18e6f4a3abcb8cfd
The patch has not been yet submitted upstream. I'll do it once it is
running on our production.
Anyway, you may easily test it by performing the small change directly
into the script.
Hope that helps.
Regards,
Loic Lambiel
On 10/03/15 19:40, Andrija Panic wrote:
> Hi guys,
>
> Kernel is Elrepo 3.10.67 (CentOS 6.6)
> Libvirt manually compiled 1.2.3 with RBD support.
> Qemu 0.12.1.2 - Inktank patched original RHEL 6.4 rpms with RBD support.
>
> And here is the output, with debug mode, from Agent.
>
> At the begining I see agent deleting routes for 169.254.x.x network...:
>
> Basically, It seems like the
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl is
> failing, not being able to connect to socket...
> And I could not ping VR (169.254.2.99)...
>
> LOG:
>
> 2015-03-10 16:43:46,976{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:) Processing command:
> com.cloud.agent.api.GetVmStatsCommand
> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:) Request:Seq 788-1319632926: { Cmd , MgmtId:
> 161344838950, via: 788, Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.StartCommand":{"vm":{"id":2191,"name":"r-2191-VM","type":"DomainRouter","cpus":1,"minSpeed":166,"maxSpeed":1000,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian
> GNU/Linux 7(64-bit)","bootArgs":" vpccidr=10.0.0.0/8
> domain=cs2cloud.internal dns1=8.8.8.8 dns2= template=domP name=r-2191-VM
> eth0ip=169.254.2.99 eth0mask=255.255.0.0 type=vpcrouter
> disable_rp_filter=true","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"e9e24eb0b1ff8ec2","params":{"memoryOvercommitRatio":"1.0","cpuOvercommitRatio":"6.0"},"uuid":"485587f7-4a04-4d77-bc36-2367f2adf77e","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"d042e003-5278-461d-8a35-3b1261164f93","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
> cephmon.domaina.net","path":"cloudstack","port":6789,"url":"RBD://
> cephmon.domaina.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6
> "}},"name":"ROOT-2191","size":2621440000,"path":"d042e003-5278-461d-8a35-3b1261164f93","volumeId":2857,"vmName":"r-2191-VM","accountId":2,"format":"RAW","id":2857,"deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"d042e003-5278-461d-8a35-3b1261164f93","type":"ROOT","_details":{"managed":"false","storagePort":"6789","storageHost":"
> cephmon.domaina.net
> ","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"uuid":"d27a7aad-2dcc-45c7-bf30-3302d011da66","ip":"169.254.2.99","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:63","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.44.253.180","executeInSequence":false,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.2.99","port":3922,"interval":6,"retries":100,"name":"r-2191-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
> router.name
> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"com.cloud.agent.api.PlugNicCommand":{"nic":{"deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid":"fe848dbe-cf4b-473e-8124-62289f86b144","ip":"5.226.146.62","netmask":"255.255.255.128","gateway":"5.226.146.1","mac":"06:f0:d6:00:01:49","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"},"instanceName":"r-2191-VM","vmType":"DomainRouter","wait":0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"ipAddresses":[{"accountId":2,"publicIp":"5.226.146.62","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":false,"broadcastUri":"vlan://untagged","vlanGateway":"5.226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","networkName":"breth1-500"}],"accessDetails":{"router.guest.ip":"5.226.146.62","zone.network.type":"Advanced","
> router.name
> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"com.cloud.agent.api.routing.SetSourceNatCommand":{"ipAddress":{"accountId":2,"publicIp":"5.226.146.62","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":false,"broadcastUri":"vlan://untagged","vlanGateway":"5.226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","networkName":"breth1-500"},"add":true,"accessDetails":{"zone.network.type":"Advanced","
> router.name":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{}] }
> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:) Processing command:
> com.cloud.agent.api.StartCommand
> 2015-03-10 16:43:58,257{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:) Processing command:
> com.cloud.agent.api.GetStorageStatsCommand
> 2015-03-10 16:44:01,747{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
> GNU/Linux 7(64-bit)
> 2015-03-10 16:44:06,309{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:) Processing command:
> com.cloud.agent.api.NetworkUsageCommand
> 2015-03-10 16:44:06,309{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Executing:
> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
> vpc_netusage.sh 169.254.2.1 -l 46.232.180.55 -g
> 2015-03-10 16:44:06,439{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Execution is successful.
> 2015-03-10 16:44:06,441{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:) Seq 788-1319632928: { Ans: , MgmtId:
> 161344838950, via: 788, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-1873-VM","bytesSent":198379,"bytesReceived":7380903,"result":true,"details":"","wait":0}}]
> }
> 2015-03-10 16:44:11,055{GMT} DEBUG [kvm.resource.BridgeVifDriver]
> (agentRequest-Handler-1:) nic=[Nic:Control-169.254.2.99-null]
> 2015-03-10 16:44:11,055{GMT} DEBUG [utils.script.Script]
> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route | grep
> 169.254.0.0/16
> 2015-03-10 16:44:11,060{GMT} DEBUG [utils.script.Script]
> (agentRequest-Handler-1:) Execution is successful.
> 2015-03-10 16:44:11,061{GMT} DEBUG [utils.script.Script]
> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route del
> 169.254.0.0/16
> 2015-03-10 16:44:11,064{GMT} DEBUG [utils.script.Script]
> (agentRequest-Handler-1:) Execution is successful.
> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
> GNU/Linux 7(64-bit)
> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) starting r-2191-VM: <domain type='kvm'>
> <name>r-2191-VM</name>
> <uuid>485587f7-4a04-4d77-bc36-2367f2adf77e</uuid>
> <description>Debian GNU/Linux 7(64-bit)</description>
> <clock offset='utc'>
> </clock>
> <features>
> <pae/>
> <apic/>
> <acpi/>
> </features>
> <devices>
> <emulator>/usr/libexec/qemu-kvm</emulator>
> <interface type='bridge'>
> <source bridge='cloud0'/>
> <mac address='0e:00:a9:fe:02:63'/>
> <model type='virtio'/>
> </interface>
> <serial type='pty'>
> <target port='0'/>
> </serial>
> <graphics type='vnc' autoport='yes' listen='10.44.253.180'
> passwd='e9e24eb0b1ff8ec2'/>
> <disk device='disk' type='network'>
> <driver name='qemu' type='raw' cache='none' />
> <source protocol='rbd'
> name='cloudstack/d042e003-5278-461d-8a35-3b1261164f93'>
> <host name='cephmon.domain.internal' port='6789'/>
> </source>
> <auth username='cloudstack'>
> <secret type='ceph' uuid='5b93422e-1a66-353d-88a8-2203f79b1dc6'/>
> </auth>
> <target dev='vda' bus='virtio'/>
> </disk>
> <disk device='cdrom' type='file'>
> <driver name='qemu' type='raw' cache='none' />
> <source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
> <target dev='hdc' bus='ide'/>
> </disk>
> <console type='pty'>
> <target port='0'/>
> </console>
> <input type='tablet' bus='usb'/>
> <channel type='unix'>
> <source mode='bind' path='/var/lib/libvirt/qemu/r-2191-VM.agent'/>
> <target type='virtio' name='r-2191-VM.vport'/>
> <address type='virtio-serial'/>
> </channel>
> </devices>
> <memory>262144</memory>
> <devices>
> <memballoon model='none'/>
> </devices>
> <vcpu>1</vcpu>
> <os>
> <type arch='x86_64' machine='pc'>hvm</type>
> <boot dev='cdrom'/>
> <boot dev='hd'/>
> </os>
> <cputune>
> <shares>166</shares>
> </cputune>
> <cpu></cpu><on_reboot>restart</on_reboot>
> <on_poweroff>destroy</on_poweroff>
> <on_crash>destroy</on_crash>
> </domain>
>
> 2015-03-10 16:44:11,467{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-2191-VM -p %vpccidr=
> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
>
> 2015-03-10 16:44:11,494{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Execution is successful.
> 2015-03-10 16:44:11,495{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:44:14,501{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor] (Thread-10:)
> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in libvirt,
> continuing
> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor] (Thread-10:)
> Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
> 2015-03-10 16:44:17,019{GMT} DEBUG [kvm.resource.KVMHAMonitor] (Thread-10:)
> Execution is successful.
> 2015-03-10 16:44:17,953{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
> Sending ping: Seq 788-8: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags: 11,
> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net"}},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":788,"wait":0}}]
> }
> 2015-03-10 16:44:17,964{GMT} DEBUG [cloud.agent.Agent] (Agent-Handler-3:)
> Received response: Seq 788-8: { Ans: , MgmtId: 161344838950, via: 788,
> Ver: v1, Flags: 100010,
> [{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
> }
> 2015-03-10 16:44:19,502{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:44:22,509{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:44:27,510{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Unable to logon to 169.254.2.99
> 2015-03-10 16:44:27,512{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-2191-VM -p %vpccidr=
> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
>
> 2015-03-10 16:44:27,540{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Execution is successful.
> 2015-03-10 16:44:27,541{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:44:30,548{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:44:31,378{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:) Processing command:
> com.cloud.agent.api.GetHostStatsCommand
> 2015-03-10 16:44:31,379{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:) Executing: /bin/bash -c idle=$(top -b -n 1|grep
> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
> 2015-03-10 16:44:31,910{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:) Execution is successful.
> 2015-03-10 16:44:31,911{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:) Executing: /bin/bash -c freeMem=$(free|grep
> cache:|awk '{print $4}');echo $freeMem
> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:) Execution is successful.
> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:) Executing: /bin/bash -c free|grep Mem:|awk
> '{print $2}'
> 2015-03-10 16:44:31,920{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:) Execution is successful.
> 2015-03-10 16:44:35,549{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:44:38,556{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:44:43,557{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Unable to logon to 169.254.2.99
> 2015-03-10 16:44:43,557{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-2191-VM -p %vpccidr=
> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
>
> 2015-03-10 16:44:43,587{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Execution is successful.
> 2015-03-10 16:44:43,588{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:44:46,594{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:44:49,449{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:) Processing command:
> com.cloud.agent.api.GetVmStatsCommand
> 2015-03-10 16:44:51,595{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:44:54,600{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:44:59,601{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Unable to logon to 169.254.2.99
> 2015-03-10 16:44:59,602{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-2191-VM -p %vpccidr=
> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
>
> 2015-03-10 16:44:59,628{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Execution is successful.
> 2015-03-10 16:44:59,628{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:45:02,634{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:45:07,635{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:45:10,640{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:45:12,066{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:) Processing command:
> com.cloud.agent.api.GetStorageStatsCommand
> 2015-03-10 16:45:15,641{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Unable to logon to 169.254.2.99
> 2015-03-10 16:45:15,641{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-2191-VM -p %vpccidr=
> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
>
> 2015-03-10 16:45:15,667{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Execution is successful.
> 2015-03-10 16:45:15,667{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:45:17,021{GMT} DEBUG [kvm.resource.KVMHAMonitor] (Thread-11:)
> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in libvirt,
> continuing
> 2015-03-10 16:45:17,022{GMT} DEBUG [kvm.resource.KVMHAMonitor] (Thread-11:)
> Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
> 2015-03-10 16:45:17,036{GMT} DEBUG [kvm.resource.KVMHAMonitor] (Thread-11:)
> Execution is successful.
> 2015-03-10 16:45:17,951{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
> Sending ping: Seq 788-9: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags: 11,
> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net"}},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":788,"wait":0}}]
> }
> 2015-03-10 16:45:18,002{GMT} DEBUG [cloud.agent.Agent] (Agent-Handler-2:)
> Received response: Seq 788-9: { Ans: , MgmtId: 161344838950, via: 788,
> Ver: v1, Flags: 100010,
> [{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
> }
> 2015-03-10 16:45:18,672{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:45:23,673{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:45:26,680{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:45:31,682{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Unable to logon to 169.254.2.99
> 2015-03-10 16:45:31,683{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-2191-VM -p %vpccidr=
> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
>
> 2015-03-10 16:45:36,684{GMT} WARN [kvm.resource.LibvirtComputingResource]
> (Script-5:) Interrupting script.
> 2015-03-10 16:45:36,686{GMT} WARN [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Timed out:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-2191-VM -p %vpccidr=
> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
> . Output is:
> 2015-03-10 16:45:36,686{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) passcmd failed:timeout
> 2015-03-10 16:45:36,686{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:45:37,436{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:) Processing command:
> com.cloud.agent.api.GetHostStatsCommand
> 2015-03-10 16:45:37,436{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Executing: /bin/bash -c idle=$(top -b -n 1|grep
> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
> 2015-03-10 16:45:37,967{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Execution is successful.
> 2015-03-10 16:45:37,968{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Executing: /bin/bash -c freeMem=$(free|grep
> cache:|awk '{print $4}');echo $freeMem
> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Execution is successful.
> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Executing: /bin/bash -c free|grep Mem:|awk
> '{print $2}'
> 2015-03-10 16:45:37,977{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Execution is successful.
> 2015-03-10 16:45:39,692{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:45:44,693{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:45:47,700{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:45:51,807{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:) Processing command:
> com.cloud.agent.api.GetVmStatsCommand
> 2015-03-10 16:45:52,701{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Unable to logon to 169.254.2.99
>
> On 10 March 2015 at 17:50, ilya musayev <il...@gmail.com>
> wrote:
>
>> What kernel are you running? Please include Libvirt and Qemu version.
>>
>>
>> On 3/10/15 8:21 AM, Andrija Panic wrote:
>>
>>> Hi,
>>>
>>> we are having issues from time to time with starting VR - while creating
>>> VPC.
>>>
>>> Here is the few lines from MGMT log:
>>>
>>> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
>>> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start vpc [VPC
>>> [621-VPC-delete] due to
>>> com.cloud.exception.AgentUnavailableException: Resource [Host:788] is
>>> unreachable: Host 788: Unable to start instance due to Unable to start
>>> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not retrying
>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
>>> VirtualMachineManagerImpl.java:1072)
>>>
>>>
>>> The interesting part is on the Agent itself:
>>>
>>>
>>> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-2:) Timed out:
>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>>> -n
>>> r-2189-VM -p %vpccidr=
>>> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%
>>> template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=
>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> . Output is:
>>>
>>>
>>> Interesting stuff is - it's seems somewhat random, 3-4 out of 5 works.
>>>
>>> Any suggestion are really appreciated, since this is live system :(
>>> ACS 4.3.0, KVM, CEPH
>>>
>>> Thanks,
>>>
>>
>
Re: [HELP] VR sometimes fail to start
Posted by Marcus <sh...@gmail.com>.
I think I've seen some bugfixes for this in newer versions of 4.3+
On Tue, Mar 10, 2015 at 11:40 AM, Andrija Panic <an...@gmail.com> wrote:
> Hi guys,
>
> Kernel is Elrepo 3.10.67 (CentOS 6.6)
> Libvirt manually compiled 1.2.3 with RBD support.
> Qemu 0.12.1.2 - Inktank patched original RHEL 6.4 rpms with RBD support.
>
> And here is the output, with debug mode, from Agent.
>
> At the begining I see agent deleting routes for 169.254.x.x network...:
>
> Basically, It seems like the
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl is
> failing, not being able to connect to socket...
> And I could not ping VR (169.254.2.99)...
>
> LOG:
>
> 2015-03-10 16:43:46,976{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:) Processing command:
> com.cloud.agent.api.GetVmStatsCommand
> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:) Request:Seq 788-1319632926: { Cmd , MgmtId:
> 161344838950, via: 788, Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.StartCommand":{"vm":{"id":2191,"name":"r-2191-VM","type":"DomainRouter","cpus":1,"minSpeed":166,"maxSpeed":1000,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian
> GNU/Linux 7(64-bit)","bootArgs":" vpccidr=10.0.0.0/8
> domain=cs2cloud.internal dns1=8.8.8.8 dns2= template=domP name=r-2191-VM
> eth0ip=169.254.2.99 eth0mask=255.255.0.0 type=vpcrouter
> disable_rp_filter=true","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"e9e24eb0b1ff8ec2","params":{"memoryOvercommitRatio":"1.0","cpuOvercommitRatio":"6.0"},"uuid":"485587f7-4a04-4d77-bc36-2367f2adf77e","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"d042e003-5278-461d-8a35-3b1261164f93","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
> cephmon.domaina.net","path":"cloudstack","port":6789,"url":"RBD://
> cephmon.domaina.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6
> "}},"name":"ROOT-2191","size":2621440000,"path":"d042e003-5278-461d-8a35-3b1261164f93","volumeId":2857,"vmName":"r-2191-VM","accountId":2,"format":"RAW","id":2857,"deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"d042e003-5278-461d-8a35-3b1261164f93","type":"ROOT","_details":{"managed":"false","storagePort":"6789","storageHost":"
> cephmon.domaina.net
> ","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"uuid":"d27a7aad-2dcc-45c7-bf30-3302d011da66","ip":"169.254.2.99","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:63","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.44.253.180","executeInSequence":false,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.2.99","port":3922,"interval":6,"retries":100,"name":"r-2191-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
> router.name
> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"com.cloud.agent.api.PlugNicCommand":{"nic":{"deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid":"fe848dbe-cf4b-473e-8124-62289f86b144","ip":"5.226.146.62","netmask":"255.255.255.128","gateway":"5.226.146.1","mac":"06:f0:d6:00:01:49","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"},"instanceName":"r-2191-VM","vmType":"DomainRouter","wait":0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"ipAddresses":[{"accountId":2,"publicIp":"5.226.146.62","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":false,"broadcastUri":"vlan://untagged","vlanGateway":"5.226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","networkName":"breth1-500"}],"accessDetails":{"router.guest.ip":"5.226.146.62","zone.network.type":"Advanced","
> router.name
> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"com.cloud.agent.api.routing.SetSourceNatCommand":{"ipAddress":{"accountId":2,"publicIp":"5.226.146.62","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":false,"broadcastUri":"vlan://untagged","vlanGateway":"5.226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","networkName":"breth1-500"},"add":true,"accessDetails":{"zone.network.type":"Advanced","
> router.name":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{}] }
> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:) Processing command:
> com.cloud.agent.api.StartCommand
> 2015-03-10 16:43:58,257{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:) Processing command:
> com.cloud.agent.api.GetStorageStatsCommand
> 2015-03-10 16:44:01,747{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
> GNU/Linux 7(64-bit)
> 2015-03-10 16:44:06,309{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:) Processing command:
> com.cloud.agent.api.NetworkUsageCommand
> 2015-03-10 16:44:06,309{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Executing:
> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
> vpc_netusage.sh 169.254.2.1 -l 46.232.180.55 -g
> 2015-03-10 16:44:06,439{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Execution is successful.
> 2015-03-10 16:44:06,441{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:) Seq 788-1319632928: { Ans: , MgmtId:
> 161344838950, via: 788, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-1873-VM","bytesSent":198379,"bytesReceived":7380903,"result":true,"details":"","wait":0}}]
> }
> 2015-03-10 16:44:11,055{GMT} DEBUG [kvm.resource.BridgeVifDriver]
> (agentRequest-Handler-1:) nic=[Nic:Control-169.254.2.99-null]
> 2015-03-10 16:44:11,055{GMT} DEBUG [utils.script.Script]
> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route | grep
> 169.254.0.0/16
> 2015-03-10 16:44:11,060{GMT} DEBUG [utils.script.Script]
> (agentRequest-Handler-1:) Execution is successful.
> 2015-03-10 16:44:11,061{GMT} DEBUG [utils.script.Script]
> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route del
> 169.254.0.0/16
> 2015-03-10 16:44:11,064{GMT} DEBUG [utils.script.Script]
> (agentRequest-Handler-1:) Execution is successful.
> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
> GNU/Linux 7(64-bit)
> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) starting r-2191-VM: <domain type='kvm'>
> <name>r-2191-VM</name>
> <uuid>485587f7-4a04-4d77-bc36-2367f2adf77e</uuid>
> <description>Debian GNU/Linux 7(64-bit)</description>
> <clock offset='utc'>
> </clock>
> <features>
> <pae/>
> <apic/>
> <acpi/>
> </features>
> <devices>
> <emulator>/usr/libexec/qemu-kvm</emulator>
> <interface type='bridge'>
> <source bridge='cloud0'/>
> <mac address='0e:00:a9:fe:02:63'/>
> <model type='virtio'/>
> </interface>
> <serial type='pty'>
> <target port='0'/>
> </serial>
> <graphics type='vnc' autoport='yes' listen='10.44.253.180'
> passwd='e9e24eb0b1ff8ec2'/>
> <disk device='disk' type='network'>
> <driver name='qemu' type='raw' cache='none' />
> <source protocol='rbd'
> name='cloudstack/d042e003-5278-461d-8a35-3b1261164f93'>
> <host name='cephmon.domain.internal' port='6789'/>
> </source>
> <auth username='cloudstack'>
> <secret type='ceph' uuid='5b93422e-1a66-353d-88a8-2203f79b1dc6'/>
> </auth>
> <target dev='vda' bus='virtio'/>
> </disk>
> <disk device='cdrom' type='file'>
> <driver name='qemu' type='raw' cache='none' />
> <source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
> <target dev='hdc' bus='ide'/>
> </disk>
> <console type='pty'>
> <target port='0'/>
> </console>
> <input type='tablet' bus='usb'/>
> <channel type='unix'>
> <source mode='bind' path='/var/lib/libvirt/qemu/r-2191-VM.agent'/>
> <target type='virtio' name='r-2191-VM.vport'/>
> <address type='virtio-serial'/>
> </channel>
> </devices>
> <memory>262144</memory>
> <devices>
> <memballoon model='none'/>
> </devices>
> <vcpu>1</vcpu>
> <os>
> <type arch='x86_64' machine='pc'>hvm</type>
> <boot dev='cdrom'/>
> <boot dev='hd'/>
> </os>
> <cputune>
> <shares>166</shares>
> </cputune>
> <cpu></cpu><on_reboot>restart</on_reboot>
> <on_poweroff>destroy</on_poweroff>
> <on_crash>destroy</on_crash>
> </domain>
>
> 2015-03-10 16:44:11,467{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-2191-VM -p %vpccidr=
> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
>
> 2015-03-10 16:44:11,494{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Execution is successful.
> 2015-03-10 16:44:11,495{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:44:14,501{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor] (Thread-10:)
> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in libvirt,
> continuing
> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor] (Thread-10:)
> Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
> 2015-03-10 16:44:17,019{GMT} DEBUG [kvm.resource.KVMHAMonitor] (Thread-10:)
> Execution is successful.
> 2015-03-10 16:44:17,953{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
> Sending ping: Seq 788-8: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags: 11,
> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net"}},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":788,"wait":0}}]
> }
> 2015-03-10 16:44:17,964{GMT} DEBUG [cloud.agent.Agent] (Agent-Handler-3:)
> Received response: Seq 788-8: { Ans: , MgmtId: 161344838950, via: 788,
> Ver: v1, Flags: 100010,
> [{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
> }
> 2015-03-10 16:44:19,502{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:44:22,509{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:44:27,510{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Unable to logon to 169.254.2.99
> 2015-03-10 16:44:27,512{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-2191-VM -p %vpccidr=
> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
>
> 2015-03-10 16:44:27,540{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Execution is successful.
> 2015-03-10 16:44:27,541{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:44:30,548{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:44:31,378{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:) Processing command:
> com.cloud.agent.api.GetHostStatsCommand
> 2015-03-10 16:44:31,379{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:) Executing: /bin/bash -c idle=$(top -b -n 1|grep
> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
> 2015-03-10 16:44:31,910{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:) Execution is successful.
> 2015-03-10 16:44:31,911{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:) Executing: /bin/bash -c freeMem=$(free|grep
> cache:|awk '{print $4}');echo $freeMem
> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:) Execution is successful.
> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:) Executing: /bin/bash -c free|grep Mem:|awk
> '{print $2}'
> 2015-03-10 16:44:31,920{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:) Execution is successful.
> 2015-03-10 16:44:35,549{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:44:38,556{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:44:43,557{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Unable to logon to 169.254.2.99
> 2015-03-10 16:44:43,557{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-2191-VM -p %vpccidr=
> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
>
> 2015-03-10 16:44:43,587{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Execution is successful.
> 2015-03-10 16:44:43,588{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:44:46,594{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:44:49,449{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:) Processing command:
> com.cloud.agent.api.GetVmStatsCommand
> 2015-03-10 16:44:51,595{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:44:54,600{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:44:59,601{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Unable to logon to 169.254.2.99
> 2015-03-10 16:44:59,602{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-2191-VM -p %vpccidr=
> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
>
> 2015-03-10 16:44:59,628{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Execution is successful.
> 2015-03-10 16:44:59,628{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:45:02,634{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:45:07,635{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:45:10,640{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:45:12,066{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:) Processing command:
> com.cloud.agent.api.GetStorageStatsCommand
> 2015-03-10 16:45:15,641{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Unable to logon to 169.254.2.99
> 2015-03-10 16:45:15,641{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-2191-VM -p %vpccidr=
> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
>
> 2015-03-10 16:45:15,667{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Execution is successful.
> 2015-03-10 16:45:15,667{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:45:17,021{GMT} DEBUG [kvm.resource.KVMHAMonitor] (Thread-11:)
> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in libvirt,
> continuing
> 2015-03-10 16:45:17,022{GMT} DEBUG [kvm.resource.KVMHAMonitor] (Thread-11:)
> Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
> 2015-03-10 16:45:17,036{GMT} DEBUG [kvm.resource.KVMHAMonitor] (Thread-11:)
> Execution is successful.
> 2015-03-10 16:45:17,951{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
> Sending ping: Seq 788-9: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags: 11,
> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net"}},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":788,"wait":0}}]
> }
> 2015-03-10 16:45:18,002{GMT} DEBUG [cloud.agent.Agent] (Agent-Handler-2:)
> Received response: Seq 788-9: { Ans: , MgmtId: 161344838950, via: 788,
> Ver: v1, Flags: 100010,
> [{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
> }
> 2015-03-10 16:45:18,672{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:45:23,673{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:45:26,680{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:45:31,682{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Unable to logon to 169.254.2.99
> 2015-03-10 16:45:31,683{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-2191-VM -p %vpccidr=
> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
>
> 2015-03-10 16:45:36,684{GMT} WARN [kvm.resource.LibvirtComputingResource]
> (Script-5:) Interrupting script.
> 2015-03-10 16:45:36,686{GMT} WARN [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) Timed out:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-2191-VM -p %vpccidr=
> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
> . Output is:
> 2015-03-10 16:45:36,686{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-1:) passcmd failed:timeout
> 2015-03-10 16:45:36,686{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:45:37,436{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:) Processing command:
> com.cloud.agent.api.GetHostStatsCommand
> 2015-03-10 16:45:37,436{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Executing: /bin/bash -c idle=$(top -b -n 1|grep
> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
> 2015-03-10 16:45:37,967{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Execution is successful.
> 2015-03-10 16:45:37,968{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Executing: /bin/bash -c freeMem=$(free|grep
> cache:|awk '{print $4}');echo $freeMem
> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Execution is successful.
> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Executing: /bin/bash -c free|grep Mem:|awk
> '{print $2}'
> 2015-03-10 16:45:37,977{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Execution is successful.
> 2015-03-10 16:45:39,692{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:45:44,693{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Trying to connect to 169.254.2.99
> 2015-03-10 16:45:47,700{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Could not connect to 169.254.2.99
> 2015-03-10 16:45:51,807{GMT} DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:) Processing command:
> com.cloud.agent.api.GetVmStatsCommand
> 2015-03-10 16:45:52,701{GMT} DEBUG
> [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
> Unable to logon to 169.254.2.99
>
> On 10 March 2015 at 17:50, ilya musayev <il...@gmail.com>
> wrote:
>
>> What kernel are you running? Please include Libvirt and Qemu version.
>>
>>
>> On 3/10/15 8:21 AM, Andrija Panic wrote:
>>
>>> Hi,
>>>
>>> we are having issues from time to time with starting VR - while creating
>>> VPC.
>>>
>>> Here is the few lines from MGMT log:
>>>
>>> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
>>> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start vpc [VPC
>>> [621-VPC-delete] due to
>>> com.cloud.exception.AgentUnavailableException: Resource [Host:788] is
>>> unreachable: Host 788: Unable to start instance due to Unable to start
>>> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not retrying
>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
>>> VirtualMachineManagerImpl.java:1072)
>>>
>>>
>>> The interesting part is on the Agent itself:
>>>
>>>
>>> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.
>>> LibvirtComputingResource]
>>> (agentRequest-Handler-2:) Timed out:
>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>>> -n
>>> r-2189-VM -p %vpccidr=
>>> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%
>>> template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=
>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> . Output is:
>>>
>>>
>>> Interesting stuff is - it's seems somewhat random, 3-4 out of 5 works.
>>>
>>> Any suggestion are really appreciated, since this is live system :(
>>> ACS 4.3.0, KVM, CEPH
>>>
>>> Thanks,
>>>
>>
>>
>
>
> --
>
> Andrija Panić
Re: [HELP] VR sometimes fail to start
Posted by Andrija Panic <an...@gmail.com>.
Hi guys,
Kernel is Elrepo 3.10.67 (CentOS 6.6)
Libvirt manually compiled 1.2.3 with RBD support.
Qemu 0.12.1.2 - Inktank patched original RHEL 6.4 rpms with RBD support.
And here is the output, with debug mode, from Agent.
At the begining I see agent deleting routes for 169.254.x.x network...:
Basically, It seems like the
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl is
failing, not being able to connect to socket...
And I could not ping VR (169.254.2.99)...
LOG:
2015-03-10 16:43:46,976{GMT} DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:) Processing command:
com.cloud.agent.api.GetVmStatsCommand
2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:) Request:Seq 788-1319632926: { Cmd , MgmtId:
161344838950, via: 788, Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.StartCommand":{"vm":{"id":2191,"name":"r-2191-VM","type":"DomainRouter","cpus":1,"minSpeed":166,"maxSpeed":1000,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian
GNU/Linux 7(64-bit)","bootArgs":" vpccidr=10.0.0.0/8
domain=cs2cloud.internal dns1=8.8.8.8 dns2= template=domP name=r-2191-VM
eth0ip=169.254.2.99 eth0mask=255.255.0.0 type=vpcrouter
disable_rp_filter=true","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"e9e24eb0b1ff8ec2","params":{"memoryOvercommitRatio":"1.0","cpuOvercommitRatio":"6.0"},"uuid":"485587f7-4a04-4d77-bc36-2367f2adf77e","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"d042e003-5278-461d-8a35-3b1261164f93","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
cephmon.domaina.net","path":"cloudstack","port":6789,"url":"RBD://
cephmon.domaina.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6
"}},"name":"ROOT-2191","size":2621440000,"path":"d042e003-5278-461d-8a35-3b1261164f93","volumeId":2857,"vmName":"r-2191-VM","accountId":2,"format":"RAW","id":2857,"deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"d042e003-5278-461d-8a35-3b1261164f93","type":"ROOT","_details":{"managed":"false","storagePort":"6789","storageHost":"
cephmon.domaina.net
","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"uuid":"d27a7aad-2dcc-45c7-bf30-3302d011da66","ip":"169.254.2.99","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:63","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.44.253.180","executeInSequence":false,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.2.99","port":3922,"interval":6,"retries":100,"name":"r-2191-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
router.name
":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"com.cloud.agent.api.PlugNicCommand":{"nic":{"deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid":"fe848dbe-cf4b-473e-8124-62289f86b144","ip":"5.226.146.62","netmask":"255.255.255.128","gateway":"5.226.146.1","mac":"06:f0:d6:00:01:49","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"},"instanceName":"r-2191-VM","vmType":"DomainRouter","wait":0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"ipAddresses":[{"accountId":2,"publicIp":"5.226.146.62","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":false,"broadcastUri":"vlan://untagged","vlanGateway":"5.226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","networkName":"breth1-500"}],"accessDetails":{"router.guest.ip":"5.226.146.62","zone.network.type":"Advanced","
router.name
":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"com.cloud.agent.api.routing.SetSourceNatCommand":{"ipAddress":{"accountId":2,"publicIp":"5.226.146.62","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":false,"broadcastUri":"vlan://untagged","vlanGateway":"5.226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","networkName":"breth1-500"},"add":true,"accessDetails":{"zone.network.type":"Advanced","
router.name":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{}] }
2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:) Processing command:
com.cloud.agent.api.StartCommand
2015-03-10 16:43:58,257{GMT} DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:) Processing command:
com.cloud.agent.api.GetStorageStatsCommand
2015-03-10 16:44:01,747{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
(agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
GNU/Linux 7(64-bit)
2015-03-10 16:44:06,309{GMT} DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:) Processing command:
com.cloud.agent.api.NetworkUsageCommand
2015-03-10 16:44:06,309{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:) Executing:
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
vpc_netusage.sh 169.254.2.1 -l 46.232.180.55 -g
2015-03-10 16:44:06,439{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:) Execution is successful.
2015-03-10 16:44:06,441{GMT} DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:) Seq 788-1319632928: { Ans: , MgmtId:
161344838950, via: 788, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-1873-VM","bytesSent":198379,"bytesReceived":7380903,"result":true,"details":"","wait":0}}]
}
2015-03-10 16:44:11,055{GMT} DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-1:) nic=[Nic:Control-169.254.2.99-null]
2015-03-10 16:44:11,055{GMT} DEBUG [utils.script.Script]
(agentRequest-Handler-1:) Executing: /bin/bash -c ip route | grep
169.254.0.0/16
2015-03-10 16:44:11,060{GMT} DEBUG [utils.script.Script]
(agentRequest-Handler-1:) Execution is successful.
2015-03-10 16:44:11,061{GMT} DEBUG [utils.script.Script]
(agentRequest-Handler-1:) Executing: /bin/bash -c ip route del
169.254.0.0/16
2015-03-10 16:44:11,064{GMT} DEBUG [utils.script.Script]
(agentRequest-Handler-1:) Execution is successful.
2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
(agentRequest-Handler-1:) Can't find the mapping of guest os: Debian
GNU/Linux 7(64-bit)
2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:) starting r-2191-VM: <domain type='kvm'>
<name>r-2191-VM</name>
<uuid>485587f7-4a04-4d77-bc36-2367f2adf77e</uuid>
<description>Debian GNU/Linux 7(64-bit)</description>
<clock offset='utc'>
</clock>
<features>
<pae/>
<apic/>
<acpi/>
</features>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<interface type='bridge'>
<source bridge='cloud0'/>
<mac address='0e:00:a9:fe:02:63'/>
<model type='virtio'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<graphics type='vnc' autoport='yes' listen='10.44.253.180'
passwd='e9e24eb0b1ff8ec2'/>
<disk device='disk' type='network'>
<driver name='qemu' type='raw' cache='none' />
<source protocol='rbd'
name='cloudstack/d042e003-5278-461d-8a35-3b1261164f93'>
<host name='cephmon.domain.internal' port='6789'/>
</source>
<auth username='cloudstack'>
<secret type='ceph' uuid='5b93422e-1a66-353d-88a8-2203f79b1dc6'/>
</auth>
<target dev='vda' bus='virtio'/>
</disk>
<disk device='cdrom' type='file'>
<driver name='qemu' type='raw' cache='none' />
<source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
<target dev='hdc' bus='ide'/>
</disk>
<console type='pty'>
<target port='0'/>
</console>
<input type='tablet' bus='usb'/>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/r-2191-VM.agent'/>
<target type='virtio' name='r-2191-VM.vport'/>
<address type='virtio-serial'/>
</channel>
</devices>
<memory>262144</memory>
<devices>
<memballoon model='none'/>
</devices>
<vcpu>1</vcpu>
<os>
<type arch='x86_64' machine='pc'>hvm</type>
<boot dev='cdrom'/>
<boot dev='hd'/>
</os>
<cputune>
<shares>166</shares>
</cputune>
<cpu></cpu><on_reboot>restart</on_reboot>
<on_poweroff>destroy</on_poweroff>
<on_crash>destroy</on_crash>
</domain>
2015-03-10 16:44:11,467{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:) Executing:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
r-2191-VM -p %vpccidr=
10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
2015-03-10 16:44:11,494{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:) Execution is successful.
2015-03-10 16:44:11,495{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Trying to connect to 169.254.2.99
2015-03-10 16:44:14,501{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Could not connect to 169.254.2.99
2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor] (Thread-10:)
Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in libvirt,
continuing
2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor] (Thread-10:)
Executing:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
/mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
2015-03-10 16:44:17,019{GMT} DEBUG [kvm.resource.KVMHAMonitor] (Thread-10:)
Execution is successful.
2015-03-10 16:44:17,953{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
Sending ping: Seq 788-8: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net"}},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":788,"wait":0}}]
}
2015-03-10 16:44:17,964{GMT} DEBUG [cloud.agent.Agent] (Agent-Handler-3:)
Received response: Seq 788-8: { Ans: , MgmtId: 161344838950, via: 788,
Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
}
2015-03-10 16:44:19,502{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Trying to connect to 169.254.2.99
2015-03-10 16:44:22,509{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Could not connect to 169.254.2.99
2015-03-10 16:44:27,510{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Unable to logon to 169.254.2.99
2015-03-10 16:44:27,512{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:) Executing:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
r-2191-VM -p %vpccidr=
10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
2015-03-10 16:44:27,540{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:) Execution is successful.
2015-03-10 16:44:27,541{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Trying to connect to 169.254.2.99
2015-03-10 16:44:30,548{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Could not connect to 169.254.2.99
2015-03-10 16:44:31,378{GMT} DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:) Processing command:
com.cloud.agent.api.GetHostStatsCommand
2015-03-10 16:44:31,379{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:) Executing: /bin/bash -c idle=$(top -b -n 1|grep
Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
2015-03-10 16:44:31,910{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:) Execution is successful.
2015-03-10 16:44:31,911{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:) Executing: /bin/bash -c freeMem=$(free|grep
cache:|awk '{print $4}');echo $freeMem
2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:) Execution is successful.
2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:) Executing: /bin/bash -c free|grep Mem:|awk
'{print $2}'
2015-03-10 16:44:31,920{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:) Execution is successful.
2015-03-10 16:44:35,549{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Trying to connect to 169.254.2.99
2015-03-10 16:44:38,556{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Could not connect to 169.254.2.99
2015-03-10 16:44:43,557{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Unable to logon to 169.254.2.99
2015-03-10 16:44:43,557{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:) Executing:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
r-2191-VM -p %vpccidr=
10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
2015-03-10 16:44:43,587{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:) Execution is successful.
2015-03-10 16:44:43,588{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Trying to connect to 169.254.2.99
2015-03-10 16:44:46,594{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Could not connect to 169.254.2.99
2015-03-10 16:44:49,449{GMT} DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:) Processing command:
com.cloud.agent.api.GetVmStatsCommand
2015-03-10 16:44:51,595{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Trying to connect to 169.254.2.99
2015-03-10 16:44:54,600{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Could not connect to 169.254.2.99
2015-03-10 16:44:59,601{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Unable to logon to 169.254.2.99
2015-03-10 16:44:59,602{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:) Executing:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
r-2191-VM -p %vpccidr=
10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
2015-03-10 16:44:59,628{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:) Execution is successful.
2015-03-10 16:44:59,628{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Trying to connect to 169.254.2.99
2015-03-10 16:45:02,634{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Could not connect to 169.254.2.99
2015-03-10 16:45:07,635{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Trying to connect to 169.254.2.99
2015-03-10 16:45:10,640{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Could not connect to 169.254.2.99
2015-03-10 16:45:12,066{GMT} DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:) Processing command:
com.cloud.agent.api.GetStorageStatsCommand
2015-03-10 16:45:15,641{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Unable to logon to 169.254.2.99
2015-03-10 16:45:15,641{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:) Executing:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
r-2191-VM -p %vpccidr=
10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
2015-03-10 16:45:15,667{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:) Execution is successful.
2015-03-10 16:45:15,667{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Trying to connect to 169.254.2.99
2015-03-10 16:45:17,021{GMT} DEBUG [kvm.resource.KVMHAMonitor] (Thread-11:)
Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in libvirt,
continuing
2015-03-10 16:45:17,022{GMT} DEBUG [kvm.resource.KVMHAMonitor] (Thread-11:)
Executing:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
/mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
2015-03-10 16:45:17,036{GMT} DEBUG [kvm.resource.KVMHAMonitor] (Thread-11:)
Execution is successful.
2015-03-10 16:45:17,951{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
Sending ping: Seq 788-9: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
"},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net"}},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":788,"wait":0}}]
}
2015-03-10 16:45:18,002{GMT} DEBUG [cloud.agent.Agent] (Agent-Handler-2:)
Received response: Seq 788-9: { Ans: , MgmtId: 161344838950, via: 788,
Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
}
2015-03-10 16:45:18,672{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Could not connect to 169.254.2.99
2015-03-10 16:45:23,673{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Trying to connect to 169.254.2.99
2015-03-10 16:45:26,680{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Could not connect to 169.254.2.99
2015-03-10 16:45:31,682{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Unable to logon to 169.254.2.99
2015-03-10 16:45:31,683{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:) Executing:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
r-2191-VM -p %vpccidr=
10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
2015-03-10 16:45:36,684{GMT} WARN [kvm.resource.LibvirtComputingResource]
(Script-5:) Interrupting script.
2015-03-10 16:45:36,686{GMT} WARN [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:) Timed out:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
r-2191-VM -p %vpccidr=
10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
. Output is:
2015-03-10 16:45:36,686{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:) passcmd failed:timeout
2015-03-10 16:45:36,686{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Trying to connect to 169.254.2.99
2015-03-10 16:45:37,436{GMT} DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:) Processing command:
com.cloud.agent.api.GetHostStatsCommand
2015-03-10 16:45:37,436{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:) Executing: /bin/bash -c idle=$(top -b -n 1|grep
Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
2015-03-10 16:45:37,967{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:) Execution is successful.
2015-03-10 16:45:37,968{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:) Executing: /bin/bash -c freeMem=$(free|grep
cache:|awk '{print $4}');echo $freeMem
2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:) Execution is successful.
2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:) Executing: /bin/bash -c free|grep Mem:|awk
'{print $2}'
2015-03-10 16:45:37,977{GMT} DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:) Execution is successful.
2015-03-10 16:45:39,692{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Could not connect to 169.254.2.99
2015-03-10 16:45:44,693{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Trying to connect to 169.254.2.99
2015-03-10 16:45:47,700{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Could not connect to 169.254.2.99
2015-03-10 16:45:51,807{GMT} DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:) Processing command:
com.cloud.agent.api.GetVmStatsCommand
2015-03-10 16:45:52,701{GMT} DEBUG
[resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-1:)
Unable to logon to 169.254.2.99
On 10 March 2015 at 17:50, ilya musayev <il...@gmail.com>
wrote:
> What kernel are you running? Please include Libvirt and Qemu version.
>
>
> On 3/10/15 8:21 AM, Andrija Panic wrote:
>
>> Hi,
>>
>> we are having issues from time to time with starting VR - while creating
>> VPC.
>>
>> Here is the few lines from MGMT log:
>>
>> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
>> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start vpc [VPC
>> [621-VPC-delete] due to
>> com.cloud.exception.AgentUnavailableException: Resource [Host:788] is
>> unreachable: Host 788: Unable to start instance due to Unable to start
>> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not retrying
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
>> VirtualMachineManagerImpl.java:1072)
>>
>>
>> The interesting part is on the Agent itself:
>>
>>
>> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.
>> LibvirtComputingResource]
>> (agentRequest-Handler-2:) Timed out:
>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
>> -n
>> r-2189-VM -p %vpccidr=
>> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%
>> template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=
>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>> . Output is:
>>
>>
>> Interesting stuff is - it's seems somewhat random, 3-4 out of 5 works.
>>
>> Any suggestion are really appreciated, since this is live system :(
>> ACS 4.3.0, KVM, CEPH
>>
>> Thanks,
>>
>
>
--
Andrija Panić
Re: [HELP] VR sometimes fail to start
Posted by ilya musayev <il...@gmail.com>.
What kernel are you running? Please include Libvirt and Qemu version.
On 3/10/15 8:21 AM, Andrija Panic wrote:
> Hi,
>
> we are having issues from time to time with starting VR - while creating
> VPC.
>
> Here is the few lines from MGMT log:
>
> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start vpc [VPC
> [621-VPC-delete] due to
> com.cloud.exception.AgentUnavailableException: Resource [Host:788] is
> unreachable: Host 788: Unable to start instance due to Unable to start
> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not retrying
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1072)
>
>
> The interesting part is on the Agent itself:
>
>
> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Timed out:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-2189-VM -p %vpccidr=
> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
> . Output is:
>
>
> Interesting stuff is - it's seems somewhat random, 3-4 out of 5 works.
>
> Any suggestion are really appreciated, since this is live system :(
> ACS 4.3.0, KVM, CEPH
>
> Thanks,
Re: [HELP] VR sometimes fail to start
Posted by Nux! <nu...@li.nux.ro>.
Hi,
I think more logs are needed, in DEBUG mode if you can.
Also agent logs and perhaps libvirt logs, just in case.
Lucian
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Andrija Panic" <an...@gmail.com>
> To: dev@cloudstack.apache.org, users@cloudstack.apache.org
> Sent: Tuesday, 10 March, 2015 15:21:24
> Subject: [HELP] VR sometimes fail to start
> Hi,
>
> we are having issues from time to time with starting VR - while creating
> VPC.
>
> Here is the few lines from MGMT log:
>
> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start vpc [VPC
> [621-VPC-delete] due to
> com.cloud.exception.AgentUnavailableException: Resource [Host:788] is
> unreachable: Host 788: Unable to start instance due to Unable to start
> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not retrying
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1072)
>
>
> The interesting part is on the Agent itself:
>
>
> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Timed out:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-2189-VM -p %vpccidr=
> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
> . Output is:
>
>
> Interesting stuff is - it's seems somewhat random, 3-4 out of 5 works.
>
> Any suggestion are really appreciated, since this is live system :(
> ACS 4.3.0, KVM, CEPH
>
> Thanks,
> --
>
> Andrija Panić
Re: [HELP] VR sometimes fail to start
Posted by Nux! <nu...@li.nux.ro>.
Hi,
I think more logs are needed, in DEBUG mode if you can.
Also agent logs and perhaps libvirt logs, just in case.
Lucian
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Andrija Panic" <an...@gmail.com>
> To: dev@cloudstack.apache.org, users@cloudstack.apache.org
> Sent: Tuesday, 10 March, 2015 15:21:24
> Subject: [HELP] VR sometimes fail to start
> Hi,
>
> we are having issues from time to time with starting VR - while creating
> VPC.
>
> Here is the few lines from MGMT log:
>
> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start vpc [VPC
> [621-VPC-delete] due to
> com.cloud.exception.AgentUnavailableException: Resource [Host:788] is
> unreachable: Host 788: Unable to start instance due to Unable to start
> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not retrying
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1072)
>
>
> The interesting part is on the Agent itself:
>
>
> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:) Timed out:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-2189-VM -p %vpccidr=
> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
> . Output is:
>
>
> Interesting stuff is - it's seems somewhat random, 3-4 out of 5 works.
>
> Any suggestion are really appreciated, since this is live system :(
> ACS 4.3.0, KVM, CEPH
>
> Thanks,
> --
>
> Andrija Panić