You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Serg Senko <ke...@gmail.com> on 2014/04/11 23:53:38 UTC
KVM, Re-create VR failed
Hi,
It's can be some know bug?
Possible it's already solved in new releases of CS but i need the
work-around or fix before upgrade or reference to bug id.
Environment:
CS 4.1.1
libvirt-1.0.1
qemu-kvm-1.2
NFS Storage ( as primary for VR's )
Advanced VLAN isolation
After hypervisor host crashing, one of VR's has failed to start in failover
case,
I have stopped it through UI with force, then was removed the VR for
re-create it again by start/create VM API call.
Try to start the Instance associated with this network, but failed because
the VR can't be started when newly created.
cloudstack-agent:
2014-04-11 07:05:34,546 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
2014-04-11 07:05:34,547 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
2014-04-11 07:05:34,548 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
2014-04-11 07:05:34,548 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Executing:
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
destroy_network_rules_for_vm --vmname r-377-VM
2014-04-11 07:05:34,663 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Execution is successful.
2014-04-11 07:05:34,664 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Try to stop the vm at first
2014-04-11 07:05:34,665 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to stop VM :r-377-VM :
org.libvirt.LibvirtException: Domain not found: no domain with matching
uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.processError(Unknown Source)
at org.libvirt.Connect.domainLookupByUUIDString(Unknown Source)
at org.libvirt.Connect.domainLookupByUUID(Unknown Source)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.stopVM(LibvirtComputingResource.java:4021)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.stopVM(LibvirtComputingResource.java:3970)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2894)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1032)
at com.cloud.agent.Agent.processRequest(Agent.java:525)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
at com.cloud.utils.nio.Task.run(Task.java:83)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2014-04-11 07:05:34,666 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
2014-04-11 07:05:34,667 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
2014-04-11 07:05:34,668 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
Management CS:
2014-04-11 07:05:40,503 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(Job-Executor-114:job-3001) Found 5 ip(s) to apply as a part of domR
VM[DomainRouter|r-377-VM] start.
2014-04-11 07:05:40,528 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(Job-Executor-114:job-3001) Resending ipAssoc, port forwarding, load
balancing rules as a part of Virtual router start
2014-04-11 07:05:40,542 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(Job-Executor-114:job-3001) Found 1 firewall Egress rule(s) to apply as a
part of domR VM[DomainRouter|r-377-VM] start.
2014-04-11 07:05:40,581 ERROR [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-114:job-3001) Failed to start instance
VM[DomainRouter|r-377-VM]
java.lang.NullPointerException
at
com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
at
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
at
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
at
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175)
at
com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518)
at
com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1434)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:1596)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:746)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:212)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3871)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2579)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:120)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2014-04-11 07:05:40,584 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-114:job-3001) Cleaning up resources for the vm
VM[DomainRouter|r-377-VM] in Starting state
2014-04-11 07:05:40,585 DEBUG [agent.transport.Request]
(Job-Executor-114:job-3001) Seq 5-187434248: Sending { Cmd , MgmtId:
66290989385104, via: 5, Ver: v1, Flags: 100111,
[{"StopCommand":{"isProxy":false,"vmName":"r-377-VM","wait":0}}] }
2014-04-11 07:05:40,707 DEBUG [agent.transport.Request]
(AgentManager-Handler-16:null) Seq 5-187434248: Processing: { Ans: ,
MgmtId: 66290989385104, via: 5, Ver: v1, Flags: 110,
[{"StopAnswer":{"vncPort":0,"result":true,"wait":0}}] }
2014-04-11 07:05:40,707 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-16:null) Seq 5-187434248: No more commands found
2014-04-11 07:05:40,707 DEBUG [agent.transport.Request]
(Job-Executor-114:job-3001) Seq 5-187434248: Received: { Ans: , MgmtId:
66290989385104, via: 5, Ver: v1, Flags: 110, { StopAnswer } }
2014-04-11 07:05:40,710 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(Job-Executor-114:job-3001) Successfully updated user statistics as a part
of domR VM[DomainRouter|r-377-VM] reboot/stop
2014-04-11 07:05:40,718 DEBUG [cloud.network.NetworkModelImpl]
(Job-Executor-114:job-3001) Service SecurityGroup is not supported in the
network id=204
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking NiciraNvp to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,722 DEBUG [network.element.NiciraNvpElement]
(Job-Executor-114:job-3001) Checking if NiciraNvpElement can handle service
Connectivity on network inewdate
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking JuniperSRX to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking Netscaler to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking F5BigIP to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking CiscoNexus1000vVSM to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking BigSwitchVnsElement to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,722 DEBUG [network.element.BigSwitchVnsElement]
(Job-Executor-114:job-3001) Checking if BigSwitchVnsElement can handle
service Connectivity on network inewdate
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking VirtualRouter to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking Ovs to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking SecurityGroupProvider to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking VpcVirtualRouter to release
Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
2014-04-11 07:05:40,724 DEBUG [network.guru.ControlNetworkGuru]
(Job-Executor-114:job-3001) Released nic: NicProfile[491-377-null-null-null
2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking NiciraNvp to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,725 DEBUG [network.element.NiciraNvpElement]
(Job-Executor-114:job-3001) Checking if NiciraNvpElement can handle service
Connectivity on network null
2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking JuniperSRX to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking Netscaler to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking F5BigIP to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking CiscoNexus1000vVSM to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking BigSwitchVnsElement to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,726 DEBUG [network.element.BigSwitchVnsElement]
(Job-Executor-114:job-3001) Checking if BigSwitchVnsElement can handle
service Connectivity on network null
2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking VirtualRouter to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking Ovs to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking SecurityGroupProvider to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-114:job-3001) Asking VpcVirtualRouter to release
Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
2014-04-11 07:05:40,728 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-114:job-3001) Successfully released network resources for the
vm VM[DomainRouter|r-377-VM]
2014-04-11 07:05:40,728 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-114:job-3001) Successfully cleanued up resources for the vm
VM[DomainRouter|r-377-VM] in Starting state
2014-04-11 07:05:40,731 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-114:job-3001) VM state transitted from :Starting to Stopped
with event: OperationFailedvm's original host id: null new host id: null
host id before state transition: 5
Someone know such problem? Or help me to debug it please.
--
ttyv0 "/usr/libexec/gmail Pc" webcons on secure
Re: KVM, Re-create VR failed
Posted by Marcus <sh...@gmail.com>.
Your agent snippet just looks like the system trying to stop the vm.
If a vm fails to start, it will also run through the stop routine to
clean up all of the prework, so the 'failed to stop' debug is all
normal. You may need to go above and look at why it failed to start.
On Fri, Apr 11, 2014 at 3:53 PM, Serg Senko <ke...@gmail.com> wrote:
> Hi,
>
> It's can be some know bug?
> Possible it's already solved in new releases of CS but i need the
> work-around or fix before upgrade or reference to bug id.
>
> Environment:
> CS 4.1.1
> libvirt-1.0.1
> qemu-kvm-1.2
> NFS Storage ( as primary for VR's )
> Advanced VLAN isolation
>
> After hypervisor host crashing, one of VR's has failed to start in failover
> case,
> I have stopped it through UI with force, then was removed the VR for
> re-create it again by start/create VM API call.
>
>
> Try to start the Instance associated with this network, but failed because
> the VR can't be started when newly created.
>
> cloudstack-agent:
>
> 2014-04-11 07:05:34,546 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
> 2014-04-11 07:05:34,547 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
> 2014-04-11 07:05:34,548 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
> 2014-04-11 07:05:34,548 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Executing:
> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> destroy_network_rules_for_vm --vmname r-377-VM
>
> 2014-04-11 07:05:34,663 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Execution is successful.
>
> 2014-04-11 07:05:34,664 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Try to stop the vm at first
>
> 2014-04-11 07:05:34,665 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to stop VM :r-377-VM :
>
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
> at org.libvirt.ErrorHandler.processError(Unknown Source)
>
> at org.libvirt.Connect.processError(Unknown Source)
>
> at org.libvirt.Connect.domainLookupByUUIDString(Unknown Source)
>
> at org.libvirt.Connect.domainLookupByUUID(Unknown Source)
>
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.stopVM(LibvirtComputingResource.java:4021)
>
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.stopVM(LibvirtComputingResource.java:3970)
>
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2894)
>
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1032)
>
> at com.cloud.agent.Agent.processRequest(Agent.java:525)
>
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
>
> at com.cloud.utils.nio.Task.run(Task.java:83)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>
> at java.lang.Thread.run(Thread.java:679)
>
> 2014-04-11 07:05:34,666 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
> domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
> 2014-04-11 07:05:34,667 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
> domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
> 2014-04-11 07:05:34,668 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
> domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
>
>
>
> Management CS:
>
> 2014-04-11 07:05:40,503 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-114:job-3001) Found 5 ip(s) to apply as a part of domR
> VM[DomainRouter|r-377-VM] start.
>
> 2014-04-11 07:05:40,528 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-114:job-3001) Resending ipAssoc, port forwarding, load
> balancing rules as a part of Virtual router start
>
> 2014-04-11 07:05:40,542 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-114:job-3001) Found 1 firewall Egress rule(s) to apply as a
> part of domR VM[DomainRouter|r-377-VM] start.
>
> 2014-04-11 07:05:40,581 ERROR [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-114:job-3001) Failed to start instance
> VM[DomainRouter|r-377-VM]
>
> java.lang.NullPointerException
>
> at
> com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>
> at
> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>
> at
> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
>
> at
> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175)
>
> at
> com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518)
>
> at
> com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1434)
>
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>
> at
> com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:1596)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:746)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
>
> at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:212)
>
> at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
>
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3871)
>
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2579)
>
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>
> at
> org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:120)
>
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162)
>
> at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>
> at java.lang.Thread.run(Thread.java:679)
>
> 2014-04-11 07:05:40,584 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-114:job-3001) Cleaning up resources for the vm
> VM[DomainRouter|r-377-VM] in Starting state
>
> 2014-04-11 07:05:40,585 DEBUG [agent.transport.Request]
> (Job-Executor-114:job-3001) Seq 5-187434248: Sending { Cmd , MgmtId:
> 66290989385104, via: 5, Ver: v1, Flags: 100111,
> [{"StopCommand":{"isProxy":false,"vmName":"r-377-VM","wait":0}}] }
>
> 2014-04-11 07:05:40,707 DEBUG [agent.transport.Request]
> (AgentManager-Handler-16:null) Seq 5-187434248: Processing: { Ans: ,
> MgmtId: 66290989385104, via: 5, Ver: v1, Flags: 110,
> [{"StopAnswer":{"vncPort":0,"result":true,"wait":0}}] }
>
> 2014-04-11 07:05:40,707 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-16:null) Seq 5-187434248: No more commands found
>
> 2014-04-11 07:05:40,707 DEBUG [agent.transport.Request]
> (Job-Executor-114:job-3001) Seq 5-187434248: Received: { Ans: , MgmtId:
> 66290989385104, via: 5, Ver: v1, Flags: 110, { StopAnswer } }
>
> 2014-04-11 07:05:40,710 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-114:job-3001) Successfully updated user statistics as a part
> of domR VM[DomainRouter|r-377-VM] reboot/stop
>
> 2014-04-11 07:05:40,718 DEBUG [cloud.network.NetworkModelImpl]
> (Job-Executor-114:job-3001) Service SecurityGroup is not supported in the
> network id=204
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking NiciraNvp to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [network.element.NiciraNvpElement]
> (Job-Executor-114:job-3001) Checking if NiciraNvpElement can handle service
> Connectivity on network inewdate
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking JuniperSRX to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking Netscaler to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking F5BigIP to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking CiscoNexus1000vVSM to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking BigSwitchVnsElement to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [network.element.BigSwitchVnsElement]
> (Job-Executor-114:job-3001) Checking if BigSwitchVnsElement can handle
> service Connectivity on network inewdate
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking VirtualRouter to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking Ovs to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking SecurityGroupProvider to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking VpcVirtualRouter to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,724 DEBUG [network.guru.ControlNetworkGuru]
> (Job-Executor-114:job-3001) Released nic: NicProfile[491-377-null-null-null
>
> 2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking NiciraNvp to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,725 DEBUG [network.element.NiciraNvpElement]
> (Job-Executor-114:job-3001) Checking if NiciraNvpElement can handle service
> Connectivity on network null
>
> 2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking JuniperSRX to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking Netscaler to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking F5BigIP to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking CiscoNexus1000vVSM to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking BigSwitchVnsElement to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,726 DEBUG [network.element.BigSwitchVnsElement]
> (Job-Executor-114:job-3001) Checking if BigSwitchVnsElement can handle
> service Connectivity on network null
>
> 2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking VirtualRouter to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking Ovs to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking SecurityGroupProvider to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking VpcVirtualRouter to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,728 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-114:job-3001) Successfully released network resources for the
> vm VM[DomainRouter|r-377-VM]
>
> 2014-04-11 07:05:40,728 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-114:job-3001) Successfully cleanued up resources for the vm
> VM[DomainRouter|r-377-VM] in Starting state
>
> 2014-04-11 07:05:40,731 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-114:job-3001) VM state transitted from :Starting to Stopped
> with event: OperationFailedvm's original host id: null new host id: null
> host id before state transition: 5
>
>
> Someone know such problem? Or help me to debug it please.
>
>
>
> --
> ttyv0 "/usr/libexec/gmail Pc" webcons on secure
Re: KVM, Re-create VR failed
Posted by Marcus <sh...@gmail.com>.
Your agent snippet just looks like the system trying to stop the vm.
If a vm fails to start, it will also run through the stop routine to
clean up all of the prework, so the 'failed to stop' debug is all
normal. You may need to go above and look at why it failed to start.
On Fri, Apr 11, 2014 at 3:53 PM, Serg Senko <ke...@gmail.com> wrote:
> Hi,
>
> It's can be some know bug?
> Possible it's already solved in new releases of CS but i need the
> work-around or fix before upgrade or reference to bug id.
>
> Environment:
> CS 4.1.1
> libvirt-1.0.1
> qemu-kvm-1.2
> NFS Storage ( as primary for VR's )
> Advanced VLAN isolation
>
> After hypervisor host crashing, one of VR's has failed to start in failover
> case,
> I have stopped it through UI with force, then was removed the VR for
> re-create it again by start/create VM API call.
>
>
> Try to start the Instance associated with this network, but failed because
> the VR can't be started when newly created.
>
> cloudstack-agent:
>
> 2014-04-11 07:05:34,546 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
> 2014-04-11 07:05:34,547 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
> 2014-04-11 07:05:34,548 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
> 2014-04-11 07:05:34,548 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Executing:
> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> destroy_network_rules_for_vm --vmname r-377-VM
>
> 2014-04-11 07:05:34,663 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Execution is successful.
>
> 2014-04-11 07:05:34,664 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Try to stop the vm at first
>
> 2014-04-11 07:05:34,665 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to stop VM :r-377-VM :
>
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
> at org.libvirt.ErrorHandler.processError(Unknown Source)
>
> at org.libvirt.Connect.processError(Unknown Source)
>
> at org.libvirt.Connect.domainLookupByUUIDString(Unknown Source)
>
> at org.libvirt.Connect.domainLookupByUUID(Unknown Source)
>
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.stopVM(LibvirtComputingResource.java:4021)
>
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.stopVM(LibvirtComputingResource.java:3970)
>
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2894)
>
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1032)
>
> at com.cloud.agent.Agent.processRequest(Agent.java:525)
>
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
>
> at com.cloud.utils.nio.Task.run(Task.java:83)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>
> at java.lang.Thread.run(Thread.java:679)
>
> 2014-04-11 07:05:34,666 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
> domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
> 2014-04-11 07:05:34,667 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
> domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
> 2014-04-11 07:05:34,668 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
> domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
>
>
>
> Management CS:
>
> 2014-04-11 07:05:40,503 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-114:job-3001) Found 5 ip(s) to apply as a part of domR
> VM[DomainRouter|r-377-VM] start.
>
> 2014-04-11 07:05:40,528 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-114:job-3001) Resending ipAssoc, port forwarding, load
> balancing rules as a part of Virtual router start
>
> 2014-04-11 07:05:40,542 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-114:job-3001) Found 1 firewall Egress rule(s) to apply as a
> part of domR VM[DomainRouter|r-377-VM] start.
>
> 2014-04-11 07:05:40,581 ERROR [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-114:job-3001) Failed to start instance
> VM[DomainRouter|r-377-VM]
>
> java.lang.NullPointerException
>
> at
> com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>
> at
> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>
> at
> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
>
> at
> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175)
>
> at
> com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518)
>
> at
> com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1434)
>
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>
> at
> com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:1596)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:746)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
>
> at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:212)
>
> at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
>
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3871)
>
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2579)
>
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>
> at
> org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:120)
>
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162)
>
> at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>
> at java.lang.Thread.run(Thread.java:679)
>
> 2014-04-11 07:05:40,584 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-114:job-3001) Cleaning up resources for the vm
> VM[DomainRouter|r-377-VM] in Starting state
>
> 2014-04-11 07:05:40,585 DEBUG [agent.transport.Request]
> (Job-Executor-114:job-3001) Seq 5-187434248: Sending { Cmd , MgmtId:
> 66290989385104, via: 5, Ver: v1, Flags: 100111,
> [{"StopCommand":{"isProxy":false,"vmName":"r-377-VM","wait":0}}] }
>
> 2014-04-11 07:05:40,707 DEBUG [agent.transport.Request]
> (AgentManager-Handler-16:null) Seq 5-187434248: Processing: { Ans: ,
> MgmtId: 66290989385104, via: 5, Ver: v1, Flags: 110,
> [{"StopAnswer":{"vncPort":0,"result":true,"wait":0}}] }
>
> 2014-04-11 07:05:40,707 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-16:null) Seq 5-187434248: No more commands found
>
> 2014-04-11 07:05:40,707 DEBUG [agent.transport.Request]
> (Job-Executor-114:job-3001) Seq 5-187434248: Received: { Ans: , MgmtId:
> 66290989385104, via: 5, Ver: v1, Flags: 110, { StopAnswer } }
>
> 2014-04-11 07:05:40,710 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-114:job-3001) Successfully updated user statistics as a part
> of domR VM[DomainRouter|r-377-VM] reboot/stop
>
> 2014-04-11 07:05:40,718 DEBUG [cloud.network.NetworkModelImpl]
> (Job-Executor-114:job-3001) Service SecurityGroup is not supported in the
> network id=204
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking NiciraNvp to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [network.element.NiciraNvpElement]
> (Job-Executor-114:job-3001) Checking if NiciraNvpElement can handle service
> Connectivity on network inewdate
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking JuniperSRX to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking Netscaler to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking F5BigIP to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking CiscoNexus1000vVSM to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking BigSwitchVnsElement to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [network.element.BigSwitchVnsElement]
> (Job-Executor-114:job-3001) Checking if BigSwitchVnsElement can handle
> service Connectivity on network inewdate
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking VirtualRouter to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking Ovs to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking SecurityGroupProvider to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking VpcVirtualRouter to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,724 DEBUG [network.guru.ControlNetworkGuru]
> (Job-Executor-114:job-3001) Released nic: NicProfile[491-377-null-null-null
>
> 2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking NiciraNvp to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,725 DEBUG [network.element.NiciraNvpElement]
> (Job-Executor-114:job-3001) Checking if NiciraNvpElement can handle service
> Connectivity on network null
>
> 2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking JuniperSRX to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking Netscaler to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking F5BigIP to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking CiscoNexus1000vVSM to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking BigSwitchVnsElement to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,726 DEBUG [network.element.BigSwitchVnsElement]
> (Job-Executor-114:job-3001) Checking if BigSwitchVnsElement can handle
> service Connectivity on network null
>
> 2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking VirtualRouter to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking Ovs to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking SecurityGroupProvider to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking VpcVirtualRouter to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,728 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-114:job-3001) Successfully released network resources for the
> vm VM[DomainRouter|r-377-VM]
>
> 2014-04-11 07:05:40,728 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-114:job-3001) Successfully cleanued up resources for the vm
> VM[DomainRouter|r-377-VM] in Starting state
>
> 2014-04-11 07:05:40,731 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-114:job-3001) VM state transitted from :Starting to Stopped
> with event: OperationFailedvm's original host id: null new host id: null
> host id before state transition: 5
>
>
> Someone know such problem? Or help me to debug it please.
>
>
>
> --
> ttyv0 "/usr/libexec/gmail Pc" webcons on secure