You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Sugandh S <s....@rocketmail.com> on 2014/03/19 09:59:35 UTC
vm stuck in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management
server and Cloudstack-agent. Primary storage and secondary storage are
also provided by this server via NFS. For primary storage, export
location is /export/primary and for secondary storage, it is
/export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
Re: vm stuck in starting state, unable to delete it
Posted by Sugandh S <s....@rocketmail.com>.
Hi,
I followed your instructions and I managed to delete the vms. I restarted the cloudstack services and my virtual router is stuck in starting state. It was initially in stopped state but after hitting play button, it is now like that.
Here is the part of log file:
2014-03-19 15:51:59,669 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-11:null) Seq 4-1161232387: For
warding Seq 4-1161232387: { Cmd , MgmtId: 279278805451040, via: 4, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.Start
Command":{"vm":{"id":4,"name":"r-4-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"
maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-4-VM eth0ip=10.
208.67.224 eth0mask=255.255.254.0 gateway=10.208.66.1 domain=cs1cloud.internal dhcprange=10.208.66.1 eth1ip=169.254.0.22
6 eth1mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=10.10.1.2 dns2=10.10.2.2","rebootOnCrash":false,"enable
HA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8f98cc61ecec605","params":{},"uuid":"305f66
97-1322-4167-b56e-8165c463ae2a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"63d080b2-33
cf-41f8-8c6a-2f796c922c30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid
":"0e1328fc-61df-3bab-9cea-0e5e4796ac5d","id":1,"poolType":"NetworkFilesystem","host":"10.208.66.23","path":"/export/pri
mary","port":2049}},"name":"ROOT-4","size":276162048,"path":"b69b0a65-abd3-4c4b-b574-9568a8847fe0","volumeId":5,"vmName"
:"r-4-VM","accountId":1,"format":"QCOW2","id":5,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId"
:0,"networkRateMbps":200,"defaultNic":true,"uuid":"f4c6a16b-4a12-42fb-9dbe-b084b41a1960","ip":"10.208.67.224","netmask":
"255.255.254.0","gateway":"10.208.66.1","mac":"06:be:74:00:00:19","dns1":"10.10.1.2","dns2":"10.10.2.2","broadcastType":
"Native","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":true},
{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"3cfcc4df-2e17-4fa7-aafd-72b9badc6df9","ip":"169.254.0.226"
,"netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:e2","broadcastType":"LinkLocal","type":"Control",
"isSecurityGroupEnabled":false}]},"hostIp":"10.208.66.24","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.chec
k.CheckSshCommand":{"ip":"169.254.0.226","port":3922,"interval":6,"retries":100,"name":"r-4-VM","wait":0}},{"com.cloud.a
gent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-4-VM","router.ip":"169.254.0.226"},"wait":0}},{}] } to 27
9278805451034
2014-03-19 15:51:59,670 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 4-1161232387: For
warding Seq 4-1161232387: { Cmd , MgmtId: 279278805451040, via: 4, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.Start
Command":{"vm":{"id":4,"name":"r-4-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"
maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-4-VM eth0ip=10.
208.67.224 eth0mask=255.255.254.0 gateway=10.208.66.1 domain=cs1cloud.internal dhcprange=10.208.66.1 eth1ip=169.254.0.22
6 eth1mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=10.10.1.2 dns2=10.10.2.2","rebootOnCrash":false,"enable
HA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8f98cc61ecec605","params":{},"uuid":"305f66
97-1322-4167-b56e-8165c463ae2a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"63d080b2-33
cf-41f8-8c6a-2f796c922c30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid
":"0e1328fc-61df-3bab-9cea-0e5e4796ac5d","id":1,"poolType":"NetworkFilesystem","host":"10.208.66.23","path":"/export/pri
mary","port":2049}},"name":"ROOT-4","size":276162048,"path":"b69b0a65-abd3-4c4b-b574-9568a8847fe0","volumeId":5,"vmName"
:"r-4-VM","accountId":1,"format":"QCOW2","id":5,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId"
:0,"networkRateMbps":200,"defaultNic":true,"uuid":"f4c6a16b-4a12-42fb-9dbe-b084b41a1960","ip":"10.208.67.224","netmask":
"255.255.254.0","gateway":"10.208.66.1","mac":"06:be:74:00:00:19","dns1":"10.10.1.2","dns2":"10.10.2.2","broadcastType":
"Native","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":true},
{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"3cfcc4df-2e17-4fa7-aafd-72b9badc6df9","ip":"169.254.0.226"
,"netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:e2","broadcastType":"LinkLocal","type":"Control",
"isSecurityGroupEnabled":false}]},"hostIp":"10.208.66.24","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.chec
k.CheckSshCommand":{"ip":"169.254.0.226","port":3922,"interval":6,"retries":100,"name":"r-4-VM","wait":0}},{"com.cloud.a
gent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-4-VM","router.ip":"169.254.0.226"},"wait":0}},{}] } to 27
9278805451034
2014-03-19 15:51:59,671 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-14:null) Seq 4-1161232387: Forwarding Seq 4-1161232387: { Cmd , MgmtId: 279278805451040, via: 4, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":4,"name":"r-4-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-4-VM eth0ip=10.208.67.224 eth0mask=255.255.254.0 gateway=10.208.66.1 domain=cs1cloud.internal dhcprange=10.208.66.1 eth1ip=169.254.0.226 eth1mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=10.10.1.2
dns2=10.10.2.2","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8f98cc61ecec605","params":{},"uuid":"305f6697-1322-4167-b56e-8165c463ae2a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"63d080b2-33cf-41f8-8c6a-2f796c922c30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"0e1328fc-61df-3bab-9cea-0e5e4796ac5d","id":1,"poolType":"NetworkFilesystem","host":"10.208.66.23","path":"/export/primary","port":2049}},"name":"ROOT-4","size":276162048,"path":"b69b0a65-abd3-4c4b-b574-9568a8847fe0","volumeId":5,"vmName":"r-4-VM","accountId":1,"format":"QCOW2","id":5,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"f4c6a16b-4a12-42fb-9dbe-b084b41a1960","ip":"10.208.67.224","netmask":"255.255.254.0","gateway":"10.208.66.1","mac":"06:be:74:00:00:19","dns1":"10.10.1
.2","dns2":"10.10.2.2","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":true},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"3cfcc4df-2e17-4fa7-aafd-72b9badc6df9","ip":"169.254.0.226","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:e2","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.208.66.24","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.226","port":3922,"interval":6,"retries":100,"name":"r-4-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-4-VM","router.ip":"169.254.0.226"},"wait":0}},{}] } to 279278805451034
2014-03-19 15:51:59,673 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 4-1161232387: Forwarding Seq 4-1161232387: { Cmd , MgmtId: 279278805451040, via: 4, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":4,"name":"r-4-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-4-VM eth0ip=10.208.67.224 eth0mask=255.255.254.0 gateway=10.208.66.1 domain=cs1cloud.internal dhcprange=10.208.66.1 eth1ip=169.254.0.226 eth1mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=10.10.1.2
dns2=10.10.2.2","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8f98cc61ecec605","params":{},"uuid":"305f6697-1322-4167-b56e-8165c463ae2a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"63d080b2-33cf-41f8-8c6a-2f796c922c30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"0e1328fc-61df-3bab-9cea-0e5e4796ac5d","id":1,"poolType":"NetworkFilesystem","host":"10.208.66.23","path":"/export/primary","port":2049}},"name":"ROOT-4","size":276162048,"path":"b69b0a65-abd3-4c4b-b574-9568a8847fe0","volumeId":5,"vmName":"r-4-VM","accountId":1,"format":"QCOW2","id":5,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"f4c6a16b-4a12-42fb-9dbe-b084b41a1960","ip":"10.208.67.224","netmask":"255.255.254.0","gateway":"10.208.66.1","mac":"06:be:74:00:00:19","dns1":"10.10.1
.2","dns2":"10.10.2.2","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":true},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"3cfcc4df-2e17-4fa7-aafd-72b9badc6df9","ip":"169.254.0.226","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:e2","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.208.66.24","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.226","port":3922,"interval":6,"retries":100,"name":"r-4-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-4-VM","router.ip":"169.254.0.226"},"wait":0}},{}] } to 279278805451034
2014-03-19 15:51:59,674 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 4-1161232387: Forwarding Seq 4-1161232387: { Cmd , MgmtId: 279278805451040, via: 4, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":4,"name":"r-4-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-4-VM eth0ip=10.208.67.224 eth0mask=255.255.254.0 gateway=10.208.66.1 domain=cs1cloud.internal dhcprange=10.208.66.1 eth1ip=169.254.0.226 eth1mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=10.10.1.2
dns2=10.10.2.2","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8f98cc61ecec605","params":{},"uuid":"305f6697-1322-4167-b56e-8165c463ae2a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"63d080b2-33cf-41f8-8c6a-2f796c922c30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"0e1328fc-61df-3bab-9cea-0e5e4796ac5d","id":1,"poolType":"NetworkFilesystem","host":"10.208.66.23","path":"/export/primary","port":2049}},"name":"ROOT-4","size":276162048,"path":"b69b0a65-abd3-4c4b-b574-9568a8847fe0","volumeId":5,"vmName":"r-4-VM","accountId":1,"format":"QCOW2","id":5,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"f4c6a16b-4a12-42fb-9dbe-b084b41a1960","ip":"10.208.67.224","netmask":"255.255.254.0","gateway":"10.208.66.1","mac":"06:be:74:00:00:19","dns1":"10.10.1
.2","dns2":"10.10.2.2","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":true},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"3cfcc4df-2e17-4fa7-aafd-72b9badc6df9","ip":"169.254.0.226","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:e2","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.208.66.24","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.226","port":3922,"interval":6,"retries":100,"name":"r-4-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-4-VM","router.ip":"169.254.0.226"},"wait":0}},{}] } to 279278805451034
2014-03-19 15:51:59,675 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-4:null) Seq 4-1161232387: Forwarding Seq 4-1161232387: { Cmd , MgmtId: 279278805451040, via: 4, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":4,"name":"r-4-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-4-VM eth0ip=10.208.67.224 eth0mask=255.255.254.0 gateway=10.208.66.1 domain=cs1cloud.internal dhcprange=10.208.66.1 eth1ip=169.254.0.226 eth1mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=10.10.1.2
dns2=10.10.2.2","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8f98cc61ecec605","params":{},"uuid":"305f6697-1322-4167-b56e-8165c463ae2a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"63d080b2-33cf-41f8-8c6a-2f796c922c30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"0e1328fc-61df-3bab-9cea-0e5e4796ac5d","id":1,"poolType":"NetworkFilesystem","host":"10.208.66.23","path":"/export/primary","port":2049}},"name":"ROOT-4","size":276162048,"path":"b69b0a65-abd3-4c4b-b574-9568a8847fe0","volumeId":5,"vmName":"r-4-VM","accountId":1,"format":"QCOW2","id":5,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"f4c6a16b-4a12-42fb-9dbe-b084b41a1960","ip":"10.208.67.224","netmask":"255.255.254.0","gateway":"10.208.66.1","mac":"06:be:74:00:00:19","dns1":"10.10.1
.2","dns2":"10.10.2.2","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":true},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"3cfcc4df-2e17-4fa7-aafd-72b9badc6df9","ip":"169.254.0.226","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:e2","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.208.66.24","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.226","port":3922,"interval":6,"retries":100,"name":"r-4-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-4-VM","router.ip":"169.254.0.226"},"wait":0}},{}] } to 279278805451034
2014-03-19 15:51:59,676 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-3:null) Seq 4-1161232387: Forw
arding Seq 4-1161232387: { Cmd , MgmtId: 279278805451040, via: 4, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":4,"name":"r-4-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-4-VM eth0ip=10.208.67.224 eth0mask=255.255.254.0 gateway=10.208.66.1 domain=cs1cloud.internal dhcprange=10.208.66.1 eth1ip=169.254.0.226 eth1mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=10.10.1.2
dns2=10.10.2.2","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8f98cc61ecec605","params":{},"uuid":"305f6697-1322-4167-b56e-8165c463ae2a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"63d080b2-33cf-41f8-8c6a-2f796c922c30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"0e1328fc-61df-3bab-9cea-0e5e4796ac5d","id":1,"poolType":"NetworkFilesystem","host":"10.208.66.23","path":"/export/primary","port":2049}},"name":"ROOT-4","size":276162048,"path":"b69b0a65-abd3-4c4b-b574-9568a8847fe0","volumeId":5,"vmName":"r-4-VM","accountId":1,"format":"QCOW2","id":5,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"f4c6a16b-4a12-42fb-9dbe-b084b41a1960","ip":"10.208.67.224","netmask":"255.255.254.0","gateway":"10.208.66.1","mac":"06:be:74:00:00:19","dns1":"10.10.1
.2","dns2":"10.10.2.2","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":true},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"3cfcc4df-2e17-4fa7-aafd-72b9badc6df9","ip":"169.254.0.226","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:e2","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.208.66.24","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.226","port":3922,"interval":6,"retries":100,"name":"r-4-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-4-VM","router.ip":"169.254.0.226"},"wait":0}},{}] } to 279278805451034
2014-03-19 15:51:59,677 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-6:null) Seq 4-1161232387: Forwarding Seq 4-1161232387: { Cmd , MgmtId: 279278805451040, via: 4, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":4,"name":"r-4-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-4-VM eth0ip=10.208.67.224 eth0mask=255.255.254.0 gateway=10.208.66.1 domain=cs1cloud.internal dhcprange=10.208.66.1 eth1ip=169.254.0.226 eth1mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=10.10.1.2
dns2=10.10.2.2","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8f98cc61ecec605","params":{},"uuid":"305f6697-1322-4167-b56e-8165c463ae2a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"63d080b2-33cf-41f8-8c6a-2f796c922c30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"0e1328fc-61df-3bab-9cea-0e5e4796ac5d","id":1,"poolType":"NetworkFilesystem","host":"10.208.66.23","path":"/export/primary","port":2049}},"name":"ROOT-4","size":276162048,"path":"b69b0a65-abd3-4c4b-b574-9568a8847fe0","volumeId":5,"vmName":"r-4-VM","accountId":1,"format":"QCOW2","id":5,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"f4c6a16b-4a12-42fb-9dbe-b084b41a1960","ip":"10.208.67.224","netmask":"255.255.254.0","gateway":"10.208.66.1","mac":"06:be:74:00:00:19","dns1":"10.10.1
.2","dns2":"10.10.2.2","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":true},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"3cfcc4df-2e17-4fa7-aafd-72b9badc6df9","ip":"169.254.0.226","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:e2","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.208.66.24","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.226","port":3922,"interval":6,"retries":100,"name":"r-4-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-4-VM","router.ip":"169.254.0.226"},"wait":0}},{}] } to 279
278805451034
2014-03-19 15:51:59,678 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-12:null) Seq 4-1161232387: Forwarding Seq 4-1161232387: { Cmd , MgmtId: 279278805451040, via: 4, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":4,"name":"r-4-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-4-VM eth0ip=10.208.67.224 eth0mask=255.255.254.0 gateway=10.208.66.1 domain=cs1cloud.internal dhcprange=10.208.66.1 eth1ip=169.254.0.226 eth1mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=10.10.1.2
dns2=10.10.2.2","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8f98cc61ecec605","params":{},"uuid":"305f6697-1322-4167-b56e-8165c463ae2a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"63d080b2-33cf-41f8-8c6a-2f796c922c30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"0e1328fc-61df-3bab-9cea-0e5e4796ac5d","id":1,"poolType":"NetworkFilesystem","host":"10.208.66.23","path":"/export/primary","port":2049}},"name":"ROOT-4","size":276162048,"path":"b69b0a65-abd3-4c4b-b574-9568a8847fe0","volumeId":5,"vmName":"r-4-VM","accountId":1,"format":"QCOW2","id":5,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"f4c6a16b-4a12-42fb-9dbe-b084b41a1960","ip":"10.208.67.224","netmask":"255.255.254.0","gateway":"10.208.66.1","mac":"06:be:74:00:00:19","dns1":"10.10.1
.2","dns2":"10.10.2.2","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":true},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"3cfcc4df-2e17-4fa7-aafd-72b9badc6df9","ip":"169.254.0.226","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:e2","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.208.66.24","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.226","port":3922,"interval":6,"retries":100,"name":"r-4-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-4-VM","router.ip":"169.254.0.226"},"wait":0}},{}] } to 279278805451034
2014-03-19 15:51:59,679 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 4-1161232387: Forwarding Seq 4-1161232387: { Cmd , MgmtId: 279278805451040, via: 4, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":4,"name":"r-4-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-4-VM eth0ip=10.208.67.224 eth0mask=255.255.254.0 gateway=10.208.66.1 domain=cs1cloud.internal dhcprange=10.208.66.1 eth1ip=169.254.0.226 eth1mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=10.10.1.2
dns2=10.10.2.2","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8f98cc61ecec605","params":{},"uuid":"305f6697-1322-4167-b56e-8165c463ae2a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"63d080b2-33cf-41f8-8c6a-2f796c922c30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"0e1328fc-61df-3bab-9cea-0e5e4796ac5d","id":1,"poolType":"NetworkFilesystem","host":"10.208.66.23","path":"/export/primary","port":2049}},"name":"ROOT-4","size":276162048,"path":"b69b0a65-abd3-4c4b-b574-9568a8847fe0","volumeId":5,"vmName":"r-4-VM","accountId":1,"format":"QCOW2","id":5,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"f4c6a16b-4a12-42fb-9dbe-b084b41a1960","ip":"10.208.67.224","netmask":"255.255.254.0","gateway":"10.208.66.1","mac":"06:be:74:00:00:19","dns1":"10.10.1
.2","dns2":"10.10.2.2","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":true},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"3cfcc4df-2e17-4fa7-aafd-72b9badc6df9","ip":"169.254.0.226","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:e2","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.208.66.24","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.226","port":3922,"interval":6,"retries":100,"name":"r-4-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-4-VM","router.ip":"169.254.0.226"},"wait":0}},{}] } to 279278805451034
2014-03-19 15:51:59,680 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-7:null) Seq 4-1161232387: Forwarding Seq 4-1161232387: { Cmd , MgmtId: 279278805451040, via: 4, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":4,"name":"r-4-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-4-VM eth0ip=10.208.67.224 eth0mask=255.255.254.0 gateway=10.208.66.1 domain=cs1cloud.internal dhcprange=10.208.66.1 eth1ip=169.254.0.226 eth1mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=10.10.1.2
dns2=10.10.2.2","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8f98cc61ecec605","params":{},"uuid":"305f6697-1322-4167-b56e-8165c463ae2a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"63d080b2-33cf-41f8-8c6a-2f796c922c30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"0e1328fc-61df-3bab-9cea-0e5e4796ac5d","id":1,"poolType":"NetworkFilesystem","host":"10.208.66.23","path":"/export/primary","port":2049}},"name":"ROOT-4","size":276162048,"path":"b69b0a65-abd3-4c4b-b574-9568a8847fe0","volumeId":5,"vmName":"r-4-VM","accountId":1,"format":"QCOW2","id":5,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"f4c6a16b-4a12-42fb-9dbe-b084b41a1960","ip":"10.208.67.224","netmask":"255.255.254.0","gateway":"10.208.66.1","mac":"06:be:74:00:00:19","dns1":"10.10.1
.2","dns2":"10.10.2.2","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":true},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"3cfcc4df-2e17-4fa7-aafd-72b9badc6df9","ip":"169.254.0.226","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:e2","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.208.66.24","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.226","port":3922,"interval":6,"retries":100,"name":"r-4-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-4-VM","router.ip":"169.254.0.226"},"wait":0}},{}] } to 279278805451034
2014-03-19 15:51:59,681 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 4-1161232387: Forwarding Seq 4-1161232387: { Cmd , MgmtId: 279278805451040, via: 4, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":4,"name":"r-4-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-4-VM eth0ip=10.208.67.224 eth0mask=255.255.254.0 gateway=10.208.66.1 domain=cs1cloud.internal dhcprange=10.208.66.1 eth1ip=169.254.0.226 eth1mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=10.10.1.2
dns2=10.10.2.2","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8f98cc61ecec605","params":{},"uuid":"305f6697-1322-4167-b56e-8165c463ae2a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"63d080b2-33cf-41f8-8c6a-2f796c922c30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"0e1328fc-61df-3bab-9cea-0e5e4796ac5d","id":1,"poolType":"NetworkFilesystem","host":"10.208.66.23","path":"/export/primary","port":2049}},"name":"ROOT-4","size":276162048,"path":"b69b0a65-abd3-4c4b-b574-9568a8847fe0","volumeId":5,"vmName":"r-4-VM","accountId":1,"format":"QCOW2","id":5,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"f4c6a16b-4a12-42fb-9dbe-b084b41a1960","ip":"10.208.67.224","netmask":"255.255.254.0","gateway":"10.208.66.1","mac":"06:be:74:00:00:19","dns1":"10.10.1
.2","dns2":"10.10.2.2","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":true},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"3cfcc4df-2e17-4fa7-aafd-72b9badc6df9","ip":"169.254.0.226","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:e2","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.208.66.24","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.226","port":3922,"interval":6,"retries":100,"name":"r-4-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-4-VM","router.ip":"169.254.0.226"},"wait":0}},{}] } to 279278805451034
2014-03-19 15:51:59,682 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-9:null) Seq 4-1161232387: Forwarding Seq 4-1161232387: { Cmd , MgmtId: 279278805451040, via: 4, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":4,"name":"r-4-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-4-VM eth0ip=10.208.67.224 eth0mask=255.255.254.0 gateway=10.208.66.1 domain=cs1cloud.internal dhcprange=10.208.66.1 eth1ip=169.254.0.226 eth1mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=10.10.1.2
dns2=10.10.2.2","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8f98cc61ecec605","params":{},"uuid":"305f6697-1322-4167-b56e-8165c463ae2a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"63d080b2-33cf-41f8-8c6a-2f796c922c30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"0e1328fc-61df-3bab-9cea-0e5e4796ac5d","id":1,"poolType":"NetworkFilesystem","host":"10.208.66.23","path":"/export/primary","port":2049}},"name":"ROOT-4","size":276162048,"path":"b69b0a65-abd3-4c4b-b574-9568a8847fe0","volumeId":5,"vmName":"r-4-VM","accountId":1,"format":"QCOW2","id":5,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"f4c6a16b-4a12-42fb-9dbe-b084b41a1960","ip":"10.208.67.224","netmask":"255.255.254.0","gateway":"10.208.66.1","mac":"06:be:74:00:00:19","dns1":"10.10.1
.2","dns2":"10.10.2.2","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":true},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"3cfcc4df-2e17-4fa7-aafd-72b9badc6df9","ip":"169.254.0.226","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:e2","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.208.66.24","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.226","port":3922,"interval":6,"retries":100,"name":"r-4-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-4-VM","router.ip":"169.254.0.226"},"wait":0}},{}] } to 279278805451034
2014-03-19 15:51:59,683 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 4-1161232387: Forwarding Seq 4-1161232387: { Cmd , MgmtId: 279278805451040, via: 4, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":4,"name":"r-4-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-4-VM eth0ip=10.208.67.224 eth0mask=255.255.254.0 gateway=10.208.66.1 domain=cs1cloud.internal dhcprange=10.208.66.1 eth1ip=169.254.0.226 eth1mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=10.10.1.2
dns2=10.10.2.2","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8f98cc61ecec605","params":{},"uuid":"305f6697-1322-4167-b56e-8165c463ae2a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"63d080b2-33cf-41f8-8c6a-2f796c922c30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"0e1328fc-61df-3bab-9cea-0e5e4796ac5d","id":1,"poolType":"NetworkFilesystem","host":"10.208.66.23","path":"/export/primary","port":2049}},"name":"ROOT-4","size":276162048,"path":"b69b0a65-abd3-4c4b-b574-9568a8847fe0","volumeId":5,"vmName":"r-4-VM","accountId":1,"format":"QCOW2","id":5,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"f4c6a16b-4a12-42fb-9dbe-b084b41a1960","ip":"10.208.67.224","netmask":"255.255.254.0","gateway":"10.208.66.1","mac":"06:be:74:00:00:19","dns1":"10.10.1
.2","dns2":"10.10.2.2","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":true},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"3cfcc4df-2e17-4fa7-aafd-72b9badc6df9","ip":"169.254.0.226","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:e2","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.208.66.24","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.226","port":3922,"interval":6,"retries":100,"name":"r-4-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-4-VM","router.ip":"169.254.0.226"},"wait":0}},{}] } to 279278805451034
2014-03-19 15:51:59,684 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-10:null) Seq 4-1161232387: Forwarding Seq 4-1161232387: { Cmd , MgmtId: 279278805451040, via: 4, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":4,"name":"r-4-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-4-VM eth0ip=10.208.67.224 eth0mask=255.255.254.0 gateway=10.208.66.1 domain=cs1cloud.internal dhcprange=10.208.66.1 eth1ip=169.254.0.226 eth1mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=10.10.1.2
dns2=10.10.2.2","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8f98cc61ecec605","params":{},"uuid":"305f6697-1322-4167-b56e-8165c463ae2a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"63d080b2-33cf-41f8-8c6a-2f796c922c30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"0e1328fc-61df-3bab-9cea-0e5e4796ac5d","id":1,"poolType":"NetworkFilesystem","host":"10.208.66.23","path":"/export/primary","port":2049}},"name":"ROOT-4","size":276162048,"path":"b69b0a65-abd3-4c4b-b574-9568a8847fe0","volumeId":5,"vmName":"r-4-VM","accountId":1,"format":"QCOW2","id":5,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"f4c6a16b-4a12-42fb-9dbe-b084b41a1960","ip":"10.208.67.224","netmask":"255.255.254.0","gateway":"10.208.66.1","mac":"06:be:74:00:00:19","dns1":"10.10.1
.2","dns2":"10.10.2.2","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":true},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"3cfcc4df-2e17-4fa7-aafd-72b9badc6df9","ip":"169.254.0.226","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:e2","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.208.66.24","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.226","port":3922,"interval":6,"retries":100,"name":"r-4-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-4-VM","router.ip":"169.254.0.226"},"wait":0}},{}] } to 279278805451034
2014-03-19 15:51:59,685 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-11:null) Seq 4-1161232387: Forwarding Seq 4-1161232387: { Cmd , MgmtId: 279278805451040, via: 4, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":4,"name":"r-4-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-4-VM eth0ip=10.208.67.224 eth0mask=255.255.254.0 gateway=10.208.66.1 domain=cs1cloud.internal dhcprange=10.208.66.1 eth1ip=169.254.0.226 eth1mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=10.10.1.2
dns2=10.10.2.2","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8f98cc61ecec605","params":{},"uuid":"305f6697-1322-4167-b56e-8165c463ae2a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"63d080b2-33cf-41f8-8c6a-2f796c922c30","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"0e1328fc-61df-3bab-9cea-0e5e4796ac5d","id":1,"poolType":"NetworkFilesystem","host":"10.208.66.23","path":"/export/primary","port":2049}},"name":"ROOT-4","size":276162048,"path":"b69b0a65-abd3-4c4b-b574-9568a8847fe0","volumeId":5,"vmName":"r-4-VM","accountId":1,"format":"QCOW2","id":5,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"f4c6a16b-4a12-42fb-9dbe-b084b41a1960","ip":"10.208.67.224","netmask":"255.255.254.0","gateway":"10.208.66.1","mac":"06:be:74:00:00:19","dns1":"10.10.1
.2","dns2":"10.10.2.2","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":true},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"3cfcc4df-2e17-4fa7-aafd-72b9badc6df9","ip":"169.254.0.226","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:e2","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.208.66.24","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.226","port":3922,"interval":6,"retries":100,"name":"r-4-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-4-VM","router.ip":"169.254.0.226"},"wait":0}},{}] } to 279278805451034
On Wednesday, 19 March 2014 3:38 PM, Suresh Sadhu <Su...@citrix.com> wrote:
Go to management server (Hope your db is running on same machine)
Type the below commands:
>mysql cloud
> update vm_instance set state="Stopped" where state="Starting";
Delete this vms from CS UI.
Hope your agent status is UP and running.
Check the nfs moutn points--- Manually mount your nfs and copy some text file and check whether its successful or not.
please refer this below link also. Might be useful to you
http://www.greenhills.co.uk/2013/08/30/cloudstack-single-server-on-ubuntu-with-kvm.html
regards
sadhu
From:Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:53
To: Suresh Sadhu; users@cloudstack.apache.org
Subject: Re: vm stuck in starting state, unable to delete it
> You can update the vm state in db as Stopped and try to delete them from CS.
How to go about doing this? Sorry, I am very new to CS.
This is what I kept on getting when I ran this:
grep -i -E 'exception|unable|fail|invalid|leak|warn|error' /var/log/cloudstack/management/management-server.log
2014-03-19 13:22:36,156 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744864 to Host 4 timed out after 3600
2014-03-19 13:22:36,156 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:22:36,156 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:23:22,786 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:23:22,786 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489624: Timed out on null
2014-03-19 13:23:22,788 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489624 to Host 3 timed out after 3600
2014-03-19 13:23:22,788 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489624 to Host 3 timed out after 3600
2014-03-19 13:23:36,747 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:23:36,748 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744865: Timed out on null
2014-03-19 13:23:36,748 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744865 to Host 4 timed out after 3600
2014-03-19 13:23:36,748 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 13:23:36,748 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 13:24:22,792 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:24:22,792 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489625: Timed out on null
2014-03-19 13:24:22,794 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489625 to Host 3 timed out after 3600
2014-03-19 13:24:22,794 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489625 to Host 3 timed out after 3600
2014-03-19 13:24:37,336 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:24:37,336 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744866: Timed out on null
2014-03-19 13:24:37,336 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744866 to Host 4 timed out after 3600
2014-03-19 13:24:37,336 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:24:37,336 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:25:22,799 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:25:22,799 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489626: Timed out on null
2014-03-19 13:25:22,799 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489626 to Host 3 timed out after 3600
2014-03-19 13:25:22,799 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489626 to Host 3 timed out after 3600
2014-03-19 13:25:38,015 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:25:38,015 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744867: Timed out on null
2014-03-19 13:25:38,015 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744867 to Host 4 timed out after 3600
2014-03-19 13:25:38,015 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:25:38,015 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:26:22,804 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:26:22,804 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489627: Timed out on null
2014-03-19 13:26:22,804 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489627 to Host 3 timed out after 3600
2014-03-19 13:26:22,804 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489627 to Host 3 timed out after 3600
2014-03-19 13:26:38,611 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:26:38,612 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744868: Timed out on null
2014-03-19 13:26:38,612 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744868 to Host 4 timed out after 3600
2014-03-19 13:26:38,612 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:26:38,612 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:27:22,808 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:27:22,809 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489628: Timed out on null
2014-03-19 13:27:22,809 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489628 to Host 3 timed out after 3600
2014-03-19 13:27:22,809 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489628 to Host 3 timed out after 3600
2014-03-19 13:27:39,200 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:27:39,200 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744869: Timed out on null
2014-03-19 13:27:39,200 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744869 to Host 4 timed out after 3600
2014-03-19 13:27:39,200 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:27:39,200 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:28:22,813 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:28:22,813 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489629: Timed out on null
2014-03-19 13:28:22,813 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489629 to Host 3 timed out after 3600
2014-03-19 13:28:22,813 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489629 to Host 3 timed out after 3600
2014-03-19 13:28:39,795 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:28:39,795 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744870: Timed out on null
2014-03-19 13:28:39,796 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744870 to Host 4 timed out after 3600
2014-03-19 13:28:39,796 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:28:39,796 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:29:22,818 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:29:22,818 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489630: Timed out on null
2014-03-19 13:29:22,819 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489630 to Host 3 timed out after 3600
2014-03-19 13:29:22,819 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489630 to Host 3 timed out after 3600
2014-03-19 13:29:40,395 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:29:40,395 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744871: Timed out on null
2014-03-19 13:29:40,396 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744871 to Host 4 timed out after 3600
2014-03-19 13:29:40,396 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:29:40,396 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:30:22,824 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:30:22,824 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489631: Timed out on null
2014-03-19 13:30:22,824 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489631 to Host 3 timed out after 3600
2014-03-19 13:30:22,824 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489631 to Host 3 timed out after 3600
2014-03-19 13:30:40,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:30:40,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744872: Timed out on null
2014-03-19 13:30:40,988 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744872 to Host 4 timed out after 3600
2014-03-19 13:30:40,988 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 13:30:40,988 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 13:31:22,828 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:31:22,828 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489632: Timed out on null
2014-03-19 13:31:22,828 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489632 to Host 3 timed out after 3600
2014-03-19 13:31:22,828 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489632 to Host 3 timed out after 3600
2014-03-19 13:31:41,580 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:31:41,580 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744873: Timed out on null
2014-03-19 13:31:41,580 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744873 to Host 4 timed out after 3600
2014-03-19 13:31:41,580 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:31:41,580 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:32:22,833 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:32:22,833 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489633: Timed out on null
2014-03-19 13:32:22,833 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489633 to Host 3 timed out after 3600
2014-03-19 13:32:22,833 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489633 to Host 3 timed out after 3600
2014-03-19 13:32:42,171 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:32:42,172 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744874: Timed out on null
2014-03-19 13:32:42,172 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744874 to Host 4 timed out after 3600
2014-03-19 13:32:42,172 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 13:32:42,172 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 13:33:22,838 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:33:22,838 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489634: Timed out on null
2014-03-19 13:33:22,838 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489634 to Host 3 timed out after 3600
2014-03-19 13:33:22,838 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489634 to Host 3 timed out after 3600
2014-03-19 13:33:42,763 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:33:42,763 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744875: Timed out on null
2014-03-19 13:33:42,764 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744875 to Host 4 timed out after 3600
2014-03-19 13:33:42,764 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:33:42,764 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:34:22,843 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:34:22,843 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489635: Timed out on null
2014-03-19 13:34:22,843 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489635 to Host 3 timed out after 3600
2014-03-19 13:34:22,843 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489635 to Host 3 timed out after 3600
2014-03-19 13:34:43,356 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:34:43,356 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744876: Timed out on null
2014-03-19 13:34:43,356 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744876 to Host 4 timed out after 3600
2014-03-19 13:34:43,356 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:34:43,356 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:35:22,847 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:35:22,848 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489636: Timed out on null
2014-03-19 13:35:22,848 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489636 to Host 3 timed out after 3600
2014-03-19 13:35:22,848 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489636 to Host 3 timed out after 3600
2014-03-19 13:35:43,948 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:35:43,948 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744877: Timed out on null
2014-03-19 13:35:43,948 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744877 to Host 4 timed out after 3600
2014-03-19 13:35:43,948 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:35:43,948 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:36:22,852 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:36:22,852 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489637: Timed out on null
2014-03-19 13:36:22,852 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489637 to Host 3 timed out after 3600
2014-03-19 13:36:22,852 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489637 to Host 3 timed out after 3600
2014-03-19 13:36:44,501 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:36:44,501 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744878: Timed out on null
2014-03-19 13:36:44,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744878 to Host 4 timed out after 3600
2014-03-19 13:36:44,502 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:36:44,502 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:37:22,857 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:37:22,857 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489638: Timed out on null
2014-03-19 13:37:22,857 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489638 to Host 3 timed out after 3600
2014-03-19 13:37:22,857 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489638 to Host 3 timed out after 3600
2014-03-19 13:37:45,091 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:37:45,091 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744879: Timed out on null
2014-03-19 13:37:45,092 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744879 to Host 4 timed out after 3600
2014-03-19 13:37:45,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:37:45,092 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:38:22,862 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:38:22,862 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489639: Timed out on null
2014-03-19 13:38:22,862 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489639 to Host 3 timed out after 3600
2014-03-19 13:38:22,862 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489639 to Host 3 timed out after 3600
2014-03-19 13:38:45,692 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:38:45,692 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744880: Timed out on null
2014-03-19 13:38:45,692 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744880 to Host 4 timed out after 3600
2014-03-19 13:38:45,692 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 13:38:45,692 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 13:39:22,866 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:39:22,866 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489640: Timed out on null
2014-03-19 13:39:22,867 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489640 to Host 3 timed out after 3600
2014-03-19 13:39:22,867 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489640 to Host 3 timed out after 3600
2014-03-19 13:39:46,284 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:39:46,284 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744881: Timed out on null
2014-03-19 13:39:46,284 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744881 to Host 4 timed out after 3600
2014-03-19 13:39:46,284 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:39:46,284 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:40:22,871 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:40:22,871 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489641: Timed out on null
2014-03-19 13:40:22,871 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489641 to Host 3 timed out after 3600
2014-03-19 13:40:22,871 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489641 to Host 3 timed out after 3600
2014-03-19 13:40:46,876 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:40:46,876 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744882: Timed out on null
2014-03-19 13:40:46,876 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744882 to Host 4 timed out after 3600
2014-03-19 13:40:46,876 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:40:46,876 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:41:22,876 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:41:22,876 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489642: Timed out on null
2014-03-19 13:41:22,876 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489642 to Host 3 timed out after 3600
2014-03-19 13:41:22,876 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489642 to Host 3 timed out after 3600
2014-03-19 13:41:47,467 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:41:47,468 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744883: Timed out on null
2014-03-19 13:41:47,468 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744883 to Host 4 timed out after 3600
2014-03-19 13:41:47,468 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 13:41:47,468 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 13:42:22,881 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:42:22,881 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489643: Timed out on null
2014-03-19 13:42:22,881 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489643 to Host 3 timed out after 3600
2014-03-19 13:42:22,881 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489643 to Host 3 timed out after 3600
2014-03-19 13:42:48,059 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:42:48,060 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744884: Timed out on null
2014-03-19 13:42:48,060 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744884 to Host 4 timed out after 3600
2014-03-19 13:42:48,060 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:42:48,060 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:43:22,886 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:43:22,886 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489644: Timed out on null
2014-03-19 13:43:22,886 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489644 to Host 3 timed out after 3600
2014-03-19 13:43:22,886 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org
Subject: vm stuck in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary
storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
RE: vm stuck in starting state, unable to delete it
Posted by Suresh Sadhu <Su...@citrix.com>.
Go to management server (Hope your db is running on same machine)
Type the below commands:
>mysql cloud
> update vm_instance set state="Stopped" where state="Starting";
Delete this vms from CS UI.
Hope your agent status is UP and running.
Check the nfs moutn points--- Manually mount your nfs and copy some text file and check whether its successful or not.
please refer this below link also. Might be useful to you
http://www.greenhills.co.uk/2013/08/30/cloudstack-single-server-on-ubuntu-with-kvm.html
regards
sadhu
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:53
To: Suresh Sadhu; users@cloudstack.apache.org
Subject: Re: vm stuck in starting state, unable to delete it
> You can update the vm state in db as Stopped and try to delete them from CS.
How to go about doing this? Sorry, I am very new to CS.
This is what I kept on getting when I ran this:
grep -i -E 'exception|unable|fail|invalid|leak|warn|error' /var/log/cloudstack/management/management-server.log
2014-03-19 13:22:36,156 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744864 to Host 4 timed out after 3600
2014-03-19 13:22:36,156 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:22:36,156 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:23:22,786 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:23:22,786 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489624: Timed out on null
2014-03-19 13:23:22,788 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489624 to Host 3 timed out after 3600
2014-03-19 13:23:22,788 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489624 to Host 3 timed out after 3600
2014-03-19 13:23:36,747 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:23:36,748 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744865: Timed out on null
2014-03-19 13:23:36,748 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744865 to Host 4 timed out after 3600
2014-03-19 13:23:36,748 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 13:23:36,748 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 13:24:22,792 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:24:22,792 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489625: Timed out on null
2014-03-19 13:24:22,794 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489625 to Host 3 timed out after 3600
2014-03-19 13:24:22,794 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489625 to Host 3 timed out after 3600
2014-03-19 13:24:37,336 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:24:37,336 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744866: Timed out on null
2014-03-19 13:24:37,336 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744866 to Host 4 timed out after 3600
2014-03-19 13:24:37,336 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:24:37,336 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:25:22,799 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:25:22,799 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489626: Timed out on null
2014-03-19 13:25:22,799 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489626 to Host 3 timed out after 3600
2014-03-19 13:25:22,799 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489626 to Host 3 timed out after 3600
2014-03-19 13:25:38,015 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:25:38,015 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744867: Timed out on null
2014-03-19 13:25:38,015 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744867 to Host 4 timed out after 3600
2014-03-19 13:25:38,015 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:25:38,015 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:26:22,804 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:26:22,804 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489627: Timed out on null
2014-03-19 13:26:22,804 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489627 to Host 3 timed out after 3600
2014-03-19 13:26:22,804 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489627 to Host 3 timed out after 3600
2014-03-19 13:26:38,611 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:26:38,612 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744868: Timed out on null
2014-03-19 13:26:38,612 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744868 to Host 4 timed out after 3600
2014-03-19 13:26:38,612 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:26:38,612 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:27:22,808 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:27:22,809 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489628: Timed out on null
2014-03-19 13:27:22,809 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489628 to Host 3 timed out after 3600
2014-03-19 13:27:22,809 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489628 to Host 3 timed out after 3600
2014-03-19 13:27:39,200 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:27:39,200 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744869: Timed out on null
2014-03-19 13:27:39,200 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744869 to Host 4 timed out after 3600
2014-03-19 13:27:39,200 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:27:39,200 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:28:22,813 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:28:22,813 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489629: Timed out on null
2014-03-19 13:28:22,813 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489629 to Host 3 timed out after 3600
2014-03-19 13:28:22,813 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489629 to Host 3 timed out after 3600
2014-03-19 13:28:39,795 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:28:39,795 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744870: Timed out on null
2014-03-19 13:28:39,796 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744870 to Host 4 timed out after 3600
2014-03-19 13:28:39,796 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:28:39,796 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:29:22,818 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:29:22,818 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489630: Timed out on null
2014-03-19 13:29:22,819 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489630 to Host 3 timed out after 3600
2014-03-19 13:29:22,819 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489630 to Host 3 timed out after 3600
2014-03-19 13:29:40,395 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:29:40,395 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744871: Timed out on null
2014-03-19 13:29:40,396 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744871 to Host 4 timed out after 3600
2014-03-19 13:29:40,396 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:29:40,396 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:30:22,824 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:30:22,824 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489631: Timed out on null
2014-03-19 13:30:22,824 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489631 to Host 3 timed out after 3600
2014-03-19 13:30:22,824 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489631 to Host 3 timed out after 3600
2014-03-19 13:30:40,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:30:40,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744872: Timed out on null
2014-03-19 13:30:40,988 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744872 to Host 4 timed out after 3600
2014-03-19 13:30:40,988 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 13:30:40,988 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 13:31:22,828 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:31:22,828 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489632: Timed out on null
2014-03-19 13:31:22,828 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489632 to Host 3 timed out after 3600
2014-03-19 13:31:22,828 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489632 to Host 3 timed out after 3600
2014-03-19 13:31:41,580 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:31:41,580 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744873: Timed out on null
2014-03-19 13:31:41,580 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744873 to Host 4 timed out after 3600
2014-03-19 13:31:41,580 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:31:41,580 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:32:22,833 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:32:22,833 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489633: Timed out on null
2014-03-19 13:32:22,833 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489633 to Host 3 timed out after 3600
2014-03-19 13:32:22,833 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489633 to Host 3 timed out after 3600
2014-03-19 13:32:42,171 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:32:42,172 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744874: Timed out on null
2014-03-19 13:32:42,172 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744874 to Host 4 timed out after 3600
2014-03-19 13:32:42,172 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 13:32:42,172 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 13:33:22,838 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:33:22,838 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489634: Timed out on null
2014-03-19 13:33:22,838 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489634 to Host 3 timed out after 3600
2014-03-19 13:33:22,838 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489634 to Host 3 timed out after 3600
2014-03-19 13:33:42,763 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:33:42,763 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744875: Timed out on null
2014-03-19 13:33:42,764 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744875 to Host 4 timed out after 3600
2014-03-19 13:33:42,764 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:33:42,764 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:34:22,843 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:34:22,843 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489635: Timed out on null
2014-03-19 13:34:22,843 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489635 to Host 3 timed out after 3600
2014-03-19 13:34:22,843 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489635 to Host 3 timed out after 3600
2014-03-19 13:34:43,356 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:34:43,356 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744876: Timed out on null
2014-03-19 13:34:43,356 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744876 to Host 4 timed out after 3600
2014-03-19 13:34:43,356 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:34:43,356 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:35:22,847 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:35:22,848 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489636: Timed out on null
2014-03-19 13:35:22,848 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489636 to Host 3 timed out after 3600
2014-03-19 13:35:22,848 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489636 to Host 3 timed out after 3600
2014-03-19 13:35:43,948 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:35:43,948 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744877: Timed out on null
2014-03-19 13:35:43,948 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744877 to Host 4 timed out after 3600
2014-03-19 13:35:43,948 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:35:43,948 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:36:22,852 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:36:22,852 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489637: Timed out on null
2014-03-19 13:36:22,852 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489637 to Host 3 timed out after 3600
2014-03-19 13:36:22,852 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489637 to Host 3 timed out after 3600
2014-03-19 13:36:44,501 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:36:44,501 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744878: Timed out on null
2014-03-19 13:36:44,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744878 to Host 4 timed out after 3600
2014-03-19 13:36:44,502 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:36:44,502 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:37:22,857 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:37:22,857 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489638: Timed out on null
2014-03-19 13:37:22,857 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489638 to Host 3 timed out after 3600
2014-03-19 13:37:22,857 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489638 to Host 3 timed out after 3600
2014-03-19 13:37:45,091 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:37:45,091 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744879: Timed out on null
2014-03-19 13:37:45,092 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744879 to Host 4 timed out after 3600
2014-03-19 13:37:45,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:37:45,092 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:38:22,862 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:38:22,862 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489639: Timed out on null
2014-03-19 13:38:22,862 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489639 to Host 3 timed out after 3600
2014-03-19 13:38:22,862 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489639 to Host 3 timed out after 3600
2014-03-19 13:38:45,692 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:38:45,692 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744880: Timed out on null
2014-03-19 13:38:45,692 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744880 to Host 4 timed out after 3600
2014-03-19 13:38:45,692 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 13:38:45,692 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 13:39:22,866 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:39:22,866 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489640: Timed out on null
2014-03-19 13:39:22,867 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489640 to Host 3 timed out after 3600
2014-03-19 13:39:22,867 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489640 to Host 3 timed out after 3600
2014-03-19 13:39:46,284 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:39:46,284 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744881: Timed out on null
2014-03-19 13:39:46,284 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744881 to Host 4 timed out after 3600
2014-03-19 13:39:46,284 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:39:46,284 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:40:22,871 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:40:22,871 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489641: Timed out on null
2014-03-19 13:40:22,871 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489641 to Host 3 timed out after 3600
2014-03-19 13:40:22,871 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489641 to Host 3 timed out after 3600
2014-03-19 13:40:46,876 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:40:46,876 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744882: Timed out on null
2014-03-19 13:40:46,876 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744882 to Host 4 timed out after 3600
2014-03-19 13:40:46,876 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:40:46,876 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:41:22,876 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:41:22,876 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489642: Timed out on null
2014-03-19 13:41:22,876 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489642 to Host 3 timed out after 3600
2014-03-19 13:41:22,876 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489642 to Host 3 timed out after 3600
2014-03-19 13:41:47,467 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:41:47,468 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744883: Timed out on null
2014-03-19 13:41:47,468 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744883 to Host 4 timed out after 3600
2014-03-19 13:41:47,468 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 13:41:47,468 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 13:42:22,881 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:42:22,881 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489643: Timed out on null
2014-03-19 13:42:22,881 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489643 to Host 3 timed out after 3600
2014-03-19 13:42:22,881 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489643 to Host 3 timed out after 3600
2014-03-19 13:42:48,059 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:42:48,060 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744884: Timed out on null
2014-03-19 13:42:48,060 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744884 to Host 4 timed out after 3600
2014-03-19 13:42:48,060 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:42:48,060 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:43:22,886 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:43:22,886 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489644: Timed out on null
2014-03-19 13:43:22,886 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489644 to Host 3 timed out after 3600
2014-03-19 13:43:22,886 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com>> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com<ma...@rocketmail.com>]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Subject: vm stuck in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
Re: vm stuck in starting state, unable to delete it
Posted by Sugandh S <s....@rocketmail.com>.
> You can update the vm state in db as Stopped and try to delete them from CS.
How to go about doing this? Sorry, I am very new to CS.
This is what I kept on getting when I ran this:
grep -i -E 'exception|unable|fail|invalid|leak|warn|error' /var/log/cloudstack/management/management-server.log
2014-03-19 13:22:36,156 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744864 to Host 4 timed out after 3600
2014-03-19 13:22:36,156 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:22:36,156 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:23:22,786 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:23:22,786 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489624: Timed out on null
2014-03-19 13:23:22,788 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489624 to Host 3 timed out after 3600
2014-03-19 13:23:22,788 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489624 to Host 3 timed out after 3600
2014-03-19 13:23:36,747 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:23:36,748 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744865: Timed out on null
2014-03-19 13:23:36,748 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744865 to Host 4 timed out after 3600
2014-03-19 13:23:36,748 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 13:23:36,748 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 13:24:22,792 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:24:22,792 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489625: Timed out on null
2014-03-19 13:24:22,794 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489625 to Host 3 timed out after 3600
2014-03-19 13:24:22,794 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489625 to Host 3 timed out after 3600
2014-03-19 13:24:37,336 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:24:37,336 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744866: Timed out on null
2014-03-19 13:24:37,336 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744866 to Host 4 timed out after 3600
2014-03-19 13:24:37,336 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:24:37,336 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:25:22,799 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:25:22,799 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489626: Timed out on null
2014-03-19 13:25:22,799 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489626 to Host 3 timed out after 3600
2014-03-19 13:25:22,799 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489626 to Host 3 timed out after 3600
2014-03-19 13:25:38,015 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:25:38,015 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744867: Timed out on null
2014-03-19 13:25:38,015 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744867 to Host 4 timed out after 3600
2014-03-19 13:25:38,015 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:25:38,015 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:26:22,804 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:26:22,804 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489627: Timed out on null
2014-03-19 13:26:22,804 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489627 to Host 3 timed out after 3600
2014-03-19 13:26:22,804 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489627 to Host 3 timed out after 3600
2014-03-19 13:26:38,611 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:26:38,612 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744868: Timed out on null
2014-03-19 13:26:38,612 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744868 to Host 4 timed out after 3600
2014-03-19 13:26:38,612 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:26:38,612 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:27:22,808 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:27:22,809 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489628: Timed out on null
2014-03-19 13:27:22,809 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489628 to Host 3 timed out after 3600
2014-03-19 13:27:22,809 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489628 to Host 3 timed out after 3600
2014-03-19 13:27:39,200 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:27:39,200 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744869: Timed out on null
2014-03-19 13:27:39,200 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744869 to Host 4 timed out after 3600
2014-03-19 13:27:39,200 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:27:39,200 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:28:22,813 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:28:22,813 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489629: Timed out on null
2014-03-19 13:28:22,813 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489629 to Host 3 timed out after 3600
2014-03-19 13:28:22,813 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489629 to Host 3 timed out after 3600
2014-03-19 13:28:39,795 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:28:39,795 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744870: Timed out on null
2014-03-19 13:28:39,796 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744870 to Host 4 timed out after 3600
2014-03-19 13:28:39,796 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:28:39,796 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:29:22,818 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:29:22,818 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489630: Timed out on null
2014-03-19 13:29:22,819 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489630 to Host 3 timed out after 3600
2014-03-19 13:29:22,819 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489630 to Host 3 timed out after 3600
2014-03-19 13:29:40,395 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:29:40,395 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744871: Timed out on null
2014-03-19 13:29:40,396 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744871 to Host 4 timed out after 3600
2014-03-19 13:29:40,396 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:29:40,396 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:30:22,824 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:30:22,824 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489631: Timed out on null
2014-03-19 13:30:22,824 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489631 to Host 3 timed out after 3600
2014-03-19 13:30:22,824 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489631 to Host 3 timed out after 3600
2014-03-19 13:30:40,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:30:40,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744872: Timed out on null
2014-03-19 13:30:40,988 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744872 to Host 4 timed out after 3600
2014-03-19 13:30:40,988 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 13:30:40,988 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 13:31:22,828 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:31:22,828 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489632: Timed out on null
2014-03-19 13:31:22,828 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489632 to Host 3 timed out after 3600
2014-03-19 13:31:22,828 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489632 to Host 3 timed out after 3600
2014-03-19 13:31:41,580 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:31:41,580 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744873: Timed out on null
2014-03-19 13:31:41,580 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744873 to Host 4 timed out after 3600
2014-03-19 13:31:41,580 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:31:41,580 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:32:22,833 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:32:22,833 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489633: Timed out on null
2014-03-19 13:32:22,833 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489633 to Host 3 timed out after 3600
2014-03-19 13:32:22,833 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489633 to Host 3 timed out after 3600
2014-03-19 13:32:42,171 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:32:42,172 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744874: Timed out on null
2014-03-19 13:32:42,172 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744874 to Host 4 timed out after 3600
2014-03-19 13:32:42,172 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 13:32:42,172 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 13:33:22,838 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:33:22,838 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489634: Timed out on null
2014-03-19 13:33:22,838 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489634 to Host 3 timed out after 3600
2014-03-19 13:33:22,838 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489634 to Host 3 timed out after 3600
2014-03-19 13:33:42,763 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:33:42,763 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744875: Timed out on null
2014-03-19 13:33:42,764 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744875 to Host 4 timed out after 3600
2014-03-19 13:33:42,764 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:33:42,764 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:34:22,843 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:34:22,843 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489635: Timed out on null
2014-03-19 13:34:22,843 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489635 to Host 3 timed out after 3600
2014-03-19 13:34:22,843 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489635 to Host 3 timed out after 3600
2014-03-19 13:34:43,356 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:34:43,356 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744876: Timed out on null
2014-03-19 13:34:43,356 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744876 to Host 4 timed out after 3600
2014-03-19 13:34:43,356 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:34:43,356 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:35:22,847 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:35:22,848 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489636: Timed out on null
2014-03-19 13:35:22,848 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489636 to Host 3 timed out after 3600
2014-03-19 13:35:22,848 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489636 to Host 3 timed out after 3600
2014-03-19 13:35:43,948 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:35:43,948 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744877: Timed out on null
2014-03-19 13:35:43,948 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744877 to Host 4 timed out after 3600
2014-03-19 13:35:43,948 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:35:43,948 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:36:22,852 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:36:22,852 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489637: Timed out on null
2014-03-19 13:36:22,852 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489637 to Host 3 timed out after 3600
2014-03-19 13:36:22,852 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489637 to Host 3 timed out after 3600
2014-03-19 13:36:44,501 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:36:44,501 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744878: Timed out on null
2014-03-19 13:36:44,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744878 to Host 4 timed out after 3600
2014-03-19 13:36:44,502 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:36:44,502 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:37:22,857 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:37:22,857 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489638: Timed out on null
2014-03-19 13:37:22,857 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489638 to Host 3 timed out after 3600
2014-03-19 13:37:22,857 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489638 to Host 3 timed out after 3600
2014-03-19 13:37:45,091 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:37:45,091 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744879: Timed out on null
2014-03-19 13:37:45,092 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744879 to Host 4 timed out after 3600
2014-03-19 13:37:45,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:37:45,092 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:38:22,862 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:38:22,862 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489639: Timed out on null
2014-03-19 13:38:22,862 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489639 to Host 3 timed out after 3600
2014-03-19 13:38:22,862 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489639 to Host 3 timed out after 3600
2014-03-19 13:38:45,692 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:38:45,692 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744880: Timed out on null
2014-03-19 13:38:45,692 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744880 to Host 4 timed out after 3600
2014-03-19 13:38:45,692 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 13:38:45,692 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 13:39:22,866 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:39:22,866 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489640: Timed out on null
2014-03-19 13:39:22,867 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489640 to Host 3 timed out after 3600
2014-03-19 13:39:22,867 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489640 to Host 3 timed out after 3600
2014-03-19 13:39:46,284 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:39:46,284 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744881: Timed out on null
2014-03-19 13:39:46,284 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744881 to Host 4 timed out after 3600
2014-03-19 13:39:46,284 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:39:46,284 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:40:22,871 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:40:22,871 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489641: Timed out on null
2014-03-19 13:40:22,871 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489641 to Host 3 timed out after 3600
2014-03-19 13:40:22,871 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489641 to Host 3 timed out after 3600
2014-03-19 13:40:46,876 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:40:46,876 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744882: Timed out on null
2014-03-19 13:40:46,876 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744882 to Host 4 timed out after 3600
2014-03-19 13:40:46,876 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 13:40:46,876 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 13:41:22,876 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:41:22,876 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489642: Timed out on null
2014-03-19 13:41:22,876 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489642 to Host 3 timed out after 3600
2014-03-19 13:41:22,876 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489642 to Host 3 timed out after 3600
2014-03-19 13:41:47,467 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:41:47,468 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744883: Timed out on null
2014-03-19 13:41:47,468 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698744883 to Host 4 timed out after 3600
2014-03-19 13:41:47,468 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 13:41:47,468 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 13:42:22,881 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:42:22,881 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489643: Timed out on null
2014-03-19 13:42:22,881 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489643 to Host 3 timed out after 3600
2014-03-19 13:42:22,881 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489643 to Host 3 timed out after 3600
2014-03-19 13:42:48,059 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:42:48,060 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744884: Timed out on null
2014-03-19 13:42:48,060 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744884 to Host 4 timed out after 3600
2014-03-19 13:42:48,060 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 13:42:48,060 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host: 4
2014-03-19 13:43:22,886 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:43:22,886 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489644: Timed out on null
2014-03-19 13:43:22,886 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489644 to Host 3 timed out after 3600
2014-03-19 13:43:22,886 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage stats
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org
Subject: vm stuck in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
RE: vm stuck in starting state, unable to delete it
Posted by Suresh Sadhu <Su...@citrix.com>.
Great :) .... Enjoy the journey with cloudstack.also try to attend the meetups happening around your area.
Like
http://www.meetup.com/CloudStack-Hyderabad-Group/events/172106682/
http://www.meetup.com/CloudStack-Bangalore-Group/events/169340552/
Bay Area(US): http://clds.co/1lx1GgN
Regards
Sadhu
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 20 March 2014 17:10
To: users@cloudstack.apache.org; Suresh Sadhu; Sugandh S
Subject: Re: vm stuck in starting state, unable to delete it
Okay, so I managed to solve the problem. I had to reboot the router vm and once that was done, I could create instances.
Thanks everyone. Keep up the good work!
Cheers,
Sugandh
On Thursday, 20 March 2014 3:02 PM, Sugandh S <s....@rocketmail.com>> wrote:
Hi
> Try stopping the firewall on your machine.
>
> Service iptables stop.
I have stopped the firewall, it was done by stopping ufw by issuing "ufw disable".
> Also try this ..check whether your able to access system vms including router vm from host:
I am able to access console proxy and ssvm but I cant access router vm, I can ping it though.
Sugandh
On Thursday, 20 March 2014 12:53 PM, Suresh Sadhu <Su...@citrix.com>> wrote:
Try stopping the firewall on your machine.
Service iptables stop.
Also try this ..check whether your able to access system vms including router vm from host:
enServer/KVM Hypervisors
Connect to the Host of which the System VM is running.
SSH the 'Link Local IP Address' of the System VM from the Host on which the VM is running.
Format: ssh -i <path-to-private-key> <link-local-ip> -p 3922
Example: root@faith<ma...@faith>:~# ssh -i /root/.ssh/id_rsa.cloud 169.254.3.93 -p 3922
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com<ma...@rocketmail.com>]
Sent: 20 March 2014 12:46
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>; Sugandh S
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
So, I have finally managed to access console proxy vm but I am still unable to create instances.
I have pasted the log file here: http://pastebin.com/UrjLuBiM
Here are the exceptions from the log that I got:
2014-03-20 12:26:10,766 DEBUG [agent.transport.Request] (AgentManager-Handler-15:null) Seq 1-1879572498: Processing: { Ans: ,
MgmtId: 279278805451363, via: 1, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"ssh: connect to host 169.254.3.71 port 3922: No route to host","wait":0}}] }
2014-03-20 12:26:10,766 DEBUG
[agent.manager.AgentAttache] (AgentManager-Handler-15:null) Seq
1-1879572498: No more commands found
2014-03-20 12:26:10,766 DEBUG [agent.transport.Request] (Job-Executor-1:job-15 = [ 838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Seq
1-1879572498: Received: { Ans: , MgmtId: 279278805451363, via: 1, Ver:
v1, Flags: 110, { Answer } }
2014-03-20 12:26:10,766 INFO
[cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-15 = [
838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Unable to contact resource.
com.cloud.exception.ResourceUnavailableException: Resource [Pod:1] is unreachable: Unable to apply dhcp entry on router
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3826)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2943)
at com.cloud.network.element.VirtualRouterElement.addDhcpEntry(VirtualRouterElement.java:902)
at com.cloud.network.NetworkManagerImpl.prepareElement(NetworkManagerImpl.java:2079)
at com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2200)
at com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2136)
at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3440)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3000)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2986)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
2014-03-20 12:26:10,859 DEBUG
[cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-15 = [
838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Cleaning up resources for the vm VM[User|test] in Starting state
2014-03-20 12:26:10,861 DEBUG [agent.transport.Request] (Job-Executor-1:job-15 = [ 838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Seq
1-1879572499: Sending { Cmd , MgmtId: 279278805451363, via: 1, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"vmName":"i-2-10-VM","wait":0}}] }
and this:
2014-03-20 12:26:12,826 INFO [user.vm.DeployVMCmd]
(Job-Executor-1:job-15 = [ 838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ])
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|test]Scope=interface com.cloud.dc.DataCenter; id=1
2014-03-20 12:26:12,826 INFO [user.vm.DeployVMCmd]
(Job-Executor-1:job-15 = [ 838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Unable to create a deployment for VM[User|test]
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|test]Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:237)
at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3440)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3000)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2986)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
2014-03-20 12:26:12,828 DEBUG
[cloud.async.AsyncJobManagerImpl] (Job-Executor-1:job-15 = [
838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Complete async job-15 = [
838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ], jobStatus: 2, resultCode: 530,
result: Error Code: 533 Error text: Unable to create a deployment for VM[User|test]
On Thursday, 20 March 2014 9:16 AM, Sugandh S <s....@rocketmail.com>> wrote:
Hi,
Yes, I did try that but it gave me this error "ssh: connect to host 169.254.2.158 port 3922: No route to host".
Sugandh
On Wednesday, 19 March 2014 7:16 PM, Hugues Lepesant <hu...@lepesant.com>> wrote:
Hi,
Do you try to ssh from the host running KVM and hosting the SystemVM ?
hug
-----Message initial-----
De:Sugandh S <s....@rocketmail.com>>
Envoyé:mer. 19-03-2014 13:00
Sujet:Re: vm stuck in starting state, unable to delete it À:Rajesh Battala <ra...@citrix.com>>; users@cloudstack.apache.org<ma...@cloudstack.apache.org>; Sailaja Mada <sa...@citrix.com>>; Suresh Sadhu <Su...@citrix.com>>; Well, the router state seems to be "running" but I am not able to ping any of the system vms via their link local ip address or their public or private ips.
When I try to ssh into console proxy vm, I get this:
root@server2<ma...@server2>:~# ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.2.158<ma...@169.254.2.158>
ssh: connect to host 169.254.2.158 port 3922: No route to host
Thanks,
Sugandh
On Wednesday, 19 March 2014 5:31 PM, Rajesh Battala <ra...@citrix.com>> wrote:
>From the log, issue is while applying the dhcp entry in the VR hence deployment fails.
Can you check whether the VR is up and network is in implemented state.?
Thanks
Rajesh Battala
From:Sugandh S [mailto:s.sugandh@rocketmail.com<ma...@rocketmail.com>]
Sent: Wednesday, March 19, 2014 5:07 PM
To: Rajesh Battala; users@cloudstack.apache.org<ma...@cloudstack.apache.org>; Sailaja Mada; Suresh Sadhu
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
It took a couple of reboots to get system vms and router working again but now I have got another problem, whenever I create an instance I get "Unable to create a deployment for VM[User|<vmname>]" error.
I have pasted the log here:
http://tny.cz/1ee21d5e
On Wednesday, 19 March 2014 4:39 PM, Rajesh Battala <ra...@citrix.com>> wrote:
Can you just capture the log from when you started the action till you see the error.
Thanks
Rajesh Battala
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com<ma...@rocketmail.com>]
Sent: Wednesday, March 19, 2014 4:23 PM
To: Sailaja Mada; users@cloudstack.apache.org<ma...@cloudstack.apache.org>; Suresh Sadhu
Subject: Re: vm stuck in starting state, unable to
delete it
Hi,
The log file is around 10gb, I used pastebinit to upload it and it gave me "memory error".
Is there any other way to provide the log file?
Thanks,
Sugandh
On Wednesday, 19 March 2014 3:56 PM, Sailaja Mada <sa...@citrix.com>> wrote:
Hi,
Can you please send the complete log using PasteBin.
Thanks,
Sailaja.M
From:Sugandh S [mailto:s.sugandh@rocketmail.com<ma...@rocketmail.com>]
Sent: 19 March 2014 15:30
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>; Sailaja Mada; Suresh Sadhu; Sugandh S
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
I just noticed that my domain router is also stuck in starting state and one of the vms I created is now showing error state.
On Wednesday, 19 March 2014 3:25 PM, Sugandh S <s....@rocketmail.com>> wrote:
Hi,
> I have noticed VM in starting state when Template is getting Copied from
Secondary Storage to > Primary Storage .
It's been over 150 minutes and I don't think it should take this long to copy the template.
> size of the template and also value of global config parameter "wait"
size of the iso is 700.29 MB and "wait" value is default "1800".
Sugandh
On Wednesday, 19 March 2014 3:16 PM, Sailaja Mada <sa...@citrix.com>> wrote:
Hi,
I have noticed VM in starting state when Template is getting Copied from Secondary Storage to Primary Storage .
VM gets deployed and will move to
running state after this copy is completed. Can you please share the size of the template and also value of global config parameter "wait"
One reason could be Storage Server is slow and Copy operation is taking longer time. It would help not to time out if you increase the "wait" value . But you may have to wait for the copy operation to complete to get the VM into running state.
Thanks,
Sailaja.M
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com<ma...@rocketmail.com>]
Sent: 19 March 2014 15:01
To: Suresh Sadhu; users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Subject: Re: vm stuck in starting state, unable to delete it
Here is another part of log
e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:58,120 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:59:58,120 WARN [agent.manager.AgentAttache] (StatsCollector-1:null)
Seq 4-698744901: Timed out on null
2014-03-19 13:59:58,120 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698
744901 to Host 4 timed out after 3600
2014-03-19 13:59:58,122 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 stati stics.
2014-03-19 13:59:58,122 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host
: 4
2014-03-19 14:00:07,507 WARN [apache.cloudstack.alerts] (HA-2:null) alertType:: 13 // dataCenterId:: 0 // podId:: 0 //
clusterId:: null // message:: No usage server process running
2014-03-19 14:00:22,973 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in
error code list for exceptions
2014-03-19 14:00:22,973 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661: Timed out on null
2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:58,715 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c
om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:58,715 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902: Timed out on null
2014-03-19 14:00:58,716 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698
744902 to Host 4 timed out after 3600
2014-03-19 14:00:58,716 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 stati stics.
2014-03-19 14:00:58,716 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:01:22,978 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:22,978 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662: Timed out on null
2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:35,900 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3660
2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql]
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:01:59,312 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:59,312 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904: Timed out on null
2014-03-19 14:01:59,312 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744904 to Host 4 timed out after
3600
2014-03-19 14:01:59,312 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:01:59,312 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:02:07,765 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,766 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Failed to deploy vm 4 with original planner, sending HAPlanner
2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable to transition into Starting state due to Unable to transition to a new state from Starting via StartRequested
2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(HA-Worker-0:work-5) Determining why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM]. Retry=4
2014-03-19 14:02:07,812 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,812 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:02:07,812 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-5) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM which was running on host name: server2(id:1), availability zone: zone1,
pod: pod1
2014-03-19 14:02:22,983 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find
exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:22,983 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663: Timed out on null
2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:59,904 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception:
com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:59,904 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905: Timed out on null
2014-03-19 14:02:59,905 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698
744905 to Host 4 timed out after 3600
2014-03-19 14:02:59,905 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 14:02:59,905 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 14:03:22,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19
14:03:22,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664: Timed out on null
2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:04:00,500 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:00,500
WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906: Timed out on null
2014-03-19 14:04:00,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744906 to Host 4 timed out after 3600
2014-03-19 14:04:00,501 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:04:00,501 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:04:22,993 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:22,993 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665: Timed out on null
2014-03-19
14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:05:01,092 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:01,092 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907: Timed out on null
2014-03-19 14:05:01,092
WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744907 to Host 4 timed out after 3600
2014-03-19 14:05:01,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 14:05:01,092 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:05:23,000 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:23,000 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666: Timed out on null
2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3,
com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:06:01,680 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:01,680 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908: Timed out on null
2014-03-19 14:06:01,680 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744908 to Host 4 timed
out after 3600
2014-03-19 14:06:01,680 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:06:01,681 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:06:23,004 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:23,004 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667: Timed out on null
2014-03-19 14:06:23,004 DEB
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com>> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com<ma...@rocketmail.com>]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Subject: vm stuck
in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary
storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
Re: vm stuck in starting state, unable to delete it
Posted by Sugandh S <s....@rocketmail.com>.
Okay, so I managed to solve the problem. I had to reboot the router vm and once that was done, I could create instances.
Thanks everyone. Keep up the good work!
Cheers,
Sugandh
On Thursday, 20 March 2014 3:02 PM, Sugandh S <s....@rocketmail.com> wrote:
Hi
> Try stopping the firewall on your machine.
>
> Service iptables stop.
I have stopped the firewall, it was done by stopping ufw by issuing "ufw disable".
> Also try this ..check whether your able to access system vms including router vm from host:
I am able to access console proxy and ssvm but I cant access router vm, I can ping it though.
Sugandh
On Thursday, 20 March 2014 12:53 PM, Suresh Sadhu <Su...@citrix.com> wrote:
Try stopping the firewall on your machine.
Service iptables stop.
Also try this ..check whether your able to access system vms including router vm from host:
enServer/KVM Hypervisors
Connect to the Host of which the System VM is running.
SSH the 'Link Local IP Address' of the System VM from the Host on which the VM is running.
Format: ssh -i <path-to-private-key> <link-local-ip> -p 3922
Example: root@faith:~# ssh -i /root/.ssh/id_rsa.cloud 169.254.3.93 -p 3922
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 20 March 2014 12:46
To: users@cloudstack.apache.org; Sugandh S
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
So, I have finally managed to access console proxy vm but I am still unable to create instances.
I have pasted the log file here: http://pastebin.com/UrjLuBiM
Here are the exceptions from the log that I got:
2014-03-20 12:26:10,766 DEBUG [agent.transport.Request] (AgentManager-Handler-15:null) Seq 1-1879572498: Processing: { Ans: ,
MgmtId: 279278805451363, via: 1, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"ssh: connect to host 169.254.3.71 port 3922: No route to host","wait":0}}] }
2014-03-20 12:26:10,766 DEBUG
[agent.manager.AgentAttache] (AgentManager-Handler-15:null) Seq
1-1879572498: No more commands found
2014-03-20 12:26:10,766 DEBUG [agent.transport.Request] (Job-Executor-1:job-15 = [ 838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Seq
1-1879572498: Received: { Ans: , MgmtId: 279278805451363, via: 1, Ver:
v1, Flags: 110, { Answer } }
2014-03-20 12:26:10,766 INFO
[cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-15 = [
838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Unable to contact resource.
com.cloud.exception.ResourceUnavailableException: Resource [Pod:1] is unreachable: Unable to apply dhcp entry on router
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3826)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2943)
at com.cloud.network.element.VirtualRouterElement.addDhcpEntry(VirtualRouterElement.java:902)
at com.cloud.network.NetworkManagerImpl.prepareElement(NetworkManagerImpl.java:2079)
at com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2200)
at com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2136)
at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3440)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3000)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2986)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
2014-03-20 12:26:10,859 DEBUG
[cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-15 = [
838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Cleaning up resources for the vm VM[User|test] in Starting state
2014-03-20 12:26:10,861 DEBUG [agent.transport.Request] (Job-Executor-1:job-15 = [ 838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Seq
1-1879572499: Sending { Cmd , MgmtId: 279278805451363, via: 1, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"vmName":"i-2-10-VM","wait":0}}] }
and this:
2014-03-20 12:26:12,826 INFO [user.vm.DeployVMCmd]
(Job-Executor-1:job-15 = [ 838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ])
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|test]Scope=interface com.cloud.dc.DataCenter; id=1
2014-03-20 12:26:12,826 INFO [user.vm.DeployVMCmd]
(Job-Executor-1:job-15 = [ 838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Unable to create a deployment for VM[User|test]
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|test]Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:237)
at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3440)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3000)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2986)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
2014-03-20 12:26:12,828 DEBUG
[cloud.async.AsyncJobManagerImpl] (Job-Executor-1:job-15 = [
838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Complete async job-15 = [
838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ], jobStatus: 2, resultCode: 530,
result: Error Code: 533 Error text: Unable to create a deployment for VM[User|test]
On Thursday, 20 March 2014 9:16 AM, Sugandh S <s....@rocketmail.com> wrote:
Hi,
Yes, I did try that but it gave me this error "ssh: connect to host 169.254.2.158 port 3922: No route to host".
Sugandh
On Wednesday, 19 March 2014 7:16 PM, Hugues Lepesant <hu...@lepesant.com> wrote:
Hi,
Do you try to ssh from the host running KVM and hosting the SystemVM ?
hug
-----Message initial-----
De:Sugandh S <s....@rocketmail.com>
Envoyé:mer. 19-03-2014 13:00
Sujet:Re: vm stuck in starting state, unable to delete it À:Rajesh Battala <ra...@citrix.com>; users@cloudstack.apache.org; Sailaja Mada <sa...@citrix.com>; Suresh Sadhu <Su...@citrix.com>; Well, the router state seems to be "running" but I am not able to ping any of the system vms via their link local ip address or their public or private ips.
When I try to ssh into console proxy vm, I get this:
root@server2:~# ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.2.158
ssh: connect to host 169.254.2.158 port 3922: No route to host
Thanks,
Sugandh
On Wednesday, 19 March 2014 5:31 PM, Rajesh Battala <ra...@citrix.com> wrote:
From the log, issue is while applying the dhcp entry in the VR hence deployment fails.
Can you check whether the VR is up and network is in implemented state.?
Thanks
Rajesh Battala
From:Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: Wednesday, March 19, 2014 5:07 PM
To: Rajesh Battala; users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
It took a couple of reboots to get system vms and router working again but now I have got another problem, whenever I create an instance I get "Unable to create a deployment for VM[User|<vmname>]" error.
I have pasted the log here:
http://tny.cz/1ee21d5e
On Wednesday, 19 March 2014 4:39 PM, Rajesh Battala <ra...@citrix.com> wrote:
Can you just capture the log from when you started the action till you see the error.
Thanks
Rajesh Battala
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: Wednesday, March 19, 2014 4:23 PM
To: Sailaja Mada; users@cloudstack.apache.org; Suresh Sadhu
Subject: Re: vm stuck in starting state, unable to
delete it
Hi,
The log file is around 10gb, I used pastebinit to upload it and it gave me "memory error".
Is there any other way to provide the log file?
Thanks,
Sugandh
On Wednesday, 19 March 2014 3:56 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
Can you please send the complete log using PasteBin.
Thanks,
Sailaja.M
From:Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:30
To: users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu; Sugandh S
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
I just noticed that my domain router is also stuck in starting state and one of the vms I created is now showing error state.
On Wednesday, 19 March 2014 3:25 PM, Sugandh S <s....@rocketmail.com> wrote:
Hi,
> I have noticed VM in starting state when Template is getting Copied from
Secondary Storage to > Primary Storage .
It's been over 150 minutes and I don't think it should take this long to copy the template.
> size of the template and also value of global config parameter "wait"
size of the iso is 700.29 MB and "wait" value is default "1800".
Sugandh
On Wednesday, 19 March 2014 3:16 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
I have noticed VM in starting state when Template is getting Copied from Secondary Storage to Primary Storage .
VM gets deployed and will move to
running state after this copy is completed. Can you please share the size of the template and also value of global config parameter "wait"
One reason could be Storage Server is slow and Copy operation is taking longer time. It would help not to time out if you increase the "wait" value . But you may have to wait for the copy operation to complete to get the VM into running state.
Thanks,
Sailaja.M
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:01
To: Suresh Sadhu; users@cloudstack.apache.org
Subject: Re: vm stuck in starting state, unable to delete it
Here is another part of log
e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:58,120 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:59:58,120 WARN [agent.manager.AgentAttache] (StatsCollector-1:null)
Seq 4-698744901: Timed out on null
2014-03-19 13:59:58,120 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698
744901 to Host 4 timed out after 3600
2014-03-19 13:59:58,122 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 stati stics.
2014-03-19 13:59:58,122 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host
: 4
2014-03-19 14:00:07,507 WARN [apache.cloudstack.alerts] (HA-2:null) alertType:: 13 // dataCenterId:: 0 // podId:: 0 //
clusterId:: null // message:: No usage server process running
2014-03-19 14:00:22,973 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in
error code list for exceptions
2014-03-19 14:00:22,973 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661: Timed out on null
2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:58,715 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c
om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:58,715 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902: Timed out on null
2014-03-19 14:00:58,716 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698
744902 to Host 4 timed out after 3600
2014-03-19 14:00:58,716 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 stati stics.
2014-03-19 14:00:58,716 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:01:22,978 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:22,978 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662: Timed out on null
2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:35,900 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3660
2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql]
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:01:59,312 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:59,312 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904: Timed out on null
2014-03-19 14:01:59,312 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744904 to Host 4 timed out after
3600
2014-03-19 14:01:59,312 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:01:59,312 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:02:07,765 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,766 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Failed to deploy vm 4 with original planner, sending HAPlanner
2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable to transition into Starting state due to Unable to transition to a new state from Starting via StartRequested
2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(HA-Worker-0:work-5) Determining why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM]. Retry=4
2014-03-19 14:02:07,812 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,812 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:02:07,812 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-5) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM which was running on host name: server2(id:1), availability zone: zone1,
pod: pod1
2014-03-19 14:02:22,983 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find
exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:22,983 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663: Timed out on null
2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:59,904 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception:
com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:59,904 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905: Timed out on null
2014-03-19 14:02:59,905 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698
744905 to Host 4 timed out after 3600
2014-03-19 14:02:59,905 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 14:02:59,905 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 14:03:22,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19
14:03:22,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664: Timed out on null
2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:04:00,500 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:00,500
WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906: Timed out on null
2014-03-19 14:04:00,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744906 to Host 4 timed out after 3600
2014-03-19 14:04:00,501 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:04:00,501 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:04:22,993 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:22,993 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665: Timed out on null
2014-03-19
14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:05:01,092 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:01,092 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907: Timed out on null
2014-03-19 14:05:01,092
WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744907 to Host 4 timed out after 3600
2014-03-19 14:05:01,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 14:05:01,092 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:05:23,000 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:23,000 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666: Timed out on null
2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3,
com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:06:01,680 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:01,680 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908: Timed out on null
2014-03-19 14:06:01,680 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744908 to Host 4 timed
out after 3600
2014-03-19 14:06:01,680 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:06:01,681 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:06:23,004 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:23,004 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667: Timed out on null
2014-03-19 14:06:23,004 DEB
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org
Subject: vm stuck
in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary
storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
Re: vm stuck in starting state, unable to delete it
Posted by Sugandh S <s....@rocketmail.com>.
Hi
> Try stopping the firewall on your machine.
>
> Service iptables stop.
I have stopped the firewall, it was done by stopping ufw by issuing "ufw disable".
> Also try this ..check whether your able to access system vms including router vm from host:
I am able to access console proxy and ssvm but I cant access router vm, I can ping it though.
Sugandh
On Thursday, 20 March 2014 12:53 PM, Suresh Sadhu <Su...@citrix.com> wrote:
Try stopping the firewall on your machine.
Service iptables stop.
Also try this ..check whether your able to access system vms including router vm from host:
enServer/KVM Hypervisors
Connect to the Host of which the System VM is running.
SSH the 'Link Local IP Address' of the System VM from the Host on which the VM is running.
Format: ssh -i <path-to-private-key> <link-local-ip> -p 3922
Example: root@faith:~# ssh -i /root/.ssh/id_rsa.cloud 169.254.3.93 -p 3922
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 20 March 2014 12:46
To: users@cloudstack.apache.org; Sugandh S
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
So, I have finally managed to access console proxy vm but I am still unable to create instances.
I have pasted the log file here: http://pastebin.com/UrjLuBiM
Here are the exceptions from the log that I got:
2014-03-20 12:26:10,766 DEBUG [agent.transport.Request] (AgentManager-Handler-15:null) Seq 1-1879572498: Processing: { Ans: ,
MgmtId: 279278805451363, via: 1, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"ssh: connect to host 169.254.3.71 port 3922: No route to host","wait":0}}] }
2014-03-20 12:26:10,766 DEBUG
[agent.manager.AgentAttache] (AgentManager-Handler-15:null) Seq
1-1879572498: No more commands found
2014-03-20 12:26:10,766 DEBUG [agent.transport.Request] (Job-Executor-1:job-15 = [ 838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Seq
1-1879572498: Received: { Ans: , MgmtId: 279278805451363, via: 1, Ver:
v1, Flags: 110, { Answer } }
2014-03-20 12:26:10,766 INFO
[cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-15 = [
838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Unable to contact resource.
com.cloud.exception.ResourceUnavailableException: Resource [Pod:1] is unreachable: Unable to apply dhcp entry on router
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3826)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2943)
at com.cloud.network.element.VirtualRouterElement.addDhcpEntry(VirtualRouterElement.java:902)
at com.cloud.network.NetworkManagerImpl.prepareElement(NetworkManagerImpl.java:2079)
at com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2200)
at com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2136)
at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3440)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3000)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2986)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
2014-03-20 12:26:10,859 DEBUG
[cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-15 = [
838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Cleaning up resources for the vm VM[User|test] in Starting state
2014-03-20 12:26:10,861 DEBUG [agent.transport.Request] (Job-Executor-1:job-15 = [ 838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Seq
1-1879572499: Sending { Cmd , MgmtId: 279278805451363, via: 1, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"vmName":"i-2-10-VM","wait":0}}] }
and this:
2014-03-20 12:26:12,826 INFO [user.vm.DeployVMCmd]
(Job-Executor-1:job-15 = [ 838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ])
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|test]Scope=interface com.cloud.dc.DataCenter; id=1
2014-03-20 12:26:12,826 INFO [user.vm.DeployVMCmd]
(Job-Executor-1:job-15 = [ 838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Unable to create a deployment for VM[User|test]
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|test]Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:237)
at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3440)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3000)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2986)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
2014-03-20 12:26:12,828 DEBUG
[cloud.async.AsyncJobManagerImpl] (Job-Executor-1:job-15 = [
838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Complete async job-15 = [
838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ], jobStatus: 2, resultCode: 530,
result: Error Code: 533 Error text: Unable to create a deployment for VM[User|test]
On Thursday, 20 March 2014 9:16 AM, Sugandh S <s....@rocketmail.com> wrote:
Hi,
Yes, I did try that but it gave me this error "ssh: connect to host 169.254.2.158 port 3922: No route to host".
Sugandh
On Wednesday, 19 March 2014 7:16 PM, Hugues Lepesant <hu...@lepesant.com> wrote:
Hi,
Do you try to ssh from the host running KVM and hosting the SystemVM ?
hug
-----Message initial-----
De:Sugandh S <s....@rocketmail.com>
Envoyé:mer. 19-03-2014 13:00
Sujet:Re: vm stuck in starting state, unable to delete it À:Rajesh Battala <ra...@citrix.com>; users@cloudstack.apache.org; Sailaja Mada <sa...@citrix.com>; Suresh Sadhu <Su...@citrix.com>; Well, the router state seems to be "running" but I am not able to ping any of the system vms via their link local ip address or their public or private ips.
When I try to ssh into console proxy vm, I get this:
root@server2:~# ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.2.158
ssh: connect to host 169.254.2.158 port 3922: No route to host
Thanks,
Sugandh
On Wednesday, 19 March 2014 5:31 PM, Rajesh Battala <ra...@citrix.com> wrote:
From the log, issue is while applying the dhcp entry in the VR hence deployment fails.
Can you check whether the VR is up and network is in implemented state.?
Thanks
Rajesh Battala
From:Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: Wednesday, March 19, 2014 5:07 PM
To: Rajesh Battala; users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
It took a couple of reboots to get system vms and router working again but now I have got another problem, whenever I create an instance I get "Unable to create a deployment for VM[User|<vmname>]" error.
I have pasted the log here:
http://tny.cz/1ee21d5e
On Wednesday, 19 March 2014 4:39 PM, Rajesh Battala <ra...@citrix.com> wrote:
Can you just capture the log from when you started the action till you see the error.
Thanks
Rajesh Battala
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: Wednesday, March 19, 2014 4:23 PM
To: Sailaja Mada; users@cloudstack.apache.org; Suresh Sadhu
Subject: Re: vm stuck in starting state, unable to
delete it
Hi,
The log file is around 10gb, I used pastebinit to upload it and it gave me "memory error".
Is there any other way to provide the log file?
Thanks,
Sugandh
On Wednesday, 19 March 2014 3:56 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
Can you please send the complete log using PasteBin.
Thanks,
Sailaja.M
From:Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:30
To: users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu; Sugandh S
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
I just noticed that my domain router is also stuck in starting state and one of the vms I created is now showing error state.
On Wednesday, 19 March 2014 3:25 PM, Sugandh S <s....@rocketmail.com> wrote:
Hi,
> I have noticed VM in starting state when Template is getting Copied from
Secondary Storage to > Primary Storage .
It's been over 150 minutes and I don't think it should take this long to copy the template.
> size of the template and also value of global config parameter "wait"
size of the iso is 700.29 MB and "wait" value is default "1800".
Sugandh
On Wednesday, 19 March 2014 3:16 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
I have noticed VM in starting state when Template is getting Copied from Secondary Storage to Primary Storage .
VM gets deployed and will move to
running state after this copy is completed. Can you please share the size of the template and also value of global config parameter "wait"
One reason could be Storage Server is slow and Copy operation is taking longer time. It would help not to time out if you increase the "wait" value . But you may have to wait for the copy operation to complete to get the VM into running state.
Thanks,
Sailaja.M
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:01
To: Suresh Sadhu; users@cloudstack.apache.org
Subject: Re: vm stuck in starting state, unable to delete it
Here is another part of log
e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:58,120 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:59:58,120 WARN [agent.manager.AgentAttache] (StatsCollector-1:null)
Seq 4-698744901: Timed out on null
2014-03-19 13:59:58,120 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698
744901 to Host 4 timed out after 3600
2014-03-19 13:59:58,122 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 stati stics.
2014-03-19 13:59:58,122 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host
: 4
2014-03-19 14:00:07,507 WARN [apache.cloudstack.alerts] (HA-2:null) alertType:: 13 // dataCenterId:: 0 // podId:: 0 //
clusterId:: null // message:: No usage server process running
2014-03-19 14:00:22,973 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in
error code list for exceptions
2014-03-19 14:00:22,973 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661: Timed out on null
2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:58,715 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c
om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:58,715 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902: Timed out on null
2014-03-19 14:00:58,716 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698
744902 to Host 4 timed out after 3600
2014-03-19 14:00:58,716 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 stati stics.
2014-03-19 14:00:58,716 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:01:22,978 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:22,978 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662: Timed out on null
2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:35,900 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3660
2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql]
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:01:59,312 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:59,312 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904: Timed out on null
2014-03-19 14:01:59,312 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744904 to Host 4 timed out after
3600
2014-03-19 14:01:59,312 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:01:59,312 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:02:07,765 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,766 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Failed to deploy vm 4 with original planner, sending HAPlanner
2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable to transition into Starting state due to Unable to transition to a new state from Starting via StartRequested
2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(HA-Worker-0:work-5) Determining why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM]. Retry=4
2014-03-19 14:02:07,812 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,812 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:02:07,812 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-5) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM which was running on host name: server2(id:1), availability zone: zone1,
pod: pod1
2014-03-19 14:02:22,983 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find
exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:22,983 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663: Timed out on null
2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:59,904 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception:
com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:59,904 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905: Timed out on null
2014-03-19 14:02:59,905 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698
744905 to Host 4 timed out after 3600
2014-03-19 14:02:59,905 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 14:02:59,905 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 14:03:22,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19
14:03:22,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664: Timed out on null
2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:04:00,500 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:00,500
WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906: Timed out on null
2014-03-19 14:04:00,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744906 to Host 4 timed out after 3600
2014-03-19 14:04:00,501 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:04:00,501 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:04:22,993 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:22,993 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665: Timed out on null
2014-03-19
14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:05:01,092 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:01,092 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907: Timed out on null
2014-03-19 14:05:01,092
WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744907 to Host 4 timed out after 3600
2014-03-19 14:05:01,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 14:05:01,092 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:05:23,000 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:23,000 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666: Timed out on null
2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3,
com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:06:01,680 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:01,680 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908: Timed out on null
2014-03-19 14:06:01,680 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744908 to Host 4 timed
out after 3600
2014-03-19 14:06:01,680 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:06:01,681 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:06:23,004 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:23,004 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667: Timed out on null
2014-03-19 14:06:23,004 DEB
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org
Subject: vm stuck
in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary
storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
Re: vm stuck in starting state, unable to delete it
Posted by Sugandh S <s....@rocketmail.com>.
Hi,
So, I have finally managed to access console proxy vm but I am still unable to create instances.
I have pasted the log file here: http://pastebin.com/UrjLuBiM
Here are the exceptions from the log that I got:
2014-03-20 12:26:10,766 DEBUG [agent.transport.Request] (AgentManager-Handler-15:null) Seq 1-1879572498: Processing: { Ans: ,
MgmtId: 279278805451363, via: 1, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"ssh: connect
to host 169.254.3.71 port 3922: No route to host","wait":0}}] }
2014-03-20 12:26:10,766 DEBUG
[agent.manager.AgentAttache] (AgentManager-Handler-15:null) Seq
1-1879572498: No more commands found
2014-03-20 12:26:10,766 DEBUG [agent.transport.Request] (Job-Executor-1:job-15 = [ 838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Seq
1-1879572498: Received: { Ans: , MgmtId: 279278805451363, via: 1, Ver:
v1, Flags: 110, { Answer } }
2014-03-20 12:26:10,766 INFO
[cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-15 = [
838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Unable to contact resource.
com.cloud.exception.ResourceUnavailableException: Resource [Pod:1] is unreachable: Unable to apply dhcp entry on router
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3826)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2943)
at com.cloud.network.element.VirtualRouterElement.addDhcpEntry(VirtualRouterElement.java:902)
at com.cloud.network.NetworkManagerImpl.prepareElement(NetworkManagerImpl.java:2079)
at com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2200)
at com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2136)
at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3440)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3000)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2986)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
2014-03-20 12:26:10,859 DEBUG
[cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-15 = [
838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Cleaning up resources for the vm VM[User|test] in Starting state
2014-03-20 12:26:10,861 DEBUG [agent.transport.Request] (Job-Executor-1:job-15 = [ 838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Seq
1-1879572499: Sending { Cmd , MgmtId: 279278805451363, via: 1, Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"vmName":"i-2-10-VM","wait":0}}] }
and this:
2014-03-20 12:26:12,826 INFO [user.vm.DeployVMCmd]
(Job-Executor-1:job-15 = [ 838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ])
com.cloud.exception.InsufficientServerCapacityException: Unable to
create a deployment for VM[User|test]Scope=interface
com.cloud.dc.DataCenter; id=1
2014-03-20 12:26:12,826 INFO [user.vm.DeployVMCmd]
(Job-Executor-1:job-15 = [ 838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ])
Unable to create a deployment for VM[User|test]
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|test]Scope=interface
com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:237)
at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3440)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3000)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2986)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
2014-03-20 12:26:12,828 DEBUG
[cloud.async.AsyncJobManagerImpl] (Job-Executor-1:job-15 = [
838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ]) Complete async job-15 = [
838a99c8-ebe5-4ce6-bcf9-5187cddbd945 ], jobStatus: 2, resultCode: 530,
result: Error Code: 533 Error text: Unable to create a deployment for
VM[User|test]
On Thursday, 20 March 2014 9:16 AM, Sugandh S <s....@rocketmail.com> wrote:
Hi,
Yes, I did try that but it gave me this error "ssh: connect to host 169.254.2.158 port 3922: No route to
host".
Sugandh
On Wednesday, 19 March 2014 7:16 PM, Hugues Lepesant <hu...@lepesant.com> wrote:
Hi,
Do you try to ssh from the host running KVM and hosting the SystemVM ?
hug
-----Message initial-----
De:Sugandh S <s....@rocketmail.com>
Envoyé:mer. 19-03-2014 13:00
Sujet:Re: vm stuck in starting state, unable to delete it
À:Rajesh Battala <ra...@citrix.com>; users@cloudstack.apache.org; Sailaja Mada <sa...@citrix.com>; Suresh Sadhu <Su...@citrix.com>;
Well, the router state seems to be "running" but I am not able to ping any of the system vms via their link local ip address or their public or private ips.
When I try to ssh into console proxy vm, I get this:
root@server2:~# ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.2.158
ssh: connect to host 169.254.2.158 port 3922: No route to host
Thanks,
Sugandh
On Wednesday, 19 March 2014 5:31 PM, Rajesh Battala <ra...@citrix.com> wrote:
>From the log, issue is while applying the dhcp entry in the VR hence deployment fails.
Can you check whether the VR is up and network is in implemented state.?
Thanks
Rajesh Battala
From:Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: Wednesday, March 19, 2014 5:07 PM
To: Rajesh Battala; users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
It took a couple of reboots to get system vms and router working again but now I have got another problem, whenever I create an instance I get "Unable to create a deployment for VM[User|<vmname>]" error.
I have pasted the log here:
http://tny.cz/1ee21d5e
On Wednesday, 19 March 2014 4:39 PM, Rajesh Battala <ra...@citrix.com> wrote:
Can you just capture the log from when you started the action till you see the error.
Thanks
Rajesh Battala
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: Wednesday, March 19, 2014 4:23 PM
To: Sailaja Mada; users@cloudstack.apache.org; Suresh Sadhu
Subject: Re: vm stuck in starting state, unable to
delete it
Hi,
The log file is around 10gb, I used pastebinit to upload it and it gave me "memory error".
Is there any other way to provide the log file?
Thanks,
Sugandh
On Wednesday, 19 March 2014 3:56 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
Can you please send the complete log using PasteBin.
Thanks,
Sailaja.M
From:Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:30
To: users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu; Sugandh S
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
I just noticed that my domain router is also stuck in starting state and one of the vms I created is now showing error state.
On Wednesday, 19 March 2014 3:25 PM, Sugandh S <s....@rocketmail.com> wrote:
Hi,
> I have noticed VM in starting state when Template is getting Copied from
Secondary Storage to > Primary Storage .
It's been over 150 minutes and I don't think it should take this long to copy the template.
> size of the template and also value of global config parameter "wait"
size of the iso is 700.29 MB and "wait" value is default "1800".
Sugandh
On Wednesday, 19 March 2014 3:16 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
I have noticed VM in starting state when Template is getting Copied from Secondary Storage to Primary Storage .
VM gets deployed and will move to
running state after this copy is completed. Can you please share the size of the template and also value of global config parameter "wait"
One reason could be Storage Server is slow and Copy operation is taking longer time. It would help not to time out if you increase the "wait" value . But you may have to wait for the copy operation to complete to get the VM into running state.
Thanks,
Sailaja.M
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:01
To: Suresh Sadhu; users@cloudstack.apache.org
Subject: Re: vm stuck in starting state, unable to delete it
Here is another part of log
e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:58,120 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:59:58,120 WARN [agent.manager.AgentAttache] (StatsCollector-1:null)
Seq 4-698744901: Timed out on null
2014-03-19 13:59:58,120 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698
744901 to Host 4 timed out after 3600
2014-03-19 13:59:58,122 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 stati stics.
2014-03-19 13:59:58,122 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host
: 4
2014-03-19 14:00:07,507 WARN [apache.cloudstack.alerts] (HA-2:null) alertType:: 13 // dataCenterId:: 0 // podId:: 0 //
clusterId:: null // message:: No usage server process running
2014-03-19 14:00:22,973 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in
error code list for exceptions
2014-03-19 14:00:22,973 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661: Timed out on null
2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:58,715 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c
om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:58,715 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902: Timed out on null
2014-03-19 14:00:58,716 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698
744902 to Host 4 timed out after 3600
2014-03-19 14:00:58,716 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 stati stics.
2014-03-19 14:00:58,716 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:01:22,978 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:22,978 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662: Timed out on null
2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:35,900 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3660
2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql]
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:01:59,312 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:59,312 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904: Timed out on null
2014-03-19 14:01:59,312 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744904 to Host 4 timed out after
3600
2014-03-19 14:01:59,312 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:01:59,312 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:02:07,765 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,766 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Failed to deploy vm 4 with original planner, sending HAPlanner
2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable to transition into Starting state due to Unable to transition to a new state from Starting via StartRequested
2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(HA-Worker-0:work-5) Determining why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM]. Retry=4
2014-03-19 14:02:07,812 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,812 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:02:07,812 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-5) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM which was running on host name: server2(id:1), availability zone: zone1,
pod: pod1
2014-03-19 14:02:22,983 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find
exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:22,983 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663: Timed out on null
2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:59,904 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception:
com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:59,904 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905: Timed out on null
2014-03-19 14:02:59,905 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698
744905 to Host 4 timed out after 3600
2014-03-19 14:02:59,905 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 14:02:59,905 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 14:03:22,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19
14:03:22,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664: Timed out on null
2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:04:00,500 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:00,500
WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906: Timed out on null
2014-03-19 14:04:00,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744906 to Host 4 timed out after 3600
2014-03-19 14:04:00,501 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:04:00,501 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:04:22,993 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:22,993 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665: Timed out on null
2014-03-19
14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:05:01,092 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:01,092 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907: Timed out on null
2014-03-19 14:05:01,092
WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744907 to Host 4 timed out after 3600
2014-03-19 14:05:01,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 14:05:01,092 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:05:23,000 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:23,000 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666: Timed out on null
2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3,
com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:06:01,680 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:01,680 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908: Timed out on null
2014-03-19 14:06:01,680 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744908 to Host 4 timed
out after 3600
2014-03-19 14:06:01,680 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:06:01,681 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:06:23,004 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:23,004 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667: Timed out on null
2014-03-19 14:06:23,004 DEB
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org
Subject: vm stuck
in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary
storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
Re: vm stuck in starting state, unable to delete it
Posted by Sugandh S <s....@rocketmail.com>.
Hi,
Yes, I did try that but it gave me this error "ssh: connect to host 169.254.2.158 port 3922: No route to host".
Sugandh
On Wednesday, 19 March 2014 7:16 PM, Hugues Lepesant <hu...@lepesant.com> wrote:
Hi,
Do you try to ssh from the host running KVM and hosting the SystemVM ?
hug
-----Message initial-----
De:Sugandh S <s....@rocketmail.com>
Envoyé:mer. 19-03-2014 13:00
Sujet:Re: vm stuck in starting state, unable to delete it
À:Rajesh Battala <ra...@citrix.com>; users@cloudstack.apache.org; Sailaja Mada <sa...@citrix.com>; Suresh Sadhu <Su...@citrix.com>;
Well, the router state seems to be "running" but I am not able to ping any of the system vms via their link local ip address or their public or private ips.
When I try to ssh into console proxy vm, I get this:
root@server2:~# ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.2.158
ssh: connect to host 169.254.2.158 port 3922: No route to host
Thanks,
Sugandh
On Wednesday, 19 March 2014 5:31 PM, Rajesh Battala <ra...@citrix.com> wrote:
From the log, issue is while applying the dhcp entry in the VR hence deployment fails.
Can you check whether the VR is up and network is in implemented state.?
Thanks
Rajesh Battala
From:Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: Wednesday, March 19, 2014 5:07 PM
To: Rajesh Battala; users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
It took a couple of reboots to get system vms and router working again but now I have got another problem, whenever I create an instance I get "Unable to create a deployment for VM[User|<vmname>]" error.
I have pasted the log here:
http://tny.cz/1ee21d5e
On Wednesday, 19 March 2014 4:39 PM, Rajesh Battala <ra...@citrix.com> wrote:
Can you just capture the log from when you started the action till you see the error.
Thanks
Rajesh Battala
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: Wednesday, March 19, 2014 4:23 PM
To: Sailaja Mada; users@cloudstack.apache.org; Suresh Sadhu
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
The log file is around 10gb, I used pastebinit to upload it and it gave me "memory error".
Is there any other way to provide the log file?
Thanks,
Sugandh
On Wednesday, 19 March 2014 3:56 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
Can you please send the complete log using PasteBin.
Thanks,
Sailaja.M
From:Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:30
To: users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu; Sugandh S
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
I just noticed that my domain router is also stuck in starting state and one of the vms I created is now showing error state.
On Wednesday, 19 March 2014 3:25 PM, Sugandh S <s....@rocketmail.com> wrote:
Hi,
> I have noticed VM in starting state when Template is getting Copied from Secondary Storage to > Primary Storage .
It's been over 150 minutes and I don't think it should take this long to copy the template.
> size of the template and also value of global config parameter "wait"
size of the iso is 700.29 MB and "wait" value is default "1800".
Sugandh
On Wednesday, 19 March 2014 3:16 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
I have noticed VM in starting state when Template is getting Copied from Secondary Storage to Primary Storage .
VM gets deployed and will move to running state after this copy is completed. Can you please share the size of the template and also value of global config parameter "wait"
One reason could be Storage Server is slow and Copy operation is taking longer time. It would help not to time out if you increase the "wait" value . But you may have to wait for the copy operation to complete to get the VM into running state.
Thanks,
Sailaja.M
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:01
To: Suresh Sadhu; users@cloudstack.apache.org
Subject: Re: vm stuck in starting state, unable to delete it
Here is another part of log
e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:58,120 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:59:58,120 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744901: Timed out on null
2014-03-19 13:59:58,120 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698
744901 to Host 4 timed out after 3600
2014-03-19 13:59:58,122 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 stati stics.
2014-03-19 13:59:58,122 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host
: 4
2014-03-19 14:00:07,507 WARN [apache.cloudstack.alerts] (HA-2:null) alertType:: 13 // dataCenterId:: 0 // podId:: 0 //
clusterId:: null // message:: No usage server process running
2014-03-19 14:00:22,973 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:22,973 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661: Timed out on null
2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:58,715 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:58,715 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902: Timed out on null
2014-03-19 14:00:58,716 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698
744902 to Host 4 timed out after 3600
2014-03-19 14:00:58,716 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 stati stics.
2014-03-19 14:00:58,716 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:01:22,978 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:22,978 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662: Timed out on null
2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:35,900 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3660
2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql]
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:01:59,312 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:59,312 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904: Timed out on null
2014-03-19 14:01:59,312 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744904 to Host 4 timed out after 3600
2014-03-19 14:01:59,312 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:01:59,312 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:02:07,765 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,766 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Failed to deploy vm 4 with original planner, sending HAPlanner
2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable to transition into Starting state due to Unable to transition to a new state from Starting via StartRequested
2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Determining why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM]. Retry=4
2014-03-19 14:02:07,812 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,812 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:02:07,812 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-5) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM which was running on host name: server2(id:1), availability zone: zone1,
pod: pod1
2014-03-19 14:02:22,983 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:22,983 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663: Timed out on null
2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:59,904 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:59,904 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905: Timed out on null
2014-03-19 14:02:59,905 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698
744905 to Host 4 timed out after 3600
2014-03-19 14:02:59,905 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 14:02:59,905 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 14:03:22,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:03:22,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664: Timed out on null
2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:04:00,500 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:00,500 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906: Timed out on null
2014-03-19 14:04:00,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744906 to Host 4 timed out after 3600
2014-03-19 14:04:00,501 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:04:00,501 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:04:22,993 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:22,993 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665: Timed out on null
2014-03-19 14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:05:01,092 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:01,092 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907: Timed out on null
2014-03-19 14:05:01,092 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744907 to Host 4 timed out after 3600
2014-03-19 14:05:01,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 14:05:01,092 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:05:23,000 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:23,000 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666: Timed out on null
2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:06:01,680 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:01,680 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908: Timed out on null
2014-03-19 14:06:01,680 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744908 to Host 4 timed out after 3600
2014-03-19 14:06:01,680 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:06:01,681 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:06:23,004 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:23,004 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667: Timed out on null
2014-03-19 14:06:23,004 DEB
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org
Subject: vm stuck in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary
storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
RE: vm stuck in starting state, unable to delete it
Posted by Hugues Lepesant <hu...@lepesant.com>.
Hi,
Do you try to ssh from the host running KVM and hosting the SystemVM ?
hug
-----Message initial-----
De:Sugandh S <s....@rocketmail.com>
Envoyé:mer. 19-03-2014 13:00
Sujet:Re: vm stuck in starting state, unable to delete it
À:Rajesh Battala <ra...@citrix.com>; users@cloudstack.apache.org; Sailaja Mada <sa...@citrix.com>; Suresh Sadhu <Su...@citrix.com>;
Well, the router state seems to be "running" but I am not able to ping any of the system vms via their link local ip address or their public or private ips.
When I try to ssh into console proxy vm, I get this:
root@server2:~# ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.2.158
ssh: connect to host 169.254.2.158 port 3922: No route to host
Thanks,
Sugandh
On Wednesday, 19 March 2014 5:31 PM, Rajesh Battala <ra...@citrix.com> wrote:
From the log, issue is while applying the dhcp entry in the VR hence deployment fails.
Can you check whether the VR is up and network is in implemented state.?
Thanks
Rajesh Battala
From:Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: Wednesday, March 19, 2014 5:07 PM
To: Rajesh Battala; users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
It took a couple of reboots to get system vms and router working again but now I have got another problem, whenever I create an instance I get "Unable to create a deployment for VM[User|<vmname>]" error.
I have pasted the log here:
http://tny.cz/1ee21d5e
On Wednesday, 19 March 2014 4:39 PM, Rajesh Battala <ra...@citrix.com> wrote:
Can you just capture the log from when you started the action till you see the error.
Thanks
Rajesh Battala
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: Wednesday, March 19, 2014 4:23 PM
To: Sailaja Mada; users@cloudstack.apache.org; Suresh Sadhu
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
The log file is around 10gb, I used pastebinit to upload it and it gave me "memory error".
Is there any other way to provide the log file?
Thanks,
Sugandh
On Wednesday, 19 March 2014 3:56 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
Can you please send the complete log using PasteBin.
Thanks,
Sailaja.M
From:Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:30
To: users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu; Sugandh S
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
I just noticed that my domain router is also stuck in starting state and one of the vms I created is now showing error state.
On Wednesday, 19 March 2014 3:25 PM, Sugandh S <s....@rocketmail.com> wrote:
Hi,
> I have noticed VM in starting state when Template is getting Copied from Secondary Storage to > Primary Storage .
It's been over 150 minutes and I don't think it should take this long to copy the template.
> size of the template and also value of global config parameter "wait"
size of the iso is 700.29 MB and "wait" value is default "1800".
Sugandh
On Wednesday, 19 March 2014 3:16 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
I have noticed VM in starting state when Template is getting Copied from Secondary Storage to Primary Storage .
VM gets deployed and will move to running state after this copy is completed. Can you please share the size of the template and also value of global config parameter "wait"
One reason could be Storage Server is slow and Copy operation is taking longer time. It would help not to time out if you increase the "wait" value . But you may have to wait for the copy operation to complete to get the VM into running state.
Thanks,
Sailaja.M
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:01
To: Suresh Sadhu; users@cloudstack.apache.org
Subject: Re: vm stuck in starting state, unable to delete it
Here is another part of log
e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:58,120 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:59:58,120 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744901: Timed out on null
2014-03-19 13:59:58,120 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698
744901 to Host 4 timed out after 3600
2014-03-19 13:59:58,122 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 stati stics.
2014-03-19 13:59:58,122 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host
: 4
2014-03-19 14:00:07,507 WARN [apache.cloudstack.alerts] (HA-2:null) alertType:: 13 // dataCenterId:: 0 // podId:: 0 //
clusterId:: null // message:: No usage server process running
2014-03-19 14:00:22,973 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:22,973 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661: Timed out on null
2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:58,715 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:58,715 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902: Timed out on null
2014-03-19 14:00:58,716 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698
744902 to Host 4 timed out after 3600
2014-03-19 14:00:58,716 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 stati stics.
2014-03-19 14:00:58,716 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:01:22,978 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:22,978 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662: Timed out on null
2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:35,900 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3660
2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql]
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:01:59,312 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:59,312 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904: Timed out on null
2014-03-19 14:01:59,312 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744904 to Host 4 timed out after 3600
2014-03-19 14:01:59,312 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:01:59,312 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:02:07,765 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,766 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Failed to deploy vm 4 with original planner, sending HAPlanner
2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable to transition into Starting state due to Unable to transition to a new state from Starting via StartRequested
2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Determining why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM]. Retry=4
2014-03-19 14:02:07,812 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,812 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:02:07,812 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-5) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM which was running on host name: server2(id:1), availability zone: zone1,
pod: pod1
2014-03-19 14:02:22,983 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:22,983 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663: Timed out on null
2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:59,904 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:59,904 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905: Timed out on null
2014-03-19 14:02:59,905 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698
744905 to Host 4 timed out after 3600
2014-03-19 14:02:59,905 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 14:02:59,905 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 14:03:22,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:03:22,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664: Timed out on null
2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:04:00,500 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:00,500 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906: Timed out on null
2014-03-19 14:04:00,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744906 to Host 4 timed out after 3600
2014-03-19 14:04:00,501 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:04:00,501 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:04:22,993 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:22,993 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665: Timed out on null
2014-03-19 14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:05:01,092 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:01,092 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907: Timed out on null
2014-03-19 14:05:01,092 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744907 to Host 4 timed out after 3600
2014-03-19 14:05:01,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 14:05:01,092 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:05:23,000 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:23,000 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666: Timed out on null
2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:06:01,680 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:01,680 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908: Timed out on null
2014-03-19 14:06:01,680 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744908 to Host 4 timed out after 3600
2014-03-19 14:06:01,680 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:06:01,681 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:06:23,004 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:23,004 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667: Timed out on null
2014-03-19 14:06:23,004 DEB
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org
Subject: vm stuck in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary
storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
Re: vm stuck in starting state, unable to delete it
Posted by Sugandh S <s....@rocketmail.com>.
Well, the router state seems to be "running" but I am not able to ping any of the system vms via their link local ip address or their public or private ips.
When I try to ssh into console proxy vm, I get this:
root@server2:~# ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.2.158
ssh: connect to host 169.254.2.158 port 3922: No route to host
Thanks,
Sugandh
On Wednesday, 19 March 2014 5:31 PM, Rajesh Battala <ra...@citrix.com> wrote:
From the log, issue is while applying the dhcp entry in the VR hence deployment fails.
Can you check whether the VR is up and network is in implemented state.?
Thanks
Rajesh Battala
From:Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: Wednesday, March 19, 2014 5:07 PM
To: Rajesh Battala; users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
It took a couple of reboots to get system vms and router working again but now I have got another problem, whenever I create an instance I get "Unable to create a deployment for VM[User|<vmname>]" error.
I have pasted the log here:
http://tny.cz/1ee21d5e
On Wednesday, 19 March 2014 4:39 PM, Rajesh Battala <ra...@citrix.com> wrote:
Can you just capture the log from when you started the action till you see the error.
Thanks
Rajesh Battala
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: Wednesday, March 19, 2014 4:23 PM
To: Sailaja Mada; users@cloudstack.apache.org; Suresh Sadhu
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
The log file is around 10gb, I used pastebinit to upload it and it gave me "memory error".
Is there any other way to provide the log file?
Thanks,
Sugandh
On Wednesday, 19 March 2014 3:56 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
Can you please send the complete log using PasteBin.
Thanks,
Sailaja.M
From:Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:30
To: users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu; Sugandh S
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
I just noticed that my domain router is also stuck in starting state and one of the vms I created is now showing error state.
On Wednesday, 19 March 2014 3:25 PM, Sugandh S <s....@rocketmail.com> wrote:
Hi,
> I have noticed VM in starting state when Template is getting Copied from Secondary Storage to > Primary Storage .
It's been over 150 minutes and I don't think it should take this long to copy the template.
> size of the template and also value of global config parameter "wait"
size of the iso is 700.29 MB and "wait" value is default "1800".
Sugandh
On Wednesday, 19 March 2014 3:16 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
I have noticed VM in starting state when Template is getting Copied from Secondary Storage to Primary Storage .
VM gets deployed and will move to running state after this copy is completed. Can you please share the size of the template and also value of global config parameter "wait"
One reason could be Storage Server is slow and Copy operation is taking longer time. It would help not to time out if you increase the "wait" value . But you may have to wait for the copy operation to complete to get the VM into running state.
Thanks,
Sailaja.M
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:01
To: Suresh Sadhu; users@cloudstack.apache.org
Subject: Re: vm stuck in starting state, unable to delete it
Here is another part of log
e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:58,120 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:59:58,120 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744901: Timed out on null
2014-03-19 13:59:58,120 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698
744901 to Host 4 timed out after 3600
2014-03-19 13:59:58,122 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 stati stics.
2014-03-19 13:59:58,122 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host
: 4
2014-03-19 14:00:07,507 WARN [apache.cloudstack.alerts] (HA-2:null) alertType:: 13 // dataCenterId:: 0 // podId:: 0 //
clusterId:: null // message:: No usage server process running
2014-03-19 14:00:22,973 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:22,973 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661: Timed out on null
2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:58,715 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:58,715 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902: Timed out on null
2014-03-19 14:00:58,716 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698
744902 to Host 4 timed out after 3600
2014-03-19 14:00:58,716 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 stati stics.
2014-03-19 14:00:58,716 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:01:22,978 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:22,978 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662: Timed out on null
2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:35,900 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3660
2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql]
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:01:59,312 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:59,312 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904: Timed out on null
2014-03-19 14:01:59,312 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744904 to Host 4 timed out after 3600
2014-03-19 14:01:59,312 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:01:59,312 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:02:07,765 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,766 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Failed to deploy vm 4 with original planner, sending HAPlanner
2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable to transition into Starting state due to Unable to transition to a new state from Starting via StartRequested
2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Determining why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM]. Retry=4
2014-03-19 14:02:07,812 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,812 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:02:07,812 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-5) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM which was running on host name: server2(id:1), availability zone: zone1,
pod: pod1
2014-03-19 14:02:22,983 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:22,983 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663: Timed out on null
2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:59,904 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:59,904 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905: Timed out on null
2014-03-19 14:02:59,905 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698
744905 to Host 4 timed out after 3600
2014-03-19 14:02:59,905 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 14:02:59,905 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 14:03:22,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:03:22,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664: Timed out on null
2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:04:00,500 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:00,500 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906: Timed out on null
2014-03-19 14:04:00,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744906 to Host 4 timed out after 3600
2014-03-19 14:04:00,501 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:04:00,501 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:04:22,993 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:22,993 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665: Timed out on null
2014-03-19 14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:05:01,092 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:01,092 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907: Timed out on null
2014-03-19 14:05:01,092 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744907 to Host 4 timed out after 3600
2014-03-19 14:05:01,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 14:05:01,092 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:05:23,000 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:23,000 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666: Timed out on null
2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:06:01,680 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:01,680 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908: Timed out on null
2014-03-19 14:06:01,680 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744908 to Host 4 timed out after 3600
2014-03-19 14:06:01,680 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:06:01,681 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:06:23,004 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:23,004 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667: Timed out on null
2014-03-19 14:06:23,004 DEB
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org
Subject: vm stuck in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary
storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
RE: vm stuck in starting state, unable to delete it
Posted by Rajesh Battala <ra...@citrix.com>.
>From the log, issue is while applying the dhcp entry in the VR hence deployment fails.
Can you check whether the VR is up and network is in implemented state.?
Thanks
Rajesh Battala
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: Wednesday, March 19, 2014 5:07 PM
To: Rajesh Battala; users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
It took a couple of reboots to get system vms and router working again but now I have got another problem, whenever I create an instance I get "Unable to create a deployment for VM[User|<vmname>]" error.
I have pasted the log here:
http://tny.cz/1ee21d5e
On Wednesday, 19 March 2014 4:39 PM, Rajesh Battala <ra...@citrix.com>> wrote:
Can you just capture the log from when you started the action till you see the error.
Thanks
Rajesh Battala
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com<ma...@rocketmail.com>]
Sent: Wednesday, March 19, 2014 4:23 PM
To: Sailaja Mada; users@cloudstack.apache.org<ma...@cloudstack.apache.org>; Suresh Sadhu
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
The log file is around 10gb, I used pastebinit to upload it and it gave me "memory error".
Is there any other way to provide the log file?
Thanks,
Sugandh
On Wednesday, 19 March 2014 3:56 PM, Sailaja Mada <sa...@citrix.com>> wrote:
Hi,
Can you please send the complete log using PasteBin.
Thanks,
Sailaja.M
From:Sugandh S [mailto:s.sugandh@rocketmail.com<ma...@rocketmail.com>]
Sent: 19 March 2014 15:30
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>; Sailaja Mada; Suresh Sadhu; Sugandh S
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
I just noticed that my domain router is also stuck in starting state and one of the vms I created is now showing error state.
On Wednesday, 19 March 2014 3:25 PM, Sugandh S <s....@rocketmail.com>> wrote:
Hi,
> I have noticed VM in starting state when Template is getting Copied from Secondary Storage to > Primary Storage .
It's been over 150 minutes and I don't think it should take this long to copy the template.
> size of the template and also value of global config parameter "wait"
size of the iso is 700.29 MB and "wait" value is default "1800".
Sugandh
On Wednesday, 19 March 2014 3:16 PM, Sailaja Mada <sa...@citrix.com>> wrote:
Hi,
I have noticed VM in starting state when Template is getting Copied from Secondary Storage to Primary Storage .
VM gets deployed and will move to running state after this copy is completed. Can you please share the size of the template and also value of global config parameter "wait"
One reason could be Storage Server is slow and Copy operation is taking longer time. It would help not to time out if you increase the "wait" value . But you may have to wait for the copy operation to complete to get the VM into running state.
Thanks,
Sailaja.M
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com<ma...@rocketmail.com>]
Sent: 19 March 2014 15:01
To: Suresh Sadhu; users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Subject: Re: vm stuck in starting state, unable to delete it
Here is another part of log
e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:58,120 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:59:58,120 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744901: Timed out on null
2014-03-19 13:59:58,120 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698
744901 to Host 4 timed out after 3600
2014-03-19 13:59:58,122 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 stati stics.
2014-03-19 13:59:58,122 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host
: 4
2014-03-19 14:00:07,507 WARN [apache.cloudstack.alerts] (HA-2:null) alertType:: 13 // dataCenterId:: 0 // podId:: 0 //
clusterId:: null // message:: No usage server process running
2014-03-19 14:00:22,973 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:22,973 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661: Timed out on null
2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:58,715 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:58,715 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902: Timed out on null
2014-03-19 14:00:58,716 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698
744902 to Host 4 timed out after 3600
2014-03-19 14:00:58,716 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 stati stics.
2014-03-19 14:00:58,716 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:01:22,978 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:22,978 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662: Timed out on null
2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:35,900 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3660
2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql]
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:01:59,312 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:59,312 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904: Timed out on null
2014-03-19 14:01:59,312 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744904 to Host 4 timed out after 3600
2014-03-19 14:01:59,312 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:01:59,312 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:02:07,765 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,766 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Failed to deploy vm 4 with original planner, sending HAPlanner
2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable to transition into Starting state due to Unable to transition to a new state from Starting via StartRequested
2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Determining why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM]. Retry=4
2014-03-19 14:02:07,812 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,812 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:02:07,812 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-5) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM which was running on host name: server2(id:1), availability zone: zone1,
pod: pod1
2014-03-19 14:02:22,983 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:22,983 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663: Timed out on null
2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:59,904 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:59,904 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905: Timed out on null
2014-03-19 14:02:59,905 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698
744905 to Host 4 timed out after 3600
2014-03-19 14:02:59,905 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 14:02:59,905 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 14:03:22,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:03:22,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664: Timed out on null
2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:04:00,500 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:00,500 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906: Timed out on null
2014-03-19 14:04:00,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744906 to Host 4 timed out after 3600
2014-03-19 14:04:00,501 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:04:00,501 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:04:22,993 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:22,993 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665: Timed out on null
2014-03-19 14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:05:01,092 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:01,092 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907: Timed out on null
2014-03-19 14:05:01,092 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744907 to Host 4 timed out after 3600
2014-03-19 14:05:01,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 14:05:01,092 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:05:23,000 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:23,000 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666: Timed out on null
2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:06:01,680 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:01,680 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908: Timed out on null
2014-03-19 14:06:01,680 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744908 to Host 4 timed out after 3600
2014-03-19 14:06:01,680 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:06:01,681 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:06:23,004 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:23,004 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667: Timed out on null
2014-03-19 14:06:23,004 DEB
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com>> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com<ma...@rocketmail.com>]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Subject: vm stuck in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
Re: vm stuck in starting state, unable to delete it
Posted by Sugandh S <s....@rocketmail.com>.
Hi,
It took a couple of reboots to get system vms and router working again but now I have got another problem, whenever I create an instance I get "Unable to create a deployment for VM[User|<vmname>]" error.
I have pasted the log here:
http://tny.cz/1ee21d5e
On Wednesday, 19 March 2014 4:39 PM, Rajesh Battala <ra...@citrix.com> wrote:
Can you just capture the log from when you started the action till you see the error.
Thanks
Rajesh Battala
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: Wednesday, March 19, 2014 4:23 PM
To: Sailaja Mada; users@cloudstack.apache.org; Suresh Sadhu
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
The log file is around 10gb, I used pastebinit to upload it and it gave me "memory error".
Is there any other way to provide the log file?
Thanks,
Sugandh
On Wednesday, 19 March 2014 3:56 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
Can you please send the complete log using PasteBin.
Thanks,
Sailaja.M
From:Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:30
To: users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu; Sugandh S
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
I just noticed that my domain router is also stuck in starting state and one of the vms I created is now showing error state.
On Wednesday, 19 March 2014 3:25 PM, Sugandh S <s....@rocketmail.com> wrote:
Hi,
> I have noticed VM in starting state when Template is getting Copied from Secondary Storage to > Primary Storage .
It's been over 150 minutes and I don't think it should take this long to copy the template.
> size of the template and also value of global config parameter "wait"
size of the iso is 700.29 MB and "wait" value is default "1800".
Sugandh
On Wednesday, 19 March 2014 3:16 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
I have noticed VM in starting state when Template is getting Copied from Secondary Storage to Primary Storage .
VM gets deployed and will move to running state after this copy is completed. Can you please share the size of the template and also value of global config parameter "wait"
One reason could be Storage Server is slow and Copy operation is taking longer time. It would help not to time out if you increase the "wait" value . But you may have to wait for the copy operation to complete to get the VM into running state.
Thanks,
Sailaja.M
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:01
To: Suresh Sadhu; users@cloudstack.apache.org
Subject: Re: vm stuck in starting state, unable to delete it
Here is another part of log
e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:58,120 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:59:58,120 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744901: Timed out on null
2014-03-19 13:59:58,120 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698
744901 to Host 4 timed out after 3600
2014-03-19 13:59:58,122 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 stati stics.
2014-03-19 13:59:58,122 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host
: 4
2014-03-19 14:00:07,507 WARN [apache.cloudstack.alerts] (HA-2:null) alertType:: 13 // dataCenterId:: 0 // podId:: 0 //
clusterId:: null // message:: No usage server process running
2014-03-19 14:00:22,973 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:22,973 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661: Timed out on null
2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:58,715 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:58,715 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902: Timed out on null
2014-03-19 14:00:58,716 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698
744902 to Host 4 timed out after 3600
2014-03-19 14:00:58,716 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 stati stics.
2014-03-19 14:00:58,716 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:01:22,978 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:22,978 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662: Timed out on null
2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:35,900 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3660
2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql]
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:01:59,312 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:59,312 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904: Timed out on null
2014-03-19 14:01:59,312 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744904 to Host 4 timed out after 3600
2014-03-19 14:01:59,312 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:01:59,312 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:02:07,765 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,766 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Failed to deploy vm 4 with original planner, sending HAPlanner
2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable to transition into Starting state due to Unable to transition to a new state from Starting via StartRequested
2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Determining why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM]. Retry=4
2014-03-19 14:02:07,812 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,812 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:02:07,812 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-5) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM which was running on host name: server2(id:1), availability zone: zone1,
pod: pod1
2014-03-19 14:02:22,983 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:22,983 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663: Timed out on null
2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:59,904 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:59,904 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905: Timed out on null
2014-03-19 14:02:59,905 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698
744905 to Host 4 timed out after 3600
2014-03-19 14:02:59,905 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 14:02:59,905 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 14:03:22,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:03:22,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664: Timed out on null
2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:04:00,500 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:00,500 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906: Timed out on null
2014-03-19 14:04:00,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744906 to Host 4 timed out after 3600
2014-03-19 14:04:00,501 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:04:00,501 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:04:22,993 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:22,993 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665: Timed out on null
2014-03-19 14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:05:01,092 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:01,092 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907: Timed out on null
2014-03-19 14:05:01,092 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744907 to Host 4 timed out after 3600
2014-03-19 14:05:01,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 14:05:01,092 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:05:23,000 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:23,000 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666: Timed out on null
2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:06:01,680 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:01,680 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908: Timed out on null
2014-03-19 14:06:01,680 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744908 to Host 4 timed out after 3600
2014-03-19 14:06:01,680 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:06:01,681 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:06:23,004 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:23,004 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667: Timed out on null
2014-03-19 14:06:23,004 DEB
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org
Subject: vm stuck in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
RE: vm stuck in starting state, unable to delete it
Posted by Rajesh Battala <ra...@citrix.com>.
Can you just capture the log from when you started the action till you see the error.
Thanks
Rajesh Battala
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: Wednesday, March 19, 2014 4:23 PM
To: Sailaja Mada; users@cloudstack.apache.org; Suresh Sadhu
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
The log file is around 10gb, I used pastebinit to upload it and it gave me "memory error".
Is there any other way to provide the log file?
Thanks,
Sugandh
On Wednesday, 19 March 2014 3:56 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
Can you please send the complete log using PasteBin.
Thanks,
Sailaja.M
From:Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:30
To: users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu; Sugandh S
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
I just noticed that my domain router is also stuck in starting state and one of the vms I created is now showing error state.
On Wednesday, 19 March 2014 3:25 PM, Sugandh S <s....@rocketmail.com> wrote:
Hi,
> I have noticed VM in starting state when Template is getting Copied from Secondary Storage to > Primary Storage .
It's been over 150 minutes and I don't think it should take this long to copy the template.
> size of the template and also value of global config parameter "wait"
size of the iso is 700.29 MB and "wait" value is default "1800".
Sugandh
On Wednesday, 19 March 2014 3:16 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
I have noticed VM in starting state when Template is getting Copied from Secondary Storage to Primary Storage .
VM gets deployed and will move to running state after this copy is completed. Can you please share the size of the template and also value of global config parameter "wait"
One reason could be Storage Server is slow and Copy operation is taking longer time. It would help not to time out if you increase the "wait" value . But you may have to wait for the copy operation to complete to get the VM into running state.
Thanks,
Sailaja.M
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:01
To: Suresh Sadhu; users@cloudstack.apache.org
Subject: Re: vm stuck in starting state, unable to delete it
Here is another part of log
e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:58,120 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:59:58,120 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744901: Timed out on null
2014-03-19 13:59:58,120 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698
744901 to Host 4 timed out after 3600
2014-03-19 13:59:58,122 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 stati stics.
2014-03-19 13:59:58,122 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host
: 4
2014-03-19 14:00:07,507 WARN [apache.cloudstack.alerts] (HA-2:null) alertType:: 13 // dataCenterId:: 0 // podId:: 0 //
clusterId:: null // message:: No usage server process running
2014-03-19 14:00:22,973 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:22,973 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661: Timed out on null
2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:58,715 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:58,715 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902: Timed out on null
2014-03-19 14:00:58,716 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698
744902 to Host 4 timed out after 3600
2014-03-19 14:00:58,716 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 stati stics.
2014-03-19 14:00:58,716 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:01:22,978 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:22,978 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662: Timed out on null
2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:35,900 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3660
2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql]
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:01:59,312 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:59,312 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904: Timed out on null
2014-03-19 14:01:59,312 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744904 to Host 4 timed out after 3600
2014-03-19 14:01:59,312 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:01:59,312 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:02:07,765 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,766 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Failed to deploy vm 4 with original planner, sending HAPlanner
2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable to transition into Starting state due to Unable to transition to a new state from Starting via StartRequested
2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Determining why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM]. Retry=4
2014-03-19 14:02:07,812 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,812 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:02:07,812 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-5) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM which was running on host name: server2(id:1), availability zone: zone1,
pod: pod1
2014-03-19 14:02:22,983 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:22,983 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663: Timed out on null
2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:59,904 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:59,904 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905: Timed out on null
2014-03-19 14:02:59,905 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698
744905 to Host 4 timed out after 3600
2014-03-19 14:02:59,905 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 14:02:59,905 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 14:03:22,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:03:22,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664: Timed out on null
2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:04:00,500 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:00,500 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906: Timed out on null
2014-03-19 14:04:00,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744906 to Host 4 timed out after 3600
2014-03-19 14:04:00,501 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:04:00,501 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:04:22,993 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:22,993 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665: Timed out on null
2014-03-19 14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:05:01,092 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:01,092 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907: Timed out on null
2014-03-19 14:05:01,092 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744907 to Host 4 timed out after 3600
2014-03-19 14:05:01,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 14:05:01,092 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:05:23,000 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:23,000 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666: Timed out on null
2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:06:01,680 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:01,680 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908: Timed out on null
2014-03-19 14:06:01,680 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744908 to Host 4 timed out after 3600
2014-03-19 14:06:01,680 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:06:01,681 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:06:23,004 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:23,004 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667: Timed out on null
2014-03-19 14:06:23,004 DEB
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org
Subject: vm stuck in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
Re: vm stuck in starting state, unable to delete it
Posted by Sugandh S <s....@rocketmail.com>.
Hi,
The log file is around 10gb, I used pastebinit to upload it and it gave me "memory error".
Is there any other way to provide the log file?
Thanks,
Sugandh
On Wednesday, 19 March 2014 3:56 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
Can you please send the complete log using PasteBin.
Thanks,
Sailaja.M
From:Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:30
To: users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu; Sugandh S
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
I just noticed that my domain router is also stuck in starting state and one of the vms I created is now showing error state.
On Wednesday, 19 March 2014 3:25 PM, Sugandh S <s....@rocketmail.com> wrote:
Hi,
> I have noticed VM in starting state when Template is getting Copied from Secondary Storage to > Primary Storage .
It's been over 150 minutes and I don't think it should take this long to copy the template.
> size of the template and also value of global config parameter "wait"
size of the iso is 700.29 MB and "wait" value is default "1800".
Sugandh
On Wednesday, 19 March 2014 3:16 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
I have noticed VM in starting state when Template is getting Copied from Secondary Storage to Primary Storage .
VM gets deployed and will move to running state after this copy is completed. Can you please share the size of the template and also value of global config parameter "wait"
One reason could be Storage Server is slow and Copy operation is taking longer time. It would help not to time out if you increase the "wait" value . But you may have to wait for the copy operation to complete to get the VM into running state.
Thanks,
Sailaja.M
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:01
To: Suresh Sadhu; users@cloudstack.apache.org
Subject: Re: vm stuck in starting state, unable to delete it
Here is another part of log
e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:58,120 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:59:58,120 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744901: Timed out on null
2014-03-19 13:59:58,120 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698
744901 to Host 4 timed out after 3600
2014-03-19 13:59:58,122 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 stati stics.
2014-03-19 13:59:58,122 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host
: 4
2014-03-19 14:00:07,507 WARN [apache.cloudstack.alerts] (HA-2:null) alertType:: 13 // dataCenterId:: 0 // podId:: 0 //
clusterId:: null // message:: No usage server process running
2014-03-19 14:00:22,973 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:22,973 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661: Timed out on null
2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:58,715 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:58,715 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902: Timed out on null
2014-03-19 14:00:58,716 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698
744902 to Host 4 timed out after 3600
2014-03-19 14:00:58,716 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 stati stics.
2014-03-19 14:00:58,716 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:01:22,978 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:22,978 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662: Timed out on null
2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:35,900 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3660
2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql]
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:01:59,312 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:59,312 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904: Timed out on null
2014-03-19 14:01:59,312 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744904 to Host 4 timed out after 3600
2014-03-19 14:01:59,312 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:01:59,312 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:02:07,765 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,766 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Failed to deploy vm 4 with original planner, sending HAPlanner
2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable to transition into Starting state due to Unable to transition to a new state from Starting via StartRequested
2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Determining why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM]. Retry=4
2014-03-19 14:02:07,812 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,812 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:02:07,812 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-5) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM which was running on host name: server2(id:1), availability zone: zone1,
pod: pod1
2014-03-19 14:02:22,983 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:22,983 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663: Timed out on null
2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:59,904 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:59,904 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905: Timed out on null
2014-03-19 14:02:59,905 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698
744905 to Host 4 timed out after 3600
2014-03-19 14:02:59,905 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 14:02:59,905 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 14:03:22,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:03:22,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664: Timed out on null
2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:04:00,500 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:00,500 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906: Timed out on null
2014-03-19 14:04:00,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744906 to Host 4 timed out after 3600
2014-03-19 14:04:00,501 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:04:00,501 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:04:22,993 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:22,993 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665: Timed out on null
2014-03-19 14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:05:01,092 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:01,092 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907: Timed out on null
2014-03-19 14:05:01,092 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744907 to Host 4 timed out after 3600
2014-03-19 14:05:01,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 14:05:01,092 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:05:23,000 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:23,000 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666: Timed out on null
2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:06:01,680 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:01,680 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908: Timed out on null
2014-03-19 14:06:01,680 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744908 to Host 4 timed out after 3600
2014-03-19 14:06:01,680 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:06:01,681 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:06:23,004 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:23,004 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667: Timed out on null
2014-03-19 14:06:23,004 DEB
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org
Subject: vm stuck in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary
storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
RE: vm stuck in starting state, unable to delete it
Posted by Sailaja Mada <sa...@citrix.com>.
Hi,
Can you please send the complete log using PasteBin.
Thanks,
Sailaja.M
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:30
To: users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu; Sugandh S
Subject: Re: vm stuck in starting state, unable to delete it
Hi,
I just noticed that my domain router is also stuck in starting state and one of the vms I created is now showing error state.
On Wednesday, 19 March 2014 3:25 PM, Sugandh S <s....@rocketmail.com>> wrote:
Hi,
> I have noticed VM in starting state when Template is getting Copied from Secondary Storage to > Primary Storage .
It's been over 150 minutes and I don't think it should take this long to copy the template.
> size of the template and also value of global config parameter "wait"
size of the iso is 700.29 MB and "wait" value is default "1800".
Sugandh
On Wednesday, 19 March 2014 3:16 PM, Sailaja Mada <sa...@citrix.com>> wrote:
Hi,
I have noticed VM in starting state when Template is getting Copied from Secondary Storage to Primary Storage .
VM gets deployed and will move to running state after this copy is completed. Can you please share the size of the template and also value of global config parameter "wait"
One reason could be Storage Server is slow and Copy operation is taking longer time. It would help not to time out if you increase the "wait" value . But you may have to wait for the copy operation to complete to get the VM into running state.
Thanks,
Sailaja.M
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com<ma...@rocketmail.com>]
Sent: 19 March 2014 15:01
To: Suresh Sadhu; users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Subject: Re: vm stuck in starting state, unable to delete it
Here is another part of log
e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:58,120 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:59:58,120 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744901: Timed out on null
2014-03-19 13:59:58,120 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698
744901 to Host 4 timed out after 3600
2014-03-19 13:59:58,122 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 stati stics.
2014-03-19 13:59:58,122 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host
: 4
2014-03-19 14:00:07,507 WARN [apache.cloudstack.alerts] (HA-2:null) alertType:: 13 // dataCenterId:: 0 // podId:: 0 //
clusterId:: null // message:: No usage server process running
2014-03-19 14:00:22,973 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:22,973 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661: Timed out on null
2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:58,715 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:58,715 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902: Timed out on null
2014-03-19 14:00:58,716 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698
744902 to Host 4 timed out after 3600
2014-03-19 14:00:58,716 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 stati stics.
2014-03-19 14:00:58,716 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:01:22,978 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:22,978 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662: Timed out on null
2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:35,900 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3660
2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql]
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:01:59,312 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:59,312 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904: Timed out on null
2014-03-19 14:01:59,312 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744904 to Host 4 timed out after 3600
2014-03-19 14:01:59,312 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:01:59,312 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:02:07,765 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,766 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Failed to deploy vm 4 with original planner, sending HAPlanner
2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable to transition into Starting state due to Unable to transition to a new state from Starting via StartRequested
2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Determining why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM]. Retry=4
2014-03-19 14:02:07,812 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,812 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:02:07,812 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-5) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM which was running on host name: server2(id:1), availability zone: zone1, pod: pod1
2014-03-19 14:02:22,983 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:22,983 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663: Timed out on null
2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:59,904 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:59,904 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905: Timed out on null
2014-03-19 14:02:59,905 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698
744905 to Host 4 timed out after 3600
2014-03-19 14:02:59,905 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 14:02:59,905 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 14:03:22,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:03:22,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664: Timed out on null
2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:04:00,500 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:00,500 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906: Timed out on null
2014-03-19 14:04:00,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744906 to Host 4 timed out after 3600
2014-03-19 14:04:00,501 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:04:00,501 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:04:22,993 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:22,993 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665: Timed out on null
2014-03-19 14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:05:01,092 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:01,092 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907: Timed out on null
2014-03-19 14:05:01,092 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744907 to Host 4 timed out after 3600
2014-03-19 14:05:01,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 14:05:01,092 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:05:23,000 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:23,000 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666: Timed out on null
2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:06:01,680 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:01,680 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908: Timed out on null
2014-03-19 14:06:01,680 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744908 to Host 4 timed out after 3600
2014-03-19 14:06:01,680 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:06:01,681 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:06:23,004 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:23,004 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667: Timed out on null
2014-03-19 14:06:23,004 DEB
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com>> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com<ma...@rocketmail.com>]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Subject: vm stuck in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
Re: vm stuck in starting state, unable to delete it
Posted by Sugandh S <s....@rocketmail.com>.
Hi,
I just noticed that my domain router is also stuck in starting state and one of the vms I created is now showing error state.
On Wednesday, 19 March 2014 3:25 PM, Sugandh S <s....@rocketmail.com> wrote:
Hi,
> I have noticed VM in starting state when Template is getting Copied from Secondary Storage to > Primary Storage .
It's been over 150 minutes and I don't think it should take this long to copy the template.
> size of the template and also value of global config parameter "wait"
size of the iso is 700.29 MB and "wait" value is default "1800".
Sugandh
On Wednesday, 19 March 2014 3:16 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
I have noticed VM in starting state when Template is getting Copied from Secondary Storage to Primary Storage .
VM gets deployed and will move to running state after this copy is completed. Can you please share the size of the template and also value of global config parameter "wait"
One reason could be Storage Server is slow and Copy operation is taking longer time. It would help not to time out if you increase the "wait" value . But you may have to wait for the copy operation to complete to get the VM into running state.
Thanks,
Sailaja.M
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:01
To: Suresh Sadhu; users@cloudstack.apache.org
Subject: Re: vm stuck in starting state, unable to delete it
Here is another part of log
e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:58,120 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:59:58,120 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744901: Timed out on null
2014-03-19 13:59:58,120 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698
744901 to Host 4 timed out after 3600
2014-03-19 13:59:58,122 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 stati stics.
2014-03-19 13:59:58,122 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host
: 4
2014-03-19 14:00:07,507 WARN [apache.cloudstack.alerts] (HA-2:null) alertType:: 13 // dataCenterId:: 0 // podId:: 0 //
clusterId:: null // message:: No usage server process running
2014-03-19 14:00:22,973 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:22,973 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661: Timed out on null
2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:58,715 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:58,715 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902: Timed out on null
2014-03-19 14:00:58,716 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698
744902 to Host 4 timed out after 3600
2014-03-19 14:00:58,716 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 stati stics.
2014-03-19 14:00:58,716 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:01:22,978 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:22,978 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662: Timed out on null
2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:35,900 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3660
2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql]
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:01:59,312 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:59,312 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904: Timed out on null
2014-03-19 14:01:59,312 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744904 to Host 4 timed out after 3600
2014-03-19 14:01:59,312 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:01:59,312 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:02:07,765 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,766 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Failed to deploy vm 4 with original planner, sending HAPlanner
2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable to transition into Starting state due to Unable to transition to a new state from Starting via StartRequested
2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Determining why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM]. Retry=4
2014-03-19 14:02:07,812 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,812 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:02:07,812 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-5) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM which was running on host name: server2(id:1), availability zone: zone1, pod: pod1
2014-03-19 14:02:22,983 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:22,983 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663: Timed out on null
2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:59,904 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:59,904 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905: Timed out on null
2014-03-19 14:02:59,905 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698
744905 to Host 4 timed out after 3600
2014-03-19 14:02:59,905 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 14:02:59,905 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 14:03:22,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:03:22,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664: Timed out on null
2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:04:00,500 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:00,500 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906: Timed out on null
2014-03-19 14:04:00,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744906 to Host 4 timed out after 3600
2014-03-19 14:04:00,501 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:04:00,501 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:04:22,993 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:22,993 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665: Timed out on null
2014-03-19 14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:05:01,092 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:01,092 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907: Timed out on null
2014-03-19 14:05:01,092 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744907 to Host 4 timed out after 3600
2014-03-19 14:05:01,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 14:05:01,092 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:05:23,000 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:23,000 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666: Timed out on null
2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:06:01,680 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:01,680 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908: Timed out on null
2014-03-19 14:06:01,680 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744908 to Host 4 timed out after 3600
2014-03-19 14:06:01,680 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:06:01,681 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:06:23,004 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:23,004 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667: Timed out on null
2014-03-19 14:06:23,004 DEB
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org
Subject: vm stuck in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
Re: vm stuck in starting state, unable to delete it
Posted by Sugandh S <s....@rocketmail.com>.
Hi,
> I have noticed VM in starting state when Template is getting Copied from Secondary Storage to > Primary Storage .
It's been over 150 minutes and I don't think it should take this long to copy the template.
> size of the template and also value of global config parameter "wait"
size of the iso is 700.29 MB and "wait" value is default "1800".
Sugandh
On Wednesday, 19 March 2014 3:16 PM, Sailaja Mada <sa...@citrix.com> wrote:
Hi,
I have noticed VM in starting state when Template is getting Copied from Secondary Storage to Primary Storage .
VM gets deployed and will move to running state after this copy is completed. Can you please share the size of the template and also value of global config parameter "wait"
One reason could be Storage Server is slow and Copy operation is taking longer time. It would help not to time out if you increase the "wait" value . But you may have to wait for the copy operation to complete to get the VM into running state.
Thanks,
Sailaja.M
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:01
To: Suresh Sadhu; users@cloudstack.apache.org
Subject: Re: vm stuck in starting state, unable to delete it
Here is another part of log
e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:58,120 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:59:58,120 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744901: Timed out on null
2014-03-19 13:59:58,120 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698
744901 to Host 4 timed out after 3600
2014-03-19 13:59:58,122 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 stati stics.
2014-03-19 13:59:58,122 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host
: 4
2014-03-19 14:00:07,507 WARN [apache.cloudstack.alerts] (HA-2:null) alertType:: 13 // dataCenterId:: 0 // podId:: 0 //
clusterId:: null // message:: No usage server process running
2014-03-19 14:00:22,973 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:22,973 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661: Timed out on null
2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:58,715 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:58,715 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902: Timed out on null
2014-03-19 14:00:58,716 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698
744902 to Host 4 timed out after 3600
2014-03-19 14:00:58,716 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 stati stics.
2014-03-19 14:00:58,716 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:01:22,978 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:22,978 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662: Timed out on null
2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:35,900 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3660
2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql]
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:01:59,312 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:59,312 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904: Timed out on null
2014-03-19 14:01:59,312 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744904 to Host 4 timed out after 3600
2014-03-19 14:01:59,312 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:01:59,312 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:02:07,765 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,766 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Failed to deploy vm 4 with original planner, sending HAPlanner
2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable to transition into Starting state due to Unable to transition to a new state from Starting via StartRequested
2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Determining why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM]. Retry=4
2014-03-19 14:02:07,812 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,812 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:02:07,812 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-5) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM which was running on host name: server2(id:1), availability zone: zone1, pod: pod1
2014-03-19 14:02:22,983 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:22,983 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663: Timed out on null
2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:59,904 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:59,904 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905: Timed out on null
2014-03-19 14:02:59,905 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698
744905 to Host 4 timed out after 3600
2014-03-19 14:02:59,905 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 14:02:59,905 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 14:03:22,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:03:22,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664: Timed out on null
2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:04:00,500 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:00,500 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906: Timed out on null
2014-03-19 14:04:00,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744906 to Host 4 timed out after 3600
2014-03-19 14:04:00,501 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:04:00,501 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:04:22,993 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:22,993 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665: Timed out on null
2014-03-19 14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:05:01,092 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:01,092 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907: Timed out on null
2014-03-19 14:05:01,092 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744907 to Host 4 timed out after 3600
2014-03-19 14:05:01,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 14:05:01,092 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:05:23,000 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:23,000 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666: Timed out on null
2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:06:01,680 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:01,680 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908: Timed out on null
2014-03-19 14:06:01,680 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744908 to Host 4 timed out after 3600
2014-03-19 14:06:01,680 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:06:01,681 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:06:23,004 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:23,004 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667: Timed out on null
2014-03-19 14:06:23,004 DEB
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org
Subject: vm stuck in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
RE: vm stuck in starting state, unable to delete it
Posted by Sailaja Mada <sa...@citrix.com>.
Hi,
I have noticed VM in starting state when Template is getting Copied from Secondary Storage to Primary Storage .
VM gets deployed and will move to running state after this copy is completed. Can you please share the size of the template and also value of global config parameter "wait"
One reason could be Storage Server is slow and Copy operation is taking longer time. It would help not to time out if you increase the "wait" value . But you may have to wait for the copy operation to complete to get the VM into running state.
Thanks,
Sailaja.M
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 15:01
To: Suresh Sadhu; users@cloudstack.apache.org
Subject: Re: vm stuck in starting state, unable to delete it
Here is another part of log
e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:58,120 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:59:58,120 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744901: Timed out on null
2014-03-19 13:59:58,120 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698
744901 to Host 4 timed out after 3600
2014-03-19 13:59:58,122 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 stati stics.
2014-03-19 13:59:58,122 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host
: 4
2014-03-19 14:00:07,507 WARN [apache.cloudstack.alerts] (HA-2:null) alertType:: 13 // dataCenterId:: 0 // podId:: 0 //
clusterId:: null // message:: No usage server process running
2014-03-19 14:00:22,973 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:22,973 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661: Timed out on null
2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:58,715 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:58,715 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902: Timed out on null
2014-03-19 14:00:58,716 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698
744902 to Host 4 timed out after 3600
2014-03-19 14:00:58,716 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 stati stics.
2014-03-19 14:00:58,716 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:01:22,978 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:22,978 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662: Timed out on null
2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:35,900 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3660
2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql]
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:01:59,312 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:59,312 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904: Timed out on null
2014-03-19 14:01:59,312 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744904 to Host 4 timed out after 3600
2014-03-19 14:01:59,312 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:01:59,312 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:02:07,765 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,766 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Failed to deploy vm 4 with original planner, sending HAPlanner
2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable to transition into Starting state due to Unable to transition to a new state from Starting via StartRequested
2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Determining why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM]. Retry=4
2014-03-19 14:02:07,812 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,812 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:02:07,812 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-5) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM which was running on host name: server2(id:1), availability zone: zone1, pod: pod1
2014-03-19 14:02:22,983 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:22,983 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663: Timed out on null
2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:59,904 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:59,904 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905: Timed out on null
2014-03-19 14:02:59,905 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698
744905 to Host 4 timed out after 3600
2014-03-19 14:02:59,905 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 14:02:59,905 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 14:03:22,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:03:22,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664: Timed out on null
2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:04:00,500 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:00,500 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906: Timed out on null
2014-03-19 14:04:00,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744906 to Host 4 timed out after 3600
2014-03-19 14:04:00,501 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:04:00,501 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:04:22,993 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:22,993 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665: Timed out on null
2014-03-19 14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:05:01,092 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:01,092 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907: Timed out on null
2014-03-19 14:05:01,092 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744907 to Host 4 timed out after 3600
2014-03-19 14:05:01,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 14:05:01,092 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:05:23,000 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:23,000 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666: Timed out on null
2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:06:01,680 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:01,680 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908: Timed out on null
2014-03-19 14:06:01,680 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744908 to Host 4 timed out after 3600
2014-03-19 14:06:01,680 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:06:01,681 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:06:23,004 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:23,004 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667: Timed out on null
2014-03-19 14:06:23,004 DEB
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org
Subject: vm stuck in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
Re: vm stuck in starting state, unable to delete it
Posted by Sugandh S <s....@rocketmail.com>.
Here is another part of log
e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta
ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:58,120 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c
om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 13:59:58,120 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744901: Timed out on null
2014-03-19 13:59:58,120 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698
744901 to Host 4 timed out after 3600
2014-03-19 13:59:58,122 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 stati
stics.
2014-03-19 13:59:58,122 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host
: 4
2014-03-19 14:00:07,507 WARN [apache.cloudstack.alerts] (HA-2:null) alertType:: 13 // dataCenterId:: 0 // podId:: 0 //
clusterId:: null // message:: No usage server process running
2014-03-19 14:00:22,973 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c
om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:22,973 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661: Timed out on null
2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, du
e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta
ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:58,715 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c
om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:00:58,715 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902: Timed out on null
2014-03-19 14:00:58,716 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698
744902 to Host 4 timed out after 3600
2014-03-19 14:00:58,716 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 stati
stics.
2014-03-19 14:00:58,716 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:01:22,978 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c
om.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:22,978 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662: Timed out on null
2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, du
e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage sta
ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:35,900 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3660
2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql]
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation
Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:01:59,312 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:01:59,312 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904: Timed out on null
2014-03-19 14:01:59,312 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744904 to Host 4 timed out after 3600
2014-03-19 14:01:59,312 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:01:59,312 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:02:07,765 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,766 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Failed to deploy vm 4 with original planner, sending HAPlanner
2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable to transition into Starting state due to Unable to transition to a new state from Starting via StartRequested
2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Determining why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM]. Retry=4
2014-03-19 14:02:07,812 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,812 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:02:07,812 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-5) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM which was running on host name: server2(id:1), availability zone: zone1, pod: pod1
2014-03-19 14:02:22,983 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:22,983 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663: Timed out on null
2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:59,904 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:02:59,904 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905: Timed out on null
2014-03-19 14:02:59,905 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698
744905 to Host 4 timed out after 3600
2014-03-19 14:02:59,905 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics.
2014-03-19 14:02:59,905 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4
2014-03-19 14:03:22,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:03:22,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664: Timed out on null
2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:04:00,500 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:00,500 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906: Timed out on null
2014-03-19 14:04:00,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744906 to Host 4 timed out after 3600
2014-03-19 14:04:00,501 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:04:00,501 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:04:22,993 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:04:22,993 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665: Timed out on null
2014-03-19 14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:05:01,092 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:01,092 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907: Timed out on null
2014-03-19 14:05:01,092 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744907 to Host 4 timed out after 3600
2014-03-19 14:05:01,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics.
2014-03-19 14:05:01,092 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host
: 4
2014-03-19 14:05:23,000 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:05:23,000 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666: Timed out on null
2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:06:01,680 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:01,680 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908: Timed out on null
2014-03-19 14:06:01,680 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744908 to Host 4 timed out after 3600
2014-03-19 14:06:01,680 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics.
2014-03-19 14:06:01,681 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4
2014-03-19 14:06:23,004 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2014-03-19 14:06:23,004 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667: Timed out on null
2014-03-19 14:06:23,004 DEB
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Su...@citrix.com> wrote:
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org
Subject: vm stuck in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh
RE: vm stuck in starting state, unable to delete it
Posted by Suresh Sadhu <Su...@citrix.com>.
Can you please provide the logs and also did you notice any exception in the management log.
For deleting vm :
You can update the vm state in db as Stopped and try to delete them from CS.
Regards
Sadhu
-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org
Subject: vm stuck in starting state, unable to delete it
Hello all,
I am using CS 4.2 and my setup is as follows:
One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary storage, it is /export/secondary.
Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent.
Now, when I create vms they are stuck in starting state and I am unable to delete them.
Any and all help would be greatly appreciated.
Thanks ahead,
Sugandh