You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@cloudstack.apache.org by Giri Prasad <g_...@yahoo.com.INVALID> on 2014/07/28 07:23:12 UTC

Unable to stop or delete vm's

Hello All,

 I installed CS 4.1 on Ubuntu 12.04 LTS. Installed two vm's, one each with centos 6.5 and ubuntu 12.04. Single server, local hdd storage.

 After a few days, I powerdown the entire linux box, and after booting up and starting the CS ms console, I started the vm's (26-Jul-2014). The vm's are telling me a status of 'Starting' for a long time.  Powered down the entire linux box, after a few hours.


Today once again booted the linux box, started CS ms console, the CS gui says, the vm's state as 'Starting'.

 I tried to stop or delete the vm thru cloudmonkey, but it is giving some errors. Any insights?

Thanks & Regards,
Giri

mycloudmonkeytpj>  stopVirtualMachine id="7757211b-d8a6-47aa-ba9f-24967499ba97"
.Async job 08cf9cb9-d2f7-46b7-af56-12d16a9c1290 failed
Error 530, We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting
{
  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "cmd": "org.apache.cloudstack.api.command.user.vm.StopVMCmd",
  "created": "2014-07-28T10:38:28+0530",
  "jobid": "08cf9cb9-d2f7-46b7-af56-12d16a9c1290",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting"
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}

mycloudmonkeytpj> stopVirtualMachine id="b74e6737-ec83-4483-b1a2-ab60045d941b"
.Async job d52f900d-dfa4-4b70-9d6a-8d4ba983e1aa failed
Error 530, We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting
{
  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "cmd": "org.apache.cloudstack.api.command.user.vm.StopVMCmd",
  "created": "2014-07-28T10:39:11+0530",
  "jobid": "d52f900d-dfa4-4b70-9d6a-8d4ba983e1aa",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting"
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}

mycloudmonkeytpj> destroyVirtualMachine id="7757211b-d8a6-47aa-ba9f-24967499ba97"
.Async job b3dc2bab-1b03-45d1-ba02-0c861f5043df failed
Error 530, We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting
{
  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "cmd": "org.apache.cloudstack.api.command.user.vm.DestroyVMCmd",
  "created": "2014-07-28T10:39:21+0530",
  "jobid": "b3dc2bab-1b03-45d1-ba02-0c861f5043df",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting"
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}

mycloudmonkeytpj> destroyVirtualMachine id="b74e6737-ec83-4483-b1a2-ab60045d941b"
.Async job 81e7e2c4-efc9-4673-90f7-8a69287cfa80 failed
Error 530, We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting
{
  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "cmd": "org.apache.cloudstack.api.command.user.vm.DestroyVMCmd",
  "created": "2014-07-28T10:39:31+0530",
  "jobid": "81e7e2c4-efc9-4673-90f7-8a69287cfa80",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting"
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}

Re: Unable to stop or delete vm's

Posted by Giri Prasad <g_...@yahoo.com.INVALID>.

And some more logs:
tail -20 /var/log/cloudstack/management/localhost.2014-07-28.log
28 Jul, 2014 10:31:26 AM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/usr/share/cloudstack-management/webapps/client/]
28 Jul, 2014 10:31:26 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing log4j from [classpath:log4j-cloud.xml]
28 Jul, 2014 10:31:27 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
28 Jul, 2014 10:32:47 AM org.apache.catalina.core.ApplicationContext log
INFO: Closing Spring root WebApplicationContext
28 Jul, 2014 10:32:47 AM org.apache.catalina.core.ApplicationContext log
INFO: Shutting down log4j
28 Jul, 2014 10:33:05 AM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/usr/share/cloudstack-management/webapps/client/]
28 Jul, 2014 10:33:05 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing log4j from [classpath:log4j-cloud.xml]
28 Jul, 2014 10:33:05 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext

root@ubuntu:/home/giri#
root@ubuntu:/home/giri#
root@ubuntu:/home/giri# tail -20 /var/log/cloudstack/management/access_log.2014-07-28.txtXXX.XXX.X.5 - - [28/Jul/2014:12:10:17 +0530] "GET /client/api?command=listCapacity&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&fetchLatest=false&_=1406529617462 HTTP/1.1" 200 1112
XXX.XXX.X.5 - - [28/Jul/2014:12:10:17 +0530] "GET /client/api?command=listAlerts&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&page=1&pageSize=4&_=1406529617478 HTTP/1.1" 200 661
XXX.XXX.X.5 - - [28/Jul/2014:12:10:17 +0530] "GET /client/api?command=listHosts&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&state=Alert&page=1&pageSize=4&_=1406529617489 HTTP/1.1" 200 550
XXX.XXX.X.5 - - [28/Jul/2014:12:10:17 +0530] "GET /client/api?command=listCapacity&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&fetchLatest=false&sortBy=usage&page=0&pagesize=8&_=1406529617502 HTTP/1.1" 200 1658
XXX.XXX.X.5 - - [28/Jul/2014:12:10:18 +0530] "GET /client/api?command=listVirtualMachines&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&listAll=true&page=1&pagesize=20&_=1406529618114 HTTP/1.1" 200 11396
XXX.XXX.X.5 - - [28/Jul/2014:12:10:24 +0530] "GET /client/api?command=listVirtualMachines&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&listAll=true&page=1&pagesize=20&_=1406529624507 HTTP/1.1" 200 11396
XXX.XXX.X.5 - - [28/Jul/2014:12:10:59 +0530] "GET /client/api?command=listZones&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&_=1406529659774 HTTP/1.1" 200 363
XXX.XXX.X.5 - - [28/Jul/2014:12:10:59 +0530] "GET /client/api?command=listCapacity&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&fetchLatest=false&_=1406529659835 HTTP/1.1" 200 1112
XXX.XXX.X.5 - - [28/Jul/2014:12:10:59 +0530] "GET /client/api?command=listAlerts&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&page=1&pageSize=4&_=1406529659845 HTTP/1.1" 200 661
XXX.XXX.X.5 - - [28/Jul/2014:12:10:59 +0530] "GET /client/api?command=listHosts&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&state=Alert&page=1&pageSize=4&_=1406529659853 HTTP/1.1" 200 550
XXX.XXX.X.5 - - [28/Jul/2014:12:10:59 +0530] "GET /client/api?command=listCapacity&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&fetchLatest=false&sortBy=usage&page=0&pagesize=8&_=1406529659862 HTTP/1.1" 200 1658
XXX.XXX.X.5 - - [28/Jul/2014:12:11:00 +0530] "GET /client/api?command=listVirtualMachines&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&listAll=true&page=1&pagesize=20&_=1406529660883 HTTP/1.1" 200 11396
XXX.XXX.X.5 - - [28/Jul/2014:12:11:05 +0530] "GET /client/api?command=listVolumes&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&listAll=true&page=1&pagesize=20&_=1406529665134 HTTP/1.1" 200 4418
XXX.XXX.X.5 - - [28/Jul/2014:12:11:06 +0530] "GET /client/api?command=listVirtualMachines&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&listAll=true&page=1&pagesize=20&_=1406529666545 HTTP/1.1" 200 11396
XXX.XXX.X.5 - - [28/Jul/2014:12:11:08 +0530] "GET /client/api?command=listVolumes&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&listAll=true&page=1&pagesize=20&_=1406529668754 HTTP/1.1" 200 4418
XXX.XXX.X.5 - - [28/Jul/2014:12:11:09 +0530] "GET /client/api?command=listVirtualMachines&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&listAll=true&page=1&pagesize=20&_=1406529669975 HTTP/1.1" 200 11396
XXX.XXX.X.5 - - [28/Jul/2014:12:11:12 +0530] "GET /client/api?command=listVolumes&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&listAll=true&page=1&pagesize=20&_=1406529672424 HTTP/1.1" 200 4418
XXX.XXX.X.5 - - [28/Jul/2014:12:11:13 +0530] "GET /client/api?command=listVirtualMachines&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&listAll=true&page=1&pagesize=20&_=1406529673820 HTTP/1.1" 200 11396
XXX.XXX.X.5 - - [28/Jul/2014:12:11:23 +0530] "GET /client/api?command=listVolumes&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&listAll=true&page=1&pagesize=20&_=1406529683363 HTTP/1.1" 200 4418
XXX.XXX.X.5 - - [28/Jul/2014:12:11:24 +0530] "GET /client/api?command=listVirtualMachines&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&listAll=true&page=1&pagesize=20&_=1406529684865 HTTP/1.1" 200 11396

root@ubuntu:/home/giri#

root@ubuntu:/home/giri# tail -20 /var/log/cloudstack/management/manager.2014-07-28.log

root@ubuntu:/home/giri# tail -20 /var/log/cloudstack/management/host-manager.2014-07-28.log







________________________________
 From: Giri Prasad <g_...@yahoo.com.INVALID>
To: Rajani Karuturi <Ra...@citrix.com> 
Cc: "users@cloudstack.apache.org" <us...@cloudstack.apache.org> 
Sent: Monday, 28 July 2014 12:16 PM
Subject: Re: Unable to stop or delete vm's
 

Thanks.

Here is the agent.log:

tail -50 /var/log/cloudstack/agent/agent.log 
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:13:59,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 12:14:02,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:14:07,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 12:14:10,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:14:15,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 12:14:18,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:14:23,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 12:14:26,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:14:31,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 12:14:34,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:14:39,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 12:14:42,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:14:47,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250




________________________________

From: Rajani Karuturi <Ra...@citrix.com>
To: Giri Prasad <g_...@yahoo.com> 
Cc: "users@cloudstack.apache.org" <us...@cloudstack.apache.org> 
Sent: Monday, 28 July 2014 12:11 PM
Subject: Re: Unable to stop or delete vm's



DHCP command failed.
Did you check the agent.log on the host and /var/log/cloud.log on VR?  


~Rajani



On 28-Jul-2014, at 11:49 am, Giri Prasad <g_...@yahoo.com> wrote:

Thanks for the update.
>
>
>I updated the db to set the vm state as Stopped, and then destroyed the instance via cloudmonkey. Went thru fine, the gui shows the vm state as Destroyed.
>
>
> Then, I created a new vm, using the ISO option (already uploaded ISO into CS infra), for centos 6.5, the management server log repeats as given below. These log entries keep coming out for a few minutes to hours, and then says: Unable to create a new instance.
>
>
>Giri
>
>
>2014-07-28 11:40:14,863 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
>2014-07-28 11:40:14,864 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>2014-07-28 11:40:14,865 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>2014-07-28 11:40:14,866 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>2014-07-28 11:40:14,867 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>2014-07-28 11:40:14,868 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-6:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>2014-07-28 11:40:14,869 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>2014-07-28 11:40:14,870 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-9:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>2014-07-28 11:40:14,871 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-7:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>2014-07-28 11:40:14,872 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-11:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>2014-07-28 11:40:14,874 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-10:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>2014-07-28 11:40:14,875 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-4:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>2014-07-28 11:40:14,876 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-3:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>2014-07-28 11:40:14,877 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-14:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>2014-07-28 11:40:14,878 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-12:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>2014-07-28 11:40:14,879 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>2014-07-28 11:40:14,880 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>2014-07-28 11:40:14,881 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>2014-07-28 11:40:14,883 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>2014-07-28 11:40:14,884 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
>
>
>
>
>
>
>
>________________________________
> From: Rajani Karuturi <Ra...@citrix.com>
>To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>; Giri Prasad <g_...@yahoo.com> 
>Sent: Monday, 28 July 2014 11:01 AM
>Subject: Re: Unable to stop or delete vm's
>
>
>You cannot stop a vm in starting state. 
>Can you change the state of it manually in db and share the logs?
>
>mysq> update vm_instance set state='Stopped' where state=‘Starting’ and id=‘PUT_ID_HERE’;
>
>~Rajani
>
>
>
>
>On 28-Jul-2014, at 10:53 am, Giri Prasad <g_...@yahoo.com.INVALID> wrote:
>
>> Hello All,
>> 
>>  I installed CS 4.1 on Ubuntu 12.04 LTS. Installed two vm's, one each with centos 6.5 and ubuntu 12.04. Single server, local hdd storage.
>> 
>>  After a few days, I powerdown the entire linux box, and after booting up and starting the CS ms console, I started the vm's (26-Jul-2014). The vm's are telling me a status of 'Starting' for a long time.  Powered down the entire linux box, after a few hours.
>> 
>> 
>> Today once again booted the linux box, started CS ms console, the CS gui says, the vm's state as 'Starting'.
>> 
>>  I tried to stop or delete the vm thru cloudmonkey, but it is giving some errors. Any insights?
>> 
>> Thanks & Regards,
>> Giri
>> 
>> mycloudmonkeytpj>  stopVirtualMachine id="7757211b-d8a6-47aa-ba9f-24967499ba97"
>> .Async job 08cf9cb9-d2f7-46b7-af56-12d16a9c1290 failed
>> Error 530, We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting
>> {
>>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>>  "cmd": "org.apache.cloudstack.api.command.user.vm.StopVMCmd",
>>  "created": "2014-07-28T10:38:28+0530",
>>  "jobid": "08cf9cb9-d2f7-46b7-af56-12d16a9c1290",
>>  "jobprocstatus": 0,
>>  "jobresult": {
>>    "errorcode": 530,
>>    "errortext": "We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting"
>>  },
>>  "jobresultcode": 530,
>>  "jobresulttype": "object",
>>  "jobstatus": 2,
>>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>> }
>> 
>> mycloudmonkeytpj> stopVirtualMachine id="b74e6737-ec83-4483-b1a2-ab60045d941b"
>> .Async job d52f900d-dfa4-4b70-9d6a-8d4ba983e1aa failed
>> Error 530, We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting
>> {
>>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>>  "cmd": "org.apache.cloudstack.api.command.user.vm.StopVMCmd",
>>  "created": "2014-07-28T10:39:11+0530",
>>  "jobid": "d52f900d-dfa4-4b70-9d6a-8d4ba983e1aa",
>>  "jobprocstatus": 0,
>>  "jobresult": {
>>    "errorcode": 530,
>>    "errortext": "We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting"
>>  },
>>  "jobresultcode": 530,
>>  "jobresulttype": "object",
>>  "jobstatus": 2,
>>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>> }
>> 
>> mycloudmonkeytpj> destroyVirtualMachine id="7757211b-d8a6-47aa-ba9f-24967499ba97"
>> .Async job b3dc2bab-1b03-45d1-ba02-0c861f5043df failed
>> Error 530, We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting
>> {
>>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>>  "cmd": "org.apache.cloudstack.api.command.user.vm.DestroyVMCmd",
>>  "created": "2014-07-28T10:39:21+0530",
>>  "jobid": "b3dc2bab-1b03-45d1-ba02-0c861f5043df",
>>  "jobprocstatus": 0,
>>  "jobresult": {
>>    "errorcode": 530,
>>    "errortext": "We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting"
>>  },
>>  "jobresultcode": 530,
>>  "jobresulttype": "object",
>>  "jobstatus": 2,
>>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>> }
>> 
>> mycloudmonkeytpj> destroyVirtualMachine id="b74e6737-ec83-4483-b1a2-ab60045d941b"
>> .Async job 81e7e2c4-efc9-4673-90f7-8a69287cfa80 failed
>> Error 530, We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting
>> {
>>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>>  "cmd": "org.apache.cloudstack.api.command.user.vm.DestroyVMCmd",
>>  "created": "2014-07-28T10:39:31+0530",
>>  "jobid": "81e7e2c4-efc9-4673-90f7-8a69287cfa80",
>>  "jobprocstatus": 0,
>>  "jobresult": {
>>    "errorcode": 530,
>>    "errortext": "We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting"
>>  },
>>  "jobresultcode": 530,
>>  "jobresulttype": "object",
>>  "jobstatus": 2,
>>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>> }
>
>
>

Re: Unable to stop or delete vm's

Posted by Giri Prasad <g_...@yahoo.com.INVALID>.

And some more logs:


tail -20 /var/log/cloudstack/management/localhost.2014-07-28.log
28 Jul, 2014 10:31:26 AM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/usr/share/cloudstack-management/webapps/client/]
28 Jul, 2014 10:31:26 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing log4j from [classpath:log4j-cloud.xml]
28 Jul, 2014 10:31:27 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
28 Jul, 2014 10:32:47 AM org.apache.catalina.core.ApplicationContext log
INFO: Closing Spring root WebApplicationContext
28 Jul, 2014 10:32:47 AM org.apache.catalina.core.ApplicationContext log
INFO: Shutting down log4j
28 Jul, 2014 10:33:05 AM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/usr/share/cloudstack-management/webapps/client/]
28 Jul, 2014 10:33:05 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing log4j from [classpath:log4j-cloud.xml]
28 Jul, 2014 10:33:05 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext

root@ubuntu:/home/giri#
root@ubuntu:/home/giri#
root@ubuntu:/home/giri# tail -20 /var/log/cloudstack/management/access_log.2014-07-28.txtXXX.XXX.X.5 - - [28/Jul/2014:12:10:17 +0530] "GET /client/api?command=listCapacity&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&fetchLatest=false&_=1406529617462 HTTP/1.1" 200 1112
XXX.XXX.X.5 - - [28/Jul/2014:12:10:17 +0530] "GET /client/api?command=listAlerts&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&page=1&pageSize=4&_=1406529617478 HTTP/1.1" 200 661
XXX.XXX.X.5 - - [28/Jul/2014:12:10:17 +0530] "GET /client/api?command=listHosts&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&state=Alert&page=1&pageSize=4&_=1406529617489 HTTP/1.1" 200 550
XXX.XXX.X.5 - - [28/Jul/2014:12:10:17 +0530] "GET /client/api?command=listCapacity&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&fetchLatest=false&sortBy=usage&page=0&pagesize=8&_=1406529617502 HTTP/1.1" 200 1658
XXX.XXX.X.5 - - [28/Jul/2014:12:10:18 +0530] "GET /client/api?command=listVirtualMachines&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&listAll=true&page=1&pagesize=20&_=1406529618114 HTTP/1.1" 200 11396
XXX.XXX.X.5 - - [28/Jul/2014:12:10:24 +0530] "GET /client/api?command=listVirtualMachines&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&listAll=true&page=1&pagesize=20&_=1406529624507 HTTP/1.1" 200 11396
XXX.XXX.X.5 - - [28/Jul/2014:12:10:59 +0530] "GET /client/api?command=listZones&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&_=1406529659774 HTTP/1.1" 200 363
XXX.XXX.X.5 - - [28/Jul/2014:12:10:59 +0530] "GET /client/api?command=listCapacity&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&fetchLatest=false&_=1406529659835 HTTP/1.1" 200 1112
XXX.XXX.X.5 - - [28/Jul/2014:12:10:59 +0530] "GET /client/api?command=listAlerts&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&page=1&pageSize=4&_=1406529659845 HTTP/1.1" 200 661
XXX.XXX.X.5 - - [28/Jul/2014:12:10:59 +0530] "GET /client/api?command=listHosts&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&state=Alert&page=1&pageSize=4&_=1406529659853 HTTP/1.1" 200 550
XXX.XXX.X.5 - - [28/Jul/2014:12:10:59 +0530] "GET /client/api?command=listCapacity&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&fetchLatest=false&sortBy=usage&page=0&pagesize=8&_=1406529659862 HTTP/1.1" 200 1658
XXX.XXX.X.5 - - [28/Jul/2014:12:11:00 +0530] "GET /client/api?command=listVirtualMachines&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&listAll=true&page=1&pagesize=20&_=1406529660883 HTTP/1.1" 200 11396
XXX.XXX.X.5 - - [28/Jul/2014:12:11:05 +0530] "GET /client/api?command=listVolumes&response=json&sessionkey=Bak6hEyxBkOULTv0IMzCxFypJMY%3D&listAll=true&page=1&pagesize=20&_=1406529665134 HTTP/1.1" 200 4418

root@ubuntu:/home/giri#

root@ubuntu:/home/giri# tail -20 /var/log/cloudstack/management/manager.2014-07-28.log

root@ubuntu:/home/giri# tail -20 /var/log/cloudstack/management/host-manager.2014-07-28.log

Re: Unable to stop or delete vm's

Posted by Giri Prasad <g_...@yahoo.com.INVALID>.

After running for about 30-40 minutes, the new vm instance creation (via gui) fails, with the following error :

2014-07-28 13:37:04,831 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 13:37:04,832 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-14:null) Seq 1-988152032: Forwarding Seq 1-988152032:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"i-2-11-VM","wait":0}}] } to 279278805451651
2014-07-28 13:37:04,832 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-9:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 13:37:04,832 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-4:null) Seq 1-988152032: Forwarding Seq 1-988152032:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"i-2-11-VM","wait":0}}] } to 279278805451651
2014-07-28 13:37:04,832 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-12:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 13:37:04,833 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 1-988152032: Forwarding Seq 1-988152032:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"i-2-11-VM","wait":0}}] } to 279278805451651
tail -20 /var/log/cloudstack/agent/agent.log

2014-07-28 13:37:03,430 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 13:37:06,429 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 13:37:11,430 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 13:37:14,227 INFO  [cloud.agent.Agent] (AgentShutdownThread:null) Stopping the agent: Reason = sig.kill
2014-07-28 13:37:14,429 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)

tail -10 /var/log/cloudstack/management/access_log.2014-07-28.txt

XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET /client/scripts/system.js?t=1406534804765 HTTP/1.1" 200 -
XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET /client/scripts/docs.js?t=1406534804765 HTTP/1.1" 200 31975
XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET /client/scripts/zoneWizard.js?t=1406534804765 HTTP/1.1" 200 -
XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET /client/scripts/globalSettings.js?t=1406534804765 HTTP/1.1" 200 5870
XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET /client/api?command=listCapabilities&response=json&sessionkey=null&_=1406534804986 HTTP/1.1" 401 98
XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET /client/images/logo-login-oss.png HTTP/1.1" 304 -
XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET /client/images/sprites.png HTTP/1.1" 304 -
XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET /client/images/bg-login.png HTTP/1.1" 304 -








________________________________
 From: Rajani Karuturi <Ra...@citrix.com>
To: Giri Prasad <g_...@yahoo.com> 
Cc: "users@cloudstack.apache.org" <us...@cloudstack.apache.org> 
Sent: Monday, 28 July 2014 12:39 PM
Subject: Re: Unable to stop or delete vm's
 


looks like agent isn’t able to connect to management server.
Did your management server ip change? is the agent state connected in cloudstack UI for the system vms?  


~Rajani

Re: Unable to stop or delete vm's

Posted by Giri Prasad <g_...@yahoo.com.INVALID>.

The management server ip did not change.

Two system vms are there:

s-1-vm and v-2-vm, both says State:Running in the gui.

One VR: r-4-VM - State : Running.

I installed a fresh Ubuntu 12.04 LTS and CS 4.1 around 1st week of July-2014. Then created two vm's, one each for centos 6.5 and ubuntu 12.04. Installed different apps on these 2 vm's. Everything was running ok. Then I did not use the infra at at for about 2 weeks (i.e. never started the management server).

After this gap of 2 weeks, I started the management server. The vm instance is not starting up(2 user vm's). Not allowing to create a new vm.

I did not do, any change in the system or network or infra setup. 


From: Rajani Karuturi <Ra...@citrix.com>

To: Giri Prasad <g_...@yahoo.com> 
Cc: "users@cloudstack.apache.org" <us...@cloudstack.apache.org> 
Sent: Monday, 28 July 2014 12:39 PM
Subject: Re: Unable to stop or delete vm's
 


looks like agent isn’t able to connect to management server.
Did your management server ip change? is the agent state connected in cloudstack UI for the system vms?  


~Rajani



On 28-Jul-2014, at 12:16 pm, Giri Prasad <g_...@yahoo.com> wrote:

Thanks.
>
>
>Here is the agent.log:
>
>
>tail -50 /var/log/cloudstack/agent/agent.log 
>    at java.lang.Thread.run(Thread.java:701)
>2014-07-28 12:13:59,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
>2014-07-28 12:14:02,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
>java.net.NoRouteToHostException: No route to host
>    at sun.nio.ch.Net.connect(Native Method)
>    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
>    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
>    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
>    at java.lang.Thread.run(Thread.java:701)
>2014-07-28 12:14:07,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
>2014-07-28 12:14:10,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
>java.net.NoRouteToHostException: No route to host
>    at sun.nio.ch.Net.connect(Native Method)
>    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
>    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
>    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
>    at java.lang.Thread.run(Thread.java:701)
>2014-07-28 12:14:15,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
>2014-07-28 12:14:18,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
>java.net.NoRouteToHostException: No route to host
>    at sun.nio.ch.Net.connect(Native Method)
>    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
>    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
>    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
>    at java.lang.Thread.run(Thread.java:701)
>2014-07-28 12:14:23,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
>2014-07-28 12:14:26,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
>java.net.NoRouteToHostException: No route to host
>    at sun.nio.ch.Net.connect(Native Method)
>    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
>    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
>    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
>    at java.lang.Thread.run(Thread.java:701)
>2014-07-28 12:14:31,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
>2014-07-28 12:14:34,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
>java.net.NoRouteToHostException: No route to host
>    at sun.nio.ch.Net.connect(Native Method)
>    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
>    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
>    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
>    at java.lang.Thread.run(Thread.java:701)
>2014-07-28 12:14:39,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
>2014-07-28 12:14:42,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
>java.net.NoRouteToHostException: No route to host
>    at sun.nio.ch.Net.connect(Native Method)
>    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
>    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
>    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
>    at java.lang.Thread.run(Thread.java:701)
>2014-07-28 12:14:47,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
>
>
>
>
>
>
>________________________________
> From: Rajani Karuturi <Ra...@citrix.com>
>To: Giri Prasad <g_...@yahoo.com> 
>Cc: "users@cloudstack.apache.org" <us...@cloudstack.apache.org> 
>Sent: Monday, 28 July 2014 12:11 PM
>Subject: Re: Unable to stop or delete vm's
>
>
>
>DHCP command failed.
Did you check the agent.log on the host and /var/log/cloud.log on VR?  
>
>
>~Rajani
>
>
>
>
>On 28-Jul-2014, at 11:49 am, Giri Prasad <g_...@yahoo.com> wrote:
>
>Thanks for the update.
>>
>>
>>I updated the db to set the vm state as Stopped, and then destroyed the instance via cloudmonkey. Went thru fine, the gui shows the vm state as Destroyed.
>>
>>
>> Then, I created a new vm, using the ISO option (already uploaded ISO into CS infra), for centos 6.5, the management server log repeats as given below. These log entries keep coming out for a few minutes to hours, and then says: Unable to create a new instance.
>>
>>
>>Giri
>>
>>
>>2014-07-28 11:40:14,863 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
>>2014-07-28 11:40:14,864 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>2014-07-28 11:40:14,865 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>2014-07-28 11:40:14,866 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>2014-07-28 11:40:14,867 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>2014-07-28 11:40:14,868 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-6:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>2014-07-28 11:40:14,869 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>2014-07-28 11:40:14,870 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-9:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>2014-07-28 11:40:14,871 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-7:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>2014-07-28 11:40:14,872 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-11:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>2014-07-28 11:40:14,874 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-10:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>2014-07-28 11:40:14,875 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-4:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>2014-07-28 11:40:14,876 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-3:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>2014-07-28 11:40:14,877 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-14:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>2014-07-28 11:40:14,878 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-12:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>2014-07-28 11:40:14,879 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>2014-07-28 11:40:14,880 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>2014-07-28 11:40:14,881 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>2014-07-28 11:40:14,883 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>2014-07-28 11:40:14,884 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>>
>>
>>
>>
>>
>>
>>
>>________________________________
>> From: Rajani Karuturi <Ra...@citrix.com>
>>To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>; Giri Prasad <g_...@yahoo.com> 
>>Sent: Monday, 28 July 2014 11:01 AM
>>Subject: Re: Unable to stop or delete vm's
>>
>>
>>You cannot stop a vm in starting state. 
>>Can you change the state of it manually in db and share the logs?
>>
>>mysq> update vm_instance set state='Stopped' where state=‘Starting’ and id=‘PUT_ID_HERE’;
>>
>>~Rajani
>>
>>
>>
>>
>>On 28-Jul-2014, at 10:53 am, Giri Prasad <g_...@yahoo.com.INVALID> wrote:
>>
>>> Hello All,
>>> 
>>>  I installed CS 4.1 on Ubuntu 12.04 LTS. Installed two vm's, one each with centos 6.5 and ubuntu 12.04. Single server, local hdd storage.
>>> 
>>>  After a few days, I powerdown the entire linux box, and after booting up and starting the CS ms console, I started the vm's (26-Jul-2014). The vm's are telling me a status of 'Starting' for a long time.  Powered down the entire linux box, after a few hours.
>>> 
>>> 
>>> Today once again booted the linux box, started CS ms console, the CS gui says, the vm's state as 'Starting'.
>>> 
>>>  I tried to stop or delete the vm thru cloudmonkey, but it is giving some errors. Any insights?
>>> 
>>> Thanks & Regards,
>>> Giri
>>> 
>>> mycloudmonkeytpj>  stopVirtualMachine id="7757211b-d8a6-47aa-ba9f-24967499ba97"
>>> .Async job 08cf9cb9-d2f7-46b7-af56-12d16a9c1290 failed
>>> Error 530, We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting
>>> {
>>>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>>>  "cmd": "org.apache.cloudstack.api.command.user.vm.StopVMCmd",
>>>  "created": "2014-07-28T10:38:28+0530",
>>>  "jobid": "08cf9cb9-d2f7-46b7-af56-12d16a9c1290",
>>>  "jobprocstatus": 0,
>>>  "jobresult": {
>>>    "errorcode": 530,
>>>    "errortext": "We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting"
>>>  },
>>>  "jobresultcode": 530,
>>>  "jobresulttype": "object",
>>>  "jobstatus": 2,
>>>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>>> }
>>> 
>>> mycloudmonkeytpj> stopVirtualMachine id="b74e6737-ec83-4483-b1a2-ab60045d941b"
>>> .Async job d52f900d-dfa4-4b70-9d6a-8d4ba983e1aa failed
>>> Error 530, We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting
>>> {
>>>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>>>  "cmd": "org.apache.cloudstack.api.command.user.vm.StopVMCmd",
>>>  "created": "2014-07-28T10:39:11+0530",
>>>  "jobid": "d52f900d-dfa4-4b70-9d6a-8d4ba983e1aa",
>>>  "jobprocstatus": 0,
>>>  "jobresult": {
>>>    "errorcode": 530,
>>>    "errortext": "We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting"
>>>  },
>>>  "jobresultcode": 530,
>>>  "jobresulttype": "object",
>>>  "jobstatus": 2,
>>>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>>> }
>>> 
>>> mycloudmonkeytpj> destroyVirtualMachine id="7757211b-d8a6-47aa-ba9f-24967499ba97"
>>> .Async job b3dc2bab-1b03-45d1-ba02-0c861f5043df failed
>>> Error 530, We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting
>>> {
>>>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>>>  "cmd": "org.apache.cloudstack.api.command.user.vm.DestroyVMCmd",
>>>  "created": "2014-07-28T10:39:21+0530",
>>>  "jobid": "b3dc2bab-1b03-45d1-ba02-0c861f5043df",
>>>  "jobprocstatus": 0,
>>>  "jobresult": {
>>>    "errorcode": 530,
>>>    "errortext": "We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting"
>>>  },
>>>  "jobresultcode": 530,
>>>  "jobresulttype": "object",
>>>  "jobstatus": 2,
>>>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>>> }
>>> 
>>> mycloudmonkeytpj> destroyVirtualMachine id="b74e6737-ec83-4483-b1a2-ab60045d941b"
>>> .Async job 81e7e2c4-efc9-4673-90f7-8a69287cfa80 failed
>>> Error 530, We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting
>>> {
>>>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>>>  "cmd": "org.apache.cloudstack.api.command.user.vm.DestroyVMCmd",
>>>  "created": "2014-07-28T10:39:31+0530",
>>>  "jobid": "81e7e2c4-efc9-4673-90f7-8a69287cfa80",
>>>  "jobprocstatus": 0,
>>>  "jobresult": {
>>>    "errorcode": 530,
>>>    "errortext": "We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting"
>>>  },
>>>  "jobresultcode": 530,
>>>  "jobresulttype": "object",
>>>  "jobstatus": 2,
>>>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>>> }
>>
>>
>>
>
>
>

Re: Unable to stop or delete vm's

Posted by Rajani Karuturi <Ra...@citrix.com>.

looks like agent isn’t able to connect to management server.
Did your management server ip change? is the agent state connected in cloudstack UI for the system vms?

~Rajani



On 28-Jul-2014, at 12:16 pm, Giri Prasad <g_...@yahoo.com>> wrote:

Thanks.

Here is the agent.log:

tail -50 /var/log/cloudstack/agent/agent.log
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:13:59,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 12:14:02,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:14:07,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 12:14:10,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:14:15,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 12:14:18,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:14:23,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 12:14:26,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:14:31,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 12:14:34,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:14:39,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 12:14:42,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:14:47,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250


________________________________
From: Rajani Karuturi <Ra...@citrix.com>>
To: Giri Prasad <g_...@yahoo.com>>
Cc: "users@cloudstack.apache.org<ma...@cloudstack.apache.org>" <us...@cloudstack.apache.org>>
Sent: Monday, 28 July 2014 12:11 PM
Subject: Re: Unable to stop or delete vm's

DHCP command failed.
Did you check the agent.log on the host and /var/log/cloud.log on VR?

~Rajani



On 28-Jul-2014, at 11:49 am, Giri Prasad <g_...@yahoo.com>> wrote:

Thanks for the update.

I updated the db to set the vm state as Stopped, and then destroyed the instance via cloudmonkey. Went thru fine, the gui shows the vm state as Destroyed.

 Then, I created a new vm, using the ISO option (already uploaded ISO into CS infra), for centos 6.5, the management server log repeats as given below. These log entries keep coming out for a few minutes to hours, and then says: Unable to create a new instance.

Giri

2014-07-28 11:40:14,863 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,864 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,865 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,866 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,867 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,868 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-6:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,869 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,870 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-9:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,871 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-7:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,872 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-11:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,874 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-10:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,875 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-4:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,876 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-3:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,877 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-14:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,878 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-12:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,879 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,880 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,881 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,883 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,884 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651




________________________________
From: Rajani Karuturi <Ra...@citrix.com>>
To: "users@cloudstack.apache.org<ma...@cloudstack.apache.org>" <us...@cloudstack.apache.org>>; Giri Prasad <g_...@yahoo.com>>
Sent: Monday, 28 July 2014 11:01 AM
Subject: Re: Unable to stop or delete vm's

You cannot stop a vm in starting state.
Can you change the state of it manually in db and share the logs?

mysq> update vm_instance set state='Stopped' where state=‘Starting’ and id=‘PUT_ID_HERE’;

~Rajani



On 28-Jul-2014, at 10:53 am, Giri Prasad <g_...@yahoo.com.INVALID>> wrote:

> Hello All,
>
>  I installed CS 4.1 on Ubuntu 12.04 LTS. Installed two vm's, one each with centos 6.5 and ubuntu 12.04. Single server, local hdd storage.
>
>  After a few days, I powerdown the entire linux box, and after booting up and starting the CS ms console, I started the vm's (26-Jul-2014). The vm's are telling me a status of 'Starting' for a long time.  Powered down the entire linux box, after a few hours.
>
>
> Today once again booted the linux box, started CS ms console, the CS gui says, the vm's state as 'Starting'.
>
>  I tried to stop or delete the vm thru cloudmonkey, but it is giving some errors. Any insights?
>
> Thanks & Regards,
> Giri
>
> mycloudmonkeytpj>  stopVirtualMachine id="7757211b-d8a6-47aa-ba9f-24967499ba97"
> .Async job 08cf9cb9-d2f7-46b7-af56-12d16a9c1290 failed
> Error 530, We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting
> {
>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>  "cmd": "org.apache.cloudstack.api.command.user.vm.StopVMCmd",
>  "created": "2014-07-28T10:38:28+0530",
>  "jobid": "08cf9cb9-d2f7-46b7-af56-12d16a9c1290",
>  "jobprocstatus": 0,
>  "jobresult": {
>    "errorcode": 530,
>    "errortext": "We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting"
>  },
>  "jobresultcode": 530,
>  "jobresulttype": "object",
>  "jobstatus": 2,
>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> }
>
> mycloudmonkeytpj> stopVirtualMachine id="b74e6737-ec83-4483-b1a2-ab60045d941b"
> .Async job d52f900d-dfa4-4b70-9d6a-8d4ba983e1aa failed
> Error 530, We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting
> {
>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>  "cmd": "org.apache.cloudstack.api.command.user.vm.StopVMCmd",
>  "created": "2014-07-28T10:39:11+0530",
>  "jobid": "d52f900d-dfa4-4b70-9d6a-8d4ba983e1aa",
>  "jobprocstatus": 0,
>  "jobresult": {
>    "errorcode": 530,
>    "errortext": "We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting"
>  },
>  "jobresultcode": 530,
>  "jobresulttype": "object",
>  "jobstatus": 2,
>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> }
>
> mycloudmonkeytpj> destroyVirtualMachine id="7757211b-d8a6-47aa-ba9f-24967499ba97"
> .Async job b3dc2bab-1b03-45d1-ba02-0c861f5043df failed
> Error 530, We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting
> {
>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>  "cmd": "org.apache.cloudstack.api.command.user.vm.DestroyVMCmd",
>  "created": "2014-07-28T10:39:21+0530",
>  "jobid": "b3dc2bab-1b03-45d1-ba02-0c861f5043df",
>  "jobprocstatus": 0,
>  "jobresult": {
>    "errorcode": 530,
>    "errortext": "We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting"
>  },
>  "jobresultcode": 530,
>  "jobresulttype": "object",
>  "jobstatus": 2,
>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> }
>
> mycloudmonkeytpj> destroyVirtualMachine id="b74e6737-ec83-4483-b1a2-ab60045d941b"
> .Async job 81e7e2c4-efc9-4673-90f7-8a69287cfa80 failed
> Error 530, We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting
> {
>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>  "cmd": "org.apache.cloudstack.api.command.user.vm.DestroyVMCmd",
>  "created": "2014-07-28T10:39:31+0530",
>  "jobid": "81e7e2c4-efc9-4673-90f7-8a69287cfa80",
>  "jobprocstatus": 0,
>  "jobresult": {
>    "errorcode": 530,
>    "errortext": "We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting"
>  },
>  "jobresultcode": 530,
>  "jobresulttype": "object",
>  "jobstatus": 2,
>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> }

Re: Unable to stop or delete vm's

Posted by Giri Prasad <g_...@yahoo.com.INVALID>.

Thanks.

Here is the agent.log:

tail -50 /var/log/cloudstack/agent/agent.log 
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:13:59,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 12:14:02,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:14:07,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 12:14:10,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:14:15,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 12:14:18,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:14:23,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 12:14:26,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:14:31,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 12:14:34,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:14:39,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250
2014-07-28 12:14:42,273 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.Net.connect(Native Method)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
    at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
    at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
    at java.lang.Thread.run(Thread.java:701)
2014-07-28 12:14:47,274 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250




________________________________
 From: Rajani Karuturi <Ra...@citrix.com>
To: Giri Prasad <g_...@yahoo.com> 
Cc: "users@cloudstack.apache.org" <us...@cloudstack.apache.org> 
Sent: Monday, 28 July 2014 12:11 PM
Subject: Re: Unable to stop or delete vm's
 


DHCP command failed.
Did you check the agent.log on the host and /var/log/cloud.log on VR?  


~Rajani



On 28-Jul-2014, at 11:49 am, Giri Prasad <g_...@yahoo.com> wrote:

Thanks for the update.
>
>
>I updated the db to set the vm state as Stopped, and then destroyed the instance via cloudmonkey. Went thru fine, the gui shows the vm state as Destroyed.
>
>
> Then, I created a new vm, using the ISO option (already uploaded ISO into CS infra), for centos 6.5, the management server log repeats as given below. These log entries keep coming out for a few minutes to hours, and then says: Unable to create a new instance.
>
>
>Giri
>
>
>2014-07-28 11:40:14,863 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
>2014-07-28 11:40:14,864 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>2014-07-28 11:40:14,865 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>2014-07-28 11:40:14,866 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>2014-07-28 11:40:14,867 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>2014-07-28 11:40:14,868 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-6:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>2014-07-28 11:40:14,869 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>2014-07-28 11:40:14,870 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-9:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>2014-07-28 11:40:14,871 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-7:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>2014-07-28 11:40:14,872 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-11:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>2014-07-28 11:40:14,874 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-10:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>2014-07-28 11:40:14,875 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-4:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>2014-07-28 11:40:14,876 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-3:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>2014-07-28 11:40:14,877 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-14:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>2014-07-28 11:40:14,878 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-12:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>2014-07-28 11:40:14,879 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>2014-07-28 11:40:14,880 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>2014-07-28 11:40:14,881 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>2014-07-28 11:40:14,883 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>2014-07-28 11:40:14,884 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
 } to 279278805451651
>
>
>
>
>
>
>
>________________________________
> From: Rajani Karuturi <Ra...@citrix.com>
>To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>; Giri Prasad <g_...@yahoo.com> 
>Sent: Monday, 28 July 2014 11:01 AM
>Subject: Re: Unable to stop or delete vm's
>
>
>You cannot stop a vm in starting state. 
>Can you change the state of it manually in db and share the logs?
>
>mysq> update vm_instance set state='Stopped' where state=‘Starting’ and id=‘PUT_ID_HERE’;
>
>~Rajani
>
>
>
>
>On 28-Jul-2014, at 10:53 am, Giri Prasad <g_...@yahoo.com.INVALID> wrote:
>
>> Hello All,
>> 
>>  I installed CS 4.1 on Ubuntu 12.04 LTS. Installed two vm's, one each with centos 6.5 and ubuntu 12.04. Single server, local hdd storage.
>> 
>>  After a few days, I powerdown the entire linux box, and after booting up and starting the CS ms console, I started the vm's (26-Jul-2014). The vm's are telling me a status of 'Starting' for a long time.  Powered down the entire linux box, after a few hours.
>> 
>> 
>> Today once again booted the linux box, started CS ms console, the CS gui says, the vm's state as 'Starting'.
>> 
>>  I tried to stop or delete the vm thru cloudmonkey, but it is giving some errors. Any insights?
>> 
>> Thanks & Regards,
>> Giri
>> 
>> mycloudmonkeytpj>  stopVirtualMachine id="7757211b-d8a6-47aa-ba9f-24967499ba97"
>> .Async job 08cf9cb9-d2f7-46b7-af56-12d16a9c1290 failed
>> Error 530, We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting
>> {
>>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>>  "cmd": "org.apache.cloudstack.api.command.user.vm.StopVMCmd",
>>  "created": "2014-07-28T10:38:28+0530",
>>  "jobid": "08cf9cb9-d2f7-46b7-af56-12d16a9c1290",
>>  "jobprocstatus": 0,
>>  "jobresult": {
>>    "errorcode": 530,
>>    "errortext": "We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting"
>>  },
>>  "jobresultcode": 530,
>>  "jobresulttype": "object",
>>  "jobstatus": 2,
>>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>> }
>> 
>> mycloudmonkeytpj> stopVirtualMachine id="b74e6737-ec83-4483-b1a2-ab60045d941b"
>> .Async job d52f900d-dfa4-4b70-9d6a-8d4ba983e1aa failed
>> Error 530, We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting
>> {
>>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>>  "cmd": "org.apache.cloudstack.api.command.user.vm.StopVMCmd",
>>  "created": "2014-07-28T10:39:11+0530",
>>  "jobid": "d52f900d-dfa4-4b70-9d6a-8d4ba983e1aa",
>>  "jobprocstatus": 0,
>>  "jobresult": {
>>    "errorcode": 530,
>>    "errortext": "We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting"
>>  },
>>  "jobresultcode": 530,
>>  "jobresulttype": "object",
>>  "jobstatus": 2,
>>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>> }
>> 
>> mycloudmonkeytpj> destroyVirtualMachine id="7757211b-d8a6-47aa-ba9f-24967499ba97"
>> .Async job b3dc2bab-1b03-45d1-ba02-0c861f5043df failed
>> Error 530, We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting
>> {
>>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>>  "cmd": "org.apache.cloudstack.api.command.user.vm.DestroyVMCmd",
>>  "created": "2014-07-28T10:39:21+0530",
>>  "jobid": "b3dc2bab-1b03-45d1-ba02-0c861f5043df",
>>  "jobprocstatus": 0,
>>  "jobresult": {
>>    "errorcode": 530,
>>    "errortext": "We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting"
>>  },
>>  "jobresultcode": 530,
>>  "jobresulttype": "object",
>>  "jobstatus": 2,
>>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>> }
>> 
>> mycloudmonkeytpj> destroyVirtualMachine id="b74e6737-ec83-4483-b1a2-ab60045d941b"
>> .Async job 81e7e2c4-efc9-4673-90f7-8a69287cfa80 failed
>> Error 530, We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting
>> {
>>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>>  "cmd": "org.apache.cloudstack.api.command.user.vm.DestroyVMCmd",
>>  "created": "2014-07-28T10:39:31+0530",
>>  "jobid": "81e7e2c4-efc9-4673-90f7-8a69287cfa80",
>>  "jobprocstatus": 0,
>>  "jobresult": {
>>    "errorcode": 530,
>>    "errortext": "We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting"
>>  },
>>  "jobresultcode": 530,
>>  "jobresulttype": "object",
>>  "jobstatus": 2,
>>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>> }
>
>
>

Re: Unable to stop or delete vm's

Posted by Rajani Karuturi <Ra...@citrix.com>.

DHCP command failed.
Did you check the agent.log on the host and /var/log/cloud.log on VR?

~Rajani



On 28-Jul-2014, at 11:49 am, Giri Prasad <g_...@yahoo.com>> wrote:

Thanks for the update.

I updated the db to set the vm state as Stopped, and then destroyed the instance via cloudmonkey. Went thru fine, the gui shows the vm state as Destroyed.

 Then, I created a new vm, using the ISO option (already uploaded ISO into CS infra), for centos 6.5, the management server log repeats as given below. These log entries keep coming out for a few minutes to hours, and then says: Unable to create a new instance.

Giri

2014-07-28 11:40:14,863 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,864 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,865 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,866 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,867 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,868 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-6:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,869 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,870 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-9:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,871 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-7:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,872 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-11:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,874 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-10:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,875 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-4:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,876 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-3:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,877 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-14:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,878 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-12:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,879 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,880 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,881 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,883 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,884 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651




________________________________
From: Rajani Karuturi <Ra...@citrix.com>>
To: "users@cloudstack.apache.org<ma...@cloudstack.apache.org>" <us...@cloudstack.apache.org>>; Giri Prasad <g_...@yahoo.com>>
Sent: Monday, 28 July 2014 11:01 AM
Subject: Re: Unable to stop or delete vm's

You cannot stop a vm in starting state.
Can you change the state of it manually in db and share the logs?

mysq> update vm_instance set state='Stopped' where state=‘Starting’ and id=‘PUT_ID_HERE’;

~Rajani



On 28-Jul-2014, at 10:53 am, Giri Prasad <g_...@yahoo.com.INVALID>> wrote:

> Hello All,
>
>  I installed CS 4.1 on Ubuntu 12.04 LTS. Installed two vm's, one each with centos 6.5 and ubuntu 12.04. Single server, local hdd storage.
>
>  After a few days, I powerdown the entire linux box, and after booting up and starting the CS ms console, I started the vm's (26-Jul-2014). The vm's are telling me a status of 'Starting' for a long time.  Powered down the entire linux box, after a few hours.
>
>
> Today once again booted the linux box, started CS ms console, the CS gui says, the vm's state as 'Starting'.
>
>  I tried to stop or delete the vm thru cloudmonkey, but it is giving some errors. Any insights?
>
> Thanks & Regards,
> Giri
>
> mycloudmonkeytpj>  stopVirtualMachine id="7757211b-d8a6-47aa-ba9f-24967499ba97"
> .Async job 08cf9cb9-d2f7-46b7-af56-12d16a9c1290 failed
> Error 530, We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting
> {
>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>  "cmd": "org.apache.cloudstack.api.command.user.vm.StopVMCmd",
>  "created": "2014-07-28T10:38:28+0530",
>  "jobid": "08cf9cb9-d2f7-46b7-af56-12d16a9c1290",
>  "jobprocstatus": 0,
>  "jobresult": {
>    "errorcode": 530,
>    "errortext": "We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting"
>  },
>  "jobresultcode": 530,
>  "jobresulttype": "object",
>  "jobstatus": 2,
>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> }
>
> mycloudmonkeytpj> stopVirtualMachine id="b74e6737-ec83-4483-b1a2-ab60045d941b"
> .Async job d52f900d-dfa4-4b70-9d6a-8d4ba983e1aa failed
> Error 530, We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting
> {
>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>  "cmd": "org.apache.cloudstack.api.command.user.vm.StopVMCmd",
>  "created": "2014-07-28T10:39:11+0530",
>  "jobid": "d52f900d-dfa4-4b70-9d6a-8d4ba983e1aa",
>  "jobprocstatus": 0,
>  "jobresult": {
>    "errorcode": 530,
>    "errortext": "We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting"
>  },
>  "jobresultcode": 530,
>  "jobresulttype": "object",
>  "jobstatus": 2,
>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> }
>
> mycloudmonkeytpj> destroyVirtualMachine id="7757211b-d8a6-47aa-ba9f-24967499ba97"
> .Async job b3dc2bab-1b03-45d1-ba02-0c861f5043df failed
> Error 530, We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting
> {
>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>  "cmd": "org.apache.cloudstack.api.command.user.vm.DestroyVMCmd",
>  "created": "2014-07-28T10:39:21+0530",
>  "jobid": "b3dc2bab-1b03-45d1-ba02-0c861f5043df",
>  "jobprocstatus": 0,
>  "jobresult": {
>    "errorcode": 530,
>    "errortext": "We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting"
>  },
>  "jobresultcode": 530,
>  "jobresulttype": "object",
>  "jobstatus": 2,
>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> }
>
> mycloudmonkeytpj> destroyVirtualMachine id="b74e6737-ec83-4483-b1a2-ab60045d941b"
> .Async job 81e7e2c4-efc9-4673-90f7-8a69287cfa80 failed
> Error 530, We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting
> {
>  "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>  "cmd": "org.apache.cloudstack.api.command.user.vm.DestroyVMCmd",
>  "created": "2014-07-28T10:39:31+0530",
>  "jobid": "81e7e2c4-efc9-4673-90f7-8a69287cfa80",
>  "jobprocstatus": 0,
>  "jobresult": {
>    "errorcode": 530,
>    "errortext": "We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting"
>  },
>  "jobresultcode": 530,
>  "jobresulttype": "object",
>  "jobstatus": 2,
>  "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> }

Re: Unable to stop or delete vm's

Posted by Giri Prasad <g_...@yahoo.com.INVALID>.

Thanks for the update.

I updated the db to set the vm state as Stopped, and then destroyed the instance via cloudmonkey. Went thru fine, the gui shows the vm state as Destroyed.

 Then, I created a new vm, using the ISO option (already uploaded ISO into CS infra), for centos 6.5, the management server log repeats as given below. These log entries keep coming out for a few minutes to hours, and then says: Unable to create a new instance.

Giri

2014-07-28 11:40:14,863 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,864 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,865 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,866 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,867 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,868 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-6:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,869 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-15:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,870 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-9:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,871 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-7:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,872 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-11:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,874 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-10:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,875 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-4:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,876 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-3:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,877 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-14:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,878 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-12:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,879 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-1:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,880 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-13:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,881 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,883 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651
2014-07-28 11:40:14,884 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-5:null) Seq 1-988151911: Forwarding Seq 1-988151911:  { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651






________________________________
 From: Rajani Karuturi <Ra...@citrix.com>
To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>; Giri Prasad <g_...@yahoo.com> 
Sent: Monday, 28 July 2014 11:01 AM
Subject: Re: Unable to stop or delete vm's
 

You cannot stop a vm in starting state. 
Can you change the state of it manually in db and share the logs?

mysq> update vm_instance set state='Stopped' where state=‘Starting’ and id=‘PUT_ID_HERE’;

~Rajani




On 28-Jul-2014, at 10:53 am, Giri Prasad <g_...@yahoo.com.INVALID> wrote:

> Hello All,
> 
>  I installed CS 4.1 on Ubuntu 12.04 LTS. Installed two vm's, one each with centos 6.5 and ubuntu 12.04. Single server, local hdd storage.
> 
>  After a few days, I powerdown the entire linux box, and after booting up and starting the CS ms console, I started the vm's (26-Jul-2014). The vm's are telling me a status of 'Starting' for a long time.  Powered down the entire linux box, after a few hours.
> 
> 
> Today once again booted the linux box, started CS ms console, the CS gui says, the vm's state as 'Starting'.
> 
>  I tried to stop or delete the vm thru cloudmonkey, but it is giving some errors. Any insights?
> 
> Thanks & Regards,
> Giri
> 
> mycloudmonkeytpj>  stopVirtualMachine id="7757211b-d8a6-47aa-ba9f-24967499ba97"
> .Async job 08cf9cb9-d2f7-46b7-af56-12d16a9c1290 failed
> Error 530, We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting
> {
>   "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>   "cmd": "org.apache.cloudstack.api.command.user.vm.StopVMCmd",
>   "created": "2014-07-28T10:38:28+0530",
>   "jobid": "08cf9cb9-d2f7-46b7-af56-12d16a9c1290",
>   "jobprocstatus": 0,
>   "jobresult": {
>     "errorcode": 530,
>     "errortext": "We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting"
>   },
>   "jobresultcode": 530,
>   "jobresulttype": "object",
>   "jobstatus": 2,
>   "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> }
> 
> mycloudmonkeytpj> stopVirtualMachine id="b74e6737-ec83-4483-b1a2-ab60045d941b"
> .Async job d52f900d-dfa4-4b70-9d6a-8d4ba983e1aa failed
> Error 530, We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting
> {
>   "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>   "cmd": "org.apache.cloudstack.api.command.user.vm.StopVMCmd",
>   "created": "2014-07-28T10:39:11+0530",
>   "jobid": "d52f900d-dfa4-4b70-9d6a-8d4ba983e1aa",
>   "jobprocstatus": 0,
>   "jobresult": {
>     "errorcode": 530,
>     "errortext": "We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting"
>   },
>   "jobresultcode": 530,
>   "jobresulttype": "object",
>   "jobstatus": 2,
>   "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> }
> 
> mycloudmonkeytpj> destroyVirtualMachine id="7757211b-d8a6-47aa-ba9f-24967499ba97"
> .Async job b3dc2bab-1b03-45d1-ba02-0c861f5043df failed
> Error 530, We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting
> {
>   "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>   "cmd": "org.apache.cloudstack.api.command.user.vm.DestroyVMCmd",
>   "created": "2014-07-28T10:39:21+0530",
>   "jobid": "b3dc2bab-1b03-45d1-ba02-0c861f5043df",
>   "jobprocstatus": 0,
>   "jobresult": {
>     "errorcode": 530,
>     "errortext": "We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting"
>   },
>   "jobresultcode": 530,
>   "jobresulttype": "object",
>   "jobstatus": 2,
>   "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> }
> 
> mycloudmonkeytpj> destroyVirtualMachine id="b74e6737-ec83-4483-b1a2-ab60045d941b"
> .Async job 81e7e2c4-efc9-4673-90f7-8a69287cfa80 failed
> Error 530, We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting
> {
>   "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>   "cmd": "org.apache.cloudstack.api.command.user.vm.DestroyVMCmd",
>   "created": "2014-07-28T10:39:31+0530",
>   "jobid": "81e7e2c4-efc9-4673-90f7-8a69287cfa80",
>   "jobprocstatus": 0,
>   "jobresult": {
>     "errorcode": 530,
>     "errortext": "We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting"
>   },
>   "jobresultcode": 530,
>   "jobresulttype": "object",
>   "jobstatus": 2,
>   "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> }

Re: Unable to stop or delete vm's

Posted by Rajani Karuturi <Ra...@citrix.com>.

You cannot stop a vm in starting state. 
Can you change the state of it manually in db and share the logs?

mysq> update vm_instance set state='Stopped' where state=‘Starting’ and id=‘PUT_ID_HERE’;

~Rajani



On 28-Jul-2014, at 10:53 am, Giri Prasad <g_...@yahoo.com.INVALID> wrote:

> Hello All,
> 
>  I installed CS 4.1 on Ubuntu 12.04 LTS. Installed two vm's, one each with centos 6.5 and ubuntu 12.04. Single server, local hdd storage.
> 
>  After a few days, I powerdown the entire linux box, and after booting up and starting the CS ms console, I started the vm's (26-Jul-2014). The vm's are telling me a status of 'Starting' for a long time.  Powered down the entire linux box, after a few hours.
> 
> 
> Today once again booted the linux box, started CS ms console, the CS gui says, the vm's state as 'Starting'.
> 
>  I tried to stop or delete the vm thru cloudmonkey, but it is giving some errors. Any insights?
> 
> Thanks & Regards,
> Giri
> 
> mycloudmonkeytpj>  stopVirtualMachine id="7757211b-d8a6-47aa-ba9f-24967499ba97"
> .Async job 08cf9cb9-d2f7-46b7-af56-12d16a9c1290 failed
> Error 530, We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting
> {
>   "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>   "cmd": "org.apache.cloudstack.api.command.user.vm.StopVMCmd",
>   "created": "2014-07-28T10:38:28+0530",
>   "jobid": "08cf9cb9-d2f7-46b7-af56-12d16a9c1290",
>   "jobprocstatus": 0,
>   "jobresult": {
>     "errorcode": 530,
>     "errortext": "We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting"
>   },
>   "jobresultcode": 530,
>   "jobresulttype": "object",
>   "jobstatus": 2,
>   "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> }
> 
> mycloudmonkeytpj> stopVirtualMachine id="b74e6737-ec83-4483-b1a2-ab60045d941b"
> .Async job d52f900d-dfa4-4b70-9d6a-8d4ba983e1aa failed
> Error 530, We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting
> {
>   "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>   "cmd": "org.apache.cloudstack.api.command.user.vm.StopVMCmd",
>   "created": "2014-07-28T10:39:11+0530",
>   "jobid": "d52f900d-dfa4-4b70-9d6a-8d4ba983e1aa",
>   "jobprocstatus": 0,
>   "jobresult": {
>     "errorcode": 530,
>     "errortext": "We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting"
>   },
>   "jobresultcode": 530,
>   "jobresulttype": "object",
>   "jobstatus": 2,
>   "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> }
> 
> mycloudmonkeytpj> destroyVirtualMachine id="7757211b-d8a6-47aa-ba9f-24967499ba97"
> .Async job b3dc2bab-1b03-45d1-ba02-0c861f5043df failed
> Error 530, We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting
> {
>   "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>   "cmd": "org.apache.cloudstack.api.command.user.vm.DestroyVMCmd",
>   "created": "2014-07-28T10:39:21+0530",
>   "jobid": "b3dc2bab-1b03-45d1-ba02-0c861f5043df",
>   "jobprocstatus": 0,
>   "jobresult": {
>     "errorcode": 530,
>     "errortext": "We cannot stop VM[User|ubuntu-12-04-64-bit] when it is in state Starting"
>   },
>   "jobresultcode": 530,
>   "jobresulttype": "object",
>   "jobstatus": 2,
>   "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> }
> 
> mycloudmonkeytpj> destroyVirtualMachine id="b74e6737-ec83-4483-b1a2-ab60045d941b"
> .Async job 81e7e2c4-efc9-4673-90f7-8a69287cfa80 failed
> Error 530, We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting
> {
>   "accountid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
>   "cmd": "org.apache.cloudstack.api.command.user.vm.DestroyVMCmd",
>   "created": "2014-07-28T10:39:31+0530",
>   "jobid": "81e7e2c4-efc9-4673-90f7-8a69287cfa80",
>   "jobprocstatus": 0,
>   "jobresult": {
>     "errorcode": 530,
>     "errortext": "We cannot stop VM[User|CentOS-6-5-x86-64] when it is in state Starting"
>   },
>   "jobresultcode": 530,
>   "jobresulttype": "object",
>   "jobstatus": 2,
>   "userid": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> }