You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Valery Ciareszka <va...@gmail.com> on 2013/04/16 16:53:53 UTC
system VMs failure
Hi all,
I have the following issue:
After emergency reboot of hypervisor node (i.e. reset via ipmi), system vms
got stuck in "starting state":
http://thesuki.org/temp/ss/SS-20130416154808.png
When I look onto console through vnc I see either prompt for fsck or errors
related to r/o mounted root fs:
http://thesuki.org/temp/ss/SS-20130416155009.png
There is no way to destroy vm either from web interface or api
(cloudmonkey):
cloudmonkey> destroy systemvm id=eb3adb37-96d1-4785-884d-5958526b75fc
Async query failed for jobid 3c2fd0d7-a0a0-427a-ae3e-dda96b7cf9e0
Error 530 We cannot stop VM[ConsoleProxy|v-1058-VM] when it is in state
Starting
cloudmonkey>
I have to manually update state in mysql before I am able to destroy this
vm:
mysql> update vm_instance set state='Stopped' where name='v-1058-VM';
\Query OK, 1 row affected (0.08 sec)
Rows matched: 1 Changed: 1 Warnings: 0
cloudmonkey> destroy systemvm id=eb3adb37-96d1-4785-884d-5958526b75fc
.
....
...............
................
..............................................................
accountid =
b080d9c2-c3b2-4ffd-8a1c-04e99d03f5ef.......................................................
cmd = com.cloud.api.commands.DestroySystemVmCmd
created = 2013-04-16T14:54:47+0200
jobid = 23942741-a26c-4dba-b65b-1a4536aab1d7
jobprocstatus = 0
jobresult:
=========
systemvm:
========
id = eb3adb37-96d1-4785-884d-5958526b75fc
created = 2013-04-16T13:02:03+0200
dns1 = 8.8.8.8
hostid = b069e873-675b-435d-a7f1-a7edf87ce9b1
hostname = ad112.colobridge.net
name = v-1058-VM
objectId = 1058
podid = f04bddae-fd73-47ac-b40e-32a4d2a8b7d5
state = Stopped
systemvmtype = consoleproxy
templateid = e54cbb23-bc5b-48d1-925b-1225ef938ea4
zoneid = 6ff79c9f-eccc-466f-9d43-3ce941855002
zonename = Zone1
jobresultcode = 0
jobresulttype = object
jobstatus = 1
userid = 79c285d1-54df-4169-84d1-72ff99f52998
First question is: why cloudstack does not handle such vm start errors ?
Second question is: how to force deletion of such VMs without mysql hack ?
environment:
CS 4.0.1, Centos 6.4 (management+node1+node2), OpenIndiana NFS server as
primary and secondary storage
--
Regards,
Valery
http://protocol.by/slayer
Re: system VMs failure
Posted by Ahmad Emneina <ae...@gmail.com>.
A bug should be filed for this, if there isnt one already. Maybe a force
option for the destroySystemVM API call. Something that will clobber the
vm's regardless of state.
On Fri, Apr 19, 2013 at 12:23 PM, Jeronimo Garcia <ga...@gmail.com>wrote:
> Any udpates on this ? I've just experienced the exact same thing.
>
> Thanks!
>
>
> On Tue, Apr 16, 2013 at 9:53 AM, Valery Ciareszka <
> valery.tereshko@gmail.com
> > wrote:
>
> > Hi all,
> >
> > I have the following issue:
> > After emergency reboot of hypervisor node (i.e. reset via ipmi), system
> vms
> > got stuck in "starting state":
> > http://thesuki.org/temp/ss/SS-20130416154808.png
> >
> > When I look onto console through vnc I see either prompt for fsck or
> errors
> > related to r/o mounted root fs:
> > http://thesuki.org/temp/ss/SS-20130416155009.png
> >
> > There is no way to destroy vm either from web interface or api
> > (cloudmonkey):
> >
> > cloudmonkey> destroy systemvm id=eb3adb37-96d1-4785-884d-5958526b75fc
> > Async query failed for jobid 3c2fd0d7-a0a0-427a-ae3e-dda96b7cf9e0
> > Error 530 We cannot stop VM[ConsoleProxy|v-1058-VM] when it is in state
> > Starting
> > cloudmonkey>
> >
> > I have to manually update state in mysql before I am able to destroy this
> > vm:
> >
> > mysql> update vm_instance set state='Stopped' where name='v-1058-VM';
> > \Query OK, 1 row affected (0.08 sec)
> > Rows matched: 1 Changed: 1 Warnings: 0
> >
> > cloudmonkey> destroy systemvm id=eb3adb37-96d1-4785-884d-5958526b75fc
> > .
> >
> >
> >
> > ....
> >
> >
> > ...............
> >
> >
> >
> > ................
> >
> > ..............................................................
> >
> >
> >
> >
> >
> > accountid =
> >
> >
> b080d9c2-c3b2-4ffd-8a1c-04e99d03f5ef.......................................................
> > cmd = com.cloud.api.commands.DestroySystemVmCmd
> > created = 2013-04-16T14:54:47+0200
> > jobid = 23942741-a26c-4dba-b65b-1a4536aab1d7
> > jobprocstatus = 0
> > jobresult:
> > =========
> > systemvm:
> > ========
> > id = eb3adb37-96d1-4785-884d-5958526b75fc
> > created = 2013-04-16T13:02:03+0200
> > dns1 = 8.8.8.8
> > hostid = b069e873-675b-435d-a7f1-a7edf87ce9b1
> > hostname = ad112.colobridge.net
> > name = v-1058-VM
> > objectId = 1058
> > podid = f04bddae-fd73-47ac-b40e-32a4d2a8b7d5
> > state = Stopped
> > systemvmtype = consoleproxy
> > templateid = e54cbb23-bc5b-48d1-925b-1225ef938ea4
> > zoneid = 6ff79c9f-eccc-466f-9d43-3ce941855002
> > zonename = Zone1
> > jobresultcode = 0
> > jobresulttype = object
> > jobstatus = 1
> > userid = 79c285d1-54df-4169-84d1-72ff99f52998
> >
> >
> > First question is: why cloudstack does not handle such vm start errors ?
> >
> > Second question is: how to force deletion of such VMs without mysql hack
> ?
> >
> > environment:
> > CS 4.0.1, Centos 6.4 (management+node1+node2), OpenIndiana NFS server as
> > primary and secondary storage
> > --
> > Regards,
> > Valery
> >
> > http://protocol.by/slayer
> >
>
Re: system VMs failure
Posted by Jeronimo Garcia <ga...@gmail.com>.
Any udpates on this ? I've just experienced the exact same thing.
Thanks!
On Tue, Apr 16, 2013 at 9:53 AM, Valery Ciareszka <valery.tereshko@gmail.com
> wrote:
> Hi all,
>
> I have the following issue:
> After emergency reboot of hypervisor node (i.e. reset via ipmi), system vms
> got stuck in "starting state":
> http://thesuki.org/temp/ss/SS-20130416154808.png
>
> When I look onto console through vnc I see either prompt for fsck or errors
> related to r/o mounted root fs:
> http://thesuki.org/temp/ss/SS-20130416155009.png
>
> There is no way to destroy vm either from web interface or api
> (cloudmonkey):
>
> cloudmonkey> destroy systemvm id=eb3adb37-96d1-4785-884d-5958526b75fc
> Async query failed for jobid 3c2fd0d7-a0a0-427a-ae3e-dda96b7cf9e0
> Error 530 We cannot stop VM[ConsoleProxy|v-1058-VM] when it is in state
> Starting
> cloudmonkey>
>
> I have to manually update state in mysql before I am able to destroy this
> vm:
>
> mysql> update vm_instance set state='Stopped' where name='v-1058-VM';
> \Query OK, 1 row affected (0.08 sec)
> Rows matched: 1 Changed: 1 Warnings: 0
>
> cloudmonkey> destroy systemvm id=eb3adb37-96d1-4785-884d-5958526b75fc
> .
>
>
>
> ....
>
>
> ...............
>
>
>
> ................
>
> ..............................................................
>
>
>
>
>
> accountid =
>
> b080d9c2-c3b2-4ffd-8a1c-04e99d03f5ef.......................................................
> cmd = com.cloud.api.commands.DestroySystemVmCmd
> created = 2013-04-16T14:54:47+0200
> jobid = 23942741-a26c-4dba-b65b-1a4536aab1d7
> jobprocstatus = 0
> jobresult:
> =========
> systemvm:
> ========
> id = eb3adb37-96d1-4785-884d-5958526b75fc
> created = 2013-04-16T13:02:03+0200
> dns1 = 8.8.8.8
> hostid = b069e873-675b-435d-a7f1-a7edf87ce9b1
> hostname = ad112.colobridge.net
> name = v-1058-VM
> objectId = 1058
> podid = f04bddae-fd73-47ac-b40e-32a4d2a8b7d5
> state = Stopped
> systemvmtype = consoleproxy
> templateid = e54cbb23-bc5b-48d1-925b-1225ef938ea4
> zoneid = 6ff79c9f-eccc-466f-9d43-3ce941855002
> zonename = Zone1
> jobresultcode = 0
> jobresulttype = object
> jobstatus = 1
> userid = 79c285d1-54df-4169-84d1-72ff99f52998
>
>
> First question is: why cloudstack does not handle such vm start errors ?
>
> Second question is: how to force deletion of such VMs without mysql hack ?
>
> environment:
> CS 4.0.1, Centos 6.4 (management+node1+node2), OpenIndiana NFS server as
> primary and secondary storage
> --
> Regards,
> Valery
>
> http://protocol.by/slayer
>