You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Andrew Kirch <tr...@trelane.net> on 2015/12/30 17:45:16 UTC
System VM's down
I logged into cloudstack 4.5.2 this morning to find both system vm's
down. It appears the system rebooted overnight uncleanly. This is a
single-host standalone cloudstack. In "System Capacity" the
Management IP % shows:
/ 10
The secondary VM is continuously trying to restart, and the console vm
will not start.
MySQL claims the databases are clean.
According to the database, none of the management IP's are in use.
(nic_id, and reservation_id are null).
> select * FROM op_dc_ip_address_alloc;
+----+----------------+----------------+--------+--------+----------------+-------+-------------+
| id | ip_address | data_center_id | pod_id | nic_id |
reservation_id | taken | mac_address |
+----+----------------+----------------+--------+--------+----------------+-------+-------------+
| 1 | 192.168.100.30 | 1 | 1 | NULL | NULL
| NULL | 1 |
| 2 | 192.168.100.31 | 1 | 1 | NULL | NULL
| NULL | 2 |
| 3 | 192.168.100.32 | 1 | 1 | NULL | NULL
| NULL | 3 |
| 4 | 192.168.100.33 | 1 | 1 | NULL | NULL
| NULL | 4 |
| 5 | 192.168.100.34 | 1 | 1 | NULL | NULL
| NULL | 5 |
| 6 | 192.168.100.35 | 1 | 1 | NULL | NULL
| NULL | 6 |
| 7 | 192.168.100.36 | 1 | 1 | NULL | NULL
| NULL | 7 |
| 8 | 192.168.100.37 | 1 | 1 | NULL | NULL
| NULL | 8 |
| 9 | 192.168.100.38 | 1 | 1 | NULL | NULL
| NULL | 9 |
| 10 | 192.168.100.39 | 1 | 1 | NULL | NULL
| NULL | 10 |
+----+----------------+----------------+--------+--------+----------------+-------+-------------+
10 rows in set (0.00 sec)
Any thoughts on how I can get the management vm's restarted?
RE: System VM's down
Posted by Paul Angus <pa...@shapeblue.com>.
Yes, they're hard to read; The error is here (line 188)
result":false,"details":"internal error: process exited while connecting to monitor: qemu-system-x86_64: -drive file=/mnt/86a6ae68-8ee2-3f7c-bfc6-85d4b7060f58/f45d6699-3cf9-44bf-9565-3bf75ad09405,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: could not open disk image /mnt/86a6ae68-8ee2-3f7c-bfc6-85d4b7060f58/f45d6699-3cf9-44bf-9565-3bf75ad09405: Could not open '/mnt/86a6ae68-8ee2-3f7c-bfc6-85d4b7060f58/f45d6699-3cf9-44bf-9565-3bf75ad09405': Is a directory\n","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}}] }
I'm not sure *exactly* what's going wrong from this, but there seems to be an issue with your NFS mount on the KVM host.
It's worth checking permissions on the NFS server.
[ShapeBlue]<http://www.shapeblue.com>
Paul Angus
VP Technology , ShapeBlue
d: +44 203 617 0528 | s: +44 203 603 0540<tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540> | m: +44 7711 418784<tel:+44%207711%20418784>
e: paul.angus@shapeblue.com | t: @cloudyangus<ma...@cloudyangus> | w: www.shapeblue.com<http://www.shapeblue.com>
a: 53 Chandos Place, Covent Garden London WC2N 4HS UK
[cid:imagefbc895.png@ff4a0bb0.468c5bb3]
Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.
-----Original Message-----
From: Andrew Kirch [mailto:trelane@trelane.net]
Sent: Wednesday, December 30, 2015 6:16 PM
To: users@cloudstack.apache.org
Subject: Re: System VM's down
I've never found that there was much sanity in logs, so I've left them as is. I doubt that it will shock people that I'm using RFC-1918 space. :)
*http://pastebin.com/YUXbwgkw <http://pastebin.com/YUXbwgkw>*
On Wed, Dec 30, 2015 at 12:58 PM, Paul Angus <pa...@shapeblue.com>
wrote:
> Hi Andrew,
>
> Can you sanitise your mgmt. log and post them in pastebin so we can
> take a look.
>
>
> [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP Technology
> , ShapeBlue d: *+44 203 617 0528 | s: +44 203 603 0540*
> <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> | m:
> *+44 7711 418784* <+44%207711%20418784> e:
> *paul.angus@shapeblue.com | t: @cloudyangus*
> <pa...@cloudyangus> | w:
> *www.shapeblue.com* <http://www.shapeblue.com> a:
> 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue Ltd is a
> company incorporated in England & Wales. ShapeBlue Services India LLP
> is a company incorporated in India and is operated under license from
> Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company
> incorporated in Brasil and is operated under license from Shape Blue
> Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
> South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
> This email and any attachments to it may be confidential and are
> intended solely for the use of the individual to whom it is addressed.
> Any views or opinions expressed are solely those of the author and do
> not necessarily represent those of Shape Blue Ltd or related
> companies. If you are not the intended recipient of this email, you
> must neither take any action based upon its contents, nor copy or show
> it to anyone. Please contact the sender if you believe you have received this email in error.
>
>
> -----Original Message-----
> From: Andrew Kirch [mailto:trelane@trelane.net]
> Sent: Wednesday, December 30, 2015 4:45 PM
> To: users@cloudstack.apache.org
> Subject: System VM's down
>
> I logged into cloudstack 4.5.2 this morning to find both system vm's down.
> It appears the system rebooted overnight uncleanly. This is a
> single-host standalone cloudstack. In "System Capacity" the Management IP % shows:
> / 10
> The secondary VM is continuously trying to restart, and the console vm
> will not start.
>
> MySQL claims the databases are clean.
>
> According to the database, none of the management IP's are in use.
> (nic_id, and reservation_id are null).
> > select * FROM op_dc_ip_address_alloc;
>
> +----+----------------+----------------+--------+--------+----------------+-------+-------------+
> | id | ip_address | data_center_id | pod_id | nic_id |
> reservation_id | taken | mac_address |
>
> +----+----------------+----------------+--------+--------+----------------+-------+-------------+
> | 1 | 192.168.100.30 | 1 | 1 | NULL | NULL NULL | 1 |
> | 2 | 192.168.100.31 | 1 | 1 | NULL | NULL NULL | 2 |
> | 3 | 192.168.100.32 | 1 | 1 | NULL | NULL NULL | 3 |
> | 4 | 192.168.100.33 | 1 | 1 | NULL | NULL NULL | 4 |
> | 5 | 192.168.100.34 | 1 | 1 | NULL | NULL NULL | 5 |
> | 6 | 192.168.100.35 | 1 | 1 | NULL | NULL NULL | 6 |
> | 7 | 192.168.100.36 | 1 | 1 | NULL | NULL NULL | 7 |
> | 8 | 192.168.100.37 | 1 | 1 | NULL | NULL NULL | 8 |
> | 9 | 192.168.100.38 | 1 | 1 | NULL | NULL NULL | 9 |
> | 10 | 192.168.100.39 | 1 | 1 | NULL | NULL NULL | 10 |
>
> +----+----------------+----------------+--------+--------+----------------+-------+-------------+
> 10 rows in set (0.00 sec)
>
> Any thoughts on how I can get the management vm's restarted?
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build
> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> IaaS deployment framework <http://shapeblue.com/csforge/> CloudStack
> Consulting <http://shapeblue.com/cloudstack-consultancy/> | CloudStack
> Software Engineering
> <http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support
> <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
>
Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
Re: System VM's down
Posted by Andrew Kirch <tr...@trelane.net>.
I've never found that there was much sanity in logs, so I've left them as
is. I doubt that it will shock people that I'm using RFC-1918 space. :)
*http://pastebin.com/YUXbwgkw <http://pastebin.com/YUXbwgkw>*
On Wed, Dec 30, 2015 at 12:58 PM, Paul Angus <pa...@shapeblue.com>
wrote:
> Hi Andrew,
>
> Can you sanitise your mgmt. log and post them in pastebin so we can take a
> look.
>
>
> [image: ShapeBlue] <http://www.shapeblue.com> Paul Angus VP Technology ,
> ShapeBlue d: *+44 203 617 0528 | s: +44 203 603 0540*
> <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> | m:
> *+44 7711 418784* <+44%207711%20418784> e:
> *paul.angus@shapeblue.com | t: @cloudyangus*
> <pa...@cloudyangus> | w:
> *www.shapeblue.com* <http://www.shapeblue.com> a:
> 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue Ltd is a
> company incorporated in England & Wales. ShapeBlue Services India LLP is a
> company incorporated in India and is operated under license from Shape Blue
> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil
> and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is
> a company registered by The Republic of South Africa and is traded under
> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based
> upon its contents, nor copy or show it to anyone. Please contact the sender
> if you believe you have received this email in error.
>
>
> -----Original Message-----
> From: Andrew Kirch [mailto:trelane@trelane.net]
> Sent: Wednesday, December 30, 2015 4:45 PM
> To: users@cloudstack.apache.org
> Subject: System VM's down
>
> I logged into cloudstack 4.5.2 this morning to find both system vm's down.
> It appears the system rebooted overnight uncleanly. This is a single-host
> standalone cloudstack. In "System Capacity" the Management IP % shows:
> / 10
> The secondary VM is continuously trying to restart, and the console vm
> will not start.
>
> MySQL claims the databases are clean.
>
> According to the database, none of the management IP's are in use.
> (nic_id, and reservation_id are null).
> > select * FROM op_dc_ip_address_alloc;
>
> +----+----------------+----------------+--------+--------+----------------+-------+-------------+
> | id | ip_address | data_center_id | pod_id | nic_id |
> reservation_id | taken | mac_address |
>
> +----+----------------+----------------+--------+--------+----------------+-------+-------------+
> | 1 | 192.168.100.30 | 1 | 1 | NULL | NULL
> | NULL | 1 |
> | 2 | 192.168.100.31 | 1 | 1 | NULL | NULL
> | NULL | 2 |
> | 3 | 192.168.100.32 | 1 | 1 | NULL | NULL
> | NULL | 3 |
> | 4 | 192.168.100.33 | 1 | 1 | NULL | NULL
> | NULL | 4 |
> | 5 | 192.168.100.34 | 1 | 1 | NULL | NULL
> | NULL | 5 |
> | 6 | 192.168.100.35 | 1 | 1 | NULL | NULL
> | NULL | 6 |
> | 7 | 192.168.100.36 | 1 | 1 | NULL | NULL
> | NULL | 7 |
> | 8 | 192.168.100.37 | 1 | 1 | NULL | NULL
> | NULL | 8 |
> | 9 | 192.168.100.38 | 1 | 1 | NULL | NULL
> | NULL | 9 |
> | 10 | 192.168.100.39 | 1 | 1 | NULL | NULL
> | NULL | 10 |
>
> +----+----------------+----------------+--------+--------+----------------+-------+-------------+
> 10 rows in set (0.00 sec)
>
> Any thoughts on how I can get the management vm's restarted?
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build
> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> IaaS deployment framework <http://shapeblue.com/csforge/>
> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> | CloudStack
> Software Engineering
> <http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support
> <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
>
RE: System VM's down
Posted by Paul Angus <pa...@shapeblue.com>.
Hi Andrew,
Can you sanitise your mgmt. log and post them in pastebin so we can take a look.
[ShapeBlue]<http://www.shapeblue.com>
Paul Angus
VP Technology , ShapeBlue
d: +44 203 617 0528 | s: +44 203 603 0540<tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540> | m: +44 7711 418784<tel:+44%207711%20418784>
e: paul.angus@shapeblue.com | t: @cloudyangus<ma...@cloudyangus> | w: www.shapeblue.com<http://www.shapeblue.com>
a: 53 Chandos Place, Covent Garden London WC2N 4HS UK
[cid:imagec7e870.png@0ed20489.43a895d3]
Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.
-----Original Message-----
From: Andrew Kirch [mailto:trelane@trelane.net]
Sent: Wednesday, December 30, 2015 4:45 PM
To: users@cloudstack.apache.org
Subject: System VM's down
I logged into cloudstack 4.5.2 this morning to find both system vm's down. It appears the system rebooted overnight uncleanly. This is a single-host standalone cloudstack. In "System Capacity" the Management IP % shows:
/ 10
The secondary VM is continuously trying to restart, and the console vm will not start.
MySQL claims the databases are clean.
According to the database, none of the management IP's are in use.
(nic_id, and reservation_id are null).
> select * FROM op_dc_ip_address_alloc;
+----+----------------+----------------+--------+--------+----------------+-------+-------------+
| id | ip_address | data_center_id | pod_id | nic_id |
reservation_id | taken | mac_address |
+----+----------------+----------------+--------+--------+----------------+-------+-------------+
| 1 | 192.168.100.30 | 1 | 1 | NULL | NULL
| NULL | 1 |
| 2 | 192.168.100.31 | 1 | 1 | NULL | NULL
| NULL | 2 |
| 3 | 192.168.100.32 | 1 | 1 | NULL | NULL
| NULL | 3 |
| 4 | 192.168.100.33 | 1 | 1 | NULL | NULL
| NULL | 4 |
| 5 | 192.168.100.34 | 1 | 1 | NULL | NULL
| NULL | 5 |
| 6 | 192.168.100.35 | 1 | 1 | NULL | NULL
| NULL | 6 |
| 7 | 192.168.100.36 | 1 | 1 | NULL | NULL
| NULL | 7 |
| 8 | 192.168.100.37 | 1 | 1 | NULL | NULL
| NULL | 8 |
| 9 | 192.168.100.38 | 1 | 1 | NULL | NULL
| NULL | 9 |
| 10 | 192.168.100.39 | 1 | 1 | NULL | NULL
| NULL | 10 |
+----+----------------+----------------+--------+--------+----------------+-------+-------------+
10 rows in set (0.00 sec)
Any thoughts on how I can get the management vm's restarted?
Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>