You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Matthew Midgett <cl...@trick-solutions.com.INVALID> on 2014/12/03 12:38:42 UTC
RE: Fresh 4.4.2 - Unable to get status on .... my XenServer
Thanks Geoff I really appreciate the response. I guess that every other time
over the last year I've reinstalled XenServer and didn't remember it. Maybe
we should add this to the Docs as I don't ever remember seeing it, I could
be wrong though. I do have the patch management automated and it checks in
on every reboot. I will have to use Ilo to reinstall the OS today which is
not that big of a deal. I do admit that I need to PXE boot the servers I
just hadn't had the time. Yesterday I spent working out a builder for CS and
a network mirror. Then I noticed about 3 hours later that you guys posted a
public repo for 4.4.2. :)
CONFIDENTIAL: This electronic mail, including any attachment(s) may contain
information that is privileged, confidential, and/or otherwise protected
from disclosure to anyone other than its intended recipient(s). Any
dissemination or use of this electronic email or its contents (including any
attachments) by persons other than the intended recipient(s) is strictly
prohibited. If you have received this message in error, please notify me
immediately by reply email so that I may correct my internal records.
Please then delete the original message (including any attachments) in its
entirety. Thank you.
-----Original Message-----
From: Matthew Midgett [mailto:cloudstck@trick-solutions.com.INVALID]
Sent: Wednesday, December 03, 2014 1:11 AM
To: users@cloudstack.apache.org
Subject: Fresh 4.4.2 - Unable to get status on .... my xenservers
First I am not sure what is causing this, I have been testing 4.3 and 4.4
multiple times and didn't notice this problem till middle of last month when
I processed all xenserver updates and then tried to go to 4.4.1. I was
getting the same error that I am unable to get status of my xenservers. I
wiped out everything but the xenservers, those I just removed from the pool
and let them reboot and added them to a new pool. In ACS it says its trying
to start the SSVM and the Console VM but they stay in the starting state.
I'm sure it is because of this error. The agent show up in ACS so I don't
know why it can't talk to the xenservers. I can ping them from the
management servers.
Last login: Wed Dec 3 00:57:04 2014 from 192.168.255.6
[root@CSTW1 ~]# grep -i -E 'exception|unable|fail|invalid|leak|warn|error'
/var/log/cloudstack/management/management-server.log
2014-12-03 00:00:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-5:ctx-a0a23e42) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:00:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-21:ctx-f78a59e4) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:01:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-11:ctx-2e791a18) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:01:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-22:ctx-7bca20e8) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:02:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-12:ctx-28d48099) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:02:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-24:ctx-bc593a63) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:03:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-1:ctx-615b171d) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:03:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-25:ctx-07f6ba50) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:04:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-13:ctx-c2287fb5) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:04:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-26:ctx-f7d92459) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:05:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-14:ctx-a4e62e4e) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:05:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-28:ctx-01bb91e3) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:06:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-2:ctx-30f576ea) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:06:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-29:ctx-4d6690de) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:07:27,540 ERROR [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-2:ctx-645639e7) Caught this throwable,
com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
Caused by: com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
com.mysql.jdbc.JDBC4PreparedStatement@32602693: UPDATE op_ha_work SET
op_ha_work.taken='2014-12-03 05:07:27',
op_ha_work.mgmt_server_id=223162458780723 WHERE op_ha_work.id = 5
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock
found when trying to get lock; try restarting transaction
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1065)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
2014-12-03 00:07:27,595 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-3:ctx-c7c94958 work-5) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 00:07:27,645 WARN [c.c.u.d.Merovingian2]
(HA-Worker-3:ctx-c7c94958 work-5) Was unable to find lock for the
key vm_instance22 and thread id 818653380
2014-12-03 00:07:29,425 ERROR [c.c.v.VirtualMachineManagerImpl]
(Vm-Operations-Cleanup-1:ctx-35d886d3) VM Operations failed due to
com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
com.mysql.jdbc.JDBC4PreparedStatement@61807bba: DELETE FROM async_job WHERE
async_job.id= 401
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
Cannot delete or update a parent row: a foreign key constraint fails
(`cloud`.`async_job_join_map`, CONSTRAINT
`fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES
`async_job` (`id`))
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
2014-12-03 00:07:30,702 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-3:ctx-c7c94958 work-5) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
2014-12-03 00:07:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-15:ctx-c7dd62d6) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:07:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-30:ctx-99d14f4c) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:08:27,621 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-2:ctx-94cc502e work-2) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 00:08:27,678 WARN [c.c.u.d.Merovingian2]
(HA-Worker-2:ctx-94cc502e work-2) Was unable to find lock for the
key vm_instance15 and thread id 52609881
2014-12-03 00:08:29,421 ERROR [c.c.v.VirtualMachineManagerImpl]
(Vm-Operations-Cleanup-1:ctx-ed5f2ff6) VM Operations failed due to
com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
com.mysql.jdbc.PreparedStatement@15fe9d6d: DELETE FROM async_job WHERE
async_job.id= 406
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
Cannot delete or update a parent row: a foreign key constraint fails
(`cloud`.`async_job_join_map`, CONSTRAINT
`fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES
`async_job` (`id`))
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
2014-12-03 00:08:30,736 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-2:ctx-94cc502e work-2) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:08:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-16:ctx-acc87f49) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:08:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-32:ctx-efd0036e) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:09:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-3:ctx-1d3eecd0) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:09:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-33:ctx-3271eeea) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:10:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-17:ctx-27637ed9) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:10:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-34:ctx-b1f98667) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:11:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-18:ctx-45760fd3) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:11:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-36:ctx-3367d63d) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:12:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-4:ctx-ee2683aa) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:12:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-37:ctx-d0b02603) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:13:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-19:ctx-ce10b0fa) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:13:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-38:ctx-e7e4ae6b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:14:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-20:ctx-c158dd2c) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:14:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-40:ctx-7a662bea) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:15:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-5:ctx-0a8f1c90) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:15:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-41:ctx-4c163515) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:16:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-21:ctx-51c2dfd8) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:16:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-42:ctx-c670489b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:17:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-22:ctx-9df53feb) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:17:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-44:ctx-a921b13b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:18:29,546 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-1:ctx-4fdb5161 work-5) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 00:18:29,582 WARN [c.c.u.d.Merovingian2]
(HA-Worker-1:ctx-4fdb5161 work-5) Was unable to find lock for the
key vm_instance22 and thread id 2132920636
2014-12-03 00:18:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-6:ctx-a54d1732) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:18:32,638 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-1:ctx-4fdb5161 work-5) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
2014-12-03 00:18:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-45:ctx-5d1c0c7e) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:19:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Unable to
transition the state but we're moving on because it's forced stop
2014-12-03 00:19:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Unable to
cleanup VM: VM[ConsoleProxy|v-15-VM] ,since outstanding work item is not
found
2014-12-03 00:19:30,871 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Invocation
exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
item not found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state
Starting
2014-12-03 00:19:30,872 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Rethrow
exception com.cloud.utils.exception.CloudRuntimeException: Work item not
found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:19:30,872 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-3:ctx-d59db286 job-408/job-411) Unable to
complete AsyncJobVO {id:411, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
AAAAD3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
created: Wed Dec 03 00:19:02 EST 2014}, job origin:408
com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:19:30,874 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-3:ctx-d59db286 job-408/job-411) Complete async job-411,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAdUpvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
cCBWTVtDb25zb2xlUHJveHl8di0xNS1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFydGluZ3Vy
AB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZh
LmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJp
bmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNv
bS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZh
dAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9i
cy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQA
DHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUu
amF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250
ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQu
amF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29u
dGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EA
fgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hl
LmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEA
fgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3Vy
cmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4A
CwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh
cQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y
dAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51
dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsA
AALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNv
bGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZh
LnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QA
FkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0D
AAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
2014-12-03 00:19:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-23:ctx-0414dcdd) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:19:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-46:ctx-fc02b226) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:20:32,282 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-24:ctx-821b5aca) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:20:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-48:ctx-70dbfde8) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:21:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-1:ctx-de7b1636) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:21:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-49:ctx-154bac45) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:22:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-25:ctx-5c37a417) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:22:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-50:ctx-a34d4acf) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:23:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-52:ctx-4ee4cdd0) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:23:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-7:ctx-de025548) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:24:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-53:ctx-198b066a) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:24:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-27:ctx-e4b78a1b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:25:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-14:ctx-d0466cfa) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:25:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-55:ctx-6c330223) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:26:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-2:ctx-98d6b222) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:26:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-57:ctx-031552f0) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:27:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-29:ctx-a436d05f) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:27:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-58:ctx-0f3496a2) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:28:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-15:ctx-40cf6a78) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:28:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-59:ctx-1537f233) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:29:30,874 ERROR [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-0:ctx-84af6112) Caught this throwable,
com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to commit
or close the connection.
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock
found when trying to get lock; try restarting transaction
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1065)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
2014-12-03 00:29:30,881 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Unable to
transition the state but we're moving on because it's forced stop
2014-12-03 00:29:30,882 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Unable to
cleanup VM: VM[SecondaryStorageVm|s-22-VM] ,since outstanding work item is
not found
2014-12-03 00:29:30,882 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Invocation
exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
item not found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in
state Starting
2014-12-03 00:29:30,883 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Rethrow
exception com.cloud.utils.exception.CloudRuntimeException: Work item not
found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state
Starting
2014-12-03 00:29:30,883 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-4:ctx-b25792e9 job-414/job-417) Unable to
complete AsyncJobVO {id:417, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
AAAAFnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
created: Wed Dec 03 00:29:02 EST 2014}, job origin:414
com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
2014-12-03 00:29:30,896 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-4:ctx-b25792e9 job-414/job-417) Complete async job-417,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAe0pvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
cCBWTVtTZWNvbmRhcnlTdG9yYWdlVm18cy0yMi1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFy
dGluZ3VyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNy
ABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5k
ZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAA
cnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hl
ci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdv
cmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwu
amF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFu
YWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVu
bmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdl
ZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENv
bnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFn
ZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250
ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcu
YXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVx
AH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwu
Y29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4A
G3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFz
ay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4
ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAu
amF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdz
cQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51
dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhy
ACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgAB
TAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHS
HZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
2014-12-03 00:29:30,919 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-4:ctx-10fea75f work-2) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 00:29:30,967 WARN [c.c.u.d.Merovingian2]
(HA-Worker-4:ctx-10fea75f work-2) Was unable to find lock for the
key vm_instance15 and thread id 139852089
2014-12-03 00:29:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-8:ctx-bc77e0da) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:29:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-61:ctx-de44c2a9) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:29:34,025 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-4:ctx-10fea75f work-2) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:30:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-31:ctx-8417fc60) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:30:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-62:ctx-d118067c) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:31:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-16:ctx-393a2cde) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:31:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-63:ctx-fd5d5e54) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:32:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-3:ctx-07ab5550) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:32:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-65:ctx-0345383e) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:33:32,286 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-33:ctx-60af4723) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:33:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-66:ctx-0b95563b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:34:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-17:ctx-8e9c36a2) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:34:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-67:ctx-397176fb) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:35:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-9:ctx-3c967d9b) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:35:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-69:ctx-8bc1cfaf) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:36:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-35:ctx-8186e45f) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:36:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-70:ctx-557403b1) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:37:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-18:ctx-8539dc77) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:37:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-71:ctx-61ea3221) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:38:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-4:ctx-0dc9a587) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:38:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-73:ctx-7fba328c) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:39:31,006 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-0:ctx-7a356c50 work-5) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 00:39:31,051 WARN [c.c.u.d.Merovingian2]
(HA-Worker-0:ctx-7a356c50 work-5) Was unable to find lock for the
key vm_instance22 and thread id 84459079
2014-12-03 00:39:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-37:ctx-bc7a8f81) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:39:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-74:ctx-ebe4657b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:39:34,105 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-0:ctx-7a356c50 work-5) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
2014-12-03 00:40:30,865 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Unable to
transition the state but we're moving on because it's forced stop
2014-12-03 00:40:30,866 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Unable to
cleanup VM: VM[ConsoleProxy|v-15-VM] ,since outstanding work item is not
found
2014-12-03 00:40:30,866 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Invocation
exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
item not found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state
Starting
2014-12-03 00:40:30,866 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Rethrow
exception com.cloud.utils.exception.CloudRuntimeException: Work item not
found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:40:30,866 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-5:ctx-32edffa5 job-423/job-426) Unable to
complete AsyncJobVO {id:426, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
AAAAD3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
created: Wed Dec 03 00:40:02 EST 2014}, job origin:423
com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:40:30,868 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-5:ctx-32edffa5 job-423/job-426) Complete async job-426,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAdUpvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
cCBWTVtDb25zb2xlUHJveHl8di0xNS1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFydGluZ3Vy
AB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZh
LmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJp
bmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNv
bS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZh
dAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9i
cy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQA
DHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUu
amF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250
ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQu
amF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29u
dGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EA
fgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hl
LmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEA
fgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3Vy
cmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4A
CwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh
cQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y
dAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51
dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsA
AALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNv
bGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZh
LnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QA
FkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0D
AAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
2014-12-03 00:40:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-19:ctx-338898ef) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:40:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-75:ctx-790ca221) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:41:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-10:ctx-0e2f52eb) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:41:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-77:ctx-f8287f50) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:42:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-39:ctx-1af09356) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:42:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-78:ctx-e1b2675f) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:43:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-20:ctx-c9aea423) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:43:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-79:ctx-c459450e) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:44:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-5:ctx-f7fee74e) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:44:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-81:ctx-bc615860) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:45:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-41:ctx-4f287927) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:45:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-82:ctx-092eb488) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:46:27,281 WARN [o.a.c.alerts] (HA-1:ctx-b34246c5)
alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null //
message:: No usage server process running
2014-12-03 00:46:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-21:ctx-62eca8f2) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:46:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-83:ctx-272e6406) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:47:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-11:ctx-23f28317) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:47:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-85:ctx-905ed14b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:48:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-43:ctx-4a673855) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:48:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-86:ctx-1285bc0f) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:49:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-22:ctx-60a35d36) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:49:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-87:ctx-687282ce) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:50:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Unable to
transition the state but we're moving on because it's forced stop
2014-12-03 00:50:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Unable to
cleanup VM: VM[SecondaryStorageVm|s-22-VM] ,since outstanding work item is
not found
2014-12-03 00:50:30,871 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Invocation
exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
item not found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in
state Starting
2014-12-03 00:50:30,871 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Rethrow
exception com.cloud.utils.exception.CloudRuntimeException: Work item not
found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state
Starting
2014-12-03 00:50:30,871 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-6:ctx-46fe9139 job-429/job-432) Unable to
complete AsyncJobVO {id:432, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
AAAAFnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
created: Wed Dec 03 00:50:02 EST 2014}, job origin:429
com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
2014-12-03 00:50:30,873 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-6:ctx-46fe9139 job-429/job-432) Complete async job-432,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAe0pvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
cCBWTVtTZWNvbmRhcnlTdG9yYWdlVm18cy0yMi1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFy
dGluZ3VyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNy
ABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5k
ZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAA
cnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hl
ci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdv
cmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwu
amF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFu
YWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVu
bmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdl
ZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENv
bnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFn
ZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250
ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcu
YXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVx
AH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwu
Y29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4A
G3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFz
ay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4
ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAu
amF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdz
cQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51
dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhy
ACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgAB
TAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHS
HZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
2014-12-03 00:50:31,095 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-3:ctx-0dbe91b2 work-2) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 00:50:31,129 WARN [c.c.u.d.Merovingian2]
(HA-Worker-3:ctx-0dbe91b2 work-2) Was unable to find lock for the
key vm_instance15 and thread id 818653380
2014-12-03 00:50:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-6:ctx-2758188b) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:50:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-89:ctx-eb95a1c2) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:50:34,182 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-3:ctx-0dbe91b2 work-2) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:51:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-45:ctx-7be8bae6) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:51:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-90:ctx-3f76c1d7) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:52:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-23:ctx-e902ca80) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:52:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-91:ctx-4d56948f) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:53:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-12:ctx-0876aaa4) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:53:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-93:ctx-c057ee22) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:54:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-47:ctx-8083fccf) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:54:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-94:ctx-56cebf3c) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:55:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-24:ctx-36c965de) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:55:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-95:ctx-86b1db16) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:56:27,241 WARN [o.a.c.alerts] (HA-2:ctx-627e96a3)
alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null //
message:: No usage server process running
2014-12-03 00:56:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-1:ctx-6000a728) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:56:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-97:ctx-ffbe388e) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:57:32,279 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-49:ctx-a54e4fa0) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:57:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-98:ctx-34e48d1f) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:58:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-25:ctx-ffdd759e) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:58:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-99:ctx-0c1a9fa7) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:59:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-101:ctx-8442d937) Unable to get current
status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:59:32,963 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-51:ctx-9bd2b3cc) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:00:31,170 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-2:ctx-f17ce3b5 work-5) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 01:00:31,208 WARN [c.c.u.d.Merovingian2]
(HA-Worker-2:ctx-f17ce3b5 work-5) Was unable to find lock for the
key vm_instance22 and thread id 52609881
2014-12-03 01:00:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-102:ctx-138c6123) Unable to get current
status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:00:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-26:ctx-3e13656a) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:00:34,263 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-2:ctx-f17ce3b5 work-5) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
2014-12-03 01:01:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-103:ctx-334d11fc) Unable to get current
status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:01:32,856 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Unable to
transition the state but we're moving on because it's forced stop
2014-12-03 01:01:32,856 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Unable to
cleanup VM: VM[ConsoleProxy|v-15-VM] ,since outstanding work item is not
found
2014-12-03 01:01:32,856 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Invocation
exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
item not found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state
Starting
2014-12-03 01:01:32,857 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Rethrow
exception com.cloud.utils.exception.CloudRuntimeException: Work item not
found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 01:01:32,857 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-7:ctx-0a1f299d job-429/job-438) Unable to
complete AsyncJobVO {id:438, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
AAAAD3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
created: Wed Dec 03 01:01:03 EST 2014}, job origin:429
com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 01:01:32,865 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-7:ctx-0a1f299d job-429/job-438) Complete async job-438,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAdUpvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
cCBWTVtDb25zb2xlUHJveHl8di0xNS1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFydGluZ3Vy
AB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZh
LmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJp
bmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNv
bS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZh
dAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9i
cy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQA
DHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUu
amF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250
ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQu
amF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29u
dGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EA
fgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hl
LmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEA
fgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3Vy
cmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4A
CwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh
cQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y
dAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51
dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsA
AALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNv
bGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZh
LnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QA
FkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0D
AAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
2014-12-03 01:01:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-52:ctx-76332a49) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:02:32,282 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-105:ctx-b73de834) Unable to get current
status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:02:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-53:ctx-9b938dc5) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:03:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-27:ctx-28735fd7) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:03:32,967 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-107:ctx-e290c580) Unable to get current
status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:04:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-54:ctx-a53679d7) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:04:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-108:ctx-e1d3829f) Unable to get current
status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:05:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-55:ctx-65e26d72) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:05:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-110:ctx-b755e6ac) Unable to get current
status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:06:27,239 WARN [o.a.c.alerts] (HA-3:ctx-b686e09d)
alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null //
message:: No usage server process running
2014-12-03 01:06:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-28:ctx-230f4b8f) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:06:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-111:ctx-77cd282d) Unable to get current
status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
[root@CSTW1 ~]#
Re: Fresh 4.4.2 - Unable to get status on .... my XenServer
Posted by Alessandro Caviglione <c....@gmail.com>.
Yes, I do not believe that the issue is related to a host already
assigned...
Me too have this issue, CS4.4.1 and XS 6.2 hosts Zone installed from
scratch, "Unable to get status on..." error appear in management server
logs from the beginning.
However, I can work with my hosts, create instance, template, networks and
so on...
I can think that this is a bug... :-/
On Wed, Dec 3, 2014 at 4:50 PM, Matthew Midgett <
cloudstck@trick-solutions.com.invalid> wrote:
> I just finished installing XenServer again and started from a setup and I'm
> still getting the same errors that I was before about not being able to get
> a status. What to do now?
>
>
> JrSm9iRGlzcGF0Y2hlcnQAGFZtV29ya0pvYkRpc3BhdGNoZXIuamF2YXQABnJ1bkpvYnNxAH4ACw
>
> AAAfd0AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk
>
> 1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRzcQ
>
> B-AAsAAAAxdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ2
>
> 9udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5zcQB-AA
>
> sAAAA4dABCb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bH
>
> RNYW5hZ2VkQ29udGV4dCQxdAAaRGVmYXVsdE1hbmFnZWRDb250ZXh0LmphdmF0AARjYWxsc3EAfg
>
> ALAAAAZ3QAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdW
>
> x0TWFuYWdlZENvbnRleHRxAH4AGnQAD2NhbGxXaXRoQ29udGV4dHNxAH4ACwAAADVxAH4AHXEAfg
>
> AadAAOcnVuV2l0aENvbnRleHRzcQB-AAsAAAAudAA8b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbm
>
> FnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlcQB-ABZxAH4AF3NxAH4ACwAAAcxxAH
>
> 4AEXEAfgAScQB-ABdzcQB-AAsAAAHXdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuRXhlY3V0b3JzJF
>
> J1bm5hYmxlQWRhcHRlcnQADkV4ZWN1dG9ycy5qYXZhcQB-ABtzcQB-AAsAAAEGdAAfamF2YS51dG
>
> lsLmNvbmN1cnJlbnQuRnV0dXJlVGFza3QAD0Z1dHVyZVRhc2suamF2YXEAfgAXc3EAfgALAAAEeX
>
> QAJ2phdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvcnQAF1RocmVhZFBvb2xFeG
>
> VjdXRvci5qYXZhdAAJcnVuV29ya2Vyc3EAfgALAAACZ3QALmphdmEudXRpbC5jb25jdXJyZW50Ll
>
> RocmVhZFBvb2xFeGVjdXRvciRXb3JrZXJxAH4ALHEAfgAXc3EAfgALAAAC6XQAEGphdmEubGFuZy
>
> 5UaHJlYWR0AAtUaHJlYWQuamF2YXEAfgAXc3IAJmphdmEudXRpbC5Db2xsZWN0aW9ucyRVbm1vZG
>
> lmaWFibGVMaXN0_A8lMbXsjhACAAFMAARsaXN0cQB-AAZ4cgAsamF2YS51dGlsLkNvbGxlY3Rpb2
>
> 5zJFVubW9kaWZpYWJsZUNvbGxlY3Rpb24ZQgCAy173HgIAAUwAAWN0ABZMamF2YS91dGlsL0NvbG
>
> xlY3Rpb247eHBzcgATamF2YS51dGlsLkFycmF5TGlzdHiB0h2Zx2GdAwABSQAEc2l6ZXhwAAAAAH
> cEAAAAAHhxAH4AOHg
> 2014-12-03 10:28:23,558 WARN [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-2d3ea815) Exception while trying
> to start secondary storage vm
> java.lang.RuntimeException: Job failed due to exception Resource [Host:4]
> is
> unreachable: Host 4: Unable to start instance due to DB Exception on:
> com.mysql.jdbc.JDBC4PreparedStatement@7268fbbc: UPDATE
> op_dc_link_local_ip_address_alloc SET
> op_dc_link_local_ip_address_alloc.nic_id=123,
>
> op_dc_link_local_ip_address_alloc.reservation_id=_binary'97c33776-2f60-494b-
> 8f3c-7b20a0d7881f', op_dc_link_local_ip_address_alloc.taken='2014-12-03
> 15:28:23' WHERE op_dc_link_local_ip_address_alloc.id = 1806
> 2014-12-03 10:28:23,562 INFO [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-2d3ea815) Unable to start secondary storage vm for
> standby
> capacity, secStorageVm vm Id : 13, will recycle it and start a new one
> 2014-12-03 10:28:23,697 WARN [c.c.u.d.Merovingian2]
> (secstorage-1:ctx-00d2f7e3) Was unable to find lock for the key
> vm_instance13 and thread id 819805219
> 2014-12-03 10:28:37,991 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-15:ctx-077ec0bf) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:28:56,362 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-30:ctx-7833f032) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:29:37,990 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-8:ctx-40da93df) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:29:52,876 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-db89f3ef) Task (job-393) has been pending for
> 90 seconds
> 2014-12-03 10:29:56,361 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-61:ctx-80573bda) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:30:02,536 DEBUG [o.a.c.s.i.s.TemplateObject]
> (Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) failed to
> process event and answer
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> com.mysql.jdbc.JDBC4PreparedStatement@571a2d18: SELECT
> template_spool_ref.id, template_spool_ref.pool_id,
> template_spool_ref.template_id, template_spool_ref.created,
> template_spool_ref.last_updated, template_spool_ref.download_pct,
> template_spool_ref.download_state, template_spool_ref.local_path,
> template_spool_ref.error_str, template_spool_ref.job_id,
> template_spool_ref.install_path, template_spool_ref.template_size,
> template_spool_ref.marked_for_gc, template_spool_ref.update_count,
> template_spool_ref.updated, template_spool_ref.state FROM
> template_spool_ref
> WHERE template_spool_ref.pool_id = 2 AND template_spool_ref.template_id =
> 1
> ORDER BY RAND() LIMIT 1
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:30:02,560 DEBUG [o.a.c.s.v.VolumeServiceImpl]
> (Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) failed to
> create template on storage
> java.lang.RuntimeException: InvocationTargetException when invoking RPC
> callback for command: copyBaseImageCallback
> Caused by: java.lang.reflect.InvocationTargetException
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to
> process event
> Caused by: com.cloud.utils.exception.CloudRuntimeException: DB Exception
> on:
> com.mysql.jdbc.JDBC4PreparedStatement@571a2d18: SELECT
> template_spool_ref.id, template_spool_ref.pool_id,
> template_spool_ref.template_id, template_spool_ref.created,
> template_spool_ref.last_updated, template_spool_ref.download_pct,
> template_spool_ref.download_state, template_spool_ref.local_path,
> template_spool_ref.error_str, template_spool_ref.job_id,
> template_spool_ref.install_path, template_spool_ref.template_size,
> template_spool_ref.marked_for_gc, template_spool_ref.update_count,
> template_spool_ref.updated, template_spool_ref.state FROM
> template_spool_ref
> WHERE template_spool_ref.pool_id = 2 AND template_spool_ref.template_id =
> 1
> ORDER BY RAND() LIMIT 1
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:30:02,568 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
> (Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) Template 1
> is not found on storage pool 2, so no need to delete
> 2014-12-03 10:30:02,570 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) Failed to
> start instance VM[ConsoleProxy|v-10-VM]
> com.cloud.utils.exception.CloudRuntimeException: can't find mapping in
> ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
> 2014-12-03 10:30:02,693 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) VM state
> transitted from :Starting to Stopped with event: OperationFailedvm's
> original host id: null new host id: null host id before state transition: 1
> 2014-12-03 10:30:02,721 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) Invocation
> exception, caused by: com.cloud.exception.AgentUnavailableException:
> Resource [Host:1] is unreachable: Host 1: Unable to start instance due to
> can't find mapping in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
> 2014-12-03 10:30:02,721 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) Rethrow
> exception com.cloud.exception.AgentUnavailableException: Resource [Host:1]
> is unreachable: Host 1: Unable to start instance due to can't find mapping
> in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
> 2014-12-03 10:30:02,722 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-61:ctx-927aa31e job-388/job-393) Unable to complete
> AsyncJobVO {id:393, userId: 1, accountId: 1, instanceType: null,
> instanceId:
> null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
>
> ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
>
> bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
>
> EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
>
> AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
>
> cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
>
> SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAB
>
> AAAAAAAAAAEAAAAAAAAACnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBw
> cHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
> 0, result: null,
> initMsid: 223162458780723, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Wed Dec 03 10:28:21 EST 2014}, job origin:388
> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
> unreachable: Host 1: Unable to start instance due to can't find mapping in
> ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
> Caused by: com.cloud.utils.exception.CloudRuntimeException: can't find
> mapping in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
> 2014-12-03 10:30:02,723 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-61:ctx-927aa31e job-388/job-393) Complete async job-393,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QA2kpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBSZXNvdXJjZSBbSG9zdDoxXSBpcyB1bnJlYWNoYWJsZTog
>
> SG9zdCAxOiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgZHVlIHRvIGNhbid0IGZpbmQgbWFwcGlu
>
> ZyBpbiBPYmplY3RJbkRhdGFTdG9yZSB0YWJsZSBmb3I6IG9yZy5hcGFjaGUuY2xvdWRzdGFjay5z
>
> dG9yYWdlLmltYWdlLnN0b3JlLlRlbXBsYXRlT2JqZWN0QDc4NWRlMjlidXIAHltMamF2YS5sYW5n
>
> LlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_SI5AgAAeHAAAAANc3IAG2phdmEubGFuZy5TdGFja1Ry
>
> YWNlRWxlbWVudGEJxZomNt2FAgAESQAKbGluZU51bWJlckwADmRlY2xhcmluZ0NsYXNzcQB-AARM
>
> AAhmaWxlTmFtZXEAfgAETAAKbWV0aG9kTmFtZXEAfgAEeHAAAABydAAgY29tLmNsb3VkLnZtLlZt
>
> V29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JzcQB-
>
> AAsAAAH3dAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNK
>
> b2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0
>
> c3EAfgALAAAAMXQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdl
>
> ZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVuc3EA
>
> fgALAAAAOHQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZh
>
> dWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHNx
>
> AH4ACwAAAGd0AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVm
>
> YXVsdE1hbmFnZWRDb250ZXh0cQB-ABp0AA9jYWxsV2l0aENvbnRleHRzcQB-AAsAAAA1cQB-AB1x
>
> AH4AGnQADnJ1bldpdGhDb250ZXh0c3EAfgALAAAALnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5t
>
> YW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAWcQB-ABdzcQB-AAsAAAHM
>
> cQB-ABFxAH4AEnEAfgAXc3EAfgALAAAB13QALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9y
>
> cyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAbc3EAfgALAAABBnQAH2phdmEu
>
> dXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4AF3NxAH4ACwAA
>
> BHl0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29s
>
> RXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnNxAH4ACwAAAmd0AC5qYXZhLnV0aWwuY29uY3VycmVu
>
> dC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-ACxxAH4AF3NxAH4ACwAAAul0ABBqYXZhLmxh
>
> bmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4AF3NyACZqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5t
>
> b2RpZmlhYmxlTGlzdPwPJTG17I4QAgABTAAEbGlzdHEAfgAGeHIALGphdmEudXRpbC5Db2xsZWN0
>
> aW9ucyRVbm1vZGlmaWFibGVDb2xsZWN0aW9uGUIAgMte9x4CAAFMAAFjdAAWTGphdmEvdXRpbC9D
>
> b2xsZWN0aW9uO3hwc3IAE2phdmEudXRpbC5BcnJheUxpc3R4gdIdmcdhnQMAAUkABHNpemV4cAAA
> AAB3BAAAAAB4cQB-ADh4
> 2014-12-03 10:30:02,768 ERROR [c.c.u.d.DbUtil]
> (consoleproxy-1:ctx-8e76fad8)
> RELEASE_LOCK() throws exception
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link
> failure
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:30:02,770 WARN [c.c.v.SystemVmLoadScanner]
> (consoleproxy-1:ctx-8e76fad8) Unexpected exception Job failed due to
> exception Resource [Host:1] is unreachable: Host 1: Unable to start
> instance
> due to can't find mapping in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
> java.lang.RuntimeException: Job failed due to exception Resource [Host:1]
> is
> unreachable: Host 1: Unable to start instance due to can't find mapping in
> ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
> 2014-12-03 10:30:02,861 WARN [c.c.u.d.Merovingian2]
> (consoleproxy-1:ctx-c282e93d) Was unable to find lock for the key
> vm_instance10 and thread id 117755041
> 2014-12-03 10:30:05,371 INFO [o.a.c.s.v.VolumeServiceImpl]
> (Work-Job-Executor-62:ctx-21f67d3c job-384/job-394 ctx-55123741) Unable to
> acquire lock on VMTemplateStoragePool 54
> 2014-12-03 10:30:05,375 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-62:ctx-21f67d3c job-384/job-394 ctx-55123741) Failed to
> start instance VM[SecondaryStorageVm|s-13-VM]
> com.cloud.utils.exception.CloudRuntimeException: Unable to acquire lock on
> VMTemplateStoragePool: 54
> 2014-12-03 10:30:05,527 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-62:ctx-21f67d3c job-384/job-394 ctx-55123741) VM state
> transitted from :Starting to Stopped with event: OperationFailedvm's
> original host id: null new host id: null host id before state transition: 4
> 2014-12-03 10:30:05,551 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-62:ctx-21f67d3c job-384/job-394 ctx-55123741) Invocation
> exception, caused by: com.cloud.exception.AgentUnavailableException:
> Resource [Host:4] is unreachable: Host 4: Unable to start instance due to
> Unable to acquire lock on VMTemplateStoragePool: 54
> 2014-12-03 10:30:05,551 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-62:ctx-21f67d3c job-384/job-394 ctx-55123741) Rethrow
> exception com.cloud.exception.AgentUnavailableException: Resource [Host:4]
> is unreachable: Host 4: Unable to start instance due to Unable to acquire
> lock on VMTemplateStoragePool: 54
> 2014-12-03 10:30:05,551 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-62:ctx-21f67d3c job-384/job-394) Unable to complete
> AsyncJobVO {id:394, userId: 1, accountId: 1, instanceType: null,
> instanceId:
> null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
>
> ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
>
> bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
>
> EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
>
> AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
>
> cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
>
> SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAB
>
> AAAAAAAAAAEAAAAAAAAADXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBw
> cHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
> 0, result: null,
> initMsid: 223162458780723, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Wed Dec 03 10:28:23 EST 2014}, job origin:384
> com.cloud.exception.AgentUnavailableException: Resource [Host:4] is
> unreachable: Host 4: Unable to start instance due to Unable to acquire lock
> on VMTemplateStoragePool: 54
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to
> acquire lock on VMTemplateStoragePool: 54
> 2014-12-03 10:30:05,553 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-62:ctx-21f67d3c job-384/job-394) Complete async job-394,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAmUpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBSZXNvdXJjZSBbSG9zdDo0XSBpcyB1bnJlYWNoYWJsZTog
>
> SG9zdCA0OiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgZHVlIHRvIFVuYWJsZSB0byBhY3F1aXJl
>
> IGxvY2sgb24gVk1UZW1wbGF0ZVN0b3JhZ2VQb29sOiA1NHVyAB5bTGphdmEubGFuZy5TdGFja1Ry
>
> YWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1l
>
> bnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5h
>
> bWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JE
>
> aXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QA
>
> P29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdl
>
> ckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAA
>
> ADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
>
> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0
>
> AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFn
>
> ZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABn
>
> dABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5h
>
> Z2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5y
>
> dW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
>
> b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-
>
> ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFi
>
> bGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29u
>
> Y3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2
>
> YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9y
>
> LmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFk
>
> UG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVh
>
> ZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJs
>
> ZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5t
>
> b2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlv
>
> bjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAA
> eHEAfgA4eA
> 2014-12-03 10:30:05,596 WARN [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-00d2f7e3) Exception while trying
> to start secondary storage vm
> java.lang.RuntimeException: Job failed due to exception Resource [Host:4]
> is
> unreachable: Host 4: Unable to start instance due to Unable to acquire lock
> on VMTemplateStoragePool: 54
> 2014-12-03 10:30:05,597 ERROR [c.c.u.d.DbUtil] (secstorage-1:ctx-00d2f7e3)
> RELEASE_LOCK() throws exception
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link
> failure
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:30:05,599 INFO [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-00d2f7e3) Unable to start secondary storage vm for
> standby
> capacity, secStorageVm vm Id : 13, will recycle it and start a new one
> 2014-12-03 10:30:05,663 WARN [c.c.u.d.Merovingian2]
> (secstorage-1:ctx-00d2f7e3) Was unable to find lock for the key
> vm_instance13 and thread id 819805219
> 2014-12-03 10:30:07,062 ERROR [c.c.u.d.DbUtil] (secstorage-1:ctx-00d2f7e3)
> RELEASE_LOCK() throws exception
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link
> failure
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:30:07,459 WARN [c.c.u.d.Merovingian2]
> (secstorage-1:ctx-5b27f0f0) Was unable to find lock for the key
> vm_instance14 and thread id 819805219
> 2014-12-03 10:30:37,990 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-31:ctx-37ebcabd) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:30:56,361 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-62:ctx-f52b8d19) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:31:37,991 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-16:ctx-4133ae03) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:31:46,486 DEBUG [o.a.c.s.i.s.TemplateObject]
> (Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) failed to
> process event and answer
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> com.mysql.jdbc.JDBC4PreparedStatement@311d349d: SELECT
> template_spool_ref.id, template_spool_ref.pool_id,
> template_spool_ref.template_id, template_spool_ref.created,
> template_spool_ref.last_updated, template_spool_ref.download_pct,
> template_spool_ref.download_state, template_spool_ref.local_path,
> template_spool_ref.error_str, template_spool_ref.job_id,
> template_spool_ref.install_path, template_spool_ref.template_size,
> template_spool_ref.marked_for_gc, template_spool_ref.update_count,
> template_spool_ref.updated, template_spool_ref.state FROM
> template_spool_ref
> WHERE template_spool_ref.pool_id = 2 AND template_spool_ref.template_id =
> 1
> ORDER BY RAND() LIMIT 1
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:31:46,510 DEBUG [o.a.c.s.v.VolumeServiceImpl]
> (Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) failed to
> create template on storage
> java.lang.RuntimeException: InvocationTargetException when invoking RPC
> callback for command: copyBaseImageCallback
> Caused by: java.lang.reflect.InvocationTargetException
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to
> process event
> Caused by: com.cloud.utils.exception.CloudRuntimeException: DB Exception
> on:
> com.mysql.jdbc.JDBC4PreparedStatement@311d349d: SELECT
> template_spool_ref.id, template_spool_ref.pool_id,
> template_spool_ref.template_id, template_spool_ref.created,
> template_spool_ref.last_updated, template_spool_ref.download_pct,
> template_spool_ref.download_state, template_spool_ref.local_path,
> template_spool_ref.error_str, template_spool_ref.job_id,
> template_spool_ref.install_path, template_spool_ref.template_size,
> template_spool_ref.marked_for_gc, template_spool_ref.update_count,
> template_spool_ref.updated, template_spool_ref.state FROM
> template_spool_ref
> WHERE template_spool_ref.pool_id = 2 AND template_spool_ref.template_id =
> 1
> ORDER BY RAND() LIMIT 1
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:31:46,517 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
> (Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) Template 1
> is not found on storage pool 2, so no need to delete
> 2014-12-03 10:31:46,519 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) Failed to
> start instance VM[ConsoleProxy|v-10-VM]
> com.cloud.utils.exception.CloudRuntimeException: can't find mapping in
> ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
> 2014-12-03 10:31:46,617 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) VM state
> transitted from :Starting to Stopped with event: OperationFailedvm's
> original host id: null new host id: null host id before state transition: 1
> 2014-12-03 10:31:46,642 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) Invocation
> exception, caused by: com.cloud.exception.AgentUnavailableException:
> Resource [Host:1] is unreachable: Host 1: Unable to start instance due to
> can't find mapping in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
> 2014-12-03 10:31:46,642 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) Rethrow
> exception com.cloud.exception.AgentUnavailableException: Resource [Host:1]
> is unreachable: Host 1: Unable to start instance due to can't find mapping
> in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
> 2014-12-03 10:31:46,642 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396) Unable to complete
> AsyncJobVO {id:396, userId: 1, accountId: 1, instanceType: null,
> instanceId:
> null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
>
> ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
>
> bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
>
> EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
>
> AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
>
> cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
>
> SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAB
>
> AAAAAAAAAAEAAAAAAAAACnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBw
> cHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
> 0, result: null,
> initMsid: 223162458780723, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Wed Dec 03 10:30:02 EST 2014}, job origin:388
> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
> unreachable: Host 1: Unable to start instance due to can't find mapping in
> ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
> Caused by: com.cloud.utils.exception.CloudRuntimeException: can't find
> mapping in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
> 2014-12-03 10:31:46,644 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396) Complete async job-396,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QA2kpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBSZXNvdXJjZSBbSG9zdDoxXSBpcyB1bnJlYWNoYWJsZTog
>
> SG9zdCAxOiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgZHVlIHRvIGNhbid0IGZpbmQgbWFwcGlu
>
> ZyBpbiBPYmplY3RJbkRhdGFTdG9yZSB0YWJsZSBmb3I6IG9yZy5hcGFjaGUuY2xvdWRzdGFjay5z
>
> dG9yYWdlLmltYWdlLnN0b3JlLlRlbXBsYXRlT2JqZWN0QDI1ZTg0ZWVldXIAHltMamF2YS5sYW5n
>
> LlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_SI5AgAAeHAAAAANc3IAG2phdmEubGFuZy5TdGFja1Ry
>
> YWNlRWxlbWVudGEJxZomNt2FAgAESQAKbGluZU51bWJlckwADmRlY2xhcmluZ0NsYXNzcQB-AARM
>
> AAhmaWxlTmFtZXEAfgAETAAKbWV0aG9kTmFtZXEAfgAEeHAAAABydAAgY29tLmNsb3VkLnZtLlZt
>
> V29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JzcQB-
>
> AAsAAAH3dAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNK
>
> b2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0
>
> c3EAfgALAAAAMXQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdl
>
> ZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVuc3EA
>
> fgALAAAAOHQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZh
>
> dWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHNx
>
> AH4ACwAAAGd0AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVm
>
> YXVsdE1hbmFnZWRDb250ZXh0cQB-ABp0AA9jYWxsV2l0aENvbnRleHRzcQB-AAsAAAA1cQB-AB1x
>
> AH4AGnQADnJ1bldpdGhDb250ZXh0c3EAfgALAAAALnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5t
>
> YW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAWcQB-ABdzcQB-AAsAAAHM
>
> cQB-ABFxAH4AEnEAfgAXc3EAfgALAAAB13QALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9y
>
> cyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAbc3EAfgALAAABBnQAH2phdmEu
>
> dXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4AF3NxAH4ACwAA
>
> BHl0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29s
>
> RXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnNxAH4ACwAAAmd0AC5qYXZhLnV0aWwuY29uY3VycmVu
>
> dC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-ACxxAH4AF3NxAH4ACwAAAul0ABBqYXZhLmxh
>
> bmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4AF3NyACZqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5t
>
> b2RpZmlhYmxlTGlzdPwPJTG17I4QAgABTAAEbGlzdHEAfgAGeHIALGphdmEudXRpbC5Db2xsZWN0
>
> aW9ucyRVbm1vZGlmaWFibGVDb2xsZWN0aW9uGUIAgMte9x4CAAFMAAFjdAAWTGphdmEvdXRpbC9D
>
> b2xsZWN0aW9uO3hwc3IAE2phdmEudXRpbC5BcnJheUxpc3R4gdIdmcdhnQMAAUkABHNpemV4cAAA
> AAB3BAAAAAB4cQB-ADh4
> 2014-12-03 10:31:46,689 ERROR [c.c.u.d.DbUtil]
> (consoleproxy-1:ctx-c282e93d)
> RELEASE_LOCK() throws exception
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link
> failure
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:31:46,691 WARN [c.c.v.SystemVmLoadScanner]
> (consoleproxy-1:ctx-c282e93d) Unexpected exception Job failed due to
> exception Resource [Host:1] is unreachable: Host 1: Unable to start
> instance
> due to can't find mapping in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
> java.lang.RuntimeException: Job failed due to exception Resource [Host:1]
> is
> unreachable: Host 1: Unable to start instance due to can't find mapping in
> ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
> 2014-12-03 10:31:46,788 WARN [c.c.u.d.Merovingian2]
> (consoleproxy-1:ctx-5c69b42f) Was unable to find lock for the key
> vm_instance10 and thread id 117755041
> 2014-12-03 10:31:49,395 INFO [o.a.c.s.v.VolumeServiceImpl]
> (Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400 ctx-2ee9073d) Unable to
> acquire lock on VMTemplateStoragePool 57
> 2014-12-03 10:31:49,399 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400 ctx-2ee9073d) Failed to
> start instance VM[SecondaryStorageVm|s-14-VM]
> com.cloud.utils.exception.CloudRuntimeException: Unable to acquire lock on
> VMTemplateStoragePool: 57
> 2014-12-03 10:31:49,554 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400 ctx-2ee9073d) VM state
> transitted from :Starting to Stopped with event: OperationFailedvm's
> original host id: null new host id: null host id before state transition: 4
> 2014-12-03 10:31:49,578 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400 ctx-2ee9073d) Invocation
> exception, caused by: com.cloud.exception.AgentUnavailableException:
> Resource [Host:4] is unreachable: Host 4: Unable to start instance due to
> Unable to acquire lock on VMTemplateStoragePool: 57
> 2014-12-03 10:31:49,578 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400 ctx-2ee9073d) Rethrow
> exception com.cloud.exception.AgentUnavailableException: Resource [Host:4]
> is unreachable: Host 4: Unable to start instance due to Unable to acquire
> lock on VMTemplateStoragePool: 57
> 2014-12-03 10:31:49,579 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400) Unable to complete
> AsyncJobVO {id:400, userId: 1, accountId: 1, instanceType: null,
> instanceId:
> null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
>
> ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
>
> bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
>
> EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
>
> AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
>
> cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
>
> SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAB
>
> AAAAAAAAAAEAAAAAAAAADnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBw
> cHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
> 0, result: null,
> initMsid: 223162458780723, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Wed Dec 03 10:30:07 EST 2014}, job origin:384
> com.cloud.exception.AgentUnavailableException: Resource [Host:4] is
> unreachable: Host 4: Unable to start instance due to Unable to acquire lock
> on VMTemplateStoragePool: 57
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to
> acquire lock on VMTemplateStoragePool: 57
> 2014-12-03 10:31:49,580 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400) Complete async job-400,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAmUpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBSZXNvdXJjZSBbSG9zdDo0XSBpcyB1bnJlYWNoYWJsZTog
>
> SG9zdCA0OiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgZHVlIHRvIFVuYWJsZSB0byBhY3F1aXJl
>
> IGxvY2sgb24gVk1UZW1wbGF0ZVN0b3JhZ2VQb29sOiA1N3VyAB5bTGphdmEubGFuZy5TdGFja1Ry
>
> YWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1l
>
> bnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5h
>
> bWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JE
>
> aXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QA
>
> P29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdl
>
> ckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAA
>
> ADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
>
> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0
>
> AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFn
>
> ZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABn
>
> dABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5h
>
> Z2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5y
>
> dW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
>
> b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-
>
> ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFi
>
> bGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29u
>
> Y3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2
>
> YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9y
>
> LmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFk
>
> UG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVh
>
> ZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJs
>
> ZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5t
>
> b2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlv
>
> bjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAA
> eHEAfgA4eA
> 2014-12-03 10:31:49,620 WARN [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-5b27f0f0) Exception while trying
> to start secondary storage vm
> java.lang.RuntimeException: Job failed due to exception Resource [Host:4]
> is
> unreachable: Host 4: Unable to start instance due to Unable to acquire lock
> on VMTemplateStoragePool: 57
> 2014-12-03 10:31:49,622 ERROR [c.c.u.d.DbUtil] (secstorage-1:ctx-5b27f0f0)
> RELEASE_LOCK() throws exception
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link
> failure
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:31:49,624 INFO [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-5b27f0f0) Unable to start secondary storage vm for
> standby
> capacity, secStorageVm vm Id : 14, will recycle it and start a new one
> 2014-12-03 10:31:49,624 ERROR [c.c.u.d.DbUtil] (secstorage-1:ctx-5b27f0f0)
> RELEASE_LOCK() throws exception
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link
> failure
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:31:49,781 WARN [c.c.u.d.Merovingian2]
> (secstorage-1:ctx-bc09a930) Was unable to find lock for the key
> vm_instance14 and thread id 819805219
> 2014-12-03 10:31:56,361 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-32:ctx-853d1e1b) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:32:37,989 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-3:ctx-fefa154b) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:32:56,361 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-65:ctx-51ba659a) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:33:37,990 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-33:ctx-e578dc8f) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:33:47,948 DEBUG [o.a.c.s.i.s.TemplateObject]
> (Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) failed to
> process event and answer
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> com.mysql.jdbc.JDBC4PreparedStatement@37aaee74: SELECT
> template_spool_ref.id, template_spool_ref.pool_id,
> template_spool_ref.template_id, template_spool_ref.created,
> template_spool_ref.last_updated, template_spool_ref.download_pct,
> template_spool_ref.download_state, template_spool_ref.local_path,
> template_spool_ref.error_str, template_spool_ref.job_id,
> template_spool_ref.install_path, template_spool_ref.template_size,
> template_spool_ref.marked_for_gc, template_spool_ref.update_count,
> template_spool_ref.updated, template_spool_ref.state FROM
> template_spool_ref
> WHERE template_spool_ref.pool_id = 2 AND template_spool_ref.template_id =
> 1
> ORDER BY RAND() LIMIT 1
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:33:47,970 DEBUG [o.a.c.s.v.VolumeServiceImpl]
> (Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) failed to
> create template on storage
> java.lang.RuntimeException: InvocationTargetException when invoking RPC
> callback for command: copyBaseImageCallback
> Caused by: java.lang.reflect.InvocationTargetException
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to
> process event
> Caused by: com.cloud.utils.exception.CloudRuntimeException: DB Exception
> on:
> com.mysql.jdbc.JDBC4PreparedStatement@37aaee74: SELECT
> template_spool_ref.id, template_spool_ref.pool_id,
> template_spool_ref.template_id, template_spool_ref.created,
> template_spool_ref.last_updated, template_spool_ref.download_pct,
> template_spool_ref.download_state, template_spool_ref.local_path,
> template_spool_ref.error_str, template_spool_ref.job_id,
> template_spool_ref.install_path, template_spool_ref.template_size,
> template_spool_ref.marked_for_gc, template_spool_ref.update_count,
> template_spool_ref.updated, template_spool_ref.state FROM
> template_spool_ref
> WHERE template_spool_ref.pool_id = 2 AND template_spool_ref.template_id =
> 1
> ORDER BY RAND() LIMIT 1
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:33:47,976 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
> (Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) Template 1
> is not found on storage pool 2, so no need to delete
> 2014-12-03 10:33:47,979 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) Failed to
> start instance VM[ConsoleProxy|v-10-VM]
> com.cloud.utils.exception.CloudRuntimeException: can't find mapping in
> ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
> 2014-12-03 10:33:48,085 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) VM state
> transitted from :Starting to Stopped with event: OperationFailedvm's
> original host id: null new host id: null host id before state transition: 4
> 2014-12-03 10:33:48,112 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) Invocation
> exception, caused by: com.cloud.exception.AgentUnavailableException:
> Resource [Host:4] is unreachable: Host 4: Unable to start instance due to
> can't find mapping in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
> 2014-12-03 10:33:48,113 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) Rethrow
> exception com.cloud.exception.AgentUnavailableException: Resource [Host:4]
> is unreachable: Host 4: Unable to start instance due to can't find mapping
> in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
> 2014-12-03 10:33:48,113 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-66:ctx-38c8d928 job-388/job-402) Unable to complete
> AsyncJobVO {id:402, userId: 1, accountId: 1, instanceType: null,
> instanceId:
> null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
>
> ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
>
> bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
>
> EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
>
> AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
>
> cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
>
> SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAB
>
> AAAAAAAAAAEAAAAAAAAACnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBw
> cHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
> 0, result: null,
> initMsid: 223162458780723, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Wed Dec 03 10:31:46 EST 2014}, job origin:388
> com.cloud.exception.AgentUnavailableException: Resource [Host:4] is
> unreachable: Host 4: Unable to start instance due to can't find mapping in
> ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
> Caused by: com.cloud.utils.exception.CloudRuntimeException: can't find
> mapping in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
> 2014-12-03 10:33:48,115 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-66:ctx-38c8d928 job-388/job-402) Complete async job-402,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QA2kpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBSZXNvdXJjZSBbSG9zdDo0XSBpcyB1bnJlYWNoYWJsZTog
>
> SG9zdCA0OiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgZHVlIHRvIGNhbid0IGZpbmQgbWFwcGlu
>
> ZyBpbiBPYmplY3RJbkRhdGFTdG9yZSB0YWJsZSBmb3I6IG9yZy5hcGFjaGUuY2xvdWRzdGFjay5z
>
> dG9yYWdlLmltYWdlLnN0b3JlLlRlbXBsYXRlT2JqZWN0QDRjN2JjNzc0dXIAHltMamF2YS5sYW5n
>
> LlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_SI5AgAAeHAAAAANc3IAG2phdmEubGFuZy5TdGFja1Ry
>
> YWNlRWxlbWVudGEJxZomNt2FAgAESQAKbGluZU51bWJlckwADmRlY2xhcmluZ0NsYXNzcQB-AARM
>
> AAhmaWxlTmFtZXEAfgAETAAKbWV0aG9kTmFtZXEAfgAEeHAAAABydAAgY29tLmNsb3VkLnZtLlZt
>
> V29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JzcQB-
>
> AAsAAAH3dAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNK
>
> b2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0
>
> c3EAfgALAAAAMXQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdl
>
> ZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVuc3EA
>
> fgALAAAAOHQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZh
>
> dWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHNx
>
> AH4ACwAAAGd0AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVm
>
> YXVsdE1hbmFnZWRDb250ZXh0cQB-ABp0AA9jYWxsV2l0aENvbnRleHRzcQB-AAsAAAA1cQB-AB1x
>
> AH4AGnQADnJ1bldpdGhDb250ZXh0c3EAfgALAAAALnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5t
>
> YW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAWcQB-ABdzcQB-AAsAAAHM
>
> cQB-ABFxAH4AEnEAfgAXc3EAfgALAAAB13QALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9y
>
> cyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAbc3EAfgALAAABBnQAH2phdmEu
>
> dXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4AF3NxAH4ACwAA
>
> BHl0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29s
>
> RXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnNxAH4ACwAAAmd0AC5qYXZhLnV0aWwuY29uY3VycmVu
>
> dC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-ACxxAH4AF3NxAH4ACwAAAul0ABBqYXZhLmxh
>
> bmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4AF3NyACZqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5t
>
> b2RpZmlhYmxlTGlzdPwPJTG17I4QAgABTAAEbGlzdHEAfgAGeHIALGphdmEudXRpbC5Db2xsZWN0
>
> aW9ucyRVbm1vZGlmaWFibGVDb2xsZWN0aW9uGUIAgMte9x4CAAFMAAFjdAAWTGphdmEvdXRpbC9D
>
> b2xsZWN0aW9uO3hwc3IAE2phdmEudXRpbC5BcnJheUxpc3R4gdIdmcdhnQMAAUkABHNpemV4cAAA
> AAB3BAAAAAB4cQB-ADh4
> 2014-12-03 10:33:48,158 ERROR [c.c.u.d.DbUtil]
> (consoleproxy-1:ctx-5c69b42f)
> RELEASE_LOCK() throws exception
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link
> failure
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:33:48,160 WARN [c.c.v.SystemVmLoadScanner]
> (consoleproxy-1:ctx-5c69b42f) Unexpected exception Job failed due to
> exception Resource [Host:4] is unreachable: Host 4: Unable to start
> instance
> due to can't find mapping in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
> java.lang.RuntimeException: Job failed due to exception Resource [Host:4]
> is
> unreachable: Host 4: Unable to start instance due to can't find mapping in
> ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
> 2014-12-03 10:33:48,251 WARN [c.c.u.d.Merovingian2]
> (consoleproxy-1:ctx-a282718d) Was unable to find lock for the key
> vm_instance10 and thread id 117755041
> 2014-12-03 10:33:51,376 INFO [o.a.c.s.v.VolumeServiceImpl]
> (Work-Job-Executor-67:ctx-b6aee29d job-384/job-403 ctx-05dc78c7) Unable to
> acquire lock on VMTemplateStoragePool 60
> 2014-12-03 10:33:51,380 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-67:ctx-b6aee29d job-384/job-403 ctx-05dc78c7) Failed to
> start instance VM[SecondaryStorageVm|s-14-VM]
> com.cloud.utils.exception.CloudRuntimeException: Unable to acquire lock on
> VMTemplateStoragePool: 60
> 2014-12-03 10:33:51,508 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-67:ctx-b6aee29d job-384/job-403 ctx-05dc78c7) VM state
> transitted from :Starting to Stopped with event: OperationFailedvm's
> original host id: null new host id: null host id before state transition: 1
> 2014-12-03 10:33:51,533 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-67:ctx-b6aee29d job-384/job-403 ctx-05dc78c7) Invocation
> exception, caused by: com.cloud.exception.AgentUnavailableException:
> Resource [Host:1] is unreachable: Host 1: Unable to start instance due to
> Unable to acquire lock on VMTemplateStoragePool: 60
> 2014-12-03 10:33:51,534 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-67:ctx-b6aee29d job-384/job-403 ctx-05dc78c7) Rethrow
> exception com.cloud.exception.AgentUnavailableException: Resource [Host:1]
> is unreachable: Host 1: Unable to start instance due to Unable to acquire
> lock on VMTemplateStoragePool: 60
> 2014-12-03 10:33:51,534 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-67:ctx-b6aee29d job-384/job-403) Unable to complete
> AsyncJobVO {id:403, userId: 1, accountId: 1, instanceType: null,
> instanceId:
> null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
>
> ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
>
> bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
>
> EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
>
> AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
>
> cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
>
> SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAB
>
> AAAAAAAAAAEAAAAAAAAADnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBw
> cHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
> 0, result: null,
> initMsid: 223162458780723, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Wed Dec 03 10:31:49 EST 2014}, job origin:384
> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
> unreachable: Host 1: Unable to start instance due to Unable to acquire lock
> on VMTemplateStoragePool: 60
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to
> acquire lock on VMTemplateStoragePool: 60
> 2014-12-03 10:33:51,536 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-67:ctx-b6aee29d job-384/job-403) Complete async job-403,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAmUpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBSZXNvdXJjZSBbSG9zdDoxXSBpcyB1bnJlYWNoYWJsZTog
>
> SG9zdCAxOiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgZHVlIHRvIFVuYWJsZSB0byBhY3F1aXJl
>
> IGxvY2sgb24gVk1UZW1wbGF0ZVN0b3JhZ2VQb29sOiA2MHVyAB5bTGphdmEubGFuZy5TdGFja1Ry
>
> YWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1l
>
> bnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5h
>
> bWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JE
>
> aXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QA
>
> P29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdl
>
> ckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAA
>
> ADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
>
> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0
>
> AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFn
>
> ZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABn
>
> dABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5h
>
> Z2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5y
>
> dW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
>
> b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-
>
> ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFi
>
> bGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29u
>
> Y3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2
>
> YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9y
>
> LmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFk
>
> UG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVh
>
> ZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJs
>
> ZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5t
>
> b2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlv
>
> bjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAA
> eHEAfgA4eA
> 2014-12-03 10:33:51,576 WARN [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-bc09a930) Exception while trying
> to start secondary storage vm
> java.lang.RuntimeException: Job failed due to exception Resource [Host:1]
> is
> unreachable: Host 1: Unable to start instance due to Unable to acquire lock
> on VMTemplateStoragePool: 60
> 2014-12-03 10:33:51,577 ERROR [c.c.u.d.DbUtil] (secstorage-1:ctx-bc09a930)
> RELEASE_LOCK() throws exception
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link
> failure
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:33:51,579 INFO [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-bc09a930) Unable to start secondary storage vm for
> standby
> capacity, secStorageVm vm Id : 14, will recycle it and start a new one
> 2014-12-03 10:33:51,609 WARN [c.c.u.d.Merovingian2]
> (secstorage-1:ctx-bc09a930) Was unable to find lock for the key
> vm_instance14 and thread id 819805219
> 2014-12-03 10:33:52,060 ERROR [c.c.u.d.DbUtil] (secstorage-1:ctx-bc09a930)
> RELEASE_LOCK() throws exception
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link
> failure
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:33:52,397 WARN [c.c.u.d.Merovingian2]
> (secstorage-1:ctx-0a61fb70) Was unable to find lock for the key
> vm_instance15 and thread id 819805219
> 2014-12-03 10:33:56,362 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-66:ctx-580f5bb1) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:35:31,403 ERROR [c.c.c.ClusterManagerImpl] (main:null) Unable
> to ping management server at 172.16.0.69:9090 due to ConnectException
> java.net.ConnectException: Connection refused
> 2014-12-03 10:35:31,591 WARN [c.c.c.ConsoleProxyManagerImpl] (main:null)
> Empty console proxy domain, explicitly disabling SSL
> 2014-12-03 10:35:31,993 WARN [c.c.s.d.DownloadMonitorImpl] (main:null)
> Only
> realhostip.com ssl cert is supported, ignoring self-signed and other certs
> 2014-12-03 10:35:49,150 INFO [c.c.h.x.r.XenServerConnectionPool]
> (main:null) XenServer Connection Pool Configs:
> sleep.interval.on.error=10000
> 2014-12-03 10:35:49,732 ERROR [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-3:ctx-c81f933c) Caught this throwable,
> com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to
> commit
> or close the connection.
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock
> found when trying to get lock; try restarting transaction
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1065)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> 2014-12-03 10:35:49,750 ERROR [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-4:ctx-ebb2805b) Caught this throwable,
> com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
> Caused by: com.cloud.utils.exception.CloudRuntimeException: DB Exception
> on:
> com.mysql.jdbc.JDBC4PreparedStatement@30b1baa6: UPDATE op_ha_work SET
> op_ha_work.taken='2014-12-03 15:35:49',
> op_ha_work.mgmt_server_id=223162458780723 WHERE op_ha_work.id = 4
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock
> found when trying to get lock; try restarting transaction
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1065)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> 2014-12-03 10:35:49,808 WARN [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
> Mark job as failed as its correspoding queue-item has been discarded. job
> id: 405
> 2014-12-03 10:35:49,808 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
> Complete async job-405, jobStatus: FAILED,
> resultCode: 0, result: Execution was cancelled because of server shutdown
> 2014-12-03 10:35:49,813 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
> job-405 no longer exists, we just log completion info here. FAILED,
> resultCode: 0, result: Execution was cancelled because of server shutdown
> 2014-12-03 10:35:49,850 DEBUG [c.c.h.CheckOnAgentInvestigator]
> (HA-Worker-2:ctx-b2a9dc97 work-1) Unable to reach the agent for
> VM[SecondaryStorageVm|s-15-VM]: Resource [Host:4] is unreachable: Host 4:
> Host with specified id is not in the
> right state: Disconnected
> 2014-12-03 10:35:49,863 WARN [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
> Mark job as failed as its correspoding queue-item has been discarded. job
> id: 411
> 2014-12-03 10:35:49,863 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
> Complete async job-411, jobStatus: FAILED,
> resultCode: 0, result: Execution was cancelled because of server shutdown
> 2014-12-03 10:35:49,859 DEBUG [c.c.h.CheckOnAgentInvestigator]
> (HA-Worker-1:ctx-415457e8 work-4) Unable to reach the agent for
> VM[ConsoleProxy|v-10-VM]: Resource [Host:4] is unreachable: Host 4: Host
> with specified id is not in the right
> state: Disconnected
> 2014-12-03 10:35:49,885 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
> job-411 no longer exists, we just log completion info here. FAILED,
> resultCode: 0, result: Execution was cancelled because of server shutdown
> 2014-12-03 10:35:49,901 DEBUG [c.c.h.UserVmDomRInvestigator]
> (HA-Worker-1:ctx-415457e8 work-4) Not a User Vm, unable to determine state
> of VM[ConsoleProxy|v-10-VM] returning null
> 2014-12-03 10:35:49,902 DEBUG [c.c.h.UserVmDomRInvestigator]
> (HA-Worker-2:ctx-b2a9dc97 work-1) Not a User Vm, unable to determine state
> of VM[SecondaryStorageVm|s-15-VM] returning null
> 2014-12-03 10:35:49,921 DEBUG [c.c.h.ManagementIPSystemVMInvestigator]
> (HA-Worker-2:ctx-b2a9dc97 work-1) unable to determine state of
> VM[SecondaryStorageVm|s-15-VM] returning null
> 2014-12-03 10:35:49,922 WARN [o.a.c.f.j.AsyncJobExecutionContext]
> (HA-Worker-2:ctx-b2a9dc97 work-1) Job is executed without a context, setup
> psudo job for the executing thread
> 2014-12-03 10:35:49,929 DEBUG [c.c.h.ManagementIPSystemVMInvestigator]
> (HA-Worker-1:ctx-415457e8 work-4) unable to determine state of
> VM[ConsoleProxy|v-10-VM] returning null
> 2014-12-03 10:35:49,930 WARN [o.a.c.f.j.AsyncJobExecutionContext]
> (HA-Worker-1:ctx-415457e8 work-4) Job is executed without a context, setup
> psudo job for the executing thread
> 2014-12-03 10:35:50,055 WARN [c.c.u.d.Merovingian2]
> (HA-Worker-1:ctx-415457e8 work-4) Was unable to find lock for the
> key vm_instance10 and thread id 1329615500
> 2014-12-03 10:35:50,059 WARN [c.c.u.d.Merovingian2]
> (HA-Worker-2:ctx-b2a9dc97 work-1) Was unable to find lock for the
> key vm_instance15 and thread id 9330563
> 2014-12-03 10:35:50,849 ERROR [c.c.c.ClusterManagerImpl]
> (Cluster-Heartbeat-1:ctx-375d995b) Unable to ping management server at
> 172.16.0.70:9090 due to ConnectException
> java.net.ConnectException: Connection refused
> 2014-12-03 10:35:50,850 WARN [c.c.c.ClusterManagerImpl]
> (Cluster-Heartbeat-1:ctx-375d995b) Management node 4 is detected inactive
> by
> timestamp and also not pingable
> 2014-12-03 10:35:52,166 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-1:ctx-e4c35b10 job-413/job-415 ctx-a901375d) Unable to
> transition the state but we're moving on because it's forced stop
> 2014-12-03 10:35:52,166 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-1:ctx-e4c35b10 job-413/job-415 ctx-a901375d) Unable to
> cleanup VM: VM[ConsoleProxy|v-10-VM] ,since outstanding work item is not
> found
> 2014-12-03 10:35:52,166 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-1:ctx-e4c35b10 job-413/job-415 ctx-a901375d) Invocation
> exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
> item not found, We cannot stop VM[ConsoleProxy|v-10-VM] when it is in state
> Starting
> 2014-12-03 10:35:52,167 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-1:ctx-e4c35b10 job-413/job-415 ctx-a901375d) Rethrow
> exception com.cloud.utils.exception.CloudRuntimeException: Work item not
> found, We cannot stop VM[ConsoleProxy|v-10-VM] when it is in state Starting
> 2014-12-03 10:35:52,168 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-1:ctx-e4c35b10 job-413/job-415) Unable to
> complete AsyncJobVO {id:415, userId: 1, accountId: 1, instanceType: null,
> instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
>
> rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
>
> b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
>
> TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
> AAAACnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
> IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
> 223162458780723, completeMsid: null, lastUpdated: null, lastPolled: null,
> created: Wed Dec 03 10:35:50 EST 2014}, job origin:413
> com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
> cannot stop VM[ConsoleProxy|v-10-VM] when it is in state Starting
> 2014-12-03 10:35:52,173 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-1:ctx-e4c35b10 job-413/job-415) Complete async job-415,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAdUpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
>
> cCBWTVtDb25zb2xlUHJveHl8di0xMC1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFydGluZ3Vy
>
> AB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZh
>
> LmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJp
>
> bmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNv
>
> bS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZh
>
> dAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9i
>
> cy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQA
>
> DHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
>
> b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUu
>
> amF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250
>
> ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQu
>
> amF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29u
>
> dGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EA
>
> fgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hl
>
> LmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEA
>
> fgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3Vy
>
> cmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4A
>
> CwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh
>
> cQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y
>
> dAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51
>
> dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsA
>
> AALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNv
>
> bGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZh
>
> LnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QA
>
> FkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0D
> AAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
> 2014-12-03 10:35:52,175 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-2:ctx-fd2be5c1 job-412/job-416 ctx-c11ef29e) Unable to
> transition the state but we're moving on because it's forced stop
> 2014-12-03 10:35:52,175 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-2:ctx-fd2be5c1 job-412/job-416 ctx-c11ef29e) Unable to
> cleanup VM: VM[SecondaryStorageVm|s-15-VM] ,since outstanding work item is
> not found
> 2014-12-03 10:35:52,175 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-2:ctx-fd2be5c1 job-412/job-416 ctx-c11ef29e) Invocation
> exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
> item not found, We cannot stop VM[SecondaryStorageVm|s-15-VM] when it is in
> state Starting
> 2014-12-03 10:35:52,176 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-2:ctx-fd2be5c1 job-412/job-416 ctx-c11ef29e) Rethrow
> exception com.cloud.utils.exception.CloudRuntimeException: Work item not
> found, We cannot stop VM[SecondaryStorageVm|s-15-VM] when it is in state
> Starting
> 2014-12-03 10:35:52,176 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-2:ctx-fd2be5c1 job-412/job-416) Unable to
> complete AsyncJobVO {id:416, userId: 1, accountId: 1, instanceType: null,
> instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
>
> rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
>
> b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
>
> TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
> AAAAD3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
> IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
> 223162458780723, completeMsid: null, lastUpdated: null, lastPolled: null,
> created: Wed Dec 03 10:35:50 EST 2014}, job origin:412
> com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
> cannot stop VM[SecondaryStorageVm|s-15-VM] when it is in state Starting
> 2014-12-03 10:35:52,178 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-2:ctx-fd2be5c1 job-412/job-416) Complete async job-416,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAe0pvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
>
> cCBWTVtTZWNvbmRhcnlTdG9yYWdlVm18cy0xNS1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFy
>
> dGluZ3VyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNy
>
> ABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5k
>
> ZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAA
>
> cnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hl
>
> ci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdv
>
> cmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwu
>
> amF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFu
>
> YWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVu
>
> bmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdl
>
> ZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENv
>
> bnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFn
>
> ZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250
>
> ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcu
>
> YXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVx
>
> AH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwu
>
> Y29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4A
>
> G3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFz
>
> ay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4
>
> ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAu
>
> amF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdz
>
> cQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51
>
> dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhy
>
> ACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgAB
>
> TAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHS
> HZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
> 2014-12-03 10:35:52,274 WARN [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-1:ctx-415457e8 work-4) Encountered unhandled exception during HA
> process, reschedule retry
> java.lang.RuntimeException: Job failed due to exception Work item not
> found,
> We cannot stop VM[ConsoleProxy|v-10-VM] when it is in state Starting
> 2014-12-03 10:35:52,276 WARN [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-2:ctx-b2a9dc97 work-1) Encountered unhandled exception during HA
> process, reschedule retry
> java.lang.RuntimeException: Job failed due to exception Work item not
> found,
> We cannot stop VM[SecondaryStorageVm|s-15-VM] when it is in state Starting
> 2014-12-03 10:35:52,394 WARN [c.c.c.ClusterManagerImpl]
> (Cluster-Notification-1:ctx-e8ae6c9b) Notifying management server leave
> event took 68 ms
> 2014-12-03 10:35:52,395 WARN [o.a.c.alerts]
> (Cluster-Notification-1:ctx-e8ae6c9b) alertType:: 14 // dataCenterId:: 0
> // podId:: 0 // clusterId:: null // message:: Management server node
> 172.16.0.69 is up
> 2014-12-03 10:35:52,418 WARN [c.c.c.ClusterManagerImpl]
> (Cluster-Notification-1:ctx-e8ae6c9b) Notifying management server join
> event
> took 24 ms
> 2014-12-03 10:36:57,546 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-1:ctx-dada7b54) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:36:58,203 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-2:ctx-41be4ac5) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:37:57,548 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-3:ctx-85d0acb3) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:37:58,204 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-4:ctx-68aea530) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:38:09,839 WARN [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-3:ctx-c510e983 ctx-c0a51a4e) Received unknown parameters for
> command listRegions. Unknown parameters : listall
> 2014-12-03 10:38:18,800 WARN [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-13:ctx-0ca7247c ctx-7869b51b) Received unknown parameters
> for
> command listZones. Unknown parameters : listall
> 2014-12-03 10:38:18,987 WARN [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-18:ctx-85e4fffe ctx-819e52d4) Received unknown parameters
> for
> command listPods. Unknown parameters : listall
> 2014-12-03 10:38:19,118 WARN [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-17:ctx-cb8c1540 ctx-5a4a6061) Received unknown parameters
> for
> command listClusters. Unknown parameters : listall
> 2014-12-03 10:38:19,244 WARN [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-19:ctx-be72b381 ctx-d0874501) Received unknown parameters
> for
> command listHosts. Unknown parameters : listall
> 2014-12-03 10:38:19,391 WARN [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-21:ctx-e2218038 ctx-eed3052d) Received unknown parameters
> for
> command listStoragePools. Unknown parameters : listall
> 2014-12-03 10:38:19,533 WARN [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-22:ctx-20552766 ctx-83dccf23) Received unknown parameters
> for
> command listImageStores. Unknown parameters : listall type
> 2014-12-03 10:38:19,655 WARN [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-24:ctx-968df022 ctx-d4de7d4e) Received unknown parameters
> for
> command listSystemVms. Unknown parameters : listall
> 2014-12-03 10:38:24,050 WARN [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-7:ctx-193b8617 ctx-c4ab4021) Received unknown parameters for
> command listSystemVms. Unknown parameters : listall
> 2014-12-03 10:38:24,557 WARN [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-11:ctx-0e796a3f ctx-6e24a560) Received unknown parameters
> for
> command listHosts. Unknown parameters : listall
> 2014-12-03 10:38:26,590 WARN [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-3:ctx-c278bf56 ctx-e32c7268) Received unknown parameters for
> command listZones. Unknown parameters : listall
> 2014-12-03 10:38:26,693 WARN [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-12:ctx-1afb82ed ctx-051fbce8) Received unknown parameters
> for
> command listPods. Unknown parameters : listall
> 2014-12-03 10:38:26,781 WARN [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-14:ctx-aadb7b92 ctx-cb2ede4c) Received unknown parameters
> for
> command listClusters. Unknown parameters : listall
> 2014-12-03 10:38:26,878 WARN [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-15:ctx-96f8cd1d ctx-87a1426e) Received unknown parameters
> for
> command listHosts. Unknown parameters : listall
> 2014-12-03 10:38:27,032 WARN [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-16:ctx-3593bc62 ctx-bab3e4e6) Received unknown parameters
> for
> command listStoragePools. Unknown parameters : listall
> 2014-12-03 10:38:27,133 WARN [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-13:ctx-004ef7e9 ctx-528af3a7) Received unknown parameters
> for
> command listImageStores. Unknown parameters : listall type
> 2014-12-03 10:38:27,294 WARN [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-18:ctx-fee8e727 ctx-f817e3df) Received unknown parameters
> for
> command listSystemVms. Unknown parameters : listall
> 2014-12-03 10:38:28,720 WARN [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-5:ctx-4e425560 ctx-dc515ed7) Received unknown parameters for
> command listHosts. Unknown parameters : listall
> 2014-12-03 10:38:57,545 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-5:ctx-3c160491) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:38:58,202 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-6:ctx-f2f80cb3) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:39:57,545 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-2:ctx-954a2d0f) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:39:58,202 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-8:ctx-1a58d0d9) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:40:57,545 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-3:ctx-353674d2) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:40:58,203 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-9:ctx-3761b53b) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:41:57,547 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-4:ctx-c0120a72) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:41:58,203 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-10:ctx-f1fdf064) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:42:57,546 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-6:ctx-efcc44be) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:42:58,203 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-12:ctx-9f51f8e3) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:43:57,547 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-1:ctx-3e62beae) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:43:58,204 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-13:ctx-3893f12e) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> [root@CSTW1 ~]#
>
>
>
> -----Original Message-----
> From: Matthew Midgett [mailto:cloudstck@trick-solutions.com.INVALID]
> Sent: Wednesday, December 03, 2014 6:39 AM
> To: users@cloudstack.apache.org
> Subject: RE: Fresh 4.4.2 - Unable to get status on .... my XenServer
>
> Thanks Geoff I really appreciate the response. I guess that every other
> time
> over the last year I've reinstalled XenServer and didn't remember it. Maybe
> we should add this to the Docs as I don't ever remember seeing it, I could
> be wrong though. I do have the patch management automated and it checks in
> on every reboot. I will have to use Ilo to reinstall the OS today which is
> not that big of a deal. I do admit that I need to PXE boot the servers I
> just hadn't had the time. Yesterday I spent working out a builder for CS
> and
> a network mirror. Then I noticed about 3 hours later that you guys posted a
> public repo for 4.4.2. :)
>
>
> CONFIDENTIAL: This electronic mail, including any attachment(s) may contain
> information that is privileged, confidential, and/or otherwise protected
> from disclosure to anyone other than its intended recipient(s). Any
> dissemination or use of this electronic email or its contents (including
> any
> attachments) by persons other than the intended recipient(s) is strictly
> prohibited. If you have received this message in error, please notify me
> immediately by reply email so that I may correct my internal records.
> Please then delete the original message (including any attachments) in its
> entirety. Thank you.
>
> -----Original Message-----
> From: Matthew Midgett [mailto:cloudstck@trick-solutions.com.INVALID]
> Sent: Wednesday, December 03, 2014 1:11 AM
> To: users@cloudstack.apache.org
> Subject: Fresh 4.4.2 - Unable to get status on .... my xenservers
>
> First I am not sure what is causing this, I have been testing 4.3 and 4.4
> multiple times and didn't notice this problem till middle of last month
> when
> I processed all xenserver updates and then tried to go to 4.4.1. I was
> getting the same error that I am unable to get status of my xenservers. I
> wiped out everything but the xenservers, those I just removed from the pool
> and let them reboot and added them to a new pool. In ACS it says its trying
> to start the SSVM and the Console VM but they stay in the starting state.
> I'm sure it is because of this error. The agent show up in ACS so I don't
> know why it can't talk to the xenservers. I can ping them from the
> management servers.
>
> Last login: Wed Dec 3 00:57:04 2014 from 192.168.255.6
> [root@CSTW1 ~]# grep -i -E 'exception|unable|fail|invalid|leak|warn|error'
> /var/log/cloudstack/management/management-server.log
> 2014-12-03 00:00:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-5:ctx-a0a23e42) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:00:32,966 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-21:ctx-f78a59e4) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:01:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-11:ctx-2e791a18) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:01:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-22:ctx-7bca20e8) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:02:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-12:ctx-28d48099) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:02:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-24:ctx-bc593a63) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:03:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-1:ctx-615b171d) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:03:32,966 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-25:ctx-07f6ba50) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:04:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-13:ctx-c2287fb5) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:04:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-26:ctx-f7d92459) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:05:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-14:ctx-a4e62e4e) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:05:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-28:ctx-01bb91e3) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:06:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-2:ctx-30f576ea) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:06:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-29:ctx-4d6690de) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:07:27,540 ERROR [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-2:ctx-645639e7) Caught this throwable,
> com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
> Caused by: com.cloud.utils.exception.CloudRuntimeException: DB Exception
> on:
> com.mysql.jdbc.JDBC4PreparedStatement@32602693: UPDATE op_ha_work SET
> op_ha_work.taken='2014-12-03 05:07:27',
> op_ha_work.mgmt_server_id=223162458780723 WHERE op_ha_work.id = 5
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock
> found when trying to get lock; try restarting transaction
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1065)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> 2014-12-03 00:07:27,595 WARN [o.a.c.f.j.AsyncJobExecutionContext]
> (HA-Worker-3:ctx-c7c94958 work-5) Job is executed without a context, setup
> psudo job for the executing thread
> 2014-12-03 00:07:27,645 WARN [c.c.u.d.Merovingian2]
> (HA-Worker-3:ctx-c7c94958 work-5) Was unable to find lock for the
> key vm_instance22 and thread id 818653380
> 2014-12-03 00:07:29,425 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Vm-Operations-Cleanup-1:ctx-35d886d3) VM Operations failed due to
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> com.mysql.jdbc.JDBC4PreparedStatement@61807bba: DELETE FROM async_job
> WHERE
> async_job.id= 401
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Cannot delete or update a parent row: a foreign key constraint fails
> (`cloud`.`async_job_join_map`, CONSTRAINT
> `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES
> `async_job` (`id`))
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> 2014-12-03 00:07:30,702 WARN [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-3:ctx-c7c94958 work-5) Encountered unhandled exception during HA
> process, reschedule retry
> java.lang.RuntimeException: Job failed due to exception Work item not
> found,
> We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
> 2014-12-03 00:07:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-15:ctx-c7dd62d6) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:07:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-30:ctx-99d14f4c) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:08:27,621 WARN [o.a.c.f.j.AsyncJobExecutionContext]
> (HA-Worker-2:ctx-94cc502e work-2) Job is executed without a context, setup
> psudo job for the executing thread
> 2014-12-03 00:08:27,678 WARN [c.c.u.d.Merovingian2]
> (HA-Worker-2:ctx-94cc502e work-2) Was unable to find lock for the
> key vm_instance15 and thread id 52609881
> 2014-12-03 00:08:29,421 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Vm-Operations-Cleanup-1:ctx-ed5f2ff6) VM Operations failed due to
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> com.mysql.jdbc.PreparedStatement@15fe9d6d: DELETE FROM async_job WHERE
> async_job.id= 406
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Cannot delete or update a parent row: a foreign key constraint fails
> (`cloud`.`async_job_join_map`, CONSTRAINT
> `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES
> `async_job` (`id`))
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> 2014-12-03 00:08:30,736 WARN [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-2:ctx-94cc502e work-2) Encountered unhandled exception during HA
> process, reschedule retry
> java.lang.RuntimeException: Job failed due to exception Work item not
> found,
> We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
> 2014-12-03 00:08:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-16:ctx-acc87f49) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:08:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-32:ctx-efd0036e) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:09:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-3:ctx-1d3eecd0) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:09:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-33:ctx-3271eeea) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:10:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-17:ctx-27637ed9) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:10:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-34:ctx-b1f98667) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:11:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-18:ctx-45760fd3) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:11:32,966 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-36:ctx-3367d63d) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:12:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-4:ctx-ee2683aa) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:12:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-37:ctx-d0b02603) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:13:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-19:ctx-ce10b0fa) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:13:32,964 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-38:ctx-e7e4ae6b) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:14:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-20:ctx-c158dd2c) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:14:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-40:ctx-7a662bea) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:15:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-5:ctx-0a8f1c90) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:15:32,964 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-41:ctx-4c163515) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:16:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-21:ctx-51c2dfd8) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:16:32,966 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-42:ctx-c670489b) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:17:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-22:ctx-9df53feb) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:17:32,966 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-44:ctx-a921b13b) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:18:29,546 WARN [o.a.c.f.j.AsyncJobExecutionContext]
> (HA-Worker-1:ctx-4fdb5161 work-5) Job is executed without a context, setup
> psudo job for the executing thread
> 2014-12-03 00:18:29,582 WARN [c.c.u.d.Merovingian2]
> (HA-Worker-1:ctx-4fdb5161 work-5) Was unable to find lock for the
> key vm_instance22 and thread id 2132920636
> 2014-12-03 00:18:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-6:ctx-a54d1732) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:18:32,638 WARN [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-1:ctx-4fdb5161 work-5) Encountered unhandled exception during HA
> process, reschedule retry
> java.lang.RuntimeException: Job failed due to exception Work item not
> found,
> We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
> 2014-12-03 00:18:32,964 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-45:ctx-5d1c0c7e) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:19:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Unable to
> transition the state but we're moving on because it's forced stop
> 2014-12-03 00:19:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Unable to
> cleanup VM: VM[ConsoleProxy|v-15-VM] ,since outstanding work item is not
> found
> 2014-12-03 00:19:30,871 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Invocation
> exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
> item not found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state
> Starting
> 2014-12-03 00:19:30,872 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Rethrow
> exception com.cloud.utils.exception.CloudRuntimeException: Work item not
> found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
> 2014-12-03 00:19:30,872 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-3:ctx-d59db286 job-408/job-411) Unable to
> complete AsyncJobVO {id:411, userId: 1, accountId: 1, instanceType: null,
> instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
>
> rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
>
> b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
>
> TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
> AAAAD3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
> IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
> 68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
> created: Wed Dec 03 00:19:02 EST 2014}, job origin:408
> com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
> cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
> 2014-12-03 00:19:30,874 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-3:ctx-d59db286 job-408/job-411) Complete async job-411,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAdUpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
>
> cCBWTVtDb25zb2xlUHJveHl8di0xNS1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFydGluZ3Vy
>
> AB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZh
>
> LmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJp
>
> bmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNv
>
> bS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZh
>
> dAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9i
>
> cy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQA
>
> DHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
>
> b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUu
>
> amF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250
>
> ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQu
>
> amF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29u
>
> dGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EA
>
> fgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hl
>
> LmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEA
>
> fgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3Vy
>
> cmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4A
>
> CwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh
>
> cQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y
>
> dAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51
>
> dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsA
>
> AALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNv
>
> bGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZh
>
> LnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QA
>
> FkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0D
> AAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
> 2014-12-03 00:19:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-23:ctx-0414dcdd) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:19:32,966 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-46:ctx-fc02b226) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:20:32,282 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-24:ctx-821b5aca) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:20:32,966 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-48:ctx-70dbfde8) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:21:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-1:ctx-de7b1636) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:21:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-49:ctx-154bac45) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:22:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-25:ctx-5c37a417) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:22:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-50:ctx-a34d4acf) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:23:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-52:ctx-4ee4cdd0) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:23:32,964 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-7:ctx-de025548) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:24:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-53:ctx-198b066a) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:24:32,966 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-27:ctx-e4b78a1b) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:25:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-14:ctx-d0466cfa) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:25:32,964 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-55:ctx-6c330223) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:26:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-2:ctx-98d6b222) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:26:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-57:ctx-031552f0) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:27:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-29:ctx-a436d05f) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:27:32,966 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-58:ctx-0f3496a2) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:28:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-15:ctx-40cf6a78) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:28:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-59:ctx-1537f233) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:29:30,874 ERROR [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-0:ctx-84af6112) Caught this throwable,
> com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to
> commit
> or close the connection.
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock
> found when trying to get lock; try restarting transaction
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1065)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> 2014-12-03 00:29:30,881 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Unable to
> transition the state but we're moving on because it's forced stop
> 2014-12-03 00:29:30,882 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Unable to
> cleanup VM: VM[SecondaryStorageVm|s-22-VM] ,since outstanding work item is
> not found
> 2014-12-03 00:29:30,882 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Invocation
> exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
> item not found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in
> state Starting
> 2014-12-03 00:29:30,883 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Rethrow
> exception com.cloud.utils.exception.CloudRuntimeException: Work item not
> found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state
> Starting
> 2014-12-03 00:29:30,883 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-4:ctx-b25792e9 job-414/job-417) Unable to
> complete AsyncJobVO {id:417, userId: 1, accountId: 1, instanceType: null,
> instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
>
> rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
>
> b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
>
> TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
> AAAAFnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
> IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
> 68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
> created: Wed Dec 03 00:29:02 EST 2014}, job origin:414
> com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
> cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
> 2014-12-03 00:29:30,896 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-4:ctx-b25792e9 job-414/job-417) Complete async job-417,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAe0pvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
>
> cCBWTVtTZWNvbmRhcnlTdG9yYWdlVm18cy0yMi1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFy
>
> dGluZ3VyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNy
>
> ABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5k
>
> ZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAA
>
> cnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hl
>
> ci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdv
>
> cmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwu
>
> amF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFu
>
> YWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVu
>
> bmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdl
>
> ZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENv
>
> bnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFn
>
> ZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250
>
> ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcu
>
> YXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVx
>
> AH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwu
>
> Y29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4A
>
> G3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFz
>
> ay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4
>
> ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAu
>
> amF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdz
>
> cQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51
>
> dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhy
>
> ACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgAB
>
> TAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHS
> HZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
> 2014-12-03 00:29:30,919 WARN [o.a.c.f.j.AsyncJobExecutionContext]
> (HA-Worker-4:ctx-10fea75f work-2) Job is executed without a context, setup
> psudo job for the executing thread
> 2014-12-03 00:29:30,967 WARN [c.c.u.d.Merovingian2]
> (HA-Worker-4:ctx-10fea75f work-2) Was unable to find lock for the
> key vm_instance15 and thread id 139852089
> 2014-12-03 00:29:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-8:ctx-bc77e0da) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:29:32,966 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-61:ctx-de44c2a9) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:29:34,025 WARN [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-4:ctx-10fea75f work-2) Encountered unhandled exception during HA
> process, reschedule retry
> java.lang.RuntimeException: Job failed due to exception Work item not
> found,
> We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
> 2014-12-03 00:30:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-31:ctx-8417fc60) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:30:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-62:ctx-d118067c) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:31:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-16:ctx-393a2cde) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:31:32,966 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-63:ctx-fd5d5e54) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:32:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-3:ctx-07ab5550) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:32:32,966 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-65:ctx-0345383e) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:33:32,286 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-33:ctx-60af4723) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:33:32,966 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-66:ctx-0b95563b) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:34:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-17:ctx-8e9c36a2) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:34:32,966 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-67:ctx-397176fb) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:35:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-9:ctx-3c967d9b) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:35:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-69:ctx-8bc1cfaf) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:36:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-35:ctx-8186e45f) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:36:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-70:ctx-557403b1) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:37:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-18:ctx-8539dc77) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:37:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-71:ctx-61ea3221) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:38:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-4:ctx-0dc9a587) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:38:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-73:ctx-7fba328c) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:39:31,006 WARN [o.a.c.f.j.AsyncJobExecutionContext]
> (HA-Worker-0:ctx-7a356c50 work-5) Job is executed without a context, setup
> psudo job for the executing thread
> 2014-12-03 00:39:31,051 WARN [c.c.u.d.Merovingian2]
> (HA-Worker-0:ctx-7a356c50 work-5) Was unable to find lock for the
> key vm_instance22 and thread id 84459079
> 2014-12-03 00:39:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-37:ctx-bc7a8f81) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:39:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-74:ctx-ebe4657b) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:39:34,105 WARN [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-0:ctx-7a356c50 work-5) Encountered unhandled exception during HA
> process, reschedule retry
> java.lang.RuntimeException: Job failed due to exception Work item not
> found,
> We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
> 2014-12-03 00:40:30,865 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Unable to
> transition the state but we're moving on because it's forced stop
> 2014-12-03 00:40:30,866 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Unable to
> cleanup VM: VM[ConsoleProxy|v-15-VM] ,since outstanding work item is not
> found
> 2014-12-03 00:40:30,866 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Invocation
> exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
> item not found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state
> Starting
> 2014-12-03 00:40:30,866 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Rethrow
> exception com.cloud.utils.exception.CloudRuntimeException: Work item not
> found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
> 2014-12-03 00:40:30,866 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-5:ctx-32edffa5 job-423/job-426) Unable to
> complete AsyncJobVO {id:426, userId: 1, accountId: 1, instanceType: null,
> instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
>
> rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
>
> b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
>
> TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
> AAAAD3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
> IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
> 68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
> created: Wed Dec 03 00:40:02 EST 2014}, job origin:423
> com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
> cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
> 2014-12-03 00:40:30,868 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-5:ctx-32edffa5 job-423/job-426) Complete async job-426,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAdUpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
>
> cCBWTVtDb25zb2xlUHJveHl8di0xNS1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFydGluZ3Vy
>
> AB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZh
>
> LmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJp
>
> bmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNv
>
> bS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZh
>
> dAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9i
>
> cy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQA
>
> DHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
>
> b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUu
>
> amF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250
>
> ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQu
>
> amF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29u
>
> dGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EA
>
> fgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hl
>
> LmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEA
>
> fgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3Vy
>
> cmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4A
>
> CwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh
>
> cQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y
>
> dAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51
>
> dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsA
>
> AALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNv
>
> bGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZh
>
> LnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QA
>
> FkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0D
> AAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
> 2014-12-03 00:40:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-19:ctx-338898ef) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:40:32,966 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-75:ctx-790ca221) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:41:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-10:ctx-0e2f52eb) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:41:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-77:ctx-f8287f50) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:42:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-39:ctx-1af09356) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:42:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-78:ctx-e1b2675f) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:43:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-20:ctx-c9aea423) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:43:32,964 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-79:ctx-c459450e) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:44:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-5:ctx-f7fee74e) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:44:32,966 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-81:ctx-bc615860) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:45:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-41:ctx-4f287927) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:45:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-82:ctx-092eb488) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:46:27,281 WARN [o.a.c.alerts] (HA-1:ctx-b34246c5)
> alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null //
> message:: No usage server process running
> 2014-12-03 00:46:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-21:ctx-62eca8f2) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:46:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-83:ctx-272e6406) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:47:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-11:ctx-23f28317) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:47:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-85:ctx-905ed14b) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:48:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-43:ctx-4a673855) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:48:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-86:ctx-1285bc0f) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:49:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-22:ctx-60a35d36) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:49:32,964 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-87:ctx-687282ce) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:50:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Unable to
> transition the state but we're moving on because it's forced stop
> 2014-12-03 00:50:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Unable to
> cleanup VM: VM[SecondaryStorageVm|s-22-VM] ,since outstanding work item is
> not found
> 2014-12-03 00:50:30,871 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Invocation
> exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
> item not found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in
> state Starting
> 2014-12-03 00:50:30,871 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Rethrow
> exception com.cloud.utils.exception.CloudRuntimeException: Work item not
> found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state
> Starting
> 2014-12-03 00:50:30,871 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-6:ctx-46fe9139 job-429/job-432) Unable to
> complete AsyncJobVO {id:432, userId: 1, accountId: 1, instanceType: null,
> instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
>
> rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
>
> b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
>
> TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
> AAAAFnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
> IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
> 68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
> created: Wed Dec 03 00:50:02 EST 2014}, job origin:429
> com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
> cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
> 2014-12-03 00:50:30,873 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-6:ctx-46fe9139 job-429/job-432) Complete async job-432,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAe0pvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
>
> cCBWTVtTZWNvbmRhcnlTdG9yYWdlVm18cy0yMi1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFy
>
> dGluZ3VyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNy
>
> ABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5k
>
> ZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAA
>
> cnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hl
>
> ci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdv
>
> cmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwu
>
> amF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFu
>
> YWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVu
>
> bmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdl
>
> ZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENv
>
> bnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFn
>
> ZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250
>
> ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcu
>
> YXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVx
>
> AH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwu
>
> Y29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4A
>
> G3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFz
>
> ay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4
>
> ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAu
>
> amF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdz
>
> cQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51
>
> dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhy
>
> ACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgAB
>
> TAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHS
> HZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
> 2014-12-03 00:50:31,095 WARN [o.a.c.f.j.AsyncJobExecutionContext]
> (HA-Worker-3:ctx-0dbe91b2 work-2) Job is executed without a context, setup
> psudo job for the executing thread
> 2014-12-03 00:50:31,129 WARN [c.c.u.d.Merovingian2]
> (HA-Worker-3:ctx-0dbe91b2 work-2) Was unable to find lock for the
> key vm_instance15 and thread id 818653380
> 2014-12-03 00:50:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-6:ctx-2758188b) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:50:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-89:ctx-eb95a1c2) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:50:34,182 WARN [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-3:ctx-0dbe91b2 work-2) Encountered unhandled exception during HA
> process, reschedule retry
> java.lang.RuntimeException: Job failed due to exception Work item not
> found,
> We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
> 2014-12-03 00:51:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-45:ctx-7be8bae6) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:51:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-90:ctx-3f76c1d7) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:52:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-23:ctx-e902ca80) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:52:32,964 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-91:ctx-4d56948f) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:53:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-12:ctx-0876aaa4) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:53:32,964 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-93:ctx-c057ee22) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:54:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-47:ctx-8083fccf) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:54:32,964 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-94:ctx-56cebf3c) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:55:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-24:ctx-36c965de) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:55:32,964 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-95:ctx-86b1db16) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:56:27,241 WARN [o.a.c.alerts] (HA-2:ctx-627e96a3)
> alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null //
> message:: No usage server process running
> 2014-12-03 00:56:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-1:ctx-6000a728) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:56:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-97:ctx-ffbe388e) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:57:32,279 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-49:ctx-a54e4fa0) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:57:32,964 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-98:ctx-34e48d1f) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:58:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-25:ctx-ffdd759e) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:58:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-99:ctx-0c1a9fa7) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:59:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-101:ctx-8442d937) Unable to get current
> status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:59:32,963 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-51:ctx-9bd2b3cc) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 01:00:31,170 WARN [o.a.c.f.j.AsyncJobExecutionContext]
> (HA-Worker-2:ctx-f17ce3b5 work-5) Job is executed without a context, setup
> psudo job for the executing thread
> 2014-12-03 01:00:31,208 WARN [c.c.u.d.Merovingian2]
> (HA-Worker-2:ctx-f17ce3b5 work-5) Was unable to find lock for the
> key vm_instance22 and thread id 52609881
> 2014-12-03 01:00:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-102:ctx-138c6123) Unable to get current
> status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 01:00:32,964 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-26:ctx-3e13656a) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 01:00:34,263 WARN [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-2:ctx-f17ce3b5 work-5) Encountered unhandled exception during HA
> process, reschedule retry
> java.lang.RuntimeException: Job failed due to exception Work item not
> found,
> We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
> 2014-12-03 01:01:32,280 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-103:ctx-334d11fc) Unable to get current
> status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 01:01:32,856 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Unable to
> transition the state but we're moving on because it's forced stop
> 2014-12-03 01:01:32,856 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Unable to
> cleanup VM: VM[ConsoleProxy|v-15-VM] ,since outstanding work item is not
> found
> 2014-12-03 01:01:32,856 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Invocation
> exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
> item not found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state
> Starting
> 2014-12-03 01:01:32,857 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Rethrow
> exception com.cloud.utils.exception.CloudRuntimeException: Work item not
> found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
> 2014-12-03 01:01:32,857 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-7:ctx-0a1f299d job-429/job-438) Unable to
> complete AsyncJobVO {id:438, userId: 1, accountId: 1, instanceType: null,
> instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
>
> rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
>
> b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
>
> TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
> AAAAD3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
> IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
> 68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
> created: Wed Dec 03 01:01:03 EST 2014}, job origin:429
> com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
> cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
> 2014-12-03 01:01:32,865 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-7:ctx-0a1f299d job-429/job-438) Complete async job-438,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAdUpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
>
> cCBWTVtDb25zb2xlUHJveHl8di0xNS1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFydGluZ3Vy
>
> AB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZh
>
> LmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJp
>
> bmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNv
>
> bS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZh
>
> dAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9i
>
> cy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQA
>
> DHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
>
> b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUu
>
> amF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250
>
> ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQu
>
> amF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29u
>
> dGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EA
>
> fgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hl
>
> LmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEA
>
> fgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3Vy
>
> cmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4A
>
> CwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh
>
> cQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y
>
> dAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51
>
> dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsA
>
> AALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNv
>
> bGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZh
>
> LnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QA
>
> FkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0D
> AAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
> 2014-12-03 01:01:32,964 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-52:ctx-76332a49) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 01:02:32,282 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-105:ctx-b73de834) Unable to get current
> status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 01:02:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-53:ctx-9b938dc5) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 01:03:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-27:ctx-28735fd7) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 01:03:32,967 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-107:ctx-e290c580) Unable to get current
> status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 01:04:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-54:ctx-a53679d7) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 01:04:32,964 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-108:ctx-e1d3829f) Unable to get current
> status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 01:05:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-55:ctx-65e26d72) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 01:05:32,965 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-110:ctx-b755e6ac) Unable to get current
> status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 01:06:27,239 WARN [o.a.c.alerts] (HA-3:ctx-b686e09d)
> alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null //
> message:: No usage server process running
> 2014-12-03 01:06:32,281 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-28:ctx-230f4b8f) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 01:06:32,966 WARN [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-111:ctx-77cd282d) Unable to get current
> status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> [root@CSTW1 ~]#
>
>
>
>
RE: Fresh 4.4.2 - Unable to get status on .... my XenServer
Posted by Matthew Midgett <cl...@trick-solutions.com.INVALID>.
I just finished installing XenServer again and started from a setup and I'm
still getting the same errors that I was before about not being able to get
a status. What to do now?
JrSm9iRGlzcGF0Y2hlcnQAGFZtV29ya0pvYkRpc3BhdGNoZXIuamF2YXQABnJ1bkpvYnNxAH4ACw
AAAfd0AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk
1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRzcQ
B-AAsAAAAxdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ2
9udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5zcQB-AA
sAAAA4dABCb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bH
RNYW5hZ2VkQ29udGV4dCQxdAAaRGVmYXVsdE1hbmFnZWRDb250ZXh0LmphdmF0AARjYWxsc3EAfg
ALAAAAZ3QAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdW
x0TWFuYWdlZENvbnRleHRxAH4AGnQAD2NhbGxXaXRoQ29udGV4dHNxAH4ACwAAADVxAH4AHXEAfg
AadAAOcnVuV2l0aENvbnRleHRzcQB-AAsAAAAudAA8b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbm
FnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlcQB-ABZxAH4AF3NxAH4ACwAAAcxxAH
4AEXEAfgAScQB-ABdzcQB-AAsAAAHXdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuRXhlY3V0b3JzJF
J1bm5hYmxlQWRhcHRlcnQADkV4ZWN1dG9ycy5qYXZhcQB-ABtzcQB-AAsAAAEGdAAfamF2YS51dG
lsLmNvbmN1cnJlbnQuRnV0dXJlVGFza3QAD0Z1dHVyZVRhc2suamF2YXEAfgAXc3EAfgALAAAEeX
QAJ2phdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvcnQAF1RocmVhZFBvb2xFeG
VjdXRvci5qYXZhdAAJcnVuV29ya2Vyc3EAfgALAAACZ3QALmphdmEudXRpbC5jb25jdXJyZW50Ll
RocmVhZFBvb2xFeGVjdXRvciRXb3JrZXJxAH4ALHEAfgAXc3EAfgALAAAC6XQAEGphdmEubGFuZy
5UaHJlYWR0AAtUaHJlYWQuamF2YXEAfgAXc3IAJmphdmEudXRpbC5Db2xsZWN0aW9ucyRVbm1vZG
lmaWFibGVMaXN0_A8lMbXsjhACAAFMAARsaXN0cQB-AAZ4cgAsamF2YS51dGlsLkNvbGxlY3Rpb2
5zJFVubW9kaWZpYWJsZUNvbGxlY3Rpb24ZQgCAy173HgIAAUwAAWN0ABZMamF2YS91dGlsL0NvbG
xlY3Rpb247eHBzcgATamF2YS51dGlsLkFycmF5TGlzdHiB0h2Zx2GdAwABSQAEc2l6ZXhwAAAAAH
cEAAAAAHhxAH4AOHg
2014-12-03 10:28:23,558 WARN [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-2d3ea815) Exception while trying
to start secondary storage vm
java.lang.RuntimeException: Job failed due to exception Resource [Host:4] is
unreachable: Host 4: Unable to start instance due to DB Exception on:
com.mysql.jdbc.JDBC4PreparedStatement@7268fbbc: UPDATE
op_dc_link_local_ip_address_alloc SET
op_dc_link_local_ip_address_alloc.nic_id=123,
op_dc_link_local_ip_address_alloc.reservation_id=_binary'97c33776-2f60-494b-
8f3c-7b20a0d7881f', op_dc_link_local_ip_address_alloc.taken='2014-12-03
15:28:23' WHERE op_dc_link_local_ip_address_alloc.id = 1806
2014-12-03 10:28:23,562 INFO [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-2d3ea815) Unable to start secondary storage vm for standby
capacity, secStorageVm vm Id : 13, will recycle it and start a new one
2014-12-03 10:28:23,697 WARN [c.c.u.d.Merovingian2]
(secstorage-1:ctx-00d2f7e3) Was unable to find lock for the key
vm_instance13 and thread id 819805219
2014-12-03 10:28:37,991 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-15:ctx-077ec0bf) Unable to get current status on
1(xenhv2.pvdc.chlt.charlottecolo.com)
2014-12-03 10:28:56,362 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-30:ctx-7833f032) Unable to get current status on
4(xenhv1.pvdc.chlt.charlottecolo.com)
2014-12-03 10:29:37,990 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-8:ctx-40da93df) Unable to get current status on
1(xenhv2.pvdc.chlt.charlottecolo.com)
2014-12-03 10:29:52,876 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-1:ctx-db89f3ef) Task (job-393) has been pending for
90 seconds
2014-12-03 10:29:56,361 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-61:ctx-80573bda) Unable to get current status on
4(xenhv1.pvdc.chlt.charlottecolo.com)
2014-12-03 10:30:02,536 DEBUG [o.a.c.s.i.s.TemplateObject]
(Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) failed to
process event and answer
com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
com.mysql.jdbc.JDBC4PreparedStatement@571a2d18: SELECT
template_spool_ref.id, template_spool_ref.pool_id,
template_spool_ref.template_id, template_spool_ref.created,
template_spool_ref.last_updated, template_spool_ref.download_pct,
template_spool_ref.download_state, template_spool_ref.local_path,
template_spool_ref.error_str, template_spool_ref.job_id,
template_spool_ref.install_path, template_spool_ref.template_size,
template_spool_ref.marked_for_gc, template_spool_ref.update_count,
template_spool_ref.updated, template_spool_ref.state FROM template_spool_ref
WHERE template_spool_ref.pool_id = 2 AND template_spool_ref.template_id = 1
ORDER BY RAND() LIMIT 1
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
Caused by: java.io.EOFException: Can not read response from server. Expected
to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2014-12-03 10:30:02,560 DEBUG [o.a.c.s.v.VolumeServiceImpl]
(Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) failed to
create template on storage
java.lang.RuntimeException: InvocationTargetException when invoking RPC
callback for command: copyBaseImageCallback
Caused by: java.lang.reflect.InvocationTargetException
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to
process event
Caused by: com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
com.mysql.jdbc.JDBC4PreparedStatement@571a2d18: SELECT
template_spool_ref.id, template_spool_ref.pool_id,
template_spool_ref.template_id, template_spool_ref.created,
template_spool_ref.last_updated, template_spool_ref.download_pct,
template_spool_ref.download_state, template_spool_ref.local_path,
template_spool_ref.error_str, template_spool_ref.job_id,
template_spool_ref.install_path, template_spool_ref.template_size,
template_spool_ref.marked_for_gc, template_spool_ref.update_count,
template_spool_ref.updated, template_spool_ref.state FROM template_spool_ref
WHERE template_spool_ref.pool_id = 2 AND template_spool_ref.template_id = 1
ORDER BY RAND() LIMIT 1
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
Caused by: java.io.EOFException: Can not read response from server. Expected
to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2014-12-03 10:30:02,568 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
(Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) Template 1
is not found on storage pool 2, so no need to delete
2014-12-03 10:30:02,570 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) Failed to
start instance VM[ConsoleProxy|v-10-VM]
com.cloud.utils.exception.CloudRuntimeException: can't find mapping in
ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
2014-12-03 10:30:02,693 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) VM state
transitted from :Starting to Stopped with event: OperationFailedvm's
original host id: null new host id: null host id before state transition: 1
2014-12-03 10:30:02,721 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) Invocation
exception, caused by: com.cloud.exception.AgentUnavailableException:
Resource [Host:1] is unreachable: Host 1: Unable to start instance due to
can't find mapping in ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
2014-12-03 10:30:02,721 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) Rethrow
exception com.cloud.exception.AgentUnavailableException: Resource [Host:1]
is unreachable: Host 1: Unable to start instance due to can't find mapping
in ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
2014-12-03 10:30:02,722 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-61:ctx-927aa31e job-388/job-393) Unable to complete
AsyncJobVO {id:393, userId: 1, accountId: 1, instanceType: null, instanceId:
null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAB
AAAAAAAAAAEAAAAAAAAACnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBw
cHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
0, result: null,
initMsid: 223162458780723, completeMsid: null, lastUpdated: null,
lastPolled: null, created: Wed Dec 03 10:28:21 EST 2014}, job origin:388
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
unreachable: Host 1: Unable to start instance due to can't find mapping in
ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
Caused by: com.cloud.utils.exception.CloudRuntimeException: can't find
mapping in ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
2014-12-03 10:30:02,723 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-61:ctx-927aa31e job-388/job-393) Complete async job-393,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QA2kpvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBSZXNvdXJjZSBbSG9zdDoxXSBpcyB1bnJlYWNoYWJsZTog
SG9zdCAxOiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgZHVlIHRvIGNhbid0IGZpbmQgbWFwcGlu
ZyBpbiBPYmplY3RJbkRhdGFTdG9yZSB0YWJsZSBmb3I6IG9yZy5hcGFjaGUuY2xvdWRzdGFjay5z
dG9yYWdlLmltYWdlLnN0b3JlLlRlbXBsYXRlT2JqZWN0QDc4NWRlMjlidXIAHltMamF2YS5sYW5n
LlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_SI5AgAAeHAAAAANc3IAG2phdmEubGFuZy5TdGFja1Ry
YWNlRWxlbWVudGEJxZomNt2FAgAESQAKbGluZU51bWJlckwADmRlY2xhcmluZ0NsYXNzcQB-AARM
AAhmaWxlTmFtZXEAfgAETAAKbWV0aG9kTmFtZXEAfgAEeHAAAABydAAgY29tLmNsb3VkLnZtLlZt
V29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JzcQB-
AAsAAAH3dAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNK
b2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0
c3EAfgALAAAAMXQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdl
ZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVuc3EA
fgALAAAAOHQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZh
dWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHNx
AH4ACwAAAGd0AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVm
YXVsdE1hbmFnZWRDb250ZXh0cQB-ABp0AA9jYWxsV2l0aENvbnRleHRzcQB-AAsAAAA1cQB-AB1x
AH4AGnQADnJ1bldpdGhDb250ZXh0c3EAfgALAAAALnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5t
YW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAWcQB-ABdzcQB-AAsAAAHM
cQB-ABFxAH4AEnEAfgAXc3EAfgALAAAB13QALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9y
cyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAbc3EAfgALAAABBnQAH2phdmEu
dXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4AF3NxAH4ACwAA
BHl0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29s
RXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnNxAH4ACwAAAmd0AC5qYXZhLnV0aWwuY29uY3VycmVu
dC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-ACxxAH4AF3NxAH4ACwAAAul0ABBqYXZhLmxh
bmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4AF3NyACZqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5t
b2RpZmlhYmxlTGlzdPwPJTG17I4QAgABTAAEbGlzdHEAfgAGeHIALGphdmEudXRpbC5Db2xsZWN0
aW9ucyRVbm1vZGlmaWFibGVDb2xsZWN0aW9uGUIAgMte9x4CAAFMAAFjdAAWTGphdmEvdXRpbC9D
b2xsZWN0aW9uO3hwc3IAE2phdmEudXRpbC5BcnJheUxpc3R4gdIdmcdhnQMAAUkABHNpemV4cAAA
AAB3BAAAAAB4cQB-ADh4
2014-12-03 10:30:02,768 ERROR [c.c.u.d.DbUtil] (consoleproxy-1:ctx-8e76fad8)
RELEASE_LOCK() throws exception
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link
failure
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
Caused by: java.io.EOFException: Can not read response from server. Expected
to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2014-12-03 10:30:02,770 WARN [c.c.v.SystemVmLoadScanner]
(consoleproxy-1:ctx-8e76fad8) Unexpected exception Job failed due to
exception Resource [Host:1] is unreachable: Host 1: Unable to start instance
due to can't find mapping in ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
java.lang.RuntimeException: Job failed due to exception Resource [Host:1] is
unreachable: Host 1: Unable to start instance due to can't find mapping in
ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
2014-12-03 10:30:02,861 WARN [c.c.u.d.Merovingian2]
(consoleproxy-1:ctx-c282e93d) Was unable to find lock for the key
vm_instance10 and thread id 117755041
2014-12-03 10:30:05,371 INFO [o.a.c.s.v.VolumeServiceImpl]
(Work-Job-Executor-62:ctx-21f67d3c job-384/job-394 ctx-55123741) Unable to
acquire lock on VMTemplateStoragePool 54
2014-12-03 10:30:05,375 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-62:ctx-21f67d3c job-384/job-394 ctx-55123741) Failed to
start instance VM[SecondaryStorageVm|s-13-VM]
com.cloud.utils.exception.CloudRuntimeException: Unable to acquire lock on
VMTemplateStoragePool: 54
2014-12-03 10:30:05,527 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-62:ctx-21f67d3c job-384/job-394 ctx-55123741) VM state
transitted from :Starting to Stopped with event: OperationFailedvm's
original host id: null new host id: null host id before state transition: 4
2014-12-03 10:30:05,551 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-62:ctx-21f67d3c job-384/job-394 ctx-55123741) Invocation
exception, caused by: com.cloud.exception.AgentUnavailableException:
Resource [Host:4] is unreachable: Host 4: Unable to start instance due to
Unable to acquire lock on VMTemplateStoragePool: 54
2014-12-03 10:30:05,551 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-62:ctx-21f67d3c job-384/job-394 ctx-55123741) Rethrow
exception com.cloud.exception.AgentUnavailableException: Resource [Host:4]
is unreachable: Host 4: Unable to start instance due to Unable to acquire
lock on VMTemplateStoragePool: 54
2014-12-03 10:30:05,551 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-62:ctx-21f67d3c job-384/job-394) Unable to complete
AsyncJobVO {id:394, userId: 1, accountId: 1, instanceType: null, instanceId:
null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAB
AAAAAAAAAAEAAAAAAAAADXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBw
cHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
0, result: null,
initMsid: 223162458780723, completeMsid: null, lastUpdated: null,
lastPolled: null, created: Wed Dec 03 10:28:23 EST 2014}, job origin:384
com.cloud.exception.AgentUnavailableException: Resource [Host:4] is
unreachable: Host 4: Unable to start instance due to Unable to acquire lock
on VMTemplateStoragePool: 54
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to
acquire lock on VMTemplateStoragePool: 54
2014-12-03 10:30:05,553 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-62:ctx-21f67d3c job-384/job-394) Complete async job-394,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAmUpvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBSZXNvdXJjZSBbSG9zdDo0XSBpcyB1bnJlYWNoYWJsZTog
SG9zdCA0OiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgZHVlIHRvIFVuYWJsZSB0byBhY3F1aXJl
IGxvY2sgb24gVk1UZW1wbGF0ZVN0b3JhZ2VQb29sOiA1NHVyAB5bTGphdmEubGFuZy5TdGFja1Ry
YWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1l
bnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5h
bWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JE
aXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QA
P29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdl
ckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAA
ADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0
AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFn
ZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABn
dABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5h
Z2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5y
dW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-
ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFi
bGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29u
Y3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2
YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9y
LmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFk
UG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVh
ZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJs
ZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5t
b2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlv
bjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAA
eHEAfgA4eA
2014-12-03 10:30:05,596 WARN [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-00d2f7e3) Exception while trying
to start secondary storage vm
java.lang.RuntimeException: Job failed due to exception Resource [Host:4] is
unreachable: Host 4: Unable to start instance due to Unable to acquire lock
on VMTemplateStoragePool: 54
2014-12-03 10:30:05,597 ERROR [c.c.u.d.DbUtil] (secstorage-1:ctx-00d2f7e3)
RELEASE_LOCK() throws exception
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link
failure
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
Caused by: java.io.EOFException: Can not read response from server. Expected
to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2014-12-03 10:30:05,599 INFO [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-00d2f7e3) Unable to start secondary storage vm for standby
capacity, secStorageVm vm Id : 13, will recycle it and start a new one
2014-12-03 10:30:05,663 WARN [c.c.u.d.Merovingian2]
(secstorage-1:ctx-00d2f7e3) Was unable to find lock for the key
vm_instance13 and thread id 819805219
2014-12-03 10:30:07,062 ERROR [c.c.u.d.DbUtil] (secstorage-1:ctx-00d2f7e3)
RELEASE_LOCK() throws exception
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link
failure
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
Caused by: java.io.EOFException: Can not read response from server. Expected
to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2014-12-03 10:30:07,459 WARN [c.c.u.d.Merovingian2]
(secstorage-1:ctx-5b27f0f0) Was unable to find lock for the key
vm_instance14 and thread id 819805219
2014-12-03 10:30:37,990 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-31:ctx-37ebcabd) Unable to get current status on
1(xenhv2.pvdc.chlt.charlottecolo.com)
2014-12-03 10:30:56,361 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-62:ctx-f52b8d19) Unable to get current status on
4(xenhv1.pvdc.chlt.charlottecolo.com)
2014-12-03 10:31:37,991 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-16:ctx-4133ae03) Unable to get current status on
1(xenhv2.pvdc.chlt.charlottecolo.com)
2014-12-03 10:31:46,486 DEBUG [o.a.c.s.i.s.TemplateObject]
(Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) failed to
process event and answer
com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
com.mysql.jdbc.JDBC4PreparedStatement@311d349d: SELECT
template_spool_ref.id, template_spool_ref.pool_id,
template_spool_ref.template_id, template_spool_ref.created,
template_spool_ref.last_updated, template_spool_ref.download_pct,
template_spool_ref.download_state, template_spool_ref.local_path,
template_spool_ref.error_str, template_spool_ref.job_id,
template_spool_ref.install_path, template_spool_ref.template_size,
template_spool_ref.marked_for_gc, template_spool_ref.update_count,
template_spool_ref.updated, template_spool_ref.state FROM template_spool_ref
WHERE template_spool_ref.pool_id = 2 AND template_spool_ref.template_id = 1
ORDER BY RAND() LIMIT 1
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
Caused by: java.io.EOFException: Can not read response from server. Expected
to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2014-12-03 10:31:46,510 DEBUG [o.a.c.s.v.VolumeServiceImpl]
(Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) failed to
create template on storage
java.lang.RuntimeException: InvocationTargetException when invoking RPC
callback for command: copyBaseImageCallback
Caused by: java.lang.reflect.InvocationTargetException
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to
process event
Caused by: com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
com.mysql.jdbc.JDBC4PreparedStatement@311d349d: SELECT
template_spool_ref.id, template_spool_ref.pool_id,
template_spool_ref.template_id, template_spool_ref.created,
template_spool_ref.last_updated, template_spool_ref.download_pct,
template_spool_ref.download_state, template_spool_ref.local_path,
template_spool_ref.error_str, template_spool_ref.job_id,
template_spool_ref.install_path, template_spool_ref.template_size,
template_spool_ref.marked_for_gc, template_spool_ref.update_count,
template_spool_ref.updated, template_spool_ref.state FROM template_spool_ref
WHERE template_spool_ref.pool_id = 2 AND template_spool_ref.template_id = 1
ORDER BY RAND() LIMIT 1
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
Caused by: java.io.EOFException: Can not read response from server. Expected
to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2014-12-03 10:31:46,517 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
(Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) Template 1
is not found on storage pool 2, so no need to delete
2014-12-03 10:31:46,519 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) Failed to
start instance VM[ConsoleProxy|v-10-VM]
com.cloud.utils.exception.CloudRuntimeException: can't find mapping in
ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
2014-12-03 10:31:46,617 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) VM state
transitted from :Starting to Stopped with event: OperationFailedvm's
original host id: null new host id: null host id before state transition: 1
2014-12-03 10:31:46,642 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) Invocation
exception, caused by: com.cloud.exception.AgentUnavailableException:
Resource [Host:1] is unreachable: Host 1: Unable to start instance due to
can't find mapping in ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
2014-12-03 10:31:46,642 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) Rethrow
exception com.cloud.exception.AgentUnavailableException: Resource [Host:1]
is unreachable: Host 1: Unable to start instance due to can't find mapping
in ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
2014-12-03 10:31:46,642 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396) Unable to complete
AsyncJobVO {id:396, userId: 1, accountId: 1, instanceType: null, instanceId:
null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAB
AAAAAAAAAAEAAAAAAAAACnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBw
cHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
0, result: null,
initMsid: 223162458780723, completeMsid: null, lastUpdated: null,
lastPolled: null, created: Wed Dec 03 10:30:02 EST 2014}, job origin:388
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
unreachable: Host 1: Unable to start instance due to can't find mapping in
ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
Caused by: com.cloud.utils.exception.CloudRuntimeException: can't find
mapping in ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
2014-12-03 10:31:46,644 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396) Complete async job-396,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QA2kpvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBSZXNvdXJjZSBbSG9zdDoxXSBpcyB1bnJlYWNoYWJsZTog
SG9zdCAxOiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgZHVlIHRvIGNhbid0IGZpbmQgbWFwcGlu
ZyBpbiBPYmplY3RJbkRhdGFTdG9yZSB0YWJsZSBmb3I6IG9yZy5hcGFjaGUuY2xvdWRzdGFjay5z
dG9yYWdlLmltYWdlLnN0b3JlLlRlbXBsYXRlT2JqZWN0QDI1ZTg0ZWVldXIAHltMamF2YS5sYW5n
LlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_SI5AgAAeHAAAAANc3IAG2phdmEubGFuZy5TdGFja1Ry
YWNlRWxlbWVudGEJxZomNt2FAgAESQAKbGluZU51bWJlckwADmRlY2xhcmluZ0NsYXNzcQB-AARM
AAhmaWxlTmFtZXEAfgAETAAKbWV0aG9kTmFtZXEAfgAEeHAAAABydAAgY29tLmNsb3VkLnZtLlZt
V29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JzcQB-
AAsAAAH3dAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNK
b2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0
c3EAfgALAAAAMXQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdl
ZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVuc3EA
fgALAAAAOHQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZh
dWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHNx
AH4ACwAAAGd0AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVm
YXVsdE1hbmFnZWRDb250ZXh0cQB-ABp0AA9jYWxsV2l0aENvbnRleHRzcQB-AAsAAAA1cQB-AB1x
AH4AGnQADnJ1bldpdGhDb250ZXh0c3EAfgALAAAALnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5t
YW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAWcQB-ABdzcQB-AAsAAAHM
cQB-ABFxAH4AEnEAfgAXc3EAfgALAAAB13QALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9y
cyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAbc3EAfgALAAABBnQAH2phdmEu
dXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4AF3NxAH4ACwAA
BHl0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29s
RXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnNxAH4ACwAAAmd0AC5qYXZhLnV0aWwuY29uY3VycmVu
dC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-ACxxAH4AF3NxAH4ACwAAAul0ABBqYXZhLmxh
bmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4AF3NyACZqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5t
b2RpZmlhYmxlTGlzdPwPJTG17I4QAgABTAAEbGlzdHEAfgAGeHIALGphdmEudXRpbC5Db2xsZWN0
aW9ucyRVbm1vZGlmaWFibGVDb2xsZWN0aW9uGUIAgMte9x4CAAFMAAFjdAAWTGphdmEvdXRpbC9D
b2xsZWN0aW9uO3hwc3IAE2phdmEudXRpbC5BcnJheUxpc3R4gdIdmcdhnQMAAUkABHNpemV4cAAA
AAB3BAAAAAB4cQB-ADh4
2014-12-03 10:31:46,689 ERROR [c.c.u.d.DbUtil] (consoleproxy-1:ctx-c282e93d)
RELEASE_LOCK() throws exception
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link
failure
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
Caused by: java.io.EOFException: Can not read response from server. Expected
to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2014-12-03 10:31:46,691 WARN [c.c.v.SystemVmLoadScanner]
(consoleproxy-1:ctx-c282e93d) Unexpected exception Job failed due to
exception Resource [Host:1] is unreachable: Host 1: Unable to start instance
due to can't find mapping in ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
java.lang.RuntimeException: Job failed due to exception Resource [Host:1] is
unreachable: Host 1: Unable to start instance due to can't find mapping in
ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
2014-12-03 10:31:46,788 WARN [c.c.u.d.Merovingian2]
(consoleproxy-1:ctx-5c69b42f) Was unable to find lock for the key
vm_instance10 and thread id 117755041
2014-12-03 10:31:49,395 INFO [o.a.c.s.v.VolumeServiceImpl]
(Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400 ctx-2ee9073d) Unable to
acquire lock on VMTemplateStoragePool 57
2014-12-03 10:31:49,399 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400 ctx-2ee9073d) Failed to
start instance VM[SecondaryStorageVm|s-14-VM]
com.cloud.utils.exception.CloudRuntimeException: Unable to acquire lock on
VMTemplateStoragePool: 57
2014-12-03 10:31:49,554 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400 ctx-2ee9073d) VM state
transitted from :Starting to Stopped with event: OperationFailedvm's
original host id: null new host id: null host id before state transition: 4
2014-12-03 10:31:49,578 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400 ctx-2ee9073d) Invocation
exception, caused by: com.cloud.exception.AgentUnavailableException:
Resource [Host:4] is unreachable: Host 4: Unable to start instance due to
Unable to acquire lock on VMTemplateStoragePool: 57
2014-12-03 10:31:49,578 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400 ctx-2ee9073d) Rethrow
exception com.cloud.exception.AgentUnavailableException: Resource [Host:4]
is unreachable: Host 4: Unable to start instance due to Unable to acquire
lock on VMTemplateStoragePool: 57
2014-12-03 10:31:49,579 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400) Unable to complete
AsyncJobVO {id:400, userId: 1, accountId: 1, instanceType: null, instanceId:
null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAB
AAAAAAAAAAEAAAAAAAAADnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBw
cHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
0, result: null,
initMsid: 223162458780723, completeMsid: null, lastUpdated: null,
lastPolled: null, created: Wed Dec 03 10:30:07 EST 2014}, job origin:384
com.cloud.exception.AgentUnavailableException: Resource [Host:4] is
unreachable: Host 4: Unable to start instance due to Unable to acquire lock
on VMTemplateStoragePool: 57
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to
acquire lock on VMTemplateStoragePool: 57
2014-12-03 10:31:49,580 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400) Complete async job-400,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAmUpvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBSZXNvdXJjZSBbSG9zdDo0XSBpcyB1bnJlYWNoYWJsZTog
SG9zdCA0OiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgZHVlIHRvIFVuYWJsZSB0byBhY3F1aXJl
IGxvY2sgb24gVk1UZW1wbGF0ZVN0b3JhZ2VQb29sOiA1N3VyAB5bTGphdmEubGFuZy5TdGFja1Ry
YWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1l
bnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5h
bWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JE
aXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QA
P29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdl
ckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAA
ADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0
AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFn
ZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABn
dABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5h
Z2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5y
dW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-
ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFi
bGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29u
Y3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2
YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9y
LmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFk
UG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVh
ZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJs
ZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5t
b2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlv
bjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAA
eHEAfgA4eA
2014-12-03 10:31:49,620 WARN [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-5b27f0f0) Exception while trying
to start secondary storage vm
java.lang.RuntimeException: Job failed due to exception Resource [Host:4] is
unreachable: Host 4: Unable to start instance due to Unable to acquire lock
on VMTemplateStoragePool: 57
2014-12-03 10:31:49,622 ERROR [c.c.u.d.DbUtil] (secstorage-1:ctx-5b27f0f0)
RELEASE_LOCK() throws exception
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link
failure
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
Caused by: java.io.EOFException: Can not read response from server. Expected
to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2014-12-03 10:31:49,624 INFO [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-5b27f0f0) Unable to start secondary storage vm for standby
capacity, secStorageVm vm Id : 14, will recycle it and start a new one
2014-12-03 10:31:49,624 ERROR [c.c.u.d.DbUtil] (secstorage-1:ctx-5b27f0f0)
RELEASE_LOCK() throws exception
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link
failure
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
Caused by: java.io.EOFException: Can not read response from server. Expected
to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2014-12-03 10:31:49,781 WARN [c.c.u.d.Merovingian2]
(secstorage-1:ctx-bc09a930) Was unable to find lock for the key
vm_instance14 and thread id 819805219
2014-12-03 10:31:56,361 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-32:ctx-853d1e1b) Unable to get current status on
4(xenhv1.pvdc.chlt.charlottecolo.com)
2014-12-03 10:32:37,989 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-3:ctx-fefa154b) Unable to get current status on
1(xenhv2.pvdc.chlt.charlottecolo.com)
2014-12-03 10:32:56,361 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-65:ctx-51ba659a) Unable to get current status on
4(xenhv1.pvdc.chlt.charlottecolo.com)
2014-12-03 10:33:37,990 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-33:ctx-e578dc8f) Unable to get current status on
1(xenhv2.pvdc.chlt.charlottecolo.com)
2014-12-03 10:33:47,948 DEBUG [o.a.c.s.i.s.TemplateObject]
(Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) failed to
process event and answer
com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
com.mysql.jdbc.JDBC4PreparedStatement@37aaee74: SELECT
template_spool_ref.id, template_spool_ref.pool_id,
template_spool_ref.template_id, template_spool_ref.created,
template_spool_ref.last_updated, template_spool_ref.download_pct,
template_spool_ref.download_state, template_spool_ref.local_path,
template_spool_ref.error_str, template_spool_ref.job_id,
template_spool_ref.install_path, template_spool_ref.template_size,
template_spool_ref.marked_for_gc, template_spool_ref.update_count,
template_spool_ref.updated, template_spool_ref.state FROM template_spool_ref
WHERE template_spool_ref.pool_id = 2 AND template_spool_ref.template_id = 1
ORDER BY RAND() LIMIT 1
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
Caused by: java.io.EOFException: Can not read response from server. Expected
to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2014-12-03 10:33:47,970 DEBUG [o.a.c.s.v.VolumeServiceImpl]
(Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) failed to
create template on storage
java.lang.RuntimeException: InvocationTargetException when invoking RPC
callback for command: copyBaseImageCallback
Caused by: java.lang.reflect.InvocationTargetException
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to
process event
Caused by: com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
com.mysql.jdbc.JDBC4PreparedStatement@37aaee74: SELECT
template_spool_ref.id, template_spool_ref.pool_id,
template_spool_ref.template_id, template_spool_ref.created,
template_spool_ref.last_updated, template_spool_ref.download_pct,
template_spool_ref.download_state, template_spool_ref.local_path,
template_spool_ref.error_str, template_spool_ref.job_id,
template_spool_ref.install_path, template_spool_ref.template_size,
template_spool_ref.marked_for_gc, template_spool_ref.update_count,
template_spool_ref.updated, template_spool_ref.state FROM template_spool_ref
WHERE template_spool_ref.pool_id = 2 AND template_spool_ref.template_id = 1
ORDER BY RAND() LIMIT 1
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
Caused by: java.io.EOFException: Can not read response from server. Expected
to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2014-12-03 10:33:47,976 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
(Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) Template 1
is not found on storage pool 2, so no need to delete
2014-12-03 10:33:47,979 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) Failed to
start instance VM[ConsoleProxy|v-10-VM]
com.cloud.utils.exception.CloudRuntimeException: can't find mapping in
ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
2014-12-03 10:33:48,085 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) VM state
transitted from :Starting to Stopped with event: OperationFailedvm's
original host id: null new host id: null host id before state transition: 4
2014-12-03 10:33:48,112 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) Invocation
exception, caused by: com.cloud.exception.AgentUnavailableException:
Resource [Host:4] is unreachable: Host 4: Unable to start instance due to
can't find mapping in ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
2014-12-03 10:33:48,113 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) Rethrow
exception com.cloud.exception.AgentUnavailableException: Resource [Host:4]
is unreachable: Host 4: Unable to start instance due to can't find mapping
in ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
2014-12-03 10:33:48,113 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-66:ctx-38c8d928 job-388/job-402) Unable to complete
AsyncJobVO {id:402, userId: 1, accountId: 1, instanceType: null, instanceId:
null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAB
AAAAAAAAAAEAAAAAAAAACnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBw
cHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
0, result: null,
initMsid: 223162458780723, completeMsid: null, lastUpdated: null,
lastPolled: null, created: Wed Dec 03 10:31:46 EST 2014}, job origin:388
com.cloud.exception.AgentUnavailableException: Resource [Host:4] is
unreachable: Host 4: Unable to start instance due to can't find mapping in
ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
Caused by: com.cloud.utils.exception.CloudRuntimeException: can't find
mapping in ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
2014-12-03 10:33:48,115 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-66:ctx-38c8d928 job-388/job-402) Complete async job-402,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QA2kpvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBSZXNvdXJjZSBbSG9zdDo0XSBpcyB1bnJlYWNoYWJsZTog
SG9zdCA0OiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgZHVlIHRvIGNhbid0IGZpbmQgbWFwcGlu
ZyBpbiBPYmplY3RJbkRhdGFTdG9yZSB0YWJsZSBmb3I6IG9yZy5hcGFjaGUuY2xvdWRzdGFjay5z
dG9yYWdlLmltYWdlLnN0b3JlLlRlbXBsYXRlT2JqZWN0QDRjN2JjNzc0dXIAHltMamF2YS5sYW5n
LlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_SI5AgAAeHAAAAANc3IAG2phdmEubGFuZy5TdGFja1Ry
YWNlRWxlbWVudGEJxZomNt2FAgAESQAKbGluZU51bWJlckwADmRlY2xhcmluZ0NsYXNzcQB-AARM
AAhmaWxlTmFtZXEAfgAETAAKbWV0aG9kTmFtZXEAfgAEeHAAAABydAAgY29tLmNsb3VkLnZtLlZt
V29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JzcQB-
AAsAAAH3dAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNK
b2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0
c3EAfgALAAAAMXQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdl
ZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVuc3EA
fgALAAAAOHQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZh
dWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHNx
AH4ACwAAAGd0AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVm
YXVsdE1hbmFnZWRDb250ZXh0cQB-ABp0AA9jYWxsV2l0aENvbnRleHRzcQB-AAsAAAA1cQB-AB1x
AH4AGnQADnJ1bldpdGhDb250ZXh0c3EAfgALAAAALnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5t
YW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAWcQB-ABdzcQB-AAsAAAHM
cQB-ABFxAH4AEnEAfgAXc3EAfgALAAAB13QALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9y
cyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAbc3EAfgALAAABBnQAH2phdmEu
dXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4AF3NxAH4ACwAA
BHl0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29s
RXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnNxAH4ACwAAAmd0AC5qYXZhLnV0aWwuY29uY3VycmVu
dC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-ACxxAH4AF3NxAH4ACwAAAul0ABBqYXZhLmxh
bmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4AF3NyACZqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5t
b2RpZmlhYmxlTGlzdPwPJTG17I4QAgABTAAEbGlzdHEAfgAGeHIALGphdmEudXRpbC5Db2xsZWN0
aW9ucyRVbm1vZGlmaWFibGVDb2xsZWN0aW9uGUIAgMte9x4CAAFMAAFjdAAWTGphdmEvdXRpbC9D
b2xsZWN0aW9uO3hwc3IAE2phdmEudXRpbC5BcnJheUxpc3R4gdIdmcdhnQMAAUkABHNpemV4cAAA
AAB3BAAAAAB4cQB-ADh4
2014-12-03 10:33:48,158 ERROR [c.c.u.d.DbUtil] (consoleproxy-1:ctx-5c69b42f)
RELEASE_LOCK() throws exception
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link
failure
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
Caused by: java.io.EOFException: Can not read response from server. Expected
to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2014-12-03 10:33:48,160 WARN [c.c.v.SystemVmLoadScanner]
(consoleproxy-1:ctx-5c69b42f) Unexpected exception Job failed due to
exception Resource [Host:4] is unreachable: Host 4: Unable to start instance
due to can't find mapping in ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
java.lang.RuntimeException: Job failed due to exception Resource [Host:4] is
unreachable: Host 4: Unable to start instance due to can't find mapping in
ObjectInDataStore table for:
org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
2014-12-03 10:33:48,251 WARN [c.c.u.d.Merovingian2]
(consoleproxy-1:ctx-a282718d) Was unable to find lock for the key
vm_instance10 and thread id 117755041
2014-12-03 10:33:51,376 INFO [o.a.c.s.v.VolumeServiceImpl]
(Work-Job-Executor-67:ctx-b6aee29d job-384/job-403 ctx-05dc78c7) Unable to
acquire lock on VMTemplateStoragePool 60
2014-12-03 10:33:51,380 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-67:ctx-b6aee29d job-384/job-403 ctx-05dc78c7) Failed to
start instance VM[SecondaryStorageVm|s-14-VM]
com.cloud.utils.exception.CloudRuntimeException: Unable to acquire lock on
VMTemplateStoragePool: 60
2014-12-03 10:33:51,508 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-67:ctx-b6aee29d job-384/job-403 ctx-05dc78c7) VM state
transitted from :Starting to Stopped with event: OperationFailedvm's
original host id: null new host id: null host id before state transition: 1
2014-12-03 10:33:51,533 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-67:ctx-b6aee29d job-384/job-403 ctx-05dc78c7) Invocation
exception, caused by: com.cloud.exception.AgentUnavailableException:
Resource [Host:1] is unreachable: Host 1: Unable to start instance due to
Unable to acquire lock on VMTemplateStoragePool: 60
2014-12-03 10:33:51,534 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-67:ctx-b6aee29d job-384/job-403 ctx-05dc78c7) Rethrow
exception com.cloud.exception.AgentUnavailableException: Resource [Host:1]
is unreachable: Host 1: Unable to start instance due to Unable to acquire
lock on VMTemplateStoragePool: 60
2014-12-03 10:33:51,534 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-67:ctx-b6aee29d job-384/job-403) Unable to complete
AsyncJobVO {id:403, userId: 1, accountId: 1, instanceType: null, instanceId:
null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAB
AAAAAAAAAAEAAAAAAAAADnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBw
cHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
0, result: null,
initMsid: 223162458780723, completeMsid: null, lastUpdated: null,
lastPolled: null, created: Wed Dec 03 10:31:49 EST 2014}, job origin:384
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
unreachable: Host 1: Unable to start instance due to Unable to acquire lock
on VMTemplateStoragePool: 60
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to
acquire lock on VMTemplateStoragePool: 60
2014-12-03 10:33:51,536 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-67:ctx-b6aee29d job-384/job-403) Complete async job-403,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAmUpvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBSZXNvdXJjZSBbSG9zdDoxXSBpcyB1bnJlYWNoYWJsZTog
SG9zdCAxOiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgZHVlIHRvIFVuYWJsZSB0byBhY3F1aXJl
IGxvY2sgb24gVk1UZW1wbGF0ZVN0b3JhZ2VQb29sOiA2MHVyAB5bTGphdmEubGFuZy5TdGFja1Ry
YWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1l
bnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5h
bWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JE
aXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QA
P29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdl
ckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAA
ADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0
AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFn
ZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABn
dABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5h
Z2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5y
dW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-
ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFi
bGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29u
Y3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2
YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9y
LmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFk
UG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVh
ZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJs
ZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5t
b2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlv
bjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAA
eHEAfgA4eA
2014-12-03 10:33:51,576 WARN [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-bc09a930) Exception while trying
to start secondary storage vm
java.lang.RuntimeException: Job failed due to exception Resource [Host:1] is
unreachable: Host 1: Unable to start instance due to Unable to acquire lock
on VMTemplateStoragePool: 60
2014-12-03 10:33:51,577 ERROR [c.c.u.d.DbUtil] (secstorage-1:ctx-bc09a930)
RELEASE_LOCK() throws exception
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link
failure
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
Caused by: java.io.EOFException: Can not read response from server. Expected
to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2014-12-03 10:33:51,579 INFO [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-bc09a930) Unable to start secondary storage vm for standby
capacity, secStorageVm vm Id : 14, will recycle it and start a new one
2014-12-03 10:33:51,609 WARN [c.c.u.d.Merovingian2]
(secstorage-1:ctx-bc09a930) Was unable to find lock for the key
vm_instance14 and thread id 819805219
2014-12-03 10:33:52,060 ERROR [c.c.u.d.DbUtil] (secstorage-1:ctx-bc09a930)
RELEASE_LOCK() throws exception
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link
failure
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
Caused by: java.io.EOFException: Can not read response from server. Expected
to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
2014-12-03 10:33:52,397 WARN [c.c.u.d.Merovingian2]
(secstorage-1:ctx-0a61fb70) Was unable to find lock for the key
vm_instance15 and thread id 819805219
2014-12-03 10:33:56,362 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-66:ctx-580f5bb1) Unable to get current status on
4(xenhv1.pvdc.chlt.charlottecolo.com)
2014-12-03 10:35:31,403 ERROR [c.c.c.ClusterManagerImpl] (main:null) Unable
to ping management server at 172.16.0.69:9090 due to ConnectException
java.net.ConnectException: Connection refused
2014-12-03 10:35:31,591 WARN [c.c.c.ConsoleProxyManagerImpl] (main:null)
Empty console proxy domain, explicitly disabling SSL
2014-12-03 10:35:31,993 WARN [c.c.s.d.DownloadMonitorImpl] (main:null) Only
realhostip.com ssl cert is supported, ignoring self-signed and other certs
2014-12-03 10:35:49,150 INFO [c.c.h.x.r.XenServerConnectionPool]
(main:null) XenServer Connection Pool Configs: sleep.interval.on.error=10000
2014-12-03 10:35:49,732 ERROR [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-3:ctx-c81f933c) Caught this throwable,
com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to commit
or close the connection.
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock
found when trying to get lock; try restarting transaction
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1065)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
2014-12-03 10:35:49,750 ERROR [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-4:ctx-ebb2805b) Caught this throwable,
com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
Caused by: com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
com.mysql.jdbc.JDBC4PreparedStatement@30b1baa6: UPDATE op_ha_work SET
op_ha_work.taken='2014-12-03 15:35:49',
op_ha_work.mgmt_server_id=223162458780723 WHERE op_ha_work.id = 4
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock
found when trying to get lock; try restarting transaction
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1065)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
2014-12-03 10:35:49,808 WARN [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
Mark job as failed as its correspoding queue-item has been discarded. job
id: 405
2014-12-03 10:35:49,808 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
Complete async job-405, jobStatus: FAILED,
resultCode: 0, result: Execution was cancelled because of server shutdown
2014-12-03 10:35:49,813 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
job-405 no longer exists, we just log completion info here. FAILED,
resultCode: 0, result: Execution was cancelled because of server shutdown
2014-12-03 10:35:49,850 DEBUG [c.c.h.CheckOnAgentInvestigator]
(HA-Worker-2:ctx-b2a9dc97 work-1) Unable to reach the agent for
VM[SecondaryStorageVm|s-15-VM]: Resource [Host:4] is unreachable: Host 4:
Host with specified id is not in the
right state: Disconnected
2014-12-03 10:35:49,863 WARN [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
Mark job as failed as its correspoding queue-item has been discarded. job
id: 411
2014-12-03 10:35:49,863 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
Complete async job-411, jobStatus: FAILED,
resultCode: 0, result: Execution was cancelled because of server shutdown
2014-12-03 10:35:49,859 DEBUG [c.c.h.CheckOnAgentInvestigator]
(HA-Worker-1:ctx-415457e8 work-4) Unable to reach the agent for
VM[ConsoleProxy|v-10-VM]: Resource [Host:4] is unreachable: Host 4: Host
with specified id is not in the right
state: Disconnected
2014-12-03 10:35:49,885 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
job-411 no longer exists, we just log completion info here. FAILED,
resultCode: 0, result: Execution was cancelled because of server shutdown
2014-12-03 10:35:49,901 DEBUG [c.c.h.UserVmDomRInvestigator]
(HA-Worker-1:ctx-415457e8 work-4) Not a User Vm, unable to determine state
of VM[ConsoleProxy|v-10-VM] returning null
2014-12-03 10:35:49,902 DEBUG [c.c.h.UserVmDomRInvestigator]
(HA-Worker-2:ctx-b2a9dc97 work-1) Not a User Vm, unable to determine state
of VM[SecondaryStorageVm|s-15-VM] returning null
2014-12-03 10:35:49,921 DEBUG [c.c.h.ManagementIPSystemVMInvestigator]
(HA-Worker-2:ctx-b2a9dc97 work-1) unable to determine state of
VM[SecondaryStorageVm|s-15-VM] returning null
2014-12-03 10:35:49,922 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-2:ctx-b2a9dc97 work-1) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 10:35:49,929 DEBUG [c.c.h.ManagementIPSystemVMInvestigator]
(HA-Worker-1:ctx-415457e8 work-4) unable to determine state of
VM[ConsoleProxy|v-10-VM] returning null
2014-12-03 10:35:49,930 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-1:ctx-415457e8 work-4) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 10:35:50,055 WARN [c.c.u.d.Merovingian2]
(HA-Worker-1:ctx-415457e8 work-4) Was unable to find lock for the
key vm_instance10 and thread id 1329615500
2014-12-03 10:35:50,059 WARN [c.c.u.d.Merovingian2]
(HA-Worker-2:ctx-b2a9dc97 work-1) Was unable to find lock for the
key vm_instance15 and thread id 9330563
2014-12-03 10:35:50,849 ERROR [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-375d995b) Unable to ping management server at
172.16.0.70:9090 due to ConnectException
java.net.ConnectException: Connection refused
2014-12-03 10:35:50,850 WARN [c.c.c.ClusterManagerImpl]
(Cluster-Heartbeat-1:ctx-375d995b) Management node 4 is detected inactive by
timestamp and also not pingable
2014-12-03 10:35:52,166 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-1:ctx-e4c35b10 job-413/job-415 ctx-a901375d) Unable to
transition the state but we're moving on because it's forced stop
2014-12-03 10:35:52,166 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-1:ctx-e4c35b10 job-413/job-415 ctx-a901375d) Unable to
cleanup VM: VM[ConsoleProxy|v-10-VM] ,since outstanding work item is not
found
2014-12-03 10:35:52,166 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-1:ctx-e4c35b10 job-413/job-415 ctx-a901375d) Invocation
exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
item not found, We cannot stop VM[ConsoleProxy|v-10-VM] when it is in state
Starting
2014-12-03 10:35:52,167 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-1:ctx-e4c35b10 job-413/job-415 ctx-a901375d) Rethrow
exception com.cloud.utils.exception.CloudRuntimeException: Work item not
found, We cannot stop VM[ConsoleProxy|v-10-VM] when it is in state Starting
2014-12-03 10:35:52,168 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-1:ctx-e4c35b10 job-413/job-415) Unable to
complete AsyncJobVO {id:415, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
AAAACnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
223162458780723, completeMsid: null, lastUpdated: null, lastPolled: null,
created: Wed Dec 03 10:35:50 EST 2014}, job origin:413
com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
cannot stop VM[ConsoleProxy|v-10-VM] when it is in state Starting
2014-12-03 10:35:52,173 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-1:ctx-e4c35b10 job-413/job-415) Complete async job-415,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAdUpvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
cCBWTVtDb25zb2xlUHJveHl8di0xMC1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFydGluZ3Vy
AB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZh
LmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJp
bmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNv
bS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZh
dAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9i
cy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQA
DHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUu
amF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250
ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQu
amF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29u
dGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EA
fgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hl
LmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEA
fgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3Vy
cmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4A
CwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh
cQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y
dAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51
dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsA
AALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNv
bGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZh
LnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QA
FkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0D
AAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
2014-12-03 10:35:52,175 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-2:ctx-fd2be5c1 job-412/job-416 ctx-c11ef29e) Unable to
transition the state but we're moving on because it's forced stop
2014-12-03 10:35:52,175 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-2:ctx-fd2be5c1 job-412/job-416 ctx-c11ef29e) Unable to
cleanup VM: VM[SecondaryStorageVm|s-15-VM] ,since outstanding work item is
not found
2014-12-03 10:35:52,175 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-2:ctx-fd2be5c1 job-412/job-416 ctx-c11ef29e) Invocation
exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
item not found, We cannot stop VM[SecondaryStorageVm|s-15-VM] when it is in
state Starting
2014-12-03 10:35:52,176 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-2:ctx-fd2be5c1 job-412/job-416 ctx-c11ef29e) Rethrow
exception com.cloud.utils.exception.CloudRuntimeException: Work item not
found, We cannot stop VM[SecondaryStorageVm|s-15-VM] when it is in state
Starting
2014-12-03 10:35:52,176 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-2:ctx-fd2be5c1 job-412/job-416) Unable to
complete AsyncJobVO {id:416, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
AAAAD3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
223162458780723, completeMsid: null, lastUpdated: null, lastPolled: null,
created: Wed Dec 03 10:35:50 EST 2014}, job origin:412
com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
cannot stop VM[SecondaryStorageVm|s-15-VM] when it is in state Starting
2014-12-03 10:35:52,178 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-2:ctx-fd2be5c1 job-412/job-416) Complete async job-416,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAe0pvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
cCBWTVtTZWNvbmRhcnlTdG9yYWdlVm18cy0xNS1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFy
dGluZ3VyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNy
ABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5k
ZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAA
cnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hl
ci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdv
cmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwu
amF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFu
YWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVu
bmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdl
ZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENv
bnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFn
ZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250
ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcu
YXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVx
AH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwu
Y29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4A
G3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFz
ay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4
ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAu
amF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdz
cQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51
dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhy
ACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgAB
TAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHS
HZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
2014-12-03 10:35:52,274 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-1:ctx-415457e8 work-4) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[ConsoleProxy|v-10-VM] when it is in state Starting
2014-12-03 10:35:52,276 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-2:ctx-b2a9dc97 work-1) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[SecondaryStorageVm|s-15-VM] when it is in state Starting
2014-12-03 10:35:52,394 WARN [c.c.c.ClusterManagerImpl]
(Cluster-Notification-1:ctx-e8ae6c9b) Notifying management server leave
event took 68 ms
2014-12-03 10:35:52,395 WARN [o.a.c.alerts]
(Cluster-Notification-1:ctx-e8ae6c9b) alertType:: 14 // dataCenterId:: 0
// podId:: 0 // clusterId:: null // message:: Management server node
172.16.0.69 is up
2014-12-03 10:35:52,418 WARN [c.c.c.ClusterManagerImpl]
(Cluster-Notification-1:ctx-e8ae6c9b) Notifying management server join event
took 24 ms
2014-12-03 10:36:57,546 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-1:ctx-dada7b54) Unable to get current status on
1(xenhv2.pvdc.chlt.charlottecolo.com)
2014-12-03 10:36:58,203 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-2:ctx-41be4ac5) Unable to get current status on
4(xenhv1.pvdc.chlt.charlottecolo.com)
2014-12-03 10:37:57,548 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-3:ctx-85d0acb3) Unable to get current status on
1(xenhv2.pvdc.chlt.charlottecolo.com)
2014-12-03 10:37:58,204 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-4:ctx-68aea530) Unable to get current status on
4(xenhv1.pvdc.chlt.charlottecolo.com)
2014-12-03 10:38:09,839 WARN [c.c.a.d.ParamGenericValidationWorker]
(catalina-exec-3:ctx-c510e983 ctx-c0a51a4e) Received unknown parameters for
command listRegions. Unknown parameters : listall
2014-12-03 10:38:18,800 WARN [c.c.a.d.ParamGenericValidationWorker]
(catalina-exec-13:ctx-0ca7247c ctx-7869b51b) Received unknown parameters for
command listZones. Unknown parameters : listall
2014-12-03 10:38:18,987 WARN [c.c.a.d.ParamGenericValidationWorker]
(catalina-exec-18:ctx-85e4fffe ctx-819e52d4) Received unknown parameters for
command listPods. Unknown parameters : listall
2014-12-03 10:38:19,118 WARN [c.c.a.d.ParamGenericValidationWorker]
(catalina-exec-17:ctx-cb8c1540 ctx-5a4a6061) Received unknown parameters for
command listClusters. Unknown parameters : listall
2014-12-03 10:38:19,244 WARN [c.c.a.d.ParamGenericValidationWorker]
(catalina-exec-19:ctx-be72b381 ctx-d0874501) Received unknown parameters for
command listHosts. Unknown parameters : listall
2014-12-03 10:38:19,391 WARN [c.c.a.d.ParamGenericValidationWorker]
(catalina-exec-21:ctx-e2218038 ctx-eed3052d) Received unknown parameters for
command listStoragePools. Unknown parameters : listall
2014-12-03 10:38:19,533 WARN [c.c.a.d.ParamGenericValidationWorker]
(catalina-exec-22:ctx-20552766 ctx-83dccf23) Received unknown parameters for
command listImageStores. Unknown parameters : listall type
2014-12-03 10:38:19,655 WARN [c.c.a.d.ParamGenericValidationWorker]
(catalina-exec-24:ctx-968df022 ctx-d4de7d4e) Received unknown parameters for
command listSystemVms. Unknown parameters : listall
2014-12-03 10:38:24,050 WARN [c.c.a.d.ParamGenericValidationWorker]
(catalina-exec-7:ctx-193b8617 ctx-c4ab4021) Received unknown parameters for
command listSystemVms. Unknown parameters : listall
2014-12-03 10:38:24,557 WARN [c.c.a.d.ParamGenericValidationWorker]
(catalina-exec-11:ctx-0e796a3f ctx-6e24a560) Received unknown parameters for
command listHosts. Unknown parameters : listall
2014-12-03 10:38:26,590 WARN [c.c.a.d.ParamGenericValidationWorker]
(catalina-exec-3:ctx-c278bf56 ctx-e32c7268) Received unknown parameters for
command listZones. Unknown parameters : listall
2014-12-03 10:38:26,693 WARN [c.c.a.d.ParamGenericValidationWorker]
(catalina-exec-12:ctx-1afb82ed ctx-051fbce8) Received unknown parameters for
command listPods. Unknown parameters : listall
2014-12-03 10:38:26,781 WARN [c.c.a.d.ParamGenericValidationWorker]
(catalina-exec-14:ctx-aadb7b92 ctx-cb2ede4c) Received unknown parameters for
command listClusters. Unknown parameters : listall
2014-12-03 10:38:26,878 WARN [c.c.a.d.ParamGenericValidationWorker]
(catalina-exec-15:ctx-96f8cd1d ctx-87a1426e) Received unknown parameters for
command listHosts. Unknown parameters : listall
2014-12-03 10:38:27,032 WARN [c.c.a.d.ParamGenericValidationWorker]
(catalina-exec-16:ctx-3593bc62 ctx-bab3e4e6) Received unknown parameters for
command listStoragePools. Unknown parameters : listall
2014-12-03 10:38:27,133 WARN [c.c.a.d.ParamGenericValidationWorker]
(catalina-exec-13:ctx-004ef7e9 ctx-528af3a7) Received unknown parameters for
command listImageStores. Unknown parameters : listall type
2014-12-03 10:38:27,294 WARN [c.c.a.d.ParamGenericValidationWorker]
(catalina-exec-18:ctx-fee8e727 ctx-f817e3df) Received unknown parameters for
command listSystemVms. Unknown parameters : listall
2014-12-03 10:38:28,720 WARN [c.c.a.d.ParamGenericValidationWorker]
(catalina-exec-5:ctx-4e425560 ctx-dc515ed7) Received unknown parameters for
command listHosts. Unknown parameters : listall
2014-12-03 10:38:57,545 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-5:ctx-3c160491) Unable to get current status on
1(xenhv2.pvdc.chlt.charlottecolo.com)
2014-12-03 10:38:58,202 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-6:ctx-f2f80cb3) Unable to get current status on
4(xenhv1.pvdc.chlt.charlottecolo.com)
2014-12-03 10:39:57,545 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-2:ctx-954a2d0f) Unable to get current status on
1(xenhv2.pvdc.chlt.charlottecolo.com)
2014-12-03 10:39:58,202 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-8:ctx-1a58d0d9) Unable to get current status on
4(xenhv1.pvdc.chlt.charlottecolo.com)
2014-12-03 10:40:57,545 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-3:ctx-353674d2) Unable to get current status on
1(xenhv2.pvdc.chlt.charlottecolo.com)
2014-12-03 10:40:58,203 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-9:ctx-3761b53b) Unable to get current status on
4(xenhv1.pvdc.chlt.charlottecolo.com)
2014-12-03 10:41:57,547 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-4:ctx-c0120a72) Unable to get current status on
1(xenhv2.pvdc.chlt.charlottecolo.com)
2014-12-03 10:41:58,203 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-10:ctx-f1fdf064) Unable to get current status on
4(xenhv1.pvdc.chlt.charlottecolo.com)
2014-12-03 10:42:57,546 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-6:ctx-efcc44be) Unable to get current status on
1(xenhv2.pvdc.chlt.charlottecolo.com)
2014-12-03 10:42:58,203 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-12:ctx-9f51f8e3) Unable to get current status on
4(xenhv1.pvdc.chlt.charlottecolo.com)
2014-12-03 10:43:57,547 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-1:ctx-3e62beae) Unable to get current status on
1(xenhv2.pvdc.chlt.charlottecolo.com)
2014-12-03 10:43:58,204 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-13:ctx-3893f12e) Unable to get current status on
4(xenhv1.pvdc.chlt.charlottecolo.com)
[root@CSTW1 ~]#
-----Original Message-----
From: Matthew Midgett [mailto:cloudstck@trick-solutions.com.INVALID]
Sent: Wednesday, December 03, 2014 6:39 AM
To: users@cloudstack.apache.org
Subject: RE: Fresh 4.4.2 - Unable to get status on .... my XenServer
Thanks Geoff I really appreciate the response. I guess that every other time
over the last year I've reinstalled XenServer and didn't remember it. Maybe
we should add this to the Docs as I don't ever remember seeing it, I could
be wrong though. I do have the patch management automated and it checks in
on every reboot. I will have to use Ilo to reinstall the OS today which is
not that big of a deal. I do admit that I need to PXE boot the servers I
just hadn't had the time. Yesterday I spent working out a builder for CS and
a network mirror. Then I noticed about 3 hours later that you guys posted a
public repo for 4.4.2. :)
CONFIDENTIAL: This electronic mail, including any attachment(s) may contain
information that is privileged, confidential, and/or otherwise protected
from disclosure to anyone other than its intended recipient(s). Any
dissemination or use of this electronic email or its contents (including any
attachments) by persons other than the intended recipient(s) is strictly
prohibited. If you have received this message in error, please notify me
immediately by reply email so that I may correct my internal records.
Please then delete the original message (including any attachments) in its
entirety. Thank you.
-----Original Message-----
From: Matthew Midgett [mailto:cloudstck@trick-solutions.com.INVALID]
Sent: Wednesday, December 03, 2014 1:11 AM
To: users@cloudstack.apache.org
Subject: Fresh 4.4.2 - Unable to get status on .... my xenservers
First I am not sure what is causing this, I have been testing 4.3 and 4.4
multiple times and didn't notice this problem till middle of last month when
I processed all xenserver updates and then tried to go to 4.4.1. I was
getting the same error that I am unable to get status of my xenservers. I
wiped out everything but the xenservers, those I just removed from the pool
and let them reboot and added them to a new pool. In ACS it says its trying
to start the SSVM and the Console VM but they stay in the starting state.
I'm sure it is because of this error. The agent show up in ACS so I don't
know why it can't talk to the xenservers. I can ping them from the
management servers.
Last login: Wed Dec 3 00:57:04 2014 from 192.168.255.6
[root@CSTW1 ~]# grep -i -E 'exception|unable|fail|invalid|leak|warn|error'
/var/log/cloudstack/management/management-server.log
2014-12-03 00:00:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-5:ctx-a0a23e42) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:00:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-21:ctx-f78a59e4) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:01:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-11:ctx-2e791a18) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:01:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-22:ctx-7bca20e8) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:02:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-12:ctx-28d48099) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:02:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-24:ctx-bc593a63) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:03:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-1:ctx-615b171d) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:03:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-25:ctx-07f6ba50) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:04:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-13:ctx-c2287fb5) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:04:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-26:ctx-f7d92459) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:05:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-14:ctx-a4e62e4e) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:05:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-28:ctx-01bb91e3) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:06:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-2:ctx-30f576ea) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:06:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-29:ctx-4d6690de) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:07:27,540 ERROR [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-2:ctx-645639e7) Caught this throwable,
com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
Caused by: com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
com.mysql.jdbc.JDBC4PreparedStatement@32602693: UPDATE op_ha_work SET
op_ha_work.taken='2014-12-03 05:07:27',
op_ha_work.mgmt_server_id=223162458780723 WHERE op_ha_work.id = 5
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock
found when trying to get lock; try restarting transaction
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1065)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
2014-12-03 00:07:27,595 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-3:ctx-c7c94958 work-5) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 00:07:27,645 WARN [c.c.u.d.Merovingian2]
(HA-Worker-3:ctx-c7c94958 work-5) Was unable to find lock for the
key vm_instance22 and thread id 818653380
2014-12-03 00:07:29,425 ERROR [c.c.v.VirtualMachineManagerImpl]
(Vm-Operations-Cleanup-1:ctx-35d886d3) VM Operations failed due to
com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
com.mysql.jdbc.JDBC4PreparedStatement@61807bba: DELETE FROM async_job WHERE
async_job.id= 401
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
Cannot delete or update a parent row: a foreign key constraint fails
(`cloud`.`async_job_join_map`, CONSTRAINT
`fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES
`async_job` (`id`))
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
2014-12-03 00:07:30,702 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-3:ctx-c7c94958 work-5) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
2014-12-03 00:07:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-15:ctx-c7dd62d6) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:07:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-30:ctx-99d14f4c) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:08:27,621 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-2:ctx-94cc502e work-2) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 00:08:27,678 WARN [c.c.u.d.Merovingian2]
(HA-Worker-2:ctx-94cc502e work-2) Was unable to find lock for the
key vm_instance15 and thread id 52609881
2014-12-03 00:08:29,421 ERROR [c.c.v.VirtualMachineManagerImpl]
(Vm-Operations-Cleanup-1:ctx-ed5f2ff6) VM Operations failed due to
com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
com.mysql.jdbc.PreparedStatement@15fe9d6d: DELETE FROM async_job WHERE
async_job.id= 406
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
Cannot delete or update a parent row: a foreign key constraint fails
(`cloud`.`async_job_join_map`, CONSTRAINT
`fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES
`async_job` (`id`))
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
2014-12-03 00:08:30,736 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-2:ctx-94cc502e work-2) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:08:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-16:ctx-acc87f49) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:08:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-32:ctx-efd0036e) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:09:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-3:ctx-1d3eecd0) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:09:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-33:ctx-3271eeea) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:10:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-17:ctx-27637ed9) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:10:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-34:ctx-b1f98667) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:11:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-18:ctx-45760fd3) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:11:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-36:ctx-3367d63d) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:12:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-4:ctx-ee2683aa) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:12:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-37:ctx-d0b02603) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:13:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-19:ctx-ce10b0fa) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:13:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-38:ctx-e7e4ae6b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:14:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-20:ctx-c158dd2c) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:14:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-40:ctx-7a662bea) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:15:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-5:ctx-0a8f1c90) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:15:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-41:ctx-4c163515) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:16:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-21:ctx-51c2dfd8) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:16:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-42:ctx-c670489b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:17:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-22:ctx-9df53feb) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:17:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-44:ctx-a921b13b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:18:29,546 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-1:ctx-4fdb5161 work-5) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 00:18:29,582 WARN [c.c.u.d.Merovingian2]
(HA-Worker-1:ctx-4fdb5161 work-5) Was unable to find lock for the
key vm_instance22 and thread id 2132920636
2014-12-03 00:18:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-6:ctx-a54d1732) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:18:32,638 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-1:ctx-4fdb5161 work-5) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
2014-12-03 00:18:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-45:ctx-5d1c0c7e) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:19:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Unable to
transition the state but we're moving on because it's forced stop
2014-12-03 00:19:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Unable to
cleanup VM: VM[ConsoleProxy|v-15-VM] ,since outstanding work item is not
found
2014-12-03 00:19:30,871 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Invocation
exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
item not found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state
Starting
2014-12-03 00:19:30,872 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Rethrow
exception com.cloud.utils.exception.CloudRuntimeException: Work item not
found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:19:30,872 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-3:ctx-d59db286 job-408/job-411) Unable to
complete AsyncJobVO {id:411, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
AAAAD3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
created: Wed Dec 03 00:19:02 EST 2014}, job origin:408
com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:19:30,874 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-3:ctx-d59db286 job-408/job-411) Complete async job-411,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAdUpvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
cCBWTVtDb25zb2xlUHJveHl8di0xNS1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFydGluZ3Vy
AB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZh
LmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJp
bmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNv
bS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZh
dAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9i
cy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQA
DHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUu
amF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250
ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQu
amF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29u
dGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EA
fgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hl
LmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEA
fgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3Vy
cmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4A
CwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh
cQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y
dAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51
dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsA
AALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNv
bGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZh
LnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QA
FkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0D
AAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
2014-12-03 00:19:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-23:ctx-0414dcdd) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:19:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-46:ctx-fc02b226) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:20:32,282 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-24:ctx-821b5aca) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:20:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-48:ctx-70dbfde8) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:21:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-1:ctx-de7b1636) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:21:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-49:ctx-154bac45) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:22:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-25:ctx-5c37a417) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:22:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-50:ctx-a34d4acf) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:23:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-52:ctx-4ee4cdd0) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:23:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-7:ctx-de025548) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:24:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-53:ctx-198b066a) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:24:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-27:ctx-e4b78a1b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:25:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-14:ctx-d0466cfa) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:25:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-55:ctx-6c330223) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:26:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-2:ctx-98d6b222) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:26:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-57:ctx-031552f0) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:27:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-29:ctx-a436d05f) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:27:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-58:ctx-0f3496a2) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:28:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-15:ctx-40cf6a78) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:28:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-59:ctx-1537f233) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:29:30,874 ERROR [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-0:ctx-84af6112) Caught this throwable,
com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to commit
or close the connection.
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock
found when trying to get lock; try restarting transaction
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1065)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
2014-12-03 00:29:30,881 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Unable to
transition the state but we're moving on because it's forced stop
2014-12-03 00:29:30,882 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Unable to
cleanup VM: VM[SecondaryStorageVm|s-22-VM] ,since outstanding work item is
not found
2014-12-03 00:29:30,882 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Invocation
exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
item not found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in
state Starting
2014-12-03 00:29:30,883 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Rethrow
exception com.cloud.utils.exception.CloudRuntimeException: Work item not
found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state
Starting
2014-12-03 00:29:30,883 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-4:ctx-b25792e9 job-414/job-417) Unable to
complete AsyncJobVO {id:417, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
AAAAFnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
created: Wed Dec 03 00:29:02 EST 2014}, job origin:414
com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
2014-12-03 00:29:30,896 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-4:ctx-b25792e9 job-414/job-417) Complete async job-417,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAe0pvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
cCBWTVtTZWNvbmRhcnlTdG9yYWdlVm18cy0yMi1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFy
dGluZ3VyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNy
ABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5k
ZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAA
cnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hl
ci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdv
cmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwu
amF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFu
YWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVu
bmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdl
ZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENv
bnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFn
ZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250
ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcu
YXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVx
AH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwu
Y29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4A
G3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFz
ay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4
ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAu
amF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdz
cQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51
dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhy
ACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgAB
TAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHS
HZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
2014-12-03 00:29:30,919 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-4:ctx-10fea75f work-2) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 00:29:30,967 WARN [c.c.u.d.Merovingian2]
(HA-Worker-4:ctx-10fea75f work-2) Was unable to find lock for the
key vm_instance15 and thread id 139852089
2014-12-03 00:29:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-8:ctx-bc77e0da) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:29:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-61:ctx-de44c2a9) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:29:34,025 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-4:ctx-10fea75f work-2) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:30:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-31:ctx-8417fc60) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:30:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-62:ctx-d118067c) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:31:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-16:ctx-393a2cde) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:31:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-63:ctx-fd5d5e54) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:32:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-3:ctx-07ab5550) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:32:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-65:ctx-0345383e) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:33:32,286 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-33:ctx-60af4723) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:33:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-66:ctx-0b95563b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:34:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-17:ctx-8e9c36a2) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:34:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-67:ctx-397176fb) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:35:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-9:ctx-3c967d9b) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:35:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-69:ctx-8bc1cfaf) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:36:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-35:ctx-8186e45f) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:36:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-70:ctx-557403b1) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:37:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-18:ctx-8539dc77) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:37:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-71:ctx-61ea3221) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:38:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-4:ctx-0dc9a587) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:38:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-73:ctx-7fba328c) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:39:31,006 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-0:ctx-7a356c50 work-5) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 00:39:31,051 WARN [c.c.u.d.Merovingian2]
(HA-Worker-0:ctx-7a356c50 work-5) Was unable to find lock for the
key vm_instance22 and thread id 84459079
2014-12-03 00:39:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-37:ctx-bc7a8f81) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:39:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-74:ctx-ebe4657b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:39:34,105 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-0:ctx-7a356c50 work-5) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
2014-12-03 00:40:30,865 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Unable to
transition the state but we're moving on because it's forced stop
2014-12-03 00:40:30,866 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Unable to
cleanup VM: VM[ConsoleProxy|v-15-VM] ,since outstanding work item is not
found
2014-12-03 00:40:30,866 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Invocation
exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
item not found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state
Starting
2014-12-03 00:40:30,866 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Rethrow
exception com.cloud.utils.exception.CloudRuntimeException: Work item not
found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:40:30,866 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-5:ctx-32edffa5 job-423/job-426) Unable to
complete AsyncJobVO {id:426, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
AAAAD3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
created: Wed Dec 03 00:40:02 EST 2014}, job origin:423
com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:40:30,868 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-5:ctx-32edffa5 job-423/job-426) Complete async job-426,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAdUpvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
cCBWTVtDb25zb2xlUHJveHl8di0xNS1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFydGluZ3Vy
AB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZh
LmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJp
bmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNv
bS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZh
dAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9i
cy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQA
DHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUu
amF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250
ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQu
amF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29u
dGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EA
fgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hl
LmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEA
fgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3Vy
cmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4A
CwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh
cQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y
dAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51
dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsA
AALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNv
bGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZh
LnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QA
FkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0D
AAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
2014-12-03 00:40:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-19:ctx-338898ef) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:40:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-75:ctx-790ca221) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:41:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-10:ctx-0e2f52eb) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:41:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-77:ctx-f8287f50) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:42:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-39:ctx-1af09356) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:42:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-78:ctx-e1b2675f) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:43:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-20:ctx-c9aea423) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:43:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-79:ctx-c459450e) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:44:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-5:ctx-f7fee74e) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:44:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-81:ctx-bc615860) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:45:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-41:ctx-4f287927) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:45:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-82:ctx-092eb488) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:46:27,281 WARN [o.a.c.alerts] (HA-1:ctx-b34246c5)
alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null //
message:: No usage server process running
2014-12-03 00:46:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-21:ctx-62eca8f2) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:46:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-83:ctx-272e6406) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:47:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-11:ctx-23f28317) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:47:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-85:ctx-905ed14b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:48:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-43:ctx-4a673855) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:48:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-86:ctx-1285bc0f) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:49:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-22:ctx-60a35d36) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:49:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-87:ctx-687282ce) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:50:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Unable to
transition the state but we're moving on because it's forced stop
2014-12-03 00:50:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Unable to
cleanup VM: VM[SecondaryStorageVm|s-22-VM] ,since outstanding work item is
not found
2014-12-03 00:50:30,871 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Invocation
exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
item not found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in
state Starting
2014-12-03 00:50:30,871 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Rethrow
exception com.cloud.utils.exception.CloudRuntimeException: Work item not
found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state
Starting
2014-12-03 00:50:30,871 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-6:ctx-46fe9139 job-429/job-432) Unable to
complete AsyncJobVO {id:432, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
AAAAFnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
created: Wed Dec 03 00:50:02 EST 2014}, job origin:429
com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
2014-12-03 00:50:30,873 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-6:ctx-46fe9139 job-429/job-432) Complete async job-432,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAe0pvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
cCBWTVtTZWNvbmRhcnlTdG9yYWdlVm18cy0yMi1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFy
dGluZ3VyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNy
ABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5k
ZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAA
cnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hl
ci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdv
cmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwu
amF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFu
YWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVu
bmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdl
ZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENv
bnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFn
ZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250
ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcu
YXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVx
AH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwu
Y29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4A
G3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFz
ay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4
ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAu
amF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdz
cQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51
dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhy
ACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgAB
TAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHS
HZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
2014-12-03 00:50:31,095 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-3:ctx-0dbe91b2 work-2) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 00:50:31,129 WARN [c.c.u.d.Merovingian2]
(HA-Worker-3:ctx-0dbe91b2 work-2) Was unable to find lock for the
key vm_instance15 and thread id 818653380
2014-12-03 00:50:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-6:ctx-2758188b) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:50:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-89:ctx-eb95a1c2) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:50:34,182 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-3:ctx-0dbe91b2 work-2) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:51:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-45:ctx-7be8bae6) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:51:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-90:ctx-3f76c1d7) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:52:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-23:ctx-e902ca80) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:52:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-91:ctx-4d56948f) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:53:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-12:ctx-0876aaa4) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:53:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-93:ctx-c057ee22) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:54:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-47:ctx-8083fccf) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:54:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-94:ctx-56cebf3c) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:55:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-24:ctx-36c965de) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:55:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-95:ctx-86b1db16) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:56:27,241 WARN [o.a.c.alerts] (HA-2:ctx-627e96a3)
alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null //
message:: No usage server process running
2014-12-03 00:56:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-1:ctx-6000a728) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:56:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-97:ctx-ffbe388e) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:57:32,279 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-49:ctx-a54e4fa0) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:57:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-98:ctx-34e48d1f) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:58:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-25:ctx-ffdd759e) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:58:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-99:ctx-0c1a9fa7) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:59:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-101:ctx-8442d937) Unable to get current
status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:59:32,963 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-51:ctx-9bd2b3cc) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:00:31,170 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-2:ctx-f17ce3b5 work-5) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 01:00:31,208 WARN [c.c.u.d.Merovingian2]
(HA-Worker-2:ctx-f17ce3b5 work-5) Was unable to find lock for the
key vm_instance22 and thread id 52609881
2014-12-03 01:00:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-102:ctx-138c6123) Unable to get current
status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:00:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-26:ctx-3e13656a) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:00:34,263 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-2:ctx-f17ce3b5 work-5) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
2014-12-03 01:01:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-103:ctx-334d11fc) Unable to get current
status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:01:32,856 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Unable to
transition the state but we're moving on because it's forced stop
2014-12-03 01:01:32,856 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Unable to
cleanup VM: VM[ConsoleProxy|v-15-VM] ,since outstanding work item is not
found
2014-12-03 01:01:32,856 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Invocation
exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
item not found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state
Starting
2014-12-03 01:01:32,857 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Rethrow
exception com.cloud.utils.exception.CloudRuntimeException: Work item not
found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 01:01:32,857 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-7:ctx-0a1f299d job-429/job-438) Unable to
complete AsyncJobVO {id:438, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
AAAAD3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
created: Wed Dec 03 01:01:03 EST 2014}, job origin:429
com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 01:01:32,865 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-7:ctx-0a1f299d job-429/job-438) Complete async job-438,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAdUpvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
cCBWTVtDb25zb2xlUHJveHl8di0xNS1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFydGluZ3Vy
AB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZh
LmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJp
bmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNv
bS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZh
dAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9i
cy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQA
DHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUu
amF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250
ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQu
amF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29u
dGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EA
fgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hl
LmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEA
fgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3Vy
cmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4A
CwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh
cQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y
dAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51
dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsA
AALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNv
bGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZh
LnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QA
FkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0D
AAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
2014-12-03 01:01:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-52:ctx-76332a49) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:02:32,282 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-105:ctx-b73de834) Unable to get current
status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:02:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-53:ctx-9b938dc5) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:03:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-27:ctx-28735fd7) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:03:32,967 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-107:ctx-e290c580) Unable to get current
status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:04:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-54:ctx-a53679d7) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:04:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-108:ctx-e1d3829f) Unable to get current
status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:05:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-55:ctx-65e26d72) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:05:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-110:ctx-b755e6ac) Unable to get current
status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:06:27,239 WARN [o.a.c.alerts] (HA-3:ctx-b686e09d)
alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null //
message:: No usage server process running
2014-12-03 01:06:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-28:ctx-230f4b8f) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:06:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-111:ctx-77cd282d) Unable to get current
status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
[root@CSTW1 ~]#