You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by cs user <ac...@gmail.com> on 2015/10/01 11:02:48 UTC
Re: 4.5.1 - Xen 6.5 SP1 - Live migrations failing - Local disk
Hi Koushik,
We've managed to remove the error now regarding host id 27708.... this was
related to a very old secondary storage vm which was in a status of
disconnected but had not been removed. It had long since been deleted.
However, we are still seeing the error, it is very similar to what is
described here:
https://issues.apache.org/jira/browse/CLOUDSTACK-3715
Still seeing this:
ERROR [o.a.c.s.m.XenServerStorageMotionStrategy]
(Work-Job-Executor-6:ctx-8e19de38 job-346708/job-346709 ctx-65bccb97) Error
while migrating vm VM[User|i-2-26631-VM] to host Host[-181-Routing]
com.cloud.exception.OperationTimedoutException: Commands 964614745187418300
to Host 181 timed out after 1200
The migration never happens, its as though the call to the xen host is
never made, or never received by xen.
What ports are used to make this call? I assume either 22 or 443?
Cheers!
On Wed, Sep 30, 2015 at 12:50 PM, Koushik Das <ko...@citrix.com>
wrote:
> As per the logs the host id 27708 is not in the right state and operation
> is failing. Check in DB for the status of host id 27708.
> Also grep for job-343376 and job-343377 in all the MS logs to trace in
> which MS the job originated and where it is getting executed.
>
> -----Original Message-----
> From: cs user [mailto:acldstkusr@gmail.com]
> Sent: Wednesday, 30 September 2015 16:01
> To: users@cloudstack.apache.org
> Subject: Re: 4.5.1 - Xen 6.5 SP1 - Live migrations failing - Local disk
>
> Hi Koushik,
>
> We see errors such as :
>
> com.cloud.utils.exception.CloudRuntimeException: Failed to send command,
> due to Agent:27708, com.cloud.exception.AgentUnavailableException: Resource
> [Host:27708] is unreachable: Host 27708: Host with specified id is not in
> the right state: Disconnected ERROR
> [o.a.c.s.m.XenServerStorageMotionStrategy]
> (Work-Job-Executor-159:ctx-6e40204e job-343376/job-343377 ctx-818f1a51)
> copy failed ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-159:ctx-6e40204e
> job-343376/job-343377 ctx-818f1a51) Invocation exception, caused by:
> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
> VM[User|i-2-26383-VM] along with its volumes.
> INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-159:ctx-6e40204e
> job-343376/job-343377 ctx-818f1a51) Rethrow exception
> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
> VM[User|i-2-26383-VM] along with its volumes.
> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
> VM[User|i-2-26383-VM] along with its volumes.
> com.cloud.utils.exception.CloudRuntimeException: Failed to send command,
> due to Agent:27708, com.cloud.exception.AgentUnavailableException: Resource
> [Host:27708] is unreachable: Host 27708: Host with specified id is not in
> the right state: Disconnected
>
> However, when I look at the hosts in cloudstack, all hosts are connected
> and green, none are reported by cloudstack as being disconnected. So I'm
> not really quite sure what is going wrong. As I say, when only one
> cloudstack management server is running, everything works perfectly fine.
>
> Cheers
>
> On Wed, Sep 30, 2015 at 10:27 AM, Koushik Das <ko...@citrix.com>
> wrote:
>
> > Check logs on all 4 MS in the cluster around the time the operation
> > was done. If you see any error/exceptions that can point to the problem.
> >
> > -----Original Message-----
> > From: cs user [mailto:acldstkusr@gmail.com]
> > Sent: Tuesday, 29 September 2015 20:53
> > To: users@cloudstack.apache.org
> > Subject: 4.5.1 - Xen 6.5 SP1 - Live migrations failing - Local disk
> >
> > Hi Folks,
> >
> > We are seeing some problems when attempting live migrations in a
> > clustered cloudstack environment. We have 4 management servers, and
> > when we attempt a migration of an instance (local storage) from one
> > host to another, the migration times out.
> >
> > However, if we only have one management server running, the migration
> > works fine and we can move the instance around between hosts in the
> > same pod without any problems at all.
> >
> > Just wanted to know if anyone else had come across this issue?
> >
> > Cheers
> >
>
Re: 4.5.1 - Xen 6.5 SP1 - Live migrations failing - Local disk
Posted by cs user <ac...@gmail.com>.
Apologies for the spam.... :-)
Looks like it is this problem.
https://issues.apache.org/jira/browse/CLOUDSTACK-8412
No idea why this didn't appear during my earlier googling.
However, the commit which fixes this appears to already be in 4.5.1 and
4.5.2:
user@ubuntu01:~/cloudstack/cloudstack$ git tag --contains
27b7e49b395d089ff45037faa51e73d979ceb62b
4.5.1
4.5.2
However we have two different environments installed, one on version 4.5.1
and one with 4.5.2, and are seeing the same issue on both....
Cheers
On Thu, Oct 1, 2015 at 2:58 PM, cs user <ac...@gmail.com> wrote:
> Hello,
>
> Found a better error in the logs:
>
> ERROR [c.c.a.t.Request] (AgentManager-Handler-11:null) Caught problem with
> [{"com.cloud.agent.api.MigrateWithStorageCommand":{"vm":{"id":26631,"name":"i-2-26631-VM","bootloader":"PyGrub","type":"Us
> er","cpus":1,"minSpeed":1200,"maxSpeed":1200,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS
> 5.6 (64-bit)","platformEmulator":"CentOS 5
> (64-bit)","bootArgs":"","enableHA":false,"
>
> limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"","params":{"memoryOvercommitRatio":"1","platform":"viridian:true;acpi:1;apic:true;pae:true;nx:t
>
> rue","hypervisortoolsversion":"xenserver56","cpuOvercommitRatio":"4"},"uuid":"b37e76e8-e19a-4100-bf1c-1ee8eba674f2","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"8f51
>
> e73f-9bfc-4a37-bf86-664067a2cf40","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"b114d58e-a37a-6433-c0ff-07f3613a3c68","id":219,"poolType":"LVM","ho
> st":"192.168.97.2","path":"lvm","port":0,"url":"LVM://
> 192.168.97.2/lvm/?ROLE\u003dPrimary\u0026STOREUUID\u003db114d58e-a37a-6433-c0ff-07f3613a3c68
> <http://192.168.97.2/lvm/?ROLE%5Cu003dPrimary%5Cu0026STOREUUID%5Cu003db114d58e-a37a-6433-c0ff-07f3613a3c68>
> "}},"name":"ROOT-26631","size":21474836480,"path":"48
>
> fe494b-ee02-4f08-b7aa-975baecd7b3e","volumeId":119075,"vmName":"i-2-26631-VM","accountId":2,"format":"VHD","provisioningType":"THIN","id":119075,"deviceId":0,"cacheMode":"NONE","hypervisorType":"Xe
>
> nServer"}},"diskSeq":0,"path":"48fe494b-ee02-4f08-b7aa-975baecd7b3e","type":"ROOT","_details":{"managed":"false","storagePort":"0","storageHost":"192.168.97.2","volumeSize":"21474836480"}},{"data":{
>
> "org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,"format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":2000,"defaultNic":true,"pxe
>
> Disable":false,"nicUuid":"aaef8c14-784c-403f-91fc-b6a3167f2595","uuid":"508d0384-5bf0-4434-8155-0b5bb1b33b43","ip":"192.168.8.112","netmask":"255.255.254.0","gateway":"192.168.9.254","mac":"06:68:80:
>
> 00:00:7f","dns1":"192.168.0.224","dns2":"192.168.0.224","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://untagged","isSecurityGroupEnabled":true,"name":"CS-Guest-Public"}],"vcpuMaxLimit":
>
> 16},"volumeToFiler":{"Vol[119075|ROOT|48fe494b-ee02-4f08-b7aa-975baecd7b3e|21474836480]":{"id":223,"uuid":"bc01ca2b-e6d8-a856-27b5-6f6b55a06b1d","host":"192.168.97.1","path":"lvm","port":0,"type":"L
> VM"}},"contextMap":{"job":"job-347068/job-347069"},"wait":0}}]
> com.google.gson.JsonParseException: Expecting object found:
> "Vol[119075|ROOT|48fe494b-ee02-4f08-b7aa-975baecd7b3e|21474836480]"
> at
> com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:100)
> at
> com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
> at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
> at
> com.google.gson.JsonDeserializationContextDefault.fromJsonPrimitive(JsonDeserializationContextDefault.java:85)
> at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:56)
> at
> com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:67)
> at
> com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:33)
> at
> com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
> at
> com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
> at
> com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:117)
> at
> com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
> at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
> at
> com.google.gson.JsonDeserializationContextDefault.fromJsonObject(JsonDeserializationContextDefault.java:76)
> at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:54)
> at com.google.gson.Gson.fromJson(Gson.java:551)
> at com.google.gson.Gson.fromJson(Gson.java:521)
> at
> com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:80)
> at
> com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:40)
> at
> com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
> at
> com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
> at
> com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDeserializationVisitor.java:80)
> at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
> at
> com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeserializationContextDefault.java:67)
> at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:52)
> at com.google.gson.Gson.fromJson(Gson.java:551)
> at com.google.gson.Gson.fromJson(Gson.java:498)
> at com.cloud.agent.transport.Request.getCommands(Request.java:250)
> at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:346)
> at
> com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:666)
> at com.cloud.utils.nio.Task.run(Task.java:84)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> WARN [c.c.u.n.Task] (AgentManager-Handler-11:null) Caught the following
> exception but pushing on
> com.google.gson.JsonParseException: Expecting object found:
> "Vol[119075|ROOT|48fe494b-ee02-4f08-b7aa-975baecd7b3e|21474836480]"
> at
> com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:100)
> at
> com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
> at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
> at
> com.google.gson.JsonDeserializationContextDefault.fromJsonPrimitive(JsonDeserializationContextDefault.java:85)
> at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:56)
> at
> com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:67)
> at
> com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:33)
> at
> com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
> at
> com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
> at
> com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:117)
> at
> com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
> at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
> at
> com.google.gson.JsonDeserializationContextDefault.fromJsonObject(JsonDeserializationContextDefault.java:76)
> at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:54)
> at com.google.gson.Gson.fromJson(Gson.java:551)
> at com.google.gson.Gson.fromJson(Gson.java:521)
> at
> com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:80)
> at
> com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:40)
> at
> com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
> at
> com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
> at
> com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDeserializationVisitor.java:80)
> at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
> at
> com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeserializationContextDefault.java:67)
> at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:52)
> at com.google.gson.Gson.fromJson(Gson.java:551)
> at com.google.gson.Gson.fromJson(Gson.java:498)
> at com.cloud.agent.transport.Request.getCommands(Request.java:250)
> at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:346)
> at
> com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:666)
> at com.cloud.utils.nio.Task.run(Task.java:84)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
>
>
>
> Not sure if that is more helpful?
>
> Cheers!
>
>
> On Thu, Oct 1, 2015 at 10:02 AM, cs user <ac...@gmail.com> wrote:
>
>> Hi Koushik,
>>
>> We've managed to remove the error now regarding host id 27708.... this
>> was related to a very old secondary storage vm which was in a status of
>> disconnected but had not been removed. It had long since been deleted.
>>
>> However, we are still seeing the error, it is very similar to what is
>> described here:
>>
>> https://issues.apache.org/jira/browse/CLOUDSTACK-3715
>>
>> Still seeing this:
>>
>> ERROR [o.a.c.s.m.XenServerStorageMotionStrategy]
>> (Work-Job-Executor-6:ctx-8e19de38 job-346708/job-346709 ctx-65bccb97) Error
>> while migrating vm VM[User|i-2-26631-VM] to host Host[-181-Routing]
>> com.cloud.exception.OperationTimedoutException: Commands
>> 964614745187418300 to Host 181 timed out after 1200
>>
>> The migration never happens, its as though the call to the xen host is
>> never made, or never received by xen.
>>
>> What ports are used to make this call? I assume either 22 or 443?
>>
>> Cheers!
>>
>>
>>
>> On Wed, Sep 30, 2015 at 12:50 PM, Koushik Das <ko...@citrix.com>
>> wrote:
>>
>>> As per the logs the host id 27708 is not in the right state and
>>> operation is failing. Check in DB for the status of host id 27708.
>>> Also grep for job-343376 and job-343377 in all the MS logs to trace in
>>> which MS the job originated and where it is getting executed.
>>>
>>> -----Original Message-----
>>> From: cs user [mailto:acldstkusr@gmail.com]
>>> Sent: Wednesday, 30 September 2015 16:01
>>> To: users@cloudstack.apache.org
>>> Subject: Re: 4.5.1 - Xen 6.5 SP1 - Live migrations failing - Local disk
>>>
>>> Hi Koushik,
>>>
>>> We see errors such as :
>>>
>>> com.cloud.utils.exception.CloudRuntimeException: Failed to send command,
>>> due to Agent:27708, com.cloud.exception.AgentUnavailableException: Resource
>>> [Host:27708] is unreachable: Host 27708: Host with specified id is not in
>>> the right state: Disconnected ERROR
>>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>>> (Work-Job-Executor-159:ctx-6e40204e job-343376/job-343377 ctx-818f1a51)
>>> copy failed ERROR [c.c.v.VmWorkJobHandlerProxy]
>>> (Work-Job-Executor-159:ctx-6e40204e
>>> job-343376/job-343377 ctx-818f1a51) Invocation exception, caused by:
>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>> VM[User|i-2-26383-VM] along with its volumes.
>>> INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-159:ctx-6e40204e
>>> job-343376/job-343377 ctx-818f1a51) Rethrow exception
>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>> VM[User|i-2-26383-VM] along with its volumes.
>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>> VM[User|i-2-26383-VM] along with its volumes.
>>> com.cloud.utils.exception.CloudRuntimeException: Failed to send command,
>>> due to Agent:27708, com.cloud.exception.AgentUnavailableException: Resource
>>> [Host:27708] is unreachable: Host 27708: Host with specified id is not in
>>> the right state: Disconnected
>>>
>>> However, when I look at the hosts in cloudstack, all hosts are connected
>>> and green, none are reported by cloudstack as being disconnected. So I'm
>>> not really quite sure what is going wrong. As I say, when only one
>>> cloudstack management server is running, everything works perfectly fine.
>>>
>>> Cheers
>>>
>>> On Wed, Sep 30, 2015 at 10:27 AM, Koushik Das <ko...@citrix.com>
>>> wrote:
>>>
>>> > Check logs on all 4 MS in the cluster around the time the operation
>>> > was done. If you see any error/exceptions that can point to the
>>> problem.
>>> >
>>> > -----Original Message-----
>>> > From: cs user [mailto:acldstkusr@gmail.com]
>>> > Sent: Tuesday, 29 September 2015 20:53
>>> > To: users@cloudstack.apache.org
>>> > Subject: 4.5.1 - Xen 6.5 SP1 - Live migrations failing - Local disk
>>> >
>>> > Hi Folks,
>>> >
>>> > We are seeing some problems when attempting live migrations in a
>>> > clustered cloudstack environment. We have 4 management servers, and
>>> > when we attempt a migration of an instance (local storage) from one
>>> > host to another, the migration times out.
>>> >
>>> > However, if we only have one management server running, the migration
>>> > works fine and we can move the instance around between hosts in the
>>> > same pod without any problems at all.
>>> >
>>> > Just wanted to know if anyone else had come across this issue?
>>> >
>>> > Cheers
>>> >
>>>
>>
>>
>
Re: 4.5.1 - Xen 6.5 SP1 - Live migrations failing - Local disk
Posted by cs user <ac...@gmail.com>.
Hello,
Found a better error in the logs:
ERROR [c.c.a.t.Request] (AgentManager-Handler-11:null) Caught problem with
[{"com.cloud.agent.api.MigrateWithStorageCommand":{"vm":{"id":26631,"name":"i-2-26631-VM","bootloader":"PyGrub","type":"Us
er","cpus":1,"minSpeed":1200,"maxSpeed":1200,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS
5.6 (64-bit)","platformEmulator":"CentOS 5
(64-bit)","bootArgs":"","enableHA":false,"
limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"","params":{"memoryOvercommitRatio":"1","platform":"viridian:true;acpi:1;apic:true;pae:true;nx:t
rue","hypervisortoolsversion":"xenserver56","cpuOvercommitRatio":"4"},"uuid":"b37e76e8-e19a-4100-bf1c-1ee8eba674f2","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"8f51
e73f-9bfc-4a37-bf86-664067a2cf40","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"b114d58e-a37a-6433-c0ff-07f3613a3c68","id":219,"poolType":"LVM","ho
st":"192.168.97.2","path":"lvm","port":0,"url":"LVM://
192.168.97.2/lvm/?ROLE\u003dPrimary\u0026STOREUUID\u003db114d58e-a37a-6433-c0ff-07f3613a3c68
"}},"name":"ROOT-26631","size":21474836480,"path":"48
fe494b-ee02-4f08-b7aa-975baecd7b3e","volumeId":119075,"vmName":"i-2-26631-VM","accountId":2,"format":"VHD","provisioningType":"THIN","id":119075,"deviceId":0,"cacheMode":"NONE","hypervisorType":"Xe
nServer"}},"diskSeq":0,"path":"48fe494b-ee02-4f08-b7aa-975baecd7b3e","type":"ROOT","_details":{"managed":"false","storagePort":"0","storageHost":"192.168.97.2","volumeSize":"21474836480"}},{"data":{
"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,"format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":2000,"defaultNic":true,"pxe
Disable":false,"nicUuid":"aaef8c14-784c-403f-91fc-b6a3167f2595","uuid":"508d0384-5bf0-4434-8155-0b5bb1b33b43","ip":"192.168.8.112","netmask":"255.255.254.0","gateway":"192.168.9.254","mac":"06:68:80:
00:00:7f","dns1":"192.168.0.224","dns2":"192.168.0.224","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://untagged","isSecurityGroupEnabled":true,"name":"CS-Guest-Public"}],"vcpuMaxLimit":
16},"volumeToFiler":{"Vol[119075|ROOT|48fe494b-ee02-4f08-b7aa-975baecd7b3e|21474836480]":{"id":223,"uuid":"bc01ca2b-e6d8-a856-27b5-6f6b55a06b1d","host":"192.168.97.1","path":"lvm","port":0,"type":"L
VM"}},"contextMap":{"job":"job-347068/job-347069"},"wait":0}}]
com.google.gson.JsonParseException: Expecting object found:
"Vol[119075|ROOT|48fe494b-ee02-4f08-b7aa-975baecd7b3e|21474836480]"
at
com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:100)
at
com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
at
com.google.gson.JsonDeserializationContextDefault.fromJsonPrimitive(JsonDeserializationContextDefault.java:85)
at
com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:56)
at
com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:67)
at
com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:33)
at
com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
at
com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
at
com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:117)
at
com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
at
com.google.gson.JsonDeserializationContextDefault.fromJsonObject(JsonDeserializationContextDefault.java:76)
at
com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:54)
at com.google.gson.Gson.fromJson(Gson.java:551)
at com.google.gson.Gson.fromJson(Gson.java:521)
at
com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:80)
at
com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:40)
at
com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
at
com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
at
com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDeserializationVisitor.java:80)
at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
at
com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeserializationContextDefault.java:67)
at
com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:52)
at com.google.gson.Gson.fromJson(Gson.java:551)
at com.google.gson.Gson.fromJson(Gson.java:498)
at com.cloud.agent.transport.Request.getCommands(Request.java:250)
at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:346)
at
com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:666)
at com.cloud.utils.nio.Task.run(Task.java:84)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
WARN [c.c.u.n.Task] (AgentManager-Handler-11:null) Caught the following
exception but pushing on
com.google.gson.JsonParseException: Expecting object found:
"Vol[119075|ROOT|48fe494b-ee02-4f08-b7aa-975baecd7b3e|21474836480]"
at
com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:100)
at
com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
at
com.google.gson.JsonDeserializationContextDefault.fromJsonPrimitive(JsonDeserializationContextDefault.java:85)
at
com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:56)
at
com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:67)
at
com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:33)
at
com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
at
com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
at
com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:117)
at
com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
at
com.google.gson.JsonDeserializationContextDefault.fromJsonObject(JsonDeserializationContextDefault.java:76)
at
com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:54)
at com.google.gson.Gson.fromJson(Gson.java:551)
at com.google.gson.Gson.fromJson(Gson.java:521)
at
com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:80)
at
com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:40)
at
com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
at
com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
at
com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDeserializationVisitor.java:80)
at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
at
com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeserializationContextDefault.java:67)
at
com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:52)
at com.google.gson.Gson.fromJson(Gson.java:551)
at com.google.gson.Gson.fromJson(Gson.java:498)
at com.cloud.agent.transport.Request.getCommands(Request.java:250)
at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:346)
at
com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:666)
at com.cloud.utils.nio.Task.run(Task.java:84)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Not sure if that is more helpful?
Cheers!
On Thu, Oct 1, 2015 at 10:02 AM, cs user <ac...@gmail.com> wrote:
> Hi Koushik,
>
> We've managed to remove the error now regarding host id 27708.... this was
> related to a very old secondary storage vm which was in a status of
> disconnected but had not been removed. It had long since been deleted.
>
> However, we are still seeing the error, it is very similar to what is
> described here:
>
> https://issues.apache.org/jira/browse/CLOUDSTACK-3715
>
> Still seeing this:
>
> ERROR [o.a.c.s.m.XenServerStorageMotionStrategy]
> (Work-Job-Executor-6:ctx-8e19de38 job-346708/job-346709 ctx-65bccb97) Error
> while migrating vm VM[User|i-2-26631-VM] to host Host[-181-Routing]
> com.cloud.exception.OperationTimedoutException: Commands
> 964614745187418300 to Host 181 timed out after 1200
>
> The migration never happens, its as though the call to the xen host is
> never made, or never received by xen.
>
> What ports are used to make this call? I assume either 22 or 443?
>
> Cheers!
>
>
>
> On Wed, Sep 30, 2015 at 12:50 PM, Koushik Das <ko...@citrix.com>
> wrote:
>
>> As per the logs the host id 27708 is not in the right state and operation
>> is failing. Check in DB for the status of host id 27708.
>> Also grep for job-343376 and job-343377 in all the MS logs to trace in
>> which MS the job originated and where it is getting executed.
>>
>> -----Original Message-----
>> From: cs user [mailto:acldstkusr@gmail.com]
>> Sent: Wednesday, 30 September 2015 16:01
>> To: users@cloudstack.apache.org
>> Subject: Re: 4.5.1 - Xen 6.5 SP1 - Live migrations failing - Local disk
>>
>> Hi Koushik,
>>
>> We see errors such as :
>>
>> com.cloud.utils.exception.CloudRuntimeException: Failed to send command,
>> due to Agent:27708, com.cloud.exception.AgentUnavailableException: Resource
>> [Host:27708] is unreachable: Host 27708: Host with specified id is not in
>> the right state: Disconnected ERROR
>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>> (Work-Job-Executor-159:ctx-6e40204e job-343376/job-343377 ctx-818f1a51)
>> copy failed ERROR [c.c.v.VmWorkJobHandlerProxy]
>> (Work-Job-Executor-159:ctx-6e40204e
>> job-343376/job-343377 ctx-818f1a51) Invocation exception, caused by:
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>> VM[User|i-2-26383-VM] along with its volumes.
>> INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-159:ctx-6e40204e
>> job-343376/job-343377 ctx-818f1a51) Rethrow exception
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>> VM[User|i-2-26383-VM] along with its volumes.
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>> VM[User|i-2-26383-VM] along with its volumes.
>> com.cloud.utils.exception.CloudRuntimeException: Failed to send command,
>> due to Agent:27708, com.cloud.exception.AgentUnavailableException: Resource
>> [Host:27708] is unreachable: Host 27708: Host with specified id is not in
>> the right state: Disconnected
>>
>> However, when I look at the hosts in cloudstack, all hosts are connected
>> and green, none are reported by cloudstack as being disconnected. So I'm
>> not really quite sure what is going wrong. As I say, when only one
>> cloudstack management server is running, everything works perfectly fine.
>>
>> Cheers
>>
>> On Wed, Sep 30, 2015 at 10:27 AM, Koushik Das <ko...@citrix.com>
>> wrote:
>>
>> > Check logs on all 4 MS in the cluster around the time the operation
>> > was done. If you see any error/exceptions that can point to the problem.
>> >
>> > -----Original Message-----
>> > From: cs user [mailto:acldstkusr@gmail.com]
>> > Sent: Tuesday, 29 September 2015 20:53
>> > To: users@cloudstack.apache.org
>> > Subject: 4.5.1 - Xen 6.5 SP1 - Live migrations failing - Local disk
>> >
>> > Hi Folks,
>> >
>> > We are seeing some problems when attempting live migrations in a
>> > clustered cloudstack environment. We have 4 management servers, and
>> > when we attempt a migration of an instance (local storage) from one
>> > host to another, the migration times out.
>> >
>> > However, if we only have one management server running, the migration
>> > works fine and we can move the instance around between hosts in the
>> > same pod without any problems at all.
>> >
>> > Just wanted to know if anyone else had come across this issue?
>> >
>> > Cheers
>> >
>>
>
>
RE: CloudStack Users survey
Posted by Giles Sirett <gi...@shapeblue.com>.
I can help with that Ingo
What's the problem ?
Regards
Giles
-----Original Message-----
From: Jochim, Ingo [mailto:Ingo.Jochim@bautzen-it.de]
Sent: 01 October 2015 11:29
To: users@cloudstack.apache.org
Subject: CloudStack Users survey
Hi all,
I just liked to do the survey on the cloudstack website.
But the form seems not to work.
https://cloudstack.apache.org/survey.html
Any ideas who can help to fix it?
Regards,
Ingo
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
CloudStack Users survey
Posted by "Jochim, Ingo" <In...@bautzen-it.de>.
Hi all,
I just liked to do the survey on the cloudstack website.
But the form seems not to work.
https://cloudstack.apache.org/survey.html
Any ideas who can help to fix it?
Regards,
Ingo