You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Nicolas Bouige <n....@dimsi.fr> on 2016/12/01 22:39:28 UTC
Volume migration unable to serialize
Hello team,
We are trying to migrate a volume (1TB) from a primary storage to antoher one.
The VM is stopped but we got this error log :
2016-12-01 20:31:42,629 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-43:ctx-5883082d job-89033) (logid:df7aaf8e) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd
com.cloud.utils.exception.CloudRuntimeException: Unable to serialize: Job is cancelled as it has been blocking others for too long
at org.apache.cloudstack.framework.jobs.impl.JobSerializerHelper.fromObjectSerializedString(JobSerializerHelper.java:134)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.unmarshallResultObject(AsyncJobManagerImpl.java:759)
at com.cloud.vm.VirtualMachineManagerImpl.storageMigration(VirtualMachineManagerImpl.java:1854)
at com.cloud.vm.UserVmManagerImpl.vmStorageMigration(UserVmManagerImpl.java:4390)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy198.vmStorageMigration(Unknown Source)
at org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd.execute(MigrateVMCmd.java:176)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:132)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:109)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:594)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:541)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.StreamCorruptedException: invalid stream header: 2686E2B1
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:806)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
at org.apache.cloudstack.framework.jobs.impl.JobSerializerHelper.fromObjectSerializedString(JobSerializerHelper.java:128)
... 29 more
2016-12-01 20:31:42,630 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-43:ctx-5883082d job-89033) (logid:df7aaf8e) Complete async job-89033, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable to serialize: Job is cancelled as it has been blocking others for too long"}
The probleme seems to be only a kind of timeout threshold, we have changed these global options to "0" :
migratewait: 3600
storage.pool.max.waitseconds: 3600
vm.op.cancel.interval: 3600
vm.op.cleanup.wait: 3600
wait:1800
vm.tranisition.wait.interval:3600
But we still get the error after one hour and on xenserver side the job is pending...
There is way to migrate big volume between two primary storage ? or to increase the threshold ?
Thanks,
N.B
Nicolas Bouige
RE: Volume migration unable to serialize
Posted by Nicolas Bouige <n....@dimsi.fr>.
Juste one more question :
These settings works for a live migration as well ?
Nicolas Bouige
DIMSI
cloud.dimsi.fr<http://www.cloud.dimsi.fr>
4, avenue Laurent Cely
Tour d'Asnière - 92600 Asnière sur Seine
T/ +33 (0)6 28 98 53 40
________________________________
De : Nicolas Bouige
Envoyé : vendredi 2 décembre 2016 09:10:28
À : users@cloudstack.apache.org
Objet : RE: Volume migration unable to serialize
Hello,
Thanks a lot Makrand and Sergey, im going to try today !
Have a good day,
Best regards,
Nicolas Bouige
________________________________
De : Makrand <ma...@gmail.com>
Envoyé : vendredi 2 décembre 2016 06:38:49
À : users@cloudstack.apache.org
Objet : Re: Volume migration unable to serialize
Hi Nicolas,
With xenserver its always issue with moving bigger volumes.
Here are the parameters that worked for me for bigger volume migrations.
Just need to put 1 additional zero at end really.
migratewait: 36000
storage.pool.max.waitseconds: 36000
wait:18000
*wait *is most critical of all and bit confusing in name.
--
Makrand
On Fri, Dec 2, 2016 at 4:32 AM, Sergey Levitskiy <
Sergey.Levitskiy@autodesk.com> wrote:
> You need to adjust this threshold
>
> job.cancel.threshold.minutes
>
>
>
> Thanks,
> Sergey
>
>
>
RE: Volume migration unable to serialize
Posted by Nicolas Bouige <n....@dimsi.fr>.
Hello,
Thanks a lot Makrand and Sergey, im going to try today !
Have a good day,
Best regards,
Nicolas Bouige
________________________________
De : Makrand <ma...@gmail.com>
Envoyé : vendredi 2 décembre 2016 06:38:49
À : users@cloudstack.apache.org
Objet : Re: Volume migration unable to serialize
Hi Nicolas,
With xenserver its always issue with moving bigger volumes.
Here are the parameters that worked for me for bigger volume migrations.
Just need to put 1 additional zero at end really.
migratewait: 36000
storage.pool.max.waitseconds: 36000
wait:18000
*wait *is most critical of all and bit confusing in name.
--
Makrand
On Fri, Dec 2, 2016 at 4:32 AM, Sergey Levitskiy <
Sergey.Levitskiy@autodesk.com> wrote:
> You need to adjust this threshold
>
> job.cancel.threshold.minutes
>
>
>
> Thanks,
> Sergey
>
>
>
Re: Volume migration unable to serialize
Posted by Makrand <ma...@gmail.com>.
Hi Nicolas,
With xenserver its always issue with moving bigger volumes.
Here are the parameters that worked for me for bigger volume migrations.
Just need to put 1 additional zero at end really.
migratewait: 36000
storage.pool.max.waitseconds: 36000
wait:18000
*wait *is most critical of all and bit confusing in name.
--
Makrand
On Fri, Dec 2, 2016 at 4:32 AM, Sergey Levitskiy <
Sergey.Levitskiy@autodesk.com> wrote:
> You need to adjust this threshold
>
> job.cancel.threshold.minutes
>
>
>
> Thanks,
> Sergey
>
>
>
Re: Volume migration unable to serialize
Posted by Sergey Levitskiy <Se...@autodesk.com>.
You need to adjust this threshold
job.cancel.threshold.minutes
Thanks,
Sergey