You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@cloudstack.apache.org by Rohit Yadav <ro...@shapeblue.com> on 2015/08/13 14:06:21 UTC

[SHARE] VM migrations failing with Ubuntu 14.04.3

Hi all,

I’ve been testing a bug on KVM which involves VM migration, and in that process I found that VM migrations failed in my environment twice even with clean setups of Ubuntu 14.04.3 (lastest and updated packages) based KVM hosts against CloudStack 4.5.2-SNAPSHOT (latest 4.5).

I wanted to share this issue to confirm if this is just my environment or if someone experienced similar issue with latest Ubuntu 14.04 based KVM hosts.

The specific errors I found in the logs were something like "qemuMigrationCancelDriveMirror:1421 : Unable to stop block job”, which are likely reproduced by others such as:

https://www.redhat.com/archives/libvirt-users/2014-June/msg00057.html
http://lists.nongnu.org/archive/html/qemu-devel/2014-12/msg03093.html

A possible solution could be to download libvirt to 1.1.x.

Regards,
Rohit Yadav
Software Architect, ShapeBlue

[cid:9DD97B41-04C5-45F0-92A7-951F3E962F7A]

M. +91 88 262 30892 | rohit.yadav@shapeblue.com<ma...@shapeblue.com>
Blog: bhaisaab.org<http://bhaisaab.org> | Twitter: @_bhaisaab

Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.

Re: [SHARE] VM migrations failing with Ubuntu 14.04.3

Posted by Rohit Yadav <ro...@shapeblue.com>.

Hi Wido,

On 14-Aug-2015, at 2:41 pm, Rohit Yadav <ro...@shapeblue.com>> wrote:

Have you disabled AppArmor on that machine? We've seen many machines
with random weird issues when AppArmor was enabled.

I remember disabling apparmor before the upgrade, but did not check if the upgrade re-enabled it. Will check that out.

I found that Ubuntu 14.04.3 somehow installed/setup/enabled apparmor, I removed it completely and but even after that migration failed for me with the same error.

I finally got the VM migration to work by downgrading libvirt to the version "1.2.2-0ubuntu13.1.7”.

Regards,
Rohit Yadav
Software Architect, ShapeBlue

[cid:9DD97B41-04C5-45F0-92A7-951F3E962F7A]

M. +91 88 262 30892 | rohit.yadav@shapeblue.com<ma...@shapeblue.com>
Blog: bhaisaab.org<http://bhaisaab.org> | Twitter: @_bhaisaab

Find out more about ShapeBlue and our range of CloudStack related services

Re: [SHARE] VM migrations failing with Ubuntu 14.04.3

Posted by Rohit Yadav <ro...@shapeblue.com>.

Hi,

On 14-Aug-2015, at 2:19 pm, Wido den Hollander <wi...@widodh.nl>> wrote:

Have you disabled AppArmor on that machine? We've seen many machines
with random weird issues when AppArmor was enabled.

I remember disabling apparmor before the upgrade, but did not check if the upgrade re-enabled it. Will check that out.

Regards,
Rohit Yadav
Software Architect, ShapeBlue

[cid:9DD97B41-04C5-45F0-92A7-951F3E962F7A]

M. +91 88 262 30892 | rohit.yadav@shapeblue.com<ma...@shapeblue.com>
Blog: bhaisaab.org<http://bhaisaab.org> | Twitter: @_bhaisaab

Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge - rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

Re: [SHARE] VM migrations failing with Ubuntu 14.04.3

Posted by Rohit Yadav <ro...@shapeblue.com>.

Hi Bruno,

Thanks for checking, while it could be an environment related issue - the VM migration finally started working for me after I downgraded the libvirt-bin and libvirt0 package on my KVM hosts.

On 14-Aug-2015, at 4:59 pm, Milamber <mi...@apache.org>> wrote:

Hello Rohit,

No issue for migration in my test environment (KVM over KVM) with the 4.5.2 RC1 on Ubuntu 10.04.3 (up to date)


AppArmor is disabled for libvirtd and virt-aa-helper

(see my ansible script: https://github.com/milamberspace/ansible-cloudstack-ubuntu-aio/blob/master/roles/cloudstack-node/tasks/libvirtd.yml)

Milamber

Regards,
Rohit Yadav
Software Architect, ShapeBlue


[cid:9DD97B41-04C5-45F0-92A7-951F3E962F7A]


M. +91 88 262 30892 | rohit.yadav@shapeblue.com<ma...@shapeblue.com>
Blog: bhaisaab.org<http://bhaisaab.org> | Twitter: @_bhaisaab




Find out more about ShapeBlue and our range of CloudStack related services

IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge - rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>

This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.

Re: [SHARE] VM migrations failing with Ubuntu 14.04.3

Posted by Milamber <mi...@apache.org>.

Hello Rohit,

No issue for migration in my test environment (KVM over KVM) with the 
4.5.2 RC1 on Ubuntu 10.04.3 (up to date)


AppArmor is disabled for libvirtd and virt-aa-helper

(see my ansible script: 
https://github.com/milamberspace/ansible-cloudstack-ubuntu-aio/blob/master/roles/cloudstack-node/tasks/libvirtd.yml)

Milamber



On 14/08/2015 09:49, Wido den Hollander wrote:
>
> On 13-08-15 14:06, Rohit Yadav wrote:
>> Hi all,
>>
>> I’ve been testing a bug on KVM which involves VM migration, and in that
>> process I found that VM migrations failed in my environment twice even
>> with clean setups of Ubuntu 14.04.3 (lastest and updated packages) based
>> KVM hosts against CloudStack 4.5.2-SNAPSHOT (latest 4.5).
>>
> We haven't seen issue with migrations in production, but I'd have to
> look more closely.
>
>> I wanted to share this issue to confirm if this is just my environment
>> or if someone experienced similar issue with latest Ubuntu 14.04 based
>> KVM hosts.
>>
>> The specific errors I found in the logs were something like
>> "qemuMigrationCancelDriveMirror:1421 : Unable to stop block job”, which
>> are likely reproduced by others such as:
>>
>> https://www.redhat.com/archives/libvirt-users/2014-June/msg00057.html
>> http://lists.nongnu.org/archive/html/qemu-devel/2014-12/msg03093.html
>>
>> A possible solution could be to download libvirt to 1.1.x.
>>
> Have you disabled AppArmor on that machine? We've seen many machines
> with random weird issues when AppArmor was enabled.
>
> Wido
>
>> Regards,
>> Rohit Yadav
>> Software Architect, ShapeBlue
>>
>>
>>
>>
>> M. +91 88 262 30892 | rohit.yadav@shapeblue.com
>> <ma...@shapeblue.com>
>> Blog: bhaisaab.org <http://bhaisaab.org> | Twitter: @_bhaisaab
>>
>>
>>
>>
>> Find out more about ShapeBlue and our range of CloudStack related services
>>
>> IaaS Cloud Design & Build
>> <http://shapeblue.com/iaas-cloud-design-and-build//>
>> CSForge – rapid IaaS deployment framework <http://shapeblue.com/csforge/>
>> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/>
>> CloudStack Software Engineering
>> <http://shapeblue.com/cloudstack-software-engineering/>
>> CloudStack Infrastructure Support
>> <http://shapeblue.com/cloudstack-infrastructure-support/>
>> CloudStack Bootcamp Training Courses
>> <http://shapeblue.com/cloudstack-training/>
>>
>> This email and any attachments to it may be confidential and are
>> intended solely for the use of the individual to whom it is addressed.
>> Any views or opinions expressed are solely those of the author and do
>> not necessarily represent those of Shape Blue Ltd or related companies.
>> If you are not the intended recipient of this email, you must neither
>> take any action based upon its contents, nor copy or show it to anyone.
>> Please contact the sender if you believe you have received this email in
>> error. Shape Blue Ltd is a company incorporated in England & Wales.
>> ShapeBlue Services India LLP is a company incorporated in India and is
>> operated under license from Shape Blue Ltd. Shape Blue Brasil
>> Consultoria Ltda is a company incorporated in Brasil and is operated
>> under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
>> registered by The Republic of South Africa and is traded under license
>> from Shape Blue Ltd. ShapeBlue is a registered trademark.

Re: [SHARE] VM migrations failing with Ubuntu 14.04.3

Posted by Wido den Hollander <wi...@widodh.nl>.


On 13-08-15 14:06, Rohit Yadav wrote:
> Hi all,
> 
> I’ve been testing a bug on KVM which involves VM migration, and in that
> process I found that VM migrations failed in my environment twice even
> with clean setups of Ubuntu 14.04.3 (lastest and updated packages) based
> KVM hosts against CloudStack 4.5.2-SNAPSHOT (latest 4.5).
> 

We haven't seen issue with migrations in production, but I'd have to
look more closely.

> I wanted to share this issue to confirm if this is just my environment
> or if someone experienced similar issue with latest Ubuntu 14.04 based
> KVM hosts.
> 
> The specific errors I found in the logs were something like
> "qemuMigrationCancelDriveMirror:1421 : Unable to stop block job”, which
> are likely reproduced by others such as:
> 
> https://www.redhat.com/archives/libvirt-users/2014-June/msg00057.html
> http://lists.nongnu.org/archive/html/qemu-devel/2014-12/msg03093.html
> 
> A possible solution could be to download libvirt to 1.1.x.
> 

Have you disabled AppArmor on that machine? We've seen many machines
with random weird issues when AppArmor was enabled.

Wido

> Regards,
> Rohit Yadav
> Software Architect, ShapeBlue
> 
> 
> 
> 
> M. +91 88 262 30892 | rohit.yadav@shapeblue.com
> <ma...@shapeblue.com>
> Blog: bhaisaab.org <http://bhaisaab.org> | Twitter: @_bhaisaab
> 
> 
> 
> 
> Find out more about ShapeBlue and our range of CloudStack related services
> 
> IaaS Cloud Design & Build
> <http://shapeblue.com/iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment framework <http://shapeblue.com/csforge/>
> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/>
> CloudStack Software Engineering
> <http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support
> <http://shapeblue.com/cloudstack-infrastructure-support/>
> CloudStack Bootcamp Training Courses
> <http://shapeblue.com/cloudstack-training/>
> 
> This email and any attachments to it may be confidential and are
> intended solely for the use of the individual to whom it is addressed.
> Any views or opinions expressed are solely those of the author and do
> not necessarily represent those of Shape Blue Ltd or related companies.
> If you are not the intended recipient of this email, you must neither
> take any action based upon its contents, nor copy or show it to anyone.
> Please contact the sender if you believe you have received this email in
> error. Shape Blue Ltd is a company incorporated in England & Wales.
> ShapeBlue Services India LLP is a company incorporated in India and is
> operated under license from Shape Blue Ltd. Shape Blue Brasil
> Consultoria Ltda is a company incorporated in Brasil and is operated
> under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
> registered by The Republic of South Africa and is traded under license
> from Shape Blue Ltd. ShapeBlue is a registered trademark.

Re: [SHARE] VM migrations failing with Ubuntu 14.04.3

Posted by Rohit Yadav <ro...@shapeblue.com>.

On 14-Aug-2015, at 4:13 am, Milamber <mi...@apache.org>> wrote:

On physical environment or a virtual environment (cloudstack under KVM?) ?

In my case I was running Ubuntu 14.04.3 KVM hosts as VMs (nested virtualization), in one test on top of a Ubuntu 15.04 host and in another on top of Ubuntu 14.04.2 host, it failed for me in both cases.

Regards,
Rohit Yadav
Software Architect, ShapeBlue

[cid:9DD97B41-04C5-45F0-92A7-951F3E962F7A]

M. +91 88 262 30892 | rohit.yadav@shapeblue.com<ma...@shapeblue.com>
Blog: bhaisaab.org<http://bhaisaab.org> | Twitter: @_bhaisaab

Find out more about ShapeBlue and our range of CloudStack related services

Re: [SHARE] VM migrations failing with Ubuntu 14.04.3

Posted by Milamber <mi...@apache.org>.

Hello Rohit,

On physical environment or a virtual environment (cloudstack under KVM?) ?

I will try to setup the same env here to validate or not the issue.

Milamber

On 13/08/2015 13:06, Rohit Yadav wrote:
> Hi all,
>
> I’ve been testing a bug on KVM which involves VM migration, and in 
> that process I found that VM migrations failed in my environment twice 
> even with clean setups of Ubuntu 14.04.3 (lastest and updated 
> packages) based KVM hosts against CloudStack 4.5.2-SNAPSHOT (latest 4.5).
>
> I wanted to share this issue to confirm if this is just my environment 
> or if someone experienced similar issue with latest Ubuntu 14.04 based 
> KVM hosts.
>
> The specific errors I found in the logs were something like 
> "qemuMigrationCancelDriveMirror:1421 : Unable to stop block job”, 
> which are likely reproduced by others such as:
>
> https://www.redhat.com/archives/libvirt-users/2014-June/msg00057.html
> http://lists.nongnu.org/archive/html/qemu-devel/2014-12/msg03093.html
>
> A possible solution could be to download libvirt to 1.1.x.
>
> Regards,
> Rohit Yadav
> Software Architect, ShapeBlue
>
>
>
>
> M. +91 88 262 30892 | rohit.yadav@shapeblue.com 
> <ma...@shapeblue.com>
> Blog: bhaisaab.org <http://bhaisaab.org> | Twitter: @_bhaisaab
>
>
>
>
> Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build 
> <http://shapeblue.com/iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment framework <http://shapeblue.com/csforge/>
> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/>
> CloudStack Software Engineering 
> <http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support 
> <http://shapeblue.com/cloudstack-infrastructure-support/>
> CloudStack Bootcamp Training Courses 
> <http://shapeblue.com/cloudstack-training/>
>
> This email and any attachments to it may be confidential and are 
> intended solely for the use of the individual to whom it is addressed. 
> Any views or opinions expressed are solely those of the author and do 
> not necessarily represent those of Shape Blue Ltd or related 
> companies. If you are not the intended recipient of this email, you 
> must neither take any action based upon its contents, nor copy or show 
> it to anyone. Please contact the sender if you believe you have 
> received this email in error. Shape Blue Ltd is a company incorporated 
> in England & Wales. ShapeBlue Services India LLP is a company 
> incorporated in India and is operated under license from Shape Blue 
> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in 
> Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA 
> Pty Ltd is a company registered by The Republic of South Africa and is 
> traded under license from Shape Blue Ltd. ShapeBlue is a registered 
> trademark.