You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Vladislav Nazarenko <vl...@gmail.com> on 2016/02/03 16:08:18 UTC
Self-fencing when storage not available
Hi All,
I'm testing Cloudstack 4.6 now
When doing some tests with storage (nfs), I was wondering, that the
hosts self-fence itself by reboot, when a storage not writable ...
Even more, my cluster had a working storage and I added one more, which
was not writable due to missing user permissions... So effectively all
the VMs hard drive were located on the working storage,
but the problem with permissions on the new one causes the entire
cluster to reboot :(
Is this a bug or a correct behavior?
Is it able to avoid rebooting at least?
Thank you in advance
Vlad
Re: Self-fencing when storage not available
Posted by Vladislav Nazarenko <vl...@gmail.com>.
I saw your suggestion in the ticket ...
Thanx, Paul
On 07.02.2016 20:51, Paul Angus wrote:
> Well, to start off with I'll add the suggestion to the ticket. https://issues.apache.org/jira/browse/CLOUDSTACK-3367
>
>
>
>
> Paul Angus
> VP Technology , ShapeBlue
>
>
> t: @cloudyangus<te...@cloudyangus>
>
> e: paul.angus@shapeblue.com<ma...@shapeblue.com> | w: www.shapeblue.com<http://www.shapeblue.com>
>
>
>
>
>
> -----Original Message-----
> From: Vladislav Nazarenko [mailto:vladislav.nazarenko@gmail.com]
> Sent: Friday, February 5, 2016 5:29 PM
> To: users@cloudstack.apache.org
> Subject: Re: Self-fencing when storage not available
>
> right, Paul ... this might be a good suggestion ... How could we bring it to the developers?
>
>
> On 04.02.2016 20:59, Paul Angus wrote:
>> Just to give some background (as I asked the same questions years ago). *IF* you only have one shared storage pool per cluster (and no local storage) and it goes wonky, then most likely fix is to restart the host(s). The VMs have already crashed because the primary storage is disconnected and the host is unable to reconnect it.
>>
>> That’s why it reboots.
>>
>> As everyone who has a cloud larger than their laptop knows, it also nukes the good VMs on other storage pools and local storage.
>>
>> A global setting which allows operators to choose the behaviour they want might be a solution?
>>
>>
>>
>>
>> Paul Angus
>> VP Technology , ShapeBlue
>>
>>
>> t: @cloudyangus<te...@cloudyangus>
>>
>> e: paul.angus@shapeblue.com<ma...@shapeblue.com> | w: www.shapeblue.com<http://www.shapeblue.com>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Vladislav Nazarenko [mailto:vladislav.nazarenko@gmail.com]
>> Sent: Thursday, February 4, 2016 5:35 PM
>> To: users@cloudstack.apache.org
>> Subject: Re: Self-fencing when storage not available
>>
>> So ... found some information about this ... looks like this is an annoying problem (bug), which isn't going to be fixed ... But at least there is a possibility to change the behavior modifying the script
>>
>> - https://admintweets.wordpress.com/tag/cloudstack-2/
>> - https://www.rootusers.com/edit-the-xenserver-storage-heartbeat/
>> - https://issues.apache.org/jira/browse/CLOUDSTACK-3367
>>
>> Thanks a lot, guys ..
>>
>> On 03.02.2016 20:05, Glenn Wagner wrote:
>>> Hi
>>>
>>> As Lucian said , you can edit the script and remove the reboot , but
>>> yes this is the normal behaviour of the script
>>>
>>> Glenn
>>>
>>>
>>> -----Original Message-----
>>> From: Nux! [mailto:nux@li.nux.ro]
>>> Sent: Wednesday, 03 February 2016 8:32 PM
>>> To: users@cloudstack.apache.org
>>> Subject: Re: Self-fencing when storage not available
>>>
>>> You can modify the script to not reboot, but until we find a better way to deal with it this is correct behaviour. It sucks it reboots VMs on healthy storage though.
>>>
>>> --
>>> Sent from the Delta quadrant using Borg technology!
>>>
>>> Nux!
>>> www.nux.ro
>>>
>>> ----- Original Message -----
>>>> From: "Vladislav Nazarenko" <vl...@gmail.com>
>>>> To: users@cloudstack.apache.org
>>>> Sent: Wednesday, 3 February, 2016 16:24:34
>>>> Subject: Re: Self-fencing when storage not available Hi Glenn,
>>>>
>>>> we use KVM ... I also found the script:
>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.s
>>>> h
>>>>
>>>> Just wondering if we can avoid rebooting ...
>>>>
>>>> Thanks
>>>> Vlad
>>>>
>>>> On 03.02.2016 16:57, Glenn Wagner wrote:
>>>>> Hi Vlad,
>>>>>
>>>>> Can I ask what hypervisor are you using?
>>>>> The default action is to reboot the hosts
>>>>>
>>>>> This is done in the heatbeat file on xenserver
>>>>> /opt/xensource/bin/xenheartbeat.sh
>>>>>
>>>>> Thanks
>>>>> Glenn
>>>>>
>>>>>
>>>>>
>>>>> ShapeBlue <http://www.shapeblue.com> Glenn Wagner
>>>>> Senior Consultant , ShapeBlue
>>>>>
>>>>> d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091>
>>>>> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111>
>>>>>
>>>>> e: *glenn.wagner@shapeblue.com | t: *
>>>>> <mailto:glenn.wagner@shapeblue.com%20%7C%20t:> | w:
>>>>> *www.shapeblue.com* <http://www.shapeblue.com>
>>>>>
>>>>> a:
>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town
>>>>> 7130 South Africa
>>>>>
>>>>>
>>>>> Shape Blue Ltd is a company incorporated in England & Wales.
>>>>> ShapeBlue Services India LLP is a company incorporated in India and
>>>>> is operated under license from Shape Blue Ltd. Shape Blue Brasil
>>>>> Consultoria Ltda is a company incorporated in Brasil and is operated
>>>>> under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
>>>>> registered by The Republic of South Africa and is traded under
>>>>> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>>>>> This email and any attachments to it may be confidential and are
>>>>> intended solely for the use of the individual to whom it is addressed.
>>>>> Any views or opinions expressed are solely those of the author and
>>>>> do not necessarily represent those of Shape Blue Ltd or related
>>>>> companies. If you are not the intended recipient of this email, you
>>>>> must neither take any action based upon its contents, nor copy or
>>>>> show it to anyone. Please contact the sender if you believe you have
>>>>> received this email in error.
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Vladislav Nazarenko [mailto:vladislav.nazarenko@gmail.com]
>>>>> Sent: Wednesday, 03 February 2016 5:08 PM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: Self-fencing when storage not available
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I'm testing Cloudstack 4.6 now
>>>>>
>>>>> When doing some tests with storage (nfs), I was wondering, that the
>>>>> hosts self-fence itself by reboot, when a storage not writable ...
>>>>>
>>>>> Even more, my cluster had a working storage and I added one more,
>>>>> which was not writable due to missing user permissions... So
>>>>> effectively all the VMs hard drive were located on the working
>>>>> storage, but the problem with permissions on the new one causes the
>>>>> entire cluster to reboot :(
>>>>>
>>>>> Is this a bug or a correct behavior?
>>>>>
>>>>> Is it able to avoid rebooting at least?
>>>>>
>>>>> Thank you in advance
>>>>> Vlad
>>>>>
>>>>>
>>>>> Find out more about ShapeBlue and our range of CloudStack related
>>>>> services:
>>>>> IaaS Cloud Design & Build
>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
>>>>> rapid IaaS deployment framework <http://shapeblue.com/csforge/>
>>>>> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/>
>>>>> | CloudStack Software Engineering
>>>>> <http://shapeblue.com/cloudstack-software-engineering/>
>>>>> CloudStack Infrastructure Support
>>>>> <http://shapeblue.com/cloudstack-infrastructure-support/> |
>>>>> CloudStack Bootcamp Training Courses
>>>>> <http://shapeblue.com/cloudstack-training/>
>>> Find out more about ShapeBlue and our range of CloudStack related services:
>>> IaaS Cloud Design &
>>> Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
>>> rapid IaaS deployment framework<http://shapeblue.com/csforge/>
>>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
>>> CloudStack Software
>>> Engineering<http://shapeblue.com/cloudstack-software-engineering/>
>>> CloudStack Infrastructure
>>> Support<http://shapeblue.com/cloudstack-infrastructure-support/> |
>>> CloudStack Bootcamp Training
>>> Courses<http://shapeblue.com/cloudstack-training/>
>> Find out more about ShapeBlue and our range of CloudStack related services:
>> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
>> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>>
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>
RE: Self-fencing when storage not available
Posted by Paul Angus <pa...@shapeblue.com>.
Well, to start off with I'll add the suggestion to the ticket. https://issues.apache.org/jira/browse/CLOUDSTACK-3367
Paul Angus
VP Technology , ShapeBlue
t: @cloudyangus<te...@cloudyangus>
e: paul.angus@shapeblue.com<ma...@shapeblue.com> | w: www.shapeblue.com<http://www.shapeblue.com>
-----Original Message-----
From: Vladislav Nazarenko [mailto:vladislav.nazarenko@gmail.com]
Sent: Friday, February 5, 2016 5:29 PM
To: users@cloudstack.apache.org
Subject: Re: Self-fencing when storage not available
right, Paul ... this might be a good suggestion ... How could we bring it to the developers?
On 04.02.2016 20:59, Paul Angus wrote:
> Just to give some background (as I asked the same questions years ago). *IF* you only have one shared storage pool per cluster (and no local storage) and it goes wonky, then most likely fix is to restart the host(s). The VMs have already crashed because the primary storage is disconnected and the host is unable to reconnect it.
>
> That’s why it reboots.
>
> As everyone who has a cloud larger than their laptop knows, it also nukes the good VMs on other storage pools and local storage.
>
> A global setting which allows operators to choose the behaviour they want might be a solution?
>
>
>
>
> Paul Angus
> VP Technology , ShapeBlue
>
>
> t: @cloudyangus<te...@cloudyangus>
>
> e: paul.angus@shapeblue.com<ma...@shapeblue.com> | w: www.shapeblue.com<http://www.shapeblue.com>
>
>
>
>
>
> -----Original Message-----
> From: Vladislav Nazarenko [mailto:vladislav.nazarenko@gmail.com]
> Sent: Thursday, February 4, 2016 5:35 PM
> To: users@cloudstack.apache.org
> Subject: Re: Self-fencing when storage not available
>
> So ... found some information about this ... looks like this is an annoying problem (bug), which isn't going to be fixed ... But at least there is a possibility to change the behavior modifying the script
>
> - https://admintweets.wordpress.com/tag/cloudstack-2/
> - https://www.rootusers.com/edit-the-xenserver-storage-heartbeat/
> - https://issues.apache.org/jira/browse/CLOUDSTACK-3367
>
> Thanks a lot, guys ..
>
> On 03.02.2016 20:05, Glenn Wagner wrote:
>> Hi
>>
>> As Lucian said , you can edit the script and remove the reboot , but
>> yes this is the normal behaviour of the script
>>
>> Glenn
>>
>>
>> -----Original Message-----
>> From: Nux! [mailto:nux@li.nux.ro]
>> Sent: Wednesday, 03 February 2016 8:32 PM
>> To: users@cloudstack.apache.org
>> Subject: Re: Self-fencing when storage not available
>>
>> You can modify the script to not reboot, but until we find a better way to deal with it this is correct behaviour. It sucks it reboots VMs on healthy storage though.
>>
>> --
>> Sent from the Delta quadrant using Borg technology!
>>
>> Nux!
>> www.nux.ro
>>
>> ----- Original Message -----
>>> From: "Vladislav Nazarenko" <vl...@gmail.com>
>>> To: users@cloudstack.apache.org
>>> Sent: Wednesday, 3 February, 2016 16:24:34
>>> Subject: Re: Self-fencing when storage not available Hi Glenn,
>>>
>>> we use KVM ... I also found the script:
>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.s
>>> h
>>>
>>> Just wondering if we can avoid rebooting ...
>>>
>>> Thanks
>>> Vlad
>>>
>>> On 03.02.2016 16:57, Glenn Wagner wrote:
>>>> Hi Vlad,
>>>>
>>>> Can I ask what hypervisor are you using?
>>>> The default action is to reboot the hosts
>>>>
>>>> This is done in the heatbeat file on xenserver
>>>> /opt/xensource/bin/xenheartbeat.sh
>>>>
>>>> Thanks
>>>> Glenn
>>>>
>>>>
>>>>
>>>> ShapeBlue <http://www.shapeblue.com> Glenn Wagner
>>>> Senior Consultant , ShapeBlue
>>>>
>>>> d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091>
>>>> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111>
>>>>
>>>> e: *glenn.wagner@shapeblue.com | t: *
>>>> <mailto:glenn.wagner@shapeblue.com%20%7C%20t:> | w:
>>>> *www.shapeblue.com* <http://www.shapeblue.com>
>>>>
>>>> a:
>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town
>>>> 7130 South Africa
>>>>
>>>>
>>>> Shape Blue Ltd is a company incorporated in England & Wales.
>>>> ShapeBlue Services India LLP is a company incorporated in India and
>>>> is operated under license from Shape Blue Ltd. Shape Blue Brasil
>>>> Consultoria Ltda is a company incorporated in Brasil and is operated
>>>> under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
>>>> registered by The Republic of South Africa and is traded under
>>>> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>>>> This email and any attachments to it may be confidential and are
>>>> intended solely for the use of the individual to whom it is addressed.
>>>> Any views or opinions expressed are solely those of the author and
>>>> do not necessarily represent those of Shape Blue Ltd or related
>>>> companies. If you are not the intended recipient of this email, you
>>>> must neither take any action based upon its contents, nor copy or
>>>> show it to anyone. Please contact the sender if you believe you have
>>>> received this email in error.
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Vladislav Nazarenko [mailto:vladislav.nazarenko@gmail.com]
>>>> Sent: Wednesday, 03 February 2016 5:08 PM
>>>> To: users@cloudstack.apache.org
>>>> Subject: Self-fencing when storage not available
>>>>
>>>> Hi All,
>>>>
>>>> I'm testing Cloudstack 4.6 now
>>>>
>>>> When doing some tests with storage (nfs), I was wondering, that the
>>>> hosts self-fence itself by reboot, when a storage not writable ...
>>>>
>>>> Even more, my cluster had a working storage and I added one more,
>>>> which was not writable due to missing user permissions... So
>>>> effectively all the VMs hard drive were located on the working
>>>> storage, but the problem with permissions on the new one causes the
>>>> entire cluster to reboot :(
>>>>
>>>> Is this a bug or a correct behavior?
>>>>
>>>> Is it able to avoid rebooting at least?
>>>>
>>>> Thank you in advance
>>>> Vlad
>>>>
>>>>
>>>> Find out more about ShapeBlue and our range of CloudStack related
>>>> services:
>>>> IaaS Cloud Design & Build
>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
>>>> rapid IaaS deployment framework <http://shapeblue.com/csforge/>
>>>> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/>
>>>> | CloudStack Software Engineering
>>>> <http://shapeblue.com/cloudstack-software-engineering/>
>>>> CloudStack Infrastructure Support
>>>> <http://shapeblue.com/cloudstack-infrastructure-support/> |
>>>> CloudStack Bootcamp Training Courses
>>>> <http://shapeblue.com/cloudstack-training/>
>> Find out more about ShapeBlue and our range of CloudStack related services:
>> IaaS Cloud Design &
>> Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
>> rapid IaaS deployment framework<http://shapeblue.com/csforge/>
>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
>> CloudStack Software
>> Engineering<http://shapeblue.com/cloudstack-software-engineering/>
>> CloudStack Infrastructure
>> Support<http://shapeblue.com/cloudstack-infrastructure-support/> |
>> CloudStack Bootcamp Training
>> Courses<http://shapeblue.com/cloudstack-training/>
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>
Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
Re: Self-fencing when storage not available
Posted by Vladislav Nazarenko <vl...@gmail.com>.
right, Paul ... this might be a good suggestion ... How could we bring
it to the developers?
On 04.02.2016 20:59, Paul Angus wrote:
> Just to give some background (as I asked the same questions years ago). *IF* you only have one shared storage pool per cluster (and no local storage) and it goes wonky, then most likely fix is to restart the host(s). The VMs have already crashed because the primary storage is disconnected and the host is unable to reconnect it.
>
> That’s why it reboots.
>
> As everyone who has a cloud larger than their laptop knows, it also nukes the good VMs on other storage pools and local storage.
>
> A global setting which allows operators to choose the behaviour they want might be a solution?
>
>
>
>
> Paul Angus
> VP Technology , ShapeBlue
>
>
> t: @cloudyangus<te...@cloudyangus>
>
> e: paul.angus@shapeblue.com<ma...@shapeblue.com> | w: www.shapeblue.com<http://www.shapeblue.com>
>
>
>
>
>
> -----Original Message-----
> From: Vladislav Nazarenko [mailto:vladislav.nazarenko@gmail.com]
> Sent: Thursday, February 4, 2016 5:35 PM
> To: users@cloudstack.apache.org
> Subject: Re: Self-fencing when storage not available
>
> So ... found some information about this ... looks like this is an annoying problem (bug), which isn't going to be fixed ... But at least there is a possibility to change the behavior modifying the script
>
> - https://admintweets.wordpress.com/tag/cloudstack-2/
> - https://www.rootusers.com/edit-the-xenserver-storage-heartbeat/
> - https://issues.apache.org/jira/browse/CLOUDSTACK-3367
>
> Thanks a lot, guys ..
>
> On 03.02.2016 20:05, Glenn Wagner wrote:
>> Hi
>>
>> As Lucian said , you can edit the script and remove the reboot , but
>> yes this is the normal behaviour of the script
>>
>> Glenn
>>
>>
>> -----Original Message-----
>> From: Nux! [mailto:nux@li.nux.ro]
>> Sent: Wednesday, 03 February 2016 8:32 PM
>> To: users@cloudstack.apache.org
>> Subject: Re: Self-fencing when storage not available
>>
>> You can modify the script to not reboot, but until we find a better way to deal with it this is correct behaviour. It sucks it reboots VMs on healthy storage though.
>>
>> --
>> Sent from the Delta quadrant using Borg technology!
>>
>> Nux!
>> www.nux.ro
>>
>> ----- Original Message -----
>>> From: "Vladislav Nazarenko" <vl...@gmail.com>
>>> To: users@cloudstack.apache.org
>>> Sent: Wednesday, 3 February, 2016 16:24:34
>>> Subject: Re: Self-fencing when storage not available Hi Glenn,
>>>
>>> we use KVM ... I also found the script:
>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.s
>>> h
>>>
>>> Just wondering if we can avoid rebooting ...
>>>
>>> Thanks
>>> Vlad
>>>
>>> On 03.02.2016 16:57, Glenn Wagner wrote:
>>>> Hi Vlad,
>>>>
>>>> Can I ask what hypervisor are you using?
>>>> The default action is to reboot the hosts
>>>>
>>>> This is done in the heatbeat file on xenserver
>>>> /opt/xensource/bin/xenheartbeat.sh
>>>>
>>>> Thanks
>>>> Glenn
>>>>
>>>>
>>>>
>>>> ShapeBlue <http://www.shapeblue.com> Glenn Wagner
>>>> Senior Consultant , ShapeBlue
>>>>
>>>> d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091>
>>>> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111>
>>>>
>>>> e: *glenn.wagner@shapeblue.com | t: *
>>>> <mailto:glenn.wagner@shapeblue.com%20%7C%20t:> | w:
>>>> *www.shapeblue.com* <http://www.shapeblue.com>
>>>>
>>>> a:
>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town
>>>> 7130 South Africa
>>>>
>>>>
>>>> Shape Blue Ltd is a company incorporated in England & Wales.
>>>> ShapeBlue Services India LLP is a company incorporated in India and
>>>> is operated under license from Shape Blue Ltd. Shape Blue Brasil
>>>> Consultoria Ltda is a company incorporated in Brasil and is operated
>>>> under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
>>>> registered by The Republic of South Africa and is traded under
>>>> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>>>> This email and any attachments to it may be confidential and are
>>>> intended solely for the use of the individual to whom it is addressed.
>>>> Any views or opinions expressed are solely those of the author and
>>>> do not necessarily represent those of Shape Blue Ltd or related
>>>> companies. If you are not the intended recipient of this email, you
>>>> must neither take any action based upon its contents, nor copy or
>>>> show it to anyone. Please contact the sender if you believe you have
>>>> received this email in error.
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Vladislav Nazarenko [mailto:vladislav.nazarenko@gmail.com]
>>>> Sent: Wednesday, 03 February 2016 5:08 PM
>>>> To: users@cloudstack.apache.org
>>>> Subject: Self-fencing when storage not available
>>>>
>>>> Hi All,
>>>>
>>>> I'm testing Cloudstack 4.6 now
>>>>
>>>> When doing some tests with storage (nfs), I was wondering, that the
>>>> hosts self-fence itself by reboot, when a storage not writable ...
>>>>
>>>> Even more, my cluster had a working storage and I added one more,
>>>> which was not writable due to missing user permissions... So
>>>> effectively all the VMs hard drive were located on the working
>>>> storage, but the problem with permissions on the new one causes the
>>>> entire cluster to reboot :(
>>>>
>>>> Is this a bug or a correct behavior?
>>>>
>>>> Is it able to avoid rebooting at least?
>>>>
>>>> Thank you in advance
>>>> Vlad
>>>>
>>>>
>>>> Find out more about ShapeBlue and our range of CloudStack related
>>>> services:
>>>> IaaS Cloud Design & Build
>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
>>>> rapid IaaS deployment framework <http://shapeblue.com/csforge/>
>>>> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/>
>>>> | CloudStack Software Engineering
>>>> <http://shapeblue.com/cloudstack-software-engineering/>
>>>> CloudStack Infrastructure Support
>>>> <http://shapeblue.com/cloudstack-infrastructure-support/> |
>>>> CloudStack Bootcamp Training Courses
>>>> <http://shapeblue.com/cloudstack-training/>
>> Find out more about ShapeBlue and our range of CloudStack related services:
>> IaaS Cloud Design &
>> Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
>> rapid IaaS deployment framework<http://shapeblue.com/csforge/>
>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
>> CloudStack Software
>> Engineering<http://shapeblue.com/cloudstack-software-engineering/>
>> CloudStack Infrastructure
>> Support<http://shapeblue.com/cloudstack-infrastructure-support/> |
>> CloudStack Bootcamp Training
>> Courses<http://shapeblue.com/cloudstack-training/>
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
>
RE: Self-fencing when storage not available
Posted by Paul Angus <pa...@shapeblue.com>.
Just to give some background (as I asked the same questions years ago). *IF* you only have one shared storage pool per cluster (and no local storage) and it goes wonky, then most likely fix is to restart the host(s). The VMs have already crashed because the primary storage is disconnected and the host is unable to reconnect it.
That’s why it reboots.
As everyone who has a cloud larger than their laptop knows, it also nukes the good VMs on other storage pools and local storage.
A global setting which allows operators to choose the behaviour they want might be a solution?
Paul Angus
VP Technology , ShapeBlue
t: @cloudyangus<te...@cloudyangus>
e: paul.angus@shapeblue.com<ma...@shapeblue.com> | w: www.shapeblue.com<http://www.shapeblue.com>
-----Original Message-----
From: Vladislav Nazarenko [mailto:vladislav.nazarenko@gmail.com]
Sent: Thursday, February 4, 2016 5:35 PM
To: users@cloudstack.apache.org
Subject: Re: Self-fencing when storage not available
So ... found some information about this ... looks like this is an annoying problem (bug), which isn't going to be fixed ... But at least there is a possibility to change the behavior modifying the script
- https://admintweets.wordpress.com/tag/cloudstack-2/
- https://www.rootusers.com/edit-the-xenserver-storage-heartbeat/
- https://issues.apache.org/jira/browse/CLOUDSTACK-3367
Thanks a lot, guys ..
On 03.02.2016 20:05, Glenn Wagner wrote:
> Hi
>
> As Lucian said , you can edit the script and remove the reboot , but
> yes this is the normal behaviour of the script
>
> Glenn
>
>
> -----Original Message-----
> From: Nux! [mailto:nux@li.nux.ro]
> Sent: Wednesday, 03 February 2016 8:32 PM
> To: users@cloudstack.apache.org
> Subject: Re: Self-fencing when storage not available
>
> You can modify the script to not reboot, but until we find a better way to deal with it this is correct behaviour. It sucks it reboots VMs on healthy storage though.
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
>> From: "Vladislav Nazarenko" <vl...@gmail.com>
>> To: users@cloudstack.apache.org
>> Sent: Wednesday, 3 February, 2016 16:24:34
>> Subject: Re: Self-fencing when storage not available Hi Glenn,
>>
>> we use KVM ... I also found the script:
>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.s
>> h
>>
>> Just wondering if we can avoid rebooting ...
>>
>> Thanks
>> Vlad
>>
>> On 03.02.2016 16:57, Glenn Wagner wrote:
>>> Hi Vlad,
>>>
>>> Can I ask what hypervisor are you using?
>>> The default action is to reboot the hosts
>>>
>>> This is done in the heatbeat file on xenserver
>>> /opt/xensource/bin/xenheartbeat.sh
>>>
>>> Thanks
>>> Glenn
>>>
>>>
>>>
>>> ShapeBlue <http://www.shapeblue.com> Glenn Wagner
>>> Senior Consultant , ShapeBlue
>>>
>>> d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091>
>>> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111>
>>>
>>> e: *glenn.wagner@shapeblue.com | t: *
>>> <mailto:glenn.wagner@shapeblue.com%20%7C%20t:> | w:
>>> *www.shapeblue.com* <http://www.shapeblue.com>
>>>
>>> a:
>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town
>>> 7130 South Africa
>>>
>>>
>>> Shape Blue Ltd is a company incorporated in England & Wales.
>>> ShapeBlue Services India LLP is a company incorporated in India and
>>> is operated under license from Shape Blue Ltd. Shape Blue Brasil
>>> Consultoria Ltda is a company incorporated in Brasil and is operated
>>> under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
>>> registered by The Republic of South Africa and is traded under
>>> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>>> This email and any attachments to it may be confidential and are
>>> intended solely for the use of the individual to whom it is addressed.
>>> Any views or opinions expressed are solely those of the author and
>>> do not necessarily represent those of Shape Blue Ltd or related
>>> companies. If you are not the intended recipient of this email, you
>>> must neither take any action based upon its contents, nor copy or
>>> show it to anyone. Please contact the sender if you believe you have
>>> received this email in error.
>>>
>>>
>>> -----Original Message-----
>>> From: Vladislav Nazarenko [mailto:vladislav.nazarenko@gmail.com]
>>> Sent: Wednesday, 03 February 2016 5:08 PM
>>> To: users@cloudstack.apache.org
>>> Subject: Self-fencing when storage not available
>>>
>>> Hi All,
>>>
>>> I'm testing Cloudstack 4.6 now
>>>
>>> When doing some tests with storage (nfs), I was wondering, that the
>>> hosts self-fence itself by reboot, when a storage not writable ...
>>>
>>> Even more, my cluster had a working storage and I added one more,
>>> which was not writable due to missing user permissions... So
>>> effectively all the VMs hard drive were located on the working
>>> storage, but the problem with permissions on the new one causes the
>>> entire cluster to reboot :(
>>>
>>> Is this a bug or a correct behavior?
>>>
>>> Is it able to avoid rebooting at least?
>>>
>>> Thank you in advance
>>> Vlad
>>>
>>>
>>> Find out more about ShapeBlue and our range of CloudStack related
>>> services:
>>> IaaS Cloud Design & Build
>>> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
>>> rapid IaaS deployment framework <http://shapeblue.com/csforge/>
>>> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/>
>>> | CloudStack Software Engineering
>>> <http://shapeblue.com/cloudstack-software-engineering/>
>>> CloudStack Infrastructure Support
>>> <http://shapeblue.com/cloudstack-infrastructure-support/> |
>>> CloudStack Bootcamp Training Courses
>>> <http://shapeblue.com/cloudstack-training/>
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design &
> Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
> rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> |
> CloudStack Software
> Engineering<http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure
> Support<http://shapeblue.com/cloudstack-infrastructure-support/> |
> CloudStack Bootcamp Training
> Courses<http://shapeblue.com/cloudstack-training/>
Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
Re: Self-fencing when storage not available
Posted by Vladislav Nazarenko <vl...@gmail.com>.
So ... found some information about this ... looks like this is an
annoying problem (bug), which isn't going to be fixed ... But at least
there is a possibility to change the behavior modifying the script
- https://admintweets.wordpress.com/tag/cloudstack-2/
- https://www.rootusers.com/edit-the-xenserver-storage-heartbeat/
- https://issues.apache.org/jira/browse/CLOUDSTACK-3367
Thanks a lot, guys ..
On 03.02.2016 20:05, Glenn Wagner wrote:
> Hi
>
> As Lucian said , you can edit the script and remove the reboot , but yes this is the normal behaviour of the script
>
> Glenn
>
>
> -----Original Message-----
> From: Nux! [mailto:nux@li.nux.ro]
> Sent: Wednesday, 03 February 2016 8:32 PM
> To: users@cloudstack.apache.org
> Subject: Re: Self-fencing when storage not available
>
> You can modify the script to not reboot, but until we find a better way to deal with it this is correct behaviour. It sucks it reboots VMs on healthy storage though.
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
>> From: "Vladislav Nazarenko" <vl...@gmail.com>
>> To: users@cloudstack.apache.org
>> Sent: Wednesday, 3 February, 2016 16:24:34
>> Subject: Re: Self-fencing when storage not available
>> Hi Glenn,
>>
>> we use KVM ... I also found the script:
>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>>
>> Just wondering if we can avoid rebooting ...
>>
>> Thanks
>> Vlad
>>
>> On 03.02.2016 16:57, Glenn Wagner wrote:
>>> Hi Vlad,
>>>
>>> Can I ask what hypervisor are you using?
>>> The default action is to reboot the hosts
>>>
>>> This is done in the heatbeat file on xenserver
>>> /opt/xensource/bin/xenheartbeat.sh
>>>
>>> Thanks
>>> Glenn
>>>
>>>
>>>
>>> ShapeBlue <http://www.shapeblue.com>
>>> Glenn Wagner
>>> Senior Consultant , ShapeBlue
>>>
>>> d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091>
>>> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111>
>>>
>>> e: *glenn.wagner@shapeblue.com | t: *
>>> <mailto:glenn.wagner@shapeblue.com%20%7C%20t:> | w:
>>> *www.shapeblue.com* <http://www.shapeblue.com>
>>>
>>> a:
>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town 7130
>>> South Africa
>>>
>>>
>>> Shape Blue Ltd is a company incorporated in England & Wales.
>>> ShapeBlue Services India LLP is a company incorporated in India and
>>> is operated under license from Shape Blue Ltd. Shape Blue Brasil
>>> Consultoria Ltda is a company incorporated in Brasil and is operated
>>> under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
>>> registered by The Republic of South Africa and is traded under
>>> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>>> This email and any attachments to it may be confidential and are
>>> intended solely for the use of the individual to whom it is addressed.
>>> Any views or opinions expressed are solely those of the author and do
>>> not necessarily represent those of Shape Blue Ltd or related
>>> companies. If you are not the intended recipient of this email, you
>>> must neither take any action based upon its contents, nor copy or
>>> show it to anyone. Please contact the sender if you believe you have
>>> received this email in error.
>>>
>>>
>>> -----Original Message-----
>>> From: Vladislav Nazarenko [mailto:vladislav.nazarenko@gmail.com]
>>> Sent: Wednesday, 03 February 2016 5:08 PM
>>> To: users@cloudstack.apache.org
>>> Subject: Self-fencing when storage not available
>>>
>>> Hi All,
>>>
>>> I'm testing Cloudstack 4.6 now
>>>
>>> When doing some tests with storage (nfs), I was wondering, that the
>>> hosts self-fence itself by reboot, when a storage not writable ...
>>>
>>> Even more, my cluster had a working storage and I added one more,
>>> which was not writable due to missing user permissions... So
>>> effectively all the VMs hard drive were located on the working
>>> storage, but the problem with permissions on the new one causes the
>>> entire cluster to reboot :(
>>>
>>> Is this a bug or a correct behavior?
>>>
>>> Is it able to avoid rebooting at least?
>>>
>>> Thank you in advance
>>> Vlad
>>>
>>>
>>> Find out more about ShapeBlue and our range of CloudStack related
>>> services:
>>> IaaS Cloud Design & Build
>>> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
>>> rapid IaaS deployment framework <http://shapeblue.com/csforge/>
>>> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/>
>>> | CloudStack Software Engineering
>>> <http://shapeblue.com/cloudstack-software-engineering/>
>>> CloudStack Infrastructure Support
>>> <http://shapeblue.com/cloudstack-infrastructure-support/> |
>>> CloudStack
>>> Bootcamp Training Courses
>>> <http://shapeblue.com/cloudstack-training/>
> Find out more about ShapeBlue and our range of CloudStack related services:
> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
RE: Self-fencing when storage not available
Posted by Glenn Wagner <gl...@shapeblue.com>.
Hi
As Lucian said , you can edit the script and remove the reboot , but yes this is the normal behaviour of the script
Glenn
-----Original Message-----
From: Nux! [mailto:nux@li.nux.ro]
Sent: Wednesday, 03 February 2016 8:32 PM
To: users@cloudstack.apache.org
Subject: Re: Self-fencing when storage not available
You can modify the script to not reboot, but until we find a better way to deal with it this is correct behaviour. It sucks it reboots VMs on healthy storage though.
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Vladislav Nazarenko" <vl...@gmail.com>
> To: users@cloudstack.apache.org
> Sent: Wednesday, 3 February, 2016 16:24:34
> Subject: Re: Self-fencing when storage not available
> Hi Glenn,
>
> we use KVM ... I also found the script:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>
> Just wondering if we can avoid rebooting ...
>
> Thanks
> Vlad
>
> On 03.02.2016 16:57, Glenn Wagner wrote:
>> Hi Vlad,
>>
>> Can I ask what hypervisor are you using?
>> The default action is to reboot the hosts
>>
>> This is done in the heatbeat file on xenserver
>> /opt/xensource/bin/xenheartbeat.sh
>>
>> Thanks
>> Glenn
>>
>>
>>
>> ShapeBlue <http://www.shapeblue.com>
>> Glenn Wagner
>> Senior Consultant , ShapeBlue
>>
>> d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091>
>> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111>
>>
>> e: *glenn.wagner@shapeblue.com | t: *
>> <mailto:glenn.wagner@shapeblue.com%20%7C%20t:> | w:
>> *www.shapeblue.com* <http://www.shapeblue.com>
>>
>> a:
>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town 7130
>> South Africa
>>
>>
>> Shape Blue Ltd is a company incorporated in England & Wales.
>> ShapeBlue Services India LLP is a company incorporated in India and
>> is operated under license from Shape Blue Ltd. Shape Blue Brasil
>> Consultoria Ltda is a company incorporated in Brasil and is operated
>> under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
>> registered by The Republic of South Africa and is traded under
>> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
>> This email and any attachments to it may be confidential and are
>> intended solely for the use of the individual to whom it is addressed.
>> Any views or opinions expressed are solely those of the author and do
>> not necessarily represent those of Shape Blue Ltd or related
>> companies. If you are not the intended recipient of this email, you
>> must neither take any action based upon its contents, nor copy or
>> show it to anyone. Please contact the sender if you believe you have
>> received this email in error.
>>
>>
>> -----Original Message-----
>> From: Vladislav Nazarenko [mailto:vladislav.nazarenko@gmail.com]
>> Sent: Wednesday, 03 February 2016 5:08 PM
>> To: users@cloudstack.apache.org
>> Subject: Self-fencing when storage not available
>>
>> Hi All,
>>
>> I'm testing Cloudstack 4.6 now
>>
>> When doing some tests with storage (nfs), I was wondering, that the
>> hosts self-fence itself by reboot, when a storage not writable ...
>>
>> Even more, my cluster had a working storage and I added one more,
>> which was not writable due to missing user permissions... So
>> effectively all the VMs hard drive were located on the working
>> storage, but the problem with permissions on the new one causes the
>> entire cluster to reboot :(
>>
>> Is this a bug or a correct behavior?
>>
>> Is it able to avoid rebooting at least?
>>
>> Thank you in advance
>> Vlad
>>
>>
>> Find out more about ShapeBlue and our range of CloudStack related
>> services:
>> IaaS Cloud Design & Build
>> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge –
>> rapid IaaS deployment framework <http://shapeblue.com/csforge/>
>> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/>
>> | CloudStack Software Engineering
>> <http://shapeblue.com/cloudstack-software-engineering/>
>> CloudStack Infrastructure Support
>> <http://shapeblue.com/cloudstack-infrastructure-support/> |
>> CloudStack
> > Bootcamp Training Courses
> > <http://shapeblue.com/cloudstack-training/>
Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
Re: Self-fencing when storage not available
Posted by Vladislav Nazarenko <vl...@gmail.com>.
There were probably many discussions around this implementation? Do you
have any links to read about that? I'm really interesting in the
background, would like to understand
why implemented this way ...
On 03.02.2016 19:34, Frank Louwers wrote:
>> On 03 Feb 2016, at 19:32, Nux! <nu...@li.nux.ro> wrote:
>>
>> You can modify the script to not reboot, but until we find a better way to deal with it this is correct behaviour. It sucks it reboots VMs on healthy storage though.
> Not only does it reboot on healty nodes, it only works on NFS. Not on another type of primary storage, not on local storage etc...
>> --
>> Sent from the Delta quadrant using Borg technology!
>>
>> Nux!
>> www.nux.ro
>>
>> ----- Original Message -----
>>> From: "Vladislav Nazarenko" <vl...@gmail.com>
>>> To: users@cloudstack.apache.org
>>> Sent: Wednesday, 3 February, 2016 16:24:34
>>> Subject: Re: Self-fencing when storage not available
>>> Hi Glenn,
>>>
>>> we use KVM ... I also found the script:
>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>>>
>>> Just wondering if we can avoid rebooting ...
>>>
>>> Thanks
>>> Vlad
>>>
>>> On 03.02.2016 16:57, Glenn Wagner wrote:
>>>> Hi Vlad,
>>>>
>>>> Can I ask what hypervisor are you using?
>>>> The default action is to reboot the hosts
>>>>
>>>> This is done in the heatbeat file on xenserver
>>>> /opt/xensource/bin/xenheartbeat.sh
>>>>
>>>> Thanks
>>>> Glenn
>>>>
>>>>
>>>>
>>>> ShapeBlue <http://www.shapeblue.com>
>>>> Glenn Wagner
>>>> Senior Consultant , ShapeBlue
>>>>
>>>> d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091>
>>>> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111>
>>>>
>>>> e: *glenn.wagner@shapeblue.com | t: *
>>>> <mailto:glenn.wagner@shapeblue.com%20%7C%20t:> | w:
>>>> *www.shapeblue.com* <http://www.shapeblue.com>
>>>>
>>>> a:
>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town 7130 South
>>>> Africa
>>>>
>>>>
>>>> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
>>>> Services India LLP is a company incorporated in India and is operated
>>>> under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
>>>> is a company incorporated in Brasil and is operated under license from
>>>> Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
>>>> Republic of South Africa and is traded under license from Shape Blue
>>>> Ltd. ShapeBlue is a registered trademark.
>>>> This email and any attachments to it may be confidential and are
>>>> intended solely for the use of the individual to whom it is addressed.
>>>> Any views or opinions expressed are solely those of the author and do
>>>> not necessarily represent those of Shape Blue Ltd or related
>>>> companies. If you are not the intended recipient of this email, you
>>>> must neither take any action based upon its contents, nor copy or show
>>>> it to anyone. Please contact the sender if you believe you have
>>>> received this email in error.
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Vladislav Nazarenko [mailto:vladislav.nazarenko@gmail.com]
>>>> Sent: Wednesday, 03 February 2016 5:08 PM
>>>> To: users@cloudstack.apache.org
>>>> Subject: Self-fencing when storage not available
>>>>
>>>> Hi All,
>>>>
>>>> I'm testing Cloudstack 4.6 now
>>>>
>>>> When doing some tests with storage (nfs), I was wondering, that the
>>>> hosts self-fence itself by reboot, when a storage not writable ...
>>>>
>>>> Even more, my cluster had a working storage and I added one more,
>>>> which was not writable due to missing user permissions... So
>>>> effectively all the VMs hard drive were located on the working
>>>> storage, but the problem with permissions on the new one causes the
>>>> entire cluster to reboot :(
>>>>
>>>> Is this a bug or a correct behavior?
>>>>
>>>> Is it able to avoid rebooting at least?
>>>>
>>>> Thank you in advance
>>>> Vlad
>>>>
>>>>
>>>> Find out more about ShapeBlue and our range of CloudStack related
>>>> services:
>>>> IaaS Cloud Design & Build
>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
>>>> IaaS deployment framework <http://shapeblue.com/csforge/>
>>>> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> |
>>>> CloudStack Software Engineering
>>>> <http://shapeblue.com/cloudstack-software-engineering/>
>>>> CloudStack Infrastructure Support
>>>> <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
>>>> Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
>
Re: Self-fencing when storage not available
Posted by Frank Louwers <fr...@openminds.be>.
> On 03 Feb 2016, at 19:32, Nux! <nu...@li.nux.ro> wrote:
>
> You can modify the script to not reboot, but until we find a better way to deal with it this is correct behaviour. It sucks it reboots VMs on healthy storage though.
Not only does it reboot on healty nodes, it only works on NFS. Not on another type of primary storage, not on local storage etc...
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
>> From: "Vladislav Nazarenko" <vl...@gmail.com>
>> To: users@cloudstack.apache.org
>> Sent: Wednesday, 3 February, 2016 16:24:34
>> Subject: Re: Self-fencing when storage not available
>
>> Hi Glenn,
>>
>> we use KVM ... I also found the script:
>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>>
>> Just wondering if we can avoid rebooting ...
>>
>> Thanks
>> Vlad
>>
>> On 03.02.2016 16:57, Glenn Wagner wrote:
>>> Hi Vlad,
>>>
>>> Can I ask what hypervisor are you using?
>>> The default action is to reboot the hosts
>>>
>>> This is done in the heatbeat file on xenserver
>>> /opt/xensource/bin/xenheartbeat.sh
>>>
>>> Thanks
>>> Glenn
>>>
>>>
>>>
>>> ShapeBlue <http://www.shapeblue.com>
>>> Glenn Wagner
>>> Senior Consultant , ShapeBlue
>>>
>>> d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091>
>>> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111>
>>>
>>> e: *glenn.wagner@shapeblue.com | t: *
>>> <mailto:glenn.wagner@shapeblue.com%20%7C%20t:> | w:
>>> *www.shapeblue.com* <http://www.shapeblue.com>
>>>
>>> a:
>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town 7130 South
>>> Africa
>>>
>>>
>>> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
>>> Services India LLP is a company incorporated in India and is operated
>>> under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
>>> is a company incorporated in Brasil and is operated under license from
>>> Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
>>> Republic of South Africa and is traded under license from Shape Blue
>>> Ltd. ShapeBlue is a registered trademark.
>>> This email and any attachments to it may be confidential and are
>>> intended solely for the use of the individual to whom it is addressed.
>>> Any views or opinions expressed are solely those of the author and do
>>> not necessarily represent those of Shape Blue Ltd or related
>>> companies. If you are not the intended recipient of this email, you
>>> must neither take any action based upon its contents, nor copy or show
>>> it to anyone. Please contact the sender if you believe you have
>>> received this email in error.
>>>
>>>
>>> -----Original Message-----
>>> From: Vladislav Nazarenko [mailto:vladislav.nazarenko@gmail.com]
>>> Sent: Wednesday, 03 February 2016 5:08 PM
>>> To: users@cloudstack.apache.org
>>> Subject: Self-fencing when storage not available
>>>
>>> Hi All,
>>>
>>> I'm testing Cloudstack 4.6 now
>>>
>>> When doing some tests with storage (nfs), I was wondering, that the
>>> hosts self-fence itself by reboot, when a storage not writable ...
>>>
>>> Even more, my cluster had a working storage and I added one more,
>>> which was not writable due to missing user permissions... So
>>> effectively all the VMs hard drive were located on the working
>>> storage, but the problem with permissions on the new one causes the
>>> entire cluster to reboot :(
>>>
>>> Is this a bug or a correct behavior?
>>>
>>> Is it able to avoid rebooting at least?
>>>
>>> Thank you in advance
>>> Vlad
>>>
>>>
>>> Find out more about ShapeBlue and our range of CloudStack related
>>> services:
>>> IaaS Cloud Design & Build
>>> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
>>> IaaS deployment framework <http://shapeblue.com/csforge/>
>>> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> |
>>> CloudStack Software Engineering
>>> <http://shapeblue.com/cloudstack-software-engineering/>
>>> CloudStack Infrastructure Support
>>> <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
>>> Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
Re: Self-fencing when storage not available
Posted by Nux! <nu...@li.nux.ro>.
You can modify the script to not reboot, but until we find a better way to deal with it this is correct behaviour. It sucks it reboots VMs on healthy storage though.
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Vladislav Nazarenko" <vl...@gmail.com>
> To: users@cloudstack.apache.org
> Sent: Wednesday, 3 February, 2016 16:24:34
> Subject: Re: Self-fencing when storage not available
> Hi Glenn,
>
> we use KVM ... I also found the script:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>
> Just wondering if we can avoid rebooting ...
>
> Thanks
> Vlad
>
> On 03.02.2016 16:57, Glenn Wagner wrote:
>> Hi Vlad,
>>
>> Can I ask what hypervisor are you using?
>> The default action is to reboot the hosts
>>
>> This is done in the heatbeat file on xenserver
>> /opt/xensource/bin/xenheartbeat.sh
>>
>> Thanks
>> Glenn
>>
>>
>>
>> ShapeBlue <http://www.shapeblue.com>
>> Glenn Wagner
>> Senior Consultant , ShapeBlue
>>
>> d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091>
>> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111>
>>
>> e: *glenn.wagner@shapeblue.com | t: *
>> <mailto:glenn.wagner@shapeblue.com%20%7C%20t:> | w:
>> *www.shapeblue.com* <http://www.shapeblue.com>
>>
>> a:
>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town 7130 South
>> Africa
>>
>>
>> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
>> Services India LLP is a company incorporated in India and is operated
>> under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
>> is a company incorporated in Brasil and is operated under license from
>> Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
>> Republic of South Africa and is traded under license from Shape Blue
>> Ltd. ShapeBlue is a registered trademark.
>> This email and any attachments to it may be confidential and are
>> intended solely for the use of the individual to whom it is addressed.
>> Any views or opinions expressed are solely those of the author and do
>> not necessarily represent those of Shape Blue Ltd or related
>> companies. If you are not the intended recipient of this email, you
>> must neither take any action based upon its contents, nor copy or show
>> it to anyone. Please contact the sender if you believe you have
>> received this email in error.
>>
>>
>> -----Original Message-----
>> From: Vladislav Nazarenko [mailto:vladislav.nazarenko@gmail.com]
>> Sent: Wednesday, 03 February 2016 5:08 PM
>> To: users@cloudstack.apache.org
>> Subject: Self-fencing when storage not available
>>
>> Hi All,
>>
>> I'm testing Cloudstack 4.6 now
>>
>> When doing some tests with storage (nfs), I was wondering, that the
>> hosts self-fence itself by reboot, when a storage not writable ...
>>
>> Even more, my cluster had a working storage and I added one more,
>> which was not writable due to missing user permissions... So
>> effectively all the VMs hard drive were located on the working
>> storage, but the problem with permissions on the new one causes the
>> entire cluster to reboot :(
>>
>> Is this a bug or a correct behavior?
>>
>> Is it able to avoid rebooting at least?
>>
>> Thank you in advance
>> Vlad
>>
>>
>> Find out more about ShapeBlue and our range of CloudStack related
>> services:
>> IaaS Cloud Design & Build
>> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
>> IaaS deployment framework <http://shapeblue.com/csforge/>
>> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> |
>> CloudStack Software Engineering
>> <http://shapeblue.com/cloudstack-software-engineering/>
>> CloudStack Infrastructure Support
>> <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> > Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
Re: Self-fencing when storage not available
Posted by Vladislav Nazarenko <vl...@gmail.com>.
Hi Glenn,
we use KVM ... I also found the script:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
Just wondering if we can avoid rebooting ...
Thanks
Vlad
On 03.02.2016 16:57, Glenn Wagner wrote:
> Hi Vlad,
>
> Can I ask what hypervisor are you using?
> The default action is to reboot the hosts
>
> This is done in the heatbeat file on xenserver
> /opt/xensource/bin/xenheartbeat.sh
>
> Thanks
> Glenn
>
>
>
> ShapeBlue <http://www.shapeblue.com>
> Glenn Wagner
> Senior Consultant , ShapeBlue
>
> d: * | s: +27 21 527 0091* <tel:%7C%20s:%20+27%2021%20527%200091>
> | m: *+27 73 917 4111* <tel:+27%2073%20917%204111>
>
> e: *glenn.wagner@shapeblue.com | t: *
> <mailto:glenn.wagner@shapeblue.com%20%7C%20t:> | w:
> *www.shapeblue.com* <http://www.shapeblue.com>
>
> a:
> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town 7130 South Africa
>
>
> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> Services India LLP is a company incorporated in India and is operated
> under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda
> is a company incorporated in Brasil and is operated under license from
> Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The
> Republic of South Africa and is traded under license from Shape Blue
> Ltd. ShapeBlue is a registered trademark.
> This email and any attachments to it may be confidential and are
> intended solely for the use of the individual to whom it is addressed.
> Any views or opinions expressed are solely those of the author and do
> not necessarily represent those of Shape Blue Ltd or related
> companies. If you are not the intended recipient of this email, you
> must neither take any action based upon its contents, nor copy or show
> it to anyone. Please contact the sender if you believe you have
> received this email in error.
>
>
> -----Original Message-----
> From: Vladislav Nazarenko [mailto:vladislav.nazarenko@gmail.com]
> Sent: Wednesday, 03 February 2016 5:08 PM
> To: users@cloudstack.apache.org
> Subject: Self-fencing when storage not available
>
> Hi All,
>
> I'm testing Cloudstack 4.6 now
>
> When doing some tests with storage (nfs), I was wondering, that the
> hosts self-fence itself by reboot, when a storage not writable ...
>
> Even more, my cluster had a working storage and I added one more,
> which was not writable due to missing user permissions... So
> effectively all the VMs hard drive were located on the working
> storage, but the problem with permissions on the new one causes the
> entire cluster to reboot :(
>
> Is this a bug or a correct behavior?
>
> Is it able to avoid rebooting at least?
>
> Thank you in advance
> Vlad
>
>
> Find out more about ShapeBlue and our range of CloudStack related
> services:
> IaaS Cloud Design & Build
> <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid
> IaaS deployment framework <http://shapeblue.com/csforge/>
> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/> |
> CloudStack Software Engineering
> <http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure Support
> <http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack
> Bootcamp Training Courses <http://shapeblue.com/cloudstack-training/>
RE: Self-fencing when storage not available
Posted by Glenn Wagner <gl...@shapeblue.com>.
Hi Vlad,
Can I ask what hypervisor are you using?
The default action is to reboot the hosts
This is done in the heatbeat file on xenserver
/opt/xensource/bin/xenheartbeat.sh
Thanks
Glenn
[ShapeBlue]<http://www.shapeblue.com>
Glenn Wagner
Senior Consultant , ShapeBlue
d: | s: +27 21 527 0091<tel:|%20s:%20+27%2021%20527%200091> | m: +27 73 917 4111<tel:+27%2073%20917%204111>
e: glenn.wagner@shapeblue.com | t: <mailto:glenn.wagner@shapeblue.com%20|%20t:> | w: www.shapeblue.com<http://www.shapeblue.com>
a: 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town 7130 South Africa
[cid:image24bf30.png@0df405d7.41a47b58]
Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.
-----Original Message-----
From: Vladislav Nazarenko [mailto:vladislav.nazarenko@gmail.com]
Sent: Wednesday, 03 February 2016 5:08 PM
To: users@cloudstack.apache.org
Subject: Self-fencing when storage not available
Hi All,
I'm testing Cloudstack 4.6 now
When doing some tests with storage (nfs), I was wondering, that the hosts self-fence itself by reboot, when a storage not writable ...
Even more, my cluster had a working storage and I added one more, which was not writable due to missing user permissions... So effectively all the VMs hard drive were located on the working storage, but the problem with permissions on the new one causes the entire cluster to reboot :(
Is this a bug or a correct behavior?
Is it able to avoid rebooting at least?
Thank you in advance
Vlad
Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>