You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Josh Harshman <Jo...@intrinium.com> on 2015/08/27 18:36:32 UTC

Discuss reserving memory on KVM hypervisors ref: CLOUDSTACK-8678

In a KVM cluster, CloudStack sees 100% of the compute node's RAM and treats it as allocatable space which eventually leads to OOM killing guests.


There is an agent property named host.reserved.mem.mb which is able to be set in the agent.properties file and passed to the management server.  This value is stored as dom0MinMem, however, it appears to be ignored.


If we could tweak the host capacity calculation and have it take this into account, I believe that would be ideal.


Side note: the variable dom0MinMem is declared as an int and can be overflowed. Suggested change would be make it a long and add a check especially if we are going to make this a configurable parameter.




Josh Harshman

Cloud Engineer


Intrinium
Tel: (509) 465-1234 x5259
Fax: (866) 565-4578
Lync / Skype: Josh.Harshman@intrinium.com
Web: http://intrinium.com<http://intrinium.com/>


[Intrinium Long Sig Logo]<http://intrinium.com/>


[Facebook]<http://www.facebook.com/pages/intrinium/154891544582822> [Twitter] <http://twitter.com/intrinium>  [Linkedin] <http://www.linkedin.com/company/intrinium_networks_it_security?trk=fc_badge>  [Youtube] <http://www.youtube.com/user/intrinium>  [Blog] <http://intrinium.com/blog/>

Information Security and Compliance Consulting | Managed IT and Security Services | Cloud Services





________________________________

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

RE: Discuss reserving memory on KVM hypervisors ref: CLOUDSTACK-8678

Posted by Kevin McCormick <Ke...@intrinium.com>.
I've submitted a pull request here: https://github.com/apache/cloudstack/pull/766.

Basically just does a totalRAM - host.reserved.mem.mb before reporting total RAM to the management server.

Remi, if you can test, that would be great. We can get it done later this week, but we don't have automation setup for testing, etc.



Kevin McCormick
Director of Cloud Services

Intrinium
-----Original Message-----
From: Remi Bergsma [mailto:RBergsma@schubergphilis.com]
Sent: Friday, August 28, 2015 23:41
To: <de...@cloudstack.apache.org> <de...@cloudstack.apache.org>
Subject: Re: Discuss reserving memory on KVM hypervisors ref: CLOUDSTACK-8678

Making the agent property work (host.reserved.mem.mb) would be awesome! On labs you can then set it to something low.

Let me know if you need help with testing!

Thanks, Remi

Sent from my iPhone

> On 29 Aug 2015, at 07:49, Erik Weber <te...@gmail.com> wrote:
>
> Make it configurable to enable/disable the check? Or even the size.
>
> For labs etc you might not have or want that much to spare.
>
>
> Erik
>
> Den lørdag 29. august 2015 skrev Josh Harshman <Jo...@intrinium.com>
> følgende:
>
>> Wido -
>> Looks like the problem resides in the logic that does the detection of the
>> host resources.  It looks like most, if not all, the KVM host detection
>> occurs in the LibvirtComputingResource class.  Here I can modify it so it
>> will take the value stored in dom0MinMem into account when calculating the
>> ram available.
>>
>> I've forked the repo and am working on a fix.
>> Does anyone have anything to add?  Suggestions?
>>
>> ________________________________________
>> From: Wido den Hollander <wido@widodh.nl <javascript:;>>
>> Sent: Friday, August 28, 2015 12:38 AM
>> To: dev@cloudstack.apache.org <javascript:;>
>> Subject: Re: Discuss reserving memory on KVM hypervisors ref:
>> CLOUDSTACK-8678
>>
>>
>>
>> Josh Harshman
>> Cloud Engineer
>>
>> Intrinium
>>
>>> On 27-08-15 18:36, Josh Harshman wrote:
>>> In a KVM cluster, CloudStack sees 100% of the compute node's RAM and
>> treats it as allocatable space which eventually leads to OOM killing guests.
>>>
>>>
>>> There is an agent property named host.reserved.mem.mb which is able to
>> be set in the agent.properties file and passed to the management server.
>> This value is stored as dom0MinMem, however, it appears to be ignored.
>>>
>>>
>>> If we could tweak the host capacity calculation and have it take this
>> into account, I believe that would be ideal.
>>>
>>>
>>> Side note: the variable dom0MinMem is declared as an int and can be
>> overflowed. Suggested change would be make it a long and add a check
>> especially if we are going to make this a configurable parameter.
>>
>> Yes, that seems like a sane thing to do. We should be able to say that
>> eg 8GB of memory should stay available for the HV.
>>
>> Don't know where the problem lies though. A PR is welcome :)
>>
>> Wido
>>
>>>
>>>
>>> Josh Harshman
>>>
>>> Cloud Engineer
>>>
>>>
>>> Intrinium
>>> Tel: (509) 465-1234 x5259
>>> Fax: (866) 565-4578
>>> Lync / Skype: Josh.Harshman@intrinium.com <javascript:;>
>>> Web: http://intrinium.com<http://intrinium.com/>
>>>
>>>
>>> [Intrinium Long Sig Logo]<http://intrinium.com/>
>>>
>>>
>>> [Facebook]<http://www.facebook.com/pages/intrinium/154891544582822>
>> [Twitter] <http://twitter.com/intrinium>  [Linkedin] <
>> http://www.linkedin.com/company/intrinium_networks_it_security?trk=fc_badge>
>> [Youtube] <http://www.youtube.com/user/intrinium> [Blog] <
>> http://intrinium.com/blog/>
>>>
>>> Information Security and Compliance Consulting | Managed IT and Security
>> Services | Cloud Services
>>>
>>>
>>>
>>>
>>>
>>> ________________________________
>>>
>>> This email and any files transmitted with it are confidential and
>> intended solely for the use of the individual or entity to whom they are
>> addressed. If you have received this email in error please notify the
>> system manager. This message contains confidential information and is
>> intended only for the individual named. If you are not the named addressee
>> you should not disseminate, distribute or copy this e-mail. Please notify
>> the sender immediately by e-mail if you have received this e-mail by
>> mistake and delete this e-mail from your system. If you are not the
>> intended recipient you are notified that disclosing, copying, distributing
>> or taking any action in reliance on the contents of this information is
>> strictly prohibited.
>>

Re: Discuss reserving memory on KVM hypervisors ref: CLOUDSTACK-8678

Posted by Remi Bergsma <RB...@schubergphilis.com>.
Making the agent property work (host.reserved.mem.mb) would be awesome! On labs you can then set it to something low. 

Let me know if you need help with testing!

Thanks, Remi 

Sent from my iPhone

> On 29 Aug 2015, at 07:49, Erik Weber <te...@gmail.com> wrote:
> 
> Make it configurable to enable/disable the check? Or even the size.
> 
> For labs etc you might not have or want that much to spare.
> 
> 
> Erik
> 
> Den lørdag 29. august 2015 skrev Josh Harshman <Jo...@intrinium.com>
> følgende:
> 
>> Wido -
>> Looks like the problem resides in the logic that does the detection of the
>> host resources.  It looks like most, if not all, the KVM host detection
>> occurs in the LibvirtComputingResource class.  Here I can modify it so it
>> will take the value stored in dom0MinMem into account when calculating the
>> ram available.
>> 
>> I've forked the repo and am working on a fix.
>> Does anyone have anything to add?  Suggestions?
>> 
>> ________________________________________
>> From: Wido den Hollander <wido@widodh.nl <javascript:;>>
>> Sent: Friday, August 28, 2015 12:38 AM
>> To: dev@cloudstack.apache.org <javascript:;>
>> Subject: Re: Discuss reserving memory on KVM hypervisors ref:
>> CLOUDSTACK-8678
>> 
>> 
>> 
>> Josh Harshman
>> Cloud Engineer
>> 
>> Intrinium
>> 
>>> On 27-08-15 18:36, Josh Harshman wrote:
>>> In a KVM cluster, CloudStack sees 100% of the compute node's RAM and
>> treats it as allocatable space which eventually leads to OOM killing guests.
>>> 
>>> 
>>> There is an agent property named host.reserved.mem.mb which is able to
>> be set in the agent.properties file and passed to the management server.
>> This value is stored as dom0MinMem, however, it appears to be ignored.
>>> 
>>> 
>>> If we could tweak the host capacity calculation and have it take this
>> into account, I believe that would be ideal.
>>> 
>>> 
>>> Side note: the variable dom0MinMem is declared as an int and can be
>> overflowed. Suggested change would be make it a long and add a check
>> especially if we are going to make this a configurable parameter.
>> 
>> Yes, that seems like a sane thing to do. We should be able to say that
>> eg 8GB of memory should stay available for the HV.
>> 
>> Don't know where the problem lies though. A PR is welcome :)
>> 
>> Wido
>> 
>>> 
>>> 
>>> Josh Harshman
>>> 
>>> Cloud Engineer
>>> 
>>> 
>>> Intrinium
>>> Tel: (509) 465-1234 x5259
>>> Fax: (866) 565-4578
>>> Lync / Skype: Josh.Harshman@intrinium.com <javascript:;>
>>> Web: http://intrinium.com<http://intrinium.com/>
>>> 
>>> 
>>> [Intrinium Long Sig Logo]<http://intrinium.com/>
>>> 
>>> 
>>> [Facebook]<http://www.facebook.com/pages/intrinium/154891544582822>
>> [Twitter] <http://twitter.com/intrinium>  [Linkedin] <
>> http://www.linkedin.com/company/intrinium_networks_it_security?trk=fc_badge>
>> [Youtube] <http://www.youtube.com/user/intrinium> [Blog] <
>> http://intrinium.com/blog/>
>>> 
>>> Information Security and Compliance Consulting | Managed IT and Security
>> Services | Cloud Services
>>> 
>>> 
>>> 
>>> 
>>> 
>>> ________________________________
>>> 
>>> This email and any files transmitted with it are confidential and
>> intended solely for the use of the individual or entity to whom they are
>> addressed. If you have received this email in error please notify the
>> system manager. This message contains confidential information and is
>> intended only for the individual named. If you are not the named addressee
>> you should not disseminate, distribute or copy this e-mail. Please notify
>> the sender immediately by e-mail if you have received this e-mail by
>> mistake and delete this e-mail from your system. If you are not the
>> intended recipient you are notified that disclosing, copying, distributing
>> or taking any action in reliance on the contents of this information is
>> strictly prohibited.
>>

Re: Discuss reserving memory on KVM hypervisors ref: CLOUDSTACK-8678

Posted by Erik Weber <te...@gmail.com>.
Make it configurable to enable/disable the check? Or even the size.

For labs etc you might not have or want that much to spare.


Erik

Den lørdag 29. august 2015 skrev Josh Harshman <Jo...@intrinium.com>
følgende:

> Wido -
> Looks like the problem resides in the logic that does the detection of the
> host resources.  It looks like most, if not all, the KVM host detection
> occurs in the LibvirtComputingResource class.  Here I can modify it so it
> will take the value stored in dom0MinMem into account when calculating the
> ram available.
>
> I've forked the repo and am working on a fix.
> Does anyone have anything to add?  Suggestions?
>
> ________________________________________
> From: Wido den Hollander <wido@widodh.nl <javascript:;>>
> Sent: Friday, August 28, 2015 12:38 AM
> To: dev@cloudstack.apache.org <javascript:;>
> Subject: Re: Discuss reserving memory on KVM hypervisors ref:
> CLOUDSTACK-8678
>
>
>
> Josh Harshman
> Cloud Engineer
>
> Intrinium
>
> On 27-08-15 18:36, Josh Harshman wrote:
> > In a KVM cluster, CloudStack sees 100% of the compute node's RAM and
> treats it as allocatable space which eventually leads to OOM killing guests.
> >
> >
> > There is an agent property named host.reserved.mem.mb which is able to
> be set in the agent.properties file and passed to the management server.
> This value is stored as dom0MinMem, however, it appears to be ignored.
> >
> >
> > If we could tweak the host capacity calculation and have it take this
> into account, I believe that would be ideal.
> >
> >
> > Side note: the variable dom0MinMem is declared as an int and can be
> overflowed. Suggested change would be make it a long and add a check
> especially if we are going to make this a configurable parameter.
> >
> >
>
> Yes, that seems like a sane thing to do. We should be able to say that
> eg 8GB of memory should stay available for the HV.
>
> Don't know where the problem lies though. A PR is welcome :)
>
> Wido
>
> >
> >
> > Josh Harshman
> >
> > Cloud Engineer
> >
> >
> > Intrinium
> > Tel: (509) 465-1234 x5259
> > Fax: (866) 565-4578
> > Lync / Skype: Josh.Harshman@intrinium.com <javascript:;>
> > Web: http://intrinium.com<http://intrinium.com/>
> >
> >
> > [Intrinium Long Sig Logo]<http://intrinium.com/>
> >
> >
> > [Facebook]<http://www.facebook.com/pages/intrinium/154891544582822>
> [Twitter] <http://twitter.com/intrinium>  [Linkedin] <
> http://www.linkedin.com/company/intrinium_networks_it_security?trk=fc_badge>
> [Youtube] <http://www.youtube.com/user/intrinium>  [Blog] <
> http://intrinium.com/blog/>
> >
> > Information Security and Compliance Consulting | Managed IT and Security
> Services | Cloud Services
> >
> >
> >
> >
> >
> > ________________________________
> >
> > This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom they are
> addressed. If you have received this email in error please notify the
> system manager. This message contains confidential information and is
> intended only for the individual named. If you are not the named addressee
> you should not disseminate, distribute or copy this e-mail. Please notify
> the sender immediately by e-mail if you have received this e-mail by
> mistake and delete this e-mail from your system. If you are not the
> intended recipient you are notified that disclosing, copying, distributing
> or taking any action in reliance on the contents of this information is
> strictly prohibited.
> >
>

RE: Discuss reserving memory on KVM hypervisors ref: CLOUDSTACK-8678

Posted by Josh Harshman <Jo...@intrinium.com>.
Wido -
Looks like the problem resides in the logic that does the detection of the host resources.  It looks like most, if not all, the KVM host detection occurs in the LibvirtComputingResource class.  Here I can modify it so it will take the value stored in dom0MinMem into account when calculating the ram available.

I've forked the repo and am working on a fix.
Does anyone have anything to add?  Suggestions?

________________________________________
From: Wido den Hollander <wi...@widodh.nl>
Sent: Friday, August 28, 2015 12:38 AM
To: dev@cloudstack.apache.org
Subject: Re: Discuss reserving memory on KVM hypervisors ref: CLOUDSTACK-8678



Josh Harshman
Cloud Engineer

Intrinium

On 27-08-15 18:36, Josh Harshman wrote:
> In a KVM cluster, CloudStack sees 100% of the compute node's RAM and treats it as allocatable space which eventually leads to OOM killing guests.
>
>
> There is an agent property named host.reserved.mem.mb which is able to be set in the agent.properties file and passed to the management server.  This value is stored as dom0MinMem, however, it appears to be ignored.
>
>
> If we could tweak the host capacity calculation and have it take this into account, I believe that would be ideal.
>
>
> Side note: the variable dom0MinMem is declared as an int and can be overflowed. Suggested change would be make it a long and add a check especially if we are going to make this a configurable parameter.
>
>

Yes, that seems like a sane thing to do. We should be able to say that
eg 8GB of memory should stay available for the HV.

Don't know where the problem lies though. A PR is welcome :)

Wido

>
>
> Josh Harshman
>
> Cloud Engineer
>
>
> Intrinium
> Tel: (509) 465-1234 x5259
> Fax: (866) 565-4578
> Lync / Skype: Josh.Harshman@intrinium.com
> Web: http://intrinium.com<http://intrinium.com/>
>
>
> [Intrinium Long Sig Logo]<http://intrinium.com/>
>
>
> [Facebook]<http://www.facebook.com/pages/intrinium/154891544582822> [Twitter] <http://twitter.com/intrinium>  [Linkedin] <http://www.linkedin.com/company/intrinium_networks_it_security?trk=fc_badge>  [Youtube] <http://www.youtube.com/user/intrinium>  [Blog] <http://intrinium.com/blog/>
>
> Information Security and Compliance Consulting | Managed IT and Security Services | Cloud Services
>
>
>
>
>
> ________________________________
>
> This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
>

Re: Discuss reserving memory on KVM hypervisors ref: CLOUDSTACK-8678

Posted by Wido den Hollander <wi...@widodh.nl>.

On 27-08-15 18:36, Josh Harshman wrote:
> In a KVM cluster, CloudStack sees 100% of the compute node's RAM and treats it as allocatable space which eventually leads to OOM killing guests.
> 
> 
> There is an agent property named host.reserved.mem.mb which is able to be set in the agent.properties file and passed to the management server.  This value is stored as dom0MinMem, however, it appears to be ignored.
> 
> 
> If we could tweak the host capacity calculation and have it take this into account, I believe that would be ideal.
> 
> 
> Side note: the variable dom0MinMem is declared as an int and can be overflowed. Suggested change would be make it a long and add a check especially if we are going to make this a configurable parameter.
> 
> 

Yes, that seems like a sane thing to do. We should be able to say that
eg 8GB of memory should stay available for the HV.

Don't know where the problem lies though. A PR is welcome :)

Wido

> 
> 
> Josh Harshman
> 
> Cloud Engineer
> 
> 
> Intrinium
> Tel: (509) 465-1234 x5259
> Fax: (866) 565-4578
> Lync / Skype: Josh.Harshman@intrinium.com
> Web: http://intrinium.com<http://intrinium.com/>
> 
> 
> [Intrinium Long Sig Logo]<http://intrinium.com/>
> 
> 
> [Facebook]<http://www.facebook.com/pages/intrinium/154891544582822> [Twitter] <http://twitter.com/intrinium>  [Linkedin] <http://www.linkedin.com/company/intrinium_networks_it_security?trk=fc_badge>  [Youtube] <http://www.youtube.com/user/intrinium>  [Blog] <http://intrinium.com/blog/>
> 
> Information Security and Compliance Consulting | Managed IT and Security Services | Cloud Services
> 
> 
> 
> 
> 
> ________________________________
> 
> This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
>