You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by chris snow <ch...@gmail.com> on 2014/05/26 13:57:37 UTC

autoscaling - load average and memory consumption on docker/lxc instances

Hi Guys,

I noticed on my stratos+openstack+docker installation that stratos was
getting into a cycle trying to spawn new instances.

I think this is because load average (and probably memory consumption)
on lxc/docker instances is the same value as for the host [1].

Question: Am I correct in thinking that load average and memory
consumption autoscaling parameters aren't a good idea for docker
guests?

Many thanks,

Chris

---
[1] http://unix.stackexchange.com/questions/132021/is-a-lxc-docker-guest-load-average-the-same-as-the-hosts-load-average

Re: autoscaling - load average and memory consumption on docker/lxc instances

Posted by Shaheed Haque <sh...@cisco.com>.
On Saturday 31 May 2014 10:00:33 Lahiru Sandaruwan wrote:
> Hi Shaheed,
> 
> Could you explain the implementation a bit? Which technologies should be
> used in plug-able code, e.g. bash script etc.

Certainly. What we have in mind is a Java interface, maybe like this (ignore the exact 
names):

public class MonitorResult
{
    // range 0-100, notionally memory usage
    private double metric1;

    // range 0-100, notionally cpu usage
    private double metric2;

    public MonitorResult(double metric1, double metric2)
    {
...
    }
}

public interface IMonitorPlugin
{
    public boolean init();

    public void delete();

    public MonitorResult getMetrics();
        //throws Unobtainable;
}

The reason for Java and the init() and delete() hooks is to allow a .jar file to create - for 
example a TCP session or similar stateful object - and then use that to get the results 
each time.
     *  as hinted above, it is not clear to me what should happen on failure to get the 
values - suggestions welcome!
Bash could be supported by providing a Bash-calling .jar file. The selection of the plugin 
is via a System.property.

> Also can we extend this model to collect different stats user wants?

I considered that and concluded "yes and no":
     *  "Yes" in the sense that for pretty much any case I can think of, one value, 
normalised to the current 0-100% seems perfectly adequate.
         *   Given that we already have two, we may as well keep that (for the cases I cannot 
conceive of :-)).
             *  "No" in the sense that generalising support for more than two values seems 
redundant.
                 *  also support for N > 2 values seems like a LOT of work.

I hope that all sounds sane.

Thanks, Shaheed

> Thanks.
> 
> On Sat, May 31, 2014 at 1:51 AM, Shaheed Haque <sh...@cisco.com> wrote:
> > BTW, we are experimenting with a patch whose intent is to make the
> > memory/CPU
> > stats collection pluggable. We would definitely be interested in pushing
> > this
> > upstream - once we have something working.
> > 
> > On Wednesday 28 May 2014 22:37:32 Imesh Gunaratne wrote:
> > > Great! Thanks for the clarification Chris!
> > > 
> > > I just noted following from Docker's article on collecting metrics in
> > 
> > LXCs:
> > > "Docker will take care of this, or at least expose hooks to make it more
> > > straightforward. It is one of the reasons why we repeat over and over
> > > “Docker is not production ready yet”"
> > > 
> > > On Wed, May 28, 2014 at 9:01 PM, chris snow <ch...@gmail.com> wrote:
> > > > Thanks for the confirmation Nirmal.
> > > > 
> > > > On Wed, May 28, 2014 at 3:20 PM, Nirmal Fernando <
> > 
> > nirmal070125@gmail.com>
> > 
> > > > wrote:
> > > > > On Wed, May 28, 2014 at 4:49 PM, chris snow <ch...@gmail.com>
> > 
> > wrote:
> > > > >> Hi Imesh,
> > > > >> 
> > > > >> The autoscale policy relates to the cartridge, but on LXC you
> > > > >> aren't
> > > > >> monitoring the cartridge value, instead you are monitoring the
> > > > >> host.
> > > > >> If the load average hits a threshold because the host is presumably
> > > > >> under duress so the last thing you will want to do is spawn more
> > > > >> LXC
> > > > >> instances on the host?
> > > > > 
> > > > > Correct!
> > > > > 
> > > > >> Cheers,
> > > > >> 
> > > > >> Chris
> > > > >> 
> > > > >> 
> > > > >> 
> > > > >> On Wed, May 28, 2014 at 11:16 AM, Imesh Gunaratne <imesh@apache.org

Re: autoscaling - load average and memory consumption on docker/lxc instances

Posted by Imesh Gunaratne <im...@apache.org>.
Really nice to hear that Shaheed! Please create a Jira and attach the patch
once you are done.

Thanks


On Sat, May 31, 2014 at 1:51 AM, Shaheed Haque <sh...@cisco.com> wrote:

>
> BTW, we are experimenting with a patch whose intent is to make the
> memory/CPU
> stats collection pluggable. We would definitely be interested in pushing
> this
> upstream - once we have something working.
>
> On Wednesday 28 May 2014 22:37:32 Imesh Gunaratne wrote:
> > Great! Thanks for the clarification Chris!
> >
> > I just noted following from Docker's article on collecting metrics in
> LXCs:
> > "Docker will take care of this, or at least expose hooks to make it more
> > straightforward. It is one of the reasons why we repeat over and over
> > “Docker is not production ready yet”"
> >
> > On Wed, May 28, 2014 at 9:01 PM, chris snow <ch...@gmail.com> wrote:
> > > Thanks for the confirmation Nirmal.
> > >
> > > On Wed, May 28, 2014 at 3:20 PM, Nirmal Fernando <
> nirmal070125@gmail.com>
> > >
> > > wrote:
> > > > On Wed, May 28, 2014 at 4:49 PM, chris snow <ch...@gmail.com>
> wrote:
> > > >> Hi Imesh,
> > > >>
> > > >> The autoscale policy relates to the cartridge, but on LXC you aren't
> > > >> monitoring the cartridge value, instead you are monitoring the host.
> > > >> If the load average hits a threshold because the host is presumably
> > > >> under duress so the last thing you will want to do is spawn more LXC
> > > >> instances on the host?
> > > >
> > > > Correct!
> > > >
> > > >> Cheers,
> > > >>
> > > >> Chris
> > > >>
> > > >>
> > > >>
> > > >> On Wed, May 28, 2014 at 11:16 AM, Imesh Gunaratne <imesh@apache.org
> >
> > > >>
> > > >> wrote:
> > > >> > Hi Chris,
> > > >> >
> > > >> > Thanks for the quick response.
> > > >> > Right, I still don't get why we cannot adjust the threshold values
> > > >> > and
> > > >> > fix
> > > >> > it.
> > > >> >
> > > >> > Thanks
> > > >> >
> > > >> >
> > > >> > On Wed, May 28, 2014 at 11:37 AM, chris snow <chsnow123@gmail.com
> >
> > > >> >
> > > >> > wrote:
> > > >> >> Hi Imesh,
> > > >> >>
> > > >> >> I haven't looked into this in any more detail. All I have done is
> > > >> >> removed
> > > >> >> those parameters from my auto scaling policy.
> > > >> >>
> > > >> >> Many thanks,
> > > >> >> Chris
> > > >> >>
> > > >> >> On 28 May 2014 05:58, "Imesh Gunaratne" <im...@apache.org>
> wrote:
> > > >> >>> Hi Chris,
> > > >> >>>
> > > >> >>> Were you able to check the memory consumption and load average
> > >
> > > values
> > >
> > > >> >>> sent to Autoscaler by any chance?
> > > >> >>>
> > > >> >>> Thanks
> > > >> >>>
> > > >> >>>
> > > >> >>> On Mon, May 26, 2014 at 6:31 PM, chris snow <
> chsnow123@gmail.com>
> > > >> >>>
> > > >> >>> wrote:
> > > >> >>>> Hi Lahiru,
> > > >> >>>>
> > > >> >>>> It looks like there's some good info here [1].  However, as a
> > > >> >>>> first
> > > >> >>>> step, I was wondering whether we should add a note to the wiki
> to
> > > >> >>>> recommend not using the memory consumption or load average
> > >
> > > parameters
> > >
> > > >> >>>> for docker/lxc guests [2]?
> > > >> >>>>
> > > >> >>>> Cheers,
> > > >> >>>>
> > > >> >>>> Chris
> > > >> >>>>
> > > >> >>>> ---
> > > >> >>>> [1]
> > >
> > > http://blog.docker.io/2013/10/gathering-lxc-docker-containers-metrics/
> > >
> > > >> >>>> [2
> > >
> > >
> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Auto-scaling+Gui
> > > de#id-4.0.0Auto-scalingGuide-memoryConsumptio>
> > > >> >>>> On Mon, May 26, 2014 at 1:24 PM, Lahiru Sandaruwan <
> > >
> > > lahirus@wso2.com>
> > >
> > > >> >>>> wrote:
> > > >> >>>> > Hi Chis.
> > > >> >>>> >
> > > >> >>>> >
> > > >> >>>> > On Mon, May 26, 2014 at 5:27 PM, chris snow <
> chsnow123@gmail.com
> > > >> >>>> >
> > > >> >>>> > wrote:
> > > >> >>>> >> Hi Guys,
> > > >> >>>> >>
> > > >> >>>> >> I noticed on my stratos+openstack+docker installation that
> > >
> > > stratos
> > >
> > > >> >>>> >> was
> > > >> >>>> >> getting into a cycle trying to spawn new instances.
> > > >> >>>> >>
> > > >> >>>> >> I think this is because load average (and probably memory
> > > >> >>>> >> consumption)
> > > >> >>>> >> on lxc/docker instances is the same value as for the host
> [1].
> > > >> >>>> >>
> > > >> >>>> >> Question: Am I correct in thinking that load average and
> memory
> > > >> >>>> >> consumption autoscaling parameters aren't a good idea for
> > > >> >>>> >> docker
> > > >> >>>> >> guests?
> > > >> >>>> >
> > > >> >>>> > Is there a way to find out the load average and memory
> > >
> > > consumption
> > >
> > > >> >>>> > of
> > > >> >>>> > the
> > > >> >>>> > guest?
> > > >> >>>> >
> > > >> >>>> > I think the proper solution is to customize/fix Cartridge
> agent
> > > >> >>>> > find
> > > >> >>>> > out
> > > >> >>>> > relevant values in these type of environments and publish
> stats
> > >
> > > to
> > >
> > > >> >>>> > CEP.
> > > >> >>>> >
> > > >> >>>> > Thanks.
> > > >> >>>> >
> > > >> >>>> >> Many thanks,
> > > >> >>>> >>
> > > >> >>>> >> Chris
> > > >> >>>> >>
> > > >> >>>> >> ---
> > > >> >>>> >> [1]
> > >
> > >
> http://unix.stackexchange.com/questions/132021/is-a-lxc-docker-guest-load->
> > average-the-same-as-the-hosts-load-average>
> > > >> >>>> > --
> > > >> >>>> > --
> > > >> >>>> > Lahiru Sandaruwan
> > > >> >>>> > Committer and PMC member, Apache Stratos,
> > > >> >>>> > Senior Software Engineer,
> > > >> >>>> > WSO2 Inc., http://wso2.com
> > > >> >>>> > lean.enterprise.middleware
> > > >> >>>> >
> > > >> >>>> > email: lahirus@wso2.com cell: (+94) 773 325 954
> > > >> >>>> > blog: http://lahiruwrites.blogspot.com/
> > > >> >>>> > twitter: http://twitter.com/lahirus
> > >
> > > >> >>>> > linked-in:
> > > http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
> > >
> > > >> >>>> --
> > > >> >>>> Check out my professional profile and connect with me on
> LinkedIn.
> > > >> >>>> http://lnkd.in/cw5k69
> > > >> >>>
> > > >> >>> --
> > > >> >>> Imesh Gunaratne
> > > >> >>>
> > > >> >>> Technical Lead, WSO2
> > > >> >>> Committer & PPMC Member, Apache Stratos
> > > >> >
> > > >> > --
> > > >> > Imesh Gunaratne
> > > >> >
> > > >> > Technical Lead, WSO2
> > > >> > Committer & PPMC Member, Apache Stratos
> > > >>
> > > >> --
> > > >> Check out my professional profile and connect with me on LinkedIn.
> > > >> http://lnkd.in/cw5k69
> > > >
> > > > --
> > > > Best Regards,
> > > > Nirmal
> > > >
> > > > Nirmal Fernando.
> > > > PPMC Member & Committer of Apache Stratos,
> > > >
> > > > Senior Software Engineer, WSO2 Inc.
> > > >
> > > > Blog: http://nirmalfdo.blogspot.com/
> > >
> > > --
> > > Check out my professional profile and connect with me on LinkedIn.
> > > http://lnkd.in/cw5k69
>
>


-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PPMC Member, Apache Stratos

Re: autoscaling - load average and memory consumption on docker/lxc instances

Posted by Lahiru Sandaruwan <la...@wso2.com>.
Hi Shaheed,

Could you explain the implementation a bit? Which technologies should be
used in plug-able code, e.g. bash script etc.

Also can we extend this model to collect different stats user wants?

Thanks.


On Sat, May 31, 2014 at 1:51 AM, Shaheed Haque <sh...@cisco.com> wrote:

>
> BTW, we are experimenting with a patch whose intent is to make the
> memory/CPU
> stats collection pluggable. We would definitely be interested in pushing
> this
> upstream - once we have something working.
>
> On Wednesday 28 May 2014 22:37:32 Imesh Gunaratne wrote:
> > Great! Thanks for the clarification Chris!
> >
> > I just noted following from Docker's article on collecting metrics in
> LXCs:
> > "Docker will take care of this, or at least expose hooks to make it more
> > straightforward. It is one of the reasons why we repeat over and over
> > “Docker is not production ready yet”"
> >
> > On Wed, May 28, 2014 at 9:01 PM, chris snow <ch...@gmail.com> wrote:
> > > Thanks for the confirmation Nirmal.
> > >
> > > On Wed, May 28, 2014 at 3:20 PM, Nirmal Fernando <
> nirmal070125@gmail.com>
> > >
> > > wrote:
> > > > On Wed, May 28, 2014 at 4:49 PM, chris snow <ch...@gmail.com>
> wrote:
> > > >> Hi Imesh,
> > > >>
> > > >> The autoscale policy relates to the cartridge, but on LXC you aren't
> > > >> monitoring the cartridge value, instead you are monitoring the host.
> > > >> If the load average hits a threshold because the host is presumably
> > > >> under duress so the last thing you will want to do is spawn more LXC
> > > >> instances on the host?
> > > >
> > > > Correct!
> > > >
> > > >> Cheers,
> > > >>
> > > >> Chris
> > > >>
> > > >>
> > > >>
> > > >> On Wed, May 28, 2014 at 11:16 AM, Imesh Gunaratne <imesh@apache.org
> >
> > > >>
> > > >> wrote:
> > > >> > Hi Chris,
> > > >> >
> > > >> > Thanks for the quick response.
> > > >> > Right, I still don't get why we cannot adjust the threshold values
> > > >> > and
> > > >> > fix
> > > >> > it.
> > > >> >
> > > >> > Thanks
> > > >> >
> > > >> >
> > > >> > On Wed, May 28, 2014 at 11:37 AM, chris snow <chsnow123@gmail.com
> >
> > > >> >
> > > >> > wrote:
> > > >> >> Hi Imesh,
> > > >> >>
> > > >> >> I haven't looked into this in any more detail. All I have done is
> > > >> >> removed
> > > >> >> those parameters from my auto scaling policy.
> > > >> >>
> > > >> >> Many thanks,
> > > >> >> Chris
> > > >> >>
> > > >> >> On 28 May 2014 05:58, "Imesh Gunaratne" <im...@apache.org>
> wrote:
> > > >> >>> Hi Chris,
> > > >> >>>
> > > >> >>> Were you able to check the memory consumption and load average
> > >
> > > values
> > >
> > > >> >>> sent to Autoscaler by any chance?
> > > >> >>>
> > > >> >>> Thanks
> > > >> >>>
> > > >> >>>
> > > >> >>> On Mon, May 26, 2014 at 6:31 PM, chris snow <
> chsnow123@gmail.com>
> > > >> >>>
> > > >> >>> wrote:
> > > >> >>>> Hi Lahiru,
> > > >> >>>>
> > > >> >>>> It looks like there's some good info here [1].  However, as a
> > > >> >>>> first
> > > >> >>>> step, I was wondering whether we should add a note to the wiki
> to
> > > >> >>>> recommend not using the memory consumption or load average
> > >
> > > parameters
> > >
> > > >> >>>> for docker/lxc guests [2]?
> > > >> >>>>
> > > >> >>>> Cheers,
> > > >> >>>>
> > > >> >>>> Chris
> > > >> >>>>
> > > >> >>>> ---
> > > >> >>>> [1]
> > >
> > > http://blog.docker.io/2013/10/gathering-lxc-docker-containers-metrics/
> > >
> > > >> >>>> [2
> > >
> > >
> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Auto-scaling+Gui
> > > de#id-4.0.0Auto-scalingGuide-memoryConsumptio>
> > > >> >>>> On Mon, May 26, 2014 at 1:24 PM, Lahiru Sandaruwan <
> > >
> > > lahirus@wso2.com>
> > >
> > > >> >>>> wrote:
> > > >> >>>> > Hi Chis.
> > > >> >>>> >
> > > >> >>>> >
> > > >> >>>> > On Mon, May 26, 2014 at 5:27 PM, chris snow <
> chsnow123@gmail.com
> > > >> >>>> >
> > > >> >>>> > wrote:
> > > >> >>>> >> Hi Guys,
> > > >> >>>> >>
> > > >> >>>> >> I noticed on my stratos+openstack+docker installation that
> > >
> > > stratos
> > >
> > > >> >>>> >> was
> > > >> >>>> >> getting into a cycle trying to spawn new instances.
> > > >> >>>> >>
> > > >> >>>> >> I think this is because load average (and probably memory
> > > >> >>>> >> consumption)
> > > >> >>>> >> on lxc/docker instances is the same value as for the host
> [1].
> > > >> >>>> >>
> > > >> >>>> >> Question: Am I correct in thinking that load average and
> memory
> > > >> >>>> >> consumption autoscaling parameters aren't a good idea for
> > > >> >>>> >> docker
> > > >> >>>> >> guests?
> > > >> >>>> >
> > > >> >>>> > Is there a way to find out the load average and memory
> > >
> > > consumption
> > >
> > > >> >>>> > of
> > > >> >>>> > the
> > > >> >>>> > guest?
> > > >> >>>> >
> > > >> >>>> > I think the proper solution is to customize/fix Cartridge
> agent
> > > >> >>>> > find
> > > >> >>>> > out
> > > >> >>>> > relevant values in these type of environments and publish
> stats
> > >
> > > to
> > >
> > > >> >>>> > CEP.
> > > >> >>>> >
> > > >> >>>> > Thanks.
> > > >> >>>> >
> > > >> >>>> >> Many thanks,
> > > >> >>>> >>
> > > >> >>>> >> Chris
> > > >> >>>> >>
> > > >> >>>> >> ---
> > > >> >>>> >> [1]
> > >
> > >
> http://unix.stackexchange.com/questions/132021/is-a-lxc-docker-guest-load->
> > average-the-same-as-the-hosts-load-average>
> > > >> >>>> > --
> > > >> >>>> > --
> > > >> >>>> > Lahiru Sandaruwan
> > > >> >>>> > Committer and PMC member, Apache Stratos,
> > > >> >>>> > Senior Software Engineer,
> > > >> >>>> > WSO2 Inc., http://wso2.com
> > > >> >>>> > lean.enterprise.middleware
> > > >> >>>> >
> > > >> >>>> > email: lahirus@wso2.com cell: (+94) 773 325 954
> > > >> >>>> > blog: http://lahiruwrites.blogspot.com/
> > > >> >>>> > twitter: http://twitter.com/lahirus
> > >
> > > >> >>>> > linked-in:
> > > http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
> > >
> > > >> >>>> --
> > > >> >>>> Check out my professional profile and connect with me on
> LinkedIn.
> > > >> >>>> http://lnkd.in/cw5k69
> > > >> >>>
> > > >> >>> --
> > > >> >>> Imesh Gunaratne
> > > >> >>>
> > > >> >>> Technical Lead, WSO2
> > > >> >>> Committer & PPMC Member, Apache Stratos
> > > >> >
> > > >> > --
> > > >> > Imesh Gunaratne
> > > >> >
> > > >> > Technical Lead, WSO2
> > > >> > Committer & PPMC Member, Apache Stratos
> > > >>
> > > >> --
> > > >> Check out my professional profile and connect with me on LinkedIn.
> > > >> http://lnkd.in/cw5k69
> > > >
> > > > --
> > > > Best Regards,
> > > > Nirmal
> > > >
> > > > Nirmal Fernando.
> > > > PPMC Member & Committer of Apache Stratos,
> > > >
> > > > Senior Software Engineer, WSO2 Inc.
> > > >
> > > > Blog: http://nirmalfdo.blogspot.com/
> > >
> > > --
> > > Check out my professional profile and connect with me on LinkedIn.
> > > http://lnkd.in/cw5k69
>
>


-- 
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware

email: lahirus@wso2.com cell: (+94) 773 325 954
blog: http://lahiruwrites.blogspot.com/
twitter: http://twitter.com/lahirus
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146

Re: autoscaling - load average and memory consumption on docker/lxc instances

Posted by Shaheed Haque <sh...@cisco.com>.
BTW, we are experimenting with a patch whose intent is to make the memory/CPU 
stats collection pluggable. We would definitely be interested in pushing this 
upstream - once we have something working.

On Wednesday 28 May 2014 22:37:32 Imesh Gunaratne wrote:
> Great! Thanks for the clarification Chris!
> 
> I just noted following from Docker's article on collecting metrics in LXCs:
> "Docker will take care of this, or at least expose hooks to make it more
> straightforward. It is one of the reasons why we repeat over and over
> “Docker is not production ready yet”"
> 
> On Wed, May 28, 2014 at 9:01 PM, chris snow <ch...@gmail.com> wrote:
> > Thanks for the confirmation Nirmal.
> > 
> > On Wed, May 28, 2014 at 3:20 PM, Nirmal Fernando <ni...@gmail.com>
> > 
> > wrote:
> > > On Wed, May 28, 2014 at 4:49 PM, chris snow <ch...@gmail.com> wrote:
> > >> Hi Imesh,
> > >> 
> > >> The autoscale policy relates to the cartridge, but on LXC you aren't
> > >> monitoring the cartridge value, instead you are monitoring the host.
> > >> If the load average hits a threshold because the host is presumably
> > >> under duress so the last thing you will want to do is spawn more LXC
> > >> instances on the host?
> > > 
> > > Correct!
> > > 
> > >> Cheers,
> > >> 
> > >> Chris
> > >> 
> > >> 
> > >> 
> > >> On Wed, May 28, 2014 at 11:16 AM, Imesh Gunaratne <im...@apache.org>
> > >> 
> > >> wrote:
> > >> > Hi Chris,
> > >> > 
> > >> > Thanks for the quick response.
> > >> > Right, I still don't get why we cannot adjust the threshold values
> > >> > and
> > >> > fix
> > >> > it.
> > >> > 
> > >> > Thanks
> > >> > 
> > >> > 
> > >> > On Wed, May 28, 2014 at 11:37 AM, chris snow <ch...@gmail.com>
> > >> > 
> > >> > wrote:
> > >> >> Hi Imesh,
> > >> >> 
> > >> >> I haven't looked into this in any more detail. All I have done is
> > >> >> removed
> > >> >> those parameters from my auto scaling policy.
> > >> >> 
> > >> >> Many thanks,
> > >> >> Chris
> > >> >> 
> > >> >> On 28 May 2014 05:58, "Imesh Gunaratne" <im...@apache.org> wrote:
> > >> >>> Hi Chris,
> > >> >>> 
> > >> >>> Were you able to check the memory consumption and load average
> > 
> > values
> > 
> > >> >>> sent to Autoscaler by any chance?
> > >> >>> 
> > >> >>> Thanks
> > >> >>> 
> > >> >>> 
> > >> >>> On Mon, May 26, 2014 at 6:31 PM, chris snow <ch...@gmail.com>
> > >> >>> 
> > >> >>> wrote:
> > >> >>>> Hi Lahiru,
> > >> >>>> 
> > >> >>>> It looks like there's some good info here [1].  However, as a
> > >> >>>> first
> > >> >>>> step, I was wondering whether we should add a note to the wiki to
> > >> >>>> recommend not using the memory consumption or load average
> > 
> > parameters
> > 
> > >> >>>> for docker/lxc guests [2]?
> > >> >>>> 
> > >> >>>> Cheers,
> > >> >>>> 
> > >> >>>> Chris
> > >> >>>> 
> > >> >>>> ---
> > >> >>>> [1]
> > 
> > http://blog.docker.io/2013/10/gathering-lxc-docker-containers-metrics/
> > 
> > >> >>>> [2
> > 
> > https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Auto-scaling+Gui
> > de#id-4.0.0Auto-scalingGuide-memoryConsumptio> 
> > >> >>>> On Mon, May 26, 2014 at 1:24 PM, Lahiru Sandaruwan <
> > 
> > lahirus@wso2.com>
> > 
> > >> >>>> wrote:
> > >> >>>> > Hi Chis.
> > >> >>>> > 
> > >> >>>> > 
> > >> >>>> > On Mon, May 26, 2014 at 5:27 PM, chris snow <chsnow123@gmail.com
> > >> >>>> > 
> > >> >>>> > wrote:
> > >> >>>> >> Hi Guys,
> > >> >>>> >> 
> > >> >>>> >> I noticed on my stratos+openstack+docker installation that
> > 
> > stratos
> > 
> > >> >>>> >> was
> > >> >>>> >> getting into a cycle trying to spawn new instances.
> > >> >>>> >> 
> > >> >>>> >> I think this is because load average (and probably memory
> > >> >>>> >> consumption)
> > >> >>>> >> on lxc/docker instances is the same value as for the host [1].
> > >> >>>> >> 
> > >> >>>> >> Question: Am I correct in thinking that load average and memory
> > >> >>>> >> consumption autoscaling parameters aren't a good idea for
> > >> >>>> >> docker
> > >> >>>> >> guests?
> > >> >>>> > 
> > >> >>>> > Is there a way to find out the load average and memory
> > 
> > consumption
> > 
> > >> >>>> > of
> > >> >>>> > the
> > >> >>>> > guest?
> > >> >>>> > 
> > >> >>>> > I think the proper solution is to customize/fix Cartridge agent
> > >> >>>> > find
> > >> >>>> > out
> > >> >>>> > relevant values in these type of environments and publish stats
> > 
> > to
> > 
> > >> >>>> > CEP.
> > >> >>>> > 
> > >> >>>> > Thanks.
> > >> >>>> > 
> > >> >>>> >> Many thanks,
> > >> >>>> >> 
> > >> >>>> >> Chris
> > >> >>>> >> 
> > >> >>>> >> ---
> > >> >>>> >> [1]
> > 
> > http://unix.stackexchange.com/questions/132021/is-a-lxc-docker-guest-load-> > average-the-same-as-the-hosts-load-average> 
> > >> >>>> > --
> > >> >>>> > --
> > >> >>>> > Lahiru Sandaruwan
> > >> >>>> > Committer and PMC member, Apache Stratos,
> > >> >>>> > Senior Software Engineer,
> > >> >>>> > WSO2 Inc., http://wso2.com
> > >> >>>> > lean.enterprise.middleware
> > >> >>>> > 
> > >> >>>> > email: lahirus@wso2.com cell: (+94) 773 325 954
> > >> >>>> > blog: http://lahiruwrites.blogspot.com/
> > >> >>>> > twitter: http://twitter.com/lahirus
> > 
> > >> >>>> > linked-in:
> > http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
> > 
> > >> >>>> --
> > >> >>>> Check out my professional profile and connect with me on LinkedIn.
> > >> >>>> http://lnkd.in/cw5k69
> > >> >>> 
> > >> >>> --
> > >> >>> Imesh Gunaratne
> > >> >>> 
> > >> >>> Technical Lead, WSO2
> > >> >>> Committer & PPMC Member, Apache Stratos
> > >> > 
> > >> > --
> > >> > Imesh Gunaratne
> > >> > 
> > >> > Technical Lead, WSO2
> > >> > Committer & PPMC Member, Apache Stratos
> > >> 
> > >> --
> > >> Check out my professional profile and connect with me on LinkedIn.
> > >> http://lnkd.in/cw5k69
> > > 
> > > --
> > > Best Regards,
> > > Nirmal
> > > 
> > > Nirmal Fernando.
> > > PPMC Member & Committer of Apache Stratos,
> > > 
> > > Senior Software Engineer, WSO2 Inc.
> > > 
> > > Blog: http://nirmalfdo.blogspot.com/
> > 
> > --
> > Check out my professional profile and connect with me on LinkedIn.
> > http://lnkd.in/cw5k69


Re: autoscaling - load average and memory consumption on docker/lxc instances

Posted by Imesh Gunaratne <im...@apache.org>.
Great! Thanks for the clarification Chris!

I just noted following from Docker's article on collecting metrics in LXCs:
"Docker will take care of this, or at least expose hooks to make it more
straightforward. It is one of the reasons why we repeat over and over
“Docker is not production ready yet”"


On Wed, May 28, 2014 at 9:01 PM, chris snow <ch...@gmail.com> wrote:

> Thanks for the confirmation Nirmal.
>
> On Wed, May 28, 2014 at 3:20 PM, Nirmal Fernando <ni...@gmail.com>
> wrote:
> >
> >
> >
> > On Wed, May 28, 2014 at 4:49 PM, chris snow <ch...@gmail.com> wrote:
> >>
> >> Hi Imesh,
> >>
> >> The autoscale policy relates to the cartridge, but on LXC you aren't
> >> monitoring the cartridge value, instead you are monitoring the host.
> >> If the load average hits a threshold because the host is presumably
> >> under duress so the last thing you will want to do is spawn more LXC
> >> instances on the host?
> >
> >
> > Correct!
> >>
> >>
> >> Cheers,
> >>
> >> Chris
> >>
> >>
> >>
> >> On Wed, May 28, 2014 at 11:16 AM, Imesh Gunaratne <im...@apache.org>
> >> wrote:
> >> > Hi Chris,
> >> >
> >> > Thanks for the quick response.
> >> > Right, I still don't get why we cannot adjust the threshold values and
> >> > fix
> >> > it.
> >> >
> >> > Thanks
> >> >
> >> >
> >> > On Wed, May 28, 2014 at 11:37 AM, chris snow <ch...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Hi Imesh,
> >> >>
> >> >> I haven't looked into this in any more detail. All I have done is
> >> >> removed
> >> >> those parameters from my auto scaling policy.
> >> >>
> >> >> Many thanks,
> >> >> Chris
> >> >>
> >> >> On 28 May 2014 05:58, "Imesh Gunaratne" <im...@apache.org> wrote:
> >> >>>
> >> >>> Hi Chris,
> >> >>>
> >> >>> Were you able to check the memory consumption and load average
> values
> >> >>> sent to Autoscaler by any chance?
> >> >>>
> >> >>> Thanks
> >> >>>
> >> >>>
> >> >>> On Mon, May 26, 2014 at 6:31 PM, chris snow <ch...@gmail.com>
> >> >>> wrote:
> >> >>>>
> >> >>>> Hi Lahiru,
> >> >>>>
> >> >>>> It looks like there's some good info here [1].  However, as a first
> >> >>>> step, I was wondering whether we should add a note to the wiki to
> >> >>>> recommend not using the memory consumption or load average
> parameters
> >> >>>> for docker/lxc guests [2]?
> >> >>>>
> >> >>>> Cheers,
> >> >>>>
> >> >>>> Chris
> >> >>>>
> >> >>>> ---
> >> >>>> [1]
> >> >>>>
> >> >>>>
> http://blog.docker.io/2013/10/gathering-lxc-docker-containers-metrics/
> >> >>>> [2
> >> >>>>
> >> >>>>
> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Auto-scaling+Guide#id-4.0.0Auto-scalingGuide-memoryConsumptio
> >> >>>>
> >> >>>> On Mon, May 26, 2014 at 1:24 PM, Lahiru Sandaruwan <
> lahirus@wso2.com>
> >> >>>> wrote:
> >> >>>> > Hi Chis.
> >> >>>> >
> >> >>>> >
> >> >>>> > On Mon, May 26, 2014 at 5:27 PM, chris snow <chsnow123@gmail.com
> >
> >> >>>> > wrote:
> >> >>>> >>
> >> >>>> >> Hi Guys,
> >> >>>> >>
> >> >>>> >> I noticed on my stratos+openstack+docker installation that
> stratos
> >> >>>> >> was
> >> >>>> >> getting into a cycle trying to spawn new instances.
> >> >>>> >>
> >> >>>> >> I think this is because load average (and probably memory
> >> >>>> >> consumption)
> >> >>>> >> on lxc/docker instances is the same value as for the host [1].
> >> >>>> >>
> >> >>>> >> Question: Am I correct in thinking that load average and memory
> >> >>>> >> consumption autoscaling parameters aren't a good idea for docker
> >> >>>> >> guests?
> >> >>>> >
> >> >>>> >
> >> >>>> > Is there a way to find out the load average and memory
> consumption
> >> >>>> > of
> >> >>>> > the
> >> >>>> > guest?
> >> >>>> >
> >> >>>> > I think the proper solution is to customize/fix Cartridge agent
> >> >>>> > find
> >> >>>> > out
> >> >>>> > relevant values in these type of environments and publish stats
> to
> >> >>>> > CEP.
> >> >>>> >
> >> >>>> > Thanks.
> >> >>>> >>
> >> >>>> >>
> >> >>>> >> Many thanks,
> >> >>>> >>
> >> >>>> >> Chris
> >> >>>> >>
> >> >>>> >> ---
> >> >>>> >> [1]
> >> >>>> >>
> >> >>>> >>
> >> >>>> >>
> http://unix.stackexchange.com/questions/132021/is-a-lxc-docker-guest-load-average-the-same-as-the-hosts-load-average
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > --
> >> >>>> > --
> >> >>>> > Lahiru Sandaruwan
> >> >>>> > Committer and PMC member, Apache Stratos,
> >> >>>> > Senior Software Engineer,
> >> >>>> > WSO2 Inc., http://wso2.com
> >> >>>> > lean.enterprise.middleware
> >> >>>> >
> >> >>>> > email: lahirus@wso2.com cell: (+94) 773 325 954
> >> >>>> > blog: http://lahiruwrites.blogspot.com/
> >> >>>> > twitter: http://twitter.com/lahirus
> >> >>>> > linked-in:
> http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
> >> >>>> >
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>> Check out my professional profile and connect with me on LinkedIn.
> >> >>>> http://lnkd.in/cw5k69
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Imesh Gunaratne
> >> >>>
> >> >>> Technical Lead, WSO2
> >> >>> Committer & PPMC Member, Apache Stratos
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Imesh Gunaratne
> >> >
> >> > Technical Lead, WSO2
> >> > Committer & PPMC Member, Apache Stratos
> >>
> >>
> >>
> >> --
> >> Check out my professional profile and connect with me on LinkedIn.
> >> http://lnkd.in/cw5k69
> >
> >
> >
> >
> > --
> > Best Regards,
> > Nirmal
> >
> > Nirmal Fernando.
> > PPMC Member & Committer of Apache Stratos,
> >
> > Senior Software Engineer, WSO2 Inc.
> >
> > Blog: http://nirmalfdo.blogspot.com/
>
>
>
> --
> Check out my professional profile and connect with me on LinkedIn.
> http://lnkd.in/cw5k69
>



-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PPMC Member, Apache Stratos

Re: autoscaling - load average and memory consumption on docker/lxc instances

Posted by chris snow <ch...@gmail.com>.
Thanks for the confirmation Nirmal.

On Wed, May 28, 2014 at 3:20 PM, Nirmal Fernando <ni...@gmail.com> wrote:
>
>
>
> On Wed, May 28, 2014 at 4:49 PM, chris snow <ch...@gmail.com> wrote:
>>
>> Hi Imesh,
>>
>> The autoscale policy relates to the cartridge, but on LXC you aren't
>> monitoring the cartridge value, instead you are monitoring the host.
>> If the load average hits a threshold because the host is presumably
>> under duress so the last thing you will want to do is spawn more LXC
>> instances on the host?
>
>
> Correct!
>>
>>
>> Cheers,
>>
>> Chris
>>
>>
>>
>> On Wed, May 28, 2014 at 11:16 AM, Imesh Gunaratne <im...@apache.org>
>> wrote:
>> > Hi Chris,
>> >
>> > Thanks for the quick response.
>> > Right, I still don't get why we cannot adjust the threshold values and
>> > fix
>> > it.
>> >
>> > Thanks
>> >
>> >
>> > On Wed, May 28, 2014 at 11:37 AM, chris snow <ch...@gmail.com>
>> > wrote:
>> >>
>> >> Hi Imesh,
>> >>
>> >> I haven't looked into this in any more detail. All I have done is
>> >> removed
>> >> those parameters from my auto scaling policy.
>> >>
>> >> Many thanks,
>> >> Chris
>> >>
>> >> On 28 May 2014 05:58, "Imesh Gunaratne" <im...@apache.org> wrote:
>> >>>
>> >>> Hi Chris,
>> >>>
>> >>> Were you able to check the memory consumption and load average values
>> >>> sent to Autoscaler by any chance?
>> >>>
>> >>> Thanks
>> >>>
>> >>>
>> >>> On Mon, May 26, 2014 at 6:31 PM, chris snow <ch...@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> Hi Lahiru,
>> >>>>
>> >>>> It looks like there's some good info here [1].  However, as a first
>> >>>> step, I was wondering whether we should add a note to the wiki to
>> >>>> recommend not using the memory consumption or load average parameters
>> >>>> for docker/lxc guests [2]?
>> >>>>
>> >>>> Cheers,
>> >>>>
>> >>>> Chris
>> >>>>
>> >>>> ---
>> >>>> [1]
>> >>>>
>> >>>> http://blog.docker.io/2013/10/gathering-lxc-docker-containers-metrics/
>> >>>> [2
>> >>>>
>> >>>> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Auto-scaling+Guide#id-4.0.0Auto-scalingGuide-memoryConsumptio
>> >>>>
>> >>>> On Mon, May 26, 2014 at 1:24 PM, Lahiru Sandaruwan <la...@wso2.com>
>> >>>> wrote:
>> >>>> > Hi Chis.
>> >>>> >
>> >>>> >
>> >>>> > On Mon, May 26, 2014 at 5:27 PM, chris snow <ch...@gmail.com>
>> >>>> > wrote:
>> >>>> >>
>> >>>> >> Hi Guys,
>> >>>> >>
>> >>>> >> I noticed on my stratos+openstack+docker installation that stratos
>> >>>> >> was
>> >>>> >> getting into a cycle trying to spawn new instances.
>> >>>> >>
>> >>>> >> I think this is because load average (and probably memory
>> >>>> >> consumption)
>> >>>> >> on lxc/docker instances is the same value as for the host [1].
>> >>>> >>
>> >>>> >> Question: Am I correct in thinking that load average and memory
>> >>>> >> consumption autoscaling parameters aren't a good idea for docker
>> >>>> >> guests?
>> >>>> >
>> >>>> >
>> >>>> > Is there a way to find out the load average and memory consumption
>> >>>> > of
>> >>>> > the
>> >>>> > guest?
>> >>>> >
>> >>>> > I think the proper solution is to customize/fix Cartridge agent
>> >>>> > find
>> >>>> > out
>> >>>> > relevant values in these type of environments and publish stats to
>> >>>> > CEP.
>> >>>> >
>> >>>> > Thanks.
>> >>>> >>
>> >>>> >>
>> >>>> >> Many thanks,
>> >>>> >>
>> >>>> >> Chris
>> >>>> >>
>> >>>> >> ---
>> >>>> >> [1]
>> >>>> >>
>> >>>> >>
>> >>>> >> http://unix.stackexchange.com/questions/132021/is-a-lxc-docker-guest-load-average-the-same-as-the-hosts-load-average
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > --
>> >>>> > --
>> >>>> > Lahiru Sandaruwan
>> >>>> > Committer and PMC member, Apache Stratos,
>> >>>> > Senior Software Engineer,
>> >>>> > WSO2 Inc., http://wso2.com
>> >>>> > lean.enterprise.middleware
>> >>>> >
>> >>>> > email: lahirus@wso2.com cell: (+94) 773 325 954
>> >>>> > blog: http://lahiruwrites.blogspot.com/
>> >>>> > twitter: http://twitter.com/lahirus
>> >>>> > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>> >>>> >
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Check out my professional profile and connect with me on LinkedIn.
>> >>>> http://lnkd.in/cw5k69
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Imesh Gunaratne
>> >>>
>> >>> Technical Lead, WSO2
>> >>> Committer & PPMC Member, Apache Stratos
>> >
>> >
>> >
>> >
>> > --
>> > Imesh Gunaratne
>> >
>> > Technical Lead, WSO2
>> > Committer & PPMC Member, Apache Stratos
>>
>>
>>
>> --
>> Check out my professional profile and connect with me on LinkedIn.
>> http://lnkd.in/cw5k69
>
>
>
>
> --
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
>
> Senior Software Engineer, WSO2 Inc.
>
> Blog: http://nirmalfdo.blogspot.com/



-- 
Check out my professional profile and connect with me on LinkedIn.
http://lnkd.in/cw5k69

Re: autoscaling - load average and memory consumption on docker/lxc instances

Posted by Nirmal Fernando <ni...@gmail.com>.
On Wed, May 28, 2014 at 4:49 PM, chris snow <ch...@gmail.com> wrote:

> Hi Imesh,
>
> The autoscale policy relates to the cartridge, but on LXC you aren't
> monitoring the cartridge value, instead you are monitoring the host.
> If the load average hits a threshold because the host is presumably
> under duress so the last thing you will want to do is spawn more LXC
> instances on the host?
>

Correct!

>
> Cheers,
>
> Chris
>
>
>
> On Wed, May 28, 2014 at 11:16 AM, Imesh Gunaratne <im...@apache.org>
> wrote:
> > Hi Chris,
> >
> > Thanks for the quick response.
> > Right, I still don't get why we cannot adjust the threshold values and
> fix
> > it.
> >
> > Thanks
> >
> >
> > On Wed, May 28, 2014 at 11:37 AM, chris snow <ch...@gmail.com>
> wrote:
> >>
> >> Hi Imesh,
> >>
> >> I haven't looked into this in any more detail. All I have done is
> removed
> >> those parameters from my auto scaling policy.
> >>
> >> Many thanks,
> >> Chris
> >>
> >> On 28 May 2014 05:58, "Imesh Gunaratne" <im...@apache.org> wrote:
> >>>
> >>> Hi Chris,
> >>>
> >>> Were you able to check the memory consumption and load average values
> >>> sent to Autoscaler by any chance?
> >>>
> >>> Thanks
> >>>
> >>>
> >>> On Mon, May 26, 2014 at 6:31 PM, chris snow <ch...@gmail.com>
> wrote:
> >>>>
> >>>> Hi Lahiru,
> >>>>
> >>>> It looks like there's some good info here [1].  However, as a first
> >>>> step, I was wondering whether we should add a note to the wiki to
> >>>> recommend not using the memory consumption or load average parameters
> >>>> for docker/lxc guests [2]?
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Chris
> >>>>
> >>>> ---
> >>>> [1]
> >>>>
> http://blog.docker.io/2013/10/gathering-lxc-docker-containers-metrics/
> >>>> [2
> >>>>
> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Auto-scaling+Guide#id-4.0.0Auto-scalingGuide-memoryConsumptio
> >>>>
> >>>> On Mon, May 26, 2014 at 1:24 PM, Lahiru Sandaruwan <la...@wso2.com>
> >>>> wrote:
> >>>> > Hi Chis.
> >>>> >
> >>>> >
> >>>> > On Mon, May 26, 2014 at 5:27 PM, chris snow <ch...@gmail.com>
> >>>> > wrote:
> >>>> >>
> >>>> >> Hi Guys,
> >>>> >>
> >>>> >> I noticed on my stratos+openstack+docker installation that stratos
> >>>> >> was
> >>>> >> getting into a cycle trying to spawn new instances.
> >>>> >>
> >>>> >> I think this is because load average (and probably memory
> >>>> >> consumption)
> >>>> >> on lxc/docker instances is the same value as for the host [1].
> >>>> >>
> >>>> >> Question: Am I correct in thinking that load average and memory
> >>>> >> consumption autoscaling parameters aren't a good idea for docker
> >>>> >> guests?
> >>>> >
> >>>> >
> >>>> > Is there a way to find out the load average and memory consumption
> of
> >>>> > the
> >>>> > guest?
> >>>> >
> >>>> > I think the proper solution is to customize/fix Cartridge agent find
> >>>> > out
> >>>> > relevant values in these type of environments and publish stats to
> >>>> > CEP.
> >>>> >
> >>>> > Thanks.
> >>>> >>
> >>>> >>
> >>>> >> Many thanks,
> >>>> >>
> >>>> >> Chris
> >>>> >>
> >>>> >> ---
> >>>> >> [1]
> >>>> >>
> >>>> >>
> http://unix.stackexchange.com/questions/132021/is-a-lxc-docker-guest-load-average-the-same-as-the-hosts-load-average
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > --
> >>>> > --
> >>>> > Lahiru Sandaruwan
> >>>> > Committer and PMC member, Apache Stratos,
> >>>> > Senior Software Engineer,
> >>>> > WSO2 Inc., http://wso2.com
> >>>> > lean.enterprise.middleware
> >>>> >
> >>>> > email: lahirus@wso2.com cell: (+94) 773 325 954
> >>>> > blog: http://lahiruwrites.blogspot.com/
> >>>> > twitter: http://twitter.com/lahirus
> >>>> > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
> >>>> >
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Check out my professional profile and connect with me on LinkedIn.
> >>>> http://lnkd.in/cw5k69
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Imesh Gunaratne
> >>>
> >>> Technical Lead, WSO2
> >>> Committer & PPMC Member, Apache Stratos
> >
> >
> >
> >
> > --
> > Imesh Gunaratne
> >
> > Technical Lead, WSO2
> > Committer & PPMC Member, Apache Stratos
>
>
>
> --
> Check out my professional profile and connect with me on LinkedIn.
> http://lnkd.in/cw5k69
>



-- 
Best Regards,
Nirmal

Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.

Blog: http://nirmalfdo.blogspot.com/

Re: autoscaling - load average and memory consumption on docker/lxc instances

Posted by chris snow <ch...@gmail.com>.
Imesh, does my previous email make sense, or am I missing the point somewhere?

On Wed, May 28, 2014 at 12:19 PM, chris snow <ch...@gmail.com> wrote:
> Hi Imesh,
>
> The autoscale policy relates to the cartridge, but on LXC you aren't
> monitoring the cartridge value, instead you are monitoring the host.
> If the load average hits a threshold because the host is presumably
> under duress so the last thing you will want to do is spawn more LXC
> instances on the host?
>
> Cheers,
>
> Chris
>
>
>
> On Wed, May 28, 2014 at 11:16 AM, Imesh Gunaratne <im...@apache.org> wrote:
>> Hi Chris,
>>
>> Thanks for the quick response.
>> Right, I still don't get why we cannot adjust the threshold values and fix
>> it.
>>
>> Thanks
>>
>>
>> On Wed, May 28, 2014 at 11:37 AM, chris snow <ch...@gmail.com> wrote:
>>>
>>> Hi Imesh,
>>>
>>> I haven't looked into this in any more detail. All I have done is removed
>>> those parameters from my auto scaling policy.
>>>
>>> Many thanks,
>>> Chris
>>>
>>> On 28 May 2014 05:58, "Imesh Gunaratne" <im...@apache.org> wrote:
>>>>
>>>> Hi Chris,
>>>>
>>>> Were you able to check the memory consumption and load average values
>>>> sent to Autoscaler by any chance?
>>>>
>>>> Thanks
>>>>
>>>>
>>>> On Mon, May 26, 2014 at 6:31 PM, chris snow <ch...@gmail.com> wrote:
>>>>>
>>>>> Hi Lahiru,
>>>>>
>>>>> It looks like there's some good info here [1].  However, as a first
>>>>> step, I was wondering whether we should add a note to the wiki to
>>>>> recommend not using the memory consumption or load average parameters
>>>>> for docker/lxc guests [2]?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Chris
>>>>>
>>>>> ---
>>>>> [1]
>>>>> http://blog.docker.io/2013/10/gathering-lxc-docker-containers-metrics/
>>>>> [2
>>>>> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Auto-scaling+Guide#id-4.0.0Auto-scalingGuide-memoryConsumptio
>>>>>
>>>>> On Mon, May 26, 2014 at 1:24 PM, Lahiru Sandaruwan <la...@wso2.com>
>>>>> wrote:
>>>>> > Hi Chis.
>>>>> >
>>>>> >
>>>>> > On Mon, May 26, 2014 at 5:27 PM, chris snow <ch...@gmail.com>
>>>>> > wrote:
>>>>> >>
>>>>> >> Hi Guys,
>>>>> >>
>>>>> >> I noticed on my stratos+openstack+docker installation that stratos
>>>>> >> was
>>>>> >> getting into a cycle trying to spawn new instances.
>>>>> >>
>>>>> >> I think this is because load average (and probably memory
>>>>> >> consumption)
>>>>> >> on lxc/docker instances is the same value as for the host [1].
>>>>> >>
>>>>> >> Question: Am I correct in thinking that load average and memory
>>>>> >> consumption autoscaling parameters aren't a good idea for docker
>>>>> >> guests?
>>>>> >
>>>>> >
>>>>> > Is there a way to find out the load average and memory consumption of
>>>>> > the
>>>>> > guest?
>>>>> >
>>>>> > I think the proper solution is to customize/fix Cartridge agent find
>>>>> > out
>>>>> > relevant values in these type of environments and publish stats to
>>>>> > CEP.
>>>>> >
>>>>> > Thanks.
>>>>> >>
>>>>> >>
>>>>> >> Many thanks,
>>>>> >>
>>>>> >> Chris
>>>>> >>
>>>>> >> ---
>>>>> >> [1]
>>>>> >>
>>>>> >> http://unix.stackexchange.com/questions/132021/is-a-lxc-docker-guest-load-average-the-same-as-the-hosts-load-average
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > --
>>>>> > Lahiru Sandaruwan
>>>>> > Committer and PMC member, Apache Stratos,
>>>>> > Senior Software Engineer,
>>>>> > WSO2 Inc., http://wso2.com
>>>>> > lean.enterprise.middleware
>>>>> >
>>>>> > email: lahirus@wso2.com cell: (+94) 773 325 954
>>>>> > blog: http://lahiruwrites.blogspot.com/
>>>>> > twitter: http://twitter.com/lahirus
>>>>> > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Check out my professional profile and connect with me on LinkedIn.
>>>>> http://lnkd.in/cw5k69
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Imesh Gunaratne
>>>>
>>>> Technical Lead, WSO2
>>>> Committer & PPMC Member, Apache Stratos
>>
>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PPMC Member, Apache Stratos
>
>
>
> --
> Check out my professional profile and connect with me on LinkedIn.
> http://lnkd.in/cw5k69



-- 
Check out my professional profile and connect with me on LinkedIn.
http://lnkd.in/cw5k69

Re: autoscaling - load average and memory consumption on docker/lxc instances

Posted by chris snow <ch...@gmail.com>.
Hi Imesh,

The autoscale policy relates to the cartridge, but on LXC you aren't
monitoring the cartridge value, instead you are monitoring the host.
If the load average hits a threshold because the host is presumably
under duress so the last thing you will want to do is spawn more LXC
instances on the host?

Cheers,

Chris



On Wed, May 28, 2014 at 11:16 AM, Imesh Gunaratne <im...@apache.org> wrote:
> Hi Chris,
>
> Thanks for the quick response.
> Right, I still don't get why we cannot adjust the threshold values and fix
> it.
>
> Thanks
>
>
> On Wed, May 28, 2014 at 11:37 AM, chris snow <ch...@gmail.com> wrote:
>>
>> Hi Imesh,
>>
>> I haven't looked into this in any more detail. All I have done is removed
>> those parameters from my auto scaling policy.
>>
>> Many thanks,
>> Chris
>>
>> On 28 May 2014 05:58, "Imesh Gunaratne" <im...@apache.org> wrote:
>>>
>>> Hi Chris,
>>>
>>> Were you able to check the memory consumption and load average values
>>> sent to Autoscaler by any chance?
>>>
>>> Thanks
>>>
>>>
>>> On Mon, May 26, 2014 at 6:31 PM, chris snow <ch...@gmail.com> wrote:
>>>>
>>>> Hi Lahiru,
>>>>
>>>> It looks like there's some good info here [1].  However, as a first
>>>> step, I was wondering whether we should add a note to the wiki to
>>>> recommend not using the memory consumption or load average parameters
>>>> for docker/lxc guests [2]?
>>>>
>>>> Cheers,
>>>>
>>>> Chris
>>>>
>>>> ---
>>>> [1]
>>>> http://blog.docker.io/2013/10/gathering-lxc-docker-containers-metrics/
>>>> [2
>>>> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Auto-scaling+Guide#id-4.0.0Auto-scalingGuide-memoryConsumptio
>>>>
>>>> On Mon, May 26, 2014 at 1:24 PM, Lahiru Sandaruwan <la...@wso2.com>
>>>> wrote:
>>>> > Hi Chis.
>>>> >
>>>> >
>>>> > On Mon, May 26, 2014 at 5:27 PM, chris snow <ch...@gmail.com>
>>>> > wrote:
>>>> >>
>>>> >> Hi Guys,
>>>> >>
>>>> >> I noticed on my stratos+openstack+docker installation that stratos
>>>> >> was
>>>> >> getting into a cycle trying to spawn new instances.
>>>> >>
>>>> >> I think this is because load average (and probably memory
>>>> >> consumption)
>>>> >> on lxc/docker instances is the same value as for the host [1].
>>>> >>
>>>> >> Question: Am I correct in thinking that load average and memory
>>>> >> consumption autoscaling parameters aren't a good idea for docker
>>>> >> guests?
>>>> >
>>>> >
>>>> > Is there a way to find out the load average and memory consumption of
>>>> > the
>>>> > guest?
>>>> >
>>>> > I think the proper solution is to customize/fix Cartridge agent find
>>>> > out
>>>> > relevant values in these type of environments and publish stats to
>>>> > CEP.
>>>> >
>>>> > Thanks.
>>>> >>
>>>> >>
>>>> >> Many thanks,
>>>> >>
>>>> >> Chris
>>>> >>
>>>> >> ---
>>>> >> [1]
>>>> >>
>>>> >> http://unix.stackexchange.com/questions/132021/is-a-lxc-docker-guest-load-average-the-same-as-the-hosts-load-average
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > --
>>>> > Lahiru Sandaruwan
>>>> > Committer and PMC member, Apache Stratos,
>>>> > Senior Software Engineer,
>>>> > WSO2 Inc., http://wso2.com
>>>> > lean.enterprise.middleware
>>>> >
>>>> > email: lahirus@wso2.com cell: (+94) 773 325 954
>>>> > blog: http://lahiruwrites.blogspot.com/
>>>> > twitter: http://twitter.com/lahirus
>>>> > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Check out my professional profile and connect with me on LinkedIn.
>>>> http://lnkd.in/cw5k69
>>>
>>>
>>>
>>>
>>> --
>>> Imesh Gunaratne
>>>
>>> Technical Lead, WSO2
>>> Committer & PPMC Member, Apache Stratos
>
>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PPMC Member, Apache Stratos



-- 
Check out my professional profile and connect with me on LinkedIn.
http://lnkd.in/cw5k69

Re: autoscaling - load average and memory consumption on docker/lxc instances

Posted by Imesh Gunaratne <im...@apache.org>.
Hi Chris,

Thanks for the quick response.
Right, I still don't get why we cannot adjust the threshold values and fix
it.

Thanks


On Wed, May 28, 2014 at 11:37 AM, chris snow <ch...@gmail.com> wrote:

> Hi Imesh,
>
> I haven't looked into this in any more detail. All I have done is removed
> those parameters from my auto scaling policy.
>
> Many thanks,
> Chris
> On 28 May 2014 05:58, "Imesh Gunaratne" <im...@apache.org> wrote:
>
>> Hi Chris,
>>
>> Were you able to check the memory consumption and load average values
>> sent to Autoscaler by any chance?
>>
>> Thanks
>>
>>
>> On Mon, May 26, 2014 at 6:31 PM, chris snow <ch...@gmail.com> wrote:
>>
>>> Hi Lahiru,
>>>
>>> It looks like there's some good info here [1].  However, as a first
>>> step, I was wondering whether we should add a note to the wiki to
>>> recommend not using the memory consumption or load average parameters
>>> for docker/lxc guests [2]?
>>>
>>> Cheers,
>>>
>>> Chris
>>>
>>> ---
>>> [1]
>>> http://blog.docker.io/2013/10/gathering-lxc-docker-containers-metrics/
>>> [2
>>> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Auto-scaling+Guide#id-4.0.0Auto-scalingGuide-memoryConsumptio
>>>
>>> On Mon, May 26, 2014 at 1:24 PM, Lahiru Sandaruwan <la...@wso2.com>
>>> wrote:
>>> > Hi Chis.
>>> >
>>> >
>>> > On Mon, May 26, 2014 at 5:27 PM, chris snow <ch...@gmail.com>
>>> wrote:
>>> >>
>>> >> Hi Guys,
>>> >>
>>> >> I noticed on my stratos+openstack+docker installation that stratos was
>>> >> getting into a cycle trying to spawn new instances.
>>> >>
>>> >> I think this is because load average (and probably memory consumption)
>>> >> on lxc/docker instances is the same value as for the host [1].
>>> >>
>>> >> Question: Am I correct in thinking that load average and memory
>>> >> consumption autoscaling parameters aren't a good idea for docker
>>> >> guests?
>>> >
>>> >
>>> > Is there a way to find out the load average and memory consumption of
>>> the
>>> > guest?
>>> >
>>> > I think the proper solution is to customize/fix Cartridge agent find
>>> out
>>> > relevant values in these type of environments and publish stats to CEP.
>>> >
>>> > Thanks.
>>> >>
>>> >>
>>> >> Many thanks,
>>> >>
>>> >> Chris
>>> >>
>>> >> ---
>>> >> [1]
>>> >>
>>> http://unix.stackexchange.com/questions/132021/is-a-lxc-docker-guest-load-average-the-same-as-the-hosts-load-average
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > --
>>> > Lahiru Sandaruwan
>>> > Committer and PMC member, Apache Stratos,
>>> > Senior Software Engineer,
>>> > WSO2 Inc., http://wso2.com
>>> > lean.enterprise.middleware
>>> >
>>> > email: lahirus@wso2.com cell: (+94) 773 325 954
>>> > blog: http://lahiruwrites.blogspot.com/
>>> > twitter: http://twitter.com/lahirus
>>> > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>> >
>>>
>>>
>>>
>>> --
>>> Check out my professional profile and connect with me on LinkedIn.
>>> http://lnkd.in/cw5k69
>>>
>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PPMC Member, Apache Stratos
>>
>


-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PPMC Member, Apache Stratos

Re: autoscaling - load average and memory consumption on docker/lxc instances

Posted by chris snow <ch...@gmail.com>.
Hi Imesh,

I haven't looked into this in any more detail. All I have done is removed
those parameters from my auto scaling policy.

Many thanks,
Chris
On 28 May 2014 05:58, "Imesh Gunaratne" <im...@apache.org> wrote:

> Hi Chris,
>
> Were you able to check the memory consumption and load average values sent
> to Autoscaler by any chance?
>
> Thanks
>
>
> On Mon, May 26, 2014 at 6:31 PM, chris snow <ch...@gmail.com> wrote:
>
>> Hi Lahiru,
>>
>> It looks like there's some good info here [1].  However, as a first
>> step, I was wondering whether we should add a note to the wiki to
>> recommend not using the memory consumption or load average parameters
>> for docker/lxc guests [2]?
>>
>> Cheers,
>>
>> Chris
>>
>> ---
>> [1]
>> http://blog.docker.io/2013/10/gathering-lxc-docker-containers-metrics/
>> [2
>> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Auto-scaling+Guide#id-4.0.0Auto-scalingGuide-memoryConsumptio
>>
>> On Mon, May 26, 2014 at 1:24 PM, Lahiru Sandaruwan <la...@wso2.com>
>> wrote:
>> > Hi Chis.
>> >
>> >
>> > On Mon, May 26, 2014 at 5:27 PM, chris snow <ch...@gmail.com>
>> wrote:
>> >>
>> >> Hi Guys,
>> >>
>> >> I noticed on my stratos+openstack+docker installation that stratos was
>> >> getting into a cycle trying to spawn new instances.
>> >>
>> >> I think this is because load average (and probably memory consumption)
>> >> on lxc/docker instances is the same value as for the host [1].
>> >>
>> >> Question: Am I correct in thinking that load average and memory
>> >> consumption autoscaling parameters aren't a good idea for docker
>> >> guests?
>> >
>> >
>> > Is there a way to find out the load average and memory consumption of
>> the
>> > guest?
>> >
>> > I think the proper solution is to customize/fix Cartridge agent find out
>> > relevant values in these type of environments and publish stats to CEP.
>> >
>> > Thanks.
>> >>
>> >>
>> >> Many thanks,
>> >>
>> >> Chris
>> >>
>> >> ---
>> >> [1]
>> >>
>> http://unix.stackexchange.com/questions/132021/is-a-lxc-docker-guest-load-average-the-same-as-the-hosts-load-average
>> >
>> >
>> >
>> >
>> > --
>> > --
>> > Lahiru Sandaruwan
>> > Committer and PMC member, Apache Stratos,
>> > Senior Software Engineer,
>> > WSO2 Inc., http://wso2.com
>> > lean.enterprise.middleware
>> >
>> > email: lahirus@wso2.com cell: (+94) 773 325 954
>> > blog: http://lahiruwrites.blogspot.com/
>> > twitter: http://twitter.com/lahirus
>> > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>> >
>>
>>
>>
>> --
>> Check out my professional profile and connect with me on LinkedIn.
>> http://lnkd.in/cw5k69
>>
>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PPMC Member, Apache Stratos
>

Re: autoscaling - load average and memory consumption on docker/lxc instances

Posted by Imesh Gunaratne <im...@apache.org>.
Hi Chris,

Were you able to check the memory consumption and load average values sent
to Autoscaler by any chance?

Thanks


On Mon, May 26, 2014 at 6:31 PM, chris snow <ch...@gmail.com> wrote:

> Hi Lahiru,
>
> It looks like there's some good info here [1].  However, as a first
> step, I was wondering whether we should add a note to the wiki to
> recommend not using the memory consumption or load average parameters
> for docker/lxc guests [2]?
>
> Cheers,
>
> Chris
>
> ---
> [1] http://blog.docker.io/2013/10/gathering-lxc-docker-containers-metrics/
> [2
> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Auto-scaling+Guide#id-4.0.0Auto-scalingGuide-memoryConsumptio
>
> On Mon, May 26, 2014 at 1:24 PM, Lahiru Sandaruwan <la...@wso2.com>
> wrote:
> > Hi Chis.
> >
> >
> > On Mon, May 26, 2014 at 5:27 PM, chris snow <ch...@gmail.com> wrote:
> >>
> >> Hi Guys,
> >>
> >> I noticed on my stratos+openstack+docker installation that stratos was
> >> getting into a cycle trying to spawn new instances.
> >>
> >> I think this is because load average (and probably memory consumption)
> >> on lxc/docker instances is the same value as for the host [1].
> >>
> >> Question: Am I correct in thinking that load average and memory
> >> consumption autoscaling parameters aren't a good idea for docker
> >> guests?
> >
> >
> > Is there a way to find out the load average and memory consumption of the
> > guest?
> >
> > I think the proper solution is to customize/fix Cartridge agent find out
> > relevant values in these type of environments and publish stats to CEP.
> >
> > Thanks.
> >>
> >>
> >> Many thanks,
> >>
> >> Chris
> >>
> >> ---
> >> [1]
> >>
> http://unix.stackexchange.com/questions/132021/is-a-lxc-docker-guest-load-average-the-same-as-the-hosts-load-average
> >
> >
> >
> >
> > --
> > --
> > Lahiru Sandaruwan
> > Committer and PMC member, Apache Stratos,
> > Senior Software Engineer,
> > WSO2 Inc., http://wso2.com
> > lean.enterprise.middleware
> >
> > email: lahirus@wso2.com cell: (+94) 773 325 954
> > blog: http://lahiruwrites.blogspot.com/
> > twitter: http://twitter.com/lahirus
> > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
> >
>
>
>
> --
> Check out my professional profile and connect with me on LinkedIn.
> http://lnkd.in/cw5k69
>



-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PPMC Member, Apache Stratos

Re: autoscaling - load average and memory consumption on docker/lxc instances

Posted by chris snow <ch...@gmail.com>.
Hi Lahiru,

It looks like there's some good info here [1].  However, as a first
step, I was wondering whether we should add a note to the wiki to
recommend not using the memory consumption or load average parameters
for docker/lxc guests [2]?

Cheers,

Chris

---
[1] http://blog.docker.io/2013/10/gathering-lxc-docker-containers-metrics/
[2 https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Auto-scaling+Guide#id-4.0.0Auto-scalingGuide-memoryConsumptio

On Mon, May 26, 2014 at 1:24 PM, Lahiru Sandaruwan <la...@wso2.com> wrote:
> Hi Chis.
>
>
> On Mon, May 26, 2014 at 5:27 PM, chris snow <ch...@gmail.com> wrote:
>>
>> Hi Guys,
>>
>> I noticed on my stratos+openstack+docker installation that stratos was
>> getting into a cycle trying to spawn new instances.
>>
>> I think this is because load average (and probably memory consumption)
>> on lxc/docker instances is the same value as for the host [1].
>>
>> Question: Am I correct in thinking that load average and memory
>> consumption autoscaling parameters aren't a good idea for docker
>> guests?
>
>
> Is there a way to find out the load average and memory consumption of the
> guest?
>
> I think the proper solution is to customize/fix Cartridge agent find out
> relevant values in these type of environments and publish stats to CEP.
>
> Thanks.
>>
>>
>> Many thanks,
>>
>> Chris
>>
>> ---
>> [1]
>> http://unix.stackexchange.com/questions/132021/is-a-lxc-docker-guest-load-average-the-same-as-the-hosts-load-average
>
>
>
>
> --
> --
> Lahiru Sandaruwan
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
> lean.enterprise.middleware
>
> email: lahirus@wso2.com cell: (+94) 773 325 954
> blog: http://lahiruwrites.blogspot.com/
> twitter: http://twitter.com/lahirus
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>



-- 
Check out my professional profile and connect with me on LinkedIn.
http://lnkd.in/cw5k69

Re: autoscaling - load average and memory consumption on docker/lxc instances

Posted by Lahiru Sandaruwan <la...@wso2.com>.
Hi Chis.


On Mon, May 26, 2014 at 5:27 PM, chris snow <ch...@gmail.com> wrote:

> Hi Guys,
>
> I noticed on my stratos+openstack+docker installation that stratos was
> getting into a cycle trying to spawn new instances.
>
> I think this is because load average (and probably memory consumption)
> on lxc/docker instances is the same value as for the host [1].
>
> Question: Am I correct in thinking that load average and memory
> consumption autoscaling parameters aren't a good idea for docker
> guests?
>

Is there a way to find out the load average and memory consumption of the
guest?

I think the proper solution is to customize/fix Cartridge agent find out
relevant values in these type of environments and publish stats to CEP.

Thanks.

>
> Many thanks,
>
> Chris
>
> ---
> [1]
> http://unix.stackexchange.com/questions/132021/is-a-lxc-docker-guest-load-average-the-same-as-the-hosts-load-average
>



-- 
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware

email: lahirus@wso2.com cell: (+94) 773 325 954
blog: http://lahiruwrites.blogspot.com/
twitter: http://twitter.com/lahirus
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146