You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Francois Gaudreault <fg...@cloudops.com> on 2013/07/03 16:14:38 UTC
Re: Auto-scaling won't scale down
I did some more tests around this, and the NetScaler does trigger the
scale down, but it looks like CS never reacts.
Netscaler did:
Autoscale: Evaluating autoscale policy
Cloud-AutoScale-Policy-xxxxx-80-12Autoscale: Triggering autoscale
action: SCALE_DOWN for policy:
Cloud-AutoScale-ScaleDownAction-xxxxx-80Autoscale: :::925###POST
/provision HTTP/1.1
Host: NS
Content-Length: 866
{ "autoscale" : { "ns" : { "sysid" : "98310000cb254307ee78" }, "vserver"
: { "name" : "Cloud-VirtualServer-xxxxx-80", "servicegroupname" :
"Cloudf84eaf4ed3fe43aebbc4d2ff9f7311bc" }, "cloudprofile" : { "type" :
"CLOUDSTACK", "url" : "http://172.30.100.15:8080/client/api", "apiKey" :
"x2134OIL1TKNyW6PahIq4NubxMn1tK3krpQFyDWks8fv_[...]", "sharedsecret" :
"HtpqjvHENGMqZLBcn_o-_[...]" }, "action" : { "type" : "SCALE_DOWN",
"params" :
"command=destroyVirtualMachine&lbruleid=24aab38f-f466-45c5-9878-484841ba1b9d",
"quiettime" : "300", "destroyVMGracePeriod" : "30" }, "autoprovisiondns"
: { "nsdomainname" :
"autoscale-internal_server_Cloudf84eaf4ed3fe43aebbc4d2ff9f7311bc",
"nsservername" :
"autoscale-internal_server_Cloudf84eaf4ed3fe43aebbc4d2ff9f7311bc" } } }:::
current time is Tue Jul 2 21:09:09 2013
Autoscale: Timer Event: Cloud-AutoScale-Timer-xxxxx-80
Returning avg_val: 67
If I look in CloudStack, I do see the deployVirtualMachine calls coming
from the NetScaler during the scale up, but I don't see any
destroyVirtualMachine hitting the server for the scale down.
I also added this issue regarding autoscaling:
https://issues.apache.org/jira/browse/CLOUDSTACK-3330
I am not sure if this is an expected behavior or not. Apparently, if the
VR is not setup, and you create the autoscale rules, nothing will happen
until you spin at least one vm in the network.
Thanks!
On 2013-06-27 7:35 PM, Sowmya Krishnan wrote:
> Configurations look ok.
>
> Could you check to see if there are destroyVirtualMachine API calls in the management server logs? If not then the condition for Scale down is probably not getting hit.
> To find out if the condition is being hit in Netscaler, run "nsconmsg" in the shell. When scale down is triggered, you should see logs something like:
>
> Evaluating autoscale policy.. : Triggering autoscale action SCALE_DOWN for action ...
>
>
>> -----Original Message-----
>> From: Francois Gaudreault [mailto:fgaudreault@cloudops.com]
>> Sent: Friday, June 28, 2013 1:44 AM
>> To: dev@cloudstack.apache.org
>> Subject: Auto-scaling won't scale down
>>
>> Hi,
>>
>> So I tried the AutoScaling on our setup today, and while it's working fine to scale
>> up, it won't scale down. We are on CS 4.1.0 nonoss with NetScaler 10.1 build
>> 112.15.
>>
>> I configured a Scale Up policy to trigger when servers are under 30% CPU idle for
>> a duration of 60sec, and Scale Down policy to trigger when servers are over 40%
>> CPU idle for a duration of 120secs. I configured the AutoScaling to have a
>> minimum of 1 machine, and a maximum of 3.
>>
>> I do see the SNMP calls coming from the NetScaler, and the response.
>>
>> Did I configured it wrong? Let me know what you need for troubleshooting.
>>
>> Thanks!
>>
>> --
>> Francois Gaudreault
>> Architecte de Solution Cloud | Cloud Solutions Architect
>> fgaudreault@cloudops.com
>> 514-629-6775
>> - - -
>> CloudOps
>> 420 rue Guy
>> Montréal QC H3J 1S6
>> www.cloudops.com
>> @CloudOps_
>
>
--
Francois Gaudreault
Architecte de Solution Cloud | Cloud Solutions Architect
fgaudreault@cloudops.com
514-629-6775
- - -
CloudOps
420 rue Guy
Montréal QC H3J 1S6
www.cloudops.com
@CloudOps_
RE: Auto-scaling won't scale down
Posted by Ram Ganesh <Ra...@citrix.com>.
> -----Original Message-----
> From: Francois Gaudreault [mailto:fgaudreault@cloudops.com]
> Sent: 03 July 2013 19:46
> To: dev@cloudstack.apache.org
> Subject: Re: Auto-scaling won't scale down
>
> I did some more tests around this, and the NetScaler does trigger the scale
> down, but it looks like CS never reacts.
>
> Netscaler did:
> Autoscale: Evaluating autoscale policy
> Cloud-AutoScale-Policy-xxxxx-80-12Autoscale: Triggering autoscale
> action: SCALE_DOWN for policy:
> Cloud-AutoScale-ScaleDownAction-xxxxx-80Autoscale: :::925###POST
> /provision HTTP/1.1
> Host: NS
> Content-Length: 866
>
> { "autoscale" : { "ns" : { "sysid" : "98310000cb254307ee78" }, "vserver"
> : { "name" : "Cloud-VirtualServer-xxxxx-80", "servicegroupname" :
> "Cloudf84eaf4ed3fe43aebbc4d2ff9f7311bc" }, "cloudprofile" : { "type" :
> "CLOUDSTACK", "url" : "http://172.30.100.15:8080/client/api", "apiKey" :
> "x2134OIL1TKNyW6PahIq4NubxMn1tK3krpQFyDWks8fv_[...]",
> "sharedsecret" :
> "HtpqjvHENGMqZLBcn_o-_[...]" }, "action" : { "type" : "SCALE_DOWN",
> "params" :
> "command=destroyVirtualMachine&lbruleid=24aab38f-f466-45c5-9878-
> 484841ba1b9d",
> "quiettime" : "300", "destroyVMGracePeriod" : "30" }, "autoprovisiondns"
> : { "nsdomainname" :
> "autoscale-internal_server_Cloudf84eaf4ed3fe43aebbc4d2ff9f7311bc",
> "nsservername" :
> "autoscale-internal_server_Cloudf84eaf4ed3fe43aebbc4d2ff9f7311bc" } } }:::
> current time is Tue Jul 2 21:09:09 2013
> Autoscale: Timer Event: Cloud-AutoScale-Timer-xxxxx-80 Returning avg_val:
> 67
>
> If I look in CloudStack, I do see the deployVirtualMachine calls coming from
> the NetScaler during the scale up, but I don't see any destroyVirtualMachine
> hitting the server for the scale down.
>
> I also added this issue regarding autoscaling:
> https://issues.apache.org/jira/browse/CLOUDSTACK-3330
>
> I am not sure if this is an expected behavior or not. Apparently, if the VR is
> not setup, and you create the autoscale rules, nothing will happen until you
> spin at least one vm in the network.
The guest network should be in implemented state for autoscale to work. Or you can define a Persistent network (https://cwiki.apache.org/confluence/display/CLOUDSTACK/FS+-+Persistent+Networks) and then configure autoscale without a need for a guest VM to be deployed manually.
>
> Thanks!
>
> On 2013-06-27 7:35 PM, Sowmya Krishnan wrote:
> > Configurations look ok.
> >
> > Could you check to see if there are destroyVirtualMachine API calls in the
> management server logs? If not then the condition for Scale down is
> probably not getting hit.
> > To find out if the condition is being hit in Netscaler, run "nsconmsg" in the
> shell. When scale down is triggered, you should see logs something like:
> >
> > Evaluating autoscale policy.. : Triggering autoscale action SCALE_DOWN for
> action ...
> >
> >
> >> -----Original Message-----
> >> From: Francois Gaudreault [mailto:fgaudreault@cloudops.com]
> >> Sent: Friday, June 28, 2013 1:44 AM
> >> To: dev@cloudstack.apache.org
> >> Subject: Auto-scaling won't scale down
> >>
> >> Hi,
> >>
> >> So I tried the AutoScaling on our setup today, and while it's working
> >> fine to scale up, it won't scale down. We are on CS 4.1.0 nonoss
> >> with NetScaler 10.1 build 112.15.
> >>
> >> I configured a Scale Up policy to trigger when servers are under 30%
> >> CPU idle for a duration of 60sec, and Scale Down policy to trigger
> >> when servers are over 40% CPU idle for a duration of 120secs. I
> >> configured the AutoScaling to have a minimum of 1 machine, and a
> maximum of 3.
> >>
> >> I do see the SNMP calls coming from the NetScaler, and the response.
> >>
> >> Did I configured it wrong? Let me know what you need for
> troubleshooting.
> >>
> >> Thanks!
> >>
> >> --
> >> Francois Gaudreault
> >> Architecte de Solution Cloud | Cloud Solutions Architect
> >> fgaudreault@cloudops.com
> >> 514-629-6775
> >> - - -
> >> CloudOps
> >> 420 rue Guy
> >> Montréal QC H3J 1S6
> >> www.cloudops.com
> >> @CloudOps_
> >
> >
>
>
> --
> Francois Gaudreault
> Architecte de Solution Cloud | Cloud Solutions Architect
> fgaudreault@cloudops.com
> 514-629-6775
> - - -
> CloudOps
> 420 rue Guy
> Montréal QC H3J 1S6
> www.cloudops.com
> @CloudOps_
Re: Auto-scaling won't scale down
Posted by Francois Gaudreault <fg...@cloudops.com>.
Since the discussion diverted on another related issue, this one still
need to be investigated :)
Francois
On 2013-07-03 10:14 AM, Francois Gaudreault wrote:
> I did some more tests around this, and the NetScaler does trigger the
> scale down, but it looks like CS never reacts.
>
> Netscaler did:
> Autoscale: Evaluating autoscale policy
> Cloud-AutoScale-Policy-xxxxx-80-12Autoscale: Triggering autoscale
> action: SCALE_DOWN for policy:
> Cloud-AutoScale-ScaleDownAction-xxxxx-80Autoscale: :::925###POST
> /provision HTTP/1.1
> Host: NS
> Content-Length: 866
>
> { "autoscale" : { "ns" : { "sysid" : "98310000cb254307ee78" },
> "vserver" : { "name" : "Cloud-VirtualServer-xxxxx-80",
> "servicegroupname" : "Cloudf84eaf4ed3fe43aebbc4d2ff9f7311bc" },
> "cloudprofile" : { "type" : "CLOUDSTACK", "url" :
> "http://172.30.100.15:8080/client/api", "apiKey" :
> "x2134OIL1TKNyW6PahIq4NubxMn1tK3krpQFyDWks8fv_[...]", "sharedsecret" :
> "HtpqjvHENGMqZLBcn_o-_[...]" }, "action" : { "type" : "SCALE_DOWN",
> "params" :
> "command=destroyVirtualMachine&lbruleid=24aab38f-f466-45c5-9878-484841ba1b9d",
> "quiettime" : "300", "destroyVMGracePeriod" : "30" },
> "autoprovisiondns" : { "nsdomainname" :
> "autoscale-internal_server_Cloudf84eaf4ed3fe43aebbc4d2ff9f7311bc",
> "nsservername" :
> "autoscale-internal_server_Cloudf84eaf4ed3fe43aebbc4d2ff9f7311bc" } }
> }:::
> current time is Tue Jul 2 21:09:09 2013
> Autoscale: Timer Event: Cloud-AutoScale-Timer-xxxxx-80
> Returning avg_val: 67
>
> If I look in CloudStack, I do see the deployVirtualMachine calls
> coming from the NetScaler during the scale up, but I don't see any
> destroyVirtualMachine hitting the server for the scale down.
--
Francois Gaudreault
Architecte de Solution Cloud | Cloud Solutions Architect
fgaudreault@cloudops.com
514-629-6775
- - -
CloudOps
420 rue Guy
Montréal QC H3J 1S6
www.cloudops.com
@CloudOps_
RE: Auto-scaling won't scale down
Posted by Ram Ganesh <Ra...@citrix.com>.
> -----Original Message-----
> From: Chip Childers [mailto:chip.childers@sungard.com]
> Sent: 03 July 2013 20:13
> To: <de...@cloudstack.apache.org>
> Cc: fgaudreault@cloudops.com
> Subject: Re: Auto-scaling won't scale down
>
> On Wed, Jul 3, 2013 at 10:39 AM, Ram Ganesh <Ra...@citrix.com>
> wrote:
> >> -----Original Message-----
> >> From: Chip Childers [mailto:chip.childers@sungard.com]
> >> Sent: 03 July 2013 19:58
> >> To: dev@cloudstack.apache.org; fgaudreault@cloudops.com
> >> Subject: Re: Auto-scaling won't scale down
> >>
> >> On Wed, Jul 03, 2013 at 10:14:38AM -0400, Francois Gaudreault wrote:
> >> > I am not sure if this is an expected behavior or not. Apparently,
> >> > if the VR is not setup, and you create the autoscale rules, nothing
> >> > will happen until you spin at least one vm in the network.
> >>
> >> The reason makes sense technically, since you have 0 usage for the
> >> environment at that point. Until there is actual usage, it will
> >> never trigger a threshold that would cause a scaling event.
> >>
> >> Looking at this from a user's perspective, I actually don't see an
> >> issue with this implementation. Does anyone know how AWS reacts in
> this scenario?
> >
> > Chip,
> >
> > One can configure minMembers while defining an autoscale rule in which
> case CS will provision minMember of guestVMs even when there is no
> traffic. It should be true in case of AWS also.
> >
> > Thanks,
> > RamG
> >
> >
>
> Sounds like a valid bug then, based on Francois not seeing it behave this way.
> ;-)
Don't think so. In case of Francois the network was not in implemented state. If I recollect correctly CloudStack UI does not allow one to configure an LB rule when a guest network is not in implemented state. Better would be to use PersistentNetwork feature of 4.2 to configure one(autoscale) without the need for deploying a guest VM
Re: Auto-scaling won't scale down
Posted by Chip Childers <ch...@sungard.com>.
On Wed, Jul 3, 2013 at 10:39 AM, Ram Ganesh <Ra...@citrix.com> wrote:
>> -----Original Message-----
>> From: Chip Childers [mailto:chip.childers@sungard.com]
>> Sent: 03 July 2013 19:58
>> To: dev@cloudstack.apache.org; fgaudreault@cloudops.com
>> Subject: Re: Auto-scaling won't scale down
>>
>> On Wed, Jul 03, 2013 at 10:14:38AM -0400, Francois Gaudreault wrote:
>> > I am not sure if this is an expected behavior or not. Apparently, if
>> > the VR is not setup, and you create the autoscale rules, nothing will
>> > happen until you spin at least one vm in the network.
>>
>> The reason makes sense technically, since you have 0 usage for the
>> environment at that point. Until there is actual usage, it will never trigger a
>> threshold that would cause a scaling event.
>>
>> Looking at this from a user's perspective, I actually don't see an issue with
>> this implementation. Does anyone know how AWS reacts in this scenario?
>
> Chip,
>
> One can configure minMembers while defining an autoscale rule in which case CS will provision minMember of guestVMs even when there is no traffic. It should be true in case of AWS also.
>
> Thanks,
> RamG
>
>
Sounds like a valid bug then, based on Francois not seeing it behave
this way. ;-)
RE: Auto-scaling won't scale down
Posted by Ram Ganesh <Ra...@citrix.com>.
> -----Original Message-----
> From: Chip Childers [mailto:chip.childers@sungard.com]
> Sent: 03 July 2013 19:58
> To: dev@cloudstack.apache.org; fgaudreault@cloudops.com
> Subject: Re: Auto-scaling won't scale down
>
> On Wed, Jul 03, 2013 at 10:14:38AM -0400, Francois Gaudreault wrote:
> > I am not sure if this is an expected behavior or not. Apparently, if
> > the VR is not setup, and you create the autoscale rules, nothing will
> > happen until you spin at least one vm in the network.
>
> The reason makes sense technically, since you have 0 usage for the
> environment at that point. Until there is actual usage, it will never trigger a
> threshold that would cause a scaling event.
>
> Looking at this from a user's perspective, I actually don't see an issue with
> this implementation. Does anyone know how AWS reacts in this scenario?
Chip,
One can configure minMembers while defining an autoscale rule in which case CS will provision minMember of guestVMs even when there is no traffic. It should be true in case of AWS also.
Thanks,
RamG
Re: Auto-scaling won't scale down
Posted by Francois Gaudreault <fg...@cloudops.com>.
Chip,
> The reason makes sense technically, since you have 0 usage for the
> environment at that point. Until there is actual usage, it will never
> trigger a threshold that would cause a scaling event.
>
> Looking at this from a user's perspective, I actually don't see an issue
> with this implementation. Does anyone know how AWS reacts in this
> scenario?
Fair enough, but I would assume that if you configure the autoscaling
with a minimum instance (say 1), CS should do all the necessary steps to
be able to spin that minimum. So if the network is not in an
implemented state, CS should make the necessary to put it into an
implemented state. That's my thoughts as a user ;)
--
Francois Gaudreault
Architecte de Solution Cloud | Cloud Solutions Architect
fgaudreault@cloudops.com
514-629-6775
- - -
CloudOps
420 rue Guy
Montréal QC H3J 1S6
www.cloudops.com
@CloudOps_
Re: Auto-scaling won't scale down
Posted by Chip Childers <ch...@sungard.com>.
On Wed, Jul 03, 2013 at 10:14:38AM -0400, Francois Gaudreault wrote:
> I am not sure if this is an expected behavior or not. Apparently, if
> the VR is not setup, and you create the autoscale rules, nothing
> will happen until you spin at least one vm in the network.
The reason makes sense technically, since you have 0 usage for the
environment at that point. Until there is actual usage, it will never
trigger a threshold that would cause a scaling event.
Looking at this from a user's perspective, I actually don't see an issue
with this implementation. Does anyone know how AWS reacts in this
scenario?