[jira] [Commented] (BROOKLYN-394) "Request limit exceeded" on Amazon

["Aled Sage (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BROOKLYN-394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15681595#comment-15681595 ] 

Aled Sage commented on BROOKLYN-394:
------------------------------------

This is at one level surprising because jclouds does automatic retry with exponential backoff. However, I guess if we're doing too many calls then even with the backoff we might still get it repeatedly rejected.

My understanding is that AWS will rate-limit based on the nature (as well as number) of API calls. For example, if creating/modifying security groups is a more exensive operation (from AWS's perspective) than a simple poll for a machine's state, then those requests would cause rate-limiting sooner.

Your list sounds good. A couple of other suggestions spring to mind:

1. We could also tweak {{jclouds.retries-delay-start}} and {{jclouds.max-retries}}, which default to 50ms and 5 respectively. The max period between retries is {{10 * delayStart}}, so would just be 500ms; the time increases exponentially in powers of 2. See https://github.com/jclouds/jclouds/blob/master/core/src/main/java/org/jclouds/http/handlers/BackoffLimitedRetryHandler.java#L116-L122

2. We should try to be smart about how we open ports in a security group, to minimise the number of API calls. We should review the jclouds code more closely to see how it behaves when using the standard jclouds {{inboundPorts: ...}} - including looking at the wire log for what actually happens. It looks like it does make a single API call for aws-ec2 (see https://github.com/jclouds/jclouds/blob/rel/jclouds-1.9.2/providers/aws-ec2/src/main/java/org/jclouds/aws/ec2/compute/loaders/AWSEC2CreateSecurityGroupIfNeeded.java#L89-L128); it also looks at first blush that the use of {{RangeSet}} will squash together contiguous ports into a minimum number of ranges.

Option (1) is probably worth adding to Brooklyn asap to override the default (e.g. {{jclouds.retries-delay-start=500}}).

> "Request limit exceeded" on Amazon
> ----------------------------------
>
>                 Key: BROOKLYN-394
>                 URL: https://issues.apache.org/jira/browse/BROOKLYN-394
>             Project: Brooklyn
>          Issue Type: Bug
>            Reporter: Svetoslav Neykov
>
> Any moderately sized blueprint could trigger {{Request limit exceeded}} on Amazon (say kubernetes). The only way users have control over the request rate is by setting {{maxConcurrentMachineCreations}} with the current recommended value of 3 (see clocker.io).
> It's bad user experience if one needs to adapt the location based on the blueprint.
> Possible steps to improve:
> * Add to troubleshooting documentation
> * Make maxConcurrentMachineCreations default to 3
> * Check are we polling for machine creation too often.
> * Check how many requests are we hitting Amazon with (per created machine)
> * The number of requests per machine could vary from blueprint to blueprint (say if the blueprint is creating security networks, using other amazon services). Is there a way to throttle our requests to amazon and stay below a certain limit per second?
> * I've hit the error during machine tear down as well, so {{maxConcurrentMachineCreations}} is not enough to work around
> Some docs on rate limits at http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html.
> Related: https://github.com/jclouds/legacy-jclouds/issues/1214



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)