An old challenge: bulk queries for node polling

[Andrew Bayer <an...@gmail.com>]

So I'm trying to figure out how to make it possible to provision 100+
instances on EC2 in one createNodesInGroup call without getting crushed by
API rate limits. The combination of the min count/max count parameters and
the idempotency with a client token (see
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Run_Instance_Idempotency.html)
makes the raw creation of 100+ instances without a thundering herd of
RunInstances calls very realistic, but then we hit the issue of polling
those nodes (even with the idempotency, we'd still end up having at least
one is-node-running polling call per node, and then we'd hit the same thing
but much bigger at terminate/pause/resume/etc)...

So the problem I'm trying to tackle is bigger than just EC2, I think. A
decent number of provider APIs allow querying for more than one node in a
single DescribeInstances-equivalent call, and I've done some work in the
past that allowed for switching over ComputeService.listNodesByIds(...) to
use those approaches when possible. But polling is very much about doing
one call per node, i.e. in
https://github.com/jclouds/jclouds/blob/master/compute/src/main/java/org/jclouds/compute/predicates/internal/RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java?source=c-
I'm honestly completely unclear as to what the best approach would be
to
switch this to using a cached/periodically refreshed list instead.

Does anyone have any thoughts on this before I just start beating my head
against it?

A.

Re: An old challenge: bulk queries for node polling

[Andrew Bayer <an...@gmail.com>]

That's the best workaround I've got at this point, yeah, but I'd really
like to find a good way of doing bulk polling - I don't understand
LoadingCache/CacheLoader well enough, but was thinking maybe something
along the lines of
having RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid check a cache,
where that cache expires entries after the polling period has passed, and
when the cache expires (or it needs to load for the first time), it can do
a bulk get nodes query (if available) for all the nodes it needs to know
about...but I'm not sure how to be sure it knows all the relevant nodes,
how to keep it from slamming with a bunch of bulk queries if you're
creating nodes consecutively, etc...

I'll play with this, in the context of JCLOUDS-450 (which I just opened)...

A.


On Tue, Feb 4, 2014 at 6:46 AM, Everett Toews
<ev...@rackspace.com>wrote:

> On Feb 3, 2014, at 6:28 PM, Andrew Bayer <an...@gmail.com> wrote:
>
> > So I'm trying to figure out how to make it possible to provision 100+
> > instances on EC2 in one createNodesInGroup call without getting crushed
> by
> > API rate limits. The combination of the min count/max count parameters
> and
> > the idempotency with a client token (see
> >
> http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Run_Instance_Idempotency.html
> )
> > makes the raw creation of 100+ instances without a thundering herd of
> > RunInstances calls very realistic, but then we hit the issue of polling
> > those nodes (even with the idempotency, we'd still end up having at least
> > one is-node-running polling call per node, and then we'd hit the same
> thing
> > but much bigger at terminate/pause/resume/etc)...
>
> The way I've dealt with this in the past is by using POLL_INITIAL_PERIOD
> and POLL_MAX_PERIOD [1].
>
> Does that work for your use case?
>
> Everett
>
> [1]
> https://github.com/jclouds/jclouds-examples/blob/master/rackspace/src/main/java/org/jclouds/examples/rackspace/cloudservers/CloudServersPublish.java#L102-103

Re: An old challenge: bulk queries for node polling

[Everett Toews <ev...@RACKSPACE.COM>]

On Feb 3, 2014, at 6:28 PM, Andrew Bayer <an...@gmail.com> wrote:

> So I'm trying to figure out how to make it possible to provision 100+
> instances on EC2 in one createNodesInGroup call without getting crushed by
> API rate limits. The combination of the min count/max count parameters and
> the idempotency with a client token (see
> http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Run_Instance_Idempotency.html)
> makes the raw creation of 100+ instances without a thundering herd of
> RunInstances calls very realistic, but then we hit the issue of polling
> those nodes (even with the idempotency, we'd still end up having at least
> one is-node-running polling call per node, and then we'd hit the same thing
> but much bigger at terminate/pause/resume/etc)…

The way I’ve dealt with this in the past is by using POLL_INITIAL_PERIOD and POLL_MAX_PERIOD [1].

Does that work for your use case?

Everett

[1] https://github.com/jclouds/jclouds-examples/blob/master/rackspace/src/main/java/org/jclouds/examples/rackspace/cloudservers/CloudServersPublish.java#L102-103