AutoScalerPolicy: can resize too large

[Aled Sage <al...@gmail.com>]

Hi all,

We have a problem with AutoScalerPolicy, as demonstrated by 
`TomcatAutoScalerPolicyTest` which fails non-deterministically.

*--- SUMMARY OF PROBLEM ---*
(for a more detailed description, jump to the "problem details" section 
at the end).

Imagine one has an enricher to calculate average-cpu-load across a 
cluster, and a policy to keep that average between 30% and 60% per node.

When the AutoScalerPolicy calculates the optimal cluster size based on 
the metric's value and the current cluster size - its "current size" is 
*all* nodes, including those that are still starting, which have not 
contributed to the load. Therefore it acts as though those new nodes 
were already pulling their weight, and resizes too big.

  * AutoScalerPolicy uses a different "current size" operator from that
    used by the enricher calculating the average.
  * AutoScaler's extrapolation logic (to calculate desired size)
    duplicates (part of) the logic in the original enricher (but using a
    different number!).

Note that I really like our separation of enricher from policy. The 
enricher can perform complex aggregation of multiple metrics (e.g. CPU, 
queue-length, request count, etc), which is independent of there being 
an auto-scaler. The metric could also come from an external source that 
is just summarising the cluster (i.e. Brooklyn might not have the 
individual metrics for each member; but I don't think we've hit that 
use-case in the wild yet).

Also note that the AutoScalerPolicy just acts on a Resizable: it calls 
`int getCurrentSize()` and `int resize(int desiredSize)`. Therefore it 
could easily be applied to vertically scaling a component as well as 
horizontally scaling a cluster.

*--- PROPOSAL ---*

Add to Resizable a new method `getActiveSize()`, which returns the 
number of active nodes. This would often be nodes reporting 
serviceUp=true && state=running.

Add support for configuring an Aggregator enricher to only use nodes 
matching some state (e.g. serviceUp=true && state=running).
(Current it just accepts any non-null value from the members).

Rely on the stabilisation-delay config for any races where the policy 
and enricher get different counts for the "active size".
i.e. we only resize up or down if we consistently need to do so for 
several seconds/minutes.

*---**ALTERNATIVE SOLUTIONS ---*
Alternatives include:

 1. Change resizable.getCurrentSize() implementation of cluster to
    return only the active-member count.
    (currently it inherits getCurrentSize() from `AbstractGroup`, which
    just does a simple count of the members.
 2. Don't use the aggregating enricher.
    We could configure AutoScalerPolicy with the metric to read from
    each member, and have the auto-scaler do the aggregation across the
    members.

For (1), I like that it doesn't change the interface (which breaks 
backwards compatibility for implementers).
But I prefer the simplicity of current size telling us the member count.

For (2), that feels like putting too much responsibility into the single 
AutoScalerPolicy class.
Better to strive for composability of policies/enrichers.

*--- PROBLEM DETAILS ---*
The formula used by the AutoScalerPolicy (simplified to strip out 
stabilisation delays, max pool size, etc) is:

    // metric to be kept within the given range
    double currentMetric = ...;
    double metricUpperBound = ...;
    double metricLowerBound = ...;

    int currentSize = ...;
    double currentTotalActivity = currentSize * currentMetric;

    if (currentMetric > metricUpperBound) {
         // scale out
         desiredSize =
    (int)Math.ceil(currentTotalActivity/metricUpperBound);

A simple example would be to use the cpu-usage average per node, to be 
kept in the range 30% to 60%.

  * An enricher would aggregate the CPU usage per node, to have a single
    metric on the cluster (e.g.
    PROCESS_CPU_TIME_FRACTION_IN_WINDOW_PER_NODE).
    This enricher would ignore null values for nodes still starting up.
  * The auto-scaler policy finds the currentSize by calling
    resizable.currentSize(), which just counts the members (rather than
    those with serviceUp).

  * So if we have two running members and a third still starting, with
    cpu values 95%, 95% and null respectively...
    The cluster's PROCESS_CPU_TIME_FRACTION_IN_WINDOW_PER_NODE will be 95%.
    The currentSize will be 3.
    The auto-scaler will calculate desired size as (0.95 * 3) / 0.6 =
    4.75; rounded up to 5.
    It should have done (0.95 * 2) / 0.6 = 3.167; rounded up to 4.