[jira] [Commented] (FLINK-40405) [FLINK-autoscaler] Add reason tags to `autoscaler.balanced` counter to distinguish genuinely balanced from blocked scaling

["Kaustubh Butte (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-40405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18122183#comment-18122183 ] 

Kaustubh Butte commented on FLINK-40405:
----------------------------------------

I opened a pull request for this ticket:
https://github.com/apache/flink-kubernetes-operator/pull/1218 

The change adds a ScaleResult enum, with one constant for each outcome of an
evaluation cycle. AutoscalerFlinkMetrics.incrementBalanced increments the
untagged autoscaler.balanced counter, then a counter below the "reason" metric
variable. The untagged counter keeps its total, so existing dashboards continue
to operate.

The change is in the flink-autoscaler module. A standalone or YARN deployment
that uses the library also gets the tags.

Two constants are not in the description above. BLOCKED_BY_CUSTOM_EXECUTOR
covers the ScalingExecutorPlugin veto, which is newer than this ticket.
BLOCKED_BY_CLUSTER_RESOURCES separates the cluster capacity check from the
quota check, because an operator corrects the two differently.

Please assign the ticket to me.

> [FLINK-autoscaler] Add reason tags to `autoscaler.balanced` counter to distinguish genuinely balanced from blocked scaling
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-40405
>                 URL: https://issues.apache.org/jira/browse/FLINK-40405
>             Project: Flink
>          Issue Type: Improvement
>          Components: Autoscaler
>    Affects Versions: 2.0.0, 1.20.1
>            Reporter: Kaustubh Butte
>            Priority: Major
>              Labels: autoscaling, observability, pull-request-available
>
> Currently, the `autoscaler.balanced` counter in `AutoscalerFlinkMetrics` is incremented every time `ScalingExecutor.scaleResource()` returns `false`. However, there are at least 7 distinct reasons why scaling might not happen, and all of them are collapsed into a single counter:
> 1. *Genuinely balanced* - all vertices are within utilization bounds (`ScalingExecutor.java`, `!anyVertexOutsideBound`)
> 2. *Memory pressure* - GC pressure or heap usage exceeds threshold (`ScalingExecutor.isJobUnderMemoryPressure()`)
> 3. *Scaling disabled by config* - `SCALING_ENABLED=false` (`ScalingExecutor.checkIfBlockedAndTriggerScalingEvent()`)
> 4. *Excluded period* - calendar blackout window active (`CalendarUtils.inExcludedPeriods()`)
> 5. *Resource quota exceeded* - CPU/memory quota or cluster capacity reached (`ScalingExecutor.scalingWouldExceedMaxResources()`)
> 6. *Ineffective scaling detected* - last scale-up did not improve throughput (`JobVertexScaler.detectIneffectiveScaleUp()`)
> 7. *Scale-down cooldown* - `SCALE_DOWN_INTERVAL` has not elapsed (`JobVertexScaler.applyScaleDownInterval()`)
> 8. *Data unavailable* – TPR or target rate is NaN (`JobVertexScaler.computeScaleTargetParallelism()`)
> This makes the `balanced` metric unusable for monitoring and alerting. A job showing `balanced=100, scalings=0` could be perfectly healthy OR completely blocked by memory pressure, and there is no way to tell from the metric alone.
> *Proposed change:*
> 1. Introduce a `ScaleResult` enum with values for each blocking reason
> 2. Change `ScalingExecutor.scaleResource()` to return `ScaleResult` instead of `boolean`
> 3. In `AutoscalerFlinkMetrics`, register tagged sub-counters using `MetricGroup.addGroup("reason", reasonTag).counter("balanced")`
> 4. Keep the existing `numBalanced` counter incrementing for backward compatibility
> 5. Enrich `ParallelismChange.noChange()` with a `NoChangeReason` to propagate vertex-level blocking reasons
> *Result in metric reporters:*
>  - `autoscaler.balanced` (total, backward compatible)
>  - `autoscaler.balanced.reason.genuinely_balanced`
>  - `autoscaler.balanced.reason.memory_pressure`
>  - `autoscaler.balanced.reason.config_disabled`
>  - `autoscaler.balanced.reason.excluded_period`
>  - `autoscaler.balanced.reason.resource_quota`
>  - `autoscaler.balanced.reason.ineffective_scaling`
>  - `autoscaler.balanced.reason.cooldown`
>  - `autoscaler.balanced.reason.data_unavailable`
> *Files affected:*
>  - `AutoscalerFlinkMetrics.java` – add tagged counters
>  - `ScalingExecutor.java` – return `ScaleResult` instead of `boolean`
>  - `JobAutoScalerImpl.java` – switch on `ScaleResult`
>  - `JobVertexScaler.java` – add `NoChangeReason` to `ParallelismChange`
>  - New: `ScaleResult.java`
> All changes are in the `flink-autoscaler` module (standalone, platform-agnostic). Works with both K8s and YARN deployments.
> *Why this matters:*
> The Flink autoscaler is increasingly used in large-scale production deployments with thousands of jobs. Operators need to distinguish "this job does not need scaling" from "this job needs scaling but is blocked" to build effective dashboards and alerts. Today, the only way to get this information is through K8s events (which are not available in non-K8s deployments and are not suitable for metric-based monitoring).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)