[jira] [Created] (FLINK-40534) Memory tuning config overrides are computed before the ScalingExecutorPlugin chain and persisted unchanged when a plugin modifies the scaling decision

["Shubhankar Sunil Unhale (Jira)" <ji...@apache.org>]

Shubhankar Sunil Unhale created FLINK-40534:
-----------------------------------------------

             Summary: Memory tuning config overrides are computed before the ScalingExecutorPlugin chain and persisted unchanged when a plugin modifies the scaling decision
                 Key: FLINK-40534
                 URL: https://issues.apache.org/jira/browse/FLINK-40534
             Project: Flink
          Issue Type: Bug
          Components: Kubernetes Operator
            Reporter: Shubhankar Sunil Unhale


When memory tuning is enabled via {{job.autoscaler.memory.tuning.enabled}} and a {{ScalingExecutorPlugin}} modifies the scaling decision, which FLIP-575 explicitly supports ("Modified -> use modified decision"), the persisted memory overrides still reflect the original pre-plugin decision.

In {{ScalingExecutor#scaleResource}}, on main at commit 38a9f197:

# {{computeScalingSummary}} builds the summaries ({{ScalingExecutor.java:172-179}})
# {{MemoryTuning.tuneTaskManagerMemory}} computes {{configOverrides}} from them ({{:192-198}})
# the max-resource/quota check also runs on them ({{:200-207}})
# the plugin chain may modify the summaries ({{:210-214}})
# the modified parallelism is persisted ({{:231-234}}), but {{storeConfigChanges(context, configOverrides)}} persists the step-2 result unchanged ({{:236}})

As a result, the applied TM memory sizing, including total process size, network min/max, managed fraction and metaspace, is computed for a decision that never deploys. The tuning output depends on the per-vertex decision: network memory comes from {{computeNewParallelisms}} and heap from {{MemoryScaling.applyMemoryScaling}}, so changing even one vertex leaves the whole job-wide TM config mismatched ({{MemoryTuning.java:158-159}}, {{:175-177}}).

A plugin can't correct this itself. The SPI method {{apply(Context, Map<JobVertexID, ScalingSummary>)}} exposes no {{ConfigChanges}} surface, and the config reachable from the context is a defensive copy ({{ScalingExecutorPlugin.java:88-89}}, {{ConfigChanges.java:60-66}}). Since FLINK-39959 a plugin can _see_ the tuned config, but has no write-back path.

A related consequence of the same ordering: since the max-resource/quota check in step 3 also runs before the chain, a plugin that increases parallelism bypasses the cluster-resource and quota guards entirely.

Scope: the veto path is safe since it returns before anything is persisted, and memory tuning is off by default, so hitting this needs tuning enabled plus a decision-modifying plugin ({{ScalingExecutor.java:215-217}}, {{AutoScalerOptions.java:286-289}}). {{ScalingExecutorPluginTest}} doesn't cover the tuning interaction, and the plugins documentation page doesn't mention it.

A possible fix without touching the SPI would be to recompute the tuning and the max-resource check from the post-chain summaries before persisting, or to skip persisting {{configOverrides}} when the chain modified the decision. That said, FLINK-40455 plans to decouple tuning to run after the scaling decision and see its outcome, which would resolve this ordering naturally, so this may just be useful input for that work.

Happy to put up a PR with the recompute approach and a regression test if that seems like the right direction.

Related: FLINK-39555, FLINK-40455, [PR #1085|https://github.com/apache/flink-kubernetes-operator/pull/1085], [FLIP-575|https://cwiki.apache.org/confluence/display/FLINK/FLIP-575%3A+Support+Customized+Autoscale+Algorithm], [autotuning redesign discussion|https://lists.apache.org/thread/vk6j8793rx2y78vhh1yvfjf4qntpfkm9]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)