[jira] [Created] (FLINK-31215) Backpropagate processing rate limits from non-scalable bottlenecks to upstream operators

["Gyula Fora (Jira)" <ji...@apache.org>]

Gyula Fora created FLINK-31215:
----------------------------------

             Summary: Backpropagate processing rate limits from non-scalable bottlenecks to upstream operators
                 Key: FLINK-31215
                 URL: https://issues.apache.org/jira/browse/FLINK-31215
             Project: Flink
          Issue Type: New Feature
          Components: Autoscaler, Kubernetes Operator
            Reporter: Gyula Fora


The current algorithm scales operators based on input data rates by propagating it forward through the graph.

However there are cases where a certain operators processing capacity is limited either because it has a set maxParallelism or the users excludes it from scaling (or otherwise the capacity doesnt increase with scaling).

In these cases it doesn't make sense to scale upstream operators to the target data rate if the job is going to be bottlenecked by a downstream operator. But instead we should backpropagate the limit based on the non-scalable bottleneck.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)