[jira] [Created] (FLINK-36411) Allow configuring job vertex throughput limits for auto scaler

["Sai Sharath Dandi (Jira)" <ji...@apache.org>]

Sai Sharath Dandi created FLINK-36411:
-----------------------------------------

             Summary: Allow configuring job vertex throughput limits for auto scaler
                 Key: FLINK-36411
                 URL: https://issues.apache.org/jira/browse/FLINK-36411
             Project: Flink
          Issue Type: Improvement
          Components: Autoscaler
            Reporter: Sai Sharath Dandi


*Problem Statement*

 

Currently, auto scaler has the ability to detect ineffective scalings and prevent further scale ups. However, the ineffective scaling detection does not work when there is no scaling history. Moreover, the ineffective scaling detection does not prevent the job from making an ineffective scaling for the first time.

 

This is particularly important for some of the sinks that could enforce throttling (For example, quota limit for Kafka sink). In these cases, we can avoid ineffective scalings by comparing the throughput limit and current processing rate.


*Solution*

Some high level ideas below

 
 # Allow user to specify the throughput limit at job vertex level

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)