[jira] [Created] (FLINK-30571) Compute scale parallelism based on observed scalability

["Gyula Fora (Jira)" <ji...@apache.org>]

Gyula Fora created FLINK-30571:
----------------------------------

             Summary: Compute scale parallelism based on observed scalability 
                 Key: FLINK-30571
                 URL: https://issues.apache.org/jira/browse/FLINK-30571
             Project: Flink
          Issue Type: New Feature
          Components: Kubernetes Operator
            Reporter: Gyula Fora
            Assignee: Gyula Fora


When computing target parallelism for job vertices we currently assume linear scaling with a fixed (1) coefficient.

This assumes that in order to double the capacity we simply double the parallelism.

While linearity already might be violated by many real time workloads this form of strong linearity rarely holds due to the overhead of increased network traffic, coordination etc.

As we can access past (parallelism, processingRate) information based on the scaling history we should estimate the scalability coefficient either using a simple or weighted linear regression.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)