[jira] [Updated] (FLINK-31827) Incorrect estimation of the target data rate of a vertex when only a subset of its upstream vertex's output is consumed

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-31827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated FLINK-31827:
-----------------------------------
    Labels: pull-request-available  (was: )

> Incorrect estimation of the target data rate of a vertex when only a subset of its upstream vertex's output is consumed
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-31827
>                 URL: https://issues.apache.org/jira/browse/FLINK-31827
>             Project: Flink
>          Issue Type: Bug
>          Components: Autoscaler
>            Reporter: Zhanghao Chen
>            Assignee: Gyula Fora
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: image-2023-04-17-23-37-35-280.png
>
>
> Currently, the target data rate of a vertex = SUM(target data rate * input/output ratio) for all of its upstream vertices. This assumes that all output records of an upstream vertex is consumed by the downstream vertex. However, it does not always hold. Consider the following job plan generated by a Flink SQL job. The middle vertex contains multiple chained Calc(select xx) operators, each connecting to a separate downstream sink tasks. As a result, each sink task only consumes a sub-portion of the middle vertex's output.
> To fix it, we need operator level edge info to infer the upstream-downstream relationship as well as operator level output metrics. The metrics part is easy but AFAIK, there's no way to get the operator level edge info from the Flink REST API yet.
> !image-2023-04-17-23-37-35-280.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)