You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by "Lijie Wang (Jira)" <ji...@apache.org> on 2023/01/11 08:47:00 UTC

[jira] [Created] (FLINK-30631) Limit the max number of subpartitons consumed by each downstream task

Lijie Wang created FLINK-30631:
----------------------------------

             Summary: Limit the max number of subpartitons consumed by each downstream task
                 Key: FLINK-30631
                 URL: https://issues.apache.org/jira/browse/FLINK-30631
             Project: Flink
          Issue Type: Sub-task
          Components: Runtime / Coordination
            Reporter: Lijie Wang
             Fix For: 1.17.0


In the current implementation(FLINK-25035), when the upstream vertex parallelism is much greater than the downstream vertex parallelism, it may lead to a large number of channels in the downstream tasks(for example, A -> B, all to all edge, max parallelism is 1000. If parallelism of A is 1000, parallelism of B is decided to be 1, then the only subtask of B will consume 1000 * 1000 subpartitions), resulting in a large overhead for processing channels.

In this ticket, we temporarily address this issue by limiting the max number of subpartitons consumed by each downstream task. The ultimate solution should be to support single channel consume multiple subpartitons.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)