[jira] [Comment Edited] (FLINK-31079) Release Testing: Verify FLINK-29663 Further improvements of adaptive batch scheduler

["Lijie Wang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-31079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691969#comment-17691969 ] 

Lijie Wang edited comment on FLINK-31079 at 2/22/23 7:14 AM:
-------------------------------------------------------------

Thanks [~lsy]. Currently, the [{{execution.batch.adaptive.auto-parallelism.avg-data-volume-per-task}}|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#execution-batch-adaptive-auto-parallelism-avg-data-volume-per-ta] is cluster level, but I personally think it makes sense to make it job level, especially when using session cluster, I will evaluate it in the future version.


was (Author: wanglijie95):
Thanks [~lsy]. Currently, the [{{execution.batch.adaptive.auto-parallelism.avg-data-volume-per-task}}|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#execution-batch-adaptive-auto-parallelism-avg-data-volume-per-ta] is cluster level, but I personally think it makes sense to make it a job level, especially when using session cluster, I will evaluate it in the future version.

> Release Testing: Verify FLINK-29663 Further improvements of adaptive batch scheduler
> ------------------------------------------------------------------------------------
>
>                 Key: FLINK-31079
>                 URL: https://issues.apache.org/jira/browse/FLINK-31079
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Runtime / Coordination
>            Reporter: Lijie Wang
>            Assignee: miamiaoxyz
>            Priority: Blocker
>             Fix For: 1.17.0
>
>         Attachments: image-2023-02-22-14-00-13-646.png
>
>
> This task aims to verify FLINK-29663 which improves the adaptive batch scheduler.
> Before the change of FLINK-29663, adaptive batch scheduler will distribute subpartitoins according to the number of subpartitions, make different downstream subtasks consume roughly the same number of subpartitions. This will lead to imbalance loads of different downstream tasks when the subpartitions contain different amounts of data.
> To solve this problem, in FLINK-29663, we let the adaptive batch scheduler distribute subpartitoins according to the amount of data, so that different downstream subtasks consume roughly the same amount of data. Note that currently it only takes effect for All-To-All edges.
> The documentation of adaptive scheduler can be found [here|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#adaptive-batch-scheduler]
> One can verify it by creating intended data skew on All-To-All edges.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)