[jira] [Commented] (FLINK-31706) The default source parallelism should be the same as execution's default parallelism under adaptive batch scheduler

["Lijie Wang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-31706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721776#comment-17721776 ] 

Lijie Wang commented on FLINK-31706:
------------------------------------

I think it's a good idea to use {{paralleism.default}} instread of the {{execution.batch.adaptive.auto-parallelism.default-source-parallelism}}.

Regarding the parallelism of Source in the adaptive batch scheduler, we also have some other ideas/actions in plan: dynamically infer the Source paralleism at runtime (according to the amount of data that Source actually needs to read after Dynamic Partition Pruning). One possible way is that the source coordinator can infer the parallelism based on the splits information actually consumed.

At that time, if the parallelism of Source are not specified by the user, the source coorinator will be responseible for inferring the parallelism automatically(if it supports). If the Source does not support inferring parallelism automatically, {{parallelism.default}} will be used as the parallelism of the Source. (An initial thought :))

> The default source parallelism should be the same as execution's default parallelism under adaptive batch scheduler
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-31706
>                 URL: https://issues.apache.org/jira/browse/FLINK-31706
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Coordination
>            Reporter: Yun Tang
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.18.0
>
>
> Currently, the sources need to set {{execution.batch.adaptive.auto-parallelism.default-source-parallelism }} in the adaptive batch scheduler mode, otherwise, the source parallelism is only 1 by default. A better solution might be set as the default execution parallelism if no user configured. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)