You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by Junrui Lee <jr...@gmail.com> on 2024/10/09 02:44:00 UTC

[DISCUSS] FLIP-478: Introduce Config Option Scope

Hi devs,

I would like to initiate a discussion about FLIP-478: Introduce Config
Option Scope

Currently, Flink can be configured in various ways. Taking a session job as
an example:

1. When the session cluster starts, the JobManager (JM) and TaskManager
(TM) load the configuration files config.yaml from specified directory on
their respective machines, merging them with command line parameters (-D)
to form their component configuration.

2. When submitting a job, the client loads its local configuration file and
merges it with configuration within the job code and command line
parameters to obtain the job configuration.

When this job is submitted to the session cluster, the values of
cluster-level config options, such as the memory for JM and TM, will be
defined by the JM/TM configuration. And the values of job-level config
options, such as parallelism, will be defined by the job configuration.
Additionally, some config options will fall back to the values defined in
the configuration from JM/TM if their values are not defined in job
configuration, such as restart strategies.

However, it can be confusing for users to know where to set a config
option. For example, we've received several bug reports from users saying
that taskmanager.numberOfTaskSlots doesn’t take effect, which turned out to
be because they mistakenly configured it in the job code.

As a result, we intend to introduce the concept of Config Option Scope to
help users understand the proper way to configure each config option. And
the scopes should be displayed in Flink configuration documentation.

For more details, please refer to FLIP-478 [1]. Looking forward to your
feedback.

Best,

Junrui Lee

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-478+Introduce+Config+Option+Scope

Re: [DISCUSS] FLIP-478: Introduce Config Option Scope

Posted by Junrui Lee <jr...@gmail.com>.

Hi, devs

After discussing offline with Lincoln Lee and Shengkai Fang, we realized
that some SQL config options, like table.sql-dialect, may also server DDL
in which case no Flink job will run. Including these options under the JOB
scope is not accurate.

So we propose introducing a new SQL_EXECUTION scope, alongside the proposed
CLUSTER, JOB, JOB_AND_CLUSTER, SQL_CLIENT, SQL_GATEWAY, and HISTORY_SERVER
scopes.

The SQL_EXECUTION scope includes config options related to Flink SQL/Table
API that impact SQL or Table API execution, such as SQL translation,
execution plan generation, and operator behavior. Users can set these
options in a SQL/Table Program, or within the Sql-gateway or Sql-client.
Config values specified in the code take precedence; if not present, values
from the compiling service are used.

And since SQL-related config options are already on a separate page with
explanations on how to configure them, this FLIP will not detail the scope
of SQL config options in the common configuration page[1]. So it will add
little burden for users to understand this new scope.

If there are no further comments, we will open the voting thread on
Thursday, October 17, 2024.

[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/

Best,
Junrui

Rui Fan <19...@gmail.com> 于2024年10月10日周四 11:33写道：

> Thanks Junrui for the quick clarification!
>
> Sounds good to me, +1 for this FLIP.
>
> Best,
> Rui
>
> On Thu, Oct 10, 2024 at 10:52 AM Junrui Lee <jr...@gmail.com> wrote:
>
> > Hi Rui,
> >
> > Thank you for your response. Here are my replies to your questions:
> >
> > Regarding question 1: We plan to label the config options on the Flink
> > configuration page in the upcoming 2.0 version. In the long term, our
> goal
> > is to ensure that all config options visible to users are properly
> labeled.
> >
> > Regarding question 2: That’s a good question, and we have indeed
> considered
> > similar approaches for pre-validating configuration items. However, due
> to
> > the current limitations of Flink, the ConfigOption acts as a format used
> > when retrieving config values from the Configuration object.
> Consequently,
> > the relationship between ConfigOptions and the key-value pairs stored in
> > the Configuration object is currently one-way; there is no way to reverse
> > lookup a ConfigOption from a string key.
> >
> > This implies that we might need to construct a complete collection of
> > ConfigOptions in advance, which is not an easy or straightforward task.
> > Therefore, we prefer to focus this FLIP on introducing the concept of
> > config option scope, while any potential functionality extensions can be
> > addressed in future FLIPs.
> >
> > WDYT?
> >
> > Best,
> >
> > Junrui
> >
> > Rui Fan <19...@gmail.com> 于2024年10月10日周四 10:18写道：
> >
> > > Thanks Junrui for driving this proposal!
> > >
> > > It's valuable for users to distinguish all flink options.
> > > I have 2 minor questions about this proposal:
> > >
> > > 1. Does this FLIP enrich the scope for all old flink options?
> > > 2. If the option is configured at the wrong level, could we
> > > output some warn log to remind users? For example,
> > > an option of cluster level is added at job configuration.
> > >
> > > Best,
> > > Rui
> > >
> > > On Thu, Oct 10, 2024 at 9:34 AM Yanquan Lv <de...@gmail.com>
> wrote:
> > >
> > > > Thanks Junrui for starting this discussion!
> > > > Prior to this, it takes some experience to correctly distinguish
> > whether
> > > a
> > > > configuration item is effective at CLUSTER or JOB level, and this
> > > proposal
> > > > avoids such a burden.
> > > > And the FLIP looks good to me.
> > > >
> > > >
> > > > Best,
> > > > Yanquan
> > > >
> > > > Zhu Zhu <re...@gmail.com> 于2024年10月9日周三 17:14写道：
> > > >
> > > > > Thanks Junrui for starting this discussion!
> > > > > +1 for the proposal
> > > > >
> > > > > Clarifying the scopes of config options can be beneficial for
> users.
> > > > > Even Flink developers occasionally encounter confusion regarding
> the
> > > > > scopes of various configurations, such as attempting to retrieve
> > > > > job-specific settings from `TaskExecutor#taskManagerConfiguration`.
> > > > > Clarifying the options' scopes in documentation can be a good
> > reference
> > > > > and reminder.
> > > > >
> > > > > Thanks,
> > > > > Zhu
> > > > >
> > > > > Junrui Lee <jr...@gmail.com> 于2024年10月9日周三 11:08写道：
> > > > >
> > > > > > Hi devs,
> > > > > >
> > > > > > I would like to initiate a discussion about FLIP-478: Introduce
> > > Config
> > > > > > Option Scope
> > > > > >
> > > > > > Currently, Flink can be configured in various ways. Taking a
> > session
> > > > job
> > > > > as
> > > > > > an example:
> > > > > >
> > > > > > 1. When the session cluster starts, the JobManager (JM) and
> > > TaskManager
> > > > > > (TM) load the configuration files config.yaml from specified
> > > directory
> > > > on
> > > > > > their respective machines, merging them with command line
> > parameters
> > > > (-D)
> > > > > > to form their component configuration.
> > > > > >
> > > > > > 2. When submitting a job, the client loads its local
> configuration
> > > file
> > > > > and
> > > > > > merges it with configuration within the job code and command line
> > > > > > parameters to obtain the job configuration.
> > > > > >
> > > > > > When this job is submitted to the session cluster, the values of
> > > > > > cluster-level config options, such as the memory for JM and TM,
> > will
> > > be
> > > > > > defined by the JM/TM configuration. And the values of job-level
> > > config
> > > > > > options, such as parallelism, will be defined by the job
> > > configuration.
> > > > > > Additionally, some config options will fall back to the values
> > > defined
> > > > in
> > > > > > the configuration from JM/TM if their values are not defined in
> job
> > > > > > configuration, such as restart strategies.
> > > > > >
> > > > > > However, it can be confusing for users to know where to set a
> > config
> > > > > > option. For example, we've received several bug reports from
> users
> > > > saying
> > > > > > that taskmanager.numberOfTaskSlots doesn’t take effect, which
> > turned
> > > > out
> > > > > to
> > > > > > be because they mistakenly configured it in the job code.
> > > > > >
> > > > > > As a result, we intend to introduce the concept of Config Option
> > > Scope
> > > > to
> > > > > > help users understand the proper way to configure each config
> > option.
> > > > And
> > > > > > the scopes should be displayed in Flink configuration
> > documentation.
> > > > > >
> > > > > > For more details, please refer to FLIP-478 [1]. Looking forward
> to
> > > your
> > > > > > feedback.
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Junrui Lee
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-478+Introduce+Config+Option+Scope
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-478: Introduce Config Option Scope

Posted by Rui Fan <19...@gmail.com>.

Thanks Junrui for the quick clarification!

Sounds good to me, +1 for this FLIP.

Best,
Rui

On Thu, Oct 10, 2024 at 10:52 AM Junrui Lee <jr...@gmail.com> wrote:

> Hi Rui,
>
> Thank you for your response. Here are my replies to your questions:
>
> Regarding question 1: We plan to label the config options on the Flink
> configuration page in the upcoming 2.0 version. In the long term, our goal
> is to ensure that all config options visible to users are properly labeled.
>
> Regarding question 2: That’s a good question, and we have indeed considered
> similar approaches for pre-validating configuration items. However, due to
> the current limitations of Flink, the ConfigOption acts as a format used
> when retrieving config values from the Configuration object. Consequently,
> the relationship between ConfigOptions and the key-value pairs stored in
> the Configuration object is currently one-way; there is no way to reverse
> lookup a ConfigOption from a string key.
>
> This implies that we might need to construct a complete collection of
> ConfigOptions in advance, which is not an easy or straightforward task.
> Therefore, we prefer to focus this FLIP on introducing the concept of
> config option scope, while any potential functionality extensions can be
> addressed in future FLIPs.
>
> WDYT?
>
> Best,
>
> Junrui
>
> Rui Fan <19...@gmail.com> 于2024年10月10日周四 10:18写道：
>
> > Thanks Junrui for driving this proposal!
> >
> > It's valuable for users to distinguish all flink options.
> > I have 2 minor questions about this proposal:
> >
> > 1. Does this FLIP enrich the scope for all old flink options?
> > 2. If the option is configured at the wrong level, could we
> > output some warn log to remind users? For example,
> > an option of cluster level is added at job configuration.
> >
> > Best,
> > Rui
> >
> > On Thu, Oct 10, 2024 at 9:34 AM Yanquan Lv <de...@gmail.com> wrote:
> >
> > > Thanks Junrui for starting this discussion!
> > > Prior to this, it takes some experience to correctly distinguish
> whether
> > a
> > > configuration item is effective at CLUSTER or JOB level, and this
> > proposal
> > > avoids such a burden.
> > > And the FLIP looks good to me.
> > >
> > >
> > > Best,
> > > Yanquan
> > >
> > > Zhu Zhu <re...@gmail.com> 于2024年10月9日周三 17:14写道：
> > >
> > > > Thanks Junrui for starting this discussion!
> > > > +1 for the proposal
> > > >
> > > > Clarifying the scopes of config options can be beneficial for users.
> > > > Even Flink developers occasionally encounter confusion regarding the
> > > > scopes of various configurations, such as attempting to retrieve
> > > > job-specific settings from `TaskExecutor#taskManagerConfiguration`.
> > > > Clarifying the options' scopes in documentation can be a good
> reference
> > > > and reminder.
> > > >
> > > > Thanks,
> > > > Zhu
> > > >
> > > > Junrui Lee <jr...@gmail.com> 于2024年10月9日周三 11:08写道：
> > > >
> > > > > Hi devs,
> > > > >
> > > > > I would like to initiate a discussion about FLIP-478: Introduce
> > Config
> > > > > Option Scope
> > > > >
> > > > > Currently, Flink can be configured in various ways. Taking a
> session
> > > job
> > > > as
> > > > > an example:
> > > > >
> > > > > 1. When the session cluster starts, the JobManager (JM) and
> > TaskManager
> > > > > (TM) load the configuration files config.yaml from specified
> > directory
> > > on
> > > > > their respective machines, merging them with command line
> parameters
> > > (-D)
> > > > > to form their component configuration.
> > > > >
> > > > > 2. When submitting a job, the client loads its local configuration
> > file
> > > > and
> > > > > merges it with configuration within the job code and command line
> > > > > parameters to obtain the job configuration.
> > > > >
> > > > > When this job is submitted to the session cluster, the values of
> > > > > cluster-level config options, such as the memory for JM and TM,
> will
> > be
> > > > > defined by the JM/TM configuration. And the values of job-level
> > config
> > > > > options, such as parallelism, will be defined by the job
> > configuration.
> > > > > Additionally, some config options will fall back to the values
> > defined
> > > in
> > > > > the configuration from JM/TM if their values are not defined in job
> > > > > configuration, such as restart strategies.
> > > > >
> > > > > However, it can be confusing for users to know where to set a
> config
> > > > > option. For example, we've received several bug reports from users
> > > saying
> > > > > that taskmanager.numberOfTaskSlots doesn’t take effect, which
> turned
> > > out
> > > > to
> > > > > be because they mistakenly configured it in the job code.
> > > > >
> > > > > As a result, we intend to introduce the concept of Config Option
> > Scope
> > > to
> > > > > help users understand the proper way to configure each config
> option.
> > > And
> > > > > the scopes should be displayed in Flink configuration
> documentation.
> > > > >
> > > > > For more details, please refer to FLIP-478 [1]. Looking forward to
> > your
> > > > > feedback.
> > > > >
> > > > > Best,
> > > > >
> > > > > Junrui Lee
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-478+Introduce+Config+Option+Scope
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-478: Introduce Config Option Scope

Posted by Junrui Lee <jr...@gmail.com>.

Hi Rui,

Thank you for your response. Here are my replies to your questions:

Regarding question 1: We plan to label the config options on the Flink
configuration page in the upcoming 2.0 version. In the long term, our goal
is to ensure that all config options visible to users are properly labeled.

Regarding question 2: That’s a good question, and we have indeed considered
similar approaches for pre-validating configuration items. However, due to
the current limitations of Flink, the ConfigOption acts as a format used
when retrieving config values from the Configuration object. Consequently,
the relationship between ConfigOptions and the key-value pairs stored in
the Configuration object is currently one-way; there is no way to reverse
lookup a ConfigOption from a string key.

This implies that we might need to construct a complete collection of
ConfigOptions in advance, which is not an easy or straightforward task.
Therefore, we prefer to focus this FLIP on introducing the concept of
config option scope, while any potential functionality extensions can be
addressed in future FLIPs.

WDYT?

Best,

Junrui

Rui Fan <19...@gmail.com> 于2024年10月10日周四 10:18写道：

> Thanks Junrui for driving this proposal!
>
> It's valuable for users to distinguish all flink options.
> I have 2 minor questions about this proposal:
>
> 1. Does this FLIP enrich the scope for all old flink options?
> 2. If the option is configured at the wrong level, could we
> output some warn log to remind users? For example,
> an option of cluster level is added at job configuration.
>
> Best,
> Rui
>
> On Thu, Oct 10, 2024 at 9:34 AM Yanquan Lv <de...@gmail.com> wrote:
>
> > Thanks Junrui for starting this discussion!
> > Prior to this, it takes some experience to correctly distinguish whether
> a
> > configuration item is effective at CLUSTER or JOB level, and this
> proposal
> > avoids such a burden.
> > And the FLIP looks good to me.
> >
> >
> > Best,
> > Yanquan
> >
> > Zhu Zhu <re...@gmail.com> 于2024年10月9日周三 17:14写道：
> >
> > > Thanks Junrui for starting this discussion!
> > > +1 for the proposal
> > >
> > > Clarifying the scopes of config options can be beneficial for users.
> > > Even Flink developers occasionally encounter confusion regarding the
> > > scopes of various configurations, such as attempting to retrieve
> > > job-specific settings from `TaskExecutor#taskManagerConfiguration`.
> > > Clarifying the options' scopes in documentation can be a good reference
> > > and reminder.
> > >
> > > Thanks,
> > > Zhu
> > >
> > > Junrui Lee <jr...@gmail.com> 于2024年10月9日周三 11:08写道：
> > >
> > > > Hi devs,
> > > >
> > > > I would like to initiate a discussion about FLIP-478: Introduce
> Config
> > > > Option Scope
> > > >
> > > > Currently, Flink can be configured in various ways. Taking a session
> > job
> > > as
> > > > an example:
> > > >
> > > > 1. When the session cluster starts, the JobManager (JM) and
> TaskManager
> > > > (TM) load the configuration files config.yaml from specified
> directory
> > on
> > > > their respective machines, merging them with command line parameters
> > (-D)
> > > > to form their component configuration.
> > > >
> > > > 2. When submitting a job, the client loads its local configuration
> file
> > > and
> > > > merges it with configuration within the job code and command line
> > > > parameters to obtain the job configuration.
> > > >
> > > > When this job is submitted to the session cluster, the values of
> > > > cluster-level config options, such as the memory for JM and TM, will
> be
> > > > defined by the JM/TM configuration. And the values of job-level
> config
> > > > options, such as parallelism, will be defined by the job
> configuration.
> > > > Additionally, some config options will fall back to the values
> defined
> > in
> > > > the configuration from JM/TM if their values are not defined in job
> > > > configuration, such as restart strategies.
> > > >
> > > > However, it can be confusing for users to know where to set a config
> > > > option. For example, we've received several bug reports from users
> > saying
> > > > that taskmanager.numberOfTaskSlots doesn’t take effect, which turned
> > out
> > > to
> > > > be because they mistakenly configured it in the job code.
> > > >
> > > > As a result, we intend to introduce the concept of Config Option
> Scope
> > to
> > > > help users understand the proper way to configure each config option.
> > And
> > > > the scopes should be displayed in Flink configuration documentation.
> > > >
> > > > For more details, please refer to FLIP-478 [1]. Looking forward to
> your
> > > > feedback.
> > > >
> > > > Best,
> > > >
> > > > Junrui Lee
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-478+Introduce+Config+Option+Scope
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-478: Introduce Config Option Scope

Posted by Rui Fan <19...@gmail.com>.

Thanks Junrui for driving this proposal!

It's valuable for users to distinguish all flink options.
I have 2 minor questions about this proposal:

1. Does this FLIP enrich the scope for all old flink options?
2. If the option is configured at the wrong level, could we
output some warn log to remind users? For example,
an option of cluster level is added at job configuration.

Best,
Rui

On Thu, Oct 10, 2024 at 9:34 AM Yanquan Lv <de...@gmail.com> wrote:

> Thanks Junrui for starting this discussion!
> Prior to this, it takes some experience to correctly distinguish whether a
> configuration item is effective at CLUSTER or JOB level, and this proposal
> avoids such a burden.
> And the FLIP looks good to me.
>
>
> Best,
> Yanquan
>
> Zhu Zhu <re...@gmail.com> 于2024年10月9日周三 17:14写道：
>
> > Thanks Junrui for starting this discussion!
> > +1 for the proposal
> >
> > Clarifying the scopes of config options can be beneficial for users.
> > Even Flink developers occasionally encounter confusion regarding the
> > scopes of various configurations, such as attempting to retrieve
> > job-specific settings from `TaskExecutor#taskManagerConfiguration`.
> > Clarifying the options' scopes in documentation can be a good reference
> > and reminder.
> >
> > Thanks,
> > Zhu
> >
> > Junrui Lee <jr...@gmail.com> 于2024年10月9日周三 11:08写道：
> >
> > > Hi devs,
> > >
> > > I would like to initiate a discussion about FLIP-478: Introduce Config
> > > Option Scope
> > >
> > > Currently, Flink can be configured in various ways. Taking a session
> job
> > as
> > > an example:
> > >
> > > 1. When the session cluster starts, the JobManager (JM) and TaskManager
> > > (TM) load the configuration files config.yaml from specified directory
> on
> > > their respective machines, merging them with command line parameters
> (-D)
> > > to form their component configuration.
> > >
> > > 2. When submitting a job, the client loads its local configuration file
> > and
> > > merges it with configuration within the job code and command line
> > > parameters to obtain the job configuration.
> > >
> > > When this job is submitted to the session cluster, the values of
> > > cluster-level config options, such as the memory for JM and TM, will be
> > > defined by the JM/TM configuration. And the values of job-level config
> > > options, such as parallelism, will be defined by the job configuration.
> > > Additionally, some config options will fall back to the values defined
> in
> > > the configuration from JM/TM if their values are not defined in job
> > > configuration, such as restart strategies.
> > >
> > > However, it can be confusing for users to know where to set a config
> > > option. For example, we've received several bug reports from users
> saying
> > > that taskmanager.numberOfTaskSlots doesn’t take effect, which turned
> out
> > to
> > > be because they mistakenly configured it in the job code.
> > >
> > > As a result, we intend to introduce the concept of Config Option Scope
> to
> > > help users understand the proper way to configure each config option.
> And
> > > the scopes should be displayed in Flink configuration documentation.
> > >
> > > For more details, please refer to FLIP-478 [1]. Looking forward to your
> > > feedback.
> > >
> > > Best,
> > >
> > > Junrui Lee
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-478+Introduce+Config+Option+Scope
> > >
> >
>

Re: [DISCUSS] FLIP-478: Introduce Config Option Scope

Posted by Yanquan Lv <de...@gmail.com>.

Thanks Junrui for starting this discussion!
Prior to this, it takes some experience to correctly distinguish whether a
configuration item is effective at CLUSTER or JOB level, and this proposal
avoids such a burden.
And the FLIP looks good to me.


Best,
Yanquan

Zhu Zhu <re...@gmail.com> 于2024年10月9日周三 17:14写道：

> Thanks Junrui for starting this discussion!
> +1 for the proposal
>
> Clarifying the scopes of config options can be beneficial for users.
> Even Flink developers occasionally encounter confusion regarding the
> scopes of various configurations, such as attempting to retrieve
> job-specific settings from `TaskExecutor#taskManagerConfiguration`.
> Clarifying the options' scopes in documentation can be a good reference
> and reminder.
>
> Thanks,
> Zhu
>
> Junrui Lee <jr...@gmail.com> 于2024年10月9日周三 11:08写道：
>
> > Hi devs,
> >
> > I would like to initiate a discussion about FLIP-478: Introduce Config
> > Option Scope
> >
> > Currently, Flink can be configured in various ways. Taking a session job
> as
> > an example:
> >
> > 1. When the session cluster starts, the JobManager (JM) and TaskManager
> > (TM) load the configuration files config.yaml from specified directory on
> > their respective machines, merging them with command line parameters (-D)
> > to form their component configuration.
> >
> > 2. When submitting a job, the client loads its local configuration file
> and
> > merges it with configuration within the job code and command line
> > parameters to obtain the job configuration.
> >
> > When this job is submitted to the session cluster, the values of
> > cluster-level config options, such as the memory for JM and TM, will be
> > defined by the JM/TM configuration. And the values of job-level config
> > options, such as parallelism, will be defined by the job configuration.
> > Additionally, some config options will fall back to the values defined in
> > the configuration from JM/TM if their values are not defined in job
> > configuration, such as restart strategies.
> >
> > However, it can be confusing for users to know where to set a config
> > option. For example, we've received several bug reports from users saying
> > that taskmanager.numberOfTaskSlots doesn’t take effect, which turned out
> to
> > be because they mistakenly configured it in the job code.
> >
> > As a result, we intend to introduce the concept of Config Option Scope to
> > help users understand the proper way to configure each config option. And
> > the scopes should be displayed in Flink configuration documentation.
> >
> > For more details, please refer to FLIP-478 [1]. Looking forward to your
> > feedback.
> >
> > Best,
> >
> > Junrui Lee
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-478+Introduce+Config+Option+Scope
> >
>

Re: [DISCUSS] FLIP-478: Introduce Config Option Scope

Posted by Zhu Zhu <re...@gmail.com>.

Thanks Junrui for starting this discussion!
+1 for the proposal

Clarifying the scopes of config options can be beneficial for users.
Even Flink developers occasionally encounter confusion regarding the
scopes of various configurations, such as attempting to retrieve
job-specific settings from `TaskExecutor#taskManagerConfiguration`.
Clarifying the options' scopes in documentation can be a good reference
and reminder.

Thanks,
Zhu

Junrui Lee <jr...@gmail.com> 于2024年10月9日周三 11:08写道：

> Hi devs,
>
> I would like to initiate a discussion about FLIP-478: Introduce Config
> Option Scope
>
> Currently, Flink can be configured in various ways. Taking a session job as
> an example:
>
> 1. When the session cluster starts, the JobManager (JM) and TaskManager
> (TM) load the configuration files config.yaml from specified directory on
> their respective machines, merging them with command line parameters (-D)
> to form their component configuration.
>
> 2. When submitting a job, the client loads its local configuration file and
> merges it with configuration within the job code and command line
> parameters to obtain the job configuration.
>
> When this job is submitted to the session cluster, the values of
> cluster-level config options, such as the memory for JM and TM, will be
> defined by the JM/TM configuration. And the values of job-level config
> options, such as parallelism, will be defined by the job configuration.
> Additionally, some config options will fall back to the values defined in
> the configuration from JM/TM if their values are not defined in job
> configuration, such as restart strategies.
>
> However, it can be confusing for users to know where to set a config
> option. For example, we've received several bug reports from users saying
> that taskmanager.numberOfTaskSlots doesn’t take effect, which turned out to
> be because they mistakenly configured it in the job code.
>
> As a result, we intend to introduce the concept of Config Option Scope to
> help users understand the proper way to configure each config option. And
> the scopes should be displayed in Flink configuration documentation.
>
> For more details, please refer to FLIP-478 [1]. Looking forward to your
> feedback.
>
> Best,
>
> Junrui Lee
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-478+Introduce+Config+Option+Scope
>