[jira] [Comment Edited] (FLINK-30530) Flink configuration from user-provided ConfigMap

["Arseniy Tashoyan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-30530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17653159#comment-17653159 ] 

Arseniy Tashoyan edited comment on FLINK-30530 at 12/30/22 4:43 PM:
--------------------------------------------------------------------

Hi [~gyfora] 

The YAML descriptor contains settings, that can be common for all applications. For example, the *podTemplate* can be identical for all. Then, *serviceAccount* - same for all applications running in this Kubernetes cluster. {*}image{*}, *flinkVersion* and even *jarURI* - can be same for a number of applications.

Flink settings are more specific to a particular application. Flink memory settings is a matter of fine tuning per application. Same for checkpointing settings.  Same for parallelism and many other settings.

For me it would be fine to have separate user-provided ConfigMaps for Flink and Log4J settings: *flinkConfigMap* and {*}log4jConfigMap{*}. For example, we could have two possible values for {*}log4jConfigMap{*}: console logging and file logging. It would be better, than to copy-paste the same logging configuration across many YAML descriptors.

Finally, it is more convenient to group related settings in respective configuration files (in Kubernetes - ConfigMaps). That's why people have separate flink-conf.yaml, log4j.properties, etc.


was (Author: tashoyan):
Hi [~gyfora] 

The YAML descriptor contains settings, that can be common for all applications. For example, the *podTemplate* can be identical for all. Then, *serviceAccount* - same for all applications running in this Kubernetes cluster. {*}image{*}, *flinkVersion* and even *jarURI* - can be same for a number of applications.

Flink settings are more specific to a particular application. Flink memory settings is a matter of fine tuning per application. Same for checkpointing settings.  Same for parallelism and many other settings.

For me it would be fine to have separate user-provided ConfigMaps for Flink and Log4J settings: *flinkConfigMap* and {*}log4jConfigMap{*}. For example, we could have two possible values for {*}log4jConfigMap{*}: console logging and file logging. It would be better, than to copy-paste the same logging configuration across many YAML descriptors.

Finally, it is more convenient to group related settings in respective configuration files. That's why people have separate flink-conf.yaml, log4j.properties, etc.

> Flink configuration from user-provided ConfigMap
> ------------------------------------------------
>
>                 Key: FLINK-30530
>                 URL: https://issues.apache.org/jira/browse/FLINK-30530
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kubernetes Operator
>         Environment: Flink 1.15.2
> Flink Kubernetes operator 1.2.0
>            Reporter: Arseniy Tashoyan
>            Priority: Major
>
> Currently the Flink configuration can be specified in the YAML descriptor of FlinkDeployment via the _flinkConfiguration_ setting:
> {code:yaml}
> flinkConfiguration: 
>   taskmanager.numberOfTaskSlots: "2"
>   ...
> {code}
> Same for the logging configuration:
> {code:yaml}
> logConfiguration: 
>     "log4j-console.properties": |
>       rootLogger.level = DEBUG
>       ...{code}
> This makes the YAML descriptor overloaded and huge. In addition, Flink and logging configuration may differ for different applications, while the Kubernetes settings maybe same for all applications. Therefore it makes sense to extract Flink and logging configurations from the YAML descriptor.
> This can be done via a user-provided ConfigMap:
> {code:yaml}
> flinkConfigMap: basic-example-flink-config
> {code}
> In this example we have a Flink application {_}basic-example{_}. The _basic-example-flink-config_ ConfigMap contains all config files used by Flink: flink-conf.yaml, log4j-console.properties, possibly other files. The content of this ConfigMap gets mounted as a volume to {_}/opt/flink/conf{_}.
> Therefore we can have different Flink settings for different applications and the same YAML descriptor for all of them (only the value for _flinkConfigMap_ differs).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)