[jira] [Comment Edited] (FLINK-21036) Consider removing automatic configuration fo number of slots from docker

["Chesnay Schepler (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-21036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270268#comment-17270268 ] 

Chesnay Schepler edited comment on FLINK-21036 at 1/22/21, 4:50 PM:
--------------------------------------------------------------------

hmm...for plain docker that certainly works, but for kubernetes I suppose the issue is that'd you have to modify the container args parameter, with which you end up mixing what process you run (e.g., taskmanager) and what config options you want to set.
I don't know enough about the kubernetes configuration, but isn't there something where you can share bits across different files? I think those things would be difficult to unify.

However, in the ML discussion about configuring Flink via environment variables I brought forth the idea of a special environment variable DYNAMIC_PROPERTIES where you could set such options. We may be able to converge these issues.



was (Author: zentol):
hmm...for plain docker that certainly works, but for kubernetes I suppose the issue is that'd you have to modify the container args parameter, with which you end up mixing what process you run (e.g., taskmanager) and what config options you want to set.
I don't know enough about the kubernetes configuration, but isn't there something where you can share bits across different files?


> Consider removing automatic configuration fo number of slots from docker
> ------------------------------------------------------------------------
>
>                 Key: FLINK-21036
>                 URL: https://issues.apache.org/jira/browse/FLINK-21036
>             Project: Flink
>          Issue Type: Improvement
>          Components: Deployment / Scripts
>            Reporter: Chesnay Schepler
>            Assignee: Chesnay Schepler
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.13.0
>
>
> The {{docker-entrypoint.sh}} supports setting the number of task slots via the {{TASK_MANAGER_NUMBER_OF_TASK_SLOTS}} environment variable, which defaults to the number of cpu cores via {{$(grep -c ^processor /proc/cpuinfo)}}.
> The environment variable itself is redundant nowadays since we introduced {{FLINK_PROPERTIES}}, and is no longer documented.
> Defaulting to the number of CPU cores can be considered convenience, but it seems odd to have this specific to docker while the distribution defaults to {{1}}.
> The bigger issue in my mind though is that this creates a configuration mismatch between the Job- and TaskManager processes; the ResourceManager specifically needs to know how many slots a worker has to make decisions about redundancy and allocating resources.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)