[jira] [Commented] (FLINK-18551) Don't need to download flink-conf.yaml when launching taskmanager on Yarn mode

["Xintong Song (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17156527#comment-17156527 ] 

Xintong Song commented on FLINK-18551:
--------------------------------------

Hi [~Yumeng],

Thanks for reporting this.

Correct me if I'm wrong. Are you suggesting serialize the {{Configuration}} object on the JM side, and deserialize it on the TM side, to avoid downloading the {{flink-conf.yaml}} file?

I'm not sure about this proposal. I wonder how many time we can save from this proposed change. According to our experiences, downloading the config file takes almost no time.

On the other hand, passing the configuration object through environment variables introduces more complexity and maintain overhead, by requiring TM to load configurations with a different approach, instead of reusing the approach for other deployments. Such complexity does not seem necessary to me.

Maybe you can share some insight on how this proposed improvement is necessary to you.

> Don't need to download flink-conf.yaml when launching taskmanager on Yarn mode
> ------------------------------------------------------------------------------
>
>                 Key: FLINK-18551
>                 URL: https://issues.apache.org/jira/browse/FLINK-18551
>             Project: Flink
>          Issue Type: Improvement
>    Affects Versions: 1.10.1, 1.11.0
>            Reporter: Yumeng Zhang
>            Priority: Minor
>
> In current flink version, launching a taskmanager on Yarn will follow these steps:
> 1. download flink-conf.yaml file to the local working directory
> 2. get Configuration object by reading local flink-conf.yaml file
> We could reduce the IO operation by setting flinkConfig to container env when createTaskExecutorContext in jobmanager side and getting Configuration object by system env in taskmanager side. This way we don't need to download flink-conf.yaml when launching taskmanager on Yarn and could accelerate the start process a little bit.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)