[jira] [Commented] (FLINK-13938) Use yarn public distributed cache to speed up containers launch

["Zili Chen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969172#comment-16969172 ] 

Zili Chen commented on FLINK-13938:
-----------------------------------

Thanks for proposing this [~fly_in_gis]! We meet a similar requirement here that instead uploading share libs every time, let AM/TM localize the resource directly from HDFS. However, as you said your change would "eliminate the unnecessary jars downloading and make launching container faster.", IIUC there is always overhead localize resources from HDFS to container?

BTW, what's the progress on this issue now?

> Use yarn public distributed cache to speed up containers launch
> ---------------------------------------------------------------
>
>                 Key: FLINK-13938
>                 URL: https://issues.apache.org/jira/browse/FLINK-13938
>             Project: Flink
>          Issue Type: New Feature
>          Components: Deployment / YARN
>            Reporter: Yang Wang
>            Assignee: Yang Wang
>            Priority: Major
>
> By default, the LocalResourceVisibility is APPLICATION, so they will be downloaded only once and shared for all taskmanager containers of a same application in the same node. However, different applications will have to download all jars every time, including the flink-dist.jar. I think we could use the yarn public cache to eliminate the unnecessary jars downloading and make launching container faster.
>  
> How to use the shared lib feature?
>  # Upload a copy of flink release binary to hdfs.
>  # Use the -ysl argument to specify the shared lib
> {code:java}
> ./bin/flink run -d -m yarn-cluster -p 20 -ysl hdfs:///flink/release/flink-1.9.0/lib examples/streaming/WindowJoin.jar{code}
>  
> -ysl, --yarnsharedLib <path>          Upload a copy of flink lib beforehand
>                                                           and specify the path to use public
>                                                           visibility feature of YARN NodeManager
>                                                           localizing resources.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)