You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Yang Wang (Jira)" <ji...@apache.org> on 2020/04/20 05:21:00 UTC
[jira] [Updated] (FLINK-13938) Use pre-uploaded flink binary to
accelerate flink submission
[ https://issues.apache.org/jira/browse/FLINK-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Wang updated FLINK-13938:
------------------------------
Description:
Currently, every time we start a flink cluster, flink lib jars need to be uploaded to hdfs and then register Yarn local resource so that it could be downloaded to jobmanager and all taskmanager container. I think we could have two optimizations.
# Use pre-uploaded flink binary to avoid uploading of flink system jars
# Use the yarn public cache to eliminate the unnecessary jars downloading and make launching container faster. The public cache could be shared by different applications.
By default, the LocalResourceVisibility is APPLICATION, so they will be downloaded only once and shared for all taskmanager containers of a same application in the same node. However, different applications will have to download all jars every time, including the flink-dist.jar. We could use the yarn public cache to eliminate the unnecessary jars downloading and make launching container faster.
Following the discussion in the user ML. [https://lists.apache.org/list.html?user@flink.apache.org:lte=1M:Flink%20Conf%20%22yarn.flink-dist-jar%22%20Question]
Take both FLINK-13938[1] and FLINK-14964[2] into account, this feature will be done in the following steps.
* Enrich "-yt/--yarnship" to support HDFS directory
* Enrich "-yt/--yarnship" to specify local resource visibility. It is "APPLICATION" by default. It could be also configured to "PUBLIC", which means shared by all applications, or "PRIVATE" which means shared by a same user.
* Add a new config option to control whether to optimize the submission(default is false). When configured to true, Flink client will try to filter the jars and files by name and size to avoid unnecessary uploading.
How to use this feature?
1. First, upload the Flink binary and user jars to the HDFS directories
2. Use "-yt/–yarnship" to specify the pre-uploaded libs
3. Enable the support optimization
A final submission command could be issued like following.
{code:java}
./bin/flink run -m yarn-cluster -d \
-yt hdfs://myhdfs/flink/release/flink-1.11:PUBLIC,hdfs://myhdfs/user/someone/mylib \
-yD yarn.submission-optimization.enable=true \
examples/streaming/WindowJoin.jar
{code}
was:
Currently, every time we start a flink cluster, flink lib jars need to be uploaded to hdfs and then register Yarn local resource so that it could be downloaded to jobmanager and all taskmanager container. I think we could have two optimizations.
# Use pre-uploaded flink binary to avoid uploading of flink system jars
# Use the yarn public cache to eliminate the unnecessary jars downloading and make launching container faster. The public cache could be shared by different applications.
By default, the LocalResourceVisibility is APPLICATION, so they will be downloaded only once and shared for all taskmanager containers of a same application in the same node. However, different applications will have to download all jars every time, including the flink-dist.jar. We could use the yarn public cache to eliminate the unnecessary jars downloading and make launching container faster.
How to use the shared lib feature?
# Upload a copy of flink release binary to hdfs.
# Use the -ypl argument to specify the shared lib
{code:java}
./bin/flink run -d -m yarn-cluster -p 20 -ypl hdfs://hdpdev/flink/release/flink-1.9.0 examples/streaming/WindowJoin.jar{code}
-ypl, --yarnpreUpload <path> Upload a copy of flink binary beforehand
and specify the path to use public
visibility feature of YARN NodeManager
localizing resources.
> Use pre-uploaded flink binary to accelerate flink submission
> ------------------------------------------------------------
>
> Key: FLINK-13938
> URL: https://issues.apache.org/jira/browse/FLINK-13938
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / YARN
> Reporter: Yang Wang
> Assignee: Yang Wang
> Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently, every time we start a flink cluster, flink lib jars need to be uploaded to hdfs and then register Yarn local resource so that it could be downloaded to jobmanager and all taskmanager container. I think we could have two optimizations.
> # Use pre-uploaded flink binary to avoid uploading of flink system jars
> # Use the yarn public cache to eliminate the unnecessary jars downloading and make launching container faster. The public cache could be shared by different applications.
>
> By default, the LocalResourceVisibility is APPLICATION, so they will be downloaded only once and shared for all taskmanager containers of a same application in the same node. However, different applications will have to download all jars every time, including the flink-dist.jar. We could use the yarn public cache to eliminate the unnecessary jars downloading and make launching container faster.
>
>
> Following the discussion in the user ML. [https://lists.apache.org/list.html?user@flink.apache.org:lte=1M:Flink%20Conf%20%22yarn.flink-dist-jar%22%20Question]
> Take both FLINK-13938[1] and FLINK-14964[2] into account, this feature will be done in the following steps.
> * Enrich "-yt/--yarnship" to support HDFS directory
> * Enrich "-yt/--yarnship" to specify local resource visibility. It is "APPLICATION" by default. It could be also configured to "PUBLIC", which means shared by all applications, or "PRIVATE" which means shared by a same user.
> * Add a new config option to control whether to optimize the submission(default is false). When configured to true, Flink client will try to filter the jars and files by name and size to avoid unnecessary uploading.
>
> How to use this feature?
> 1. First, upload the Flink binary and user jars to the HDFS directories
> 2. Use "-yt/–yarnship" to specify the pre-uploaded libs
> 3. Enable the support optimization
>
> A final submission command could be issued like following.
> {code:java}
> ./bin/flink run -m yarn-cluster -d \
> -yt hdfs://myhdfs/flink/release/flink-1.11:PUBLIC,hdfs://myhdfs/user/someone/mylib \
> -yD yarn.submission-optimization.enable=true \
> examples/streaming/WindowJoin.jar
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)