[jira] [Commented] (FLINK-14964) Support configure remote flink jar

["Andreas Hailu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17053737#comment-17053737 ] 

Andreas Hailu commented on FLINK-14964:
---------------------------------------

Hi - I was linked to this ticket through the user distribution list as it's a feature our team is looking for. Subject: _Flink Conf "yarn.flink-dist-jar" Question_.

As we submit thousands of Flink apps a day, we run out of Hadoop user disk space quota, causing our apps to fail. We noticed that a lot of disk space was consumed by /user/<ourUser>/.flink/application_<appId>/flink-dist_2.11-1.9.1.jar. Spark solves this solution by passing in the HDFS location of the spark assembly package as a option during submission (see "spark.yarn.jar" option).

It would be great if we could have a similar command line option behavior, or a flink-conf option.

> Support configure remote flink jar
> ----------------------------------
>
>                 Key: FLINK-14964
>                 URL: https://issues.apache.org/jira/browse/FLINK-14964
>             Project: Flink
>          Issue Type: Improvement
>          Components: Deployment / YARN
>            Reporter: Zili Chen
>            Assignee: Zili Chen
>            Priority: Major
>
> Corresponding discussion happens on [https://lists.apache.org/x/thread.html/a2fef19f6925d43dd2cf3f9a46a2263b20562bd61d4da970f609568d@%3Cdev.flink.apache.org%3E]
> This ticket focuses on support configure remote flink jar. That is, when user configure flink jar as "hdfs://..." we also respect the pattern and instead of uploading from local, directly register as YARN's {{LocalResource}}.
> This improvement will speed up the deployment on YARN.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)