[jira] [Commented] (FLINK-6949) Add ability to ship custom resource files to YARN cluster

["Mikhail Pryakhin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16182562#comment-16182562 ] 

Mikhail Pryakhin commented on FLINK-6949:
-----------------------------------------

Unfortunately, the workaround you suggested doesn't work for me.

I tried to leverage the _StreamExecutionEnvironment#registerCachedFile_ method but failed because
this instance is created when the application master has already been started therefore the classpath to run the application somewhere on YARN cluster has already been created by means of _org.apache.flink.yarn.YarnClusterClient_. In my case, I need to be able to pass a local folder at the moment I submit the application so that it is included in the application YARN classpath. 

The option you suggested works well if I need to cache a file that is available for me at the moment I want to register it (for example a file on HDFS).

Is there any way we can extend _org.apache.flink.yarn.YarnClusterClient_ to pass user-specified folders to the YARN application classpath?

> Add ability to ship custom resource files to YARN cluster
> ---------------------------------------------------------
>
>                 Key: FLINK-6949
>                 URL: https://issues.apache.org/jira/browse/FLINK-6949
>             Project: Flink
>          Issue Type: Improvement
>          Components: Client, YARN
>    Affects Versions: 1.3.0
>            Reporter: Mikhail Pryakhin
>            Priority: Critical
>
> *The problem:*
> When deploying a flink job on YARN it is not possible to specify custom resource files to be shipped to YARN cluster.
>  
> *The use case description:*
> When running a flink job on multiple environments it becomes necessary to pass environment-related configuration files to the job's runtime. It can be accomplished by packaging configuration files within the job's jar. But having tens of different environments one can easily end up packaging as many jars as there are environments. It would be great to have an ability to separate configuration files from the job artifacts. 
>  
> *The possible solution:*
> add the --yarnship-files option to flink cli to specify files that should be shipped to the YARN cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)