[jira] [Commented] (FLINK-14626) User jar packaged with hadoop dependencies may cause class conflit with hadoop jars on yarn

["Chesnay Schepler (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-14626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969267#comment-16969267 ] 

Chesnay Schepler commented on FLINK-14626:
------------------------------------------

This would change existing behavior and could result in subtle bugs. I'm also a bit concerned about making the existing YARN code even more complicated (it is a minefield for bugs as is), in particular without a use-case that can't be summed up as "user error".

> User jar packaged with hadoop dependencies may cause class conflit with hadoop jars on yarn
> -------------------------------------------------------------------------------------------
>
>                 Key: FLINK-14626
>                 URL: https://issues.apache.org/jira/browse/FLINK-14626
>             Project: Flink
>          Issue Type: Bug
>          Components: Deployment / YARN
>    Affects Versions: 1.9.1
>            Reporter: Victor Wong
>            Priority: Major
>
> Currently, the yarn application classpath is placed behind Flink classpath (including user jars), which will cause conflict if the user jar accidentally included hadoop dependencies.
> {code:java}
> // org.apache.flink.yarn.Utils#setupYarnClassPath
> public static void setupYarnClassPath(Configuration conf, Map<String, String> appMasterEnv) {
>    addToEnvironment(
>       appMasterEnv,
>       Environment.CLASSPATH.name(),
>       appMasterEnv.get(ENV_FLINK_CLASSPATH));
>    String[] applicationClassPathEntries = conf.getStrings(
>       YarnConfiguration.YARN_APPLICATION_CLASSPATH,
>       YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH);
>    for (String c : applicationClassPathEntries) {
>       addToEnvironment(appMasterEnv, Environment.CLASSPATH.name(), c.trim());
>    }
> }
> {code}
> Maybe we should place the user jars behind yarn application classpath when `org.apache.flink.yarn.configuration.YarnConfigOptions.UserJarInclusion` is set to LAST, like this "flink-xxx.jar:hadoop-xxx.jar:user-xxx.jar".
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)