[jira] [Created] (FLINK-14626) User jar packaged with hadoop dependencies may cause class conflit with hadoop jars on yarn

["Victor Wong (Jira)" <ji...@apache.org>]

Victor Wong created FLINK-14626:
-----------------------------------

             Summary: User jar packaged with hadoop dependencies may cause class conflit with hadoop jars on yarn
                 Key: FLINK-14626
                 URL: https://issues.apache.org/jira/browse/FLINK-14626
             Project: Flink
          Issue Type: Bug
          Components: Deployment / YARN
    Affects Versions: 1.9.1
            Reporter: Victor Wong


Currently, the yarn application classpath is placed behind Flink classpath (including user jars), which will cause conflict if the user jar accidentally included hadoop dependencies.
{code:java}
// org.apache.flink.yarn.Utils#setupYarnClassPath
public static void setupYarnClassPath(Configuration conf, Map<String, String> appMasterEnv) {
   addToEnvironment(
      appMasterEnv,
      Environment.CLASSPATH.name(),
      appMasterEnv.get(ENV_FLINK_CLASSPATH));
   String[] applicationClassPathEntries = conf.getStrings(
      YarnConfiguration.YARN_APPLICATION_CLASSPATH,
      YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH);
   for (String c : applicationClassPathEntries) {
      addToEnvironment(appMasterEnv, Environment.CLASSPATH.name(), c.trim());
   }
}
{code}
Maybe we should place the user jars behind yarn application classpath when `org.apache.flink.yarn.configuration.YarnConfigOptions.UserJarInclusion` is set to LAST, like this "flink-xxx.jar:hadoop-xxx.jar:user-xxx.jar".

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)