[jira] [Created] (FLINK-17858) Yarn mode, windows and linux environment should be interlinked

["ChengWei Ye (Jira)" <ji...@apache.org>]

ChengWei Ye created FLINK-17858:
-----------------------------------

             Summary: Yarn mode, windows and linux environment should be interlinked
                 Key: FLINK-17858
                 URL: https://issues.apache.org/jira/browse/FLINK-17858
             Project: Flink
          Issue Type: Improvement
          Components: Deployment / YARN
    Affects Versions: 1.10.0
            Reporter: ChengWei Ye


my flink version: 1.10.0

my yarn version: 2.7.6

When I use windows idea to try to submit a job to Linux's yarn cluster using java, am container throws an exception "Error: Could not find or load main class org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint".

Then find out that the CLASSPATH and _FLINK_CLASSPATH delimiter of the container startup script "launch_container.sh" are different, windows is ";", linux is ":", so the container cannot get the dependent packages.

I think the operating environment should be left to Yarn to judge.

Locate in the flink-yarn module
1. The startAppMaster method of org.apache.flink.yarn.YarnClusterDescriptor
Change "File.pathSeparator" to "ApplicationConstants.CLASS_PATH_SEPARATOR"
Yarn will replace special characters according to the environment, but this only solves _FLINK_CLASSPATH

 
{code:java}
private ApplicationReport startAppMaster(.....) {
    .....
    StringBuilder classPathBuilder = new StringBuilder();
    if (userJarInclusion == YarnConfigOptions.UserJarInclusion.FIRST) {
       for (String userClassPath : userClassPaths) {
          // here
          classPathBuilder.append(userClassPath).append(File.pathSeparator);
       }
    }
    for (String classPath : systemClassPaths) {
       // here
       classPathBuilder.append(classPath).append(File.pathSeparator);
    }    
    // Setup jar for ApplicationMaster
    .....    
    // here 
    classPathBuilder.append(flinkJarPath.getName()).append(File.pathSeparator);
    .....
}{code}
 

2. The addToEnvironment method of org.apache.flink.yarn.Utils
Change "File.pathSeparator" to "ApplicationConstants.CLASS_PATH_SEPARATOR"
Can solve the CLASSPATH
{code:java}
public static void addToEnvironment(Map<String, String> environment,
      String variable, String value) {
   String val = environment.get(variable);
   if (val == null) {
      val = value;
   } else {
      // here
      val = val + File.pathSeparator + value;
   }
   environment.put(StringInterner.weakIntern(variable),
         StringInterner.weakIntern(val));
}
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)