[jira] [Commented] (FLINK-17858) Yarn mode, windows and linux environment should be interlinked

["Andrey Zagrebin (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-17858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17177656#comment-17177656 ] 

Andrey Zagrebin commented on FLINK-17858:
-----------------------------------------

[~barneyye]

any feedback?
what about java.io.File.pathSeparatorChar?

> Yarn mode, windows and linux environment should be interlinked
> --------------------------------------------------------------
>
>                 Key: FLINK-17858
>                 URL: https://issues.apache.org/jira/browse/FLINK-17858
>             Project: Flink
>          Issue Type: Improvement
>          Components: Deployment / YARN
>    Affects Versions: 1.10.0
>            Reporter: ChengWei Ye
>            Priority: Minor
>
> my flink version: 1.10.0
> my yarn version: 2.7.6
> When I use windows idea to try to submit a job to Linux's yarn cluster using java, am container throws an exception "Error: Could not find or load main class org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint".
> Then find out that the CLASSPATH and _FLINK_CLASSPATH delimiter of the container startup script "launch_container.sh" are different, windows is ";", linux is ":", so the container cannot get the dependent packages.
> I think the operating environment should be left to Yarn to judge.
> Locate in the flink-yarn module
> 1. The startAppMaster method of org.apache.flink.yarn.YarnClusterDescriptor
> Change "File.pathSeparator" to "ApplicationConstants.CLASS_PATH_SEPARATOR"
> Yarn will replace special characters according to the environment, but this only solves _FLINK_CLASSPATH
>  
> {code:java}
> private ApplicationReport startAppMaster(.....) {
>     .....
>     StringBuilder classPathBuilder = new StringBuilder();
>     if (userJarInclusion == YarnConfigOptions.UserJarInclusion.FIRST) {
>        for (String userClassPath : userClassPaths) {
>           // here
>           classPathBuilder.append(userClassPath).append(File.pathSeparator);
>        }
>     }
>     for (String classPath : systemClassPaths) {
>        // here
>        classPathBuilder.append(classPath).append(File.pathSeparator);
>     }    
>     // Setup jar for ApplicationMaster
>     .....    
>     // here 
>     classPathBuilder.append(flinkJarPath.getName()).append(File.pathSeparator);
>     .....
> }{code}
>  
> 2. The addToEnvironment method of org.apache.flink.yarn.Utils
> Change "File.pathSeparator" to "ApplicationConstants.CLASS_PATH_SEPARATOR"
> Can solve the CLASSPATH
> {code:java}
> public static void addToEnvironment(Map<String, String> environment,
>       String variable, String value) {
>    String val = environment.get(variable);
>    if (val == null) {
>       val = value;
>    } else {
>       // here
>       val = val + File.pathSeparator + value;
>    }
>    environment.put(StringInterner.weakIntern(variable),
>          StringInterner.weakIntern(val));
> }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)