You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Till Rohrmann (Jira)" <ji...@apache.org> on 2020/10/09 09:54:00 UTC
[jira] [Closed] (FLINK-17858) Yarn mode, windows and linux
environment should be interlinked
[ https://issues.apache.org/jira/browse/FLINK-17858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Till Rohrmann closed FLINK-17858.
---------------------------------
Resolution: Later
Closing this ticket because of inactivity. Please re-open this ticket if you want to work on it.
> Yarn mode, windows and linux environment should be interlinked
> --------------------------------------------------------------
>
> Key: FLINK-17858
> URL: https://issues.apache.org/jira/browse/FLINK-17858
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
> Affects Versions: 1.10.0
> Reporter: ChengWei Ye
> Priority: Minor
>
> my flink version: 1.10.0
> my yarn version: 2.7.6
> When I use windows idea to try to submit a job to Linux's yarn cluster using java, am container throws an exception "Error: Could not find or load main class org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint".
> Then find out that the CLASSPATH and _FLINK_CLASSPATH delimiter of the container startup script "launch_container.sh" are different, windows is ";", linux is ":", so the container cannot get the dependent packages.
> I think the operating environment should be left to Yarn to judge.
> Locate in the flink-yarn module
> 1. The startAppMaster method of org.apache.flink.yarn.YarnClusterDescriptor
> Change "File.pathSeparator" to "ApplicationConstants.CLASS_PATH_SEPARATOR"
> Yarn will replace special characters according to the environment, but this only solves _FLINK_CLASSPATH
>
> {code:java}
> private ApplicationReport startAppMaster(.....) {
> .....
> StringBuilder classPathBuilder = new StringBuilder();
> if (userJarInclusion == YarnConfigOptions.UserJarInclusion.FIRST) {
> for (String userClassPath : userClassPaths) {
> // here
> classPathBuilder.append(userClassPath).append(File.pathSeparator);
> }
> }
> for (String classPath : systemClassPaths) {
> // here
> classPathBuilder.append(classPath).append(File.pathSeparator);
> }
> // Setup jar for ApplicationMaster
> .....
> // here
> classPathBuilder.append(flinkJarPath.getName()).append(File.pathSeparator);
> .....
> }{code}
>
> 2. The addToEnvironment method of org.apache.flink.yarn.Utils
> Change "File.pathSeparator" to "ApplicationConstants.CLASS_PATH_SEPARATOR"
> Can solve the CLASSPATH
> {code:java}
> public static void addToEnvironment(Map<String, String> environment,
> String variable, String value) {
> String val = environment.get(variable);
> if (val == null) {
> val = value;
> } else {
> // here
> val = val + File.pathSeparator + value;
> }
> environment.put(StringInterner.weakIntern(variable),
> StringInterner.weakIntern(val));
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)