You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Zhankun Tang (JIRA)" <ji...@apache.org> on 2018/11/01 03:34:00 UTC
[jira] [Updated] (YARN-7277) Container Launch expand environment
needs to consider bracket matching
[ https://issues.apache.org/jira/browse/YARN-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhankun Tang updated YARN-7277:
-------------------------------
Attachment: YARN-7277-trunk.004.patch
> Container Launch expand environment needs to consider bracket matching
> ----------------------------------------------------------------------
>
> Key: YARN-7277
> URL: https://issues.apache.org/jira/browse/YARN-7277
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Reporter: balloons
> Assignee: Zhankun Tang
> Priority: Critical
> Attachments: YARN-7277-trunk.001.patch, YARN-7277-trunk.002.patch, YARN-7277-trunk.003.patch, YARN-7277-trunk.004.patch
>
>
> The SPARK application I submitted always failed and I finally found that the commands I specified to launch AM Container were changed by NM.
> *The following is part of the excerpt I submitted to RM to see the command:*
> {code:java}
> *'{\"handler\":\"FILLER\",\"inputTable\":\"engine_arch.adult_train\",\"outputTable\":[\"ether_features_filler_\$experimentId_\$taskId_out0\"],\"params\":{\"age\":{\"param\":[\"0\"]}}}'*
> {code}
> *The following is an excerpt from the corresponding command used when I observe the NM launch container:*
> {code:java}
> *'{\"handler\":\"FILLER\",\"inputTable\":\"engine_arch.adult_train\",\"outputTable\":[\"ether_features_filler_\$experimentId_\$taskId_out0\"],\"params\":{\"age\":{\"param\":[\"0\"]}*
> {code}
> Finally, I found that NM made the following transformation in launch container which led to this situation:
> {code:java}
> @VisibleForTesting
> public static String expandEnvironment(String var,
> Path containerLogDir) {
> var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR,
> containerLogDir.toString());
> var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR,
> File.pathSeparator);
> // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced
> // as %VAR% and on Linux replaced as "$VAR"
> if (Shell.WINDOWS) {
> var = var.replaceAll("(\\{\\{)|(\\}\\})", "%");
> } else {
> var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$");
> *var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");*
> }
> return var;
> }
> {code}
> I think this is a Bug that doesn't even consider the pairing of "*PARAMETER_EXPANSION_LEFT*" and "*PARAMETER_EXPANSION_RIGHT*" when substituting. But simply substituting for simple violence.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org