You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Zhankun Tang (JIRA)" <ji...@apache.org> on 2018/11/02 04:06:00 UTC
[jira] [Comment Edited] (YARN-7277) Container Launch expand
environment needs to consider bracket matching
[ https://issues.apache.org/jira/browse/YARN-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672547#comment-16672547 ]
Zhankun Tang edited comment on YARN-7277 at 11/2/18 4:05 AM:
-------------------------------------------------------------
[~cheersyang]. The error output is quite weird. And output is different when I run "mvn test -Dtest=TestContainerLaunch" in different directory.
In nodemanager dir, it will fail as Yetus. But in parent folder hadoop-yarn, it will succeed.
Has submitted a JIRA https://issues.apache.org/jira/browse/YETUS-711 asking for help.
was (Author: tangzhankun):
[~cheersyang]. The error output is quit weird. And output is different when I run "mvn test -Dtest=TestContainerLaunch" in different directory.
In nodemanager dir, it will fail as Yetus. But in parent folder hadoop-yarn, it will succeed.
Has submitted a JIRA https://issues.apache.org/jira/browse/YETUS-711 asking for help.
> Container Launch expand environment needs to consider bracket matching
> ----------------------------------------------------------------------
>
> Key: YARN-7277
> URL: https://issues.apache.org/jira/browse/YARN-7277
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Reporter: balloons
> Assignee: Zhankun Tang
> Priority: Critical
> Attachments: YARN-7277-trunk.001.patch, YARN-7277-trunk.002.patch, YARN-7277-trunk.003.patch, YARN-7277-trunk.004.patch, YARN-7277-trunk.005.patch
>
>
> The SPARK application I submitted always failed and I finally found that the commands I specified to launch AM Container were changed by NM.
> *The following is part of the excerpt I submitted to RM to see the command:*
> {code:java}
> *'{\"handler\":\"FILLER\",\"inputTable\":\"engine_arch.adult_train\",\"outputTable\":[\"ether_features_filler_\$experimentId_\$taskId_out0\"],\"params\":{\"age\":{\"param\":[\"0\"]}}}'*
> {code}
> *The following is an excerpt from the corresponding command used when I observe the NM launch container:*
> {code:java}
> *'{\"handler\":\"FILLER\",\"inputTable\":\"engine_arch.adult_train\",\"outputTable\":[\"ether_features_filler_\$experimentId_\$taskId_out0\"],\"params\":{\"age\":{\"param\":[\"0\"]}*
> {code}
> Finally, I found that NM made the following transformation in launch container which led to this situation:
> {code:java}
> @VisibleForTesting
> public static String expandEnvironment(String var,
> Path containerLogDir) {
> var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR,
> containerLogDir.toString());
> var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR,
> File.pathSeparator);
> // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced
> // as %VAR% and on Linux replaced as "$VAR"
> if (Shell.WINDOWS) {
> var = var.replaceAll("(\\{\\{)|(\\}\\})", "%");
> } else {
> var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$");
> *var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");*
> }
> return var;
> }
> {code}
> I think this is a Bug that doesn't even consider the pairing of "*PARAMETER_EXPANSION_LEFT*" and "*PARAMETER_EXPANSION_RIGHT*" when substituting. But simply substituting for simple violence.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org