You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Bruno Mahé (Created JIRA)" <ji...@apache.org> on 2011/12/03 00:59:40 UTC
[jira] [Created] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH needs
to be overridable
yarn APPLICATION_CLASSPATH needs to be overridable
--------------------------------------------------
Key: MAPREDUCE-3505
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.23.1
Reporter: Bruno Mahé
Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
For convenience, here is its current content:
{quote}
/**
* Classpath for typical applications.
*/
public static final String[] APPLICATION_CLASSPATH =
new String[] {
"$HADOOP_CONF_DIR",
"$HADOOP_COMMON_HOME/share/hadoop/common/*",
"$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
"$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
"$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
"$YARN_HOME/modules/*",
"$YARN_HOME/lib/*"
};
{quote}
Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH needs
to be overridable
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-3505:
------------------------------------
Status: Patch Available (was: Open)
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.23.1
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Attachments: MAPREDUCE-3505.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH needs
to be overridable
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-3505:
------------------------------------
Attachment: MAPREDUCE-3505.patch
A possible way of addressing that is to push this classpath as part of configuration, where the default value will be the current value in ApplicationConstants.APPLICATION_CLASSPATH. This value can be then overridden (if needed) in the conf files.
The attached patch does that.
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.23.1
> Reporter: Bruno Mahé
> Labels: bigtop
> Attachments: MAPREDUCE-3505.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193078#comment-13193078 ]
Hudson commented on MAPREDUCE-3505:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #969 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/969/])
MAPREDUCE-3505. yarn APPLICATION_CLASSPATH needs to be overridable. (ahmed via tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235391
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3505.patch, MAPREDUCE-3505_rev2.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188045#comment-13188045 ]
Arun C Murthy commented on MAPREDUCE-3505:
------------------------------------------
Seems fair. We should add a test case though.
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3505.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH needs
to be overridable
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-3505:
------------------------------------
Status: Patch Available (was: Open)
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3505.patch, MAPREDUCE-3505_rev2.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Roman Shaposhnik (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13177525#comment-13177525 ]
Roman Shaposhnik commented on MAPREDUCE-3505:
---------------------------------------------
@Arun,
yes it would be very nice if this can be auto-detected. Short of that, an alternative that would
avoid run-time configuration might be compile-time configuration (something like an autogenerated class a'la VersionInfo).
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Attachments: MAPREDUCE-3505.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192347#comment-13192347 ]
Hudson commented on MAPREDUCE-3505:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1578 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1578/])
MAPREDUCE-3505. yarn APPLICATION_CLASSPATH needs to be overridable. (ahmed via tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235391
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3505.patch, MAPREDUCE-3505_rev2.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193012#comment-13193012 ]
Hudson commented on MAPREDUCE-3505:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #936 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/936/])
MAPREDUCE-3505. yarn APPLICATION_CLASSPATH needs to be overridable. (ahmed via tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235391
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3505.patch, MAPREDUCE-3505_rev2.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193032#comment-13193032 ]
Hudson commented on MAPREDUCE-3505:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #171 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/171/])
Merge -r 1235390:1235391 from trunk to branch. FIXES: MAPREDUCE-3505
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235397
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3505.patch, MAPREDUCE-3505_rev2.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH needs
to be overridable
Posted by "Bruno Mahé (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bruno Mahé updated MAPREDUCE-3505:
----------------------------------
Description:
Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
For convenience, here is its current content:
{noformat}
/**
* Classpath for typical applications.
*/
public static final String[] APPLICATION_CLASSPATH =
new String[] {
"$HADOOP_CONF_DIR",
"$HADOOP_COMMON_HOME/share/hadoop/common/*",
"$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
"$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
"$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
"$YARN_HOME/modules/*",
"$YARN_HOME/lib/*"
};
{noformat}
Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
was:
Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
For convenience, here is its current content:
{quote}
/**
* Classpath for typical applications.
*/
public static final String[] APPLICATION_CLASSPATH =
new String[] {
"$HADOOP_CONF_DIR",
"$HADOOP_COMMON_HOME/share/hadoop/common/*",
"$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
"$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
"$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
"$YARN_HOME/modules/*",
"$YARN_HOME/lib/*"
};
{quote}
Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.23.1
> Reporter: Bruno Mahé
> Labels: bigtop
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Roman Shaposhnik (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182753#comment-13182753 ]
Roman Shaposhnik commented on MAPREDUCE-3505:
---------------------------------------------
+1 on the current patch. Can we, please, have it committed? I understand the interest in a more general solution, but committing this would be an incremental positive step.
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Attachments: MAPREDUCE-3505.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192365#comment-13192365 ]
Hudson commented on MAPREDUCE-3505:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #428 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/428/])
Merge -r 1235390:1235391 from trunk to branch. FIXES: MAPREDUCE-3505
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235397
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3505.patch, MAPREDUCE-3505_rev2.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Bruno Mahé (Commented JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182747#comment-13182747 ]
Bruno Mahé commented on MAPREDUCE-3505:
---------------------------------------
+1 to the patch.
It solves a big issue I have had to hack around. This patch makes it way better than the current situation and we can always improve it later on.
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Attachments: MAPREDUCE-3505.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH needs
to be overridable
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-3505:
------------------------------------
Attachment: MAPREDUCE-3505_rev2.patch
Attaching an updated patch where a new test case is added to TestMRApps (Per Arun's comment). Thanks!
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3505.patch, MAPREDUCE-3505_rev2.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH needs
to be overridable
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-3505:
------------------------------------
Fix Version/s: 0.23.1
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3505.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167458#comment-13167458 ]
Hadoop QA commented on MAPREDUCE-3505:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506990/MAPREDUCE-3505.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1420//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1420//console
This message is automatically generated.
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.23.1
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Attachments: MAPREDUCE-3505.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183517#comment-13183517 ]
Alejandro Abdelnur commented on MAPREDUCE-3505:
-----------------------------------------------
+1
@Arun, I agree that we should try to infer this somehow in an automatic why. Until we do so, would it be OK to have this patch as a temporary workaround? (effectively this patch is just moving the constructed classpath from being hardcoded in the Java code to be in the yarn config file).
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Attachments: MAPREDUCE-3505.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192391#comment-13192391 ]
Hudson commented on MAPREDUCE-3505:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1595 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1595/])
MAPREDUCE-3505. yarn APPLICATION_CLASSPATH needs to be overridable. (ahmed via tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235391
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3505.patch, MAPREDUCE-3505_rev2.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192359#comment-13192359 ]
Hudson commented on MAPREDUCE-3505:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1651 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1651/])
MAPREDUCE-3505. yarn APPLICATION_CLASSPATH needs to be overridable. (ahmed via tucu)
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235391
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3505.patch, MAPREDUCE-3505_rev2.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH needs
to be overridable
Posted by "Alejandro Abdelnur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated MAPREDUCE-3505:
------------------------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Ahmed. Committed to trunk and branch-0.23
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3505.patch, MAPREDUCE-3505_rev2.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193056#comment-13193056 ]
Hudson commented on MAPREDUCE-3505:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #149 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/149/])
Merge -r 1235390:1235391 from trunk to branch. FIXES: MAPREDUCE-3505
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235397
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3505.patch, MAPREDUCE-3505_rev2.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13177056#comment-13177056 ]
Arun C Murthy commented on MAPREDUCE-3505:
------------------------------------------
The APPLICATION_CLASSPATH represents 'standard dependencies' (common, hdfs and yarn) needed by all applications.
What does bigtop want this to be?
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Attachments: MAPREDUCE-3505.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191887#comment-13191887 ]
Hadoop QA commented on MAPREDUCE-3505:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511621/MAPREDUCE-3505_rev2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1656//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1656//console
This message is automatically generated.
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3505.patch, MAPREDUCE-3505_rev2.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188039#comment-13188039 ]
Alejandro Abdelnur commented on MAPREDUCE-3505:
-----------------------------------------------
Unless there are objections I'll be committing this patch later today.
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3505.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH needs
to be overridable
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3505:
-------------------------------------
Affects Version/s: (was: 0.23.1)
Status: Open (was: Patch Available)
My preference would be to avoid using configuration for this. Maybe we can detect the paths from the NM's classpath or some clever hack? Thus, we could avoid the hardcoding too...
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Attachments: MAPREDUCE-3505.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192353#comment-13192353 ]
Hudson commented on MAPREDUCE-3505:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #412 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/412/])
Merge -r 1235390:1235391 from trunk to branch. FIXES: MAPREDUCE-3505
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235397
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3505.patch, MAPREDUCE-3505_rev2.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH
needs to be overridable
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192362#comment-13192362 ]
Hudson commented on MAPREDUCE-3505:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #403 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/403/])
Merge -r 1235390:1235391 from trunk to branch. FIXES: MAPREDUCE-3505
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235397
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3505.patch, MAPREDUCE-3505_rev2.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3505) yarn APPLICATION_CLASSPATH needs
to be overridable
Posted by "Ahmed Radwan (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan reassigned MAPREDUCE-3505:
---------------------------------------
Assignee: Ahmed Radwan
> yarn APPLICATION_CLASSPATH needs to be overridable
> --------------------------------------------------
>
> Key: MAPREDUCE-3505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0, 0.23.1
> Reporter: Bruno Mahé
> Assignee: Ahmed Radwan
> Labels: bigtop
> Attachments: MAPREDUCE-3505.patch
>
>
> Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
> If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
> The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
> For convenience, here is its current content:
> {noformat}
> /**
> * Classpath for typical applications.
> */
> public static final String[] APPLICATION_CLASSPATH =
> new String[] {
> "$HADOOP_CONF_DIR",
> "$HADOOP_COMMON_HOME/share/hadoop/common/*",
> "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
> "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
> "$YARN_HOME/modules/*",
> "$YARN_HOME/lib/*"
> };
> {noformat}
> Not all deployment scenarii fit in this layout and therefore we need a standardized way to customize this class path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira