You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by "Ivan Veselovsky (JIRA)" <ji...@apache.org> on 2015/02/27 13:00:16 UTC
[jira] [Updated] (IGNITE-372) improve Ignite node Hadoop classpath
composition reliability
[ https://issues.apache.org/jira/browse/IGNITE-372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Veselovsky updated IGNITE-372:
-----------------------------------
Description:
User encountered a problem with Hadoop mapred jobs execution when incorrectly set in environment
{code}
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
{code}
. That caused incorrect GRIDGAIN_HADOOP_CLASSPATH and , as a consequence, job failure with NoClassDefFoundError.
Suggestions to improve reliability and tracability of the Hadoop classpath composition:
1) Simplify the logic of the classpath composition: do not take into account HADOOP_COMMON_HOME, HADOOP_MAPRED_HOME, HADOOP_HDFS_HOME, also do not read /etc/default/hadoop .
Compose the classpath basing only on the value of $HADOOP_HOME.
2) when GRIDGAIN_HADOOP_CLASSPATH composed, print out its value if ignite.sh as run with "-v" option.
3) add the Hadoop classpath validation as a part of Node startup sequence: try to find (load?) at least one class from each of the key Hadoop APIs (FS, MapRed, Yarn API).
was:
User encountered a problem with Hadoop mapred jobs execution when incorrectly set in environment
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
. That caused incorrect GRIDGAIN_HADOOP_CLASSPATH and , as a consequence, job failure with NoClassDefFoundError.
Suggestions to improve reliability and tracability of the Hadoop classpath composition:
1) Simplify the logic of the classpath composition: do not take into account HADOOP_COMMON_HOME, HADOOP_MAPRED_HOME, HADOOP_HDFS_HOME, also do not read /etc/default/hadoop .
Compose the classpath basing only on the value of $HADOOP_HOME.
2) when GRIDGAIN_HADOOP_CLASSPATH composed, print out its value if ignite.sh as run with "-v" option.
3) add the Hadoop classpath validation as a part of Node startup sequence: try to find (load?) at least one class from each of the key Hadoop APIs (FS, MapRed, Yarn API).
> improve Ignite node Hadoop classpath composition reliability
> -------------------------------------------------------------
>
> Key: IGNITE-372
> URL: https://issues.apache.org/jira/browse/IGNITE-372
> Project: Ignite
> Issue Type: Improvement
> Affects Versions: sprint-2
> Reporter: Ivan Veselovsky
> Assignee: Ivan Veselovsky
> Priority: Minor
>
> User encountered a problem with Hadoop mapred jobs execution when incorrectly set in environment
> {code}
> export HADOOP_MAPRED_HOME=${HADOOP_HOME}
> export HADOOP_COMMON_HOME=${HADOOP_HOME}
> export HADOOP_HDFS_HOME=${HADOOP_HOME}
> {code}
> . That caused incorrect GRIDGAIN_HADOOP_CLASSPATH and , as a consequence, job failure with NoClassDefFoundError.
> Suggestions to improve reliability and tracability of the Hadoop classpath composition:
> 1) Simplify the logic of the classpath composition: do not take into account HADOOP_COMMON_HOME, HADOOP_MAPRED_HOME, HADOOP_HDFS_HOME, also do not read /etc/default/hadoop .
> Compose the classpath basing only on the value of $HADOOP_HOME.
> 2) when GRIDGAIN_HADOOP_CLASSPATH composed, print out its value if ignite.sh as run with "-v" option.
> 3) add the Hadoop classpath validation as a part of Node startup sequence: try to find (load?) at least one class from each of the key Hadoop APIs (FS, MapRed, Yarn API).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)