[jira] [Comment Edited] (IGNITE-3185) Hadoop: Improve Hadoop JAR search logic.

["Ivan Veselovsky (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IGNITE-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334440#comment-15334440 ] 

Ivan Veselovsky edited comment on IGNITE-3185 at 6/16/16 6:56 PM:
------------------------------------------------------------------

As I understand, in case of almost all non-apache distributions the hdfs & mapred homes are not sub-folders of HADOOP_HOME. 
HADOOP_HOME by its original sense represents hadoop client base folder (${HADOOP_HOME}/bin/hadoop executable file assumed to exist). 
So, common , hdfs and mapred base folders typically have some common root , but this root is not a value of HADOOP_HOME.
E.g. out docs v1.0 say ( http://apacheignite.gridgain.org/v1.0/docs/installing-on-hortonworks-hdp ) :
{code}
HDP=/usr/hdp/current
export HADOOP_HOME=$HDP/hadoop-client/
export HADOOP_COMMON_HOME=$HDP/hadoop-client/
export HADOOP_HDFS_HOME=$HDP/hadoop-hdfs-client/ 
export HADOOP_MAPRED_HOME=$HDP/hadoop-mapreduce-client/
{code}

Note that more late version of that page defines HADOOP_HOME differently ( https://apacheignite-fs.readme.io/docs/installing-on-hortonworks-hdp ): here HADOOP_HOME is not a client home folder. 

I would suggest another solution: specify a directory location profile for each supported distribution. The user will just need to chose the right profile name to get correct layout.


was (Author: iveselovskiy):
As I understand, in case of almost all non-apache distributions the hdfs & mapred homes are not sub-folders of HADOOP_HOME. 
HADOOP_HOME by its original sense represents hadoop client base folder (${HADOOP_HOME}/bin/hadoop executable file assumed to exist). 
So, common , hdfs and mapred base folders typically have some common root , but this root is not a value of HADOOP_HOME.
E.g. out docs v1.0 say ( http://apacheignite.gridgain.org/v1.0/docs/installing-on-hortonworks-hdp ) :
{code}
HDP=/usr/hdp/current
export HADOOP_HOME=$HDP/hadoop-client/
export HADOOP_COMMON_HOME=$HDP/hadoop-client/
export HADOOP_HDFS_HOME=$HDP/hadoop-hdfs-client/ 
export HADOOP_MAPRED_HOME=$HDP/hadoop-mapreduce-client/
{code}

Note that more late version of that page defines HADOOP_HOME differently ( https://apacheignite-fs.readme.io/docs/installing-on-hortonworks-hdp ): here HADOOP_HOME is not a client home folder. 

> Hadoop: Improve Hadoop JAR search logic.
> ----------------------------------------
>
>                 Key: IGNITE-3185
>                 URL: https://issues.apache.org/jira/browse/IGNITE-3185
>             Project: Ignite
>          Issue Type: Bug
>          Components: hadoop
>    Affects Versions: 1.5.0.final
>            Reporter: Vladimir Ozerov
>            Assignee: Ivan Veselovsky
>             Fix For: 1.7
>
>
> *Problem*
> Currently our {{HadoopClassLoader}} is trying to find common/hdfs/mapred JARs automatically if {{HADOOP_HOME}} is set. However, it checks only for standard Apache Hadoop distribution paths (/share/hadoop/*). For other distributions (CDH, HDP), user has to bother with additional env variables manually.
> *Solution*
> Try to search for the following folders as well:
> $HADOOP_HOME/hadoop
> $HADOOP_HOME/hadoop-hdfs
> $HADOOP_HOME/hadoop-mapreduce
> If all these folders are found, then this is either HDP or CDH deployment and we can create the classpath without additional env vars.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)