[jira] [Created] (SPARK-5241) spark-ec2 spark init scripts do not handle all hadoop (or tachyon?) dependencies correctly

["Florian Verhein (JIRA)" <ji...@apache.org>]

Florian Verhein created SPARK-5241:
--------------------------------------

             Summary: spark-ec2 spark init scripts do not handle all hadoop (or tachyon?) dependencies correctly
                 Key: SPARK-5241
                 URL: https://issues.apache.org/jira/browse/SPARK-5241
             Project: Spark
          Issue Type: Bug
          Components: Build, EC2
            Reporter: Florian Verhein



spark-ec2/spark/init.sh doesn't completely adhere to hadoop dependencies. This may also be an issue for tachyon dependencies. Related: tachyon appears require builds against the right version of hadoop also (probably causes this: SPARK-3185). 

Applies to the spark build from git checkout in spark/init.sh (I suspect this should also be changed to using mvn as that's the reference build according to the docs?).

May apply to pre-built spark in spark/init.sh as well, but I'm not sure about this. E.g. I thought that the hadoop2.4 and cdh4.2 builds of spark are different.

Also note that hadoop native is built from hadoop 2.4.1 on the AMI, and this is used regardless of HADOOP_MAJOR_VERSION in the *-hdfs modules.

Tachyon is hard coded to 0.4.1 (which is probably built against hadoop1.x?)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org