[jira] [Resolved] (BIGTOP-2959) Add SPARK_DIST_CLASSPATH to include libraries for HDFS and YARN

["YoungWoo Kim (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BIGTOP-2959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

YoungWoo Kim resolved BIGTOP-2959.
----------------------------------
    Resolution: Fixed

All patches related Hive, Spark and Zeppelin are pushed to master. Let me see how it goes...

> Add SPARK_DIST_CLASSPATH to include libraries for HDFS and YARN
> ---------------------------------------------------------------
>
>                 Key: BIGTOP-2959
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-2959
>             Project: Bigtop
>          Issue Type: Improvement
>          Components: build, spark
>    Affects Versions: 1.2.1
>            Reporter: YoungWoo Kim
>            Assignee: YoungWoo Kim
>            Priority: Minor
>             Fix For: 1.3.0
>
>
> It might be a better way to use SPARK_DIST_CLASSPATH for including the Hadoop libs instead of  CLASSPATH.
> {code}
> export SPARK_DIST_CLASSPATH=$(hadoop classpath)
> {code}
> Refs. Using Spark's "Hadoop Free" Build, https://spark.apache.org/docs/2.1.1/hadoop-provided.html
> Current spark-env.sh:
> {code}
> # Let's make sure that all needed hadoop libs are added properly
> export CLASSPATH="$CLASSPATH:$HADOOP_HOME/*:$HADOOP_HDFS_HOME/*:$HADOOP_YARN_HOME/*:$HADOOP_MAPRED_HOME/*"
> {code} 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)