[jira] [Created] (BIGTOP-2959) Add SPARK_DIST_CLASSPATH to include libraries for HDFS and YARN

["YoungWoo Kim (JIRA)" <ji...@apache.org>]

YoungWoo Kim created BIGTOP-2959:
------------------------------------

             Summary: Add SPARK_DIST_CLASSPATH to include libraries for HDFS and YARN
                 Key: BIGTOP-2959
                 URL: https://issues.apache.org/jira/browse/BIGTOP-2959
             Project: Bigtop
          Issue Type: Improvement
          Components: build, spark
    Affects Versions: 1.2.1
            Reporter: YoungWoo Kim
            Priority: Minor


It might be a better way to use SPARK_DIST_CLASSPATH for including the Hadoop libs instead of  CLASSPATH.
{code}
export SPARK_DIST_CLASSPATH=$(hadoop classpath)
{code}

Refs. Using Spark's "Hadoop Free" Build, https://spark.apache.org/docs/2.1.1/hadoop-provided.html

Current spark-env.sh:
{code}
# Let's make sure that all needed hadoop libs are added properly
export CLASSPATH="$CLASSPATH:$HADOOP_HOME/*:$HADOOP_HDFS_HOME/*:$HADOOP_YARN_HOME/*:$HADOOP_MAPRED_HOME/*"
{code} 




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)