You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Yana Kadiyska <ya...@gmail.com> on 2015/06/30 16:17:59 UTC

Difference between spark-defaults.conf and SparkConf.set

Hi folks, running into a pretty strange issue:

I'm setting
spark.executor.extraClassPath
spark.driver.extraClassPath

to point to some external JARs. If I set them in spark-defaults.conf
everything works perfectly.
However, if I remove spark-defaults.conf and just create a SparkConf and
call
.set("spark.executor.extraClassPath","...)
.set("spark.driver.extraClassPath",...)

I get ClassNotFound exceptions from Hadoop Conf:

Caused by: java.lang.ClassNotFoundException: Class
org.apache.hadoop.fs.ceph.CephFileSystem not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1493)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1585)

​

This seems like a bug to me -- or does spark-defaults.conf somehow get
processed differently?

I have dumped out sparkConf.toDebugString and in both cases
(spark-defaults.conf/in code sets) it seems to have the same values in it...

Re: Difference between spark-defaults.conf and SparkConf.set

Posted by "Kelly, Jonathan" <jo...@amazon.com>.
I assume the exception happens in the driver? I think by the time it's executing your code to add jars to the classpath, it's already too late, whereas when you put them in spark-defaults.conf I'm pretty sure spark-submit does something special to pick up those properties so that it can append to the classpath before it even launches the driver's java process.

Sent from Nine<http://www.9folders.com/>

From: Yana Kadiyska <ya...@gmail.com>
Sent: Jun 30, 2015 7:18 AM
To: user@spark.apache.org
Subject: Difference between spark-defaults.conf and SparkConf.set

Hi folks, running into a pretty strange issue:

I'm setting
spark.executor.extraClassPath
spark.driver.extraClassPath

to point to some external JARs. If I set them in spark-defaults.conf everything works perfectly.
However, if I remove spark-defaults.conf and just create a SparkConf and call
.set("spark.executor.extraClassPath","...)
.set("spark.driver.extraClassPath",...)

I get ClassNotFound exceptions from Hadoop Conf:


Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.ceph.CephFileSystem not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1493)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1585)


?

This seems like a bug to me -- or does spark-defaults.conf somehow get processed differently?

I have dumped out sparkConf.toDebugString and in both cases (spark-defaults.conf/in code sets) it seems to have the same values in it...

Re: Difference between spark-defaults.conf and SparkConf.set

Posted by Akhil Das <ak...@sigmoidanalytics.com>.
.addJar works for me when i run it as a stand-alone application (without
using spark-submit)

Thanks
Best Regards

On Tue, Jun 30, 2015 at 7:47 PM, Yana Kadiyska <ya...@gmail.com>
wrote:

> Hi folks, running into a pretty strange issue:
>
> I'm setting
> spark.executor.extraClassPath
> spark.driver.extraClassPath
>
> to point to some external JARs. If I set them in spark-defaults.conf
> everything works perfectly.
> However, if I remove spark-defaults.conf and just create a SparkConf and
> call
> .set("spark.executor.extraClassPath","...)
> .set("spark.driver.extraClassPath",...)
>
> I get ClassNotFound exceptions from Hadoop Conf:
>
> Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.ceph.CephFileSystem not found
>         at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1493)
>         at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1585)
>
> ​
>
> This seems like a bug to me -- or does spark-defaults.conf somehow get
> processed differently?
>
> I have dumped out sparkConf.toDebugString and in both cases
> (spark-defaults.conf/in code sets) it seems to have the same values in it...
>