You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Koert Kuipers <ko...@tresata.com> on 2014/09/15 17:16:52 UTC
Found both spark.driver.extraClassPath and SPARK_CLASSPATH
in spark 1.1.0 i get this error:
2014-09-14 23:17:01 ERROR actor.OneForOneStrategy: Found both
spark.driver.extraClassPath and SPARK_CLASSPATH. Use only the former.
i checked my application. i do not set spark.driver.extraClassPath or
SPARK_CLASSPATH.
SPARK_CLASSPATH is set in spark-env.sh since the machine is a worker and
this is how lzo is added to classpath. as a user i cannot modify this.
looking at the logs i see the value of SPARK_CLASSPATH ends up in
spark.driver.extraClassPath, but thats not done by me. i presume some logic
in spark-submit does this.
Re: Found both spark.driver.extraClassPath and SPARK_CLASSPATH
Posted by Koert Kuipers <ko...@tresata.com>.
i have found no way around this. basically this makes SPARK_CLASSPATH
unusable. and the alternative for enabling lzo on a cluster is not
reasonable.
one has to set in spark-defaults.conf:
spark.executor.extraClassPath
/usr/lib/hadoop/lib/hadoop-lzo-cdh4-0.4.15-gplextras.jar
spark.executor.extraLibraryPath
/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native
and then for every job submitted one has to do using spark-submit (i get
errors if i leave this out):
--driver-class-path /usr/lib/hadoop/lib/hadoop-lzo-cdh4-0.4.15-gplextras.jar
--driver-library-path
/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native
i think the settings in spark-defaults.conf are fine, but expecting the
user to set the driver-class-path and driver-library-path like that is
cumbersome and requires knowledge of locations of certain files on the
cluster. thats not workable to me.
On Mon, Sep 15, 2014 at 11:16 AM, Koert Kuipers <ko...@tresata.com> wrote:
> in spark 1.1.0 i get this error:
>
> 2014-09-14 23:17:01 ERROR actor.OneForOneStrategy: Found both
> spark.driver.extraClassPath and SPARK_CLASSPATH. Use only the former.
>
> i checked my application. i do not set spark.driver.extraClassPath or
> SPARK_CLASSPATH.
>
> SPARK_CLASSPATH is set in spark-env.sh since the machine is a worker and
> this is how lzo is added to classpath. as a user i cannot modify this.
>
> looking at the logs i see the value of SPARK_CLASSPATH ends up in
> spark.driver.extraClassPath, but thats not done by me. i presume some logic
> in spark-submit does this.
>
>