You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Eric Friedman <er...@gmail.com> on 2014/07/17 22:25:07 UTC

replacement for SPARK_LIBRARY_PATH ?

I used to use SPARK_LIBRARY_PATH to specify the location of native libs
for lzo compression when using spark 0.9.0.

The references to that environment variable have disappeared from the docs
for
spark 1.0.1 and it's not clear how to specify the location for lzo.

Any guidance?

Re: replacement for SPARK_LIBRARY_PATH ?

Posted by Koert Kuipers <ko...@tresata.com>.

but be aware that spark-defaults.conf is only used if you use spark-submit
On Jul 17, 2014 4:29 PM, "Zongheng Yang" <zo...@gmail.com> wrote:

> One way is to set this in your conf/spark-defaults.conf:
>
> spark.executor.extraLibraryPath /path/to/native/lib
>
> The key is documented here:
> http://spark.apache.org/docs/latest/configuration.html
>
> On Thu, Jul 17, 2014 at 1:25 PM, Eric Friedman
> <er...@gmail.com> wrote:
> > I used to use SPARK_LIBRARY_PATH to specify the location of native libs
> > for lzo compression when using spark 0.9.0.
> >
> > The references to that environment variable have disappeared from the
> docs
> > for
> > spark 1.0.1 and it's not clear how to specify the location for lzo.
> >
> > Any guidance?
>

Re: replacement for SPARK_LIBRARY_PATH ?

Posted by Zongheng Yang <zo...@gmail.com>.

One way is to set this in your conf/spark-defaults.conf:

spark.executor.extraLibraryPath /path/to/native/lib

The key is documented here:
http://spark.apache.org/docs/latest/configuration.html

On Thu, Jul 17, 2014 at 1:25 PM, Eric Friedman
<er...@gmail.com> wrote:
> I used to use SPARK_LIBRARY_PATH to specify the location of native libs
> for lzo compression when using spark 0.9.0.
>
> The references to that environment variable have disappeared from the docs
> for
> spark 1.0.1 and it's not clear how to specify the location for lzo.
>
> Any guidance?