Cannot find Native Library in "cluster" deploy-mode

[jtgenesis <jt...@gmail.com>]

hey guys,

I'm hoping someone could provide some assistance. I'm having issues
(UnsatisfiedLinkError) calling some native libraries when submitting the
application in "cluster" mode. When running the application in local mode,
the application runs fine. Here's what my setup looks like.

The .so files are stored in the same location across all cluster nodes. The
node that submits the application has its LD_LIBRARY_PATH pointing to the
native libraries. Within the application i set the environment variables
like so:

SparkContext(new SparkConf()
            .setAppName("SparkDriver")
            .setExecutorEnv("LD_LIBRARY_PATH", sys.env("LD_LIBRARY_PATH"))

I thought the .setExecutorEnv would give each node the correct path to the
native libraries, but I guess not. 

I'm not really sure what I'm missing in my setup, any help is greatly
appreciated.

Thanks!





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Cannot-find-Native-Library-in-cluster-deploy-mode-tp28072.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Cannot find Native Library in "cluster" deploy-mode

[jtgenesis <jt...@gmail.com>]

This link helped solved the issue for me!

http://permalink.gmane.org/gmane.comp.lang.scala.spark.user/22700 



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Cannot-find-Native-Library-in-cluster-deploy-mode-tp28072p28081.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Cannot find Native Library in "cluster" deploy-mode

[jtgenesis <jt...@gmail.com>]

thanks for the feedback.

I tried your suggestion using the --files option, but still no luck. I've
also checked and made sure that the libraries the .so files need are also
there. When I look at the SparkUI/Environment tab, the
`spark.executorEnv.LD_LIBRARY_PATH` is pointing to the correct libraries.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Cannot-find-Native-Library-in-cluster-deploy-mode-tp28072p28080.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Cannot find Native Library in "cluster" deploy-mode

[Mo Tao <my...@qq.com>]

Hi jtgenesis,

UnsatisfiedLinkError could be caused by the missing library that your .so
files require, so you may have a look at the exception message.

You can also try setExecutorEnv("LD_LIBRARY_PATH", ".:" +
sys.env("LD_LIBRARY_PATH")) and then submit your job with your .so files
using the --files option. Spark would put these .so files in the working
directory of each exectuor.

Best



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Cannot-find-Native-Library-in-cluster-deploy-mode-tp28072p28074.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org