You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Christoph Böhm <li...@gmx.net> on 2014/03/26 15:52:10 UTC
Re: spark-shell on standalone cluster gives error " no mesos in
java.library.path"
Hi,
I have a similar issue like the user below:
I’m running Spark 0.8.1 (standalone). When I test the streaming NetworkWordCount example as in the docs with local[2] it works fine. As soon as I want to connect to my cluster using [NetworkWordCount master …] it says:
---
Failed to load native Mesos library from /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Exception in thread "main" java.lang.UnsatisfiedLinkError: no mesos in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)
at java.lang.Runtime.loadLibrary0(Runtime.java:849)
at java.lang.System.loadLibrary(System.java:1088)
at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:52)
at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:64)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:260)
at org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:559)
at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:84)
at org.apache.spark.streaming.api.java.JavaStreamingContext.<init>(JavaStreamingContext.scala:76)
at org.apache.spark.streaming.examples.JavaNetworkWordCount.main(JavaNetworkWordCount.java:50)
---
I built mesos 0.13 and added the MESOS_NATIVE_LIBRARY entry in spark-env.sh. But then I get:
---
A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fed89801ce9, pid=13580, tid=140657358776064
#
# JRE version: Java(TM) SE Runtime Environment (7.0_51-b13) (build 1.7.0_51-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.51-b03 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V [libjvm.so+0x632ce9] jni_GetByteArrayElements+0x89
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/vagrant/hs_err_pid13580.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
---
The error lag says:
---
Current thread (0x00007fed8473d000): JavaThread "MesosSchedulerBackend driver" daemon [_thread_in_vm, id=13638, stack(0x00007fed57d7a000,0x00007fed57e7b000)]
…
---
Working on Ubuntu 12.04 in Virtual Box. Tried it with OpenJDK 6 and Oracle Java 7.
Any ideas??
Many thanks.
Christoph
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>Please ignore this error - I found the issue.
>
>Thanks !
>
>
>On Mon, Jan 20, 2014 at 3:14 PM, Manoj Samel <ma...@gmail.com>wrote:
>
>> Hi
>>
>> I deployed spark 0.8.1 on standalone cluster per
>> https://spark.incubator.apache.org/docs/0.8.1/spark-standalone.html
>>
>> When i start a spark-shell , I get following error
>>
>> I thought mesos should not be required for standalone cluster. Do I have
>> to change any parameters in make-distribution.sh that I used to build the
>> spark distribution for this cluster ? I left all to default (and noticed
>> that the default HADOOP version is 1.0.4 which is not my hadoop version -
>> but I am not using Hadoop here).
>>
>> Creating SparkContext...
>> Failed to load native Mesos library from
>> java.lang.UnsatisfiedLinkError: no mesos in java.library.path
>> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1738)
>> at java.lang.Runtime.loadLibrary0(Runtime.java:823)
>> at java.lang.System.loadLibrary(System.java:1028)
>> at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:52)
>> at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:64)
>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:260)
>> at
>> org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:862)
Re: spark-shell on standalone cluster gives error " no mesos in java.library.path"
Posted by Christoph Böhm <li...@gmx.net>.
Forgot to post the solution. I messed up the master URL. In particular, I gave the host (master), not a URL. My bad. The error message is weird, though. Seems like the URL regex matches master for mesos://...
No idea about the Java Runtime Environment Error.
On Mar 26, 2014, at 3:52 PM, Christoph Böhm wrote:
> Hi,
>
> I have a similar issue like the user below:
> I’m running Spark 0.8.1 (standalone). When I test the streaming NetworkWordCount example as in the docs with local[2] it works fine. As soon as I want to connect to my cluster using [NetworkWordCount master …] it says:
> ---
> Failed to load native Mesos library from /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> Exception in thread "main" java.lang.UnsatisfiedLinkError: no mesos in java.library.path
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)
> at java.lang.Runtime.loadLibrary0(Runtime.java:849)
> at java.lang.System.loadLibrary(System.java:1088)
> at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:52)
> at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:64)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:260)
> at org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:559)
> at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:84)
> at org.apache.spark.streaming.api.java.JavaStreamingContext.<init>(JavaStreamingContext.scala:76)
> at org.apache.spark.streaming.examples.JavaNetworkWordCount.main(JavaNetworkWordCount.java:50)
> ---
>
> I built mesos 0.13 and added the MESOS_NATIVE_LIBRARY entry in spark-env.sh. But then I get:
> ---
> A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x00007fed89801ce9, pid=13580, tid=140657358776064
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_51-b13) (build 1.7.0_51-b13)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.51-b03 mixed mode linux-amd64 compressed oops)
> # Problematic frame:
> # V [libjvm.so+0x632ce9] jni_GetByteArrayElements+0x89
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /home/vagrant/hs_err_pid13580.log
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.sun.com/bugreport/crash.jsp
> ---
>
> The error lag says:
> ---
> Current thread (0x00007fed8473d000): JavaThread "MesosSchedulerBackend driver" daemon [_thread_in_vm, id=13638, stack(0x00007fed57d7a000,0x00007fed57e7b000)]
> …
> ---
>
> Working on Ubuntu 12.04 in Virtual Box. Tried it with OpenJDK 6 and Oracle Java 7.
>
>
> Any ideas??
> Many thanks.
>
> Christoph