[Spark 1.2] zeppelin with mesos

[Ayoub Benali <ay...@nugg.ad>]

Hi,

Yes there is but now I have a different problem although the application is
connected now.

I am trying to use zeppelin in cluster mode with mesos and I get exceptions
when interpreting spark command like "sc.version"

INFO: Adding the following classes declared in
META-INF/services/jersey-server-components to the resource configuration:
  class org.atmosphere.jersey.AtmosphereResourceConfigurator
Feb 06, 2015 10:34:31 AM com.sun.jersey.spi.inject.Errors
processErrorMessages
WARNING: The following warnings have been detected with resource and/or
provider classes:
  WARNING: A HTTP GET method, public javax.ws.rs.core.Response
com.nflabs.zeppelin.rest.InterpreterRestApi.listInterpreter(java.lang.String),
should not consume any entity.
java.lang.NoClassDefFoundError: scala/tools/nsc/CompilerCommand
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:412)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at
com.nflabs.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:192)
at
com.nflabs.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:83)
at
com.nflabs.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:52)
at
com.nflabs.zeppelin.interpreter.LazyOpenInterpreter.bindValue(LazyOpenInterpreter.java:88)
at com.nflabs.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:175)
at com.nflabs.zeppelin.scheduler.Job.run(Job.java:147)
at com.nflabs.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:85)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException: scala.tools.nsc.CompilerCommand
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 27 more
Exception in thread "Thread-18" java.lang.NoClassDefFoundError:
org/apache/spark/repl/SparkCommandLine
at
com.nflabs.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:192)
at
com.nflabs.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:83)
at
com.nflabs.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:52)
at
com.nflabs.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:99)
at com.nflabs.zeppelin.notebook.Paragraph.progress(Paragraph.java:150)
at
com.nflabs.zeppelin.scheduler.JobProgressPoller.run(JobProgressPoller.java:35)

According to this message in the mailing list:
https://groups.google.com/forum/#!msg/zeppelin-developers/i7oXKHYfB38/uxZr_3XHvhkJ

Zeppeling should take the same arguments as spark-shell so here is the
content of zeppelin-env.sh which is the same as conf/spark-defaults.conf
and conf/spark-env.sh in my spark installation to run the spark-shell:

export JAVA_HOME=/usr/lib/jvm/j2sdk1.7-oracle/
export SPARK_HOME=/opt/spark-v1.2.1-snapshot
export SPARK_MASTER=mesos://zk://****:2181,****:2181,***:2181/mesos
export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so
# for spark-history-server
export SPARK_PID_DIR=/opt/spark-v1.2.1-snapshot/tmp
export SPARK_LOCAL_DIRS=/opt/tmp
export HADOOP_CONF_DIR=/etc/hadoop/conf

Could the problem be due to "MASTER" is not set in my script? if it is the
case what should be the value given that I use mesos and zookeeper?

the cluster have spark1.2.1-snapshot installed and I compiled zeppelin from
this pull request https://github.com/NFLabs/zeppelin/pull/294

Does anyone know what could be the issue ?

Thanks,
Ayoub.



On 6 February 2015 at 00:40, Anthony Corbacho <an...@apache.org>
wrote:

> hi,
> can you check in your server if there is a Service that already use the
> port 8081?
> On Feb 6, 2015 2:05 AM, "Ayoub Benali" <ay...@nugg.ad> wrote:
>
>> Hello,
>>
>> I build the application but when I start it, the console in Chrome shows
>> the following error message:
>>
>> WebSocket connection to 'ws://****:8081/' failed: Error during WebSocket
>> handshake: Unexpected response code: 200
>>
>> So "create new notebook" don't do anything.
>>
>> I checked in the network and there is nothing that would bloc websockets
>> and there is nothing in application logs.
>>
>> Does anyone knows what would cause the issue ?
>>
>> Thanks,
>> Ayoub
>>
>


-- 
___________________________
*Ayoub Benali*
Software Developer

nugg.ad AG - Predictive Behavioural Targeting
Rotherstraße 16 - 10245 Berlin

t:   +49 (0)30 29 38 19 99 18
f:   +49 (0)30 29 38 19 99 -9
m: +49 (0)175 294 85 25
mareike.oetting@nugg.ad

www.nugg.ad * http://blog.nugg.ad/ * www.twitter.com/nuggad *
www.facebook.com/nuggad

*Registergericht/District court: *Charlottenburg HRB 102226 B
*Vorsitzender des Aufsichtsrates/Chairman of the supervisory board: *Dr.
Detlev Ruland
*Vorstand/Executive board: *Stephan Noller, Karim H. Attia, Richard Hutton

*nugg.ad <http://nugg.ad/> is a company of Deutsche Post DHL.*