Zeppelin with external cluster

[Jongyoul Lee <jo...@gmail.com>]

Hi dev,

I've succeeded zeppelin with spark 1.2. Thanks, Moon. Now, I'm trying to
use zeppelin with external cluster. I've tested yesterday with standalone,
mesos, but the results are not good. In case of standalone, No snappyjava
error occurs, and in case of mesos, Nothing's happened. Do you have any
reference to run zeppelin with external cluster? If you don't have anyone,
I can write references for running with external cluster.

Regards,
JL

-- 
이종열, Jongyoul Lee, 李宗烈
http://madeng.net

Fwd: Zeppelin with external cluster

[Jongyoul Lee <jo...@gmail.com>]

I'll resend email 'cause my attachment's size if larger than 1000000 bytes


---------- Forwarded message ----------
From: Jongyoul Lee <jo...@gmail.com>
Date: Thu, Jan 29, 2015 at 2:14 PM
Subject: Re: Zeppelin with external cluster
To: dev@zeppelin.incubator.apache.org


Hi Kevin,

I also change master to spark://dicc-m002:7077. Actually, I think
interpreter.json affect what cluster is used on running codes. Anyway, my
interpreter screenshot is below, and my error is like this.

org.apache.spark.SparkException: Job aborted due to stage failure: Task 1
in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage
0.0 (TID 6, DICc-r1n029): java.lang.UnsatisfiedLinkError: no snappyjava in
java.library.path at
java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886) at
java.lang.Runtime.loadLibrary0(Runtime.java:849) at
java.lang.System.loadLibrary(System.java:1088) at
org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:170) at
org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:145) at
org.xerial.snappy.Snappy.<clinit>(Snappy.java:47) at
org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:358)
at org.xerial.snappy.SnappyInputStream.rawRead(SnappyInputStream.java:167)
at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:150) at
java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2310)
at
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2323)
at
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at
org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:57)
at
org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:57)
at
org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:95)
at
org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:215)
at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:177)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1000) at
org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:164)
at
org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
at
org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
at
org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:87)
at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58) at
org.apache.spark.scheduler.Task.run(Task.scala:56) at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744) Driver stacktrace: at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1214)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1203)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1202)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1202)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:696)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:696)
at scala.Option.foreach(Option.scala:236) at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:696)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1420)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465) at
org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1375)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) at
akka.actor.ActorCell.invoke(ActorCell.scala:487) at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) at
akka.dispatch.Mailbox.run(Mailbox.scala:220) at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

I think that this error is about class path. I'm running zeppelin under
/home/1001079/apache-zeppelin. Which means all classes are located under
this directory. Because zeppelin adds classes to SPARK_CLASSPATH, if slave
doesn't have that libraries on the same path, It might be no class error
occurs.

I want to contribute by fixing this issue. Could you please tell me regular
steps for dealing with an issue? Or Is it ok to make a PR without JIRA
issue?

Regards,
JL

On Thu, Jan 29, 2015 at 1:55 PM, Kevin (Sangwoo) Kim <ke...@apache.org>
wrote:

> Hi Jongyoul,
> I'm using Zeppelin with external cluster.
> (standalone mode)
>
> All I needed to do is, writing master setting like
> export MASTER="spark://IP-ADDRESS:7077"
> in $ZEPPELIN/conf/zeppelin-env.sh
>
> If your error persists, plz post the error message in reply!
> I'm gonna looking at it.
>
> Regards,
> Kevin
>
>
> On Thu Jan 29 2015 at 12:58:41 PM Jongyoul Lee <jo...@gmail.com> wrote:
>
> > Hi dev,
> >
> > I've succeeded zeppelin with spark 1.2. Thanks, Moon. Now, I'm trying to
> > use zeppelin with external cluster. I've tested yesterday with
> standalone,
> > mesos, but the results are not good. In case of standalone, No snappyjava
> > error occurs, and in case of mesos, Nothing's happened. Do you have any
> > reference to run zeppelin with external cluster? If you don't have
> anyone,
> > I can write references for running with external cluster.
> >
> > Regards,
> > JL
> >
> > --
> > 이종열, Jongyoul Lee, 李宗烈
> > http://madeng.net
> >
>



-- 
이종열, Jongyoul Lee, 李宗烈
http://madeng.net



-- 
이종열, Jongyoul Lee, 李宗烈
http://madeng.net

Re: Zeppelin with external cluster

["Kevin (Sangwoo) Kim" <ke...@apache.org>]

Hi Jongyoul,
I'm using Zeppelin with external cluster.
(standalone mode)

All I needed to do is, writing master setting like
export MASTER="spark://IP-ADDRESS:7077"
in $ZEPPELIN/conf/zeppelin-env.sh

If your error persists, plz post the error message in reply!
I'm gonna looking at it.

Regards,
Kevin


On Thu Jan 29 2015 at 12:58:41 PM Jongyoul Lee <jo...@gmail.com> wrote:

> Hi dev,
>
> I've succeeded zeppelin with spark 1.2. Thanks, Moon. Now, I'm trying to
> use zeppelin with external cluster. I've tested yesterday with standalone,
> mesos, but the results are not good. In case of standalone, No snappyjava
> error occurs, and in case of mesos, Nothing's happened. Do you have any
> reference to run zeppelin with external cluster? If you don't have anyone,
> I can write references for running with external cluster.
>
> Regards,
> JL
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>