You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by lisak <li...@gmail.com> on 2015/12/14 20:44:36 UTC
Re: Cannot use arbitrary external jar files in spark_submit through
Zeppelin - should be fixed
Hey,
I'm still struggling with %deps so I tried :
SPARK_SUBMIT_OPTIONS="--repositories
https://oss.sonatype.org/content/repositories/snapshots --packages
com.example:spark-extensions_2.10:0.09-SNAPSHOT,org.apache.hadoop:hadoop-aws:2.6.0"
and when I first run a notebook I can see log entries about downloading the
deps and all its transitive dependencies BUT in the mean time I get this
error in notebook :
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at
org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:139)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.init(RemoteInterpreter.java:139)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:266)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType(LazyOpenInterpreter.java:104)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:199)
at org.apache.zeppelin.scheduler.Job.run(Job.java:170)
at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:320)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
and it never recovers from it, if I run it again I get :
org.apache.thrift.TApplicationException: Internal error processing
getFormType
at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_getFormType(RemoteInterpreterService.java:288)
at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.getFormType(RemoteInterpreterService.java:275)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:281)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType(LazyOpenInterpreter.java:104)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:199)
at org.apache.zeppelin.scheduler.Job.run(Job.java:170)
at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:320)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
--
View this message in context: http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Cannot-use-arbitrary-external-jar-files-in-spark-submit-through-Zeppelin-should-be-fixed-tp1277p1783.html
Sent from the Apache Zeppelin Users (incubating) mailing list mailing list archive at Nabble.com.
Re: Cannot use arbitrary external jar files in spark_submit through
Zeppelin - should be fixed
Posted by lisak <li...@gmail.com>.
Hi,
the problems with %deps were varying error messages about that it shouldn't
be executed before %spark interpreter and also it seemed to me that it was
always working for the first time and if I ran it for second time I got
weird errors about different content in jvm classes...
Anyway what works for me is : using `--jars` for libraries that executors
need and doing only :
z.addRepo("sonatype-snapshots",
"https://oss.sonatype.org/content/repositories/snapshots", true)
z.load("com.example:spark-extensions_2.10:0.09-SNAPSHOT")
without %deps, for dependencies that are used in driver/notebook ...
^^ this combination seems to be working and not causing any problems at all
Btw for using hadoop-aws the best thing one can do is :
--jars=file:/some/path/aws-java-sdk-1.7.14.jar,file:/some/path/hadoop-aws-2.6.0.jar
Otherwise one is always fighting problems with clashes of transitive
dependencies
--
View this message in context: http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Cannot-use-arbitrary-external-jar-files-in-spark-submit-through-Zeppelin-should-be-fixed-tp1277p1797.html
Sent from the Apache Zeppelin Users (incubating) mailing list mailing list archive at Nabble.com.
Re: Cannot use arbitrary external jar files in spark_submit through
Zeppelin - should be fixed
Posted by Felix Cheung <fe...@hotmail.com>.
There might be some difference since the former has some dependencies resolutions.
What is the issue you have with %dep?What is the dependency it has conflict with Hadoop-aws?
_____________________________
From: lisak <li...@gmail.com>
Sent: Monday, December 14, 2015 12:03 PM
Subject: Re: Cannot use arbitrary external jar files in spark_submit through Zeppelin - should be fixed
To: <us...@zeppelin.incubator.apache.org>
I guess that :
%deps
z.reset()
z.load("org.apache.hadoop:hadoop-aws:2.6.0")
isn't the same as doing :
SPARK_SUBMIT_OPTIONS="--packages org.apache.hadoop:hadoop-aws:2.6.0"
because the latter leads to weird dependency conflicts
--
View this message in context: http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Cannot-use-arbitrary-external-jar-files-in-spark-submit-through-Zeppelin-should-be-fixed-tp1277p1784.html
Sent from the Apache Zeppelin Users (incubating) mailing list mailing list archive at Nabble.com.
Re: Cannot use arbitrary external jar files in spark_submit through
Zeppelin - should be fixed
Posted by lisak <li...@gmail.com>.
I guess that :
%deps
z.reset()
z.load("org.apache.hadoop:hadoop-aws:2.6.0")
isn't the same as doing :
SPARK_SUBMIT_OPTIONS="--packages org.apache.hadoop:hadoop-aws:2.6.0"
because the latter leads to weird dependency conflicts
--
View this message in context: http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Cannot-use-arbitrary-external-jar-files-in-spark-submit-through-Zeppelin-should-be-fixed-tp1277p1784.html
Sent from the Apache Zeppelin Users (incubating) mailing list mailing list archive at Nabble.com.