You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by lisak <li...@gmail.com> on 2015/12/14 20:44:36 UTC

Re: Cannot use arbitrary external jar files in spark_submit through Zeppelin - should be fixed

Hey,

I'm still struggling with %deps so I tried : 

SPARK_SUBMIT_OPTIONS="--repositories
https://oss.sonatype.org/content/repositories/snapshots --packages
com.example:spark-extensions_2.10:0.09-SNAPSHOT,org.apache.hadoop:hadoop-aws:2.6.0"

and when I first run a notebook I can see log entries about downloading the
deps and all its transitive dependencies BUT in the mean time I get this
error in notebook : 

java.net.ConnectException: Connection refused 
at java.net.PlainSocketImpl.socketConnect(Native Method) 
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) 
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) 
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) 
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) 
at java.net.Socket.connect(Socket.java:589) 
at org.apache.thrift.transport.TSocket.open(TSocket.java:182) 
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51) 
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37) 
at
org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60) 
at
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861) 
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435) 
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) 
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:139) 
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.init(RemoteInterpreter.java:139) 
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:266) 
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType(LazyOpenInterpreter.java:104) 
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:199) 
at org.apache.zeppelin.scheduler.Job.run(Job.java:170) 
at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:320) 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) 
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) 
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745)

and it never recovers from it, if I run it again I get : 

org.apache.thrift.TApplicationException: Internal error processing
getFormType 
at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111) 
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71) 
at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_getFormType(RemoteInterpreterService.java:288) 
at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.getFormType(RemoteInterpreterService.java:275) 
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:281) 
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType(LazyOpenInterpreter.java:104) 
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:199) 
at org.apache.zeppelin.scheduler.Job.run(Job.java:170) 
at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:320) 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) 
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) 
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745)





--
View this message in context: http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Cannot-use-arbitrary-external-jar-files-in-spark-submit-through-Zeppelin-should-be-fixed-tp1277p1783.html
Sent from the Apache Zeppelin Users (incubating) mailing list mailing list archive at Nabble.com.

Re: Cannot use arbitrary external jar files in spark_submit through Zeppelin - should be fixed

Posted by lisak <li...@gmail.com>.
Hi,

the problems with %deps were varying error messages about that it shouldn't
be executed before %spark interpreter and also it seemed to me that it was
always working for the first time and if I ran it for second time I got
weird errors about different content in jvm classes...

Anyway what works for me is : using `--jars` for libraries that executors
need and doing only : 

z.addRepo("sonatype-snapshots",
"https://oss.sonatype.org/content/repositories/snapshots", true)
z.load("com.example:spark-extensions_2.10:0.09-SNAPSHOT")

without %deps, for dependencies that are used in driver/notebook ...

^^ this combination seems to be working and not causing any problems at all

Btw for using hadoop-aws the best thing one can do is : 

--jars=file:/some/path/aws-java-sdk-1.7.14.jar,file:/some/path/hadoop-aws-2.6.0.jar

Otherwise one is always fighting problems with clashes of transitive
dependencies 



--
View this message in context: http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Cannot-use-arbitrary-external-jar-files-in-spark-submit-through-Zeppelin-should-be-fixed-tp1277p1797.html
Sent from the Apache Zeppelin Users (incubating) mailing list mailing list archive at Nabble.com.

Re: Cannot use arbitrary external jar files in spark_submit through Zeppelin - should be fixed

Posted by Felix Cheung <fe...@hotmail.com>.
There might be some difference since the former has some dependencies resolutions.
What is the issue you have with %dep?What is the dependency it has conflict with Hadoop-aws?


    _____________________________
From: lisak <li...@gmail.com>
Sent: Monday, December 14, 2015 12:03 PM
Subject: Re: Cannot use arbitrary external jar files in spark_submit through Zeppelin - should be fixed
To:  <us...@zeppelin.incubator.apache.org>


                   I guess that :    
    
 %deps   
 z.reset()   
 z.load("org.apache.hadoop:hadoop-aws:2.6.0")   
    
 isn't the same as doing :   
    
 SPARK_SUBMIT_OPTIONS="--packages org.apache.hadoop:hadoop-aws:2.6.0"    
    
 because the latter leads to weird dependency conflicts    
    
    
    
    
    
 --   
 View this message in context:    http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Cannot-use-arbitrary-external-jar-files-in-spark-submit-through-Zeppelin-should-be-fixed-tp1277p1784.html   
 Sent from the Apache Zeppelin Users (incubating) mailing list mailing list archive at Nabble.com.   
       


  

Re: Cannot use arbitrary external jar files in spark_submit through Zeppelin - should be fixed

Posted by lisak <li...@gmail.com>.
I guess that : 

%deps
z.reset()
z.load("org.apache.hadoop:hadoop-aws:2.6.0")

isn't the same as doing :

SPARK_SUBMIT_OPTIONS="--packages org.apache.hadoop:hadoop-aws:2.6.0" 

because the latter leads to weird dependency conflicts 





--
View this message in context: http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Cannot-use-arbitrary-external-jar-files-in-spark-submit-through-Zeppelin-should-be-fixed-tp1277p1784.html
Sent from the Apache Zeppelin Users (incubating) mailing list mailing list archive at Nabble.com.