You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Theofilos Kakantousis <tk...@kth.se> on 2017/05/30 16:05:15 UTC
Livy - add external libraries from additional maven repo
Hi everyone,
I'm using Zeppelin with Livy 0.4 and trying to add external libraries
from an additional maven repo to my application according to the
documentation available here
<https://zeppelin.apache.org/docs/0.7.0/interpreter/livy.html#adding-external-libraries>.
The example works fine, but when I set the livy.spark.jars.packages to
my library the interpreter throws an unresolved dependency error.
I have added the additional maven repository in the interpreter settings
and have also tried setting livy.spark.jars.ivy but without luck.
However, if I use the Spark interpreter with the following code it works
fine.
"%dep
z.reset();
z.addRepo("my repo").url("http://myrepo").snapshot
z.load("mygroup:myartifact:myversion");
Has anyone managed to do that with Livy? Thanks!
Cheers,
Theo
Re: Livy - add external libraries from additional maven repo
Posted by Jeff Zhang <zj...@gmail.com>.
I am afraid for now only official maven repo is supported.
Theofilos Kakantousis <tk...@kth.se>于2017年6月1日周四 下午4:04写道：
> Hi Jeff,
>
> That is the first thing I tried, as per the documentation as well, but it
> only works for the main maven repo and not my custom maven repo. The
> documentation is not clear if it is supposed to work for custom maven
> repos. Have you tried it?
>
> Kind regards,
> Theo
>
>
> On 2017-06-01 01:45, Jeff Zhang wrote:
>
> Ben's right, for livy interpreter, you need to specify
> livy.spark.jars.packages.
>
> Specifying interpreter dependency doesn't work for livy interpreter,
> because livy run in yarn-cluster mode, while interpreter dependencies is
> downloaded in the client JVM.
>
>
>
>
> Felix Cheung <fe...@hotmail.com>于2017年6月1日周四 上午2:52写道：
>
>> How are you setting this:
>> When setting my local maven repo and the "groupId:artifactId:version" in
>> the interpreter settings
>>
>> ------------------------------
>> *From:* Theofilos Kakantousis <tk...@kth.se>
>> *Sent:* Wednesday, May 31, 2017 1:56:07 AM
>> *To:* users@zeppelin.apache.org
>>
>> *Subject:* Re: Livy - add external libraries from additional maven repo
>> Thanks everyone for the feedback!
>>
>> Indeed %dep works only for Spark interpreter, just mentioned it to show
>> the interpreter behavior I expected with Livy.
>> When setting my local maven repo and the "groupId:artifactId:version" in
>> the interpreter settings, I can see the dependency (i.e. a jar file) being
>> downloaded to the local OS under "local-repo" directory but the dependency
>> is not deployed with the Spark application in YARN.
>>
>> Cheers,
>> Theo
>>
>> On 2017-05-31 01:19, Ben Vogan wrote:
>>
>> For what it's worth I have successfully added jar files and maven
>> packages to sessions using zeppelin & livy 0.3 - although not using %dep.
>> In the interpreter settings I set the livy.spark.jars setting for jars that
>> are on my HDFS cluster, and livy.spark.jars.packages for maven packages
>> - although only using maven central and not a local repo.
>>
>> --Ben
>>
>> On Tue, May 30, 2017 at 12:36 PM, Felix Cheung <felixcheung_m@hotmail.com
>> > wrote:
>>
>>> To add, this might be an issue with Livy.
>>>
>>> I'm seeing something similar as well.
>>>
>>> If you can get a repo with calling the Livy REST API directly it will be
>>> worthwhile to follow up with the Livy community separately.
>>>
>>>
>>> ------------------------------
>>> *From:* Felix Cheung <fe...@hotmail.com>
>>> *Sent:* Tuesday, May 30, 2017 11:34:31 AM
>>> *To:* users@zeppelin.apache.org; users@zeppelin.apache.org
>>> *Subject:* Re: Livy - add external libraries from additional maven repo
>>>
>>> if I recall, %dep only works with the built in Spark interpreter and not
>>> the Livy interpreter.
>>>
>>> To manage dependency win Livy you will need to set Spark conf with Livy.
>>>
>>> ------------------------------
>>> *From:* Theofilos Kakantousis <tk...@kth.se>
>>> *Sent:* Tuesday, May 30, 2017 9:05:15 AM
>>> *To:* users@zeppelin.apache.org
>>> *Subject:* Livy - add external libraries from additional maven repo
>>>
>>> Hi everyone,
>>>
>>> I'm using Zeppelin with Livy 0.4 and trying to add external libraries
>>> from an additional maven repo to my application according to the
>>> documentation available here
>>> <https://zeppelin.apache.org/docs/0.7.0/interpreter/livy.html#adding-external-libraries>.
>>> The example works fine, but when I set the livy.spark.jars.packages to my
>>> library the interpreter throws an unresolved dependency error.
>>>
>>> I have added the additional maven repository in the interpreter settings
>>> and have also tried setting livy.spark.jars.ivy but without luck. However,
>>> if I use the Spark interpreter with the following code it works fine.
>>>
>>> "%dep
>>> z.reset();
>>> z.addRepo("my repo").url("http://myrepo" <http://myrepo>).snapshot
>>> z.load("mygroup:myartifact:myversion");
>>>
>>> Has anyone managed to do that with Livy? Thanks!
>>>
>>> Cheers,
>>> Theo
>>>
>>
>>
>>
>> --
>> *BENJAMIN VOGAN* | Data Platform Team Lead
>>
>> <http://www.shopkick.com/>
>> <https://www.facebook.com/shopkick> <https://www.instagram.com/shopkick/>
>> <https://www.pinterest.com/shopkick/> <https://twitter.com/shopkickbiz>
>> <https://www.linkedin.com/company-beta/831240/?pathWildcard=831240>
>>
>>
>>
>
Re: Livy - add external libraries from additional maven repo
Posted by Theofilos Kakantousis <tk...@kth.se>.
Hi Jeff,
That is the first thing I tried, as per the documentation as well, but
it only works for the main maven repo and not my custom maven repo. The
documentation is not clear if it is supposed to work for custom maven
repos. Have you tried it?
Kind regards,
Theo
On 2017-06-01 01:45, Jeff Zhang wrote:
> Ben's right, for livy interpreter, you need to specify
> livy.spark.jars.packages.
>
> Specifying interpreter dependency doesn't work for livy interpreter,
> because livy run in yarn-cluster mode, while interpreter dependencies
> is downloaded in the client JVM.
>
>
>
>
> Felix Cheung <felixcheung_m@hotmail.com
> <ma...@hotmail.com>>于2017年6月1日周四 上午2:52写道：
>
> How are you setting this:
> When setting my local maven repo and the
> "groupId:artifactId:version" in the interpreter settings
>
> ------------------------------------------------------------------------
> *From:* Theofilos Kakantousis <tkak@kth.se <ma...@kth.se>>
> *Sent:* Wednesday, May 31, 2017 1:56:07 AM
> *To:* users@zeppelin.apache.org <ma...@zeppelin.apache.org>
>
> *Subject:* Re: Livy - add external libraries from additional maven
> repo
> Thanks everyone for the feedback!
>
> Indeed %dep works only for Spark interpreter, just mentioned it to
> show the interpreter behavior I expected with Livy.
> When setting my local maven repo and the
> "groupId:artifactId:version" in the interpreter settings, I can
> see the dependency (i.e. a jar file) being downloaded to the local
> OS under "local-repo" directory but the dependency is not deployed
> with the Spark application in YARN.
>
> Cheers,
> Theo
>
> On 2017-05-31 01:19, Ben Vogan wrote:
>> For what it's worth I have successfully added jar files and maven
>> packages to sessions using zeppelin & livy 0.3 - although not
>> using %dep. In the interpreter settings I set the
>> livy.spark.jars setting for jars that are on my HDFS cluster, and
>> livy.spark.jars.packages for maven packages - although only using
>> maven central and not a local repo.
>>
>> --Ben
>>
>> On Tue, May 30, 2017 at 12:36 PM, Felix Cheung
>> <felixcheung_m@hotmail.com <ma...@hotmail.com>> wrote:
>>
>> To add, this might be an issue with Livy.
>>
>> I'm seeing something similar as well.
>>
>> If you can get a repo with calling the Livy REST API directly
>> it will be worthwhile to follow up with the Livy community
>> separately.
>>
>>
>> ------------------------------------------------------------------------
>> *From:* Felix Cheung <felixcheung_m@hotmail.com
>> <ma...@hotmail.com>>
>> *Sent:* Tuesday, May 30, 2017 11:34:31 AM
>> *To:* users@zeppelin.apache.org
>> <ma...@zeppelin.apache.org>; users@zeppelin.apache.org
>> <ma...@zeppelin.apache.org>
>> *Subject:* Re: Livy - add external libraries from additional
>> maven repo
>> if I recall, %dep only works with the built in Spark
>> interpreter and not the Livy interpreter.
>>
>> To manage dependency win Livy you will need to set Spark conf
>> with Livy.
>>
>> ------------------------------------------------------------------------
>> *From:* Theofilos Kakantousis <tkak@kth.se <ma...@kth.se>>
>> *Sent:* Tuesday, May 30, 2017 9:05:15 AM
>> *To:* users@zeppelin.apache.org
>> <ma...@zeppelin.apache.org>
>> *Subject:* Livy - add external libraries from additional
>> maven repo
>> Hi everyone,
>>
>> I'm using Zeppelin with Livy 0.4 and trying to add external
>> libraries from an additional maven repo to my application
>> according to the documentation available here
>> <https://zeppelin.apache.org/docs/0.7.0/interpreter/livy.html#adding-external-libraries>.
>> The example works fine, but when I set the
>> livy.spark.jars.packages to my library the interpreter throws
>> an unresolved dependency error.
>>
>> I have added the additional maven repository in the
>> interpreter settings and have also tried setting
>> livy.spark.jars.ivy but without luck. However, if I use the
>> Spark interpreter with the following code it works fine.
>>
>> "%dep
>> z.reset();
>> z.addRepo("my repo").url("http://myrepo"
>> <http://myrepo>).snapshot
>> z.load("mygroup:myartifact:myversion");
>>
>> Has anyone managed to do that with Livy? Thanks!
>>
>> Cheers,
>> Theo
>>
>>
>>
>>
>> --
>> *BENJAMIN VOGAN*| Data Platform Team Lead
>>
>> <http://www.shopkick.com/>
>> <https://www.facebook.com/shopkick><https://www.instagram.com/shopkick/><https://www.pinterest.com/shopkick/><https://twitter.com/shopkickbiz><https://www.linkedin.com/company-beta/831240/?pathWildcard=831240>
>
>
Re: Livy - add external libraries from additional maven repo
Posted by Jeff Zhang <zj...@gmail.com>.
Ben's right, for livy interpreter, you need to specify
livy.spark.jars.packages.
Specifying interpreter dependency doesn't work for livy interpreter,
because livy run in yarn-cluster mode, while interpreter dependencies is
downloaded in the client JVM.
Felix Cheung <fe...@hotmail.com>于2017年6月1日周四 上午2:52写道：
> How are you setting this:
> When setting my local maven repo and the "groupId:artifactId:version" in
> the interpreter settings
>
> ------------------------------
> *From:* Theofilos Kakantousis <tk...@kth.se>
> *Sent:* Wednesday, May 31, 2017 1:56:07 AM
> *To:* users@zeppelin.apache.org
>
> *Subject:* Re: Livy - add external libraries from additional maven repo
> Thanks everyone for the feedback!
>
> Indeed %dep works only for Spark interpreter, just mentioned it to show
> the interpreter behavior I expected with Livy.
> When setting my local maven repo and the "groupId:artifactId:version" in
> the interpreter settings, I can see the dependency (i.e. a jar file) being
> downloaded to the local OS under "local-repo" directory but the dependency
> is not deployed with the Spark application in YARN.
>
> Cheers,
> Theo
>
> On 2017-05-31 01:19, Ben Vogan wrote:
>
> For what it's worth I have successfully added jar files and maven packages
> to sessions using zeppelin & livy 0.3 - although not using %dep. In the
> interpreter settings I set the livy.spark.jars setting for jars that are on
> my HDFS cluster, and livy.spark.jars.packages for maven packages -
> although only using maven central and not a local repo.
>
> --Ben
>
> On Tue, May 30, 2017 at 12:36 PM, Felix Cheung <fe...@hotmail.com>
> wrote:
>
>> To add, this might be an issue with Livy.
>>
>> I'm seeing something similar as well.
>>
>> If you can get a repo with calling the Livy REST API directly it will be
>> worthwhile to follow up with the Livy community separately.
>>
>>
>> ------------------------------
>> *From:* Felix Cheung <fe...@hotmail.com>
>> *Sent:* Tuesday, May 30, 2017 11:34:31 AM
>> *To:* users@zeppelin.apache.org; users@zeppelin.apache.org
>> *Subject:* Re: Livy - add external libraries from additional maven repo
>>
>> if I recall, %dep only works with the built in Spark interpreter and not
>> the Livy interpreter.
>>
>> To manage dependency win Livy you will need to set Spark conf with Livy.
>>
>> ------------------------------
>> *From:* Theofilos Kakantousis <tk...@kth.se>
>> *Sent:* Tuesday, May 30, 2017 9:05:15 AM
>> *To:* users@zeppelin.apache.org
>> *Subject:* Livy - add external libraries from additional maven repo
>>
>> Hi everyone,
>>
>> I'm using Zeppelin with Livy 0.4 and trying to add external libraries
>> from an additional maven repo to my application according to the
>> documentation available here
>> <https://zeppelin.apache.org/docs/0.7.0/interpreter/livy.html#adding-external-libraries>.
>> The example works fine, but when I set the livy.spark.jars.packages to my
>> library the interpreter throws an unresolved dependency error.
>>
>> I have added the additional maven repository in the interpreter settings
>> and have also tried setting livy.spark.jars.ivy but without luck. However,
>> if I use the Spark interpreter with the following code it works fine.
>>
>> "%dep
>> z.reset();
>> z.addRepo("my repo").url("http://myrepo" <http://myrepo>).snapshot
>> z.load("mygroup:myartifact:myversion");
>>
>> Has anyone managed to do that with Livy? Thanks!
>>
>> Cheers,
>> Theo
>>
>
>
>
> --
> *BENJAMIN VOGAN* | Data Platform Team Lead
>
> <http://www.shopkick.com/>
> <https://www.facebook.com/shopkick> <https://www.instagram.com/shopkick/>
> <https://www.pinterest.com/shopkick/> <https://twitter.com/shopkickbiz>
> <https://www.linkedin.com/company-beta/831240/?pathWildcard=831240>
>
>
>
Re: Livy - add external libraries from additional maven repo
Posted by Felix Cheung <fe...@hotmail.com>.
How are you setting this:
When setting my local maven repo and the "groupId:artifactId:version" in the interpreter settings
________________________________
From: Theofilos Kakantousis <tk...@kth.se>
Sent: Wednesday, May 31, 2017 1:56:07 AM
To: users@zeppelin.apache.org
Subject: Re: Livy - add external libraries from additional maven repo
Thanks everyone for the feedback!
Indeed %dep works only for Spark interpreter, just mentioned it to show the interpreter behavior I expected with Livy.
When setting my local maven repo and the "groupId:artifactId:version" in the interpreter settings, I can see the dependency (i.e. a jar file) being downloaded to the local OS under "local-repo" directory but the dependency is not deployed with the Spark application in YARN.
Cheers,
Theo
On 2017-05-31 01:19, Ben Vogan wrote:
For what it's worth I have successfully added jar files and maven packages to sessions using zeppelin & livy 0.3 - although not using %dep. In the interpreter settings I set the livy.spark.jars setting for jars that are on my HDFS cluster, and livy.spark.jars.packages for maven packages - although only using maven central and not a local repo.
--Ben
On Tue, May 30, 2017 at 12:36 PM, Felix Cheung <fe...@hotmail.com>> wrote:
To add, this might be an issue with Livy.
I'm seeing something similar as well.
If you can get a repo with calling the Livy REST API directly it will be worthwhile to follow up with the Livy community separately.
________________________________
From: Felix Cheung <fe...@hotmail.com>>
Sent: Tuesday, May 30, 2017 11:34:31 AM
To: users@zeppelin.apache.org<ma...@zeppelin.apache.org>; users@zeppelin.apache.org<ma...@zeppelin.apache.org>
Subject: Re: Livy - add external libraries from additional maven repo
if I recall, %dep only works with the built in Spark interpreter and not the Livy interpreter.
To manage dependency win Livy you will need to set Spark conf with Livy.
________________________________
From: Theofilos Kakantousis <tk...@kth.se>>
Sent: Tuesday, May 30, 2017 9:05:15 AM
To: users@zeppelin.apache.org<ma...@zeppelin.apache.org>
Subject: Livy - add external libraries from additional maven repo
Hi everyone,
I'm using Zeppelin with Livy 0.4 and trying to add external libraries from an additional maven repo to my application according to the documentation available here<https://zeppelin.apache.org/docs/0.7.0/interpreter/livy.html#adding-external-libraries>. The example works fine, but when I set the livy.spark.jars.packages to my library the interpreter throws an unresolved dependency error.
I have added the additional maven repository in the interpreter settings and have also tried setting livy.spark.jars.ivy but without luck. However, if I use the Spark interpreter with the following code it works fine.
"%dep
z.reset();
z.addRepo("my repo").url("http://myrepo"<http://myrepo>).snapshot
z.load("mygroup:myartifact:myversion");
Has anyone managed to do that with Livy? Thanks!
Cheers,
Theo
--
BENJAMIN VOGAN | Data Platform Team Lead
[https://docs.google.com/uc?export=download&id=0B2SW57jgJhgaOGNLSXhCaUNoMzQ&revid=0B2SW57jgJhgaNm1lM1Q4NXlrN1VkcmQ5MG1TNWs1L0pOdkZBPQ]<http://www.shopkick.com/>
[https://docs.google.com/uc?export=download&id=0B2SW57jgJhgaU0o1Zy1YWmhNN0k&revid=0B2SW57jgJhgaMG1hZXJ0b0hQYzNvVnhyRlhFV3R0aDh6eEhVPQ]<https://www.facebook.com/shopkick>[https://docs.google.com/uc?export=download&id=0B2SW57jgJhgaNjNwelg2Y1lwdkE&revid=0B2SW57jgJhgaR0xZWG5xajFGQk5VRWtTNC9yV09lbmVQeDRRPQ]<https://www.instagram.com/shopkick/>[https://docs.google.com/uc?export=download&id=0B2SW57jgJhgaTU1MQjdpaG96N0E&revid=0B2SW57jgJhgaazkyTkU1VlF5UFBrYmFuZklhMFA1T3NqZklvPQ]<https://www.pinterest.com/shopkick/>[https://docs.google.com/uc?export=download&id=0B2SW57jgJhgaQVJzbkc2TUVqOEU&revid=0B2SW57jgJhgaU2Jpb3hWQlZoVWFsQjl0aHBrVVZpMXp0VFYwPQ]<https://twitter.com/shopkickbiz>[https://docs.google.com/uc?export=download&id=0B2SW57jgJhgaMHBINExPa1NLTVE&revid=0B2SW57jgJhgacUlnaGRqNVJDNlhCTVlodHloeUR5aXNoZlp3PQ]<https://www.linkedin.com/company-beta/831240/?pathWildcard=831240>
Re: Livy - add external libraries from additional maven repo
Posted by Theofilos Kakantousis <tk...@kth.se>.
Thanks everyone for the feedback!
Indeed %dep works only for Spark interpreter, just mentioned it to show
the interpreter behavior I expected with Livy.
When setting my local maven repo and the "groupId:artifactId:version" in
the interpreter settings, I can see the dependency (i.e. a jar file)
being downloaded to the local OS under "local-repo" directory but the
dependency is not deployed with the Spark application in YARN.
Cheers,
Theo
On 2017-05-31 01:19, Ben Vogan wrote:
> For what it's worth I have successfully added jar files and maven
> packages to sessions using zeppelin & livy 0.3 - although not using
> %dep. In the interpreter settings I set the livy.spark.jars setting
> for jars that are on my HDFS cluster, and livy.spark.jars.packages for
> maven packages - although only using maven central and not a local repo.
>
> --Ben
>
> On Tue, May 30, 2017 at 12:36 PM, Felix Cheung
> <felixcheung_m@hotmail.com <ma...@hotmail.com>> wrote:
>
> To add, this might be an issue with Livy.
>
> I'm seeing something similar as well.
>
> If you can get a repo with calling the Livy REST API directly it
> will be worthwhile to follow up with the Livy community separately.
>
>
> ------------------------------------------------------------------------
> *From:* Felix Cheung <felixcheung_m@hotmail.com
> <ma...@hotmail.com>>
> *Sent:* Tuesday, May 30, 2017 11:34:31 AM
> *To:* users@zeppelin.apache.org
> <ma...@zeppelin.apache.org>; users@zeppelin.apache.org
> <ma...@zeppelin.apache.org>
> *Subject:* Re: Livy - add external libraries from additional maven
> repo
> if I recall, %dep only works with the built in Spark interpreter
> and not the Livy interpreter.
>
> To manage dependency win Livy you will need to set Spark conf with
> Livy.
>
> ------------------------------------------------------------------------
> *From:* Theofilos Kakantousis <tkak@kth.se <ma...@kth.se>>
> *Sent:* Tuesday, May 30, 2017 9:05:15 AM
> *To:* users@zeppelin.apache.org <ma...@zeppelin.apache.org>
> *Subject:* Livy - add external libraries from additional maven repo
> Hi everyone,
>
> I'm using Zeppelin with Livy 0.4 and trying to add external
> libraries from an additional maven repo to my application
> according to the documentation available here
> <https://zeppelin.apache.org/docs/0.7.0/interpreter/livy.html#adding-external-libraries>.
> The example works fine, but when I set the
> livy.spark.jars.packages to my library the interpreter throws an
> unresolved dependency error.
>
> I have added the additional maven repository in the interpreter
> settings and have also tried setting livy.spark.jars.ivy but
> without luck. However, if I use the Spark interpreter with the
> following code it works fine.
>
> "%dep
> z.reset();
> z.addRepo("my repo").url("http://myrepo" <http://myrepo>).snapshot
> z.load("mygroup:myartifact:myversion");
>
> Has anyone managed to do that with Livy? Thanks!
>
> Cheers,
> Theo
>
>
>
>
> --
> *BENJAMIN VOGAN*| Data Platform Team Lead
>
> <http://www.shopkick.com/>
> <https://www.facebook.com/shopkick><https://www.instagram.com/shopkick/><https://www.pinterest.com/shopkick/><https://twitter.com/shopkickbiz><https://www.linkedin.com/company-beta/831240/?pathWildcard=831240>
Re: Livy - add external libraries from additional maven repo
Posted by Ben Vogan <be...@shopkick.com>.
For what it's worth I have successfully added jar files and maven packages
to sessions using zeppelin & livy 0.3 - although not using %dep. In the
interpreter settings I set the livy.spark.jars setting for jars that are on
my HDFS cluster, and livy.spark.jars.packages for maven packages - although
only using maven central and not a local repo.
--Ben
On Tue, May 30, 2017 at 12:36 PM, Felix Cheung <fe...@hotmail.com>
wrote:
> To add, this might be an issue with Livy.
>
> I'm seeing something similar as well.
>
> If you can get a repo with calling the Livy REST API directly it will be
> worthwhile to follow up with the Livy community separately.
>
>
> ------------------------------
> *From:* Felix Cheung <fe...@hotmail.com>
> *Sent:* Tuesday, May 30, 2017 11:34:31 AM
> *To:* users@zeppelin.apache.org; users@zeppelin.apache.org
> *Subject:* Re: Livy - add external libraries from additional maven repo
>
> if I recall, %dep only works with the built in Spark interpreter and not
> the Livy interpreter.
>
> To manage dependency win Livy you will need to set Spark conf with Livy.
>
> ------------------------------
> *From:* Theofilos Kakantousis <tk...@kth.se>
> *Sent:* Tuesday, May 30, 2017 9:05:15 AM
> *To:* users@zeppelin.apache.org
> *Subject:* Livy - add external libraries from additional maven repo
>
> Hi everyone,
>
> I'm using Zeppelin with Livy 0.4 and trying to add external libraries from
> an additional maven repo to my application according to the documentation
> available here
> <https://zeppelin.apache.org/docs/0.7.0/interpreter/livy.html#adding-external-libraries>.
> The example works fine, but when I set the livy.spark.jars.packages to my
> library the interpreter throws an unresolved dependency error.
>
> I have added the additional maven repository in the interpreter settings
> and have also tried setting livy.spark.jars.ivy but without luck. However,
> if I use the Spark interpreter with the following code it works fine.
>
> "%dep
> z.reset();
> z.addRepo("my repo").url("http://myrepo" <http://myrepo>).snapshot
> z.load("mygroup:myartifact:myversion");
>
> Has anyone managed to do that with Livy? Thanks!
>
> Cheers,
> Theo
>
--
*BENJAMIN VOGAN* | Data Platform Team Lead
<http://www.shopkick.com/>
<https://www.facebook.com/shopkick> <https://www.instagram.com/shopkick/>
<https://www.pinterest.com/shopkick/> <https://twitter.com/shopkickbiz>
<https://www.linkedin.com/company-beta/831240/?pathWildcard=831240>
Re: Livy - add external libraries from additional maven repo
Posted by Felix Cheung <fe...@hotmail.com>.
To add, this might be an issue with Livy.
I'm seeing something similar as well.
If you can get a repo with calling the Livy REST API directly it will be worthwhile to follow up with the Livy community separately.
________________________________
From: Felix Cheung <fe...@hotmail.com>
Sent: Tuesday, May 30, 2017 11:34:31 AM
To: users@zeppelin.apache.org; users@zeppelin.apache.org
Subject: Re: Livy - add external libraries from additional maven repo
if I recall, %dep only works with the built in Spark interpreter and not the Livy interpreter.
To manage dependency win Livy you will need to set Spark conf with Livy.
________________________________
From: Theofilos Kakantousis <tk...@kth.se>
Sent: Tuesday, May 30, 2017 9:05:15 AM
To: users@zeppelin.apache.org
Subject: Livy - add external libraries from additional maven repo
Hi everyone,
I'm using Zeppelin with Livy 0.4 and trying to add external libraries from an additional maven repo to my application according to the documentation available here<https://zeppelin.apache.org/docs/0.7.0/interpreter/livy.html#adding-external-libraries>. The example works fine, but when I set the livy.spark.jars.packages to my library the interpreter throws an unresolved dependency error.
I have added the additional maven repository in the interpreter settings and have also tried setting livy.spark.jars.ivy but without luck. However, if I use the Spark interpreter with the following code it works fine.
"%dep
z.reset();
z.addRepo("my repo").url("http://myrepo"<http://myrepo>).snapshot
z.load("mygroup:myartifact:myversion");
Has anyone managed to do that with Livy? Thanks!
Cheers,
Theo
Re: Livy - add external libraries from additional maven repo
Posted by Felix Cheung <fe...@hotmail.com>.
if I recall, %dep only works with the built in Spark interpreter and not the Livy interpreter.
To manage dependency win Livy you will need to set Spark conf with Livy.
________________________________
From: Theofilos Kakantousis <tk...@kth.se>
Sent: Tuesday, May 30, 2017 9:05:15 AM
To: users@zeppelin.apache.org
Subject: Livy - add external libraries from additional maven repo
Hi everyone,
I'm using Zeppelin with Livy 0.4 and trying to add external libraries from an additional maven repo to my application according to the documentation available here<https://zeppelin.apache.org/docs/0.7.0/interpreter/livy.html#adding-external-libraries>. The example works fine, but when I set the livy.spark.jars.packages to my library the interpreter throws an unresolved dependency error.
I have added the additional maven repository in the interpreter settings and have also tried setting livy.spark.jars.ivy but without luck. However, if I use the Spark interpreter with the following code it works fine.
"%dep
z.reset();
z.addRepo("my repo").url("http://myrepo"<http://myrepo>).snapshot
z.load("mygroup:myartifact:myversion");
Has anyone managed to do that with Livy? Thanks!
Cheers,
Theo