You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by mfelgamal <gi...@git.apache.org> on 2016/07/31 08:02:27 UTC
[GitHub] zeppelin pull request #1255: Adding extra libraries to livy
GitHub user mfelgamal opened a pull request:
https://github.com/apache/zeppelin/pull/1255
Adding extra libraries to livy
### What is this PR for?
Adding extra libraries to livy interpreter which isn't exist by default.
### What type of PR is it?
[ Improvement ]
### Todos
* [Test case ] - Task
### What is the Jira issue?
* [ZEPPELIN-1258]
### How should this be tested?
- Create new livy interpreter or modify the default.
- Set `livy.spark.driver.extraClassPath` and `livy.spark.executor.extraClassPath` to path of libraries.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/novelari/incubator-zeppelin feature/modifying-livy-doc
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1255.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1255
----
commit 2586651ab9768e484e04a840978d6db8d5659f48
Author: Fouad <fu...@gmail.com>
Date: 2016-05-29T14:39:08Z
Merge pull request #2 from apache/master
update from original
commit 03a1e80f8f3115a75c42fa25fdaa06e7dd2cbc28
Author: mfelgamal <ma...@gmail.com>
Date: 2016-07-19T18:28:46Z
Merge pull request #4 from apache/master
merge master
commit 98090abc075b97e3896ae6ea0c6e48f57e8d6434
Author: mahmoudelgamal <ma...@gmail.com>
Date: 2016-07-19T22:12:54Z
Adding external libraries to livy
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1255: Adding extra libraries to livy
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1255
@jimdowling Right, there's no ZeppelinContext in livy right now. Would allowing user to specify jars/packages when creating livy session more general ? (like --jars/--packages in spark-submit)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1255: Adding extra libraries to livy
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/1255
could you update the title of this PR and JIRA ZEPPELIN-1258, if we are changing how we would approach this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1255: Adding extra libraries to livy
Posted by jimdowling <gi...@git.apache.org>.
Github user jimdowling commented on the issue:
https://github.com/apache/zeppelin/pull/1255
Do you mean by modifying the UI on notebook creation or using a param to selecting the interpreter like
'%livy --jars [path]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1255: Adding extra libraries to livy
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/1255
how about "Add Spark packages support to Livy interpreter"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1255: Adding extra libraries to livy
Posted by mfelgamal <gi...@git.apache.org>.
Github user mfelgamal commented on the issue:
https://github.com/apache/zeppelin/pull/1255
@felixcheung I suggests this title "Adding packages to livy interpreter", Is it Ok ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1255: Adding extra libraries to livy
Posted by jimdowling <gi...@git.apache.org>.
Github user jimdowling commented on the issue:
https://github.com/apache/zeppelin/pull/1255
@zjffdu - I don't think there's a ZeppelinContext available in Livy. Would this be acceptable until the ZeppelinContext becomes available? As of now, Livy is not that much use if you cannot add libraries to your Spark app.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1255: Adding extra libraries to livy
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1255
You can use --conf spark.jars=<jars> and --conf spark.jars.packages=<packages> instead
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1255: [ZEPPELIN-1258] Add Spark packages support to Livy int...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/1255
That's a fair point.
I'll merge if there is no more discussion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1255: [ZEPPELIN-1258] Add Spark packages support to Livy int...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/1255
There is a test failure?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1255: Adding extra libraries to livy
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1255
No, I mean add interpreter property to allow user to specify jars/packages.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1255: Adding extra libraries to livy
Posted by mfelgamal <gi...@git.apache.org>.
Github user mfelgamal commented on the issue:
https://github.com/apache/zeppelin/pull/1255
@zjffdu I added `spark.jars.packages` property in the recent commit, and it works correctly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1255: Adding extra libraries to livy
Posted by mfelgamal <gi...@git.apache.org>.
Github user mfelgamal commented on the issue:
https://github.com/apache/zeppelin/pull/1255
@zjffdu do you have any further comments ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1255: [ZEPPELIN-1258] Add Spark packages support to Livy int...
Posted by mfelgamal <gi...@git.apache.org>.
Github user mfelgamal commented on the issue:
https://github.com/apache/zeppelin/pull/1255
@felixcheung : This PR does not change the code. It is just a change in the configurations. So, it should not affect the tests
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1255: Add Spark packages support to Livy interpreter
Posted by mfelgamal <gi...@git.apache.org>.
Github user mfelgamal commented on the issue:
https://github.com/apache/zeppelin/pull/1255
@felixcheung done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1255: Adding extra libraries to livy
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1255
I would not recommend user to do that to add extra library. Because most of time zeppelin user don't know the cluster details like which extra library are located in the cluster machines. I would suggest livy to support dynamic library loading as native spark interpreter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1255: [ZEPPELIN-1258] Add Spark packages support to L...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/1255
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1255: [ZEPPELIN-1258] Add Spark packages support to Livy int...
Posted by mfelgamal <gi...@git.apache.org>.
Github user mfelgamal commented on the issue:
https://github.com/apache/zeppelin/pull/1255
@felixcheung do you have any further comments ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1255: Adding extra libraries to livy
Posted by mfelgamal <gi...@git.apache.org>.
Github user mfelgamal commented on the issue:
https://github.com/apache/zeppelin/pull/1255
I think that the livy itself doesn't support add (--jars/--packages property in spark-submit)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---