You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by zjffdu <gi...@git.apache.org> on 2016/09/12 02:35:57 UTC
[GitHub] zeppelin pull request #1423: ZEPPELIN-1425. sparkr.zip is not distributed to...
GitHub user zjffdu opened a pull request:
https://github.com/apache/zeppelin/pull/1423
ZEPPELIN-1425. sparkr.zip is not distributed to executors
### What is this PR for?
sparkr.zip is not distrubuted to executor, so any sparkR job that requrie R daemon in executor will fail. This PR would add sparkr.zip into `spark.yarn.dist.archives`.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1425
### How should this be tested?
Run the following code
```
%spark.r
df <- createDataFrame(sqlContext, mtcars)
showDF(df)
```
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1425
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1423.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1423
----
commit 145a8dcf54fd325bdbc49568fcd69b73987ee516
Author: Jeff Zhang <zj...@apache.org>
Date: 2016-09-12T01:51:06Z
ZEPPELIN-1425. sparkr.zip is not distributed to executors
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1423: ZEPPELIN-1425. sparkr.zip is not distributed to execut...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/1423
@zjffdu Do you think [rpkg.zip](https://github.com/apache/spark/blob/v1.6.2/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L394) is the something also need to be taken care?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1423: ZEPPELIN-1425. sparkr.zip is not distributed to execut...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1423
@Leemoonsoo The `rpkg.zip` is for the case that user have R code inside of jars specified through --jars. The reason why I didn't include it here is that I have to duplicate lots of code from spark (most of them is private, I have to copy them to zeppelin). Besides I think very few people would include R inside of jars. So I think it is acceptable to not include it for now.
BTW, I think zeppelin still duplicate lots of code of `SparkSubmit`, I plan to improve them in `ZEPPELIN-1263`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1423: ZEPPELIN-1425. sparkr.zip is not distributed to execut...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/1423
Thanks for the explanation.
LGTM and merge if there're no more discussions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1423: ZEPPELIN-1425. sparkr.zip is not distributed to execut...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1423
@Leemoonsoo Please help review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1423: ZEPPELIN-1425. sparkr.zip is not distributed to...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/1423
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---