You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by minahlee <gi...@git.apache.org> on 2016/02/21 13:08:19 UTC
[GitHub] incubator-zeppelin pull request: Fix pyspark to work on yarn mode ...
GitHub user minahlee opened a pull request:
https://github.com/apache/incubator-zeppelin/pull/736
Fix pyspark to work on yarn mode when spark version is lower than or equal to 1.4.x
### What is this PR for?
pyspark.zip, py4j-\*.zip should be distributed to yarn nodes to make pyspark function but this hasn't been working after #463 because [`if (pythonLibs.length == pythonLibUris.size())`](https://github.com/apache/incubator-zeppelin/blob/master/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java#L329) condition will never be true. This PR fixes this issue by changing this if condition to be `pythonlibUris.size() == 2`, while integer 2 refers pyspark.zip and py4j-\*.zip.
In addition, yarn-install documentation has been updated.
### What type of PR is it?
Bug Fix
### Is there a relevant Jira issue?
No. But the issue has reported via [user mailing list](http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Can-t-get-Pyspark-1-4-1-interpreter-to-work-on-Zeppelin-0-6-td2229.html#a2259) by Ian Maloney
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/minahlee/incubator-zeppelin fix/pyspark_on_yarn
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-zeppelin/pull/736.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #736
----
commit 6465ba89025919adcaa3f884459eb2992ed9f33e
Author: Mina Lee <mi...@nflabs.com>
Date: 2016-02-21T11:24:27Z
Change condition to make pyspark, py4j libraries be distributed to yarn executors
commit c544decd16555147c498caa1c14fa8b25db091c3
Author: Mina Lee <mi...@nflabs.com>
Date: 2016-02-21T11:31:01Z
[DOC] Remove redundant Zeppelin build information from yarn_install.md
[DOC] Guide users to set SPARK_HOME to use spark in yarn mode
[DOC] Change spark version to the latest in yarn config example
[DOC] Add note that spark for cdh4 doesn't support yarn
[DOC] Remove spark properties `spark.home` and `spark.yarn.jar` from doc which doesn't work on zeppelin anymore
[DOC] Fix typos
[DOC] Add info that embedded spark doesn't work on yarn mode anymore when Spark version is 1.5.0 or higher in README.md
commit 2710c46240a2c047e97e52cd52316d9429185125
Author: Mina Lee <mi...@nflabs.com>
Date: 2016-02-21T11:45:14Z
[DOC] Remove invalid information of installation location
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Fix pyspark to work on yarn mode ...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/736#issuecomment-187889549
Looks good to me. and merge if there're no more discussions
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Fix pyspark to work on yarn mode ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-zeppelin/pull/736
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---