You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by tae-jun <gi...@git.apache.org> on 2016/08/29 12:06:59 UTC
[GitHub] zeppelin pull request #1381: [ZEPPELIN-1390] SparkInterpreter does not work ...
GitHub user tae-jun opened a pull request:
https://github.com/apache/zeppelin/pull/1381
[ZEPPELIN-1390] SparkInterpreter does not work for Spark2 version of HDP 2.5
### What is this PR for?
Spark2 version of HDP 2.5 is "2.0.0.2.5.0.0-1245". Currently, Zeppelin parses this version to integer 2002500, which is higher than version "2.0.0" (i.e. 200 in integer). Therefore, Zeppelin thinks it's not supported version and fail.
I fixed some codes to consider major, minor, and patch version. Now, version parsed using only first 3 numbers (major, minor, patch) and parsed into a 5-digit integer. For example, 2.0.0 -> 20000, 1.6.2 -> 10602, 1.10.1 -> 11001. This makes HDP 2.5 compatible, and also make it more robust than before. Since maybe Spark's minor version can be a 2-digit number in the future
### What type of PR is it?
[Bug Fix]
### What is the Jira issue?
[ZEPPELIN-1390](https://issues.apache.org/jira/browse/ZEPPELIN-1390)
### How should this be tested?
I added SparkVersion test codes.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tae-jun/zeppelin ZEPPELIN-1390
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1381.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1381
----
commit 6ad8049da5cc4b9a2fadf48b06a3e42f8bee30ca
Author: Jun <i2...@gmail.com>
Date: 2016-08-29T12:02:20Z
[ZEPPELIN-1390] SparkInterpreter does not work for Spark2 version of HDP 2.5
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1381: [ZEPPELIN-1390] SparkInterpreter does not work for Spa...
Posted by minahlee <gi...@git.apache.org>.
Github user minahlee commented on the issue:
https://github.com/apache/zeppelin/pull/1381
CI is green now. Shall we merge this guy?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1381: [ZEPPELIN-1390] SparkInterpreter does not work for Spa...
Posted by tae-jun <gi...@git.apache.org>.
Github user tae-jun commented on the issue:
https://github.com/apache/zeppelin/pull/1381
Thanks everyone!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1381: [ZEPPELIN-1390] SparkInterpreter does not work for Spa...
Posted by tae-jun <gi...@git.apache.org>.
Github user tae-jun commented on the issue:
https://github.com/apache/zeppelin/pull/1381
I upgraded HDP to 2.5 from 2.4 today. sc.version printed 1.6.1 when HDP 2.4 and now HDP 2.5 spark.version prints 2.0.0.2.5.0.0-1245
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1381: [ZEPPELIN-1390] SparkInterpreter does not work for Spa...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/1381
@tae-jun Thanks for the contribution.
I think there're few more places related to this change
Pyspark interpreter
https://github.com/apache/zeppelin/blob/master/spark/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java#L181
https://github.com/apache/zeppelin/blob/master/spark/src/main/resources/python/zeppelin_pyspark.py#L109
SparkR interpreter
https://github.com/apache/zeppelin/blob/master/spark/src/main/java/org/apache/zeppelin/spark/ZeppelinR.java#L143
https://github.com/apache/zeppelin/blob/master/spark/src/main/resources/R/zeppelin_sparkr.R#L45
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1381: [ZEPPELIN-1390] SparkInterpreter does not work for Spa...
Posted by tae-jun <gi...@git.apache.org>.
Github user tae-jun commented on the issue:
https://github.com/apache/zeppelin/pull/1381
Ok thanks
Let me know when it is done.
I will reopen PR :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1381: [ZEPPELIN-1390] SparkInterpreter does not work for Spa...
Posted by tae-jun <gi...@git.apache.org>.
Github user tae-jun commented on the issue:
https://github.com/apache/zeppelin/pull/1381
@Leemoonsoo Yep, I did it :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1381: [ZEPPELIN-1390] SparkInterpreter does not work ...
Posted by tae-jun <gi...@git.apache.org>.
Github user tae-jun closed the pull request at:
https://github.com/apache/zeppelin/pull/1381
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1381: [ZEPPELIN-1390] SparkInterpreter does not work for Spa...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1381
Tested on HDP2.5, on executing both spark and pyspark `sc.version` prints `res0: String = 2.0.0.2.5.0.0-1245` and works as expected.
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1381: [ZEPPELIN-1390] SparkInterpreter does not work ...
Posted by tae-jun <gi...@git.apache.org>.
GitHub user tae-jun reopened a pull request:
https://github.com/apache/zeppelin/pull/1381
[ZEPPELIN-1390] SparkInterpreter does not work for Spark2 version of HDP 2.5
### What is this PR for?
Spark2 version of HDP 2.5 is "2.0.0.2.5.0.0-1245". Currently, Zeppelin parses this version to integer 2002500, which is higher than version "2.0.0" (i.e. 200 in integer). Therefore, Zeppelin thinks it's not supported version and fail.
I fixed some codes to consider major, minor, and patch version. Now, version parsed using only first 3 numbers (major, minor, patch) and parsed into a 5-digit integer. For example, 2.0.0 -> 20000, 1.6.2 -> 10602, 1.10.1 -> 11001. This makes HDP 2.5 compatible, and also make it more robust than before. Since maybe Spark's minor version can be a 2-digit number in the future
### What type of PR is it?
[Bug Fix]
### What is the Jira issue?
[ZEPPELIN-1390](https://issues.apache.org/jira/browse/ZEPPELIN-1390)
### How should this be tested?
I added SparkVersion test codes.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tae-jun/zeppelin ZEPPELIN-1390
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1381.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1381
----
commit 6ad8049da5cc4b9a2fadf48b06a3e42f8bee30ca
Author: Jun <i2...@gmail.com>
Date: 2016-08-29T12:02:20Z
[ZEPPELIN-1390] SparkInterpreter does not work for Spark2 version of HDP 2.5
commit f5531ab2733d2cc7fa92c93c0d2e41d58e761e35
Author: Jun <i2...@gmail.com>
Date: 2016-08-31T03:35:14Z
[ZEPPELIN-1390] Fix Python and R number versions
commit bae5264c196c0e332588e7388fd2e1bebbdd6876
Author: Jun <i2...@gmail.com>
Date: 2016-09-04T01:53:32Z
Merge remote-tracking branch 'origin/master' into ZEPPELIN-1390
* origin/master: (21 commits)
Buffer append output results + fix extra incorrect results
[ZEPPELIN-1279] Spark on Mesos Docker.
ZEPPELIN-1374. Should prevent use dot in interpreter name
[ZEPPELIN-1069]Ignore implicit interpreter when user enter wrong interpreter name
ZEPPELIN-1319 Use absolute path for ssl truststore and keystore when available
[ZEPPELIN-1366] Removed legacy JDBC alias
Cache zeppelin-web/node_modules on travis ci and see if it reduces CI failure on npm install
[MINOR] Add new line before logging paragraph content
[ZEPPELIN-1391][Interpreters] print error while existing registedInterpreter with the same key but different settings
[ZEPPELIN-1365] Error of Zeppelin Application in development mode.
ZEPPELIN-1384. Spark interpreter binary compatibility to scala 2.10 / 2.11 broken
[ZEPPELIN-1379] Flink interpreter is missing scala libraries
[ZEPPELIN-699] Add new synchronous paragraph run REST API
[DOC]fix some spelling mistakes
[ZEPPELIN-1217] Remove horizontal scrollbar in Zeppelin conf table
ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should not use ZEPPELIN_JAVA_OPTS
[ZEPPELIN-1313] NullPointerException when using Clone notebook REST API
[ZEPPELIN-1040] Show the time when the result is updated
[zeppelin] add temp directories generated by zeppelin-Rinterpreter to gitignore
[MINOR] Removed unused profiles from spark/pom.xml
...
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1381: [ZEPPELIN-1390] SparkInterpreter does not work for Spa...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/1381
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1381: [ZEPPELIN-1390] SparkInterpreter does not work for Spa...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/1381
@tae-jun, could you trigger ci (close / reopen pullrequest will trigger ci) and see if it passes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1381: [ZEPPELIN-1390] SparkInterpreter does not work ...
Posted by tae-jun <gi...@git.apache.org>.
GitHub user tae-jun reopened a pull request:
https://github.com/apache/zeppelin/pull/1381
[ZEPPELIN-1390] SparkInterpreter does not work for Spark2 version of HDP 2.5
### What is this PR for?
Spark2 version of HDP 2.5 is "2.0.0.2.5.0.0-1245". Currently, Zeppelin parses this version to integer 2002500, which is higher than version "2.0.0" (i.e. 200 in integer). Therefore, Zeppelin thinks it's not supported version and fail.
I fixed some codes to consider major, minor, and patch version. Now, version parsed using only first 3 numbers (major, minor, patch) and parsed into a 5-digit integer. For example, 2.0.0 -> 20000, 1.6.2 -> 10602, 1.10.1 -> 11001. This makes HDP 2.5 compatible, and also make it more robust than before. Since maybe Spark's minor version can be a 2-digit number in the future
### What type of PR is it?
[Bug Fix]
### What is the Jira issue?
[ZEPPELIN-1390](https://issues.apache.org/jira/browse/ZEPPELIN-1390)
### How should this be tested?
I added SparkVersion test codes.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tae-jun/zeppelin ZEPPELIN-1390
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1381.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1381
----
commit 6ad8049da5cc4b9a2fadf48b06a3e42f8bee30ca
Author: Jun <i2...@gmail.com>
Date: 2016-08-29T12:02:20Z
[ZEPPELIN-1390] SparkInterpreter does not work for Spark2 version of HDP 2.5
commit f5531ab2733d2cc7fa92c93c0d2e41d58e761e35
Author: Jun <i2...@gmail.com>
Date: 2016-08-31T03:35:14Z
[ZEPPELIN-1390] Fix Python and R number versions
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1381: [ZEPPELIN-1390] SparkInterpreter does not work ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/1381
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1381: [ZEPPELIN-1390] SparkInterpreter does not work for Spa...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/1381
I think master branch has CI build problem. Let me try to fix it first.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1381: [ZEPPELIN-1390] SparkInterpreter does not work for Spa...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/1381
Let's merge!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1381: [ZEPPELIN-1390] SparkInterpreter does not work ...
Posted by tae-jun <gi...@git.apache.org>.
Github user tae-jun closed the pull request at:
https://github.com/apache/zeppelin/pull/1381
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1381: [ZEPPELIN-1390] SparkInterpreter does not work for Spa...
Posted by tae-jun <gi...@git.apache.org>.
Github user tae-jun commented on the issue:
https://github.com/apache/zeppelin/pull/1381
@Leemoonsoo Thanks for comment!
I made some changes.
How does it look?
Thanks again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1381: [ZEPPELIN-1390] SparkInterpreter does not work ...
Posted by tae-jun <gi...@git.apache.org>.
GitHub user tae-jun reopened a pull request:
https://github.com/apache/zeppelin/pull/1381
[ZEPPELIN-1390] SparkInterpreter does not work for Spark2 version of HDP 2.5
### What is this PR for?
Spark2 version of HDP 2.5 is "2.0.0.2.5.0.0-1245". Currently, Zeppelin parses this version to integer 2002500, which is higher than version "2.0.0" (i.e. 200 in integer). Therefore, Zeppelin thinks it's not supported version and fail.
I fixed some codes to consider major, minor, and patch version. Now, version parsed using only first 3 numbers (major, minor, patch) and parsed into a 5-digit integer. For example, 2.0.0 -> 20000, 1.6.2 -> 10602, 1.10.1 -> 11001. This makes HDP 2.5 compatible, and also make it more robust than before. Since maybe Spark's minor version can be a 2-digit number in the future
### What type of PR is it?
[Bug Fix]
### What is the Jira issue?
[ZEPPELIN-1390](https://issues.apache.org/jira/browse/ZEPPELIN-1390)
### How should this be tested?
I added SparkVersion test codes.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tae-jun/zeppelin ZEPPELIN-1390
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1381.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1381
----
commit 6ad8049da5cc4b9a2fadf48b06a3e42f8bee30ca
Author: Jun <i2...@gmail.com>
Date: 2016-08-29T12:02:20Z
[ZEPPELIN-1390] SparkInterpreter does not work for Spark2 version of HDP 2.5
commit f5531ab2733d2cc7fa92c93c0d2e41d58e761e35
Author: Jun <i2...@gmail.com>
Date: 2016-08-31T03:35:14Z
[ZEPPELIN-1390] Fix Python and R number versions
commit bae5264c196c0e332588e7388fd2e1bebbdd6876
Author: Jun <i2...@gmail.com>
Date: 2016-09-04T01:53:32Z
Merge remote-tracking branch 'origin/master' into ZEPPELIN-1390
* origin/master: (21 commits)
Buffer append output results + fix extra incorrect results
[ZEPPELIN-1279] Spark on Mesos Docker.
ZEPPELIN-1374. Should prevent use dot in interpreter name
[ZEPPELIN-1069]Ignore implicit interpreter when user enter wrong interpreter name
ZEPPELIN-1319 Use absolute path for ssl truststore and keystore when available
[ZEPPELIN-1366] Removed legacy JDBC alias
Cache zeppelin-web/node_modules on travis ci and see if it reduces CI failure on npm install
[MINOR] Add new line before logging paragraph content
[ZEPPELIN-1391][Interpreters] print error while existing registedInterpreter with the same key but different settings
[ZEPPELIN-1365] Error of Zeppelin Application in development mode.
ZEPPELIN-1384. Spark interpreter binary compatibility to scala 2.10 / 2.11 broken
[ZEPPELIN-1379] Flink interpreter is missing scala libraries
[ZEPPELIN-699] Add new synchronous paragraph run REST API
[DOC]fix some spelling mistakes
[ZEPPELIN-1217] Remove horizontal scrollbar in Zeppelin conf table
ZEPPELIN-1185. ZEPPELIN_INTP_JAVA_OPTS should not use ZEPPELIN_JAVA_OPTS
[ZEPPELIN-1313] NullPointerException when using Clone notebook REST API
[ZEPPELIN-1040] Show the time when the result is updated
[zeppelin] add temp directories generated by zeppelin-Rinterpreter to gitignore
[MINOR] Removed unused profiles from spark/pom.xml
...
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1381: [ZEPPELIN-1390] SparkInterpreter does not work for Spa...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/1381
First 3 ci test profile fails with the same error.
```
No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.
The build has been terminated
```
I'm not sure this problem comes from this PR or not, but i think the problem should be resolved before we merge this PR.
@tae-jun, could you rebase or merge master and see how CI-test goes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1381: [ZEPPELIN-1390] SparkInterpreter does not work for Spa...
Posted by tae-jun <gi...@git.apache.org>.
Github user tae-jun commented on the issue:
https://github.com/apache/zeppelin/pull/1381
@Leemoonsoo Hi, thanks for your guide.
I merged master branch, but it failed again.
And I tried to build again via reopening, but it fails again as you see.
However, it seems like other PRs fail with the same error.
What should I do???
Thanks for comment ^^
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1381: [ZEPPELIN-1390] SparkInterpreter does not work ...
Posted by tae-jun <gi...@git.apache.org>.
Github user tae-jun closed the pull request at:
https://github.com/apache/zeppelin/pull/1381
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1381: [ZEPPELIN-1390] SparkInterpreter does not work for Spa...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1381
@tae-jun Have you ever tried to start spark-shell to print the spark version in HDP 2.5 ? Because I think if the version is an issue, this should also happen in spark 1.6 in HDP 2.4
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---