You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by HyukjinKwon <gi...@git.apache.org> on 2018/10/17 14:58:52 UTC
[GitHub] zeppelin pull request #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/zeppelin/pull/3206
[WIP][ZEPPELIN-3810] Support Spark 2.4
### What is this PR for?
Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418).
There are two problems for this upgrade at Zeppelin side:
1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible.
See:
- https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala
- https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala
To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection.
2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit:
```
Caused by: java.lang.NoSuchMethodError:
jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)
```
To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`.
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Wait until Spark 2.4.0 is officially released.
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3810
### How should this be tested?
Verified manually against Spark 2.4.0 RC3
### Questions:
* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/zeppelin ZEPPELIN-3810
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/3206.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3206
----
commit e2d224aadf0e54533837a9a89f8e9d3586aee3a9
Author: hyukjinkwon <gu...@...>
Date: 2018-10-17T14:41:29Z
Support Spark 2.4
----
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Hey all ~ could this get in by any change maybe?
---
[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Thanks @HyukjinKwon Have you checked this PR (https://github.com/apache/zeppelin/pull/3034) for supporting scala 2.12
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/zeppelin/pull/3206
This should be ready for a look as is. I already tested Spark 2.4.0.
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/3206
It is merged to branch-0.8 and master branch. So yes it will be released in zeppelin 0.9.0, but I believe 0.8.1 will be released before zeppelin 0.9.0
---
[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Does that happen only with this code changes? The change here does not touch signature at `class SparkScala211Interpreter(` and the error message looks pretty unrelated. The whole change proposed here does not change any signature.
---
[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Nope, it will work for both 2.11.8 and 2.11.12. I manually checked. This change only uses the methods existing in both 2.11.8 and 2.11.12 at Scala.
---
[GitHub] zeppelin pull request #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon closed the pull request at:
https://github.com/apache/zeppelin/pull/3206
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Thanks for everyone, the only remaining thing is to update `.travis.yml` to make sure the test pass again spark 2.4
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Thank you, @zjffdu and ALL!
---
[GitHub] zeppelin pull request #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon closed the pull request at:
https://github.com/apache/zeppelin/pull/3206
---
[GitHub] zeppelin pull request #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
GitHub user HyukjinKwon reopened a pull request:
https://github.com/apache/zeppelin/pull/3206
[WIP][ZEPPELIN-3810] Support Spark 2.4
### What is this PR for?
Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418).
There are two problems for this upgrade at Zeppelin side:
1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible.
See:
- https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala
- https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala
To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection.
2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit:
```
Caused by: java.lang.NoSuchMethodError:
jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)
```
To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`.
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Wait until Spark 2.4.0 is officially released.
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3810
### How should this be tested?
Verified manually against Spark 2.4.0 RC3
### Questions:
* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/zeppelin ZEPPELIN-3810
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/3206.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3206
----
commit 9ac1797a7e175a73350d61a0e62b3e4f89b29b8f
Author: hyukjinkwon <gu...@...>
Date: 2018-10-17T14:41:29Z
Support Spark 2.4
----
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Ah, yeap. sure.
---
[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Thanks for confirmation.
---
[GitHub] zeppelin pull request #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
GitHub user HyukjinKwon reopened a pull request:
https://github.com/apache/zeppelin/pull/3206
[WIP][ZEPPELIN-3810] Support Spark 2.4
### What is this PR for?
Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418).
There are two problems for this upgrade at Zeppelin side:
1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible.
See:
- https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala
- https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala
To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection.
2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit:
```
Caused by: java.lang.NoSuchMethodError:
jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)
```
To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`.
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Wait until Spark 2.4.0 is officially released.
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3810
### How should this be tested?
Verified manually against Spark 2.4.0 RC3
### Questions:
* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/zeppelin ZEPPELIN-3810
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/3206.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3206
----
commit e2d224aadf0e54533837a9a89f8e9d3586aee3a9
Author: hyukjinkwon <gu...@...>
Date: 2018-10-17T14:41:29Z
Support Spark 2.4
----
---
[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by ajaygk95 <gi...@git.apache.org>.
Github user ajaygk95 commented on the issue:
https://github.com/apache/zeppelin/pull/3206
> picked
I was able to get this working by doing cherry-picking. We needed some changes in our environment not related to zeppelin, but for the usage of spark-images.
Thanks for the help.
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by alonshoham <gi...@git.apache.org>.
Github user alonshoham commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Thank you very much!
בתאריך יום א׳, 18 בנוב׳ 2018, 15:17, מאת Jeff Zhang <
notifications@github.com>:
> I will try to do it by the end of 2018
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/zeppelin/pull/3206#issuecomment-439692204>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AP-5fC-vkwaAJVD2G81Bq8JBKLaS1PQdks5uwV4GgaJpZM4XkINL>
> .
>
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/zeppelin/pull/3206
This fix is not released yet. This PR exactly fixes the problem you faced. This fix will be available in the next release of Zeppelin.
---
[GitHub] zeppelin pull request #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
GitHub user HyukjinKwon reopened a pull request:
https://github.com/apache/zeppelin/pull/3206
[WIP][ZEPPELIN-3810] Support Spark 2.4
### What is this PR for?
Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418).
There are two problems for this upgrade at Zeppelin side:
1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible.
See:
- https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala
- https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala
To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection.
2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit:
```
Caused by: java.lang.NoSuchMethodError:
jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)
```
To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`.
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Wait until Spark 2.4.0 is officially released.
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3810
### How should this be tested?
Verified manually against Spark 2.4.0 RC3
### Questions:
* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/zeppelin ZEPPELIN-3810
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/3206.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3206
----
commit e2d224aadf0e54533837a9a89f8e9d3586aee3a9
Author: hyukjinkwon <gu...@...>
Date: 2018-10-17T14:41:29Z
Support Spark 2.4
----
---
[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/zeppelin/pull/3206
oops. I haven't. Will check that too while I am here.
BTW, my understanding is that we need this one as well since Spark still can be compiled against Scala 2.11.x, am I in the right way?
---
[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/zeppelin/pull/3206
This is a WIP. We should wait for Spark 2.4.0.
cc @zjffdu and @felixcheung
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/3206
@HyukjinKwon I don't see the official announcement of spark 2.4 release. Zeppelin test also depends on the binary distribution of spark, so we have to wait for the official binary release of spark 2.4
---
Re: [GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by Jeff Zhang <zj...@gmail.com>.
It is supposed to be in this month. Stay tuned.
G, Ajay (Nokia - IN/Bangalore) <aj...@nokia.com> 于2019年1月2日周三 下午9:09写道：
> Hello,
>
> Can you please share when zeppelin-0.8.1 will be released ?
>
> Thanks and Regards,
> Ajay G
>
> -----Original Message-----
> From: zjffdu <gi...@git.apache.org>
> Sent: Sunday, November 18, 2018 6:48 PM
> To: dev@zeppelin.incubator.apache.org
> Subject: [GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
>
> Github user zjffdu commented on the issue:
>
> https://github.com/apache/zeppelin/pull/3206
>
> I will try to do it by the end of 2018
>
>
> ---
>
--
Best Regards
Jeff Zhang
RE: [GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by "G, Ajay (Nokia - IN/Bangalore)" <aj...@nokia.com>.
Hello,
Can you please share when zeppelin-0.8.1 will be released ?
Thanks and Regards,
Ajay G
-----Original Message-----
From: zjffdu <gi...@git.apache.org>
Sent: Sunday, November 18, 2018 6:48 PM
To: dev@zeppelin.incubator.apache.org
Subject: [GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/3206
I will try to do it by the end of 2018
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/3206
I will try to do it by the end of 2018
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/3206
@HyukjinKwon I created one PR for you to add test for spark 2.4, would mind to merge that ? https://github.com/HyukjinKwon/zeppelin/pull/1
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Thank you all!!
---
[GitHub] zeppelin pull request #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon closed the pull request at:
https://github.com/apache/zeppelin/pull/3206
---
[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/zeppelin/pull/3206
I also tested this patch against Spark RC5.
---
[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/3206
is this going to break, say spark 2.3 with scala 2.11.8?
---
[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Awesome @HyukjinKwon Let's wait for spark 2.4 release
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Thank you guys!
---
[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Hi, @Leemoonsoo , @zjffdu , @felixcheung , @HyukjinKwon , @ajaygk95 .
Spark 2.4.0 is on the Maven Central. Can we restart this PR?
---
[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by ajaygk95 <gi...@git.apache.org>.
Github user ajaygk95 commented on the issue:
https://github.com/apache/zeppelin/pull/3206
>
>
> Does that happen only with this code changes? The change here does not touch signature at `class SparkScala211Interpreter(` and the error message looks pretty unrelated. The whole change proposed here does not change any signature.
Yes. With the original 0.8.0 source code I'm able to build.
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by antonkulaga <gi...@git.apache.org>.
Github user antonkulaga commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Looking forward to seeing it merged and Zeppelin 0.9.0 released, Spark 2.4 fixes some very nasty bugs
---
[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/zeppelin/pull/3206
BTW, I tested this via my Travis CI - https://travis-ci.org/HyukjinKwon/zeppelin/builds/448215776. Tests seems got passed.
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by alonshoham <gi...@git.apache.org>.
Github user alonshoham commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Hello,
Is this PR going to be officially released soon as part of zeppelin 0.9.0?
Thank you
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/3206
That's correct, @dongjoon-hyun
---
[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/zeppelin/pull/3206
It should be usable if the changes is cherry-picked properly. This PR basically just replace one line:
https://github.com/apache/zeppelin/blob/v0.8.0/spark/scala-2.11/src/main/scala/org/apache/zeppelin/spark/SparkScala211Interpreter.scala#L84
to a private function.
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by NicolasDutronc <gi...@git.apache.org>.
Github user NicolasDutronc commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Hi all, I'm using Zeppelin with the official Docker image and I'm still getting `java.lang.NoSuchMethodException: scala.tools.nsc.interpreter.ILoop.scala$tools$nsc$interpreter$ILoop$$loopPostInit()` when using Spark 2.4.0.
I've seen that on [docker store](https://store.docker.com/community/images/apache/zeppelin) the image has been updated for the last time 5 months ago. Could you please update it ? Or should I build the docker branch of the repo ?
Thank you
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/zeppelin/pull/3206
I understand the situation. FYI, the binaries (except SparkR) are ready.
1. Download: https://www-us.apache.org/dist/spark/spark-2.4.0/
2. Maven: https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/2.4.0
3. PySpark: https://pypi.org/project/pyspark/2.4.0/
SparkR seems to be rejected by CRAN. But, in the worst case, it will become 2.4.1. I guess so.
Zeppelin depends on only (1), doesn't it?
---
[GitHub] zeppelin pull request #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon closed the pull request at:
https://github.com/apache/zeppelin/pull/3206
---
[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by ajaygk95 <gi...@git.apache.org>.
Github user ajaygk95 commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Hello,
I was trying to run zeppelin with spark 2.4. I have pulled your code.
While building zeppelin I'm facing the below issue,
```
[ERROR] /home/cloud-user/ajay/code/csf-cc-zeppelin-k8szep/spark/scala-2.11/src/main/scala/org/apache/zeppelin/spark/SparkScala211Interpreter.scala:37: error: class SparkScala211Interpreter needs to be abstract, since method interpret in class BaseSparkScalaInterpreter of type (code: String, context: org.apache.zeppelin.interpreter.InterpreterContext)org.apache.zeppelin.interpreter.InterpreterResult is not defined
[ERROR] class SparkScala211Interpreter(override val conf: SparkConf,
[ERROR] ^
[ERROR] one error found
[INFO] ------------------------------------------------------------------------
```
Can you please help.
Thanks
---
[GitHub] zeppelin pull request #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
GitHub user HyukjinKwon reopened a pull request:
https://github.com/apache/zeppelin/pull/3206
[WIP][ZEPPELIN-3810] Support Spark 2.4
### What is this PR for?
Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418).
There are two problems for this upgrade at Zeppelin side:
1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible.
See:
- https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala
- https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala
To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection.
2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit:
```
Caused by: java.lang.NoSuchMethodError:
jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V
at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139)
```
To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`.
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Wait until Spark 2.4.0 is officially released.
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3810
### How should this be tested?
Verified manually against Spark 2.4.0 RC3
### Questions:
* Does the licenses files need update? Yes
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/zeppelin ZEPPELIN-3810
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/3206.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3206
----
commit e2d224aadf0e54533837a9a89f8e9d3586aee3a9
Author: hyukjinkwon <gu...@...>
Date: 2018-10-17T14:41:29Z
Support Spark 2.4
----
---
[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Yes， we need to support scala 2.11 for spark 2.4 first.
And please also update travis.yml to build it with spark-2.4 profile
---
[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/zeppelin/pull/3206
The error message:
```
[ERROR] /home/cloud-user/ajay/code/csf-cc-zeppelin-k8szep/spark/scala-2.11/src/main/scala/org/apache/zeppelin/spark/SparkScala211Interpreter.scala:37: error: class
SparkScala211Interpreter needs to be abstract, since method interpret in class
BaseSparkScalaInterpreter of type (code: String, context: org.apache.zeppelin.interpreter.InterpreterContext)org.apache.zeppelin.interpreter.InterpreterResult is
not defined
```
complains there's no `interpret` method defined at `BaseSparkScalaInterpreter`.
---
[GitHub] zeppelin pull request #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/3206
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/3206
CI is passed, will merge it if no more comments
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/3206
yes, everything is out (except for the announcement)
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/zeppelin/pull/3206
Hi, All. It's announced finally.
http://spark.apache.org/news/spark-2-4-0-released.html
---
[GitHub] zeppelin issue #3206: [ZEPPELIN-3810] Support Spark 2.4
Posted by alonshoham <gi...@git.apache.org>.
Github user alonshoham commented on the issue:
https://github.com/apache/zeppelin/pull/3206
thank you for the quick response!
does 0.8.1 have a release date?
thanks again
בתאריך יום א׳, 18 בנוב׳ 2018, 15:10, מאת Jeff Zhang <
notifications@github.com>:
> It is merged to branch-0.8 and master branch. So yes it will be released
> in zeppelin 0.9.0, but I believe 0.8.1 will be released before zeppelin
> 0.9.0
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/zeppelin/pull/3206#issuecomment-439691784>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AP-5fNF0kNk3qGODsqniD9sa1TDGaFPvks5uwVxGgaJpZM4XkINL>
> .
>
---
[GitHub] zeppelin issue #3206: [WIP][ZEPPELIN-3810] Support Spark 2.4
Posted by ajaygk95 <gi...@git.apache.org>.
Github user ajaygk95 commented on the issue:
https://github.com/apache/zeppelin/pull/3206
>
>
> The error message:
>
> ```
> [ERROR] /home/cloud-user/ajay/code/csf-cc-zeppelin-k8szep/spark/scala-
> 2.11/src/main/scala/org/apache/zeppelin/spark/SparkScala211Interpreter.scala:37: error: class
> SparkScala211Interpreter needs to be abstract, since method interpret in class
> BaseSparkScalaInterpreter of type (code: String, context:
> org.apache.zeppelin.interpreter.InterpreterContext)org.apache.zeppelin.interpreter.InterpreterResult is
> not defined
> ```
>
> complains there's no `interpret` method defined at `BaseSparkScalaInterpreter`.
Thanks for the reply.
I found the issue. I was using zeppelin 0.8.0 source code where "interpret" method was overriden. I think in the current branch this method is not present.
---