You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@livy.apache.org by GitBox <gi...@apache.org> on 2020/07/01 05:41:08 UTC
[GitHub] [incubator-livy] jerryshao opened a new pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
jerryshao opened a new pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300
## What changes were proposed in this pull request?
This PR is based @tprelle 's PR #289 , and address all the left issues in that PR:
1. multi-scala version support in one build (Scala 2.11 and 2.12 support).
2. make SparkR work.
Also reverts most of the unnecessary changes. Besides this PR remove the build below 2.4 (2.2, 2.3), since Spark 2.2 and 2.3 only ships with Scala 2.11, hard to maintain multiple version. But user could still use 2.2 and 2.3 without changes.
All credits to @tprelle.
## How was this patch tested?
Run UT and IT with Spark 2.4.5 and 3.0.0 locally.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] derSascha commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
Posted by GitBox <gi...@apache.org>.
derSascha commented on pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-771916664
The changes introduced here upgrade Kryo from 2.22 to 4.0.2. This change makes the API incompatible. The latest AWS EMR release (6.2.0) includes Spark 3 and Livy 0.7.0 together with Kryo 2.22 and works fine. Can we revert the Kryo upgrade before the Livy 0.8.0 release to keep the API compatible?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
Posted by GitBox <gi...@apache.org>.
jerryshao commented on pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-652839568
I've triggered the travis test several times, seems everything is fine. I'm going to merge this, thanks @tprelle for the contribution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
Posted by GitBox <gi...@apache.org>.
jerryshao commented on pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-652320407
@tprelle @andrasbeni would you please help to review, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] pnayek commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
Posted by GitBox <gi...@apache.org>.
pnayek commented on pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-899837371
Hi @jerryshao I have a Cloudera CDP Private Cloud 7.1.6 Hadoop system. As CDP Private Cloud 7.1.6 supports Spark 3.1 along with Spark 2.4, I have separately installed Spark version 3.1.1.3.1.7280.2-11 on the system (as per Cloudera document https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/cds-3/topics/spark-spark-3-overview.html). I am trying to use Livy (version livy-0.7.1-incubating) to connect to Spark 3.1.1.3.1.7280.2-11. Does livy-0.7.1 support Spark version 3.1? If so, are there any requirements that need to be followed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao closed pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
Posted by GitBox <gi...@apache.org>.
jerryshao closed pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] tooptoop4 commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
Posted by GitBox <gi...@apache.org>.
tooptoop4 commented on pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-706393343
@jerryshao will this be released soon?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] redsanket commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
Posted by GitBox <gi...@apache.org>.
redsanket commented on pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-758727710
@jerryshao Is there any progress with 0.8.0 release?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
Posted by GitBox <gi...@apache.org>.
jerryshao commented on pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-652203533
reopen to trigger the test again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] Dom-nik commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
Posted by GitBox <gi...@apache.org>.
Dom-nik commented on pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-721148882
@jerryshao Is it possible that you add the instruction to build the Spark 3.0 version of Livy to the README? I'd been struggling with Maven Profiles and didn't get anywhere.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] derSascha commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
Posted by GitBox <gi...@apache.org>.
derSascha commented on pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-771916664
The changes introduced here upgrade Kryo from 2.22 to 4.0.2. This change makes the API incompatible. The latest AWS EMR release (6.2.0) includes Spark 3 and Livy 0.7.0 together with Kryo 2.22 and works fine. Can we revert the Kryo upgrade before the Livy 0.8.0 release to keep the API compatible?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] fmarchand commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
Posted by GitBox <gi...@apache.org>.
fmarchand commented on pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-731576185
> @jerryshao Is it possible that you add the instruction to build the Spark 3.0 version of Livy to the README? I'd been struggling with Maven Profiles and didn't get anywhere (some tests are failing when I try to run `mvn package -P spark-3.0`)
There's 3 tests failing in module livy-rsc yes @Dom-nik, I guess we missed something ;)
```
Tests in error:
TestSparkClient.testConnectToRunningContext:338->runTest:575->runTest:588 » Execution
TestSparkClient.testImpersonation:321->runTest:575->runTest:588 » Execution ja...
TestSparkClient.testRemoteClient:213->runTest:575->runTest:588 » Execution jav...
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300#discussion_r447039881
##########
File path: repl/src/main/scala/org/apache/livy/repl/Session.scala
##########
@@ -348,8 +348,10 @@ class Session(
case "1" =>
(s"""setJobGroup(sc, "$jobGroup", "Job group for statement $jobGroup", FALSE)""",
SparkR)
- case "2" =>
+ case "2" | "3" =>
Review comment:
@tprelle this should be changed to support SparkR, otherwise it will be hung when running against spark 3.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] Dom-nik edited a comment on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
Posted by GitBox <gi...@apache.org>.
Dom-nik edited a comment on pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-721148882
@jerryshao Is it possible that you add the instruction to build the Spark 3.0 version of Livy to the README? I'd been struggling with Maven Profiles and didn't get anywhere (some tests are failing when I try to run `mvn package -P spark-3.0`)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] pdambrauskas commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
Posted by GitBox <gi...@apache.org>.
pdambrauskas commented on pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-663497351
Hey, are there any plans to release this feature :) ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
Posted by GitBox <gi...@apache.org>.
tprelle commented on pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-651137090
@jerryshao How you manage to make sparkR work ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao closed pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
Posted by GitBox <gi...@apache.org>.
jerryshao closed pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
Posted by GitBox <gi...@apache.org>.
jerryshao commented on pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-664909277
There still have some minor issues in JDBC side related to Scala versions. Will prepare a new release when everything is done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #300: [LIVY-756] Add Spark 3.0 and Scala 2.12 support
Posted by GitBox <gi...@apache.org>.
jerryshao commented on pull request #300:
URL: https://github.com/apache/incubator-livy/pull/300#issuecomment-651474672
Travis image seems too old to support Spark 3.0 R version, we should also upgrade Travis script.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org