You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@toree.apache.org by "pan3793 (via GitHub)" <gi...@apache.org> on 2023/08/04 20:59:49 UTC
[GitHub] [incubator-toree] pan3793 opened a new pull request, #201: [TOREE-537] Upgrade Spark 3.3.2
pan3793 opened a new pull request, #201:
URL: https://github.com/apache/incubator-toree/pull/201
This PR aims to upgrade the Spark version to 3.3.2
The status of maintained Spark versions are
- Spark 3.0 => EOL
- Spark 3.1 => EOL
- Spark 3.2 => EOL
- Apache Spark 3.3 => December, 2023.
- Apache Spark 3.4 => October, 2024
- Upcoming Apache Spark 3.5 => 18 months from the release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] pan3793 commented on a diff in pull request #201: [TOREE-537] Upgrade Spark 3.3.2
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #201:
URL: https://github.com/apache/incubator-toree/pull/201#discussion_r1284846810
##########
build.sbt:
##########
@@ -22,11 +22,11 @@ ThisBuild / version := Properties.envOrElse("VERSION", "0.0.0-dev") +
(if ((ThisBuild / isSnapshot ).value) "-SNAPSHOT" else "")
ThisBuild / isSnapshot := Properties.envOrElse("IS_SNAPSHOT","true").toBoolean
ThisBuild / organization := "org.apache.toree.kernel"
-ThisBuild / crossScalaVersions := Seq("2.12.15") // https://github.com/scala/bug/issues/12475, for Spark 3.2.0
+ThisBuild / crossScalaVersions := Seq("2.12.15")
Review Comment:
obviously, we should keep consistent with Spark 3.3.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] requaos commented on pull request #201: [TOREE-537] Upgrade Spark 3.3.2
Posted by "requaos (via GitHub)" <gi...@apache.org>.
requaos commented on PR #201:
URL: https://github.com/apache/incubator-toree/pull/201#issuecomment-1667948093
If this change is passable, then we can refocus #199 to introduce Scala 2.13, as upgrading Spark was ancillary to that effort. Let me know, because as is these commits will directly conflict with my branch for the overlap from copying my changes over instead of some other merge or cherrypick strategy. I will happily split up the efforts in my branch behind #199.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] lresende commented on pull request #201: [TOREE-537] Upgrade Spark 3.3.2
Posted by "lresende (via GitHub)" <gi...@apache.org>.
lresende commented on PR #201:
URL: https://github.com/apache/incubator-toree/pull/201#issuecomment-1667065396
@pan3793 Thanks for all the enhancements, it is very welcomed.
for this pr in specific, let's wait to merge it when the #199 gets merged... which, btw, if you want to help fix the remaining failing issues, we would welcome the help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] pan3793 commented on a diff in pull request #201: [TOREE-537] Upgrade Spark 3.3.2
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #201:
URL: https://github.com/apache/incubator-toree/pull/201#discussion_r1284845353
##########
Dockerfile:
##########
@@ -61,7 +62,7 @@ RUN apt-get update && \
rm -rf /var/lib/apt/lists/*
ENV SPARK_HOME /usr/local/spark
-ENV PYTHONPATH $SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.9-src.zip
+ENV PYTHONPATH $SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.9.5-src.zip
Review Comment:
keep consistent with Spark 3.3.2 https://github.com/apache/spark/blob/v3.3.2/python/lib/py4j-0.10.9.5-src.zip
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] lresende merged pull request #201: [TOREE-537] Upgrade Spark 3.3.2
Posted by "lresende (via GitHub)" <gi...@apache.org>.
lresende merged PR #201:
URL: https://github.com/apache/incubator-toree/pull/201
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] pan3793 commented on pull request #201: [TOREE-537] Upgrade Spark 3.3.2
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #201:
URL: https://github.com/apache/incubator-toree/pull/201#issuecomment-1667070491
@lresende thanks for your reply.
I actually took a look at #199 first, and thought it composes of
- upgrade spark 3.3
- upgrade scala 2.13
- upgrade some dependencies which the current version does not support scala 2.13, for example, cursior
This PR could be treated as part of that. And as I commented before
> Spark 3.0 ~ 3.5 uses Scala 2.12 by default, it rarely uses a non-default Scala version in production, so I would like to upgrade the Spark first.
would it be good to upgrade Scala 2.13 at the current stage?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] pan3793 commented on a diff in pull request #201: [TOREE-537] Upgrade Spark 3.3.2
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #201:
URL: https://github.com/apache/incubator-toree/pull/201#discussion_r1284845924
##########
project/Dependencies.scala:
##########
@@ -36,10 +36,10 @@ object Dependencies {
val coursier = "io.get-coursier" %% "coursier" % coursierVersion // Apache v2
val coursierCache = "io.get-coursier" %% "coursier-cache" % coursierVersion // Apache v2
- val ivy = "org.apache.ivy" % "ivy" % "2.4.0" // Apache v2
+ val ivy = "org.apache.ivy" % "ivy" % "2.5.1" // Apache v2
- // use the same jackson version in test than the one provided at runtime by Spark 3.0.0
- val jacksonDatabind = "com.fasterxml.jackson.core" % "jackson-databind" % "2.10.0" // Apache v2
+ // use the same jackson version in test than the one provided at runtime by Spark 3.3.2
+ val jacksonDatabind = "com.fasterxml.jackson.core" % "jackson-databind" % "2.13.4.2" // Apache v2
Review Comment:
dependencies updated to align with Spark 3.3.2
https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/3.3.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] pan3793 commented on a diff in pull request #201: [TOREE-537] Upgrade Spark 3.3.2
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #201:
URL: https://github.com/apache/incubator-toree/pull/201#discussion_r1284846463
##########
build.sbt:
##########
@@ -58,8 +58,8 @@ ThisBuild / javacOptions ++= Seq(
"-Xlint:-options",
"-Xlint:-processing",
"-Werror", // Treat warnings as errors
- "-source", "1.6",
- "-target", "1.6"
+ "-source", "1.8",
+ "-target", "1.8"
Review Comment:
Actually, Spark 3.x always releases with -target 1.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] pan3793 commented on pull request #201: [TOREE-537] Upgrade Spark 3.3.2
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #201:
URL: https://github.com/apache/incubator-toree/pull/201#issuecomment-1668813167
> ... as is these commits will directly conflict with my branch for the overlap from copying my changes over instead of some other merge or cherrypick strategy.
@requaos Yes, this change starts from scratch, because I can not achieve it by cherry-picking in a clean way. I respect your effort in #199 so I added you as co-author of this PR. If necessary, I can help with git rebase and conflict resolution, which requires write permission of your forked repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] pan3793 commented on pull request #201: [TOREE-537] Upgrade Spark 3.3.2
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #201:
URL: https://github.com/apache/incubator-toree/pull/201#issuecomment-1666377005
ping @lresende, could you review this PR when you have time?
BTW, this is an awesome project, please let me know what I can do for this project :)
I'm familiar with Spark and Scala(but sorry for my poor Python skill), and am learning how to implement a Jupyter kernel using Java/Scala.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] pan3793 commented on pull request #201: [TOREE-537] Upgrade Spark 3.3.2
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #201:
URL: https://github.com/apache/incubator-toree/pull/201#issuecomment-1667099408
I updated the PR description to add @requaos(the original author of https://github.com/apache/incubator-toree/pull/199) and me as the co-author, from the tech view, I lean toward breaking down the huge PR into dedicated ones, and keep using Scala 2.12 during Spark 3.x serial.
@lresende WDYT? Of course, if you think we should work on https://github.com/apache/incubator-toree/pull/199, I would close this one and try to fix that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] lresende commented on pull request #201: [TOREE-537] Upgrade Spark 3.3.2
Posted by "lresende (via GitHub)" <gi...@apache.org>.
lresende commented on PR #201:
URL: https://github.com/apache/incubator-toree/pull/201#issuecomment-1667147523
> @lresende WDYT? Of course, if you think we should work on #199, I would close this one and try to fix that.
It's ok, let me test this and will review/merge based on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org