You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@zeppelin.apache.org by "pan3793 (via GitHub)" <gi...@apache.org> on 2023/10/08 04:31:03 UTC
[PR] [ZEPPELIN-5966] Support Spark 3.5 [zeppelin]
pan3793 opened a new pull request, #4665:
URL: https://github.com/apache/zeppelin/pull/4665
### What is this PR for?
Spark 3.5.0 was released a few weeks ago, this PR tries to add Spark 3.5 support in Zeppelin.
https://spark.apache.org/releases/spark-release-3-5-0.html
### What type of PR is it?
Improvement
### What is the Jira issue?
ZEPPELIN-5966
### How should this be tested?
Update GitHub workflow to cover Spark 3.5 tests
### Screenshots (if appropriate)
### Questions:
* Does the license files need to update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes, updated correspondingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-5966] Support Spark 3.5 [zeppelin]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #4665:
URL: https://github.com/apache/zeppelin/pull/4665#issuecomment-1761517295
@Reamer @jongyoul I think it's ready to go, would you please take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-5966] Support Spark 3.5 [zeppelin]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #4665:
URL: https://github.com/apache/zeppelin/pull/4665#discussion_r1354372407
##########
.github/workflows/core.yml:
##########
@@ -411,6 +411,11 @@ jobs:
run: |
rm -rf spark/interpreter/metastore_db
./mvnw verify -pl spark-submit,spark/interpreter -am -Dtest=org/apache/zeppelin/spark/* -Pspark-3.4 -Pspark-scala-2.13 -Phadoop3 -Pintegration -DfailIfNoTests=false ${MAVEN_ARGS}
+ - name: run spark-3.5 tests with scala-2.13 and python-${{ matrix.python }}
+ if: matrix.python >= '3.8'
+ run: |
+ rm -rf spark/interpreter/metastore_db
+ ./mvnw verify -pl spark-submit,spark/interpreter -am -Dtest=org/apache/zeppelin/spark/* -Pspark-3.5 -Pspark-scala-2.13 -Phadoop3 -Pintegration -DfailIfNoTests=false ${MAVEN_ARGS}
Review Comment:
BTW, hadoop2 support has been deprecated since Spark 3.4(no official binary artifacts, but allows user to compile by themselves), and removed since 3.5(totally removed the hadoop2 related stuff from codebase)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-5966] Support Spark 3.5 [zeppelin]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #4665:
URL: https://github.com/apache/zeppelin/pull/4665#issuecomment-1752937437
will rebase once https://github.com/apache/zeppelin/pull/4668 get in
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-5966] Support Spark 3.5 [zeppelin]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #4665:
URL: https://github.com/apache/zeppelin/pull/4665#discussion_r1354998647
##########
.github/workflows/core.yml:
##########
@@ -411,6 +411,11 @@ jobs:
run: |
rm -rf spark/interpreter/metastore_db
./mvnw verify -pl spark-submit,spark/interpreter -am -Dtest=org/apache/zeppelin/spark/* -Pspark-3.4 -Pspark-scala-2.13 -Phadoop3 -Pintegration -DfailIfNoTests=false ${MAVEN_ARGS}
+ - name: run spark-3.5 tests with scala-2.13 and python-${{ matrix.python }}
+ if: matrix.python >= '3.8'
+ run: |
+ rm -rf spark/interpreter/metastore_db
+ ./mvnw verify -pl spark-submit,spark/interpreter -am -Dtest=org/apache/zeppelin/spark/* -Pspark-3.5 -Pspark-scala-2.13 -Phadoop3 -Pintegration -DfailIfNoTests=false ${MAVEN_ARGS}
Review Comment:
Remove Hadoop2 support would be a big change, I'm not very familiar with Flink, but for Spark, it requires at least Spark 3.x to support Hadoop 3, the Livy module still uses Spark 2.4.8.
And Spark moved to use Hadoop Shaded client since 3.2.0, which is a quite big change for Hadoop deps management.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-5966] Support Spark 3.5 [zeppelin]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #4665:
URL: https://github.com/apache/zeppelin/pull/4665#issuecomment-1759634060
kindly ping @jongyoul
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-5966] Support Spark 3.5 [zeppelin]
Posted by "Reamer (via GitHub)" <gi...@apache.org>.
Reamer commented on code in PR #4665:
URL: https://github.com/apache/zeppelin/pull/4665#discussion_r1354616041
##########
.github/workflows/core.yml:
##########
@@ -411,6 +411,11 @@ jobs:
run: |
rm -rf spark/interpreter/metastore_db
./mvnw verify -pl spark-submit,spark/interpreter -am -Dtest=org/apache/zeppelin/spark/* -Pspark-3.4 -Pspark-scala-2.13 -Phadoop3 -Pintegration -DfailIfNoTests=false ${MAVEN_ARGS}
+ - name: run spark-3.5 tests with scala-2.13 and python-${{ matrix.python }}
+ if: matrix.python >= '3.8'
+ run: |
+ rm -rf spark/interpreter/metastore_db
+ ./mvnw verify -pl spark-submit,spark/interpreter -am -Dtest=org/apache/zeppelin/spark/* -Pspark-3.5 -Pspark-scala-2.13 -Phadoop3 -Pintegration -DfailIfNoTests=false ${MAVEN_ARGS}
Review Comment:
I've been trying to remove hadoop2 for a while, but am still having problems with the Flink integration tests.
See #4674
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-5966] Support Spark 3.5 [zeppelin]
Posted by "jongyoul (via GitHub)" <gi...@apache.org>.
jongyoul merged PR #4665:
URL: https://github.com/apache/zeppelin/pull/4665
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-5966] Support Spark 3.5 [zeppelin]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #4665:
URL: https://github.com/apache/zeppelin/pull/4665#issuecomment-1764314644
@Reamer thanks for the review and approval
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-5966] Support Spark 3.5 [zeppelin]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #4665:
URL: https://github.com/apache/zeppelin/pull/4665#discussion_r1354367211
##########
spark/spark-shims/src/main/java/org/apache/zeppelin/spark/SparkVersion.java:
##########
@@ -31,8 +31,10 @@ public class SparkVersion {
public static final SparkVersion SPARK_3_5_0 = SparkVersion.fromVersionString("3.5.0");
+ public static final SparkVersion SPARK_4_0_0 = SparkVersion.fromVersionString("4.0.0");
+
public static final SparkVersion MIN_SUPPORTED_VERSION = SPARK_3_2_0;
- public static final SparkVersion UNSUPPORTED_FUTURE_VERSION = SPARK_3_5_0;
+ public static final SparkVersion UNSUPPORTED_FUTURE_VERSION = SPARK_4_0_0;
Review Comment:
3.5 will be the last feature version in Spark 3.x serials, Apache Spark master branch goes 4.0.0-SNAPSHOT now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-5966] Support Spark 3.5 [zeppelin]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #4665:
URL: https://github.com/apache/zeppelin/pull/4665#issuecomment-1751952837
BTW, Spark 3.5 drops support for Python 3.7
https://spark.apache.org/docs/latest/api/python/getting_started/install.html#python-versions-supported
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-5966] Support Spark 3.5 [zeppelin]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #4665:
URL: https://github.com/apache/zeppelin/pull/4665#issuecomment-1758883150
could the committer help trigger the failed checks? I suppose all failures are caused flaky tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [ZEPPELIN-5966] Support Spark 3.5 [zeppelin]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #4665:
URL: https://github.com/apache/zeppelin/pull/4665#issuecomment-1757100038
@Reamer @jongyoul this is ready for review (the previous attempts already passed CI), also cc @zjffdu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org