You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zjuwangg (via GitHub)" <gi...@apache.org> on 2025/08/13 11:50:27 UTC
[PR] [Doc][VL]Add support for spark 4.0.0 support [incubator-gluten]
zjuwangg opened a new pull request, #10430:
URL: https://github.com/apache/incubator-gluten/pull/10430
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/incubator-gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
Add spark 4.0.0 build support in document.
## How was this patch tested?
NA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [Doc][VL]Add documents for spark 4.0.0 support [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10430:
URL: https://github.com/apache/incubator-gluten/pull/10430#issuecomment-3561001940
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [Doc][VL]Add documents for spark 4.0.0 support [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10430:
URL: https://github.com/apache/incubator-gluten/pull/10430#issuecomment-3367756319
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [Doc][VL]Add documents for spark 4.0.0 support [gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #10430:
URL: https://github.com/apache/gluten/pull/10430#issuecomment-4543964260
closing as it's already documented
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [Doc][VL]Add support for spark 4.0.0 support [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on code in PR #10430:
URL: https://github.com/apache/incubator-gluten/pull/10430#discussion_r2274974386
##########
docs/get-started/Velox.md:
##########
@@ -7,18 +7,18 @@ parent: Getting-Started
# Supported Version
-| Type | Version |
-|-------|------------------------------|
-| Spark | 3.2.2, 3.3.1, 3.4.4, 3.5.2 |
-| OS | Ubuntu20.04/22.04, Centos7/8 |
-| jdk | openjdk8/jdk17 |
-| scala | 2.12 |
+| Type | Version |
+|-------|-------------------------------------------------|
+| Spark | 3.2.2, 3.3.1, 3.4.4, 3.5.2, 4.0.0(Experimental) |
+| OS | Ubuntu20.04/22.04, Centos7/8 |
+| jdk | openjdk8/jdk17 |
+| scala | 2.12 |
# Prerequisite
Currently, with static build Gluten+Velox backend supports all the Linux OSes, but is only tested on **Ubuntu20.04/Ubuntu22.04/Centos7/Centos8**. With dynamic build, Gluten+Velox backend support **Ubuntu20.04/Ubuntu22.04/Centos7/Centos8** and their variants.
-Currently, the officially supported Spark versions are 3.2.2, 3.3.1, 3.4.4 and 3.5.2.
+Currently, the officially supported Spark versions are 3.2.2, 3.3.1, 3.4.4, 3.5.2 and 4.0.0(Experimental).
Review Comment:
I am drafting a pr to add the CI. Will submit a PR soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [Doc][VL]Add documents for spark 4.0.0 support [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10430:
URL: https://github.com/apache/incubator-gluten/pull/10430#issuecomment-3931182873
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [Doc][VL]Add support for spark 4.0.0 support [incubator-gluten]
Posted by "zjuwangg (via GitHub)" <gi...@apache.org>.
zjuwangg commented on PR #10430:
URL: https://github.com/apache/incubator-gluten/pull/10430#issuecomment-3186889883
Got your guys meaning, let's hold until spark 400 support get more matured
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [Doc][VL]Add documents for spark 4.0.0 support [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10430:
URL: https://github.com/apache/gluten/pull/10430#issuecomment-4196053245
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [Doc][VL]Add documents for spark 4.0.0 support [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he commented on PR #10430:
URL: https://github.com/apache/gluten/pull/10430#issuecomment-4196988726
@zjuwangg, could you check the latest doc to see if this PR is still required? If yes, could you rebase and update this PR? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [Doc][VL]Add documents for spark 4.0.0 support [gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan closed pull request #10430: [Doc][VL]Add documents for spark 4.0.0 support
URL: https://github.com/apache/gluten/pull/10430
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [Doc][VL]Add support for spark 4.0.0 support [incubator-gluten]
Posted by "zjuwangg (via GitHub)" <gi...@apache.org>.
zjuwangg closed pull request #10430: [Doc][VL]Add support for spark 4.0.0 support
URL: https://github.com/apache/incubator-gluten/pull/10430
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [Doc][VL]Add documents for spark 4.0.0 support [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10430:
URL: https://github.com/apache/incubator-gluten/pull/10430#issuecomment-3708706053
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [Doc][VL]Add support for spark 4.0.0 support [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #10430:
URL: https://github.com/apache/incubator-gluten/pull/10430#issuecomment-3185749578
3 steps to add new spark version support:
1. compile pass (Done)
2. TPCH/DS pass (Done)
3. UT imported and pass (?)
4. New feature supported (list?)
We can claim to support until UTs passed. @zhouyuan @PHILO-HE any idea of the UT status?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [Doc][VL]Add support for spark 4.0.0 support [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #10430:
URL: https://github.com/apache/incubator-gluten/pull/10430#issuecomment-3186788802
> > We can claim to support until UTs passed. @zhouyuan @PHILO-HE any idea of the UT status?
>
> @FelixYBW, currently, only TPC-H passes and TPC-DS still fails. We will add Spark-4.0 UT soon.
Let's document this clearly.
```
H passed, DS still fails. CI haven't passed
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [Doc][VL]Add support for spark 4.0.0 support [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on code in PR #10430:
URL: https://github.com/apache/incubator-gluten/pull/10430#discussion_r2275213332
##########
docs/get-started/Velox.md:
##########
@@ -7,18 +7,18 @@ parent: Getting-Started
# Supported Version
-| Type | Version |
-|-------|------------------------------|
-| Spark | 3.2.2, 3.3.1, 3.4.4, 3.5.2 |
-| OS | Ubuntu20.04/22.04, Centos7/8 |
-| jdk | openjdk8/jdk17 |
-| scala | 2.12 |
+| Type | Version |
+|-------|-------------------------------------------------|
+| Spark | 3.2.2, 3.3.1, 3.4.4, 3.5.2, 4.0.0(Experimental) |
Review Comment:
ditto.
##########
docs/get-started/Velox.md:
##########
@@ -7,18 +7,18 @@ parent: Getting-Started
# Supported Version
-| Type | Version |
-|-------|------------------------------|
-| Spark | 3.2.2, 3.3.1, 3.4.4, 3.5.2 |
-| OS | Ubuntu20.04/22.04, Centos7/8 |
-| jdk | openjdk8/jdk17 |
-| scala | 2.12 |
+| Type | Version |
+|-------|-------------------------------------------------|
+| Spark | 3.2.2, 3.3.1, 3.4.4, 3.5.2, 4.0.0(Experimental) |
+| OS | Ubuntu20.04/22.04, Centos7/8 |
+| jdk | openjdk8/jdk17 |
+| scala | 2.12 |
# Prerequisite
Currently, with static build Gluten+Velox backend supports all the Linux OSes, but is only tested on **Ubuntu20.04/Ubuntu22.04/Centos7/Centos8**. With dynamic build, Gluten+Velox backend support **Ubuntu20.04/Ubuntu22.04/Centos7/Centos8** and their variants.
-Currently, the officially supported Spark versions are 3.2.2, 3.3.1, 3.4.4 and 3.5.2.
+Currently, the officially supported Spark versions are 3.2.2, 3.3.1, 3.4.4, 3.5.2 and 4.0.0(Experimental).
Review Comment:
This may be confusing for users, as experimental features are typically not "officially supported." I recommend adding Spark-4.0 to the official support list after the Spark unit test failures have been resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [Doc][VL]Add documents for spark 4.0.0 support [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10430:
URL: https://github.com/apache/gluten/pull/10430#issuecomment-4523856267
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [Doc][VL]Add support for spark 4.0.0 support [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #10430:
URL: https://github.com/apache/incubator-gluten/pull/10430#discussion_r2273413206
##########
docs/get-started/Velox.md:
##########
@@ -7,18 +7,18 @@ parent: Getting-Started
# Supported Version
-| Type | Version |
-|-------|------------------------------|
-| Spark | 3.2.2, 3.3.1, 3.4.4, 3.5.2 |
-| OS | Ubuntu20.04/22.04, Centos7/8 |
-| jdk | openjdk8/jdk17 |
-| scala | 2.12 |
+| Type | Version |
+|-------|-------------------------------------------------|
+| Spark | 3.2.2, 3.3.1, 3.4.4, 3.5.2, 4.0.0(Experimental) |
+| OS | Ubuntu20.04/22.04, Centos7/8 |
+| jdk | openjdk8/jdk17 |
+| scala | 2.12 |
# Prerequisite
Currently, with static build Gluten+Velox backend supports all the Linux OSes, but is only tested on **Ubuntu20.04/Ubuntu22.04/Centos7/Centos8**. With dynamic build, Gluten+Velox backend support **Ubuntu20.04/Ubuntu22.04/Centos7/Centos8** and their variants.
-Currently, the officially supported Spark versions are 3.2.2, 3.3.1, 3.4.4 and 3.5.2.
+Currently, the officially supported Spark versions are 3.2.2, 3.3.1, 3.4.4, 3.5.2 and 4.0.0(Experimental).
Review Comment:
Could it be better to have a basic TPC-H CI for 4.0.0 then do such statement? I assume H is passing based on description of Yuan's original [PR](https://github.com/apache/incubator-gluten/pull/9768).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [Doc][VL]Add support for spark 4.0.0 support [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on PR #10430:
URL: https://github.com/apache/incubator-gluten/pull/10430#issuecomment-3186650816
> 3 steps to add new spark version support:
>
> 1. compile pass (Done)
>
> 2. TPCH/DS pass (Done)
>
> 3. UT imported and pass (?)
>
> 4. New feature supported (list?)
>
>
> We can claim to support until UTs passed. @zhouyuan @PHILO-HE any idea of the UT status?
@FelixYBW, currently, only TPC-H passes and TPC-DS still fails. We will add Spark-4.0 UT soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [Doc][VL]Add documents for spark 4.0.0 support [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #10430:
URL: https://github.com/apache/incubator-gluten/pull/10430#issuecomment-3369729666
@zhouyuan is working on 4.0 UT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org