You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "Yohahaha (via GitHub)" <gi...@apache.org> on 2024/10/15 06:47:12 UTC
[PR] [CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Yohahaha opened a new pull request, #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
(Fixes: \#ISSUE-ID)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7336][CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537#issuecomment-2428399722
> Have you checked the iceberg with Spark 3.5.3? It seems that Spark 3.5.3 has some problems with iceberg and the iceberg community would not support the Spark 3.5.3 release in [iceberg-11160](https://github.com/apache/iceberg/pull/11160).
no, I only check the GHA passed.
IMO, Gluten only depends on Spark, we have no guarantee that gluten-iceberg will works well after this PR if iceberg has decided to skip Spark 3.5.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7336][CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537#issuecomment-2413031962
https://github.com/apache/incubator-gluten/issues/7336
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7336][CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537#issuecomment-2428682923
https://github.com/apache/incubator-gluten/issues/7637
Open above issue for gluten-iceberg if we need fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7336][CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "zzcclp (via GitHub)" <gi...@apache.org>.
zzcclp commented on PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537#issuecomment-2428668671
can wo merge this pr first if ci passed, if there are some problems with the iceberg, will raise another pr to fix ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7336][CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537#issuecomment-2415581326
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7336][CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537#issuecomment-2418480649
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7336][CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537#issuecomment-2428355077
Have you checked the iceberg with Spark 3.5.3? It seems that Spark 3.5.3 has some problems with iceberg and the iceberg community would not support the Spark 3.5.3 release in [iceberg-11160](https://github.com/apache/iceberg/pull/11160).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7336][CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537#issuecomment-2425692508
> why CH backend's performance may drop when upgrade spark version?
It's not said that performance will decrease, it's just a standard process for upgrading versions. In fact, when upgrading to 3.5.2, we found that two TPCDS queries were a bit faster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7336][CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537#issuecomment-2418413123
@Yohahaha can you please do a rebase? there's a small conflict on GHA script
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7336][CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537#issuecomment-2413031902
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7336][CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on code in PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537#discussion_r1810002438
##########
pom.xml:
##########
@@ -341,12 +341,12 @@
<sparkbundle.version>3.5</sparkbundle.version>
<sparkshim.module.name>spark35</sparkshim.module.name>
<sparkshim.artifactId>spark-sql-columnar-shims-spark35</sparkshim.artifactId>
- <spark.version>3.5.2</spark.version>
+ <spark.version>3.5.3</spark.version>
<iceberg.version>1.5.0</iceberg.version>
<delta.package.name>delta-spark</delta.package.name>
- <delta.version>3.2.0</delta.version>
- <delta.binary.version>32</delta.binary.version>
- <hudi.version>0.15.0</hudi.version>
+ <delta.version>3.2.1</delta.version>
+ <delta.binary.version>32</delta.binary.version>
+ <hudi.version>0.15.0</hudi.version>
Review Comment:
maybe it's Github diff issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7336][CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537#discussion_r1809976475
##########
pom.xml:
##########
@@ -341,12 +341,12 @@
<sparkbundle.version>3.5</sparkbundle.version>
<sparkshim.module.name>spark35</sparkshim.module.name>
<sparkshim.artifactId>spark-sql-columnar-shims-spark35</sparkshim.artifactId>
- <spark.version>3.5.2</spark.version>
+ <spark.version>3.5.3</spark.version>
<iceberg.version>1.5.0</iceberg.version>
<delta.package.name>delta-spark</delta.package.name>
- <delta.version>3.2.0</delta.version>
- <delta.binary.version>32</delta.binary.version>
- <hudi.version>0.15.0</hudi.version>
+ <delta.version>3.2.1</delta.version>
+ <delta.binary.version>32</delta.binary.version>
+ <hudi.version>0.15.0</hudi.version>
Review Comment:
Assuming it's unintentional?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7336][CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537#issuecomment-2418479120
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7336][CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537#issuecomment-2418797559
CC @baibaichen @zzcclp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7336][CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha merged PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7336][CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537#issuecomment-2428281155
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537#issuecomment-2413031505
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICEGJ-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
Could you open an issue for this pull request on Github Issues?
https://github.com/apache/incubator-gluten/issues
Then could you also rename ***commit message*** and ***pull request title*** in the following format?
[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}
See also:
* [Other pull requests](https://github.com/apache/incubator-gluten/pulls/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7336][CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537#issuecomment-2421089190
> @zhouyuan
>
> I need some time to do performance testing, maybe I'll have a chance to merge next week.
why CH backend's performance may drop when upgrade spark version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7336][CORE] Bump Spark version to v3.5.3 [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on PR #7537:
URL: https://github.com/apache/incubator-gluten/pull/7537#issuecomment-2418849282
@zhouyuan
I need some time to do performance testing, maybe I'll have a chance to merge next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org