You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "beliefer (via GitHub)" <gi...@apache.org> on 2025/08/01 09:21:55 UTC
[PR] [GLUTEN-10334] Share the results compare the spark version [incubator-gluten]
beliefer opened a new pull request, #10335:
URL: https://github.com/apache/incubator-gluten/pull/10335
## What changes were proposed in this pull request?
This PR proposes to share the results compare the spark version.
(Fixes: \#10334)
## How was this patch tested?
unit tests, integration tests, manual tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10334] Share the results compare the spark version [incubator-gluten]
Posted by "zzcclp (via GitHub)" <gi...@apache.org>.
zzcclp commented on PR #10335:
URL: https://github.com/apache/incubator-gluten/pull/10335#issuecomment-3157608590
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10334] Share the results compare the spark version [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #10335:
URL: https://github.com/apache/incubator-gluten/pull/10335#issuecomment-3154035540
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10334] Share the results compare the spark version [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #10335:
URL: https://github.com/apache/incubator-gluten/pull/10335#discussion_r2247817919
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkVersionUtil.scala:
##########
@@ -17,8 +17,15 @@
package org.apache.spark.util
object SparkVersionUtil {
+ val lteSpark32: Boolean = compareMajorMinorVersion((3, 2)) <= 0
+ private val comparedWithSpark33 = compareMajorMinorVersion((3, 3))
+ val ltSpark33: Boolean = comparedWithSpark33 < 0
Review Comment:
Would it be better not to keep `lt` / `gt`, as we can always use `lte` / `gte` for clearance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10334] Share the results compare the spark version [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #10335:
URL: https://github.com/apache/incubator-gluten/pull/10335#issuecomment-3151897448
conflicted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10334] Share the results compare the spark version [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW merged PR #10335:
URL: https://github.com/apache/incubator-gluten/pull/10335
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10334] Share the results compare the spark version [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #10335:
URL: https://github.com/apache/incubator-gluten/pull/10335#issuecomment-3157399953
```
[2025-08-05T10:15:43.063Z] 10:15:42.955 ERROR org.apache.spark.sql.execution.exchange.BroadcastExchangeExec: Could not execute broadcast in 300 secs.
[2025-08-05T10:15:43.066Z] java.util.concurrent.TimeoutException
[2025-08-05T10:15:43.066Z] at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:204)
[2025-08-05T10:15:43.066Z] at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec.doExecuteBroadcast(BroadcastExchangeExec.scala:212)
[2025-08-05T10:15:43.066Z] at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeBroadcast$1(SparkPlan.scala:208)
[2025-08-05T10:15:43.066Z] at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:246)
[2025-08-05T10:15:43.066Z] at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
[2025-08-05T10:15:43.066Z] at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:243)
[2025-08-05T10:15:43.066Z] at org.apache.spark.sql.execution.SparkPlan.executeBroadcast(SparkPlan.scala:204)
[2025-08-05T10:15:43.066Z] 'unsafe broadcast left semi join updates peak execution memory' offload to gluten
[2025-08-05T10:15:43.319Z] - unsafe broadcast left semi join updates peak execution memory *** FAILED ***
[2025-08-05T10:15:43.323Z] org.apache.spark.SparkException: Could not execute broadcast in 300 secs. You can increase the timeout for broadcasts via "spark.sql.broadcastTimeout" or disable broadcast join by setting "spark.sql.autoBroadcastJoinThreshold" to -1.
[2025-08-05T10:15:43.323Z] at org.apache.spark.sql.errors.QueryExecutionErrors$.executeBroadcastTimeoutError(QueryExecutionErrors.scala:1099)
[2025-08-05T10:15:43.323Z] at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec.doExecuteBroadcast(BroadcastExchangeExec.scala:220)
[2025-08-05T10:15:43.323Z] at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeBroadcast$1(SparkPlan.scala:208)
[2025-08-05T10:15:43.323Z] at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:246)
[2025-08-05T10:15:43.323Z] at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
[2025-08-05T10:15:43.323Z] at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:243)
[2025-08-05T10:15:43.323Z] at org.apache.spark.sql.execution.SparkPlan.executeBroadcast(SparkPlan.scala:204)
[2025-08-05T10:15:43.323Z] at org.apache.spark.sql.execution.joins.BroadcastHashJoinExec.doExecute(BroadcastHashJoinExec.scala:126)
[2025-08-05T10:15:43.323Z] at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:195)
[2025-08-05T10:15:43.323Z] at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:246)
[2025-08-05T10:15:43.323Z] ...
[2025-08-05T10:15:43.323Z] Cause: java.util.concurrent.TimeoutException:
[2025-08-05T10:15:43.323Z] at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:204)
[2025-08-05T10:15:43.323Z] at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec.doExecuteBroadcast(BroadcastExchangeExec.scala:212)
[2025-08-05T10:15:43.323Z] at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeBroadcast$1(SparkPlan.scala:208)
[2025-08-05T10:15:43.323Z] at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:246)
[2025-08-05T10:15:43.323Z] at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
[2025-08-05T10:15:43.323Z] at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:243)
[2025-08-05T10:15:43.323Z] at org.apache.spark.sql.execution.SparkPlan.executeBroadcast(SparkPlan.scala:204)
[2025-08-05T10:15:43.323Z] at org.apache.spark.sql.execution.joins.BroadcastHashJoinExec.doExecute(BroadcastHashJoinExec.scala:126)
[2025-08-05T10:15:43.323Z] at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:195)
[2025-08-05T10:15:43.323Z] at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:246)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10334] Share the results compare the spark version [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #10335:
URL: https://github.com/apache/incubator-gluten/pull/10335#issuecomment-3157399385
The CH CI failure seems not related to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10334] Share the results compare the spark version [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10335:
URL: https://github.com/apache/incubator-gluten/pull/10335#issuecomment-3143865286
https://github.com/apache/incubator-gluten/issues/10334
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10334] Share the results compare the spark version [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #10335:
URL: https://github.com/apache/incubator-gluten/pull/10335#discussion_r2248080956
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkVersionUtil.scala:
##########
@@ -17,8 +17,15 @@
package org.apache.spark.util
object SparkVersionUtil {
+ val lteSpark32: Boolean = compareMajorMinorVersion((3, 2)) <= 0
+ private val comparedWithSpark33 = compareMajorMinorVersion((3, 3))
+ val ltSpark33: Boolean = comparedWithSpark33 < 0
Review Comment:
I meant `ltSpark33` is functionally equal to `lteSpark32`, we can remove the former and use the latter everywhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10334] Share the results compare the spark version [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #10335:
URL: https://github.com/apache/incubator-gluten/pull/10335#issuecomment-3148932171
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10334] Share the results compare the spark version [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10335:
URL: https://github.com/apache/incubator-gluten/pull/10335#issuecomment-3149363201
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10334] Share the results compare the spark version [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10335:
URL: https://github.com/apache/incubator-gluten/pull/10335#issuecomment-3153064806
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10334] Share the results compare the spark version [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10335:
URL: https://github.com/apache/incubator-gluten/pull/10335#issuecomment-3143866250
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10334] Share the results compare the spark version [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on code in PR #10335:
URL: https://github.com/apache/incubator-gluten/pull/10335#discussion_r2247986817
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkVersionUtil.scala:
##########
@@ -17,8 +17,15 @@
package org.apache.spark.util
object SparkVersionUtil {
+ val lteSpark32: Boolean = compareMajorMinorVersion((3, 2)) <= 0
+ private val comparedWithSpark33 = compareMajorMinorVersion((3, 3))
+ val ltSpark33: Boolean = comparedWithSpark33 < 0
Review Comment:
I'm not sure I understand your opinion. Could you tell me more?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10334] Share the results compare the spark version [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #10335:
URL: https://github.com/apache/incubator-gluten/pull/10335#issuecomment-3157557014
@FelixYBW @zhztheplayer Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10334] Share the results compare the spark version [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on code in PR #10335:
URL: https://github.com/apache/incubator-gluten/pull/10335#discussion_r2250551289
##########
gluten-core/src/main/scala/org/apache/spark/util/SparkVersionUtil.scala:
##########
@@ -17,8 +17,15 @@
package org.apache.spark.util
object SparkVersionUtil {
+ val lteSpark32: Boolean = compareMajorMinorVersion((3, 2)) <= 0
+ private val comparedWithSpark33 = compareMajorMinorVersion((3, 3))
+ val ltSpark33: Boolean = comparedWithSpark33 < 0
Review Comment:
Good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org