You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "jinchengchenghh (via GitHub)" <gi...@apache.org> on 2025/11/25 12:35:18 UTC
[PR] [GLUTEN-11088][VL] Fix the Spark4.0 storage partition join [incubator-gluten]
jinchengchenghh opened a new pull request, #11184:
URL: https://github.com/apache/incubator-gluten/pull/11184
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix the Spark4.0 storage partition join [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11184:
URL: https://github.com/apache/incubator-gluten/pull/11184#issuecomment-3580531384
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix the Spark4.0 storage partition join [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11184:
URL: https://github.com/apache/incubator-gluten/pull/11184#issuecomment-3580999175
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix the Spark4.0 storage partition join [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11184:
URL: https://github.com/apache/incubator-gluten/pull/11184#issuecomment-3575441100
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix the Spark4.0 storage partition join [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #11184:
URL: https://github.com/apache/incubator-gluten/pull/11184#discussion_r2568154282
##########
shims/spark40/src/main/scala/org/apache/gluten/sql/shims/spark40/Spark40Shims.scala:
##########
@@ -481,10 +481,9 @@ class Spark40Shims extends SparkShims {
applyPartialClustering: Boolean,
replicatePartitions: Boolean,
joinKeyPositions: Option[Seq[Int]] = None): Seq[Seq[InputPartition]] = {
+ val original = batchScan.asInstanceOf[BatchScanExecShim]
Review Comment:
Could you help do a refactor for the orderPartitions function, this is mostly copied from BatchScanExec::inputRDD, and inputPartitionsShim may can be replaced by inputPartitions @beliefer Thanks! https://github.com/apache/incubator-gluten/blob/49389cd05ea07356f71bfdfe660410604c1461ea/shims/spark40/src/main/scala/org/apache/spark/sql/execution/datasources/v2/AbstractBatchScanExec.scala#L60
https://github.com/apache/incubator-gluten/blob/d636fa77c49e991eb02159a0c25431eb499c6da2/shims/spark40/src/main/scala/org/apache/spark/sql/execution/datasources/v2/AbstractBatchScanExec.scala#L142
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix the Spark4.0 storage partition join [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #11184:
URL: https://github.com/apache/incubator-gluten/pull/11184#discussion_r2568168691
##########
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/connector/GlutenKeyGroupedPartitioningSuite.scala:
##########
@@ -72,10 +84,46 @@ class GlutenKeyGroupedPartitioningSuite
case s: SortMergeJoinExec => s
}.flatMap(smj => collect(smj) { case s: ColumnarShuffleExchangeExec => s })
}
+
+ private def collectShuffles(plan: SparkPlan): Seq[ShuffleExchangeLike] = {
Review Comment:
Will create a PR in apache/Spark to make the function from private to protected, then we can only override the function to check the plan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix the Spark4.0 storage partition join [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11184:
URL: https://github.com/apache/incubator-gluten/pull/11184#issuecomment-3582815508
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix the Spark4.0 storage partition join [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11184:
URL: https://github.com/apache/incubator-gluten/pull/11184#issuecomment-3580559213
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix the Spark4.0 storage partition join [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on code in PR #11184:
URL: https://github.com/apache/incubator-gluten/pull/11184#discussion_r2568170193
##########
shims/spark40/src/main/scala/org/apache/gluten/sql/shims/spark40/Spark40Shims.scala:
##########
@@ -481,10 +481,9 @@ class Spark40Shims extends SparkShims {
applyPartialClustering: Boolean,
replicatePartitions: Boolean,
joinKeyPositions: Option[Seq[Int]] = None): Seq[Seq[InputPartition]] = {
+ val original = batchScan.asInstanceOf[BatchScanExecShim]
Review Comment:
@jinchengchenghh would you please help to create a issue to track on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix the Spark4.0 storage partition join [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11184:
URL: https://github.com/apache/incubator-gluten/pull/11184#issuecomment-3580537834
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix the Spark4.0 storage partition join [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11184:
URL: https://github.com/apache/incubator-gluten/pull/11184#issuecomment-3582062521
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix the Spark4.0 storage partition join [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11184:
URL: https://github.com/apache/incubator-gluten/pull/11184#issuecomment-3576398724
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix the Spark4.0 storage partition join [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11184:
URL: https://github.com/apache/incubator-gluten/pull/11184#issuecomment-3581529002
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix the Spark4.0 storage partition join [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11184:
URL: https://github.com/apache/incubator-gluten/pull/11184#issuecomment-3582596943
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix the Spark4.0 storage partition join [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh merged PR #11184:
URL: https://github.com/apache/incubator-gluten/pull/11184
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11088][VL] Fix the Spark4.0 storage partition join [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #11184:
URL: https://github.com/apache/incubator-gluten/pull/11184#discussion_r2568208498
##########
shims/spark40/src/main/scala/org/apache/gluten/sql/shims/spark40/Spark40Shims.scala:
##########
@@ -481,10 +481,9 @@ class Spark40Shims extends SparkShims {
applyPartialClustering: Boolean,
replicatePartitions: Boolean,
joinKeyPositions: Option[Seq[Int]] = None): Seq[Seq[InputPartition]] = {
+ val original = batchScan.asInstanceOf[BatchScanExecShim]
Review Comment:
Created https://github.com/apache/incubator-gluten/issues/11207
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org