You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "beliefer (via GitHub)" <gi...@apache.org> on 2025/08/27 03:05:49 UTC
[PR] [GLUTEN-10544] Remove unnecessary method separateScanRDD [incubator-gluten]
beliefer opened a new pull request, #10545:
URL: https://github.com/apache/incubator-gluten/pull/10545
## What changes are proposed in this pull request?
This PR proposes to remove unnecessary method `separateScanRDD`.
Fixes #10544
## How was this patch tested?
GA tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10544] Remove unnecessary method separateScanRDD [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on code in PR #10545:
URL: https://github.com/apache/incubator-gluten/pull/10545#discussion_r2312242124
##########
gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala:
##########
@@ -446,10 +446,11 @@ case class WholeStageTransformer(child: SparkPlan, materializeInput: Boolean = f
/**
* the whole stage contains NO [[LeafTransformSupport]]. this the default case for:
- * 1. SCAN with clickhouse backend (check ColumnarCollapseTransformStages#separateScanRDD())
- * 2. test case where query plan is constructed from simple dataframes (e.g.
- * GlutenDataFrameAggregateSuite) in these cases, separate RDDs takes care of SCAN as a
- * result, genFinalStageIterator rather than genFirstStageIterator will be invoked
+ * 1. SCAN with clickhouse backend (check
Review Comment:
I just want replace `ColumnarCollapseTransformStages#separateScanRDD()` with `BackendsApiManager.getSettings.excludeScanExecFromCollapsedStage()`, but the spotless make this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10544] Remove unnecessary method separateScanRDD [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #10545:
URL: https://github.com/apache/incubator-gluten/pull/10545#issuecomment-3272944858
@zml1206 @PHILO-HE @jinchengchenghh Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10544] Remove unnecessary method separateScanRDD [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10545:
URL: https://github.com/apache/incubator-gluten/pull/10545#issuecomment-3226587952
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10544] Remove unnecessary method separateScanRDD [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on code in PR #10545:
URL: https://github.com/apache/incubator-gluten/pull/10545#discussion_r2333071542
##########
gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala:
##########
@@ -446,10 +446,11 @@ case class WholeStageTransformer(child: SparkPlan, materializeInput: Boolean = f
/**
* the whole stage contains NO [[LeafTransformSupport]]. this the default case for:
- * 1. SCAN with clickhouse backend (check ColumnarCollapseTransformStages#separateScanRDD())
- * 2. test case where query plan is constructed from simple dataframes (e.g.
- * GlutenDataFrameAggregateSuite) in these cases, separate RDDs takes care of SCAN as a
- * result, genFinalStageIterator rather than genFirstStageIterator will be invoked
+ * 1. SCAN with clickhouse backend (check
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10544] Remove unnecessary method separateScanRDD [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10545:
URL: https://github.com/apache/incubator-gluten/pull/10545#issuecomment-3226587644
https://github.com/apache/incubator-gluten/issues/10544
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10544] Remove unnecessary method separateScanRDD [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #10545:
URL: https://github.com/apache/incubator-gluten/pull/10545#issuecomment-3240742198
ping @zzcclp @zml1206 @FelixYBW
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10544] Remove unnecessary method separateScanRDD [incubator-gluten]
Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 merged PR #10545:
URL: https://github.com/apache/incubator-gluten/pull/10545
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10544] Remove unnecessary method separateScanRDD [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #10545:
URL: https://github.com/apache/incubator-gluten/pull/10545#discussion_r2304639243
##########
gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala:
##########
@@ -446,10 +446,11 @@ case class WholeStageTransformer(child: SparkPlan, materializeInput: Boolean = f
/**
* the whole stage contains NO [[LeafTransformSupport]]. this the default case for:
- * 1. SCAN with clickhouse backend (check ColumnarCollapseTransformStages#separateScanRDD())
- * 2. test case where query plan is constructed from simple dataframes (e.g.
- * GlutenDataFrameAggregateSuite) in these cases, separate RDDs takes care of SCAN as a
- * result, genFinalStageIterator rather than genFirstStageIterator will be invoked
+ * 1. SCAN with clickhouse backend (check
Review Comment:
What's the change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10544] Remove unnecessary method separateScanRDD [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10545:
URL: https://github.com/apache/incubator-gluten/pull/10545#issuecomment-3270140985
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10544] Remove unnecessary method separateScanRDD [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #10545:
URL: https://github.com/apache/incubator-gluten/pull/10545#issuecomment-3268691048
cc @PHILO-HE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10544] Remove unnecessary method separateScanRDD [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on code in PR #10545:
URL: https://github.com/apache/incubator-gluten/pull/10545#discussion_r2332772067
##########
gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala:
##########
@@ -446,10 +446,11 @@ case class WholeStageTransformer(child: SparkPlan, materializeInput: Boolean = f
/**
* the whole stage contains NO [[LeafTransformSupport]]. this the default case for:
- * 1. SCAN with clickhouse backend (check ColumnarCollapseTransformStages#separateScanRDD())
- * 2. test case where query plan is constructed from simple dataframes (e.g.
- * GlutenDataFrameAggregateSuite) in these cases, separate RDDs takes care of SCAN as a
- * result, genFinalStageIterator rather than genFirstStageIterator will be invoked
+ * 1. SCAN with clickhouse backend (check
Review Comment:
@beliefer, I suggest to refine the comments for better readability as follows. Thanks.
```diff
diff --git a/gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala b/gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala
index 0c5e1b58b..588ba4567 100644
--- a/gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala
+++ b/gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala
@@ -438,11 +438,14 @@ case class WholeStageTransformer(child: SparkPlan, materializeInput: Boolean = f
} else {
/**
- * the whole stage contains NO [[LeafTransformSupport]]. this the default case for:
- * 1. SCAN with clickhouse backend (check ColumnarCollapseTransformStages#separateScanRDD())
- * 2. test case where query plan is constructed from simple dataframes (e.g.
- * GlutenDataFrameAggregateSuite) in these cases, separate RDDs takes care of SCAN as a
- * result, genFinalStageIterator rather than genFirstStageIterator will be invoked
+ * The whole stage contains NO [[LeafTransformSupport]]. This is the default case for:
+ * - SCAN of clickhouse backend. See
+ * BackendsApiManager.getSettings.excludeScanExecFromCollapsedStage.
+ * - Test case where query plan is constructed from simple DataFrames, e.g.
+ * GlutenDataFrameAggregateSuite.
+ *
+ * In these cases, separate RDDs take care of SCAN. As a result, genFinalStageIterator rather
+ * than genFirstStageIterator will be invoked.
*/
new WholeStageZippedPartitionsRDD(
sparkContext,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org