You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "wecharyu (via GitHub)" <gi...@apache.org> on 2026/08/05 10:28:37 UTC
[PR] [GLUTEN-12702][VL] Do not count structural Spark plans as fallback nodes [gluten]
wecharyu opened a new pull request, #12703:
URL: https://github.com/apache/gluten/pull/12703
## What changes are proposed in this pull request?
- Add `GlutenExplainUtils.isFallbackNode` to distinguish actual Spark fallback operators from Gluten plans and structural/framework nodes
- Reuse the helper in `GlutenExplainUtils` and `GlutenImplicits` fallback walkers.
Fix https://github.com/apache/gluten/issues/12702.
## How was this patch tested?
Add a new UT.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex GPT-5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12702][VL] Do not count structural Spark plans as fallback nodes [gluten]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #12703:
URL: https://github.com/apache/gluten/pull/12703#discussion_r3854590821
##########
gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenExplainUtils.scala:
##########
@@ -45,6 +45,36 @@ import scala.collection.mutable.{ArrayBuffer, BitSet}
object GlutenExplainUtils extends AdaptiveSparkPlanHelper {
type FallbackInfo = (Int, Map[String, String])
+ /**
+ * Returns whether a plan should be ignored when collecting fallback statistics.
+ *
+ * Such plans are structural wrappers rather than execution operators: they are neither fallback
+ * Spark operators nor native Gluten operators.
+ */
+ def isFallbackInsensitivePlan(plan: SparkPlan): Boolean = plan match {
+ case _: ExecutedCommandExec => true
+ case _: CommandResultExec => true
+ case p: V2CommandExec =>
+ !FallbackTags.nonEmpty(p) && p.logicalLink.forall(FallbackTags.getOption(_).isEmpty)
+ case _: DataWritingCommandExec => true
+ case _: WholeStageCodegenExec => true
+ case _: WholeStageTransformer => true
+ case _: InputAdapter => true
+ case _: ColumnarInputAdapter => true
+ case _: InputIteratorTransformer => true
+ case _: ColumnarToRowTransition => true
+ case _: RowToColumnarTransition => true
+ case _: ReusedExchangeExec => true
+ case _: NoopLeaf => true
+ case w: WriteFilesExec => w.child.isInstanceOf[NoopLeaf]
+ case _: AdaptiveSparkPlanExec => true
+ case _: QueryStageExec => true
+ case _: AQEShuffleReadExec => true
+ case _: ColumnarAQEShuffleReadExec => true
+ case i: InMemoryTableScanExec => PlanUtil.isGlutenTableCache(i)
Review Comment:
Make sense, remove it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12702][VL] Do not count structural Spark plans as fallback nodes [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he commented on code in PR #12703:
URL: https://github.com/apache/gluten/pull/12703#discussion_r3818697380
##########
gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenExplainUtils.scala:
##########
@@ -45,6 +45,36 @@ import scala.collection.mutable.{ArrayBuffer, BitSet}
object GlutenExplainUtils extends AdaptiveSparkPlanHelper {
type FallbackInfo = (Int, Map[String, String])
+ /**
+ * Returns whether a plan should be ignored when collecting fallback statistics.
+ *
+ * Such plans are structural wrappers rather than execution operators: they are neither fallback
+ * Spark operators nor native Gluten operators.
+ */
+ def isFallbackInsensitivePlan(plan: SparkPlan): Boolean = plan match {
+ case _: ExecutedCommandExec => true
+ case _: CommandResultExec => true
+ case p: V2CommandExec =>
+ !FallbackTags.nonEmpty(p) && p.logicalLink.forall(FallbackTags.getOption(_).isEmpty)
+ case _: DataWritingCommandExec => true
+ case _: WholeStageCodegenExec => true
+ case _: WholeStageTransformer => true
+ case _: InputAdapter => true
+ case _: ColumnarInputAdapter => true
+ case _: InputIteratorTransformer => true
+ case _: ColumnarToRowTransition => true
+ case _: RowToColumnarTransition => true
+ case _: ReusedExchangeExec => true
+ case _: NoopLeaf => true
+ case w: WriteFilesExec => w.child.isInstanceOf[NoopLeaf]
+ case _: AdaptiveSparkPlanExec => true
+ case _: QueryStageExec => true
+ case _: AQEShuffleReadExec => true
+ case _: ColumnarAQEShuffleReadExec => true
+ case i: InMemoryTableScanExec => PlanUtil.isGlutenTableCache(i)
Review Comment:
I think this Gluten InMemoryTableScanExec shouldn't be viewed as FallbackInsensitive plan. And I notice it is already included as a case in `GlutenExplainUtils.scala`.
##########
gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenExplainUtils.scala:
##########
@@ -45,6 +45,36 @@ import scala.collection.mutable.{ArrayBuffer, BitSet}
object GlutenExplainUtils extends AdaptiveSparkPlanHelper {
type FallbackInfo = (Int, Map[String, String])
+ /**
+ * Returns whether a plan should be ignored when collecting fallback statistics.
+ *
+ * Such plans are structural wrappers rather than execution operators: they are neither fallback
+ * Spark operators nor native Gluten operators.
+ */
+ def isFallbackInsensitivePlan(plan: SparkPlan): Boolean = plan match {
Review Comment:
I note some cases are not really structural nodes. If we'd like to make this method only focus on structural node check, the proposed method name would be `isStructuralPlanNode` for easily understanding.
In addition, we should move the following non-structural nodes to the caller side checking, e.g. let the visit method ignore them directly (or conditionally depending on the if condition).
```
ExecutedCommandExec, CommandResultExec, DataWritingCommandExec, V2CommandExec, WriteFilesExec,
AQEShuffleReadExec, ColumnarAQEShuffleReadExec and InMemoryTableScanExec
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12702][VL] Do not count structural Spark plans as fallback nodes [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12703:
URL: https://github.com/apache/gluten/pull/12703#issuecomment-5203329064
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12702][VL] Do not count structural Spark plans as fallback nodes [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12703:
URL: https://github.com/apache/gluten/pull/12703#issuecomment-5191011200
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12702][VL] Do not count structural Spark plans as fallback nodes [gluten]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #12703:
URL: https://github.com/apache/gluten/pull/12703#discussion_r3854642386
##########
gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenExplainUtils.scala:
##########
@@ -45,6 +45,36 @@ import scala.collection.mutable.{ArrayBuffer, BitSet}
object GlutenExplainUtils extends AdaptiveSparkPlanHelper {
type FallbackInfo = (Int, Map[String, String])
+ /**
+ * Returns whether a plan should be ignored when collecting fallback statistics.
+ *
+ * Such plans are structural wrappers rather than execution operators: they are neither fallback
+ * Spark operators nor native Gluten operators.
+ */
+ def isFallbackInsensitivePlan(plan: SparkPlan): Boolean = plan match {
Review Comment:
This function aims to filter out the nodes that should not be considered as native operator or fallback operator, it contains structural nodes and some wrappers, so we named it `isFallbackInsensitivePlan`.
Updated the comments of misleading structural.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12702][VL] Do not count structural Spark plans as fallback nodes [gluten]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #12703:
URL: https://github.com/apache/gluten/pull/12703#discussion_r3869890034
##########
gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenExplainUtils.scala:
##########
@@ -45,6 +45,34 @@ import scala.collection.mutable.{ArrayBuffer, BitSet}
object GlutenExplainUtils extends AdaptiveSparkPlanHelper {
type FallbackInfo = (Int, Map[String, String])
+ /**
+ * Returns whether a plan should be ignored when collecting fallback statistics.
+ *
+ * Such plans may be execution-framework or implementation wrappers.
+ */
+ def isFallbackInsensitivePlan(plan: SparkPlan): Boolean = plan match {
Review Comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12702][VL] Introduce shouldIgnoreInFallbackStats to skip certain plan nodes in fallback counting [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he merged PR #12703:
URL: https://github.com/apache/gluten/pull/12703
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12702][VL] Do not count structural Spark plans as fallback nodes [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12703:
URL: https://github.com/apache/gluten/pull/12703#issuecomment-5419657497
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12702][VL] Do not count structural Spark plans as fallback nodes [gluten]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on PR #12703:
URL: https://github.com/apache/gluten/pull/12703#issuecomment-5263762466
@philo-he @zjuwangg Could you help review this PR? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12702][VL] Do not count structural Spark plans as fallback nodes [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12703:
URL: https://github.com/apache/gluten/pull/12703#issuecomment-5204436492
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12702][VL] Do not count structural Spark plans as fallback nodes [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12703:
URL: https://github.com/apache/gluten/pull/12703#issuecomment-5412865717
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12702][VL] Do not count structural Spark plans as fallback nodes [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12703:
URL: https://github.com/apache/gluten/pull/12703#discussion_r3861243767
##########
gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenExplainUtils.scala:
##########
@@ -45,6 +45,34 @@ import scala.collection.mutable.{ArrayBuffer, BitSet}
object GlutenExplainUtils extends AdaptiveSparkPlanHelper {
type FallbackInfo = (Int, Map[String, String])
+ /**
+ * Returns whether a plan should be ignored when collecting fallback statistics.
+ *
+ * Such plans may be execution-framework or implementation wrappers.
+ */
+ def isFallbackInsensitivePlan(plan: SparkPlan): Boolean = plan match {
Review Comment:
The term “fallback insensitive” is a bit ambiguous—this helper is effectively “ignore for fallback statistics” / “structural wrapper node.” Consider renaming to something more directly descriptive (e.g., `shouldIgnoreInFallbackStats` or `isStructuralWrapperForFallbackStats`) to reduce misinterpretation by future readers.
##########
gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala:
##########
@@ -145,8 +147,13 @@ case class GlutenAutoAdjustStageResourceProfile(glutenConf: GlutenConfig, spark:
// case 2: check whether fallback exists and decide whether increase heap memory
// and decrease offheap memory.
- val fallenNodeCnt = planNodes.count(p => !p.isInstanceOf[GlutenPlan])
- val totalCount = planNodes.size
+ val countedPlanNodes = planNodes.filterNot(GlutenExplainUtils.isFallbackInsensitivePlan)
+ val fallenNodeCnt = countedPlanNodes.count {
+ case _: GlutenPlan => false
+ case i: InMemoryTableScanExec => !PlanUtil.isGlutenTableCache(i)
+ case _ => true
+ }
+ val totalCount = countedPlanNodes.size
if (1.0 * fallenNodeCnt / totalCount >= glutenConf.autoAdjustStageFallenNodeThreshold) {
Review Comment:
`totalCount` can become `0` after filtering (e.g., for plans comprised only of ignored/structural nodes like certain command/AQE wrappers). In that case `fallenNodeCnt / totalCount` evaluates to `Infinity` and can incorrectly trigger auto-adjustment. Add an explicit guard (e.g., require `totalCount > 0` before computing the ratio, otherwise skip adjustment). Also consider adding a UT for the `totalCount == 0` scenario to prevent regressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12702][VL] Do not count structural Spark plans as fallback nodes [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12703:
URL: https://github.com/apache/gluten/pull/12703#issuecomment-5422813864
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12702][VL] Do not count structural Spark plans as fallback nodes [gluten]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #12703:
URL: https://github.com/apache/gluten/pull/12703#discussion_r3869824808
##########
gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala:
##########
@@ -145,8 +147,13 @@ case class GlutenAutoAdjustStageResourceProfile(glutenConf: GlutenConfig, spark:
// case 2: check whether fallback exists and decide whether increase heap memory
// and decrease offheap memory.
- val fallenNodeCnt = planNodes.count(p => !p.isInstanceOf[GlutenPlan])
- val totalCount = planNodes.size
+ val countedPlanNodes = planNodes.filterNot(GlutenExplainUtils.isFallbackInsensitivePlan)
+ val fallenNodeCnt = countedPlanNodes.count {
+ case _: GlutenPlan => false
+ case i: InMemoryTableScanExec => !PlanUtil.isGlutenTableCache(i)
+ case _ => true
+ }
+ val totalCount = countedPlanNodes.size
if (1.0 * fallenNodeCnt / totalCount >= glutenConf.autoAdjustStageFallenNodeThreshold) {
Review Comment:
`totalCount == 0` means `fallenNode == 0`, then the fallenNodeCnt / totalCount is `NaN` rather than `Infinity`, and `NaN` compare is always false so no auto-adjustment is triggered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12702][VL] Do not count structural Spark plans as fallback nodes [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12703:
URL: https://github.com/apache/gluten/pull/12703#issuecomment-5436320948
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org