You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "beliefer (via GitHub)" <gi...@apache.org> on 2026/03/12 08:14:00 UTC
[PR] [GLUTEN-11692][CORE] Driver side should avoid submitting subqueries while validating file formats [gluten]
beliefer opened a new pull request, #11748:
URL: https://github.com/apache/gluten/pull/11748
## What changes are proposed in this pull request?
This PR proposes to avoid submitting subqueries while validating file formats.
Fixes #11692
## How was this patch tested?
Manual tests in our product environment.
## Was this patch authored or co-authored using generative AI tooling?
'No'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11692][CORE] Driver side should avoid submitting subqueries while validating file formats [gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #11748:
URL: https://github.com/apache/gluten/pull/11748#issuecomment-4046046446
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11692][CORE] Driver side should avoid submitting subqueries while validating file formats [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11748:
URL: https://github.com/apache/gluten/pull/11748#issuecomment-4053610047
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11692][CORE] Driver side should avoid submitting subqueries while validating file formats [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11748:
URL: https://github.com/apache/gluten/pull/11748#issuecomment-4044859813
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11692][CORE] Driver side should avoid submitting subqueries while validating file formats [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11748:
URL: https://github.com/apache/gluten/pull/11748#issuecomment-4064847063
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11692][CORE] Driver side should avoid submitting subqueries while validating file formats [gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on code in PR #11748:
URL: https://github.com/apache/gluten/pull/11748#discussion_r2932339055
##########
shims/spark33/src/main/scala/org/apache/spark/sql/hive/execution/AbstractHiveTableScanExec.scala:
##########
@@ -173,26 +173,30 @@ abstract private[hive] class AbstractHiveTableScanExec(
}
}
- @transient lazy val prunedPartitions: Seq[HivePartition] = {
+ // This is used on the driver side, so it is important to avoid executing subqueries
+ @transient lazy val basePrunedPartitions: Seq[HivePartition] = {
+ if (relation.prunedPartitions.nonEmpty) {
+ relation.prunedPartitions.get.map(HiveClientImpl.toHivePartition(_, hiveQlTable))
+ } else {
+ rawPartitions
+ }
+ }
+
+ @transient lazy val prunedPartitions: Seq[HivePartition] =
Review Comment:
Are you sure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11692][CORE] Driver side should avoid submitting subqueries while validating file formats [gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #11748:
URL: https://github.com/apache/gluten/pull/11748#discussion_r2931036971
##########
shims/spark33/src/main/scala/org/apache/spark/sql/hive/execution/AbstractHiveTableScanExec.scala:
##########
@@ -173,26 +173,30 @@ abstract private[hive] class AbstractHiveTableScanExec(
}
}
- @transient lazy val prunedPartitions: Seq[HivePartition] = {
+ // This is used on the driver side, so it is important to avoid executing subqueries
+ @transient lazy val basePrunedPartitions: Seq[HivePartition] = {
+ if (relation.prunedPartitions.nonEmpty) {
+ relation.prunedPartitions.get.map(HiveClientImpl.toHivePartition(_, hiveQlTable))
+ } else {
+ rawPartitions
+ }
+ }
+
+ @transient lazy val prunedPartitions: Seq[HivePartition] =
Review Comment:
nit: reduce redundant judgment
```
@transient lazy val prunedPartitions: Seq[HivePartition] =
if (partitionPruningPred.forall(!ExecSubqueryExpression.hasSubquery(_)) ||
(sparkSession.sessionState.conf.metastorePartitionPruning &&
partitionPruningPred.nonEmpty))
basePrunedPartitions
} else {
prunePartitions(basePrunedPartitions)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11692][CORE] Driver side should avoid submitting subqueries while validating file formats [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11748:
URL: https://github.com/apache/gluten/pull/11748#issuecomment-4044813541
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11692][CORE] Driver side should avoid submitting subqueries while validating file formats [gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer merged PR #11748:
URL: https://github.com/apache/gluten/pull/11748
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11692][CORE] Driver side should avoid submitting subqueries while validating file formats [gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on code in PR #11748:
URL: https://github.com/apache/gluten/pull/11748#discussion_r2937981347
##########
gluten-substrait/src/main/scala/org/apache/spark/sql/hive/HiveTableScanExecTransformer.scala:
##########
@@ -77,7 +77,16 @@ case class HiveTableScanExecTransformer(
partitionWithReadFileFormats
override def getDistinctPartitionReadFileFormats: Set[ReadFileFormat] =
Review Comment:
I updated comments at the declaration in the `WholeStageTransformer.scala`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11692][CORE] Driver side should avoid submitting subqueries while validating file formats [gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #11748:
URL: https://github.com/apache/gluten/pull/11748#discussion_r2931076334
##########
gluten-substrait/src/main/scala/org/apache/spark/sql/hive/HiveTableScanExecTransformer.scala:
##########
@@ -77,7 +77,16 @@ case class HiveTableScanExecTransformer(
partitionWithReadFileFormats
override def getDistinctPartitionReadFileFormats: Set[ReadFileFormat] =
Review Comment:
Could you also add a comment for this method? Mark it as only for validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11692][CORE] Driver side should avoid submitting subqueries while validating file formats [gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #11748:
URL: https://github.com/apache/gluten/pull/11748#issuecomment-4054559194
ping @wForget @jinchengchenghh @zhouyuan @zml1206
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org