You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/22 04:41:38 UTC
[GitHub] [spark] CodingCat opened a new pull request #29831: [SPARK-32351][SQL] pushdown filter in FileSourceScanExec and HiveTableScanExec
CodingCat opened a new pull request #29831:
URL: https://github.com/apache/spark/pull/29831
### What changes were proposed in this pull request?
This patch proposes to extract partition filters with `extractPredicatesWithinOutputSet` added in https://github.com/apache/spark/pull/29101/files#diff-6be42cfa3c62a7536b1eb1d6447c073c
it will show the pushed down partition filter in explain()
### Why are the changes needed?
without the patch, the explained plan is inconsistent with what is actually executed
### Does this PR introduce _any_ user-facing change
no
### How was this patch tested?
Unit test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702836721
Jenkins, retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liancheng commented on a change in pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
liancheng commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r493197246
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,42 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, expectedPushedDownFilterCount = 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
assertPrunedPartitions(
"""
|SELECT i, COUNT(1) FROM (
|SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)
|) tmp GROUP BY i
- """.stripMargin, 2)
+ """.stripMargin, 2, 1)
}
}
}
- protected def assertPrunedPartitions(query: String, expected: Long): Unit = {
- val plan = sql(query).queryExecution.sparkPlan
- assert(getScanExecPartitionSize(plan) == expected)
+ protected def assertPrunedPartitions(
+ query: String,
+ expectedPartitionCount: Long,
+ expectedPushedDownFilterCount: Int): Unit = {
+ val qe = sql(query).queryExecution
+ val plan = qe.sparkPlan
+ assert(getScanExecPartitionSize(plan) == expectedPartitionCount)
+ val pushedDownPartitionFilters = qe.executedPlan.collectFirst {
+ case FileSourceScanExec(_, _, _, partitionFilters, _, _, _, _) =>
+ partitionFilters
+ case HiveTableScanExec(_, _, partitionFilters) =>
+ partitionFilters
+ }
+ assert(pushedDownPartitionFilters.isDefined &&
+ pushedDownPartitionFilters.get.length == expectedPushedDownFilterCount)
Review comment:
Ah, good to know!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-708859736
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34383/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-713639134
> The current code assumes the same partition filters are generated again in FileSourceStrategy and passed into FileSourceScanExec.
A further refactor can be: remove the partition filters after they are pushed down into the catalog file index in the optimizer rule. Also keep the partition filters in the query plan so that we can explain them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-706343012
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34194/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-712543428
Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696514552
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r498568307
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,39 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
assertPrunedPartitions(
"""
|SELECT i, COUNT(1) FROM (
|SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)
|) tmp GROUP BY i
- """.stripMargin, 2)
+ """.stripMargin, 2, 1)
}
}
}
- protected def assertPrunedPartitions(query: String, expected: Long): Unit = {
- val plan = sql(query).queryExecution.sparkPlan
- assert(getScanExecPartitionSize(plan) == expected)
+ protected def assertPrunedPartitions(
+ query: String,
+ expectedPartitionCount: Long,
+ expectedPushedDownFilterCount: Int): Unit = {
Review comment:
ok, but IMO, the current comparison actually achieved the similar goal?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701061818
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696968587
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702567364
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701103067
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702535182
**[Test build #129336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129336/testReport)** for PR 29831 at commit [`15ec353`](https://github.com/apache/spark/commit/15ec3534e345631fd775d5679507e651291e0552).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696517046
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702970066
**[Test build #129367 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129367/testReport)** for PR 29831 at commit [`15ec353`](https://github.com/apache/spark/commit/15ec3534e345631fd775d5679507e651291e0552).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696505226
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceStrategy and HiveTableScans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696546731
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705377324
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-706352540
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34194/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696968005
**[Test build #128993 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128993/testReport)** for PR 29831 at commit [`428fc64`](https://github.com/apache/spark/commit/428fc64b950c856f62994e00e395135cafa43653).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701106800
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-711550645
@cloud-fan @HyukjinKwon @gengliangwang @maropu @liancheng @viirya @sunchao thanks for the review!
addressed the last comment, is it ready to go?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-698571139
Yeah, I will look at this again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r501432300
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
##########
@@ -254,22 +254,28 @@ private[hive] trait HiveStrategies {
* Retrieves data using a HiveTableScan. Partition pruning predicates are also detected and
* applied.
*/
- object HiveTableScans extends Strategy {
+ object HiveTableScans extends Strategy with PredicateHelper {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ScanOperation(projectList, predicates, relation: HiveTableRelation) =>
+ case ScanOperation(projectList, filters, relation: HiveTableRelation) =>
// Filter out all predicates that only deal with partition keys, these are given to the
// hive table scan operator to be used for partition pruning.
val partitionKeyIds = AttributeSet(relation.partitionCols)
- val (pruningPredicates, otherPredicates) = predicates.partition { predicate =>
- !predicate.references.isEmpty &&
- predicate.references.subsetOf(partitionKeyIds)
+ val normalized = DataSourceStrategy.normalizeExprs(
+ filters.filter(_.deterministic), relation.output)
+ val partitionKeyFilters = if (relation.partitionCols.isEmpty) {
+ ExpressionSet(Nil)
+ } else {
+ val predicates = ExpressionSet(normalized
+ .flatMap(extractPredicatesWithinOutputSet(_, partitionKeyIds)))
+ logInfo(s"Pruning partitions with: ${filters.mkString(",")}")
+ predicates
}
pruneFilterProject(
projectList,
- otherPredicates,
+ (ExpressionSet(filters) -- partitionKeyFilters).toSeq,
Review comment:
@cloud-fan sorry, missed this, you're right, just changed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705198973
**[Test build #129522 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129522/testReport)** for PR 29831 at commit [`e3262e3`](https://github.com/apache/spark/commit/e3262e349adf96adc2f20e464b80730d43ea93e2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702870597
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r499528767
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -484,6 +485,20 @@ object DataSourceStrategy {
}
}
+ private[sql] def getPushedDownFilters(
Review comment:
You can remove "private[sql]" since we're already in the private package `execution`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701063643
**[Test build #129260 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129260/testReport)** for PR 29831 at commit [`55672e4`](https://github.com/apache/spark/commit/55672e4cff1580a096d4c2f9b633a6edfc933185).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-703432522
@maropu any other comments?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705830748
**[Test build #129564 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129564/testReport)** for PR 29831 at commit [`3cec094`](https://github.com/apache/spark/commit/3cec094fd2c9dae15cc15c10d9219c98ec5c651b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceStrategy and HiveTableScans
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696529692
Could you add explain result before and after in the PR description? So it can be clearer to know what is changed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705199897
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705096317
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liancheng commented on a change in pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
liancheng commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r492476475
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,42 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, expectedPushedDownFilterCount = 0)
Review comment:
Why only adding the parameter name for this line?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702549334
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33949/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceStrategy and HiveTableScans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696546982
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceStrategy and HiveTableScans
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696519602
**[Test build #128965 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128965/testReport)** for PR 29831 at commit [`54b896a`](https://github.com/apache/spark/commit/54b896a2d840d926db560f987e894643a27c571b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-708910460
**[Test build #129777 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129777/testReport)** for PR 29831 at commit [`52ad2ea`](https://github.com/apache/spark/commit/52ad2eae736f2d188db93caa152d155ba42d7faf).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-700425678
**[Test build #129213 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129213/testReport)** for PR 29831 at commit [`8862aa0`](https://github.com/apache/spark/commit/8862aa034274f6c81fcaecfea7b54a74d795e46f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on a change in pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
CodingCat commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r492478729
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,42 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, expectedPushedDownFilterCount = 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
assertPrunedPartitions(
"""
|SELECT i, COUNT(1) FROM (
|SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)
|) tmp GROUP BY i
- """.stripMargin, 2)
+ """.stripMargin, 2, 1)
}
}
}
- protected def assertPrunedPartitions(query: String, expected: Long): Unit = {
- val plan = sql(query).queryExecution.sparkPlan
- assert(getScanExecPartitionSize(plan) == expected)
+ protected def assertPrunedPartitions(
+ query: String,
+ expectedPartitionCount: Long,
+ expectedPushedDownFilterCount: Int): Unit = {
+ val qe = sql(query).queryExecution
+ val plan = qe.sparkPlan
+ assert(getScanExecPartitionSize(plan) == expectedPartitionCount)
+ val pushedDownPartitionFilters = qe.executedPlan.collectFirst {
+ case FileSourceScanExec(_, _, _, partitionFilters, _, _, _, _) =>
+ partitionFilters
+ case HiveTableScanExec(_, _, partitionFilters) =>
+ partitionFilters
+ }
+ assert(pushedDownPartitionFilters.isDefined &&
+ pushedDownPartitionFilters.get.length == expectedPushedDownFilterCount)
Review comment:
thanks for the suggestion! I agree the suggested way is cleaner
but in the latest scalatest, it actually can tell something like `pushedDownPartitionFilters is defined, but `whatever` has length of 0`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r503009862
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -467,13 +468,13 @@ case class DataSourceStrategy(conf: SQLConf) extends Strategy with Logging with
}
}
-object DataSourceStrategy {
+object DataSourceStrategy extends PredicateHelper {
/**
* The attribute name may differ from the one in the schema if the query analyzer
* is case insensitive. We should change attribute names to match the ones in the schema,
* so we do not need to worry about case sensitivity anymore.
*/
- protected[sql] def normalizeExprs(
Review comment:
This was fixed at SPARK-16964
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705312425
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-706422715
**[Test build #129591 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129591/testReport)** for PR 29831 at commit [`d993c2d`](https://github.com/apache/spark/commit/d993c2df7274ba069bdb9fddaa3d39e6479f859f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] pushdown filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696505226
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705199897
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r503800662
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
##########
@@ -256,20 +256,21 @@ private[hive] trait HiveStrategies {
*/
object HiveTableScans extends Strategy {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ScanOperation(projectList, predicates, relation: HiveTableRelation) =>
+ case ScanOperation(projectList, filters, relation: HiveTableRelation) =>
// Filter out all predicates that only deal with partition keys, these are given to the
// hive table scan operator to be used for partition pruning.
val partitionKeyIds = AttributeSet(relation.partitionCols)
- val (pruningPredicates, otherPredicates) = predicates.partition { predicate =>
- !predicate.references.isEmpty &&
- predicate.references.subsetOf(partitionKeyIds)
- }
+ val normalizedFilters = DataSourceStrategy.normalizeExprs(
+ filters.filter(_.deterministic), relation.output)
+
+ val partitionKeyFilters = DataSourceStrategy.getPushedDownFilters(relation.partitionCols,
+ normalizedFilters)
pruneFilterProject(
projectList,
- otherPredicates,
+ filters.filter(f => f.references.isEmpty || !f.references.subsetOf(partitionKeyIds)),
Review comment:
`f.references.isEmpty` means it's true or false literal?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705329539
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-706152111
> FileSourceScanExec uses the partition filters when listing files, but the filters do nothing because unnecessary partitions are already pruned in advance, so the filters are mainly used for explain output in this case
Actually, the partition filters can contain dynamic partition pruning filters. So the description is wrong here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liancheng commented on a change in pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
liancheng commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r492477664
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
##########
@@ -254,22 +254,29 @@ private[hive] trait HiveStrategies {
* Retrieves data using a HiveTableScan. Partition pruning predicates are also detected and
* applied.
*/
- object HiveTableScans extends Strategy {
+ object HiveTableScans extends Strategy with PredicateHelper {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ScanOperation(projectList, predicates, relation: HiveTableRelation) =>
+ case ScanOperation(projectList, filters, relation: HiveTableRelation) =>
// Filter out all predicates that only deal with partition keys, these are given to the
// hive table scan operator to be used for partition pruning.
val partitionKeyIds = AttributeSet(relation.partitionCols)
- val (pruningPredicates, otherPredicates) = predicates.partition { predicate =>
- !predicate.references.isEmpty &&
- predicate.references.subsetOf(partitionKeyIds)
+ val normalized = DataSourceStrategy.normalizeExprs(
+ filters.filter(_.deterministic), relation.output)
+ val partitionKeyFilters = if (relation.partitionCols.isEmpty) {
+ ExpressionSet(Nil)
+ } else {
+ val predicates = ExpressionSet(normalized
+ .flatMap(extractPredicatesWithinOutputSet(_, partitionKeyIds)))
+ .filter(_.references.subsetOf(partitionKeyIds))
+ logInfo(s"Pruning directories with: ${filters.mkString(",")}")
Review comment:
```suggestion
logInfo(s"Pruning partitions with: ${filters.mkString(",")}")
```
(Conceptually, partitioning can be implemented using mechanisms other than filesystem directories.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705829750
Jenkins, retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-707450125
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceStrategy and HiveTableScans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696547123
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-711616660
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696517049
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128964/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701106793
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33877/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-700470674
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705377333
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129537/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r498561930
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,39 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
assertPrunedPartitions(
"""
|SELECT i, COUNT(1) FROM (
|SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)
|) tmp GROUP BY i
- """.stripMargin, 2)
+ """.stripMargin, 2, 1)
}
}
}
- protected def assertPrunedPartitions(query: String, expected: Long): Unit = {
- val plan = sql(query).queryExecution.sparkPlan
- assert(getScanExecPartitionSize(plan) == expected)
+ protected def assertPrunedPartitions(
+ query: String,
+ expectedPartitionCount: Long,
+ expectedPushedDownFilterCount: Int): Unit = {
Review comment:
Sorry to confuse you, but I meant it like this;
```
protected def assertPrunedPartitions(
query: String,
expectedPartitionCount: Long,
expectedPartitionFilters: Seq[Expression]): Unit = {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r505194284
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +47,66 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2,
+ "((`p` = '1') || (`p` = '2'))")
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4,
+ "")
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2,
+ "((`p` = '1') || (`p` = '3'))")
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3,
+ "((`p` = '1') || ((`p` = '2') || (`p` = '3')))")
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4,
+ "")
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1,
+ "(`p` = '1')")
assertPrunedPartitions(
"""
|SELECT i, COUNT(1) FROM (
|SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)
|) tmp GROUP BY i
- """.stripMargin, 2)
+ """.stripMargin, 2, "((`p` = '1') || (`p` = '2'))")
}
}
}
- protected def assertPrunedPartitions(query: String, expected: Long): Unit = {
- val plan = sql(query).queryExecution.sparkPlan
- assert(getScanExecPartitionSize(plan) == expected)
+ private def getCleanStringRepresentation(exp: Expression): String = exp match {
+ case attr: AttributeReference =>
+ attr.sql
+ case l: Literal =>
+ l.sql
+ case e: BinaryOperator =>
+ (s"(${getCleanStringRepresentation(e.left)} ${e.symbol} " +
+ s"${getCleanStringRepresentation(e.right)})").replaceAll("spark_catalog.default.t.", "")
Review comment:
Seems we only need to call `.replaceAll("spark_catalog.default.t.", "")` for columns? i.e. the `case attr: AttributeReference =>`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-707380450
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34315/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceStrategy and HiveTableScans
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696530528
BTW, the PR title seems not correct at all? This adds partially pushed partition filters to explain result, not pushdown partition filter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceStrategy and HiveTableScans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696519937
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-703027624
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-711549797
**[Test build #129980 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129980/testReport)** for PR 29831 at commit [`8ac7e5a`](https://github.com/apache/spark/commit/8ac7e5aba2f0e75cb7717eeb561903c9acda9830).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705329527
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34143/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-697069212
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702187042
@maropu @cloud-fan any further comments about this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r494671598
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
##########
@@ -254,22 +254,28 @@ private[hive] trait HiveStrategies {
* Retrieves data using a HiveTableScan. Partition pruning predicates are also detected and
* applied.
*/
- object HiveTableScans extends Strategy {
+ object HiveTableScans extends Strategy with PredicateHelper {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ScanOperation(projectList, predicates, relation: HiveTableRelation) =>
+ case ScanOperation(projectList, filters, relation: HiveTableRelation) =>
// Filter out all predicates that only deal with partition keys, these are given to the
// hive table scan operator to be used for partition pruning.
val partitionKeyIds = AttributeSet(relation.partitionCols)
- val (pruningPredicates, otherPredicates) = predicates.partition { predicate =>
- !predicate.references.isEmpty &&
- predicate.references.subsetOf(partitionKeyIds)
+ val normalized = DataSourceStrategy.normalizeExprs(
+ filters.filter(_.deterministic), relation.output)
+ val partitionKeyFilters = if (relation.partitionCols.isEmpty) {
+ ExpressionSet(Nil)
+ } else {
+ val predicates = ExpressionSet(normalized
+ .flatMap(extractPredicatesWithinOutputSet(_, partitionKeyIds)))
+ logInfo(s"Pruning partitions with: ${filters.mkString(",")}")
+ predicates
}
pruneFilterProject(
projectList,
- otherPredicates,
+ (ExpressionSet(filters) -- partitionKeyFilters).toSeq,
Review comment:
Shall we use normalized filters here?
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
##########
@@ -254,22 +254,28 @@ private[hive] trait HiveStrategies {
* Retrieves data using a HiveTableScan. Partition pruning predicates are also detected and
* applied.
*/
- object HiveTableScans extends Strategy {
+ object HiveTableScans extends Strategy with PredicateHelper {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ScanOperation(projectList, predicates, relation: HiveTableRelation) =>
+ case ScanOperation(projectList, filters, relation: HiveTableRelation) =>
// Filter out all predicates that only deal with partition keys, these are given to the
// hive table scan operator to be used for partition pruning.
val partitionKeyIds = AttributeSet(relation.partitionCols)
- val (pruningPredicates, otherPredicates) = predicates.partition { predicate =>
- !predicate.references.isEmpty &&
- predicate.references.subsetOf(partitionKeyIds)
+ val normalized = DataSourceStrategy.normalizeExprs(
+ filters.filter(_.deterministic), relation.output)
+ val partitionKeyFilters = if (relation.partitionCols.isEmpty) {
+ ExpressionSet(Nil)
+ } else {
+ val predicates = ExpressionSet(normalized
+ .flatMap(extractPredicatesWithinOutputSet(_, partitionKeyIds)))
+ logInfo(s"Pruning partitions with: ${filters.mkString(",")}")
+ predicates
}
pruneFilterProject(
projectList,
- otherPredicates,
+ (ExpressionSet(filters) -- partitionKeyFilters).toSeq,
Review comment:
Hmm, looks should be fine as `ExpressionSet` will canonicalize expressions.
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,39 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
Review comment:
Isn't only one filter `p = '1'`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705849234
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34170/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-698568080
ping @viirya for review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701106800
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-706352566
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34194/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r501432300
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
##########
@@ -254,22 +254,28 @@ private[hive] trait HiveStrategies {
* Retrieves data using a HiveTableScan. Partition pruning predicates are also detected and
* applied.
*/
- object HiveTableScans extends Strategy {
+ object HiveTableScans extends Strategy with PredicateHelper {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ScanOperation(projectList, predicates, relation: HiveTableRelation) =>
+ case ScanOperation(projectList, filters, relation: HiveTableRelation) =>
// Filter out all predicates that only deal with partition keys, these are given to the
// hive table scan operator to be used for partition pruning.
val partitionKeyIds = AttributeSet(relation.partitionCols)
- val (pruningPredicates, otherPredicates) = predicates.partition { predicate =>
- !predicate.references.isEmpty &&
- predicate.references.subsetOf(partitionKeyIds)
+ val normalized = DataSourceStrategy.normalizeExprs(
+ filters.filter(_.deterministic), relation.output)
+ val partitionKeyFilters = if (relation.partitionCols.isEmpty) {
+ ExpressionSet(Nil)
+ } else {
+ val predicates = ExpressionSet(normalized
+ .flatMap(extractPredicatesWithinOutputSet(_, partitionKeyIds)))
+ logInfo(s"Pruning partitions with: ${filters.mkString(",")}")
+ predicates
}
pruneFilterProject(
projectList,
- otherPredicates,
+ (ExpressionSet(filters) -- partitionKeyFilters).toSeq,
Review comment:
@cloud-fan sorry, missed this, you're right, just changed (but it shouldn't impact correctness , as the resulting expression set will contain non-partition key filters)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-697068487
**[Test build #128993 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128993/testReport)** for PR 29831 at commit [`428fc64`](https://github.com/apache/spark/commit/428fc64b950c856f62994e00e395135cafa43653).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696968005
**[Test build #128993 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128993/testReport)** for PR 29831 at commit [`428fc64`](https://github.com/apache/spark/commit/428fc64b950c856f62994e00e395135cafa43653).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liancheng commented on a change in pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
liancheng commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r492476018
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,42 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, expectedPushedDownFilterCount = 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
assertPrunedPartitions(
"""
|SELECT i, COUNT(1) FROM (
|SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)
|) tmp GROUP BY i
- """.stripMargin, 2)
+ """.stripMargin, 2, 1)
}
}
}
- protected def assertPrunedPartitions(query: String, expected: Long): Unit = {
- val plan = sql(query).queryExecution.sparkPlan
- assert(getScanExecPartitionSize(plan) == expected)
+ protected def assertPrunedPartitions(
+ query: String,
+ expectedPartitionCount: Long,
+ expectedPushedDownFilterCount: Int): Unit = {
+ val qe = sql(query).queryExecution
+ val plan = qe.sparkPlan
+ assert(getScanExecPartitionSize(plan) == expectedPartitionCount)
+ val pushedDownPartitionFilters = qe.executedPlan.collectFirst {
+ case FileSourceScanExec(_, _, _, partitionFilters, _, _, _, _) =>
+ partitionFilters
+ case HiveTableScanExec(_, _, partitionFilters) =>
+ partitionFilters
Review comment:
```suggestion
case scan: FileSourceScanExec => scan.partitionFilters
case scan: HiveTableScanExec => scan.partitionFilters
```
I'd prefer the above style for the following reasons:
1. It's more resilient to future physical plan changes, e.g., adding, removing, or reordering fields.
2. Shorter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-711594964
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34588/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceStrategy and HiveTableScans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696547128
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128962/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r502391339
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
##########
@@ -154,15 +154,8 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
l.resolve(
fsRelation.partitionSchema, fsRelation.sparkSession.sessionState.analyzer.resolver)
val partitionSet = AttributeSet(partitionColumns)
- val partitionKeyFilters = if (partitionColumns.isEmpty) {
- ExpressionSet(Nil)
- } else {
- val predicates = ExpressionSet(normalizedFilters
- .filter(_.references.subsetOf(partitionSet)))
- logInfo(s"Pruning directories with: ${predicates.mkString(",")}")
- predicates
- }
-
+ val partitionKeyFilters = DataSourceStrategy.getPushedDownFilters(partitionColumns,
Review comment:
Nit: It would be better to add a comment saying that the `partitionKeyFilters` here is the same as the filters used in the partition pruning rule `PruneFileSourcePartitions`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-706352554
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696968032
@viirya updated the diff, could you take another look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on pull request #29831: [SPARK-32351][SQL] pushdown filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
CodingCat commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696505529
@wangyum @viirya @gengliangwang would you mind taking a look at this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696514295
**[Test build #128964 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128964/testReport)** for PR 29831 at commit [`3690a79`](https://github.com/apache/spark/commit/3690a79d7c5b8d79d2fab104587b6d0b3a165e18).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-711616660
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701061097
**[Test build #129259 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129259/testReport)** for PR 29831 at commit [`51ad7b9`](https://github.com/apache/spark/commit/51ad7b9c20793c6e6f593917eab93d626a804753).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-712280654
**[Test build #130009 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130009/testReport)** for PR 29831 at commit [`8ac7e5a`](https://github.com/apache/spark/commit/8ac7e5aba2f0e75cb7717eeb561903c9acda9830).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-712280654
**[Test build #130009 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130009/testReport)** for PR 29831 at commit [`8ac7e5a`](https://github.com/apache/spark/commit/8ac7e5aba2f0e75cb7717eeb561903c9acda9830).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-699265145
cc @maropu too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-700495802
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r495660147
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
##########
@@ -254,22 +254,28 @@ private[hive] trait HiveStrategies {
* Retrieves data using a HiveTableScan. Partition pruning predicates are also detected and
* applied.
*/
- object HiveTableScans extends Strategy {
+ object HiveTableScans extends Strategy with PredicateHelper {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ScanOperation(projectList, predicates, relation: HiveTableRelation) =>
+ case ScanOperation(projectList, filters, relation: HiveTableRelation) =>
// Filter out all predicates that only deal with partition keys, these are given to the
// hive table scan operator to be used for partition pruning.
val partitionKeyIds = AttributeSet(relation.partitionCols)
- val (pruningPredicates, otherPredicates) = predicates.partition { predicate =>
- !predicate.references.isEmpty &&
- predicate.references.subsetOf(partitionKeyIds)
+ val normalized = DataSourceStrategy.normalizeExprs(
+ filters.filter(_.deterministic), relation.output)
+ val partitionKeyFilters = if (relation.partitionCols.isEmpty) {
Review comment:
btw, this code part looks the same with `FileSourceStrategy`, so could we share the code between them? (for preventing us from updating the one side and forgetting to update the other side)
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
##########
@@ -254,22 +254,28 @@ private[hive] trait HiveStrategies {
* Retrieves data using a HiveTableScan. Partition pruning predicates are also detected and
* applied.
*/
- object HiveTableScans extends Strategy {
+ object HiveTableScans extends Strategy with PredicateHelper {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ScanOperation(projectList, predicates, relation: HiveTableRelation) =>
+ case ScanOperation(projectList, filters, relation: HiveTableRelation) =>
// Filter out all predicates that only deal with partition keys, these are given to the
// hive table scan operator to be used for partition pruning.
val partitionKeyIds = AttributeSet(relation.partitionCols)
- val (pruningPredicates, otherPredicates) = predicates.partition { predicate =>
- !predicate.references.isEmpty &&
- predicate.references.subsetOf(partitionKeyIds)
+ val normalized = DataSourceStrategy.normalizeExprs(
Review comment:
nit: `normalized` -> `normalizedFilters`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696514295
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-706352554
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r503124192
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
##########
@@ -256,20 +256,21 @@ private[hive] trait HiveStrategies {
*/
object HiveTableScans extends Strategy {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ScanOperation(projectList, predicates, relation: HiveTableRelation) =>
+ case ScanOperation(projectList, filters, relation: HiveTableRelation) =>
// Filter out all predicates that only deal with partition keys, these are given to the
// hive table scan operator to be used for partition pruning.
val partitionKeyIds = AttributeSet(relation.partitionCols)
- val (pruningPredicates, otherPredicates) = predicates.partition { predicate =>
- !predicate.references.isEmpty &&
- predicate.references.subsetOf(partitionKeyIds)
- }
+ val normalizedFilters = DataSourceStrategy.normalizeExprs(
+ filters.filter(_.deterministic), relation.output)
Review comment:
`PruneFileSourcePartitions` is still in the logical phase, and subquery is not planned yet, so we must filter subquery out as we can't execute it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705312425
**[Test build #129537 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129537/testReport)** for PR 29831 at commit [`3cec094`](https://github.com/apache/spark/commit/3cec094fd2c9dae15cc15c10d9219c98ec5c651b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-700495337
**[Test build #129213 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129213/testReport)** for PR 29831 at commit [`8862aa0`](https://github.com/apache/spark/commit/8862aa034274f6c81fcaecfea7b54a74d795e46f).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class InstallCommand(install):`
* `case class RefreshTable(child: LogicalPlan) extends Command `
* `case class RefreshTableCommand(tableIdent: TableIdentifier)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r496432526
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,39 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
assertPrunedPartitions(
"""
|SELECT i, COUNT(1) FROM (
|SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)
|) tmp GROUP BY i
- """.stripMargin, 2)
+ """.stripMargin, 2, 1)
}
}
}
- protected def assertPrunedPartitions(query: String, expected: Long): Unit = {
- val plan = sql(query).queryExecution.sparkPlan
- assert(getScanExecPartitionSize(plan) == expected)
+ protected def assertPrunedPartitions(
+ query: String,
+ expectedPartitionCount: Long,
+ expectedPushedDownFilterCount: Int): Unit = {
Review comment:
Ah, one more comment; could you check filter expressions instead of #filters?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-711715037
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129980/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705377324
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #29831:
URL: https://github.com/apache/spark/pull/29831
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702980304
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33977/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-712304113
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34616/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701106805
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33877/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-708865103
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-712319787
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34616/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-700861448
> How does this PR change the explain result?
it shows in the description part where it shows the pushed down partition filter in scanning operation
e.g. for `"SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)"`
```
FileScan parquet default.t[i#20,p#21] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/nanzhu/code/spark/sql/hive/target/tmp/hive_execution_test_group/war..., PartitionFilters: [],
```
v.s.
```
FileScan parquet default.t[i#20,p#21] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/nanzhu/code/spark/sql/hive/target/tmp/hive_execution_test_group/war..., PartitionFilters: [((p#21 = 1) OR (p#21 = 2))]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702967806
Jenkins, retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-700470251
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33828/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701061806
**[Test build #129259 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129259/testReport)** for PR 29831 at commit [`51ad7b9`](https://github.com/apache/spark/commit/51ad7b9c20793c6e6f593917eab93d626a804753).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-700470674
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-711549797
**[Test build #129980 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129980/testReport)** for PR 29831 at commit [`8ac7e5a`](https://github.com/apache/spark/commit/8ac7e5aba2f0e75cb7717eeb561903c9acda9830).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-711713834
**[Test build #129980 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129980/testReport)** for PR 29831 at commit [`8ac7e5a`](https://github.com/apache/spark/commit/8ac7e5aba2f0e75cb7717eeb561903c9acda9830).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-706152111
> FileSourceScanExec uses the partition filters when listing files, but the filters do nothing because unnecessary partitions are already pruned in advance, so the filters are mainly used for explain output in this case
Actually, the partition filters can contain dynamic partition pruning filters. So the description is wrong here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701500142
> Probably, you'd be better to describe a bit more in the PR description;
> example) currently, actual partition pruning is executed in the optimizer phase (`PruneFileSourcePartitions`) if an input relation has a catalog file index. The current code assumes the same partition filters are generated again in `FileSourceStrategy` and passed into `FileSourceScanExec`. `FileSourceScanExec` uses the partition filters when listing files, but [the filters do nothing](https://github.com/apache/spark/blob/cc06266ade5a4eb35089501a3b32736624208d4c/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L211-L213) because unnecessary partitions are already pruned in advance, so the filters are mainly used for explain output in this case. If a `WEHRE` clause has DNF-ed predicates, `FileSourceStrategy` cannot extract the same filters with `PruneFileSourcePartitions` and then `PartitionFilters` is not shown in explain output. In this PR, brabrabra....
sure, added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702557024
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705096317
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-706334081
> > FileSourceScanExec uses the partition filters when listing files, but the filters do nothing because unnecessary partitions are already pruned in advance, so the filters are mainly used for explain output in this case
>
> Actually, the partition filters can contain dynamic partition pruning filters. So the description is wrong here.
sure, updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705376019
**[Test build #129537 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129537/testReport)** for PR 29831 at commit [`3cec094`](https://github.com/apache/spark/commit/3cec094fd2c9dae15cc15c10d9219c98ec5c651b).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-706324687
**[Test build #129591 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129591/testReport)** for PR 29831 at commit [`d993c2d`](https://github.com/apache/spark/commit/d993c2df7274ba069bdb9fddaa3d39e6479f859f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r502710169
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -467,13 +468,13 @@ case class DataSourceStrategy(conf: SQLConf) extends Strategy with Logging with
}
}
-object DataSourceStrategy {
+object DataSourceStrategy extends PredicateHelper {
/**
* The attribute name may differ from the one in the schema if the query analyzer
* is case insensitive. We should change attribute names to match the ones in the schema,
* so we do not need to worry about case sensitivity anymore.
*/
- protected[sql] def normalizeExprs(
Review comment:
not sure if this is needed given `HiveStrategies.scala` is also within the `sql` scope?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liancheng commented on a change in pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
liancheng commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r492476525
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,42 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
Review comment:
Nit: Extra space.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-700495807
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129213/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-698571139
Yeah, I will look at this again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on pull request #29831: [SPARK-32351][SQL] pushdown filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
CodingCat commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696505529
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702870568
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33968/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702841009
**[Test build #129358 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129358/testReport)** for PR 29831 at commit [`15ec353`](https://github.com/apache/spark/commit/15ec3534e345631fd775d5679507e651291e0552).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705312425
**[Test build #129537 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129537/testReport)** for PR 29831 at commit [`3cec094`](https://github.com/apache/spark/commit/3cec094fd2c9dae15cc15c10d9219c98ec5c651b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r503020929
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -467,13 +468,13 @@ case class DataSourceStrategy(conf: SQLConf) extends Strategy with Logging with
}
}
-object DataSourceStrategy {
+object DataSourceStrategy extends PredicateHelper {
/**
* The attribute name may differ from the one in the schema if the query analyzer
* is case insensitive. We should change attribute names to match the ones in the schema,
* so we do not need to worry about case sensitivity anymore.
*/
- protected[sql] def normalizeExprs(
Review comment:
Ah thanks @HyukjinKwon !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-703027624
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liancheng commented on a change in pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
liancheng commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r492476018
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,42 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, expectedPushedDownFilterCount = 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
assertPrunedPartitions(
"""
|SELECT i, COUNT(1) FROM (
|SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)
|) tmp GROUP BY i
- """.stripMargin, 2)
+ """.stripMargin, 2, 1)
}
}
}
- protected def assertPrunedPartitions(query: String, expected: Long): Unit = {
- val plan = sql(query).queryExecution.sparkPlan
- assert(getScanExecPartitionSize(plan) == expected)
+ protected def assertPrunedPartitions(
+ query: String,
+ expectedPartitionCount: Long,
+ expectedPushedDownFilterCount: Int): Unit = {
+ val qe = sql(query).queryExecution
+ val plan = qe.sparkPlan
+ assert(getScanExecPartitionSize(plan) == expectedPartitionCount)
+ val pushedDownPartitionFilters = qe.executedPlan.collectFirst {
+ case FileSourceScanExec(_, _, _, partitionFilters, _, _, _, _) =>
+ partitionFilters
+ case HiveTableScanExec(_, _, partitionFilters) =>
+ partitionFilters
Review comment:
```suggestion
case scan: FileSourceScanExec => scan.partitionFilters
case scan: HiveTableScanExec => scan.partitionFilters
```
I'd prefer the above style for the following reasons:
1. It's more resilient to future physical plan changes, e.g., adding or removing fields.
2. Shorter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-707388120
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705086744
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34127/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705053052
**[Test build #129522 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129522/testReport)** for PR 29831 at commit [`e3262e3`](https://github.com/apache/spark/commit/e3262e349adf96adc2f20e464b80730d43ea93e2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-703621335
I've checked this again and I have no more comment now, so please wait for the other reviewers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r498578587
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,39 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
assertPrunedPartitions(
"""
|SELECT i, COUNT(1) FROM (
|SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)
|) tmp GROUP BY i
- """.stripMargin, 2)
+ """.stripMargin, 2, 1)
}
}
}
- protected def assertPrunedPartitions(query: String, expected: Long): Unit = {
- val plan = sql(query).queryExecution.sparkPlan
- assert(getScanExecPartitionSize(plan) == expected)
+ protected def assertPrunedPartitions(
+ query: String,
+ expectedPartitionCount: Long,
+ expectedPushedDownFilterCount: Int): Unit = {
Review comment:
hm, but I thought the current test cannot check if expected partition filters are generated in FileSourceStrategy if a filter count is the same with incorrect filters? Do I miss something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-700901174
The thing I don't get is: this PR changes the planning logic and why it only affects explain? Seems it changes the final execution plan as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-708750145
**[Test build #129777 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129777/testReport)** for PR 29831 at commit [`52ad2ea`](https://github.com/apache/spark/commit/52ad2eae736f2d188db93caa152d155ba42d7faf).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-707366373
**[Test build #129709 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129709/testReport)** for PR 29831 at commit [`0058a4e`](https://github.com/apache/spark/commit/0058a4eabbe87c85734c0c171afd245d41d97e8d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701103067
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696514295
**[Test build #128964 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128964/testReport)** for PR 29831 at commit [`3690a79`](https://github.com/apache/spark/commit/3690a79d7c5b8d79d2fab104587b6d0b3a165e18).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696968587
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702986590
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696506485
**[Test build #128962 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128962/testReport)** for PR 29831 at commit [`9938b4e`](https://github.com/apache/spark/commit/9938b4e29f967e87f41faba7ab9c89153a6cb8fa).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702862685
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33968/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-708911720
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-706423696
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on a change in pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r494734806
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,39 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
Review comment:
oh, I also noticed this one, when it's regular partition filter, Spark will impose a partitioncol not null (seems at least since spark 2.4, it works like this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702986590
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705053052
**[Test build #129522 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129522/testReport)** for PR 29831 at commit [`e3262e3`](https://github.com/apache/spark/commit/e3262e349adf96adc2f20e464b80730d43ea93e2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-708750145
**[Test build #129777 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129777/testReport)** for PR 29831 at commit [`52ad2ea`](https://github.com/apache/spark/commit/52ad2eae736f2d188db93caa152d155ba42d7faf).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705855978
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705855959
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34170/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-707366373
**[Test build #129709 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129709/testReport)** for PR 29831 at commit [`0058a4e`](https://github.com/apache/spark/commit/0058a4eabbe87c85734c0c171afd245d41d97e8d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceStrategy and HiveTableScans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696546982
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701063643
**[Test build #129260 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129260/testReport)** for PR 29831 at commit [`55672e4`](https://github.com/apache/spark/commit/55672e4cff1580a096d4c2f9b633a6edfc933185).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702535182
**[Test build #129336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129336/testReport)** for PR 29831 at commit [`15ec353`](https://github.com/apache/spark/commit/15ec3534e345631fd775d5679507e651291e0552).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702557024
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r504397253
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +47,59 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2,
+ Set("`p`='1'", "`p`='2'"))
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4,
+ Set())
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2,
+ Set("`p`='1'", "`p`='3'"))
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3,
+ Set("`p`='1'", "`p`='2'", "`p`='3'"))
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4,
+ Set())
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1,
+ Set("`p`='1'"))
assertPrunedPartitions(
"""
|SELECT i, COUNT(1) FROM (
|SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)
|) tmp GROUP BY i
- """.stripMargin, 2)
+ """.stripMargin, 2, Set("`p`='1'", "`p`='2'"))
}
}
}
- protected def assertPrunedPartitions(query: String, expected: Long): Unit = {
- val plan = sql(query).queryExecution.sparkPlan
- assert(getScanExecPartitionSize(plan) == expected)
+ private def collectEqualExp(
+ expression: Expression,
+ currentSet: Set[String]): Set[String] = expression match {
+ case e: EqualTo =>
+ currentSet + (e.left.sql + e.symbol + e.right.sql)
+ .replaceAll("spark_catalog.default.t.", "")
+ case _ =>
+ expression.children.foldLeft(currentSet)((s, exp) => collectEqualExp(exp, s))
Review comment:
can we put `and`, `or` in the result? I think the test should check `and/or` as well, not just the leaf predicates.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-712434212
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r498578587
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,39 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
assertPrunedPartitions(
"""
|SELECT i, COUNT(1) FROM (
|SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)
|) tmp GROUP BY i
- """.stripMargin, 2)
+ """.stripMargin, 2, 1)
}
}
}
- protected def assertPrunedPartitions(query: String, expected: Long): Unit = {
- val plan = sql(query).queryExecution.sparkPlan
- assert(getScanExecPartitionSize(plan) == expected)
+ protected def assertPrunedPartitions(
+ query: String,
+ expectedPartitionCount: Long,
+ expectedPushedDownFilterCount: Int): Unit = {
Review comment:
hm, but I thought the current test cannot check if expected partition filters are generated in FileSourceStrategy if the number of filters happens to be the same between correct/incorrect filters? Do I miss something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702557030
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33949/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701061825
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129259/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceStrategy and HiveTableScans
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r492497787
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
##########
@@ -158,6 +158,7 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
ExpressionSet(Nil)
} else {
val predicates = ExpressionSet(normalizedFilters
+ .flatMap(extractPredicatesWithinOutputSet(_, partitionSet))
.filter(_.references.subsetOf(partitionSet)))
Review comment:
After `extractPredicatesWithinOutputSet`, do we still need `.filter(_.references.subsetOf(partitionSet)))`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r502716842
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -467,13 +468,13 @@ case class DataSourceStrategy(conf: SQLConf) extends Strategy with Logging with
}
}
-object DataSourceStrategy {
+object DataSourceStrategy extends PredicateHelper {
/**
* The attribute name may differ from the one in the schema if the query analyzer
* is case insensitive. We should change attribute names to match the ones in the schema,
* so we do not need to worry about case sensitivity anymore.
*/
- protected[sql] def normalizeExprs(
Review comment:
yeah im not sure about the comment - perhaps @HyukjinKwon meant to replace `private[sql]` with `private` (which won't work since this is used by Hive. Otherwise it increases the scope to public which I'm not sure is a good thing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702565016
**[Test build #129336 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129336/testReport)** for PR 29831 at commit [`15ec353`](https://github.com/apache/spark/commit/15ec3534e345631fd775d5679507e651291e0552).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705830748
**[Test build #129564 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129564/testReport)** for PR 29831 at commit [`3cec094`](https://github.com/apache/spark/commit/3cec094fd2c9dae15cc15c10d9219c98ec5c651b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r502714748
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -467,13 +468,13 @@ case class DataSourceStrategy(conf: SQLConf) extends Strategy with Logging with
}
}
-object DataSourceStrategy {
+object DataSourceStrategy extends PredicateHelper {
/**
* The attribute name may differ from the one in the schema if the query analyzer
* is case insensitive. We should change attribute names to match the ones in the schema,
* so we do not need to worry about case sensitivity anymore.
*/
- protected[sql] def normalizeExprs(
Review comment:
ah, this is to follow https://github.com/apache/spark/pull/29831#discussion_r499528767 as we are in execution package, we could eliminate these per-method package-level constraint?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705323084
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34143/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701103061
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33876/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r504397419
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
##########
@@ -256,20 +256,21 @@ private[hive] trait HiveStrategies {
*/
object HiveTableScans extends Strategy {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ScanOperation(projectList, predicates, relation: HiveTableRelation) =>
+ case ScanOperation(projectList, filters, relation: HiveTableRelation) =>
// Filter out all predicates that only deal with partition keys, these are given to the
// hive table scan operator to be used for partition pruning.
val partitionKeyIds = AttributeSet(relation.partitionCols)
- val (pruningPredicates, otherPredicates) = predicates.partition { predicate =>
- !predicate.references.isEmpty &&
- predicate.references.subsetOf(partitionKeyIds)
- }
+ val normalizedFilters = DataSourceStrategy.normalizeExprs(
+ filters.filter(_.deterministic), relation.output)
+
+ val partitionKeyFilters = DataSourceStrategy.getPushedDownFilters(relation.partitionCols,
+ normalizedFilters)
pruneFilterProject(
projectList,
- otherPredicates,
+ filters.filter(f => f.references.isEmpty || !f.references.subsetOf(partitionKeyIds)),
Review comment:
It's OK for now to be the same as before, but it's better to make it consistent with `FileSourceStrategy`. We can do it in followup.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r504307502
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
##########
@@ -256,20 +256,21 @@ private[hive] trait HiveStrategies {
*/
object HiveTableScans extends Strategy {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ScanOperation(projectList, predicates, relation: HiveTableRelation) =>
+ case ScanOperation(projectList, filters, relation: HiveTableRelation) =>
// Filter out all predicates that only deal with partition keys, these are given to the
// hive table scan operator to be used for partition pruning.
val partitionKeyIds = AttributeSet(relation.partitionCols)
- val (pruningPredicates, otherPredicates) = predicates.partition { predicate =>
- !predicate.references.isEmpty &&
- predicate.references.subsetOf(partitionKeyIds)
- }
+ val normalizedFilters = DataSourceStrategy.normalizeExprs(
+ filters.filter(_.deterministic), relation.output)
+
+ val partitionKeyFilters = DataSourceStrategy.getPushedDownFilters(relation.partitionCols,
+ normalizedFilters)
pruneFilterProject(
projectList,
- otherPredicates,
+ filters.filter(f => f.references.isEmpty || !f.references.subsetOf(partitionKeyIds)),
Review comment:
I am actually just following the original behavior of HiveStrategies,
regarding scalar subquery, is it in partition filter , I missed something in `FileSourceStrategy `?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-700425678
**[Test build #129213 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129213/testReport)** for PR 29831 at commit [`8862aa0`](https://github.com/apache/spark/commit/8862aa034274f6c81fcaecfea7b54a74d795e46f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696514552
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705323084
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r494671976
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
##########
@@ -254,22 +254,28 @@ private[hive] trait HiveStrategies {
* Retrieves data using a HiveTableScan. Partition pruning predicates are also detected and
* applied.
*/
- object HiveTableScans extends Strategy {
+ object HiveTableScans extends Strategy with PredicateHelper {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ScanOperation(projectList, predicates, relation: HiveTableRelation) =>
+ case ScanOperation(projectList, filters, relation: HiveTableRelation) =>
// Filter out all predicates that only deal with partition keys, these are given to the
// hive table scan operator to be used for partition pruning.
val partitionKeyIds = AttributeSet(relation.partitionCols)
- val (pruningPredicates, otherPredicates) = predicates.partition { predicate =>
- !predicate.references.isEmpty &&
- predicate.references.subsetOf(partitionKeyIds)
+ val normalized = DataSourceStrategy.normalizeExprs(
+ filters.filter(_.deterministic), relation.output)
+ val partitionKeyFilters = if (relation.partitionCols.isEmpty) {
+ ExpressionSet(Nil)
+ } else {
+ val predicates = ExpressionSet(normalized
+ .flatMap(extractPredicatesWithinOutputSet(_, partitionKeyIds)))
+ logInfo(s"Pruning partitions with: ${filters.mkString(",")}")
+ predicates
}
pruneFilterProject(
projectList,
- otherPredicates,
+ (ExpressionSet(filters) -- partitionKeyFilters).toSeq,
Review comment:
Hmm, looks should be fine as `ExpressionSet` will canonicalize expressions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-700495802
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu edited a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
maropu edited a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701197040
Probably, you'd be better to describe a bit more in the PR description;
example) currently, actual partition pruning is executed in the optimizer phase (`PruneFileSourcePartitions`) if an input relation has a catalog file index. The current code assumes the same partition filters are generated again in `FileSourceStrategy` and passed into `FileSourceScanExec`. `FileSourceScanExec` uses the partition filters when listing files, but [the filters do nothing](https://github.com/apache/spark/blob/cc06266ade5a4eb35089501a3b32736624208d4c/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L211-L213) because unnecessary partitions are already pruned in advance, so the filters are mainly used for explain output in this case. If a `WEHRE` clause has DNF-ed predicates, `FileSourceStrategy` cannot extract the same filters with `PruneFileSourcePartitions` and then `PartitionFilters` is not shown in explain output. In this PR, brabrabra....
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701150078
**[Test build #129260 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129260/testReport)** for PR 29831 at commit [`55672e4`](https://github.com/apache/spark/commit/55672e4cff1580a096d4c2f9b633a6edfc933185).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701099537
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33877/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r502391339
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
##########
@@ -154,15 +154,8 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
l.resolve(
fsRelation.partitionSchema, fsRelation.sparkSession.sessionState.analyzer.resolver)
val partitionSet = AttributeSet(partitionColumns)
- val partitionKeyFilters = if (partitionColumns.isEmpty) {
- ExpressionSet(Nil)
- } else {
- val predicates = ExpressionSet(normalizedFilters
- .filter(_.references.subsetOf(partitionSet)))
- logInfo(s"Pruning directories with: ${predicates.mkString(",")}")
- predicates
- }
-
+ val partitionKeyFilters = DataSourceStrategy.getPushedDownFilters(partitionColumns,
Review comment:
Nit: It would be better to add a comment saying that the `partitionKeyFilters` here is the same as the filters used in the partition pruning rule `PruneFileSourcePartitions`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-700451882
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33828/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r503801753
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
##########
@@ -256,20 +256,21 @@ private[hive] trait HiveStrategies {
*/
object HiveTableScans extends Strategy {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ScanOperation(projectList, predicates, relation: HiveTableRelation) =>
+ case ScanOperation(projectList, filters, relation: HiveTableRelation) =>
// Filter out all predicates that only deal with partition keys, these are given to the
// hive table scan operator to be used for partition pruning.
val partitionKeyIds = AttributeSet(relation.partitionCols)
- val (pruningPredicates, otherPredicates) = predicates.partition { predicate =>
- !predicate.references.isEmpty &&
- predicate.references.subsetOf(partitionKeyIds)
- }
+ val normalizedFilters = DataSourceStrategy.normalizeExprs(
+ filters.filter(_.deterministic), relation.output)
+
+ val partitionKeyFilters = DataSourceStrategy.getPushedDownFilters(relation.partitionCols,
+ normalizedFilters)
pruneFilterProject(
projectList,
- otherPredicates,
+ filters.filter(f => f.references.isEmpty || !f.references.subsetOf(partitionKeyIds)),
Review comment:
In `FileSourceStrategy`, scalar subquery is put in partition filters, not data filters, shall we follow it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-706423708
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129591/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r498615640
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,39 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
assertPrunedPartitions(
"""
|SELECT i, COUNT(1) FROM (
|SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)
|) tmp GROUP BY i
- """.stripMargin, 2)
+ """.stripMargin, 2, 1)
}
}
}
- protected def assertPrunedPartitions(query: String, expected: Long): Unit = {
- val plan = sql(query).queryExecution.sparkPlan
- assert(getScanExecPartitionSize(plan) == expected)
+ protected def assertPrunedPartitions(
+ query: String,
+ expectedPartitionCount: Long,
+ expectedPushedDownFilterCount: Int): Unit = {
Review comment:
I originally meant that the information like whether partition filters are correctly picked up should have been handled in other unit test, so here only checking the count would be enough
but sure, we can make it more comprehensive, just revised by comparing based on string
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r503009620
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -467,13 +468,13 @@ case class DataSourceStrategy(conf: SQLConf) extends Strategy with Logging with
}
}
-object DataSourceStrategy {
+object DataSourceStrategy extends PredicateHelper {
/**
* The attribute name may differ from the one in the schema if the query analyzer
* is case insensitive. We should change attribute names to match the ones in the schema,
* so we do not need to worry about case sensitivity anymore.
*/
- protected[sql] def normalizeExprs(
Review comment:
Probably I wasn't clear. I meant to just don't use `private[sql]`. since `execution` is already a private package.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-708865095
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34383/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-708911720
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696517035
**[Test build #128964 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128964/testReport)** for PR 29831 at commit [`3690a79`](https://github.com/apache/spark/commit/3690a79d7c5b8d79d2fab104587b6d0b3a165e18).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-712319808
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-706423696
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r502714790
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
##########
@@ -256,20 +256,21 @@ private[hive] trait HiveStrategies {
*/
object HiveTableScans extends Strategy {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ScanOperation(projectList, predicates, relation: HiveTableRelation) =>
+ case ScanOperation(projectList, filters, relation: HiveTableRelation) =>
// Filter out all predicates that only deal with partition keys, these are given to the
// hive table scan operator to be used for partition pruning.
val partitionKeyIds = AttributeSet(relation.partitionCols)
- val (pruningPredicates, otherPredicates) = predicates.partition { predicate =>
- !predicate.references.isEmpty &&
- predicate.references.subsetOf(partitionKeyIds)
- }
+ val normalizedFilters = DataSourceStrategy.normalizeExprs(
+ filters.filter(_.deterministic), relation.output)
Review comment:
Also curious why we don't check if a filter contains subquery expression - we used to check it in the actual partition filtering (e.g., in `PruneFileSourcePartitions` and `PruneHiveTablePartitions`).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705916115
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701061818
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-707449589
**[Test build #129709 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129709/testReport)** for PR 29831 at commit [`0058a4e`](https://github.com/apache/spark/commit/0058a4eabbe87c85734c0c171afd245d41d97e8d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705329539
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701061097
**[Test build #129259 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129259/testReport)** for PR 29831 at commit [`51ad7b9`](https://github.com/apache/spark/commit/51ad7b9c20793c6e6f593917eab93d626a804753).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701062651
> The thing I don't get is: this PR changes the planning logic and why it only affects explain? Seems it changes the final execution plan as well.
@cloud-fan the reason is that in [PruneFileSourcePartitions](https://github.com/apache/spark/blob/d0c83f372b1aa87327eacc16b131c57fccf6c205/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitions.scala#L100), we have build fileIndexer with the pruned partition directories, so the execution will only touch files in interested partitions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705855978
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702567371
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129336/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702970066
**[Test build #129367 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129367/testReport)** for PR 29831 at commit [`15ec353`](https://github.com/apache/spark/commit/15ec3534e345631fd775d5679507e651291e0552).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-712319808
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705916115
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-711715030
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on a change in pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r494734806
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,39 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
Review comment:
oh, I also noticed this one, when it's regular partition filter, Spark will impose a partitioncol not null (seems at least since spark 2.4, it works like this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701150510
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705829750
Jenkins, retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-711715030
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-711616610
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34588/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceStrategy and HiveTableScans
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696519937
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] pushdown filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696505226
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696505226
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701197040
Probably, you'd be better to describe more in the PR description;
example) currently, actual partition pruning is executed in the optimizer phase (`PruneFileSourcePartitions`) if an input relation has a catalog file index. The current code assumes the same partition filters are generated again in `FileSourceStrategy` and passed into `FileSourceScanExec`. `FileSourceScanExec` uses the partition filters when listing files, but [the filters do nothing](https://github.com/apache/spark/blob/cc06266ade5a4eb35089501a3b32736624208d4c/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L211-L213) because unnecessary partitions are already pruned in advance, so the filters are mainly used for explain output in this case. If a `WEHRE` clause has DNF-ed predicates, `FileSourceStrategy` cannot extract the same filters with `PruneFileSourcePartitions` and then `PartitionFilters` is not shown in explain output. In this PR, brabrabra....
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705329539
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705096300
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34127/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-712433122
**[Test build #130009 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130009/testReport)** for PR 29831 at commit [`8ac7e5a`](https://github.com/apache/spark/commit/8ac7e5aba2f0e75cb7717eeb561903c9acda9830).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702872831
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r503800969
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
##########
@@ -256,20 +256,21 @@ private[hive] trait HiveStrategies {
*/
object HiveTableScans extends Strategy {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ScanOperation(projectList, predicates, relation: HiveTableRelation) =>
+ case ScanOperation(projectList, filters, relation: HiveTableRelation) =>
// Filter out all predicates that only deal with partition keys, these are given to the
// hive table scan operator to be used for partition pruning.
val partitionKeyIds = AttributeSet(relation.partitionCols)
- val (pruningPredicates, otherPredicates) = predicates.partition { predicate =>
- !predicate.references.isEmpty &&
- predicate.references.subsetOf(partitionKeyIds)
- }
+ val normalizedFilters = DataSourceStrategy.normalizeExprs(
+ filters.filter(_.deterministic), relation.output)
+
+ val partitionKeyFilters = DataSourceStrategy.getPushedDownFilters(relation.partitionCols,
+ normalizedFilters)
pruneFilterProject(
projectList,
- otherPredicates,
+ filters.filter(f => f.references.isEmpty || !f.references.subsetOf(partitionKeyIds)),
Review comment:
or a scalar subquery?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-700470681
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33828/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liancheng commented on a change in pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
liancheng commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r492476018
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,42 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, expectedPushedDownFilterCount = 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
assertPrunedPartitions(
"""
|SELECT i, COUNT(1) FROM (
|SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)
|) tmp GROUP BY i
- """.stripMargin, 2)
+ """.stripMargin, 2, 1)
}
}
}
- protected def assertPrunedPartitions(query: String, expected: Long): Unit = {
- val plan = sql(query).queryExecution.sparkPlan
- assert(getScanExecPartitionSize(plan) == expected)
+ protected def assertPrunedPartitions(
+ query: String,
+ expectedPartitionCount: Long,
+ expectedPushedDownFilterCount: Int): Unit = {
+ val qe = sql(query).queryExecution
+ val plan = qe.sparkPlan
+ assert(getScanExecPartitionSize(plan) == expectedPartitionCount)
+ val pushedDownPartitionFilters = qe.executedPlan.collectFirst {
+ case FileSourceScanExec(_, _, _, partitionFilters, _, _, _, _) =>
+ partitionFilters
+ case HiveTableScanExec(_, _, partitionFilters) =>
+ partitionFilters
Review comment:
```suggestion
case scan: FileSourceScanExec => scan.partitionFilters
case scan: HiveTableScanExec => scan.partitionFilters
```
I'd prefer the above style for the following reasons:
1. It's more resilient to future physical plan changes, e.g., adding or removing fields.
2. Shorter.
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,42 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, expectedPushedDownFilterCount = 0)
Review comment:
Why only adding the parameter name for this line?
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,42 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
Review comment:
Nit: Extra space.
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,42 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, expectedPushedDownFilterCount = 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
assertPrunedPartitions(
"""
|SELECT i, COUNT(1) FROM (
|SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)
|) tmp GROUP BY i
- """.stripMargin, 2)
+ """.stripMargin, 2, 1)
}
}
}
- protected def assertPrunedPartitions(query: String, expected: Long): Unit = {
- val plan = sql(query).queryExecution.sparkPlan
- assert(getScanExecPartitionSize(plan) == expected)
+ protected def assertPrunedPartitions(
+ query: String,
+ expectedPartitionCount: Long,
+ expectedPushedDownFilterCount: Int): Unit = {
+ val qe = sql(query).queryExecution
+ val plan = qe.sparkPlan
+ assert(getScanExecPartitionSize(plan) == expectedPartitionCount)
+ val pushedDownPartitionFilters = qe.executedPlan.collectFirst {
+ case FileSourceScanExec(_, _, _, partitionFilters, _, _, _, _) =>
+ partitionFilters
+ case HiveTableScanExec(_, _, partitionFilters) =>
+ partitionFilters
+ }
+ assert(pushedDownPartitionFilters.isDefined &&
+ pushedDownPartitionFilters.get.length == expectedPushedDownFilterCount)
Review comment:
How about:
```suggestion
assert(pushedDownPartitionFilters == Some(expectedPushedDownFilterCount))
```
For the current assertion, if it fails, it's hard to tell which condition fails.
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
##########
@@ -254,22 +254,29 @@ private[hive] trait HiveStrategies {
* Retrieves data using a HiveTableScan. Partition pruning predicates are also detected and
* applied.
*/
- object HiveTableScans extends Strategy {
+ object HiveTableScans extends Strategy with PredicateHelper {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ScanOperation(projectList, predicates, relation: HiveTableRelation) =>
+ case ScanOperation(projectList, filters, relation: HiveTableRelation) =>
// Filter out all predicates that only deal with partition keys, these are given to the
// hive table scan operator to be used for partition pruning.
val partitionKeyIds = AttributeSet(relation.partitionCols)
- val (pruningPredicates, otherPredicates) = predicates.partition { predicate =>
- !predicate.references.isEmpty &&
- predicate.references.subsetOf(partitionKeyIds)
+ val normalized = DataSourceStrategy.normalizeExprs(
+ filters.filter(_.deterministic), relation.output)
+ val partitionKeyFilters = if (relation.partitionCols.isEmpty) {
+ ExpressionSet(Nil)
+ } else {
+ val predicates = ExpressionSet(normalized
+ .flatMap(extractPredicatesWithinOutputSet(_, partitionKeyIds)))
+ .filter(_.references.subsetOf(partitionKeyIds))
+ logInfo(s"Pruning directories with: ${filters.mkString(",")}")
Review comment:
```suggestion
logInfo(s"Pruning partitions with: ${filters.mkString(",")}")
```
(Conceptually, partitioning can be implemented using mechanisms other than filesystem directories.)
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,42 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, expectedPushedDownFilterCount = 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
assertPrunedPartitions(
"""
|SELECT i, COUNT(1) FROM (
|SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)
|) tmp GROUP BY i
- """.stripMargin, 2)
+ """.stripMargin, 2, 1)
}
}
}
- protected def assertPrunedPartitions(query: String, expected: Long): Unit = {
- val plan = sql(query).queryExecution.sparkPlan
- assert(getScanExecPartitionSize(plan) == expected)
+ protected def assertPrunedPartitions(
+ query: String,
+ expectedPartitionCount: Long,
+ expectedPushedDownFilterCount: Int): Unit = {
+ val qe = sql(query).queryExecution
+ val plan = qe.sparkPlan
+ assert(getScanExecPartitionSize(plan) == expectedPartitionCount)
+ val pushedDownPartitionFilters = qe.executedPlan.collectFirst {
+ case FileSourceScanExec(_, _, _, partitionFilters, _, _, _, _) =>
+ partitionFilters
+ case HiveTableScanExec(_, _, partitionFilters) =>
+ partitionFilters
Review comment:
```suggestion
case scan: FileSourceScanExec => scan.partitionFilters
case scan: HiveTableScanExec => scan.partitionFilters
```
I'd prefer the above style for the following reasons:
1. It's more resilient to future physical plan changes, e.g., adding, removing, or reordering fields.
2. Shorter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-703027320
**[Test build #129367 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129367/testReport)** for PR 29831 at commit [`15ec353`](https://github.com/apache/spark/commit/15ec3534e345631fd775d5679507e651291e0552).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-708865103
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-697069212
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702567364
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702870597
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-700632500
How does this PR change the explain result?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liancheng commented on a change in pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
liancheng commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r492477024
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,42 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, expectedPushedDownFilterCount = 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
assertPrunedPartitions(
"""
|SELECT i, COUNT(1) FROM (
|SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)
|) tmp GROUP BY i
- """.stripMargin, 2)
+ """.stripMargin, 2, 1)
}
}
}
- protected def assertPrunedPartitions(query: String, expected: Long): Unit = {
- val plan = sql(query).queryExecution.sparkPlan
- assert(getScanExecPartitionSize(plan) == expected)
+ protected def assertPrunedPartitions(
+ query: String,
+ expectedPartitionCount: Long,
+ expectedPushedDownFilterCount: Int): Unit = {
+ val qe = sql(query).queryExecution
+ val plan = qe.sparkPlan
+ assert(getScanExecPartitionSize(plan) == expectedPartitionCount)
+ val pushedDownPartitionFilters = qe.executedPlan.collectFirst {
+ case FileSourceScanExec(_, _, _, partitionFilters, _, _, _, _) =>
+ partitionFilters
+ case HiveTableScanExec(_, _, partitionFilters) =>
+ partitionFilters
+ }
+ assert(pushedDownPartitionFilters.isDefined &&
+ pushedDownPartitionFilters.get.length == expectedPushedDownFilterCount)
Review comment:
How about:
```suggestion
assert(pushedDownPartitionFilters == Some(expectedPushedDownFilterCount))
```
For the current assertion, if it fails, it's hard to tell which condition fails.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r494671598
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
##########
@@ -254,22 +254,28 @@ private[hive] trait HiveStrategies {
* Retrieves data using a HiveTableScan. Partition pruning predicates are also detected and
* applied.
*/
- object HiveTableScans extends Strategy {
+ object HiveTableScans extends Strategy with PredicateHelper {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ScanOperation(projectList, predicates, relation: HiveTableRelation) =>
+ case ScanOperation(projectList, filters, relation: HiveTableRelation) =>
// Filter out all predicates that only deal with partition keys, these are given to the
// hive table scan operator to be used for partition pruning.
val partitionKeyIds = AttributeSet(relation.partitionCols)
- val (pruningPredicates, otherPredicates) = predicates.partition { predicate =>
- !predicate.references.isEmpty &&
- predicate.references.subsetOf(partitionKeyIds)
+ val normalized = DataSourceStrategy.normalizeExprs(
+ filters.filter(_.deterministic), relation.output)
+ val partitionKeyFilters = if (relation.partitionCols.isEmpty) {
+ ExpressionSet(Nil)
+ } else {
+ val predicates = ExpressionSet(normalized
+ .flatMap(extractPredicatesWithinOutputSet(_, partitionKeyIds)))
+ logInfo(s"Pruning partitions with: ${filters.mkString(",")}")
+ predicates
}
pruneFilterProject(
projectList,
- otherPredicates,
+ (ExpressionSet(filters) -- partitionKeyFilters).toSeq,
Review comment:
Shall we use normalized filters here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r494680971
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,39 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
Review comment:
Isn't only one filter `p = '1'`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceStrategy and HiveTableScans
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696529692
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceStrategy and HiveTableScans
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r492497787
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
##########
@@ -158,6 +158,7 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging {
ExpressionSet(Nil)
} else {
val predicates = ExpressionSet(normalizedFilters
+ .flatMap(extractPredicatesWithinOutputSet(_, partitionSet))
.filter(_.references.subsetOf(partitionSet)))
Review comment:
After `extractPredicatesWithinOutputSet`, do we still need `.filter(_.references.subsetOf(partitionSet)))`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702556508
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33949/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-707388120
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-698568080
ping @viirya for review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29831: [SPARK-32351][SQL] show pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r495365478
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
##########
@@ -254,22 +254,28 @@ private[hive] trait HiveStrategies {
* Retrieves data using a HiveTableScan. Partition pruning predicates are also detected and
* applied.
*/
- object HiveTableScans extends Strategy {
+ object HiveTableScans extends Strategy with PredicateHelper {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case ScanOperation(projectList, predicates, relation: HiveTableRelation) =>
+ case ScanOperation(projectList, filters, relation: HiveTableRelation) =>
// Filter out all predicates that only deal with partition keys, these are given to the
// hive table scan operator to be used for partition pruning.
val partitionKeyIds = AttributeSet(relation.partitionCols)
- val (pruningPredicates, otherPredicates) = predicates.partition { predicate =>
- !predicate.references.isEmpty &&
- predicate.references.subsetOf(partitionKeyIds)
+ val normalized = DataSourceStrategy.normalizeExprs(
+ filters.filter(_.deterministic), relation.output)
+ val partitionKeyFilters = if (relation.partitionCols.isEmpty) {
+ ExpressionSet(Nil)
+ } else {
+ val predicates = ExpressionSet(normalized
+ .flatMap(extractPredicatesWithinOutputSet(_, partitionKeyIds)))
+ logInfo(s"Pruning partitions with: ${filters.mkString(",")}")
Review comment:
filters -> predicates?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-706324687
**[Test build #129591 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129591/testReport)** for PR 29831 at commit [`d993c2d`](https://github.com/apache/spark/commit/d993c2df7274ba069bdb9fddaa3d39e6479f859f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceStrategy and HiveTableScans
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696506485
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705915587
**[Test build #129564 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129564/testReport)** for PR 29831 at commit [`3cec094`](https://github.com/apache/spark/commit/3cec094fd2c9dae15cc15c10d9219c98ec5c651b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-707450125
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-705329539
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702986570
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33977/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-707388109
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34315/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on a change in pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
CodingCat commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r497237999
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,39 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
assertPrunedPartitions(
"""
|SELECT i, COUNT(1) FROM (
|SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)
|) tmp GROUP BY i
- """.stripMargin, 2)
+ """.stripMargin, 2, 1)
}
}
}
- protected def assertPrunedPartitions(query: String, expected: Long): Unit = {
- val plan = sql(query).queryExecution.sparkPlan
- assert(getScanExecPartitionSize(plan) == expected)
+ protected def assertPrunedPartitions(
+ query: String,
+ expectedPartitionCount: Long,
+ expectedPushedDownFilterCount: Int): Unit = {
Review comment:
sure, updated @maropu would you please take a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701098598
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33876/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-701150510
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696506485
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-696517046
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-702872831
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] CodingCat commented on a change in pull request #29831: [SPARK-32351][SQL] pushdown partition filter in FileSourceScanExec and HiveTableScanExec
Posted by GitBox <gi...@apache.org>.
CodingCat commented on a change in pull request #29831:
URL: https://github.com/apache/spark/pull/29831#discussion_r492478729
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala
##########
@@ -46,30 +46,42 @@ abstract class PrunePartitionSuiteBase extends QueryTest with SQLTestUtils with
}
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2)
+ "SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (i = 1 OR p = '2')", 4, 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '3' AND i = 3 )", 2, 1)
assertPrunedPartitions(
- "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3)
+ "SELECT * FROM t WHERE (p = '1' AND i = 2) OR (p = '2' OR p = '3')", 3, 1)
assertPrunedPartitions(
- "SELECT * FROM t", 4)
+ "SELECT * FROM t", 4, expectedPushedDownFilterCount = 0)
assertPrunedPartitions(
- "SELECT * FROM t WHERE p = '1' AND i = 2", 1)
+ "SELECT * FROM t WHERE p = '1' AND i = 2", 1, 2)
assertPrunedPartitions(
"""
|SELECT i, COUNT(1) FROM (
|SELECT * FROM t WHERE p = '1' OR (p = '2' AND i = 1)
|) tmp GROUP BY i
- """.stripMargin, 2)
+ """.stripMargin, 2, 1)
}
}
}
- protected def assertPrunedPartitions(query: String, expected: Long): Unit = {
- val plan = sql(query).queryExecution.sparkPlan
- assert(getScanExecPartitionSize(plan) == expected)
+ protected def assertPrunedPartitions(
+ query: String,
+ expectedPartitionCount: Long,
+ expectedPushedDownFilterCount: Int): Unit = {
+ val qe = sql(query).queryExecution
+ val plan = qe.sparkPlan
+ assert(getScanExecPartitionSize(plan) == expectedPartitionCount)
+ val pushedDownPartitionFilters = qe.executedPlan.collectFirst {
+ case FileSourceScanExec(_, _, _, partitionFilters, _, _, _, _) =>
+ partitionFilters
+ case HiveTableScanExec(_, _, partitionFilters) =>
+ partitionFilters
+ }
+ assert(pushedDownPartitionFilters.isDefined &&
+ pushedDownPartitionFilters.get.length == expectedPushedDownFilterCount)
Review comment:
thanks for the suggestion! I agree the suggested way is cleaner
but in the latest scalatest, it actually can tell something like `pushedDownPartitionFilters is defined, but `whatever` has length of 0`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-712277326
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29831: [SPARK-32351][SQL] Show partially pushed down partition filters in explain()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29831:
URL: https://github.com/apache/spark/pull/29831#issuecomment-712434212
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org