You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/24 23:19:52 UTC
[GitHub] [spark] viirya opened a new pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
viirya opened a new pull request #34701:
URL: https://github.com/apache/spark/pull/34701
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This patch proposes an optimization rule to prune unnecessary fields from `Generate` under count-only `Aggregate`.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
As shown in the JIRA, if a query counts nested elements (structs) on an array by exploding the array in `Generate`, because there is no particular nested field is specified, Spark currently reads the full nested struct without any pruning.
An optimization can be made to pick up arbitrary nested field from the struct. So we can prune unnecessary field access and still count the number of array elements.
```
== Optimized Logical Plan ==
Aggregate [count(1) AS count(true)#20299L]
+- Project
+- Generate explode(items#20293), [0], false, [item#20296]
+- Filter ((size(items#20293, true) > 0) AND isnotnull(items#20293))
+- Relation default.table[items#20293] parquet
```
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
Added test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-979048636
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50096/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978728286
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50067/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978837934
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50078/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r760776756
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -2162,6 +2163,48 @@ object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan] {
}
}
+/**
+ * Prunes unnecessary fields from a [[Generate]] if it is under a project which does not refer
+ * any generated attributes, .e.g., count-like aggregation on an exploded array.
+ */
+object GenerateOptimization extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan.transformDownWithPruning(
+ _.containsAllPatterns(PROJECT, GENERATE), ruleId) {
+ case p @ Project(_, g: Generate) if p.references.isEmpty
+ && g.generator.isInstanceOf[ExplodeBase] =>
+ g.generator.children.head.dataType match {
+ case ArrayType(StructType(fields), _) =>
+ val atomicFields = fields.collect {
+ case f: StructField if f.dataType.isInstanceOf[AtomicType] => f
+ }
+ val extractor = if (atomicFields.size > 0) {
+ // Pick an arbitrary atomic field, if any
Review comment:
shall we pick the smallest one? e.g. prefer int over string
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984495537
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978778030
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145598/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978982831
**[Test build #145624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145624/testReport)** for PR 34701 at commit [`f63a8c5`](https://github.com/apache/spark/commit/f63a8c50450961b5f2f9ac9dc345684e728849e4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r760776370
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -2162,6 +2163,48 @@ object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan] {
}
}
+/**
+ * Prunes unnecessary fields from a [[Generate]] if it is under a project which does not refer
+ * any generated attributes, .e.g., count-like aggregation on an exploded array.
+ */
+object GenerateOptimization extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan.transformDownWithPruning(
+ _.containsAllPatterns(PROJECT, GENERATE), ruleId) {
+ case p @ Project(_, g: Generate) if p.references.isEmpty
+ && g.generator.isInstanceOf[ExplodeBase] =>
+ g.generator.children.head.dataType match {
+ case ArrayType(StructType(fields), _) =>
+ val atomicFields = fields.collect {
+ case f: StructField if f.dataType.isInstanceOf[AtomicType] => f
+ }
+ val extractor = if (atomicFields.size > 0) {
+ // Pick an arbitrary atomic field, if any
+ ExtractValue(g.generator.children.head,
Review comment:
nit: I feel it's safer to create `GetStructField` instead of doing name lookup again. It's possible that some dataframe-generated query plan has name conflicts in the struct, and `GetStructField` allows us to put the ordinal directly to avoid a name lookup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984366546
**[Test build #145849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145849/testReport)** for PR 34701 at commit [`e479e5c`](https://github.com/apache/spark/commit/e479e5c2088510f88f62e6baa89ebbd18ff135af).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984433549
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50324/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984497195
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145845/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-981961706
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50199/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-982122967
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145729/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978776783
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50070/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-981927319
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50199/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r758616810
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -2162,6 +2163,74 @@ object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan] {
}
}
+/**
+ * Prunes unnecessary fields from a [[Generate]] if it is under a count aggregation
+ * query.
+ */
+object CountAggregateOptimization extends Rule[LogicalPlan] {
+ private def isLiteralCountAggFun(func: AggregateFunction): Boolean = func match {
+ case c: Count if c.children.size == 1 && c.children(0).isInstanceOf[Literal] => true
+ case _ => false
+ }
+
+ private def isCandidate(agg: Aggregate): Boolean = {
+ if (agg.aggregateExpressions.size == 1) {
+ agg.aggregateExpressions(0) match {
+ case Alias(ae: AggregateExpression, _) if isLiteralCountAggFun(ae.aggregateFunction) =>
+ true
+ case _ =>
+ false
+ }
+ } else {
+ false
+ }
+ }
+
+ private def pruningFields(agg: Aggregate): LogicalPlan = {
+ agg.transformDownWithPruning(_.containsPattern(GENERATE), ruleId) {
+ case p @ Project(_, g: Generate) if p.references.intersect(g.outputSet).isEmpty
Review comment:
For the example query, the optimized query plan has the `Project`. I think it is because column pruning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984282092
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145826/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984529907
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50327/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984279146
**[Test build #145826 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145826/testReport)** for PR 34701 at commit [`1af505a`](https://github.com/apache/spark/commit/1af505ab52ba5d294187760ebc56cc5ac690d757).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984588904
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145849/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984450031
**[Test build #145852 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145852/testReport)** for PR 34701 at commit [`12e6528`](https://github.com/apache/spark/commit/12e65284e45d7e7c05cef881b6da3a154da274d5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978924156
**[Test build #145611 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145611/testReport)** for PR 34701 at commit [`d6f7472`](https://github.com/apache/spark/commit/d6f7472f9d669aa62e2c2d5c345c6671573961a1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-979234567
**[Test build #145624 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145624/testReport)** for PR 34701 at commit [`f63a8c5`](https://github.com/apache/spark/commit/f63a8c50450961b5f2f9ac9dc345684e728849e4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class CountAggregateOptimizationSuite extends PlanTest `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978506113
**[Test build #145595 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145595/testReport)** for PR 34701 at commit [`97c1c36`](https://github.com/apache/spark/commit/97c1c3631b0a92b5a614f75ce4cd55d1a75a5aff).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978731148
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50067/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-979096415
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50096/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984153377
**[Test build #145826 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145826/testReport)** for PR 34701 at commit [`1af505a`](https://github.com/apache/spark/commit/1af505ab52ba5d294187760ebc56cc5ac690d757).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r760827034
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -2162,6 +2163,48 @@ object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan] {
}
}
+/**
+ * Prunes unnecessary fields from a [[Generate]] if it is under a project which does not refer
+ * any generated attributes, .e.g., count-like aggregation on an exploded array.
+ */
+object GenerateOptimization extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan.transformDownWithPruning(
+ _.containsAllPatterns(PROJECT, GENERATE), ruleId) {
+ case p @ Project(_, g: Generate) if p.references.isEmpty
+ && g.generator.isInstanceOf[ExplodeBase] =>
+ g.generator.children.head.dataType match {
+ case ArrayType(StructType(fields), _) =>
+ val atomicFields = fields.collect {
+ case f: StructField if f.dataType.isInstanceOf[AtomicType] => f
+ }
+ val extractor = if (atomicFields.size > 0) {
+ // Pick an arbitrary atomic field, if any
+ ExtractValue(g.generator.children.head,
Review comment:
okay, updated to use `GetArrayStructFields`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r760826896
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -2162,6 +2163,48 @@ object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan] {
}
}
+/**
+ * Prunes unnecessary fields from a [[Generate]] if it is under a project which does not refer
+ * any generated attributes, .e.g., count-like aggregation on an exploded array.
+ */
+object GenerateOptimization extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan.transformDownWithPruning(
+ _.containsAllPatterns(PROJECT, GENERATE), ruleId) {
+ case p @ Project(_, g: Generate) if p.references.isEmpty
+ && g.generator.isInstanceOf[ExplodeBase] =>
+ g.generator.children.head.dataType match {
+ case ArrayType(StructType(fields), _) =>
+ val atomicFields = fields.collect {
+ case f: StructField if f.dataType.isInstanceOf[AtomicType] => f
+ }
+ val extractor = if (atomicFields.size > 0) {
+ // Pick an arbitrary atomic field, if any
Review comment:
good idea. updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984348042
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50320/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r760866369
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaPruningSuite.scala
##########
@@ -882,4 +882,32 @@ abstract class SchemaPruningSuite
Nil)
}
}
+
+ test("SPARK-37450: Prunes unnecessary fields from Explode for count aggregation") {
+ import testImplicits._
+
+ withTempView("table") {
+ withTempPath { dir =>
+ val path = dir.getCanonicalPath
+
+ val jsonStr =
+ """{
+ "items": [
+ {"itemId": 1, "itemData": "a"},
+ {"itemId": 2, "itemData": "b"}
+ ]}""".stripMargin
+ val df = spark.read.json(Seq(jsonStr).toDS)
+ makeDataSourceFile(df, new File(path + "/table"))
Review comment:
can't we use `path` directly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984587483
**[Test build #145849 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145849/testReport)** for PR 34701 at commit [`e479e5c`](https://github.com/apache/spark/commit/e479e5c2088510f88f62e6baa89ebbd18ff135af).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978506113
**[Test build #145595 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145595/testReport)** for PR 34701 at commit [`97c1c36`](https://github.com/apache/spark/commit/97c1c3631b0a92b5a614f75ce4cd55d1a75a5aff).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978982831
**[Test build #145624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145624/testReport)** for PR 34701 at commit [`f63a8c5`](https://github.com/apache/spark/commit/f63a8c50450961b5f2f9ac9dc345684e728849e4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-981896899
**[Test build #145729 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145729/testReport)** for PR 34701 at commit [`f391dc9`](https://github.com/apache/spark/commit/f391dc9aca652ce455d62d5a68380f3477f1c2dc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r760882517
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -2162,6 +2163,42 @@ object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan] {
}
}
+/**
+ * Prunes unnecessary fields from a [[Generate]] if it is under a project which does not refer
+ * any generated attributes, .e.g., count-like aggregation on an exploded array.
+ */
+object GenerateOptimization extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan.transformDownWithPruning(
+ _.containsAllPatterns(PROJECT, GENERATE), ruleId) {
+ case p @ Project(_, g: Generate) if p.references.isEmpty
+ && g.generator.isInstanceOf[ExplodeBase] =>
+ g.generator.children.head.dataType match {
+ case ArrayType(StructType(fields), containsNull) =>
Review comment:
added it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984678635
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145852/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984282092
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145826/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984588904
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145849/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984447353
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50324/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978778030
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145598/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-979085686
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50096/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978837934
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50078/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978836364
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50078/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978732555
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145595/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978779238
**[Test build #145611 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145611/testReport)** for PR 34701 at commit [`d6f7472`](https://github.com/apache/spark/commit/d6f7472f9d669aa62e2c2d5c345c6671573961a1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-979096415
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50096/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984360337
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50320/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984678635
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145852/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya closed pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
viirya closed pull request #34701:
URL: https://github.com/apache/spark/pull/34701
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984307445
**[Test build #145845 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145845/testReport)** for PR 34701 at commit [`099d3a8`](https://github.com/apache/spark/commit/099d3a8919189fa0b6f0d10079e327d114b657b8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r759073428
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -2162,6 +2163,74 @@ object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan] {
}
}
+/**
+ * Prunes unnecessary fields from a [[Generate]] if it is under a count aggregation
+ * query.
+ */
+object CountAggregateOptimization extends Rule[LogicalPlan] {
Review comment:
Do you mean just checking for empty project list instead of references?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r760864036
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -2162,6 +2163,42 @@ object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan] {
}
}
+/**
+ * Prunes unnecessary fields from a [[Generate]] if it is under a project which does not refer
+ * any generated attributes, .e.g., count-like aggregation on an exploded array.
+ */
+object GenerateOptimization extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan.transformDownWithPruning(
+ _.containsAllPatterns(PROJECT, GENERATE), ruleId) {
+ case p @ Project(_, g: Generate) if p.references.isEmpty
+ && g.generator.isInstanceOf[ExplodeBase] =>
+ g.generator.children.head.dataType match {
+ case ArrayType(StructType(fields), containsNull) =>
Review comment:
```suggestion
case ArrayType(StructType(fields), containsNull) if fields.length > 1 =>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984541610
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50327/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984173819
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50301/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984447353
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50324/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984153377
**[Test build #145826 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145826/testReport)** for PR 34701 at commit [`1af505a`](https://github.com/apache/spark/commit/1af505ab52ba5d294187760ebc56cc5ac690d757).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r760764995
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -2162,6 +2163,74 @@ object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan] {
}
}
+/**
+ * Prunes unnecessary fields from a [[Generate]] if it is under a count aggregation
+ * query.
+ */
+object CountAggregateOptimization extends Rule[LogicalPlan] {
+ private def isLiteralCountAggFun(func: AggregateFunction): Boolean = func match {
+ case c: Count if c.children.size == 1 && c.children(0).isInstanceOf[Literal] => true
+ case _ => false
+ }
+
+ private def isCandidate(agg: Aggregate): Boolean = {
+ if (agg.aggregateExpressions.size == 1) {
+ agg.aggregateExpressions(0) match {
+ case Alias(ae: AggregateExpression, _) if isLiteralCountAggFun(ae.aggregateFunction) =>
+ true
+ case _ =>
+ false
+ }
+ } else {
+ false
+ }
+ }
+
+ private def pruningFields(agg: Aggregate): LogicalPlan = {
+ agg.transformDownWithPruning(_.containsPattern(GENERATE), ruleId) {
+ case p @ Project(_, g: Generate) if p.references.intersect(g.outputSet).isEmpty
Review comment:
Updated. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r760776756
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -2162,6 +2163,48 @@ object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan] {
}
}
+/**
+ * Prunes unnecessary fields from a [[Generate]] if it is under a project which does not refer
+ * any generated attributes, .e.g., count-like aggregation on an exploded array.
+ */
+object GenerateOptimization extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan.transformDownWithPruning(
+ _.containsAllPatterns(PROJECT, GENERATE), ruleId) {
+ case p @ Project(_, g: Generate) if p.references.isEmpty
+ && g.generator.isInstanceOf[ExplodeBase] =>
+ g.generator.children.head.dataType match {
+ case ArrayType(StructType(fields), _) =>
+ val atomicFields = fields.collect {
+ case f: StructField if f.dataType.isInstanceOf[AtomicType] => f
+ }
+ val extractor = if (atomicFields.size > 0) {
+ // Pick an arbitrary atomic field, if any
Review comment:
shall we pick the smaller list one? e.g. prefer int over string
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r760865681
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaPruningSuite.scala
##########
@@ -882,4 +882,32 @@ abstract class SchemaPruningSuite
Nil)
}
}
+
+ test("SPARK-37450: Prunes unnecessary fields from Explode for count aggregation") {
+ import testImplicits._
+
+ withTempView("table") {
+ withTempPath { dir =>
+ val path = dir.getCanonicalPath
+
+ val jsonStr =
+ """{
Review comment:
nit: can we use the multi-line format?
```
"""
|xxx
|""".stripMargin
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984541610
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50327/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-979240610
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145624/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978728223
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50070/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-979410522
cc @cloud-fan @maropu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978777890
**[Test build #145598 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145598/testReport)** for PR 34701 at commit [`bab11cd`](https://github.com/apache/spark/commit/bab11cd6cd1ee9d882854c7a511a0eede3c4aa3d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978805529
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50078/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984202528
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50301/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r760752530
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -2162,6 +2163,74 @@ object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan] {
}
}
+/**
+ * Prunes unnecessary fields from a [[Generate]] if it is under a count aggregation
+ * query.
+ */
+object CountAggregateOptimization extends Rule[LogicalPlan] {
+ private def isLiteralCountAggFun(func: AggregateFunction): Boolean = func match {
+ case c: Count if c.children.size == 1 && c.children(0).isInstanceOf[Literal] => true
+ case _ => false
+ }
+
+ private def isCandidate(agg: Aggregate): Boolean = {
+ if (agg.aggregateExpressions.size == 1) {
+ agg.aggregateExpressions(0) match {
+ case Alias(ae: AggregateExpression, _) if isLiteralCountAggFun(ae.aggregateFunction) =>
+ true
+ case _ =>
+ false
+ }
+ } else {
+ false
+ }
+ }
+
+ private def pruningFields(agg: Aggregate): LogicalPlan = {
+ agg.transformDownWithPruning(_.containsPattern(GENERATE), ruleId) {
+ case p @ Project(_, g: Generate) if p.references.intersect(g.outputSet).isEmpty
Review comment:
is `p.references.isEmpty` clearer here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r760865681
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaPruningSuite.scala
##########
@@ -882,4 +882,32 @@ abstract class SchemaPruningSuite
Nil)
}
}
+
+ test("SPARK-37450: Prunes unnecessary fields from Explode for count aggregation") {
+ import testImplicits._
+
+ withTempView("table") {
+ withTempPath { dir =>
+ val path = dir.getCanonicalPath
+
+ val jsonStr =
+ """{
Review comment:
nit: can we use the multi-line format?
```
"""
|xxx
|""".stripMargin
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984423166
Thank you @cloud-fan !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978776783
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50070/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978642798
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50067/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978779238
**[Test build #145611 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145611/testReport)** for PR 34701 at commit [`d6f7472`](https://github.com/apache/spark/commit/d6f7472f9d669aa62e2c2d5c345c6671573961a1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978925620
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145611/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-981987228
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50199/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r758294477
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -2162,6 +2163,74 @@ object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan] {
}
}
+/**
+ * Prunes unnecessary fields from a [[Generate]] if it is under a count aggregation
+ * query.
+ */
+object CountAggregateOptimization extends Rule[LogicalPlan] {
+ private def isLiteralCountAggFun(func: AggregateFunction): Boolean = func match {
+ case c: Count if c.children.size == 1 && c.children(0).isInstanceOf[Literal] => true
+ case _ => false
+ }
+
+ private def isCandidate(agg: Aggregate): Boolean = {
+ if (agg.aggregateExpressions.size == 1) {
+ agg.aggregateExpressions(0) match {
+ case Alias(ae: AggregateExpression, _) if isLiteralCountAggFun(ae.aggregateFunction) =>
+ true
+ case _ =>
+ false
+ }
+ } else {
+ false
+ }
+ }
+
+ private def pruningFields(agg: Aggregate): LogicalPlan = {
+ agg.transformDownWithPruning(_.containsPattern(GENERATE), ruleId) {
+ case p @ Project(_, g: Generate) if p.references.intersect(g.outputSet).isEmpty
+ && g.generator.isInstanceOf[ExplodeBase] =>
+ g.generator.children.head.dataType match {
+ case ArrayType(StructType(fields), _) =>
+ val atomicFields = fields.collect {
+ case f: StructField if f.dataType.isInstanceOf[AtomicType] => f
+ }
+ val extractor = if (atomicFields.size > 0) {
+ // Pick an arbitrary atomic field, if any
+ ExtractValue(g.generator.children.head,
+ Literal(atomicFields(0).name), SQLConf.get.resolver)
Review comment:
nit: `conf.resolver`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r759073428
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -2162,6 +2163,74 @@ object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan] {
}
}
+/**
+ * Prunes unnecessary fields from a [[Generate]] if it is under a count aggregation
+ * query.
+ */
+object CountAggregateOptimization extends Rule[LogicalPlan] {
Review comment:
Do you mean just checking for empty project list?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r759506339
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -2162,6 +2163,74 @@ object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan] {
}
}
+/**
+ * Prunes unnecessary fields from a [[Generate]] if it is under a count aggregation
+ * query.
+ */
+object CountAggregateOptimization extends Rule[LogicalPlan] {
Review comment:
Oh I see. Got your point. You mean not just targeting count-only aggregation, but also generally empty project cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978773611
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50070/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-979240610
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145624/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978681665
**[Test build #145598 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145598/testReport)** for PR 34701 at commit [`bab11cd`](https://github.com/apache/spark/commit/bab11cd6cd1ee9d882854c7a511a0eede3c4aa3d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978731148
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50067/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-981896899
**[Test build #145729 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145729/testReport)** for PR 34701 at commit [`f391dc9`](https://github.com/apache/spark/commit/f391dc9aca652ce455d62d5a68380f3477f1c2dc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-982122967
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145729/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984360337
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50320/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984208905
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50301/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984662955
**[Test build #145852 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145852/testReport)** for PR 34701 at commit [`12e6528`](https://github.com/apache/spark/commit/12e65284e45d7e7c05cef881b6da3a154da274d5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-981987228
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50199/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r758930961
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -2162,6 +2163,74 @@ object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan] {
}
}
+/**
+ * Prunes unnecessary fields from a [[Generate]] if it is under a count aggregation
+ * query.
+ */
+object CountAggregateOptimization extends Rule[LogicalPlan] {
Review comment:
My understanding is, rule `ColumnPruning` will create `Project` with an empty project list under count-only aggregate. In the mean while, people can use DataFrame API to create empty `Project` as well.
I think it's simpler and more general to target `Project(Nil, Generate(...))`. We don't care about how the empty project was created, either by count-only aggregate, or by users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984827831
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-982101410
**[Test build #145729 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145729/testReport)** for PR 34701 at commit [`f391dc9`](https://github.com/apache/spark/commit/f391dc9aca652ce455d62d5a68380f3477f1c2dc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r758293697
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -2162,6 +2163,74 @@ object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan] {
}
}
+/**
+ * Prunes unnecessary fields from a [[Generate]] if it is under a count aggregation
+ * query.
+ */
+object CountAggregateOptimization extends Rule[LogicalPlan] {
+ private def isLiteralCountAggFun(func: AggregateFunction): Boolean = func match {
+ case c: Count if c.children.size == 1 && c.children(0).isInstanceOf[Literal] => true
+ case _ => false
+ }
+
+ private def isCandidate(agg: Aggregate): Boolean = {
+ if (agg.aggregateExpressions.size == 1) {
+ agg.aggregateExpressions(0) match {
+ case Alias(ae: AggregateExpression, _) if isLiteralCountAggFun(ae.aggregateFunction) =>
+ true
+ case _ =>
+ false
+ }
+ } else {
+ false
+ }
+ }
+
+ private def pruningFields(agg: Aggregate): LogicalPlan = {
+ agg.transformDownWithPruning(_.containsPattern(GENERATE), ruleId) {
+ case p @ Project(_, g: Generate) if p.references.intersect(g.outputSet).isEmpty
Review comment:
`p.references.intersect(g.outputSet).isEmpty` how is this possible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978925620
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145611/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978732418
**[Test build #145595 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145595/testReport)** for PR 34701 at commit [`97c1c36`](https://github.com/apache/spark/commit/97c1c3631b0a92b5a614f75ce4cd55d1a75a5aff).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978732555
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145595/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fiels from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-978681665
**[Test build #145598 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145598/testReport)** for PR 34701 at commit [`bab11cd`](https://github.com/apache/spark/commit/bab11cd6cd1ee9d882854c7a511a0eede3c4aa3d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984398108
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50324/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate under count-only Aggregate
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r759945029
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -2162,6 +2163,74 @@ object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan] {
}
}
+/**
+ * Prunes unnecessary fields from a [[Generate]] if it is under a count aggregation
+ * query.
+ */
+object CountAggregateOptimization extends Rule[LogicalPlan] {
Review comment:
> also generally empty project cases.
Yea, and not only empty, e.g. `Project(1, child)` can also trigger the optimization, as long as the Project has no references.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984208905
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50301/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984328016
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50320/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r760882270
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaPruningSuite.scala
##########
@@ -882,4 +882,32 @@ abstract class SchemaPruningSuite
Nil)
}
}
+
+ test("SPARK-37450: Prunes unnecessary fields from Explode for count aggregation") {
+ import testImplicits._
+
+ withTempView("table") {
+ withTempPath { dir =>
+ val path = dir.getCanonicalPath
+
+ val jsonStr =
+ """{
+ "items": [
+ {"itemId": 1, "itemData": "a"},
+ {"itemId": 2, "itemData": "b"}
+ ]}""".stripMargin
+ val df = spark.read.json(Seq(jsonStr).toDS)
+ makeDataSourceFile(df, new File(path + "/table"))
Review comment:
changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34701:
URL: https://github.com/apache/spark/pull/34701#discussion_r760882449
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaPruningSuite.scala
##########
@@ -882,4 +882,32 @@ abstract class SchemaPruningSuite
Nil)
}
}
+
+ test("SPARK-37450: Prunes unnecessary fields from Explode for count aggregation") {
+ import testImplicits._
+
+ withTempView("table") {
+ withTempPath { dir =>
+ val path = dir.getCanonicalPath
+
+ val jsonStr =
+ """{
Review comment:
use multi-line now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984497195
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145845/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984366546
**[Test build #145849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145849/testReport)** for PR 34701 at commit [`e479e5c`](https://github.com/apache/spark/commit/e479e5c2088510f88f62e6baa89ebbd18ff135af).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984450031
**[Test build #145852 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145852/testReport)** for PR 34701 at commit [`12e6528`](https://github.com/apache/spark/commit/12e65284e45d7e7c05cef881b6da3a154da274d5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34701: [SPARK-37450][SQL] Prune unnecessary fields from Generate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34701:
URL: https://github.com/apache/spark/pull/34701#issuecomment-984307445
**[Test build #145845 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145845/testReport)** for PR 34701 at commit [`099d3a8`](https://github.com/apache/spark/commit/099d3a8919189fa0b6f0d10079e327d114b657b8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org