You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/05/07 19:12:06 UTC
[GitHub] [spark] cloud-fan opened a new pull request #32470: [WIP] Simplify ResolveAggregateFunctions
cloud-fan opened a new pull request #32470:
URL: https://github.com/apache/spark/pull/32470
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error message, please read the guideline first:
https://spark.apache.org/error-message-guidelines.html
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-859276195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #32470:
URL: https://github.com/apache/spark/pull/32470
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858712516
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44173/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858777838
**[Test build #139652 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139652/testReport)** for PR 32470 at commit [`1e973ef`](https://github.com/apache/spark/commit/1e973ef66b445e3d70df92a52b39e99f7c491151).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-837884000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-834744872
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42783/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-836757567
**[Test build #138336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138336/testReport)** for PR 32470 at commit [`ab31ab1`](https://github.com/apache/spark/commit/ab31ab1b6312da694e9964fbc94c5d7842bf5797).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-862219789
thanks for the review, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-835159651
**[Test build #138281 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138281/testReport)** for PR 32470 at commit [`c67a801`](https://github.com/apache/spark/commit/c67a8013d6f848e7ee0227e50f5fec499d8648dc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858347812
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44137/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-836344091
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138326/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-836319291
**[Test build #138326 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138326/testReport)** for PR 32470 at commit [`dbc00c8`](https://github.com/apache/spark/commit/dbc00c8e40f397447f4d75c7a16c33fb62eb5112).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858467500
**[Test build #139629 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139629/testReport)** for PR 32470 at commit [`ae4d411`](https://github.com/apache/spark/commit/ae4d411bd68fcd9daf4db23a7a33b4fdca478528).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858878898
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-859276195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858290812
**[Test build #139612 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139612/testReport)** for PR 32470 at commit [`d08d8e4`](https://github.com/apache/spark/commit/d08d8e4cb6acfaff34e3d81cc8b41c65aa34f4f6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #32470:
URL: https://github.com/apache/spark/pull/32470#discussion_r649623086
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2457,164 +2450,127 @@ class Analyzer(override val catalogManager: CatalogManager)
_.containsPattern(AGGREGATE), ruleId) {
// Resolve aggregate with having clause to Filter(..., Aggregate()). Note, to avoid wrongly
// resolve the having condition expression, here we skip resolving it in ResolveReferences
- // and transform it to Filter after aggregate is resolved. See more details in SPARK-31519.
+ // and transform it to Filter after aggregate is resolved. Basically columns in HAVING should
+ // be resolved with `agg.child.output` first. See more details in SPARK-31519.
case UnresolvedHaving(cond, agg: Aggregate) if agg.resolved =>
- resolveHaving(Filter(cond, agg), agg)
-
- case f @ Filter(_, agg: Aggregate) if agg.resolved =>
- resolveHaving(f, agg)
-
- case sort @ Sort(sortOrder, global, aggregate: Aggregate) if aggregate.resolved =>
-
- // Try resolving the ordering as though it is in the aggregate clause.
- try {
- // If a sort order is unresolved, containing references not in aggregate, or containing
- // `AggregateExpression`, we need to push down it to the underlying aggregate operator.
- val unresolvedSortOrders = sortOrder.filter { s =>
- !s.resolved || !s.references.subsetOf(aggregate.outputSet) || containsAggregate(s)
+ resolveOperatorWithAggregate(Seq(cond), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Filter(cond, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = resolveExpressionByPlanOutput(cond, agg)
+ resolveOperatorWithAggregate(Seq(maybeResolved), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Sort(sortOrder, global, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = sortOrder.map(_.child).map(resolveExpressionByPlanOutput(_, agg))
+ resolveOperatorWithAggregate(maybeResolved, agg, (newExprs, newChild) => {
+ val newSortOrder = sortOrder.zip(newExprs).map {
+ case (sortOrder, expr) => sortOrder.copy(child = expr)
}
- val aliasedOrdering = unresolvedSortOrders.map(o => Alias(o.child, "aggOrder")())
-
- val aggregateWithExtraOrdering = aggregate.copy(
- aggregateExpressions = aggregate.aggregateExpressions ++ aliasedOrdering)
-
- val resolvedAggregate: Aggregate =
- executeSameContext(aggregateWithExtraOrdering).asInstanceOf[Aggregate]
-
- val (reResolvedAggExprs, resolvedAliasedOrdering) =
- resolvedAggregate.aggregateExpressions.splitAt(aggregate.aggregateExpressions.length)
-
- // If we pass the analysis check, then the ordering expressions should only reference to
- // aggregate expressions or grouping expressions, and it's safe to push them down to
- // Aggregate.
- checkAnalysis(resolvedAggregate)
-
- val originalAggExprs = aggregate.aggregateExpressions.map(trimNonTopLevelAliases)
-
- // If the ordering expression is same with original aggregate expression, we don't need
- // to push down this ordering expression and can reference the original aggregate
- // expression instead.
- val needsPushDown = ArrayBuffer.empty[NamedExpression]
- val orderToAlias = unresolvedSortOrders.zip(aliasedOrdering)
- val evaluatedOrderings =
- resolvedAliasedOrdering.asInstanceOf[Seq[Alias]].zip(orderToAlias).map {
- case (evaluated, (order, aliasOrder)) =>
- val index = reResolvedAggExprs.indexWhere {
- case Alias(child, _) => child semanticEquals evaluated.child
- case other => other semanticEquals evaluated.child
- }
-
- if (index == -1) {
- if (hasCharVarchar(evaluated)) {
- needsPushDown += aliasOrder
- order.copy(child = aliasOrder)
- } else {
- needsPushDown += evaluated
- order.copy(child = evaluated.toAttribute)
- }
- } else {
- order.copy(child = originalAggExprs(index).toAttribute)
- }
+ Sort(newSortOrder, global, newChild)
+ })
+ }
+
+ def resolveExprsWithAggregate(
+ exprs: Seq[Expression],
+ agg: Aggregate): (Seq[NamedExpression], Seq[Expression]) = {
Review comment:
How about leaving some comments about what are the returned two values?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2457,164 +2450,127 @@ class Analyzer(override val catalogManager: CatalogManager)
_.containsPattern(AGGREGATE), ruleId) {
// Resolve aggregate with having clause to Filter(..., Aggregate()). Note, to avoid wrongly
// resolve the having condition expression, here we skip resolving it in ResolveReferences
- // and transform it to Filter after aggregate is resolved. See more details in SPARK-31519.
+ // and transform it to Filter after aggregate is resolved. Basically columns in HAVING should
+ // be resolved with `agg.child.output` first. See more details in SPARK-31519.
case UnresolvedHaving(cond, agg: Aggregate) if agg.resolved =>
- resolveHaving(Filter(cond, agg), agg)
-
- case f @ Filter(_, agg: Aggregate) if agg.resolved =>
- resolveHaving(f, agg)
-
- case sort @ Sort(sortOrder, global, aggregate: Aggregate) if aggregate.resolved =>
-
- // Try resolving the ordering as though it is in the aggregate clause.
- try {
- // If a sort order is unresolved, containing references not in aggregate, or containing
- // `AggregateExpression`, we need to push down it to the underlying aggregate operator.
- val unresolvedSortOrders = sortOrder.filter { s =>
- !s.resolved || !s.references.subsetOf(aggregate.outputSet) || containsAggregate(s)
+ resolveOperatorWithAggregate(Seq(cond), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Filter(cond, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = resolveExpressionByPlanOutput(cond, agg)
+ resolveOperatorWithAggregate(Seq(maybeResolved), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Sort(sortOrder, global, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = sortOrder.map(_.child).map(resolveExpressionByPlanOutput(_, agg))
+ resolveOperatorWithAggregate(maybeResolved, agg, (newExprs, newChild) => {
+ val newSortOrder = sortOrder.zip(newExprs).map {
+ case (sortOrder, expr) => sortOrder.copy(child = expr)
}
- val aliasedOrdering = unresolvedSortOrders.map(o => Alias(o.child, "aggOrder")())
-
- val aggregateWithExtraOrdering = aggregate.copy(
- aggregateExpressions = aggregate.aggregateExpressions ++ aliasedOrdering)
-
- val resolvedAggregate: Aggregate =
- executeSameContext(aggregateWithExtraOrdering).asInstanceOf[Aggregate]
-
- val (reResolvedAggExprs, resolvedAliasedOrdering) =
- resolvedAggregate.aggregateExpressions.splitAt(aggregate.aggregateExpressions.length)
-
- // If we pass the analysis check, then the ordering expressions should only reference to
- // aggregate expressions or grouping expressions, and it's safe to push them down to
- // Aggregate.
- checkAnalysis(resolvedAggregate)
-
- val originalAggExprs = aggregate.aggregateExpressions.map(trimNonTopLevelAliases)
-
- // If the ordering expression is same with original aggregate expression, we don't need
- // to push down this ordering expression and can reference the original aggregate
- // expression instead.
- val needsPushDown = ArrayBuffer.empty[NamedExpression]
- val orderToAlias = unresolvedSortOrders.zip(aliasedOrdering)
- val evaluatedOrderings =
- resolvedAliasedOrdering.asInstanceOf[Seq[Alias]].zip(orderToAlias).map {
- case (evaluated, (order, aliasOrder)) =>
- val index = reResolvedAggExprs.indexWhere {
- case Alias(child, _) => child semanticEquals evaluated.child
- case other => other semanticEquals evaluated.child
- }
-
- if (index == -1) {
- if (hasCharVarchar(evaluated)) {
- needsPushDown += aliasOrder
- order.copy(child = aliasOrder)
- } else {
- needsPushDown += evaluated
- order.copy(child = evaluated.toAttribute)
- }
- } else {
- order.copy(child = originalAggExprs(index).toAttribute)
- }
+ Sort(newSortOrder, global, newChild)
+ })
+ }
+
+ def resolveExprsWithAggregate(
+ exprs: Seq[Expression],
+ agg: Aggregate): (Seq[NamedExpression], Seq[Expression]) = {
+ val aggregateExpressions = ArrayBuffer.empty[NamedExpression]
Review comment:
nit: how about `aggregateExpressions` -> `extraAggExprs`?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2457,164 +2450,127 @@ class Analyzer(override val catalogManager: CatalogManager)
_.containsPattern(AGGREGATE), ruleId) {
// Resolve aggregate with having clause to Filter(..., Aggregate()). Note, to avoid wrongly
// resolve the having condition expression, here we skip resolving it in ResolveReferences
- // and transform it to Filter after aggregate is resolved. See more details in SPARK-31519.
+ // and transform it to Filter after aggregate is resolved. Basically columns in HAVING should
+ // be resolved with `agg.child.output` first. See more details in SPARK-31519.
case UnresolvedHaving(cond, agg: Aggregate) if agg.resolved =>
- resolveHaving(Filter(cond, agg), agg)
-
- case f @ Filter(_, agg: Aggregate) if agg.resolved =>
- resolveHaving(f, agg)
-
- case sort @ Sort(sortOrder, global, aggregate: Aggregate) if aggregate.resolved =>
-
- // Try resolving the ordering as though it is in the aggregate clause.
- try {
- // If a sort order is unresolved, containing references not in aggregate, or containing
- // `AggregateExpression`, we need to push down it to the underlying aggregate operator.
- val unresolvedSortOrders = sortOrder.filter { s =>
- !s.resolved || !s.references.subsetOf(aggregate.outputSet) || containsAggregate(s)
+ resolveOperatorWithAggregate(Seq(cond), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Filter(cond, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = resolveExpressionByPlanOutput(cond, agg)
+ resolveOperatorWithAggregate(Seq(maybeResolved), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Sort(sortOrder, global, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = sortOrder.map(_.child).map(resolveExpressionByPlanOutput(_, agg))
+ resolveOperatorWithAggregate(maybeResolved, agg, (newExprs, newChild) => {
+ val newSortOrder = sortOrder.zip(newExprs).map {
+ case (sortOrder, expr) => sortOrder.copy(child = expr)
}
- val aliasedOrdering = unresolvedSortOrders.map(o => Alias(o.child, "aggOrder")())
-
- val aggregateWithExtraOrdering = aggregate.copy(
- aggregateExpressions = aggregate.aggregateExpressions ++ aliasedOrdering)
-
- val resolvedAggregate: Aggregate =
- executeSameContext(aggregateWithExtraOrdering).asInstanceOf[Aggregate]
-
- val (reResolvedAggExprs, resolvedAliasedOrdering) =
- resolvedAggregate.aggregateExpressions.splitAt(aggregate.aggregateExpressions.length)
-
- // If we pass the analysis check, then the ordering expressions should only reference to
- // aggregate expressions or grouping expressions, and it's safe to push them down to
- // Aggregate.
- checkAnalysis(resolvedAggregate)
-
- val originalAggExprs = aggregate.aggregateExpressions.map(trimNonTopLevelAliases)
-
- // If the ordering expression is same with original aggregate expression, we don't need
- // to push down this ordering expression and can reference the original aggregate
- // expression instead.
- val needsPushDown = ArrayBuffer.empty[NamedExpression]
- val orderToAlias = unresolvedSortOrders.zip(aliasedOrdering)
- val evaluatedOrderings =
- resolvedAliasedOrdering.asInstanceOf[Seq[Alias]].zip(orderToAlias).map {
- case (evaluated, (order, aliasOrder)) =>
- val index = reResolvedAggExprs.indexWhere {
- case Alias(child, _) => child semanticEquals evaluated.child
- case other => other semanticEquals evaluated.child
- }
-
- if (index == -1) {
- if (hasCharVarchar(evaluated)) {
- needsPushDown += aliasOrder
- order.copy(child = aliasOrder)
- } else {
- needsPushDown += evaluated
- order.copy(child = evaluated.toAttribute)
- }
- } else {
- order.copy(child = originalAggExprs(index).toAttribute)
- }
+ Sort(newSortOrder, global, newChild)
+ })
+ }
+
+ def resolveExprsWithAggregate(
+ exprs: Seq[Expression],
+ agg: Aggregate): (Seq[NamedExpression], Seq[Expression]) = {
+ val aggregateExpressions = ArrayBuffer.empty[NamedExpression]
+ val transformed = exprs.map { e =>
+ // Try resolving the expression as though it is in the aggregate clause.
+ def resolveCol(input: Expression): Expression = {
+ resolveExpressionByPlanOutput(input, agg.child)
+ }
+ def resolveSubQuery(input: Expression): Expression = {
+ if (SubqueryExpression.hasSubquery(input)) {
+ val fake = Project(Alias(input, "fake")() :: Nil, agg.child)
+ ResolveSubquery(fake).asInstanceOf[Project].projectList.head.asInstanceOf[Alias].child
+ } else {
+ input
}
-
- val sortOrdersMap = unresolvedSortOrders
- .map(new TreeNodeRef(_))
- .zip(evaluatedOrderings)
- .toMap
- val finalSortOrders = sortOrder.map(s => sortOrdersMap.getOrElse(new TreeNodeRef(s), s))
-
- // Since we don't rely on sort.resolved as the stop condition for this rule,
- // we need to check this and prevent applying this rule multiple times
- if (sortOrder == finalSortOrders) {
- sort
+ }
+ val maybeResolved = resolveSubQuery(resolveCol(e))
+ if (maybeResolved.resolved && maybeResolved.references.subsetOf(agg.outputSet) &&
+ !containsAggregate(maybeResolved)) {
+ // The given expression is valid and doesn't need extra resolution.
+ maybeResolved
+ } else if (containsUnresolvedFunc(maybeResolved)) {
+ // The given expression has unresolved functions which may be aggregate functions and we
+ // need to wait for other rules to resolve the functions first.
+ maybeResolved
+ } else {
+ // Avoid adding an extra aggregate expression if it's already present in
+ // `agg.aggregateExpressions`.
+ val index = if (maybeResolved.resolved) {
+ agg.aggregateExpressions.indexWhere {
+ case Alias(child, _) => child semanticEquals maybeResolved
+ case other => other semanticEquals maybeResolved
+ }
} else {
- Project(aggregate.output,
- Sort(finalSortOrders, global,
- aggregate.copy(aggregateExpressions = originalAggExprs ++ needsPushDown)))
+ -1
+ }
+ if (index >= 0) {
+ agg.aggregateExpressions(index).toAttribute
+ } else {
+ buildAggExprList(maybeResolved, agg, aggregateExpressions)
}
- } catch {
- // Attempting to resolve in the aggregate can result in ambiguity. When this happens,
- // just return the original plan.
- case ae: AnalysisException => sort
}
+ }
+ (aggregateExpressions.toSeq, transformed)
}
- def hasCharVarchar(expr: Alias): Boolean = {
- expr.find {
- case ne: NamedExpression => CharVarcharUtils.getRawType(ne.metadata).nonEmpty
- case _ => false
- }.nonEmpty
+ private def buildAggExprList(
+ expr: Expression,
+ agg: Aggregate,
+ aggExprList: ArrayBuffer[NamedExpression]): Expression = expr match {
+ case ae: AggregateExpression if ae.resolved =>
+ val alias = Alias(ae, ae.toString)()
+ aggExprList += alias
+ alias.toAttribute
+ // Grouping functions are handled in the rule [[ResolveGroupingAnalytics]].
+ case grouping: Expression if grouping.resolved &&
+ agg.groupingExpressions.exists(_.semanticEquals(grouping)) &&
+ !ResolveGroupingAnalytics.hasGroupingFunction(grouping) &&
+ !agg.output.exists(_.semanticEquals(grouping)) =>
+ grouping match {
+ case ne: NamedExpression =>
+ aggExprList += ne
+ ne.toAttribute
+ case _ =>
+ val alias = Alias(grouping, grouping.toString)()
+ aggExprList += alias
+ alias.toAttribute
+ }
+ case a: Attribute if agg.child.outputSet.contains(a) && !agg.outputSet.contains(a) =>
+ // Undo the resolution. This attribute is neither inside aggregate functions nor a
+ // grouping column. It shouldn't be resolved with `agg.child.output`.
+ CurrentOrigin.withOrigin(a.origin)(UnresolvedAttribute(Seq(a.name)))
Review comment:
Just out of curiosity; what's this handling for?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -623,28 +622,26 @@ class Analyzer(override val catalogManager: CatalogManager)
// groupingExpressions for condition resolving.
val aggForResolving = aggregate.copy(groupingExpressions = groupByExprs)
// Try resolving the condition of the filter as though it is in the aggregate clause
- val resolvedInfo =
- ResolveAggregateFunctions.resolveFilterCondInAggregate(h.havingCondition, aggForResolving)
+ val (extraAggExprs, Seq(afterResolve)) =
Review comment:
nit: how about `afterResolve` -> `resolvedHavingCond`?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2457,164 +2450,127 @@ class Analyzer(override val catalogManager: CatalogManager)
_.containsPattern(AGGREGATE), ruleId) {
// Resolve aggregate with having clause to Filter(..., Aggregate()). Note, to avoid wrongly
// resolve the having condition expression, here we skip resolving it in ResolveReferences
- // and transform it to Filter after aggregate is resolved. See more details in SPARK-31519.
+ // and transform it to Filter after aggregate is resolved. Basically columns in HAVING should
+ // be resolved with `agg.child.output` first. See more details in SPARK-31519.
case UnresolvedHaving(cond, agg: Aggregate) if agg.resolved =>
- resolveHaving(Filter(cond, agg), agg)
-
- case f @ Filter(_, agg: Aggregate) if agg.resolved =>
- resolveHaving(f, agg)
-
- case sort @ Sort(sortOrder, global, aggregate: Aggregate) if aggregate.resolved =>
-
- // Try resolving the ordering as though it is in the aggregate clause.
- try {
- // If a sort order is unresolved, containing references not in aggregate, or containing
- // `AggregateExpression`, we need to push down it to the underlying aggregate operator.
- val unresolvedSortOrders = sortOrder.filter { s =>
- !s.resolved || !s.references.subsetOf(aggregate.outputSet) || containsAggregate(s)
+ resolveOperatorWithAggregate(Seq(cond), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Filter(cond, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = resolveExpressionByPlanOutput(cond, agg)
+ resolveOperatorWithAggregate(Seq(maybeResolved), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Sort(sortOrder, global, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = sortOrder.map(_.child).map(resolveExpressionByPlanOutput(_, agg))
+ resolveOperatorWithAggregate(maybeResolved, agg, (newExprs, newChild) => {
+ val newSortOrder = sortOrder.zip(newExprs).map {
+ case (sortOrder, expr) => sortOrder.copy(child = expr)
}
- val aliasedOrdering = unresolvedSortOrders.map(o => Alias(o.child, "aggOrder")())
-
- val aggregateWithExtraOrdering = aggregate.copy(
- aggregateExpressions = aggregate.aggregateExpressions ++ aliasedOrdering)
-
- val resolvedAggregate: Aggregate =
- executeSameContext(aggregateWithExtraOrdering).asInstanceOf[Aggregate]
-
- val (reResolvedAggExprs, resolvedAliasedOrdering) =
- resolvedAggregate.aggregateExpressions.splitAt(aggregate.aggregateExpressions.length)
-
- // If we pass the analysis check, then the ordering expressions should only reference to
- // aggregate expressions or grouping expressions, and it's safe to push them down to
- // Aggregate.
- checkAnalysis(resolvedAggregate)
-
- val originalAggExprs = aggregate.aggregateExpressions.map(trimNonTopLevelAliases)
-
- // If the ordering expression is same with original aggregate expression, we don't need
- // to push down this ordering expression and can reference the original aggregate
- // expression instead.
- val needsPushDown = ArrayBuffer.empty[NamedExpression]
- val orderToAlias = unresolvedSortOrders.zip(aliasedOrdering)
- val evaluatedOrderings =
- resolvedAliasedOrdering.asInstanceOf[Seq[Alias]].zip(orderToAlias).map {
- case (evaluated, (order, aliasOrder)) =>
- val index = reResolvedAggExprs.indexWhere {
- case Alias(child, _) => child semanticEquals evaluated.child
- case other => other semanticEquals evaluated.child
- }
-
- if (index == -1) {
- if (hasCharVarchar(evaluated)) {
- needsPushDown += aliasOrder
- order.copy(child = aliasOrder)
- } else {
- needsPushDown += evaluated
- order.copy(child = evaluated.toAttribute)
- }
- } else {
- order.copy(child = originalAggExprs(index).toAttribute)
- }
+ Sort(newSortOrder, global, newChild)
+ })
+ }
+
+ def resolveExprsWithAggregate(
+ exprs: Seq[Expression],
+ agg: Aggregate): (Seq[NamedExpression], Seq[Expression]) = {
+ val aggregateExpressions = ArrayBuffer.empty[NamedExpression]
+ val transformed = exprs.map { e =>
+ // Try resolving the expression as though it is in the aggregate clause.
+ def resolveCol(input: Expression): Expression = {
+ resolveExpressionByPlanOutput(input, agg.child)
+ }
+ def resolveSubQuery(input: Expression): Expression = {
+ if (SubqueryExpression.hasSubquery(input)) {
+ val fake = Project(Alias(input, "fake")() :: Nil, agg.child)
+ ResolveSubquery(fake).asInstanceOf[Project].projectList.head.asInstanceOf[Alias].child
+ } else {
+ input
}
-
- val sortOrdersMap = unresolvedSortOrders
- .map(new TreeNodeRef(_))
- .zip(evaluatedOrderings)
- .toMap
- val finalSortOrders = sortOrder.map(s => sortOrdersMap.getOrElse(new TreeNodeRef(s), s))
-
- // Since we don't rely on sort.resolved as the stop condition for this rule,
- // we need to check this and prevent applying this rule multiple times
- if (sortOrder == finalSortOrders) {
- sort
+ }
+ val maybeResolved = resolveSubQuery(resolveCol(e))
+ if (maybeResolved.resolved && maybeResolved.references.subsetOf(agg.outputSet) &&
+ !containsAggregate(maybeResolved)) {
+ // The given expression is valid and doesn't need extra resolution.
+ maybeResolved
+ } else if (containsUnresolvedFunc(maybeResolved)) {
+ // The given expression has unresolved functions which may be aggregate functions and we
+ // need to wait for other rules to resolve the functions first.
+ maybeResolved
+ } else {
+ // Avoid adding an extra aggregate expression if it's already present in
+ // `agg.aggregateExpressions`.
+ val index = if (maybeResolved.resolved) {
+ agg.aggregateExpressions.indexWhere {
+ case Alias(child, _) => child semanticEquals maybeResolved
+ case other => other semanticEquals maybeResolved
+ }
} else {
- Project(aggregate.output,
- Sort(finalSortOrders, global,
- aggregate.copy(aggregateExpressions = originalAggExprs ++ needsPushDown)))
+ -1
+ }
+ if (index >= 0) {
Review comment:
It looks we don't need this if condition?
```
if (maybeResolved.resolved) {
val index = agg.aggregateExpressions.indexWhere {
case Alias(child, _) => child semanticEquals maybeResolved
case other => other semanticEquals maybeResolved
}
agg.aggregateExpressions(index).toAttribute
} else {
buildAggExprList(maybeResolved, agg, aggregateExpressions)
}
```
?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2457,164 +2450,127 @@ class Analyzer(override val catalogManager: CatalogManager)
_.containsPattern(AGGREGATE), ruleId) {
// Resolve aggregate with having clause to Filter(..., Aggregate()). Note, to avoid wrongly
// resolve the having condition expression, here we skip resolving it in ResolveReferences
- // and transform it to Filter after aggregate is resolved. See more details in SPARK-31519.
+ // and transform it to Filter after aggregate is resolved. Basically columns in HAVING should
+ // be resolved with `agg.child.output` first. See more details in SPARK-31519.
case UnresolvedHaving(cond, agg: Aggregate) if agg.resolved =>
- resolveHaving(Filter(cond, agg), agg)
-
- case f @ Filter(_, agg: Aggregate) if agg.resolved =>
- resolveHaving(f, agg)
-
- case sort @ Sort(sortOrder, global, aggregate: Aggregate) if aggregate.resolved =>
-
- // Try resolving the ordering as though it is in the aggregate clause.
- try {
- // If a sort order is unresolved, containing references not in aggregate, or containing
- // `AggregateExpression`, we need to push down it to the underlying aggregate operator.
- val unresolvedSortOrders = sortOrder.filter { s =>
- !s.resolved || !s.references.subsetOf(aggregate.outputSet) || containsAggregate(s)
+ resolveOperatorWithAggregate(Seq(cond), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Filter(cond, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = resolveExpressionByPlanOutput(cond, agg)
+ resolveOperatorWithAggregate(Seq(maybeResolved), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Sort(sortOrder, global, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = sortOrder.map(_.child).map(resolveExpressionByPlanOutput(_, agg))
+ resolveOperatorWithAggregate(maybeResolved, agg, (newExprs, newChild) => {
+ val newSortOrder = sortOrder.zip(newExprs).map {
+ case (sortOrder, expr) => sortOrder.copy(child = expr)
}
- val aliasedOrdering = unresolvedSortOrders.map(o => Alias(o.child, "aggOrder")())
-
- val aggregateWithExtraOrdering = aggregate.copy(
- aggregateExpressions = aggregate.aggregateExpressions ++ aliasedOrdering)
-
- val resolvedAggregate: Aggregate =
- executeSameContext(aggregateWithExtraOrdering).asInstanceOf[Aggregate]
-
- val (reResolvedAggExprs, resolvedAliasedOrdering) =
- resolvedAggregate.aggregateExpressions.splitAt(aggregate.aggregateExpressions.length)
-
- // If we pass the analysis check, then the ordering expressions should only reference to
- // aggregate expressions or grouping expressions, and it's safe to push them down to
- // Aggregate.
- checkAnalysis(resolvedAggregate)
-
- val originalAggExprs = aggregate.aggregateExpressions.map(trimNonTopLevelAliases)
-
- // If the ordering expression is same with original aggregate expression, we don't need
- // to push down this ordering expression and can reference the original aggregate
- // expression instead.
- val needsPushDown = ArrayBuffer.empty[NamedExpression]
- val orderToAlias = unresolvedSortOrders.zip(aliasedOrdering)
- val evaluatedOrderings =
- resolvedAliasedOrdering.asInstanceOf[Seq[Alias]].zip(orderToAlias).map {
- case (evaluated, (order, aliasOrder)) =>
- val index = reResolvedAggExprs.indexWhere {
- case Alias(child, _) => child semanticEquals evaluated.child
- case other => other semanticEquals evaluated.child
- }
-
- if (index == -1) {
- if (hasCharVarchar(evaluated)) {
- needsPushDown += aliasOrder
- order.copy(child = aliasOrder)
- } else {
- needsPushDown += evaluated
- order.copy(child = evaluated.toAttribute)
- }
- } else {
- order.copy(child = originalAggExprs(index).toAttribute)
- }
+ Sort(newSortOrder, global, newChild)
+ })
+ }
+
+ def resolveExprsWithAggregate(
+ exprs: Seq[Expression],
+ agg: Aggregate): (Seq[NamedExpression], Seq[Expression]) = {
+ val aggregateExpressions = ArrayBuffer.empty[NamedExpression]
+ val transformed = exprs.map { e =>
+ // Try resolving the expression as though it is in the aggregate clause.
+ def resolveCol(input: Expression): Expression = {
+ resolveExpressionByPlanOutput(input, agg.child)
+ }
+ def resolveSubQuery(input: Expression): Expression = {
+ if (SubqueryExpression.hasSubquery(input)) {
+ val fake = Project(Alias(input, "fake")() :: Nil, agg.child)
+ ResolveSubquery(fake).asInstanceOf[Project].projectList.head.asInstanceOf[Alias].child
+ } else {
+ input
}
-
- val sortOrdersMap = unresolvedSortOrders
- .map(new TreeNodeRef(_))
- .zip(evaluatedOrderings)
- .toMap
- val finalSortOrders = sortOrder.map(s => sortOrdersMap.getOrElse(new TreeNodeRef(s), s))
-
- // Since we don't rely on sort.resolved as the stop condition for this rule,
- // we need to check this and prevent applying this rule multiple times
- if (sortOrder == finalSortOrders) {
- sort
+ }
+ val maybeResolved = resolveSubQuery(resolveCol(e))
+ if (maybeResolved.resolved && maybeResolved.references.subsetOf(agg.outputSet) &&
+ !containsAggregate(maybeResolved)) {
+ // The given expression is valid and doesn't need extra resolution.
+ maybeResolved
+ } else if (containsUnresolvedFunc(maybeResolved)) {
+ // The given expression has unresolved functions which may be aggregate functions and we
+ // need to wait for other rules to resolve the functions first.
+ maybeResolved
+ } else {
+ // Avoid adding an extra aggregate expression if it's already present in
+ // `agg.aggregateExpressions`.
+ val index = if (maybeResolved.resolved) {
+ agg.aggregateExpressions.indexWhere {
+ case Alias(child, _) => child semanticEquals maybeResolved
+ case other => other semanticEquals maybeResolved
+ }
} else {
- Project(aggregate.output,
- Sort(finalSortOrders, global,
- aggregate.copy(aggregateExpressions = originalAggExprs ++ needsPushDown)))
+ -1
+ }
+ if (index >= 0) {
Review comment:
Ah, I see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858829181
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44180/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-861437561
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44335/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a change in pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on a change in pull request #32470:
URL: https://github.com/apache/spark/pull/32470#discussion_r650165639
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2457,164 +2450,136 @@ class Analyzer(override val catalogManager: CatalogManager)
_.containsPattern(AGGREGATE), ruleId) {
// Resolve aggregate with having clause to Filter(..., Aggregate()). Note, to avoid wrongly
// resolve the having condition expression, here we skip resolving it in ResolveReferences
- // and transform it to Filter after aggregate is resolved. See more details in SPARK-31519.
+ // and transform it to Filter after aggregate is resolved. Basically columns in HAVING should
+ // be resolved with `agg.child.output` first. See more details in SPARK-31519.
case UnresolvedHaving(cond, agg: Aggregate) if agg.resolved =>
- resolveHaving(Filter(cond, agg), agg)
-
- case f @ Filter(_, agg: Aggregate) if agg.resolved =>
- resolveHaving(f, agg)
-
- case sort @ Sort(sortOrder, global, aggregate: Aggregate) if aggregate.resolved =>
-
- // Try resolving the ordering as though it is in the aggregate clause.
- try {
- // If a sort order is unresolved, containing references not in aggregate, or containing
- // `AggregateExpression`, we need to push down it to the underlying aggregate operator.
- val unresolvedSortOrders = sortOrder.filter { s =>
- !s.resolved || !s.references.subsetOf(aggregate.outputSet) || containsAggregate(s)
+ resolveOperatorWithAggregate(Seq(cond), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Filter(cond, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = resolveExpressionByPlanOutput(cond, agg)
+ resolveOperatorWithAggregate(Seq(maybeResolved), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Sort(sortOrder, global, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = sortOrder.map(_.child).map(resolveExpressionByPlanOutput(_, agg))
+ resolveOperatorWithAggregate(maybeResolved, agg, (newExprs, newChild) => {
+ val newSortOrder = sortOrder.zip(newExprs).map {
+ case (sortOrder, expr) => sortOrder.copy(child = expr)
}
- val aliasedOrdering = unresolvedSortOrders.map(o => Alias(o.child, "aggOrder")())
-
- val aggregateWithExtraOrdering = aggregate.copy(
- aggregateExpressions = aggregate.aggregateExpressions ++ aliasedOrdering)
-
- val resolvedAggregate: Aggregate =
- executeSameContext(aggregateWithExtraOrdering).asInstanceOf[Aggregate]
-
- val (reResolvedAggExprs, resolvedAliasedOrdering) =
- resolvedAggregate.aggregateExpressions.splitAt(aggregate.aggregateExpressions.length)
-
- // If we pass the analysis check, then the ordering expressions should only reference to
- // aggregate expressions or grouping expressions, and it's safe to push them down to
- // Aggregate.
- checkAnalysis(resolvedAggregate)
-
- val originalAggExprs = aggregate.aggregateExpressions.map(trimNonTopLevelAliases)
-
- // If the ordering expression is same with original aggregate expression, we don't need
- // to push down this ordering expression and can reference the original aggregate
- // expression instead.
- val needsPushDown = ArrayBuffer.empty[NamedExpression]
- val orderToAlias = unresolvedSortOrders.zip(aliasedOrdering)
- val evaluatedOrderings =
- resolvedAliasedOrdering.asInstanceOf[Seq[Alias]].zip(orderToAlias).map {
- case (evaluated, (order, aliasOrder)) =>
- val index = reResolvedAggExprs.indexWhere {
- case Alias(child, _) => child semanticEquals evaluated.child
- case other => other semanticEquals evaluated.child
- }
+ Sort(newSortOrder, global, newChild)
+ })
+ }
- if (index == -1) {
- if (hasCharVarchar(evaluated)) {
- needsPushDown += aliasOrder
- order.copy(child = aliasOrder)
- } else {
- needsPushDown += evaluated
- order.copy(child = evaluated.toAttribute)
- }
- } else {
- order.copy(child = originalAggExprs(index).toAttribute)
- }
+ /**
+ * Resolves the given expressions as if they are in the given Aggregate operator, which means
+ * the column can be resolved using `agg.child` and aggregate functions/grouping columns are
+ * allowed. It returns a list of named expressions that need to be appended to
+ * `agg.aggregateExpressions`, and the list of resolved expressions.
+ */
+ def resolveExprsWithAggregate(
+ exprs: Seq[Expression],
+ agg: Aggregate): (Seq[NamedExpression], Seq[Expression]) = {
+ val extraAggExprs = ArrayBuffer.empty[NamedExpression]
+ val transformed = exprs.map { e =>
+ // Try resolving the expression as though it is in the aggregate clause.
+ def resolveCol(input: Expression): Expression = {
+ input.transform {
+ case u: UnresolvedAttribute =>
+ try {
+ agg.child.resolve(u.nameParts, resolver)
+ .map(TempResolvedColumn(_, u.nameParts)).getOrElse(u)
+ } catch {
+ case _: AnalysisException => u
+ }
}
-
- val sortOrdersMap = unresolvedSortOrders
- .map(new TreeNodeRef(_))
- .zip(evaluatedOrderings)
- .toMap
- val finalSortOrders = sortOrder.map(s => sortOrdersMap.getOrElse(new TreeNodeRef(s), s))
-
- // Since we don't rely on sort.resolved as the stop condition for this rule,
- // we need to check this and prevent applying this rule multiple times
- if (sortOrder == finalSortOrders) {
- sort
+ }
+ def resolveSubQuery(input: Expression): Expression = {
+ if (SubqueryExpression.hasSubquery(input)) {
+ val fake = Project(Alias(input, "fake")() :: Nil, agg.child)
+ ResolveSubquery(fake).asInstanceOf[Project].projectList.head.asInstanceOf[Alias].child
} else {
- Project(aggregate.output,
- Sort(finalSortOrders, global,
- aggregate.copy(aggregateExpressions = originalAggExprs ++ needsPushDown)))
+ input
}
- } catch {
- // Attempting to resolve in the aggregate can result in ambiguity. When this happens,
- // just return the original plan.
- case ae: AnalysisException => sort
}
- }
- def hasCharVarchar(expr: Alias): Boolean = {
- expr.find {
- case ne: NamedExpression => CharVarcharUtils.getRawType(ne.metadata).nonEmpty
- case _ => false
- }.nonEmpty
+ val maybeResolved = resolveSubQuery(resolveCol(e))
+ if (!maybeResolved.resolved) {
+ maybeResolved
+ } else {
+ buildAggExprList(maybeResolved, agg, extraAggExprs)
+ }
+ }
+ (extraAggExprs.toSeq, transformed)
}
- def containsAggregate(condition: Expression): Boolean = {
- condition.find(_.isInstanceOf[AggregateExpression]).isDefined
+ private def trimTempResolvedField(input: Expression): Expression = input.transform {
+ case t: TempResolvedColumn => t.child
}
- def resolveFilterCondInAggregate(
- filterCond: Expression, agg: Aggregate): Option[(Seq[NamedExpression], Expression)] = {
- try {
- val aggregatedCondition =
- Aggregate(
- agg.groupingExpressions,
- Alias(filterCond, "havingCondition")() :: Nil,
- agg.child)
- val resolvedOperator = executeSameContext(aggregatedCondition)
- def resolvedAggregateFilter =
- resolvedOperator
- .asInstanceOf[Aggregate]
- .aggregateExpressions.head
-
- // If resolution was successful and we see the filter has an aggregate in it, add it to
- // the original aggregate operator.
- if (resolvedOperator.resolved) {
- // Try to replace all aggregate expressions in the filter by an alias.
- val aggregateExpressions = ArrayBuffer.empty[NamedExpression]
- val transformedAggregateFilter = resolvedAggregateFilter.transform {
- case ae: AggregateExpression =>
- val alias = Alias(ae, ae.toString)()
- aggregateExpressions += alias
+ private def buildAggExprList(
+ expr: Expression,
+ agg: Aggregate,
+ aggExprList: ArrayBuffer[NamedExpression]): Expression = {
+ // Avoid adding an extra aggregate expression if it's already present in
+ // `agg.aggregateExpressions`.
+ val index = agg.aggregateExpressions.indexWhere {
Review comment:
Should we also take into account the aggregate expressions already added in `aggExprList`? For example
```sql
select c1 from t1 group by c1 having sum(c2) > 0 and sum(c2) < 10
```
Here `sum(c2)` will be added to the exprList and we don't need to create another alias for the second `sum(c2)`. Maybe use a aggExpr map instead of list?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
##########
@@ -325,8 +325,6 @@ object PullupCorrelatedPredicates extends Rule[LogicalPlan] with PredicateHelper
*/
def apply(plan: LogicalPlan): LogicalPlan = plan.transformUpWithPruning(
_.containsPattern(PLAN_EXPRESSION)) {
- case f @ Filter(_, a: Aggregate) =>
Review comment:
This is not related to this PR but I just thought about the following case:
```sql
-- t1: [c1, c2]: (0,1),(1,2) , t: [a, b]: (0,1),(1,2)
select sum(c2) as c1 from t1 group by c1 having (select sum(b) from t where c1 = a) > 0
```
Because ResolveSubquery comes before ResolveAggregateFunction, this `c1` in `c1 = a` will be resolved as `sum(c2)` instead of `t1.c1`, because `UnresolvedHaving` is also a UnaryNode which will be resolved under the UnaryNode case below. So the result here can be incorrect.
Here we need to exclude `UnresolvedHaving` from this rule.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -839,9 +839,7 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
def checkMixedReferencesInsideAggregateExpr(expr: Expression): Unit = {
expr.foreach {
case a: AggregateExpression if containsOuter(a) =>
- val outer = a.collect { case OuterReference(e) => e.toAttribute }
- val local = a.references -- outer
Review comment:
Good catch!
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
##########
@@ -325,8 +325,6 @@ object PullupCorrelatedPredicates extends Rule[LogicalPlan] with PredicateHelper
*/
def apply(plan: LogicalPlan): LogicalPlan = plan.transformUpWithPruning(
_.containsPattern(PLAN_EXPRESSION)) {
- case f @ Filter(_, a: Aggregate) =>
Review comment:
This is not related to this PR but I just thought about the following case:
```sql
-- t1: [c1, c2]: (0,1),(1,2) , t: [a, b]: (0,1),(1,2)
select sum(c2) as c1 from t1 group by c1 having (select sum(b) from t where c1 = a) > 0
```
Because ResolveSubquery comes before ResolveAggregateFunction, this `c1` in `c1 = a` will be resolved as `sum(c2)` instead of `t1.c1`, because `UnresolvedHaving` is also a UnaryNode which will be resolved under the UnaryNode case below. So the result here can be incorrect.
Here we need to exclude `UnresolvedHaving` from this rule.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2378,10 +2375,6 @@ class Analyzer(override val catalogManager: CatalogManager)
*/
def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUpWithPruning(
_.containsPattern(PLAN_EXPRESSION), ruleId) {
- // In case of HAVING (a filter after an aggregate) we use both the aggregate and
- // its child for resolution.
- case f @ Filter(_, a: Aggregate) if f.childrenResolved =>
Review comment:
This is not related to this PR but I just thought about the following case:
```sql
-- t1: [c1, c2]: (0,1),(1,2) , t: [a, b]: (0,1),(1,2)
select sum(c2) as c1 from t1 group by c1 having (select sum(b) from t where c1 = a) > 0
```
Because ResolveSubquery comes before ResolveAggregateFunction, this `c1` in `c1 = a` will be resolved as `sum(c2)` instead of `t1.c1`, because UnresolvedHaving is also a UnaryNode which will be resolved under the UnaryNode case below. So the result here can be incorrect.
Here we need to exclude UnresolvedHaving from this rule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858424090
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139612/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-835187278
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32470:
URL: https://github.com/apache/spark/pull/32470#discussion_r649008968
##########
File path: sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q35.sf100/explain.txt
##########
@@ -306,11 +306,11 @@ Input [19]: [ca_state#25, cd_gender#29, cd_marital_status#30, cd_dep_count#31, c
Keys [6]: [ca_state#25, cd_gender#29, cd_marital_status#30, cd_dep_count#31, cd_dep_employed_count#32, cd_dep_college_count#33]
Functions [10]: [count(1), min(cd_dep_count#31), max(cd_dep_count#31), avg(cd_dep_count#31), min(cd_dep_employed_count#32), max(cd_dep_employed_count#32), avg(cd_dep_employed_count#32), min(cd_dep_college_count#33), max(cd_dep_college_count#33), avg(cd_dep_college_count#33)]
Aggregate Attributes [10]: [count(1)#62, min(cd_dep_count#31)#63, max(cd_dep_count#31)#64, avg(cd_dep_count#31)#65, min(cd_dep_employed_count#32)#66, max(cd_dep_employed_count#32)#67, avg(cd_dep_employed_count#32)#68, min(cd_dep_college_count#33)#69, max(cd_dep_college_count#33)#70, avg(cd_dep_college_count#33)#71]
-Results [18]: [ca_state#25, cd_gender#29, cd_marital_status#30, count(1)#62 AS cnt1#72, min(cd_dep_count#31)#63 AS min(cd_dep_count)#73, max(cd_dep_count#31)#64 AS max(cd_dep_count)#74, avg(cd_dep_count#31)#65 AS avg(cd_dep_count)#75, cd_dep_employed_count#32, count(1)#62 AS cnt2#76, min(cd_dep_employed_count#32)#66 AS min(cd_dep_employed_count)#77, max(cd_dep_employed_count#32)#67 AS max(cd_dep_employed_count)#78, avg(cd_dep_employed_count#32)#68 AS avg(cd_dep_employed_count)#79, cd_dep_college_count#33, count(1)#62 AS cnt3#80, min(cd_dep_college_count#33)#69 AS min(cd_dep_college_count)#81, max(cd_dep_college_count#33)#70 AS max(cd_dep_college_count)#82, avg(cd_dep_college_count#33)#71 AS avg(cd_dep_college_count)#83, cd_dep_count#31 AS aggOrder#84]
Review comment:
the only change is the removed alias
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32470:
URL: https://github.com/apache/spark/pull/32470#discussion_r649026895
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveGroupingAnalyticsSuite.scala
##########
@@ -287,9 +287,9 @@ class ResolveGroupingAnalyticsSuite extends AnalysisTest {
Seq(GroupingSets(Seq(Seq(), Seq(unresolved_a), Seq(unresolved_a, unresolved_b)))),
Seq(unresolved_a, unresolved_b), r1))
val expected = Project(Seq(a, b), Sort(
- Seq(SortOrder('aggOrder.byte.withNullability(false), Ascending)), true,
+ Seq(SortOrder(grouping_a, Ascending)), true,
Aggregate(Seq(a, b, gid),
- Seq(a, b, grouping_a.as("aggOrder")),
+ Seq(a, b, gid),
Review comment:
This is a more correct plan. If you look at other tests in this file, `test("grouping function")` and `test("filter with grouping function")`, they both expect plans to group by `gid` first, then calculate the grouping function (which is `grouping_a`) in agg list or the filter node above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858467500
**[Test build #139629 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139629/testReport)** for PR 32470 at commit [`ae4d411`](https://github.com/apache/spark/commit/ae4d411bd68fcd9daf4db23a7a33b4fdca478528).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32470:
URL: https://github.com/apache/spark/pull/32470#discussion_r650982808
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
##########
@@ -652,3 +652,15 @@ case object UnresolvedSeed extends LeafExpression with Unevaluable {
override def dataType: DataType = throw new UnresolvedException("dataType")
override lazy val resolved = false
}
+
+/**
+ * An intermediate expression to hold a resolved (nested) column. Some rules may need to undo the
+ * column resolution and use this expression to keep the original column name.
+ */
+case class TempResolvedColumn(child: Expression, nameParts: Seq[String]) extends UnaryExpression
Review comment:
Yes, I'll refine the type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-861402281
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44335/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-835228683
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138281/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-834772529
**[Test build #138261 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138261/testReport)** for PR 32470 at commit [`a8c708f`](https://github.com/apache/spark/commit/a8c708f0e5dbae0cd6b2f3b8a9b6917207d8dca1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-835207742
**[Test build #138281 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138281/testReport)** for PR 32470 at commit [`c67a801`](https://github.com/apache/spark/commit/c67a8013d6f848e7ee0227e50f5fec499d8648dc).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-861573585
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139808/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-859763809
It's a bit hacky to restore an `UnresolvedAttribute` from an `AttributeReference`, I introduced `TempResolvedColumn` and updated the PR description accordingly. Please take another look, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858775626
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858871707
**[Test build #139641 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139641/testReport)** for PR 32470 at commit [`21353f3`](https://github.com/apache/spark/commit/21353f37288d4daa10bb6715a97f07e5326be618).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-836904086
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138336/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-836280820
**[Test build #138326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138326/testReport)** for PR 32470 at commit [`dbc00c8`](https://github.com/apache/spark/commit/dbc00c8e40f397447f4d75c7a16c33fb62eb5112).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-861365825
**[Test build #139808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139808/testReport)** for PR 32470 at commit [`b362a09`](https://github.com/apache/spark/commit/b362a098978be65ab1fc033fe0213a78a467b6ae).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-836757567
**[Test build #138336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138336/testReport)** for PR 32470 at commit [`ab31ab1`](https://github.com/apache/spark/commit/ab31ab1b6312da694e9964fbc94c5d7842bf5797).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-834711131
**[Test build #138261 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138261/testReport)** for PR 32470 at commit [`a8c708f`](https://github.com/apache/spark/commit/a8c708f0e5dbae0cd6b2f3b8a9b6917207d8dca1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-836347015
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42848/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-834778266
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138261/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-834744895
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42783/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32470:
URL: https://github.com/apache/spark/pull/32470#discussion_r649223245
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -839,9 +839,7 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
def checkMixedReferencesInsideAggregateExpr(expr: Expression): Unit = {
expr.foreach {
case a: AggregateExpression if containsOuter(a) =>
- val outer = a.collect { case OuterReference(e) => e.toAttribute }
- val local = a.references -- outer
Review comment:
`Expression.references` doesn't count outer refs, so this is unnecessary. It can miss some cases if the outer ref is the same as local ref (e.g. the outer query and inner query select from the same table)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858252267
**[Test build #139609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139609/testReport)** for PR 32470 at commit [`e33c0a8`](https://github.com/apache/spark/commit/e33c0a87388d587f18f1e7e21b9b2170dab7b695).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-859311626
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858680426
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139645/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858635960
**[Test build #139641 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139641/testReport)** for PR 32470 at commit [`21353f3`](https://github.com/apache/spark/commit/21353f37288d4daa10bb6715a97f07e5326be618).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-836823210
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42858/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-861573585
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139808/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858471325
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139629/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858347812
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44137/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-836890194
**[Test build #138336 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138336/testReport)** for PR 32470 at commit [`ab31ab1`](https://github.com/apache/spark/commit/ab31ab1b6312da694e9964fbc94c5d7842bf5797).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858542182
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44156/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858252747
**[Test build #139609 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139609/testReport)** for PR 32470 at commit [`e33c0a8`](https://github.com/apache/spark/commit/e33c0a87388d587f18f1e7e21b9b2170dab7b695).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-862028396
@viirya @maropu Can you take one more look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858721844
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32470:
URL: https://github.com/apache/spark/pull/32470#discussion_r650251603
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
##########
@@ -652,3 +652,15 @@ case object UnresolvedSeed extends LeafExpression with Unevaluable {
override def dataType: DataType = throw new UnresolvedException("dataType")
override lazy val resolved = false
}
+
+/**
+ * An intermediate expression to hold a resolved (nested) column. Some rules may need to undo the
+ * column resolution and use this expression to keep the original column name.
+ */
+case class TempResolvedColumn(child: Expression, nameParts: Seq[String]) extends UnaryExpression
Review comment:
Is its child always a named expression?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2457,164 +2451,133 @@ class Analyzer(override val catalogManager: CatalogManager)
_.containsPattern(AGGREGATE), ruleId) {
// Resolve aggregate with having clause to Filter(..., Aggregate()). Note, to avoid wrongly
// resolve the having condition expression, here we skip resolving it in ResolveReferences
- // and transform it to Filter after aggregate is resolved. See more details in SPARK-31519.
+ // and transform it to Filter after aggregate is resolved. Basically columns in HAVING should
+ // be resolved with `agg.child.output` first. See more details in SPARK-31519.
case UnresolvedHaving(cond, agg: Aggregate) if agg.resolved =>
- resolveHaving(Filter(cond, agg), agg)
-
- case f @ Filter(_, agg: Aggregate) if agg.resolved =>
- resolveHaving(f, agg)
-
- case sort @ Sort(sortOrder, global, aggregate: Aggregate) if aggregate.resolved =>
-
- // Try resolving the ordering as though it is in the aggregate clause.
- try {
- // If a sort order is unresolved, containing references not in aggregate, or containing
- // `AggregateExpression`, we need to push down it to the underlying aggregate operator.
- val unresolvedSortOrders = sortOrder.filter { s =>
- !s.resolved || !s.references.subsetOf(aggregate.outputSet) || containsAggregate(s)
- }
- val aliasedOrdering = unresolvedSortOrders.map(o => Alias(o.child, "aggOrder")())
-
- val aggregateWithExtraOrdering = aggregate.copy(
- aggregateExpressions = aggregate.aggregateExpressions ++ aliasedOrdering)
-
- val resolvedAggregate: Aggregate =
- executeSameContext(aggregateWithExtraOrdering).asInstanceOf[Aggregate]
-
- val (reResolvedAggExprs, resolvedAliasedOrdering) =
- resolvedAggregate.aggregateExpressions.splitAt(aggregate.aggregateExpressions.length)
-
- // If we pass the analysis check, then the ordering expressions should only reference to
- // aggregate expressions or grouping expressions, and it's safe to push them down to
- // Aggregate.
- checkAnalysis(resolvedAggregate)
-
- val originalAggExprs = aggregate.aggregateExpressions.map(trimNonTopLevelAliases)
-
- // If the ordering expression is same with original aggregate expression, we don't need
- // to push down this ordering expression and can reference the original aggregate
- // expression instead.
- val needsPushDown = ArrayBuffer.empty[NamedExpression]
- val orderToAlias = unresolvedSortOrders.zip(aliasedOrdering)
- val evaluatedOrderings =
- resolvedAliasedOrdering.asInstanceOf[Seq[Alias]].zip(orderToAlias).map {
- case (evaluated, (order, aliasOrder)) =>
- val index = reResolvedAggExprs.indexWhere {
- case Alias(child, _) => child semanticEquals evaluated.child
- case other => other semanticEquals evaluated.child
- }
-
- if (index == -1) {
- if (hasCharVarchar(evaluated)) {
- needsPushDown += aliasOrder
- order.copy(child = aliasOrder)
- } else {
- needsPushDown += evaluated
- order.copy(child = evaluated.toAttribute)
- }
- } else {
- order.copy(child = originalAggExprs(index).toAttribute)
- }
+ resolveOperatorWithAggregate(Seq(cond), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Filter(cond, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = resolveExpressionByPlanOutput(cond, agg)
Review comment:
> We should resolve the references normally based on child.output first.
But you resolve them using `agg`?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2457,164 +2451,133 @@ class Analyzer(override val catalogManager: CatalogManager)
_.containsPattern(AGGREGATE), ruleId) {
// Resolve aggregate with having clause to Filter(..., Aggregate()). Note, to avoid wrongly
// resolve the having condition expression, here we skip resolving it in ResolveReferences
- // and transform it to Filter after aggregate is resolved. See more details in SPARK-31519.
+ // and transform it to Filter after aggregate is resolved. Basically columns in HAVING should
+ // be resolved with `agg.child.output` first. See more details in SPARK-31519.
case UnresolvedHaving(cond, agg: Aggregate) if agg.resolved =>
- resolveHaving(Filter(cond, agg), agg)
-
- case f @ Filter(_, agg: Aggregate) if agg.resolved =>
- resolveHaving(f, agg)
-
- case sort @ Sort(sortOrder, global, aggregate: Aggregate) if aggregate.resolved =>
-
- // Try resolving the ordering as though it is in the aggregate clause.
- try {
- // If a sort order is unresolved, containing references not in aggregate, or containing
- // `AggregateExpression`, we need to push down it to the underlying aggregate operator.
- val unresolvedSortOrders = sortOrder.filter { s =>
- !s.resolved || !s.references.subsetOf(aggregate.outputSet) || containsAggregate(s)
- }
- val aliasedOrdering = unresolvedSortOrders.map(o => Alias(o.child, "aggOrder")())
-
- val aggregateWithExtraOrdering = aggregate.copy(
- aggregateExpressions = aggregate.aggregateExpressions ++ aliasedOrdering)
-
- val resolvedAggregate: Aggregate =
- executeSameContext(aggregateWithExtraOrdering).asInstanceOf[Aggregate]
-
- val (reResolvedAggExprs, resolvedAliasedOrdering) =
- resolvedAggregate.aggregateExpressions.splitAt(aggregate.aggregateExpressions.length)
-
- // If we pass the analysis check, then the ordering expressions should only reference to
- // aggregate expressions or grouping expressions, and it's safe to push them down to
- // Aggregate.
- checkAnalysis(resolvedAggregate)
-
- val originalAggExprs = aggregate.aggregateExpressions.map(trimNonTopLevelAliases)
-
- // If the ordering expression is same with original aggregate expression, we don't need
- // to push down this ordering expression and can reference the original aggregate
- // expression instead.
- val needsPushDown = ArrayBuffer.empty[NamedExpression]
- val orderToAlias = unresolvedSortOrders.zip(aliasedOrdering)
- val evaluatedOrderings =
- resolvedAliasedOrdering.asInstanceOf[Seq[Alias]].zip(orderToAlias).map {
- case (evaluated, (order, aliasOrder)) =>
- val index = reResolvedAggExprs.indexWhere {
- case Alias(child, _) => child semanticEquals evaluated.child
- case other => other semanticEquals evaluated.child
- }
-
- if (index == -1) {
- if (hasCharVarchar(evaluated)) {
- needsPushDown += aliasOrder
- order.copy(child = aliasOrder)
- } else {
- needsPushDown += evaluated
- order.copy(child = evaluated.toAttribute)
- }
- } else {
- order.copy(child = originalAggExprs(index).toAttribute)
- }
+ resolveOperatorWithAggregate(Seq(cond), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Filter(cond, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = resolveExpressionByPlanOutput(cond, agg)
Review comment:
Do you mean to call `resolveExprsWithAggregate` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-859033403
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139652/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-836823210
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42858/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858777838
**[Test build #139652 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139652/testReport)** for PR 32470 at commit [`1e973ef`](https://github.com/apache/spark/commit/1e973ef66b445e3d70df92a52b39e99f7c491151).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858252267
**[Test build #139609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139609/testReport)** for PR 32470 at commit [`e33c0a8`](https://github.com/apache/spark/commit/e33c0a87388d587f18f1e7e21b9b2170dab7b695).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858505961
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44156/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-835193655
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42803/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-836280820
**[Test build #138326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138326/testReport)** for PR 32470 at commit [`dbc00c8`](https://github.com/apache/spark/commit/dbc00c8e40f397447f4d75c7a16c33fb62eb5112).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-836806244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-837836239
**[Test build #138358 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138358/testReport)** for PR 32470 at commit [`c0bb807`](https://github.com/apache/spark/commit/c0bb8070cbb52f9a20da0c5e3e791db72ea4bf04).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858290812
**[Test build #139612 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139612/testReport)** for PR 32470 at commit [`d08d8e4`](https://github.com/apache/spark/commit/d08d8e4cb6acfaff34e3d81cc8b41c65aa34f4f6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-834744895
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42783/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858865055
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44180/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32470:
URL: https://github.com/apache/spark/pull/32470#discussion_r649223245
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -839,9 +839,7 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
def checkMixedReferencesInsideAggregateExpr(expr: Expression): Unit = {
expr.foreach {
case a: AggregateExpression if containsOuter(a) =>
- val outer = a.collect { case OuterReference(e) => e.toAttribute }
- val local = a.references -- outer
Review comment:
`Expression.references` doesn't count outer refs, so this is unnecessary. It can cause problems if the outer ref is the same as local ref (e.g. the outer query and inner query select from the same table)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858673855
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44169/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858680426
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139645/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-861437561
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44335/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-861547402
**[Test build #139808 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139808/testReport)** for PR 32470 at commit [`b362a09`](https://github.com/apache/spark/commit/b362a098978be65ab1fc033fe0213a78a467b6ae).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-859005261
**[Test build #139652 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139652/testReport)** for PR 32470 at commit [`1e973ef`](https://github.com/apache/spark/commit/1e973ef66b445e3d70df92a52b39e99f7c491151).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-836346923
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32470:
URL: https://github.com/apache/spark/pull/32470#discussion_r650981865
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2457,164 +2451,133 @@ class Analyzer(override val catalogManager: CatalogManager)
_.containsPattern(AGGREGATE), ruleId) {
// Resolve aggregate with having clause to Filter(..., Aggregate()). Note, to avoid wrongly
// resolve the having condition expression, here we skip resolving it in ResolveReferences
- // and transform it to Filter after aggregate is resolved. See more details in SPARK-31519.
+ // and transform it to Filter after aggregate is resolved. Basically columns in HAVING should
+ // be resolved with `agg.child.output` first. See more details in SPARK-31519.
case UnresolvedHaving(cond, agg: Aggregate) if agg.resolved =>
- resolveHaving(Filter(cond, agg), agg)
-
- case f @ Filter(_, agg: Aggregate) if agg.resolved =>
- resolveHaving(f, agg)
-
- case sort @ Sort(sortOrder, global, aggregate: Aggregate) if aggregate.resolved =>
-
- // Try resolving the ordering as though it is in the aggregate clause.
- try {
- // If a sort order is unresolved, containing references not in aggregate, or containing
- // `AggregateExpression`, we need to push down it to the underlying aggregate operator.
- val unresolvedSortOrders = sortOrder.filter { s =>
- !s.resolved || !s.references.subsetOf(aggregate.outputSet) || containsAggregate(s)
- }
- val aliasedOrdering = unresolvedSortOrders.map(o => Alias(o.child, "aggOrder")())
-
- val aggregateWithExtraOrdering = aggregate.copy(
- aggregateExpressions = aggregate.aggregateExpressions ++ aliasedOrdering)
-
- val resolvedAggregate: Aggregate =
- executeSameContext(aggregateWithExtraOrdering).asInstanceOf[Aggregate]
-
- val (reResolvedAggExprs, resolvedAliasedOrdering) =
- resolvedAggregate.aggregateExpressions.splitAt(aggregate.aggregateExpressions.length)
-
- // If we pass the analysis check, then the ordering expressions should only reference to
- // aggregate expressions or grouping expressions, and it's safe to push them down to
- // Aggregate.
- checkAnalysis(resolvedAggregate)
-
- val originalAggExprs = aggregate.aggregateExpressions.map(trimNonTopLevelAliases)
-
- // If the ordering expression is same with original aggregate expression, we don't need
- // to push down this ordering expression and can reference the original aggregate
- // expression instead.
- val needsPushDown = ArrayBuffer.empty[NamedExpression]
- val orderToAlias = unresolvedSortOrders.zip(aliasedOrdering)
- val evaluatedOrderings =
- resolvedAliasedOrdering.asInstanceOf[Seq[Alias]].zip(orderToAlias).map {
- case (evaluated, (order, aliasOrder)) =>
- val index = reResolvedAggExprs.indexWhere {
- case Alias(child, _) => child semanticEquals evaluated.child
- case other => other semanticEquals evaluated.child
- }
-
- if (index == -1) {
- if (hasCharVarchar(evaluated)) {
- needsPushDown += aliasOrder
- order.copy(child = aliasOrder)
- } else {
- needsPushDown += evaluated
- order.copy(child = evaluated.toAttribute)
- }
- } else {
- order.copy(child = originalAggExprs(index).toAttribute)
- }
+ resolveOperatorWithAggregate(Seq(cond), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Filter(cond, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = resolveExpressionByPlanOutput(cond, agg)
Review comment:
`child` means the child of `Filter`, which is `agg`. I'll refine the comment a bit more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858396906
**[Test build #139612 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139612/testReport)** for PR 32470 at commit [`d08d8e4`](https://github.com/apache/spark/commit/d08d8e4cb6acfaff34e3d81cc8b41c65aa34f4f6).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32470:
URL: https://github.com/apache/spark/pull/32470#discussion_r649009772
##########
File path: sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q6.sf100/explain.txt
##########
@@ -267,15 +267,15 @@ Input [2]: [ca_state#26, count#33]
Keys [1]: [ca_state#26]
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#35]
-Results [4]: [ca_state#26 AS state#36, count(1)#35 AS cnt#37, count(1)#35 AS count(1)#38, ca_state#26]
+Results [4]: [ca_state#26 AS state#36, count(1)#35 AS cnt#37, ca_state#26, count(1)#35 AS count(1)#38]
Review comment:
only the order changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858326389
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44137/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858703800
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44169/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-835193655
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42803/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858287872
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44136/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32470:
URL: https://github.com/apache/spark/pull/32470#discussion_r651629291
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
##########
@@ -652,3 +652,15 @@ case object UnresolvedSeed extends LeafExpression with Unevaluable {
override def dataType: DataType = throw new UnresolvedException("dataType")
override lazy val resolved = false
}
+
+/**
+ * An intermediate expression to hold a resolved (nested) column. Some rules may need to undo the
+ * column resolution and use this expression to keep the original column name.
+ */
+case class TempResolvedColumn(child: Expression, nameParts: Seq[String]) extends UnaryExpression
Review comment:
Actually, we don't require it to be a named expression, so it's more robust to keep the type as `Expression`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-838127291
**[Test build #138358 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138358/testReport)** for PR 32470 at commit [`c0bb807`](https://github.com/apache/spark/commit/c0bb8070cbb52f9a20da0c5e3e791db72ea4bf04).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858252760
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139609/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-836344091
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138326/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858252760
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139609/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858309113
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44137/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-835159651
**[Test build #138281 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138281/testReport)** for PR 32470 at commit [`c67a801`](https://github.com/apache/spark/commit/c67a8013d6f848e7ee0227e50f5fec499d8648dc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858314556
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44136/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858680360
**[Test build #139645 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139645/testReport)** for PR 32470 at commit [`1e973ef`](https://github.com/apache/spark/commit/1e973ef66b445e3d70df92a52b39e99f7c491151).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858675146
**[Test build #139645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139645/testReport)** for PR 32470 at commit [`1e973ef`](https://github.com/apache/spark/commit/1e973ef66b445e3d70df92a52b39e99f7c491151).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858424090
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139612/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a change in pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on a change in pull request #32470:
URL: https://github.com/apache/spark/pull/32470#discussion_r650734143
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2457,164 +2451,133 @@ class Analyzer(override val catalogManager: CatalogManager)
_.containsPattern(AGGREGATE), ruleId) {
// Resolve aggregate with having clause to Filter(..., Aggregate()). Note, to avoid wrongly
// resolve the having condition expression, here we skip resolving it in ResolveReferences
- // and transform it to Filter after aggregate is resolved. See more details in SPARK-31519.
+ // and transform it to Filter after aggregate is resolved. Basically columns in HAVING should
+ // be resolved with `agg.child.output` first. See more details in SPARK-31519.
case UnresolvedHaving(cond, agg: Aggregate) if agg.resolved =>
- resolveHaving(Filter(cond, agg), agg)
-
- case f @ Filter(_, agg: Aggregate) if agg.resolved =>
- resolveHaving(f, agg)
-
- case sort @ Sort(sortOrder, global, aggregate: Aggregate) if aggregate.resolved =>
-
- // Try resolving the ordering as though it is in the aggregate clause.
- try {
- // If a sort order is unresolved, containing references not in aggregate, or containing
- // `AggregateExpression`, we need to push down it to the underlying aggregate operator.
- val unresolvedSortOrders = sortOrder.filter { s =>
- !s.resolved || !s.references.subsetOf(aggregate.outputSet) || containsAggregate(s)
- }
- val aliasedOrdering = unresolvedSortOrders.map(o => Alias(o.child, "aggOrder")())
-
- val aggregateWithExtraOrdering = aggregate.copy(
- aggregateExpressions = aggregate.aggregateExpressions ++ aliasedOrdering)
-
- val resolvedAggregate: Aggregate =
- executeSameContext(aggregateWithExtraOrdering).asInstanceOf[Aggregate]
-
- val (reResolvedAggExprs, resolvedAliasedOrdering) =
- resolvedAggregate.aggregateExpressions.splitAt(aggregate.aggregateExpressions.length)
-
- // If we pass the analysis check, then the ordering expressions should only reference to
- // aggregate expressions or grouping expressions, and it's safe to push them down to
- // Aggregate.
- checkAnalysis(resolvedAggregate)
-
- val originalAggExprs = aggregate.aggregateExpressions.map(trimNonTopLevelAliases)
-
- // If the ordering expression is same with original aggregate expression, we don't need
- // to push down this ordering expression and can reference the original aggregate
- // expression instead.
- val needsPushDown = ArrayBuffer.empty[NamedExpression]
- val orderToAlias = unresolvedSortOrders.zip(aliasedOrdering)
- val evaluatedOrderings =
- resolvedAliasedOrdering.asInstanceOf[Seq[Alias]].zip(orderToAlias).map {
- case (evaluated, (order, aliasOrder)) =>
- val index = reResolvedAggExprs.indexWhere {
- case Alias(child, _) => child semanticEquals evaluated.child
- case other => other semanticEquals evaluated.child
- }
-
- if (index == -1) {
- if (hasCharVarchar(evaluated)) {
- needsPushDown += aliasOrder
- order.copy(child = aliasOrder)
- } else {
- needsPushDown += evaluated
- order.copy(child = evaluated.toAttribute)
- }
- } else {
- order.copy(child = originalAggExprs(index).toAttribute)
- }
+ resolveOperatorWithAggregate(Seq(cond), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Filter(cond, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = resolveExpressionByPlanOutput(cond, agg)
+ resolveOperatorWithAggregate(Seq(maybeResolved), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Sort(sortOrder, global, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = sortOrder.map(_.child).map(resolveExpressionByPlanOutput(_, agg))
+ resolveOperatorWithAggregate(maybeResolved, agg, (newExprs, newChild) => {
+ val newSortOrder = sortOrder.zip(newExprs).map {
+ case (sortOrder, expr) => sortOrder.copy(child = expr)
}
+ Sort(newSortOrder, global, newChild)
+ })
+ }
- val sortOrdersMap = unresolvedSortOrders
- .map(new TreeNodeRef(_))
- .zip(evaluatedOrderings)
- .toMap
- val finalSortOrders = sortOrder.map(s => sortOrdersMap.getOrElse(new TreeNodeRef(s), s))
+ /**
+ * Resolves the given expressions as if they are in the given Aggregate operator, which means
+ * the column can be resolved using `agg.child` and aggregate functions/grouping columns are
+ * allowed. It returns a list of named expressions that need to be appended to
+ * `agg.aggregateExpressions`, and the list of resolved expressions.
+ */
+ def resolveExprsWithAggregate(
+ exprs: Seq[Expression],
+ agg: Aggregate): (Seq[NamedExpression], Seq[Expression]) = {
+ def resolveCol(input: Expression): Expression = {
+ input.transform {
+ case u: UnresolvedAttribute =>
+ try {
+ // Resolve the column and wrap it with `TempResolvedColumn`. If the resolved column
+ // doesn't end up with as aggregate function input or grouping column, we should
+ // undo the column resolution to avoid confusing error message. For example, if
+ // a table `t` has two columns `c1` and `c2`, for query `SELECT ... FROM t
+ // GROUP BY c1 HAVING c2 = 0`, even though we can resolve column `c2` here, we
+ // should undo it later and fail with "Column c2 not found".
+ agg.child.resolve(u.nameParts, resolver).map(TempResolvedColumn(_, u.nameParts))
+ .getOrElse(u)
+ } catch {
+ case _: AnalysisException => u
+ }
+ }
+ }
- // Since we don't rely on sort.resolved as the stop condition for this rule,
- // we need to check this and prevent applying this rule multiple times
- if (sortOrder == finalSortOrders) {
- sort
- } else {
- Project(aggregate.output,
- Sort(finalSortOrders, global,
- aggregate.copy(aggregateExpressions = originalAggExprs ++ needsPushDown)))
- }
- } catch {
- // Attempting to resolve in the aggregate can result in ambiguity. When this happens,
- // just return the original plan.
- case ae: AnalysisException => sort
+ def resolveSubQuery(input: Expression): Expression = {
+ if (SubqueryExpression.hasSubquery(input)) {
+ val fake = Project(Alias(input, "fake")() :: Nil, agg.child)
+ ResolveSubquery(fake).asInstanceOf[Project].projectList.head.asInstanceOf[Alias].child
Review comment:
The children of the resolved subquery that are not aggregate expressions should also be wrapped in `TempResolvedColumn` (but it seems the current logic can't handle this case with proper error messages either. This can be a follow-up in the future)
```sql
select c1 from t1 group by c1 having (select sum(c2) from t2 where t1.c2 = t2.c1) > 0
-- org.apache.spark.sql.AnalysisException: Resolved attribute(s) c2#10 missing from c1#9 in operator
-- !Filter (scalar-subquery#8 [c2#10] > cast(0 as bigint)).;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858299573
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44136/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-859239440
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858527171
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44156/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32470:
URL: https://github.com/apache/spark/pull/32470#discussion_r649009205
##########
File path: sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q35/explain.txt
##########
@@ -251,11 +251,11 @@ Input [19]: [ca_state#23, cd_gender#26, cd_marital_status#27, cd_dep_count#28, c
Keys [6]: [ca_state#23, cd_gender#26, cd_marital_status#27, cd_dep_count#28, cd_dep_employed_count#29, cd_dep_college_count#30]
Functions [10]: [count(1), min(cd_dep_count#28), max(cd_dep_count#28), avg(cd_dep_count#28), min(cd_dep_employed_count#29), max(cd_dep_employed_count#29), avg(cd_dep_employed_count#29), min(cd_dep_college_count#30), max(cd_dep_college_count#30), avg(cd_dep_college_count#30)]
Aggregate Attributes [10]: [count(1)#59, min(cd_dep_count#28)#60, max(cd_dep_count#28)#61, avg(cd_dep_count#28)#62, min(cd_dep_employed_count#29)#63, max(cd_dep_employed_count#29)#64, avg(cd_dep_employed_count#29)#65, min(cd_dep_college_count#30)#66, max(cd_dep_college_count#30)#67, avg(cd_dep_college_count#30)#68]
-Results [18]: [ca_state#23, cd_gender#26, cd_marital_status#27, count(1)#59 AS cnt1#69, min(cd_dep_count#28)#60 AS min(cd_dep_count)#70, max(cd_dep_count#28)#61 AS max(cd_dep_count)#71, avg(cd_dep_count#28)#62 AS avg(cd_dep_count)#72, cd_dep_employed_count#29, count(1)#59 AS cnt2#73, min(cd_dep_employed_count#29)#63 AS min(cd_dep_employed_count)#74, max(cd_dep_employed_count#29)#64 AS max(cd_dep_employed_count)#75, avg(cd_dep_employed_count#29)#65 AS avg(cd_dep_employed_count)#76, cd_dep_college_count#30, count(1)#59 AS cnt3#77, min(cd_dep_college_count#30)#66 AS min(cd_dep_college_count)#78, max(cd_dep_college_count#30)#67 AS max(cd_dep_college_count)#79, avg(cd_dep_college_count#30)#68 AS avg(cd_dep_college_count)#80, cd_dep_count#28 AS aggOrder#81]
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-834778266
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138261/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858635960
**[Test build #139641 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139641/testReport)** for PR 32470 at commit [`21353f3`](https://github.com/apache/spark/commit/21353f37288d4daa10bb6715a97f07e5326be618).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32470:
URL: https://github.com/apache/spark/pull/32470#discussion_r649661183
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2457,164 +2450,127 @@ class Analyzer(override val catalogManager: CatalogManager)
_.containsPattern(AGGREGATE), ruleId) {
// Resolve aggregate with having clause to Filter(..., Aggregate()). Note, to avoid wrongly
// resolve the having condition expression, here we skip resolving it in ResolveReferences
- // and transform it to Filter after aggregate is resolved. See more details in SPARK-31519.
+ // and transform it to Filter after aggregate is resolved. Basically columns in HAVING should
+ // be resolved with `agg.child.output` first. See more details in SPARK-31519.
case UnresolvedHaving(cond, agg: Aggregate) if agg.resolved =>
- resolveHaving(Filter(cond, agg), agg)
-
- case f @ Filter(_, agg: Aggregate) if agg.resolved =>
- resolveHaving(f, agg)
-
- case sort @ Sort(sortOrder, global, aggregate: Aggregate) if aggregate.resolved =>
-
- // Try resolving the ordering as though it is in the aggregate clause.
- try {
- // If a sort order is unresolved, containing references not in aggregate, or containing
- // `AggregateExpression`, we need to push down it to the underlying aggregate operator.
- val unresolvedSortOrders = sortOrder.filter { s =>
- !s.resolved || !s.references.subsetOf(aggregate.outputSet) || containsAggregate(s)
+ resolveOperatorWithAggregate(Seq(cond), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Filter(cond, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = resolveExpressionByPlanOutput(cond, agg)
+ resolveOperatorWithAggregate(Seq(maybeResolved), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Sort(sortOrder, global, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = sortOrder.map(_.child).map(resolveExpressionByPlanOutput(_, agg))
+ resolveOperatorWithAggregate(maybeResolved, agg, (newExprs, newChild) => {
+ val newSortOrder = sortOrder.zip(newExprs).map {
+ case (sortOrder, expr) => sortOrder.copy(child = expr)
}
- val aliasedOrdering = unresolvedSortOrders.map(o => Alias(o.child, "aggOrder")())
-
- val aggregateWithExtraOrdering = aggregate.copy(
- aggregateExpressions = aggregate.aggregateExpressions ++ aliasedOrdering)
-
- val resolvedAggregate: Aggregate =
- executeSameContext(aggregateWithExtraOrdering).asInstanceOf[Aggregate]
-
- val (reResolvedAggExprs, resolvedAliasedOrdering) =
- resolvedAggregate.aggregateExpressions.splitAt(aggregate.aggregateExpressions.length)
-
- // If we pass the analysis check, then the ordering expressions should only reference to
- // aggregate expressions or grouping expressions, and it's safe to push them down to
- // Aggregate.
- checkAnalysis(resolvedAggregate)
-
- val originalAggExprs = aggregate.aggregateExpressions.map(trimNonTopLevelAliases)
-
- // If the ordering expression is same with original aggregate expression, we don't need
- // to push down this ordering expression and can reference the original aggregate
- // expression instead.
- val needsPushDown = ArrayBuffer.empty[NamedExpression]
- val orderToAlias = unresolvedSortOrders.zip(aliasedOrdering)
- val evaluatedOrderings =
- resolvedAliasedOrdering.asInstanceOf[Seq[Alias]].zip(orderToAlias).map {
- case (evaluated, (order, aliasOrder)) =>
- val index = reResolvedAggExprs.indexWhere {
- case Alias(child, _) => child semanticEquals evaluated.child
- case other => other semanticEquals evaluated.child
- }
-
- if (index == -1) {
- if (hasCharVarchar(evaluated)) {
- needsPushDown += aliasOrder
- order.copy(child = aliasOrder)
- } else {
- needsPushDown += evaluated
- order.copy(child = evaluated.toAttribute)
- }
- } else {
- order.copy(child = originalAggExprs(index).toAttribute)
- }
+ Sort(newSortOrder, global, newChild)
+ })
+ }
+
+ def resolveExprsWithAggregate(
+ exprs: Seq[Expression],
+ agg: Aggregate): (Seq[NamedExpression], Seq[Expression]) = {
+ val aggregateExpressions = ArrayBuffer.empty[NamedExpression]
+ val transformed = exprs.map { e =>
+ // Try resolving the expression as though it is in the aggregate clause.
+ def resolveCol(input: Expression): Expression = {
+ resolveExpressionByPlanOutput(input, agg.child)
+ }
+ def resolveSubQuery(input: Expression): Expression = {
+ if (SubqueryExpression.hasSubquery(input)) {
+ val fake = Project(Alias(input, "fake")() :: Nil, agg.child)
+ ResolveSubquery(fake).asInstanceOf[Project].projectList.head.asInstanceOf[Alias].child
+ } else {
+ input
}
-
- val sortOrdersMap = unresolvedSortOrders
- .map(new TreeNodeRef(_))
- .zip(evaluatedOrderings)
- .toMap
- val finalSortOrders = sortOrder.map(s => sortOrdersMap.getOrElse(new TreeNodeRef(s), s))
-
- // Since we don't rely on sort.resolved as the stop condition for this rule,
- // we need to check this and prevent applying this rule multiple times
- if (sortOrder == finalSortOrders) {
- sort
+ }
+ val maybeResolved = resolveSubQuery(resolveCol(e))
+ if (maybeResolved.resolved && maybeResolved.references.subsetOf(agg.outputSet) &&
+ !containsAggregate(maybeResolved)) {
+ // The given expression is valid and doesn't need extra resolution.
+ maybeResolved
+ } else if (containsUnresolvedFunc(maybeResolved)) {
+ // The given expression has unresolved functions which may be aggregate functions and we
+ // need to wait for other rules to resolve the functions first.
+ maybeResolved
+ } else {
+ // Avoid adding an extra aggregate expression if it's already present in
+ // `agg.aggregateExpressions`.
+ val index = if (maybeResolved.resolved) {
+ agg.aggregateExpressions.indexWhere {
+ case Alias(child, _) => child semanticEquals maybeResolved
+ case other => other semanticEquals maybeResolved
+ }
} else {
- Project(aggregate.output,
- Sort(finalSortOrders, global,
- aggregate.copy(aggregateExpressions = originalAggExprs ++ needsPushDown)))
+ -1
+ }
+ if (index >= 0) {
Review comment:
`val index = ...` this may return -1 if no match is found.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2457,164 +2450,127 @@ class Analyzer(override val catalogManager: CatalogManager)
_.containsPattern(AGGREGATE), ruleId) {
// Resolve aggregate with having clause to Filter(..., Aggregate()). Note, to avoid wrongly
// resolve the having condition expression, here we skip resolving it in ResolveReferences
- // and transform it to Filter after aggregate is resolved. See more details in SPARK-31519.
+ // and transform it to Filter after aggregate is resolved. Basically columns in HAVING should
+ // be resolved with `agg.child.output` first. See more details in SPARK-31519.
case UnresolvedHaving(cond, agg: Aggregate) if agg.resolved =>
- resolveHaving(Filter(cond, agg), agg)
-
- case f @ Filter(_, agg: Aggregate) if agg.resolved =>
- resolveHaving(f, agg)
-
- case sort @ Sort(sortOrder, global, aggregate: Aggregate) if aggregate.resolved =>
-
- // Try resolving the ordering as though it is in the aggregate clause.
- try {
- // If a sort order is unresolved, containing references not in aggregate, or containing
- // `AggregateExpression`, we need to push down it to the underlying aggregate operator.
- val unresolvedSortOrders = sortOrder.filter { s =>
- !s.resolved || !s.references.subsetOf(aggregate.outputSet) || containsAggregate(s)
+ resolveOperatorWithAggregate(Seq(cond), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Filter(cond, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = resolveExpressionByPlanOutput(cond, agg)
+ resolveOperatorWithAggregate(Seq(maybeResolved), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Sort(sortOrder, global, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = sortOrder.map(_.child).map(resolveExpressionByPlanOutput(_, agg))
+ resolveOperatorWithAggregate(maybeResolved, agg, (newExprs, newChild) => {
+ val newSortOrder = sortOrder.zip(newExprs).map {
+ case (sortOrder, expr) => sortOrder.copy(child = expr)
}
- val aliasedOrdering = unresolvedSortOrders.map(o => Alias(o.child, "aggOrder")())
-
- val aggregateWithExtraOrdering = aggregate.copy(
- aggregateExpressions = aggregate.aggregateExpressions ++ aliasedOrdering)
-
- val resolvedAggregate: Aggregate =
- executeSameContext(aggregateWithExtraOrdering).asInstanceOf[Aggregate]
-
- val (reResolvedAggExprs, resolvedAliasedOrdering) =
- resolvedAggregate.aggregateExpressions.splitAt(aggregate.aggregateExpressions.length)
-
- // If we pass the analysis check, then the ordering expressions should only reference to
- // aggregate expressions or grouping expressions, and it's safe to push them down to
- // Aggregate.
- checkAnalysis(resolvedAggregate)
-
- val originalAggExprs = aggregate.aggregateExpressions.map(trimNonTopLevelAliases)
-
- // If the ordering expression is same with original aggregate expression, we don't need
- // to push down this ordering expression and can reference the original aggregate
- // expression instead.
- val needsPushDown = ArrayBuffer.empty[NamedExpression]
- val orderToAlias = unresolvedSortOrders.zip(aliasedOrdering)
- val evaluatedOrderings =
- resolvedAliasedOrdering.asInstanceOf[Seq[Alias]].zip(orderToAlias).map {
- case (evaluated, (order, aliasOrder)) =>
- val index = reResolvedAggExprs.indexWhere {
- case Alias(child, _) => child semanticEquals evaluated.child
- case other => other semanticEquals evaluated.child
- }
-
- if (index == -1) {
- if (hasCharVarchar(evaluated)) {
- needsPushDown += aliasOrder
- order.copy(child = aliasOrder)
- } else {
- needsPushDown += evaluated
- order.copy(child = evaluated.toAttribute)
- }
- } else {
- order.copy(child = originalAggExprs(index).toAttribute)
- }
+ Sort(newSortOrder, global, newChild)
+ })
+ }
+
+ def resolveExprsWithAggregate(
+ exprs: Seq[Expression],
+ agg: Aggregate): (Seq[NamedExpression], Seq[Expression]) = {
+ val aggregateExpressions = ArrayBuffer.empty[NamedExpression]
+ val transformed = exprs.map { e =>
+ // Try resolving the expression as though it is in the aggregate clause.
+ def resolveCol(input: Expression): Expression = {
+ resolveExpressionByPlanOutput(input, agg.child)
+ }
+ def resolveSubQuery(input: Expression): Expression = {
+ if (SubqueryExpression.hasSubquery(input)) {
+ val fake = Project(Alias(input, "fake")() :: Nil, agg.child)
+ ResolveSubquery(fake).asInstanceOf[Project].projectList.head.asInstanceOf[Alias].child
+ } else {
+ input
}
-
- val sortOrdersMap = unresolvedSortOrders
- .map(new TreeNodeRef(_))
- .zip(evaluatedOrderings)
- .toMap
- val finalSortOrders = sortOrder.map(s => sortOrdersMap.getOrElse(new TreeNodeRef(s), s))
-
- // Since we don't rely on sort.resolved as the stop condition for this rule,
- // we need to check this and prevent applying this rule multiple times
- if (sortOrder == finalSortOrders) {
- sort
+ }
+ val maybeResolved = resolveSubQuery(resolveCol(e))
+ if (maybeResolved.resolved && maybeResolved.references.subsetOf(agg.outputSet) &&
+ !containsAggregate(maybeResolved)) {
+ // The given expression is valid and doesn't need extra resolution.
+ maybeResolved
+ } else if (containsUnresolvedFunc(maybeResolved)) {
+ // The given expression has unresolved functions which may be aggregate functions and we
+ // need to wait for other rules to resolve the functions first.
+ maybeResolved
+ } else {
+ // Avoid adding an extra aggregate expression if it's already present in
+ // `agg.aggregateExpressions`.
+ val index = if (maybeResolved.resolved) {
+ agg.aggregateExpressions.indexWhere {
+ case Alias(child, _) => child semanticEquals maybeResolved
+ case other => other semanticEquals maybeResolved
+ }
} else {
- Project(aggregate.output,
- Sort(finalSortOrders, global,
- aggregate.copy(aggregateExpressions = originalAggExprs ++ needsPushDown)))
+ -1
+ }
+ if (index >= 0) {
+ agg.aggregateExpressions(index).toAttribute
+ } else {
+ buildAggExprList(maybeResolved, agg, aggregateExpressions)
}
- } catch {
- // Attempting to resolve in the aggregate can result in ambiguity. When this happens,
- // just return the original plan.
- case ae: AnalysisException => sort
}
+ }
+ (aggregateExpressions.toSeq, transformed)
}
- def hasCharVarchar(expr: Alias): Boolean = {
- expr.find {
- case ne: NamedExpression => CharVarcharUtils.getRawType(ne.metadata).nonEmpty
- case _ => false
- }.nonEmpty
+ private def buildAggExprList(
+ expr: Expression,
+ agg: Aggregate,
+ aggExprList: ArrayBuffer[NamedExpression]): Expression = expr match {
+ case ae: AggregateExpression if ae.resolved =>
+ val alias = Alias(ae, ae.toString)()
+ aggExprList += alias
+ alias.toAttribute
+ // Grouping functions are handled in the rule [[ResolveGroupingAnalytics]].
+ case grouping: Expression if grouping.resolved &&
+ agg.groupingExpressions.exists(_.semanticEquals(grouping)) &&
+ !ResolveGroupingAnalytics.hasGroupingFunction(grouping) &&
+ !agg.output.exists(_.semanticEquals(grouping)) =>
+ grouping match {
+ case ne: NamedExpression =>
+ aggExprList += ne
+ ne.toAttribute
+ case _ =>
+ val alias = Alias(grouping, grouping.toString)()
+ aggExprList += alias
+ alias.toAttribute
+ }
+ case a: Attribute if agg.child.outputSet.contains(a) && !agg.outputSet.contains(a) =>
+ // Undo the resolution. This attribute is neither inside aggregate functions nor a
+ // grouping column. It shouldn't be resolved with `agg.child.output`.
+ CurrentOrigin.withOrigin(a.origin)(UnresolvedAttribute(Seq(a.name)))
Review comment:
Some tests failed because the error message is changed. This is to restore the previous error message, which is more pretty.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2378,10 +2375,6 @@ class Analyzer(override val catalogManager: CatalogManager)
*/
def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUpWithPruning(
_.containsPattern(PLAN_EXPRESSION), ruleId) {
- // In case of HAVING (a filter after an aggregate) we use both the aggregate and
- // its child for resolution.
- case f @ Filter(_, a: Aggregate) if f.childrenResolved =>
Review comment:
Good catch! Since it's an existing bug, let's fix it separately with a test.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2457,164 +2450,136 @@ class Analyzer(override val catalogManager: CatalogManager)
_.containsPattern(AGGREGATE), ruleId) {
// Resolve aggregate with having clause to Filter(..., Aggregate()). Note, to avoid wrongly
// resolve the having condition expression, here we skip resolving it in ResolveReferences
- // and transform it to Filter after aggregate is resolved. See more details in SPARK-31519.
+ // and transform it to Filter after aggregate is resolved. Basically columns in HAVING should
+ // be resolved with `agg.child.output` first. See more details in SPARK-31519.
case UnresolvedHaving(cond, agg: Aggregate) if agg.resolved =>
- resolveHaving(Filter(cond, agg), agg)
-
- case f @ Filter(_, agg: Aggregate) if agg.resolved =>
- resolveHaving(f, agg)
-
- case sort @ Sort(sortOrder, global, aggregate: Aggregate) if aggregate.resolved =>
-
- // Try resolving the ordering as though it is in the aggregate clause.
- try {
- // If a sort order is unresolved, containing references not in aggregate, or containing
- // `AggregateExpression`, we need to push down it to the underlying aggregate operator.
- val unresolvedSortOrders = sortOrder.filter { s =>
- !s.resolved || !s.references.subsetOf(aggregate.outputSet) || containsAggregate(s)
+ resolveOperatorWithAggregate(Seq(cond), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Filter(cond, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = resolveExpressionByPlanOutput(cond, agg)
+ resolveOperatorWithAggregate(Seq(maybeResolved), agg, (newExprs, newChild) => {
+ Filter(newExprs.head, newChild)
+ })
+
+ case Sort(sortOrder, global, agg: Aggregate) if agg.resolved =>
+ // We should resolve the references normally based on child.output first.
+ val maybeResolved = sortOrder.map(_.child).map(resolveExpressionByPlanOutput(_, agg))
+ resolveOperatorWithAggregate(maybeResolved, agg, (newExprs, newChild) => {
+ val newSortOrder = sortOrder.zip(newExprs).map {
+ case (sortOrder, expr) => sortOrder.copy(child = expr)
}
- val aliasedOrdering = unresolvedSortOrders.map(o => Alias(o.child, "aggOrder")())
-
- val aggregateWithExtraOrdering = aggregate.copy(
- aggregateExpressions = aggregate.aggregateExpressions ++ aliasedOrdering)
-
- val resolvedAggregate: Aggregate =
- executeSameContext(aggregateWithExtraOrdering).asInstanceOf[Aggregate]
-
- val (reResolvedAggExprs, resolvedAliasedOrdering) =
- resolvedAggregate.aggregateExpressions.splitAt(aggregate.aggregateExpressions.length)
-
- // If we pass the analysis check, then the ordering expressions should only reference to
- // aggregate expressions or grouping expressions, and it's safe to push them down to
- // Aggregate.
- checkAnalysis(resolvedAggregate)
-
- val originalAggExprs = aggregate.aggregateExpressions.map(trimNonTopLevelAliases)
-
- // If the ordering expression is same with original aggregate expression, we don't need
- // to push down this ordering expression and can reference the original aggregate
- // expression instead.
- val needsPushDown = ArrayBuffer.empty[NamedExpression]
- val orderToAlias = unresolvedSortOrders.zip(aliasedOrdering)
- val evaluatedOrderings =
- resolvedAliasedOrdering.asInstanceOf[Seq[Alias]].zip(orderToAlias).map {
- case (evaluated, (order, aliasOrder)) =>
- val index = reResolvedAggExprs.indexWhere {
- case Alias(child, _) => child semanticEquals evaluated.child
- case other => other semanticEquals evaluated.child
- }
+ Sort(newSortOrder, global, newChild)
+ })
+ }
- if (index == -1) {
- if (hasCharVarchar(evaluated)) {
- needsPushDown += aliasOrder
- order.copy(child = aliasOrder)
- } else {
- needsPushDown += evaluated
- order.copy(child = evaluated.toAttribute)
- }
- } else {
- order.copy(child = originalAggExprs(index).toAttribute)
- }
+ /**
+ * Resolves the given expressions as if they are in the given Aggregate operator, which means
+ * the column can be resolved using `agg.child` and aggregate functions/grouping columns are
+ * allowed. It returns a list of named expressions that need to be appended to
+ * `agg.aggregateExpressions`, and the list of resolved expressions.
+ */
+ def resolveExprsWithAggregate(
+ exprs: Seq[Expression],
+ agg: Aggregate): (Seq[NamedExpression], Seq[Expression]) = {
+ val extraAggExprs = ArrayBuffer.empty[NamedExpression]
+ val transformed = exprs.map { e =>
+ // Try resolving the expression as though it is in the aggregate clause.
+ def resolveCol(input: Expression): Expression = {
+ input.transform {
+ case u: UnresolvedAttribute =>
+ try {
+ agg.child.resolve(u.nameParts, resolver)
+ .map(TempResolvedColumn(_, u.nameParts)).getOrElse(u)
+ } catch {
+ case _: AnalysisException => u
+ }
}
-
- val sortOrdersMap = unresolvedSortOrders
- .map(new TreeNodeRef(_))
- .zip(evaluatedOrderings)
- .toMap
- val finalSortOrders = sortOrder.map(s => sortOrdersMap.getOrElse(new TreeNodeRef(s), s))
-
- // Since we don't rely on sort.resolved as the stop condition for this rule,
- // we need to check this and prevent applying this rule multiple times
- if (sortOrder == finalSortOrders) {
- sort
+ }
+ def resolveSubQuery(input: Expression): Expression = {
+ if (SubqueryExpression.hasSubquery(input)) {
+ val fake = Project(Alias(input, "fake")() :: Nil, agg.child)
+ ResolveSubquery(fake).asInstanceOf[Project].projectList.head.asInstanceOf[Alias].child
} else {
- Project(aggregate.output,
- Sort(finalSortOrders, global,
- aggregate.copy(aggregateExpressions = originalAggExprs ++ needsPushDown)))
+ input
}
- } catch {
- // Attempting to resolve in the aggregate can result in ambiguity. When this happens,
- // just return the original plan.
- case ae: AnalysisException => sort
}
- }
- def hasCharVarchar(expr: Alias): Boolean = {
- expr.find {
- case ne: NamedExpression => CharVarcharUtils.getRawType(ne.metadata).nonEmpty
- case _ => false
- }.nonEmpty
+ val maybeResolved = resolveSubQuery(resolveCol(e))
+ if (!maybeResolved.resolved) {
+ maybeResolved
+ } else {
+ buildAggExprList(maybeResolved, agg, extraAggExprs)
+ }
+ }
+ (extraAggExprs.toSeq, transformed)
}
- def containsAggregate(condition: Expression): Boolean = {
- condition.find(_.isInstanceOf[AggregateExpression]).isDefined
+ private def trimTempResolvedField(input: Expression): Expression = input.transform {
+ case t: TempResolvedColumn => t.child
}
- def resolveFilterCondInAggregate(
- filterCond: Expression, agg: Aggregate): Option[(Seq[NamedExpression], Expression)] = {
- try {
- val aggregatedCondition =
- Aggregate(
- agg.groupingExpressions,
- Alias(filterCond, "havingCondition")() :: Nil,
- agg.child)
- val resolvedOperator = executeSameContext(aggregatedCondition)
- def resolvedAggregateFilter =
- resolvedOperator
- .asInstanceOf[Aggregate]
- .aggregateExpressions.head
-
- // If resolution was successful and we see the filter has an aggregate in it, add it to
- // the original aggregate operator.
- if (resolvedOperator.resolved) {
- // Try to replace all aggregate expressions in the filter by an alias.
- val aggregateExpressions = ArrayBuffer.empty[NamedExpression]
- val transformedAggregateFilter = resolvedAggregateFilter.transform {
- case ae: AggregateExpression =>
- val alias = Alias(ae, ae.toString)()
- aggregateExpressions += alias
+ private def buildAggExprList(
+ expr: Expression,
+ agg: Aggregate,
+ aggExprList: ArrayBuffer[NamedExpression]): Expression = {
+ // Avoid adding an extra aggregate expression if it's already present in
+ // `agg.aggregateExpressions`.
+ val index = agg.aggregateExpressions.indexWhere {
Review comment:
Makes sense. The current code follows the previous one in https://github.com/apache/spark/pull/32470/files#diff-ed19f376a63eba52eea59ca71f3355d4495fad4fad4db9a3324aade0d4986a47L2502
I'll do it in a follow-up as this is an improvement, and update TPCDS query golden files if needed.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
##########
@@ -278,11 +278,6 @@ case class AttributeReference(
case _ => false
}
- override def semanticEquals(other: Expression): Boolean = other match {
Review comment:
will be done by https://github.com/apache/spark/pull/32885
##########
File path: sql/core/src/test/resources/sql-tests/results/explain-aqe.sql.out
##########
@@ -186,17 +185,13 @@ Input [2]: [key#x, max#x]
Keys [1]: [key#x]
Functions [1]: [max(val#x)]
Aggregate Attributes [1]: [max(val#x)#x]
-Results [3]: [key#x, max(val#x)#x AS max(val)#x, max(val#x)#x AS max(val#x)#x]
+Results [2]: [key#x, max(val#x)#x AS max(val)#x]
Review comment:
Removes a duplicated max function.
##########
File path: sql/core/src/test/resources/sql-tests/results/explain.sql.out
##########
@@ -188,15 +187,11 @@ Input [2]: [key#x, max#x]
Keys [1]: [key#x]
Functions [1]: [max(val#x)]
Aggregate Attributes [1]: [max(val#x)#x]
-Results [3]: [key#x, max(val#x)#x AS max(val)#x, max(val#x)#x AS max(val#x)#x]
+Results [2]: [key#x, max(val#x)#x AS max(val)#x]
Review comment:
ditto
##########
File path: sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14a.sf100/explain.txt
##########
@@ -536,38 +536,38 @@ Input [6]: [i_brand_id#54, i_class_id#55, i_category_id#56, sum#62, isEmpty#63,
Keys [3]: [i_brand_id#54, i_class_id#55, i_category_id#56]
Functions [2]: [sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true)), count(1)]
Aggregate Attributes [2]: [sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true))#66, count(1)#67]
-Results [6]: [i_brand_id#54, i_class_id#55, i_category_id#56, sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true))#66 AS sales#68, count(1)#67 AS number_sales#69, sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true))#66 AS sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true))#70]
+Results [5]: [i_brand_id#54, i_class_id#55, i_category_id#56, sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true))#66 AS sales#68, count(1)#67 AS number_sales#69]
Review comment:
removes a duplicated sum function.
##########
File path: sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14a.sf100/explain.txt
##########
@@ -577,82 +577,82 @@ Input [1]: [ss_item_sk#47]
Arguments: [ss_item_sk#47 ASC NULLS FIRST], false, 0
(98) SortMergeJoin [codegen id : 91]
-Left keys [1]: [cs_item_sk#74]
+Left keys [1]: [cs_item_sk#73]
Right keys [1]: [ss_item_sk#47]
Join condition: None
(99) ReusedExchange [Reuses operator id: 72]
-Output [1]: [d_date_sk#79]
+Output [1]: [d_date_sk#78]
(100) BroadcastHashJoin [codegen id : 91]
-Left keys [1]: [cs_sold_date_sk#77]
-Right keys [1]: [d_date_sk#79]
+Left keys [1]: [cs_sold_date_sk#76]
+Right keys [1]: [d_date_sk#78]
Join condition: None
(101) Project [codegen id : 91]
-Output [3]: [cs_item_sk#74, cs_quantity#75, cs_list_price#76]
-Input [5]: [cs_item_sk#74, cs_quantity#75, cs_list_price#76, cs_sold_date_sk#77, d_date_sk#79]
+Output [3]: [cs_item_sk#73, cs_quantity#74, cs_list_price#75]
+Input [5]: [cs_item_sk#73, cs_quantity#74, cs_list_price#75, cs_sold_date_sk#76, d_date_sk#78]
(102) ReusedExchange [Reuses operator id: 83]
-Output [4]: [i_item_sk#80, i_brand_id#81, i_class_id#82, i_category_id#83]
+Output [4]: [i_item_sk#79, i_brand_id#80, i_class_id#81, i_category_id#82]
(103) BroadcastHashJoin [codegen id : 91]
-Left keys [1]: [cs_item_sk#74]
-Right keys [1]: [i_item_sk#80]
+Left keys [1]: [cs_item_sk#73]
+Right keys [1]: [i_item_sk#79]
Join condition: None
(104) Project [codegen id : 91]
-Output [5]: [cs_quantity#75, cs_list_price#76, i_brand_id#81, i_class_id#82, i_category_id#83]
-Input [7]: [cs_item_sk#74, cs_quantity#75, cs_list_price#76, i_item_sk#80, i_brand_id#81, i_class_id#82, i_category_id#83]
+Output [5]: [cs_quantity#74, cs_list_price#75, i_brand_id#80, i_class_id#81, i_category_id#82]
+Input [7]: [cs_item_sk#73, cs_quantity#74, cs_list_price#75, i_item_sk#79, i_brand_id#80, i_class_id#81, i_category_id#82]
(105) HashAggregate [codegen id : 91]
-Input [5]: [cs_quantity#75, cs_list_price#76, i_brand_id#81, i_class_id#82, i_category_id#83]
-Keys [3]: [i_brand_id#81, i_class_id#82, i_category_id#83]
-Functions [2]: [partial_sum(CheckOverflow((promote_precision(cast(cast(cs_quantity#75 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(cs_list_price#76 as decimal(12,2)))), DecimalType(18,2), true)), partial_count(1)]
-Aggregate Attributes [3]: [sum#84, isEmpty#85, count#86]
-Results [6]: [i_brand_id#81, i_class_id#82, i_category_id#83, sum#87, isEmpty#88, count#89]
+Input [5]: [cs_quantity#74, cs_list_price#75, i_brand_id#80, i_class_id#81, i_category_id#82]
+Keys [3]: [i_brand_id#80, i_class_id#81, i_category_id#82]
+Functions [2]: [partial_sum(CheckOverflow((promote_precision(cast(cast(cs_quantity#74 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(cs_list_price#75 as decimal(12,2)))), DecimalType(18,2), true)), partial_count(1)]
+Aggregate Attributes [3]: [sum#83, isEmpty#84, count#85]
+Results [6]: [i_brand_id#80, i_class_id#81, i_category_id#82, sum#86, isEmpty#87, count#88]
(106) Exchange
-Input [6]: [i_brand_id#81, i_class_id#82, i_category_id#83, sum#87, isEmpty#88, count#89]
-Arguments: hashpartitioning(i_brand_id#81, i_class_id#82, i_category_id#83, 5), ENSURE_REQUIREMENTS, [id=#90]
+Input [6]: [i_brand_id#80, i_class_id#81, i_category_id#82, sum#86, isEmpty#87, count#88]
+Arguments: hashpartitioning(i_brand_id#80, i_class_id#81, i_category_id#82, 5), ENSURE_REQUIREMENTS, [id=#89]
(107) HashAggregate [codegen id : 92]
-Input [6]: [i_brand_id#81, i_class_id#82, i_category_id#83, sum#87, isEmpty#88, count#89]
-Keys [3]: [i_brand_id#81, i_class_id#82, i_category_id#83]
-Functions [2]: [sum(CheckOverflow((promote_precision(cast(cast(cs_quantity#75 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(cs_list_price#76 as decimal(12,2)))), DecimalType(18,2), true)), count(1)]
-Aggregate Attributes [2]: [sum(CheckOverflow((promote_precision(cast(cast(cs_quantity#75 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(cs_list_price#76 as decimal(12,2)))), DecimalType(18,2), true))#91, count(1)#92]
-Results [6]: [i_brand_id#81, i_class_id#82, i_category_id#83, sum(CheckOverflow((promote_precision(cast(cast(cs_quantity#75 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(cs_list_price#76 as decimal(12,2)))), DecimalType(18,2), true))#91 AS sales#93, count(1)#92 AS number_sales#94, sum(CheckOverflow((promote_precision(cast(cast(cs_quantity#75 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(cs_list_price#76 as decimal(12,2)))), DecimalType(18,2), true))#91 AS sum(CheckOverflow((promote_precision(cast(cast(cs_quantity#75 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(cs_list_price#76 as decimal(12,2)))), DecimalType(18,2), true))#95]
+Input [6]: [i_brand_id#80, i_class_id#81, i_category_id#82, sum#86, isEmpty#87, count#88]
+Keys [3]: [i_brand_id#80, i_class_id#81, i_category_id#82]
+Functions [2]: [sum(CheckOverflow((promote_precision(cast(cast(cs_quantity#74 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(cs_list_price#75 as decimal(12,2)))), DecimalType(18,2), true)), count(1)]
+Aggregate Attributes [2]: [sum(CheckOverflow((promote_precision(cast(cast(cs_quantity#74 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(cs_list_price#75 as decimal(12,2)))), DecimalType(18,2), true))#90, count(1)#91]
+Results [5]: [i_brand_id#80, i_class_id#81, i_category_id#82, sum(CheckOverflow((promote_precision(cast(cast(cs_quantity#74 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(cs_list_price#75 as decimal(12,2)))), DecimalType(18,2), true))#90 AS sales#92, count(1)#91 AS number_sales#93]
Review comment:
ditto
##########
File path: sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14a.sf100/explain.txt
##########
@@ -662,91 +662,91 @@ Input [1]: [ss_item_sk#47]
Arguments: [ss_item_sk#47 ASC NULLS FIRST], false, 0
(117) SortMergeJoin [codegen id : 137]
-Left keys [1]: [ws_item_sk#97]
+Left keys [1]: [ws_item_sk#95]
Right keys [1]: [ss_item_sk#47]
Join condition: None
(118) ReusedExchange [Reuses operator id: 72]
-Output [1]: [d_date_sk#102]
+Output [1]: [d_date_sk#100]
(119) BroadcastHashJoin [codegen id : 137]
-Left keys [1]: [ws_sold_date_sk#100]
-Right keys [1]: [d_date_sk#102]
+Left keys [1]: [ws_sold_date_sk#98]
+Right keys [1]: [d_date_sk#100]
Join condition: None
(120) Project [codegen id : 137]
-Output [3]: [ws_item_sk#97, ws_quantity#98, ws_list_price#99]
-Input [5]: [ws_item_sk#97, ws_quantity#98, ws_list_price#99, ws_sold_date_sk#100, d_date_sk#102]
+Output [3]: [ws_item_sk#95, ws_quantity#96, ws_list_price#97]
+Input [5]: [ws_item_sk#95, ws_quantity#96, ws_list_price#97, ws_sold_date_sk#98, d_date_sk#100]
(121) ReusedExchange [Reuses operator id: 83]
-Output [4]: [i_item_sk#103, i_brand_id#104, i_class_id#105, i_category_id#106]
+Output [4]: [i_item_sk#101, i_brand_id#102, i_class_id#103, i_category_id#104]
(122) BroadcastHashJoin [codegen id : 137]
-Left keys [1]: [ws_item_sk#97]
-Right keys [1]: [i_item_sk#103]
+Left keys [1]: [ws_item_sk#95]
+Right keys [1]: [i_item_sk#101]
Join condition: None
(123) Project [codegen id : 137]
-Output [5]: [ws_quantity#98, ws_list_price#99, i_brand_id#104, i_class_id#105, i_category_id#106]
-Input [7]: [ws_item_sk#97, ws_quantity#98, ws_list_price#99, i_item_sk#103, i_brand_id#104, i_class_id#105, i_category_id#106]
+Output [5]: [ws_quantity#96, ws_list_price#97, i_brand_id#102, i_class_id#103, i_category_id#104]
+Input [7]: [ws_item_sk#95, ws_quantity#96, ws_list_price#97, i_item_sk#101, i_brand_id#102, i_class_id#103, i_category_id#104]
(124) HashAggregate [codegen id : 137]
-Input [5]: [ws_quantity#98, ws_list_price#99, i_brand_id#104, i_class_id#105, i_category_id#106]
-Keys [3]: [i_brand_id#104, i_class_id#105, i_category_id#106]
-Functions [2]: [partial_sum(CheckOverflow((promote_precision(cast(cast(ws_quantity#98 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ws_list_price#99 as decimal(12,2)))), DecimalType(18,2), true)), partial_count(1)]
-Aggregate Attributes [3]: [sum#107, isEmpty#108, count#109]
-Results [6]: [i_brand_id#104, i_class_id#105, i_category_id#106, sum#110, isEmpty#111, count#112]
+Input [5]: [ws_quantity#96, ws_list_price#97, i_brand_id#102, i_class_id#103, i_category_id#104]
+Keys [3]: [i_brand_id#102, i_class_id#103, i_category_id#104]
+Functions [2]: [partial_sum(CheckOverflow((promote_precision(cast(cast(ws_quantity#96 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ws_list_price#97 as decimal(12,2)))), DecimalType(18,2), true)), partial_count(1)]
+Aggregate Attributes [3]: [sum#105, isEmpty#106, count#107]
+Results [6]: [i_brand_id#102, i_class_id#103, i_category_id#104, sum#108, isEmpty#109, count#110]
(125) Exchange
-Input [6]: [i_brand_id#104, i_class_id#105, i_category_id#106, sum#110, isEmpty#111, count#112]
-Arguments: hashpartitioning(i_brand_id#104, i_class_id#105, i_category_id#106, 5), ENSURE_REQUIREMENTS, [id=#113]
+Input [6]: [i_brand_id#102, i_class_id#103, i_category_id#104, sum#108, isEmpty#109, count#110]
+Arguments: hashpartitioning(i_brand_id#102, i_class_id#103, i_category_id#104, 5), ENSURE_REQUIREMENTS, [id=#111]
(126) HashAggregate [codegen id : 138]
-Input [6]: [i_brand_id#104, i_class_id#105, i_category_id#106, sum#110, isEmpty#111, count#112]
-Keys [3]: [i_brand_id#104, i_class_id#105, i_category_id#106]
-Functions [2]: [sum(CheckOverflow((promote_precision(cast(cast(ws_quantity#98 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ws_list_price#99 as decimal(12,2)))), DecimalType(18,2), true)), count(1)]
-Aggregate Attributes [2]: [sum(CheckOverflow((promote_precision(cast(cast(ws_quantity#98 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ws_list_price#99 as decimal(12,2)))), DecimalType(18,2), true))#114, count(1)#115]
-Results [6]: [i_brand_id#104, i_class_id#105, i_category_id#106, sum(CheckOverflow((promote_precision(cast(cast(ws_quantity#98 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ws_list_price#99 as decimal(12,2)))), DecimalType(18,2), true))#114 AS sales#116, count(1)#115 AS number_sales#117, sum(CheckOverflow((promote_precision(cast(cast(ws_quantity#98 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ws_list_price#99 as decimal(12,2)))), DecimalType(18,2), true))#114 AS sum(CheckOverflow((promote_precision(cast(cast(ws_quantity#98 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ws_list_price#99 as decimal(12,2)))), DecimalType(18,2), true))#118]
+Input [6]: [i_brand_id#102, i_class_id#103, i_category_id#104, sum#108, isEmpty#109, count#110]
+Keys [3]: [i_brand_id#102, i_class_id#103, i_category_id#104]
+Functions [2]: [sum(CheckOverflow((promote_precision(cast(cast(ws_quantity#96 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ws_list_price#97 as decimal(12,2)))), DecimalType(18,2), true)), count(1)]
+Aggregate Attributes [2]: [sum(CheckOverflow((promote_precision(cast(cast(ws_quantity#96 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ws_list_price#97 as decimal(12,2)))), DecimalType(18,2), true))#112, count(1)#113]
+Results [5]: [i_brand_id#102, i_class_id#103, i_category_id#104, sum(CheckOverflow((promote_precision(cast(cast(ws_quantity#96 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ws_list_price#97 as decimal(12,2)))), DecimalType(18,2), true))#112 AS sales#114, count(1)#113 AS number_sales#115]
Review comment:
ditto
##########
File path: sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14a/explain.txt
##########
@@ -465,193 +465,193 @@ Input [6]: [i_brand_id#46, i_class_id#47, i_category_id#48, sum#57, isEmpty#58,
Keys [3]: [i_brand_id#46, i_class_id#47, i_category_id#48]
Functions [2]: [sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true)), count(1)]
Aggregate Attributes [2]: [sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true))#61, count(1)#62]
-Results [6]: [i_brand_id#46, i_class_id#47, i_category_id#48, sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true))#61 AS sales#63, count(1)#62 AS number_sales#64, sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true))#61 AS sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true))#65]
+Results [5]: [i_brand_id#46, i_class_id#47, i_category_id#48, sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true))#61 AS sales#63, count(1)#62 AS number_sales#64]
Review comment:
ditto
##########
File path: sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14b.sf100/explain.txt
##########
@@ -518,352 +516,344 @@ Input [6]: [i_brand_id#55, i_class_id#56, i_category_id#57, sum#63, isEmpty#64,
Keys [3]: [i_brand_id#55, i_class_id#56, i_category_id#57]
Functions [2]: [sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true)), count(1)]
Aggregate Attributes [2]: [sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true))#67, count(1)#68]
-Results [6]: [i_brand_id#55, i_class_id#56, i_category_id#57, sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true))#67 AS sales#69, count(1)#68 AS number_sales#70, sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true))#67 AS sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true))#71]
+Results [6]: [store AS channel#69, i_brand_id#55, i_class_id#56, i_category_id#57, sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true))#67 AS sales#70, count(1)#68 AS number_sales#71]
(89) Filter [codegen id : 92]
-Input [6]: [i_brand_id#55, i_class_id#56, i_category_id#57, sales#69, number_sales#70, sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true))#71]
-Condition : (isnotnull(sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true))#71) AND (cast(sum(CheckOverflow((promote_precision(cast(cast(ss_quantity#2 as decimal(10,0)) as decimal(12,2))) * promote_precision(cast(ss_list_price#3 as decimal(12,2)))), DecimalType(18,2), true))#71 as decimal(32,6)) > cast(Subquery scalar-subquery#72, [id=#73] as decimal(32,6))))
+Input [6]: [channel#69, i_brand_id#55, i_class_id#56, i_category_id#57, sales#70, number_sales#71]
+Condition : (isnotnull(sales#70) AND (cast(sales#70 as decimal(32,6)) > cast(Subquery scalar-subquery#72, [id=#73] as decimal(32,6))))
-(90) Project [codegen id : 92]
-Output [6]: [store AS channel#74, i_brand_id#55, i_class_id#56, i_category_id#57, sales#69, number_sales#70]
Review comment:
`store AS channel#74` is in the aggregate node and the project is no longer needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-837901701
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42880/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858721845
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-836904086
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138336/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-837901701
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42880/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858471325
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139629/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858659431
cc @viirya @maropu @allisonwang-db @yaooqinn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-861423115
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44335/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-836347015
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42848/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-834711131
**[Test build #138261 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138261/testReport)** for PR 32470 at commit [`a8c708f`](https://github.com/apache/spark/commit/a8c708f0e5dbae0cd6b2f3b8a9b6917207d8dca1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858542182
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44156/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-838135134
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138358/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-838135134
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138358/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858675146
**[Test build #139645 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139645/testReport)** for PR 32470 at commit [`1e973ef`](https://github.com/apache/spark/commit/1e973ef66b445e3d70df92a52b39e99f7c491151).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-837836239
**[Test build #138358 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138358/testReport)** for PR 32470 at commit [`c0bb807`](https://github.com/apache/spark/commit/c0bb8070cbb52f9a20da0c5e3e791db72ea4bf04).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-861365825
**[Test build #139808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139808/testReport)** for PR 32470 at commit [`b362a09`](https://github.com/apache/spark/commit/b362a098978be65ab1fc033fe0213a78a467b6ae).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-835228683
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138281/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-859033403
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139652/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858471292
**[Test build #139629 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139629/testReport)** for PR 32470 at commit [`ae4d411`](https://github.com/apache/spark/commit/ae4d411bd68fcd9daf4db23a7a33b4fdca478528).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32470:
URL: https://github.com/apache/spark/pull/32470#discussion_r649027730
##########
File path: sql/core/src/test/resources/sql-tests/results/subquery/negative-cases/invalid-correlation.sql.out
##########
@@ -63,7 +63,7 @@ WHERE t1a IN (SELECT min(t2a)
struct<>
-- !query output
org.apache.spark.sql.AnalysisException
-Resolved attribute(s) t2b#x missing from min(t2a)#x,t2c#x in operator !Filter t2c#x IN (list#x [t2b#x]).
+cannot resolve 't2b' given input columns: [min(t2a), t2.t2c]; line 6 pos 24
Review comment:
Neither of the error messages is good, we can improve it later when we improve the correlated subquery resolution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [WIP] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858314556
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44136/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32470: [SPARK-35712][SQL] Simplify ResolveAggregateFunctions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32470:
URL: https://github.com/apache/spark/pull/32470#issuecomment-858878896
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org