You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by willb <gi...@git.apache.org> on 2014/07/20 04:19:18 UTC
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with unre...
GitHub user willb opened a pull request:
https://github.com/apache/spark/pull/1497
SPARK-2226: transform HAVING clauses with unresolvable attributes
This commit adds an analyzer rule to
1. find expressions in `HAVING` clause filters that depend on unresolved attributes,
2. push these expressions down to the underlying aggregates, and then
3. project them away above the filter.
It also enables the `HAVING` queries in the Hive compatibility suite.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/willb/spark spark-2226
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1497.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1497
----
commit 29a26e3ab6a21e6619f003d905bc7aa7d1cb2976
Author: William Benton <wi...@redhat.com>
Date: 2014-07-17T15:36:37Z
Added rule to handle unresolved attributes in HAVING clauses (SPARK-2226)
commit c7f2b2c8a19b09ec095a316cb965f18d474d7144
Author: William Benton <wi...@redhat.com>
Date: 2014-07-17T17:16:18Z
Whitelist HAVING queries.
Also adds golden outputs for HAVING tests.
commit 5a12647c169ee06bba5355c3956a158699247e43
Author: William Benton <wi...@redhat.com>
Date: 2014-07-19T17:08:17Z
Explanatory comments and stylistic cleanups.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with aggr...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/1497#discussion_r15157732
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -152,6 +155,34 @@ class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Bool
}
/**
+ * This rule finds expressions in HAVING clause filters that depend on
+ * unresolved attributes. It pushes these expressions down to the underlying
+ * aggregates and then projects them away above the filter.
+ */
+ object UnresolvedHavingClauseAttributes extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case pl @ Filter(fexp, agg @ Aggregate(_, ae, _)) if !fexp.childrenResolved => {
+ val alias = Alias(fexp, makeTmp())()
+ val aggExprs = Seq(alias) ++ ae
+
+ val newCond = EqualTo(Cast(alias.toAttribute, BooleanType), Literal(true, BooleanType))
+
+ val newFilter = ResolveReferences(pl.copy(condition = newCond,
+ child = agg.copy(aggregateExpressions = aggExprs)))
+
+ Project(pl.output, newFilter)
+ }
+ }
+
+ private val curId = new java.util.concurrent.atomic.AtomicLong()
+
+ private def makeTmp() = {
+ val id = curId.getAndIncrement()
+ s"tmp_cond_$id"
--- End diff --
All `NamedExpression`s have a globally unique id (i.e. the `#number` in the `toString`), so we can just create an Alias with a fixed `name`. I'd consider naming it something like `havingCondition`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with aggr...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49707966
@willb mind throwing [SQL] in the title here? It helps this filter nicely on some internal tools we have to triage PR's.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: [SQL] transform HAVING clauses wit...
Posted by willb <gi...@git.apache.org>.
Github user willb commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49726591
@marmbrus I've made the changes you suggested and moved the rule to the Resolution batch but now the newly-inserted attribute references remain unresolved after analysis. I'm going to try and figure out where things are going wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: [SQL] transform HAVING clauses wit...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49837238
Instead of creating `UnresolvedAttribute`s, we can just get the `AttributeReference` from the created `Alias` directly using `toAttribute`. Maybe like this?
```scala
/**
* This rule finds expressions in HAVING clause filters that depend on
* unresolved attributes. It pushes these expressions down to the underlying
* aggregates and then projects them away above the filter.
*/
object UnresolvedHavingClauseAttributes extends Rule[LogicalPlan] {
def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
case filter @ Filter(havingCondition, aggregate @ Aggregate(_, originalAggExprs, _))
if !filter.resolved && aggregate.resolved && containsAggregate(havingCondition) => {
val evaluatedCondition = Alias(havingCondition, "havingCondition")()
val aggExprsWithHaving = evaluatedCondition +: originalAggExprs
Project(aggregate.output,
Filter(evaluatedCondition.toAttribute,
aggregate.copy(aggregateExpressions = aggExprsWithHaving)))
}
}
protected def containsAggregate(condition: Expression): Boolean =
condition
.collect { case ae: AggregateExpression => ae }
.nonEmpty
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: [SQL] transform HAVING clauses wit...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49937859
QA tests have started for PR 1497. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17061/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: [SQL] transform HAVING clauses wit...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/1497#discussion_r15304322
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -152,6 +154,34 @@ class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Bool
}
/**
+ * This rule finds expressions in HAVING clause filters that depend on
+ * unresolved attributes. It pushes these expressions down to the underlying
+ * aggregates and then projects them away above the filter.
+ */
+ object UnresolvedHavingClauseAttributes extends Rule[LogicalPlan] {
+ val condName = "havingCondition"
+ val trueLit = Literal(true, BooleanType)
--- End diff --
Hey Will, this is looking great. Mind removing these two unused variables? and then I'll merge it in today!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: [SQL] transform HAVING clauses wit...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49775576
@willb feel free to push the broken version if you want another set of eyes to look at it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49536515
QA results for PR 1497:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16860/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with aggr...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1497#discussion_r15147996
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -152,6 +155,37 @@ class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Bool
}
/**
+ * This rule finds expressions in HAVING clause filters that depend on
+ * unresolved attributes. It pushes these expressions down to the underlying
+ * aggregates and then projects them away above the filter.
+ *
+ * For SPARK-2226
--- End diff --
Don't think you need to include the JIRA ticket, since it is a pretty common feature.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49542753
QA tests have started for PR 1497. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16869/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with aggr...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1497#discussion_r15155562
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -152,6 +155,34 @@ class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Bool
}
/**
+ * This rule finds expressions in HAVING clause filters that depend on
+ * unresolved attributes. It pushes these expressions down to the underlying
+ * aggregates and then projects them away above the filter.
+ */
+ object UnresolvedHavingClauseAttributes extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case pl @ Filter(fexp, agg @ Aggregate(_, ae, _)) if !fexp.childrenResolved => {
+ val alias = Alias(fexp, makeTmp())()
+ val aggExprs = Seq(alias) ++ ae
+
+ val newCond = EqualTo(Cast(alias.toAttribute, BooleanType), Literal(true, BooleanType))
+
+ val newFilter = ResolveReferences(pl.copy(condition = newCond,
+ child = agg.copy(aggregateExpressions = aggExprs)))
+
+ Project(pl.output, newFilter)
+ }
+ }
+
+ private val curId = new java.util.concurrent.atomic.AtomicLong()
+
+ private def makeTmp() = {
+ val id = curId.getAndIncrement()
+ s"tmp_cond_$id"
--- End diff --
@marmbrus is this the standard way of assigning temp attributes in spark sql?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with aggr...
Posted by willb <gi...@git.apache.org>.
Github user willb commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49542704
Thanks, @rxin! I've made these changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: [SQL] transform HAVING clauses wit...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49937007
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: [SQL] transform HAVING clauses wit...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1497
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: [SQL] transform HAVING clauses wit...
Posted by willb <gi...@git.apache.org>.
Github user willb commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49783600
@marmbrus I've pushed it to https://github.com/willb/spark/tree/spark-2226-broken -- if I split out the `UnresolvedHavingClauseAttributes` to a separate batch and add a `ResolveReferences` (as in willb@16b3dfc), things seem to work, but if I put `UnresolvedHavingClauseAttributes` in Resolution (as in willb@76ae5c1), things fail with
No function to evaluate expression. type: AttributeReference
I'm sure I'm missing something simple here. Thanks for taking a look!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with aggr...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49580561
Thanks for working on this @willb!
One thought: We might want to put this in in the Resolution batch instead of in its own batch. That way we don't have to call `ResolveReferences` manually. It'll also let us handle queries like `SELECT * ... FROM (SELECT * FROM ... HAVING ...) HAVING ...`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with aggr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49544751
QA results for PR 1497:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16869/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: [SQL] transform HAVING clauses wit...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49948005
QA results for PR 1497:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17061/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with aggr...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1497#discussion_r15148004
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -152,6 +155,37 @@ class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Bool
}
/**
+ * This rule finds expressions in HAVING clause filters that depend on
+ * unresolved attributes. It pushes these expressions down to the underlying
+ * aggregates and then projects them away above the filter.
+ *
+ * For SPARK-2226
+ */
+ object UnresolvedHavingClauseAttributes extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case pl@Filter(fexp, agg@Aggregate(_, ae, _)) if !fexp.childrenResolved => {
+ val alias = Alias(fexp, makeTmp)()
+ val aggExprs = Seq(alias) ++ ae
+
+ val newCond = EqualTo(Cast(alias.toAttribute, BooleanType),
--- End diff --
i think this fits on one line ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: [SQL] transform HAVING clauses wit...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49949715
Thanks! I've merged this into master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with aggr...
Posted by willb <gi...@git.apache.org>.
Github user willb commented on a diff in the pull request:
https://github.com/apache/spark/pull/1497#discussion_r15172123
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -152,6 +155,34 @@ class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Bool
}
/**
+ * This rule finds expressions in HAVING clause filters that depend on
+ * unresolved attributes. It pushes these expressions down to the underlying
+ * aggregates and then projects them away above the filter.
+ */
+ object UnresolvedHavingClauseAttributes extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case pl @ Filter(fexp, agg @ Aggregate(_, ae, _)) if !fexp.childrenResolved => {
+ val alias = Alias(fexp, makeTmp())()
+ val aggExprs = Seq(alias) ++ ae
+
+ val newCond = EqualTo(Cast(alias.toAttribute, BooleanType), Literal(true, BooleanType))
+
+ val newFilter = ResolveReferences(pl.copy(condition = newCond,
+ child = agg.copy(aggregateExpressions = aggExprs)))
+
+ Project(pl.output, newFilter)
+ }
+ }
+
+ private val curId = new java.util.concurrent.atomic.AtomicLong()
+
+ private def makeTmp() = {
+ val id = curId.getAndIncrement()
+ s"tmp_cond_$id"
--- End diff --
@marmbrus Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: [SQL] transform HAVING clauses wit...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49890618
QA results for PR 1497:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17036/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with unre...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49535141
QA tests have started for PR 1497. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16860/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with aggr...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/1497#discussion_r15158553
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -152,6 +155,34 @@ class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Bool
}
/**
+ * This rule finds expressions in HAVING clause filters that depend on
+ * unresolved attributes. It pushes these expressions down to the underlying
+ * aggregates and then projects them away above the filter.
+ */
+ object UnresolvedHavingClauseAttributes extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case pl @ Filter(fexp, agg @ Aggregate(_, ae, _)) if !fexp.childrenResolved => {
+ val alias = Alias(fexp, makeTmp())()
+ val aggExprs = Seq(alias) ++ ae
+
+ val newCond = EqualTo(Cast(alias.toAttribute, BooleanType), Literal(true, BooleanType))
+
+ val newFilter = ResolveReferences(pl.copy(condition = newCond,
+ child = agg.copy(aggregateExpressions = aggExprs)))
+
+ Project(pl.output, newFilter)
+ }
+ }
+
+ private val curId = new java.util.concurrent.atomic.AtomicLong()
+
+ private def makeTmp() = {
+ val id = curId.getAndIncrement()
+ s"tmp_cond_$id"
--- End diff --
[Some more details about NamedExpressions](http://people.apache.org/~marmbrus/docs/catalyst/#org.apache.spark.sql.catalyst.expressions.package)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with aggr...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1497#discussion_r15148014
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -152,6 +155,37 @@ class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Bool
}
/**
+ * This rule finds expressions in HAVING clause filters that depend on
+ * unresolved attributes. It pushes these expressions down to the underlying
+ * aggregates and then projects them away above the filter.
+ *
+ * For SPARK-2226
+ */
+ object UnresolvedHavingClauseAttributes extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case pl@Filter(fexp, agg@Aggregate(_, ae, _)) if !fexp.childrenResolved => {
+ val alias = Alias(fexp, makeTmp)()
+ val aggExprs = Seq(alias) ++ ae
+
+ val newCond = EqualTo(Cast(alias.toAttribute, BooleanType),
+ Literal(true, BooleanType))
+
+ val flt = ResolveReferences(pl.copy(condition=newCond,
+ child=agg.copy(aggregateExpressions=aggExprs)))
+
+ Project(pl.output, flt)
+ }
+ }
+
+ private val curId = new java.util.concurrent.atomic.AtomicLong()
+
+ def makeTmp = {
--- End diff --
private - and add parenthesis since it has side effect. make sure you update the caller to add parenthesis too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: [SQL] transform HAVING clauses wit...
Posted by willb <gi...@git.apache.org>.
Github user willb commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49726353
@pwendell Sure, and I'll do this in the future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: [SQL] transform HAVING clauses wit...
Posted by willb <gi...@git.apache.org>.
Github user willb commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49912686
@marmbrus, thanks again; sorry to have missed those earlier!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: [SQL] transform HAVING clauses wit...
Posted by willb <gi...@git.apache.org>.
Github user willb commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49876099
Thanks, @marmbrus; this works now and I understand where things were going wrong (and also, hopefully, how to use Catalyst more idiomatically).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with aggr...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1497#discussion_r15148009
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -152,6 +155,37 @@ class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Bool
}
/**
+ * This rule finds expressions in HAVING clause filters that depend on
+ * unresolved attributes. It pushes these expressions down to the underlying
+ * aggregates and then projects them away above the filter.
+ *
+ * For SPARK-2226
+ */
+ object UnresolvedHavingClauseAttributes extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case pl@Filter(fexp, agg@Aggregate(_, ae, _)) if !fexp.childrenResolved => {
+ val alias = Alias(fexp, makeTmp)()
+ val aggExprs = Seq(alias) ++ ae
+
+ val newCond = EqualTo(Cast(alias.toAttribute, BooleanType),
+ Literal(true, BooleanType))
+
+ val flt = ResolveReferences(pl.copy(condition=newCond,
--- End diff --
Scala's convention is different from Python's in that we do add space before/after "=" for named arguments. Would would be better to rename the variable instead of "flt".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with aggr...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/1497#discussion_r15156055
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -152,6 +155,34 @@ class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Bool
}
/**
+ * This rule finds expressions in HAVING clause filters that depend on
+ * unresolved attributes. It pushes these expressions down to the underlying
+ * aggregates and then projects them away above the filter.
+ */
+ object UnresolvedHavingClauseAttributes extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case pl @ Filter(fexp, agg @ Aggregate(_, ae, _)) if !fexp.childrenResolved => {
+ val alias = Alias(fexp, makeTmp())()
+ val aggExprs = Seq(alias) ++ ae
+
+ val newCond = EqualTo(Cast(alias.toAttribute, BooleanType), Literal(true, BooleanType))
+
+ val newFilter = ResolveReferences(pl.copy(condition = newCond,
+ child = agg.copy(aggregateExpressions = aggExprs)))
+
+ Project(pl.output, newFilter)
+ }
+ }
+
+ private val curId = new java.util.concurrent.atomic.AtomicLong()
+
+ private def makeTmp() = {
+ val id = curId.getAndIncrement()
+ s"tmp_cond_$id"
--- End diff --
Actually I have the same question, generally how do we avoid the alias name conflict with the existed or user specified alias name?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with aggr...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/1497#discussion_r15158261
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -152,6 +155,34 @@ class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Bool
}
/**
+ * This rule finds expressions in HAVING clause filters that depend on
+ * unresolved attributes. It pushes these expressions down to the underlying
+ * aggregates and then projects them away above the filter.
+ */
+ object UnresolvedHavingClauseAttributes extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case pl @ Filter(fexp, agg @ Aggregate(_, ae, _)) if !fexp.childrenResolved => {
+ val alias = Alias(fexp, makeTmp())()
+ val aggExprs = Seq(alias) ++ ae
+
+ val newCond = EqualTo(Cast(alias.toAttribute, BooleanType), Literal(true, BooleanType))
--- End diff --
I think we can assume that the condition of the filter has already been coerced to a boolean, so using alias.toAttribute should be fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: transform HAVING clauses with aggr...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1497#discussion_r15148000
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -152,6 +155,37 @@ class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Bool
}
/**
+ * This rule finds expressions in HAVING clause filters that depend on
+ * unresolved attributes. It pushes these expressions down to the underlying
+ * aggregates and then projects them away above the filter.
+ *
+ * For SPARK-2226
+ */
+ object UnresolvedHavingClauseAttributes extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case pl@Filter(fexp, agg@Aggregate(_, ae, _)) if !fexp.childrenResolved => {
--- End diff --
space before and after "@"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: [SQL] transform HAVING clauses wit...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/1497#discussion_r15304351
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -21,6 +21,7 @@ import org.apache.spark.sql.catalyst.errors.TreeNodeException
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules._
+import org.apache.spark.sql.catalyst.types._
--- End diff --
Also, I don't think we need this import.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2226: [SQL] transform HAVING clauses wit...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1497#issuecomment-49876182
QA tests have started for PR 1497. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17036/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---