You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by hvanhovell <gi...@git.apache.org> on 2015/11/02 10:47:10 UTC

[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

GitHub user hvanhovell opened a pull request:

    https://github.com/apache/spark/pull/9406

    [SPARK-9241][SQL] Supporting multiple DISTINCT columns (2) - Rewriting Rule

    The second PR for SPARK-9241, this adds support for multiple distinct columns to the new aggregation code path.
    
    This PR solves the multiple DISTINCT column problem by rewriting these Aggregates into an Expand-Aggregate-Aggregate combination. See the [JIRA ticket](https://issues.apache.org/jira/browse/SPARK-9241) for some information on this. The advantages over the - competing - [first PR](https://github.com/apache/spark/pull/9280) are:
    - This can use the faster TungstenAggregate code path.
    - It is impossible to OOM due to an ```OpenHashSet``` allocating to much memory. However, this will multiply the number of input rows by the number of distinct clauses (plus one), and puts a lot more memory pressure on the aggregation code path itself.
    
    The location of this Rule is a bit funny, and should probably change when the old aggregation path is changed.
    
    cc @yhuai - Could you also tell me where to add tests for this?

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/hvanhovell/spark SPARK-9241-rewriter

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/9406.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #9406
    
----
commit 9fd77c64165f79d17a12613afee89e51afaf5e00
Author: Herman van Hovell <hv...@questtec.nl>
Date:   2015-11-02T07:15:55Z

    rebase

commit 6139f473d4edec53bcbe47008832f67f3ef567fb
Author: Herman van Hovell <hv...@questtec.nl>
Date:   2015-11-02T09:30:52Z

    Fix a few small bugs.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-153777154
  
    **[Test build #45019 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45019/consoleFull)** for PR 9406 at commit [`1e705fe`](https://github.com/apache/spark/commit/1e705fe3741d3d6a8030f9ca8d08de6c62a28d82).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-154342367
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-154398776
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-154352052
  
    **[Test build #45207 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45207/consoleFull)** for PR 9406 at commit [`9be5b9d`](https://github.com/apache/spark/commit/9be5b9d9e3c9de81473ac93750b687e2de824bb2).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:\n  * `final class ShuffleSortDataFormat extends SortDataFormat<PackedRecordPointer, LongArray> `\n  * `final class UnsafeSortDataFormat extends SortDataFormat<RecordPointerAndKeyPrefix, LongArray> `\n  * `case class Expand(`\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/9406


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-153810518
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-153323375
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44915/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-154352102
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-153810523
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45019/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-153774767
  
    Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:

    https://github.com/apache/spark/pull/9406#discussion_r44199608
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Utils.scala ---
    @@ -213,3 +216,178 @@ object Utils {
         case other => None
       }
     }
    +
    +/**
    + * This rule rewrites an aggregate query with multiple distinct clauses into an expanded double
    + * aggregation in which the regular aggregation expressions and every distinct clause is aggregated
    + * in a separate group. The results are then combined in a second aggregate.
    + *
    + * TODO Expression cannocalization
    + * TODO Eliminate foldable expressions from distinct clauses.
    + * TODO This eliminates all distinct expressions. We could safely pass one to the aggregate
    + *      operator. Perhaps this is a good thing? It is much simpler to plan later on...
    + */
    +object MultipleDistinctRewriter extends Rule[LogicalPlan] {
    +
    +  def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
    +    case a: Aggregate => rewrite(a)
    +    case p => p
    +  }
    +
    +  def rewrite(a: Aggregate): Aggregate = {
    +
    +    // Collect all aggregate expressions.
    +    val aggExpressions = a.aggregateExpressions.flatMap { e =>
    +      e.collect {
    +        case ae: AggregateExpression2 => ae
    +      }
    +    }
    +
    +    // Extract distinct aggregate expressions.
    +    val distinctAggGroups = aggExpressions
    +      .filter(_.isDistinct)
    +      .groupBy(_.aggregateFunction.children.toSet)
    +
    +    // Only continue to rewrite if there is more than one distinct group.
    +    if (distinctAggGroups.size > 1) {
    +      // Create the attributes for the grouping id and the group by clause.
    +      val gid = new AttributeReference("gid", IntegerType, false)()
    +      val groupByMap = a.groupingExpressions.collect {
    +        case ne: NamedExpression => ne -> ne.toAttribute
    +        case e => e -> new AttributeReference(e.prettyName, e.dataType, e.nullable)()
    +      }
    +      val groupByAttrs = groupByMap.map(_._2)
    +
    +      // Functions used to modify aggregate functions and their inputs.
    +      def evalWithinGroup(id: Literal, e: Expression) = If(EqualTo(gid, id), e, nullify(e))
    +      def patchAggregateFunctionChildren(
    +          af: AggregateFunction2,
    +          id: Literal,
    +          attrs: Map[Expression, Expression]): AggregateFunction2 = {
    +        af.withNewChildren(af.children.map { case afc =>
    +          evalWithinGroup(id, attrs(afc))
    +        }).asInstanceOf[AggregateFunction2]
    +      }
    +
    +      // Setup unique distinct aggregate children.
    +      val distinctAggChildren = distinctAggGroups.keySet.flatten.toSeq
    +      val distinctAggChildAttrMap = distinctAggChildren.map(expressionAttributePair).toMap
    +      val distinctAggChildAttrs = distinctAggChildAttrMap.values.toSeq
    +
    +      // Setup expand & aggregate operators for distinct aggregate expressions.
    +      val distinctAggOperatorMap = distinctAggGroups.toSeq.zipWithIndex.map {
    +        case ((group, expressions), i) =>
    +          val id = Literal(i + 1)
    +
    +          // Expand projection
    +          val projection = distinctAggChildren.map {
    +            case e if group.contains(e) => e
    +            case e => nullify(e)
    +          } :+ id
    +
    +          // Final aggregate
    +          val operators = expressions.map { e =>
    +            val af = e.aggregateFunction
    +            val naf = patchAggregateFunctionChildren(af, id, distinctAggChildAttrMap)
    +            (e, e.copy(aggregateFunction = naf, isDistinct = false))
    +          }
    +
    +          (projection, operators)
    +      }
    +
    +      // Setup expand for the 'regular' aggregate expressions.
    +      val regularAggExprs = aggExpressions.filter(!_.isDistinct)
    +      val regularAggChildren = regularAggExprs.flatMap(_.aggregateFunction.children).distinct
    +      val regularAggChildAttrMap = regularAggChildren.map(expressionAttributePair).toMap
    +
    +      // Setup aggregates for 'regular' aggregate expressions.
    +      val regularGroupId = Literal(0)
    +      val regularAggOperatorMap = regularAggExprs.map { e =>
    --- End diff --
    
    Comment on what each tuple element is, or maybe even use a case class?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/9406#discussion_r44203475
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Utils.scala ---
    @@ -213,3 +216,178 @@ object Utils {
         case other => None
       }
     }
    +
    +/**
    + * This rule rewrites an aggregate query with multiple distinct clauses into an expanded double
    + * aggregation in which the regular aggregation expressions and every distinct clause is aggregated
    + * in a separate group. The results are then combined in a second aggregate.
    + *
    + * TODO Expression cannocalization
    + * TODO Eliminate foldable expressions from distinct clauses.
    + * TODO This eliminates all distinct expressions. We could safely pass one to the aggregate
    + *      operator. Perhaps this is a good thing? It is much simpler to plan later on...
    --- End diff --
    
    These are all optimizations. The last one is perhaps a question for @yhuai: we have a choice to rewrite all distinct expressions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-154398778
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45219/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-154372363
  
    **[Test build #45219 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45219/consoleFull)** for PR 9406 at commit [`d3bdb2b`](https://github.com/apache/spark/commit/d3bdb2bb0096663af7d9faf4c0963a3df00065aa).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:

    https://github.com/apache/spark/pull/9406#discussion_r44207772
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Utils.scala ---
    @@ -213,3 +216,178 @@ object Utils {
         case other => None
       }
     }
    +
    +/**
    + * This rule rewrites an aggregate query with multiple distinct clauses into an expanded double
    + * aggregation in which the regular aggregation expressions and every distinct clause is aggregated
    + * in a separate group. The results are then combined in a second aggregate.
    + *
    + * TODO Expression cannocalization
    + * TODO Eliminate foldable expressions from distinct clauses.
    + * TODO This eliminates all distinct expressions. We could safely pass one to the aggregate
    + *      operator. Perhaps this is a good thing? It is much simpler to plan later on...
    --- End diff --
    
    Yeah, we can use this path to handle all cases. If I understand correctly, this rewriting approach will first create two logical Aggregate operators and then we shuffle data twice. Our current planning rule for a single distinct agg will shuffle data once, which can be bad if we do not have group by clause (because we will have a single reducer). To make the ideal decision, we need to know the statistics of grouping columns and distinct column. However, for the cases that we have a single distinct column and we do not have a group by clause, I feel your rewriting approach should be strictly better. What do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-154398663
  
    **[Test build #45219 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45219/consoleFull)** for PR 9406 at commit [`d3bdb2b`](https://github.com/apache/spark/commit/d3bdb2bb0096663af7d9faf4c0963a3df00065aa).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:\n  * `case class Expand(`\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-154369474
  
    Jenkins is not retesting... @marmbrus could you add me to the whitelist?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-153323369
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-154342399
  
    Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-154581656
  
    Okay, I looked over this pretty quickly and it looks awesome.  We need some tests and we are super close to me cutting a preview release.  That said, I'd really like to include this in 1.6.  Here is my proposal:
     - Lets merge this as is.
     - Yin will start ripping out the old aggregation path. (with or without #9409).
     - Comments and TODOs can be addressed in a follow up.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-154368715
  
    Jenkins retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:

    https://github.com/apache/spark/pull/9406#discussion_r44199322
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Utils.scala ---
    @@ -213,3 +216,178 @@ object Utils {
         case other => None
       }
     }
    +
    +/**
    + * This rule rewrites an aggregate query with multiple distinct clauses into an expanded double
    + * aggregation in which the regular aggregation expressions and every distinct clause is aggregated
    + * in a separate group. The results are then combined in a second aggregate.
    + *
    + * TODO Expression cannocalization
    + * TODO Eliminate foldable expressions from distinct clauses.
    + * TODO This eliminates all distinct expressions. We could safely pass one to the aggregate
    + *      operator. Perhaps this is a good thing? It is much simpler to plan later on...
    --- End diff --
    
    Just making sure I understand, these are all optimizations not correctness concerns?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-153315138
  
    **[Test build #44915 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44915/consoleFull)** for PR 9406 at commit [`6139f47`](https://github.com/apache/spark/commit/6139f473d4edec53bcbe47008832f67f3ef567fb).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/9406#discussion_r44203639
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Utils.scala ---
    @@ -213,3 +216,178 @@ object Utils {
         case other => None
       }
     }
    +
    +/**
    + * This rule rewrites an aggregate query with multiple distinct clauses into an expanded double
    + * aggregation in which the regular aggregation expressions and every distinct clause is aggregated
    + * in a separate group. The results are then combined in a second aggregate.
    + *
    + * TODO Expression cannocalization
    + * TODO Eliminate foldable expressions from distinct clauses.
    + * TODO This eliminates all distinct expressions. We could safely pass one to the aggregate
    + *      operator. Perhaps this is a good thing? It is much simpler to plan later on...
    + */
    +object MultipleDistinctRewriter extends Rule[LogicalPlan] {
    +
    +  def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
    +    case a: Aggregate => rewrite(a)
    +    case p => p
    +  }
    +
    +  def rewrite(a: Aggregate): Aggregate = {
    +
    +    // Collect all aggregate expressions.
    +    val aggExpressions = a.aggregateExpressions.flatMap { e =>
    +      e.collect {
    +        case ae: AggregateExpression2 => ae
    +      }
    +    }
    +
    +    // Extract distinct aggregate expressions.
    +    val distinctAggGroups = aggExpressions
    +      .filter(_.isDistinct)
    +      .groupBy(_.aggregateFunction.children.toSet)
    +
    +    // Only continue to rewrite if there is more than one distinct group.
    +    if (distinctAggGroups.size > 1) {
    +      // Create the attributes for the grouping id and the group by clause.
    +      val gid = new AttributeReference("gid", IntegerType, false)()
    +      val groupByMap = a.groupingExpressions.collect {
    +        case ne: NamedExpression => ne -> ne.toAttribute
    +        case e => e -> new AttributeReference(e.prettyName, e.dataType, e.nullable)()
    +      }
    +      val groupByAttrs = groupByMap.map(_._2)
    +
    +      // Functions used to modify aggregate functions and their inputs.
    +      def evalWithinGroup(id: Literal, e: Expression) = If(EqualTo(gid, id), e, nullify(e))
    +      def patchAggregateFunctionChildren(
    +          af: AggregateFunction2,
    +          id: Literal,
    +          attrs: Map[Expression, Expression]): AggregateFunction2 = {
    +        af.withNewChildren(af.children.map { case afc =>
    +          evalWithinGroup(id, attrs(afc))
    +        }).asInstanceOf[AggregateFunction2]
    +      }
    +
    +      // Setup unique distinct aggregate children.
    +      val distinctAggChildren = distinctAggGroups.keySet.flatten.toSeq
    +      val distinctAggChildAttrMap = distinctAggChildren.map(expressionAttributePair).toMap
    +      val distinctAggChildAttrs = distinctAggChildAttrMap.values.toSeq
    +
    +      // Setup expand & aggregate operators for distinct aggregate expressions.
    +      val distinctAggOperatorMap = distinctAggGroups.toSeq.zipWithIndex.map {
    +        case ((group, expressions), i) =>
    +          val id = Literal(i + 1)
    +
    +          // Expand projection
    +          val projection = distinctAggChildren.map {
    +            case e if group.contains(e) => e
    +            case e => nullify(e)
    +          } :+ id
    +
    +          // Final aggregate
    +          val operators = expressions.map { e =>
    +            val af = e.aggregateFunction
    +            val naf = patchAggregateFunctionChildren(af, id, distinctAggChildAttrMap)
    +            (e, e.copy(aggregateFunction = naf, isDistinct = false))
    +          }
    +
    +          (projection, operators)
    +      }
    +
    +      // Setup expand for the 'regular' aggregate expressions.
    +      val regularAggExprs = aggExpressions.filter(!_.isDistinct)
    +      val regularAggChildren = regularAggExprs.flatMap(_.aggregateFunction.children).distinct
    +      val regularAggChildAttrMap = regularAggChildren.map(expressionAttributePair).toMap
    +
    +      // Setup aggregates for 'regular' aggregate expressions.
    +      val regularGroupId = Literal(0)
    +      val regularAggOperatorMap = regularAggExprs.map { e =>
    --- End diff --
    
    I'll add documentation in a follow-up PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-154577223
  
    @hvanhovell I have started to use this PR as the foundation of removing our old aggregation code path. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:

    https://github.com/apache/spark/pull/9406#discussion_r44198936
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Utils.scala ---
    @@ -213,3 +216,178 @@ object Utils {
         case other => None
       }
     }
    +
    +/**
    + * This rule rewrites an aggregate query with multiple distinct clauses into an expanded double
    + * aggregation in which the regular aggregation expressions and every distinct clause is aggregated
    + * in a separate group. The results are then combined in a second aggregate.
    --- End diff --
    
    It would be really helpful if there was an example of what this rewrite looks like here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/9406#discussion_r44203629
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Utils.scala ---
    @@ -213,3 +216,178 @@ object Utils {
         case other => None
       }
     }
    +
    +/**
    + * This rule rewrites an aggregate query with multiple distinct clauses into an expanded double
    + * aggregation in which the regular aggregation expressions and every distinct clause is aggregated
    + * in a separate group. The results are then combined in a second aggregate.
    --- End diff --
    
    I'll add an example in the follow-up PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-154368700
  
    Hmmmm... this is a bit of a strange error.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-153810427
  
    **[Test build #45019 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45019/consoleFull)** for PR 9406 at commit [`1e705fe`](https://github.com/apache/spark/commit/1e705fe3741d3d6a8030f9ca8d08de6c62a28d82).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:\n  * `case class Expand(`\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-154371145
  
    Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-152973657
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-153774702
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-153313342
  
    Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-154371130
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:

    https://github.com/apache/spark/pull/9406#discussion_r44199453
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Utils.scala ---
    @@ -213,3 +216,178 @@ object Utils {
         case other => None
       }
     }
    +
    +/**
    + * This rule rewrites an aggregate query with multiple distinct clauses into an expanded double
    + * aggregation in which the regular aggregation expressions and every distinct clause is aggregated
    + * in a separate group. The results are then combined in a second aggregate.
    + *
    + * TODO Expression cannocalization
    + * TODO Eliminate foldable expressions from distinct clauses.
    + * TODO This eliminates all distinct expressions. We could safely pass one to the aggregate
    + *      operator. Perhaps this is a good thing? It is much simpler to plan later on...
    + */
    +object MultipleDistinctRewriter extends Rule[LogicalPlan] {
    +
    +  def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
    +    case a: Aggregate => rewrite(a)
    +    case p => p
    +  }
    +
    +  def rewrite(a: Aggregate): Aggregate = {
    +
    +    // Collect all aggregate expressions.
    +    val aggExpressions = a.aggregateExpressions.flatMap { e =>
    +      e.collect {
    +        case ae: AggregateExpression2 => ae
    +      }
    +    }
    +
    +    // Extract distinct aggregate expressions.
    +    val distinctAggGroups = aggExpressions
    +      .filter(_.isDistinct)
    +      .groupBy(_.aggregateFunction.children.toSet)
    +
    +    // Only continue to rewrite if there is more than one distinct group.
    +    if (distinctAggGroups.size > 1) {
    +      // Create the attributes for the grouping id and the group by clause.
    +      val gid = new AttributeReference("gid", IntegerType, false)()
    +      val groupByMap = a.groupingExpressions.collect {
    +        case ne: NamedExpression => ne -> ne.toAttribute
    +        case e => e -> new AttributeReference(e.prettyName, e.dataType, e.nullable)()
    +      }
    +      val groupByAttrs = groupByMap.map(_._2)
    +
    +      // Functions used to modify aggregate functions and their inputs.
    +      def evalWithinGroup(id: Literal, e: Expression) = If(EqualTo(gid, id), e, nullify(e))
    +      def patchAggregateFunctionChildren(
    +          af: AggregateFunction2,
    +          id: Literal,
    +          attrs: Map[Expression, Expression]): AggregateFunction2 = {
    +        af.withNewChildren(af.children.map { case afc =>
    +          evalWithinGroup(id, attrs(afc))
    +        }).asInstanceOf[AggregateFunction2]
    +      }
    +
    +      // Setup unique distinct aggregate children.
    +      val distinctAggChildren = distinctAggGroups.keySet.flatten.toSeq
    +      val distinctAggChildAttrMap = distinctAggChildren.map(expressionAttributePair).toMap
    +      val distinctAggChildAttrs = distinctAggChildAttrMap.values.toSeq
    +
    +      // Setup expand & aggregate operators for distinct aggregate expressions.
    +      val distinctAggOperatorMap = distinctAggGroups.toSeq.zipWithIndex.map {
    +        case ((group, expressions), i) =>
    +          val id = Literal(i + 1)
    +
    +          // Expand projection
    +          val projection = distinctAggChildren.map {
    +            case e if group.contains(e) => e
    +            case e => nullify(e)
    +          } :+ id
    +
    +          // Final aggregate
    +          val operators = expressions.map { e =>
    +            val af = e.aggregateFunction
    +            val naf = patchAggregateFunctionChildren(af, id, distinctAggChildAttrMap)
    +            (e, e.copy(aggregateFunction = naf, isDistinct = false))
    +          }
    +
    +          (projection, operators)
    +      }
    +
    +      // Setup expand for the 'regular' aggregate expressions.
    +      val regularAggExprs = aggExpressions.filter(!_.isDistinct)
    +      val regularAggChildren = regularAggExprs.flatMap(_.aggregateFunction.children).distinct
    +      val regularAggChildAttrMap = regularAggChildren.map(expressionAttributePair).toMap
    +
    +      // Setup aggregates for 'regular' aggregate expressions.
    +      val regularGroupId = Literal(0)
    +      val regularAggOperatorMap = regularAggExprs.map { e =>
    +        // Perform the actual aggregation in the initial aggregate.
    +        val af = patchAggregateFunctionChildren(
    +          e.aggregateFunction,
    +          regularGroupId,
    +          regularAggChildAttrMap)
    +        val a = Alias(e.copy(aggregateFunction = af), e.toString)()
    +
    +        // Get the result of the first aggregate in the last aggregate.
    +        val b = AggregateExpression2(
    +          aggregate.First(evalWithinGroup(regularGroupId, a.toAttribute), Literal(true)),
    +          mode = Complete,
    +          isDistinct = false)
    +
    +        // Some aggregate functions (COUNT) have the special property that they can return a
    +        // non-null result without any input. We need to make sure we return a result in this case.
    +        val c = af.defaultResult match {
    +          case Some(lit) => Coalesce(Seq(b, lit))
    +          case None => b
    +        }
    +
    +        (e, a, c)
    +      }
    +
    +      // Construct the regular aggregate input projection only if we need one.
    +      val regularAggProjection = if (regularAggExprs.nonEmpty) {
    +        Seq(a.groupingExpressions ++
    +          distinctAggChildren.map(nullify) ++
    +          Seq(regularGroupId) ++
    +          regularAggChildren)
    +      } else {
    +        Seq.empty[Seq[Expression]]
    +      }
    +
    +      // Construct the distinct aggregate input projections.
    +      val regularAggNulls = regularAggChildren.map(nullify)
    +      val distinctAggProjections = distinctAggOperatorMap.map {
    +        case (projection, _) =>
    +          a.groupingExpressions ++
    +            projection ++
    +            regularAggNulls
    +      }
    +
    +      // Construct the expand operator.
    +      val expand = Expand(
    +        regularAggProjection ++ distinctAggProjections,
    +        groupByAttrs ++ distinctAggChildAttrs ++ Seq(gid) ++ regularAggChildAttrMap.values.toSeq,
    +        a.child)
    +
    +      // Construct the first aggregate operator. This de-duplicates the all the children of
    +      // distinct operators, and applies the regular aggregate operators.
    +      val firstAggregateGroupBy = groupByAttrs ++ distinctAggChildAttrs :+ gid
    +      val firstAggregate = Aggregate(
    +        firstAggregateGroupBy,
    +        firstAggregateGroupBy ++ regularAggOperatorMap.map(_._2),
    +        expand)
    +
    +      // Construct the second aggregate
    +      val transformations: Map[Expression, Expression] =
    +        (distinctAggOperatorMap.flatMap(_._2) ++
    +          regularAggOperatorMap.map(e => (e._1, e._3))).toMap
    +
    +      val patchedAggExpressions = a.aggregateExpressions.map { e =>
    +        e.transformDown {
    +          case e: Expression =>
    +            // The same GROUP BY clauses can have different forms (different names for instance) in
    +            // the groupBy and aggregate expressions of an aggregate. This makes a map lookup
    +            // tricky. So we do a linear search for a semantically equal group by expression.
    --- End diff --
    
    We've talked about adding an `ExpressionMap` similar to `AttributeMap` in the past.  Seems like that would be useful here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-154352105
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45207/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-153323299
  
    **[Test build #44915 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44915/consoleFull)** for PR 9406 at commit [`6139f47`](https://github.com/apache/spark/commit/6139f473d4edec53bcbe47008832f67f3ef567fb).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:\n  * `case class Expand(`\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-153312928
  
    ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-153313304
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-9241][SQL] Supporting multiple DISTINCT...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/9406#issuecomment-154342636
  
    **[Test build #45207 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45207/consoleFull)** for PR 9406 at commit [`9be5b9d`](https://github.com/apache/spark/commit/9be5b9d9e3c9de81473ac93750b687e2de824bb2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org