You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2016/03/08 20:28:41 UTC
[jira] [Commented] (SPARK-13749) Faster pivot implementation for
many distinct values with two phase aggregation
[ https://issues.apache.org/jira/browse/SPARK-13749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15185571#comment-15185571 ]
Apache Spark commented on SPARK-13749:
--------------------------------------
User 'aray' has created a pull request for this issue:
https://github.com/apache/spark/pull/11583
> Faster pivot implementation for many distinct values with two phase aggregation
> -------------------------------------------------------------------------------
>
> Key: SPARK-13749
> URL: https://issues.apache.org/jira/browse/SPARK-13749
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Andrew Ray
>
> The existing implementation of pivot translates into a single aggregation with one aggregate per distinct pivot value. When the number of distinct pivot values is large (say 1000+) this can get extremely slow since each input value gets evaluated on every aggregate even though it only affects the value of one of them.
> I'm proposing an alternate strategy for when there are 10+ (somewhat arbitrary threshold) distinct pivot values. We do two phases of aggregation. In the first we group by the grouping columns plus the pivot column and perform the specified aggregations (one or sometimes more). In the second aggregation we group by the grouping columns and use the new (non public) PivotFirst aggregate that rearranges the outputs of the first aggregation into an array indexed by the pivot value. Finally we do a project to extract the array entries into the appropriate output column.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org