You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2017/01/18 14:25:26 UTC

[jira] [Resolved] (SPARK-13335) Optimize Data Frames collect_list and collect_set with declarative aggregates

     [ https://issues.apache.org/jira/browse/SPARK-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-13335.
----------------------------------
    Resolution: Duplicate

I am resolving this per committer's comments - https://github.com/apache/spark/pull/12874#issuecomment-216632661 and https://github.com/apache/spark/pull/11688#issuecomment-220496380

Please reopen this if I misunderstood this.

> Optimize Data Frames collect_list and collect_set with declarative aggregates
> -----------------------------------------------------------------------------
>
>                 Key: SPARK-13335
>                 URL: https://issues.apache.org/jira/browse/SPARK-13335
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Matt Cheah
>            Priority: Minor
>
> Based on discussion from SPARK-9301, we can optimize collect_set and collect_list with declarative aggregate expressions, as opposed to using Hive UDAFs. The problem with Hive UDAFs is that they require converting the data items from catalyst types back to external types repeatedly. We can get around this by implementing declarative aggregate expressions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org