[jira] [Assigned] (SPARK-17187) Support using arbitrary Java object as internal aggregation buffer object

["Apache Spark (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-17187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-17187:
------------------------------------

    Assignee: Apache Spark

> Support using arbitrary Java object as internal aggregation buffer object
> -------------------------------------------------------------------------
>
>                 Key: SPARK-17187
>                 URL: https://issues.apache.org/jira/browse/SPARK-17187
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>            Reporter: Sean Zhong
>            Assignee: Apache Spark
>
> *Background*
> For aggregation functions like sum and count, Spark-Sql internally use an aggregation buffer to store the intermediate aggregation result for all aggregation functions. Each aggregation function will occupy a section of aggregation buffer.
> *Problem*
> Currently, Spark-sql only allows a small set of Spark-Sql supported storage data types stored in aggregation buffer, which is not very convenient or performant, there are several typical cases like:
> 1. If the aggregation has a complex model CountMinSketch, it is not very easy to convert the complex model so that it can be stored with limited Spark-sql supported data types.
> 2. It is hard to reuse aggregation class definition defined in existing libraries like algebird.
> 3. It may introduces heavy serialization/deserialization cost when converting a domain model to Spark sql supported data type. For example, the current implementation of `TypedAggregateExpression` requires serialization/de-serialization for each call of update or merge.
> *Proposal*
> We propose:
> 1. Introduces a TypedImperativeAggregate which allows using arbitrary java object as aggregation buffer, with requirements like:
>      -  It is flexible enough that the API allows using any java object as aggregation buffer, so that it is easier to integrate with existing Monoid libraries like algebird.
>      -  We don't need to call serialize/deserialize for each call of update/merge. Instead, only a few serialization/deserialization operations are needed. This is to guarantee the    performance.
> 2.  Refactors `TypedAggregateExpression` to use this new interface, to get higher performance.
> 3. Implements Appro-Percentile and other aggregation functions which has a complex aggregation object with this new interface.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org