You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yanbo Liang (JIRA)" <ji...@apache.org> on 2017/08/01 06:48:00 UTC
[jira] [Commented] (SPARK-21591) Implement treeAggregate on Dataset
API
[ https://issues.apache.org/jira/browse/SPARK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108476#comment-16108476 ]
Yanbo Liang commented on SPARK-21591:
-------------------------------------
cc [~cloud_fan]
> Implement treeAggregate on Dataset API
> --------------------------------------
>
> Key: SPARK-21591
> URL: https://issues.apache.org/jira/browse/SPARK-21591
> Project: Spark
> Issue Type: Brainstorming
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Yanbo Liang
>
> The Tungsten execution engine substantially improved the efficiency of memory and CPU for Spark application. However, in MLlib we still not migrate the internal computing workload from {{RDD}} to {{DataFrame}}.
> The main block issue is there is no {{treeAggregate}} on {{DataFrame}}. As we all know, {{RDD}} based {{treeAggregate}} reduces the aggregation time by an order of magnitude for lots of MLlib algorithms(https://databricks.com/blog/2014/09/22/spark-1-1-mllib-performance-improvements.html).
> I open this JIRA to discuss to implement {{treeAggregate}} on {{DataFrame}} API and do the performance benchmark related issues.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org