You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2016/11/16 15:59:59 UTC
[jira] [Assigned] (SPARK-18471) In treeAggregate, generate (big)
zeros instead of sending them.
[ https://issues.apache.org/jira/browse/SPARK-18471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-18471:
------------------------------------
Assignee: Apache Spark
> In treeAggregate, generate (big) zeros instead of sending them.
> ---------------------------------------------------------------
>
> Key: SPARK-18471
> URL: https://issues.apache.org/jira/browse/SPARK-18471
> Project: Spark
> Issue Type: Improvement
> Components: MLlib, Spark Core
> Reporter: Anthony Truchet
> Assignee: Apache Spark
> Priority: Minor
>
> When using optimization routine like LBFGS, treeAggregate curently sends the zero vector as part of the closure. This zero can be huge (e.g. ML vectors with millions of zeros) but can be easily generated.
> Several option are possible (upcoming patches to come soon for some of them).
> On is to provide a treeAggregateWithZeroGenerator method (either in core on in MLlib) which wrap treeAggregate in an option and generate the zero if None.
> Another one is to rewrite treeAggregate to wrap an underlying implementation which use a zero generator directly.
> There might be other better alternative we have not spotted...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org