You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Xiangrui Meng (JIRA)" <ji...@apache.org> on 2014/07/16 05:06:04 UTC

[jira] [Updated] (SPARK-2361) Decide whether to broadcast or serialize the weights directly in MLlib algorithms

     [ https://issues.apache.org/jira/browse/SPARK-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiangrui Meng updated SPARK-2361:
---------------------------------

    Assignee: Xiangrui Meng

> Decide whether to broadcast or serialize the weights directly in MLlib algorithms
> ---------------------------------------------------------------------------------
>
>                 Key: SPARK-2361
>                 URL: https://issues.apache.org/jira/browse/SPARK-2361
>             Project: Spark
>          Issue Type: Improvement
>          Components: MLlib
>            Reporter: Xiangrui Meng
>            Assignee: Xiangrui Meng
>
> In the current implementation, MLlib serializes weights directly into closure. This is okay for small feature dimension, but not efficient for feature dimensions beyond 1M. Especially the default akka.frameSize is 10m. We should use broadcast when the size of the serialized task is going to be large.



--
This message was sent by Atlassian JIRA
(v6.2#6252)