You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Xiangrui Meng (JIRA)" <ji...@apache.org> on 2015/01/23 07:09:40 UTC

[jira] [Resolved] (SPARK-3541) Improve ALS internal storage

     [ https://issues.apache.org/jira/browse/SPARK-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiangrui Meng resolved SPARK-3541.
----------------------------------
       Resolution: Fixed
    Fix Version/s: 1.3.0

Issue resolved by pull request 3720
[https://github.com/apache/spark/pull/3720]

> Improve ALS internal storage
> ----------------------------
>
>                 Key: SPARK-3541
>                 URL: https://issues.apache.org/jira/browse/SPARK-3541
>             Project: Spark
>          Issue Type: Improvement
>          Components: MLlib
>            Reporter: Xiangrui Meng
>            Assignee: Xiangrui Meng
>             Fix For: 1.3.0
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> The internal storage of ALS uses many small objects, which increases the GC pressure and makes ALS difficult to scale to very large scale, e.g., 50 billion ratings. In such cases, the full GC may take more than 10 minutes to finish. That is longer than the default heartbeat timeout and hence executors will be removed under default settings.
> We can use primitive arrays to reduce the number of objects significantly. This requires big change to the ALS implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org