You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Asher Krim (JIRA)" <ji...@apache.org> on 2017/01/19 19:14:26 UTC

[jira] [Created] (SPARK-19294) improve ml LDA save/load

Asher Krim created SPARK-19294:
----------------------------------

             Summary: improve ml LDA save/load
                 Key: SPARK-19294
                 URL: https://issues.apache.org/jira/browse/SPARK-19294
             Project: Spark
          Issue Type: Bug
            Reporter: Asher Krim


The LDA model in ml has some of the same problems addressed by https://issues.apache.org/jira/browse/SPARK-19247 for word2vec.

An LDA model is on order of `vocabSize` * `k`, which can easily reach 3gb for k=1000 and vocabSize=3m. It's currently saved as a single datum in 1 partition. 

Instead, we should represent the matrix as a list, and use the logic from https://issues.apache.org/jira/browse/SPARK-11994 to pick a reasonable number of partitions.

cc [~josephkb]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org