You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Chris Nardi (Jira)" <ji...@apache.org> on 2019/10/17 03:55:00 UTC

[jira] [Created] (SPARK-29496) Add ability to estimate per

Chris Nardi created SPARK-29496:
-----------------------------------

             Summary: Add ability to estimate per
                 Key: SPARK-29496
                 URL: https://issues.apache.org/jira/browse/SPARK-29496
             Project: Spark
          Issue Type: Improvement
          Components: ML
    Affects Versions: 2.4.4
            Reporter: Chris Nardi


In gensim, [the LDA model|[https://radimrehurek.com/gensim/models/ldamodel.html]] has a parameter eval_every that allows a user to specify that the model should be evaluated every X iterations to determine its log perplexity. This helps to determine convergence of the model, and whether or not the proper number of iterations has been chosen. Spark has no similar functionality in its implementation of LDA. This should be added, as it appears the only way to achieve this functionality would be to train models of varying numbers of iterations and evaluate each's log perplexity.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org