[jira] [Created] (SPARK-5563) LDA with online variational inference

["Joseph K. Bradley (JIRA)" <ji...@apache.org>]

Joseph K. Bradley created SPARK-5563:
----------------------------------------

             Summary: LDA with online variational inference
                 Key: SPARK-5563
                 URL: https://issues.apache.org/jira/browse/SPARK-5563
             Project: Spark
          Issue Type: Test
          Components: MLlib
    Affects Versions: 1.3.0
            Reporter: Joseph K. Bradley


Latent Dirichlet Allocation (LDA) parameters can be inferred using online variational inference, as in Hoffman, Blei and Bach. “Online Learning for Latent Dirichlet Allocation.”  NIPS, 2010.  This algorithm should be very efficient and should be able to handle much larger datasets than batch algorithms for LDA.

This algorithm will also be important for supporting Streaming versions of LDA.

The implementation will ideally use the same API as the existing LDA but use a different underlying optimizer.

This will require hooking in to the existing mllib.optimization frameworks.

This will require some discussion about whether batch versions of online variational inference should be supported, as well as what variational approximation should be used now or in the future.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org