[jira] [Updated] (MAHOUT-308) Improve Lanczos to handle extremely large feature sets (without hashing)

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAHOUT-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen updated MAHOUT-308:
-----------------------------

    Resolution: Won't Fix
        Status: Resolved  (was: Patch Available)

Looks like this timed out too... it would at least need to be completely redone on the current code. But do any other recent related changes actually address this? seems like some other stuff has been going on in this code.

> Improve Lanczos to handle extremely large feature sets (without hashing)
> ------------------------------------------------------------------------
>
>                 Key: MAHOUT-308
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-308
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Math
>    Affects Versions: 0.3
>         Environment: all
>            Reporter: Jake Mannix
>            Assignee: Jake Mannix
>             Fix For: 0.5
>
>         Attachments: MAHOUT-308.patch
>
>
> DistributedLanczosSolver currently keeps all Lanczos vectors in memory on the driver (client) computer while Hadoop is iterating.  The memory requirements of this is (desiredRank) * (numColumnsOfInput) * 8bytes, which for desiredRank = a few hundred, starts to cap out usefulness at some-small-number * millions of columns for most commodity hardware.
> The solution (without doing stochastic decomposition) is to persist the Lanczos basis to disk, except for the most recent two vectors.  Some care must be taken in the "orthogonalizeAgainstBasis()" method call, which uses the entire basis.  This part would be slower this way.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira