[jira] [Commented] (MAHOUT-797) MapReduce SSVD: provide alternative B-pipeline per B=R' ^{-1} Y'A

["Dmitriy Lyubimov (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAHOUT-797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159469#comment-13159469 ] 

Dmitriy Lyubimov commented on MAHOUT-797:
-----------------------------------------

so what i think is that using R'^-1YA route is more promising overall for B_0 pipeline. 

The reason being that the version that uses explicit Q must pull in both Q and A while the version that would use R^-1 would only go over A and Omega, but Omega is imaginary. So we trade slightly more flops for less network I/O. This route may turn out significantly more beneficial if we believe that size of Q >> size of A. I imagine this is not too common though, but i have high hopes. 

we may eventually even deploy some automatic default based on subsampling A for sparsity. 

For power iterations pipeline perhaps looses some of it since we don't have random matrix in the works anymore. It seems possible that i might be able to do both Y_i = AB'_i-1 and Y'_i Y_i   in one MR step there without ever forming AB' and that would be a huge plus instead. 

But all of that changes the flow significantly enough to practically build a completely new set of jobs (which is ok). 
 
                
> MapReduce SSVD: provide alternative B-pipeline per B=R' ^{-1} Y'A
> -----------------------------------------------------------------
>
>                 Key: MAHOUT-797
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-797
>             Project: Mahout
>          Issue Type: Improvement
>    Affects Versions: 0.5, 0.6
>            Reporter: Dmitriy Lyubimov
>            Assignee: Dmitriy Lyubimov
>             Fix For: Backlog
>
>         Attachments: MAHOUT-797.pdf
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> Since alternative flow using Cholesky decomposition is extremely easy to add to existing computations of BB', I am thinking of just adding an option that chooses between B-pipeline with QR step and B-pipeline with Y'Y-Cholesky step. 
> Ongoing work and some initial code (Y'Y step) is here https://github.com/dlyubimov/mahout-commits/tree/MAHOUT-797.
> I also want to fix what's left unfixed in MAHOUT-638.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira