[jira] [Commented] (MAHOUT-1035) Hotspot in recommenditembased – UnsymmetrifyMapper job

["Sebastian Schelter (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAHOUT-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403722#comment-13403722 ] 

Sebastian Schelter commented on MAHOUT-1035:
--------------------------------------------

Nice find!

I think the problem with the like() call is that the newly created vector has the same initial capacity as the vector on which like() is invoked (a row of the similarity matrix here). This doesn't make any sense here and potentially wastes a lot of memory.

I think we should simply change the creation to 

{noformat}
Vector transposedPartial = new RandomAccessSparseVector(similarities.size(), 1);
{noformat}

I don't get the second part of your patch where you change OpenKeyTypeValueTypeHashMap. IMHO inserting there shouldn't be a way to early terminate that loop, did I look over something?

                
> Hotspot in recommenditembased – UnsymmetrifyMapper job
> ------------------------------------------------------
>
>                 Key: MAHOUT-1035
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-1035
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Collaborative Filtering
>    Affects Versions: 0.7
>            Reporter: Bhaskar Devireddy
>            Assignee: Sean Owen
>            Priority: Minor
>             Fix For: 0.8
>
>         Attachments: patch_1035.patch
>
>
> While profiling the unsymmetrify mapper job in recommendations we noticed an hotspot consuming 90% of the CPU runtime in org.apache.mahout.math.map.OpenIntDoubleHashMap.keys method for the first map task.  We used the script provided in mahout examples for running ASF Email recommendations for profiling.  The attached patch addresses  the hotspot by reducing the number of for loop iterations in OpenIntDoubleHashMap.keys method by changing the initialization of transposedPartial.  This patch while retaining functionality(verified the output with and without patch) speeds up the unsymmetrify mapper task by more than 4X on x86 architectures.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira