[jira] [Commented] (MAHOUT-767) Improve RowSimilarityJob performance

["Sebastian Schelter (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAHOUT-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100265#comment-13100265 ] 

Sebastian Schelter commented on MAHOUT-767:
-------------------------------------------

A summary of my current work so far, a new patch is coming:


We should only support algebraic similarity measures which allows us to use a combiner in the most crucial phase. Furthermore we will use the stripes-pattern for in-mapper combination of cooccurrences to avoid emitting lots of cooccurrence pair objects.

This issue also touches ItemSimilarityJob and RecommenderJob as they use RowSimilarityJob internally. We will introduce a new job responsible for preparing the input data for these jobs.

As the distribution of ratings per user and ratings per item follow power-law distributions usually, appropriate down-sampling is crucial for the performance of these jobs as their runtime is dominated by the user with the largest number of interactions. We should remove the old "maxCooccurrencesPerItem" heuristic as it depends on the number of mappers that are run and the ordering of the input data. A simple random downsampling of users having a number of ratings above a threshold should work better.

> Improve RowSimilarityJob performance
> ------------------------------------
>
>                 Key: MAHOUT-767
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-767
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Collaborative Filtering
>    Affects Versions: 0.5
>            Reporter: Grant Ingersoll
>            Assignee: Sebastian Schelter
>             Fix For: 0.6
>
>         Attachments: MAHOUT-767.patch
>
>
> (See http://www.lucidimagination.com/search/document/40c4f124795c6b5/rowsimilarity_s#42ab816c27c6a9e7 for background)
> Currently, the RowSimilarityJob defers the calculation of the similarity metric until the reduce phase, while emitting many Cooccurrence objects.  For similarity metrics that are algebraic (http://pig.apache.org/docs/r0.8.1/udf.html#Aggregate+Functions) we should be able to do much of the computation during the Mapper part of this phase and also take advantage of a Combiner.  
> We should use a marker interface to know whether a similarity metric is algebraic and then make use of an appropriate Mapper implementation, otherwise we can fall back on our existing implementation.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira