[jira] Commented: (MAHOUT-559) Compare Recommender output by order of recommendations.

["Lance Norskog (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAHOUT-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12968651#action_12968651 ] 

Lance Norskog commented on MAHOUT-559:
--------------------------------------

This first version finds the common items in the two recommendations, and performs various statistical tests comparing the order of these common items. It does not evaluate the items unique to one recommendation. 

It includes a few statistical tests on the difference in order of the two common sequences:
* A sliding-window Hamming distance
** A match among neighboring positions between vectors counts as a match
* A bubble-sort of one recommendation sequence against the other sequence
** How many swaps must a sorting algorithm use to make one order match the other?
* Statistical Rank
** The Classic Ranking score from the stats world. You're on your own
* A variation on the Wilcoxon ranking score
** Based on this: [A normal-scores alternative to the Wilcoxon test|http://comp9.psych.cornell.edu/Darlington/normscor.htm]
** (Regular Wilcoxon requires a separate lookup table. This was too weird to implement.)

The score ranges from 0 on upwards. A 0 score means they're the same; larger numbers represent large differences. 

CompareRecommenders is not a unit test, but rather a sample main() program for your on testing. You may wish to disable the CSV output option: set the csvOut field to null.




> Compare Recommender output by order of recommendations.
> -------------------------------------------------------
>
>                 Key: MAHOUT-559
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-559
>             Project: Mahout
>          Issue Type: New Feature
>          Components: Collaborative Filtering
>            Reporter: Lance Norskog
>
> OrderBasedRecommenderEvaluator compares the output of two recommenders. It only checks the order of the recommended items, ignoring the returned preferences.
> Purpose: the existing RecommenderEvaluator has a very limited use model. It does not allow comparing the outputs of different recommenders against the same data model. Also, I could not figure out how its comparison criteria.
> OrderBasedRecommenderEvaluator gets a recommendation of N samples for each user available from the datamodel, from both recommenders. It finds the common items in the two recommendations, and performs various statistical tests comparing the order of these common items.  It does not evaluate the items unique to one recommendation. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.