You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Kris Jack (JIRA)" <ji...@apache.org> on 2012/05/01 12:24:51 UTC
[jira] [Created] (MAHOUT-1004) Distributed User-based Collaborative
Filtering
Kris Jack created MAHOUT-1004:
---------------------------------
Summary: Distributed User-based Collaborative Filtering
Key: MAHOUT-1004
URL: https://issues.apache.org/jira/browse/MAHOUT-1004
Project: Mahout
Issue Type: New Feature
Components: Collaborative Filtering
Reporter: Kris Jack
Assignee: Sean Owen
Priority: Minor
I'd like to contribute code that implements a distributed user-based collaborative filtering algorithm.
In brief, so far I've taken the code for the existing org.apache.mahout.cf.taste.hadoop.item.RecommenderJob and created a new org.apache.mahout.cf.taste.hadoop.user.RecommenderJob. With help from Sean Owen, I followed a similar approach to the item-based implementation, but multiplied a user-user matrix with a user-item vector rather than an item-item matrix with an item-user vector. The result of the multiplication then needs to be transposed in order to output recommendations by user id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-1004) Distributed User-based
Collaborative Filtering
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265749#comment-13265749 ]
Sean Owen commented on MAHOUT-1004:
-----------------------------------
It sounds like a fine idea. The ideal thing would be some fancy, refactored approach that didn't duplicate much of anything. The next-best approach would be a cleanly parallel implementation. Next-best would be a different approach.
I imagine the first is not realistic, and I imagine the last is probably not a great thing for the project. Kris is it mostly a parallel implementation? or could you also push back any of your improvements to the .item implementation? With that and avoiding any easily-avoided duplication I bet this would be fine.
> Distributed User-based Collaborative Filtering
> ----------------------------------------------
>
> Key: MAHOUT-1004
> URL: https://issues.apache.org/jira/browse/MAHOUT-1004
> Project: Mahout
> Issue Type: New Feature
> Components: Collaborative Filtering
> Reporter: Kris Jack
> Assignee: Sean Owen
> Priority: Minor
> Labels: Recommender, User-based
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> I'd like to contribute code that implements a distributed user-based collaborative filtering algorithm.
> In brief, so far I've taken the code for the existing org.apache.mahout.cf.taste.hadoop.item.RecommenderJob and created a new org.apache.mahout.cf.taste.hadoop.user.RecommenderJob. With help from Sean Owen, I followed a similar approach to the item-based implementation, but multiplied a user-user matrix with a user-item vector rather than an item-item matrix with an item-user vector. The result of the multiplication then needs to be transposed in order to output recommendations by user id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-1004) Distributed User-based
Collaborative Filtering
Posted by "Sebastian Schelter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265738#comment-13265738 ]
Sebastian Schelter commented on MAHOUT-1004:
--------------------------------------------
I think it would be good to also have a userbased approach. We should try to avoid code duplication however.
> Distributed User-based Collaborative Filtering
> ----------------------------------------------
>
> Key: MAHOUT-1004
> URL: https://issues.apache.org/jira/browse/MAHOUT-1004
> Project: Mahout
> Issue Type: New Feature
> Components: Collaborative Filtering
> Reporter: Kris Jack
> Assignee: Sean Owen
> Priority: Minor
> Labels: Recommender, User-based
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> I'd like to contribute code that implements a distributed user-based collaborative filtering algorithm.
> In brief, so far I've taken the code for the existing org.apache.mahout.cf.taste.hadoop.item.RecommenderJob and created a new org.apache.mahout.cf.taste.hadoop.user.RecommenderJob. With help from Sean Owen, I followed a similar approach to the item-based implementation, but multiplied a user-user matrix with a user-item vector rather than an item-item matrix with an item-user vector. The result of the multiplication then needs to be transposed in order to output recommendations by user id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-1004) Distributed User-based
Collaborative Filtering
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265808#comment-13265808 ]
Sean Owen commented on MAHOUT-1004:
-----------------------------------
If you can refactor to a reasonable degree I'm sure this is pretty fine. Use your judgment and/or post partial patches as you go for feedback, especially from Sebastian.
> Distributed User-based Collaborative Filtering
> ----------------------------------------------
>
> Key: MAHOUT-1004
> URL: https://issues.apache.org/jira/browse/MAHOUT-1004
> Project: Mahout
> Issue Type: New Feature
> Components: Collaborative Filtering
> Reporter: Kris Jack
> Assignee: Sean Owen
> Priority: Minor
> Labels: Recommender, User-based
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> I'd like to contribute code that implements a distributed user-based collaborative filtering algorithm.
> In brief, so far I've taken the code for the existing org.apache.mahout.cf.taste.hadoop.item.RecommenderJob and created a new org.apache.mahout.cf.taste.hadoop.user.RecommenderJob. With help from Sean Owen, I followed a similar approach to the item-based implementation, but multiplied a user-user matrix with a user-item vector rather than an item-item matrix with an item-user vector. The result of the multiplication then needs to be transposed in order to output recommendations by user id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-1004) Distributed User-based
Collaborative Filtering
Posted by "Kris Jack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265767#comment-13265767 ]
Kris Jack commented on MAHOUT-1004:
-----------------------------------
Sean, the user-based implementation is a variation of the item-based one. At the moment, there's lots of duplication. If you don't mind me refactoring the item-based code then I think that I can pretty much eliminate all of the duplication.
> Distributed User-based Collaborative Filtering
> ----------------------------------------------
>
> Key: MAHOUT-1004
> URL: https://issues.apache.org/jira/browse/MAHOUT-1004
> Project: Mahout
> Issue Type: New Feature
> Components: Collaborative Filtering
> Reporter: Kris Jack
> Assignee: Sean Owen
> Priority: Minor
> Labels: Recommender, User-based
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> I'd like to contribute code that implements a distributed user-based collaborative filtering algorithm.
> In brief, so far I've taken the code for the existing org.apache.mahout.cf.taste.hadoop.item.RecommenderJob and created a new org.apache.mahout.cf.taste.hadoop.user.RecommenderJob. With help from Sean Owen, I followed a similar approach to the item-based implementation, but multiplied a user-user matrix with a user-item vector rather than an item-item matrix with an item-user vector. The result of the multiplication then needs to be transposed in order to output recommendations by user id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-1004) Distributed User-based Collaborative
Filtering
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated MAHOUT-1004:
------------------------------
Assignee: (was: Sean Owen)
> Distributed User-based Collaborative Filtering
> ----------------------------------------------
>
> Key: MAHOUT-1004
> URL: https://issues.apache.org/jira/browse/MAHOUT-1004
> Project: Mahout
> Issue Type: New Feature
> Components: Collaborative Filtering
> Affects Versions: 0.7
> Reporter: Kris Jack
> Priority: Minor
> Labels: Recommender, User-based
> Attachments: MAHOUT-1004.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> I'd like to contribute code that implements a distributed user-based collaborative filtering algorithm.
> In brief, so far I've taken the code for the existing org.apache.mahout.cf.taste.hadoop.item.RecommenderJob and created a new org.apache.mahout.cf.taste.hadoop.user.RecommenderJob. With help from Sean Owen, I followed a similar approach to the item-based implementation, but multiplied a user-user matrix with a user-item vector rather than an item-item matrix with an item-user vector. The result of the multiplication then needs to be transposed in order to output recommendations by user id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-1004) Distributed User-based Collaborative
Filtering
Posted by "Kris Jack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kris Jack updated MAHOUT-1004:
------------------------------
Attachment: MAHOUT-1004.patch
This patch is applied to the core directory.
> Distributed User-based Collaborative Filtering
> ----------------------------------------------
>
> Key: MAHOUT-1004
> URL: https://issues.apache.org/jira/browse/MAHOUT-1004
> Project: Mahout
> Issue Type: New Feature
> Components: Collaborative Filtering
> Affects Versions: 0.7
> Reporter: Kris Jack
> Assignee: Sean Owen
> Priority: Minor
> Labels: Recommender, User-based
> Attachments: MAHOUT-1004.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> I'd like to contribute code that implements a distributed user-based collaborative filtering algorithm.
> In brief, so far I've taken the code for the existing org.apache.mahout.cf.taste.hadoop.item.RecommenderJob and created a new org.apache.mahout.cf.taste.hadoop.user.RecommenderJob. With help from Sean Owen, I followed a similar approach to the item-based implementation, but multiplied a user-user matrix with a user-item vector rather than an item-item matrix with an item-user vector. The result of the multiplication then needs to be transposed in order to output recommendations by user id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-1004) Distributed User-based Collaborative
Filtering
Posted by "Kris Jack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kris Jack updated MAHOUT-1004:
------------------------------
Affects Version/s: 0.7
Status: Patch Available (was: Open)
First attempt to refactor the item-based code to include user-based functionality. I have only written high level unit tests that demonstrate that the whole job runs so far because I think we're still at the stage when we need to discuss whether this refactoring is appropriate or not. I have fixed an existing unit test that was failing because of my changes though (RecommenderJobTest).
Here's a description of the changes...
org.apache.mahout.cf.taste.hadoop.user.RecommenderJob has been created. It's similar to the item-based version of this job and the two could conceivably be merged together or rely upon common classes. I've left it as a separate job for the moment so as not to make this already fat patch even fatter.
org.apache.mahout.cf.taste.hadoop.preparation.PreparePreferenceMatrixJob has the option to skip encoding item ids as ints. I had to remove the int-long mapping because it wouldn't fit in memory when I was running large scale experiments. I understand that you want to keep it in so that it's compatible with the Taste code so I've just added a flag that allows it to be skipped.
org.apache.mahout.math.hadoop.similarity.cooccurrence.RowSimilarityJob can be column or row centric with the option "outputColumns":
- 2 versions of VectorNormMapper have been created, ColumnVectorNormMapper and RowVectorNormMapper, one is needed for item-based, the other for user-based. I pulled these out to make the separate classes and removed the VectorNormMapper.
- pairwiseSimilarity operates on either the weightsPath or the input path, the first being needed for item-based, the second for user-based
org.apache.mahout.cf.taste.hadoop.item.UserVectorSplitterMapper can now run as item-based or user-based:
- item-based is the default behaviour and does not change the item-based algorithm
- user-based changes what is output by the mapper, swapping the item index and the user id around
I need to break the final PartialMultiplyMapper-AggregateAndRecommendReducer step in the item-based recommender into 2 MR jobs for the user-based version. This is because we need an extra transposition step since the results are generated keyed on item rather than user. The new MR jobs are:
- PartialMultiplyMapper-VectorAdditionReducer that multiplies and adds vectors together
- TransposeMapper-VectorToRecommendationsReducer that transposes the results and outputs them as RecommendedItemsWritable
As you can imagine though, lots of the code in these new MR Jobs is the same as the original AggregateAndRecommendReducer job so I pulled out 2 common classes, naming them VectorAdditionUtils and RecommendationsWriter. I haven't refactored the AggregateAndRecommendReducer job but it would be relatively straight-forward to eliminate code duplicate here by having it call the new VectorAdditionUtils and RecommendationsWriter.
Also, so far, I have only been testing this with the co-occurrence metric. I'm not sure if the weights are being calculated correctly in the user-based version of the RowSimilarityJob.
> Distributed User-based Collaborative Filtering
> ----------------------------------------------
>
> Key: MAHOUT-1004
> URL: https://issues.apache.org/jira/browse/MAHOUT-1004
> Project: Mahout
> Issue Type: New Feature
> Components: Collaborative Filtering
> Affects Versions: 0.7
> Reporter: Kris Jack
> Assignee: Sean Owen
> Priority: Minor
> Labels: Recommender, User-based
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> I'd like to contribute code that implements a distributed user-based collaborative filtering algorithm.
> In brief, so far I've taken the code for the existing org.apache.mahout.cf.taste.hadoop.item.RecommenderJob and created a new org.apache.mahout.cf.taste.hadoop.user.RecommenderJob. With help from Sean Owen, I followed a similar approach to the item-based implementation, but multiplied a user-user matrix with a user-item vector rather than an item-item matrix with an item-user vector. The result of the multiplication then needs to be transposed in order to output recommendations by user id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-1004) Distributed User-based
Collaborative Filtering
Posted by "Sami Nieminen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488597#comment-13488597 ]
Sami Nieminen commented on MAHOUT-1004:
---------------------------------------
Which version of mahout contains the user-based recommender? I still couldn't find it from the 0.8-SNAPSHOT.
> Distributed User-based Collaborative Filtering
> ----------------------------------------------
>
> Key: MAHOUT-1004
> URL: https://issues.apache.org/jira/browse/MAHOUT-1004
> Project: Mahout
> Issue Type: New Feature
> Components: Collaborative Filtering
> Affects Versions: 0.7
> Reporter: Kris Jack
> Priority: Minor
> Labels: Recommender, User-based
> Attachments: MAHOUT-1004.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> I'd like to contribute code that implements a distributed user-based collaborative filtering algorithm.
> In brief, so far I've taken the code for the existing org.apache.mahout.cf.taste.hadoop.item.RecommenderJob and created a new org.apache.mahout.cf.taste.hadoop.user.RecommenderJob. With help from Sean Owen, I followed a similar approach to the item-based implementation, but multiplied a user-user matrix with a user-item vector rather than an item-item matrix with an item-user vector. The result of the multiplication then needs to be transposed in order to output recommendations by user id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira