You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by sbourke <gi...@git.apache.org> on 2014/12/01 10:52:54 UTC
[GitHub] spark pull request: [MLLIB][SPARK-4675] Find similar products and ...
GitHub user sbourke opened a pull request:
https://github.com/apache/spark/pull/3536
[MLLIB][SPARK-4675] Find similar products and similar users in MatrixFactorizationModel
Using the latent feature space that is learnt in MatrixFactorizationModel, I have added 2 new functions to find similar products and similar users. A user of the API can for example pass a product ID, and get the closest products based on the feature space.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sbourke/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3536.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3536
----
commit 956ca1b86aacb22fabd52740ce0c6fef5524bae8
Author: Senior Stefano El Bour-que <st...@schibsted.es>
Date: 2014-11-28T08:40:40Z
added functionality to find similar users and similar products
commit 12e6b6b3a2cbfa1baa29449396e7e85bed1dec56
Author: Steven Bourke <st...@stevens-imac.local>
Date: 2014-11-30T23:22:46Z
added unit test to make sure id isnt teh same
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB][SPARK-4675] Find similar products and ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3536#discussion_r21078944
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala ---
@@ -95,6 +95,35 @@ class MatrixFactorizationModel(
}
/**
+ * Recommends similar products
+ *
+ * @param user the user to find similar users for
+ * @param num how many products to return. The number returned may be less than this.
+ * @return [[Rating]] objects, each of which contains the given user ID, a user ID, and a
+ * "score" in the rating field. Each represents one recommended user, and they are sorted
+ * by score, decreasing. The first returned is the one predicted to be most similar
+ * user to the specified user ID. The score is an opaque value that indicates how strongly
+ * recommended the user is.
+ */
+ def recommendSimilariUsers(user: Int, num: Int): Array[Rating] =
--- End diff --
Typo: `Similari`, also below.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB][SPARK-4675] Find similar products and ...
Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:
https://github.com/apache/spark/pull/3536#issuecomment-66503238
Can we discuss it more on the JIRA ? I updated it with my comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB][SPARK-4675] Find similar products and ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/3536#issuecomment-65393291
I wasn't necessarily suggesting changing the similarity metric although I
ended up using cosine too. Note you can skip normalizing by the target
item's norm.
I suppose my point is that the recommendation computation does not use a
dot product because it is performing a similarity computation. Those
vectors are not even in the same space. So I wouldn't reuse that logic on
the grounds that it is reusing a similarity computation.
On Dec 3, 2014 5:03 AM, "Steven" <no...@github.com> wrote:
> Re: Explaining similarity metric [image: :+1:] I'll do that.
>
> Re: Cosine - no biggie to add. I used dot product because 1) Taking the
> logic that CF is finding "similar" items based on the latent space for a
> user when recommending products and 2) Using dot product would reduce the
> new code added to MatrixFactorizationModel ( I don't want to create clutter
> :)) So [image: :+1:] will change to cosine
>
> Re: Popularity, I'll look into that as well then.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/3536#issuecomment-65391077>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB][SPARK-4675] Find similar products and ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/3536#issuecomment-168112394
I'm going to close this pull request. If this is still relevant and you are interested in pushing it forward, please open a new pull request. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB][SPARK-4675] Find similar products and ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3536#issuecomment-65042076
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB][SPARK-4675] Find similar products and ...
Posted by sbourke <gi...@git.apache.org>.
Github user sbourke commented on the pull request:
https://github.com/apache/spark/pull/3536#issuecomment-65391077
Re: Explaining similarity metric :+1: I'll do that.
Re: Cosine - no biggie to add. I used dot product because 1) Taking the logic that CF is finding "similar" items based on the latent space for a user when recommending products and 2) Using dot product would reduce the new code added to MatrixFactorizationModel ( I don't want to create clutter :)) So :+1: will change to cosine
Re: Popularity, I'll look into that as well then.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB][SPARK-4675] Find similar products and ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3536#issuecomment-96770048
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB][SPARK-4675] Find similar products and ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/3536#issuecomment-65042436
I think it's essential to explain (even in internal comments, or this PR) what the similarity metric is. It's just ranking by dot product, which makes it something like cosine similarity. The differences are that it isn't in [-1,1], and the result doesn't normalize away the length of the feature vectors. This tends to favor popular items, or mean that somewhat less similar items may rank higher because they're popular. I had traditionally viewed that as a negative, and preferred the more standard cosine similarity, but it's certainly up for debate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB][SPARK-4675] Find similar products and ...
Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:
https://github.com/apache/spark/pull/3536#issuecomment-83755168
agreed...Dense BLAS will be a common optimization to item->item, user->user and user->item APIs...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB][SPARK-4675] Find similar products and ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/3536#issuecomment-83753751
Agree, although this is no worse than the existing implementation for recommendation (it reuses it even).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB][SPARK-4675] Find similar products and ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/3536#issuecomment-83749073
I kind of like this functionality; @sbourke are you in a position to continue with this? I think it needs a few typo fixes and commentary about its function. This is really about computing similar users and items rather than recommending items to users or users to items. I also think it has to include dividing by the norms to be cosine similarity.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB][SPARK-4675] Find similar products and ...
Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the pull request:
https://github.com/apache/spark/pull/3536#issuecomment-66331985
I'd agree that cosine similarity is preferred. Can't really think of a case where I've *not* used cosine sim for a similar items or similar users computation. Of course, it could be added as an option potentially (whether to use cosine sim - default - or plain dot product.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB][SPARK-4675] Find similar products and ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3536#issuecomment-93861608
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB][SPARK-4675] Find similar products and ...
Posted by debasish83 <gi...@git.apache.org>.
Github user debasish83 commented on the pull request:
https://github.com/apache/spark/pull/3536#issuecomment-83753525
@srowen we have to be a bit careful since dense blas has to be used...I have a internal version with dot and it needs to be more faster..also one at a time is not a good idea...there has to be block dense matrix * block dense matrix operation...that way we can reuse native dgemm...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB][SPARK-4675] Find similar products and ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3536
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org