You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Debasish Das <de...@gmail.com> on 2014/10/29 19:23:57 UTC
matrix factorization cross validation
Hi,
In the current factorization flow, we cross validate on the test dataset
using the RMSE number but there are some other measures which are worth
looking into.
If we consider the problem as a regression problem and the ratings 1-5 are
considered as 5 classes, it is possible to generate a confusion matrix
using MultiClassMetrics.scala
If the ratings are only 0/1 (like from the spotify demo from spark summit)
then it is possible to use Binary Classification Metrices to come up with
the ROC curve...
For topK user/products we should also look into prec@k and pdcg@k as the
metric..
Does it make sense to add the multiclass metric and prec@k, pdcg@k in
examples.MovielensALS along with RMSE ?
Thanks.
Deb
Re: matrix factorization cross validation
Posted by Debasish Das <de...@gmail.com>.
I added the drivers for precisionAt(k: Int) driver for the movielens
test-cases...Although I am a bit confused on precisionAt(k: Int) code from
RankingMetrics.scala...
While cross validating, I am really not sure how to set K...
if (labSet.nonEmpty) { val n = math.min(pred.length, k) ... }
If I make k as a function of pred.length val n = math.min(pred.length,
k*pred.length) then I can vary k between 0 and 1 and choose the sweet spot
for K on a given dataset but I am not sure if it is a measure that makes
sense for recommendation...
MAP is something that makes sense as it is average over all test set...
On Fri, Oct 31, 2014 at 1:26 AM, Sean Owen <so...@cloudera.com> wrote:
> No, excepting approximate methods like LSH to figure out the
> relatively small set of candidates for the users in the partition, and
> broadcast or join those.
>
> On Fri, Oct 31, 2014 at 5:45 AM, Nick Pentreath
> <ni...@gmail.com> wrote:
> > Sean, re my point earlier do you know a more efficient way to compute
> top k
> > for each user, other than to broadcast the item factors?
> >
> > (I guess one can use the new asymmetric lsh paper perhaps to assist)
> >
> > —
> > Sent from Mailbox
> >
> >
> > On Thu, Oct 30, 2014 at 11:24 PM, Sean Owen <so...@cloudera.com> wrote:
> >>
> >> MAP is effectively an average over all k from 1 to min(#
> >> recommendations, # items rated) Getting first recommendations right is
> >> more important than the last.
> >>
> >> On Thu, Oct 30, 2014 at 10:21 PM, Debasish Das <
> debasish.das83@gmail.com>
> >> wrote:
> >> > Does it make sense to have a user specific K or K is considered same
> >> > over
> >> > all users ?
> >> >
> >> > Intuitively the users who watches more movies should get a higher K
> than
> >> > the
> >> > others...
> >> >
> >
> >
>
Re: matrix factorization cross validation
Posted by Sean Owen <so...@cloudera.com>.
No, excepting approximate methods like LSH to figure out the
relatively small set of candidates for the users in the partition, and
broadcast or join those.
On Fri, Oct 31, 2014 at 5:45 AM, Nick Pentreath
<ni...@gmail.com> wrote:
> Sean, re my point earlier do you know a more efficient way to compute top k
> for each user, other than to broadcast the item factors?
>
> (I guess one can use the new asymmetric lsh paper perhaps to assist)
>
> —
> Sent from Mailbox
>
>
> On Thu, Oct 30, 2014 at 11:24 PM, Sean Owen <so...@cloudera.com> wrote:
>>
>> MAP is effectively an average over all k from 1 to min(#
>> recommendations, # items rated) Getting first recommendations right is
>> more important than the last.
>>
>> On Thu, Oct 30, 2014 at 10:21 PM, Debasish Das <de...@gmail.com>
>> wrote:
>> > Does it make sense to have a user specific K or K is considered same
>> > over
>> > all users ?
>> >
>> > Intuitively the users who watches more movies should get a higher K than
>> > the
>> > others...
>> >
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: matrix factorization cross validation
Posted by Nick Pentreath <ni...@gmail.com>.
Sean, re my point earlier do you know a more efficient way to compute top k for each user, other than to broadcast the item factors?
(I guess one can use the new asymmetric lsh paper perhaps to assist)
—
Sent from Mailbox
On Thu, Oct 30, 2014 at 11:24 PM, Sean Owen <so...@cloudera.com> wrote:
> MAP is effectively an average over all k from 1 to min(#
> recommendations, # items rated) Getting first recommendations right is
> more important than the last.
> On Thu, Oct 30, 2014 at 10:21 PM, Debasish Das <de...@gmail.com> wrote:
>> Does it make sense to have a user specific K or K is considered same over
>> all users ?
>>
>> Intuitively the users who watches more movies should get a higher K than the
>> others...
>>
Re: matrix factorization cross validation
Posted by Sean Owen <so...@cloudera.com>.
MAP is effectively an average over all k from 1 to min(#
recommendations, # items rated) Getting first recommendations right is
more important than the last.
On Thu, Oct 30, 2014 at 10:21 PM, Debasish Das <de...@gmail.com> wrote:
> Does it make sense to have a user specific K or K is considered same over
> all users ?
>
> Intuitively the users who watches more movies should get a higher K than the
> others...
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: matrix factorization cross validation
Posted by Debasish Das <de...@gmail.com>.
Does it make sense to have a user specific K or K is considered same over
all users ?
Intuitively the users who watches more movies should get a higher K than
the others...
On Thu, Oct 30, 2014 at 2:15 PM, Sean Owen <so...@cloudera.com> wrote:
> The pretty standard metric for recommenders is mean average precision,
> and RankingMetrics will already do that as-is. I don't know that a
> confusion matrix for this binary classification does much.
>
>
> On Thu, Oct 30, 2014 at 9:41 PM, Debasish Das <de...@gmail.com>
> wrote:
> > I am working on it...I will open up a JIRA once I see some results..
> >
> > Idea is to come up with a test train set based on users...basically for
> > each user, we come up with 80% train data and 20% test data...
> >
> > Now we pick up a K (each user should have a different K based on the
> movies
> > he watched so some multiplier) and then we get topK for each user and see
> > the confusion matrix for each user...
> >
> > This data will also go to RankingMetrics I think...one is ground truth
> > array and the other is our prediction...I would like to see the raw
> > confusions as well..
> >
> > These measures are necessary to validate any of the topic modeling
> > algorithms as well...
> >
> > Is there a better place for it other than mllib examples ?
> >
> > On Thu, Oct 30, 2014 at 8:13 AM, Debasish Das <de...@gmail.com>
> > wrote:
> >
> >> I thought topK will save us...for each user we have 1xrank...now our
> movie
> >> factor is a RDD...we pick topK movie factors based on vector
> norm...with K
> >> = 50, we will have 50 vectors * num_executors in a RDD...with the user
> >> 1xrank we do a distributed dot product using RowMatrix APIs...
> >>
> >> May be we can't find topK using vector norm on movie factors...
> >>
> >> On Thu, Oct 30, 2014 at 1:12 AM, Nick Pentreath <
> nick.pentreath@gmail.com>
> >> wrote:
> >>
> >>> Looking at
> >>>
> https://github.com/apache/spark/blob/814a9cd7fabebf2a06f7e2e5d46b6a2b28b917c2/mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala#L82
> >>>
> >>> For each user in test set, you generate an Array of top K predicted
> item
> >>> ids (Int or String probably), and an Array of ground truth item ids
> (the
> >>> known rated or liked items in the test set for that user), and pass
> that to
> >>> precisionAt(k) to compute MAP@k (Actually this method name is a bit
> >>> misleading - it should be meanAveragePrecisionAt where the other method
> >>> there is without a cutoff at k. However, both compute MAP).
> >>>
> >>> The challenge at scale is actually computing all the top Ks for each
> >>> user, as it requires broadcasting all the item factors (unless there
> is a
> >>> smarter way?)
> >>>
> >>> I wonder if it is possible to extend the DIMSUM idea to computing top K
> >>> matrix multiply between the user and item factor matrices, as opposed
> to
> >>> all-pairs similarity of one matrix?
> >>>
> >>> On Thu, Oct 30, 2014 at 5:28 AM, Debasish Das <
> debasish.das83@gmail.com>
> >>> wrote:
> >>>
> >>>> Is there an example of how to use RankingMetrics ?
> >>>>
> >>>> Let's take the user, document example...we get user x topic and
> document
> >>>> x
> >>>> topic matrices as the model...
> >>>>
> >>>> Now for each user, we can generate topK document by doing a sort on
> (1 x
> >>>> topic)dot(topic x document) and picking topK...
> >>>>
> >>>> Is it possible to validate such a topK finding algorithm using
> >>>> RankingMetrics ?
> >>>>
> >>>>
> >>>> On Wed, Oct 29, 2014 at 12:14 PM, Xiangrui Meng <me...@gmail.com>
> >>>> wrote:
> >>>>
> >>>> > Let's narrow the context from matrix factorization to recommendation
> >>>> > via ALS. It adds extra complexity if we treat it as a multi-class
> >>>> > classification problem. ALS only outputs a single value for each
> >>>> > prediction, which is hard to convert to probability distribution
> over
> >>>> > the 5 rating levels. Treating it as a binary classification problem
> or
> >>>> > a ranking problem does make sense. The RankingMetricc is in master.
> >>>> > Free free to add prec@k and ndcg@k to examples.MovielensALS. ROC
> >>>> > should be good to add as well. -Xiangrui
> >>>> >
> >>>> >
> >>>> > On Wed, Oct 29, 2014 at 11:23 AM, Debasish Das <
> >>>> debasish.das83@gmail.com>
> >>>> > wrote:
> >>>> > > Hi,
> >>>> > >
> >>>> > > In the current factorization flow, we cross validate on the test
> >>>> dataset
> >>>> > > using the RMSE number but there are some other measures which are
> >>>> worth
> >>>> > > looking into.
> >>>> > >
> >>>> > > If we consider the problem as a regression problem and the ratings
> >>>> 1-5
> >>>> > are
> >>>> > > considered as 5 classes, it is possible to generate a confusion
> >>>> matrix
> >>>> > > using MultiClassMetrics.scala
> >>>> > >
> >>>> > > If the ratings are only 0/1 (like from the spotify demo from spark
> >>>> > summit)
> >>>> > > then it is possible to use Binary Classification Metrices to come
> up
> >>>> with
> >>>> > > the ROC curve...
> >>>> > >
> >>>> > > For topK user/products we should also look into prec@k and pdcg@k
> >>>> as the
> >>>> > > metric..
> >>>> > >
> >>>> > > Does it make sense to add the multiclass metric and prec@k,
> pdcg@k
> >>>> in
> >>>> > > examples.MovielensALS along with RMSE ?
> >>>> > >
> >>>> > > Thanks.
> >>>> > > Deb
> >>>> >
> >>>>
> >>>
> >>>
> >>
>
Re: matrix factorization cross validation
Posted by Sean Owen <so...@cloudera.com>.
The pretty standard metric for recommenders is mean average precision,
and RankingMetrics will already do that as-is. I don't know that a
confusion matrix for this binary classification does much.
On Thu, Oct 30, 2014 at 9:41 PM, Debasish Das <de...@gmail.com> wrote:
> I am working on it...I will open up a JIRA once I see some results..
>
> Idea is to come up with a test train set based on users...basically for
> each user, we come up with 80% train data and 20% test data...
>
> Now we pick up a K (each user should have a different K based on the movies
> he watched so some multiplier) and then we get topK for each user and see
> the confusion matrix for each user...
>
> This data will also go to RankingMetrics I think...one is ground truth
> array and the other is our prediction...I would like to see the raw
> confusions as well..
>
> These measures are necessary to validate any of the topic modeling
> algorithms as well...
>
> Is there a better place for it other than mllib examples ?
>
> On Thu, Oct 30, 2014 at 8:13 AM, Debasish Das <de...@gmail.com>
> wrote:
>
>> I thought topK will save us...for each user we have 1xrank...now our movie
>> factor is a RDD...we pick topK movie factors based on vector norm...with K
>> = 50, we will have 50 vectors * num_executors in a RDD...with the user
>> 1xrank we do a distributed dot product using RowMatrix APIs...
>>
>> May be we can't find topK using vector norm on movie factors...
>>
>> On Thu, Oct 30, 2014 at 1:12 AM, Nick Pentreath <ni...@gmail.com>
>> wrote:
>>
>>> Looking at
>>> https://github.com/apache/spark/blob/814a9cd7fabebf2a06f7e2e5d46b6a2b28b917c2/mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala#L82
>>>
>>> For each user in test set, you generate an Array of top K predicted item
>>> ids (Int or String probably), and an Array of ground truth item ids (the
>>> known rated or liked items in the test set for that user), and pass that to
>>> precisionAt(k) to compute MAP@k (Actually this method name is a bit
>>> misleading - it should be meanAveragePrecisionAt where the other method
>>> there is without a cutoff at k. However, both compute MAP).
>>>
>>> The challenge at scale is actually computing all the top Ks for each
>>> user, as it requires broadcasting all the item factors (unless there is a
>>> smarter way?)
>>>
>>> I wonder if it is possible to extend the DIMSUM idea to computing top K
>>> matrix multiply between the user and item factor matrices, as opposed to
>>> all-pairs similarity of one matrix?
>>>
>>> On Thu, Oct 30, 2014 at 5:28 AM, Debasish Das <de...@gmail.com>
>>> wrote:
>>>
>>>> Is there an example of how to use RankingMetrics ?
>>>>
>>>> Let's take the user, document example...we get user x topic and document
>>>> x
>>>> topic matrices as the model...
>>>>
>>>> Now for each user, we can generate topK document by doing a sort on (1 x
>>>> topic)dot(topic x document) and picking topK...
>>>>
>>>> Is it possible to validate such a topK finding algorithm using
>>>> RankingMetrics ?
>>>>
>>>>
>>>> On Wed, Oct 29, 2014 at 12:14 PM, Xiangrui Meng <me...@gmail.com>
>>>> wrote:
>>>>
>>>> > Let's narrow the context from matrix factorization to recommendation
>>>> > via ALS. It adds extra complexity if we treat it as a multi-class
>>>> > classification problem. ALS only outputs a single value for each
>>>> > prediction, which is hard to convert to probability distribution over
>>>> > the 5 rating levels. Treating it as a binary classification problem or
>>>> > a ranking problem does make sense. The RankingMetricc is in master.
>>>> > Free free to add prec@k and ndcg@k to examples.MovielensALS. ROC
>>>> > should be good to add as well. -Xiangrui
>>>> >
>>>> >
>>>> > On Wed, Oct 29, 2014 at 11:23 AM, Debasish Das <
>>>> debasish.das83@gmail.com>
>>>> > wrote:
>>>> > > Hi,
>>>> > >
>>>> > > In the current factorization flow, we cross validate on the test
>>>> dataset
>>>> > > using the RMSE number but there are some other measures which are
>>>> worth
>>>> > > looking into.
>>>> > >
>>>> > > If we consider the problem as a regression problem and the ratings
>>>> 1-5
>>>> > are
>>>> > > considered as 5 classes, it is possible to generate a confusion
>>>> matrix
>>>> > > using MultiClassMetrics.scala
>>>> > >
>>>> > > If the ratings are only 0/1 (like from the spotify demo from spark
>>>> > summit)
>>>> > > then it is possible to use Binary Classification Metrices to come up
>>>> with
>>>> > > the ROC curve...
>>>> > >
>>>> > > For topK user/products we should also look into prec@k and pdcg@k
>>>> as the
>>>> > > metric..
>>>> > >
>>>> > > Does it make sense to add the multiclass metric and prec@k, pdcg@k
>>>> in
>>>> > > examples.MovielensALS along with RMSE ?
>>>> > >
>>>> > > Thanks.
>>>> > > Deb
>>>> >
>>>>
>>>
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: matrix factorization cross validation
Posted by Debasish Das <de...@gmail.com>.
I am working on it...I will open up a JIRA once I see some results..
Idea is to come up with a test train set based on users...basically for
each user, we come up with 80% train data and 20% test data...
Now we pick up a K (each user should have a different K based on the movies
he watched so some multiplier) and then we get topK for each user and see
the confusion matrix for each user...
This data will also go to RankingMetrics I think...one is ground truth
array and the other is our prediction...I would like to see the raw
confusions as well..
These measures are necessary to validate any of the topic modeling
algorithms as well...
Is there a better place for it other than mllib examples ?
On Thu, Oct 30, 2014 at 8:13 AM, Debasish Das <de...@gmail.com>
wrote:
> I thought topK will save us...for each user we have 1xrank...now our movie
> factor is a RDD...we pick topK movie factors based on vector norm...with K
> = 50, we will have 50 vectors * num_executors in a RDD...with the user
> 1xrank we do a distributed dot product using RowMatrix APIs...
>
> May be we can't find topK using vector norm on movie factors...
>
> On Thu, Oct 30, 2014 at 1:12 AM, Nick Pentreath <ni...@gmail.com>
> wrote:
>
>> Looking at
>> https://github.com/apache/spark/blob/814a9cd7fabebf2a06f7e2e5d46b6a2b28b917c2/mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala#L82
>>
>> For each user in test set, you generate an Array of top K predicted item
>> ids (Int or String probably), and an Array of ground truth item ids (the
>> known rated or liked items in the test set for that user), and pass that to
>> precisionAt(k) to compute MAP@k (Actually this method name is a bit
>> misleading - it should be meanAveragePrecisionAt where the other method
>> there is without a cutoff at k. However, both compute MAP).
>>
>> The challenge at scale is actually computing all the top Ks for each
>> user, as it requires broadcasting all the item factors (unless there is a
>> smarter way?)
>>
>> I wonder if it is possible to extend the DIMSUM idea to computing top K
>> matrix multiply between the user and item factor matrices, as opposed to
>> all-pairs similarity of one matrix?
>>
>> On Thu, Oct 30, 2014 at 5:28 AM, Debasish Das <de...@gmail.com>
>> wrote:
>>
>>> Is there an example of how to use RankingMetrics ?
>>>
>>> Let's take the user, document example...we get user x topic and document
>>> x
>>> topic matrices as the model...
>>>
>>> Now for each user, we can generate topK document by doing a sort on (1 x
>>> topic)dot(topic x document) and picking topK...
>>>
>>> Is it possible to validate such a topK finding algorithm using
>>> RankingMetrics ?
>>>
>>>
>>> On Wed, Oct 29, 2014 at 12:14 PM, Xiangrui Meng <me...@gmail.com>
>>> wrote:
>>>
>>> > Let's narrow the context from matrix factorization to recommendation
>>> > via ALS. It adds extra complexity if we treat it as a multi-class
>>> > classification problem. ALS only outputs a single value for each
>>> > prediction, which is hard to convert to probability distribution over
>>> > the 5 rating levels. Treating it as a binary classification problem or
>>> > a ranking problem does make sense. The RankingMetricc is in master.
>>> > Free free to add prec@k and ndcg@k to examples.MovielensALS. ROC
>>> > should be good to add as well. -Xiangrui
>>> >
>>> >
>>> > On Wed, Oct 29, 2014 at 11:23 AM, Debasish Das <
>>> debasish.das83@gmail.com>
>>> > wrote:
>>> > > Hi,
>>> > >
>>> > > In the current factorization flow, we cross validate on the test
>>> dataset
>>> > > using the RMSE number but there are some other measures which are
>>> worth
>>> > > looking into.
>>> > >
>>> > > If we consider the problem as a regression problem and the ratings
>>> 1-5
>>> > are
>>> > > considered as 5 classes, it is possible to generate a confusion
>>> matrix
>>> > > using MultiClassMetrics.scala
>>> > >
>>> > > If the ratings are only 0/1 (like from the spotify demo from spark
>>> > summit)
>>> > > then it is possible to use Binary Classification Metrices to come up
>>> with
>>> > > the ROC curve...
>>> > >
>>> > > For topK user/products we should also look into prec@k and pdcg@k
>>> as the
>>> > > metric..
>>> > >
>>> > > Does it make sense to add the multiclass metric and prec@k, pdcg@k
>>> in
>>> > > examples.MovielensALS along with RMSE ?
>>> > >
>>> > > Thanks.
>>> > > Deb
>>> >
>>>
>>
>>
>
Re: matrix factorization cross validation
Posted by Debasish Das <de...@gmail.com>.
I thought topK will save us...for each user we have 1xrank...now our movie
factor is a RDD...we pick topK movie factors based on vector norm...with K
= 50, we will have 50 vectors * num_executors in a RDD...with the user
1xrank we do a distributed dot product using RowMatrix APIs...
May be we can't find topK using vector norm on movie factors...
On Thu, Oct 30, 2014 at 1:12 AM, Nick Pentreath <ni...@gmail.com>
wrote:
> Looking at
> https://github.com/apache/spark/blob/814a9cd7fabebf2a06f7e2e5d46b6a2b28b917c2/mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala#L82
>
> For each user in test set, you generate an Array of top K predicted item
> ids (Int or String probably), and an Array of ground truth item ids (the
> known rated or liked items in the test set for that user), and pass that to
> precisionAt(k) to compute MAP@k (Actually this method name is a bit
> misleading - it should be meanAveragePrecisionAt where the other method
> there is without a cutoff at k. However, both compute MAP).
>
> The challenge at scale is actually computing all the top Ks for each user,
> as it requires broadcasting all the item factors (unless there is a smarter
> way?)
>
> I wonder if it is possible to extend the DIMSUM idea to computing top K
> matrix multiply between the user and item factor matrices, as opposed to
> all-pairs similarity of one matrix?
>
> On Thu, Oct 30, 2014 at 5:28 AM, Debasish Das <de...@gmail.com>
> wrote:
>
>> Is there an example of how to use RankingMetrics ?
>>
>> Let's take the user, document example...we get user x topic and document x
>> topic matrices as the model...
>>
>> Now for each user, we can generate topK document by doing a sort on (1 x
>> topic)dot(topic x document) and picking topK...
>>
>> Is it possible to validate such a topK finding algorithm using
>> RankingMetrics ?
>>
>>
>> On Wed, Oct 29, 2014 at 12:14 PM, Xiangrui Meng <me...@gmail.com> wrote:
>>
>> > Let's narrow the context from matrix factorization to recommendation
>> > via ALS. It adds extra complexity if we treat it as a multi-class
>> > classification problem. ALS only outputs a single value for each
>> > prediction, which is hard to convert to probability distribution over
>> > the 5 rating levels. Treating it as a binary classification problem or
>> > a ranking problem does make sense. The RankingMetricc is in master.
>> > Free free to add prec@k and ndcg@k to examples.MovielensALS. ROC
>> > should be good to add as well. -Xiangrui
>> >
>> >
>> > On Wed, Oct 29, 2014 at 11:23 AM, Debasish Das <
>> debasish.das83@gmail.com>
>> > wrote:
>> > > Hi,
>> > >
>> > > In the current factorization flow, we cross validate on the test
>> dataset
>> > > using the RMSE number but there are some other measures which are
>> worth
>> > > looking into.
>> > >
>> > > If we consider the problem as a regression problem and the ratings 1-5
>> > are
>> > > considered as 5 classes, it is possible to generate a confusion matrix
>> > > using MultiClassMetrics.scala
>> > >
>> > > If the ratings are only 0/1 (like from the spotify demo from spark
>> > summit)
>> > > then it is possible to use Binary Classification Metrices to come up
>> with
>> > > the ROC curve...
>> > >
>> > > For topK user/products we should also look into prec@k and pdcg@k as
>> the
>> > > metric..
>> > >
>> > > Does it make sense to add the multiclass metric and prec@k, pdcg@k in
>> > > examples.MovielensALS along with RMSE ?
>> > >
>> > > Thanks.
>> > > Deb
>> >
>>
>
>
Re: matrix factorization cross validation
Posted by Nick Pentreath <ni...@gmail.com>.
Looking at
https://github.com/apache/spark/blob/814a9cd7fabebf2a06f7e2e5d46b6a2b28b917c2/mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala#L82
For each user in test set, you generate an Array of top K predicted item
ids (Int or String probably), and an Array of ground truth item ids (the
known rated or liked items in the test set for that user), and pass that to
precisionAt(k) to compute MAP@k (Actually this method name is a bit
misleading - it should be meanAveragePrecisionAt where the other method
there is without a cutoff at k. However, both compute MAP).
The challenge at scale is actually computing all the top Ks for each user,
as it requires broadcasting all the item factors (unless there is a smarter
way?)
I wonder if it is possible to extend the DIMSUM idea to computing top K
matrix multiply between the user and item factor matrices, as opposed to
all-pairs similarity of one matrix?
On Thu, Oct 30, 2014 at 5:28 AM, Debasish Das <de...@gmail.com>
wrote:
> Is there an example of how to use RankingMetrics ?
>
> Let's take the user, document example...we get user x topic and document x
> topic matrices as the model...
>
> Now for each user, we can generate topK document by doing a sort on (1 x
> topic)dot(topic x document) and picking topK...
>
> Is it possible to validate such a topK finding algorithm using
> RankingMetrics ?
>
>
> On Wed, Oct 29, 2014 at 12:14 PM, Xiangrui Meng <me...@gmail.com> wrote:
>
> > Let's narrow the context from matrix factorization to recommendation
> > via ALS. It adds extra complexity if we treat it as a multi-class
> > classification problem. ALS only outputs a single value for each
> > prediction, which is hard to convert to probability distribution over
> > the 5 rating levels. Treating it as a binary classification problem or
> > a ranking problem does make sense. The RankingMetricc is in master.
> > Free free to add prec@k and ndcg@k to examples.MovielensALS. ROC
> > should be good to add as well. -Xiangrui
> >
> >
> > On Wed, Oct 29, 2014 at 11:23 AM, Debasish Das <debasish.das83@gmail.com
> >
> > wrote:
> > > Hi,
> > >
> > > In the current factorization flow, we cross validate on the test
> dataset
> > > using the RMSE number but there are some other measures which are worth
> > > looking into.
> > >
> > > If we consider the problem as a regression problem and the ratings 1-5
> > are
> > > considered as 5 classes, it is possible to generate a confusion matrix
> > > using MultiClassMetrics.scala
> > >
> > > If the ratings are only 0/1 (like from the spotify demo from spark
> > summit)
> > > then it is possible to use Binary Classification Metrices to come up
> with
> > > the ROC curve...
> > >
> > > For topK user/products we should also look into prec@k and pdcg@k as
> the
> > > metric..
> > >
> > > Does it make sense to add the multiclass metric and prec@k, pdcg@k in
> > > examples.MovielensALS along with RMSE ?
> > >
> > > Thanks.
> > > Deb
> >
>
Re: matrix factorization cross validation
Posted by Debasish Das <de...@gmail.com>.
Is there an example of how to use RankingMetrics ?
Let's take the user, document example...we get user x topic and document x
topic matrices as the model...
Now for each user, we can generate topK document by doing a sort on (1 x
topic)dot(topic x document) and picking topK...
Is it possible to validate such a topK finding algorithm using
RankingMetrics ?
On Wed, Oct 29, 2014 at 12:14 PM, Xiangrui Meng <me...@gmail.com> wrote:
> Let's narrow the context from matrix factorization to recommendation
> via ALS. It adds extra complexity if we treat it as a multi-class
> classification problem. ALS only outputs a single value for each
> prediction, which is hard to convert to probability distribution over
> the 5 rating levels. Treating it as a binary classification problem or
> a ranking problem does make sense. The RankingMetricc is in master.
> Free free to add prec@k and ndcg@k to examples.MovielensALS. ROC
> should be good to add as well. -Xiangrui
>
>
> On Wed, Oct 29, 2014 at 11:23 AM, Debasish Das <de...@gmail.com>
> wrote:
> > Hi,
> >
> > In the current factorization flow, we cross validate on the test dataset
> > using the RMSE number but there are some other measures which are worth
> > looking into.
> >
> > If we consider the problem as a regression problem and the ratings 1-5
> are
> > considered as 5 classes, it is possible to generate a confusion matrix
> > using MultiClassMetrics.scala
> >
> > If the ratings are only 0/1 (like from the spotify demo from spark
> summit)
> > then it is possible to use Binary Classification Metrices to come up with
> > the ROC curve...
> >
> > For topK user/products we should also look into prec@k and pdcg@k as the
> > metric..
> >
> > Does it make sense to add the multiclass metric and prec@k, pdcg@k in
> > examples.MovielensALS along with RMSE ?
> >
> > Thanks.
> > Deb
>
Re: matrix factorization cross validation
Posted by Debasish Das <de...@gmail.com>.
Makes sense for the binary and ranking problem but for example linear
regression for multi-class also optimizes on RMSE but we still measure the
prediction efficiency using some measure on confusion matrix...Is not the
same idea should hold for ALS as well ?
On Wed, Oct 29, 2014 at 12:14 PM, Xiangrui Meng <me...@gmail.com> wrote:
> Let's narrow the context from matrix factorization to recommendation
> via ALS. It adds extra complexity if we treat it as a multi-class
> classification problem. ALS only outputs a single value for each
> prediction, which is hard to convert to probability distribution over
> the 5 rating levels. Treating it as a binary classification problem or
> a ranking problem does make sense. The RankingMetricc is in master.
> Free free to add prec@k and ndcg@k to examples.MovielensALS. ROC
> should be good to add as well. -Xiangrui
>
>
> On Wed, Oct 29, 2014 at 11:23 AM, Debasish Das <de...@gmail.com>
> wrote:
> > Hi,
> >
> > In the current factorization flow, we cross validate on the test dataset
> > using the RMSE number but there are some other measures which are worth
> > looking into.
> >
> > If we consider the problem as a regression problem and the ratings 1-5
> are
> > considered as 5 classes, it is possible to generate a confusion matrix
> > using MultiClassMetrics.scala
> >
> > If the ratings are only 0/1 (like from the spotify demo from spark
> summit)
> > then it is possible to use Binary Classification Metrices to come up with
> > the ROC curve...
> >
> > For topK user/products we should also look into prec@k and pdcg@k as the
> > metric..
> >
> > Does it make sense to add the multiclass metric and prec@k, pdcg@k in
> > examples.MovielensALS along with RMSE ?
> >
> > Thanks.
> > Deb
>
Re: matrix factorization cross validation
Posted by Xiangrui Meng <me...@gmail.com>.
Let's narrow the context from matrix factorization to recommendation
via ALS. It adds extra complexity if we treat it as a multi-class
classification problem. ALS only outputs a single value for each
prediction, which is hard to convert to probability distribution over
the 5 rating levels. Treating it as a binary classification problem or
a ranking problem does make sense. The RankingMetricc is in master.
Free free to add prec@k and ndcg@k to examples.MovielensALS. ROC
should be good to add as well. -Xiangrui
On Wed, Oct 29, 2014 at 11:23 AM, Debasish Das <de...@gmail.com> wrote:
> Hi,
>
> In the current factorization flow, we cross validate on the test dataset
> using the RMSE number but there are some other measures which are worth
> looking into.
>
> If we consider the problem as a regression problem and the ratings 1-5 are
> considered as 5 classes, it is possible to generate a confusion matrix
> using MultiClassMetrics.scala
>
> If the ratings are only 0/1 (like from the spotify demo from spark summit)
> then it is possible to use Binary Classification Metrices to come up with
> the ROC curve...
>
> For topK user/products we should also look into prec@k and pdcg@k as the
> metric..
>
> Does it make sense to add the multiclass metric and prec@k, pdcg@k in
> examples.MovielensALS along with RMSE ?
>
> Thanks.
> Deb
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org