You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Ted Dunning <te...@gmail.com> on 2013/12/01 09:29:32 UTC
Re: Question about Pearson Correlation in non-Taste mode
Good point Amit.
Not sure how much this matters. It may be that
PearsonCorrelationSimilarity is bad name that should be
PearonInspiredCorrelationSimilarity. My guess is that this implementation
is lifted directly from the very early recommendation literature and is
reflective of the way that it was used back then.
Remember that the context here is prediction of ratings. If you assume
that you really want correlation and that missing elements are zero, then
this is mathematically wrong. On the other hand, if you assume missing
elements are equal to the mean (whatever it is), then this definition is
correct.
In any case, I don't think that PearsonCorrelationSimilarity should be
"fixed" at this point. First of all, a substantial change here is somewhat
risky since there may be people who depend on current behavior. Second, I
think that this is almost never a particularly good recommendation
algorithm so even if the proposed change is a small improvement, it will
have negligible positive effect on the universe of production recommenders.
Remember that this function is not a stats routine. It is an embodiment of
recommendation practice. Were it the former, I would strongly recommend we
fix it.
On Sat, Nov 30, 2013 at 10:18 AM, Amit Nithian <an...@gmail.com> wrote:
> Hi Ted,
>
> Thanks that is what I would have thought too but I don't think that the
> Pearson Similarity (in Hadoop mode) does this:
>
> in
>
> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.PearsonCorrelationSimilarity
> around line 31
>
> double average = vector.norm(1) / vector.getNumNonZeroElements();
> Which looks like it's taking the sum and dividing by the number of defined
> elements. Which would make my [5 - 4] average be 4.5.
>
> Thanks again
> Amit
>
> On Fri, Nov 29, 2013 at 10:34 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > On Fri, Nov 29, 2013 at 10:16 PM, Amit Nithian <an...@gmail.com>
> wrote:
> >
> > > Hi Ted,
> > >
> > > Thanks for your response. I thought that the mean of a sparse vector is
> > > simply the mean of the "defined" elements? Why would the vectors become
> > > dense unless you're meaning that all the undefined elements (0?) now
> will
> > > be (0-m_x)?
> > >
> >
> > Yes. Just so. All those zero elements become non-zero and the vector is
> > thus non-dense.
> >
> >
> > >
> > > Looking at the following example:
> > > X = [5 - 4] and Y= [4 5 2].
> > >
> > > is m_x 4.5 or 3?
> >
> >
> > 3.
> >
> > This is because the elements of X are really 5, 0, and 4. The zero is
> just
> > not stored, but it still is the value of that element.
> >
> >
> > > Is m_y 11/3 or (6/2) because we ignore the "5" since it's
> > > counterpart in X is undefined?.
> > >
> >
> > 11/3
> >
>
RE: Question about Pearson Correlation in non-Taste mode
Posted by Jason Xin <Ja...@sas.com>.
Thanks, Ted
Jason Xin
-----Original Message-----
From: Ted Dunning [mailto:ted.dunning@gmail.com]
Sent: Friday, December 06, 2013 9:25 PM
To: user@mahout.apache.org
Subject: Re: Question about Pearson Correlation in non-Taste mode
The second link was an article I wrote that led eventually to the dissertation (third link).
On Fri, Dec 6, 2013 at 5:15 PM, Jason Xin <Ja...@sas.com> wrote:
> Ted,
>
> Is this your doctoral "Accurate Methods for the Statistics of Surprise
> and Coincidence" , the second one PDF you attached, or you have
> another one you can forward to me, your doctoral dissertation? Thanks.
>
> Jason Xin
>
> -----Original Message-----
> From: Ted Dunning [mailto:ted.dunning@gmail.com]
> Sent: Friday, December 06, 2013 7:56 PM
> To: user@mahout.apache.org
> Subject: Re: Question about Pearson Correlation in non-Taste mode
>
> See
>
> http://tdunning.blogspot.com/2008/03/surprise-and-coincidence.html
> http://acl.ldc.upenn.edu/J/J93/J93-1003.pdf
> http://arxiv.org/abs/1207.1847
>
>
>
>
>
> On Fri, Dec 6, 2013 at 1:09 PM, Amit Nithian <an...@gmail.com> wrote:
>
> > Hey Sebastian,
> >
> > Thanks again for the explanation. So now you have me intrigued about
> > something else. Why is it that logliklihood ratio test is a better
> > measure for essentially implicit ratings? Are there
> > resources/research papers you can point me to explaining this?
> >
> > Take care
> > Amit
> >
> >
> > On Sun, Dec 1, 2013 at 9:25 AM, Sebastian Schelter
> > <ss...@googlemail.com>wrote:
> >
> > > Hi Amit,
> > >
> > > No need to excuse for picking on me, I'm happy about anyone
> > > digging into the paper :)
> > >
> > > The reason, I implemented Pearson in this (flawed) way has to do
> > > with the way the parallel algorithm works:
> > >
> > > It never compares two item vectors in memory, instead it
> > > preprocesses the vectors and computes sparse dot products in
> > > parallel. The centering which is usually done for Pearson
> > > correlation is dependent on which pair of vectors you're currently
> > > looking at (and doesn't fit the parallel algorithm). We had an
> > > earlier implementation that didn't have this flaw, but was way
> > > slower
> than the current one.
> > >
> > > Rating prediction on explicit feedback data like ratings for which
> > > Pearson correlation is mostly used in CF, is a rather academic
> > > topic and in science there are nearly no datasets that really
> > > require you to go to Hadoop.
> > >
> > > On the other hand item prediction on implicit feedback data (like
> > > clicks) is the common scenario in the majority of industry
> > > usecases, but here count-based similarity measures like the
> > > loglikelihood ratio test give much better results. The current
> > > implementation of Mahout's distributed itembased recommender is
> > > clearly designed and tuned for the latter usecase.
> > >
> > > I hope that answers your question.
> > >
> > > --sebastian
> > >
> > > On 01.12.2013 18:10, Amit Nithian wrote:
> > > > Thanks guys! So the real question is not so much what's the
> > > > average of
> > > the
> > > > vector with the missing rating (although yes that was a
> > > > question) but what's the average of the vector with all the
> > > > ratings specified but the second rating that is not shared with
> > > > the first
> user:
> > > > [5 - 4] vs [4 5 2].
> > > >
> > > > If we agree that the first is 4.5 then is the second one 11/3 or
> > > > 3 ((4+2)/2)? Taste has this as ((4+2)/2) while distributed mode
> > > > has it as 11/3.
> > > >
> > > > Since Taste (and Lenskit) is sequential, it can (and will only)
> > > > look at co-occurring ratings whereas the Hadoop implementation
> > > > doesn't. The
> > paper
> > > > that Sebastian wrote has a pre-processing step where (for
> > > > Pearson) you subtract each element of an item-rating vector from
> > > > the average rating which implies that each item-rating vector is
> > > > treated independently of
> > > each
> > > > other whereas in the sequential/non-distributed mode it's all
> > considered
> > > > together.
> > > >
> > > > My main reason for posting is because the Taste implementation
> > > > of
> > > item-item
> > > > similarity differs from the distributed implementation. Since I
> > > > am
> > > totally
> > > > new to this space and these similarities I wanted to understand
> > > > if
> > there
> > > is
> > > > a reason for this difference and whether or not it matters.
> > > > Sounds like from the discussion it doesn't matter but
> > > > understanding why helps me explain this to others.
> > > >
> > > > My guess (and I'm glad Sebastian is on this list so he can help
> > > > confirm/deny this.. sorry I'm not picking on you just happy to
> > > > be able
> > to
> > > > talk to you about your good paper) is that considering
> > > > co-occuring
> > > ratings
> > > > in a distributed implementation would require access to the full
> > > > matrix which defeats the parallel nature of computing item-item
> similarity?
> > > >
> > > > Thanks again!
> > > > Amit
> > > >
> > > >
> > > > On Sun, Dec 1, 2013 at 2:55 AM, Sean Owen <sr...@gmail.com> wrote:
> > > >
> > > >> It's not an issue of how to be careful with sparsity and
> > > >> subtracting means, although that's a valuable point in itself.
> > > >> The question is what the mean is supposed to be.
> > > >>
> > > >> You can't think of missing ratings as 0 in general, and the
> > > >> example here shows why: you're acting as if most movies are
> > > >> hated. Instead they are excluded from the computation entirely.
> > > >>
> > > >> m_x should be 4.5 in the example here. That's consistent with
> > > >> literature and the other implementations earlier in this project.
> > > >>
> > > >> I don't know the Hadoop implementation well enough, and wasn't
> > > >> sure from the comments above, whether it does end up behaving
> > > >> as if it's "4.5" or "3". If it's not 4.5 I would call that a bug.
> > > >> Items that aren't co-rated can't meaningfully be included in
> > > >> this
> computation.
> > > >>
> > > >>
> > > >> On Sun, Dec 1, 2013 at 8:29 AM, Ted Dunning
> > > >> <te...@gmail.com>
> > > wrote:
> > > >>> Good point Amit.
> > > >>>
> > > >>> Not sure how much this matters. It may be that
> > > >>> PearsonCorrelationSimilarity is bad name that should be
> > > >>> PearonInspiredCorrelationSimilarity. My guess is that this
> > > >> implementation
> > > >>> is lifted directly from the very early recommendation
> > > >>> literature and
> > is
> > > >>> reflective of the way that it was used back then.
> > > >>
> > > >
> > >
> > >
> >
>
Re: Question about Pearson Correlation in non-Taste mode
Posted by Ted Dunning <te...@gmail.com>.
The second link was an article I wrote that led eventually to the
dissertation (third link).
On Fri, Dec 6, 2013 at 5:15 PM, Jason Xin <Ja...@sas.com> wrote:
> Ted,
>
> Is this your doctoral "Accurate Methods for the Statistics of Surprise and
> Coincidence" , the second one PDF you attached, or you have another one you
> can forward to me, your doctoral dissertation? Thanks.
>
> Jason Xin
>
> -----Original Message-----
> From: Ted Dunning [mailto:ted.dunning@gmail.com]
> Sent: Friday, December 06, 2013 7:56 PM
> To: user@mahout.apache.org
> Subject: Re: Question about Pearson Correlation in non-Taste mode
>
> See
>
> http://tdunning.blogspot.com/2008/03/surprise-and-coincidence.html
> http://acl.ldc.upenn.edu/J/J93/J93-1003.pdf
> http://arxiv.org/abs/1207.1847
>
>
>
>
>
> On Fri, Dec 6, 2013 at 1:09 PM, Amit Nithian <an...@gmail.com> wrote:
>
> > Hey Sebastian,
> >
> > Thanks again for the explanation. So now you have me intrigued about
> > something else. Why is it that logliklihood ratio test is a better
> > measure for essentially implicit ratings? Are there resources/research
> > papers you can point me to explaining this?
> >
> > Take care
> > Amit
> >
> >
> > On Sun, Dec 1, 2013 at 9:25 AM, Sebastian Schelter
> > <ss...@googlemail.com>wrote:
> >
> > > Hi Amit,
> > >
> > > No need to excuse for picking on me, I'm happy about anyone digging
> > > into the paper :)
> > >
> > > The reason, I implemented Pearson in this (flawed) way has to do
> > > with the way the parallel algorithm works:
> > >
> > > It never compares two item vectors in memory, instead it
> > > preprocesses the vectors and computes sparse dot products in
> > > parallel. The centering which is usually done for Pearson
> > > correlation is dependent on which pair of vectors you're currently
> > > looking at (and doesn't fit the parallel algorithm). We had an
> > > earlier implementation that didn't have this flaw, but was way slower
> than the current one.
> > >
> > > Rating prediction on explicit feedback data like ratings for which
> > > Pearson correlation is mostly used in CF, is a rather academic topic
> > > and in science there are nearly no datasets that really require you
> > > to go to Hadoop.
> > >
> > > On the other hand item prediction on implicit feedback data (like
> > > clicks) is the common scenario in the majority of industry usecases,
> > > but here count-based similarity measures like the loglikelihood
> > > ratio test give much better results. The current implementation of
> > > Mahout's distributed itembased recommender is clearly designed and
> > > tuned for the latter usecase.
> > >
> > > I hope that answers your question.
> > >
> > > --sebastian
> > >
> > > On 01.12.2013 18:10, Amit Nithian wrote:
> > > > Thanks guys! So the real question is not so much what's the
> > > > average of
> > > the
> > > > vector with the missing rating (although yes that was a question)
> > > > but what's the average of the vector with all the ratings
> > > > specified but the second rating that is not shared with the first
> user:
> > > > [5 - 4] vs [4 5 2].
> > > >
> > > > If we agree that the first is 4.5 then is the second one 11/3 or 3
> > > > ((4+2)/2)? Taste has this as ((4+2)/2) while distributed mode has
> > > > it as 11/3.
> > > >
> > > > Since Taste (and Lenskit) is sequential, it can (and will only)
> > > > look at co-occurring ratings whereas the Hadoop implementation
> > > > doesn't. The
> > paper
> > > > that Sebastian wrote has a pre-processing step where (for Pearson)
> > > > you subtract each element of an item-rating vector from the
> > > > average rating which implies that each item-rating vector is
> > > > treated independently of
> > > each
> > > > other whereas in the sequential/non-distributed mode it's all
> > considered
> > > > together.
> > > >
> > > > My main reason for posting is because the Taste implementation of
> > > item-item
> > > > similarity differs from the distributed implementation. Since I am
> > > totally
> > > > new to this space and these similarities I wanted to understand if
> > there
> > > is
> > > > a reason for this difference and whether or not it matters. Sounds
> > > > like from the discussion it doesn't matter but understanding why
> > > > helps me explain this to others.
> > > >
> > > > My guess (and I'm glad Sebastian is on this list so he can help
> > > > confirm/deny this.. sorry I'm not picking on you just happy to be
> > > > able
> > to
> > > > talk to you about your good paper) is that considering co-occuring
> > > ratings
> > > > in a distributed implementation would require access to the full
> > > > matrix which defeats the parallel nature of computing item-item
> similarity?
> > > >
> > > > Thanks again!
> > > > Amit
> > > >
> > > >
> > > > On Sun, Dec 1, 2013 at 2:55 AM, Sean Owen <sr...@gmail.com> wrote:
> > > >
> > > >> It's not an issue of how to be careful with sparsity and
> > > >> subtracting means, although that's a valuable point in itself.
> > > >> The question is what the mean is supposed to be.
> > > >>
> > > >> You can't think of missing ratings as 0 in general, and the
> > > >> example here shows why: you're acting as if most movies are
> > > >> hated. Instead they are excluded from the computation entirely.
> > > >>
> > > >> m_x should be 4.5 in the example here. That's consistent with
> > > >> literature and the other implementations earlier in this project.
> > > >>
> > > >> I don't know the Hadoop implementation well enough, and wasn't
> > > >> sure from the comments above, whether it does end up behaving as
> > > >> if it's "4.5" or "3". If it's not 4.5 I would call that a bug.
> > > >> Items that aren't co-rated can't meaningfully be included in this
> computation.
> > > >>
> > > >>
> > > >> On Sun, Dec 1, 2013 at 8:29 AM, Ted Dunning
> > > >> <te...@gmail.com>
> > > wrote:
> > > >>> Good point Amit.
> > > >>>
> > > >>> Not sure how much this matters. It may be that
> > > >>> PearsonCorrelationSimilarity is bad name that should be
> > > >>> PearonInspiredCorrelationSimilarity. My guess is that this
> > > >> implementation
> > > >>> is lifted directly from the very early recommendation literature
> > > >>> and
> > is
> > > >>> reflective of the way that it was used back then.
> > > >>
> > > >
> > >
> > >
> >
>
RE: Question about Pearson Correlation in non-Taste mode
Posted by Jason Xin <Ja...@sas.com>.
Ted,
Is this your doctoral "Accurate Methods for the Statistics of Surprise and Coincidence" , the second one PDF you attached, or you have another one you can forward to me, your doctoral dissertation? Thanks.
Jason Xin
-----Original Message-----
From: Ted Dunning [mailto:ted.dunning@gmail.com]
Sent: Friday, December 06, 2013 7:56 PM
To: user@mahout.apache.org
Subject: Re: Question about Pearson Correlation in non-Taste mode
See
http://tdunning.blogspot.com/2008/03/surprise-and-coincidence.html
http://acl.ldc.upenn.edu/J/J93/J93-1003.pdf
http://arxiv.org/abs/1207.1847
On Fri, Dec 6, 2013 at 1:09 PM, Amit Nithian <an...@gmail.com> wrote:
> Hey Sebastian,
>
> Thanks again for the explanation. So now you have me intrigued about
> something else. Why is it that logliklihood ratio test is a better
> measure for essentially implicit ratings? Are there resources/research
> papers you can point me to explaining this?
>
> Take care
> Amit
>
>
> On Sun, Dec 1, 2013 at 9:25 AM, Sebastian Schelter
> <ss...@googlemail.com>wrote:
>
> > Hi Amit,
> >
> > No need to excuse for picking on me, I'm happy about anyone digging
> > into the paper :)
> >
> > The reason, I implemented Pearson in this (flawed) way has to do
> > with the way the parallel algorithm works:
> >
> > It never compares two item vectors in memory, instead it
> > preprocesses the vectors and computes sparse dot products in
> > parallel. The centering which is usually done for Pearson
> > correlation is dependent on which pair of vectors you're currently
> > looking at (and doesn't fit the parallel algorithm). We had an
> > earlier implementation that didn't have this flaw, but was way slower than the current one.
> >
> > Rating prediction on explicit feedback data like ratings for which
> > Pearson correlation is mostly used in CF, is a rather academic topic
> > and in science there are nearly no datasets that really require you
> > to go to Hadoop.
> >
> > On the other hand item prediction on implicit feedback data (like
> > clicks) is the common scenario in the majority of industry usecases,
> > but here count-based similarity measures like the loglikelihood
> > ratio test give much better results. The current implementation of
> > Mahout's distributed itembased recommender is clearly designed and
> > tuned for the latter usecase.
> >
> > I hope that answers your question.
> >
> > --sebastian
> >
> > On 01.12.2013 18:10, Amit Nithian wrote:
> > > Thanks guys! So the real question is not so much what's the
> > > average of
> > the
> > > vector with the missing rating (although yes that was a question)
> > > but what's the average of the vector with all the ratings
> > > specified but the second rating that is not shared with the first user:
> > > [5 - 4] vs [4 5 2].
> > >
> > > If we agree that the first is 4.5 then is the second one 11/3 or 3
> > > ((4+2)/2)? Taste has this as ((4+2)/2) while distributed mode has
> > > it as 11/3.
> > >
> > > Since Taste (and Lenskit) is sequential, it can (and will only)
> > > look at co-occurring ratings whereas the Hadoop implementation
> > > doesn't. The
> paper
> > > that Sebastian wrote has a pre-processing step where (for Pearson)
> > > you subtract each element of an item-rating vector from the
> > > average rating which implies that each item-rating vector is
> > > treated independently of
> > each
> > > other whereas in the sequential/non-distributed mode it's all
> considered
> > > together.
> > >
> > > My main reason for posting is because the Taste implementation of
> > item-item
> > > similarity differs from the distributed implementation. Since I am
> > totally
> > > new to this space and these similarities I wanted to understand if
> there
> > is
> > > a reason for this difference and whether or not it matters. Sounds
> > > like from the discussion it doesn't matter but understanding why
> > > helps me explain this to others.
> > >
> > > My guess (and I'm glad Sebastian is on this list so he can help
> > > confirm/deny this.. sorry I'm not picking on you just happy to be
> > > able
> to
> > > talk to you about your good paper) is that considering co-occuring
> > ratings
> > > in a distributed implementation would require access to the full
> > > matrix which defeats the parallel nature of computing item-item similarity?
> > >
> > > Thanks again!
> > > Amit
> > >
> > >
> > > On Sun, Dec 1, 2013 at 2:55 AM, Sean Owen <sr...@gmail.com> wrote:
> > >
> > >> It's not an issue of how to be careful with sparsity and
> > >> subtracting means, although that's a valuable point in itself.
> > >> The question is what the mean is supposed to be.
> > >>
> > >> You can't think of missing ratings as 0 in general, and the
> > >> example here shows why: you're acting as if most movies are
> > >> hated. Instead they are excluded from the computation entirely.
> > >>
> > >> m_x should be 4.5 in the example here. That's consistent with
> > >> literature and the other implementations earlier in this project.
> > >>
> > >> I don't know the Hadoop implementation well enough, and wasn't
> > >> sure from the comments above, whether it does end up behaving as
> > >> if it's "4.5" or "3". If it's not 4.5 I would call that a bug.
> > >> Items that aren't co-rated can't meaningfully be included in this computation.
> > >>
> > >>
> > >> On Sun, Dec 1, 2013 at 8:29 AM, Ted Dunning
> > >> <te...@gmail.com>
> > wrote:
> > >>> Good point Amit.
> > >>>
> > >>> Not sure how much this matters. It may be that
> > >>> PearsonCorrelationSimilarity is bad name that should be
> > >>> PearonInspiredCorrelationSimilarity. My guess is that this
> > >> implementation
> > >>> is lifted directly from the very early recommendation literature
> > >>> and
> is
> > >>> reflective of the way that it was used back then.
> > >>
> > >
> >
> >
>
Re: Question about Pearson Correlation in non-Taste mode
Posted by Ted Dunning <te...@gmail.com>.
See
http://tdunning.blogspot.com/2008/03/surprise-and-coincidence.html
http://acl.ldc.upenn.edu/J/J93/J93-1003.pdf
http://arxiv.org/abs/1207.1847
On Fri, Dec 6, 2013 at 1:09 PM, Amit Nithian <an...@gmail.com> wrote:
> Hey Sebastian,
>
> Thanks again for the explanation. So now you have me intrigued about
> something else. Why is it that logliklihood ratio test is a better measure
> for essentially implicit ratings? Are there resources/research papers you
> can point me to explaining this?
>
> Take care
> Amit
>
>
> On Sun, Dec 1, 2013 at 9:25 AM, Sebastian Schelter
> <ss...@googlemail.com>wrote:
>
> > Hi Amit,
> >
> > No need to excuse for picking on me, I'm happy about anyone digging into
> > the paper :)
> >
> > The reason, I implemented Pearson in this (flawed) way has to do with
> > the way the parallel algorithm works:
> >
> > It never compares two item vectors in memory, instead it preprocesses
> > the vectors and computes sparse dot products in parallel. The centering
> > which is usually done for Pearson correlation is dependent on which pair
> > of vectors you're currently looking at (and doesn't fit the parallel
> > algorithm). We had an earlier implementation that didn't have this flaw,
> > but was way slower than the current one.
> >
> > Rating prediction on explicit feedback data like ratings for which
> > Pearson correlation is mostly used in CF, is a rather academic topic and
> > in science there are nearly no datasets that really require you to go to
> > Hadoop.
> >
> > On the other hand item prediction on implicit feedback data (like
> > clicks) is the common scenario in the majority of industry usecases, but
> > here count-based similarity measures like the loglikelihood ratio test
> > give much better results. The current implementation of Mahout's
> > distributed itembased recommender is clearly designed and tuned for the
> > latter usecase.
> >
> > I hope that answers your question.
> >
> > --sebastian
> >
> > On 01.12.2013 18:10, Amit Nithian wrote:
> > > Thanks guys! So the real question is not so much what's the average of
> > the
> > > vector with the missing rating (although yes that was a question) but
> > > what's the average of the vector with all the ratings specified but the
> > > second rating that is not shared with the first user:
> > > [5 - 4] vs [4 5 2].
> > >
> > > If we agree that the first is 4.5 then is the second one 11/3 or 3
> > > ((4+2)/2)? Taste has this as ((4+2)/2) while distributed mode has it as
> > > 11/3.
> > >
> > > Since Taste (and Lenskit) is sequential, it can (and will only) look at
> > > co-occurring ratings whereas the Hadoop implementation doesn't. The
> paper
> > > that Sebastian wrote has a pre-processing step where (for Pearson) you
> > > subtract each element of an item-rating vector from the average rating
> > > which implies that each item-rating vector is treated independently of
> > each
> > > other whereas in the sequential/non-distributed mode it's all
> considered
> > > together.
> > >
> > > My main reason for posting is because the Taste implementation of
> > item-item
> > > similarity differs from the distributed implementation. Since I am
> > totally
> > > new to this space and these similarities I wanted to understand if
> there
> > is
> > > a reason for this difference and whether or not it matters. Sounds like
> > > from the discussion it doesn't matter but understanding why helps me
> > > explain this to others.
> > >
> > > My guess (and I'm glad Sebastian is on this list so he can help
> > > confirm/deny this.. sorry I'm not picking on you just happy to be able
> to
> > > talk to you about your good paper) is that considering co-occuring
> > ratings
> > > in a distributed implementation would require access to the full matrix
> > > which defeats the parallel nature of computing item-item similarity?
> > >
> > > Thanks again!
> > > Amit
> > >
> > >
> > > On Sun, Dec 1, 2013 at 2:55 AM, Sean Owen <sr...@gmail.com> wrote:
> > >
> > >> It's not an issue of how to be careful with sparsity and subtracting
> > >> means, although that's a valuable point in itself. The question is
> > >> what the mean is supposed to be.
> > >>
> > >> You can't think of missing ratings as 0 in general, and the example
> > >> here shows why: you're acting as if most movies are hated. Instead
> > >> they are excluded from the computation entirely.
> > >>
> > >> m_x should be 4.5 in the example here. That's consistent with
> > >> literature and the other implementations earlier in this project.
> > >>
> > >> I don't know the Hadoop implementation well enough, and wasn't sure
> > >> from the comments above, whether it does end up behaving as if it's
> > >> "4.5" or "3". If it's not 4.5 I would call that a bug. Items that
> > >> aren't co-rated can't meaningfully be included in this computation.
> > >>
> > >>
> > >> On Sun, Dec 1, 2013 at 8:29 AM, Ted Dunning <te...@gmail.com>
> > wrote:
> > >>> Good point Amit.
> > >>>
> > >>> Not sure how much this matters. It may be that
> > >>> PearsonCorrelationSimilarity is bad name that should be
> > >>> PearonInspiredCorrelationSimilarity. My guess is that this
> > >> implementation
> > >>> is lifted directly from the very early recommendation literature and
> is
> > >>> reflective of the way that it was used back then.
> > >>
> > >
> >
> >
>
Re: Question about Pearson Correlation in non-Taste mode
Posted by Amit Nithian <an...@gmail.com>.
Hey Sebastian,
Thanks again for the explanation. So now you have me intrigued about
something else. Why is it that logliklihood ratio test is a better measure
for essentially implicit ratings? Are there resources/research papers you
can point me to explaining this?
Take care
Amit
On Sun, Dec 1, 2013 at 9:25 AM, Sebastian Schelter
<ss...@googlemail.com>wrote:
> Hi Amit,
>
> No need to excuse for picking on me, I'm happy about anyone digging into
> the paper :)
>
> The reason, I implemented Pearson in this (flawed) way has to do with
> the way the parallel algorithm works:
>
> It never compares two item vectors in memory, instead it preprocesses
> the vectors and computes sparse dot products in parallel. The centering
> which is usually done for Pearson correlation is dependent on which pair
> of vectors you're currently looking at (and doesn't fit the parallel
> algorithm). We had an earlier implementation that didn't have this flaw,
> but was way slower than the current one.
>
> Rating prediction on explicit feedback data like ratings for which
> Pearson correlation is mostly used in CF, is a rather academic topic and
> in science there are nearly no datasets that really require you to go to
> Hadoop.
>
> On the other hand item prediction on implicit feedback data (like
> clicks) is the common scenario in the majority of industry usecases, but
> here count-based similarity measures like the loglikelihood ratio test
> give much better results. The current implementation of Mahout's
> distributed itembased recommender is clearly designed and tuned for the
> latter usecase.
>
> I hope that answers your question.
>
> --sebastian
>
> On 01.12.2013 18:10, Amit Nithian wrote:
> > Thanks guys! So the real question is not so much what's the average of
> the
> > vector with the missing rating (although yes that was a question) but
> > what's the average of the vector with all the ratings specified but the
> > second rating that is not shared with the first user:
> > [5 - 4] vs [4 5 2].
> >
> > If we agree that the first is 4.5 then is the second one 11/3 or 3
> > ((4+2)/2)? Taste has this as ((4+2)/2) while distributed mode has it as
> > 11/3.
> >
> > Since Taste (and Lenskit) is sequential, it can (and will only) look at
> > co-occurring ratings whereas the Hadoop implementation doesn't. The paper
> > that Sebastian wrote has a pre-processing step where (for Pearson) you
> > subtract each element of an item-rating vector from the average rating
> > which implies that each item-rating vector is treated independently of
> each
> > other whereas in the sequential/non-distributed mode it's all considered
> > together.
> >
> > My main reason for posting is because the Taste implementation of
> item-item
> > similarity differs from the distributed implementation. Since I am
> totally
> > new to this space and these similarities I wanted to understand if there
> is
> > a reason for this difference and whether or not it matters. Sounds like
> > from the discussion it doesn't matter but understanding why helps me
> > explain this to others.
> >
> > My guess (and I'm glad Sebastian is on this list so he can help
> > confirm/deny this.. sorry I'm not picking on you just happy to be able to
> > talk to you about your good paper) is that considering co-occuring
> ratings
> > in a distributed implementation would require access to the full matrix
> > which defeats the parallel nature of computing item-item similarity?
> >
> > Thanks again!
> > Amit
> >
> >
> > On Sun, Dec 1, 2013 at 2:55 AM, Sean Owen <sr...@gmail.com> wrote:
> >
> >> It's not an issue of how to be careful with sparsity and subtracting
> >> means, although that's a valuable point in itself. The question is
> >> what the mean is supposed to be.
> >>
> >> You can't think of missing ratings as 0 in general, and the example
> >> here shows why: you're acting as if most movies are hated. Instead
> >> they are excluded from the computation entirely.
> >>
> >> m_x should be 4.5 in the example here. That's consistent with
> >> literature and the other implementations earlier in this project.
> >>
> >> I don't know the Hadoop implementation well enough, and wasn't sure
> >> from the comments above, whether it does end up behaving as if it's
> >> "4.5" or "3". If it's not 4.5 I would call that a bug. Items that
> >> aren't co-rated can't meaningfully be included in this computation.
> >>
> >>
> >> On Sun, Dec 1, 2013 at 8:29 AM, Ted Dunning <te...@gmail.com>
> wrote:
> >>> Good point Amit.
> >>>
> >>> Not sure how much this matters. It may be that
> >>> PearsonCorrelationSimilarity is bad name that should be
> >>> PearonInspiredCorrelationSimilarity. My guess is that this
> >> implementation
> >>> is lifted directly from the very early recommendation literature and is
> >>> reflective of the way that it was used back then.
> >>
> >
>
>
Re: Question about Pearson Correlation in non-Taste mode
Posted by Sebastian Schelter <ss...@googlemail.com>.
Hi Amit,
No need to excuse for picking on me, I'm happy about anyone digging into
the paper :)
The reason, I implemented Pearson in this (flawed) way has to do with
the way the parallel algorithm works:
It never compares two item vectors in memory, instead it preprocesses
the vectors and computes sparse dot products in parallel. The centering
which is usually done for Pearson correlation is dependent on which pair
of vectors you're currently looking at (and doesn't fit the parallel
algorithm). We had an earlier implementation that didn't have this flaw,
but was way slower than the current one.
Rating prediction on explicit feedback data like ratings for which
Pearson correlation is mostly used in CF, is a rather academic topic and
in science there are nearly no datasets that really require you to go to
Hadoop.
On the other hand item prediction on implicit feedback data (like
clicks) is the common scenario in the majority of industry usecases, but
here count-based similarity measures like the loglikelihood ratio test
give much better results. The current implementation of Mahout's
distributed itembased recommender is clearly designed and tuned for the
latter usecase.
I hope that answers your question.
--sebastian
On 01.12.2013 18:10, Amit Nithian wrote:
> Thanks guys! So the real question is not so much what's the average of the
> vector with the missing rating (although yes that was a question) but
> what's the average of the vector with all the ratings specified but the
> second rating that is not shared with the first user:
> [5 - 4] vs [4 5 2].
>
> If we agree that the first is 4.5 then is the second one 11/3 or 3
> ((4+2)/2)? Taste has this as ((4+2)/2) while distributed mode has it as
> 11/3.
>
> Since Taste (and Lenskit) is sequential, it can (and will only) look at
> co-occurring ratings whereas the Hadoop implementation doesn't. The paper
> that Sebastian wrote has a pre-processing step where (for Pearson) you
> subtract each element of an item-rating vector from the average rating
> which implies that each item-rating vector is treated independently of each
> other whereas in the sequential/non-distributed mode it's all considered
> together.
>
> My main reason for posting is because the Taste implementation of item-item
> similarity differs from the distributed implementation. Since I am totally
> new to this space and these similarities I wanted to understand if there is
> a reason for this difference and whether or not it matters. Sounds like
> from the discussion it doesn't matter but understanding why helps me
> explain this to others.
>
> My guess (and I'm glad Sebastian is on this list so he can help
> confirm/deny this.. sorry I'm not picking on you just happy to be able to
> talk to you about your good paper) is that considering co-occuring ratings
> in a distributed implementation would require access to the full matrix
> which defeats the parallel nature of computing item-item similarity?
>
> Thanks again!
> Amit
>
>
> On Sun, Dec 1, 2013 at 2:55 AM, Sean Owen <sr...@gmail.com> wrote:
>
>> It's not an issue of how to be careful with sparsity and subtracting
>> means, although that's a valuable point in itself. The question is
>> what the mean is supposed to be.
>>
>> You can't think of missing ratings as 0 in general, and the example
>> here shows why: you're acting as if most movies are hated. Instead
>> they are excluded from the computation entirely.
>>
>> m_x should be 4.5 in the example here. That's consistent with
>> literature and the other implementations earlier in this project.
>>
>> I don't know the Hadoop implementation well enough, and wasn't sure
>> from the comments above, whether it does end up behaving as if it's
>> "4.5" or "3". If it's not 4.5 I would call that a bug. Items that
>> aren't co-rated can't meaningfully be included in this computation.
>>
>>
>> On Sun, Dec 1, 2013 at 8:29 AM, Ted Dunning <te...@gmail.com> wrote:
>>> Good point Amit.
>>>
>>> Not sure how much this matters. It may be that
>>> PearsonCorrelationSimilarity is bad name that should be
>>> PearonInspiredCorrelationSimilarity. My guess is that this
>> implementation
>>> is lifted directly from the very early recommendation literature and is
>>> reflective of the way that it was used back then.
>>
>
Re: Question about Pearson Correlation in non-Taste mode
Posted by Amit Nithian <an...@gmail.com>.
Thanks guys! So the real question is not so much what's the average of the
vector with the missing rating (although yes that was a question) but
what's the average of the vector with all the ratings specified but the
second rating that is not shared with the first user:
[5 - 4] vs [4 5 2].
If we agree that the first is 4.5 then is the second one 11/3 or 3
((4+2)/2)? Taste has this as ((4+2)/2) while distributed mode has it as
11/3.
Since Taste (and Lenskit) is sequential, it can (and will only) look at
co-occurring ratings whereas the Hadoop implementation doesn't. The paper
that Sebastian wrote has a pre-processing step where (for Pearson) you
subtract each element of an item-rating vector from the average rating
which implies that each item-rating vector is treated independently of each
other whereas in the sequential/non-distributed mode it's all considered
together.
My main reason for posting is because the Taste implementation of item-item
similarity differs from the distributed implementation. Since I am totally
new to this space and these similarities I wanted to understand if there is
a reason for this difference and whether or not it matters. Sounds like
from the discussion it doesn't matter but understanding why helps me
explain this to others.
My guess (and I'm glad Sebastian is on this list so he can help
confirm/deny this.. sorry I'm not picking on you just happy to be able to
talk to you about your good paper) is that considering co-occuring ratings
in a distributed implementation would require access to the full matrix
which defeats the parallel nature of computing item-item similarity?
Thanks again!
Amit
On Sun, Dec 1, 2013 at 2:55 AM, Sean Owen <sr...@gmail.com> wrote:
> It's not an issue of how to be careful with sparsity and subtracting
> means, although that's a valuable point in itself. The question is
> what the mean is supposed to be.
>
> You can't think of missing ratings as 0 in general, and the example
> here shows why: you're acting as if most movies are hated. Instead
> they are excluded from the computation entirely.
>
> m_x should be 4.5 in the example here. That's consistent with
> literature and the other implementations earlier in this project.
>
> I don't know the Hadoop implementation well enough, and wasn't sure
> from the comments above, whether it does end up behaving as if it's
> "4.5" or "3". If it's not 4.5 I would call that a bug. Items that
> aren't co-rated can't meaningfully be included in this computation.
>
>
> On Sun, Dec 1, 2013 at 8:29 AM, Ted Dunning <te...@gmail.com> wrote:
> > Good point Amit.
> >
> > Not sure how much this matters. It may be that
> > PearsonCorrelationSimilarity is bad name that should be
> > PearonInspiredCorrelationSimilarity. My guess is that this
> implementation
> > is lifted directly from the very early recommendation literature and is
> > reflective of the way that it was used back then.
>
Re: Question about Pearson Correlation in non-Taste mode
Posted by Sean Owen <sr...@gmail.com>.
It's not an issue of how to be careful with sparsity and subtracting
means, although that's a valuable point in itself. The question is
what the mean is supposed to be.
You can't think of missing ratings as 0 in general, and the example
here shows why: you're acting as if most movies are hated. Instead
they are excluded from the computation entirely.
m_x should be 4.5 in the example here. That's consistent with
literature and the other implementations earlier in this project.
I don't know the Hadoop implementation well enough, and wasn't sure
from the comments above, whether it does end up behaving as if it's
"4.5" or "3". If it's not 4.5 I would call that a bug. Items that
aren't co-rated can't meaningfully be included in this computation.
On Sun, Dec 1, 2013 at 8:29 AM, Ted Dunning <te...@gmail.com> wrote:
> Good point Amit.
>
> Not sure how much this matters. It may be that
> PearsonCorrelationSimilarity is bad name that should be
> PearonInspiredCorrelationSimilarity. My guess is that this implementation
> is lifted directly from the very early recommendation literature and is
> reflective of the way that it was used back then.