NMF for Taste

[Otis Gospodnetic <ot...@yahoo.com>]

Hello,

Recently, I read "Matrix Factorization Techniques for Recommender Systems" from http://research.yahoo.com/node/2859 .  I was wondering what you think about this vs. what we have in Taste now?

It looks like Collective Intelligence talks about this on p232-239 + 302... but I haven't read that yet.

Thanks,
Otis

Re: NMF for Taste

[Jake Mannix <ja...@gmail.com>]

On Sat, Nov 28, 2009 at 10:58 AM, Ted Dunning <te...@gmail.com> wrote:

> Jake,
>
> Do you have any concrete information about how much difference there
> actually is in these decompositions?
>

I do not.  I'm not sure if this is supposed to be surprising or not, but
I've for the most
part done pretty well not even worrying about the missing data part, and I'm
not sure
why that is, because I buy most of the theoretical arguments for why you
need to take
care in treating the missing data separately.  Maybe it's because I've
always done some
pretty hand-tuned early stopping, and where you get killed with the zeroes
is in the case
of overfitting?

  -jake


>
> On Sat, Nov 28, 2009 at 8:31 AM, Jake Mannix <ja...@gmail.com>
> wrote:
>
> > or more precisely, a sparse SVD which doesn't treat
> > missing data as the numerical 0 or mean of the values
> >
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>

Re: NMF for Taste

[Ted Dunning <te...@gmail.com>]

Jake,

Do you have any concrete information about how much difference there
actually is in these decompositions?

On Sat, Nov 28, 2009 at 8:31 AM, Jake Mannix <ja...@gmail.com> wrote:

> or more precisely, a sparse SVD which doesn't treat
> missing data as the numerical 0 or mean of the values
>



-- 
Ted Dunning, CTO
DeepDyve

Re: NMF for Taste

[Jake Mannix <ja...@gmail.com>]

On Sat, Nov 28, 2009 at 10:57 AM, Ted Dunning <te...@gmail.com> wrote:

> Restricted Boltzmann are of real interest, but again, I repeat the
> obligatory warning about replicating all things from the Netflix
> competition.
>

Totally agree in principle, but I'm not saying everyone should use the best
techniques from the Netflix challenge, at it is, as you say, just minimizing
RMS over a particular recommendation type (one where popularity matters
a lot, blockbusters are a big effect, etc...), but I have a feeling that as
Mahout
gets more popular, people are going to be wanting to try out Netflix
techniques (whether they work on their data sets or not!), and will be
asking
whether we have them.


> To take a few concrete examples,
>
> - user biases were a huge advance in terms of RMS error, but they don't
> affect the ordering of the results presented to a user and thus are of no
> interest for almost all production recommender applications
>

Do you really think that correcting for user-bias doesn't affect ordering of
results?  The simplest possible thing (just making sure the final predicted
ratings for a given user take into account the user's bias) certainly
doesn't
affect the ordering, but even this affects your judgement of whether the
user will "like" the movie, right?

More importantly, there were other techniques regarding user-bias which
I seem to recall which are good for not just RMS, but ordering effects:
the fact that there is a nonlinear response in ratings - if a user has an
average rating of 3 (or we've adjusted for their bias), then it usually
turns out that a rating of 1 implies they hate said item *far* more than
the amount to which they rate a 5 means they love it (ie. user bias is
not just a centering effect).

To put it another way - finding similar users, at least in the Netflix case,

can be better done by finding groups of people who rate movies
*negatively* than by finding people who rate them positively.  This
happens to be very domain specific (this particular result), but I've found
the phenomenon is pretty general when it comes to user ratings, and it's
hard to capture using Pearson-correlation or even SVD.  I don't know if
it come out of RBM-based autocoder tecnhiques, or if you have to roll
it in by hand in a custom gradient descent decomposer, but either way,
it's something to keep in mind.

On the other side,
>
> - portfolio approaches that increase the diversity of results presented to
> the user increase the probability of user clicking, but decrease RMS score
>
> - dithering of results to give a changing set of recommendations increases
> users click rates, but decreases RMS score
>

Sure, but these two approaches can be done "after the fact" on results
which come from a system designed to optimize for RMS minimization.
These kinds of things tend to be very domain specific, don't you think?
They
depend on things you know about your audience, on business factors (how
much does your company care about increasing click-through versus
increasing some other conversion *after* clickthrough versus just please
them by showing them more different results [ie how much do you care about
*decreasing* bounce rate from your main page which has the recommendations
on it]).


>
> The take-away is that the Netflix results can't be used as a blueprint for
> all recommendation needs.
>

This I totally agree on - not basing all recommenders on those which work
well for Netflix should go without saying.  But RMS is a pretty nice generic

evaluation criterion (as well as pretty ubiquitous one in academic circles,
so
we'll have lots to compare against there), so it techniques which are good
at helping this are important to look at.

  -jake

Re: NMF for Taste

[Ted Dunning <te...@gmail.com>]

Restricted Boltzmann are of real interest, but again, I repeat the
obligatory warning about replicating all things from the Netflix
competition.

To take a few concrete examples,

- user biases were a huge advance in terms of RMS error, but they don't
affect the ordering of the results presented to a user and thus are of no
interest for almost all production recommender applications

- temporal dynamics as a time variation in user bias is just like the first
case.

- temporal dynamics in the sense of block-busters that decay quickly are
often of little interest in a production recommender because blockbusters
are typically presented outside of the context of recommendations which are
instead used to help users find back catalog items of interest *in*spite* of
current popularity trends.  This means that tracking this kind of temporal
dynamics is good in the Netflix challenge, but neutral or bad in most
recommendation applications.  An exception is magical navigation links that
populate themselves using recommendation technology.

On the other side,

- portfolio approaches that increase the diversity of results presented to
the user increase the probability of user clicking, but decrease RMS score

- dithering of results to give a changing set of recommendations increases
users click rates, but decreases RMS score

The take-away is that the Netflix results can't be used as a blueprint for
all recommendation needs.

On Sat, Nov 28, 2009 at 8:31 AM, Jake Mannix <ja...@gmail.com> wrote:

> Machine based recommender, because this makes the final leap from linear
> and quasi-linear decompositions to the truly nonlinear case (my friend on
> the executive team over at Netflix tells me that it was pretty apparent
> that the
> winners were going to be blendings of the RBM and SVD-based approaches
> pretty early on - and he was right!)
>



-- 
Ted Dunning, CTO
DeepDyve

Re: NMF for Taste

[Jake Mannix <ja...@gmail.com>]

On Fri, Nov 27, 2009 at 11:23 PM, Ted Dunning <te...@gmail.com> wrote:

> Summarize yes.
>


> But this is, actually, theoretically better because the summarization
> introduces useful smoothing.  That way you get recommendations for items
> even if there is no direct overlap.
>

Summarize, smooth, and enhance clustering: distances are *not* preserved in
truncated decompositions, and the *hope* is that the "meaningful" distances
are
decreased, and the less meaningful distances are not.  This can be seen in
a simple example of user preferences (on the netflix scale of 1-5)


user1: item1 = 4, item2: 1, item3: 5
user2: item2: 1, item5:1, item7: 3, item8: 3
user3: item4: 4, item5: 1, item6: 5

A first-order recommender won't be able to infer any similarity or
dissimilarity between
user1 and user 3 (although it can tell some similarity between 1 and 2 and 3
and 2).
A decomposing recommender will notice that user1 and 2 both hated item2, and
that
another item which user2 hated was the same item that user3 hated, and infer

transitive similarity, not just to 2nd degree as in this example, but to
nth-order.

The difference between the various decompositional approaches is how they
approximate these transitive similarities - LDA would be best in the very
low
overlap case, and SVD (or more precisely, a sparse SVD which doesn't treat
missing data as the numerical 0 or mean of the values) approaching that
level
of quality in the bigger data case (but SVD / randomized SVD should be a lot
faster than LDA on the big big data case).

What I'd really like to see (once I get this decomposer stuff in - soon!
We've
got good linear primitives now, so I'm working on it!) is also a Restricted
Boltzmann
Machine based recommender, because this makes the final leap from linear
and quasi-linear decompositions to the truly nonlinear case (my friend on
the
executive team over at Netflix tells me that it was pretty apparent that the

winners were going to be blendings of the RBM and SVD-based approaches
pretty early on - and he was right!)


  -jake



> Your point about noisy is trenchant because small count data is inherently
> noisy because you can't have an exact 0.04 of an observation.  Small counts
> dominate in recommendations.
>
> On Fri, Nov 27, 2009 at 10:00 PM, Sean Owen <sr...@gmail.com> wrote:
>
> >
> > Correct me if I'm wrong, but my impression of matrix factorization
> > approaches is that they're just a way to effectively "summarize" input
> > data. They're not a theoretically better, or even different, approach
> > to recommendation, but more a transformation of the input into
> > conventional algorithms. (Though this process of simplification could,
> > I imagine, sometimes be an improvement on the input, if it's noisy.)
>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>

Re: NMF for Taste

[Sean Owen <sr...@gmail.com>]

That's a good word -- smoothing. Yes, I can see why this is an actual
advantage in practice. You end up with less sparsity.

On Sat, Nov 28, 2009 at 7:23 AM, Ted Dunning <te...@gmail.com> wrote:
> Summarize yes.
>
> But this is, actually, theoretically better because the summarization
> introduces useful smoothing.  That way you get recommendations for items
> even if there is no direct overlap.
>
> Your point about noisy is trenchant because small count data is inherently
> noisy because you can't have an exact 0.04 of an observation.  Small counts
> dominate in recommendations.

Re: NMF for Taste

[Ted Dunning <te...@gmail.com>]

Summarize yes.

But this is, actually, theoretically better because the summarization
introduces useful smoothing.  That way you get recommendations for items
even if there is no direct overlap.

Your point about noisy is trenchant because small count data is inherently
noisy because you can't have an exact 0.04 of an observation.  Small counts
dominate in recommendations.

On Fri, Nov 27, 2009 at 10:00 PM, Sean Owen <sr...@gmail.com> wrote:

>
> Correct me if I'm wrong, but my impression of matrix factorization
> approaches is that they're just a way to effectively "summarize" input
> data. They're not a theoretically better, or even different, approach
> to recommendation, but more a transformation of the input into
> conventional algorithms. (Though this process of simplification could,
> I imagine, sometimes be an improvement on the input, if it's noisy.)




-- 
Ted Dunning, CTO
DeepDyve

Re: NMF for Taste

[Ted Dunning <te...@gmail.com>]

Hmm.... I find that factorization techniques serve as an effective smoothing
technique that helps most when data is a bit sparse.  When you have lots of
data, you need these techniques a bit less.

But we should have good factorization soon (right Jake!?)

On Fri, Nov 27, 2009 at 10:00 PM, Sean Owen <sr...@gmail.com> wrote:

> The only issue with it, from my perspective, is that it's all
> in-memory, and these approaches are appropriate exactly when you have
> a gargantuan input.
>



-- 
Ted Dunning, CTO
DeepDyve

Re: NMF for Taste

[Sean Owen <sr...@gmail.com>]

Yes there is already an SVD-based recommender in the code base. The
name of the contributor escapes me at the moment, I can't take credit
for it. It works fine.

The only issue with it, from my perspective, is that it's all
in-memory, and these approaches are appropriate exactly when you have
a gargantuan input.

Correct me if I'm wrong, but my impression of matrix factorization
approaches is that they're just a way to effectively "summarize" input
data. They're not a theoretically better, or even different, approach
to recommendation, but more a transformation of the input into
conventional algorithms. (Though this process of simplification could,
I imagine, sometimes be an improvement on the input, if it's noisy.)

That is not to say they aren't valuable -- on the contrary. But
they're a necessary evil, like Hadoop, when tackling scale. One simply
can't digest a billion ratings without needing to apply tools like
these.

Therefore yes in the short term there needs to be some properly
distributed recommendation support, and in the long term that needs to
include stuff like a dstributed SVD implementation.

On Fri, Nov 27, 2009 at 10:56 PM, Ted Dunning <te...@gmail.com> wrote:
> NMF, singular value decomposition, random indexing and LDA are all very
> interesting and useful methods for recommenders.  If you have lots of data,
> then sparsification becomes more important than smoothing and so other
> considerations come to the fore.
>
> We have LDA available, but not integrated.  We also have sparsification
> using log likelihood ratio tests integrated into Taste.  Jake has been going
> gang-busters on decomposition techniques, but mostly SVD so far.  His work
> will probably result in random indexing being supported as well.  I don't
> know if SVD and random indexing will happen right away in Taste, but it
> shouldn't be too long a walk.

Re: NMF for Taste

[Ted Dunning <te...@gmail.com>]

NMF, singular value decomposition, random indexing and LDA are all very
interesting and useful methods for recommenders.  If you have lots of data,
then sparsification becomes more important than smoothing and so other
considerations come to the fore.

We have LDA available, but not integrated.  We also have sparsification
using log likelihood ratio tests integrated into Taste.  Jake has been going
gang-busters on decomposition techniques, but mostly SVD so far.  His work
will probably result in random indexing being supported as well.  I don't
know if SVD and random indexing will happen right away in Taste, but it
shouldn't be too long a walk.

NMF has not had much support so far although it is potentially intriguing.
Depending on your constraints and optimization goal, NMF can be equivalent
to pLSI (in which case LDA should be better) or k-means (in which case we
already have it).

Regarding Koren et al's article, you have to take what they say with just a
bit of a grain of salt.  Factorization techniques are definitely very good
if what you want is smallest RMS error on a moderate sized data set that you
can tune nearly forever.  If you want the best click rate or visit length in
a system that has lots of content churn and where you need to account for
the virtuous or vicious cycle that the recommender has on what people watch
then other answers may be better.  This is particularly true when you are
severely constrained on developer time/skill/attention span.

On Fri, Nov 27, 2009 at 11:03 AM, Otis Gospodnetic <
otis_gospodnetic@yahoo.com> wrote:

> Hello,
>
> Recently, I read "Matrix Factorization Techniques for Recommender Systems"
> from http://research.yahoo.com/node/2859 .  I was wondering what you think
> about this vs. what we have in Taste now?
>
> It looks like Collective Intelligence talks about this on p232-239 + 302...
> but I haven't read that yet.
>
> Thanks,
> Otis
>



-- 
Ted Dunning, CTO
DeepDyve