You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Lance Norskog <go...@gmail.com> on 2010/11/22 08:52:39 UTC
Interpreting the output of SVD
This post is inspired by this tutorial, which talks about interpreting
the U and V matrices:
http://www.puffinwarellc.com/index.php/news-and-articles/articles/30-singular-value-decomposition-tutorial.html
Given a DataModel that generates preferences between all Users and all
Items, lets take two Users and three Items:
I I I
U 0.5 0.2 0.1
U 0.8 0.3 0.2
What can we learn from an SVD factorization?
SVD gives 3 matrices and a scalar: U, a singular value matrix that
signifies the actual rank of the matrix, and transpose(V). For
simplicity, do the 1-dimensional factorization, which gives left and
right vectors instead of scalars. Ignoring the scaling matrix, we get
the Left and Right singular vectors.
The Left Singular Vector is: (column A x rows U1 and U2)
____A__
U1
U2
The Right Singular Vector is: (row B x columns I1, I2, and I3)
___I1___I2___I3
B
Now, the question: what do the Left and Right vectors encode? What do
column A and row B mean?
--
Lance Norskog
goksron@gmail.com
Re: Interpreting the output of SVD
Posted by Lance Norskog <go...@gmail.com>.
Wow! Thanks everyone, this really helps.
2010/11/22 Fernando Fernández <fe...@gmail.com>:
> Lance,
>
> Columns of U are in some contexts called "latent factors". For example, if
> we are applying SVD over a Document(User)-Term(Items) matrix, Columns of U
> could be interpreted as a representation of groups of terms (words that have
> similar meaning or tend to appear together in documents of the same kind, so
> in this case this "latent" factors are "topics" in some way. Another example
> of this is when we apply the SVD factorization in the famous movie
> recommendation problem. The "latent" factors (columns of the U matrix)
> represent somewhat some kind of "movie topics" (Drama, terror, comedy, and
> possible combinations of these...). Note that if we are trying to make
> recommendations of movies, we will recommend movies that has a similar
> topic, i.e. we will recommend probably a whole topic, not an specific
> movie... but SVD helps us find what movies fall into that topic. Note that
> this "topic" could be in fact something more abstract than "Drama" or
> "comedy".
>
> The interpretation of V is more or less the "transpose" of these. In the
> movie example, the columns of V could be seen as a representation of users
> that have seen (or rated) the same movie. So if two movies have a similar
> topic, it has been possible been rated or seen by the same persons, so both
> movies will have similar values on the V colum representing that group of
> persons...
>
> Actually, Rows of U can be use to find distances between users (according to
> what the have rated), and rows of Vt can be used to find distances between
> movies (according to what people have rated them).
>
> Last, The values of S are as some other users pointed, can be seen as a
> "weight" of the importance of this "latent" factors when i'm trying to see
> the differences between movies or between users.
>
> Hope this helps. Please, any other user correct me if you see something
> wrong in my examples.
>
> Best,
> Fernando.
>
>
>
> 2010/11/22 Ted Dunning <te...@gmail.com>
>
>> Commonly the square root of S is applied to both U and V. S is a set of
>> importance weightings for the otherwise
>> normalized columns of U and V.
>>
>> On Mon, Nov 22, 2010 at 10:10 AM, Sean Owen <sr...@gmail.com> wrote:
>>
>> > Hmm. I think I need to fix the second half of my analogy.
>> >
>> > It's really U x S that could be said to be users' preferences for
>> > pseudo-items. and S x VT could be said to be pseudo-users preferences for
>> > real items. S itself is a diagonal matrix of course and those values are
>> > kind of like "scaling factors" ... but I actually struggle to come up
>> with
>> > a
>> > good intuitive explanation of what S itself is (or really, U and V by
>> > themselves).
>> >
>> > Anyone smarter have a nice pithy analogy?
>> >
>> > On Mon, Nov 22, 2010 at 11:06 AM, Sean Owen <sr...@gmail.com> wrote:
>> > >
>> > > In more CF-oriented terms, S is an expression of pseudo-users'
>> > preferences
>> > > for pseudo-items. And then U expresses how much each real user
>> > corresponds
>> > > to each pseudo-user, and likewise for V and items.
>> > >
>> > > To put out a speculative analogy -- let's say we're looking at users'
>> > > preferences for songs. The "pseudo-items" that the SVD comes up with
>> > might
>> > > correspond to something like genres, or logical groupings of songs.
>> > > "Pseudo-users" are something like types of listeners, perhaps
>> > corresponding
>> > > to demographics.
>> > >
>> > > Whereas an entry in the original matrix makes a statement like "Tommy
>> > likes
>> > > the band Filter", an entry in S makes a statement like "Teenage boys in
>> > > moderately affluent households like industrial metal". And U says how
>> > much
>> > > Tommy is part of this demographic, and V tells how much Filter is
>> > industrial
>> > > metal.
>> > >
>> > >
>> >
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Interpreting the output of SVD
Posted by Lance Norskog <go...@gmail.com>.
On Mon, Nov 22, 2010 at 11:46 PM, Dan Brickley <da...@danbri.org> wrote:
> 2010/11/22 Fernando Fernández <fe...@gmail.com>:
>> Lance,
>>
>> Columns of U are in some contexts called "latent factors". For example, if
>> we are applying SVD over a Document(User)-Term(Items) matrix, Columns of U
>> could be interpreted as a representation of groups of terms (words that have
>> similar meaning or tend to appear together in documents of the same kind, so
>> in this case this "latent" factors are "topics" in some way. Another example
>> of this is when we apply the SVD factorization in the famous movie
>> recommendation problem. The "latent" factors (columns of the U matrix)
>> represent somewhat some kind of "movie topics" (Drama, terror, comedy, and
>> possible combinations of these...). Note that if we are trying to make
>> recommendations of movies, we will recommend movies that has a similar
>> topic, i.e. we will recommend probably a whole topic, not an specific
>> movie... but SVD helps us find what movies fall into that topic. Note that
>> this "topic" could be in fact something more abstract than "Drama" or
>> "comedy".
>
> Naming these seems a fun project; the examples in
> http://www.timelydevelopment.com/demos/NetflixPrize.aspx made me
> smile... but also illustrate the point.
>
> 'Offbeat / Dark-Comedy' vs 'Mass-Market / 'Beniffer' Movies'; 'Good'
> vs 'Twisted'; 'What a 10 year old boy would watch' vs 'What a liberal
> woman would watch'...
>
> cheers,
>
> Dan
>
It's been done:
http://www.nytimes.com/interactive/2010/01/10/nyregion/20100110-netflix-map.html
Just wonderful- matching your local zip codes to rentals is a hoot.
--
Lance Norskog
goksron@gmail.com
Re: Interpreting the output of SVD
Posted by Dan Brickley <da...@danbri.org>.
2010/11/22 Fernando Fernández <fe...@gmail.com>:
> Lance,
>
> Columns of U are in some contexts called "latent factors". For example, if
> we are applying SVD over a Document(User)-Term(Items) matrix, Columns of U
> could be interpreted as a representation of groups of terms (words that have
> similar meaning or tend to appear together in documents of the same kind, so
> in this case this "latent" factors are "topics" in some way. Another example
> of this is when we apply the SVD factorization in the famous movie
> recommendation problem. The "latent" factors (columns of the U matrix)
> represent somewhat some kind of "movie topics" (Drama, terror, comedy, and
> possible combinations of these...). Note that if we are trying to make
> recommendations of movies, we will recommend movies that has a similar
> topic, i.e. we will recommend probably a whole topic, not an specific
> movie... but SVD helps us find what movies fall into that topic. Note that
> this "topic" could be in fact something more abstract than "Drama" or
> "comedy".
Naming these seems a fun project; the examples in
http://www.timelydevelopment.com/demos/NetflixPrize.aspx made me
smile... but also illustrate the point.
'Offbeat / Dark-Comedy' vs 'Mass-Market / 'Beniffer' Movies'; 'Good'
vs 'Twisted'; 'What a 10 year old boy would watch' vs 'What a liberal
woman would watch'...
cheers,
Dan
Re: Interpreting the output of SVD
Posted by Fernando Fernández <fe...@gmail.com>.
Lance,
Columns of U are in some contexts called "latent factors". For example, if
we are applying SVD over a Document(User)-Term(Items) matrix, Columns of U
could be interpreted as a representation of groups of terms (words that have
similar meaning or tend to appear together in documents of the same kind, so
in this case this "latent" factors are "topics" in some way. Another example
of this is when we apply the SVD factorization in the famous movie
recommendation problem. The "latent" factors (columns of the U matrix)
represent somewhat some kind of "movie topics" (Drama, terror, comedy, and
possible combinations of these...). Note that if we are trying to make
recommendations of movies, we will recommend movies that has a similar
topic, i.e. we will recommend probably a whole topic, not an specific
movie... but SVD helps us find what movies fall into that topic. Note that
this "topic" could be in fact something more abstract than "Drama" or
"comedy".
The interpretation of V is more or less the "transpose" of these. In the
movie example, the columns of V could be seen as a representation of users
that have seen (or rated) the same movie. So if two movies have a similar
topic, it has been possible been rated or seen by the same persons, so both
movies will have similar values on the V colum representing that group of
persons...
Actually, Rows of U can be use to find distances between users (according to
what the have rated), and rows of Vt can be used to find distances between
movies (according to what people have rated them).
Last, The values of S are as some other users pointed, can be seen as a
"weight" of the importance of this "latent" factors when i'm trying to see
the differences between movies or between users.
Hope this helps. Please, any other user correct me if you see something
wrong in my examples.
Best,
Fernando.
2010/11/22 Ted Dunning <te...@gmail.com>
> Commonly the square root of S is applied to both U and V. S is a set of
> importance weightings for the otherwise
> normalized columns of U and V.
>
> On Mon, Nov 22, 2010 at 10:10 AM, Sean Owen <sr...@gmail.com> wrote:
>
> > Hmm. I think I need to fix the second half of my analogy.
> >
> > It's really U x S that could be said to be users' preferences for
> > pseudo-items. and S x VT could be said to be pseudo-users preferences for
> > real items. S itself is a diagonal matrix of course and those values are
> > kind of like "scaling factors" ... but I actually struggle to come up
> with
> > a
> > good intuitive explanation of what S itself is (or really, U and V by
> > themselves).
> >
> > Anyone smarter have a nice pithy analogy?
> >
> > On Mon, Nov 22, 2010 at 11:06 AM, Sean Owen <sr...@gmail.com> wrote:
> > >
> > > In more CF-oriented terms, S is an expression of pseudo-users'
> > preferences
> > > for pseudo-items. And then U expresses how much each real user
> > corresponds
> > > to each pseudo-user, and likewise for V and items.
> > >
> > > To put out a speculative analogy -- let's say we're looking at users'
> > > preferences for songs. The "pseudo-items" that the SVD comes up with
> > might
> > > correspond to something like genres, or logical groupings of songs.
> > > "Pseudo-users" are something like types of listeners, perhaps
> > corresponding
> > > to demographics.
> > >
> > > Whereas an entry in the original matrix makes a statement like "Tommy
> > likes
> > > the band Filter", an entry in S makes a statement like "Teenage boys in
> > > moderately affluent households like industrial metal". And U says how
> > much
> > > Tommy is part of this demographic, and V tells how much Filter is
> > industrial
> > > metal.
> > >
> > >
> >
>
Re: Interpreting the output of SVD
Posted by Ted Dunning <te...@gmail.com>.
Commonly the square root of S is applied to both U and V. S is a set of
importance weightings for the otherwise
normalized columns of U and V.
On Mon, Nov 22, 2010 at 10:10 AM, Sean Owen <sr...@gmail.com> wrote:
> Hmm. I think I need to fix the second half of my analogy.
>
> It's really U x S that could be said to be users' preferences for
> pseudo-items. and S x VT could be said to be pseudo-users preferences for
> real items. S itself is a diagonal matrix of course and those values are
> kind of like "scaling factors" ... but I actually struggle to come up with
> a
> good intuitive explanation of what S itself is (or really, U and V by
> themselves).
>
> Anyone smarter have a nice pithy analogy?
>
> On Mon, Nov 22, 2010 at 11:06 AM, Sean Owen <sr...@gmail.com> wrote:
> >
> > In more CF-oriented terms, S is an expression of pseudo-users'
> preferences
> > for pseudo-items. And then U expresses how much each real user
> corresponds
> > to each pseudo-user, and likewise for V and items.
> >
> > To put out a speculative analogy -- let's say we're looking at users'
> > preferences for songs. The "pseudo-items" that the SVD comes up with
> might
> > correspond to something like genres, or logical groupings of songs.
> > "Pseudo-users" are something like types of listeners, perhaps
> corresponding
> > to demographics.
> >
> > Whereas an entry in the original matrix makes a statement like "Tommy
> likes
> > the band Filter", an entry in S makes a statement like "Teenage boys in
> > moderately affluent households like industrial metal". And U says how
> much
> > Tommy is part of this demographic, and V tells how much Filter is
> industrial
> > metal.
> >
> >
>
Re: Interpreting the output of SVD
Posted by Sean Owen <sr...@gmail.com>.
Hmm. I think I need to fix the second half of my analogy.
It's really U x S that could be said to be users' preferences for
pseudo-items. and S x VT could be said to be pseudo-users preferences for
real items. S itself is a diagonal matrix of course and those values are
kind of like "scaling factors" ... but I actually struggle to come up with a
good intuitive explanation of what S itself is (or really, U and V by
themselves).
Anyone smarter have a nice pithy analogy?
On Mon, Nov 22, 2010 at 11:06 AM, Sean Owen <sr...@gmail.com> wrote:
>
> In more CF-oriented terms, S is an expression of pseudo-users' preferences
> for pseudo-items. And then U expresses how much each real user corresponds
> to each pseudo-user, and likewise for V and items.
>
> To put out a speculative analogy -- let's say we're looking at users'
> preferences for songs. The "pseudo-items" that the SVD comes up with might
> correspond to something like genres, or logical groupings of songs.
> "Pseudo-users" are something like types of listeners, perhaps corresponding
> to demographics.
>
> Whereas an entry in the original matrix makes a statement like "Tommy likes
> the band Filter", an entry in S makes a statement like "Teenage boys in
> moderately affluent households like industrial metal". And U says how much
> Tommy is part of this demographic, and V tells how much Filter is industrial
> metal.
>
>
Re: Interpreting the output of SVD
Posted by Sean Owen <sr...@gmail.com>.
Are you asking what the left and right vectors mean in general in the SVD?
S is a re-expression of the original matrix's transformation, but in a
different and more natural basis. (Actually it's an approximation, since
small singular values are tossed out, and the rank of S is therefore much
smaller since those 0s might as well not exist.) So U and V are really
change-of-basis transformations -- transforming into S's world of basis
vectors and back out again.
In more CF-oriented terms, S is an expression of pseudo-users' preferences
for pseudo-items. And then U expresses how much each real user corresponds
to each pseudo-user, and likewise for V and items.
To put out a speculative analogy -- let's say we're looking at users'
preferences for songs. The "pseudo-items" that the SVD comes up with might
correspond to something like genres, or logical groupings of songs.
"Pseudo-users" are something like types of listeners, perhaps corresponding
to demographics.
Whereas an entry in the original matrix makes a statement like "Tommy likes
the band Filter", an entry in S makes a statement like "Teenage boys in
moderately affluent households like industrial metal". And U says how much
Tommy is part of this demographic, and V tells how much Filter is industrial
metal.
(Unfortunately, the SVD doesn't tell you these interpretations, and
interpretations of S are rarely so neat as in this made-up analogy.)
On Mon, Nov 22, 2010 at 7:52 AM, Lance Norskog <go...@gmail.com> wrote:
> This post is inspired by this tutorial, which talks about interpreting
> the U and V matrices:
>
>
> http://www.puffinwarellc.com/index.php/news-and-articles/articles/30-singular-value-decomposition-tutorial.html
>
> Given a DataModel that generates preferences between all Users and all
> Items, lets take two Users and three Items:
> I I I
> U 0.5 0.2 0.1
> U 0.8 0.3 0.2
>
> What can we learn from an SVD factorization?
>
> SVD gives 3 matrices and a scalar: U, a singular value matrix that
> signifies the actual rank of the matrix, and transpose(V). For
> simplicity, do the 1-dimensional factorization, which gives left and
> right vectors instead of scalars. Ignoring the scaling matrix, we get
> the Left and Right singular vectors.
>
> The Left Singular Vector is: (column A x rows U1 and U2)
>
> ____A__
> U1
> U2
>
> The Right Singular Vector is: (row B x columns I1, I2, and I3)
>
> ___I1___I2___I3
> B
>
> Now, the question: what do the Left and Right vectors encode? What do
> column A and row B mean?
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: Interpreting the output of SVD
Posted by Ted Dunning <te...@gmail.com>.
They are closely related to the row and column sums.
On Sun, Nov 21, 2010 at 11:52 PM, Lance Norskog <go...@gmail.com> wrote:
> Now, the question: what do the Left and Right vectors encode? What do
> column A and row B mean?
>