How to interpret recommendation strength

[Pat Ferrel <pa...@gmail.com>]

Using a boolean data model and log likelihood similarity I get recommendations with strengths.

If I were using preference rating magnitudes the recommendation strength is interpreted as the likely magnitude that a user would rate the recommendation. Using the boolean model I get values approaching 2 (this over a quick and small sample so not sure of the real range), which leads me to the question...

What is the meaning of the strength returned with the recommendation for boolean data? 

Re: How to interpret recommendation strength

[Pat Ferrel <pa...@gmail.com>]

I think I had some double talk there. Again I'm looking for a way to reasonable rank recommendations between users so globally rank them.

(((sum-of-similarities-for-user)*cooccurrences-for-user)/max-coocurrences-in-model)/max-sum-of-similarities-in-model ?

On Nov 15, 2012, at 10:52 AM, Pat Ferrel <pa...@gmail.com> wrote:

Thanks, you answered in the code too and I see how it is being used to rank returned recommendations, which all makes sense.

Maybe I'm looking for something different. For purposes other than ranking it seems like it would be nice to have value that represents the probability that the recommendation is a true preference. This would be a value between 0-1, obviously, and if it's available could be used to compare recommendation strengths between different users.

Another way to approach this magic number, not a probability here, and intuitively speaking might be to average the similarity of users in the neighborhood who preferred the recommendation but weight by the number of users and normalize by the max number of user who ever contributed to a recommendation in the entire model

On Nov 15, 2012, at 10:07 AM, Sean Owen <sr...@gmail.com> wrote:

It's an average similarity, weighted by count -- which is to say, it's a
sum of similarities. This isn't terribly principled but works reasonably in
practice. A simple average tends to over-weight unpopular items, but there
are likely better ways to account for that.


On Thu, Nov 15, 2012 at 5:59 PM, Pat Ferrel <pa...@gmail.com> wrote:

> Using a boolean data model and log likelihood similarity I get
> recommendations with strengths.
> 
> If I were using preference rating magnitudes the recommendation strength
> is interpreted as the likely magnitude that a user would rate the
> recommendation. Using the boolean model I get values approaching 2 (this
> over a quick and small sample so not sure of the real range), which leads
> me to the question...
> 
> What is the meaning of the strength returned with the recommendation for
> boolean data?

Re: How to interpret recommendation strength

[Pat Ferrel <pa...@gmail.com>]

Thanks, you answered in the code too and I see how it is being used to rank returned recommendations, which all makes sense.

Maybe I'm looking for something different. For purposes other than ranking it seems like it would be nice to have value that represents the probability that the recommendation is a true preference. This would be a value between 0-1, obviously, and if it's available could be used to compare recommendation strengths between different users.

Another way to approach this magic number, not a probability here, and intuitively speaking might be to average the similarity of users in the neighborhood who preferred the recommendation but weight by the number of users and normalize by the max number of user who ever contributed to a recommendation in the entire model

On Nov 15, 2012, at 10:07 AM, Sean Owen <sr...@gmail.com> wrote:

It's an average similarity, weighted by count -- which is to say, it's a
sum of similarities. This isn't terribly principled but works reasonably in
practice. A simple average tends to over-weight unpopular items, but there
are likely better ways to account for that.


On Thu, Nov 15, 2012 at 5:59 PM, Pat Ferrel <pa...@gmail.com> wrote:

> Using a boolean data model and log likelihood similarity I get
> recommendations with strengths.
> 
> If I were using preference rating magnitudes the recommendation strength
> is interpreted as the likely magnitude that a user would rate the
> recommendation. Using the boolean model I get values approaching 2 (this
> over a quick and small sample so not sure of the real range), which leads
> me to the question...
> 
> What is the meaning of the strength returned with the recommendation for
> boolean data?

Re: How to interpret recommendation strength

[Sean Owen <sr...@gmail.com>]

It's an average similarity, weighted by count -- which is to say, it's a
sum of similarities. This isn't terribly principled but works reasonably in
practice. A simple average tends to over-weight unpopular items, but there
are likely better ways to account for that.


On Thu, Nov 15, 2012 at 5:59 PM, Pat Ferrel <pa...@gmail.com> wrote:

> Using a boolean data model and log likelihood similarity I get
> recommendations with strengths.
>
> If I were using preference rating magnitudes the recommendation strength
> is interpreted as the likely magnitude that a user would rate the
> recommendation. Using the boolean model I get values approaching 2 (this
> over a quick and small sample so not sure of the real range), which leads
> me to the question...
>
> What is the meaning of the strength returned with the recommendation for
> boolean data?

Re: How to interpret recommendation strength

[Ted Dunning <te...@gmail.com>]

On Nov 15, 2012, at 6:02 PM, Pat Ferrel <pa...@gmail.com> wrote:

> Trying to catch up.
> 
> Isn't the sum of similarities actually a globally comparable number for strength of preference in a boolean model? I was thinking it wasn't but it is really. It may not be ideal but as an ordinal it should work, right?

Not sure without details. 

> Is the logic behind the IDF idea that very popular items are of less value in calculating recommendations?

Roughly yes. 

> If an IDF weight is to be applied isn't it to the preference values (0,1) before the similarity is calculated between users?

Not directly in my approach.  

In my approach LLR Is used to produce a binary item to item matrix.  The LLR computation has terms that are similar to IDF and which do something similar to what you say, though with corrections to avoid excess contribution by singletons and such. 

I then weight the item item matrix according to the frequency of the remaining non-zero elements in it.  I often recommend achieving this end by simply creating a document per row of the item item matrix and constructing a query of the users item history.  A text retrieval engine then does the necessary weighting as part of it normal query operations.  

> The intuition would be that people aren't all that similar just because they have puppy liking in common.
> 
> I'm afraid I got lost applying IDFish weighting to similarity strengths themselves.
> 
> On Nov 15, 2012, at 10:50 AM, Sean Owen <sr...@gmail.com> wrote:
> 
> That's kind of what it does now... though it weights everything as "1". Not
> so smart, but for sparse-ish data is not far off from a smarter answer.
> 
> 
> On Thu, Nov 15, 2012 at 6:47 PM, Ted Dunning <te...@gmail.com> wrote:
> 
>> My own preference (pun intended) is to use log-likelihood score for
>> determining which similarities are non-zero and then use simple frequency
>> weighting such as IDF for weighting the similarities.   This doesn't make
>> direct use of cooccurrence frequencies, but it works really well.  One
>> reason that it seems to work well is that by using only general occurrence
>> frequencies makes it *really* hard to overfit.
>> 
>> 
> 

Re: How to interpret recommendation strength

[Pat Ferrel <pa...@gmail.com>]

Trying to catch up.

Isn't the sum of similarities actually a globally comparable number for strength of preference in a boolean model? I was thinking it wasn't but it is really. It may not be ideal but as an ordinal it should work, right?

Is the logic behind the IDF idea that very popular items are of less value in calculating recommendations? If an IDF weight is to be applied isn't it to the preference values (0,1) before the similarity is calculated between users? The intuition would be that people aren't all that similar just because they have puppy liking in common.

I'm afraid I got lost applying IDFish weighting to similarity strengths themselves.

On Nov 15, 2012, at 10:50 AM, Sean Owen <sr...@gmail.com> wrote:

That's kind of what it does now... though it weights everything as "1". Not
so smart, but for sparse-ish data is not far off from a smarter answer.


On Thu, Nov 15, 2012 at 6:47 PM, Ted Dunning <te...@gmail.com> wrote:

> My own preference (pun intended) is to use log-likelihood score for
> determining which similarities are non-zero and then use simple frequency
> weighting such as IDF for weighting the similarities.   This doesn't make
> direct use of cooccurrence frequencies, but it works really well.  One
> reason that it seems to work well is that by using only general occurrence
> frequencies makes it *really* hard to overfit.
> 
> 

Re: How to interpret recommendation strength

[Sean Owen <sr...@gmail.com>]

That's kind of what it does now... though it weights everything as "1". Not
so smart, but for sparse-ish data is not far off from a smarter answer.


On Thu, Nov 15, 2012 at 6:47 PM, Ted Dunning <te...@gmail.com> wrote:

> My own preference (pun intended) is to use log-likelihood score for
> determining which similarities are non-zero and then use simple frequency
> weighting such as IDF for weighting the similarities.   This doesn't make
> direct use of cooccurrence frequencies, but it works really well.  One
> reason that it seems to work well is that by using only general occurrence
> frequencies makes it *really* hard to overfit.
>
>

Re: How to interpret recommendation strength

[Ted Dunning <te...@gmail.com>]

My own preference (pun intended) is to use log-likelihood score for
determining which similarities are non-zero and then use simple frequency
weighting such as IDF for weighting the similarities.   This doesn't make
direct use of cooccurrence frequencies, but it works really well.  One
reason that it seems to work well is that by using only general occurrence
frequencies makes it *really* hard to overfit.

On Thu, Nov 15, 2012 at 9:59 AM, Pat Ferrel <pa...@gmail.com> wrote:

> Using a boolean data model and log likelihood similarity I get
> recommendations with strengths.
>
> If I were using preference rating magnitudes the recommendation strength
> is interpreted as the likely magnitude that a user would rate the
> recommendation. Using the boolean model I get values approaching 2 (this
> over a quick and small sample so not sure of the real range), which leads
> me to the question...
>
> What is the meaning of the strength returned with the recommendation for
> boolean data?

Re: How to interpret recommendation strength

[Pat Ferrel <pa...@gmail.com>]

Oops, I think I found the answer. If so nevermind...

 /**
   * This computation is in a technical sense, wrong, since in the domain of "boolean preference users" where
   * all preference values are 1, this method should only ever return 1.0 or NaN. This isn't terribly useful
   * however since it means results can't be ranked by preference value (all are 1). So instead this returns a
   * sum of similarities to any other user in the neighborhood who has also rated the item.
   */

On Nov 15, 2012, at 9:59 AM, Pat Ferrel <pa...@gmail.com> wrote:

Using a boolean data model and log likelihood similarity I get recommendations with strengths.

If I were using preference rating magnitudes the recommendation strength is interpreted as the likely magnitude that a user would rate the recommendation. Using the boolean model I get values approaching 2 (this over a quick and small sample so not sure of the real range), which leads me to the question...

What is the meaning of the strength returned with the recommendation for boolean data?