You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by michal shmueli <mi...@gmail.com> on 2009/11/04 13:22:04 UTC

problems with GenericRecommenderIRStatsEvaluator:

Hi,

I'm trying to evaluate the quality of the Boolean recommender. (I have no
ratings in my data but only: (userId,itemId,1) 1 for all entries.
I'm using this setting for the recommender:

UserSimilarity userSimilarity = new
TanimotoCoefficientSimilarity(dataModel);
UserNeighborhood neighborhood =  new NearestNUserNeighborhood(10, 0.0,
userSimilarity, dataModel, 1.0);
 recommender = new CachingRecommender(new
GenericBooleanPrefUserBasedRecommender(dataModel, neighborhood,
userSimilarity));

As for the evaluator :

 RecommenderIRStatsEvaluator evaluator = new
GenericRecommenderIRStatsEvaluator();
  IRStatistics stats =      evaluator.evaluate(new
GroupLensRecommenderBuilder(), null,model, null, 50, 1.0,0.7);



When running the eval, I keep getting:
09/11/04 13:56:01 INFO eval.
GenericRecommenderIRStatsEvaluator:
processOtherUSer: debug debug --- original id: 9606925 evaluate id: 9606925
09/11/04 13:56:01 INFO model.GenericUserPreferenceArray: debug debug ---
user id is: 0 size is: 0
Exception in thread "main" java.lang.IllegalArgumentException: size is
less than 1
       at
org.apache.mahout.cf.taste.impl.model.GenericUserPreferenceArray.<init>(GenericUserPreferenceArray.java:54)
       at
org.apache.mahout.cf.taste.impl.model.GenericUserPreferenceArray.<init>(GenericUserPreferenceArray.java:66)
       at
org.apache.mahout.cf.taste.impl.eval.GenericRecommenderIRStatsEvaluator.processOtherUser(GenericRecommenderIRStatsEvaluator.java:200)
       at
org.apache.mahout.cf.taste.impl.eval.GenericRecommenderIRStatsEvaluator.evaluate(GenericRecommenderIRStatsEvaluator.java:132)
       at
org.apache.mahout.cf.taste.example.grouplens.GroupLensRecommenderEvaluatorRunner.main(GroupLensRecommenderEvaluatorRunner.java:68)


However, when I'm trying to access the recommendation of this user :9606925
from the server I'm getting:

User:9606925
Recommender: GroupLensRecommender[recommender:CachingRecommender[recommender:GenericBooleanPrefUserBasedRecommender]]

Top 20 Preferences:
1.0	552559


1.0	450097
1.0	450098
1.0	450133
1.0	450133
1.0	450133
1.0	450133
1.0	450990
1.0	450990
1.0	457282
1.0	458155
1.0	458155
1.0	458155
1.0	458155
1.0	458155
1.0	458155
1.0	458155
1.0	458155
1.0	458155
1.0	458157



Recommendations:
4.9622216	457294
3.5182436	550267
3.1865802	459571
3.1865802	459823
3.1865802	550995
2.404762	458841
2.404762	450047
2.3295739	550150
2.3295739	459390
2.3295739	550545
2.3295739	551189
2.3295739	551428
2.3295739	551552
1.5438596	551940

Is something wrong with the way I'm using the evaluator? It does evaluate
the precision/recall for some of the users.
Any suggstions?

thanks.
Michal

Re: problems with GenericRecommenderIRStatsEvaluator:

Posted by Sean Owen <sr...@gmail.com>.
You are correct, but no that is not what's in question here.

On Thu, Nov 5, 2009 at 2:45 PM, James James <re...@yahoo.com> wrote:
> -->  My understanding is that the returned recommendentation list is a ranked list of items based on their predicted scores.
> Even under the setting of the boolean, each candidate item to be recommended gets a different predicted score. So I think that you are right they are also ranked.

Re: problems with GenericRecommenderIRStatsEvaluator:

Posted by James James <re...@yahoo.com>.
 Actually, there is, in the sense that you can have items that are more
likely to be purchase (if we assume that the data is {userid, itemId,1} and
1 represents that the user bought itemId). So you can ranked the item that
you recommend. Thus, if I ask for k=3, there is some ranked order of the
item that you would recommend.
Again, maybe it's being supported already, beacuse when asking
recommendation from the servlet, i get ranked list (even under the setting
of the boolean rec.)

-->  My understanding is that the returned recommendentation list is a ranked list of items based on their predicted scores.
Even under the setting of the boolean, each candidate item to be recommended gets a different predicted score. So I think that you are right they are also ranked.

 



________________________________
From: michal shmueli <mi...@gmail..com>
To: mahout-user@lucene.apache.org
Sent: Thu, November 5, 2009 8:16:54 AM
Subject: Re: problems with GenericRecommenderIRStatsEvaluator:

On Thu, Nov 5, 2009 at 3:33 PM, Sean Owen <sr...@gmail.com> wrote:

> On Thu, Nov 5, 2009 at 1:28 PM, michal shmueli <mi...@gmail.com>
> wrote:
> >>    - which recommenderEvaluator would you suggest for Boolean data? does
> it
> > also mean that I need to change the recommender that i'm using (which use
> > the Tanimoto similarity) ?
>
> None of them will work with RecommenderEvaluator. It is trying to
> evaluate how well estimated and actual preferences match, but, with
> boolean data, these are all always 1, so it's meaningless. You are
> doing the right thing.
>


>  - OK, so maybe what i'm missing here is the recommender itself. I went
> back to the IR evaluator, and I see that it compare the preferences of the
> user (say, items that appears in her list), with the items it gets from
> similar users.
>
  So I'm going one step back, to the similarity, and the recommender:

        UserSimilarity userSimilarity = new
TanimotoCoefficientSimilarity(dataModel);
            UserNeighborhood neighborhood =  new
NearestNUserNeighborhood(10, 0.0, userSimilarity, dataModel, 1.0);
recommender = new CachingRecommender(new
GenericBooleanPrefUserBasedRecommender(dataModel, neighborhood,
userSimilarity));

      Also in here I can't see where i can define the "trainning" set size.

>
> >  - It seems that you give the same results if you report on the best-2
> > (ranked 1st and 2nd) out of k=10 or the worst-2 (9th and 10th) from the
> 10,
> > but again , I might be wrong.
>
> What do you mean by 1st, 2nd, 9th, 10th? in your data set there is no
> notion of an item being "most liked" by a user.
>


      

Re: problems with GenericRecommenderIRStatsEvaluator:

Posted by Sean Owen <sr...@gmail.com>.
On Thu, Nov 5, 2009 at 2:16 PM, michal shmueli <mi...@gmail.com> wrote:
>         UserSimilarity userSimilarity = new
> TanimotoCoefficientSimilarity(dataModel);
>            UserNeighborhood neighborhood =  new
> NearestNUserNeighborhood(10, 0.0, userSimilarity, dataModel, 1.0);
>  recommender = new CachingRecommender(new
> GenericBooleanPrefUserBasedRecommender(dataModel, neighborhood,
> userSimilarity));
>
>      Also in here I can't see where i can define the "trainning" set size.

I don't either. A training set size is not a property of a
Recommender, it is a property of a RecommenderEvaluator.

But again, you are not using a RecommenderEvaluator! there is no
training set size in what you are doing.


>
>   Actually, there is, in the sense that you can have items that are more
> likely to be purchase (if we assume that the data is {userid, itemId,1} and
> 1 represents that the user bought itemId). So you can ranked the item that
> you recommend. Thus, if I ask for k=3, there is some ranked order of the
> item that you would recommend.

Yes, but, that is not the question here. We're talking about how to
choose *among the items the user already prefers* which ones are
relevant. There is no ranking there. You are talking about a ranking
among recommended items, which the user doesn't know about.

Re: problems with GenericRecommenderIRStatsEvaluator:

Posted by michal shmueli <mi...@gmail.com>.
On Thu, Nov 5, 2009 at 3:33 PM, Sean Owen <sr...@gmail.com> wrote:

> On Thu, Nov 5, 2009 at 1:28 PM, michal shmueli <mi...@gmail.com>
> wrote:
> >>    - which recommenderEvaluator would you suggest for Boolean data? does
> it
> > also mean that I need to change the recommender that i'm using (which use
> > the Tanimoto similarity) ?
>
> None of them will work with RecommenderEvaluator. It is trying to
> evaluate how well estimated and actual preferences match, but, with
> boolean data, these are all always 1, so it's meaningless. You are
> doing the right thing.
>


>  - OK, so maybe what i'm missing here is the recommender itself. I went
> back to the IR evaluator, and I see that it compare the preferences of the
> user (say, items that appears in her list), with the items it gets from
> similar users.
>
   So I'm going one step back, to the similarity, and the recommender:

         UserSimilarity userSimilarity = new
TanimotoCoefficientSimilarity(dataModel);
            UserNeighborhood neighborhood =  new
NearestNUserNeighborhood(10, 0.0, userSimilarity, dataModel, 1.0);
 recommender = new CachingRecommender(new
GenericBooleanPrefUserBasedRecommender(dataModel, neighborhood,
userSimilarity));

      Also in here I can't see where i can define the "trainning" set size.

>
> >  - It seems that you give the same results if you report on the best-2
> > (ranked 1st and 2nd) out of k=10 or the worst-2 (9th and 10th) from the
> 10,
> > but again , I might be wrong.
>
> What do you mean by 1st, 2nd, 9th, 10th? in your data set there is no
> notion of an item being "most liked" by a user.
>

   Actually, there is, in the sense that you can have items that are more
likely to be purchase (if we assume that the data is {userid, itemId,1} and
1 represents that the user bought itemId). So you can ranked the item that
you recommend. Thus, if I ask for k=3, there is some ranked order of the
item that you would recommend.
Again, maybe it's being supported already, beacuse when asking
recommendation from the servlet, i get ranked list (even under the setting
of the boolean rec.)

Re: problems with GenericRecommenderIRStatsEvaluator:

Posted by Sean Owen <sr...@gmail.com>.
On Thu, Nov 5, 2009 at 1:28 PM, michal shmueli <mi...@gmail.com> wrote:
>>    - which recommenderEvaluator would you suggest for Boolean data? does it
> also mean that I need to change the recommender that i'm using (which use
> the Tanimoto similarity) ?

None of them will work with RecommenderEvaluator. It is trying to
evaluate how well estimated and actual preferences match, but, with
boolean data, these are all always 1, so it's meaningless. You are
doing the right thing.


>  - It seems that you give the same results if you report on the best-2
> (ranked 1st and 2nd) out of k=10 or the worst-2 (9th and 10th) from the 10,
> but again , I might be wrong.

What do you mean by 1st, 2nd, 9th, 10th? in your data set there is no
notion of an item being "most liked" by a user.

Re: problems with GenericRecommenderIRStatsEvaluator:

Posted by michal shmueli <mi...@gmail.com>.
On Thu, Nov 5, 2009 at 3:14 PM, Sean Owen <sr...@gmail.com> wrote:

> You are again partly describing what RecommenderEvaluator does, not
> RecommenderIRStatsEvaluator.
>
> The main difference between what you are describing, and what happens,
> is there is no "70%" -- instead there is a relevance threshold.
> However in *RecommenderEvaluator* there is a parameter than controls
> what percent of data is used for training. But you are not using this
> code.
>
>    - which recommenderEvaluator would you suggest for Boolean data? does it
also mean that I need to change the recommender that i'm using (which use
the Tanimoto similarity) ?


> What ranking are you talking about, that is ignored?
>

  - It seems that you give the same results if you report on the best-2
(ranked 1st and 2nd) out of k=10 or the worst-2 (9th and 10th) from the 10,
but again , I might be wrong.

Cheers,
Michal

>
> Sean
>
> On Thu, Nov 5, 2009 at 1:08 PM, michal shmueli <mi...@gmail.com>
> wrote:
> > The way i envision this is the follow: assume user rates 10 items, this
> 10
> > are the correct items. Further assume that for recommendation we use
> subset
> > of this 10 items, say 70% (leave us with 30% for test) to build the
> > similarity, etc. Now, during evaluation, we ask from the recommneder for
> say
> > k items, and we check how many from the 3 correct item (the 30% of the
> > tests) are within the k recommended items.
> > This solutions ignore the ranking on the different items, however, this
> > could be also added later.
> >
> > Does it make sense?
> >
> > thanks,
> > Michal
>

Re: problems with GenericRecommenderIRStatsEvaluator:

Posted by Sean Owen <sr...@gmail.com>.
You are again partly describing what RecommenderEvaluator does, not
RecommenderIRStatsEvaluator.

The main difference between what you are describing, and what happens,
is there is no "70%" -- instead there is a relevance threshold.
However in *RecommenderEvaluator* there is a parameter than controls
what percent of data is used for training. But you are not using this
code.

What ranking are you talking about, that is ignored?

Sean

On Thu, Nov 5, 2009 at 1:08 PM, michal shmueli <mi...@gmail.com> wrote:
> The way i envision this is the follow: assume user rates 10 items, this 10
> are the correct items. Further assume that for recommendation we use subset
> of this 10 items, say 70% (leave us with 30% for test) to build the
> similarity, etc. Now, during evaluation, we ask from the recommneder for say
> k items, and we check how many from the 3 correct item (the 30% of the
> tests) are within the k recommended items.
> This solutions ignore the ranking on the different items, however, this
> could be also added later.
>
> Does it make sense?
>
> thanks,
> Michal

Re: problems with GenericRecommenderIRStatsEvaluator:

Posted by michal shmueli <mi...@gmail.com>.
The way i envision this is the follow: assume user rates 10 items, this 10
are the correct items. Further assume that for recommendation we use subset
of this 10 items, say 70% (leave us with 30% for test) to build the
similarity, etc. Now, during evaluation, we ask from the recommneder for say
k items, and we check how many from the 3 correct item (the 30% of the
tests) are within the k recommended items.
This solutions ignore the ranking on the different items, however, this
could be also added later.

Does it make sense?

thanks,
Michal

On Thu, Nov 5, 2009 at 2:16 PM, Sean Owen <sr...@gmail.com> wrote:

> It doesn't simulate "training" and "test", that's what I'm saying.
> This concept exists in RecommenderEvaluator, not
> RecommenderIRStatsEvaluator. They're reasonably different things.
>
> In RecommenderIRStatsEvaluator, there is instead a "relevance
> threshold" parameter.
>
> But the final parameter, which you refer to, is something else still.
> It simply controls what percentage of all data to use. It's a simple
> way to use a lot less data to produce a result faster.
>
> You are right that in your 'boolean' data, all preference values are
> effectively 1.0. So passing a 1.0 means that all items are considered
> relevant. That's fine, that's reasonable. While the framework
> typically removes all relevant items from a user for test purposes, it
> will remove only up to "at" items -- that is, if you are evaluating
> precision at 5, it will remove up to 5 items. In this case they are
> effectively randomly chosen since all items are equal.
>
> How would you like it to choose the relevant and not relevant items in
> this case? we can figure out how to do it then.
>
> Sean
>
> On Thu, Nov 5, 2009 at 12:06 PM, michal shmueli
> <mi...@gmail.com> wrote:
> >    >>  I still don't get why this parameter simulates the "training" and
> > the "test". In addition, since my data is Boolean, ain't it mean that
> anyway
> > what is 1 is relevant ? Is there another way to tell the recommender how
> to
> > chose the training and test sets?
>

Re: problems with GenericRecommenderIRStatsEvaluator:

Posted by Sean Owen <sr...@gmail.com>.
It doesn't simulate "training" and "test", that's what I'm saying.
This concept exists in RecommenderEvaluator, not
RecommenderIRStatsEvaluator. They're reasonably different things.

In RecommenderIRStatsEvaluator, there is instead a "relevance
threshold" parameter.

But the final parameter, which you refer to, is something else still.
It simply controls what percentage of all data to use. It's a simple
way to use a lot less data to produce a result faster.

You are right that in your 'boolean' data, all preference values are
effectively 1.0. So passing a 1.0 means that all items are considered
relevant. That's fine, that's reasonable. While the framework
typically removes all relevant items from a user for test purposes, it
will remove only up to "at" items -- that is, if you are evaluating
precision at 5, it will remove up to 5 items. In this case they are
effectively randomly chosen since all items are equal.

How would you like it to choose the relevant and not relevant items in
this case? we can figure out how to do it then.

Sean

On Thu, Nov 5, 2009 at 12:06 PM, michal shmueli
<mi...@gmail.com> wrote:
>    >>  I still don't get why this parameter simulates the "training" and
> the "test". In addition, since my data is Boolean, ain't it mean that anyway
> what is 1 is relevant ? Is there another way to tell the recommender how to
> chose the training and test sets?

Re: problems with GenericRecommenderIRStatsEvaluator:

Posted by michal shmueli <mi...@gmail.com>.
On Thu, Nov 5, 2009 at 12:47 PM, Sean Owen <sr...@gmail.com> wrote:

> On Thu, Nov 5, 2009 at 6:58 AM, michal shmueli <mi...@gmail.com>
> wrote:
> > Is this the aggregated statistics over all users?
>
> Yes
>
>
> > Just to make it more clear, for the 0.7 (last param), does it mean that
> for
> > each user we use 70% of the data to learn and 30% for test?
>
> You sound like you're describing a parameter to RecommenderEvaluator,
> which is the thing that splits the data into training and test and
> such. This is different.
>
> No, this parameter just controls how much of the overall data to use.
> You can turn it way down for a faster evaluation or use 1.0 to use all
> data.
>
> The parameter before it, the relevance threshold, sort of controls
> what you're saying. Items are split into "relevant" and "not relevant"
> groups based on this threshold. Then all the user's preferences for
> relevant items are removed, and recommendations are made. Precision
> and recall are calculated based on how many relevant items come back.
>
> It's a slightly funky adaptation of this metric from information
> retrieval, and picking the wrong threshold will give meaningless
> results. I might pass Double.NaN there to let the framework choose a
> suitable value.
>
>
    >>  I still don't get why this parameter simulates the "training" and
the "test". In addition, since my data is Boolean, ain't it mean that anyway
what is 1 is relevant ? Is there another way to tell the recommender how to
chose the training and test sets?




> > 3. Scalability of the Boolean recommneder ? I'm using this setting:
> > Is this recommender is scalable by the number of users?
>
> Looks reasonable to me. What are you asking, how does it scale? Memory
> requirements are dominated by the number of preference values in the
> input data and scale linearly as input size grows. Running time will
> grow linearly with the number of users, and is dominated by this.
>
>
> > 4. Some issue that I still didn't understand... I know that the Taste
> demo
> > doesn't required Hadoop.
> > I wonder when the Hadoop is required? Thus, assume I implemented these 4
> > steps:
>
> Hadoop isn't required unless you want to use Hadoop! you probably only
> need that if you want to do an off-line computation so large that you
> can't get it on one computer. This would probably happen once you have
> hundreds of millions of preference values. For smaller input, or when
> you need real-time recommendations, you want to use the framework as
> you are which does not involve Hadoop.
>
> See org.apache.mahout.cf.taste.hadoop for the Hadoop integration.
> There is information in the javadoc. You won't find a lot of info
> since this still isn't the primary way to use the recommender engine.
>

Re: problems with GenericRecommenderIRStatsEvaluator:

Posted by Sean Owen <sr...@gmail.com>.
On Thu, Nov 5, 2009 at 6:58 AM, michal shmueli <mi...@gmail.com> wrote:
> Is this the aggregated statistics over all users?

Yes


> Just to make it more clear, for the 0.7 (last param), does it mean that for
> each user we use 70% of the data to learn and 30% for test?

You sound like you're describing a parameter to RecommenderEvaluator,
which is the thing that splits the data into training and test and
such. This is different.

No, this parameter just controls how much of the overall data to use.
You can turn it way down for a faster evaluation or use 1.0 to use all
data.

The parameter before it, the relevance threshold, sort of controls
what you're saying. Items are split into "relevant" and "not relevant"
groups based on this threshold. Then all the user's preferences for
relevant items are removed, and recommendations are made. Precision
and recall are calculated based on how many relevant items come back.

It's a slightly funky adaptation of this metric from information
retrieval, and picking the wrong threshold will give meaningless
results. I might pass Double.NaN there to let the framework choose a
suitable value.

> 3. Scalability of the Boolean recommneder ? I'm using this setting:
> Is this recommender is scalable by the number of users?

Looks reasonable to me. What are you asking, how does it scale? Memory
requirements are dominated by the number of preference values in the
input data and scale linearly as input size grows. Running time will
grow linearly with the number of users, and is dominated by this.


> 4. Some issue that I still didn't understand... I know that the Taste demo
> doesn't required Hadoop.
> I wonder when the Hadoop is required? Thus, assume I implemented these 4
> steps:

Hadoop isn't required unless you want to use Hadoop! you probably only
need that if you want to do an off-line computation so large that you
can't get it on one computer. This would probably happen once you have
hundreds of millions of preference values. For smaller input, or when
you need real-time recommendations, you want to use the framework as
you are which does not involve Hadoop.

See org.apache.mahout.cf.taste.hadoop for the Hadoop integration.
There is information in the javadoc. You won't find a lot of info
since this still isn't the primary way to use the recommender engine.

Re: problems with GenericRecommenderIRStatsEvaluator:

Posted by michal shmueli <mi...@gmail.com>.
Thanks ! Sean.
The eval now terminates successfully.
If I may, i have few more clarifications questions:

1. At the end of the eval run I'm getting this information:
09/11/04 15:59:48 INFO grouplens.
GroupLensRecommenderEvaluatorRunner:
IRStatisticsImpl[precision:0.2960713,recall:0.170202808,fallOut:6.6057273888005836E-5]

Is this the aggregated statistics over all users?

2. Given that these are the params that i use for eval:
 IRStatistics stats =
      evaluator.evaluate(new GroupLensRecommenderBuilder(),
null,model, null, 5, 1.0,0.7);

Just to make it more clear, for the 0.7 (last param), does it mean that for
each user we use 70% of the data to learn and 30% for test?

3. Scalability of the Boolean recommneder ? I'm using this setting:

UserSimilarity userSimilarity = new
TanimotoCoefficientSimilarity(dataModel);
UserNeighborhood neighborhood =  new NearestNUserNeighborhood(10, 0.0,
userSimilarity, dataModel, 1.0);
 recommender = new CachingRecommender(new
GenericBooleanPrefUserBasedRecommender(dataModel, neighborhood,
userSimilarity));

Is this recommender is scalable by the number of users?

4. Some issue that I still didn't understand... I know that the Taste demo
doesn't required Hadoop.
I wonder when the Hadoop is required? Thus, assume I implemented these 4
steps:

   - DataModel
   - UserSimilarity and ItemSimilarity
   - UserNeighborhood
   - Recommender

Now, assume that I use some existing classes (the Boolean), where exactly
(if at all) the Hadoop is required? I can't find good documentation that
explain this issue.

Cheers,
Michal






On Wed, Nov 4, 2009 at 2:48 PM, Sean Owen <sr...@gmail.com> wrote:

> Looks like a small bug to me. When it happens that *all* the items
> from one user are considered relevant, and, that user happens to be
> the one being evaluated, this happens.
>
> Easy to fix and I'll commit it in just a second here. Get the latest
> from SVN in a few minutes and try agian.
>
> On Wed, Nov 4, 2009 at 12:22 PM, michal shmueli
> <mi...@gmail.com> wrote:
> > Hi,
> >
> > I'm trying to evaluate the quality of the Boolean recommender. (I have no
> > ratings in my data but only: (userId,itemId,1) 1 for all entries.
> > I'm using this setting for the recommender:
>

Re: problems with GenericRecommenderIRStatsEvaluator:

Posted by Sean Owen <sr...@gmail.com>.
Looks like a small bug to me. When it happens that *all* the items
from one user are considered relevant, and, that user happens to be
the one being evaluated, this happens.

Easy to fix and I'll commit it in just a second here. Get the latest
from SVN in a few minutes and try agian.

On Wed, Nov 4, 2009 at 12:22 PM, michal shmueli
<mi...@gmail.com> wrote:
> Hi,
>
> I'm trying to evaluate the quality of the Boolean recommender. (I have no
> ratings in my data but only: (userId,itemId,1) 1 for all entries.
> I'm using this setting for the recommender: