Strange evaluation results for BookCrossingRecommender

[h....@free.fr]

Hello,

When testing the mahout example BookCrossingRecommender with default settings
(GenericUserBasedRecommender, PearsonCorrelationSimilarity,
NearestNUserNeighborhood), I noticed that the result of the evaluation
(AverageAbsoluteDifferenceRecommenderEvaluator) are
changing randomly, from one test to another. I get scores between 2.1 and 4.8.

Considering the size of the input (about 100000 users and 100000 books), I can't
imagine that the randomness in the algorithms can lead to huge evaluation
differences like that.

What do you think?

Re: Strange evaluation results for BookCrossingRecommender

[Sean Owen <sr...@gmail.com>]

It's just a mistake, a bit hasty in copying/pasting/committing before
I'd actually tested it. Let me fix that.

On Thu, Mar 11, 2010 at 2:15 PM, Julien Carme <ju...@gmail.com> wrote:
> Thanks for this clarification.Thanks for the new code as well, I am quite
> impressed by your responsiveness.
>
> About your new code, in particular BookCrossingBooleanRecommender, I am a
> bit surprised that you use GenericUserBasedRecommander, and not
> GenericBooleanPrefsUserBasedRecommander, which seems to habe been designed
> for this purpose. Maybe is it just a mistake?

Re: Strange evaluation results for BookCrossingRecommender

[Julien Carme <ju...@gmail.com>]

Thanks for this clarification.Thanks for the new code as well, I am quite
impressed by your responsiveness.

About your new code, in particular BookCrossingBooleanRecommender, I am a
bit surprised that you use GenericUserBasedRecommander, and not
GenericBooleanPrefsUserBasedRecommander, which seems to habe been designed
for this purpose. Maybe is it just a mistake?


2010/3/11 Sean Owen <sr...@gmail.com>

> You are right, you can't evaluate average error of predicted rating
> since there is no rating.
>
> What you can do is do a sort of precision-recall test on the results.
> You take out some items from a user's set of associations, and see how
> many of them are recommended back. It's an imperfect test -- it's not
> necessarily true that those removed items are the best
> recommendations, but they're not bad probably. However it's better
> than nothing.
>
> There were some pretty interesting discussions on this very topic on
> mahout-user a few weeks ago, you might check the archives.
>
> If you update your code from SVN you'll see I just checked in an
> example of this for Book Crossing.
>
> Lastly I'll plug the book which has pretty good coverage of
> evaluation: http://manning.com/owen
>
>
> On Thu, Mar 11, 2010 at 11:02 AM,  <h....@free.fr> wrote:
> >> You'll be willing to look at GenericBooleanPrefDataModel,
> >> GenericItemBasedRecommender, GenericBooleanPrefUserBasedRecommender
> >> and similarities supporting boolean preferences like Tanimoto and
> >> LogLikelihood.
> >
> > Thanks a lot, this is really helpful, I missed that. One problem with
> these kind
> > of recommender based on implicit ratings (user x bought item y) is that
> there
> > are no evaluator for them. I have to say it is not even clear to me what
> such an
> > evaluator should actually evaluate.
> >
> > Any references on document on the subject would be really appreciated.
> >
> >
> >
> >
> >
>

Re: Strange evaluation results for BookCrossingRecommender

[Sean Owen <sr...@gmail.com>]

You are right, you can't evaluate average error of predicted rating
since there is no rating.

What you can do is do a sort of precision-recall test on the results.
You take out some items from a user's set of associations, and see how
many of them are recommended back. It's an imperfect test -- it's not
necessarily true that those removed items are the best
recommendations, but they're not bad probably. However it's better
than nothing.

There were some pretty interesting discussions on this very topic on
mahout-user a few weeks ago, you might check the archives.

If you update your code from SVN you'll see I just checked in an
example of this for Book Crossing.

Lastly I'll plug the book which has pretty good coverage of
evaluation: http://manning.com/owen


On Thu, Mar 11, 2010 at 11:02 AM,  <h....@free.fr> wrote:
>> You'll be willing to look at GenericBooleanPrefDataModel,
>> GenericItemBasedRecommender, GenericBooleanPrefUserBasedRecommender
>> and similarities supporting boolean preferences like Tanimoto and
>> LogLikelihood.
>
> Thanks a lot, this is really helpful, I missed that. One problem with these kind
> of recommender based on implicit ratings (user x bought item y) is that there
> are no evaluator for them. I have to say it is not even clear to me what such an
> evaluator should actually evaluate.
>
> Any references on document on the subject would be really appreciated.
>
>
>
>
>

Re: Strange evaluation results for BookCrossingRecommender

[h....@free.fr]

> You'll be willing to look at GenericBooleanPrefDataModel,
> GenericItemBasedRecommender, GenericBooleanPrefUserBasedRecommender
> and similarities supporting boolean preferences like Tanimoto and
> LogLikelihood.

Thanks a lot, this is really helpful, I missed that. One problem with these kind
of recommender based on implicit ratings (user x bought item y) is that there
are no evaluator for them. I have to say it is not even clear to me what such an
evaluator should actually evaluate.

Any references on document on the subject would be really appreciated.

Re: Strange evaluation results for BookCrossingRecommender

[Claudio Martella <cl...@tis.bz.it>]

h.9000@free.fr wrote:
> I think I found the explanation of the poor result and, maybe, the instability.
>
> More than 60% of the ratings are 0/10. This is what the publishers of this
> dataset call "implicit rating". It means that the book was read (or purchased)
> but not rated by the user.
>
> It seems that BookCrossingDataModel is not aware of that and just considered
> them as rating 0. It is therefore not surprising that results are inconsistant.
> An obvious way to solve the problem would be to filter out these implicit
> ratings.
>
> It would be interesting as well to change all ratings to "0" and to consider all
> of them as implicit. There is so far no mahout examples dedicated to
> recommendation based on binary data (user as bought item or not), even though
> this seems to me like a more common problem than recommendation based on actual
> ratings.
>   
You'll be willing to look at GenericBooleanPrefDataModel,
GenericItemBasedRecommender, GenericBooleanPrefUserBasedRecommender
and similarities supporting boolean preferences like Tanimoto and
LogLikelihood.



> Selon Sean Owen <sr...@gmail.com>:
>
>   
>> I see the same variance, but I believe it's due to a small input size.
>> At the moment it's using only 5% of the total input, or about 50,000
>> ratings over 5,000 users. That's fairly small. From there, it's also
>> looking at only 5% of those users to form neighborhoods. These are
>> just too low, and I have increased the amount of data the evaluation
>> uses in a few ways, and get much more stable results.
>>
>> I also switched the algorithm it uses, since the average difference
>> was 4 out of 10, which is pretty poor. I think with more research one
>> could pick the optimal algorithm, but I just picked something that
>> worked a little better (< 3) for now.
>>
>> On Tue, Mar 9, 2010 at 6:30 PM, Sean Owen <sr...@gmail.com> wrote:
>>     
>>> I see, that definitely doesn't sound right. Let me run it myself
>>> tonight when I am home and see what I observe.
>>>
>>> On Tue, Mar 9, 2010 at 5:40 PM, Â <h....@free.fr> wrote:
>>>       
>>>> I did not change anything from the example provided in mahout-example,
>>>> development version. It uses 5% for evaluation, which is 5000 instances.
>>>>         
>> With
>>     
>>>> such test set size, the range should not be that big. I suspect that there
>>>>         
>> is
>>     
>>>> something wrong somewhere.
>>>>         
>
>
>
>   


-- 
Claudio Martella
Digital Technologies
Unit Research & Development - Analyst

TIS innovation park
Via Siemens 19 | Siemensstr. 19
39100 Bolzano | 39100 Bozen
Tel. +39 0471 068 123
Fax  +39 0471 068 129
claudio.martella@tis.bz.it http://www.tis.bz.it

Short information regarding use of personal data. According to Section 13 of Italian Legislative Decree no. 196 of 30 June 2003, we inform you that we process your personal data in order to fulfil contractual and fiscal obligations and also to send you information regarding our services and events. Your personal data are processed with and without electronic means and by respecting data subjects' rights, fundamental freedoms and dignity, particularly with regard to confidentiality, personal identity and the right to personal data protection. At any time and without formalities you can write an e-mail to privacy@tis.bz.it in order to object the processing of your personal data for the purpose of sending advertising materials and also to exercise the right to access personal data and other rights referred to in Section 7 of Decree 196/2003. The data controller is TIS Techno Innovation Alto Adige, Siemens Street n. 19, Bolzano. You can find the complete information on the web site www.tis.bz.it.

Re: Strange evaluation results for BookCrossingRecommender

[Sean Owen <sr...@gmail.com>]

Ah, good catch. I will adjust that.

I'm happy to make a new example for 'boolean' data, perhaps based on
BookCrossing. It would just ignore the rating data.

On Wed, Mar 10, 2010 at 2:46 PM,  <h....@free.fr> wrote:
> I think I found the explanation of the poor result and, maybe, the instability.
>
> More than 60% of the ratings are 0/10. This is what the publishers of this
> dataset call "implicit rating". It means that the book was read (or purchased)
> but not rated by the user.
>
> It seems that BookCrossingDataModel is not aware of that and just considered
> them as rating 0. It is therefore not surprising that results are inconsistant.
> An obvious way to solve the problem would be to filter out these implicit
> ratings.
>
> It would be interesting as well to change all ratings to "0" and to consider all
> of them as implicit. There is so far no mahout examples dedicated to
> recommendation based on binary data (user as bought item or not), even though
> this seems to me like a more common problem than recommendation based on actual
> ratings.
>
>
> Selon Sean Owen <sr...@gmail.com>:
>
>> I see the same variance, but I believe it's due to a small input size.
>> At the moment it's using only 5% of the total input, or about 50,000
>> ratings over 5,000 users. That's fairly small. From there, it's also
>> looking at only 5% of those users to form neighborhoods. These are
>> just too low, and I have increased the amount of data the evaluation
>> uses in a few ways, and get much more stable results.
>>
>> I also switched the algorithm it uses, since the average difference
>> was 4 out of 10, which is pretty poor. I think with more research one
>> could pick the optimal algorithm, but I just picked something that
>> worked a little better (< 3) for now.
>>
>> On Tue, Mar 9, 2010 at 6:30 PM, Sean Owen <sr...@gmail.com> wrote:
>> > I see, that definitely doesn't sound right. Let me run it myself
>> > tonight when I am home and see what I observe.
>> >
>> > On Tue, Mar 9, 2010 at 5:40 PM,  <h....@free.fr> wrote:
>> >> I did not change anything from the example provided in mahout-example,
>> >> development version. It uses 5% for evaluation, which is 5000 instances.
>> With
>> >> such test set size, the range should not be that big. I suspect that there
>> is
>> >> something wrong somewhere.
>> >
>>
>
>
>

Re: Strange evaluation results for BookCrossingRecommender

[h....@free.fr]

I think I found the explanation of the poor result and, maybe, the instability.

More than 60% of the ratings are 0/10. This is what the publishers of this
dataset call "implicit rating". It means that the book was read (or purchased)
but not rated by the user.

It seems that BookCrossingDataModel is not aware of that and just considered
them as rating 0. It is therefore not surprising that results are inconsistant.
An obvious way to solve the problem would be to filter out these implicit
ratings.

It would be interesting as well to change all ratings to "0" and to consider all
of them as implicit. There is so far no mahout examples dedicated to
recommendation based on binary data (user as bought item or not), even though
this seems to me like a more common problem than recommendation based on actual
ratings.


Selon Sean Owen <sr...@gmail.com>:

> I see the same variance, but I believe it's due to a small input size.
> At the moment it's using only 5% of the total input, or about 50,000
> ratings over 5,000 users. That's fairly small. From there, it's also
> looking at only 5% of those users to form neighborhoods. These are
> just too low, and I have increased the amount of data the evaluation
> uses in a few ways, and get much more stable results.
>
> I also switched the algorithm it uses, since the average difference
> was 4 out of 10, which is pretty poor. I think with more research one
> could pick the optimal algorithm, but I just picked something that
> worked a little better (< 3) for now.
>
> On Tue, Mar 9, 2010 at 6:30 PM, Sean Owen <sr...@gmail.com> wrote:
> > I see, that definitely doesn't sound right. Let me run it myself
> > tonight when I am home and see what I observe.
> >
> > On Tue, Mar 9, 2010 at 5:40 PM,  <h....@free.fr> wrote:
> >> I did not change anything from the example provided in mahout-example,
> >> development version. It uses 5% for evaluation, which is 5000 instances.
> With
> >> such test set size, the range should not be that big. I suspect that there
> is
> >> something wrong somewhere.
> >
>

Re: Strange evaluation results for BookCrossingRecommender

[Sean Owen <sr...@gmail.com>]

I see the same variance, but I believe it's due to a small input size.
At the moment it's using only 5% of the total input, or about 50,000
ratings over 5,000 users. That's fairly small. From there, it's also
looking at only 5% of those users to form neighborhoods. These are
just too low, and I have increased the amount of data the evaluation
uses in a few ways, and get much more stable results.

I also switched the algorithm it uses, since the average difference
was 4 out of 10, which is pretty poor. I think with more research one
could pick the optimal algorithm, but I just picked something that
worked a little better (< 3) for now.

On Tue, Mar 9, 2010 at 6:30 PM, Sean Owen <sr...@gmail.com> wrote:
> I see, that definitely doesn't sound right. Let me run it myself
> tonight when I am home and see what I observe.
>
> On Tue, Mar 9, 2010 at 5:40 PM,  <h....@free.fr> wrote:
>> I did not change anything from the example provided in mahout-example,
>> development version. It uses 5% for evaluation, which is 5000 instances. With
>> such test set size, the range should not be that big. I suspect that there is
>> something wrong somewhere.
>

Re: Strange evaluation results for BookCrossingRecommender

[Sean Owen <sr...@gmail.com>]

I see, that definitely doesn't sound right. Let me run it myself
tonight when I am home and see what I observe.

On Tue, Mar 9, 2010 at 5:40 PM,  <h....@free.fr> wrote:
> I did not change anything from the example provided in mahout-example,
> development version. It uses 5% for evaluation, which is 5000 instances. With
> such test set size, the range should not be that big. I suspect that there is
> something wrong somewhere.

Re: Strange evaluation results for BookCrossingRecommender

[h....@free.fr]

I did not change anything from the example provided in mahout-example,
development version. It uses 5% for evaluation, which is 5000 instances. With
such test set size, the range should not be that big. I suspect that there is
something wrong somewhere.

Selon Sean Owen <sr...@gmail.com>:

> It depends on how big a subset of the data you are using to evaluate,
> and also how much you are using for test versus training. Yes, that
> kind of range is undesirable. How are you executing the evaluation?
>
> On Tue, Mar 9, 2010 at 4:50 PM,  <h....@free.fr> wrote:
> > Hello,
> >
> > When testing the mahout example BookCrossingRecommender with default
> settings
> > (GenericUserBasedRecommender, PearsonCorrelationSimilarity,
> > NearestNUserNeighborhood), I noticed that the result of the evaluation
> > (AverageAbsoluteDifferenceRecommenderEvaluator) are
> > changing randomly, from one test to another. I get scores between 2.1 and
> 4.8.
> >
> > Considering the size of the input (about 100000 users and 100000 books), I
> can't
> > imagine that the randomness in the algorithms can lead to huge evaluation
> > differences like that.
> >
> > What do you think?
> >
>

Re: Strange evaluation results for BookCrossingRecommender

[Sean Owen <sr...@gmail.com>]

It depends on how big a subset of the data you are using to evaluate,
and also how much you are using for test versus training. Yes, that
kind of range is undesirable. How are you executing the evaluation?

On Tue, Mar 9, 2010 at 4:50 PM,  <h....@free.fr> wrote:
> Hello,
>
> When testing the mahout example BookCrossingRecommender with default settings
> (GenericUserBasedRecommender, PearsonCorrelationSimilarity,
> NearestNUserNeighborhood), I noticed that the result of the evaluation
> (AverageAbsoluteDifferenceRecommenderEvaluator) are
> changing randomly, from one test to another. I get scores between 2.1 and 4.8.
>
> Considering the size of the input (about 100000 users and 100000 books), I can't
> imagine that the randomness in the algorithms can lead to huge evaluation
> differences like that.
>
> What do you think?
>