Precision/Recall for GenericBooleanPrefUserBasedRecommender

[Janina <ma...@googlemail.com>]

Hi all,

currently I am implementing a recommender based on a boolean pref data
model. I have to evaluate this recommender and I have tried to use the
precision and recall measurements Mahout provides. In Mahout-in-Action it
says:

"There isn’t even a notion of relative preference on which to select
a subset of good items. The best the test can do is randomly select some
preferred items as the good ones."

Does this mean the notion of what is good is really just random? Do you
maybe have another idea what I can use to evaluate my recommender? Or do I
have to manually define what is good to test the recommender? This would be
very hard because of the large data set I use...

Thanks a lot and greetings,
Janina

Re: Precision/Recall for GenericBooleanPrefUserBasedRecommender

[Sean Owen <sr...@gmail.com>]

I would not quite say it that way. The notion of "good" itself is not
random: the "good" items are all of the ones that the user is
associated to. The issue is that they're all equally good, so you have
no basis on which to choose some as the most relevant for test
purposes. You pick randomly.

You can also define the test/training set manually if you want, yes.

If you mean, are there other metrics, I'd look at normalized
discounted cumulative gain (nDCG), also implemented. It provides
perhaps a slightly more useful metric, since it scores results higher
for putting relevant results higher.

Sean


On Tue, Apr 10, 2012 at 7:54 AM, Janina <ma...@googlemail.com> wrote:
> Hi all,
>
> currently I am implementing a recommender based on a boolean pref data
> model. I have to evaluate this recommender and I have tried to use the
> precision and recall measurements Mahout provides. In Mahout-in-Action it
> says:
>
> "There isn’t even a notion of relative preference on which to select
> a subset of good items. The best the test can do is randomly select some
> preferred items as the good ones."
>
> Does this mean the notion of what is good is really just random? Do you
> maybe have another idea what I can use to evaluate my recommender? Or do I
> have to manually define what is good to test the recommender? This would be
> very hard because of the large data set I use...
>
> Thanks a lot and greetings,
> Janina