You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Mirko <id...@googlemail.com> on 2010/02/24 20:07:48 UTC
GenericRecommenderIRStatsEvaluator results
Hi all,
I would like to evaluate the IR statistics of my item based recommender with the GenericRecommenderIRStatsEvaluator. However, precision and recall are 0.0 for each user, as I can see from my logs.
Possible, that my data are too sparse?
Regards,
Mirko
My data:
936080 preferences
162291 users
109661 items
How I call the evaluator:
IRStatistics irStats = new GenericRecommenderIRStatsEvaluator().evaluate(recBuilder, null, dm, null, 10, Double.NaN, 0.1);
log.debug("Precision: "+irStats.getPrecision());
log.debug("Recall: "+irStats.getRecall());
My RecommenderBuilder implementation:
public class SimpleRecommenderBuilder implements RecommenderBuilder{
@Override
public Recommender buildRecommender(DataModel dm) throws TasteException {
CachingItemSimilarity itemSim = new CachingItemSimilarity(new LogLikelihoodSimilarity(dm), dm);
ItemBasedRecommender itemRecommender = new GenericItemBasedRecommender(dm, itemSim);
return itemRecommender;
}
}
My logs:
[...]
2010-02-24 19:50:30,517 Processed 160000 users 2010-02-24 19:50:30,521 Processed 162291 users 2010-02-24 19:50:35,473 Retrieving new recommendations for user ID '-85512851039269613' 2010-02-24 19:50:35,473 Recommending items for user ID '-85512851039269613' 2010-02-24 19:50:59,995 Recommendations are: [RecommendedItem[item:454067076717825534, value:0.005567929], RecommendedItem[item:-2322998021761419502, value:0.005567929], RecommendedItem[item:612773236668408
7105, value:0.005567929], RecommendedItem[item:-3981461966915310408, value:0.005567929], RecommendedItem
[item:-5313445508550677921, value:0.005567929], RecommendedItem[item:-1832988160817640861, value:0.00556
7929], RecommendedItem[item:2911886255972263265, value:0.005567929], RecommendedItem[item:26217060620275
57406, value:0.005567929], RecommendedItem[item:-3536836517010980436, value:0.005567929], RecommendedIte
m[item:-6721918598917466000, value:0.005567929]]
2010-02-24 19:50:59,995 Evaluated with user -85512851039269613 in 29782ms
2010-02-24 19:50:59,995 Precision/recall/fall-out: 0.0 / 0.0 / 9.119405351853041E-5
2010-02-24 19:51:00,061 Processed 10000 users
2010-02-24 19:51:00,074 Processed 20000 users
2010-02-24 19:51:00,088 Processed 30000 users
2010-02-24 19:51:00,101 Processed 40000 users
2010-02-24 19:51:00,118 Processed 50000 users
2010-02-24 19:51:00,131 Processed 60000 users
2010-02-24 19:51:00,145 Processed 70000 users
2010-02-24 19:51:00,159 Processed 80000 users
2010-02-24 19:51:00,180 Processed 90000 users
2010-02-24 19:51:00,195 Processed 100000 users
2010-02-24 19:51:00,209 Processed 110000 users
2010-02-24 19:51:00,224 Processed 120000 users
2010-02-24 19:51:00,239 Processed 130000 users
2010-02-24 19:51:00,254 Processed 140000 users
2010-02-24 19:51:00,284 Processed 150000 users
2010-02-24 19:51:00,298 Processed 160000 users
2010-02-24 19:51:00,302 Processed 162291 users
2010-02-24 19:51:05,254 Retrieving new recommendations for user ID '124198514856103961'
2010-02-24 19:51:05,254 Recommending items for user ID '124198514856103961'
2010-02-24 19:52:00,228 Recommendations are: [RecommendedItem[item:6710022688720561421, value:0.005021
555], RecommendedItem[item:-9152989315951376207, value:0.00500819], RecommendedItem[item:-10610960939022
57025, value:0.00500819], RecommendedItem[item:-3145875224888183517, value:0.0047857966], RecommendedIte
m[item:4764651006754212795, value:0.004762556], RecommendedItem[item:6530066880626179760, value:0.004762
132], RecommendedItem[item:5360311107615737665, value:0.004639993], RecommendedItem[item:162168311040648
6210, value:0.004639993], RecommendedItem[item:-6562592997555071847, value:0.004632245], RecommendedItem
[item:8239589664710936158, value:0.0046123066]]
2010-02-24 19:52:00,228 Evaluated with user 124198514856103961 in 60233ms
2010-02-24 19:52:00,228 Precision/recall/fall-out: 0.0 / 0.0 / 9.119404271753251E-5
[...]
Re: GenericRecommenderIRStatsEvaluator results
Posted by Sean Owen <sr...@gmail.com>.
Actually one more thought here: reduce your 'at' parameter from 10 to
something like 3.
Looks like your users have on average 6 preferences each. In order to
compute these metrics you have to throw out some preferences and see
whether they are recommended back. But if your "at" is 10, then you
need more than 10 preferences to even run the test; in fact, you
should have more. The framework actually won't bother with users with
less than 20 preferences in this case.
That could be biasing the test but still doesn't really suggest why it
would come out 0 -- could be chance, but maybe not.
I think precision/recall tests are a little ill-defined for
recommenders. (We just had an interesting thread on this on
mahout-dev.) Really, the RecommenderEvaluator is what you want to use
in general.
This class exists mostly to test when your data set does not contain
any ratings. Then this is the only test you can run.
The next step is honestly to attach a debugger and see what's
happening in the evaluation. I'm having trouble guessing what the
issue or strange interaction with your data is.
On Wed, Feb 24, 2010 at 7:16 PM, Sean Owen <sr...@gmail.com> wrote:
> What is your DataModel like? what is your range of ratings?
>
> I don't think it's sparsity, no. These are results from just a handful
> of users; it's possible that the test results in 0 for just those
> users. But it's not a great sign; I do think something's wrong here
> but don't yet see an issue with what you are doing.
>
>
> On Wed, Feb 24, 2010 at 7:07 PM, Mirko
> <id...@googlemail.com> wrote:
>> Hi all,
>> I would like to evaluate the IR statistics of my item based recommender with the GenericRecommenderIRStatsEvaluator. However, precision and recall are 0.0 for each user, as I can see from my logs.
>>
>> Possible, that my data are too sparse?
>
Re: GenericRecommenderIRStatsEvaluator results
Posted by Sean Owen <sr...@gmail.com>.
I'd say recommenders answer at least one and maybe three of the
following three questions:
A. What are the top n best recommendations?
B. What are the top n best recommendations, ordered by quality?
C. What is the likely preference value for a given item and user?
All recommenders answer A. Virtually all answer B in the course of
answering A. Most answer C in the course of answering B.
Precision/recall tests measure effectiveness in answering A. MAE tests
measure effectiveness in answering C (which isn't applicable for all
recommenders).
You are only interested in A. Therefore precision/recall is the right
test to use. (Really, recall isn't going to be useful, but precision
is.)
There's no such thing as a perfect evaluator. To perfectly evaluate
recommendations you'd need to know the user's true preference for
everything, which even the user does not know. These two techniques
are standard and fine approximations to perfect evaluation. I am not
saying they're flawed, no. They do have biases though which are worth
understanding.
On Wed, Mar 3, 2010 at 10:23 AM, Mirko
<id...@googlemail.com> wrote:
>
>> In a sense, evaluating the quality of predictions is slightly the
>> wrong question to ask. After all a recommender's primary job is to
>> make ordered recommendations, only. It does not necessarily need to
>> predict preferences to do this, though most do.
>
> I see. Maybe this is part of my problem, apart from the novelty issue. I want to evaluate the quality of predictions, rather then how well they are ordered. To illustrate, I recommend an unordered list of top 5 items, it does not matter if item 1 and 4 are interchanged. But it may matter if item 4 and 7 are interchanged. Thus, it is not sufficient for my evaluation to ask whether predictions are in correct order. I rather need to evaluate the quality of the entirety of the 5 recommendations. I hoped that PR could be more appropriate then MAE to measure this 'quality of predictions' (rather then 'quality of ordering'). But if I get you correctly both, PR and MAE, are not appropriate to measure quality of predictions (directly).
>
>> I don't have a good reference for you but I think there's really one
>> way forward to evaluation: you need to collect data about how often
>> your recommended items were viewed / clicked, and how they were rated.
>> That is you'd really have to deploy the recommender and evaluate it
>> going forward. I just can't imagine any other solution since it is
>> necessarily based on information you don't have yet.
>
> Yes, I will give this a go.
>
> Thanks for your comments,
> Mirko
>
>
Re: GenericRecommenderIRStatsEvaluator results
Posted by Mirko <id...@googlemail.com>.
> In a sense, evaluating the quality of predictions is slightly the
> wrong question to ask. After all a recommender's primary job is to
> make ordered recommendations, only. It does not necessarily need to
> predict preferences to do this, though most do.
I see. Maybe this is part of my problem, apart from the novelty issue. I want to evaluate the quality of predictions, rather then how well they are ordered. To illustrate, I recommend an unordered list of top 5 items, it does not matter if item 1 and 4 are interchanged. But it may matter if item 4 and 7 are interchanged. Thus, it is not sufficient for my evaluation to ask whether predictions are in correct order. I rather need to evaluate the quality of the entirety of the 5 recommendations. I hoped that PR could be more appropriate then MAE to measure this 'quality of predictions' (rather then 'quality of ordering'). But if I get you correctly both, PR and MAE, are not appropriate to measure quality of predictions (directly).
> I don't have a good reference for you but I think there's really one
> way forward to evaluation: you need to collect data about how often
> your recommended items were viewed / clicked, and how they were rated.
> That is you'd really have to deploy the recommender and evaluate it
> going forward. I just can't imagine any other solution since it is
> necessarily based on information you don't have yet.
Yes, I will give this a go.
Thanks for your comments,
Mirko
Re: GenericRecommenderIRStatsEvaluator results
Posted by Sean Owen <sr...@gmail.com>.
On Wed, Mar 3, 2010 at 9:32 AM, Mirko
<id...@googlemail.com> wrote:
> I finally got some PR results from my evaluation. But I also became aware how painful recommender evaluation is. I don't think that RecommenderEvaluator is to be favored over PR in general. It depends on the use case of the recommender. For example, in some use cases MAE doesn't say much about the quality of recommendations.
I think they're answering different questions, and one or both might
or might not be useful in a given circumstance.
In a sense, evaluating the quality of predictions is slightly the
wrong question to ask. After all a recommender's primary job is to
make ordered recommendations, only. It does not necessarily need to
predict preferences to do this, though most do.
It's certainly a common concept and approach for recommenders (this
was how Netflix recommenders were scored for instance) so it's
available to you. But it isn't even applicable in all cases, like when
you have no preferences at all!
> In particular, I would be interested in what you think about novelty in the context of evalutation. I think novelty is a big potential of recommenders. Depending on the use case, novel recommendations are favorable over non-novels. However, both MAE and PR punish novelty, if I understand correctly. More novel recommendations mean worse results for MEA, and PR. Do you know of offline evaluation methods that consider novelty / any literatur on this issue?
Yes, this is exactly the reason I think using precision and recall
also has problems. It 'punishes' the recommender for making novel
recommendations. All it wants to see is that the recommender suggested
those items the user had already known about.
MAE doesn't really punish the recommender for this. It just can't
incorporate scores on these novel recommendations into the final
evaluation, since there is no 'real' preference to benchmark against.
I don't have a good reference for you but I think there's really one
way forward to evaluation: you need to collect data about how often
your recommended items were viewed / clicked, and how they were rated.
That is you'd really have to deploy the recommender and evaluate it
going forward. I just can't imagine any other solution since it is
necessarily based on information you don't have yet.
Re: GenericRecommenderIRStatsEvaluator results
Posted by Mirko <id...@googlemail.com>.
Hi Sean and all,
first, your book is really a helpful documentation of taste - not only as a general introduction, but also to look up things when developing or wondering about results. (And I'm not getting paid for saying this).
I finally got some PR results from my evaluation. But I also became aware how painful recommender evaluation is. I don't think that RecommenderEvaluator is to be favored over PR in general. It depends on the use case of the recommender. For example, in some use cases MAE doesn't say much about the quality of recommendations.
In particular, I would be interested in what you think about novelty in the context of evalutation. I think novelty is a big potential of recommenders. Depending on the use case, novel recommendations are favorable over non-novels. However, both MAE and PR punish novelty, if I understand correctly. More novel recommendations mean worse results for MEA, and PR. Do you know of offline evaluation methods that consider novelty / any literatur on this issue?
Regards,
- Mirko
Am 24.02.2010 um 22:09 schrieb Sean Owen:
> On Wed, Feb 24, 2010 at 8:51 PM, Mirko
> <id...@googlemail.com> wrote:
>> Is it possible that my ratings are problematic? Below is a sample from my data set. My ratings are all very similar. I derived the ratings from a "usage score" and normalized to a value between 0-1. For example, if a user used an item 1 times, the normalized rating is 0.0022271716 . This is the case for most preferences. When a user used an item more often, the rating (slightly) increases.
>
> In some cases, this could be a problem. For example, the Pearson
> correlation is undefined when one of the two data series has values
> that are all the same. If you were using PearsonCorrelationSimilarity,
> I'd believe this could be interfering with its ability to calculate
> similarities in many cases.
>
> But you're using LogLikelihoodSimilarity, which actually doesn't even
> look at the preference values.
>
>
>> I thought this approach could give better results then a boolean model. Would you agree or is this approach weird?
>
> It's really hard to say. You'd think that more data is better, but
> it's not always. You are actually using your rating values, just not
> in the similarity computation. Sometimes throwing out the data is
> better, if it's so noisy or un-representative of real preferences that
> it's just doing harm.
>
> I'm not sure there is a good rule of thumb about when to use the
> ratings or not. That's why there's a fairly robust evaluation
> framework to let you just try it out on your data.
>
> This is also the point where we should wheel in Ted D. for comment
> since he has dealt with this sort of thing for years.
>
>
>> This is very interesting! Could you please give me the subject of this thread on mahout-dev? I can't find it in the archive.
>
> Oops the discussion I was thinking of was actually here:
> http://issues.apache.org/jira/browse/MAHOUT-305
>
>
>> Is a Boolean DataModel one without ratings? Could I use the RecommenderEvaluator with a Boolean DataModel?
>
> Yes it is.
> No, you can't use RecommenderEvaluator. The framework thinks of such a
> DataModel as one where all ratings are 1.0. RecommenderEvaluator
> assesses the difference between predicted and actual ratings, but that
> makes no sense in a world where all ratings are always 1.0.
>
> Let me shamelessly plug Mahout in Action (http://manning.com/owen)
> which has a good bit of coverage about evaluation. But I don't think
> you're doing something wrong here that the book would clear up.
>
>
> I don't see anything obviously wrong. What you could do, if you have a
> bit of motivation, is simply step through the core evaluation method
> with a debugger. After 10-20 loops I'd imagine you'll get a sense of
> what's going on -- is it unable to create test data? are
> recommendations empty? etc. If you can spot anything odd like that,
> that would be a big clue to me.
>
> Then if we're still stumped perhaps I can ask you for your data,
> separately, so I can investigate myself.
Re: GenericRecommenderIRStatsEvaluator results
Posted by Sean Owen <sr...@gmail.com>.
On Wed, Feb 24, 2010 at 8:51 PM, Mirko
<id...@googlemail.com> wrote:
> Is it possible that my ratings are problematic? Below is a sample from my data set. My ratings are all very similar. I derived the ratings from a "usage score" and normalized to a value between 0-1. For example, if a user used an item 1 times, the normalized rating is 0.0022271716 . This is the case for most preferences. When a user used an item more often, the rating (slightly) increases.
In some cases, this could be a problem. For example, the Pearson
correlation is undefined when one of the two data series has values
that are all the same. If you were using PearsonCorrelationSimilarity,
I'd believe this could be interfering with its ability to calculate
similarities in many cases.
But you're using LogLikelihoodSimilarity, which actually doesn't even
look at the preference values.
> I thought this approach could give better results then a boolean model. Would you agree or is this approach weird?
It's really hard to say. You'd think that more data is better, but
it's not always. You are actually using your rating values, just not
in the similarity computation. Sometimes throwing out the data is
better, if it's so noisy or un-representative of real preferences that
it's just doing harm.
I'm not sure there is a good rule of thumb about when to use the
ratings or not. That's why there's a fairly robust evaluation
framework to let you just try it out on your data.
This is also the point where we should wheel in Ted D. for comment
since he has dealt with this sort of thing for years.
> This is very interesting! Could you please give me the subject of this thread on mahout-dev? I can't find it in the archive.
Oops the discussion I was thinking of was actually here:
http://issues.apache.org/jira/browse/MAHOUT-305
> Is a Boolean DataModel one without ratings? Could I use the RecommenderEvaluator with a Boolean DataModel?
Yes it is.
No, you can't use RecommenderEvaluator. The framework thinks of such a
DataModel as one where all ratings are 1.0. RecommenderEvaluator
assesses the difference between predicted and actual ratings, but that
makes no sense in a world where all ratings are always 1.0.
Let me shamelessly plug Mahout in Action (http://manning.com/owen)
which has a good bit of coverage about evaluation. But I don't think
you're doing something wrong here that the book would clear up.
I don't see anything obviously wrong. What you could do, if you have a
bit of motivation, is simply step through the core evaluation method
with a debugger. After 10-20 loops I'd imagine you'll get a sense of
what's going on -- is it unable to create test data? are
recommendations empty? etc. If you can spot anything odd like that,
that would be a big clue to me.
Then if we're still stumped perhaps I can ask you for your data,
separately, so I can investigate myself.
Re: GenericRecommenderIRStatsEvaluator results
Posted by Mirko <id...@googlemail.com>.
Hi Sean,
also the overall results are 0.0 for precision and recall. I will re-run the evalutation with a lower "at" parameter.
Is it possible that my ratings are problematic? Below is a sample from my data set. My ratings are all very similar. I derived the ratings from a "usage score" and normalized to a value between 0-1. For example, if a user used an item 1 times, the normalized rating is 0.0022271716 . This is the case for most preferences. When a user used an item more often, the rating (slightly) increases.
I thought this approach could give better results then a boolean model. Would you agree or is this approach weird?
> I think precision/recall tests are a little ill-defined for
> recommenders. (We just had an interesting thread on this on
> mahout-dev.) Really, the RecommenderEvaluator is what you want to use
> in general.
This is very interesting! Could you please give me the subject of this thread on mahout-dev? I can't find it in the archive.
> This class exists mostly to test when your data set does not contain
> any ratings. Then this is the only test you can run.
Is a Boolean DataModel one without ratings? Could I use the RecommenderEvaluator with a Boolean DataModel?
Thanks,
Mirko
Sample from my data set:
-3162216497309240828,2388991395777179713,0.0022271716
-3162216497309240828,-3069209537984688393,0.005567929
-3162216497309240828,-2618929863909794487,0.004454343
-3162216497309240828,5324707117640678349,0.0022271716
-3162216497309240828,-1532365936040783655,0.0022271716
-3162216497309240828,-5955221080145821903,0.0022271716
-3162216497309240828,4113222740828786538,0.0033407572
-3162216497309240828,-7695708570393034633,0.011135858
-3162216497309240828,6175940079748517486,0.0022271716
-3162216497309240828,7393161318827876898,0.0033407572
-3162216497309240828,2695305337569143674,0.0022271716
5621641483248293687,-5210224202303277338,0.0033407572
5621641483248293687,218889427960178691,0.0022271716
5621641483248293687,-1656822876488605684,0.0033407572
5621641483248293687,4455717574725992639,0.0022271716
5621641483248293687,-7490307787020003916,0.0022271716
5621641483248293687,-177903632999865810,0.0022271716
5621641483248293687,-8048798605167820211,0.0033407572
5621641483248293687,6318363976007294908,0.0022271716
5621641483248293687,-4448682744761460232,0.0077951
-7608832354152443012,-4328254679664372212,0.0022271716
-7608832354152443012,8798916565400141951,0.0022271716
-7608832354152443012,1655986988161457922,0.0022271716
-7608832354152443012,6680366060918372837,0.004454343
-7608832354152443012,9097101941836289600,0.004454343
-7608832354152443012,-4050477985649143346,0.0077951
-7608832354152443012,6483249541724522416,0.0022271716
-7608832354152443012,7888212850350411733,0.0022271716
-7608832354152443012,-4186615265961125232,0.005567929
-7608832354152443012,182029336574903084,0.0066815144
-3724542625493951560,901834265349196618,0.004454343
-3724542625493951560,2872984645083701473,0.0022271716
-3724542625493951560,710847465422191124,0.0077951
-3724542625493951560,-5723398178184638104,0.0033407572
-3724542625493951560,-4370720251872037961,0.0022271716
-3724542625493951560,810443090478857584,0.0022271716
-3724542625493951560,1954687945444914660,0.0033407572
-3724542625493951560,-434016020530942727,0.0022271716
-3724542625493951560,-8302108387647633409,0.0022271716
-3724542625493951560,507330792947537280,0.0033407572
-7503948664302479837,7084280837853105572,0.0033407572
-7503948664302479837,-1924829243914733069,0.0033407572
1649783211936870553,7857451131838754815,0.0022271716
1649783211936870553,8435274872607229275,0.0022271716
1649783211936870553,-4501948519344861350,0.0022271716
-3614918170829596350,-8740911171316545431,0.0022271716
-3614918170829596350,9111855612034415477,0.0022271716
8782629145521766207,-8261401803059057025,0.0022271716
8782629145521766207,7477297696504705454,0.0022271716
8593596540975467984,5161832021077148371,0.0022271716
8593596540975467984,-4977493117409780258,0.0077951
8593596540975467984,479989351350248267,0.0022271716
8593596540975467984,4763480195061959176,0.0033407572
4994655713734535662,1067823951843775847,0.0022271716
4994655713734535662,-8717602555723597440,0.0022271716
4994655713734535662,-900081358029843346,0.0022271716
8551803939396449658,-9070527651881014916,0.004454343
8551803939396449658,-731867567562838762,0.004454343
8551803939396449658,3585216355450215803,0.004454343
8551803939396449658,5081412625160281856,0.0022271716
8551803939396449658,7084280837853105572,0.0022271716
8551803939396449658,6309215032350562837,0.0022271716
8551803939396449658,3229581128678272400,0.0022271716
8551803939396449658,-5243266240130245406,0.0022271716
8551803939396449658,6679435817265618566,0.0022271716
-8384794006395586032,3197384946189382628,0.004454343
-8384794006395586032,306801002341480871,0.0033407572
-8384794006395586032,-6099277415666023996,0.0022271716
-8384794006395586032,1954687945444914660,0.0066815144
-1790777859396310501,1327095098958184939,0.004454343
-1790777859396310501,-4774539419328968279,0.0022271716
-1790777859396310501,7084280837853105572,0.005567929
-1790777859396310501,-7386668338099708703,0.0022271716
-1790777859396310501,1789748301850373328,0.0022271716
-1790777859396310501,5804143522977819365,0.013363029
-1790777859396310501,-334251965603959582,0.010022271
-1790777859396310501,-7576296270696146684,0.0022271716
-1790777859396310501,-5000138706628511200,0.012249443
-7542783584443801731,3585216355450215803,0.0033407572
-7542783584443801731,-2406154927270384099,0.0022271716
-7542783584443801731,-4050477985649143346,0.0022271716
-7542783584443801731,9111855612034415477,0.0022271716
-7542783584443801731,6175940079748517486,0.0022271716
-7542783584443801731,-3522223837183408650,0.0022271716
-7542783584443801731,-1960957431402637752,0.0022271716
-7542783584443801731,-4721008524353694287,0.0033407572
-7542783584443801731,507330792947537280,0.0022271716
6944365358450671125,1327095098958184939,0.0022271716
6944365358450671125,3585216355450215803,0.0033407572
-2170133603554543208,-4345051608145009077,0.0022271716
-2485483784871603234,3887954562187131337,0.004454343
-2485483784871603234,-3405987222396724810,0.0022271716
-2485483784871603234,6227280783235722609,0.004454343
-2485483784871603234,-2979754214499378340,0.004454343
-2485483784871603234,1674125821565185566,0.005567929
-2485483784871603234,-6184165426259382065,0.0022271716
-2485483784871603234,9171042243179519717,0.0022271716
-2485483784871603234,-6752075473109274310,0.0033407572
-2485483784871603234,4763480195061959176,0.0022271716
-2485483784871603234,-2616511497406263467,0.0022271716
-2485483784871603234,5557692575749358374,0.0022271716
4687451979443530426,5612030680563154297,0.0022271716
2256990898862823301,4155470905800486047,0.0033407572
2256990898862823301,-5243266240130245406,0.0022271716
4519320536426160635,-172884052995936406,0.0033407572
4519320536426160635,-731867567562838762,0.0022271716
2491341383048002353,-3183100304031888933,0.0022271716
-1703008347381719260,2249671975877176393,0.008908686
-1703008347381719260,-4025442128542569177,0.0022271716
-1703008347381719260,666567786754196295,0.0066815144
-1703008347381719260,810443090478857584,0.004454343
-1703008347381719260,1444563746925784146,0.0022271716
-1703008347381719260,-4260200376978362238,0.0022271716
2360751113445691119,3265275961432422241,0.0022271716
2360751113445691119,-6500370637084536949,0.0022271716
2360751113445691119,-9197217165883295249,0.0022271716
2360751113445691119,8458982604760233040,0.0022271716
2360751113445691119,2943943803070182192,0.0022271716
2360751113445691119,-434016020530942727,0.0033407572
6418598886840300636,3585216355450215803,0.0022271716
6418598886840300636,-3183100304031888933,0.0022271716
5849787832498006701,8474495553073610591,0.0022271716
5849787832498006701,-2159857214311297842,0.0033407572
5849787832498006701,1290396384218895189,0.0022271716
5849787832498006701,507330792947537280,0.0022271716
-1956248019161907125,2787907539422042661,0.0022271716
-1956248019161907125,-5556948485527356115,0.0022271716
-1956248019161907125,122175119313889177,0.0022271716
-1956248019161907125,-1727621483683893975,0.0022271716
-1956248019161907125,5801587430576821457,0.0033407572
-1956248019161907125,2063107611148935615,0.0033407572
-1956248019161907125,-5767997068698852509,0.0022271716
-1956248019161907125,-934444556457000840,0.004454343
-1956248019161907125,-7141354163483667787,0.0022271716
-1956248019161907125,-1223754232910636267,0.0033407572
-1956248019161907125,-405861188161280160,0.004454343
-1956248019161907125,1300974936426304050,0.0033407572
-1956248019161907125,6309541073553640753,0.012249443
-1956248019161907125,6653811804730774449,0.0022271716
-1956248019161907125,5015322369703545822,0.0033407572
-1956248019161907125,-1994696145908504504,0.0066815144
-1956248019161907125,-2259727532398580353,0.0077951
-1956248019161907125,2123995771431589373,0.0033407572
-1956248019161907125,-6251927776582057976,0.0033407572
-1956248019161907125,-1744119987828871241,0.0022271716
-1956248019161907125,-7142067797357251623,0.004454343
3010050112738655316,3585216355450215803,0.0066815144
3010050112738655316,8474495553073610591,0.0033407572
3010050112738655316,-196641029835882134,0.0022271716
3010050112738655316,-7490307787020003916,0.0033407572
3010050112738655316,6464094930452079790,0.0033407572
3010050112738655316,-3183100304031888933,0.0022271716
3010050112738655316,9111855612034415477,0.0022271716
3010050112738655316,-5138866751436494704,0.0022271716
3010050112738655316,5390783746581784855,0.0033407572
3010050112738655316,2695305337569143674,0.0022271716
3010050112738655316,507330792947537280,0.0066815144
-3997868911105540939,5539599052863627936,0.0022271716
-3997868911105540939,3486213142835327774,0.0022271716
-3997868911105540939,-343933718656203794,0.004454343
-3780514765660184299,-4060513158963517205,0.0033407572
-3780514765660184299,3585216355450215803,0.004454343
-3780514765660184299,4718872754184181836,0.0022271716
-3780514765660184299,-5938211453410237327,0.0022271716
-3780514765660184299,-2681503204555609534,0.004454343
-3780514765660184299,-139219214146380949,0.0022271716
-3780514765660184299,-2406154927270384099,0.0033407572
-3780514765660184299,-8256584327543939124,0.0022271716
-3780514765660184299,-175252550619564198,0.0022271716
-3780514765660184299,9171042243179519717,0.005567929
-3780514765660184299,5373661950899502537,0.0066815144
-3780514765660184299,393441362084330248,0.0022271716
8371470136588026854,7754872405737741929,0.0022271716
8371470136588026854,-5138866751436494704,0.0022271716
-3080349944334832757,-9096232754884560930,0.0022271716
780531655063079636,2692564121263611604,0.004454343
780531655063079636,3252772880526154546,0.004454343
780531655063079636,7313584286023766578,0.0033407572
780531655063079636,7275010564930623882,0.0022271716
780531655063079636,1308520868412794702,0.0022271716
780531655063079636,-6726755793685690793,0.004454343
-3577681554138780159,-7490307787020003916,0.0022271716
-3577681554138780159,-6431551263107086348,0.0022271716
-4142056294256867423,3585216355450215803,0.0022271716
-4142056294256867423,-3183100304031888933,0.0022271716
-4142056294256867423,507330792947537280,0.0022271716
1398542383991673232,3197384946189382628,0.0022271716
1398542383991673232,1327095098958184939,0.0033407572
1398542383991673232,3585216355450215803,0.0033407572
1398542383991673232,-3183100304031888933,0.0022271716
1398542383991673232,4182102600615078769,0.0022271716
5165462309880224755,-3061885761783178076,0.0033407572
5165462309880224755,-5041096034436565352,0.0033407572
5165462309880224755,-105623235477326811,0.004454343
5165462309880224755,6175940079748517486,0.0022271716
5165462309880224755,5531396405270558829,0.0022271716
5165462309880224755,-1887163529708310914,0.0022271716
5165462309880224755,-3003312792931996790,0.0033407572
5165462309880224755,5557692575749358374,0.004454343
-4351076531718366303,-1648771252322015010,0.0022271716
8623242532956789445,693476918119313863,0.0022271716
-8488704534466667536,-105623235477326811,0.0077951
-8488704534466667536,9108370397979208512,0.004454343
-8488704534466667536,1789748301850373328,0.004454343
-8488704534466667536,-334251965603959582,0.0022271716
-8488704534466667536,-7836088772324369683,0.004454343
-383728128810364530,244973034903943462,0.0022271716
-383728128810364530,8132838882384625094,0.0022271716
-383728128810364530,-4077513870245098023,0.0022271716
-383728128810364530,2943943803070182192,0.0022271716
-383728128810364530,4859794915733872910,0.0022271716
-383728128810364530,-6849591261276087891,0.0022271716
-2478227420826149219,-6614163799846330666,0.0022271716
-411574310920983446,1327095098958184939,0.0022271716
-411574310920983446,-3183100304031888933,0.0022271716
-411574310920983446,1954687945444914660,0.0022271716
-3744872235065329686,8474495553073610591,0.0022271716
-3744872235065329686,9111855612034415477,0.0022271716
-1112221664710802085,1045376291734989278,0.0022271716
-1112221664710802085,-5636898217307752116,0.0022271716
-1112221664710802085,-7695708570393034633,0.0033407572
4090674362190657682,-4254017485230881343,0.0033407572
4090674362190657682,-7762360570111608429,0.0033407572
-7527632671045837662,3475847555632042114,0.004454343
-7527632671045837662,-1218789744655811748,0.0022271716
-7527632671045837662,3194955804226887890,0.004454343
-7527632671045837662,1943994668912612445,0.0022271716
-7527632671045837662,-2415580176163625397,0.0022271716
-7527632671045837662,358983904694544773,0.0022271716
-7527632671045837662,-5598853064696374517,0.0022271716
-7527632671045837662,5880819958594438494,0.0022271716
-7527632671045837662,-7099283971802000295,0.0022271716
-7527632671045837662,833765922317965153,0.0033407572
-7527632671045837662,1232788054790411241,0.0033407572
-7527632671045837662,3931615472874670872,0.004454343
-7527632671045837662,8266165035809523737,0.0022271716
-7527632671045837662,-1642339538910419073,0.0022271716
-5971455341044088690,1327095098958184939,0.0022271716
-5971455341044088690,5557692575749358374,0.0022271716
6528798402096619012,-4026655064267673757,0.0022271716
6528798402096619012,4266475374638635508,0.0022271716
6528798402096619012,-4115849488040248248,0.0022271716
4935701437857785680,-5611298388783068275,0.0022271716
4935701437857785680,5373661950899502537,0.0033407572
619515850216010344,3585216355450215803,0.0033407572
619515850216010344,-5138866751436494704,0.0022271716
2711551565806425691,5557692575749358374,0.0022271716
6807977633659930168,5531396405270558829,0.0022271716
6807977633659930168,-1887163529708310914,0.0022271716
7985329964407141358,-2159857214311297842,0.0022271716
-8274771505706115768,-6500370637084536949,0.0033407572
-8274771505706115768,4718872754184181836,0.0033407572
-8274771505706115768,-7158155141925876922,0.0033407572
-5283100941349400782,-8717602555723597440,0.0022271716
-2365540731556772741,-4818542620102367091,0.0022271716
-2365540731556772741,9111855612034415477,0.0022271716
-3957361780771013460,-3304550021239085960,0.005567929
-3957361780771013460,-8256584327543939124,0.0022271716
1282620309235788693,8446104618955540150,0.0022271716
1282620309235788693,6160586547465285903,0.0022271716
1282620309235788693,8469390505592365353,0.0022271716
1282620309235788693,2224988814371259171,0.0033407572
1282620309235788693,-4220773994793408915,0.0077951
1282620309235788693,-4539189978908509352,0.0022271716
1282620309235788693,4202689417625140446,0.0033407572
1282620309235788693,-4536939205526310386,0.0022271716
1282620309235788693,-7470212307310885175,0.0022271716
1282620309235788693,257674075312929031,0.0022271716
1282620309235788693,5706559724571020282,0.0022271716
1282620309235788693,6314875735388664086,0.0022271716
1282620309235788693,6479229409862929659,0.004454343
1282620309235788693,8420903386923071274,0.0022271716
1282620309235788693,8901545724908352450,0.0022271716
1282620309235788693,5399265823182744847,0.0022271716
1282620309235788693,4237750374176848412,0.005567929
1282620309235788693,-185224489478953940,0.0066815144
1282620309235788693,-6914301623608608653,0.0022271716
1282620309235788693,-1736304954976138474,0.0033407572
1282620309235788693,8521441079177373948,0.0022271716
1282620309235788693,8437209419043462667,0.0033407572
1282620309235788693,-5254985181282420546,0.011135858
1282620309235788693,294897548447214616,0.0033407572
1282620309235788693,-5249806578735321265,0.005567929
1282620309235788693,-2018133799435298891,0.0022271716
1282620309235788693,-7845458944902666933,0.0022271716
1282620309235788693,-1895734305832357084,0.0022271716
1282620309235788693,-3268067683072738015,0.0022271716
1282620309235788693,-7880536511837829629,0.005567929
1282620309235788693,517159721434611238,0.0022271716
1282620309235788693,8474495553073610591,0.0022271716
1282620309235788693,-2623629192336893379,0.0022271716
1282620309235788693,8458982604760233040,0.0022271716
1282620309235788693,-7967693075711945398,0.005567929
1282620309235788693,-2406154927270384099,0.0022271716
1282620309235788693,2855399391619874663,0.0022271716
1282620309235788693,-730235095678282376,0.0022271716
1282620309235788693,5993210100509955931,0.0033407572
1282620309235788693,9171042243179519717,0.0022271716
1282620309235788693,9111855612034415477,0.0022271716
1282620309235788693,-122706388157359039,0.0022271716
1282620309235788693,3961542702305345114,0.0033407572
1282620309235788693,3587656892738120479,0.014476615
1282620309235788693,8184429757478391883,0.0022271716
1282620309235788693,2695305337569143674,0.0033407572
8517505098130328704,-2354315913441785226,0.0022271716
8517505098130328704,666567786754196295,0.0022271716
8517505098130328704,810443090478857584,0.0022271716
8517505098130328704,-2249891209256189917,0.005567929
8517505098130328704,-7513032036476932702,0.0022271716
-532047533149083615,-7335452605849009490,0.004454343
-532047533149083615,5384999675621513654,0.0077951
-532047533149083615,-695593973317069999,0.0077951
-532047533149083615,-4313235779939103350,0.004454343
-532047533149083615,-7658316895791758177,0.0022271716
8656820269370770173,9046507657305375823,0.0022271716
8656820269370770173,-3069209537984688393,0.004454343
8656820269370770173,-8980469299439626426,0.0022271716
8656820269370770173,-2014458257815540437,0.0022271716
8656820269370770173,-98813148661903171,0.0022271716
8656820269370770173,-4361568712254383708,0.0022271716
8656820269370770173,2578922575025387263,0.0033407572
8656820269370770173,-3542861098479182421,0.0022271716
8656820269370770173,3585216355450215803,0.0033407572
8656820269370770173,-1532365936040783655,0.0077951
8656820269370770173,5240904113957696167,0.008908686
8656820269370770173,-6184165426259382065,0.0022271716
8656820269370770173,-2243074962977168907,0.0022271716
8656820269370770173,7767977321142554240,0.0033407572
8656820269370770173,8365500024869021714,0.0033407572
8656820269370770173,1182876754474670069,0.0022271716
8656820269370770173,-9139294507937211079,0.0022271716
8656820269370770173,-2323355210788998320,0.0033407572
8656820269370770173,2597433642796298478,0.0022271716
8656820269370770173,-5901687700676273512,0.018930959
8656820269370770173,3992357852563404786,0.0022271716
8656820269370770173,5491879091387391803,0.0022271716
8656820269370770173,-5548922869462814425,0.0022271716
8656820269370770173,3784200466408481682,0.0022271716
8656820269370770173,2917321260939511995,0.0022271716
8656820269370770173,-7447276788919670229,0.0022271716
8656820269370770173,42933047741752535,0.0022271716
8656820269370770173,3127131336295163630,0.0022271716
7818183649809206896,9171042243179519717,0.0022271716
7818183649809206896,-7951233729908168098,0.0022271716
787450383326055724,-5252244730044589964,0.0022271716
787450383326055724,3849110595013876549,0.0022271716
787450383326055724,9120993680458869960,0.0022271716
787450383326055724,5373661950899502537,0.0022271716
787450383326055724,6679435817265618566,0.0022271716
2727491228189870225,4576876270343761131,0.0022271716
2727491228189870225,-9163148405595977385,0.0022271716
2727491228189870225,-623536053297067904,0.0022271716
2727491228189870225,4763480195061959176,0.0022271716
2727491228189870225,1962474163174091101,0.0022271716
-9005753702318557676,-7333766923580274632,0.0022271716
-2667310075779870210,7316684554354987832,0.0033407572
-2667310075779870210,-739929535491710828,0.0022271716
-2667310075779870210,5089810972385038852,0.0022271716
-185793595579247870,-7213150910597301204,0.004454343
-185793595579247870,-7162060775776231587,0.0033407572
-1820963272777526849,2589262924301734280,0.004454343
-1820963272777526849,-105623235477326811,0.010022271
-1820963272777526849,-3557302632169562435,0.014476615
-4479861654709369398,2097929176315093483,0.0022271716
-4479861654709369398,-1159786640174487108,0.004454343
-4479861654709369398,3197384946189382628,0.0022271716
-4479861654709369398,-8755464156197654167,0.0033407572
-4479861654709369398,-1582527952116924419,0.0022271716
-4479861654709369398,1789748301850373328,0.0022271716
-4479861654709369398,5373661950899502537,0.004454343
-4479861654709369398,4763480195061959176,0.0033407572
7886710350367444962,184839967492815792,0.0022271716
6676794222561466555,3352263812072474172,0.0022271716
7911703600617284947,-1159786640174487108,0.0022271716
7911703600617284947,9002566407876343676,0.0022271716
-3359479094931756447,3714183431931648598,0.0022271716
-3359479094931756447,-88085468050584453,0.0022271716
-3359479094931756447,2673968385134502798,0.0022271716
-3359479094931756447,1748819301509295490,0.004454343
-3359479094931756447,6941852195052765076,0.0033407572
-3359479094931756447,1644574205037202324,0.0022271716
-3359479094931756447,-6254635105268742374,0.0077951
-3359479094931756447,-860094584405193588,0.0022271716
-3359479094931756447,3872834068356954457,0.0033407572
-3359479094931756447,1383932110505565335,0.022271715
3378425205296388202,-8176217374018511991,0.0033407572
2488399755603129850,8486821392338505443,0.0033407572
2488399755603129850,-812693968430515831,0.0033407572
2488399755603129850,4637212239047360487,0.0033407572
7599985337300330930,8321197458856622752,0.0022271716
-1942163347761859682,-1159786640174487108,0.0033407572
-1942163347761859682,-1743996343649694018,0.010022271
5956696371552944225,8474495553073610591,0.0022271716
5956696371552944225,-6659696680396705782,0.0022271716
5956696371552944225,4761595023364501751,0.0022271716
8691601186461090514,-8888348574672816947,0.0022271716
3988642442405680144,-9096232754884560930,0.0022271716
-1362178633464093946,9171042243179519717,0.0022271716
5823542163485759223,-8857825663718966942,0.0022271716
-8805324042380510568,-5623595547011878951,0.0033407572
-8805324042380510568,3585216355450215803,0.0022271716
-8805324042380510568,-3405987222396724810,0.0022271716
-8805324042380510568,1182876754474670069,0.0022271716
-8805324042380510568,-3183100304031888933,0.0022271716
-8805324042380510568,2713889643137250426,0.0022271716
-8805324042380510568,1789748301850373328,0.0022271716
-8805324042380510568,2475491940931988199,0.0022271716
-8805324042380510568,-5138866751436494704,0.0033407572
-8805324042380510568,8363992834255670570,0.0022271716
-8805324042380510568,-334251965603959582,0.0022271716
-8805324042380510568,-434016020530942727,0.0033407572
-2529951479162141513,1444563746925784146,0.0022271716
-2529951479162141513,-2508452677995872982,0.0022271716
-2529951479162141513,-4660598909291459515,0.0033407572
-2529951479162141513,-7884049513323129159,0.0022271716
-2529951479162141513,-7073778008782503884,0.0022271716
2826564525103420660,5081412625160281856,0.0033407572
2826564525103420660,5968978643233812286,0.0022271716
2053206381808158133,-2079351410347776691,0.0022271716
2053206381808158133,3585216355450215803,0.0033407572
2053206381808158133,-8717602555723597440,0.0022271716
2053206381808158133,8336241169986392323,0.0022271716
2053206381808158133,1954687945444914660,0.0022271716
2053206381808158133,-5292268380911975860,0.0033407572
2053206381808158133,-334251965603959582,0.0033407572
2053206381808158133,6679435817265618566,0.0022271716
4701452358427011331,5842440240550642043,0.020044543
4701452358427011331,-4797115671829869562,0.008908686
4701452358427011331,3197384946189382628,0.0022271716
4701452358427011331,6037089375228153365,0.017817372
4701452358427011331,-8373276060180699361,0.014476615
4701452358427011331,3354457967000987769,0.0033407572
4701452358427011331,-6661633615411575332,0.023385301
4701452358427011331,-6722990168513702790,0.004454343
4701452358427011331,-8770428736294771189,0.0066815144
4701452358427011331,8233756523986964801,0.0022271716
4701452358427011331,-19886612368694133,0.0033407572
4701452358427011331,7103084986436092404,0.0022271716
4701452358427011331,-8386191125172030153,0.005567929
4701452358427011331,-6713986970818145835,0.0033407572
4701452358427011331,5832598651091914564,0.0066815144
4701452358427011331,1452857314026984865,0.0066815144
4701452358427011331,-6097320350499204087,0.008908686
4701452358427011331,2548691195714304375,0.0022271716
4701452358427011331,5701953034404396070,0.008908686
4701452358427011331,5423372641469080781,0.005567929
4701452358427011331,1482742675120422809,0.004454343
4701452358427011331,2494345084016305849,0.0066815144
4701452358427011331,-3873214899222365356,0.0022271716
4701452358427011331,80371481739827359,0.0022271716
4701452358427011331,-2107069627325048562,0.004454343
3835305033737653767,-1871166977825652103,0.0022271716
-1192197596104594601,-8651323336991110069,0.0022271716
-3639475179937121955,-1895734305832357084,0.0033407572
-3639475179937121955,3197384946189382628,0.0033407572
-3639475179937121955,3585216355450215803,0.005567929
-3639475179937121955,2525057669663079284,0.0022271716
-3639475179937121955,-6659696680396705782,0.0022271716
-3639475179937121955,-3183100304031888933,0.005567929
-3639475179937121955,-7410844972006593281,0.0022271716
-3639475179937121955,-6099277415666023996,0.005567929
-3639475179937121955,1954687945444914660,0.0066815144
-3639475179937121955,-5292268380911975860,0.008908686
-3639475179937121955,-5936999422521093309,0.0033407572
-3639475179937121955,1016248447215065870,0.0022271716
-3639475179937121955,-5138866751436494704,0.0022271716
-3639475179937121955,6175940079748517486,0.0022271716
-3639475179937121955,5390783746581784855,0.0022271716
-7606535364741002797,3585216355450215803,0.0022271716
447311178524030513,8914429597676860578,0.0022271716
447311178524030513,2195509875177744222,0.0022271716
447311178524030513,-4477744827348829019,0.0033407572
447311178524030513,-6869689457816421116,0.0022271716
Re: GenericRecommenderIRStatsEvaluator results
Posted by Sean Owen <sr...@gmail.com>.
What is your DataModel like? what is your range of ratings?
I don't think it's sparsity, no. These are results from just a handful
of users; it's possible that the test results in 0 for just those
users. But it's not a great sign; I do think something's wrong here
but don't yet see an issue with what you are doing.
On Wed, Feb 24, 2010 at 7:07 PM, Mirko
<id...@googlemail.com> wrote:
> Hi all,
> I would like to evaluate the IR statistics of my item based recommender with the GenericRecommenderIRStatsEvaluator. However, precision and recall are 0.0 for each user, as I can see from my logs.
>
> Possible, that my data are too sparse?