You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Claudia Grieco <gr...@crmpa.unisa.it> on 2009/08/26 15:55:47 UTC
Problems with evaluator.
Hi,
I'm using a version of mahout prior to MAHOUT_150, since the changes after
that version were too big.
I'm having a problem with an evaluation class
(GenericRecommenderIRStatsEvaluator): it always returns precision and recall
as NaN values. I'm using a c# port of mahout so I can't run a detailed debug
and understand what is wrong, but from some experiments I understood it
never calls my recommender (it obtains data of each user, calculate their
neighborhood and that's all).
Here is some code I'm using:
GenericRecommenderIRStatsEvaluator evaluator = new
GenericRecommenderIRStatsEvaluator();
MyDataSource source = new MyDataSource(); //a jdbc data source
ConnectionPoolDataSource poolSource = new ConnectionPoolDataSource(source);
DataModel model = new MyDataModel(poolSource, "iscrizioni", "studentID",
"ofalID", "score");
//builder is a RecommenderBuilder that returns my
UserRecommender
IRStatistics statistics = evaluator.evaluate(builder, model, null, 5,
Double.NegativeInfinity, 0.5);
string results = "Precision:" + statistics.getPrecision() + " Recall" +
statistics.getRecall();
Console.WriteLine("results:" + results);
My recommender has always worked, so I don't think it's its fault.
Thanks in advance
Re: R: R: R: R: Problems with evaluator.
Posted by Sean Owen <sr...@gmail.com>.
Right, in that case you can pass any threshold less than 1. This will
cause the framework to pick the first "at" items it encounters as
relevant. Since there is no way of differentiating the items, this is
as good as anything. So if you are evaluating precision at 3, it will
pick some 3 items as relevant. Sounds like you already tried passing
Double.NEGATIVE_INFINITY, which is fine. If it is still not giving
results, my other guesses still stand -- low evaluation percentage?
too few preferences per user? I think you need to go in with a
debugger; it's hard to guess more from here. I can tell you for what
it's worth that I am using the same evaluator (in the main Java code
of course) to evaluate results in a similar data set and it appears to
work fine for me. There could be some issue in the port, or perhaps
there was a bug fix since the port happened? I don't remember a bug
fix here in a while but could be missing something.
On Thu, Aug 27, 2009 at 10:42 AM, Claudia Grieco<gr...@crmpa.unisa.it> wrote:
> But since I'm using boolean preference data, all the preferences are relevant (score 1) so they are all erased from the training data:
>
> for (Preference pref : prefs) {
> if (pref.getValue() >= theRelevanceThreshold) {
> relevantItems.add(pref.getItem());
> }
> }
>
> -----Messaggio originale-----
> Da: Sean Owen [mailto:srowen@gmail.com]
> Inviato: giovedì 27 agosto 2009 11.29
> A: mahout-user@lucene.apache.org
> Oggetto: Re: R: R: R: Problems with evaluator.
>
> Yes, that is correct. The framework splits the user's preferences into
> "relevant" and "not relevant" items. It then takes away the relevant
> items, and leaves the non-relevant items in the training data. Then,
> it sees how many of those relevant items are recommended back to the
> user, to compute precision and recall.
>
> On Thu, Aug 27, 2009 at 10:24 AM, Claudia Grieco<gr...@crmpa.unisa.it> wrote:
>> Notice this part:
>> for (Preference pref : prefs2) {
>> if (!relevantItems.contains(pref.getItem())) {
>> trainingPrefs.add(pref);
>> }}
>> It adds a preference only if it's NOT in the relevant items
>
>
R: R: R: R: Problems with evaluator.
Posted by Claudia Grieco <gr...@crmpa.unisa.it>.
Oh, I see, I was using even an older version of the mahout code, and this problem had been fixed after. I had downloaded the before-150 code but forgot to substitute the libraries
-----Messaggio originale-----
Da: Claudia Grieco [mailto:grieco@crmpa.unisa.it]
Inviato: giovedì 27 agosto 2009 11.42
A: mahout-user@lucene.apache.org
Oggetto: R: R: R: R: Problems with evaluator.
But since I'm using boolean preference data, all the preferences are relevant (score 1) so they are all erased from the training data:
for (Preference pref : prefs) {
if (pref.getValue() >= theRelevanceThreshold) {
relevantItems.add(pref.getItem());
}
}
-----Messaggio originale-----
Da: Sean Owen [mailto:srowen@gmail.com]
Inviato: giovedì 27 agosto 2009 11.29
A: mahout-user@lucene.apache.org
Oggetto: Re: R: R: R: Problems with evaluator.
Yes, that is correct. The framework splits the user's preferences into
"relevant" and "not relevant" items. It then takes away the relevant
items, and leaves the non-relevant items in the training data. Then,
it sees how many of those relevant items are recommended back to the
user, to compute precision and recall.
On Thu, Aug 27, 2009 at 10:24 AM, Claudia Grieco<gr...@crmpa.unisa.it> wrote:
> Notice this part:
> for (Preference pref : prefs2) {
> if (!relevantItems.contains(pref.getItem())) {
> trainingPrefs.add(pref);
> }}
> It adds a preference only if it's NOT in the relevant items
R: R: R: R: Problems with evaluator.
Posted by Claudia Grieco <gr...@crmpa.unisa.it>.
But since I'm using boolean preference data, all the preferences are relevant (score 1) so they are all erased from the training data:
for (Preference pref : prefs) {
if (pref.getValue() >= theRelevanceThreshold) {
relevantItems.add(pref.getItem());
}
}
-----Messaggio originale-----
Da: Sean Owen [mailto:srowen@gmail.com]
Inviato: giovedì 27 agosto 2009 11.29
A: mahout-user@lucene.apache.org
Oggetto: Re: R: R: R: Problems with evaluator.
Yes, that is correct. The framework splits the user's preferences into
"relevant" and "not relevant" items. It then takes away the relevant
items, and leaves the non-relevant items in the training data. Then,
it sees how many of those relevant items are recommended back to the
user, to compute precision and recall.
On Thu, Aug 27, 2009 at 10:24 AM, Claudia Grieco<gr...@crmpa.unisa.it> wrote:
> Notice this part:
> for (Preference pref : prefs2) {
> if (!relevantItems.contains(pref.getItem())) {
> trainingPrefs.add(pref);
> }}
> It adds a preference only if it's NOT in the relevant items
Re: R: R: R: Problems with evaluator.
Posted by Sean Owen <sr...@gmail.com>.
Yes, that is correct. The framework splits the user's preferences into
"relevant" and "not relevant" items. It then takes away the relevant
items, and leaves the non-relevant items in the training data. Then,
it sees how many of those relevant items are recommended back to the
user, to compute precision and recall.
On Thu, Aug 27, 2009 at 10:24 AM, Claudia Grieco<gr...@crmpa.unisa.it> wrote:
> Notice this part:
> for (Preference pref : prefs2) {
> if (!relevantItems.contains(pref.getItem())) {
> trainingPrefs.add(pref);
> }}
> It adds a preference only if it's NOT in the relevant items
R: R: R: Problems with evaluator.
Posted by Claudia Grieco <gr...@crmpa.unisa.it>.
Do you think this could be the cause?
private void processOtherUser(Object id,
Collection<Item> relevantItems,
Collection<User> trainingUsers,
User user2) {
if (id.equals(user2.getID())) {
List<Preference> trainingPrefs = new ArrayList<Preference>();
Preference[] prefs2 = user2.getPreferencesAsArray();
for (Preference pref : prefs2) {
if (!relevantItems.contains(pref.getItem())) {
trainingPrefs.add(pref);
}
}
if (!trainingPrefs.isEmpty()) {
User trainingUser = new GenericUser<String>(id.toString(), trainingPrefs);
trainingUsers.add(trainingUser);
}
} else {
trainingUsers.add(user2);
}
}
Notice this part:
for (Preference pref : prefs2) {
if (!relevantItems.contains(pref.getItem())) {
trainingPrefs.add(pref);
}}
It adds a preference only if it's NOT in the relevant items
-----Messaggio originale-----
Da: Sean Owen [mailto:srowen@gmail.com]
Inviato: giovedì 27 agosto 2009 10.24
A: mahout-user@lucene.apache.org
Oggetto: Re: R: R: Problems with evaluator.
Good, and again have you evaluated the two other possible causes I
mentioned? The evaluator is basically saying it cannot evaluate given the
amount of data or settings available. Beyond this I think you need a quick
debug session to see what is happening. It should be clear because you will
find every evaluation attempt is aborting for some clear reason. I don't
have any more smart guesses from here. It could possibly be a problem with
the port? But I kind of doubt that.
On Aug 27, 2009 9:14 AM, "Claudia Grieco" <gr...@crmpa.unisa.it> wrote:
I tried letting the library pick the threshold. It picked 1 and the final
results were the same. I even tried setting the threshold to negative
infinity, with no success.
-----Messaggio originale-----
Da: Sean Owen [mailto:srowen@gmail.com]
Inviato: mercoledì 26 agosto 2009 16.57
A: mahout-user@lucene.apache.org
Oggetto: Re: R: Problems with evaluator.
It makes sense that the recommender is not called. That would be
caused by some of the reasons I listed below. The wrong relevance
threshold would indeed cause the recommender to not be called. If
there are no relevant items for a user, it can't really proceed at
all. Have you tried setting the threshold to NaN to let the library
pick? or the other items I mentioned. You really might need a debugger
to look inside and see what the real reason is.
On Wed, Aug 26, 2009 at 3:48 PM, Claudia Grieco<gr...@crmpa.unisa.it>
wrote:
> I lowered the precision @ number to 2, without results. I don't think it's
a problem of relevance threshold since it never calls my recommender, so it
can't compute any relevant or not relevant item. I'm using Boolean
preference data (an item was bought or not bought by a customer)
Re: R: R: Problems with evaluator.
Posted by Sean Owen <sr...@gmail.com>.
Good, and again have you evaluated the two other possible causes I
mentioned? The evaluator is basically saying it cannot evaluate given the
amount of data or settings available. Beyond this I think you need a quick
debug session to see what is happening. It should be clear because you will
find every evaluation attempt is aborting for some clear reason. I don't
have any more smart guesses from here. It could possibly be a problem with
the port? But I kind of doubt that.
On Aug 27, 2009 9:14 AM, "Claudia Grieco" <gr...@crmpa.unisa.it> wrote:
I tried letting the library pick the threshold. It picked 1 and the final
results were the same. I even tried setting the threshold to negative
infinity, with no success.
-----Messaggio originale-----
Da: Sean Owen [mailto:srowen@gmail.com]
Inviato: mercoledì 26 agosto 2009 16.57
A: mahout-user@lucene.apache.org
Oggetto: Re: R: Problems with evaluator.
It makes sense that the recommender is not called. That would be
caused by some of the reasons I listed below. The wrong relevance
threshold would indeed cause the recommender to not be called. If
there are no relevant items for a user, it can't really proceed at
all. Have you tried setting the threshold to NaN to let the library
pick? or the other items I mentioned. You really might need a debugger
to look inside and see what the real reason is.
On Wed, Aug 26, 2009 at 3:48 PM, Claudia Grieco<gr...@crmpa.unisa.it>
wrote:
> I lowered the precision @ number to 2, without results. I don't think it's
a problem of relevance threshold since it never calls my recommender, so it
can't compute any relevant or not relevant item. I'm using Boolean
preference data (an item was bought or not bought by a customer)
R: R: Problems with evaluator.
Posted by Claudia Grieco <gr...@crmpa.unisa.it>.
I tried letting the library pick the threshold. It picked 1 and the final results were the same. I even tried setting the threshold to negative infinity, with no success.
-----Messaggio originale-----
Da: Sean Owen [mailto:srowen@gmail.com]
Inviato: mercoledì 26 agosto 2009 16.57
A: mahout-user@lucene.apache.org
Oggetto: Re: R: Problems with evaluator.
It makes sense that the recommender is not called. That would be
caused by some of the reasons I listed below. The wrong relevance
threshold would indeed cause the recommender to not be called. If
there are no relevant items for a user, it can't really proceed at
all. Have you tried setting the threshold to NaN to let the library
pick? or the other items I mentioned. You really might need a debugger
to look inside and see what the real reason is.
On Wed, Aug 26, 2009 at 3:48 PM, Claudia Grieco<gr...@crmpa.unisa.it> wrote:
> I lowered the precision @ number to 2, without results. I don't think it's a problem of relevance threshold since it never calls my recommender, so it can't compute any relevant or not relevant item. I'm using Boolean preference data (an item was bought or not bought by a customer)
Re: R: Problems with evaluator.
Posted by Sean Owen <sr...@gmail.com>.
It makes sense that the recommender is not called. That would be
caused by some of the reasons I listed below. The wrong relevance
threshold would indeed cause the recommender to not be called. If
there are no relevant items for a user, it can't really proceed at
all. Have you tried setting the threshold to NaN to let the library
pick? or the other items I mentioned. You really might need a debugger
to look inside and see what the real reason is.
On Wed, Aug 26, 2009 at 3:48 PM, Claudia Grieco<gr...@crmpa.unisa.it> wrote:
> I lowered the precision @ number to 2, without results. I don't think it's a problem of relevance threshold since it never calls my recommender, so it can't compute any relevant or not relevant item. I'm using Boolean preference data (an item was bought or not bought by a customer)
R: Problems with evaluator.
Posted by Claudia Grieco <gr...@crmpa.unisa.it>.
I lowered the precision @ number to 2, without results. I don't think it's a problem of relevance threshold since it never calls my recommender, so it can't compute any relevant or not relevant item. I'm using Boolean preference data (an item was bought or not bought by a customer)
-----Messaggio originale-----
Da: Sean Owen [mailto:srowen@gmail.com]
Inviato: mercoledì 26 agosto 2009 16.15
A: mahout-user@lucene.apache.org
Oggetto: Re: Problems with evaluator.
I can tell you what the symptom likely means, but yeah it is hard to
help debug a port of an older version of the code, remotely. It
basically means you didn't get enough information to compute precision
or recall.
- Is your evaluation percentage too low? Might increase it if it's
close to 0. If it's very low there's a chance you will filter out all
users and evaluate nothing.
- Do your users have few preferences? If you're computing "precision @
3" for instance then the evaluator will only bother considering
evaluating for users with at least 2*3 = 6 preferences. If it has
fewer preferences, the result is probably not meaningful.
- Maybe set a lower relevance threshold? If it's too high, and no
items are 'relevant', then it can't figure precision or recall, which
depend on distinguishing 'relevant' and 'non-relevant' results
On Wed, Aug 26, 2009 at 2:55 PM, Claudia Grieco<gr...@crmpa.unisa.it> wrote:
> Hi,
>
> I'm using a version of mahout prior to MAHOUT_150, since the changes after
> that version were too big.
>
> I'm having a problem with an evaluation class
> (GenericRecommenderIRStatsEvaluator): it always returns precision and recall
> as NaN values. I'm using a c# port of mahout so I can't run a detailed debug
> and understand what is wrong, but from some experiments I understood it
> never calls my recommender (it obtains data of each user, calculate their
> neighborhood and that's all).
>
> Here is some code I'm using:
>
> GenericRecommenderIRStatsEvaluator evaluator = new
> GenericRecommenderIRStatsEvaluator();
>
> MyDataSource source = new MyDataSource(); //a jdbc data source
>
> ConnectionPoolDataSource poolSource = new ConnectionPoolDataSource(source);
>
> DataModel model = new MyDataModel(poolSource, "iscrizioni", "studentID",
> "ofalID", "score");
>
>
>
> //builder is a RecommenderBuilder that returns my
> UserRecommender
>
> IRStatistics statistics = evaluator.evaluate(builder, model, null, 5,
> Double.NegativeInfinity, 0.5);
>
> string results = "Precision:" + statistics.getPrecision() + " Recall" +
> statistics.getRecall();
>
> Console.WriteLine("results:" + results);
>
>
>
>
>
> My recommender has always worked, so I don't think it's its fault.
>
> Thanks in advance
>
>
Re: Problems with evaluator.
Posted by Sean Owen <sr...@gmail.com>.
I can tell you what the symptom likely means, but yeah it is hard to
help debug a port of an older version of the code, remotely. It
basically means you didn't get enough information to compute precision
or recall.
- Is your evaluation percentage too low? Might increase it if it's
close to 0. If it's very low there's a chance you will filter out all
users and evaluate nothing.
- Do your users have few preferences? If you're computing "precision @
3" for instance then the evaluator will only bother considering
evaluating for users with at least 2*3 = 6 preferences. If it has
fewer preferences, the result is probably not meaningful.
- Maybe set a lower relevance threshold? If it's too high, and no
items are 'relevant', then it can't figure precision or recall, which
depend on distinguishing 'relevant' and 'non-relevant' results
On Wed, Aug 26, 2009 at 2:55 PM, Claudia Grieco<gr...@crmpa.unisa.it> wrote:
> Hi,
>
> I'm using a version of mahout prior to MAHOUT_150, since the changes after
> that version were too big.
>
> I'm having a problem with an evaluation class
> (GenericRecommenderIRStatsEvaluator): it always returns precision and recall
> as NaN values. I'm using a c# port of mahout so I can't run a detailed debug
> and understand what is wrong, but from some experiments I understood it
> never calls my recommender (it obtains data of each user, calculate their
> neighborhood and that's all).
>
> Here is some code I'm using:
>
> GenericRecommenderIRStatsEvaluator evaluator = new
> GenericRecommenderIRStatsEvaluator();
>
> MyDataSource source = new MyDataSource(); //a jdbc data source
>
> ConnectionPoolDataSource poolSource = new ConnectionPoolDataSource(source);
>
> DataModel model = new MyDataModel(poolSource, "iscrizioni", "studentID",
> "ofalID", "score");
>
>
>
> //builder is a RecommenderBuilder that returns my
> UserRecommender
>
> IRStatistics statistics = evaluator.evaluate(builder, model, null, 5,
> Double.NegativeInfinity, 0.5);
>
> string results = "Precision:" + statistics.getPrecision() + " Recall" +
> statistics.getRecall();
>
> Console.WriteLine("results:" + results);
>
>
>
>
>
> My recommender has always worked, so I don't think it's its fault.
>
> Thanks in advance
>
>