You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Steven Bourke <st...@ucd.ie> on 2010/08/16 14:39:35 UTC
Boolean Recommender evaluator returning score greater than 1
Hi,
I've removed all preference values from a dataset and plugged it into a boolean recommendation configuration. When I run RMSRecommenderEvaluator on the dataset I get values back which are higher than 1 (But lower than 2). I assumed that the highest value that should come back from the evaluator is a one. If my reasoning is correct, can anyone recommend what could be throwing the incorrect results?
Thanks
Re: Boolean Recommender evaluator returning score greater than 1
Posted by Sean Owen <sr...@gmail.com>.
Yes, that's what this IR test does. It leaves out several items and
sees how many are recommended back.
On Mon, Aug 16, 2010 at 2:46 PM, Steven Bourke <st...@ucd.ie> wrote:
> Ah thanks -
>
> Is there any form of 'leave one out' in the mahout implementation? Precision and recall are obviously quite useful, but I'd like to get some form of overall picture for the recommendations.
>
Re: Boolean Recommender evaluator returning score greater than 1
Posted by Steven Bourke <st...@ucd.ie>.
Ah thanks -
Is there any form of 'leave one out' in the mahout implementation? Precision and recall are obviously quite useful, but I'd like to get some form of overall picture for the recommendations.
On 16 Aug 2010, at 14:35, Sean Owen wrote:
> RMS is root-mean-square error, which can be arbitrarily large. So, no
> it's not wrong for it to be above 1.
>
> But for boolean data, the evaluation doesn't make sense. You can only
> use simple IR stats eval -- precision and recall. Those should not be
> more than 1 as they are percentages.
>
> On Mon, Aug 16, 2010 at 1:39 PM, Steven Bourke <st...@ucd.ie> wrote:
>> Hi,
>>
>> I've removed all preference values from a dataset and plugged it into a boolean recommendation configuration. When I run RMSRecommenderEvaluator on the dataset I get values back which are higher than 1 (But lower than 2). I assumed that the highest value that should come back from the evaluator is a one. If my reasoning is correct, can anyone recommend what could be throwing the incorrect results?
>>
>> Thanks
>>
>>
>>
Re: Boolean Recommender evaluator returning score greater than 1
Posted by Sean Owen <sr...@gmail.com>.
RMS is root-mean-square error, which can be arbitrarily large. So, no
it's not wrong for it to be above 1.
But for boolean data, the evaluation doesn't make sense. You can only
use simple IR stats eval -- precision and recall. Those should not be
more than 1 as they are percentages.
On Mon, Aug 16, 2010 at 1:39 PM, Steven Bourke <st...@ucd.ie> wrote:
> Hi,
>
> I've removed all preference values from a dataset and plugged it into a boolean recommendation configuration. When I run RMSRecommenderEvaluator on the dataset I get values back which are higher than 1 (But lower than 2). I assumed that the highest value that should come back from the evaluator is a one. If my reasoning is correct, can anyone recommend what could be throwing the incorrect results?
>
> Thanks
>
>
>