Using SVD evaluator

[ashish negi <as...@gmail.com>]

Dear All,



I have Boolean data (around 0.1 milion records) which consists 50 thousand
users. Data is too sparse.

As per my understanding to test for any recommendation SVD based will be
useful ( for sparse data).

I am trying to evaluate recommender using generic Boolean evaluator as
follows.



---------------------



public static void IRStatEval()

      {



            try{

                    DataModel model = new GenericBooleanPrefDataModel(

                                GenericBooleanPrefDataModel.toDataMap(

                                  new FileDataModel(new
File("preference.txt"))));



             RecommenderBuilder recommenderBuilder = new
RecommenderBuilder() {

                 // @Override

                  public Recommender buildRecommender(DataModel model)
throws TasteException {



                    return new SVDRecommender(model, new
ALSWRFactorizer(model, 10, 0.3,5));//0.05, 10

                  }

                };



                DataModelBuilder modelBuilder = new DataModelBuilder() {

                    // @Override

                     public DataModel
buildDataModel(FastByIDMap<PreferenceArray> trainingData) {

                       return new GenericBooleanPrefDataModel(


GenericBooleanPrefDataModel.toDataMap(trainingData));

                     }

                   };



                   RecommenderIRStatsEvaluator evaluator =

                        new GenericRecommenderIRStatsEvaluator();



                   IRStatistics stats = evaluator.evaluate(

                                recommenderBuilder, modelBuilder, model,
null, 2,


GenericRecommenderIRStatsEvaluator.CHOOSE_THRESHOLD,

                                1.0);





                  System.out.println(stats.getPrecision());

                     System.out.println(stats.getRecall());



            }catch(Exception e)

            {

                  e.printStackTrace();

            }

      }

--------------



Is this right way to do things?

Is there any reference or Test file of mahout source where I can get more
pointers.

Above all if I run this code it is taking too much time while evaluating
as it is calculating for each user.

Is there any way to limit evaluation ?



Regards,

Ashish

Re: Using SVD evaluator

[Ted Dunning <te...@gmail.com>]

If you understand the domain, you can inspect things by hand.  That is only
of limited utility.

You can also test click through if your data comes from a live system and
you have set up to do testing.

(always set up to do testing!)

On Fri, Dec 14, 2012 at 2:40 AM, Sean Owen <sr...@gmail.com> wrote:

> It's still true. There aren't much better tests you can run in this
> situation, since you are not evaluating predicted ratings, but
> attempting to assess how good the recommendations are when you don't
> have any nearly complete information on what the good recommendations
> are a priori.
>
>
> On Fri, Dec 14, 2012 at 10:37 AM, ashish negi
> <as...@gmail.com> wrote:
> > Thanks for quick reply and suggestions.
> > So Is there any measure to test whether SVD is performing on data or not?
> >
> > I read following lines from you book mahout in action
> > ---
> > In the case of Boolean preference data, only a precision-recall test is
> > available
> > anyway. It’s worth understanding the test’s limitations in this context.
> > ---
> >
> > Is it true now also (with new version of mahout)?
> > Regards,
> > Ashish
>

Re: Using SVD evaluator

[Sean Owen <sr...@gmail.com>]

It's still true. There aren't much better tests you can run in this
situation, since you are not evaluating predicted ratings, but
attempting to assess how good the recommendations are when you don't
have any nearly complete information on what the good recommendations
are a priori.


On Fri, Dec 14, 2012 at 10:37 AM, ashish negi
<as...@gmail.com> wrote:
> Thanks for quick reply and suggestions.
> So Is there any measure to test whether SVD is performing on data or not?
>
> I read following lines from you book mahout in action
> ---
> In the case of Boolean preference data, only a precision-recall test is
> available
> anyway. It’s worth understanding the test’s limitations in this context.
> ---
>
> Is it true now also (with new version of mahout)?
> Regards,
> Ashish

Re: Using SVD evaluator

[ashish negi <as...@gmail.com>]

Thanks for quick reply and suggestions.
So Is there any measure to test whether SVD is performing on data or not?

I read following lines from you book mahout in action
---
In the case of Boolean preference data, only a precision-recall test is
available
anyway. It’s worth understanding the test’s limitations in this context.
---

Is it true now also (with new version of mahout)?
Regards,
Ashish

On Fri, Dec 14, 2012 at 3:55 PM, Sean Owen <sr...@gmail.com> wrote:

> The "1.0" last parameter to evaluate() can be turned down to, say, 0.1
> to use only 10% of the data for testing.
>
> Precision/recall tests are always problematic for recommenders. I
> don't think this is going to give good results, but you're doing it
> right.
>

Re: Using SVD evaluator

[Sean Owen <sr...@gmail.com>]

The "1.0" last parameter to evaluate() can be turned down to, say, 0.1
to use only 10% of the data for testing.

Precision/recall tests are always problematic for recommenders. I
don't think this is going to give good results, but you're doing it
right.