Custome metrics in Mahout Recommendation

[Vikas Kumar <ku...@umn.edu>]

I am implementing recommendation techniques in Mahout. However, I have a
requirement for a custom evaluation metrics other than predefined or
built-in ones. So,

Q: Can someone please point me to sample custom Evaluator or metric
implementation in Mahout?


Thanks
Vikas

Re: Custome metrics in Mahout Recommendation

[Pat Ferrel <pa...@occamsmachete.com>]

MAP is what I use, not precision. Mean average precision accounts for ranking better and ranking is usually what you want to optimize. As I said none are in the current iteration of the recommender code including RMSE, etc. http://mahout.apache.org/users/recommender/intro-cooccurrence-spark.html

What use did you have for offline metrics? No offline metric is as good as A/B testing. Often things built in to recommenders will lower one metric or another but produce better results in the form of user engagement or sales. Metrics must be used with great caution and should never be relied on in a real world situation where user testing is available.

On Mar 2, 2015, at 5:04 PM, Vikas Kumar <ku...@umn.edu> wrote:

Sorry for the confusion. Yes, I meant the recommender evaluation metric
such as RMSE, Precision, Recall etc which are inbuilt. But, I am planning
(or reusing - let me know if already done) to write the metrics such as
nDCG, Popularity, Avg. Rating, diversity etc.

Thanks
Vikas

On Mon, Mar 2, 2015 at 6:46 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:

> Evaluation metric? You mean like the old recommender evaluator? I’d use
> MAP mean average precision, but none are implemented in the new Spark
> recommender code.
> 
> On Mar 2, 2015, at 3:12 PM, Vikas Kumar <ku...@umn.edu> wrote:
> 
> I am implementing recommendation techniques in Mahout. However, I have a
> requirement for a custom evaluation metrics other than predefined or
> built-in ones. So,
> 
> Q: Can someone please point me to sample custom Evaluator or metric
> implementation in Mahout?
> 
> 
> Thanks
> Vikas
> 
> 

Re: Custome metrics in Mahout Recommendation

[Vikas Kumar <ku...@umn.edu>]

Sorry for the confusion. Yes, I meant the recommender evaluation metric
such as RMSE, Precision, Recall etc which are inbuilt. But, I am planning
(or reusing - let me know if already done) to write the metrics such as
nDCG, Popularity, Avg. Rating, diversity etc.

Thanks
Vikas

On Mon, Mar 2, 2015 at 6:46 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:

> Evaluation metric? You mean like the old recommender evaluator? I’d use
> MAP mean average precision, but none are implemented in the new Spark
> recommender code.
>
> On Mar 2, 2015, at 3:12 PM, Vikas Kumar <ku...@umn.edu> wrote:
>
> I am implementing recommendation techniques in Mahout. However, I have a
> requirement for a custom evaluation metrics other than predefined or
> built-in ones. So,
>
> Q: Can someone please point me to sample custom Evaluator or metric
> implementation in Mahout?
>
>
> Thanks
> Vikas
>
>

Re: Custome metrics in Mahout Recommendation

[Pat Ferrel <pa...@occamsmachete.com>]

Evaluation metric? You mean like the old recommender evaluator? I’d use MAP mean average precision, but none are implemented in the new Spark recommender code. 

On Mar 2, 2015, at 3:12 PM, Vikas Kumar <ku...@umn.edu> wrote:

I am implementing recommendation techniques in Mahout. However, I have a
requirement for a custom evaluation metrics other than predefined or
built-in ones. So,

Q: Can someone please point me to sample custom Evaluator or metric
implementation in Mahout?


Thanks
Vikas

Re: Custome metrics in Mahout Recommendation

[Andrew Musselman <an...@gmail.com>]

If you check out the latest master branch from
https://github.com/apache/mahout you'll find classes like this in the
map-reduce legacy
package:  /home/akm/mahout/mrlegacy/src/main/java/org/apache/mahout/common/distance/CosineDistanceMeasure.java

I'm not sure if/where new ones are being written..

On Mon, Mar 2, 2015 at 3:12 PM, Vikas Kumar <ku...@umn.edu> wrote:

> I am implementing recommendation techniques in Mahout. However, I have a
> requirement for a custom evaluation metrics other than predefined or
> built-in ones. So,
>
> Q: Can someone please point me to sample custom Evaluator or metric
> implementation in Mahout?
>
>
> Thanks
> Vikas
>