You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Claudia Grieco <gr...@crmpa.unisa.it> on 2009/07/20 10:47:23 UTC

Implement the related search feature with mahout

Hi guys!

I'm trying to implement a "related search" feature using the mahout
libraries. The queries are used to retrieve a set of items memorized in a
DB.

I have come up with this implementation:

-treat queries as "Users" and items in the DB as "Items"

-for each query entered in the search engine I memorize the text of the
query and the first 10 items retrieved. (the user-item column contains the
query id, the item id and the relevance score of the item for the query)

-to compute searches related to the current search I use mahout's Tanimoto
similarity to find the most similar "users", i.e. the queries which have
more result items in common with the current query.

 

Is there a way to improve what I have done? I'd like to increase the
importance of a query according to its "popularity" (i.e. how many time the
query was entered) and/or keep trace of the most clicked items instead of
the first 10 items, but I can't figure out how to do it.

Any ideas?


Re: R: Implement the related search feature with mahout

Posted by Ted Dunning <te...@gmail.com>.
AOL.

http://www.gregsadetsky.com/aol-data/

Big scandal.

On Wed, Jul 22, 2009 at 12:59 AM, Sean Owen <sr...@gmail.com> wrote:

> query log data... but seem to vaguely recall Altavista (?) releasing
> something




-- 
Ted Dunning, CTO
DeepDyve

Re: R: Implement the related search feature with mahout

Posted by Sean Owen <sr...@gmail.com>.
To your other question, my favorite large data sets are Netflix and
GroupLens. They are movie rating data, not query logs. I don't know of query
log data... but seem to vaguely recall Altavista (?) releasing something
like this years ago?

On Jul 22, 2009 8:41 AM, "Claudia Grieco" <gr...@crmpa.unisa.it> wrote:

I had a sudden idea: would it be better to use an item based recommender?
(with TanimotoCoefficientSimilarity instead of
BooleanTanimotoCoefficientSimilarity that doesn't implement ItemSimilarity)
In this way I could overcome the scalability problem of having too many
"users" (queries)

Another question: do you know of any example query log data I could use to
experiment how the algorithm performs for large data sets?
Thanks again
Claudia

-----Messaggio originale-----
Da: Sean Owen [mailto:srowen@gmail.com]
Inviato: lunedì 20 luglio 2009 13.19

A: mahout-user@lucene.apache.org
Oggetto: Re: Implement the related search feature with mahout

Actually I kind of misspoke. The fastest way to do this is in fact to

use a GenericUserBasedRecommender -- not because you need recommendations,
but because it exposes a ...
DataModel model = new MySQLBooleanPrefJDBCDataModel(...); // or
whatever you are using
UserSimilarity similarity = BooleanTanimotoCoefficientSimilarity(model);
UserNeighborhood similarity = new NearestNUserNeighborhood(10,
similarity, model);
BooleanUserGenericUserBasedRecommender recommender = new
BooleanUserGenericUserBasedRecommender(model, neighborhood,
similarity);
Rescorer<Pair<User,User>> rescorer = new Rescorer<Pair<User,User>>() {
 // implement your rescoring logic to affect how similar 'users' are
-- boost the returned value for popular queries
};
Object userID = ...; // current query ID
List<User> similarUsers = recommender.mostSimilarUsers(userID, 10,
rescorer);

Re: R: Implement the related search feature with mahout

Posted by Sean Owen <sr...@gmail.com>.
What advantage are you expecting from an item-based recommender? It sounds
like you do not care about recommendations per se. An item-based recommender
won't be able to figure most similar users, which is your primary
requirement it seems. You have also identified a drawback, which is being
unable to use a faster similarity metric.

On Jul 22, 2009 8:41 AM, "Claudia Grieco" <gr...@crmpa.unisa.it> wrote:

I had a sudden idea: would it be better to use an item based recommender?
(with TanimotoCoefficientSimilarity instead of
BooleanTanimotoCoefficientSimilarity that doesn't implement ItemSimilarity)
In this way I could overcome the scalability problem of having too many
"users" (queries)

Another question: do you know of any example query log data I could use to
experiment how the algorithm performs for large data sets?
Thanks again
Claudia

-----Messaggio originale-----
Da: Sean Owen [mailto:srowen@gmail.com]
Inviato: lunedì 20 luglio 2009 13.19

A: mahout-user@lucene.apache.org
Oggetto: Re: Implement the related search feature with mahout

Actually I kind of misspoke. The fastest way to do this is in fact to

use a GenericUserBasedRecommender -- not because you need recommendations,
but because it exposes a ...
DataModel model = new MySQLBooleanPrefJDBCDataModel(...); // or
whatever you are using
UserSimilarity similarity = BooleanTanimotoCoefficientSimilarity(model);
UserNeighborhood similarity = new NearestNUserNeighborhood(10,
similarity, model);
BooleanUserGenericUserBasedRecommender recommender = new
BooleanUserGenericUserBasedRecommender(model, neighborhood,
similarity);
Rescorer<Pair<User,User>> rescorer = new Rescorer<Pair<User,User>>() {
 // implement your rescoring logic to affect how similar 'users' are
-- boost the returned value for popular queries
};
Object userID = ...; // current query ID
List<User> similarUsers = recommender.mostSimilarUsers(userID, 10,
rescorer);

R: Implement the related search feature with mahout

Posted by Claudia Grieco <gr...@crmpa.unisa.it>.
I had a sudden idea: would it be better to use an item based recommender? (with TanimotoCoefficientSimilarity instead of BooleanTanimotoCoefficientSimilarity that doesn't implement ItemSimilarity)
In this way I could overcome the scalability problem of having too many "users" (queries)

Another question: do you know of any example query log data I could use to experiment how the algorithm performs for large data sets?
Thanks again
Claudia

-----Messaggio originale-----
Da: Sean Owen [mailto:srowen@gmail.com] 
Inviato: lunedì 20 luglio 2009 13.19
A: mahout-user@lucene.apache.org
Oggetto: Re: Implement the related search feature with mahout

Actually I kind of misspoke. The fastest way to do this is in fact to
use a GenericUserBasedRecommender -- not because you need
recommendations, but because it exposes a nice mostSimlarUsers()
method.

DataModel model = new MySQLBooleanPrefJDBCDataModel(...); // or
whatever you are using
UserSimilarity similarity = BooleanTanimotoCoefficientSimilarity(model);
UserNeighborhood similarity = new NearestNUserNeighborhood(10,
similarity, model);
BooleanUserGenericUserBasedRecommender recommender = new
BooleanUserGenericUserBasedRecommender(model, neighborhood,
similarity);
Rescorer<Pair<User,User>> rescorer = new Rescorer<Pair<User,User>>() {
  // implement your rescoring logic to affect how similar 'users' are
-- boost the returned value for popular queries
};
Object userID = ...; // current query ID
List<User> similarUsers = recommender.mostSimilarUsers(userID, 10, rescorer);




Re: R: Implement the related search feature with mahout

Posted by Sean Owen <sr...@gmail.com>.
(You can just re-use the similarity metric you fed in in the first
instance, to re-compute that figure. You could also use the Rescorer
to do this type of filtering. But, yeah, point taken, could be
useful.)

On Mon, Jul 20, 2009 at 3:54 PM, Otis
Gospodnetic<ot...@yahoo.com> wrote:
> I like Taste's mostSimilarUsers(), but I wish it exposed/returned the similarity metric, so one can use this method and see exactly how similar those most similar users are.  For example, an application may want to show or hide similar users based on this similarity "distance" from the given user.

R: R: Implement the related search feature with mahout

Posted by Claudia Grieco <gr...@crmpa.unisa.it>.
Do you think that the similar users approach of Lucene is similar to the
"most similar users" of mahout?

-----Messaggio originale-----
Da: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com] 
Inviato: lunedì 20 luglio 2009 16.54
A: mahout-user@lucene.apache.org
Oggetto: Re: R: Implement the related search feature with mahout


Hi,

Here is an example of Lucene MLT-based "similar users" functionality:
http://www.simpy.com/user/otis
(it's based on tags)
Of course, MLT can be tweaked with a couple of parameters, so it can be
adjusted depending on the need.

I like Taste's mostSimilarUsers(), but I wish it exposed/returned the
similarity metric, so one can use this method and see exactly how similar
those most similar users are.  For example, an application may want to show
or hide similar users based on this similarity "distance" from the given
user.

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch





Re: R: Implement the related search feature with mahout

Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi,

Here is an example of Lucene MLT-based "similar users" functionality: http://www.simpy.com/user/otis
(it's based on tags)
Of course, MLT can be tweaked with a couple of parameters, so it can be adjusted depending on the need.

I like Taste's mostSimilarUsers(), but I wish it exposed/returned the similarity metric, so one can use this method and see exactly how similar those most similar users are.  For example, an application may want to show or hide similar users based on this similarity "distance" from the given user.

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



----- Original Message ----
> From: Claudia Grieco <gr...@crmpa.unisa.it>
> To: mahout-user@lucene.apache.org
> Sent: Monday, July 20, 2009 8:15:26 AM
> Subject: R: Implement the related search feature with mahout
> 
> Thanks for your reply
> > The fastest way to do this is in fact to
> use a GenericUserBasedRecommender -- not because you need
> recommendations, but because it exposes a nice mostSimlarUsers()
> method.
> 
> Yes, that's the approach I was using. I was using almost the same code but with 
> GenericUserBasedRecommender instead of the Boolean version, so thanks for the 
> tip ;)
> 
> 
> > First, Lucene itself has related-search functionality.
> I have tried to look at it before, but I have only found the MoreLikeThis 
> functionality, which is useful to obtain more similar *results*, I think.


R: Implement the related search feature with mahout

Posted by Claudia Grieco <gr...@crmpa.unisa.it>.
Thanks for your reply
> The fastest way to do this is in fact to
use a GenericUserBasedRecommender -- not because you need
recommendations, but because it exposes a nice mostSimlarUsers()
method.

Yes, that's the approach I was using. I was using almost the same code but with GenericUserBasedRecommender instead of the Boolean version, so thanks for the tip ;)


> First, Lucene itself has related-search functionality.
I have tried to look at it before, but I have only found the MoreLikeThis functionality, which is useful to obtain more similar *results*, I think.




Re: Implement the related search feature with mahout

Posted by Sean Owen <sr...@gmail.com>.
Actually I kind of misspoke. The fastest way to do this is in fact to
use a GenericUserBasedRecommender -- not because you need
recommendations, but because it exposes a nice mostSimlarUsers()
method.

DataModel model = new MySQLBooleanPrefJDBCDataModel(...); // or
whatever you are using
UserSimilarity similarity = BooleanTanimotoCoefficientSimilarity(model);
UserNeighborhood similarity = new NearestNUserNeighborhood(10,
similarity, model);
BooleanUserGenericUserBasedRecommender recommender = new
BooleanUserGenericUserBasedRecommender(model, neighborhood,
similarity);
Rescorer<Pair<User,User>> rescorer = new Rescorer<Pair<User,User>>() {
  // implement your rescoring logic to affect how similar 'users' are
-- boost the returned value for popular queries
};
Object userID = ...; // current query ID
List<User> similarUsers = recommender.mostSimilarUsers(userID, 10, rescorer);

On Mon, Jul 20, 2009 at 11:18 AM, Sean Owen<sr...@gmail.com> wrote:
> First, Lucene itself has related-search functionality. If you are using it,
> I suspect it will be better to leverage that. (Even if not it is possible to
> run Lucene just for this purpose.) Others much more familiar with Lucene can
> comment.
>
> I can comment on your current approach. Yes, it seems reasonable. You are
> effectively using just a piece of a user-based recommender, and that is the
> UserNeighborhood component. This is all you need, such as
> NearestNUserNeighborhood and your similarity metric.
>
> Scalability could be an issue, since you have a 'user' for every distinct
> query. Consider normalizing queries a lot - decapitalize, remove whitespace,
> sort by query term, keep only first n terms, etc.
>
> Also for this reason, consider using
> BooleanUserTanimotoCoefficientSimilarity and BooleanPrefUserDatModel (off
> the top of my hrad even I am not sure I got those names right!)  Because you
> do not have a reliable notion of strength of preference, you should ignore
> preference value, and these implementations let you take advantage of this.
>
> Please use the very latest code from Subversion (Mahout 0.2 SNAPSHOT). I am
> right now working with a client using these implementations and have been
> fixing and optimizing them a lot recently.
>
> Finally, you want to boost queries that are popular. You can use a Rescorer
> for this to inject any score changes you like. Perhaps you devise some
> function that increases the similarity towards 1 the more the two queries
> are observed.
>
> Let us start there and follow up with questions here.

Re: Implement the related search feature with mahout

Posted by Sean Owen <sr...@gmail.com>.
First, Lucene itself has related-search functionality. If you are using it,
I suspect it will be better to leverage that. (Even if not it is possible to
run Lucene just for this purpose.) Others much more familiar with Lucene can
comment.

I can comment on your current approach. Yes, it seems reasonable. You are
effectively using just a piece of a user-based recommender, and that is the
UserNeighborhood component. This is all you need, such as
NearestNUserNeighborhood and your similarity metric.

Scalability could be an issue, since you have a 'user' for every distinct
query. Consider normalizing queries a lot - decapitalize, remove whitespace,
sort by query term, keep only first n terms, etc.

Also for this reason, consider using
BooleanUserTanimotoCoefficientSimilarity and BooleanPrefUserDatModel (off
the top of my hrad even I am not sure I got those names right!)  Because you
do not have a reliable notion of strength of preference, you should ignore
preference value, and these implementations let you take advantage of this.

Please use the very latest code from Subversion (Mahout 0.2 SNAPSHOT). I am
right now working with a client using these implementations and have been
fixing and optimizing them a lot recently.

Finally, you want to boost queries that are popular. You can use a Rescorer
for this to inject any score changes you like. Perhaps you devise some
function that increases the similarity towards 1 the more the two queries
are observed.

Let us start there and follow up with questions here.

On Jul 20, 2009 9:50 AM, "Claudia Grieco" <gr...@crmpa.unisa.it> wrote:

Hi guys!

I'm trying to implement a "related search" feature using the mahout
libraries. The queries are used to retrieve a set of items memorized in a
DB.

I have come up with this implementation:

-treat queries as "Users" and items in the DB as "Items"

-for each query entered in the search engine I memorize the text of the
query and the first 10 items retrieved. (the user-item column contains the
query id, the item id and the relevance score of the item for the query)

-to compute searches related to the current search I use mahout's Tanimoto
similarity to find the most similar "users", i.e. the queries which have
more result items in common with the current query.



Is there a way to improve what I have done? I'd like to increase the
importance of a query according to its "popularity" (i.e. how many time the
query was entered) and/or keep trace of the most clicked items instead of
the first 10 items, but I can't figure out how to do it.

Any ideas?