You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Rih <ta...@gmail.com> on 2010/05/20 08:43:45 UTC

Personalized Search

Has anybody done personalized search with Solr? I'm thinking of including
fields such as "bought" or "like" per member/visitor via dynamic fields to a
product search schema. Another option is to have a multi-value field that
can contain user IDs. What are the possible performance issues with this
setup?

Looking forward to your ideas.

Rih

Re: Personalized Search

Posted by Rih <ta...@gmail.com>.
It will likely be what you suggested, one or two multi value fields. But
with 10,000+ members, does Solr scaled with this schema?


On Thu, May 20, 2010 at 6:27 PM, findbestopensource <
findbestopensource@gmail.com> wrote:

> Hi Rih,
>
> You going to include either of the two field "bought" or "like" to per
> member/visitor OR a unique field per member / visitor?
>
> If it's one or two common fields are included then there will not be any
> impact in performance. If you want to include unique field then you need to
> consider multi value field otherwise you certainly hit the wall.
>
> Regards
> Aditya
> www.findbestopensource.com
>
>
>
>
> On Thu, May 20, 2010 at 12:13 PM, Rih <ta...@gmail.com> wrote:
>
> > Has anybody done personalized search with Solr? I'm thinking of including
> > fields such as "bought" or "like" per member/visitor via dynamic fields
> to
> > a
> > product search schema. Another option is to have a multi-value field that
> > can contain user IDs. What are the possible performance issues with this
> > setup?
> >
> > Looking forward to your ideas.
> >
> > Rih
> >
>

Re: Personalized Search

Posted by dc tech <dc...@gmail.com>.
In our specific case, we would get the user's folders and then do a
function query that provides a boost if the document.folder is in {my
folder list}.

Another approach that will work for our intranet use is to add the
userids in a multi-valued field as others have suggested.



On 5/20/10, MitchK <mi...@web.de> wrote:
>
> Hi dc,
>
>
>
>> - at query time, specify boosts for 'my items' items
>>
> Do you mean something like document-boost or do you want to include
> something like
> "OR myItemId:100^100"
> ?
>
> Can you tell us how you would specify document-boostings at query-time? Or
> are you querying something like a boolean field (i.e. isFavorite:true^10) or
> a numeric field?
>
> Kind regards
> - Mitch
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Personalized-Search-tp831070p832062.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

-- 
Sent from my mobile device

Re: Personalized Search

Posted by MitchK <mi...@web.de>.
Hi dc,



> - at query time, specify boosts for 'my items' items 
> 
Do you mean something like document-boost or do you want to include
something like
"OR myItemId:100^100"
?

Can you tell us how you would specify document-boostings at query-time? Or
are you querying something like a boolean field (i.e. isFavorite:true^10) or
a numeric field?

Kind regards
- Mitch
-- 
View this message in context: http://lucene.472066.n3.nabble.com/Personalized-Search-tp831070p832062.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Personalized Search

Posted by dc tech <dc...@gmail.com>.
Excluding favorited items is an easier problem
- get the results
- get exclude list from db
- scan results and exclude the items in the item list

You'd have to do some code to manage 'holes' in the result list ie
fetch more etc.

You could marry this with the solr batch based approach to reduce the holes :
- Every night, update the item.users field. This can be simple string
type of  field.
- query with negative criteria ie
   content:search_term AND -users:userid
- then do the steps outlined earlier

On 5/21/10, Rih <ta...@gmail.com> wrote:
>>
>> - keep the SOLR index independent of bought/like
>
> - have a db table with user prefs on a per item basis
>
>
> I have the same idea this far.
>
> at query time, specify boosts for 'my items' items
>
>
> I believe this works if you want to sort results by faved/not faved. But how
> does it scale if users already favorited/liked hundreds of item? The query
> can be quite long.
>
> Looking forward to your idea.
>
>
>
> On Thu, May 20, 2010 at 6:37 PM, dc tech <dc...@gmail.com> wrote:
>
>> Another approach would be to do query time boosts of 'my' items under
>> the assumption that count is limited:
>> - keep the SOLR index independent of bought/like
>> - have a db table with user prefs on a per item basis
>> - at query time, specify boosts for 'my items' items
>>
>> We are planning to do this in the context of document management where
>> documents in 'my (used/favorited.... ) folders' provide a boost factor
>> to the results.
>>
>>
>>
>> On 5/20/10, findbestopensource <fi...@gmail.com> wrote:
>> > Hi Rih,
>> >
>> > You going to include either of the two field "bought" or "like" to per
>> > member/visitor OR a unique field per member / visitor?
>> >
>> > If it's one or two common fields are included then there will not be any
>> > impact in performance. If you want to include unique field then you need
>> to
>> > consider multi value field otherwise you certainly hit the wall.
>> >
>> > Regards
>> > Aditya
>> > www.findbestopensource.com
>> >
>> >
>> >
>> >
>> > On Thu, May 20, 2010 at 12:13 PM, Rih <ta...@gmail.com> wrote:
>> >
>> >> Has anybody done personalized search with Solr? I'm thinking of
>> including
>> >> fields such as "bought" or "like" per member/visitor via dynamic fields
>> to
>> >> a
>> >> product search schema. Another option is to have a multi-value field
>> that
>> >> can contain user IDs. What are the possible performance issues with
>> >> this
>> >> setup?
>> >>
>> >> Looking forward to your ideas.
>> >>
>> >> Rih
>> >>
>> >
>>
>> --
>> Sent from my mobile device
>>
>

-- 
Sent from my mobile device

Re: Personalized Search

Posted by Geert-Jan Brits <gb...@gmail.com>.
Just want to throw this in: If you're worried about scaling, etc. you could
take a look at item-based collaborative filtering instead of user based.
i.e:
DO NIGHTLY/ BATCH:
- calculate the similarity between items based on their properties

DO ON EACH REQUEST
- have a user store/update it's interest as a vector of item-properties. How
to update this based on click / browse behavior is the interesting thing and
depends a lot on your environment.
- Next is to recommend 'neighboring' items that are close to the defined
'interest-vector'.

The code is similar to user-based colab. filtering, but scaling is invariant
to the nr of users.

other advantages:
- new items/ products can be recommended as soon as they are added to the
catalog (no need for users to express interest in them before the item can
be suggested)

disadvantage:
- top-N results tend to be less dynamic then when using user-based colab.
filtering.

Of course, this doesn't touch on how to integrate this with Solr. Perhaps
some combination with Mahout is indeed the best solution. I haven't given
this much thought yet I must say.
For info on Mahout Taste (+ an explanation on item-based filtering vs.
user-based filtering) see:
http://lucene.apache.org/mahout/taste.html

Cheers,
Geert-Jan

2010/5/21 Rih <ta...@gmail.com>

> >
> > - keep the SOLR index independent of bought/like
>
> - have a db table with user prefs on a per item basis
>
>
> I have the same idea this far.
>
> at query time, specify boosts for 'my items' items
>
>
> I believe this works if you want to sort results by faved/not faved. But
> how
> does it scale if users already favorited/liked hundreds of item? The query
> can be quite long.
>
> Looking forward to your idea.
>
>
>
> On Thu, May 20, 2010 at 6:37 PM, dc tech <dc...@gmail.com> wrote:
>
> > Another approach would be to do query time boosts of 'my' items under
> > the assumption that count is limited:
> > - keep the SOLR index independent of bought/like
> > - have a db table with user prefs on a per item basis
> > - at query time, specify boosts for 'my items' items
> >
> > We are planning to do this in the context of document management where
> > documents in 'my (used/favorited.... ) folders' provide a boost factor
> > to the results.
> >
> >
> >
> > On 5/20/10, findbestopensource <fi...@gmail.com> wrote:
> > > Hi Rih,
> > >
> > > You going to include either of the two field "bought" or "like" to per
> > > member/visitor OR a unique field per member / visitor?
> > >
> > > If it's one or two common fields are included then there will not be
> any
> > > impact in performance. If you want to include unique field then you
> need
> > to
> > > consider multi value field otherwise you certainly hit the wall.
> > >
> > > Regards
> > > Aditya
> > > www.findbestopensource.com
> > >
> > >
> > >
> > >
> > > On Thu, May 20, 2010 at 12:13 PM, Rih <ta...@gmail.com> wrote:
> > >
> > >> Has anybody done personalized search with Solr? I'm thinking of
> > including
> > >> fields such as "bought" or "like" per member/visitor via dynamic
> fields
> > to
> > >> a
> > >> product search schema. Another option is to have a multi-value field
> > that
> > >> can contain user IDs. What are the possible performance issues with
> this
> > >> setup?
> > >>
> > >> Looking forward to your ideas.
> > >>
> > >> Rih
> > >>
> > >
> >
> > --
> > Sent from my mobile device
> >
>

Re: Personalized Search

Posted by Rih <ta...@gmail.com>.
>
> - keep the SOLR index independent of bought/like

- have a db table with user prefs on a per item basis


I have the same idea this far.

at query time, specify boosts for 'my items' items


I believe this works if you want to sort results by faved/not faved. But how
does it scale if users already favorited/liked hundreds of item? The query
can be quite long.

Looking forward to your idea.



On Thu, May 20, 2010 at 6:37 PM, dc tech <dc...@gmail.com> wrote:

> Another approach would be to do query time boosts of 'my' items under
> the assumption that count is limited:
> - keep the SOLR index independent of bought/like
> - have a db table with user prefs on a per item basis
> - at query time, specify boosts for 'my items' items
>
> We are planning to do this in the context of document management where
> documents in 'my (used/favorited.... ) folders' provide a boost factor
> to the results.
>
>
>
> On 5/20/10, findbestopensource <fi...@gmail.com> wrote:
> > Hi Rih,
> >
> > You going to include either of the two field "bought" or "like" to per
> > member/visitor OR a unique field per member / visitor?
> >
> > If it's one or two common fields are included then there will not be any
> > impact in performance. If you want to include unique field then you need
> to
> > consider multi value field otherwise you certainly hit the wall.
> >
> > Regards
> > Aditya
> > www.findbestopensource.com
> >
> >
> >
> >
> > On Thu, May 20, 2010 at 12:13 PM, Rih <ta...@gmail.com> wrote:
> >
> >> Has anybody done personalized search with Solr? I'm thinking of
> including
> >> fields such as "bought" or "like" per member/visitor via dynamic fields
> to
> >> a
> >> product search schema. Another option is to have a multi-value field
> that
> >> can contain user IDs. What are the possible performance issues with this
> >> setup?
> >>
> >> Looking forward to your ideas.
> >>
> >> Rih
> >>
> >
>
> --
> Sent from my mobile device
>

Re: Personalized Search

Posted by dc tech <dc...@gmail.com>.
Another approach would be to do query time boosts of 'my' items under
the assumption that count is limited:
- keep the SOLR index independent of bought/like
- have a db table with user prefs on a per item basis
- at query time, specify boosts for 'my items' items

We are planning to do this in the context of document management where
documents in 'my (used/favorited.... ) folders' provide a boost factor
to the results.



On 5/20/10, findbestopensource <fi...@gmail.com> wrote:
> Hi Rih,
>
> You going to include either of the two field "bought" or "like" to per
> member/visitor OR a unique field per member / visitor?
>
> If it's one or two common fields are included then there will not be any
> impact in performance. If you want to include unique field then you need to
> consider multi value field otherwise you certainly hit the wall.
>
> Regards
> Aditya
> www.findbestopensource.com
>
>
>
>
> On Thu, May 20, 2010 at 12:13 PM, Rih <ta...@gmail.com> wrote:
>
>> Has anybody done personalized search with Solr? I'm thinking of including
>> fields such as "bought" or "like" per member/visitor via dynamic fields to
>> a
>> product search schema. Another option is to have a multi-value field that
>> can contain user IDs. What are the possible performance issues with this
>> setup?
>>
>> Looking forward to your ideas.
>>
>> Rih
>>
>

-- 
Sent from my mobile device

Re: Personalized Search

Posted by findbestopensource <fi...@gmail.com>.
Hi Rih,

You going to include either of the two field "bought" or "like" to per
member/visitor OR a unique field per member / visitor?

If it's one or two common fields are included then there will not be any
impact in performance. If you want to include unique field then you need to
consider multi value field otherwise you certainly hit the wall.

Regards
Aditya
www.findbestopensource.com




On Thu, May 20, 2010 at 12:13 PM, Rih <ta...@gmail.com> wrote:

> Has anybody done personalized search with Solr? I'm thinking of including
> fields such as "bought" or "like" per member/visitor via dynamic fields to
> a
> product search schema. Another option is to have a multi-value field that
> can contain user IDs. What are the possible performance issues with this
> setup?
>
> Looking forward to your ideas.
>
> Rih
>

Re: Personalized Search

Posted by MitchK <mi...@web.de>.
Hi RiH,

I think the idea is interesting, but the approach you think of is a little
bit difficult to implement.
Imagine you got 10.000 users, than a Solr document must actually have 10.000
fields which are responsible to managing the whole stuff. Furthermore your
data will change quite often. So the probability of reindexing documents
would be high, since you want to update your fields on the fly to create the
best user experience.
So what I want to say is: the idea is good, but Solr seems to be the wrong
tool to do so *alone*.

I can imagine that Solr + a database + Mahout would be a good mixture. The
last one, Mahout unfortunately is not well documented at the moment (please,
correct me, if I am wrong).

Hope this is some usefull input for you.

- Mitch
-- 
View this message in context: http://lucene.472066.n3.nabble.com/Personalized-Search-tp831070p831245.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Personalized Search

Posted by Rih <ta...@gmail.com>.
Well, it's not really a recommendation engine per se but more of a filter
for the user. Say, I already own some stuff from the result set, I just want
to exclude them from the results. What I'm concerned with is reindexing the
document everytime someone marks/votes/likes/boughts.


On Thu, May 20, 2010 at 11:04 PM, Ken Krugler
<kk...@transpac.com>wrote:

>
> On May 19, 2010, at 11:43pm, Rih wrote:
>
>  Has anybody done personalized search with Solr? I'm thinking of including
>> fields such as "bought" or "like" per member/visitor via dynamic fields to
>> a
>> product search schema. Another option is to have a multi-value field that
>> can contain user IDs. What are the possible performance issues with this
>> setup?
>>
>
> Mitch is right, what you're looking for here is a recommendation engine, if
> I understand your question properly.
>
> And yes, Mahout should work though the Taste recommendation engine it
> supports is pretty new. But Sean Owen & Robin Anil have a "Mahout in Action"
> book that's in early release via Manning, and it has lots of good
> information about Mahout & recommender systems.
>
> Assuming you have a list of recommendations for a given user, based on
> their past behavior and the recommendation engine, then you could use this
> to adjust search results. I'm waiting for Hoss to jump in here on how best
> to handle that :)
>
> -- Ken
>
> --------------------------------------------
> Ken Krugler
> +1 530-210-6378
> http://bixolabs.com
> e l a s t i c   w e b   m i n i n g
>
>
>
>
>

Re: Personalized Search

Posted by Ken Krugler <kk...@transpac.com>.
On May 19, 2010, at 11:43pm, Rih wrote:

> Has anybody done personalized search with Solr? I'm thinking of  
> including
> fields such as "bought" or "like" per member/visitor via dynamic  
> fields to a
> product search schema. Another option is to have a multi-value field  
> that
> can contain user IDs. What are the possible performance issues with  
> this
> setup?

Mitch is right, what you're looking for here is a recommendation  
engine, if I understand your question properly.

And yes, Mahout should work though the Taste recommendation engine it  
supports is pretty new. But Sean Owen & Robin Anil have a "Mahout in  
Action" book that's in early release via Manning, and it has lots of  
good information about Mahout & recommender systems.

Assuming you have a list of recommendations for a given user, based on  
their past behavior and the recommendation engine, then you could use  
this to adjust search results. I'm waiting for Hoss to jump in here on  
how best to handle that :)

-- Ken

--------------------------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c   w e b   m i n i n g