You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Ugo Matrangolo <ug...@gmail.com> on 2017/02/01 15:04:30 UTC
Upserting doc fields from a SearchComponent
Hi,
I'm trying to write a SearchComponent that personalizes on the fly a field
on all the docs resulting from the initial search query.
What I would like to do is to write a SearchComponent that intercepts the
documents belonging to the result sets of a search query and upsert one or
more of their field values with infos retrieved from external svc/logic.
I can't pre-compute all of these values because they are highly dependant
on the user/context.
I was wondering if someone here already did something like this and, more
broadly, has some experience in personalizing a search response in the Solr
guts.
Best
Ugo
Re: Upserting doc fields from a SearchComponent
Posted by Charlie Hull <ch...@flax.co.uk>.
On 01/02/2017 15:55, Ugo Matrangolo wrote:
> Hi Erick,
>
> Personalizing a 'price' involves using an external service (luckily we
> could cache most of the interactions) and it is accessed using a lib that
> gets dropped in the Solr classpath.
Hi Ugo,
We built a Solr plugin 'XJoin', allowing you to use results from an
external system with Solr. Here are two blog posts about it, the first
may be relevant:
http://www.flax.co.uk/blog/2016/01/25/xjoin-solr-part-1-filtering-using-price-discount-data/
http://www.flax.co.uk/blog/2016/01/29/xjoin-solr-part-2-click-example/
Cheers
Charlie
>
> What I need to do is this kind of flow:
>
> 1. Query (.. personalization params ...)
> 2. Find the initial search results (e.g. find all stuff 'converse')
> 3. From field values in search results docs (previously indexed and rarely
> changing) I need to ask this 'personalizer' what is the actual price for
> this item for this user in this moment
> 4. Add this as a normal doc field in search response somewhere (e.fg.
> personalized_price)
> 5. Use stat/fq/personalized_price:[10 TO 99]
>
> The idea is to plug a 'Pricer' search component in the query chain but I
> was wondering if it was the best idea/practice before going all-in with
> this approach.
>
> Another option could be function queries and frange to do the filtering.
>
> Hope this clarifies.
>
> Best
> Ugo
>
> On Wed, Feb 1, 2017 at 3:44 PM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> You need to be clear about what to do when. The [subquery], is
>> completely separate from _finding_ the top N docs. Your last
>> question is about finding the top N.
>>
>> There are really two choices that spring to mind, depending on where
>> you keep your data about the user. Solr can't magically know that user
>> X wants a specific price range.
>>
>> One choice would be to have the app layer contact wherever the data
>> is kept and add fq clauses.
>>
>> Another is to keep the data in Solr somewhere and write a search component
>> that tacks this kind of clause on the incoming query.
>>
>> Best,
>> Erick
>>
>> On Wed, Feb 1, 2017 at 7:39 AM, Ugo Matrangolo <ug...@gmail.com>
>> wrote:
>>> Hi,
>>>
>>> tx for the speedy response.
>>>
>>> What if I need to filter on the result matches ??
>>>
>>> Example: I have a price I need to personalize per user/request and need
>>> then to apply filter ranges on this personalized price (show only stuff
>> in
>>> the 10$ - 99$ range).
>>>
>>> WDYT ?
>>>
>>> Best
>>> Ugo
>>>
>>> On Wed, Feb 1, 2017 at 3:34 PM, Erick Erickson <er...@gmail.com>
>>> wrote:
>>>
>>>> If the data is from another Solr instance, consider the [subquery]
>>>> Document Transformer here:
>>>> https://cwiki.apache.org/confluence/display/solr/
>>>> Transforming+Result+Documents#TransformingResultDocuments-[subquery]
>>>>
>>>> More broadly, you can write a custom DocTransformer plugin do insert
>>>> anything you want in the output documents.
>>>>
>>>> NOTE: DocTransformer only works on the top N docs being returned,
>>>> which is what I think you want. IOW, if rows=10 it only "sees" 10
>>>> documents even if numFound is millions.
>>>>
>>>> Best,
>>>> Erick
>>>>
>>>>
>>>>
>>>> On Wed, Feb 1, 2017 at 7:04 AM, Ugo Matrangolo <
>> ugo.matrangolo@gmail.com>
>>>> wrote:
>>>>> Hi,
>>>>>
>>>>> I'm trying to write a SearchComponent that personalizes on the fly a
>>>> field
>>>>> on all the docs resulting from the initial search query.
>>>>>
>>>>> What I would like to do is to write a SearchComponent that intercepts
>> the
>>>>> documents belonging to the result sets of a search query and upsert
>> one
>>>> or
>>>>> more of their field values with infos retrieved from external
>> svc/logic.
>>>>>
>>>>> I can't pre-compute all of these values because they are highly
>> dependant
>>>>> on the user/context.
>>>>>
>>>>> I was wondering if someone here already did something like this and,
>> more
>>>>> broadly, has some experience in personalizing a search response in the
>>>> Solr
>>>>> guts.
>>>>>
>>>>> Best
>>>>> Ugo
>>>>
>>
>
--
Charlie Hull
Flax - Open Source Enterprise Search
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
web: www.flax.co.uk
Re: Upserting doc fields from a SearchComponent
Posted by Ugo Matrangolo <ug...@gmail.com>.
Hi Erick,
Personalizing a 'price' involves using an external service (luckily we
could cache most of the interactions) and it is accessed using a lib that
gets dropped in the Solr classpath.
What I need to do is this kind of flow:
1. Query (.. personalization params ...)
2. Find the initial search results (e.g. find all stuff 'converse')
3. From field values in search results docs (previously indexed and rarely
changing) I need to ask this 'personalizer' what is the actual price for
this item for this user in this moment
4. Add this as a normal doc field in search response somewhere (e.fg.
personalized_price)
5. Use stat/fq/personalized_price:[10 TO 99]
The idea is to plug a 'Pricer' search component in the query chain but I
was wondering if it was the best idea/practice before going all-in with
this approach.
Another option could be function queries and frange to do the filtering.
Hope this clarifies.
Best
Ugo
On Wed, Feb 1, 2017 at 3:44 PM, Erick Erickson <er...@gmail.com>
wrote:
> You need to be clear about what to do when. The [subquery], is
> completely separate from _finding_ the top N docs. Your last
> question is about finding the top N.
>
> There are really two choices that spring to mind, depending on where
> you keep your data about the user. Solr can't magically know that user
> X wants a specific price range.
>
> One choice would be to have the app layer contact wherever the data
> is kept and add fq clauses.
>
> Another is to keep the data in Solr somewhere and write a search component
> that tacks this kind of clause on the incoming query.
>
> Best,
> Erick
>
> On Wed, Feb 1, 2017 at 7:39 AM, Ugo Matrangolo <ug...@gmail.com>
> wrote:
> > Hi,
> >
> > tx for the speedy response.
> >
> > What if I need to filter on the result matches ??
> >
> > Example: I have a price I need to personalize per user/request and need
> > then to apply filter ranges on this personalized price (show only stuff
> in
> > the 10$ - 99$ range).
> >
> > WDYT ?
> >
> > Best
> > Ugo
> >
> > On Wed, Feb 1, 2017 at 3:34 PM, Erick Erickson <er...@gmail.com>
> > wrote:
> >
> >> If the data is from another Solr instance, consider the [subquery]
> >> Document Transformer here:
> >> https://cwiki.apache.org/confluence/display/solr/
> >> Transforming+Result+Documents#TransformingResultDocuments-[subquery]
> >>
> >> More broadly, you can write a custom DocTransformer plugin do insert
> >> anything you want in the output documents.
> >>
> >> NOTE: DocTransformer only works on the top N docs being returned,
> >> which is what I think you want. IOW, if rows=10 it only "sees" 10
> >> documents even if numFound is millions.
> >>
> >> Best,
> >> Erick
> >>
> >>
> >>
> >> On Wed, Feb 1, 2017 at 7:04 AM, Ugo Matrangolo <
> ugo.matrangolo@gmail.com>
> >> wrote:
> >> > Hi,
> >> >
> >> > I'm trying to write a SearchComponent that personalizes on the fly a
> >> field
> >> > on all the docs resulting from the initial search query.
> >> >
> >> > What I would like to do is to write a SearchComponent that intercepts
> the
> >> > documents belonging to the result sets of a search query and upsert
> one
> >> or
> >> > more of their field values with infos retrieved from external
> svc/logic.
> >> >
> >> > I can't pre-compute all of these values because they are highly
> dependant
> >> > on the user/context.
> >> >
> >> > I was wondering if someone here already did something like this and,
> more
> >> > broadly, has some experience in personalizing a search response in the
> >> Solr
> >> > guts.
> >> >
> >> > Best
> >> > Ugo
> >>
>
Re: Upserting doc fields from a SearchComponent
Posted by Erick Erickson <er...@gmail.com>.
You need to be clear about what to do when. The [subquery], is
completely separate from _finding_ the top N docs. Your last
question is about finding the top N.
There are really two choices that spring to mind, depending on where
you keep your data about the user. Solr can't magically know that user
X wants a specific price range.
One choice would be to have the app layer contact wherever the data
is kept and add fq clauses.
Another is to keep the data in Solr somewhere and write a search component
that tacks this kind of clause on the incoming query.
Best,
Erick
On Wed, Feb 1, 2017 at 7:39 AM, Ugo Matrangolo <ug...@gmail.com> wrote:
> Hi,
>
> tx for the speedy response.
>
> What if I need to filter on the result matches ??
>
> Example: I have a price I need to personalize per user/request and need
> then to apply filter ranges on this personalized price (show only stuff in
> the 10$ - 99$ range).
>
> WDYT ?
>
> Best
> Ugo
>
> On Wed, Feb 1, 2017 at 3:34 PM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> If the data is from another Solr instance, consider the [subquery]
>> Document Transformer here:
>> https://cwiki.apache.org/confluence/display/solr/
>> Transforming+Result+Documents#TransformingResultDocuments-[subquery]
>>
>> More broadly, you can write a custom DocTransformer plugin do insert
>> anything you want in the output documents.
>>
>> NOTE: DocTransformer only works on the top N docs being returned,
>> which is what I think you want. IOW, if rows=10 it only "sees" 10
>> documents even if numFound is millions.
>>
>> Best,
>> Erick
>>
>>
>>
>> On Wed, Feb 1, 2017 at 7:04 AM, Ugo Matrangolo <ug...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I'm trying to write a SearchComponent that personalizes on the fly a
>> field
>> > on all the docs resulting from the initial search query.
>> >
>> > What I would like to do is to write a SearchComponent that intercepts the
>> > documents belonging to the result sets of a search query and upsert one
>> or
>> > more of their field values with infos retrieved from external svc/logic.
>> >
>> > I can't pre-compute all of these values because they are highly dependant
>> > on the user/context.
>> >
>> > I was wondering if someone here already did something like this and, more
>> > broadly, has some experience in personalizing a search response in the
>> Solr
>> > guts.
>> >
>> > Best
>> > Ugo
>>
Re: Upserting doc fields from a SearchComponent
Posted by Ugo Matrangolo <ug...@gmail.com>.
Hi,
tx for the speedy response.
What if I need to filter on the result matches ??
Example: I have a price I need to personalize per user/request and need
then to apply filter ranges on this personalized price (show only stuff in
the 10$ - 99$ range).
WDYT ?
Best
Ugo
On Wed, Feb 1, 2017 at 3:34 PM, Erick Erickson <er...@gmail.com>
wrote:
> If the data is from another Solr instance, consider the [subquery]
> Document Transformer here:
> https://cwiki.apache.org/confluence/display/solr/
> Transforming+Result+Documents#TransformingResultDocuments-[subquery]
>
> More broadly, you can write a custom DocTransformer plugin do insert
> anything you want in the output documents.
>
> NOTE: DocTransformer only works on the top N docs being returned,
> which is what I think you want. IOW, if rows=10 it only "sees" 10
> documents even if numFound is millions.
>
> Best,
> Erick
>
>
>
> On Wed, Feb 1, 2017 at 7:04 AM, Ugo Matrangolo <ug...@gmail.com>
> wrote:
> > Hi,
> >
> > I'm trying to write a SearchComponent that personalizes on the fly a
> field
> > on all the docs resulting from the initial search query.
> >
> > What I would like to do is to write a SearchComponent that intercepts the
> > documents belonging to the result sets of a search query and upsert one
> or
> > more of their field values with infos retrieved from external svc/logic.
> >
> > I can't pre-compute all of these values because they are highly dependant
> > on the user/context.
> >
> > I was wondering if someone here already did something like this and, more
> > broadly, has some experience in personalizing a search response in the
> Solr
> > guts.
> >
> > Best
> > Ugo
>
Re: Upserting doc fields from a SearchComponent
Posted by Erick Erickson <er...@gmail.com>.
If the data is from another Solr instance, consider the [subquery]
Document Transformer here:
https://cwiki.apache.org/confluence/display/solr/Transforming+Result+Documents#TransformingResultDocuments-[subquery]
More broadly, you can write a custom DocTransformer plugin do insert
anything you want in the output documents.
NOTE: DocTransformer only works on the top N docs being returned,
which is what I think you want. IOW, if rows=10 it only "sees" 10
documents even if numFound is millions.
Best,
Erick
On Wed, Feb 1, 2017 at 7:04 AM, Ugo Matrangolo <ug...@gmail.com> wrote:
> Hi,
>
> I'm trying to write a SearchComponent that personalizes on the fly a field
> on all the docs resulting from the initial search query.
>
> What I would like to do is to write a SearchComponent that intercepts the
> documents belonging to the result sets of a search query and upsert one or
> more of their field values with infos retrieved from external svc/logic.
>
> I can't pre-compute all of these values because they are highly dependant
> on the user/context.
>
> I was wondering if someone here already did something like this and, more
> broadly, has some experience in personalizing a search response in the Solr
> guts.
>
> Best
> Ugo