You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Andy <an...@yahoo.com> on 2010/08/28 11:50:32 UTC
ExternalFileField best practices
I'm interested in using ExternalFileField to store a field "popularity" that is being updated frequently.
However ExternalFileField seems to be a pretty obscure feature. Have a few questions:
1) Can anyone share your experience using it?
2) What is the most efficient way to update the external file?
For example, the file could look like:
1=12 // the document with uniqueKey 1 has a popularity of 12//
2=4
3=45
5=78
Now the popularity of document 1 is updated to 13:
- What is the best way to update the file to reflect the change? Isn't this an O(n) operation?
- How to deal with concurrent updates to the file by multiple threads?
Would this method of using an external file scale?
Thanks.
Re: ExternalFileField best practices
Posted by simon <mt...@gmail.com>.
The extended dismax parser (see SOLR-1553) may do what you are looking for
From its feature list..
'Supports the "boost" parameter.. like the dismax bf param, but multiplies
the function query instead of adding it in'
On Sun, Aug 29, 2010 at 12:27 AM, Andy <an...@yahoo.com> wrote:
> But isn't it the case that bf adds the boost value while {!boost} multiply
> the boost value? In my case I think a multiplication is more appropriate.
>
> So there's no way to use ExternalFileField in {!boost}?
>
> --- On Sat, 8/28/10, Lance Norskog <go...@gmail.com> wrote:
>
> > From: Lance Norskog <go...@gmail.com>
> > Subject: Re: ExternalFileField best practices
> > To: solr-user@lucene.apache.org
> > Date: Saturday, August 28, 2010, 11:55 PM
> > You want the boost function bf=
> > parameter.
> >
> > On Sat, Aug 28, 2010 at 5:32 PM, Andy <an...@yahoo.com>
> > wrote:
> > > Lance,
> > >
> > > Thanks for the response.
> > >
> > > Can I use an ExternalFileField as an input to a boost
> > query?
> > >
> > > For example, if I put the field "popularity" in an
> > ExternalFileField, can I still use "popularity" in a boosted
> > query such as:
> > >
> > > {!boost b=log(popularity)}foo
> > >
> > > The doc says ExternalFileField can only be used in
> > FunctionQuery. Does that include a boost query like {!boost
> > b=log(popularity)}?
> > >
> > >
> > > --- On Sat, 8/28/10, Lance Norskog <go...@gmail.com>
> > wrote:
> > >
> > >> From: Lance Norskog <go...@gmail.com>
> > >> Subject: Re: ExternalFileField best practices
> > >> To: solr-user@lucene.apache.org
> > >> Date: Saturday, August 28, 2010, 5:16 PM
> > >> The file is completely reloaded when
> > >> you commit or optimize. There is
> > >> no incremental update available. And, yes, this
> > could be a
> > >> scaling
> > >> problem.
> > >>
> > >> How you update it is completely external to Solr.
> > >>
> > >> On Sat, Aug 28, 2010 at 2:50 AM, Andy <an...@yahoo.com>
> > >> wrote:
> > >> > I'm interested in using ExternalFileField to
> > store a
> > >> field "popularity" that is being updated
> > frequently.
> > >> >
> > >> > However ExternalFileField seems to be a
> > pretty obscure
> > >> feature. Have a few questions:
> > >> >
> > >> > 1) Can anyone share your experience using
> > it?
> > >> >
> > >> > 2) What is the most efficient way to update
> > the
> > >> external file?
> > >> > For example, the file could look like:
> > >> >
> > >> > 1=12 // the document with uniqueKey 1
> > has a
> > >> popularity of 12//
> > >> > 2=4
> > >> > 3=45
> > >> > 5=78
> > >> >
> > >> > Now the popularity of document 1 is updated
> > to 13:
> > >> >
> > >> > - What is the best way to update the file to
> > reflect
> > >> the change? Isn't this an O(n) operation?
> > >> > - How to deal with concurrent updates to the
> > file by
> > >> multiple threads?
> > >> >
> > >> > Would this method of using an external file
> > scale?
>
>
Re: ExternalFileField best practices
Posted by Andy <an...@yahoo.com>.
But isn't it the case that bf adds the boost value while {!boost} multiply the boost value? In my case I think a multiplication is more appropriate.
So there's no way to use ExternalFileField in {!boost}?
--- On Sat, 8/28/10, Lance Norskog <go...@gmail.com> wrote:
> From: Lance Norskog <go...@gmail.com>
> Subject: Re: ExternalFileField best practices
> To: solr-user@lucene.apache.org
> Date: Saturday, August 28, 2010, 11:55 PM
> You want the boost function bf=
> parameter.
>
> On Sat, Aug 28, 2010 at 5:32 PM, Andy <an...@yahoo.com>
> wrote:
> > Lance,
> >
> > Thanks for the response.
> >
> > Can I use an ExternalFileField as an input to a boost
> query?
> >
> > For example, if I put the field "popularity" in an
> ExternalFileField, can I still use "popularity" in a boosted
> query such as:
> >
> > {!boost b=log(popularity)}foo
> >
> > The doc says ExternalFileField can only be used in
> FunctionQuery. Does that include a boost query like {!boost
> b=log(popularity)}?
> >
> >
> > --- On Sat, 8/28/10, Lance Norskog <go...@gmail.com>
> wrote:
> >
> >> From: Lance Norskog <go...@gmail.com>
> >> Subject: Re: ExternalFileField best practices
> >> To: solr-user@lucene.apache.org
> >> Date: Saturday, August 28, 2010, 5:16 PM
> >> The file is completely reloaded when
> >> you commit or optimize. There is
> >> no incremental update available. And, yes, this
> could be a
> >> scaling
> >> problem.
> >>
> >> How you update it is completely external to Solr.
> >>
> >> On Sat, Aug 28, 2010 at 2:50 AM, Andy <an...@yahoo.com>
> >> wrote:
> >> > I'm interested in using ExternalFileField to
> store a
> >> field "popularity" that is being updated
> frequently.
> >> >
> >> > However ExternalFileField seems to be a
> pretty obscure
> >> feature. Have a few questions:
> >> >
> >> > 1) Can anyone share your experience using
> it?
> >> >
> >> > 2) What is the most efficient way to update
> the
> >> external file?
> >> > For example, the file could look like:
> >> >
> >> > 1=12 // the document with uniqueKey 1
> has a
> >> popularity of 12//
> >> > 2=4
> >> > 3=45
> >> > 5=78
> >> >
> >> > Now the popularity of document 1 is updated
> to 13:
> >> >
> >> > - What is the best way to update the file to
> reflect
> >> the change? Isn't this an O(n) operation?
> >> > - How to deal with concurrent updates to the
> file by
> >> multiple threads?
> >> >
> >> > Would this method of using an external file
> scale?
> >> >
> >> > Thanks.
> >> >
> >> >
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Lance Norskog
> >> goksron@gmail.com
> >>
> >
> >
> >
> >
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: ExternalFileField best practices
Posted by Lance Norskog <go...@gmail.com>.
You want the boost function bf= parameter.
On Sat, Aug 28, 2010 at 5:32 PM, Andy <an...@yahoo.com> wrote:
> Lance,
>
> Thanks for the response.
>
> Can I use an ExternalFileField as an input to a boost query?
>
> For example, if I put the field "popularity" in an ExternalFileField, can I still use "popularity" in a boosted query such as:
>
> {!boost b=log(popularity)}foo
>
> The doc says ExternalFileField can only be used in FunctionQuery. Does that include a boost query like {!boost b=log(popularity)}?
>
>
> --- On Sat, 8/28/10, Lance Norskog <go...@gmail.com> wrote:
>
>> From: Lance Norskog <go...@gmail.com>
>> Subject: Re: ExternalFileField best practices
>> To: solr-user@lucene.apache.org
>> Date: Saturday, August 28, 2010, 5:16 PM
>> The file is completely reloaded when
>> you commit or optimize. There is
>> no incremental update available. And, yes, this could be a
>> scaling
>> problem.
>>
>> How you update it is completely external to Solr.
>>
>> On Sat, Aug 28, 2010 at 2:50 AM, Andy <an...@yahoo.com>
>> wrote:
>> > I'm interested in using ExternalFileField to store a
>> field "popularity" that is being updated frequently.
>> >
>> > However ExternalFileField seems to be a pretty obscure
>> feature. Have a few questions:
>> >
>> > 1) Can anyone share your experience using it?
>> >
>> > 2) What is the most efficient way to update the
>> external file?
>> > For example, the file could look like:
>> >
>> > 1=12 // the document with uniqueKey 1 has a
>> popularity of 12//
>> > 2=4
>> > 3=45
>> > 5=78
>> >
>> > Now the popularity of document 1 is updated to 13:
>> >
>> > - What is the best way to update the file to reflect
>> the change? Isn't this an O(n) operation?
>> > - How to deal with concurrent updates to the file by
>> multiple threads?
>> >
>> > Would this method of using an external file scale?
>> >
>> > Thanks.
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>> Lance Norskog
>> goksron@gmail.com
>>
>
>
>
>
--
Lance Norskog
goksron@gmail.com
Re: ExternalFileField best practices
Posted by Andy <an...@yahoo.com>.
Lance,
Thanks for the response.
Can I use an ExternalFileField as an input to a boost query?
For example, if I put the field "popularity" in an ExternalFileField, can I still use "popularity" in a boosted query such as:
{!boost b=log(popularity)}foo
The doc says ExternalFileField can only be used in FunctionQuery. Does that include a boost query like {!boost b=log(popularity)}?
--- On Sat, 8/28/10, Lance Norskog <go...@gmail.com> wrote:
> From: Lance Norskog <go...@gmail.com>
> Subject: Re: ExternalFileField best practices
> To: solr-user@lucene.apache.org
> Date: Saturday, August 28, 2010, 5:16 PM
> The file is completely reloaded when
> you commit or optimize. There is
> no incremental update available. And, yes, this could be a
> scaling
> problem.
>
> How you update it is completely external to Solr.
>
> On Sat, Aug 28, 2010 at 2:50 AM, Andy <an...@yahoo.com>
> wrote:
> > I'm interested in using ExternalFileField to store a
> field "popularity" that is being updated frequently.
> >
> > However ExternalFileField seems to be a pretty obscure
> feature. Have a few questions:
> >
> > 1) Can anyone share your experience using it?
> >
> > 2) What is the most efficient way to update the
> external file?
> > For example, the file could look like:
> >
> > 1=12 // the document with uniqueKey 1 has a
> popularity of 12//
> > 2=4
> > 3=45
> > 5=78
> >
> > Now the popularity of document 1 is updated to 13:
> >
> > - What is the best way to update the file to reflect
> the change? Isn't this an O(n) operation?
> > - How to deal with concurrent updates to the file by
> multiple threads?
> >
> > Would this method of using an external file scale?
> >
> > Thanks.
> >
> >
> >
> >
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: ExternalFileField best practices
Posted by Lance Norskog <go...@gmail.com>.
The file is completely reloaded when you commit or optimize. There is
no incremental update available. And, yes, this could be a scaling
problem.
How you update it is completely external to Solr.
On Sat, Aug 28, 2010 at 2:50 AM, Andy <an...@yahoo.com> wrote:
> I'm interested in using ExternalFileField to store a field "popularity" that is being updated frequently.
>
> However ExternalFileField seems to be a pretty obscure feature. Have a few questions:
>
> 1) Can anyone share your experience using it?
>
> 2) What is the most efficient way to update the external file?
> For example, the file could look like:
>
> 1=12 // the document with uniqueKey 1 has a popularity of 12//
> 2=4
> 3=45
> 5=78
>
> Now the popularity of document 1 is updated to 13:
>
> - What is the best way to update the file to reflect the change? Isn't this an O(n) operation?
> - How to deal with concurrent updates to the file by multiple threads?
>
> Would this method of using an external file scale?
>
> Thanks.
>
>
>
>
--
Lance Norskog
goksron@gmail.com