dismax "boost query" not useful?

["Smiley, David W." <ds...@mitre.org>]

As I was reviewing the boosting capabilities of the dismax & edismax query parsers, it's not clear to me that the "boost query" has much use.  The value of boost functions, particularly with a multiplied boost that edismax supports, is very clear -- there are a variety of uses.  But I can't think of a useful case when I want to both *add* a component to the ultimate score, and for that component to be a non-function query (i.e. use the lucene query parser).

Also, you can basically get the same affect as a boost query via boost functions: bf=query(mybq)&mybq=...      and note you will probably multiply this via product(10,query(mybq)) to boost it to an appropriate number.

~ David Smiley

Re: dismax "boost query" not useful?

[Chris Hostetter <ho...@fucit.org>]

: 1. Each piece is still subject to the IDF component of the score, 
: requiring me to make each individual category have a boost factoring 
: that in.  For example, if I want meta:promote to be twice as boosted as 
: category:featured, I can't simply boost the first to 2 and the second to 
: 1 (the default) -- I have enable debugQuery and carefully skew them 
: appropriately to what I want.  And the IDF might change as the data 
: changes.

right, getting anything close to an "absolute" effect is neraly impossible 
-- but with custom similarities and omit norms/tf the complexities can be 
reduced.

most relevant to this discussion however: this isn't a specific 
shortcoming of bq -- you'd have the same problem if you used the bf or 
boost params with these type of "flag" boosts as well.

: 2. It still ads instead of multiplies which is always what I want. 
: (should I not always want it?)

i think in general multiplication makes a lot more sense, but YMMV.  the 
one hitch with always using multiplcation is that it always has the same 
relative impact on the scores -- regardless of what the main query was or 
what the range of scores was.  but an additive boost can have more or less 
impact depending on the weighting of hte main query.  Specificly: query 
time changes to boost values in the "qf" and "pf" params can make default 
"bq" and "bf" params more or less important, but a "boost" param wouldn't 
care, it will always be a multiplier.

: use.  The only way I know to give a fielded query a constant score is a 
: range query which is a total hack, e.g. meta:[promote TO promote] which 
: you could then boost.  Ick!

That's an interesting point ... I don't think we have any nice generic way
to let users make any arbitrary query be a constant scoring query.  (you 
can use a "0" boost, but then the constant score is "0") ... that would 
probably be a pretty trivial QParser to write though ... hmmm.



-Hoss

Re: dismax "boost query" not useful?

["Smiley, David W." <ds...@mitre.org>]

On Apr 5, 2011, at 3:17 PM, Chris Hostetter wrote:

> one of the original use cases for bq was for artificial keyword boosting, 
> in which case it still comes in handy...
> 
> bq=meta:promote^100 text:new^10 category:featured^100 (*:* -category:accessories)^10

Yeah I thought of this specific use-case. There are two issues with it though:
1. Each piece is still subject to the IDF component of the score, requiring me to make each individual category have a boost factoring that in.  For example, if I want meta:promote to be twice as boosted as category:featured, I can't simply boost the first to 2 and the second to 1 (the default) -- I have enable debugQuery and carefully skew them appropriately to what I want.  And the IDF might change as the data changes.
2. It still ads instead of multiplies which is always what I want. (should I not always want it?)

It's hard to actually avoid the IDF irrespective of which parameter you use.  The only way I know to give a fielded query a constant score is a range query which is a total hack, e.g. meta:[promote TO promote] which you could then boost.  Ick!

~ David Smiley
Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/

Re: dismax "boost query" not useful?

[Yonik Seeley <yo...@lucidimagination.com>]

On Wed, Apr 6, 2011 at 12:00 PM, Shawn Heisey <so...@elyograg.org> wrote:
> We aren't yet using dismax in production, but I've had it in my config for a
> while now.  I've changed it to edismax in the 3.1 setup I'm putting together
> now.  It has the following in the bf parameter:
>
> recip(ms(NOW/DAY,pd),3.16e-11,1,1)
>
> Is there a way to do this without bf?  I couldn't make heads or tails of
> what you wrote above.

"bf" parsing is fragile because it is a space-delimited list of
functions (meaning no functions may have whitespace in them).
"bf" also adds the function to the query score, but for boosting one
is normally better off multiplying.  With edismax, you can get a
multiplicative boost via
boost=recip(ms(NOW/DAY,pd),3.16e-11,1,1)

-Yonik
http://www.lucenerevolution.org -- Lucene/Solr User Conference, May
25-26, San Francisco

Re: dismax "boost query" not useful?

[Shawn Heisey <so...@elyograg.org>]

On 4/5/2011 1:17 PM, Chris Hostetter wrote:
> the "boost" param of edismax is probably a lot better choice then either
> bq/bf -- but it really depends on wether you want an additive boost or a
> multiplicitive one (of course with teh function query syntax add(),
> product() and (query() can be combined in anyway you want)
>
> in terms of the merrits of the bq vs bf, if we wanted to get rid of one or
> the other, i'd argue eliminating bf since it has *very* brittle parsing
> rules in place (for historic reasons).  while you can use variable
> derefrencing to get the "guts" of either a bf=query($a) or a bq={$func
> v=$a}, promoting the use of bq over bf makes using the param body inline
> simpler so people are less likely to run into problems (ie: bq={!func}...
> doesn't require any special escaping, but bf=query(...) does)

We aren't yet using dismax in production, but I've had it in my config 
for a while now.  I've changed it to edismax in the 3.1 setup I'm 
putting together now.  It has the following in the bf parameter:

recip(ms(NOW/DAY,pd),3.16e-11,1,1)

Is there a way to do this without bf?  I couldn't make heads or tails of 
what you wrote above.

Thanks,
Shawn

Re: dismax "boost query" not useful?

[Chris Hostetter <ho...@fucit.org>]

Short answer: the existence is entirely historic.  I added bq because i 
needed it, and then i added bf because the _val_:"..." syntax was anoying.

: can't think of a useful case when I want to both *add* a component to 
: the ultimate score, and for that component to be a non-function query 
: (i.e. use the lucene query parser).

one of the original use cases for bq was for artificial keyword boosting, 
in which case it still comes in handy...

bq=meta:promote^100 text:new^10 category:featured^100 (*:* -category:accessories)^10

: Also, you can basically get the same affect as a boost query via boost 
: functions: bf=query(mybq)&mybq=...  and note you will probably multiply 
: this via product(10,query(mybq)) to boost it to an appropriate number.

the "boost" param of edismax is probably a lot better choice then either 
bq/bf -- but it really depends on wether you want an additive boost or a 
multiplicitive one (of course with teh function query syntax add(), 
product() and (query() can be combined in anyway you want)

in terms of the merrits of the bq vs bf, if we wanted to get rid of one or 
the other, i'd argue eliminating bf since it has *very* brittle parsing 
rules in place (for historic reasons).  while you can use variable 
derefrencing to get the "guts" of either a bf=query($a) or a bq={$func 
v=$a}, promoting the use of bq over bf makes using the param body inline 
simpler so people are less likely to run into problems (ie: bq={!func}... 
doesn't require any special escaping, but bf=query(...) does)



-Hoss