You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Shawn Heisey <so...@elyograg.org> on 2013/01/17 22:11:20 UTC
Questions about boosting
I've been trying to figure this out on my own, but I've come up empty so
far. I need to boost documents from a certain provider. The idea is
that if any documents in a result match a separate query (like
provider:bigbucks), I need to multiply the score by X. It's important
that the result set of the actual query is not changed, just the order.
I've tried a few things from the relevancy page on the wiki but so far I
can't seem to get anything to work. What syntax should I be using? Is
it possible to do this at query time?
Thanks,
Shawn
Re: Questions about boosting
Posted by Shawn Heisey <so...@elyograg.org>.
On 1/18/2013 12:32 AM, Mikhail Khludnev wrote:
> Colleagues,
> fwiw bq is a DisMax parser feature. Shawn, to approach the boosting syntax
> with the standard parser you need something like q=foo:bar ip:sc^1000.
> Specifying ^1000 in bq makes no sense ever. If you show query params and
> debugQuery output, it would much easier for us to help you.
> PS omitting termfreq's and positions doesn't impact query time boosing
> ever. The closes caveat is that disabling norms indexing kills _index_ time
> boosting.
Ah! As soon as I changed to my edismax handler, suddenly it started
working! I was doing all my tests with /select.
Now to work out what the boost factor should be. A value of 0.25 seems
like it might produce good results. I was surprised, I thought it would
require a higher value. Makes me think that this is not a
multiplicative boost.
Thanks,
Shawn
Re: Questions about boosting
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Colleagues,
fwiw bq is a DisMax parser feature. Shawn, to approach the boosting syntax
with the standard parser you need something like q=foo:bar ip:sc^1000.
Specifying ^1000 in bq makes no sense ever. If you show query params and
debugQuery output, it would much easier for us to help you.
PS omitting termfreq's and positions doesn't impact query time boosing
ever. The closes caveat is that disabling norms indexing kills _index_ time
boosting.
On Fri, Jan 18, 2013 at 11:10 AM, Shawn Heisey <so...@elyograg.org> wrote:
> On 1/17/2013 11:41 PM, Walter Underwood wrote:
>
>> As I understand it, the bq parameter is a full Lucene query, but only
>> used for ranking, not for selection. This is the complement of fq.
>>
>> You can use weighting: provider:fred^8
>>
>
> I tried bq=ip:sc^1000 and it doesn't seem to be making any difference.
> Even if I add fq=ip:sc, I don't see any mention of bq, ip, sc, or 1000 in
> the debugQuery output.
>
> This is the case on both 3.5 and 4.1. In case it was caused by omitting
> termfreq and positions on the field I'm using in the bq, I tried a couple
> of other fields that don't omit anything and bq seems to be having no
> effect at all.
>
> Thanks,
> Shawn
>
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: Questions about boosting
Posted by Shawn Heisey <so...@elyograg.org>.
On 1/17/2013 11:41 PM, Walter Underwood wrote:
> As I understand it, the bq parameter is a full Lucene query, but only used for ranking, not for selection. This is the complement of fq.
>
> You can use weighting: provider:fred^8
I tried bq=ip:sc^1000 and it doesn't seem to be making any difference.
Even if I add fq=ip:sc, I don't see any mention of bq, ip, sc, or 1000
in the debugQuery output.
This is the case on both 3.5 and 4.1. In case it was caused by omitting
termfreq and positions on the field I'm using in the bq, I tried a
couple of other fields that don't omit anything and bq seems to be
having no effect at all.
Thanks,
Shawn
Re: Questions about boosting
Posted by Shawn Heisey <so...@elyograg.org>.
On 1/18/2013 8:52 AM, Walter Underwood wrote:
>
> On Jan 17, 2013, at 10:53 PM, Shawn Heisey wrote:
>
>> On 1/17/2013 11:41 PM, Walter Underwood wrote:
>>> As I understand it, the bq parameter is a full Lucene query, but only used for ranking, not for selection. This is the complement of fq.
>>>
>>> You can use weighting: provider:fred^8
>>>
>>> This will be affected by idf, so providers with fewer matches will have higher weight than those with more matches. This is a bother, but the idf-free approach requires Solr 4.0.
>>
>> I am doing my testing on Solr 4.1, so if you can give me the syntax for that, I would appreciate it. My production indexes are 3.5, but once we are confident with the 4.1 dev system, we'll upgrade.
>>
>> The provider field has omitTermFreqAndPositions="true" defined, but the fields that typically get searched don't omit anything, so IDF probably still applies in the aggregate.
>>
>> On a related note, I have rather extreme length variation in my fields, so I see quite a lot of weird results due to very short metadata. Is there any way to lessen the impact of lengthNorm without eliminating it entirely? If not, is there any way to eliminate lengthNorm without also disabling index-time boosts? At this moment I am not doing index-time boosting, but business requirements may change that in the future.
>>
>> Thanks,
>> Shawn
>
> I was experimenting with a boost function like this:
>
> if(exists(query(provider:fred)), 5, 1)
>
> That gives a constant boost if the term exists in the field, none if it does not. If you pass the provider in as a separate URL param, you could use parameter substitution.
>
> if(exists(query(provider:$provider)), 5, 1)
>
> For length norms, you could try a different similarity class or write your own, changing Similarity.computeNorm().
I tried a boost= parameter on my 4.1 server and got an error message in
the response with the entire value of the parameter:
org.apache.solr.search.SyntaxError: Nested function query must use
$param or {!v=value} forms. got 'if(exists(query(ip:sc)), 2, 1)'
I get a different error if I change it to bf instead of boost:
org.apache.solr.search.SyntaxError: Unexpected text after function: )
What am I doing wrong?
Thanks,
Shawn
Re: Questions about boosting
Posted by Walter Underwood <wu...@wunderwood.org>.
On Jan 17, 2013, at 10:53 PM, Shawn Heisey wrote:
> On 1/17/2013 11:41 PM, Walter Underwood wrote:
>> As I understand it, the bq parameter is a full Lucene query, but only used for ranking, not for selection. This is the complement of fq.
>>
>> You can use weighting: provider:fred^8
>>
>> This will be affected by idf, so providers with fewer matches will have higher weight than those with more matches. This is a bother, but the idf-free approach requires Solr 4.0.
>
> I am doing my testing on Solr 4.1, so if you can give me the syntax for that, I would appreciate it. My production indexes are 3.5, but once we are confident with the 4.1 dev system, we'll upgrade.
>
> The provider field has omitTermFreqAndPositions="true" defined, but the fields that typically get searched don't omit anything, so IDF probably still applies in the aggregate.
>
> On a related note, I have rather extreme length variation in my fields, so I see quite a lot of weird results due to very short metadata. Is there any way to lessen the impact of lengthNorm without eliminating it entirely? If not, is there any way to eliminate lengthNorm without also disabling index-time boosts? At this moment I am not doing index-time boosting, but business requirements may change that in the future.
>
> Thanks,
> Shawn
I was experimenting with a boost function like this:
if(exists(query(provider:fred)), 5, 1)
That gives a constant boost if the term exists in the field, none if it does not. If you pass the provider in as a separate URL param, you could use parameter substitution.
if(exists(query(provider:$provider)), 5, 1)
For length norms, you could try a different similarity class or write your own, changing Similarity.computeNorm().
wunder
--
Walter Underwood
wunder@wunderwood.org
Re: Questions about boosting
Posted by Shawn Heisey <so...@elyograg.org>.
On 1/17/2013 11:41 PM, Walter Underwood wrote:
> As I understand it, the bq parameter is a full Lucene query, but only used for ranking, not for selection. This is the complement of fq.
>
> You can use weighting: provider:fred^8
>
> This will be affected by idf, so providers with fewer matches will have higher weight than those with more matches. This is a bother, but the idf-free approach requires Solr 4.0.
I am doing my testing on Solr 4.1, so if you can give me the syntax for
that, I would appreciate it. My production indexes are 3.5, but once we
are confident with the 4.1 dev system, we'll upgrade.
The provider field has omitTermFreqAndPositions="true" defined, but the
fields that typically get searched don't omit anything, so IDF probably
still applies in the aggregate.
On a related note, I have rather extreme length variation in my fields,
so I see quite a lot of weird results due to very short metadata. Is
there any way to lessen the impact of lengthNorm without eliminating it
entirely? If not, is there any way to eliminate lengthNorm without also
disabling index-time boosts? At this moment I am not doing index-time
boosting, but business requirements may change that in the future.
Thanks,
Shawn
Re: Questions about boosting
Posted by Walter Underwood <wu...@wunderwood.org>.
As I understand it, the bq parameter is a full Lucene query, but only used for ranking, not for selection. This is the complement of fq.
You can use weighting: provider:fred^8
This will be affected by idf, so providers with fewer matches will have higher weight than those with more matches. This is a bother, but the idf-free approach requires Solr 4.0.
wunder
On Jan 17, 2013, at 10:31 PM, Shawn Heisey wrote:
> I did try the bq parameter. Either I'm not using it correctly, or it's not making a noticeable difference. I was not able to find any good docs, either. Can you give me complete instructions in its use? Can I control the boost factor? Is the boost additive or multiplicative?
>
> For query elevation, don't you have to know in advance the query that a user will send? There's no way for me to know this - we want to be able to apply the boost to arbitrary queries.
>
> The source data comes from MySQL, and this is a seven-shard distributed index with 74075200 documents as of a few minutes ago. Although ExternalFileField probably wouldn't be impossible, it is rather impractical.
>
> Thanks,
> Shawn
>
> On 1/17/2013 10:53 PM, Walter Underwood wrote:
>> Have you tried boost query? bq=provider:fred
>>
>> wunder
>>
>> On Jan 17, 2013, at 9:08 PM, Jack Krupansky wrote:
>>
>>> Start with "Query Elevation" and see if that helps:
>>> http://wiki.apache.org/solr/QueryElevationComponent
>>>
>>> Index-time document boost is a possibility.
>>>
>>> Maybe an ExternalFileField where every document could have a dynamic boost value that you add with a boost function.
>>>
>>> -- Jack Krupansky
>>>
>>> -----Original Message----- From: Shawn Heisey
>>> Sent: Thursday, January 17, 2013 4:11 PM
>>> To: solr-user@lucene.apache.org
>>> Subject: Questions about boosting
>>>
>>> I've been trying to figure this out on my own, but I've come up empty so
>>> far. I need to boost documents from a certain provider. The idea is
>>> that if any documents in a result match a separate query (like
>>> provider:bigbucks), I need to multiply the score by X. It's important
>>> that the result set of the actual query is not changed, just the order.
>
Re: Questions about boosting
Posted by Shawn Heisey <so...@elyograg.org>.
I did try the bq parameter. Either I'm not using it correctly, or it's
not making a noticeable difference. I was not able to find any good
docs, either. Can you give me complete instructions in its use? Can I
control the boost factor? Is the boost additive or multiplicative?
For query elevation, don't you have to know in advance the query that a
user will send? There's no way for me to know this - we want to be able
to apply the boost to arbitrary queries.
The source data comes from MySQL, and this is a seven-shard distributed
index with 74075200 documents as of a few minutes ago. Although
ExternalFileField probably wouldn't be impossible, it is rather impractical.
Thanks,
Shawn
On 1/17/2013 10:53 PM, Walter Underwood wrote:
> Have you tried boost query? bq=provider:fred
>
> wunder
>
> On Jan 17, 2013, at 9:08 PM, Jack Krupansky wrote:
>
>> Start with "Query Elevation" and see if that helps:
>> http://wiki.apache.org/solr/QueryElevationComponent
>>
>> Index-time document boost is a possibility.
>>
>> Maybe an ExternalFileField where every document could have a dynamic boost value that you add with a boost function.
>>
>> -- Jack Krupansky
>>
>> -----Original Message----- From: Shawn Heisey
>> Sent: Thursday, January 17, 2013 4:11 PM
>> To: solr-user@lucene.apache.org
>> Subject: Questions about boosting
>>
>> I've been trying to figure this out on my own, but I've come up empty so
>> far. I need to boost documents from a certain provider. The idea is
>> that if any documents in a result match a separate query (like
>> provider:bigbucks), I need to multiply the score by X. It's important
>> that the result set of the actual query is not changed, just the order.
Re: Questions about boosting
Posted by Walter Underwood <wu...@wunderwood.org>.
Have you tried boost query? bq=provider:fred
wunder
On Jan 17, 2013, at 9:08 PM, Jack Krupansky wrote:
> Start with "Query Elevation" and see if that helps:
> http://wiki.apache.org/solr/QueryElevationComponent
>
> Index-time document boost is a possibility.
>
> Maybe an ExternalFileField where every document could have a dynamic boost value that you add with a boost function.
>
> -- Jack Krupansky
>
> -----Original Message----- From: Shawn Heisey
> Sent: Thursday, January 17, 2013 4:11 PM
> To: solr-user@lucene.apache.org
> Subject: Questions about boosting
>
> I've been trying to figure this out on my own, but I've come up empty so
> far. I need to boost documents from a certain provider. The idea is
> that if any documents in a result match a separate query (like
> provider:bigbucks), I need to multiply the score by X. It's important
> that the result set of the actual query is not changed, just the order.
>
> I've tried a few things from the relevancy page on the wiki but so far I
> can't seem to get anything to work. What syntax should I be using? Is
> it possible to do this at query time?
>
> Thanks,
> Shawn
Re: Questions about boosting
Posted by Jack Krupansky <ja...@basetechnology.com>.
Start with "Query Elevation" and see if that helps:
http://wiki.apache.org/solr/QueryElevationComponent
Index-time document boost is a possibility.
Maybe an ExternalFileField where every document could have a dynamic boost
value that you add with a boost function.
-- Jack Krupansky
-----Original Message-----
From: Shawn Heisey
Sent: Thursday, January 17, 2013 4:11 PM
To: solr-user@lucene.apache.org
Subject: Questions about boosting
I've been trying to figure this out on my own, but I've come up empty so
far. I need to boost documents from a certain provider. The idea is
that if any documents in a result match a separate query (like
provider:bigbucks), I need to multiply the score by X. It's important
that the result set of the actual query is not changed, just the order.
I've tried a few things from the relevancy page on the wiki but so far I
can't seem to get anything to work. What syntax should I be using? Is
it possible to do this at query time?
Thanks,
Shawn