You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by John <fa...@gmail.com> on 2011/11/17 14:46:51 UTC
FunctionQuery score=0
Hi,
I am using a function query that based on the query of the user gives a
score for the results I am presenting.
Some of the results are receiving score=0 in my function and I would like
them not to appear in the search results.
How can I achieve that?
Thanks in advance.
Re: FunctionQuery score=0
Posted by John <fa...@gmail.com>.
Thanks Hoss,
I will give those a try and let you know.
Cheers.
On Wed, Nov 23, 2011 at 8:35 PM, Chris Hostetter
<ho...@fucit.org>wrote:
>
> : Which answers my query needs. BUT, my boost function actually changes
> some
> : of the results to be of score 0, which I want to be excluded from the
> : result set.
>
> Ok .. so the crux of the issue is that your boost function results in a
> value of 0 for some documents, and you would like those documents excluded
> from your results...
>
> eqsim(alltokens,"xyz")
>
> "eqsim" is not a function thta ships with Solr (as far as i know) so i'm
> guessing it's something custom .. can you clarify what it does?
>
> : 2) This is why I used the frange query to solve the issue with the score
> 0:
> : q={!frange l=0 incl=false}query({!type=edismax qf="abstract^0.02
> title^0.08
> : categorysearch^0.05" boost='eqsim(alltokens,"xyz")' v='+tokens5:"xyz" '})
> :
> : But this time, the remaining results lost their *boosted* scores, and
> : therefore the sort by score got all mixed up.
>
> correct: frange produces a ConstantScoreQuery, it can only be used to
> filter documents based on wether the function it wraps falls in/out of the
> range.
>
> : 3) I assume I can use filter queries, but from my understanding FQs
> : actually perform another query before the main one and these queries are
> : expensive in time and I would like to avoid it if possible.
>
> Unless you actaully see notisable performance problems I wouldn't assume
> it will be an issue -- test first, get it working, then optimize if it's
> too slow. For most people the overhead of the "fq" won't a factor.
>
> One option you might consider is the "cache=false" local param which tells
> Solr not to cache the fq (handy if you know the query you are
> filtering on is not going to be reused much) and since it's not being
> cached, Solr will execute it in parallel with the main query and ignore
> anything that it already knows isn't going to matter in the final query.
>
> In your case however, you can already "optimize" the fq solution a bit
> because what you really need to filter out isn't documents matching your
> main query with a score less then zero; that set is the same as the set of
> documents for whom your "eqsim" function returns 0, so you can just use
> *that* in your fq. Something like this should work...
>
> q={!edismax ... boost=$eqsim}
> fq={!frange l=0 incl=false v=$eqsim}
> eqsim=eqsim(alltokens,"xyz")
>
> ...but there may still be ways to clean that up and make it faster
> depending on what exactly your "eqsim" function does (ie: there may be a
> simple query that can be faster then that "frange" to identify the docs
> that get non-zero values from that function.
>
> -Hoss
>
Re: FunctionQuery score=0
Posted by Chris Hostetter <ho...@fucit.org>.
: Which answers my query needs. BUT, my boost function actually changes some
: of the results to be of score 0, which I want to be excluded from the
: result set.
Ok .. so the crux of the issue is that your boost function results in a
value of 0 for some documents, and you would like those documents excluded
from your results...
eqsim(alltokens,"xyz")
"eqsim" is not a function thta ships with Solr (as far as i know) so i'm
guessing it's something custom .. can you clarify what it does?
: 2) This is why I used the frange query to solve the issue with the score 0:
: q={!frange l=0 incl=false}query({!type=edismax qf="abstract^0.02 title^0.08
: categorysearch^0.05" boost='eqsim(alltokens,"xyz")' v='+tokens5:"xyz" '})
:
: But this time, the remaining results lost their *boosted* scores, and
: therefore the sort by score got all mixed up.
correct: frange produces a ConstantScoreQuery, it can only be used to
filter documents based on wether the function it wraps falls in/out of the
range.
: 3) I assume I can use filter queries, but from my understanding FQs
: actually perform another query before the main one and these queries are
: expensive in time and I would like to avoid it if possible.
Unless you actaully see notisable performance problems I wouldn't assume
it will be an issue -- test first, get it working, then optimize if it's
too slow. For most people the overhead of the "fq" won't a factor.
One option you might consider is the "cache=false" local param which tells
Solr not to cache the fq (handy if you know the query you are
filtering on is not going to be reused much) and since it's not being
cached, Solr will execute it in parallel with the main query and ignore
anything that it already knows isn't going to matter in the final query.
In your case however, you can already "optimize" the fq solution a bit
because what you really need to filter out isn't documents matching your
main query with a score less then zero; that set is the same as the set of
documents for whom your "eqsim" function returns 0, so you can just use
*that* in your fq. Something like this should work...
q={!edismax ... boost=$eqsim}
fq={!frange l=0 incl=false v=$eqsim}
eqsim=eqsim(alltokens,"xyz")
...but there may still be ways to clean that up and make it faster
depending on what exactly your "eqsim" function does (ie: there may be a
simple query that can be faster then that "frange" to identify the docs
that get non-zero values from that function.
-Hoss
Re: FunctionQuery score=0
Posted by John <fa...@gmail.com>.
Hi Hoss,
Thanks for the detailed response.
My XY problem is:
1) I am trying to search for a complex query:
q={!type=edismax qf="abstract^0.02 title^0.08 categorysearch^0.05"
boost='eqsim(alltokens,"xyz")' v='+tokens5:"xyz" '}
Which answers my query needs. BUT, my boost function actually changes some
of the results to be of score 0, which I want to be excluded from the
result set.
2) This is why I used the frange query to solve the issue with the score 0:
q={!frange l=0 incl=false}query({!type=edismax qf="abstract^0.02 title^0.08
categorysearch^0.05" boost='eqsim(alltokens,"xyz")' v='+tokens5:"xyz" '})
But this time, the remaining results lost their *boosted* scores, and
therefore the sort by score got all mixed up.
3) I assume I can use filter queries, but from my understanding FQs
actually perform another query before the main one and these queries are
expensive in time and I would like to avoid it if possible.
Hope this explains a bit more.
Thanks,
Lev
On Tue, Nov 22, 2011 at 9:15 PM, Chris Hostetter
<ho...@fucit.org>wrote:
>
> : > q={!frange l=0 incl=false}query({!type=edismax qf="abstract^0.02
> : > title^0.08 categorysearch^0.05" boost='eqsim(alltokens,"xyz")'
> : > v='+tokens5:"xyz" '})
> : >
> : >
> : > With the above query, I am getting only the results that I want, the
> ones
> : > whose score after my FucntionQuery are above 0, but the problem now is
> that
> : > the final score for all results is changed to 1, which affects the
> sorting.
> : >
> : > How can I keep the original score that is calculated by the edismax
> query?
>
> a) Like i said. details matter.
>
> In your earlier messages you mentioned that you were wrapping a function
> arround a query and wanted to not have the function match anythign where
> the result was 0 -- the suggestions provided have done that.
>
> this is the first time you mentioned that you needed the values returned
> by the function as the scores of the documents (had you mentioned that you
> might have gotten differnet answers)
>
> b) if you look closely at the suggestion from André, you'll see that his
> specific suggestion will actually do what you want if you follow it --
> express the query you want in the "q" param (so you get the scores from
> it) and then express an "fq" that refers to the "q" query as a variable...
>
> : >>>> q=ipod&fq={!frange l=0 incl=false}query($q)
>
> c) Based on the concrete example you've given above, i'ts not clear to me
> that you actually need any of this -- if the above query is giving you the
> results you want, but you want the scores from the edismax query to be
> used as the final scores of the function, then there is no need to wrap
> the query in any sort of function at all, or exclude any 0 values....
>
> this should be exactly what you want...
>
> q={!type=edismax qf="abstract^0.02 title^0.08 categorysearch^0.05"
> boost='eqsim(alltokens,"xyz")' v='+tokens5:"xyz" '}
>
> ...why exactly did you think you needed to wrap that query in a function?
>
> https://people.apache.org/~hossman/#xyproblem
> XY Problem
>
> Your question appears to be an "XY Problem" ... that is: you are dealing
> with "X", you are assuming "Y" will help you, and you are asking about "Y"
> without giving more details about the "X" so that we can understand the
> full issue. Perhaps the best solution doesn't involve "Y" at all?
> See Also: http://www.perlmonks.org/index.pl?node_id=542341
>
>
>
>
> -Hoss
Re: FunctionQuery score=0
Posted by Chris Hostetter <ho...@fucit.org>.
: > q={!frange l=0 incl=false}query({!type=edismax qf="abstract^0.02
: > title^0.08 categorysearch^0.05" boost='eqsim(alltokens,"xyz")'
: > v='+tokens5:"xyz" '})
: >
: >
: > With the above query, I am getting only the results that I want, the ones
: > whose score after my FucntionQuery are above 0, but the problem now is that
: > the final score for all results is changed to 1, which affects the sorting.
: >
: > How can I keep the original score that is calculated by the edismax query?
a) Like i said. details matter.
In your earlier messages you mentioned that you were wrapping a function
arround a query and wanted to not have the function match anythign where
the result was 0 -- the suggestions provided have done that.
this is the first time you mentioned that you needed the values returned
by the function as the scores of the documents (had you mentioned that you
might have gotten differnet answers)
b) if you look closely at the suggestion from André, you'll see that his
specific suggestion will actually do what you want if you follow it --
express the query you want in the "q" param (so you get the scores from
it) and then express an "fq" that refers to the "q" query as a variable...
: >>>> q=ipod&fq={!frange l=0 incl=false}query($q)
c) Based on the concrete example you've given above, i'ts not clear to me
that you actually need any of this -- if the above query is giving you the
results you want, but you want the scores from the edismax query to be
used as the final scores of the function, then there is no need to wrap
the query in any sort of function at all, or exclude any 0 values....
this should be exactly what you want...
q={!type=edismax qf="abstract^0.02 title^0.08 categorysearch^0.05" boost='eqsim(alltokens,"xyz")' v='+tokens5:"xyz" '}
...why exactly did you think you needed to wrap that query in a function?
https://people.apache.org/~hossman/#xyproblem
XY Problem
Your question appears to be an "XY Problem" ... that is: you are dealing
with "X", you are assuming "Y" will help you, and you are asking about "Y"
without giving more details about the "X" so that we can understand the
full issue. Perhaps the best solution doesn't involve "Y" at all?
See Also: http://www.perlmonks.org/index.pl?node_id=542341
-Hoss
Re: FunctionQuery score=0
Posted by John <fa...@gmail.com>.
Can this be fixed somehow? I also need the real score.
On Sun, Nov 20, 2011 at 10:44 AM, John <fa...@gmail.com> wrote:
> After playing some more with this I managed to get what I want, almost.
>
> My query now looks like:
>
> q={!frange l=0 incl=false}query({!type=edismax qf="abstract^0.02
> title^0.08 categorysearch^0.05" boost='eqsim(alltokens,"xyz")'
> v='+tokens5:"xyz" '})
>
>
> With the above query, I am getting only the results that I want, the ones
> whose score after my FucntionQuery are above 0, but the problem now is that
> the final score for all results is changed to 1, which affects the sorting.
>
> How can I keep the original score that is calculated by the edismax query?
>
> Cheers,
> John
>
>
> On Fri, Nov 18, 2011 at 10:50 AM, Andre Bois-Crettez <
> andre.bois@kelkoo.com> wrote:
>
>> Definitely worked for me, with a classic full text search on "ipod" and
>> such.
>> Changing the lower bound changed the number of results.
>>
>> Follow Chris advice, and give more details.
>>
>>
>>
>> John wrote:
>>
>>> Doesn't seem to work.
>>> I though that FilterQueries work before the search is performed and not
>>> after... no?
>>>
>>> Debug doesn't include filter query only the below (changed a bit):
>>>
>>> BoostedQuery(boost(+fieldName:**"....",boostedFunction(ord(**
>>> fieldName),query)))
>>>
>>>
>>> On Thu, Nov 17, 2011 at 5:04 PM, Andre Bois-Crettez
>>> <an...@kelkoo.com>wrote:
>>>
>>>
>>>
>>>> John wrote:
>>>>
>>>>
>>>>
>>>>> Some of the results are receiving score=0 in my function and I would
>>>>> like
>>>>> them not to appear in the search results.
>>>>>
>>>>>
>>>>>
>>>>>
>>>> you can use frange, and filter by score:
>>>>
>>>> q=ipod&fq={!frange l=0 incl=false}query($q)
>>>>
>>>> --
>>>> André Bois-Crettez
>>>>
>>>> Search technology, Kelkoo
>>>> http://www.kelkoo.com/
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>> --
>> André Bois-Crettez
>>
>> Search technology, Kelkoo
>> http://www.kelkoo.com/
>>
>>
>
Re: FunctionQuery score=0
Posted by John <fa...@gmail.com>.
After playing some more with this I managed to get what I want, almost.
My query now looks like:
q={!frange l=0 incl=false}query({!type=edismax qf="abstract^0.02 title^0.08
categorysearch^0.05" boost='eqsim(alltokens,"xyz")' v='+tokens5:"xyz" '})
With the above query, I am getting only the results that I want, the ones
whose score after my FucntionQuery are above 0, but the problem now is that
the final score for all results is changed to 1, which affects the sorting.
How can I keep the original score that is calculated by the edismax query?
Cheers,
John
On Fri, Nov 18, 2011 at 10:50 AM, Andre Bois-Crettez
<an...@kelkoo.com>wrote:
> Definitely worked for me, with a classic full text search on "ipod" and
> such.
> Changing the lower bound changed the number of results.
>
> Follow Chris advice, and give more details.
>
>
>
> John wrote:
>
>> Doesn't seem to work.
>> I though that FilterQueries work before the search is performed and not
>> after... no?
>>
>> Debug doesn't include filter query only the below (changed a bit):
>>
>> BoostedQuery(boost(+fieldName:**"....",boostedFunction(ord(**
>> fieldName),query)))
>>
>>
>> On Thu, Nov 17, 2011 at 5:04 PM, Andre Bois-Crettez
>> <an...@kelkoo.com>wrote:
>>
>>
>>
>>> John wrote:
>>>
>>>
>>>
>>>> Some of the results are receiving score=0 in my function and I would
>>>> like
>>>> them not to appear in the search results.
>>>>
>>>>
>>>>
>>>>
>>> you can use frange, and filter by score:
>>>
>>> q=ipod&fq={!frange l=0 incl=false}query($q)
>>>
>>> --
>>> André Bois-Crettez
>>>
>>> Search technology, Kelkoo
>>> http://www.kelkoo.com/
>>>
>>>
>>>
>>>
>>
>>
>>
>
> --
> André Bois-Crettez
>
> Search technology, Kelkoo
> http://www.kelkoo.com/
>
>
Re: FunctionQuery score=0
Posted by Andre Bois-Crettez <an...@kelkoo.com>.
Definitely worked for me, with a classic full text search on "ipod" and
such.
Changing the lower bound changed the number of results.
Follow Chris advice, and give more details.
John wrote:
> Doesn't seem to work.
> I though that FilterQueries work before the search is performed and not
> after... no?
>
> Debug doesn't include filter query only the below (changed a bit):
>
> BoostedQuery(boost(+fieldName:"....",boostedFunction(ord(fieldName),query)))
>
>
> On Thu, Nov 17, 2011 at 5:04 PM, Andre Bois-Crettez
> <an...@kelkoo.com>wrote:
>
>
>> John wrote:
>>
>>
>>> Some of the results are receiving score=0 in my function and I would like
>>> them not to appear in the search results.
>>>
>>>
>>>
>> you can use frange, and filter by score:
>>
>> q=ipod&fq={!frange l=0 incl=false}query($q)
>>
>> --
>> André Bois-Crettez
>>
>> Search technology, Kelkoo
>> http://www.kelkoo.com/
>>
>>
>>
>
>
--
André Bois-Crettez
Search technology, Kelkoo
http://www.kelkoo.com/
Re: FunctionQuery score=0
Posted by John <fa...@gmail.com>.
Doesn't seem to work.
I though that FilterQueries work before the search is performed and not
after... no?
Debug doesn't include filter query only the below (changed a bit):
BoostedQuery(boost(+fieldName:"....",boostedFunction(ord(fieldName),query)))
On Thu, Nov 17, 2011 at 5:04 PM, Andre Bois-Crettez
<an...@kelkoo.com>wrote:
> John wrote:
>
>> Some of the results are receiving score=0 in my function and I would like
>> them not to appear in the search results.
>>
>>
> you can use frange, and filter by score:
>
> q=ipod&fq={!frange l=0 incl=false}query($q)
>
> --
> André Bois-Crettez
>
> Search technology, Kelkoo
> http://www.kelkoo.com/
>
>
Re: FunctionQuery score=0
Posted by Andre Bois-Crettez <an...@kelkoo.com>.
John wrote:
> Some of the results are receiving score=0 in my function and I would like
> them not to appear in the search results.
>
you can use frange, and filter by score:
q=ipod&fq={!frange l=0 incl=false}query($q)
--
André Bois-Crettez
Search technology, Kelkoo
http://www.kelkoo.com/
Re: FunctionQuery score=0
Posted by Chris Hostetter <ho...@fucit.org>.
: I am using a function query that based on the query of the user gives a
: score for the results I am presenting.
please be specific -- it's not at all clear what the structure of your
query is, and the details matter.
: Some of the results are receiving score=0 in my function and I would like
: them not to appear in the search results.
this sounds expected given how functions work: by definition they match
all documents, even if one of the inputs to the function is a query that
only matches some documents.
you either need to use that query as a filter to constrain the set of
documents returned, or you need to restructure your main query using
something like the {!boost} parser (which only matches documents from it's
nested query).
if you give us an actual example of what you are doing, we can give you
suggestions on how to change it to achieve what you want.
-Hoss