You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Derek Poh <dp...@globalsources.com.INVALID> on 2021/04/13 02:14:52 UTC
de-boost documents with certain words
Hi
I am trying to "de-boost" documents that have certain word(s) so that
they are pushed back to the end of the result set. Or at least not in
the first 60 records of the result set.
The query is doing a q=*:* with a few filter query (fq) parameters.
This is an example of the query which uses bq to check for the word
"mask" and give it a negative boost factor,
http://localhost:8983/solr/product/select?start=0&rows=140&fq=P_DO_Flag:(%22Y%22)&fq=P_CategoryGroupId:(11044)&fq=-(P_SPC_Flag:(%22Y%22))&q=*:*&sort=score%20desc,P_DO_SPPRank%20asc,P_DO_Coupon_Flag%20desc,P_SupplierRanking%20desc,random_515%20asc&defType=edismax&bq=P_NewShortDescription:mask^-1&wt=xml&fl=P_ProductId,P_VeryShortDescription,score,P_DO_Flag,P_NewShortDescription
I have tried with
bq=P_NewShortDescription:mask^-1
bq=(*:* -P_NewShortDescription:"mask")^0.5
But the documents that have the word "mask" are not de-boost or pushed
back to the back of the result set.
In the result of the query example above, there are 3 documents that
have the word "Mask" in the field P_NewShortDescription.
2 of them are not de-boost as they have the same score of 1.5 as those
documents that do not have "mask" in them.
The remaining 1 document does have a lower score of 1 and it is at the
end of the result set.
Could you advice how to correctly de-boost documents that have certain
word(s) in them?
This is the definiton of the field that is checked on.
<field name="P_NewShortDescription" type="text_general" omitNorms="true"
multiValued="false" indexed="true" stored="true"/>
<fieldType name="text_general" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" words="stopwords.txt"
ignoreCase="true"/>
<filter class="solr.KeywordMarkerFilterFactory"
protected="protwords.txt"/>
<filter class="solr.SynonymGraphFilterFactory" expand="true"
ignoreCase="true" synonyms="synonyms.txt"/>
<filter class="solr.KStemFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" words="stopwords.txt"
ignoreCase="true"/>
<filter class="solr.KeywordMarkerFilterFactory"
protected="protwords.txt"/>
<filter class="solr.SynonymGraphFilterFactory" expand="true"
ignoreCase="true" synonyms="synonyms.txt"/>
<filter class="solr.KStemFilterFactory"/>
</analyzer>
</fieldType>
Solr version = 7.7.2
Regards,
Derek
----------------------
CONFIDENTIALITY NOTICE
This e-mail (including any attachments) may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error, please inform the sender immediately and delete this e-mail (including any attachments) from your computer, and you must not use, disclose to anyone else or copy this e-mail (including any attachments), whether in whole or in part.
This e-mail and any reply to it may be monitored for security, legal, regulatory compliance and/or other appropriate reasons.
Re: de-boost documents with certain words
Posted by Charlie Hull <ch...@opensourceconnections.com>.
Hi Derek,
It might be worth looking into Querqy, an open source query processor
for Solr that lets you trigger actions such as down boosts on certain
query words.
www.querqy.org & https://github.com/querqy
Best
Charlie
On 13/04/2021 03:14, Derek Poh wrote:
> Hi
>
> I am trying to "de-boost" documents that have certain word(s) so that
> they are pushed back to the end of the result set. Or at least not in
> the first 60 records of the result set.
>
> The query is doing a q=*:* with a few filter query (fq) parameters.
>
> This is an example of the query which uses bq to check for the word
> "mask" and give it a negative boost factor,
>
> http://localhost:8983/solr/product/select?start=0&rows=140&fq=P_DO_Flag:(%22Y%22)&fq=P_CategoryGroupId:(11044)&fq=-(P_SPC_Flag:(%22Y%22))&q=*:*&sort=score%20desc,P_DO_SPPRank%20asc,P_DO_Coupon_Flag%20desc,P_SupplierRanking%20desc,random_515%20asc&defType=edismax&bq=P_NewShortDescription:mask^-1&wt=xml&fl=P_ProductId,P_VeryShortDescription,score,P_DO_Flag,P_NewShortDescription
>
>
> I have tried with
> bq=P_NewShortDescription:mask^-1
>
> bq=(*:* -P_NewShortDescription:"mask")^0.5
>
> But the documents that have the word "mask" are not de-boost or pushed
> back to the back of the result set.
>
> In the result of the query example above, there are 3 documents that
> have the word "Mask" in the field P_NewShortDescription.
> 2 of them are not de-boost as they have the same score of 1.5 as those
> documents that do not have "mask" in them.
> The remaining 1 document does have a lower score of 1 and it is at
> the end of the result set.
>
> Could you advice how to correctly de-boost documents that have certain
> word(s) in them?
>
> This is the definiton of the field that is checked on.
> <field name="P_NewShortDescription" type="text_general"
> omitNorms="true" multiValued="false" indexed="true" stored="true"/>
>
> <fieldType name="text_general" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" words="stopwords.txt"
> ignoreCase="true"/>
> <filter class="solr.KeywordMarkerFilterFactory"
> protected="protwords.txt"/>
> <filter class="solr.SynonymGraphFilterFactory" expand="true"
> ignoreCase="true" synonyms="synonyms.txt"/>
> <filter class="solr.KStemFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" words="stopwords.txt"
> ignoreCase="true"/>
> <filter class="solr.KeywordMarkerFilterFactory"
> protected="protwords.txt"/>
> <filter class="solr.SynonymGraphFilterFactory" expand="true"
> ignoreCase="true" synonyms="synonyms.txt"/>
> <filter class="solr.KStemFilterFactory"/>
> </analyzer>
> </fieldType>
>
> Solr version = 7.7.2
>
> Regards,
> Derek
>
> ----------------------
> CONFIDENTIALITY NOTICE
> This e-mail (including any attachments) may contain confidential
> and/or privileged information. If you are not the intended recipient
> or have received this e-mail in error, please inform the sender
> immediately and delete this e-mail (including any attachments) from
> your computer, and you must not use, disclose to anyone else or copy
> this e-mail (including any attachments), whether in whole or in part.
> This e-mail and any reply to it may be monitored for security, legal,
> regulatory compliance and/or other appropriate reasons.
>
--
Charlie Hull - Managing Consultant at OpenSource Connections Limited
<www.o19s.com>
Founding member of The Search Network <https://thesearchnetwork.com/>
and co-author of Searching the Enterprise
<https://opensourceconnections.com/about-us/books-resources/>
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
Re: de-boost documents with certain words
Posted by Alessandro Benedetti <a....@sease.io>.
Hi Derek,
the BQ (Boost Query) parameter is an additive contribution to the score.
Solr calculates the score contribution of each BQ query and adds it to the
score calculated so far.
*bq=P_NewShortDescription:mask^-1 *
This is not supported: boost must be a positive float, got -1.0
org.apache.lucene.search.BoostQuery#BoostQuery
I thought to achieve this effect you could just use boost function queries.
For example for an additive negative contribution:
bf=mul(query($qq),-10)&qq=P_NewShortDescription:mask
But the score contribution is zero :
0.0 = FunctionQuery(product(query(title:cities,def=0.0),const(-10))),
product of:\n -32.01146 =
product(query(title:cities,def=0.0)=3.201146,const(-10))\n 1.0 = boost\n"
So I suspect for additive modifications to the score you have also some
code check that prevents that internally.
The same if your multiplicative :
boost=mul(query($qq),-10)&qq=P_NewShortDescription:mask
0 = truncated score, max of:
0.0 = minimum score
-298.690542362805 = product of:
9.330738 = sum of:
...
Playing a bit with boost function you should be able to get your effect
anyway.
*bq=(*:* -P_NewShortDescription:"mask")^0.5*
Using this approach you are boosting up all documents that don't contain
the term 'mask' in such field.
Fine-tuning the amount of boost this works.
*Classic Question*
*The classic question remains: how much shoud I boost? (negative or
positive)*
You could improve one query and break 10 others.
The recommendation here is:
set up a search quality evaluation framework to measure your favourite
metrics across multiple queries
We contributed Rated Ranking Evaluator to the Open Source community, a
powerful tool to measure search quality, check it out, it could help you:
https://github.com/SeaseLtd/rated-ranking-evaluator
For enterprise needs, we have also developed RRE-Enterprise( coming
soon: https://sease.io/2019/12/road-to-rated-ranking-evaluator-enterprise.html
)
Cheers
--------------------------
Alessandro Benedetti
Apache Lucene/Solr Committer
Director, R&D Software Engineer, Search Consultant
www.sease.io
On Tue, 13 Apr 2021 at 03:15, Derek Poh <dp...@globalsources.com.invalid>
wrote:
> Hi
>
> I am trying to "de-boost" documents that have certain word(s) so that
> they are pushed back to the end of the result set. Or at least not in
> the first 60 records of the result set.
>
> The query is doing a q=*:* with a few filter query (fq) parameters.
>
> This is an example of the query which uses bq to check for the word
> "mask" and give it a negative boost factor,
>
>
> http://localhost:8983/solr/product/select?start=0&rows=140&fq=P_DO_Flag:(%22Y%22)&fq=P_CategoryGroupId:(11044)&fq=-(P_SPC_Flag:(%22Y%22))&q=*:*&sort=score%20desc,P_DO_SPPRank%20asc,P_DO_Coupon_Flag%20desc,P_SupplierRanking%20desc,random_515%20asc&defType=edismax&bq=P_NewShortDescription:mask
> ^-1&wt=xml&fl=P_ProductId,P_VeryShortDescription,score,P_DO_Flag,P_NewShortDescription
>
> I have tried with
> bq=P_NewShortDescription:mask^-1
>
> bq=(*:* -P_NewShortDescription:"mask")^0.5
>
> But the documents that have the word "mask" are not de-boost or pushed
> back to the back of the result set.
>
> In the result of the query example above, there are 3 documents that
> have the word "Mask" in the field P_NewShortDescription.
> 2 of them are not de-boost as they have the same score of 1.5 as those
> documents that do not have "mask" in them.
> The remaining 1 document does have a lower score of 1 and it is at the
> end of the result set.
>
> Could you advice how to correctly de-boost documents that have certain
> word(s) in them?
>
> This is the definiton of the field that is checked on.
> <field name="P_NewShortDescription" type="text_general" omitNorms="true"
> multiValued="false" indexed="true" stored="true"/>
>
> <fieldType name="text_general" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" words="stopwords.txt"
> ignoreCase="true"/>
> <filter class="solr.KeywordMarkerFilterFactory"
> protected="protwords.txt"/>
> <filter class="solr.SynonymGraphFilterFactory" expand="true"
> ignoreCase="true" synonyms="synonyms.txt"/>
> <filter class="solr.KStemFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StopFilterFactory" words="stopwords.txt"
> ignoreCase="true"/>
> <filter class="solr.KeywordMarkerFilterFactory"
> protected="protwords.txt"/>
> <filter class="solr.SynonymGraphFilterFactory" expand="true"
> ignoreCase="true" synonyms="synonyms.txt"/>
> <filter class="solr.KStemFilterFactory"/>
> </analyzer>
> </fieldType>
>
> Solr version = 7.7.2
>
> Regards,
> Derek
>
> ----------------------
> CONFIDENTIALITY NOTICE
>
> This e-mail (including any attachments) may contain confidential and/or
> privileged information. If you are not the intended recipient or have
> received this e-mail in error, please inform the sender immediately and
> delete this e-mail (including any attachments) from your computer, and you
> must not use, disclose to anyone else or copy this e-mail (including any
> attachments), whether in whole or in part.
>
> This e-mail and any reply to it may be monitored for security, legal,
> regulatory compliance and/or other appropriate reasons.
>
>