You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Jan Høydahl (Created JIRA)" <ji...@apache.org> on 2012/02/01 15:14:59 UTC
[jira] [Created] (SOLR-3085) Fix the dismax/edismax stopwords mm
issue
Fix the dismax/edismax stopwords mm issue
-----------------------------------------
Key: SOLR-3085
URL: https://issues.apache.org/jira/browse/SOLR-3085
Project: Solr
Issue Type: Bug
Components: search
Reporter: Jan Høydahl
Fix For: 3.6, 4.0
As discussed here http://search-lucene.com/m/Wr7iz1a95jx and here http://search-lucene.com/m/Yne042qEyCq1 and here http://search-lucene.com/m/RfAp82nSsla DisMax has an issue with stopwords if not all fields used in QF have exactly same stopword lists.
Typical solution is to not use stopwords or harmonize stopword lists across all fields in your QF, or relax the MM to a lower percentag. Sometimes these are not acceptable workarounds, and we should find a better solution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3085) Fix the dismax/edismax stopwords mm
issue
Posted by "Jonathan Rochkind (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251828#comment-13251828 ]
Jonathan Rochkind commented on SOLR-3085:
-----------------------------------------
Hoss says: i have a nagging feeling that there are non-stopword cases that would be indistinguishable (to the parser) from this type of stopword case, and thus would also trigger this logic undesirably, but i can't articulate what they might be off the top of my head.
Indeed there are, pretty much anything where analysis differs between two fields in a way that can effect number of tokens produced. Punctuation stripping can sometimes do this, and I ran into such a case in my real world use. More info http://bibwild.wordpress.com/2011/06/15/more-dismax-gotchas-varying-field-analysis-and-mm/
This is a difficult problem to fix in the general case, at one point I think there was a solr listserv discussion where I tried to brainstorm general case solutions, but they were all shot down by people who knew more about Solr than me. :) I can't find the archive of that discussion now though.
> Fix the dismax/edismax stopwords mm issue
> -----------------------------------------
>
> Key: SOLR-3085
> URL: https://issues.apache.org/jira/browse/SOLR-3085
> Project: Solr
> Issue Type: Bug
> Components: query parsers
> Reporter: Jan Høydahl
> Labels: MinimumShouldMatch, dismax, stopwords
> Fix For: 4.0
>
>
> As discussed here http://search-lucene.com/m/Wr7iz1a95jx and here http://search-lucene.com/m/Yne042qEyCq1 and here http://search-lucene.com/m/RfAp82nSsla DisMax has an issue with stopwords if not all fields used in QF have exactly same stopword lists.
> Typical solution is to not use stopwords or harmonize stopword lists across all fields in your QF, or relax the MM to a lower percentag. Sometimes these are not acceptable workarounds, and we should find a better solution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3085) Fix the dismax/edismax stopwords mm
issue
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man updated SOLR-3085:
---------------------------
Fix Version/s: (was: 4.0)
removing fixVersion=4.0 since there is no evidence that anyone is currently working on this issue. (this can certainly be revisited if volunteers step forward)
> Fix the dismax/edismax stopwords mm issue
> -----------------------------------------
>
> Key: SOLR-3085
> URL: https://issues.apache.org/jira/browse/SOLR-3085
> Project: Solr
> Issue Type: Bug
> Components: query parsers
> Reporter: Jan Høydahl
> Labels: MinimumShouldMatch, dismax, stopwords
>
> As discussed here http://search-lucene.com/m/Wr7iz1a95jx and here http://search-lucene.com/m/Yne042qEyCq1 and here http://search-lucene.com/m/RfAp82nSsla DisMax has an issue with stopwords if not all fields used in QF have exactly same stopword lists.
> Typical solution is to not use stopwords or harmonize stopword lists across all fields in your QF, or relax the MM to a lower percentag. Sometimes these are not acceptable workarounds, and we should find a better solution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3085) Fix the dismax/edismax stopwords mm
issue
Posted by "Jan Høydahl (Commented JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198347#comment-13198347 ]
Jan Høydahl commented on SOLR-3085:
-----------------------------------
You're right that technically it's not marked as required, but in the context of this "feature" we're discussing, the reason why people get 0 hits is that mm=100%, counted from all (SHOULD) clauses. And that means effectively that alltags:the is required.
What James suggested, and what most people tricked by this "feature" expects, is that if "the" is a stopword for one of the qf fields, it becomes optional in some way.
So how can we get that end result? First we need a way to safely detect that we're in this scenario, perhaps by inspecting whether each DisMax clause contains a field query for every field listed in QF. If one or more is missing, we can assume that the query term is a stopword in one or more of the fields. Then, one way may be to subtract the MM count accordingly, so that in our case above, when we detect that the DisMax clause for "the" does not contain "title_en", we do mm=mm-1 which will give us an MM of 1 instead of 2 and we'll get hits. This is probably the easiest solution.
Another way would be to keep mm as is, and move the affected clause out of the BooleanQuery and add it as a BoostQuery instead?
This behavior should be parameter driven, e.g. {{&mm.sw=false}} reading "Minimum should match does not require Stop Words"
> Fix the dismax/edismax stopwords mm issue
> -----------------------------------------
>
> Key: SOLR-3085
> URL: https://issues.apache.org/jira/browse/SOLR-3085
> Project: Solr
> Issue Type: Bug
> Components: search
> Reporter: Jan Høydahl
> Labels: MinimumShouldMatch, dismax, stopwords
> Fix For: 3.6, 4.0
>
>
> As discussed here http://search-lucene.com/m/Wr7iz1a95jx and here http://search-lucene.com/m/Yne042qEyCq1 and here http://search-lucene.com/m/RfAp82nSsla DisMax has an issue with stopwords if not all fields used in QF have exactly same stopword lists.
> Typical solution is to not use stopwords or harmonize stopword lists across all fields in your QF, or relax the MM to a lower percentag. Sometimes these are not acceptable workarounds, and we should find a better solution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3085) Fix the dismax/edismax stopwords mm
issue
Posted by "Hoss Man (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198104#comment-13198104 ]
Hoss Man commented on SOLR-3085:
--------------------------------
bq. So we get a required DisMax Query for alltags:the which does not match any docs.
I think you are missreading that output...
{code{
+( ( DisjunctionMaxQuery((alltags:the)~0.01)
DisjunctionMaxQuery((title_en:contract | alltags:contract)~0.01)
)~2
)
{code}
The "DisjunctionMaxQuery((alltags:the)~0.01)" clause is not required in that query. it is one of two SHOULD clauses in a boolean query, and becomes subject to the same "mm" rule. both clauses in that BooleanQuery are already SHOULD clauses, so i don't know what it would mean to make then more "optional".
> Fix the dismax/edismax stopwords mm issue
> -----------------------------------------
>
> Key: SOLR-3085
> URL: https://issues.apache.org/jira/browse/SOLR-3085
> Project: Solr
> Issue Type: Bug
> Components: search
> Reporter: Jan Høydahl
> Labels: MinimumShouldMatch, dismax, stopwords
> Fix For: 3.6, 4.0
>
>
> As discussed here http://search-lucene.com/m/Wr7iz1a95jx and here http://search-lucene.com/m/Yne042qEyCq1 and here http://search-lucene.com/m/RfAp82nSsla DisMax has an issue with stopwords if not all fields used in QF have exactly same stopword lists.
> Typical solution is to not use stopwords or harmonize stopword lists across all fields in your QF, or relax the MM to a lower percentag. Sometimes these are not acceptable workarounds, and we should find a better solution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3085) Fix the dismax/edismax stopwords mm
issue
Posted by "Jan Høydahl (Commented JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197854#comment-13197854 ]
Jan Høydahl commented on SOLR-3085:
-----------------------------------
In this thread http://search-lucene.com/m/Tzktd1a95jx James Dyer suggests:
bq. I do wonder...what if (e)dismax had a flag you could set that would tell it that if any analyzers removed a term, then that term would become optional for any fields for which it remained? I'm not sure what the development effort would perhaps it would be a nice way to circumvent this problem in a future release...
I like the suggestion. Would this be possible?
Take as example this parsed query (q=the contract&qf=alltags title_en&mm=100%defType=edismax)
{noformat}
+((DisjunctionMaxQuery((alltags:the)~0.01) DisjunctionMaxQuery((title_en:contract | alltags:contract)~0.01))~2)
{noformat}
The field "alltags" does not use stopwords, but "title_en" does. So we get a required DisMax Query for alltags:the which does not match any docs. Is it possible in the (e)DisMax code to detect this and make the first DisMax query optional?
> Fix the dismax/edismax stopwords mm issue
> -----------------------------------------
>
> Key: SOLR-3085
> URL: https://issues.apache.org/jira/browse/SOLR-3085
> Project: Solr
> Issue Type: Bug
> Components: search
> Reporter: Jan Høydahl
> Labels: MinimumShouldMatch, dismax, stopwords
> Fix For: 3.6, 4.0
>
>
> As discussed here http://search-lucene.com/m/Wr7iz1a95jx and here http://search-lucene.com/m/Yne042qEyCq1 and here http://search-lucene.com/m/RfAp82nSsla DisMax has an issue with stopwords if not all fields used in QF have exactly same stopword lists.
> Typical solution is to not use stopwords or harmonize stopword lists across all fields in your QF, or relax the MM to a lower percentag. Sometimes these are not acceptable workarounds, and we should find a better solution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3085) Fix the dismax/edismax stopwords mm
issue
Posted by "Hoss Man (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198361#comment-13198361 ]
Hoss Man commented on SOLR-3085:
--------------------------------
bq. Then, one way may be to subtract the MM count accordingly, so that in our case above, when we detect that the DisMax clause for "the" does not contain "title_en", we do mm=mm-1 which will give us an MM of 1 instead of 2 and we'll get hits. This is probably the easiest solution.
that wouldn't make any sense ... in your example that would result in the query matching every doc containing "alltags:the" (or "title_en:contract", or "alltags:contract") which hardly seems like what the user is likely to expect if they used mm=100% (with or w/o a "mm.sw=false" param)
bq. Another way would be to keep mm as is, and move the affected clause out of the BooleanQuery and add it as a BoostQuery instead?
something like that might work .. but i haven't thought it through very hard ... i have a nagging feeling that there are non-stopword cases that would be indistinguishable (to the parser) from this type of stopword case, and thus would also trigger this logic undesirably, but i can't articulate what they might be off the top of my head.
> Fix the dismax/edismax stopwords mm issue
> -----------------------------------------
>
> Key: SOLR-3085
> URL: https://issues.apache.org/jira/browse/SOLR-3085
> Project: Solr
> Issue Type: Bug
> Components: search
> Reporter: Jan Høydahl
> Labels: MinimumShouldMatch, dismax, stopwords
> Fix For: 3.6, 4.0
>
>
> As discussed here http://search-lucene.com/m/Wr7iz1a95jx and here http://search-lucene.com/m/Yne042qEyCq1 and here http://search-lucene.com/m/RfAp82nSsla DisMax has an issue with stopwords if not all fields used in QF have exactly same stopword lists.
> Typical solution is to not use stopwords or harmonize stopword lists across all fields in your QF, or relax the MM to a lower percentag. Sometimes these are not acceptable workarounds, and we should find a better solution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3085) Fix the dismax/edismax stopwords mm
issue
Posted by "Jan Høydahl (Commented JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251915#comment-13251915 ]
Jan Høydahl commented on SOLR-3085:
-----------------------------------
Good article, Jonathan. I agree that it may be very hard to completely fix this 100%, but an option to at least avoid the most common frustrations around this would be welcome, even if it is only fixing the symptoms. I.e. having a configuration parameter relaxMmHack=true which relaxes mm if one of the fields yields fewer tokens than the others would be fixing the felt effect of the problem for many peo, simply by adding a param.
> Fix the dismax/edismax stopwords mm issue
> -----------------------------------------
>
> Key: SOLR-3085
> URL: https://issues.apache.org/jira/browse/SOLR-3085
> Project: Solr
> Issue Type: Bug
> Components: query parsers
> Reporter: Jan Høydahl
> Labels: MinimumShouldMatch, dismax, stopwords
> Fix For: 4.0
>
>
> As discussed here http://search-lucene.com/m/Wr7iz1a95jx and here http://search-lucene.com/m/Yne042qEyCq1 and here http://search-lucene.com/m/RfAp82nSsla DisMax has an issue with stopwords if not all fields used in QF have exactly same stopword lists.
> Typical solution is to not use stopwords or harmonize stopword lists across all fields in your QF, or relax the MM to a lower percentag. Sometimes these are not acceptable workarounds, and we should find a better solution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3085) Fix the dismax/edismax stopwords mm
issue
Posted by "Jan Høydahl (Commented JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199147#comment-13199147 ]
Jan Høydahl commented on SOLR-3085:
-----------------------------------
bq. i have a nagging feeling that there are non-stopword cases that would be indistinguishable (to the parser) from this type of stopword case, and thus would also trigger this logic undesirably, but i can't articulate what they might be off the top of my head.
A potential difficult one is this multi language example: {{&qf=title_no title_en tags}}. Each of these fields may have their separate stopwords list, say title_no has a stopword "men" (norwegian for but) and title_en has stopword "the". Then we query {{q=the men}}. The user expectation would be that it would return ENGLISH docs matching "men", since "the" is an english stopword.
Today we'd get:
{noformat}
+((DisjunctionMaxQuery((title_no:the | tags:the)~0.01) DisjunctionMaxQuery((title_en:men | tags:men)~0.01))~2)
{noformat}
In this case with mm=100% we'd likely get 0 hits, given that "the" is not common in either of title_no or tags. However, the parser cannot know whether the user's real information need is "the" or "men" - since both are stopwords for different fields.
Now, all DisMax clauses in this example have had at least one stopword pruned, so using the "mm decrement" strategy would change mm from 2 to 0 which would turn this into an OR query - and of course return results. This is a compromise, so a better option in this special case would probably be to use eDismax's "smart" conditional stopword removal [1], but that requires change of fieldType.
The "convert to boost query" approach would only work when we have at least one clause without stop words, since we cannot query ONLY with bq. Say two of my four query terms {{q=the best cheap holiday}} are stop words, and mm=100%. So we remove the two stop clauses from the BooleanQuery and reduce mm accordingly from 4 (100%) to 2, and add the two stop clauses as BQs. This approach would also work for mm<100% cases, since we only count mm clauses from the non-stop clauses.
----
[1] For the special case of all clauses being stop clauses, eDisMax's existing "smart" conditional stopword handling could perhaps be another solution? For those unfamiliar with it, you can specify {{&stopwords=true}} (which is the default) and eDismax will remove stopwords for you instead of letting Analysis do it. It requires that you don't have StopFilterFactory in your Analysis. Now, if ALL query terms are stopwords, disMax will not remove them, to support queries like "Who is the who?". (Q: How does edismax pick up which stopword dicationary(ies) to use?). It's of no use to those removing stop-words in their "index" analysis though.
> Fix the dismax/edismax stopwords mm issue
> -----------------------------------------
>
> Key: SOLR-3085
> URL: https://issues.apache.org/jira/browse/SOLR-3085
> Project: Solr
> Issue Type: Bug
> Components: search
> Reporter: Jan Høydahl
> Labels: MinimumShouldMatch, dismax, stopwords
> Fix For: 3.6, 4.0
>
>
> As discussed here http://search-lucene.com/m/Wr7iz1a95jx and here http://search-lucene.com/m/Yne042qEyCq1 and here http://search-lucene.com/m/RfAp82nSsla DisMax has an issue with stopwords if not all fields used in QF have exactly same stopword lists.
> Typical solution is to not use stopwords or harmonize stopword lists across all fields in your QF, or relax the MM to a lower percentag. Sometimes these are not acceptable workarounds, and we should find a better solution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3085) Fix the dismax/edismax stopwords mm
issue
Posted by "Jan Høydahl (Commented JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229083#comment-13229083 ]
Jan Høydahl commented on SOLR-3085:
-----------------------------------
@Bill, since this is a bit off topic, I moved your "loophole" to SOLR-3243. It is certainly something that is dangerous and I cannot see a single usecase for allowing an un-fielded range! Good catch.
> Fix the dismax/edismax stopwords mm issue
> -----------------------------------------
>
> Key: SOLR-3085
> URL: https://issues.apache.org/jira/browse/SOLR-3085
> Project: Solr
> Issue Type: Bug
> Components: search
> Reporter: Jan Høydahl
> Labels: MinimumShouldMatch, dismax, stopwords
> Fix For: 3.6, 4.0
>
>
> As discussed here http://search-lucene.com/m/Wr7iz1a95jx and here http://search-lucene.com/m/Yne042qEyCq1 and here http://search-lucene.com/m/RfAp82nSsla DisMax has an issue with stopwords if not all fields used in QF have exactly same stopword lists.
> Typical solution is to not use stopwords or harmonize stopword lists across all fields in your QF, or relax the MM to a lower percentag. Sometimes these are not acceptable workarounds, and we should find a better solution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3085) Fix the dismax/edismax stopwords mm
issue
Posted by "Jan Høydahl (Commented JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228351#comment-13228351 ]
Jan Høydahl commented on SOLR-3085:
-----------------------------------
How about we add a new fieldType to exampel schema.xml {{text_general_smartstopwords}} and in it document how to use eDisMax to conditionally remove stopwords on query side only?
> Fix the dismax/edismax stopwords mm issue
> -----------------------------------------
>
> Key: SOLR-3085
> URL: https://issues.apache.org/jira/browse/SOLR-3085
> Project: Solr
> Issue Type: Bug
> Components: search
> Reporter: Jan Høydahl
> Labels: MinimumShouldMatch, dismax, stopwords
> Fix For: 3.6, 4.0
>
>
> As discussed here http://search-lucene.com/m/Wr7iz1a95jx and here http://search-lucene.com/m/Yne042qEyCq1 and here http://search-lucene.com/m/RfAp82nSsla DisMax has an issue with stopwords if not all fields used in QF have exactly same stopword lists.
> Typical solution is to not use stopwords or harmonize stopword lists across all fields in your QF, or relax the MM to a lower percentag. Sometimes these are not acceptable workarounds, and we should find a better solution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3085) Fix the dismax/edismax stopwords mm
issue
Posted by "Bill Bell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228990#comment-13228990 ]
Bill Bell commented on SOLR-3085:
---------------------------------
We also found another loophole. If we send [* TO *] to edismax we also can bring down the server.
Some chars are not being escaped before being sent to SOLR. Eg I can send queries like this to SOLR by searching on ([* TO *] OR [* TO *] OR [* TO *]) in the search box - it took 72 seconds to return:
webapp=/solr path=/select params={d=160.9344&start=0&q=([*+TO+*]+OR+[*+TO+*]+OR+[*+TO+*])&pt=40.7146,-74.0071&qt=providersearchdist&wt=json&qq=city_state_lower:"new+york,+ny"&rows=20} hits=276442 status=0 QTime=72458
> Fix the dismax/edismax stopwords mm issue
> -----------------------------------------
>
> Key: SOLR-3085
> URL: https://issues.apache.org/jira/browse/SOLR-3085
> Project: Solr
> Issue Type: Bug
> Components: search
> Reporter: Jan Høydahl
> Labels: MinimumShouldMatch, dismax, stopwords
> Fix For: 3.6, 4.0
>
>
> As discussed here http://search-lucene.com/m/Wr7iz1a95jx and here http://search-lucene.com/m/Yne042qEyCq1 and here http://search-lucene.com/m/RfAp82nSsla DisMax has an issue with stopwords if not all fields used in QF have exactly same stopword lists.
> Typical solution is to not use stopwords or harmonize stopword lists across all fields in your QF, or relax the MM to a lower percentag. Sometimes these are not acceptable workarounds, and we should find a better solution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (SOLR-3085) Fix the dismax/edismax
stopwords mm issue
Posted by "Hoss Man (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198104#comment-13198104 ]
Hoss Man edited comment on SOLR-3085 at 2/2/12 12:12 AM:
---------------------------------------------------------
bq. So we get a required DisMax Query for alltags:the which does not match any docs.
I think you are missreading that output...
{code}
+( ( DisjunctionMaxQuery((alltags:the)~0.01)
DisjunctionMaxQuery((title_en:contract | alltags:contract)~0.01)
)~2
)
{code}
The "DisjunctionMaxQuery((alltags:the)~0.01)" clause is not required in that query. it is one of two SHOULD clauses in a boolean query, and becomes subject to the same "mm" rule. both clauses in that BooleanQuery are already SHOULD clauses, so i don't know what it would mean to make then more "optional".
was (Author: hossman):
bq. So we get a required DisMax Query for alltags:the which does not match any docs.
I think you are missreading that output...
{code{
+( ( DisjunctionMaxQuery((alltags:the)~0.01)
DisjunctionMaxQuery((title_en:contract | alltags:contract)~0.01)
)~2
)
{code}
The "DisjunctionMaxQuery((alltags:the)~0.01)" clause is not required in that query. it is one of two SHOULD clauses in a boolean query, and becomes subject to the same "mm" rule. both clauses in that BooleanQuery are already SHOULD clauses, so i don't know what it would mean to make then more "optional".
> Fix the dismax/edismax stopwords mm issue
> -----------------------------------------
>
> Key: SOLR-3085
> URL: https://issues.apache.org/jira/browse/SOLR-3085
> Project: Solr
> Issue Type: Bug
> Components: search
> Reporter: Jan Høydahl
> Labels: MinimumShouldMatch, dismax, stopwords
> Fix For: 3.6, 4.0
>
>
> As discussed here http://search-lucene.com/m/Wr7iz1a95jx and here http://search-lucene.com/m/Yne042qEyCq1 and here http://search-lucene.com/m/RfAp82nSsla DisMax has an issue with stopwords if not all fields used in QF have exactly same stopword lists.
> Typical solution is to not use stopwords or harmonize stopword lists across all fields in your QF, or relax the MM to a lower percentag. Sometimes these are not acceptable workarounds, and we should find a better solution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org