You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Christiaan Fluit <ch...@aduna.biz> on 2004/12/20 16:32:05 UTC
determination of matching hits
Hello all,
I have a question regarding the determination of the set of matching
documents, in particular (I guess) related to the NOT operator.
In my case I have a document containing the terms A and B. When I query
for either A or for B, I get this document back, just as expected. Now
when I query for A -B, I once again get this document back. In other
words: this document matches both B and a query containing the clause
-B, which theoretically should never happen.
I've seen this happen with various keywords, sometimes with multiple
"conflicting" documents. In each case, the B query returned the document
with a very low relevance (e.g. 0.007...).
Based on these low relevancies and a quick peek in the Lucene code, I
strongly suspect that this is caused by rounding errors, as it seems to
me that floating point numbers are used to both express the membership
of a set as well as its score. Can somebody confirm this?
And if this is the case, is there a workaround to eliminate or at least
significantly suppress this problem? A colleague mentioned boosting
every term in a query, would this solve anything?
For most search engine-like applications, which order documents on
relevance, I think this problem is not a real issue since such
conflicting documents appear at the end of the result list and are not
likely to be seen by the user. However, in our case we have an
application which displays overlaps of entire result sets and these
documents show up very prominently (I can show screenshots if desired).
We have already been asked by customers to explain these results :)
FYI, in case it may be relevant: I'm still using Lucene 1.4.2. Every
document has the same set of five fields. The above queries are parsed
by MultiFieldQueryParser, using all five fields. I haven't touched the
default operator, but the queries A AND -B and A AND NOT B give the same
conflicting overlap in the result set.
Thanks in advance,
Christiaan Fluit
Aduna.biz
--
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: determination of matching hits
Posted by Christiaan Fluit <ch...@aduna.biz>.
ok, I feel a bit stupid now ;) Turns out this issue has been discussed a
while ago on both mailing lists and I even participated in one of
them... shame on me.
The problem is indeed in how MFQP parses my query: the query A -B becomes:
(text:A -text:B) (title:A -title:B) (path:A -path:B) (summary:A
-summary:B) (agent:A -agent:B)
whereas I intuitively expexted it to be evaluated as "A in any field and
not B in any field". When I use a normal QueryParser and let it use a
single field only, everything works as expected.
Browsing the lists archives I see that there were some efforts from
different people in solving this issue, but I'm a bit confused about the
final outcome. Was this solved in the MFQP in 1.4.3? If not, what
alternative implementation of MFPQ can I currently use best?
Kind regards,
Chris
--
Erik Hatcher wrote:
> Christian,
>
> Please simplify your situation. Use a plain TermQuery for "B" and see
> what is returned. Then use a simple BooleanQuery for "A -B". I suspect
> MultiFieldQueryParser is the culprit. What does the toString of the
> generated Query return? MFQP is known to be trouble, and an overhaul to
> it has been contributed recently.
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: determination of matching hits
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
Christian,
Please simplify your situation. Use a plain TermQuery for "B" and see
what is returned. Then use a simple BooleanQuery for "A -B". I
suspect MultiFieldQueryParser is the culprit. What does the toString
of the generated Query return? MFQP is known to be trouble, and an
overhaul to it has been contributed recently.
Erik
On Dec 20, 2004, at 10:32 AM, Christiaan Fluit wrote:
> Hello all,
>
> I have a question regarding the determination of the set of matching
> documents, in particular (I guess) related to the NOT operator.
>
> In my case I have a document containing the terms A and B. When I
> query for either A or for B, I get this document back, just as
> expected. Now when I query for A -B, I once again get this document
> back. In other words: this document matches both B and a query
> containing the clause -B, which theoretically should never happen.
>
> I've seen this happen with various keywords, sometimes with multiple
> "conflicting" documents. In each case, the B query returned the
> document with a very low relevance (e.g. 0.007...).
>
> Based on these low relevancies and a quick peek in the Lucene code, I
> strongly suspect that this is caused by rounding errors, as it seems
> to me that floating point numbers are used to both express the
> membership of a set as well as its score. Can somebody confirm this?
>
> And if this is the case, is there a workaround to eliminate or at
> least significantly suppress this problem? A colleague mentioned
> boosting every term in a query, would this solve anything?
>
> For most search engine-like applications, which order documents on
> relevance, I think this problem is not a real issue since such
> conflicting documents appear at the end of the result list and are not
> likely to be seen by the user. However, in our case we have an
> application which displays overlaps of entire result sets and these
> documents show up very prominently (I can show screenshots if
> desired). We have already been asked by customers to explain these
> results :)
>
> FYI, in case it may be relevant: I'm still using Lucene 1.4.2. Every
> document has the same set of five fields. The above queries are parsed
> by MultiFieldQueryParser, using all five fields. I haven't touched the
> default operator, but the queries A AND -B and A AND NOT B give the
> same conflicting overlap in the result set.
>
>
> Thanks in advance,
>
> Christiaan Fluit
> Aduna.biz
> --
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org