You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Ulrich Mayring <ul...@denic.de> on 2004/09/23 11:09:20 UTC
Strange search results with wildcard - Bug?
Hi all,
first, here's how to reproduce the problem:
Go to http://www.denic.de/en/special/index.jsp and enter "obscure
service" in the search field. You'll get 132 hits. Now enter "obscure
service*" - and you only get 1 hit.
The above website is running Lucene 1.3rc3, but I was able to reproduce
this locally with 1.4.1. Here are my local results with "controlled
pseudo" documents, perhaps you can see a pattern:
searching for "00700*" gets two documents:
"007001 action" and "007002 handle"
searching for "handle" gets two documents:
"007002 handle" and "011010 handle"
searching for "00700* handle" gets two documents:
"007002 handle" and "011010 handle"
But where is "007001 action"?
searching for "handle 00700*" gets two documents:
"007001 action" and "007002 handle"
But where is "001010 handle"?
We're using the MultiFieldQueryParser and the Snowball Stemmers, if that
makes any difference.
Many thanks in advance for any pointers,
Ulrich
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Strange search results with wildcard - Bug?
Posted by Morus Walter <mo...@tanto.de>.
Ulrich Mayring writes:
> Daniel Naber wrote:
> >
> > AND always refers to the terms on both sides, +/- only refers to the term
> > on the right. So "a AND b" -> "+a +b" is correct.
>
> *slap forehead* - you're right. Wasn't there something about operator
> precedence way back when.... ;-)
>
Yes. January. And it's still in bugzilla. :-(
But it would not make a difference in this case, since AND has higher
precedence, so
a OR b AND c
is
a OR (b AND c)
which is correctly done as
a (+b +c)
in boolean queries.
a +b +c is different, since it won't find documents containing only a.
Occurences of a only modify score in this case.
Morus
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Strange search results with wildcard - Bug?
Posted by Ulrich Mayring <ul...@denic.de>.
Daniel Naber wrote:
>
> AND always refers to the terms on both sides, +/- only refers to the term
> on the right. So "a AND b" -> "+a +b" is correct.
*slap forehead* - you're right. Wasn't there something about operator
precedence way back when.... ;-)
Anyway, thanks to my stupidity and the help on this list I now have some
cool debugging code and can ask better questions next time :)
Ulrich
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Strange search results with wildcard - Bug?
Posted by Daniel Naber <da...@t-online.de>.
On Thursday 23 September 2004 17:53, Ulrich Mayring wrote:
> field1:foo field2:bar AND field3:true
>
> turns into
>
> field1:foo +field2:bar +field3:true
AND always refers to the terms on both sides, +/- only refers to the term
on the right. So "a AND b" -> "+a +b" is correct.
Regards
Daniel
--
http://www.danielnaber.de
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Strange search results with wildcard - Bug?
Posted by Ulrich Mayring <ul...@denic.de>.
Ulrich Mayring wrote:
>
> If the user searches for "007001 handle", the MultiFieldQueryParser,
> which searches in the fields "title" and "contents", changes that query to:
>
> (title:007001 +title:handl) (contents:007001 +contents:handl)
Ok, I cleared this up, there was some invisible magic going on in the
code, sorry for the inconvenience. Anyway:
field1:foo field2:bar AND field3:true
turns into
field1:foo +field2:bar +field3:true
If I lose the "AND" and use a "+" instead, then everything works as
expected. Now, is this a bug or a feature that I haven't quite grasped? :)
Ulrich
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Strange search results with wildcard - Bug?
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Sep 23, 2004, at 11:00 AM, Ulrich Mayring wrote:
> Erik Hatcher wrote:
>> Look at AnalysisDemo referred to here:
>> http://wiki.apache.org/jakarta-lucene/AnalysisParalysis
>> Keep in mind that phrase queries do not support wildcards - they are
>> analyzed and any wildcard characters are likely stripped and cause
>> tokens to split.
>
> Ok, I did all that and identified a basic case:
>
> If the user searches for "007001 handle", the MultiFieldQueryParser,
> which searches in the fields "title" and "contents", changes that
> query to:
>
> (title:007001 +title:handl) (contents:007001 +contents:handl)
>
> So, actually it has nothing to do with the wildcard, the problem comes
> from the + modifier - where does it originate? Obviously, this way I
> can never find a document without the term "handle", but with the
> number 007001.
How are you using MultiFieldQueryParser? It's surely coming from that.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Strange search results with wildcard - Bug?
Posted by Ulrich Mayring <ul...@denic.de>.
Erik Hatcher wrote:
>
> Look at AnalysisDemo referred to here:
>
> http://wiki.apache.org/jakarta-lucene/AnalysisParalysis
>
> Keep in mind that phrase queries do not support wildcards - they are
> analyzed and any wildcard characters are likely stripped and cause
> tokens to split.
Ok, I did all that and identified a basic case:
If the user searches for "007001 handle", the MultiFieldQueryParser,
which searches in the fields "title" and "contents", changes that query to:
(title:007001 +title:handl) (contents:007001 +contents:handl)
So, actually it has nothing to do with the wildcard, the problem comes
from the + modifier - where does it originate? Obviously, this way I can
never find a document without the term "handle", but with the number 007001.
Kind regards,
Ulrich
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Strange search results with wildcard - Bug?
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Sep 23, 2004, at 5:49 AM, Morus Walter wrote:
> Ulrich Mayring writes:
>>
>> Will do, thank you very much. However, how do I get at the analyzed
>> form
>> of my terms?
>>
> Instanciate the analyzer, create a token stream feeding your input,
> loop over the tokens, output the results.
Look at AnalysisDemo referred to here:
http://wiki.apache.org/jakarta-lucene/AnalysisParalysis
Keep in mind that phrase queries do not support wildcards - they are
analyzed and any wildcard characters are likely stripped and cause
tokens to split.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Strange search results with wildcard - Bug?
Posted by Morus Walter <mo...@tanto.de>.
Ulrich Mayring writes:
>
> Will do, thank you very much. However, how do I get at the analyzed form
> of my terms?
>
Instanciate the analyzer, create a token stream feeding your input,
loop over the tokens, output the results.
Morus
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Strange search results with wildcard - Bug?
Posted by Ulrich Mayring <ul...@denic.de>.
Morus Walter wrote:
>
> Your number/handle samples look ok to me if the default operator is AND.
But it's OR ;-)
Using AND explicitly I get different results and using OR explicitly I
get the same results as documented.
> Note that wildcard expressions are not analyzed so if service is
> stemmed to anything different from service, it's not surprising that
> service* doesn't find it.
Ok, I didn't know that, but it makes sense. Perhaps the phenomenon on
the live pages is different from my local test installation. I was just
looking for a comparable case on our live pages, but the real problem is
in pages that I'm just developing locally and which look similar to the
number/handle example.
> I think you should look at a) what's the analyzed form of your terms
> and b) how does the rewritten query look like (there's a rewrite method
> for query that expands wildcard queries into basic queries).
Will do, thank you very much. However, how do I get at the analyzed form
of my terms?
Ulrich
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Strange search results with wildcard - Bug?
Posted by Morus Walter <mo...@tanto.de>.
Ulrich Mayring writes:
> Hi all,
>
> first, here's how to reproduce the problem:
>
> Go to http://www.denic.de/en/special/index.jsp and enter "obscure
> service" in the search field. You'll get 132 hits. Now enter "obscure
> service*" - and you only get 1 hit.
>
> The above website is running Lucene 1.3rc3, but I was able to reproduce
> this locally with 1.4.1. Here are my local results with "controlled
> pseudo" documents, perhaps you can see a pattern:
>
> searching for "00700*" gets two documents:
> "007001 action" and "007002 handle"
>
>
> searching for "handle" gets two documents:
> "007002 handle" and "011010 handle"
>
>
> searching for "00700* handle" gets two documents:
> "007002 handle" and "011010 handle"
> But where is "007001 action"?
>
>
> searching for "handle 00700*" gets two documents:
> "007001 action" and "007002 handle"
> But where is "001010 handle"?
>
>
> We're using the MultiFieldQueryParser and the Snowball Stemmers, if that
> makes any difference.
>
Your number/handle samples look ok to me if the default operator is AND.
Note that wildcard expressions are not analyzed so if service is
stemmed to anything different from service, it's not surprising that
service* doesn't find it.
I think you should look at a) what's the analyzed form of your terms
and b) how does the rewritten query look like (there's a rewrite method
for query that expands wildcard queries into basic queries).
HTH
Morus
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org