Wildcard with Words Ending in 'e'

[Gordon Rogers <g....@intrallect.com>]

Hi

I'm currently upgrading our version of intraLibrary from 1.3 to 1.9.1 (about time!) and have come 
across a failure in one of our wildcard query unit tests. Previously searching for the phrase peopl? 
would return results and now it doesn't. Further investigation has found that for any words I could 
think of that ended with an 'e' you can't use the single character wildcard to find the word.

Has anyone else seen this problem or know of a workaround?

Many thanks

Gordon


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Wildcard with Words Ending in 'e'

[Chris Hostetter <ho...@fucit.org>]

: now very useful when combined. I'm going to investigate other means of
: taking into account the wildcard queries and handling them in a
: different way. Rest assured that if I find a solution I'll post it to
: the user group.

the most straight forward solution is to have two indexed files -- one
that's stemmed which you use for normal "word granularity" searching, and
one that's unstemmed (probably whitespaced tokenized with a lowercase
filter) that you use when people give you wildcard like characters.

what i was refering to was the problem of trying to use wildcards on a
stemmed field -- there's no simple solution to that problem, because
untill you've looked up all the possible matching values, you don't know
what the possible stemms might be.




-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Wildcard with Words Ending in 'e'

[Gordon Rogers <g....@intrallect.com>]

Chris

Your shot in the dark was correct, I am using Porter Stemming. Thanks to your excellent response I 
understand now why wildcards and stemming are now very useful when combined. I'm going to 
investigate other means of taking into account the wildcard queries and handling them in a different 
way. Rest assured that if I find a solution I'll post it to the user group.

Gordon

Chris Hostetter wrote:
> I'm going to take a shot in the dark and guess that the field in question
> is analyzed using some form of Stemming which strips off trailing "e"
> characters (Porter Stemmer seems to do this).
> 
> In the past, there was a bug in the way WildcardQuery works, so that the
> metacharacter "?" matches 0 or 1 characters even though it's documented to
> match exactly one character -- so "peopl?" would match both "peopl" and
> peoplX" -- which wasn't correct.
> 
>    http://issues.apache.org/jira/browse/LUCENE-306
> 
> If i'm right about why you are encountering this problem, then your test
> case isn't really valid.  Either you should do you wildcard queries on a
> field that isn't stemmed, or you should take the stemming into account
> (somehow ... if you think of a good way i'm sure others would be
> interested) when building your wildcard query.
> 
> Keep in mind that trying to use wildcard queries on a field which uses
> porter stemming isn't generally a good idea regardless of wether you like
> the old behavior of "?" or not ... if your goal is to find documents
> containing words like "elephant" and "elephantine" so you search for
> "elephan*" you are going to miss any documents containing "elephant"
> because Porter Stemmer trims that to "eleph"
> 
> 
> -Hoss
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Wildcard with Words Ending in 'e'

[Chris Hostetter <ho...@fucit.org>]

I'm going to take a shot in the dark and guess that the field in question
is analyzed using some form of Stemming which strips off trailing "e"
characters (Porter Stemmer seems to do this).

In the past, there was a bug in the way WildcardQuery works, so that the
metacharacter "?" matches 0 or 1 characters even though it's documented to
match exactly one character -- so "peopl?" would match both "peopl" and
peoplX" -- which wasn't correct.

   http://issues.apache.org/jira/browse/LUCENE-306

If i'm right about why you are encountering this problem, then your test
case isn't really valid.  Either you should do you wildcard queries on a
field that isn't stemmed, or you should take the stemming into account
(somehow ... if you think of a good way i'm sure others would be
interested) when building your wildcard query.

Keep in mind that trying to use wildcard queries on a field which uses
porter stemming isn't generally a good idea regardless of wether you like
the old behavior of "?" or not ... if your goal is to find documents
containing words like "elephant" and "elephantine" so you search for
"elephan*" you are going to miss any documents containing "elephant"
because Porter Stemmer trims that to "eleph"


-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org