You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Christoph Kiehl <ck...@sulu3000.de> on 2003/02/11 21:13:46 UTC
Wildcard search and analyzers
Hi,
I was just faced with the problem of wildcard queries not being analyzed.
While it might be a good idea to have no real stemming for wildcard terms, I
think one should do some basic filtering like convert to lowercase and for
german replacing �, �, � with vowels.
For the english stemmer it might be a fix to simply lowercase these terms,
but with the german one it gets more complicated. I mean I could replace
these vowels manually, but that's a bad idea. IMHO we should separate
between real stemming and filtering, where filtering would also apply to
wildcard terms.
What do you think about that? Any better ideas?
Regards
Christoph
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: Wildcard search and analyzers
Posted by Tatu Saloranta <ta...@hypermall.net>.
On Tuesday 11 February 2003 13:13, Christoph Kiehl wrote:
> Hi,
>
> I was just faced with the problem of wildcard queries not being analyzed.
> While it might be a good idea to have no real stemming for wildcard terms,
> I think one should do some basic filtering like convert to lowercase and
> for german replacing ö, ü, ä with vowels.
Just curious... what would you replace those with? (I'm not saying there's no
need for that, there are many cases where one might want to do it, just
interested in this case)
> For the english stemmer it might be a fix to simply lowercase these terms,
> but with the german one it gets more complicated. I mean I could replace
> these vowels manually, but that's a bad idea. IMHO we should separate
> between real stemming and filtering, where filtering would also apply to
> wildcard terms.
> What do you think about that? Any better ideas?
I don't think there's a one-size-fits-all solution, as using same analyzer for
both normal and wildcard/prefix terms is unlikely to work well generally.
One reason is that it's common to use short terms for prefixes, and those
would be likely to be removed by stop word list.
One easy solution would be to add couple of flags to QueryParser, for the most
common cases (lowercasing at least).
Since I was thinking of doing some small refactoring to QueryParser class
(just making factory methods to replace new XxxQuery() constructors, so that
those can be overridden), I can also refactor wildcard/prefix query term
handling, so that it'll be easy to override handling (that is, one needs not
modify Lucene source code but it'll be easy to extend parser class and
redefine functionality)?
-+ Tatu +-
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: [PATCH] Refactoring QueryParser.jj, setLowercaseWildcardTerms()
Posted by Christoph Kiehl <ck...@sulu3000.de>.
D.L.B. wrote:
> Given that this is the case, I don't think it's possible to come up
> with a solution that will cover every case. That said, I believe it
> is still worthwhile to try to do something reasonable to cover most
> cases.
>
> The company I work for has public text searchable websites in the
> following languages: English, Danish, Spanish, French, Dutch,
> Norwegian, Finnish, and Swedish. The approach we took, as I
> mentioned in an earlier mail, was to only stem prefix and "suffix"
> queries (of the form *someText). In these cases, don't pass the
> wildcard character to the stemmer and only use the stemmed result if
> it is a single word.
> [...]
> It turns out that this wildcard policy works well for us -- the users
> tend to get the results they expect. Whatever solution falls out of
> this argument, I just wanted to mention what is working for us. I'm
> thinking that adding a suffix term notion, parallel to prefix term in
> QueryParser.jj, creating subclassable methods to handle these, maybe
> providing a subclass that performs the imperfect stemming solution
> mentioned above, might be enough to please a lot of users.
This might be the way to go. Perhaps we could extend this, and provide a
special flag like "%men*" or simply enclose the query in "" to prevent
automatic stemming. This way one would also be able to find "menigitis".
Christoph
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: [PATCH] Refactoring QueryParser.jj, setLowercaseWildcardTerms()
Posted by "D.L.B." <au...@rcn.com>.
On Thursday 13 February 2003 07:49 am, Christoph Kiehl wrote:
> Tatu Saloranta wrote:
> > - Stemming can only be done for prefix queries (what is stem of,
> > say, "hä*er"?), and even then it might not produce stem one would
> > want. For example, for prefix query "men*" might be 'stemmed' to
> > "man*", and user might be perplexed at why documents with
> > words like "meningitis" and "menstrual" did not match (ok, that is
> > a contrived example, but hope you get the idea).
>
> Good point. It's is really amazing how different and complex languages are
> ;)
Given that this is the case, I don't think it's possible to come up with a
solution that will cover every case. That said, I believe it is still
worthwhile to try to do something reasonable to cover most cases.
The company I work for has public text searchable websites in the following
languages: English, Danish, Spanish, French, Dutch, Norwegian, Finnish, and
Swedish. The approach we took, as I mentioned in an earlier mail, was to
only stem prefix and "suffix" queries (of the form *someText). In these
cases, don't pass the wildcard character to the stemmer and only use the
stemmed result if it is a single word.
We didn't have time to analyze all the stemming possibilities of each language
and how our wildcard policy might perform in all cases. Instead, we just
threw it out there and had the native speakers run their QA and see what
happened.
It turns out that this wildcard policy works well for us -- the users tend to
get the results they expect. Whatever solution falls out of this argument, I
just wanted to mention what is working for us. I'm thinking that adding a
suffix term notion, parallel to prefix term in QueryParser.jj, creating
subclassable methods to handle these, maybe providing a subclass that
performs the imperfect stemming solution mentioned above, might be enough to
please a lot of users.
DaveB
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: [PATCH] Refactoring QueryParser.jj, setLowercaseWildcardTerms()
Posted by Christoph Kiehl <ck...@sulu3000.de>.
Tatu Saloranta wrote:
> - Stemming can only be done for prefix queries (what is stem of,
> say, "h�*er"?), and even then it might not produce stem one would
> want. For example, for prefix query "men*" might be 'stemmed' to
> "man*", and user might be perplexed at why documents with
> words like "meningitis" and "menstrual" did not match (ok, that is
> a contrived example, but hope you get the idea).
Good point. It's is really amazing how different and complex languages are
;)
> In a way, you could think that user is doing "manual stemming", using
> a stem of a word with prefix query.
Yup, but for example the german word "M�llemann" is a surname so there is
nothing to stem. If you search for "m�llema*" now, you won't get any results
because "M�llemann" is indexed as "mollemann". Ok, I admit it would be
uncommon to search for "m�llema*" if you want to find occurences of
"M�llemann". But this is only an example.
> In case of german, if umlaut chars are typically converted, perhaps
> you could create a GermanQueryParser.java that just extends default
> query parser, and does necessary transformation for wildcard/prefix
> queries? Since there already exists separate language-dependant
> stemmers, this might make sense?
Yep, this would be worth a try. But I'm not sure if this really beats all
problems. I'm still trying to get a whole picture of the problem ;)
>> is also the problem with Umlauts (�,�,�) turned into vowels (a,o,u)
>> while indexing. An example: "H�user" is the plural of "Haus". If I
>> index "H�user" it is stemmed to "hau". If I do for example a search
>> for "h�us*" nothing is
>
> Not "haus"?
I meant "h�us*", but I admit it would be more natural searching for haus ;).
Perhaps I'm trying to find problems where there are none ;). But it really
depends on how you use Lucene.
>> and defaults to off. Hope you understand my problem ;)
>
> Yes I do... I don't even dare to think of problems finnish analyzer
> might have, with stemming. :-)
;)
A bit confused
Christoph
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: [PATCH] Refactoring QueryParser.jj, setLowercaseWildcardTerms()
Posted by Christoph Kiehl <ck...@sulu3000.de>.
maurits van wijland wrote:
> Hi all,
>
> Maybe it we should start using stemming in a different maner. Look at
> it from the perspective
> of queryexpansion. In case we store stems in a different table, we
> will not have this problem!
>
> So, each token in stored in the index as a term.
> Each term is stemmed with the appropriate stemmer
> Store each stem and unstemed term in a separate index.
>
> We could then, search using the terms entered, and firstfind all the
> terms that match the WildcardQuery. Next,you coulde use the terms
> found, and then stem them.
> From there, you retrieve all the terms related to that stem!
> Finally, search for documents with all terms retrieved.
This, might be an idea. But it would slow down everything by factor 3 if I
understand you correctly.
This problem is more complicated then I thought first. Hm, are we really the
first people on earth facing this problem? There must be a common way
solving this ;)
I tried to find out how google handles wildcard, but they seem to be
ignored.
Thoughtfully
Christoph
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: [PATCH] Refactoring QueryParser.jj, setLowercaseWildcardTerms()
Posted by maurits van wijland <m....@quicknet.nl>.
Hi all,
Maybe it we should start using stemming in a different maner. Look at it
from the perspective
of queryexpansion. In case we store stems in a different table, we will not
have this problem!
So, each token in stored in the index as a term.
Each term is stemmed with the appropriate stemmer
Store each stem and unstemed term in a separate index.
We could then, search using the terms entered, and firstfind all the terms
that match the WildcardQuery. Next,you coulde use the terms found, and then
stem them.
>From there, you retrieve all the terms related to that stem!
Finally, search for documents with all terms retrieved.
This would give an extra option for end users, turning query expansion on or
off.
Your thoughts, please.
kind regards,
Maurits.
----- Original Message -----
From: "Tatu Saloranta" <ta...@hypermall.net>
To: "Lucene Developers List" <lu...@jakarta.apache.org>
Sent: Thursday, February 13, 2003 2:43 AM
Subject: Re: [PATCH] Refactoring QueryParser.jj, setLowercaseWildcardTerms()
> On Wednesday 12 February 2003 11:39, Christoph Kiehl wrote:
> > Hi Doug,
> >
> > > Also, I think we should lowercase prefix and wildcard queries by
> ...
> > > wildcard searches. What do others think?
> >
> > For the StandardAnalyzer this might work, but for the GermanAnalyzer,
there
>
> Solving this problem should be easier after refactoring, just
> override 'getPrefixQuery()' and 'getWildcardQuery' (see below for one
possible
> idea of what could be done).
>
> Another possibility would be to have another property for enabling use of
same
> analyzer used for normal terms for wildcard/prefix queries.
>
> However, using typical analyzers is not something one usually wants to do
> for couple of reasons:
>
> - Wildcards are discarded by analyzer, so wildcard query will get broken
(ie.
> one needs wildcard-char - aware analyzer)
> - Stemming can only be done for prefix queries (what is stem of,
> say, "hä*er"?), and even then it might not produce stem one would
> want. For example, for prefix query "men*" might be 'stemmed' to
> "man*", and user might be perplexed at why documents with
> words like "meningitis" and "menstrual" did not match (ok, that is
> a contrived example, but hope you get the idea).
> In a way, you could think that user is doing "manual stemming", using
> a stem of a word with prefix query.
>
> In case of german, if umlaut chars are typically converted, perhaps you
could
> create a GermanQueryParser.java that just extends default query parser,
and
> does necessary transformation for wildcard/prefix queries? Since there
> already exists separate language-dependant stemmers, this might make
sense?
>
> > is also the problem with Umlauts (ä,ö,ü) turned into vowels (a,o,u)
while
> > indexing. An example: "Häuser" is the plural of "Haus". If I index
"Häuser"
> > it is stemmed to "hau". If I do for example a search for "häus*" nothing
is
>
> Not "haus"?
>
> > found, because "häus" is not stemmed. If I would analyze "häus*" I
should
> > get "hau*". The problem is, that now you do not only get "Häuser" but
also
> > "Haus" as result. But I think it is better to get more results than no
> > result. This is perhaps a special problem with the GermanAnalyzer. May
be
> > there could be an option to use the Analyzer also for wildcard queries.
So
> > I can turn it on in my case and defaults to off.
> > Hope you understand my problem ;)
>
> Yes I do... I don't even dare to think of problems finnish analyzer might
> have, with stemming. :-)
>
> -+ Tatu +-
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: [PATCH] Refactoring QueryParser.jj, setLowercaseWildcardTerms()
Posted by Tatu Saloranta <ta...@hypermall.net>.
On Wednesday 12 February 2003 11:39, Christoph Kiehl wrote:
> Hi Doug,
>
> > Also, I think we should lowercase prefix and wildcard queries by
...
> > wildcard searches. What do others think?
>
> For the StandardAnalyzer this might work, but for the GermanAnalyzer, there
Solving this problem should be easier after refactoring, just
override 'getPrefixQuery()' and 'getWildcardQuery' (see below for one possible
idea of what could be done).
Another possibility would be to have another property for enabling use of same
analyzer used for normal terms for wildcard/prefix queries.
However, using typical analyzers is not something one usually wants to do
for couple of reasons:
- Wildcards are discarded by analyzer, so wildcard query will get broken (ie.
one needs wildcard-char - aware analyzer)
- Stemming can only be done for prefix queries (what is stem of,
say, "hä*er"?), and even then it might not produce stem one would
want. For example, for prefix query "men*" might be 'stemmed' to
"man*", and user might be perplexed at why documents with
words like "meningitis" and "menstrual" did not match (ok, that is
a contrived example, but hope you get the idea).
In a way, you could think that user is doing "manual stemming", using
a stem of a word with prefix query.
In case of german, if umlaut chars are typically converted, perhaps you could
create a GermanQueryParser.java that just extends default query parser, and
does necessary transformation for wildcard/prefix queries? Since there
already exists separate language-dependant stemmers, this might make sense?
> is also the problem with Umlauts (ä,ö,ü) turned into vowels (a,o,u) while
> indexing. An example: "Häuser" is the plural of "Haus". If I index "Häuser"
> it is stemmed to "hau". If I do for example a search for "häus*" nothing is
Not "haus"?
> found, because "häus" is not stemmed. If I would analyze "häus*" I should
> get "hau*". The problem is, that now you do not only get "Häuser" but also
> "Haus" as result. But I think it is better to get more results than no
> result. This is perhaps a special problem with the GermanAnalyzer. May be
> there could be an option to use the Analyzer also for wildcard queries. So
> I can turn it on in my case and defaults to off.
> Hope you understand my problem ;)
Yes I do... I don't even dare to think of problems finnish analyzer might
have, with stemming. :-)
-+ Tatu +-
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
RE : [PATCH] Refactoring QueryParser.jj, setLowercaseWildcardTerms()
Posted by Martin Sevigny <se...@ajlsm.com>.
Hi,
> > Also, I think we should lowercase prefix and wildcard queries by
> > default. This would fix one of the most frequently
> reported problems.
> > Yes, it might also break folks who currently do case-sensitive
> > wildcard queries, but I suspect they are far fewer than
> those who will
> > continue to complain about the default case-sensitivity of wildcard
> > searches. What do others think?
>
> For the StandardAnalyzer this might work, but for the
> GermanAnalyzer, there is also the problem with Umlauts
> (ä,ö,ü) turned into vowels (a,o,u) while indexing. An
> example: "Häuser" is the plural of "Haus". If I index
> "Häuser" it is stemmed to "hau". If I do for example a search
> for "häus*" nothing is found, because "häus" is not stemmed.
> If I would analyze "häus*" I should get "hau*". The problem
> is, that now you do not only get "Häuser" but also "Haus" as
> result. But I think it is better to get more results than no
> result. This is perhaps a special problem with the
> GermanAnalyzer. May be there could be an option to use the
> Analyzer also for wildcard queries. So I can turn it on in my
> case and defaults to off. Hope you understand my problem ;)
I second that, it is true for many languages where a "standard" analyzer
will most of the time do more than removing uppercase, it will remove
"diacritics" like in the above example. Along with possibly stemming.
Lucene is a wonderful tool for building i18n-ready search engines, let's
not forget it ;-)
Martin Sévigny
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: [PATCH] Refactoring QueryParser.jj, setLowercaseWildcardTerms()
Posted by Christoph Kiehl <ck...@sulu3000.de>.
Hi Doug,
> Also, I think we should lowercase prefix and wildcard queries by
> default. This would fix one of the most frequently reported problems.
> Yes, it might also break folks who currently do case-sensitive
> wildcard queries, but I suspect they are far fewer than those who
> will continue to complain about the default case-sensitivity of
> wildcard searches. What do others think?
For the StandardAnalyzer this might work, but for the GermanAnalyzer, there
is also the problem with Umlauts (�,�,�) turned into vowels (a,o,u) while
indexing. An example: "H�user" is the plural of "Haus". If I index "H�user"
it is stemmed to "hau". If I do for example a search for "h�us*" nothing is
found, because "h�us" is not stemmed. If I would analyze "h�us*" I should
get "hau*". The problem is, that now you do not only get "H�user" but also
"Haus" as result. But I think it is better to get more results than no
result. This is perhaps a special problem with the GermanAnalyzer. May be
there could be an option to use the Analyzer also for wildcard queries. So I
can turn it on in my case and defaults to off.
Hope you understand my problem ;)
Regards
Christoph
ps: Lucene otherwise really rocks ;)
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: [PATCH] Refactoring QueryParser.jj, setLowercaseWildcardTerms()
Posted by Matt Tucker <ma...@jivesoftware.com>.
Doug,
> Also, I think we should lowercase prefix and wildcard queries by
> default. This would fix one of the most frequently reported problems.
> Yes, it might also break folks who currently do case-sensitive wildcard
> queries, but I suspect they are far fewer than those who will continue
> to complain about the default case-sensitivity of wildcard searches.
> What do others think?
Seems like a good idea to me.
Regards,
-Matt
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: [PATCH] Refactoring QueryParser.jj, setLowercaseWildcardTerms()
Posted by Terry Steichen <te...@net-frame.com>.
> Also, I think we should lowercase prefix and wildcard queries by
> default. This would fix one of the most frequently reported problems.
> Yes, it might also break folks who currently do case-sensitive wildcard
> queries, but I suspect they are far fewer than those who will continue
> to complain about the default case-sensitivity of wildcard searches.
> What do others think?
+1
Terry
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: [PATCH] Refactoring QueryParser.jj, setLowercaseWildcardTerms()
Posted by Tatu Saloranta <ta...@hypermall.net>.
On Sunday 23 February 2003 21:41, Otis Gospodnetic wrote:
> Hello,
>
> --- Doug Cutting <cu...@lucene.com> wrote:
> > +1
> >
> > I like this approach of modifying the query parser through
> > subclassing.
> >
> > We should consider taking this approach further, e.g., perhaps by
> > making
> > addClause(), getFieldQuery() and getRangeQuery() into protected
> > methods,
> > so that folks can modify their behavior too. Thoughts?
>
> I don't see a problem with that. +1.
>
> > Also, I think we should lowercase prefix and wildcard queries by
> > default. This would fix one of the most frequently reported
> > problems.
...
> > to complain about the default case-sensitivity of wildcard searches.
> > What do others think?
>
> I'm pro lowercase! :)
> Tatu, are you okay with that?
Sure, no problem. I just wasn't sure what should be the default, so started
with backward compatibility. :-)
There are ways to make this even more reliable, for example analyzer could
have a method for checking if it always does lowercasing
('isLowerCaseFilter()' or such), but perhaps that'd be an overkill.
It's probably ok to default to lower-casing; deriving classes can override the
methods to change behaviour.
-+ Tatu +-
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: [PATCH] Refactoring QueryParser.jj, setLowercaseWildcardTerms()
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hello,
--- Doug Cutting <cu...@lucene.com> wrote:
> +1
>
> I like this approach of modifying the query parser through
> subclassing.
>
> We should consider taking this approach further, e.g., perhaps by
> making
> addClause(), getFieldQuery() and getRangeQuery() into protected
> methods,
> so that folks can modify their behavior too. Thoughts?
I don't see a problem with that. +1.
> Also, I think we should lowercase prefix and wildcard queries by
> default. This would fix one of the most frequently reported
> problems.
> Yes, it might also break folks who currently do case-sensitive
> wildcard
> queries, but I suspect they are far fewer than those who will
> continue
> to complain about the default case-sensitivity of wildcard searches.
> What do others think?
I'm pro lowercase! :)
Tatu, are you okay with that?
Otis
> Tatu Saloranta wrote:
> > (since this is the first patch I've done to Lucene, let me know if
> there are
> > some preferences or if I missed something...)
> >
> > I did some small refactorization to QueryParser.jj, to make it
> easier to
> > create subclasses that override actual query building (without
> modifying
> > actual parser part). I'll probably try to create my own
> MultiFieldQueryParser
> > with some changes (different weights for different fields, not
> necessarily
> > duplicating the whole query etc.), and these changes should make it
> possible
> > to do that without touching QueryParser.jj itself.
> >
> > I also added simple improvement to prefix/wildcard query handling;
> now it is
> > possible to set a property to enable automatic lowercasing of
> prefix/wildcard
> > terms (default is false to stay backwards compatible). This is not
> a complete
> > solution to wildcard query problems, but should help a bit.
> >
> > I ran unit tests (without errors), but I wasn't sure if there are
> unit tests
> > for QueryParser. Hopefully I didn't break anything; changes are
> fairly minor.
> >
> > Let me know if proposed changes make sense,
> >
> > -+ Tatu +-
__________________________________________________
Do you Yahoo!?
Yahoo! Tax Center - forms, calculators, tips, more
http://taxes.yahoo.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: [PATCH] Refactoring QueryParser.jj, setLowercaseWildcardTerms()
Posted by Doug Cutting <cu...@lucene.com>.
+1
I like this approach of modifying the query parser through subclassing.
We should consider taking this approach further, e.g., perhaps by making
addClause(), getFieldQuery() and getRangeQuery() into protected methods,
so that folks can modify their behavior too. Thoughts?
Also, I think we should lowercase prefix and wildcard queries by
default. This would fix one of the most frequently reported problems.
Yes, it might also break folks who currently do case-sensitive wildcard
queries, but I suspect they are far fewer than those who will continue
to complain about the default case-sensitivity of wildcard searches.
What do others think?
Doug
Tatu Saloranta wrote:
> (since this is the first patch I've done to Lucene, let me know if there are
> some preferences or if I missed something...)
>
> I did some small refactorization to QueryParser.jj, to make it easier to
> create subclasses that override actual query building (without modifying
> actual parser part). I'll probably try to create my own MultiFieldQueryParser
> with some changes (different weights for different fields, not necessarily
> duplicating the whole query etc.), and these changes should make it possible
> to do that without touching QueryParser.jj itself.
>
> I also added simple improvement to prefix/wildcard query handling; now it is
> possible to set a property to enable automatic lowercasing of prefix/wildcard
> terms (default is false to stay backwards compatible). This is not a complete
> solution to wildcard query problems, but should help a bit.
>
> I ran unit tests (without errors), but I wasn't sure if there are unit tests
> for QueryParser. Hopefully I didn't break anything; changes are fairly minor.
>
> Let me know if proposed changes make sense,
>
> -+ Tatu +-
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: [PATCH] Refactoring QueryParser.jj, setLowercaseWildcardTerms()
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hello Tatu,
I like these changes. The patch applied without any problems, and all
unit tests still pass. It would be good to add some unit tests that
exercise the new setLowercaseWildcardTerms(boolean) method. That
should be done by adding some tests to:
org.apache.lucene.queryParser.TestQueryParser
and/or
org.apache.lucene.search.Test*
Thanks,
Otis
--- Tatu Saloranta <ta...@hypermall.net> wrote:
> (since this is the first patch I've done to Lucene, let me know if
> there are
> some preferences or if I missed something...)
>
> I did some small refactorization to QueryParser.jj, to make it easier
> to
> create subclasses that override actual query building (without
> modifying
> actual parser part). I'll probably try to create my own
> MultiFieldQueryParser
> with some changes (different weights for different fields, not
> necessarily
> duplicating the whole query etc.), and these changes should make it
> possible
> to do that without touching QueryParser.jj itself.
>
> I also added simple improvement to prefix/wildcard query handling;
> now it is
> possible to set a property to enable automatic lowercasing of
> prefix/wildcard
> terms (default is false to stay backwards compatible). This is not a
> complete
> solution to wildcard query problems, but should help a bit.
>
> I ran unit tests (without errors), but I wasn't sure if there are
> unit tests
> for QueryParser. Hopefully I didn't break anything; changes are
> fairly minor.
>
> Let me know if proposed changes make sense,
>
> -+ Tatu +-
> > --- QueryParser.jj.orig 2003-02-11 21:25:19.000000000 -0700
> +++ QueryParser.jj 2003-02-11 21:29:03.000000000 -0700
> @@ -129,6 +129,12 @@
> Analyzer analyzer;
> String field;
> int phraseSlop = 0;
> + /**
> + * Whether terms of wildcard and prefix queries are to be
> automatically
> + * lowercased or not. Default is false (for backwards
> compatibility)
> + * to indicate those terms are not modified in any way.
> + */
> + boolean lowercaseWildcardTerms = false;
>
> /** Constructs a query parser.
> * @param field the default field for query terms.
> @@ -181,6 +187,12 @@
> return this.operator;
> }
>
> + public void setLowercaseWildcardTerms(boolean b) {
> + lowercaseWildcardTerms = b;
> + }
> +
> + public boolean getLowercaseWildcardTerms() { return
> lowercaseWildcardTerms; }
> +
> private void addClause(Vector clauses, int conj, int mods, Query
> q) {
> boolean required, prohibited;
>
> @@ -291,6 +303,103 @@
> inclusive);
> }
>
> + /**
> + * Factory method for generating query, given set of clauses.
> + * By default creates a boolean query composed of clauses passed
> in.
> + *
> + * Can be overridden by extending classes, to modify query being
> + * returned.
> + *
> + * @param clauses Vector that contains {@link BooleanClause}
> instances
> + * to join.
> + *
> + * @return Resulting {@link Query} object.
> + */
> + protected Query getBooleanQuery(Vector clauses)
> + {
> + BooleanQuery query = new BooleanQuery();
> + for (int i = 0; i < clauses.size(); i++) {
> + query.add((BooleanClause)clauses.elementAt(i));
> + }
> + return query;
> + }
> +
> + /**
> + * Factory method for generating a query. Called when parser
> + * parses an input term token that contains one or more wildcard
> + * characters (? and *), but is not a prefix term token (one
> + * that has just a single * character at the end)
> + *<p>
> + * Depending on settings, prefix term may be lower-cased
> + * automatically. It will not go through the default analyzer,
> + * however, since normal analyzers are unlikely to work properly
> + * with wildcard templates.
> + *<p>
> + * Can be overridden by extending classes, to provide custom
> handling for
> + * wild card queries (which may be necessary due to missing
> analyzer calls)
> + *
> + * @param field Name of the field query will use.
> + * @param termStr Term token that contains one or more wild card
> + * characters (? or *), but is not simple prefix term
> + *
> + * @return Resulting query build for the term
> + */
> + protected Query getWildcardQuery(String field, String termStr)
> + {
> + if (lowercaseWildcardTerms) {
> + termStr = termStr.toLowerCase();
> + }
> + Term t = new Term(field, termStr);
> + return new WildcardQuery(t);
> + }
> +
> + /**
> + * Factory method for generating a query (similar to
> + * (@link getWildcardQuery}). Called when parser parses an input
> term
> + * token that uses prefix notation; that is, contains a single '*'
> wild
> + * char character as it's last character. Since this is a special
> case
> + * of generic wild card term, and such a query can be optimized
> easily,
> + * this usually results in different query object.
> + *<p>
> + * Depending on settings, prefix term may be lower-cased
> + * automatically. It will not go through the default analyzer,
> + * however, since normal analyzers are unlikely to work properly
> + * with wildcard templates.
> + *<p>
> + * Can be overridden by extending classes, to provide custom
> handling for
> + * wild card queries (which may be necessary due to missing
> analyzer calls)
> + *
> + * @param field Name of the field query will use.
> + * @param termStr Term token to use for building term for the
> query
> + * (<b>without</b> trailing '*' character!)
> + *
> + * @return Resulting query build for the term
> + */
> + protected Query getPrefixQuery(String field, String termStr)
> + {
> + if (lowercaseWildcardTerms) {
> + termStr = termStr.toLowerCase();
> + }
> + Term t = new Term(field, termStr);
> + return new PrefixQuery(t);
> + }
> +
> + /**
> + * Factory method for generating a query (similar to
> + * (@link getWildcardQuery}). Called when parser parses
> + * an input term token that has the fuzzy suffix (~) appended.
> + *
> + * @param field Name of the field query will use.
> + * @param termStr Term token to use for building term for the
> query
> + *
> + * @return Resulting query build for the term
> + */
> + protected Query getFuzzyQuery(String field, String termStr)
> + {
> + Term t = new Term(field, termStr);
> + return new FuzzyQuery(t);
> + }
> +
> public static void main(String[] args) throws Exception {
> QueryParser qp = new QueryParser("field",
> new
> org.apache.lucene.analysis.SimpleAnalyzer());
> @@ -423,10 +532,7 @@
> if (clauses.size() == 1 && firstQuery != null)
> return firstQuery;
> else {
> - BooleanQuery query = new BooleanQuery();
> - for (int i = 0; i < clauses.size(); i++)
> - query.add((BooleanClause)clauses.elementAt(i));
> - return query;
> + return getBooleanQuery(clauses);
> }
> }
> }
> @@ -478,15 +584,16 @@
> [ <FUZZY> { fuzzy=true; } ]
> [ <CARAT> boost=<NUMBER> [ <FUZZY> { fuzzy=true; } ] ]
> {
> - if (wildcard)
> - q = new WildcardQuery(new Term(field, term.image));
> - else if (prefix)
> - q = new PrefixQuery(new Term(field, term.image.substring
> - (0, term.image.length()-1)));
> - else if (fuzzy)
> - q = new FuzzyQuery(new Term(field, term.image));
> - else
> + if (wildcard) {
> + q = getWildcardQuery(field, term.image);
> + } else if (prefix) {
> + q = getPrefixQuery(field, term.image.substring
> + (0, term.image.length()-1));
> + } else if (fuzzy) {
> + q = getFuzzyQuery(field, term.image);
> + } else {
> q = getFieldQuery(field, analyzer, term.image);
> + }
> }
> | ( <RANGEIN_START> (
> goop1=<RANGEIN_GOOP>|goop1=<RANGEIN_QUOTED> )
> [ <RANGEIN_TO> ] (
> goop2=<RANGEIN_GOOP>|goop2=<RANGEIN_QUOTED> )
> @@ -533,7 +640,11 @@
> try {
> f = Float.valueOf(boost.image).floatValue();
> }
> - catch (Exception ignored) { }
> + catch (Exception ignored) {
> + /* Should this be handled somehow? (defaults to "no boost", if
> + * boost number is invalid)
> + */
> + }
>
> // avoid boosting null queries, such as those caused by stop
> words
> if (q != null) {
>
> >
---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
__________________________________________________
Do you Yahoo!?
Yahoo! Tax Center - forms, calculators, tips, more
http://taxes.yahoo.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
[PATCH] Refactoring QueryParser.jj, setLowercaseWildcardTerms()
Posted by Tatu Saloranta <ta...@hypermall.net>.
(since this is the first patch I've done to Lucene, let me know if there are
some preferences or if I missed something...)
I did some small refactorization to QueryParser.jj, to make it easier to
create subclasses that override actual query building (without modifying
actual parser part). I'll probably try to create my own MultiFieldQueryParser
with some changes (different weights for different fields, not necessarily
duplicating the whole query etc.), and these changes should make it possible
to do that without touching QueryParser.jj itself.
I also added simple improvement to prefix/wildcard query handling; now it is
possible to set a property to enable automatic lowercasing of prefix/wildcard
terms (default is false to stay backwards compatible). This is not a complete
solution to wildcard query problems, but should help a bit.
I ran unit tests (without errors), but I wasn't sure if there are unit tests
for QueryParser. Hopefully I didn't break anything; changes are fairly minor.
Let me know if proposed changes make sense,
-+ Tatu +-