You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Sascha Szott <sz...@zib.de> on 2010/05/21 16:11:43 UTC
Wildcard queries
Hi folks,
what's the idea behind the fact that no text analysis (e.g. lowercasing)
is performed on wildcarded search terms?
In my context this behaviour seems to be counter-intuitive (I guess
that's the case in the majority of applications) and my application
needs to lowercase any input term before sending the HTTP request to my
Solr server.
Would it be easy to disable this behaviour in Solr (1.5)? I would like
to see a config parameter (per field type) that allows to disable this
"odd" behaviour if needed. To ensure backward compatibility the "odd"
behaviour should be the default anymore.
Am I missing any drawbacks?
Best,
Sascha
Re: Wildcard queries
Posted by Robert Muir <rc...@gmail.com>.
On Fri, May 21, 2010 at 10:40 AM, Smiley, David W. <ds...@mitre.org> wrote:
> Then why are you talking about stemming in the following example? We know stemming is problematic with wildcard searching. But casing... I argue not.
>
I just mentioned an example stemmer that properly case-folds this
german character.
Another tokenstream that does, is the unicode case-folding algorithm
[requires code dependent on ICU at the moment]
LowerCaseFilter is *not* unicode-compliant as far as casing goes.
toLowerCase is intended for display, not for case-insensitive
matching.
--
Robert Muir
rcmuir@gmail.com
Re: Wildcard queries
Posted by "Smiley, David W." <ds...@mitre.org>.
On May 21, 2010, at 10:35 AM, Robert Muir wrote:
> I honestly do not know the rationale behind this in Solr, except to
> say similar problems exist even if you reduce the scope to just
> casing:
Then why are you talking about stemming in the following example? We know stemming is problematic with wildcard searching. But casing... I argue not.
> For example, if you are using a german stemmer, it will case-fold ß to
> 'ss' (such that it will match SS).
>
> So doing some lowercasing at query-time will not correct the situation
> for that character, and furthermore it will be inconsistent with the
> '?' operator... (which only matches one character)
>
> On Fri, May 21, 2010 at 10:28 AM, Sascha Szott <sz...@zib.de> wrote:
>> Hi Robert,
>>
>> thanks, you're absolutely right. I should better refine my initial question
>> to: What's the idea behind the fact that no *lowercasing* is performed on
>> wildcarded search terms if the field in question contains a LowercaseFilter
>> in its associated field type definition?
>>
>> -Sascha
>>
>> Robert Muir wrote:
>>>
>>> we can use stemming as an example:
>>>
>>> lets say your query is c?ns?st?nt?y
>>>
>>> how will this match "consistently", which the porter stemmer
>>> transforms to 'consistent'.
>>> furthermore, note that i replaced the vowels with ?'s here. The porter
>>> stemmer doesnt just rip stuff off the end, but attempts to guess
>>> syllables as part of the process, so it cannot possibly work.
>>>
>>> the only way it would work in this situation would be if you formed
>>> permutations of all the possible words this wildcard would match, and
>>> then did analysis on each form, and searched on all stems.
>>>
>>> but, this is impossible, since the * operator allows an infinite language.
>>>
>>> On Fri, May 21, 2010 at 10:11 AM, Sascha Szott<sz...@zib.de> wrote:
>>>>
>>>> Hi folks,
>>>>
>>>> what's the idea behind the fact that no text analysis (e.g. lowercasing)
>>>> is
>>>> performed on wildcarded search terms?
>>>>
>>>> In my context this behaviour seems to be counter-intuitive (I guess
>>>> that's
>>>> the case in the majority of applications) and my application needs to
>>>> lowercase any input term before sending the HTTP request to my Solr
>>>> server.
>>>>
>>>> Would it be easy to disable this behaviour in Solr (1.5)? I would like to
>>>> see a config parameter (per field type) that allows to disable this "odd"
>>>> behaviour if needed. To ensure backward compatibility the "odd" behaviour
>>>> should be the default anymore.
>>>>
>>>> Am I missing any drawbacks?
>>>>
>>>> Best,
>>>> Sascha
>>
>>
>>
>>
>
>
>
> --
> Robert Muir
> rcmuir@gmail.com
Re: Wildcard queries
Posted by Robert Muir <rc...@gmail.com>.
I honestly do not know the rationale behind this in Solr, except to
say similar problems exist even if you reduce the scope to just
casing:
For example, if you are using a german stemmer, it will case-fold ß to
'ss' (such that it will match SS).
So doing some lowercasing at query-time will not correct the situation
for that character, and furthermore it will be inconsistent with the
'?' operator... (which only matches one character)
On Fri, May 21, 2010 at 10:28 AM, Sascha Szott <sz...@zib.de> wrote:
> Hi Robert,
>
> thanks, you're absolutely right. I should better refine my initial question
> to: What's the idea behind the fact that no *lowercasing* is performed on
> wildcarded search terms if the field in question contains a LowercaseFilter
> in its associated field type definition?
>
> -Sascha
>
> Robert Muir wrote:
>>
>> we can use stemming as an example:
>>
>> lets say your query is c?ns?st?nt?y
>>
>> how will this match "consistently", which the porter stemmer
>> transforms to 'consistent'.
>> furthermore, note that i replaced the vowels with ?'s here. The porter
>> stemmer doesnt just rip stuff off the end, but attempts to guess
>> syllables as part of the process, so it cannot possibly work.
>>
>> the only way it would work in this situation would be if you formed
>> permutations of all the possible words this wildcard would match, and
>> then did analysis on each form, and searched on all stems.
>>
>> but, this is impossible, since the * operator allows an infinite language.
>>
>> On Fri, May 21, 2010 at 10:11 AM, Sascha Szott<sz...@zib.de> wrote:
>>>
>>> Hi folks,
>>>
>>> what's the idea behind the fact that no text analysis (e.g. lowercasing)
>>> is
>>> performed on wildcarded search terms?
>>>
>>> In my context this behaviour seems to be counter-intuitive (I guess
>>> that's
>>> the case in the majority of applications) and my application needs to
>>> lowercase any input term before sending the HTTP request to my Solr
>>> server.
>>>
>>> Would it be easy to disable this behaviour in Solr (1.5)? I would like to
>>> see a config parameter (per field type) that allows to disable this "odd"
>>> behaviour if needed. To ensure backward compatibility the "odd" behaviour
>>> should be the default anymore.
>>>
>>> Am I missing any drawbacks?
>>>
>>> Best,
>>> Sascha
>
>
>
>
--
Robert Muir
rcmuir@gmail.com
Re: Wildcard queries
Posted by Sascha Szott <sz...@zib.de>.
Hi Robert,
thanks, you're absolutely right. I should better refine my initial
question to: What's the idea behind the fact that no *lowercasing* is
performed on wildcarded search terms if the field in question contains a
LowercaseFilter in its associated field type definition?
-Sascha
Robert Muir wrote:
> we can use stemming as an example:
>
> lets say your query is c?ns?st?nt?y
>
> how will this match "consistently", which the porter stemmer
> transforms to 'consistent'.
> furthermore, note that i replaced the vowels with ?'s here. The porter
> stemmer doesnt just rip stuff off the end, but attempts to guess
> syllables as part of the process, so it cannot possibly work.
>
> the only way it would work in this situation would be if you formed
> permutations of all the possible words this wildcard would match, and
> then did analysis on each form, and searched on all stems.
>
> but, this is impossible, since the * operator allows an infinite language.
>
> On Fri, May 21, 2010 at 10:11 AM, Sascha Szott<sz...@zib.de> wrote:
>> Hi folks,
>>
>> what's the idea behind the fact that no text analysis (e.g. lowercasing) is
>> performed on wildcarded search terms?
>>
>> In my context this behaviour seems to be counter-intuitive (I guess that's
>> the case in the majority of applications) and my application needs to
>> lowercase any input term before sending the HTTP request to my Solr server.
>>
>> Would it be easy to disable this behaviour in Solr (1.5)? I would like to
>> see a config parameter (per field type) that allows to disable this "odd"
>> behaviour if needed. To ensure backward compatibility the "odd" behaviour
>> should be the default anymore.
>>
>> Am I missing any drawbacks?
>>
>> Best,
>> Sascha
RE: solr connection question
Posted by Chris Hostetter <ho...@fucit.org>.
: Yes I mean HTTP-requests
: How can I log them?
it's entirely dependent on your Servlet Container (ie: jetty, tomcat,
resin, weblogic, etc...)
If you are using the example jetty provided in the solr releases (ie: java
-jar start.jar) they show up in examples/logs
-Hoss
RE: solr connection question
Posted by "ZAROGKIKAS,GIORGOS" <g....@multirama.gr>.
Yes I mean HTTP-requests
How can I log them?
-----Original Message-----
From: Sven Maurmann [mailto:sven.maurmann@kippdata.de]
Sent: Thursday, July 08, 2010 3:56 PM
To: solr-user@lucene.apache.org
Subject: Re: solr connection question
Hi,
Solr runs as a Web application. The requests you most probably mean
are just HTTP-requests to the underlying container. Internally each
request is processed against the Lucene index, usually being a file-
based one. Therefore there are no connections like in a database
application, where you have a pool of connections to your remote
databse server.
Best,
Sven
--On Donnerstag, 8. Juli 2010 15:46 +0300 "ZAROGKIKAS,GIORGOS"
<g....@multirama.gr> wrote:
> Hi solr users
>
> I need to know how solr manages the connections when we make a
> request(select update commit) Is there any connection pooling or an
> article to learn about it connection management?? How can I log in a
file
> the connections solr server
>
> I have setup my solr 1.4 with tomcat
>
> Thanks in advance
Re: solr connection question
Posted by Sven Maurmann <sv...@kippdata.de>.
Hi,
Solr runs as a Web application. The requests you most probably mean
are just HTTP-requests to the underlying container. Internally each
request is processed against the Lucene index, usually being a file-
based one. Therefore there are no connections like in a database
application, where you have a pool of connections to your remote
databse server.
Best,
Sven
--On Donnerstag, 8. Juli 2010 15:46 +0300 "ZAROGKIKAS,GIORGOS"
<g....@multirama.gr> wrote:
> Hi solr users
>
> I need to know how solr manages the connections when we make a
> request(select update commit) Is there any connection pooling or an
> article to learn about it connection management?? How can I log in a file
> the connections solr server
>
> I have setup my solr 1.4 with tomcat
>
> Thanks in advance
Re: solr connection question
Posted by Leonardo Menezes <le...@googlemail.com>.
jarrlll
On Fri, Jul 9, 2010 at 10:20 AM, Óscar Marín Miró
<os...@gmail.com>wrote:
> xD
>
> On Thu, Jul 8, 2010 at 2:58 PM, Alejandro Gonzalez
> <al...@gmail.com> wrote:
> > ok please don't forget it :)
> >
> > 2010/7/8 Ruben Abad <ru...@gmail.com>
> >
> >> Jorl, ok.... tendré que modificar mi petición de vacaciones :(
> >> Rubén Abad <ru...@gmail.com>
> >>
> >>
> >> On Thu, Jul 8, 2010 at 2:46 PM, ZAROGKIKAS,GIORGOS <
> >> g.zarogkikas@multirama.gr> wrote:
> >>
> >> > Hi solr users
> >> >
> >> > I need to know how solr manages the connections when we make a
> >> > request(select update commit)
> >> > Is there any connection pooling or an article to learn about it
> >> connection
> >> > management??
> >> > How can I log in a file the connections solr server
> >> >
> >> > I have setup my solr 1.4 with tomcat
> >> >
> >> > Thanks in advance
> >> >
> >> >
> >> >
> >> >
> >>
> >
>
>
>
> --
> Whether it's science, technology, personal experience, true love,
> astrology, or gut feelings, each of us has confidence in something
> that we will never fully comprehend.
> --Roy H. William
>
Re: solr connection question
Posted by Óscar Marín Miró <os...@gmail.com>.
xD
On Thu, Jul 8, 2010 at 2:58 PM, Alejandro Gonzalez
<al...@gmail.com> wrote:
> ok please don't forget it :)
>
> 2010/7/8 Ruben Abad <ru...@gmail.com>
>
>> Jorl, ok.... tendré que modificar mi petición de vacaciones :(
>> Rubén Abad <ru...@gmail.com>
>>
>>
>> On Thu, Jul 8, 2010 at 2:46 PM, ZAROGKIKAS,GIORGOS <
>> g.zarogkikas@multirama.gr> wrote:
>>
>> > Hi solr users
>> >
>> > I need to know how solr manages the connections when we make a
>> > request(select update commit)
>> > Is there any connection pooling or an article to learn about it
>> connection
>> > management??
>> > How can I log in a file the connections solr server
>> >
>> > I have setup my solr 1.4 with tomcat
>> >
>> > Thanks in advance
>> >
>> >
>> >
>> >
>>
>
--
Whether it's science, technology, personal experience, true love,
astrology, or gut feelings, each of us has confidence in something
that we will never fully comprehend.
--Roy H. William
Re: solr connection question
Posted by Alejandro Gonzalez <al...@gmail.com>.
ok please don't forget it :)
2010/7/8 Ruben Abad <ru...@gmail.com>
> Jorl, ok.... tendré que modificar mi petición de vacaciones :(
> Rubén Abad <ru...@gmail.com>
>
>
> On Thu, Jul 8, 2010 at 2:46 PM, ZAROGKIKAS,GIORGOS <
> g.zarogkikas@multirama.gr> wrote:
>
> > Hi solr users
> >
> > I need to know how solr manages the connections when we make a
> > request(select update commit)
> > Is there any connection pooling or an article to learn about it
> connection
> > management??
> > How can I log in a file the connections solr server
> >
> > I have setup my solr 1.4 with tomcat
> >
> > Thanks in advance
> >
> >
> >
> >
>
Re: solr connection question
Posted by Ruben Abad <ru...@gmail.com>.
Jorl, ok.... tendré que modificar mi petición de vacaciones :(
Rubén Abad <ru...@gmail.com>
On Thu, Jul 8, 2010 at 2:46 PM, ZAROGKIKAS,GIORGOS <
g.zarogkikas@multirama.gr> wrote:
> Hi solr users
>
> I need to know how solr manages the connections when we make a
> request(select update commit)
> Is there any connection pooling or an article to learn about it connection
> management??
> How can I log in a file the connections solr server
>
> I have setup my solr 1.4 with tomcat
>
> Thanks in advance
>
>
>
>
solr connection question
Posted by "ZAROGKIKAS,GIORGOS" <g....@multirama.gr>.
Hi solr users
I need to know how solr manages the connections when we make a request(select update commit)
Is there any connection pooling or an article to learn about it connection management??
How can I log in a file the connections solr server
I have setup my solr 1.4 with tomcat
Thanks in advance
Re: Wildcard queries
Posted by Robert Muir <rc...@gmail.com>.
we can use stemming as an example:
lets say your query is c?ns?st?nt?y
how will this match "consistently", which the porter stemmer
transforms to 'consistent'.
furthermore, note that i replaced the vowels with ?'s here. The porter
stemmer doesnt just rip stuff off the end, but attempts to guess
syllables as part of the process, so it cannot possibly work.
the only way it would work in this situation would be if you formed
permutations of all the possible words this wildcard would match, and
then did analysis on each form, and searched on all stems.
but, this is impossible, since the * operator allows an infinite language.
On Fri, May 21, 2010 at 10:11 AM, Sascha Szott <sz...@zib.de> wrote:
> Hi folks,
>
> what's the idea behind the fact that no text analysis (e.g. lowercasing) is
> performed on wildcarded search terms?
>
> In my context this behaviour seems to be counter-intuitive (I guess that's
> the case in the majority of applications) and my application needs to
> lowercase any input term before sending the HTTP request to my Solr server.
>
> Would it be easy to disable this behaviour in Solr (1.5)? I would like to
> see a config parameter (per field type) that allows to disable this "odd"
> behaviour if needed. To ensure backward compatibility the "odd" behaviour
> should be the default anymore.
>
> Am I missing any drawbacks?
>
> Best,
> Sascha
>
>
--
Robert Muir
rcmuir@gmail.com
Re: Wildcard queries
Posted by Robert Muir <rc...@gmail.com>.
this lowercasing can 'sort of work' (depending on your analysis, and
even language, not all case folding is as simple as english).
But the more general problem cannot be a bug, as its mathematically
not possible to do with queries like wildcard that allow an infinite
language, and non-reversible analysis.
On Fri, May 21, 2010 at 10:21 AM, Smiley, David W. <ds...@mitre.org> wrote:
> I absolutely consider this a bug too. Cast your vote:
> https://issues.apache.org/jira/browse/SOLR-219
>
> ~ David
>
> On May 21, 2010, at 10:11 AM, Sascha Szott wrote:
>
>> Hi folks,
>>
>> what's the idea behind the fact that no text analysis (e.g. lowercasing)
>> is performed on wildcarded search terms?
>>
>> In my context this behaviour seems to be counter-intuitive (I guess
>> that's the case in the majority of applications) and my application
>> needs to lowercase any input term before sending the HTTP request to my
>> Solr server.
>>
>> Would it be easy to disable this behaviour in Solr (1.5)? I would like
>> to see a config parameter (per field type) that allows to disable this
>> "odd" behaviour if needed. To ensure backward compatibility the "odd"
>> behaviour should be the default anymore.
>>
>> Am I missing any drawbacks?
>>
>> Best,
>> Sascha
>>
>
>
--
Robert Muir
rcmuir@gmail.com
Re: Wildcard queries
Posted by "Smiley, David W." <ds...@mitre.org>.
I absolutely consider this a bug too. Cast your vote:
https://issues.apache.org/jira/browse/SOLR-219
~ David
On May 21, 2010, at 10:11 AM, Sascha Szott wrote:
> Hi folks,
>
> what's the idea behind the fact that no text analysis (e.g. lowercasing)
> is performed on wildcarded search terms?
>
> In my context this behaviour seems to be counter-intuitive (I guess
> that's the case in the majority of applications) and my application
> needs to lowercase any input term before sending the HTTP request to my
> Solr server.
>
> Would it be easy to disable this behaviour in Solr (1.5)? I would like
> to see a config parameter (per field type) that allows to disable this
> "odd" behaviour if needed. To ensure backward compatibility the "odd"
> behaviour should be the default anymore.
>
> Am I missing any drawbacks?
>
> Best,
> Sascha
>