You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by David Whalen <dw...@enr-corp.com> on 2007/08/13 21:11:39 UTC
Problem with stemming
Hi All.
We're running into a problem with stemming that I can't
figure out. For example, searching for the word "transit"
(whether in quotes or not) returns documents with the word
"transition" in them.
How do I disable this? We want our engine to be as literal
as possible. If a user mis-types a word, that's too bad for
them....
TIA
DW
Re: Problem with stemming
Posted by Yonik Seeley <yo...@apache.org>.
On 8/13/07, David Whalen <dw...@enr-corp.com> wrote:
> Yonik:
>
> I only raised the question to the group after I had looked in
> the schema.xml. There are a lot of comments in that file, but
> they make no sense to me.
>
> I'd appreciate some specific help on what to do...
Every field has a fieldType.
A fieldType for a full-text field specifies how to do analysis. You
need to have a basic understanding of how Lucene does text analysis,
then modify your fieldType for the field (removing the stemmer, among
other things).
Look at these:
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters
http://wiki.apache.org/solr/SolrRelevancyFAQ
-Yonik
RE: Problem with stemming
Posted by Lance Norskog <go...@gmail.com>.
You need this book:
-----Original Message-----
From: David Whalen [mailto:dwhalen@enr-corp.com]
Sent: Monday, August 13, 2007 1:00 PM
To: solr-user@lucene.apache.org
Subject: RE: Problem with stemming
Yonik:
I only raised the question to the group after I had looked in the
schema.xml. There are a lot of comments in that file, but they make no
sense to me.
I'd appreciate some specific help on what to do...
DW
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik
> Seeley
> Sent: Monday, August 13, 2007 3:28 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Problem with stemming
>
> On 8/13/07, David Whalen <dw...@enr-corp.com> wrote:
> > Hi All.
> >
> > We're running into a problem with stemming that I can't
> figure out.
> > For example, searching for the word "transit"
> > (whether in quotes or not) returns documents with the word
> > "transition" in them.
> >
> > How do I disable this? We want our engine to be as literal as
> > possible. If a user mis-types a word, that's too bad for them....
>
> Use a different field-type for those fields that you want exact
> matching for (and then re-index).
> Read through schema.xml if you haven't... there are quite a few
> comments in there.
> You may want a field type with just a whitespace tokenizer followed by
> a lowercase filter.
>
> -Yonik
>
Re: Problem with stemming
Posted by Yonik Seeley <yo...@apache.org>.
On 8/13/07, David Whalen <dw...@enr-corp.com> wrote:
> So I shut it off by removing these tags from my schema.xml
> file? Seems like it's this Porter thing that's messing
> me up.
Yes. As I said in my first email, if you want very exact matching,
you may just want a whitespace tokenizer followed by a lowercase
filter in the analyzer for that fieldType. I don't know if you have
synonyms configured or if you want them, if you have/want
WordDelimiterFilter, etc. If you have both a query analyzer and an
index analyzer for that fieldType, then remove stemming from both of
them.
You also must re-index for a change like this (this is not just a
query-time config since stemming also affects what's in the index).
-Yonik
RE: Problem with stemming
Posted by David Whalen <dw...@enr-corp.com>.
So I shut it off by removing these tags from my schema.xml
file? Seems like it's this Porter thing that's messing
me up.
> -----Original Message-----
> From: Tom Mastre [mailto:tmmastre@nps.edu]
> Sent: Monday, August 13, 2007 5:19 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Problem with stemming
>
> Go here
>
> http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters?hi
ghlight=%28stemming%29> #head-88cc86e4432b359030cffdb32d095062b843d4f5
>
> Look for this
>
> solr.PorterStemFilterFactory
>
>
> On 8/13/07 1:50 PM, "David Whalen" <dw...@enr-corp.com> wrote:
>
> > Thanks, guys. I'm sure that by the time I get the book and
> > learn all about Lucene the CEO of my company will have insisted
> > we find another search engine. But the book will look great
> > on my coffee table....
> >
> >
> >
> >
> >> -----Original Message-----
> >> From: Lance Norskog [mailto:goksron@gmail.com]
> >> Sent: Monday, August 13, 2007 4:37 PM
> >> To: solr-user@lucene.apache.org
> >> Subject: RE: Problem with stemming
> >>
> >> (Oops, try again.)
> >>
> >> You need this book:
> >>
> >> http://www.amazon.com/Lucene-Action-Erik-Hatcher/dp/1932394281
> >> /ref=pd_bbs_sr
> >> _1/103-4871137-7111056?ie=UTF8&s=books&qid=1187037246&sr=8-1
> >>
> >> Lucene in Action by Eric Hatcher and Otis Gospodnetic. It
> >> does not cover Solr really, but you will understand what
> >> Lucene does and how it works.
> >> Until then you will not really get anywhere.
> >>
> >> Cheers,
> >>
> >> Lance
> >>
> >> -----Original Message-----
> >> From: David Whalen [mailto:dwhalen@enr-corp.com]
> >> Sent: Monday, August 13, 2007 1:00 PM
> >> To: solr-user@lucene.apache.org
> >> Subject: RE: Problem with stemming
> >>
> >> Yonik:
> >>
> >> I only raised the question to the group after I had looked in
> >> the schema.xml. There are a lot of comments in that file,
> >> but they make no sense to me.
> >>
> >> I'd appreciate some specific help on what to do...
> >>
> >> DW
> >>
> >>> -----Original Message-----
> >>> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On
> >> Behalf Of Yonik
> >>> Seeley
> >>> Sent: Monday, August 13, 2007 3:28 PM
> >>> To: solr-user@lucene.apache.org
> >>> Subject: Re: Problem with stemming
> >>>
> >>> On 8/13/07, David Whalen <dw...@enr-corp.com> wrote:
> >>>> Hi All.
> >>>>
> >>>> We're running into a problem with stemming that I can't
> >>> figure out.
> >>>> For example, searching for the word "transit"
> >>>> (whether in quotes or not) returns documents with the word
> >>>> "transition" in them.
> >>>>
> >>>> How do I disable this? We want our engine to be as literal as
> >>>> possible. If a user mis-types a word, that's too bad
> for them....
> >>>
> >>> Use a different field-type for those fields that you want exact
> >>> matching for (and then re-index).
> >>> Read through schema.xml if you haven't... there are quite a few
> >>> comments in there.
> >>> You may want a field type with just a whitespace tokenizer
> >> followed by
> >>> a lowercase filter.
> >>>
> >>> -Yonik
> >>>
> >>
> >>
>
> Thomas M. Mastre
> Manager, Homeland Security Digital Library
>
>
> Center for Homeland Defense and Security
> The Nation's Homeland Security Educator
> 1 University Circle
> DKL, Room 112
> Monterey, Ca. 93943
> Phone: 831.656.1095, Cell:831.238.1451
> Fax:831.656.2619
> email: tmmastre@nps.edu
> http://www.hsdl.org
>
>
Re: Problem with stemming
Posted by Tom Mastre <tm...@nps.edu>.
Go here
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters?highlight=%28ste
mming%29#head-88cc86e4432b359030cffdb32d095062b843d4f5
Look for this
solr.PorterStemFilterFactory
On 8/13/07 1:50 PM, "David Whalen" <dw...@enr-corp.com> wrote:
> Thanks, guys. I'm sure that by the time I get the book and
> learn all about Lucene the CEO of my company will have insisted
> we find another search engine. But the book will look great
> on my coffee table....
>
>
>
>
>> -----Original Message-----
>> From: Lance Norskog [mailto:goksron@gmail.com]
>> Sent: Monday, August 13, 2007 4:37 PM
>> To: solr-user@lucene.apache.org
>> Subject: RE: Problem with stemming
>>
>> (Oops, try again.)
>>
>> You need this book:
>>
>> http://www.amazon.com/Lucene-Action-Erik-Hatcher/dp/1932394281
>> /ref=pd_bbs_sr
>> _1/103-4871137-7111056?ie=UTF8&s=books&qid=1187037246&sr=8-1
>>
>> Lucene in Action by Eric Hatcher and Otis Gospodnetic. It
>> does not cover Solr really, but you will understand what
>> Lucene does and how it works.
>> Until then you will not really get anywhere.
>>
>> Cheers,
>>
>> Lance
>>
>> -----Original Message-----
>> From: David Whalen [mailto:dwhalen@enr-corp.com]
>> Sent: Monday, August 13, 2007 1:00 PM
>> To: solr-user@lucene.apache.org
>> Subject: RE: Problem with stemming
>>
>> Yonik:
>>
>> I only raised the question to the group after I had looked in
>> the schema.xml. There are a lot of comments in that file,
>> but they make no sense to me.
>>
>> I'd appreciate some specific help on what to do...
>>
>> DW
>>
>>> -----Original Message-----
>>> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On
>> Behalf Of Yonik
>>> Seeley
>>> Sent: Monday, August 13, 2007 3:28 PM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: Problem with stemming
>>>
>>> On 8/13/07, David Whalen <dw...@enr-corp.com> wrote:
>>>> Hi All.
>>>>
>>>> We're running into a problem with stemming that I can't
>>> figure out.
>>>> For example, searching for the word "transit"
>>>> (whether in quotes or not) returns documents with the word
>>>> "transition" in them.
>>>>
>>>> How do I disable this? We want our engine to be as literal as
>>>> possible. If a user mis-types a word, that's too bad for them....
>>>
>>> Use a different field-type for those fields that you want exact
>>> matching for (and then re-index).
>>> Read through schema.xml if you haven't... there are quite a few
>>> comments in there.
>>> You may want a field type with just a whitespace tokenizer
>> followed by
>>> a lowercase filter.
>>>
>>> -Yonik
>>>
>>
>>
Thomas M. Mastre
Manager, Homeland Security Digital Library
Center for Homeland Defense and Security
The Nation's Homeland Security Educator
1 University Circle
DKL, Room 112
Monterey, Ca. 93943
Phone: 831.656.1095, Cell:831.238.1451
Fax:831.656.2619
email: tmmastre@nps.edu
http://www.hsdl.org
RE: Problem with stemming
Posted by David Whalen <dw...@enr-corp.com>.
Thanks, guys. I'm sure that by the time I get the book and
learn all about Lucene the CEO of my company will have insisted
we find another search engine. But the book will look great
on my coffee table....
> -----Original Message-----
> From: Lance Norskog [mailto:goksron@gmail.com]
> Sent: Monday, August 13, 2007 4:37 PM
> To: solr-user@lucene.apache.org
> Subject: RE: Problem with stemming
>
> (Oops, try again.)
>
> You need this book:
>
> http://www.amazon.com/Lucene-Action-Erik-Hatcher/dp/1932394281
> /ref=pd_bbs_sr
> _1/103-4871137-7111056?ie=UTF8&s=books&qid=1187037246&sr=8-1
>
> Lucene in Action by Eric Hatcher and Otis Gospodnetic. It
> does not cover Solr really, but you will understand what
> Lucene does and how it works.
> Until then you will not really get anywhere.
>
> Cheers,
>
> Lance
>
> -----Original Message-----
> From: David Whalen [mailto:dwhalen@enr-corp.com]
> Sent: Monday, August 13, 2007 1:00 PM
> To: solr-user@lucene.apache.org
> Subject: RE: Problem with stemming
>
> Yonik:
>
> I only raised the question to the group after I had looked in
> the schema.xml. There are a lot of comments in that file,
> but they make no sense to me.
>
> I'd appreciate some specific help on what to do...
>
> DW
>
> > -----Original Message-----
> > From: yseeley@gmail.com [mailto:yseeley@gmail.com] On
> Behalf Of Yonik
> > Seeley
> > Sent: Monday, August 13, 2007 3:28 PM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Problem with stemming
> >
> > On 8/13/07, David Whalen <dw...@enr-corp.com> wrote:
> > > Hi All.
> > >
> > > We're running into a problem with stemming that I can't
> > figure out.
> > > For example, searching for the word "transit"
> > > (whether in quotes or not) returns documents with the word
> > > "transition" in them.
> > >
> > > How do I disable this? We want our engine to be as literal as
> > > possible. If a user mis-types a word, that's too bad for them....
> >
> > Use a different field-type for those fields that you want exact
> > matching for (and then re-index).
> > Read through schema.xml if you haven't... there are quite a few
> > comments in there.
> > You may want a field type with just a whitespace tokenizer
> followed by
> > a lowercase filter.
> >
> > -Yonik
> >
>
>
RE: Problem with stemming
Posted by Lance Norskog <go...@gmail.com>.
(Oops, try again.)
You need this book:
http://www.amazon.com/Lucene-Action-Erik-Hatcher/dp/1932394281/ref=pd_bbs_sr
_1/103-4871137-7111056?ie=UTF8&s=books&qid=1187037246&sr=8-1
Lucene in Action by Eric Hatcher and Otis Gospodnetic. It does not cover
Solr really, but you will understand what Lucene does and how it works.
Until then you will not really get anywhere.
Cheers,
Lance
-----Original Message-----
From: David Whalen [mailto:dwhalen@enr-corp.com]
Sent: Monday, August 13, 2007 1:00 PM
To: solr-user@lucene.apache.org
Subject: RE: Problem with stemming
Yonik:
I only raised the question to the group after I had looked in the
schema.xml. There are a lot of comments in that file, but they make no
sense to me.
I'd appreciate some specific help on what to do...
DW
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik
> Seeley
> Sent: Monday, August 13, 2007 3:28 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Problem with stemming
>
> On 8/13/07, David Whalen <dw...@enr-corp.com> wrote:
> > Hi All.
> >
> > We're running into a problem with stemming that I can't
> figure out.
> > For example, searching for the word "transit"
> > (whether in quotes or not) returns documents with the word
> > "transition" in them.
> >
> > How do I disable this? We want our engine to be as literal as
> > possible. If a user mis-types a word, that's too bad for them....
>
> Use a different field-type for those fields that you want exact
> matching for (and then re-index).
> Read through schema.xml if you haven't... there are quite a few
> comments in there.
> You may want a field type with just a whitespace tokenizer followed by
> a lowercase filter.
>
> -Yonik
>
RE: Problem with stemming
Posted by David Whalen <dw...@enr-corp.com>.
Yonik:
I only raised the question to the group after I had looked in
the schema.xml. There are a lot of comments in that file, but
they make no sense to me.
I'd appreciate some specific help on what to do...
DW
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf
> Of Yonik Seeley
> Sent: Monday, August 13, 2007 3:28 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Problem with stemming
>
> On 8/13/07, David Whalen <dw...@enr-corp.com> wrote:
> > Hi All.
> >
> > We're running into a problem with stemming that I can't
> figure out.
> > For example, searching for the word "transit"
> > (whether in quotes or not) returns documents with the word
> > "transition" in them.
> >
> > How do I disable this? We want our engine to be as literal as
> > possible. If a user mis-types a word, that's too bad for them....
>
> Use a different field-type for those fields that you want
> exact matching for (and then re-index).
> Read through schema.xml if you haven't... there are quite a
> few comments in there.
> You may want a field type with just a whitespace tokenizer
> followed by a lowercase filter.
>
> -Yonik
>
Re: Problem with stemming
Posted by Yonik Seeley <yo...@apache.org>.
On 8/13/07, David Whalen <dw...@enr-corp.com> wrote:
> Hi All.
>
> We're running into a problem with stemming that I can't
> figure out. For example, searching for the word "transit"
> (whether in quotes or not) returns documents with the word
> "transition" in them.
>
> How do I disable this? We want our engine to be as literal
> as possible. If a user mis-types a word, that's too bad for
> them....
Use a different field-type for those fields that you want exact
matching for (and then re-index).
Read through schema.xml if you haven't... there are quite a few
comments in there.
You may want a field type with just a whitespace tokenizer followed by
a lowercase filter.
-Yonik
Re: Problem with stemming
Posted by Chris Hostetter <ho...@fucit.org>.
: Message-ID:
: <D6...@vsharex.share.acnusa.net>
: In-Reply-To: <f7...@mail.gmail.com>
http://people.apache.org/~hossman/#threadhijack
When starting a new discussion on a mailing list, please do not reply to
an existing message, instead start a fresh email. Even if you change the
subject line of your email, other mail headers still track which thread
you replied to and your question is "hidden" in that thread and gets less
attention. It makes following discussions in the mailing list archives
particularly difficult.
See Also: http://en.wikipedia.org/wiki/Thread_hijacking
-Hoss