You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Jamie <ja...@stimulussoft.com> on 2010/01/08 21:01:35 UTC
Search query problem
Hi There
We are trying to search for the exact word "Lowe's" across a large set
of indexed data. Our results include everything with "low" in it. Thus,
we are receiving a much larger data set that we expected. The data is
indexing using the analyzer:
TokenStream result = new StandardTokenizer(reader);
result = new StandardFilter(result);
result = new LowerCaseFilter(result);
result = new StopFilter(result, stopTable);
result = new PorterStemFilter(result);
return result;
Thanks
Jamie
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Search query problem
Posted by Ahmet Arslan <io...@yahoo.com>.
> Is there another stemmer we can use that is perhaps not as
> aggressive as the Porter Stemmer.
"KStem is an alternative to Porter for developers looking for a less agressive stemmer. It was written by Bob Krovetz, ported to Lucene by Sergio Guzman-Lara (UMASS Amherst)." [1]
[1]http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters/Kstem?highlight=(kstemmer)
> i.e. the stemming could remove ing's,
> er's, but not
> something so significant as to convert ""Lowe's" to "Low"
By the way Apostrophes are removed by StandardFilter in your analyzer chain.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Search query problem
Posted by Shashi Kant <sk...@sloan.mit.edu>.
Couldn't you just mod the PorterStemmer class for your requirements?
(we did and provided it a list of ignore words & phrases specific to
our needs)
On Sat, Jan 9, 2010 at 4:00 AM, Jamie <ja...@stimulussoft.com> wrote:
> Hi All
>
> Is there another stemmer we can use that is perhaps not as aggressive as the
> Porter Stemmer. i.e. the stemming could remove ing's, er's, but not
> something so significant as to convert ""Lowe's" to "Low"
>
> Thanks
>
> Jamie
>
> Will Murnane wrote:
>>
>> On Fri, Jan 8, 2010 at 16:27, Jamie <ja...@stimulussoft.com> wrote:
>>
>>>
>>> Hi Ian / Will
>>>
>>> Thanks. Surely, the Porter Stemmer should not stem proper noun's. i.e. it
>>> could check the capitalization of the first letter of a word and whether
>>> or
>>> not the word is the start of sentence. If so, it could choose not apply
>>> any
>>> stemming. Or am I completely out of whack?
>>>
>>
>> Look again: you're downcasing the terms before the Porter filter ever
>> sees them (which is, AIUI, necessary). You might do well to combine
>> the tokenizing and downcasing step with some heuristic to find proper
>> nouns and not downcase or stem them.
>>
>> Will
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> --
> Stimulus Software - MailArchiva
> Email Archiving And Compliance
> USA Tel: +1-713-343-8824 ext 100
> UK Tel: +44-20-80991035 ext 100
> Email: jamie@stimulussoft.com
> Web: http://www.mailarchiva.com
> To receive MailArchiva Enterprise Edition product announcements, send a
> message to: <ma...@stimulussoft.com>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Search query problem
Posted by Jamie <ja...@stimulussoft.com>.
Hi All
Is there another stemmer we can use that is perhaps not as aggressive as
the Porter Stemmer. i.e. the stemming could remove ing's, er's, but not
something so significant as to convert ""Lowe's" to "Low"
Thanks
Jamie
Will Murnane wrote:
> On Fri, Jan 8, 2010 at 16:27, Jamie <ja...@stimulussoft.com> wrote:
>
>> Hi Ian / Will
>>
>> Thanks. Surely, the Porter Stemmer should not stem proper noun's. i.e. it
>> could check the capitalization of the first letter of a word and whether or
>> not the word is the start of sentence. If so, it could choose not apply any
>> stemming. Or am I completely out of whack?
>>
> Look again: you're downcasing the terms before the Porter filter ever
> sees them (which is, AIUI, necessary). You might do well to combine
> the tokenizing and downcasing step with some heuristic to find proper
> nouns and not downcase or stem them.
>
> Will
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Stimulus Software - MailArchiva
Email Archiving And Compliance
USA Tel: +1-713-343-8824 ext 100
UK Tel: +44-20-80991035 ext 100
Email: jamie@stimulussoft.com
Web: http://www.mailarchiva.com
To receive MailArchiva Enterprise Edition product announcements, send a message to: <ma...@stimulussoft.com>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Search query problem
Posted by Will Murnane <wi...@gmail.com>.
On Fri, Jan 8, 2010 at 16:27, Jamie <ja...@stimulussoft.com> wrote:
> Hi Ian / Will
>
> Thanks. Surely, the Porter Stemmer should not stem proper noun's. i.e. it
> could check the capitalization of the first letter of a word and whether or
> not the word is the start of sentence. If so, it could choose not apply any
> stemming. Or am I completely out of whack?
Look again: you're downcasing the terms before the Porter filter ever
sees them (which is, AIUI, necessary). You might do well to combine
the tokenizing and downcasing step with some heuristic to find proper
nouns and not downcase or stem them.
Will
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Search query problem
Posted by Jamie <ja...@stimulussoft.com>.
Hi Ian / Will
Thanks. Surely, the Porter Stemmer should not stem proper noun's. i.e.
it could check the capitalization of the first letter of a word and
whether or not the word is the start of sentence. If so, it could choose
not apply any stemming. Or am I completely out of whack?
Jamie
Ian Lea wrote:
> Looks like PorterStemFilter converts "Lowe's" to low. Not very surprising.
>
> Options include
>
> . Drop the stemming
>
> . Index stemmed and non-stemmed variants and search both, maybe
> boosting the non-stemmed variant.
>
> If you really want exact matches only, you may also/instead want
> untokenized fields. Apostrophes etc can be a problem. Look into what
> analyzers do and use Luke to see what is indexed.
>
> --
> Ian.
>
>
> On Fri, Jan 8, 2010 at 8:01 PM, Jamie <ja...@stimulussoft.com> wrote:
>
>> Hi There
>>
>> We are trying to search for the exact word "Lowe's" across a large set of
>> indexed data. Our results include everything with "low" in it. Thus, we are
>> receiving a much larger data set that we expected. The data is indexing
>> using the analyzer:
>> TokenStream result = new StandardTokenizer(reader);
>> result = new StandardFilter(result);
>> result = new LowerCaseFilter(result);
>> result = new StopFilter(result, stopTable);
>> result = new PorterStemFilter(result);
>> return result;
>>
>> Thanks
>>
>> Jamie
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Stimulus Software - MailArchiva
Email Archiving And Compliance
USA Tel: +1-713-343-8824 ext 100
UK Tel: +44-20-80991035 ext 100
Email: jamie@stimulussoft.com
Web: http://www.mailarchiva.com
To receive MailArchiva Enterprise Edition product announcements, send a message to: <ma...@stimulussoft.com>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Search query problem
Posted by Ian Lea <ia...@gmail.com>.
Looks like PorterStemFilter converts "Lowe's" to low. Not very surprising.
Options include
. Drop the stemming
. Index stemmed and non-stemmed variants and search both, maybe
boosting the non-stemmed variant.
If you really want exact matches only, you may also/instead want
untokenized fields. Apostrophes etc can be a problem. Look into what
analyzers do and use Luke to see what is indexed.
--
Ian.
On Fri, Jan 8, 2010 at 8:01 PM, Jamie <ja...@stimulussoft.com> wrote:
> Hi There
>
> We are trying to search for the exact word "Lowe's" across a large set of
> indexed data. Our results include everything with "low" in it. Thus, we are
> receiving a much larger data set that we expected. The data is indexing
> using the analyzer:
> TokenStream result = new StandardTokenizer(reader);
> result = new StandardFilter(result);
> result = new LowerCaseFilter(result);
> result = new StopFilter(result, stopTable);
> result = new PorterStemFilter(result);
> return result;
>
> Thanks
>
> Jamie
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Search query problem
Posted by Will Murnane <wi...@gmail.com>.
On Fri, Jan 8, 2010 at 15:01, Jamie <ja...@stimulussoft.com> wrote:
> Hi There
>
> We are trying to search for the exact word "Lowe's" across a large set of
> indexed data. Our results include everything with "low" in it. Thus, we are
> receiving a much larger data set that we expected. The data is indexing
> using the analyzer:
> TokenStream result = new StandardTokenizer(reader);
> result = new StandardFilter(result);
> result = new LowerCaseFilter(result);
> result = new StopFilter(result, stopTable);
> result = new PorterStemFilter(result);
PorterStemFilter is changing Lowe to low. Change your tokenizer so
that Lowe's is tokenized as a single token, and that should avoid it.
Will
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org