You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Ahmet Arslan <io...@yahoo.com> on 2010/01/02 12:57:44 UTC
ShingleFilter with outputUnigrams=false
I am using lucene 2.9.1 and I was trying to understand the ShingleFilter and wrote the code below.
String test = "please divide this sentence";
Tokenizer wsTokenizer = new WhitespaceTokenizer(new StringReader(test));
ShingleFilter filter = new ShingleFilter(wsTokenizer, 3);
filter.setOutputUnigrams(false);
TermAttribute termAtt = (TermAttribute) filter.getAttribute(TermAttribute.class);
while (filter.incrementToken()) System.out.println(termAtt.term());
I noticed that if I set outputUnigrams to false it gives me the same output for maxShingleSize=2 and maxShingleSize=3.
please divide
divide this
this sentence
when i set maxShingleSize to 4 output is:
please divide
please divide this sentence
divide this
this sentence
I was expecting the output as follows with maxShingleSize=3 and outputUnigrams=false :
please divide this
divide this sentence
Am I missing something or this is the expected behavior?
I checked source code of ShingleFilterTest (lucene 3.0.0) and see that TRI_GRAM_TOKENS are tested with only outputUnigrams=true but not with outputUnigrams=false.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: ShingleFilter with outputUnigrams=false
Posted by Simon Willnauer <si...@googlemail.com>.
You can find the issue for this here
https://issues.apache.org/jira/browse/LUCENE-2199
On Fri, Jan 8, 2010 at 8:53 PM, Simon Willnauer
<si...@googlemail.com> wrote:
> This is truly a bug. The outputUnigram internally only works if you
> request bi-grams.
> If the outputUnigram is set to false the filter increment the
> shingleposition by one and therefore skips every even shingle. The
> position should only be incremented if shingleBufferPosition %
> maxShingle == 0
>
> I have a test and the fix - will open an issue soon.
>
> simon
>
> On Fri, Jan 8, 2010 at 7:48 PM, Chris Hostetter
> <ho...@fucit.org> wrote:
>>
>> : I am using lucene 2.9.1 and I was trying to understand the ShingleFilter and wrote the code below.
>> ...
>> : I was expecting the output as follows with maxShingleSize=3 and outputUnigrams=false :
>> ...
>> : Am I missing something or this is the expected behavior?
>>
>> I'm not very familiar with ShingleFilter, and i'n not 100% sure i
>> understand the example you describe, but it *seems* like there may be a
>> bug here ... the easieest way to verify that is if you could tweak your
>> example code into the form of a (failing) JUnit test and open a new Jira
>> issue -- then other devs (who know more about SHingleFilter) could look at
>> it and either verify that there is a bug, or point out what's invalid
>> about hte test.
>>
>>
>>
>> -Hoss
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Search query problem
Posted by Ahmet Arslan <io...@yahoo.com>.
> Is there another stemmer we can use that is perhaps not as
> aggressive as the Porter Stemmer.
"KStem is an alternative to Porter for developers looking for a less agressive stemmer. It was written by Bob Krovetz, ported to Lucene by Sergio Guzman-Lara (UMASS Amherst)." [1]
[1]http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters/Kstem?highlight=(kstemmer)
> i.e. the stemming could remove ing's,
> er's, but not
> something so significant as to convert ""Lowe's" to "Low"
By the way Apostrophes are removed by StandardFilter in your analyzer chain.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Search query problem
Posted by Shashi Kant <sk...@sloan.mit.edu>.
Couldn't you just mod the PorterStemmer class for your requirements?
(we did and provided it a list of ignore words & phrases specific to
our needs)
On Sat, Jan 9, 2010 at 4:00 AM, Jamie <ja...@stimulussoft.com> wrote:
> Hi All
>
> Is there another stemmer we can use that is perhaps not as aggressive as the
> Porter Stemmer. i.e. the stemming could remove ing's, er's, but not
> something so significant as to convert ""Lowe's" to "Low"
>
> Thanks
>
> Jamie
>
> Will Murnane wrote:
>>
>> On Fri, Jan 8, 2010 at 16:27, Jamie <ja...@stimulussoft.com> wrote:
>>
>>>
>>> Hi Ian / Will
>>>
>>> Thanks. Surely, the Porter Stemmer should not stem proper noun's. i.e. it
>>> could check the capitalization of the first letter of a word and whether
>>> or
>>> not the word is the start of sentence. If so, it could choose not apply
>>> any
>>> stemming. Or am I completely out of whack?
>>>
>>
>> Look again: you're downcasing the terms before the Porter filter ever
>> sees them (which is, AIUI, necessary). You might do well to combine
>> the tokenizing and downcasing step with some heuristic to find proper
>> nouns and not downcase or stem them.
>>
>> Will
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> --
> Stimulus Software - MailArchiva
> Email Archiving And Compliance
> USA Tel: +1-713-343-8824 ext 100
> UK Tel: +44-20-80991035 ext 100
> Email: jamie@stimulussoft.com
> Web: http://www.mailarchiva.com
> To receive MailArchiva Enterprise Edition product announcements, send a
> message to: <ma...@stimulussoft.com>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Search query problem
Posted by Jamie <ja...@stimulussoft.com>.
Hi All
Is there another stemmer we can use that is perhaps not as aggressive as
the Porter Stemmer. i.e. the stemming could remove ing's, er's, but not
something so significant as to convert ""Lowe's" to "Low"
Thanks
Jamie
Will Murnane wrote:
> On Fri, Jan 8, 2010 at 16:27, Jamie <ja...@stimulussoft.com> wrote:
>
>> Hi Ian / Will
>>
>> Thanks. Surely, the Porter Stemmer should not stem proper noun's. i.e. it
>> could check the capitalization of the first letter of a word and whether or
>> not the word is the start of sentence. If so, it could choose not apply any
>> stemming. Or am I completely out of whack?
>>
> Look again: you're downcasing the terms before the Porter filter ever
> sees them (which is, AIUI, necessary). You might do well to combine
> the tokenizing and downcasing step with some heuristic to find proper
> nouns and not downcase or stem them.
>
> Will
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Stimulus Software - MailArchiva
Email Archiving And Compliance
USA Tel: +1-713-343-8824 ext 100
UK Tel: +44-20-80991035 ext 100
Email: jamie@stimulussoft.com
Web: http://www.mailarchiva.com
To receive MailArchiva Enterprise Edition product announcements, send a message to: <ma...@stimulussoft.com>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Search query problem
Posted by Will Murnane <wi...@gmail.com>.
On Fri, Jan 8, 2010 at 16:27, Jamie <ja...@stimulussoft.com> wrote:
> Hi Ian / Will
>
> Thanks. Surely, the Porter Stemmer should not stem proper noun's. i.e. it
> could check the capitalization of the first letter of a word and whether or
> not the word is the start of sentence. If so, it could choose not apply any
> stemming. Or am I completely out of whack?
Look again: you're downcasing the terms before the Porter filter ever
sees them (which is, AIUI, necessary). You might do well to combine
the tokenizing and downcasing step with some heuristic to find proper
nouns and not downcase or stem them.
Will
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Search query problem
Posted by Jamie <ja...@stimulussoft.com>.
Hi Ian / Will
Thanks. Surely, the Porter Stemmer should not stem proper noun's. i.e.
it could check the capitalization of the first letter of a word and
whether or not the word is the start of sentence. If so, it could choose
not apply any stemming. Or am I completely out of whack?
Jamie
Ian Lea wrote:
> Looks like PorterStemFilter converts "Lowe's" to low. Not very surprising.
>
> Options include
>
> . Drop the stemming
>
> . Index stemmed and non-stemmed variants and search both, maybe
> boosting the non-stemmed variant.
>
> If you really want exact matches only, you may also/instead want
> untokenized fields. Apostrophes etc can be a problem. Look into what
> analyzers do and use Luke to see what is indexed.
>
> --
> Ian.
>
>
> On Fri, Jan 8, 2010 at 8:01 PM, Jamie <ja...@stimulussoft.com> wrote:
>
>> Hi There
>>
>> We are trying to search for the exact word "Lowe's" across a large set of
>> indexed data. Our results include everything with "low" in it. Thus, we are
>> receiving a much larger data set that we expected. The data is indexing
>> using the analyzer:
>> TokenStream result = new StandardTokenizer(reader);
>> result = new StandardFilter(result);
>> result = new LowerCaseFilter(result);
>> result = new StopFilter(result, stopTable);
>> result = new PorterStemFilter(result);
>> return result;
>>
>> Thanks
>>
>> Jamie
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Stimulus Software - MailArchiva
Email Archiving And Compliance
USA Tel: +1-713-343-8824 ext 100
UK Tel: +44-20-80991035 ext 100
Email: jamie@stimulussoft.com
Web: http://www.mailarchiva.com
To receive MailArchiva Enterprise Edition product announcements, send a message to: <ma...@stimulussoft.com>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Search query problem
Posted by Ian Lea <ia...@gmail.com>.
Looks like PorterStemFilter converts "Lowe's" to low. Not very surprising.
Options include
. Drop the stemming
. Index stemmed and non-stemmed variants and search both, maybe
boosting the non-stemmed variant.
If you really want exact matches only, you may also/instead want
untokenized fields. Apostrophes etc can be a problem. Look into what
analyzers do and use Luke to see what is indexed.
--
Ian.
On Fri, Jan 8, 2010 at 8:01 PM, Jamie <ja...@stimulussoft.com> wrote:
> Hi There
>
> We are trying to search for the exact word "Lowe's" across a large set of
> indexed data. Our results include everything with "low" in it. Thus, we are
> receiving a much larger data set that we expected. The data is indexing
> using the analyzer:
> TokenStream result = new StandardTokenizer(reader);
> result = new StandardFilter(result);
> result = new LowerCaseFilter(result);
> result = new StopFilter(result, stopTable);
> result = new PorterStemFilter(result);
> return result;
>
> Thanks
>
> Jamie
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Search query problem
Posted by Will Murnane <wi...@gmail.com>.
On Fri, Jan 8, 2010 at 15:01, Jamie <ja...@stimulussoft.com> wrote:
> Hi There
>
> We are trying to search for the exact word "Lowe's" across a large set of
> indexed data. Our results include everything with "low" in it. Thus, we are
> receiving a much larger data set that we expected. The data is indexing
> using the analyzer:
> TokenStream result = new StandardTokenizer(reader);
> result = new StandardFilter(result);
> result = new LowerCaseFilter(result);
> result = new StopFilter(result, stopTable);
> result = new PorterStemFilter(result);
PorterStemFilter is changing Lowe to low. Change your tokenizer so
that Lowe's is tokenized as a single token, and that should avoid it.
Will
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Search query problem
Posted by Jamie <ja...@stimulussoft.com>.
Hi There
We are trying to search for the exact word "Lowe's" across a large set
of indexed data. Our results include everything with "low" in it. Thus,
we are receiving a much larger data set that we expected. The data is
indexing using the analyzer:
TokenStream result = new StandardTokenizer(reader);
result = new StandardFilter(result);
result = new LowerCaseFilter(result);
result = new StopFilter(result, stopTable);
result = new PorterStemFilter(result);
return result;
Thanks
Jamie
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: ShingleFilter with outputUnigrams=false
Posted by Simon Willnauer <si...@googlemail.com>.
This is truly a bug. The outputUnigram internally only works if you
request bi-grams.
If the outputUnigram is set to false the filter increment the
shingleposition by one and therefore skips every even shingle. The
position should only be incremented if shingleBufferPosition %
maxShingle == 0
I have a test and the fix - will open an issue soon.
simon
On Fri, Jan 8, 2010 at 7:48 PM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> : I am using lucene 2.9.1 and I was trying to understand the ShingleFilter and wrote the code below.
> ...
> : I was expecting the output as follows with maxShingleSize=3 and outputUnigrams=false :
> ...
> : Am I missing something or this is the expected behavior?
>
> I'm not very familiar with ShingleFilter, and i'n not 100% sure i
> understand the example you describe, but it *seems* like there may be a
> bug here ... the easieest way to verify that is if you could tweak your
> example code into the form of a (failing) JUnit test and open a new Jira
> issue -- then other devs (who know more about SHingleFilter) could look at
> it and either verify that there is a bug, or point out what's invalid
> about hte test.
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: ShingleFilter with outputUnigrams=false
Posted by Chris Hostetter <ho...@fucit.org>.
: I am using lucene 2.9.1 and I was trying to understand the ShingleFilter and wrote the code below.
...
: I was expecting the output as follows with maxShingleSize=3 and outputUnigrams=false :
...
: Am I missing something or this is the expected behavior?
I'm not very familiar with ShingleFilter, and i'n not 100% sure i
understand the example you describe, but it *seems* like there may be a
bug here ... the easieest way to verify that is if you could tweak your
example code into the form of a (failing) JUnit test and open a new Jira
issue -- then other devs (who know more about SHingleFilter) could look at
it and either verify that there is a bug, or point out what's invalid
about hte test.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org