You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Tanguy Moal <ta...@gmail.com> on 2012/05/09 15:39:49 UTC
FrenchLightStemFilterFactory : normalizing tokens longer than 4
characters and having repeated characters in it
Dear list,
I recently figured out that the FrenchLightStemFilterFactory performs some
interestingly undocumented normalization on tokens...
There's a norm() helper called for each produced token that performs,
amongst other things, deletions on repeated characters... Only for tokens
with more than 4 characters... Examples :
aabb => aabb
aabbcc => abc
aaaaaa => a
aaaaab => ab
1 => 1
11 => 11
111 => 111
1111 => 1111
11111 => 1
111111 => 1
12355 => 1235
121221 => 12121
Although it might be interesting for real words in order to hopefully
correct common typographic errors, I'm not so sure of the correctness of
doing so on numbers.
Can anyone confirm it is normal behaviour ?
I use a StandardTokenizer marking number tokens as NUM... Should
FrenchLightStemmer use this information to avoid unnecessary stemming ?
Thanks in advance for your help.
--
Tanguy
PS: I previously sent this email by mistake with an unsubscribed email and
a slightly different object, I hope it won't mess things up.
Re: FrenchLightStemFilterFactory : normalizing tokens longer than 4
characters and having repeated characters in it
Posted by Robert Muir <rc...@gmail.com>.
On Wed, May 16, 2012 at 8:28 AM, Tanguy Moal <ta...@gmail.com> wrote:
> Any idea someone ?
>
> I think this is important since this could produce weird results on
> collections with numbers mixed in text.
I agree, i think we should just add '&& Character.isLetter(ch)' to the
undoublet check?
Thanks for bringing this up. do you want to open a JIRA issue?
http://wiki.apache.org/solr/HowToContribute
--
lucidimagination.com
Re: FrenchLightStemFilterFactory : normalizing tokens longer than 4
characters and having repeated characters in it
Posted by Tanguy Moal <ta...@gmail.com>.
Thank you!
JIRA issue filed : https://issues.apache.org/jira/browse/SOLR-3463
--
Tanguy
2012/5/16 Steven A Rowe <sa...@syr.edu>
> Hi Tanguy,
>
> I looked at the code, and I can see where the problem you describe is
> happening.
>
> I think it's a bug: if numbers are search terms, "stemming" them by
> compressing repeated digits makes little sense.
>
> Could you file a bug in JIRA? Please include the examples you gave in
> your earlier email when you describe the problem on the issue.
>
> I checked all of the other *LightStemmer's on trunk, and
> FrenchLightStemmer is the only one of them that does this arbitrary
> duplicate sequence compression. (FinnishLightStemmer does repetition
> compression too, but restricts the operation to chars 'k', 'p', and 't'.)
>
> Thanks,
> Steve
>
> -----Original Message-----
> From: Tanguy Moal [mailto:tanguy.moal@gmail.com]
> Sent: Wednesday, May 16, 2012 8:29 AM
> To: solr-user@lucene.apache.org
> Subject: Re: FrenchLightStemFilterFactory : normalizing tokens longer than
> 4 characters and having repeated characters in it
>
> Any idea someone ?
>
> I think this is important since this could produce weird results on
> collections with numbers mixed in text.
>
> From my understanding, there are a few options to address the issue :
> 1) Make *LightStemmer token type aware and don't try to stem on things
> that are not text (alpha/alphanum whatever :)) : this pulls the fix into
> the faulty component but makes it dependant on the StandardTokenizer which
> may not be what people want...
> 2) Enable StandardTokenizer to mark NUM tokens with the keyword attribute
> so that NUM tokens are not stemmed by contract (this could be a
> configuration flag markNumbersWithKeywordAttribute=true, false by default)
> 3) Use a custom processor to mark NUM tokens as keywords (the solution I
> chose since it doesn't require modifying lucene/solr's code base, it's a
> very simply contrib module)
>
> I chose solution #3.
>
> Maybe #2 is the way to go since most people using
> FrenchLightStemFilterFactory will also want to use StandardTokenizer...
>
> Any advice is welcome
>
> --
> Tanguy
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/FrenchLightStemFilterFactory-normalizing-tokens-longer-than-4-characters-and-having-repeated-charactt-tp3974148p3984080.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
RE: FrenchLightStemFilterFactory : normalizing tokens longer than 4
characters and having repeated characters in it
Posted by Steven A Rowe <sa...@syr.edu>.
Hi Tanguy,
I looked at the code, and I can see where the problem you describe is happening.
I think it's a bug: if numbers are search terms, "stemming" them by compressing repeated digits makes little sense.
Could you file a bug in JIRA? Please include the examples you gave in your earlier email when you describe the problem on the issue.
I checked all of the other *LightStemmer's on trunk, and FrenchLightStemmer is the only one of them that does this arbitrary duplicate sequence compression. (FinnishLightStemmer does repetition compression too, but restricts the operation to chars 'k', 'p', and 't'.)
Thanks,
Steve
-----Original Message-----
From: Tanguy Moal [mailto:tanguy.moal@gmail.com]
Sent: Wednesday, May 16, 2012 8:29 AM
To: solr-user@lucene.apache.org
Subject: Re: FrenchLightStemFilterFactory : normalizing tokens longer than 4 characters and having repeated characters in it
Any idea someone ?
I think this is important since this could produce weird results on collections with numbers mixed in text.
>From my understanding, there are a few options to address the issue :
1) Make *LightStemmer token type aware and don't try to stem on things that are not text (alpha/alphanum whatever :)) : this pulls the fix into the faulty component but makes it dependant on the StandardTokenizer which may not be what people want...
2) Enable StandardTokenizer to mark NUM tokens with the keyword attribute so that NUM tokens are not stemmed by contract (this could be a configuration flag markNumbersWithKeywordAttribute=true, false by default)
3) Use a custom processor to mark NUM tokens as keywords (the solution I chose since it doesn't require modifying lucene/solr's code base, it's a very simply contrib module)
I chose solution #3.
Maybe #2 is the way to go since most people using FrenchLightStemFilterFactory will also want to use StandardTokenizer...
Any advice is welcome
--
Tanguy
--
View this message in context: http://lucene.472066.n3.nabble.com/FrenchLightStemFilterFactory-normalizing-tokens-longer-than-4-characters-and-having-repeated-charactt-tp3974148p3984080.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: FrenchLightStemFilterFactory : normalizing tokens longer than 4
characters and having repeated characters in it
Posted by Tanguy Moal <ta...@gmail.com>.
Any idea someone ?
I think this is important since this could produce weird results on
collections with numbers mixed in text.
>From my understanding, there are a few options to address the issue :
1) Make *LightStemmer token type aware and don't try to stem on things that
are not text (alpha/alphanum whatever :)) : this pulls the fix into the
faulty component but makes it dependant on the StandardTokenizer which may
not be what people want...
2) Enable StandardTokenizer to mark NUM tokens with the keyword attribute so
that NUM tokens are not stemmed by contract (this could be a configuration
flag markNumbersWithKeywordAttribute=true, false by default)
3) Use a custom processor to mark NUM tokens as keywords (the solution I
chose since it doesn't require modifying lucene/solr's code base, it's a
very simply contrib module)
I chose solution #3.
Maybe #2 is the way to go since most people using
FrenchLightStemFilterFactory will also want to use StandardTokenizer...
Any advice is welcome
--
Tanguy
--
View this message in context: http://lucene.472066.n3.nabble.com/FrenchLightStemFilterFactory-normalizing-tokens-longer-than-4-characters-and-having-repeated-charactt-tp3974148p3984080.html
Sent from the Solr - User mailing list archive at Nabble.com.