You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Uwe Schindler (JIRA)" <ji...@apache.org> on 2010/01/27 12:20:34 UTC
[jira] Resolved: (LUCENE-2198) support protected words in Stemming
TokenFilters
[ https://issues.apache.org/jira/browse/LUCENE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler resolved LUCENE-2198.
-----------------------------------
Resolution: Fixed
Fix Version/s: 3.1
Committed revision: 903608
Thanks Simon!
> support protected words in Stemming TokenFilters
> ------------------------------------------------
>
> Key: LUCENE-2198
> URL: https://issues.apache.org/jira/browse/LUCENE-2198
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.0
> Reporter: Robert Muir
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2198.patch, LUCENE-2198.patch
>
>
> This is from LUCENE-1515
> I propose that all stemming TokenFilters have an 'exclusion set' that bypasses any stemming for words in this set.
> Some stemming tokenfilters have this, some do not.
> This would be one way for Karl to implement his new swedish stemmer (as a text file of ignore words).
> Additionally, it would remove duplication between lucene and solr, as they reimplement snowballfilter since it does not have this functionality.
> Finally, I think this is a pretty common use case, where people want to ignore things like proper nouns in the stemming.
> As an alternative design I considered a case where we generalized this to CharArrayMap (and ignoring words would mean mapping them to themselves), which would also provide a mechanism to override the stemming algorithm. But I think this is too expert, could be its own filter, and the only example of this i can find is in the Dutch stemmer.
> So I think we should just provide ignore with CharArraySet, but if you feel otherwise please comment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org