You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2010/08/29 00:06:53 UTC
[jira] Resolved: (SOLR-2059) Allow customizing how
WordDelimiterFilter tokenizes text.
[ https://issues.apache.org/jira/browse/SOLR-2059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir resolved SOLR-2059.
-------------------------------
Resolution: Fixed
Committed revision 990451 (trunk) 990456 (3x)
> Allow customizing how WordDelimiterFilter tokenizes text.
> ---------------------------------------------------------
>
> Key: SOLR-2059
> URL: https://issues.apache.org/jira/browse/SOLR-2059
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Robert Muir
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: SOLR-2059.patch
>
>
> By default, WordDelimiterFilter assigns 'types' to each character (computed from Unicode Properties).
> Based on these types and the options provided, it splits and concatenates text.
> In some circumstances, you might need to tweak the behavior of how this works.
> It seems the filter already had this in mind, since you can pass in a custom byte[] type table.
> But its not exposed in the factory.
> I think you should be able to customize the defaults with a configuration file:
> {noformat}
> # A customized type mapping for WordDelimiterFilterFactory
> # the allowable types are: LOWER, UPPER, ALPHA, DIGIT, ALPHANUM, SUBWORD_DELIM
> #
> # the default for any character without a mapping is always computed from
> # Unicode character properties
> # Map the $, %, '.', and ',' characters to DIGIT
> # This might be useful for financial data.
> $ => DIGIT
> % => DIGIT
> . => DIGIT
> \u002C => DIGIT
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org