[jira] Commented: (SOLR-1653) add PatternReplaceCharFilter

["Paul taylor (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797601#action_12797601 ] 

Paul taylor commented on SOLR-1653:
-----------------------------------

Hi, Im using in non Solr in an analyser, and think there maybe a performance issue because you cannot pass a compiled Pattern. In the reusableTokenStream() method you cannot reset a charfilter like you can a tokenizer so it as to recompile the pattern everytime 

i.e. 
 public TokenStream reusableTokenStream(String fieldName, Reader reader) throws IOException {
        SavedStreams streams = (SavedStreams)getPreviousTokenStream();
        if (streams == null) {
            streams = new SavedStreams();
            setPreviousTokenStream(streams);
            streams.tokenStream = new StandardTokenizer(Version.LUCENE_CURRENT,new PatternReplaceCharFilter("(no\\.) ([0-9]+)","$1$2,reader));
            streams.filteredTokenStream = new StandardFilter(streams.filteredTokenStream);
            streams.filteredTokenStream = new AccentFilter(streams.filteredTokenStream);
            streams.filteredTokenStream = new LowercaseFilter(streams.filteredTokenStream);
        }
        else {
            streams.tokenStream.reset(new PatternReplaceCharFilter("(no\\.) ([0-9]+)","$1$2",reader));
        }
        return streams.filteredTokenStream;
    }

> add PatternReplaceCharFilter
> ----------------------------
>
>                 Key: SOLR-1653
>                 URL: https://issues.apache.org/jira/browse/SOLR-1653
>             Project: Solr
>          Issue Type: New Feature
>          Components: Schema and Analysis
>    Affects Versions: 1.4
>            Reporter: Koji Sekiguchi
>            Assignee: Koji Sekiguchi
>            Priority: Minor
>             Fix For: 1.5
>
>         Attachments: SOLR-1653.patch, SOLR-1653.patch
>
>
> Add a new CharFilter that uses a regular expression for the target of replace string in char stream.
> Usage:
> {code:title=schema.xml}
> <fieldType name="textCharNorm" class="solr.TextField" positionIncrementGap="100" >
>   <analyzer>
>     <charFilter class="solr.PatternReplaceCharFilterFactory"
>                 groupedPattern="([nN][oO]\.)\s*(\d+)"
>                 replaceGroups="1,2" blockDelimiters=":;"/>
>     <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt"/>
>     <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>   </analyzer>
> </fieldType>
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.