[jira] [Commented] (LUCENE-5437) ASCIIFoldingFilter that emits both unfolded and folded tokens

["Ahmet Arslan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-5437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000274#comment-14000274 ] 

Ahmet Arslan commented on LUCENE-5437:
--------------------------------------

ASCIIFoldingFilter's preserveOriginal=true breaks wildcard queries, since it is a MultiTermAwareComponent.

{code:xml}
  <fieldType name="text_ascii_preserve" class="solr.TextField" positionIncrementGap="100">
      <analyzer>
        <tokenizer class="solr.StandardTokenizerFactory"/>            
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.ASCIIFoldingFilterFactory" preserveOriginal="true"/>
      </analyzer>      
    </fieldType>
{code}

With the above type, {{q=manu:Belkı*}} yields: 

{code}
 "error": {
    "msg": "analyzer returned too many terms for multiTerm term: Belkı",
    "code": 400
  } 
{code}

I think preserveOriginal is dangerous for token filters that implement MultiTermAwareComponent. 
What is the preferred action here? Document this limitation/behavior? Or consider this as a bug and open a jira?

> ASCIIFoldingFilter that emits both unfolded and folded tokens
> -------------------------------------------------------------
>
>                 Key: LUCENE-5437
>                 URL: https://issues.apache.org/jira/browse/LUCENE-5437
>             Project: Lucene - Core
>          Issue Type: Improvement
>          Components: modules/analysis
>    Affects Versions: 4.7, 5.0
>            Reporter: Nik Everett
>            Assignee: Simon Willnauer
>            Priority: Minor
>             Fix For: 4.7, 5.0
>
>         Attachments: LUCENE-5437.patch
>
>
> I've found myself wanting an ASCIIFoldingFilter that emits both the folded tokens and the original, unfolded tokens.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org