You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@lucene.apache.org by "Tomoko Uchida (Jira)" <ji...@apache.org> on 2021/03/21 08:47:00 UTC

[jira] [Created] (LUCENE-9853) Use CJKWidthCharFilter as the default character normalizer for JapaneseAnalyzer instead of CJKWidthFilter

Tomoko Uchida created LUCENE-9853:
-------------------------------------

             Summary: Use CJKWidthCharFilter as the default character normalizer for JapaneseAnalyzer instead of CJKWidthFilter
                 Key: LUCENE-9853
                 URL: https://issues.apache.org/jira/browse/LUCENE-9853
             Project: Lucene - Core
          Issue Type: Improvement
          Components: modules/analysis
    Affects Versions: main (9.0)
            Reporter: Tomoko Uchida
            Assignee: Tomoko Uchida


Follow-up issue of LUCENE-9413.

We now have CJKWidthCharFilter in analyzers-common. I believe in many situations it is recommended applying character width normalization before tokenization for consistency in analysis.

The change slightly affects on the analyzer's outputs. We can provide a parameter to switch back to CJKWidthFilter for backward compatibility.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org