[jira] [Updated] (OPENNLP-1905) Default-locale case folding breaks Morfologik dictionary lookup

["Martin Wiesner (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OPENNLP-1905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martin Wiesner updated OPENNLP-1905:
------------------------------------
    Priority: Minor  (was: Major)

> Default-locale case folding breaks Morfologik dictionary lookup
> ---------------------------------------------------------------
>
>                 Key: OPENNLP-1905
>                 URL: https://issues.apache.org/jira/browse/OPENNLP-1905
>             Project: OpenNLP
>          Issue Type: Bug
>    Affects Versions: 3.0.0-M5
>            Reporter: Kristian Rickert
>            Assignee: Kristian Rickert
>            Priority: Minor
>             Fix For: 3.0.0-M6
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> h2. Summary
> Both Morfologik lookups fold the query word with the JVM default locale before
> searching a prebuilt FSA dictionary. Under {{tr}} or {{az}} the Turkish dotless i
> means a capitalised word containing {{I}} folds to something the dictionary does
> not contain, so the lookup misses. Nothing throws; lemmatization and POS tag
> constraints silently stop working.
> The dictionary is an external prebuilt artifact with fixed content, so a user
> cannot work around this by editing their dictionary.
> h2. Reproduction
> Assertion failures observed against unfixed code with {{Locale.setDefault}} of {{tr}}:
> {noformat}
> MorfologikLemmatizer.lemmatize   expected <Illinois>   but was <null>
> MorfologikTagDictionary.getTags  actual array was <null>
> {noformat}
> {{"Illinois".toLowerCase()}} under {{tr}} yields {{ıllinois}}, which is absent from
> an ASCII-lowercase FSA.
> h2. Affected sites
> || Location || Expression ||
> | {{opennlp-extensions/opennlp-morfologik/src/main/java/opennlp/morfologik/lemmatizer/MorfologikLemmatizer.java:66}} | {{lookup(word.toLowerCase())}} |
> | {{opennlp-extensions/opennlp-morfologik/src/main/java/opennlp/morfologik/tagdict/MorfologikTagDictionary.java:67}} | {{word = word.toLowerCase()}} (case-insensitive mode) |
> {{MorfologikTagDictionary}} is the more consequential of the two, because it feeds
> POS tagging constraints. Its failure does not merely lose a lemma, it changes the
> tagger's output depending on the JVM locale it runs under.
> h2. Fix
> Both folds become {{toLowerCase(Locale.ROOT)}}.
> {{Locale.ROOT}} rather than the project's own {{StringUtil.toLowerCase}}: the
> latter maps code points through {{Character.toLowerCase}}, which is context-free
> and disagrees with {{String.toLowerCase(Locale.ROOT)}} on context-sensitive
> mappings (Greek final sigma: {{ΟΔΟΣ}} folds to {{οδοσ}} rather than {{οδος}}) and
> on one-to-many mappings such as {{ß}}. Existing FSA dictionaries were built against
> the English-locale fold, which is byte-identical to {{Locale.ROOT}}, so this keeps
> them working unchanged. The code-point fold would silently alter lookups against
> already-published dictionaries.
> h2. Test coverage
> There was no locale test coverage anywhere in the project before this work. There
> is no {{Locale.setDefault}} in main or test sources.
> {{MorfologikLocaleTest}} adds two tests, both verified to fail before the fix and
> pass after (the fix was stashed, the build rerun, the failure observed, then
> restored). Failure messages are in the Reproduction section.
> New fixture {{dictionaryLocaleSafety.txt}} and {{dictionaryLocaleSafety.info}} in
> the morfologik test resources, plus a {{createMorfologikDictionary(String)}}
> overload on {{AbstractMorfologikTest}} (the existing no-arg method delegates to it)
> so the shared {{dictionaryWithLemma}} fixture is not perturbed.
> h2. Build
> {{mvn -o clean install}} over the full 23-module reactor: BUILD SUCCESS, zero test
> failures or errors, with checkstyle, rat and the existing forbiddenapis
> configuration active.
> h2. Note for the reviewer
> No dictionary regeneration is required. That is the point of choosing
> {{Locale.ROOT}} over the code-point fold. If a reviewer proposes switching to
> {{StringUtil.toLowerCase}} for consistency with the rest of the codebase, that
> would change lookup behaviour against every already-published FSA dictionary.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)