[PR] OPENNLP-1905: Default-locale case folding breaks Morfologik dictionary lookup (opennlp)

["krickert (via GitHub)" <gi...@apache.org>]

krickert opened a new pull request, #1207:
URL: https://github.com/apache/opennlp/pull/1207

   Fixes OPENNLP-1905.
   
   Both Morfologik lookups fold the query word with the JVM default locale before searching a prebuilt FSA dictionary. Under `tr` or `az` the dotless i means a capitalised word containing `I` folds to something the dictionary does not contain, so the lookup misses and returns null. Nothing throws.
   
   `MorfologikTagDictionary` is the more consequential of the two, because it feeds POS tagging constraints. Its failure does not merely lose a lemma, it changes tagger output depending on the JVM locale the tagger runs under.
   
   The dictionary is an external prebuilt artifact with fixed content, so a user cannot work around this by editing their own data.
   
   ### Reproduction
   
   Assertion failures against unfixed code with `Locale.setDefault` of `tr`:
   
   ```
   MorfologikLemmatizer.lemmatize   expected <Illinois>   but was <null>
   MorfologikTagDictionary.getTags  actual array was <null>
   ```
   
   `"Illinois".toLowerCase()` under `tr` yields `ıllinois`, which is absent from an ASCII-lowercase FSA.
   
   ### Sites
   
   | Location | Expression |
   |---|---|
   | `MorfologikLemmatizer.java:66` | `lookup(word.toLowerCase())` |
   | `MorfologikTagDictionary.java:67` | `word = word.toLowerCase()` (case-insensitive mode) |
   
   ### Why `Locale.ROOT`
   
   Same reasoning as OPENNLP-1904. `StringUtil.toLowerCase` maps code points through `Character.toLowerCase`, which is context free and disagrees with `String.toLowerCase(Locale.ROOT)` on context-sensitive mappings such as Greek final sigma and on one-to-many mappings such as `ß`. Existing FSA dictionaries were built against the English-locale fold, which is byte-identical to `Locale.ROOT`, so this keeps them working unchanged. **No dictionary regeneration is required.**
   
   ### Tests
   
   `MorfologikLocaleTest` adds two tests, both verified to fail before the fix and pass after. New fixture `dictionaryLocaleSafety.{txt,info}`, plus a `createMorfologikDictionary(String)` overload on `AbstractMorfologikTest` (the existing no-arg method delegates to it) so the shared `dictionaryWithLemma` fixture is not perturbed.
   
   Full 23-module reactor: BUILD SUCCESS, zero failures, with checkstyle, rat and forbiddenapis active.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] OPENNLP-1905: Default-locale case folding breaks Morfologik dictionary lookup (opennlp)

["mawiesne (via GitHub)" <gi...@apache.org>]

mawiesne merged PR #1207:
URL: https://github.com/apache/opennlp/pull/1207


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] OPENNLP-1905: Default-locale case folding breaks Morfologik dictionary lookup (opennlp)

["krickert (via GitHub)" <gi...@apache.org>]

krickert commented on PR #1207:
URL: https://github.com/apache/opennlp/pull/1207#issuecomment-5204104783

   Trailing newline on `dictionaryLocaleSafety.info` added in b837cf74d.
   
   On forbiddenapis `jdk-unsafe-*`: agree that would catch this family. Separate follow-up once the remaining sites are cleaned up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opennlp.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org