[jira] [Created] (OPENNLP-1841) Provide SymSpell spell-correction dictionary models (opennlp-models-spellcheck-{lang})

["Richard Zowalla (Jira)" <ji...@apache.org>]

Richard Zowalla created OPENNLP-1841:
----------------------------------------

             Summary: Provide SymSpell spell-correction dictionary models (opennlp-models-spellcheck-{lang})
                 Key: OPENNLP-1841
                 URL: https://issues.apache.org/jira/browse/OPENNLP-1841
             Project: OpenNLP
          Issue Type: New Feature
            Reporter: Richard Zowalla
             Fix For: 3.0.0-M4, 3.0.0


  OPENNLP-1832 landed the engine and API (opennlp-spellcheck module: SpellChecker, the SymSpell engine, SymSpellModel, serializer, model-resolver, CLI). What it deliberately did not ship is data: the module contains no production dictionaries, and SymSpellModelResolver resolves nothing on a stock classpath.

This issue covers producing, licensing, and publishing the actual dictionary model artifacts - one Maven jar per language, named opennlp-models-spellcheck-{lang} — so that new  SymSpellModelResolver().resolveByLanguage("en") returns a usable model out of the box, consistent with how every other OpenNLP component ships pre-trained models.

Proposed work

  1. Decide the home for the artifacts
  
I would recommend to produce them in apache/opennlp-models (same place other pre-trained models are released and where opennlp-models-* GAV coordinates live

2. Source MIT-/ASLv2-compatible frequency data

  For each target language, obtain a permissively-licensed unigram (and where available bigram) frequency list:
  - English: SymSpell reference frequency_dictionary_en_82_765.txt + frequency_bigramdictionary_en_243_342.txt (MIT) - gives us compound-correction (lookupCompound) coverage immediately.
  - Other languages: Hermit Dave's FrequencyWords (OpenSubtitles-derived, MIT) and/or Wortschatz/Leipzig corpora and/or Wikipedia Snapshots - license must be confirmed per language and recorded.
  - Every source's license, URL, and retrieval date go into NOTICE / LICENSE and the per-jar provenance, same discipline as OPENNLP-1832's data-licensing work.

  3. Initial language set (proposal)

  en (with bigrams) first as the reference; then de, es, fr, nl, it as a second wave, gated on confirmed licensing. 

For each language, run SpellCheckModelBuilder (default maxEditDistance=2, prefixLength=7) and package via writePackage(...) so each jar contains *.bin + model.properties. Pin and   record:
  - model.version (start at 1.0),
  - model.sha256 (computed by SymSpellModels over the binary),
  - the build parameters and source-data version, for reproducibility.

Can also be scripted like it is done for the other models and data.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)