[jira] [Created] (OPENNLP-1984) Preserve hyphenated words by default in Arvores Deitadas readers and detokenize them losslessly

["Kristian Rickert (Jira)" <ji...@apache.org>]

Kristian Rickert created OPENNLP-1984:
-----------------------------------------

             Summary: Preserve hyphenated words by default in Arvores Deitadas readers and detokenize them losslessly
                 Key: OPENNLP-1984
                 URL: https://issues.apache.org/jira/browse/OPENNLP-1984
             Project: OpenNLP
          Issue Type: Task
            Reporter: Kristian Rickert
            Assignee: Kristian Rickert


Configure the Arvores Deitadas (AD) corpus readers to preserve hyphenated words as single tokens by default, and provide lossless detokenization for reconstructed samples.

h2. Details

* Set \{{splitHyphenatedTokens}} to default to \{{false}} in \{{ADNameSampleStreamFactory}}. The \{{-splitHyphenatedTokens}} CLI option remains available for splitting when needed.
* Add package-private \{{ADDetokenizer}} wrapping the configured detokenizer in \{{ADTokenSampleStreamFactory}}. This handles hyphen joining for clitics (such as \{{ofereceu-me}}) and ensures combining accents stay attached to base characters across precomposed and decomposed forms.
* Update CLI and corpora documentation in \{{opennlp-docs}} covering the \{{-splitHyphenatedTokens}} option and clitic detokenization rules.

h2. Compatibility

Training models with \{{-format ad}} without specifying \{{-splitHyphenatedTokens}} retains hyphenated compounds as single tokens. Callers requiring split hyphens can pass \{{-splitHyphenatedTokens true}}.

h2. Validation

* Verified via \{{ADHyphenatedTokenTest}}, covering hyphen preservation, explicit splitting, clitics, and combining marks.
* 10-fold cross-validation in \{{ArvoresDeitadasEval}} (\{{evalPortugueseTokenizerDefaultNaiveBayes}}) passes on FlorestaVirgem.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)